Repository: kaito-project/kaito

Issue:
Update transformer library in base image - QWEN 3 New architecture.
**Describe the bug**
https://huggingface.co/Qwen/Qwen3-8B. Architecture has been updated (See error) would be great to update image transformer library to allow for QWEN3 deployment.

**Steps To Reproduce**
Deploy following YAML using qwen3/qwen3-8B model ID:
```
apiVersion: v1
kind: Namespace
metadata:
  name: qwen4
--- 
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: qwen4
  name: my-inference-params
data:
  inference_config.yaml: |
    max_probe_steps: 6
    vllm:
      gpu-memory-utilization: 0.95  # Controls GPU memory usage (0.0-1.0)
      tensor-parallel-size: 2        # Number of GPUs for tensor parallelism
      max-model-len: 11072         # Maximum sequence length
      swap-space: 4                 # CPU swap space in GB
      cpu-offload-gb: 0             # Amount of GPU memory to offload to CPU
--- 
apiVersion: kaito.sh/v1beta1
kind: Workspace
metadata:
  namespace: qwen4
  name: workspace-custom-llm
resource:
  instanceType: "Standard_NC24ads_A100_v4"   # Replace with the required VM SKU based on model requirements
  labelSelector:
    matchLabels:
      apps: custom-llm
#  preferredNodes:
#    - <PREFERRED_NODE_HERE>
inference:
  config: "my-inference-params"
  template:
    spec:
      containers:
        - name: custom-llm-container
          image: ghcr.io/kaito-project/kaito/llm-reference-preset:latest # KAITO Reference Image
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 5000
              scheme: HTTP
            initialDelaySeconds: 600
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 5000
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            requests:
              nvidia.com/gpu: 1  # Request 1 GPU; adjust as needed
            limits:
              nvidia.com/gpu: 1  # Optional: Limit to 1 GPU
          command:
            - "accelerate"
          args:
            - "launch"
            - "--num_processes"
            - "1"
            - "--num_machines"
            - "1"
            - "--gpu_ids"
            - "all"
            - "inference_api.py"
            - "--pipeline"
            - "text-generation"
            - "--trust_remote_code"
            - "--allow_remote_files"
            - "--pretrained_model_name_or_path"
            - "Qwen/Qwen3-8B"  # Replace <MODEL_ID> with the specific HuggingFace model identifier
            - "--torch_dtype"
            - "bfloat16"  # Set to "float16" for compatibility with V100 GPUs; use "bfloat16" for A100, H100 or newer GPUs
          # env:
          #   HF_TOKEN is required only for private or gated Hugging Face models
          #   Uncomment and configure this block if needed
          #   - name: HF_TOKEN
          #     valueFrom:
          #       secretKeyRef:
          #         name: hf-token-secret     # Replace with your Kubernetes Secret name
          #         key: HF_TOKEN             # Replace with the specific key holding the token
          volumeMounts:
            - name: dshm
              mountPath: /dev/shm
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
 ```

**Expected behavior**
Can deploy QWEN 3 Model
**Logs**
```uring handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/tfs/inference_api.py", line 127, in <module>
    base_model = AutoModelForCausalLM.from_pretrained(**model_args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1094, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `qwen3` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/usr/local/lib/python3.12/site-packages/accelerate/commands/launch.py", line 1168, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.12/site-packages/accelerate/commands/launch.py", line 763, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python3.12', 'inference_api.py', '--pipeline', 'text-generation', '--trust_remote_code', '--allow_remote_files', '--pretrained_model_name_or_path', 'Qwen/Qwen3-8B', '--torch_dtype', 'bfloat16']' returned non-zero exit status 1.
```

**Environment**

- Kubernetes version (use `kubectl version`): 1.32.7


**Additional context**


Hints from issue comments:
@kaito-pr-agent: ### :bulb: Auto-generated documentation-based answer:
KAITO (Kubernetes AI Toolchain Operator) is an open-source Kubernetes operator that automates the deployment and management of AI/ML model inference and tuning workloads on Kubernetes clusters. It specializes in simplifying the deployment of large open-source models like Phi, Llama, and Mistral by:

1. Managing model files using container images with built-in OpenAI-compatible servers
2. Providing preset configurations optimized for different GPU hardware
3. Supporting popular inference runtimes like vLLM and HuggingFace Transformers
4. Auto-provisioning GPU nodes based on model requirements
5. Hosting model images in Microsoft Container Registry (MCR) when licenses permit

KAITO follows the Kubernetes CRD/controller pattern where users define `Workspace` custom resources describing GPU requirements and inference specifications. Key components include:
- Workspace Controller: Manages node provisioning and workload deployment
- GPU Provisioner: Integrates with cloud APIs to add GPU nodes
- RAGEngine (new): Supports Retrieval Augmented Generation workflows

The architecture enables simplified onboarding of large AI models to Kubernetes while optimizing resource utilization through features like distributed inference and auto-scaling.

#### Relevant Sources:

> - https://github.com/kaito-project/kaito/blob/main/README.md
> - https://github.com/kaito-project/kaito/blob/main/docs/proposals/20250630-kaito-cli.md
> - https://github.com/kaito-project/kaito/blob/main/docs/proposals/20250325-distributed-inference.md


Solution patch:
diff --git a/Makefile b/Makefile
index 7c0862a..5b380c6 100644
--- a/Makefile
+++ b/Makefile
@@ -361,16 +361,6 @@ docker-build-dataset: docker-buildx ## Build Docker images for datasets.
 		--pull \
 		--tag $(REGISTRY)/e2e-dataset2:0.0.1 .
 
-.PHONY: docker-build-llm-reference-preset
-docker-build-llm-reference-preset: docker-buildx ## Build Docker image for LLM reference preset.
-	docker buildx build \
-		-t ghcr.io/kaito-repo/kaito/llm-reference-preset:$(VERSION) \
-		-t ghcr.io/kaito-repo/kaito/llm-reference-preset:latest \
-		-f docs/custom-model-integration/Dockerfile.reference \
-		$(BUILD_FLAGS) \
-		--build-arg MODEL_TYPE=text-generation \
-		--build-arg VERSION=$(VERSION) .
-
 ## --------------------------------------
 ## KAITO Installation
 ## --------------------------------------
diff --git a/api/v1alpha1/condition_types.go b/api/v1alpha1/condition_types.go
index 1cca03c..7bc98b2 100644
--- a/api/v1alpha1/condition_types.go
+++ b/api/v1alpha1/condition_types.go
@@ -41,6 +41,12 @@ const (
 	//WorkspaceConditionTypeDeleting is the Workspace state when starts to get deleted.
 	WorkspaceConditionTypeDeleting = ConditionType("WorkspaceDeleting")
 
+	//InferenceSetConditionTypeDeleting is the InferenceSet state when starts to get deleted.
+	InferenceSetConditionTypeDeleting = ConditionType("InferenceSetDeleting")
+
+	// InferenceSetConditionTypeReady is the InferenceSet state when starts to get ready.
+	InferenceSetConditionTypeReady = ConditionType("InferenceSetReady")
+
 	//WorkspaceConditionTypeSucceeded is the Workspace state that summarizes all operations' states.
 	//For inference, the "True" condition means the inference service is ready to serve requests.
 	//For fine tuning, the "True" condition means the tuning job completes successfully.
diff --git a/api/v1alpha1/inferenceset_default.go b/api/v1alpha1/inferenceset_default.go
new file mode 100644
index 0000000..c13f056
--- /dev/null
+++ b/api/v1alpha1/inferenceset_default.go
@@ -0,0 +1,22 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package v1alpha1
+
+import (
+	"context"
+)
+
+// SetDefaults for the InferenceSet
+func (w *InferenceSet) SetDefaults(_ context.Context) {
+}
diff --git a/api/v1alpha1/inferenceset_types.go b/api/v1alpha1/inferenceset_types.go
new file mode 100644
index 0000000..b69db71
--- /dev/null
+++ b/api/v1alpha1/inferenceset_types.go
@@ -0,0 +1,102 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package v1alpha1
+
+import (
+	appsv1 "k8s.io/api/apps/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
+)
+
+type InferenceSetResourceSpec struct {
+	// InstanceType specifies the GPU node SKU.
+	// +required
+	InstanceType string `json:"instanceType"`
+}
+
+// InferenceSetTemplate defines the template for creating InferenceSet instances.
+type InferenceSetTemplate struct {
+	Resource  InferenceSetResourceSpec   `json:"resource"`
+	Inference kaitov1beta1.InferenceSpec `json:"inference"`
+}
+
+// InferenceSetSpec defines the desired state of InferenceSet
+type InferenceSetSpec struct {
+	// Template is the template used to create the InferenceSet.
+	Template InferenceSetTemplate `json:"template"`
+	// Replicas is the desired number of workspaces to be created.
+	// +required
+	Replicas int `json:"replicas,omitempty"`
+	// NodeCountLimit is the maximum number of GPU nodes that can be created for the InferenceSet.
+	// If not specified, there is no limit on the number of GPU nodes that can be created.
+	// +optional
+	NodeCountLimit int `json:"nodeCountLimit,omitempty"`
+	// workspace created by InferenceSet controller would use this label in resource.labelSelector
+	// +required
+	Selector *metav1.LabelSelector `json:"labelSelector"`
+	// UpdateStrategy indicates the strategy to use to replace existing workspaces with new ones.
+	// +optional
+	// +kubebuilder:default:={"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":1}}
+	// +kubebuilder:validation:Schemaless
+	// +kubebuilder:pruning:PreserveUnknownFields
+	UpdateStrategy appsv1.StatefulSetUpdateStrategy `json:"updateStrategy,omitempty"`
+}
+
+// InferenceSetStatus defines the observed state of InferenceSet
+type InferenceSetStatus struct {
+	// Replicas is the total number of workspaces created by the InferenceSet.
+	Replicas int `json:"replicas,omitempty"`
+	// ReadyReplicas is the number of workspaces that are in ready state.
+	ReadyReplicas int `json:"readyReplicas,omitempty"`
+	// Selector is used to select the pods that provide metrics for making scaling action decisions.
+	// This field must be set when HPA and VPA is used for scaling.
+	Selector string `json:"selector,omitempty"`
+	// Conditions report the current conditions of the InferenceSet.
+	// +optional
+	Conditions []metav1.Condition `json:"conditions,omitempty"`
+}
+
+// InferenceSet is the Schema for the InferenceSet API
+// +kubebuilder:object:root=true
+// +kubebuilder:subresource:status
+// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.selector
+// +kubebuilder:resource:path=inferencesets,scope=Namespaced,categories=inferenceset,shortName={is,isets}
+// +kubebuilder:storageversion
+// +kubebuilder:printcolumn:name="Replicas",type="integer",JSONPath=".spec.replicas",description=""
+// +kubebuilder:printcolumn:name="ReadyReplicas",type="integer",JSONPath=".status.readyReplicas",description=""
+// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp",description=""
+type InferenceSet struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	// Spec defines the desired state of InferenceSet
+	// +required
+	Spec InferenceSetSpec `json:"spec"`
+
+	// Status defines the observed state of InferenceSet
+	Status InferenceSetStatus `json:"status,omitempty"`
+}
+
+// InferenceSetList contains a list of InferenceSet
+// +kubebuilder:object:root=true
+type InferenceSetList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata,omitempty"`
+	Items           []InferenceSet `json:"items"`
+}
+
+func init() {
+	SchemeBuilder.Register(&InferenceSet{}, &InferenceSetList{})
+}
diff --git a/api/v1alpha1/inferenceset_validation.go b/api/v1alpha1/inferenceset_validation.go
new file mode 100644
index 0000000..3f45c68
--- /dev/null
+++ b/api/v1alpha1/inferenceset_validation.go
@@ -0,0 +1,59 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package v1alpha1
+
+import (
+	"context"
+	"fmt"
+	"strings"
+
+	admissionregistrationv1 "k8s.io/api/admissionregistration/v1"
+	"k8s.io/apimachinery/pkg/util/validation"
+	"k8s.io/klog/v2"
+	"knative.dev/pkg/apis"
+)
+
+func (is *InferenceSet) SupportedVerbs() []admissionregistrationv1.OperationType {
+	return []admissionregistrationv1.OperationType{
+		admissionregistrationv1.Create,
+		admissionregistrationv1.Update,
+	}
+}
+
+func (is *InferenceSet) Validate(ctx context.Context) (errs *apis.FieldError) {
+	errmsgs := validation.IsDNS1123Label(is.Name)
+	if len(errmsgs) > 0 {
+		errs = errs.Also(apis.ErrInvalidValue(strings.Join(errmsgs, ", "), "name"))
+	}
+	base := apis.GetBaseline(ctx)
+	if base == nil {
+		klog.InfoS("Validate creation", "inferenceset", fmt.Sprintf("%s/%s", is.Namespace, is.Name))
+		errs = errs.Also(is.validateCreate().ViaField("spec"))
+	} else {
+		klog.InfoS("Validate update", "inferenceset", fmt.Sprintf("%s/%s", is.Namespace, is.Name))
+		old := base.(*InferenceSet)
+		errs = errs.Also(
+			is.validateUpdate(old).ViaField("spec"),
+		)
+	}
+	return errs
+}
+
+func (is *InferenceSet) validateCreate() (errs *apis.FieldError) {
+	return errs
+}
+
+func (is *InferenceSet) validateUpdate(_ *InferenceSet) (errs *apis.FieldError) {
+	return errs
+}
diff --git a/api/v1alpha1/labels.go b/api/v1alpha1/labels.go
index 9120a93..6525ef3 100644
--- a/api/v1alpha1/labels.go
+++ b/api/v1alpha1/labels.go
@@ -47,6 +47,9 @@ const (
 	// RAGEngineRevisionAnnotation is the Annotations for revision number
 	RAGEngineRevisionAnnotation = "ragengine.kaito.io/revision"
 
+	// InferenceSetRevisionAnnotation is the Annotations for revision number
+	InferenceSetRevisionAnnotation = "inferenceset.kaito.io/revision"
+
 	// AnnotationWorkspaceRuntime is the annotation for runtime selection.
 	AnnotationWorkspaceRuntime = KAITOPrefix + "runtime"
 
diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go
index 3b04644..de4f596 100644
--- a/api/v1alpha1/zz_generated.deepcopy.go
+++ b/api/v1alpha1/zz_generated.deepcopy.go
@@ -1,3 +1,5 @@
+//go:build !ignore_autogenerated
+
 // Copyright (c) KAITO authors.
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -174,6 +176,141 @@ func (in *InferenceServiceSpec) DeepCopy() *InferenceServiceSpec {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *InferenceSet) DeepCopyInto(out *InferenceSet) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	in.Status.DeepCopyInto(&out.Status)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new InferenceSet.
+func (in *InferenceSet) DeepCopy() *InferenceSet {
+	if in == nil {
+		return nil
+	}
+	out := new(InferenceSet)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *InferenceSet) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *InferenceSetList) DeepCopyInto(out *InferenceSetList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]InferenceSet, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new InferenceSetList.
+func (in *InferenceSetList) DeepCopy() *InferenceSetList {
+	if in == nil {
+		return nil
+	}
+	out := new(InferenceSetList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *InferenceSetList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *InferenceSetResourceSpec) DeepCopyInto(out *InferenceSetResourceSpec) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new InferenceSetResourceSpec.
+func (in *InferenceSetResourceSpec) DeepCopy() *InferenceSetResourceSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(InferenceSetResourceSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *InferenceSetSpec) DeepCopyInto(out *InferenceSetSpec) {
+	*out = *in
+	in.Template.DeepCopyInto(&out.Template)
+	if in.Selector != nil {
+		in, out := &in.Selector, &out.Selector
+		*out = new(v1.LabelSelector)
+		(*in).DeepCopyInto(*out)
+	}
+	in.UpdateStrategy.DeepCopyInto(&out.UpdateStrategy)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new InferenceSetSpec.
+func (in *InferenceSetSpec) DeepCopy() *InferenceSetSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(InferenceSetSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *InferenceSetStatus) DeepCopyInto(out *InferenceSetStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]v1.Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new InferenceSetStatus.
+func (in *InferenceSetStatus) DeepCopy() *InferenceSetStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(InferenceSetStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *InferenceSetTemplate) DeepCopyInto(out *InferenceSetTemplate) {
+	*out = *in
+	out.Resource = in.Resource
+	in.Inference.DeepCopyInto(&out.Inference)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new InferenceSetTemplate.
+func (in *InferenceSetTemplate) DeepCopy() *InferenceSetTemplate {
+	if in == nil {
+		return nil
+	}
+	out := new(InferenceSetTemplate)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *InferenceSpec) DeepCopyInto(out *InferenceSpec) {
 	*out = *in
diff --git a/charts/kaito/workspace/templates/inferenceset_clusterrole.yaml b/charts/kaito/workspace/templates/inferenceset_clusterrole.yaml
new file mode 100644
index 0000000..241000c
--- /dev/null
+++ b/charts/kaito/workspace/templates/inferenceset_clusterrole.yaml
@@ -0,0 +1,33 @@
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: {{ include "kaito.fullname" . }}-inferenceset-clusterrole
+  labels:
+    {{- include "kaito.labels" . | nindent 4 }}
+rules:
+  - apiGroups: ["kaito.sh"]
+    resources: ["workspaces"]
+    verbs: ["update", "patch","get","list","watch", "create", "delete"]
+  - apiGroups: ["kaito.sh"]
+    resources: ["workspaces/status"]
+    verbs: ["update", "patch","get","list","watch"]
+  - apiGroups: ["kaito.sh"]
+    resources: ["inferencesets"]
+    verbs: ["update", "patch","get","list","watch"]
+  - apiGroups: ["kaito.sh"]
+    resources: ["inferencesets/status"]
+    verbs: ["update", "patch","get","list","watch"]
+  - apiGroups: [ "apps" ]
+    resources: ["controllerrevisions" ]
+    verbs: [ "get","list","watch","create", "delete","update", "patch"]
+  - apiGroups: ["admissionregistration.k8s.io"]
+    resources: ["validatingwebhookconfigurations"]
+    verbs: ["get","list","watch"]
+  - apiGroups: ["admissionregistration.k8s.io"]
+    resources: ["validatingwebhookconfigurations"]
+    verbs: ["update"]
+    resourceNames: ["validation.inferenceset.kaito.sh"]
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["create", "patch", "update"]
diff --git a/charts/kaito/workspace/templates/inferenceset_clusterrole_binding.yaml b/charts/kaito/workspace/templates/inferenceset_clusterrole_binding.yaml
new file mode 100644
index 0000000..8e6c277
--- /dev/null
+++ b/charts/kaito/workspace/templates/inferenceset_clusterrole_binding.yaml
@@ -0,0 +1,14 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: {{ include "kaito.fullname" . }}-inferenceset-rolebinding
+  labels:
+   {{- include "kaito.labels" . | nindent 4 }}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: {{ include "kaito.fullname" . }}-inferenceset-clusterrole
+subjects:
+- kind: ServiceAccount
+  name: {{ include "kaito.fullname" . }}-sa
+  namespace: {{ .Release.Namespace }}
diff --git a/charts/kaito/workspace/templates/webhooks.yaml b/charts/kaito/workspace/templates/webhooks.yaml
index 478352a..3e2b909 100644
--- a/charts/kaito/workspace/templates/webhooks.yaml
+++ b/charts/kaito/workspace/templates/webhooks.yaml
@@ -25,3 +25,30 @@ webhooks:
         operations:
           - CREATE
           - UPDATE
+---
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingWebhookConfiguration
+metadata:
+  name: validation.inferenceset.kaito.sh
+  labels:
+    {{- include "kaito.labels" . | nindent 4 }}
+webhooks:
+  - name: validation.inferenceset.kaito.sh
+    admissionReviewVersions: ["v1"]
+    clientConfig:
+      service:
+        name: {{ include "kaito.fullname" . }}-svc
+        namespace: {{ .Release.Namespace }}
+        port: {{ .Values.webhook.port }}
+    failurePolicy: Fail
+    sideEffects: None
+    rules:
+      - apiGroups:
+          - kaito.sh
+        apiVersions:
+          - v1alpha1
+        resources:
+          - inferencesets
+        operations:
+          - CREATE
+          - UPDATE
diff --git a/cmd/workspace/main.go b/cmd/workspace/main.go
index 9eb255a..dabff11 100644
--- a/cmd/workspace/main.go
+++ b/cmd/workspace/main.go
@@ -42,12 +42,14 @@ import (
 
 	// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
 	// to ensure that exec-entrypoint and run can make use of them.
+	kaitov1alpha1 "github.com/kaito-project/kaito/api/v1alpha1"
 	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
 	"github.com/kaito-project/kaito/pkg/featuregates"
 	"github.com/kaito-project/kaito/pkg/k8sclient"
 	kaitoutils "github.com/kaito-project/kaito/pkg/utils"
 	"github.com/kaito-project/kaito/pkg/workspace/controllers"
 	"github.com/kaito-project/kaito/pkg/workspace/controllers/garbagecollect"
+	"github.com/kaito-project/kaito/pkg/workspace/controllers/inferenceset"
 	"github.com/kaito-project/kaito/pkg/workspace/webhooks"
 )
 
@@ -67,6 +69,7 @@ var (
 
 func init() {
 	utilruntime.Must(clientgoscheme.AddToScheme(scheme))
+	utilruntime.Must(kaitov1alpha1.AddToScheme(scheme))
 	utilruntime.Must(kaitov1beta1.AddToScheme(scheme))
 	utilruntime.Must(kaitoutils.KarpenterSchemeBuilder.AddToScheme(scheme))
 	utilruntime.Must(azurev1beta1.SchemeBuilder.AddToScheme(scheme))
@@ -150,6 +153,18 @@ func main() {
 		exitWithErrorFunc()
 	}
 
+	inferenceSetReconciler := inferenceset.NewInferenceSetReconciler(
+		kClient,
+		mgr.GetScheme(),
+		log.Log.WithName("controllers").WithName("InferenceSet"),
+		mgr.GetEventRecorderFor("KAITO-InferenceSet-controller"),
+	)
+
+	if err = inferenceSetReconciler.SetupWithManager(mgr); err != nil {
+		klog.ErrorS(err, "unable to create controller", "controller", "InferenceSet")
+		exitWithErrorFunc()
+	}
+
 	pvGCReconciler := garbagecollect.NewPersistentVolumeGCReconciler(
 		kClient,
 		mgr.GetEventRecorderFor("KAITO-PersistentVolumeGC-controller"),
@@ -184,7 +199,7 @@ func main() {
 		})
 		ctx = sharedmain.WithHealthProbesDisabled(ctx)
 		ctx = sharedmain.WithHADisabled(ctx)
-		go sharedmain.MainWithConfig(ctx, "webhook", ctrl.GetConfigOrDie(), webhooks.NewWorkspaceWebhooks()...)
+		go sharedmain.MainWithConfig(ctx, "webhook", ctrl.GetConfigOrDie(), webhooks.NewControllerWebhooks()...)
 
 		// wait 2 seconds to allow reconciling webhookconfiguration and service endpoint.
 		time.Sleep(2 * time.Second)
diff --git a/config/crd/bases/kaito.sh_inferencesets.yaml b/config/crd/bases/kaito.sh_inferencesets.yaml
new file mode 100644
index 0000000..f55c192
--- /dev/null
+++ b/config/crd/bases/kaito.sh_inferencesets.yaml
@@ -0,0 +1,336 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.15.0
+  name: inferencesets.kaito.sh
+spec:
+  group: kaito.sh
+  names:
+    categories:
+    - inferenceset
+    kind: InferenceSet
+    listKind: InferenceSetList
+    plural: inferencesets
+    shortNames:
+    - is
+    - isets
+    singular: inferenceset
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.replicas
+      name: Replicas
+      type: integer
+    - jsonPath: .status.readyReplicas
+      name: ReadyReplicas
+      type: integer
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: InferenceSet is the Schema for the InferenceSet API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Spec defines the desired state of InferenceSet
+            properties:
+              labelSelector:
+                description: workspace created by InferenceSet controller would use
+                  this label in resource.labelSelector
+                properties:
+                  matchExpressions:
+                    description: matchExpressions is a list of label selector requirements.
+                      The requirements are ANDed.
+                    items:
+                      description: |-
+                        A label selector requirement is a selector that contains values, a key, and an operator that
+                        relates the key and values.
+                      properties:
+                        key:
+                          description: key is the label key that the selector applies
+                            to.
+                          type: string
+                        operator:
+                          description: |-
+                            operator represents a key's relationship to a set of values.
+                            Valid operators are In, NotIn, Exists and DoesNotExist.
+                          type: string
+                        values:
+                          description: |-
+                            values is an array of string values. If the operator is In or NotIn,
+                            the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                            the values array must be empty. This array is replaced during a strategic
+                            merge patch.
+                          items:
+                            type: string
+                          type: array
+                          x-kubernetes-list-type: atomic
+                      required:
+                      - key
+                      - operator
+                      type: object
+                    type: array
+                    x-kubernetes-list-type: atomic
+                  matchLabels:
+                    additionalProperties:
+                      type: string
+                    description: |-
+                      matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                      map is equivalent to an element of matchExpressions, whose key field is "key", the
+                      operator is "In", and the values array contains only "value". The requirements are ANDed.
+                    type: object
+                type: object
+                x-kubernetes-map-type: atomic
+              nodeCountLimit:
+                description: |-
+                  NodeCountLimit is the maximum number of GPU nodes that can be created for the InferenceSet.
+                  If not specified, there is no limit on the number of GPU nodes that can be created.
+                type: integer
+              replicas:
+                description: Replicas is the desired number of workspaces to be created.
+                type: integer
+              template:
+                description: Template is the template used to create the InferenceSet.
+                properties:
+                  inference:
+                    properties:
+                      adapters:
+                        description: |-
+                          Adapters are integrated into the base model for inference.
+                          Users can specify multiple adapters for the model and the respective weight of using each of them.
+                        items:
+                          properties:
+                            source:
+                              description: Source describes where to obtain the adapter
+                                data.
+                              properties:
+                                image:
+                                  description: |-
+                                    The name of the image that contains the source data. The assumption is that the source data locates in the
+                                    `data` directory in the image.
+                                  type: string
+                                imagePullSecrets:
+                                  description: ImagePullSecrets is a list of secret
+                                    names in the same namespace used for pulling the
+                                    data image.
+                                  items:
+                                    type: string
+                                  type: array
+                                name:
+                                  description: |-
+                                    The name of the dataset. The same name will be used as a container name.
+                                    It must be a valid DNS subdomain value,
+                                  type: string
+                                urls:
+                                  description: URLs specifies the links to the public
+                                    data sources. E.g., files in a public github repository.
+                                  items:
+                                    type: string
+                                  type: array
+                                volumeSource:
+                                  description: The mounted volume that contains the
+                                    data.
+                                  x-kubernetes-preserve-unknown-fields: true
+                              type: object
+                            strength:
+                              description: |-
+                                Strength specifies the default multiplier for applying the adapter weights to the raw model weights.
+                                It is usually a float number between 0 and 1. It is defined as a string type to be language agnostic.
+                              type: string
+                          type: object
+                        type: array
+                      config:
+                        description: |-
+                          Config specifies the name of a custom ConfigMap that contains inference arguments.
+                          If specified, the ConfigMap must be in the same namespace as the Workspace custom resource.
+                        type: string
+                      preset:
+                        description: Preset describes the base model that will be
+                          deployed with preset configurations.
+                        properties:
+                          accessMode:
+                            default: public
+                            description: |-
+                              Deprecated: This field is deprecated in v1beta1 and will be removed in a future version.
+                              AccessMode specifies whether the containerized model image is accessible via public registry
+                              or private registry. This field defaults to "public" if not specified.
+                              If this field is "private", user needs to provide the private image information in PresetOptions.
+                            enum:
+                            - public
+                            - private
+                            type: string
+                          name:
+                            description: Name of the supported models with preset
+                              configurations.
+                            type: string
+                          presetOptions:
+                            properties:
+                              image:
+                                description: |-
+                                  Deprecated: This field is deprecated in v1beta1 and will be removed in a future version.
+                                  Image is the name of the containerized model image.
+                                type: string
+                              imagePullSecrets:
+                                description: |-
+                                  Deprecated: This field is deprecated in v1beta1 and will be removed in a future version.
+                                  ImagePullSecrets is a list of secret names in the same namespace used for pulling the model image.
+                                items:
+                                  type: string
+                                type: array
+                              modelAccessSecret:
+                                description: ModelAccessSecret is the name of the
+                                  secret that contains the huggingface access token.
+                                type: string
+                            type: object
+                        required:
+                        - name
+                        type: object
+                      template:
+                        description: |-
+                          Template specifies the Pod template used to run the inference service. Users can specify custom Pod settings
+                          if the preset configurations cannot meet the requirements. Note that if Preset is specified, Template should not
+                          be specified and vice versa.
+                        x-kubernetes-preserve-unknown-fields: true
+                    type: object
+                  resource:
+                    properties:
+                      instanceType:
+                        description: InstanceType specifies the GPU node SKU.
+                        type: string
+                    required:
+                    - instanceType
+                    type: object
+                required:
+                - inference
+                - resource
+                type: object
+              updateStrategy:
+                default:
+                  rollingUpdate:
+                    maxUnavailable: 1
+                  type: RollingUpdate
+                description: UpdateStrategy indicates the strategy to use to replace
+                  existing workspaces with new ones.
+                x-kubernetes-preserve-unknown-fields: true
+            required:
+            - labelSelector
+            - template
+            type: object
+          status:
+            description: Status defines the observed state of InferenceSet
+            properties:
+              conditions:
+                description: Conditions report the current conditions of the InferenceSet.
+                items:
+                  description: "Condition contains details for one aspect of the current
+                    state of this API Resource.\n---\nThis struct is intended for
+                    direct use as an array at the field path .status.conditions.  For
+                    example,\n\n\n\ttype FooStatus struct{\n\t    // Represents the
+                    observations of a foo's current state.\n\t    // Known .status.conditions.type
+                    are: \"Available\", \"Progressing\", and \"Degraded\"\n\t    //
+                    +patchMergeKey=type\n\t    // +patchStrategy=merge\n\t    // +listType=map\n\t
+                    \   // +listMapKey=type\n\t    Conditions []metav1.Condition `json:\"conditions,omitempty\"
+                    patchStrategy:\"merge\" patchMergeKey:\"type\" protobuf:\"bytes,1,rep,name=conditions\"`\n\n\n\t
+                    \   // other fields\n\t}"
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: |-
+                        type of condition in CamelCase or in foo.example.com/CamelCase.
+                        ---
+                        Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
+                        useful (see .node.status.conditions), the ability to deconflict is important.
+                        The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+              readyReplicas:
+                description: ReadyReplicas is the number of workspaces that are in
+                  ready state.
+                type: integer
+              replicas:
+                description: Replicas is the total number of workspaces created by
+                  the InferenceSet.
+                type: integer
+              selector:
+                description: |-
+                  Selector is used to select the pods that provide metrics for making scaling action decisions.
+                  This field must be set when HPA and VPA is used for scaling.
+                type: string
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources:
+      scale:
+        labelSelectorPath: .status.selector
+        specReplicasPath: .spec.replicas
+        statusReplicasPath: .status.replicas
+      status: {}
diff --git a/config/rbac/inferenceset_editor_role.yaml b/config/rbac/inferenceset_editor_role.yaml
new file mode 100644
index 0000000..90e23cb
--- /dev/null
+++ b/config/rbac/inferenceset_editor_role.yaml
@@ -0,0 +1,25 @@
+# permissions for end users to edit inferencesets.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    app.kubernetes.io/name: clusterrole
+    app.kubernetes.io/instance: inferenceset-editor-role
+    app.kubernetes.io/component: rbac
+    app.kubernetes.io/created-by: kaito
+    app.kubernetes.io/part-of: kaito
+    app.kubernetes.io/managed-by: kustomize
+  name: inferenceset-editor-role
+rules:
+- apiGroups: ["kaito.sh"]
+  resources: ["workspaces"]
+  verbs: ["update","patch","get","list","watch","create","delete"]
+- apiGroups: ["kaito.sh"]
+  resources: ["workspaces/status"]
+  verbs: ["update","patch","get","list","watch"]
+- apiGroups: ["kaito.sh"]
+  resources: ["inferencesets"]
+  verbs: ["update","patch","get","list","watch"]
+- apiGroups: ["kaito.sh"]
+  resources: ["inferencesets/status"]
+  verbs: ["update","patch","get","list","watch"]
diff --git a/config/rbac/inferenceset_editor_role_binding.yaml b/config/rbac/inferenceset_editor_role_binding.yaml
new file mode 100644
index 0000000..d7ce4d7
--- /dev/null
+++ b/config/rbac/inferenceset_editor_role_binding.yaml
@@ -0,0 +1,19 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  labels:
+    app.kubernetes.io/name: clusterrolebinding
+    app.kubernetes.io/instance: inferenceset-editor-role-rolebinding
+    app.kubernetes.io/component: rbac
+    app.kubernetes.io/created-by: kaito
+    app.kubernetes.io/part-of: kaito
+    app.kubernetes.io/managed-by: kustomize
+  name: inferenceset-editor-role-rolebinding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: inferenceset-editor-role
+subjects:
+- kind: ServiceAccount
+  name: controller-manager
+  namespace: system
diff --git a/config/rbac/inferenceset_viewer_role.yaml b/config/rbac/inferenceset_viewer_role.yaml
new file mode 100644
index 0000000..3362668
--- /dev/null
+++ b/config/rbac/inferenceset_viewer_role.yaml
@@ -0,0 +1,25 @@
+# permissions for end users to view inferencesets.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    app.kubernetes.io/name: clusterrole
+    app.kubernetes.io/instance: inferenceset-viewer-role
+    app.kubernetes.io/component: rbac
+    app.kubernetes.io/created-by: kaito
+    app.kubernetes.io/part-of: kaito
+    app.kubernetes.io/managed-by: kustomize
+  name: inferenceset-viewer-role
+rules:
+- apiGroups: ["kaito.sh"]
+  resources: ["workspaces"]
+  verbs: ["get","list","watch"]
+- apiGroups: ["kaito.sh"]
+  resources: ["workspaces/status"]
+  verbs: ["get","list","watch"]
+- apiGroups: ["kaito.sh"]
+  resources: ["inferencesets"]
+  verbs: ["get","list","watch"]
+- apiGroups: ["kaito.sh"]
+  resources: ["inferencesets/status"]
+  verbs: ["get","list","watch"]
diff --git a/config/rbac/inferenceset_viewer_role_binding.yaml b/config/rbac/inferenceset_viewer_role_binding.yaml
new file mode 100644
index 0000000..cef3756
--- /dev/null
+++ b/config/rbac/inferenceset_viewer_role_binding.yaml
@@ -0,0 +1,19 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  labels:
+    app.kubernetes.io/name: clusterrolebinding
+    app.kubernetes.io/instance: inferenceset-viewer-role-rolebinding
+    app.kubernetes.io/component: rbac
+    app.kubernetes.io/created-by: kaito
+    app.kubernetes.io/part-of: kaito
+    app.kubernetes.io/managed-by: kustomize
+  name: inferenceset-viewer-role-rolebinding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: inferenceset-viewer-role
+subjects:
+- kind: ServiceAccount
+  name: controller-manager
+  namespace: system
diff --git a/examples/custom-model-integration/Dockerfile.reference b/examples/custom-model-integration/Dockerfile.reference
deleted file mode 100644
index 3eb01ae..0000000
--- a/examples/custom-model-integration/Dockerfile.reference
+++ /dev/null
@@ -1,22 +0,0 @@
-FROM python:3.12-slim
-
-# Specify the repository source URL for reference and access in KAITO packages.
-LABEL org.opencontainers.image.source=https://github.com/kaito-project/kaito
-
-ARG MODEL_TYPE
-ARG VERSION
-
-# Set the working directory
-WORKDIR /workspace/tfs
-
-# Write the version to a file
-RUN echo $VERSION > /workspace/tfs/version.txt
-
-# First, copy just the preset files and install dependencies
-# This is done before copying the code to utilize Docker's layer caching and
-# avoid reinstalling dependencies unless the requirements file changes.
-# Inference
-COPY presets/workspace/dependencies/requirements.txt /workspace/tfs/inference-requirements.txt
-RUN pip install --no-cache-dir -r inference-requirements.txt
-
-COPY presets/workspace/inference/${MODEL_TYPE}/inference_api.py /workspace/tfs/inference_api.py
diff --git a/examples/custom-model-integration/custom-deployment-template.yaml b/examples/custom-model-integration/custom-deployment-template.yaml
deleted file mode 100644
index 969d05a..0000000
--- a/examples/custom-model-integration/custom-deployment-template.yaml
+++ /dev/null
@@ -1,56 +0,0 @@
-apiVersion: kaito.sh/v1beta1
-kind: Workspace
-metadata:
-  name: workspace-custom-llm
-resource:
-  instanceType: "Standard_NC24ads_A100_v4" # Replace with the required VM SKU based on model requirements
-  labelSelector:
-    matchLabels:
-      apps: custom-llm
-inference:
-  template: 
-    spec:
-      containers:
-      - name: custom-llm-container
-        image: modelsregistry.azurecr.io/custom-llm:0.0.1 # Replace with the actual image name
-        livenessProbe:
-          failureThreshold: 3
-          httpGet:
-            path: /health
-            port: 5000
-            scheme: HTTP
-          initialDelaySeconds: 600
-          periodSeconds: 10
-          successThreshold: 1
-          timeoutSeconds: 1
-        readinessProbe:
-          failureThreshold: 3
-          httpGet:
-            path: /health
-            port: 5000
-            scheme: HTTP
-          initialDelaySeconds: 30
-          periodSeconds: 10
-          successThreshold: 1
-          timeoutSeconds: 1
-        command: ["accelerate"]
-        args:
-          - "launch"
-          - "--num_processes"
-          - "1"
-          - "--num_machines"
-          - "1"
-          - "--gpu_ids"
-          - "all"
-          - "tfs/inference_api.py"
-          - "--pipeline"
-          - "text-generation"
-          - "--torch_dtype"
-          - "float16"  # Set to "float16" for compatibility with V100 GPUs; use "bfloat16" for A100, H100 or newer GPUs
-        volumeMounts:
-        - name: dshm
-          mountPath: /dev/shm
-      volumes:
-      - name: dshm
-        emptyDir:
-          medium: Memory
diff --git a/examples/custom-model-integration/reference-image-deployment.yaml b/examples/custom-model-integration/custom-model-deployment.yaml
similarity index 92%
rename from examples/custom-model-integration/reference-image-deployment.yaml
rename to examples/custom-model-integration/custom-model-deployment.yaml
index fc85c77..9aa2070 100644
--- a/examples/custom-model-integration/reference-image-deployment.yaml
+++ b/examples/custom-model-integration/custom-model-deployment.yaml
@@ -7,14 +7,12 @@ resource:
   labelSelector:
     matchLabels:
       apps: custom-llm
-#  preferredNodes:
-#    - <PREFERRED_NODE_HERE>
 inference:
   template:
     spec:
       containers:
         - name: custom-llm-container
-          image: ghcr.io/kaito-project/kaito/llm-reference-preset:latest # KAITO Reference Image
+          image: mcr.microsoft.com/aks/kaito/kaito-base:0.0.8 # KAITO base image which includes hf runtime
           livenessProbe:
             failureThreshold: 3
             httpGet:
@@ -50,7 +48,7 @@ inference:
             - "1"
             - "--gpu_ids"
             - "all"
-            - "inference_api.py"
+            - "tfs/inference_api.py"
             - "--pipeline"
             - "text-generation"
             - "--trust_remote_code"
diff --git a/examples/inference/kaito_inferenceset_llama-3.1_8b_instruct.yaml b/examples/inference/kaito_inferenceset_llama-3.1_8b_instruct.yaml
new file mode 100644
index 0000000..fc4ef40
--- /dev/null
+++ b/examples/inference/kaito_inferenceset_llama-3.1_8b_instruct.yaml
@@ -0,0 +1,30 @@
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: hf-token
+type: Opaque
+data:
+  HF_TOKEN: <base64-encoded-huggingface-token>
+---
+apiVersion: kaito.sh/v1alpha1
+kind: InferenceSet
+metadata:
+  name: llama-3-1-8b
+spec:
+  replicas: 2 # number of workspace CR created by InferenceSet controller
+  nodeCountLimit: 10 # optional, total GPU node count limit for InferenceSet
+  labelSelector:
+    matchLabels:
+      # workspace created by InferenceSet controller would use this label in resource.labelSelector
+      apps: large-model
+  template:
+    resource:
+      instanceType: "Standard_NC24ads_A100_v4"
+    inference: # fields in inference are the same as in workspace.resource.inference
+      preset:
+        name: llama-3.1-8b-instruct
+        presetOptions:
+          modelAccessSecret: hf-token
+  updateStrategy:
+    type: RollingUpdate
diff --git a/pkg/utils/consts/consts.go b/pkg/utils/consts/consts.go
index d6b0dc0..ae23633 100644
--- a/pkg/utils/consts/consts.go
+++ b/pkg/utils/consts/consts.go
@@ -18,6 +18,8 @@ import "time"
 const (
 	// WorkspaceFinalizer is used to make sure that workspace controller handles garbage collection.
 	WorkspaceFinalizer = "workspace.finalizer.kaito.sh"
+	// InferenceSetFinalizer is used to make sure that inferenceset controller handles garbage collection.
+	InferenceSetFinalizer = "inferenceset.finalizer.kaito.sh"
 	// RAGEngineFinalizer is used to make sure that ragengine controller handles garbage collection.
 	RAGEngineFinalizer            = "ragengine.finalizer.kaito.sh"
 	DefaultReleaseNamespaceEnvVar = "RELEASE_NAMESPACE"
@@ -67,6 +69,9 @@ const (
 
 	// ConditionReady is the condition type for a ready condition.
 	ConditionReady = "Ready"
+
+	WorkspaceCreatedByInferenceSetLabel = "inferenceset.kaito.sh/created-by"
+	InferenceSetMemberLabel             = "inferenceset.kaito.sh/member"
 )
 
 var (
diff --git a/pkg/utils/inferenceset/inferenceset.go b/pkg/utils/inferenceset/inferenceset.go
new file mode 100644
index 0000000..6805bc6
--- /dev/null
+++ b/pkg/utils/inferenceset/inferenceset.go
@@ -0,0 +1,160 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package inferenceset
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/api/meta"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/client-go/util/retry"
+	"k8s.io/klog/v2"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	kaitov1alpha1 "github.com/kaito-project/kaito/api/v1alpha1"
+	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
+	"github.com/kaito-project/kaito/pkg/utils/consts"
+)
+
+// UpdateStatusConditionIfNotMatch updates the inferenceset status condition if it doesn't match the current values
+func UpdateStatusConditionIfNotMatch(ctx context.Context, c client.Client, iObj *kaitov1alpha1.InferenceSet, cType kaitov1alpha1.ConditionType,
+	cStatus metav1.ConditionStatus, cReason, cMessage string) error {
+	if curCondition := meta.FindStatusCondition(iObj.Status.Conditions, string(cType)); curCondition != nil {
+		if curCondition.Status == cStatus && curCondition.Reason == cReason && curCondition.Message == cMessage {
+			// Nothing to change
+			return nil
+		}
+	}
+	klog.InfoS("updateStatusCondition", "inferenceset", klog.KObj(iObj), "conditionType", cType, "status", cStatus, "reason", cReason, "message", cMessage)
+	condition := metav1.Condition{
+		Type:               string(cType),
+		Status:             cStatus,
+		Reason:             cReason,
+		ObservedGeneration: iObj.GetGeneration(),
+		Message:            cMessage,
+		LastTransitionTime: metav1.Now(),
+	}
+	return UpdateInferenceSetStatus(ctx, c, &client.ObjectKey{Name: iObj.Name, Namespace: iObj.Namespace}, func(status *kaitov1alpha1.InferenceSetStatus) error {
+		meta.SetStatusCondition(&status.Conditions, condition)
+		return nil
+	})
+}
+
+// UpdateInferenceSetStatus updates the inferenceset status with the provided condition
+func UpdateInferenceSetStatus(ctx context.Context, c client.Client, name *client.ObjectKey, modifyFn func(*kaitov1alpha1.InferenceSetStatus) error) error {
+	return retry.OnError(retry.DefaultRetry,
+		func(err error) bool {
+			return apierrors.IsServiceUnavailable(err) || apierrors.IsServerTimeout(err) || apierrors.IsTooManyRequests(err) || apierrors.IsConflict(err)
+		},
+		func() error {
+			// Read the latest version to avoid update conflict.
+			iObj := &kaitov1alpha1.InferenceSet{}
+			if err := c.Get(ctx, *name, iObj); err != nil {
+				if !apierrors.IsNotFound(err) {
+					return err
+				}
+				return nil
+			}
+			if modifyFn != nil {
+				if err := modifyFn(&iObj.Status); err != nil {
+					return err
+				}
+			}
+			return c.Status().Update(ctx, iObj)
+		})
+}
+
+// UpdateInferenceSetWithRetry gets the latest inferenceset object, applies the modify function, and retries on conflict
+func UpdateInferenceSetWithRetry(ctx context.Context, c client.Client, iObj *kaitov1alpha1.InferenceSet, modifyFn func(*kaitov1alpha1.InferenceSet) error) error {
+	return retry.RetryOnConflict(retry.DefaultRetry, func() error {
+		latestInferenceSet := &kaitov1alpha1.InferenceSet{}
+		if err := c.Get(ctx, client.ObjectKeyFromObject(iObj), latestInferenceSet); err != nil {
+			return err
+		}
+		if err := modifyFn(latestInferenceSet); err != nil {
+			return err
+		}
+		return c.Update(ctx, latestInferenceSet)
+	})
+}
+
+// ListWorkspaces lists all workspace objects in the cluster that are created by the given InferenceSet.
+func ListWorkspaces(ctx context.Context, iObj *kaitov1alpha1.InferenceSet, kubeClient client.Client) (*kaitov1beta1.WorkspaceList, error) {
+	if iObj == nil {
+		return nil, fmt.Errorf("InferenceSet object is nil")
+	}
+	workspaceList := &kaitov1beta1.WorkspaceList{}
+
+	// List all the workspaces that are created by this inferenceset.
+	// We use label selector to find the workspaces.
+	// The label is "inferenceset.kaito.sh/created-by": <inferenceset-name>
+	ls := labels.Set{
+		consts.WorkspaceCreatedByInferenceSetLabel: iObj.Name,
+	}
+
+	err := retry.OnError(retry.DefaultBackoff, func(err error) bool {
+		return true
+	}, func() error {
+		return kubeClient.List(ctx, workspaceList, &client.MatchingLabelsSelector{Selector: ls.AsSelector()})
+	})
+	return workspaceList, err
+}
+
+// GetWorkspace retrieves a workspace by name from a list of workspaces.
+// Returns nil if the workspace is not found.
+func GetWorkspace(workspaceName string, workspaceList *kaitov1beta1.WorkspaceList) *kaitov1beta1.Workspace {
+	if workspaceName == "" || workspaceList == nil {
+		return nil
+	}
+	for _, ws := range workspaceList.Items {
+		if ws.Name == workspaceName {
+			return &ws
+		}
+	}
+	return nil
+}
+
+func ComputeInferenceSetHash(iObj *kaitov1alpha1.InferenceSet) string {
+	if iObj == nil {
+		return ""
+	}
+
+	hasher := sha256.New()
+	encoder := json.NewEncoder(hasher)
+	encoder.Encode(iObj.Spec)
+	return hex.EncodeToString(hasher.Sum(nil))
+}
+
+func MarshalInferenceSetFields(iObj *kaitov1alpha1.InferenceSet) ([]byte, error) {
+	if iObj == nil {
+		return nil, fmt.Errorf("InferenceSet object is nil")
+	}
+
+	partialMap := map[string]interface{}{
+		"Spec": iObj.Spec,
+	}
+
+	jsonData, err := json.Marshal(partialMap)
+	if err != nil {
+		return nil, fmt.Errorf("failed to marshal selected fields: %w", err)
+	}
+
+	return jsonData, nil
+}
diff --git a/pkg/utils/workspace/workspace.go b/pkg/utils/workspace/workspace.go
index a726dcb..81d006d 100644
--- a/pkg/utils/workspace/workspace.go
+++ b/pkg/utils/workspace/workspace.go
@@ -15,15 +15,81 @@ package workspace
 
 import (
 	"context"
+	"fmt"
+	"math/rand"
+	"reflect"
+	"time"
 
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	"k8s.io/apimachinery/pkg/api/meta"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/client-go/util/retry"
 	"k8s.io/klog/v2"
 	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/event"
+	"sigs.k8s.io/controller-runtime/pkg/predicate"
 
 	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
+	"github.com/kaito-project/kaito/pkg/utils/consts"
+)
+
+const WorkspaceNameSuffixLength = 12
+
+var (
+	InferenceSetSelector, _ = metav1.LabelSelectorAsSelector(&metav1.LabelSelector{
+		MatchExpressions: []metav1.LabelSelectorRequirement{
+			{Key: consts.WorkspaceCreatedByInferenceSetLabel, Operator: metav1.LabelSelectorOpExists},
+		},
+	})
+
+	WorkspacePredicate = predicate.Funcs{
+		CreateFunc: func(e event.CreateEvent) bool {
+			workspace, ok := e.Object.(*kaitov1beta1.Workspace)
+			if !ok {
+				return false
+			}
+			if !InferenceSetSelector.Matches(labels.Set(workspace.GetLabels())) {
+				return false
+			}
+			return true
+		},
+		UpdateFunc: func(e event.UpdateEvent) bool {
+			oldWorkspace, ok := e.ObjectOld.(*kaitov1beta1.Workspace)
+			if !ok {
+				return false
+			}
+
+			newWorkspace, ok := e.ObjectNew.(*kaitov1beta1.Workspace)
+			if !ok {
+				return false
+			}
+			if !InferenceSetSelector.Matches(labels.Set(oldWorkspace.GetLabels())) {
+				return false
+			}
+
+			if !InferenceSetSelector.Matches(labels.Set(newWorkspace.GetLabels())) {
+				return false
+			}
+
+			oldWorkspaceCopy := oldWorkspace.DeepCopy()
+			newWorkspaceCopy := newWorkspace.DeepCopy()
+
+			oldWorkspaceCopy.ResourceVersion = ""
+			newWorkspaceCopy.ResourceVersion = ""
+			return !reflect.DeepEqual(oldWorkspaceCopy, newWorkspaceCopy)
+		},
+		DeleteFunc: func(e event.DeleteEvent) bool {
+			workspace, ok := e.Object.(*kaitov1beta1.Workspace)
+			if !ok {
+				return false
+			}
+			if !InferenceSetSelector.Matches(labels.Set(workspace.GetLabels())) {
+				return false
+			}
+			return true
+		},
+	}
 )
 
 // UpdateStatusConditionIfNotMatch updates the workspace status condition if it doesn't match the current values
@@ -87,3 +153,22 @@ func UpdateWorkspaceWithRetry(ctx context.Context, c client.Client, wObj *kaitov
 		return c.Update(ctx, latestWorkspace)
 	})
 }
+
+// generateRandomString generates a random string of the specified length using digits and lowercase letters.
+func generateRandomString(length int) string {
+	if length <= 0 {
+		return ""
+	}
+	source := rand.NewSource(time.Now().UnixNano())
+	r := rand.New(source)
+	charset := "abcdefghijklmnopqrstuvwxyz0123456789"
+	result := make([]byte, length)
+	for i := range result {
+		result[i] = charset[r.Intn(len(charset))]
+	}
+	return string(result)
+}
+
+func GetWorkspaceNameWithRandomSuffix(baseName string) string {
+	return fmt.Sprintf("%s-%s", baseName, generateRandomString(WorkspaceNameSuffixLength))
+}
diff --git a/pkg/workspace/controllers/inferenceset/inferenceset_controller.go b/pkg/workspace/controllers/inferenceset/inferenceset_controller.go
new file mode 100644
index 0000000..bcc69ce
--- /dev/null
+++ b/pkg/workspace/controllers/inferenceset/inferenceset_controller.go
@@ -0,0 +1,406 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package inferenceset
+
+import (
+	"context"
+	"fmt"
+	"slices"
+	"sort"
+	"strconv"
+
+	"github.com/go-logr/logr"
+	appsv1 "k8s.io/api/apps/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/tools/record"
+	"k8s.io/klog/v2"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/builder"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/controller"
+	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
+	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+
+	kaitov1alpha1 "github.com/kaito-project/kaito/api/v1alpha1"
+	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
+	"github.com/kaito-project/kaito/pkg/utils"
+	"github.com/kaito-project/kaito/pkg/utils/consts"
+	"github.com/kaito-project/kaito/pkg/utils/inferenceset"
+	"github.com/kaito-project/kaito/pkg/utils/workspace"
+	"github.com/kaito-project/kaito/pkg/workspace/controllers"
+)
+
+const (
+	InferenceSetHashAnnotation = "inferenceset.kaito.io/hash"
+	InferenceSetNameLabel      = "inferenceset.kaito.io/name"
+	revisionHashSuffix         = 5
+)
+
+type InferenceSetReconciler struct {
+	client.Client
+	Log      logr.Logger
+	Scheme   *runtime.Scheme
+	Recorder record.EventRecorder
+
+	klogger      klog.Logger
+	expectations *utils.ControllerExpectations
+}
+
+func NewInferenceSetReconciler(client client.Client, scheme *runtime.Scheme, log logr.Logger, Recorder record.EventRecorder) *InferenceSetReconciler {
+	expectations := utils.NewControllerExpectations()
+	return &InferenceSetReconciler{
+		Client:       client,
+		Scheme:       scheme,
+		Log:          log,
+		klogger:      klog.NewKlogr().WithName("InferenceSetController"),
+		Recorder:     Recorder,
+		expectations: expectations,
+	}
+}
+
+func (c *InferenceSetReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
+	iObj := &kaitov1alpha1.InferenceSet{}
+	if err := c.Client.Get(ctx, req.NamespacedName, iObj); err != nil {
+		if apierrors.IsNotFound(err) {
+			c.expectations.DeleteExpectations(c.klogger, req.String())
+			klog.InfoS("Inference set not found, might be deleted already", "inference set", req.Name)
+			return reconcile.Result{}, nil
+		}
+		klog.ErrorS(err, "failed to get inference set", "inference set", req.Name)
+		return reconcile.Result{}, err
+	}
+
+	klog.InfoS("Reconciling", "inference set", req.NamespacedName, "name", req.Name)
+	if iObj.DeletionTimestamp.IsZero() {
+		if err := c.ensureFinalizer(ctx, iObj); err != nil {
+			return reconcile.Result{}, err
+		}
+	} else {
+		// Handle deleting inferenceset, garbage collect all the resources.
+		return c.deleteInferenceSet(ctx, iObj)
+	}
+
+	if err := c.syncControllerRevision(ctx, iObj); err != nil {
+		return reconcile.Result{}, err
+	}
+
+	return c.addOrUpdateInferenceSet(ctx, iObj)
+}
+
+func (c *InferenceSetReconciler) ensureFinalizer(ctx context.Context, iObj *kaitov1alpha1.InferenceSet) error {
+	if !controllerutil.ContainsFinalizer(iObj, consts.InferenceSetFinalizer) {
+		patch := client.MergeFrom(iObj.DeepCopy())
+		controllerutil.AddFinalizer(iObj, consts.InferenceSetFinalizer)
+		if err := c.Client.Patch(ctx, iObj, patch); err != nil {
+			klog.ErrorS(err, "failed to ensure the finalizer to the inference set", "inference set", klog.KObj(iObj))
+			return err
+		}
+	}
+	return nil
+}
+
+func (c *InferenceSetReconciler) deleteInferenceSet(ctx context.Context, iObj *kaitov1alpha1.InferenceSet) (reconcile.Result, error) {
+	klog.InfoS("deleteInferenceSet", "inferenceset", klog.KObj(iObj))
+	err := inferenceset.UpdateStatusConditionIfNotMatch(ctx, c.Client, iObj, kaitov1alpha1.InferenceSetConditionTypeDeleting, metav1.ConditionTrue, "inferencesetDeleted", "inferenceset is being deleted")
+	if err != nil {
+		klog.ErrorS(err, "failed to update inferenceset status", "inferenceset", klog.KObj(iObj))
+		return reconcile.Result{}, err
+	}
+
+	return c.garbageCollectInferenceSet(ctx, iObj)
+}
+
+// garbageCollectInferenceSet remove finalizer associated with inferenceset object.
+func (c *InferenceSetReconciler) garbageCollectInferenceSet(ctx context.Context, iObj *kaitov1alpha1.InferenceSet) (ctrl.Result, error) {
+	klog.InfoS("garbageCollectInferenceSet", "inferenceset", klog.KObj(iObj))
+	// Check if there are any workspaces associated with this inferenceset.
+	wsList, err := inferenceset.ListWorkspaces(ctx, iObj, c.Client)
+	if err != nil {
+		return ctrl.Result{}, err
+	}
+	// We should delete all the workspaces that are created by this inferenceset
+	for i := range wsList.Items {
+		if wsList.Items[i].DeletionTimestamp.IsZero() {
+			klog.InfoS("Deleting associated Workspace...", "workspace", wsList.Items[i].Name)
+			if deleteErr := c.Delete(ctx, &wsList.Items[i], &client.DeleteOptions{}); deleteErr != nil {
+				klog.ErrorS(deleteErr, "failed to delete the workspace", "workspace", klog.KObj(&wsList.Items[i]))
+				return ctrl.Result{}, deleteErr
+			}
+		}
+	}
+
+	updateErr := inferenceset.UpdateInferenceSetWithRetry(ctx, c.Client, iObj, func(ws *kaitov1alpha1.InferenceSet) error {
+		controllerutil.RemoveFinalizer(ws, consts.InferenceSetFinalizer)
+		return nil
+	})
+	if updateErr != nil {
+		if apierrors.IsNotFound(updateErr) {
+			return ctrl.Result{}, nil
+		}
+		klog.ErrorS(updateErr, "failed to update the inferenceset to remove finalizer", "inferenceset", klog.KObj(iObj))
+		return ctrl.Result{}, updateErr
+	}
+
+	klog.InfoS("successfully removed the inferenceset finalizers", "inferenceset", klog.KObj(iObj))
+	return ctrl.Result{}, nil
+}
+
+func (c *InferenceSetReconciler) addOrUpdateInferenceSet(ctx context.Context, iObj *kaitov1alpha1.InferenceSet) (reconcile.Result, error) {
+	if iObj == nil {
+		return reconcile.Result{}, nil
+	}
+
+	isKey := client.ObjectKeyFromObject(iObj).String()
+	if !c.expectations.SatisfiedExpectations(c.Log, isKey) {
+		klog.V(4).InfoS("Waiting for expectations to be satisfied", "inferenceset", isKey)
+		return reconcile.Result{}, nil
+	}
+
+	// Check if there are any existing workspaces associated with this inferenceset.
+	wsList, err := inferenceset.ListWorkspaces(ctx, iObj, c.Client)
+	if err != nil {
+		return ctrl.Result{}, err
+	}
+	klog.InfoS("Found workspaces for inference set", "name", iObj.Name, "current", len(wsList.Items), "desired", iObj.Spec.Replicas)
+
+	replicaNumToDelete := len(wsList.Items) - iObj.Spec.Replicas
+	var deletingWorkspaces []string
+	if replicaNumToDelete > 0 {
+		klog.InfoS("Found extra workspaces, deleting...", "current", len(wsList.Items), "desired", iObj.Spec.Replicas)
+		// first delete workspace that is not in ready state
+		for _, ws := range wsList.Items {
+			if !ws.DeletionTimestamp.IsZero() {
+				deletingWorkspaces = append(deletingWorkspaces, ws.Name)
+				replicaNumToDelete--
+				klog.InfoS("Skipping workspace that is already being deleted...", "workspace", klog.KObj(&ws))
+			} else if controllers.DetermineWorkspacePhase(&ws) != "succeeded" {
+				klog.InfoS("Deleting non-ready workspace...", "workspace", klog.KObj(&ws))
+				if err := c.Client.Delete(ctx, &ws, &client.DeleteOptions{}); err != nil {
+					klog.ErrorS(err, "failed to delete non-ready workspace", "workspace", klog.KObj(&ws))
+					return ctrl.Result{}, err
+				}
+				deletingWorkspaces = append(deletingWorkspaces, ws.Name)
+				replicaNumToDelete--
+			}
+			if replicaNumToDelete <= 0 {
+				break
+			}
+		}
+
+		// delete rest of extra workspaces
+		if replicaNumToDelete > 0 {
+			for _, ws := range wsList.Items {
+				// check whether ws.Name is already in deletingWorkspaces
+				if slices.Contains(deletingWorkspaces, ws.Name) {
+					continue
+				}
+
+				if !ws.DeletionTimestamp.IsZero() {
+					replicaNumToDelete--
+					klog.InfoS("Skipping workspace that is already being deleted...", "workspace", klog.KObj(&ws))
+				} else {
+					klog.InfoS("Deleting extra workspace...", "workspace", klog.KObj(&ws))
+					if err := c.Client.Delete(ctx, &ws, &client.DeleteOptions{}); err != nil {
+						klog.ErrorS(err, "failed to delete extra workspace", "workspace", klog.KObj(&ws))
+						return ctrl.Result{}, err
+					}
+					replicaNumToDelete--
+				}
+				if replicaNumToDelete <= 0 {
+					break
+				}
+			}
+		}
+
+		// After deleting the extra workspaces, we should requeue to wait for the deletion to complete
+		if wsList, err = inferenceset.ListWorkspaces(ctx, iObj, c.Client); err != nil {
+			return ctrl.Result{}, err
+		}
+	}
+
+	replicaNumToCreate := iObj.Spec.Replicas - len(wsList.Items)
+	if replicaNumToCreate > 0 {
+		klog.InfoS("Need to create more workspaces...", "current", len(wsList.Items), "desired", iObj.Spec.Replicas)
+		for i := range replicaNumToCreate {
+			workspaceName := workspace.GetWorkspaceNameWithRandomSuffix(iObj.Name)
+			workspaceObj := inferenceset.GetWorkspace(workspaceName, wsList)
+			if workspaceObj == nil {
+				workspaceObj = &kaitov1beta1.Workspace{}
+			}
+			workspaceObj.Name = workspaceName
+			workspaceObj.Namespace = iObj.Namespace
+			workspaceObj.Labels = map[string]string{
+				consts.InferenceSetMemberLabel:             workspaceObj.Name,
+				consts.WorkspaceCreatedByInferenceSetLabel: iObj.Name,
+			}
+			workspaceObj.OwnerReferences = []metav1.OwnerReference{
+				*metav1.NewControllerRef(iObj, kaitov1alpha1.GroupVersion.WithKind("InferenceSet")),
+			}
+			workspaceObj.Resource = kaitov1beta1.ResourceSpec{
+				InstanceType:  iObj.Spec.Template.Resource.InstanceType,
+				LabelSelector: iObj.Spec.Selector,
+			}
+			workspaceObj.Inference = &iObj.Spec.Template.Inference
+
+			klog.InfoS("creating workspace", "workspace", workspaceObj.Name, "index", i)
+			if err := c.Client.Create(ctx, workspaceObj); err != nil {
+				klog.ErrorS(err, "failed to create workspace", "workspace", workspaceObj.Name)
+				return reconcile.Result{}, err
+			}
+		}
+	}
+
+	// check whether all the workspaces are ready
+	readyReplicas := 0
+	for _, ws := range wsList.Items {
+		if controllers.DetermineWorkspacePhase(&ws) == "succeeded" {
+			readyReplicas++
+		}
+	}
+
+	// update the replicas in the status
+	if err = inferenceset.UpdateInferenceSetStatus(ctx, c.Client, &client.ObjectKey{Name: iObj.Name, Namespace: iObj.Namespace}, func(status *kaitov1alpha1.InferenceSetStatus) error {
+		status.Replicas = iObj.Spec.Replicas
+		status.ReadyReplicas = readyReplicas
+		return nil
+	}); err != nil {
+		klog.ErrorS(err, "failed to update inferenceset replicas", "inferenceset", klog.KObj(iObj))
+		return reconcile.Result{}, err
+	}
+
+	if readyReplicas == iObj.Spec.Replicas {
+		if err = inferenceset.UpdateStatusConditionIfNotMatch(ctx, c.Client, iObj, kaitov1alpha1.InferenceSetConditionTypeReady, metav1.ConditionTrue,
+			"inferencesetReady", "inferenceset is ready"); err != nil {
+			klog.ErrorS(err, "failed to update inferenceset status", "inferenceset", klog.KObj(iObj))
+			return reconcile.Result{}, err
+		}
+	} else {
+		if err = inferenceset.UpdateStatusConditionIfNotMatch(ctx, c.Client, iObj, kaitov1alpha1.InferenceSetConditionTypeReady, metav1.ConditionFalse,
+			"inferencesetNotReady", fmt.Sprintf("inferenceset is not ready, %d/%d replicas are ready", readyReplicas, iObj.Spec.Replicas)); err != nil {
+			klog.ErrorS(err, "failed to update inferenceset status", "inferenceset", klog.KObj(iObj))
+			return reconcile.Result{}, err
+		}
+	}
+	return reconcile.Result{}, nil
+}
+
+func (c *InferenceSetReconciler) syncControllerRevision(ctx context.Context, iObj *kaitov1alpha1.InferenceSet) error {
+	currentHash := inferenceset.ComputeInferenceSetHash(iObj)
+	annotations := iObj.GetAnnotations()
+	if annotations == nil {
+		annotations = make(map[string]string)
+	} // nil checking.
+
+	revisionNum := int64(1)
+
+	revisions := &appsv1.ControllerRevisionList{}
+	if err := c.List(ctx, revisions, client.InNamespace(iObj.Namespace), client.MatchingLabels{InferenceSetNameLabel: iObj.Name}); err != nil {
+		return fmt.Errorf("failed to list revisions: %w", err)
+	}
+	sort.Slice(revisions.Items, func(i, j int) bool {
+		return revisions.Items[i].Revision < revisions.Items[j].Revision
+	})
+
+	var latestRevision *appsv1.ControllerRevision
+
+	jsonData, err := inferenceset.MarshalInferenceSetFields(iObj)
+	if err != nil {
+		return fmt.Errorf("failed to marshal revision data: %w", err)
+	}
+
+	if len(revisions.Items) > 0 {
+		latestRevision = &revisions.Items[len(revisions.Items)-1]
+		revisionNum = latestRevision.Revision + 1
+	}
+	newRevision := &appsv1.ControllerRevision{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      fmt.Sprintf("%s-%s", iObj.Name, currentHash[:revisionHashSuffix]),
+			Namespace: iObj.Namespace,
+			Annotations: map[string]string{
+				InferenceSetHashAnnotation: currentHash,
+			},
+			Labels: map[string]string{
+				InferenceSetNameLabel: iObj.Name,
+			},
+			OwnerReferences: []metav1.OwnerReference{
+				*metav1.NewControllerRef(iObj, kaitov1alpha1.GroupVersion.WithKind("InferenceSet")),
+			},
+		},
+		Revision: revisionNum,
+		Data:     runtime.RawExtension{Raw: jsonData},
+	}
+
+	annotations[InferenceSetHashAnnotation] = currentHash
+	iObj.SetAnnotations(annotations)
+	controllerRevision := &appsv1.ControllerRevision{}
+	if err := c.Get(ctx, types.NamespacedName{
+		Name:      newRevision.Name,
+		Namespace: newRevision.Namespace,
+	}, controllerRevision); err != nil {
+		if apierrors.IsNotFound(err) {
+			if err := c.Create(ctx, newRevision); err != nil {
+				return fmt.Errorf("failed to create new ControllerRevision: %w", err)
+			} else {
+				annotations[kaitov1alpha1.InferenceSetRevisionAnnotation] = strconv.FormatInt(revisionNum, 10)
+			}
+
+			if len(revisions.Items) > consts.MaxRevisionHistoryLimit {
+				if err := c.Delete(ctx, &revisions.Items[0]); err != nil {
+					return fmt.Errorf("failed to delete old revision: %w", err)
+				}
+			}
+		} else {
+			return fmt.Errorf("failed to get controller revision: %w", err)
+		}
+	} else {
+		if controllerRevision.Annotations[InferenceSetHashAnnotation] != newRevision.Annotations[InferenceSetHashAnnotation] {
+			return fmt.Errorf("revision name conflicts, the hash values are different, old hash: %s, new hash: %s", controllerRevision.Annotations[InferenceSetHashAnnotation], newRevision.Annotations[InferenceSetHashAnnotation])
+		}
+		annotations[kaitov1alpha1.InferenceSetRevisionAnnotation] = strconv.FormatInt(controllerRevision.Revision, 10)
+	}
+	annotations[InferenceSetHashAnnotation] = currentHash
+
+	err = inferenceset.UpdateInferenceSetWithRetry(ctx, c.Client, iObj, func(ws *kaitov1alpha1.InferenceSet) error {
+		ws.SetAnnotations(annotations)
+		return nil
+	})
+	if err != nil {
+		return fmt.Errorf("failed to update InferenceSet annotations: %w", err)
+	}
+	return nil
+}
+
+// SetupWithManager sets up the controller with the Manager.
+func (c *InferenceSetReconciler) SetupWithManager(mgr ctrl.Manager) error {
+	c.Recorder = mgr.GetEventRecorderFor("InferenceSet")
+
+	builder := ctrl.NewControllerManagedBy(mgr).
+		For(&kaitov1alpha1.InferenceSet{}).
+		Owns(&appsv1.ControllerRevision{}).
+		Watches(&kaitov1beta1.Workspace{},
+			&workspaceEventHandler{
+				logger:         c.klogger,
+				expectations:   c.expectations,
+				enqueueHandler: enqueueInferenceSetForWorkspace,
+			},
+			builder.WithPredicates(workspace.WorkspacePredicate),
+		).
+		WithOptions(controller.Options{MaxConcurrentReconciles: 5})
+
+	go monitorInferenceSets(context.Background(), c.Client)
+	return builder.Complete(c)
+}
diff --git a/pkg/workspace/controllers/inferenceset/inferenceset_event_handler.go b/pkg/workspace/controllers/inferenceset/inferenceset_event_handler.go
new file mode 100644
index 0000000..58a7d25
--- /dev/null
+++ b/pkg/workspace/controllers/inferenceset/inferenceset_event_handler.go
@@ -0,0 +1,87 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package inferenceset
+
+import (
+	"context"
+
+	"k8s.io/client-go/util/workqueue"
+	"k8s.io/klog/v2"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/event"
+	"sigs.k8s.io/controller-runtime/pkg/handler"
+	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+
+	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
+	"github.com/kaito-project/kaito/pkg/utils"
+	"github.com/kaito-project/kaito/pkg/utils/consts"
+)
+
+type workspaceEventHandler struct {
+	enqueueHandler handler.TypedEventHandler[client.Object, reconcile.Request]
+	expectations   *utils.ControllerExpectations
+	logger         klog.Logger
+}
+
+var _ handler.TypedEventHandler[client.Object, reconcile.Request] = (*workspaceEventHandler)(nil)
+
+func getControllerKeyForWorkspace(ws *kaitov1beta1.Workspace) *client.ObjectKey {
+	name, ok := ws.Labels[consts.WorkspaceCreatedByInferenceSetLabel]
+	if !ok {
+		return nil
+	}
+	return &client.ObjectKey{Namespace: ws.Namespace, Name: name}
+}
+
+func (n *workspaceEventHandler) Create(ctx context.Context, evt event.TypedCreateEvent[client.Object], q workqueue.TypedRateLimitingInterface[reconcile.Request]) {
+	ws := evt.Object.(*kaitov1beta1.Workspace)
+	if ws.DeletionTimestamp != nil {
+		n.Delete(ctx, event.TypedDeleteEvent[client.Object]{Object: ws}, q)
+		return
+	}
+	if key := getControllerKeyForWorkspace(ws); key != nil {
+		n.expectations.CreationObserved(n.logger, key.String())
+		n.enqueueHandler.Create(ctx, evt, q)
+	}
+}
+
+func (n *workspaceEventHandler) Delete(ctx context.Context, evt event.TypedDeleteEvent[client.Object], q workqueue.TypedRateLimitingInterface[reconcile.Request]) {
+	ws := evt.Object.(*kaitov1beta1.Workspace)
+	if key := getControllerKeyForWorkspace(ws); key != nil {
+		n.expectations.DeletionObserved(n.logger, key.String())
+	}
+
+	n.enqueueHandler.Delete(ctx, evt, q)
+}
+
+func (n *workspaceEventHandler) Generic(ctx context.Context, evt event.TypedGenericEvent[client.Object], q workqueue.TypedRateLimitingInterface[reconcile.Request]) {
+}
+
+func (n *workspaceEventHandler) Update(ctx context.Context, evt event.TypedUpdateEvent[client.Object], q workqueue.TypedRateLimitingInterface[reconcile.Request]) {
+	n.enqueueHandler.Update(ctx, evt, q)
+}
+
+var enqueueInferenceSetForWorkspace = handler.EnqueueRequestsFromMapFunc(
+	func(ctx context.Context, o client.Object) []reconcile.Request {
+		workspaceObj := o.(*kaitov1beta1.Workspace)
+		key := getControllerKeyForWorkspace(workspaceObj)
+		if key == nil {
+			return nil
+		}
+		return []reconcile.Request{
+			{
+				NamespacedName: *key,
+			},
+		}
+	})
diff --git a/pkg/workspace/controllers/inferenceset/metrics.go b/pkg/workspace/controllers/inferenceset/metrics.go
new file mode 100644
index 0000000..f89690d
--- /dev/null
+++ b/pkg/workspace/controllers/inferenceset/metrics.go
@@ -0,0 +1,99 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package inferenceset
+
+import (
+	"context"
+	"time"
+
+	"github.com/prometheus/client_golang/prometheus"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/klog/v2"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/metrics"
+
+	kaitov1alpha1 "github.com/kaito-project/kaito/api/v1alpha1"
+)
+
+var (
+	inferencesetPhaseCount = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "kaito_inferenceset_count",
+			Help: "Number of InferenceSets in a certain phase (succeeded, error, pending, deleting)",
+		},
+		[]string{"phase"},
+	)
+)
+
+func init() {
+	metrics.Registry.MustRegister(inferencesetPhaseCount)
+}
+
+func monitorInferenceSets(ctx context.Context, k8sClient client.Client) {
+	ticker := time.NewTicker(1 * time.Minute)
+	defer ticker.Stop()
+
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-ticker.C:
+			var isList kaitov1alpha1.InferenceSetList
+
+			if err := k8sClient.List(ctx, &isList); err != nil {
+				klog.Errorf("failed to list all inferencesets: %v", err)
+				inferencesetPhaseCount.Reset()
+				continue
+			}
+
+			phaseCounts := map[string]float64{
+				"succeeded": 0,
+				"error":     0,
+				"pending":   0,
+				"deleting":  0,
+			}
+
+			for _, is := range isList.Items {
+				phase := determineInferenceSetPhase(&is)
+				if _, ok := phaseCounts[phase]; !ok {
+					phaseCounts[phase] = 0
+				}
+				phaseCounts[phase]++
+			}
+
+			for phase, count := range phaseCounts {
+				inferencesetPhaseCount.WithLabelValues(phase).Set(count)
+			}
+		}
+	}
+}
+
+func determineInferenceSetPhase(is *kaitov1alpha1.InferenceSet) string {
+	for _, cond := range is.Status.Conditions {
+		switch kaitov1alpha1.ConditionType(cond.Type) {
+		case kaitov1alpha1.InferenceSetConditionTypeDeleting:
+			if cond.Status == metav1.ConditionTrue {
+				return "deleting"
+			}
+		case kaitov1alpha1.InferenceSetConditionTypeReady:
+			if cond.Status == metav1.ConditionTrue {
+				return "ready"
+			}
+			if cond.Status == metav1.ConditionFalse {
+				return "error"
+			}
+		}
+	}
+	return "pending"
+}
diff --git a/pkg/workspace/controllers/metrics.go b/pkg/workspace/controllers/metrics.go
index 5dfb1e4..9fae917 100644
--- a/pkg/workspace/controllers/metrics.go
+++ b/pkg/workspace/controllers/metrics.go
@@ -65,7 +65,7 @@ func monitorWorkspaces(ctx context.Context, k8sClient client.Client) {
 			}
 
 			for _, ws := range wsList.Items {
-				phase := determineWorkspacePhase(&ws)
+				phase := DetermineWorkspacePhase(&ws)
 				if _, ok := phaseCounts[phase]; !ok {
 					phaseCounts[phase] = 0
 				}
@@ -79,7 +79,7 @@ func monitorWorkspaces(ctx context.Context, k8sClient client.Client) {
 	}
 }
 
-func determineWorkspacePhase(ws *kaitov1beta1.Workspace) string {
+func DetermineWorkspacePhase(ws *kaitov1beta1.Workspace) string {
 	for _, cond := range ws.Status.Conditions {
 		switch kaitov1beta1.ConditionType(cond.Type) {
 		case kaitov1beta1.WorkspaceConditionTypeDeleting:
diff --git a/pkg/workspace/controllers/workspace_controller.go b/pkg/workspace/controllers/workspace_controller.go
index 5e183db..29eb1b2 100644
--- a/pkg/workspace/controllers/workspace_controller.go
+++ b/pkg/workspace/controllers/workspace_controller.go
@@ -263,7 +263,7 @@ func (c *WorkspaceReconciler) deleteWorkspace(ctx context.Context, wObj *kaitov1
 	return c.garbageCollectWorkspace(ctx, wObj)
 }
 func (c *WorkspaceReconciler) syncControllerRevision(ctx context.Context, wObj *kaitov1beta1.Workspace) error {
-	currentHash := computeHash(wObj)
+	currentHash := ComputeHash(wObj)
 	annotations := wObj.GetAnnotations()
 	if annotations == nil {
 		annotations = make(map[string]string)
@@ -365,7 +365,7 @@ func marshalSelectedFields(wObj *kaitov1beta1.Workspace) ([]byte, error) {
 	return jsonData, nil
 }
 
-func computeHash(w *kaitov1beta1.Workspace) string {
+func ComputeHash(w *kaitov1beta1.Workspace) string {
 	hasher := sha256.New()
 	encoder := json.NewEncoder(hasher)
 	encoder.Encode(w.Resource)
diff --git a/pkg/workspace/webhooks/webhooks.go b/pkg/workspace/webhooks/webhooks.go
index 13d5918..0f35d88 100644
--- a/pkg/workspace/webhooks/webhooks.go
+++ b/pkg/workspace/webhooks/webhooks.go
@@ -28,10 +28,11 @@ import (
 	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
 )
 
-func NewWorkspaceWebhooks() []knativeinjection.ControllerConstructor {
+func NewControllerWebhooks() []knativeinjection.ControllerConstructor {
 	return []knativeinjection.ControllerConstructor{
 		certificates.NewController,
 		NewWorkspaceCRDValidationWebhook,
+		NewInferenceSetCRDValidationWebhook,
 	}
 }
 
@@ -45,7 +46,21 @@ func NewWorkspaceCRDValidationWebhook(ctx context.Context, _ configmap.Watcher)
 	)
 }
 
+func NewInferenceSetCRDValidationWebhook(ctx context.Context, _ configmap.Watcher) *controller.Impl {
+	return validation.NewAdmissionController(ctx,
+		"validation.inferenceset.kaito.sh",
+		"/validate/inferenceset.kaito.sh",
+		InferenceSetResources,
+		func(ctx context.Context) context.Context { return ctx },
+		true,
+	)
+}
+
 var WorkspaceResources = map[schema.GroupVersionKind]resourcesemantics.GenericCRD{
 	kaitov1alpha1.GroupVersion.WithKind("Workspace"): &kaitov1alpha1.Workspace{},
 	kaitov1beta1.GroupVersion.WithKind("Workspace"):  &kaitov1beta1.Workspace{},
 }
+
+var InferenceSetResources = map[schema.GroupVersionKind]resourcesemantics.GenericCRD{
+	kaitov1alpha1.GroupVersion.WithKind("InferenceSet"): &kaitov1alpha1.InferenceSet{},
+}
diff --git a/website/docs/custom-model.md b/website/docs/custom-model.md
index c3b7db7..14b33c5 100644
--- a/website/docs/custom-model.md
+++ b/website/docs/custom-model.md
@@ -2,82 +2,21 @@
 title: Custom Model Integration
 ---
 
-## Option 1: Use Pre-Built Docker Image Without Model Weights
-If you want to avoid building a Docker image with model weights, use our pre-built reference image (`ghcr.io/kaito-project/kaito/llm-reference-preset:latest`). This image, built with [Dockerfile.reference](../../examples/custom-model-integration/Dockerfile.reference), dynamically downloads model weights from HuggingFace at runtime, reducing the need to create and maintain custom images.
+## Using the KAITO base image
 
+The KAITO base image includes both HuggingFace and vLLM runtime libraries along with corresponding FastAPI server scripts. This provides a convenient way to run **any** HuggingFace model using the **HuggingFace** runtime.
 
-- **[Sample Deployment YAML](../../examples/custom-model-integration/reference-image-deployment.yaml)**
+Note that the vLLM runtime is not supported for arbitrary custom model deployment.
 
+Here is a **[sample deployment YAML](https://github.com/kaito-project/kaito/tree/main/examples/custom-model-integration/custom-model-deployment.yaml)**. To use it:
+1. Specify the HuggingFace model ID in the container command.
+2. For models that require a HuggingFace token to download, users need to add the token to the specified secret.
 
-## Option 2: Build a Custom Docker Image with Model Weights
+The script downloads model weights during server bootstrap, eliminating the need to pre-bake them into the container image.
 
-### Step 1: Clone the Repository
+## Limitations
 
-```sh
-git clone https://github.com/kaito-project/kaito.git
-```
-
-### Step 2: Download Your Private/Custom Model Weights
-
-For example, assuming HuggingFace weights:
-```sh
-git lfs install
-git clone git@hf.co:<MODEL_ID>  # Example: git clone git@hf.co:bigscience/bloom
-# OR
-git clone https://huggingface.co/bigscience/bloom
-```
-
-Alternatively, use curl:
-```
-curl -sSL https://huggingface.co/bigscience/bloom/resolve/main/config.json?download=true -o config.json
-```
-
-More information on downloading models from HuggingFace can be found [here](https://huggingface.co/docs/hub/en/models-downloading).
-
-
-### Step 3: Log In to Your Container Registry
-
-Before pushing the Docker image, ensure you’re logged into the appropriate container registry. Here are general login methods depending on the registry you use:
-
-1. GitHub Container Registry (ghcr.io):
-```sh
-echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin
-```
-Replace CR_PAT with your GitHub Personal Access Token and USERNAME with your GitHub username. This token should have the write:packages and read:packages permissions.
-
-2. Azure Container Registry (ACR): If you're using ACR:
-
-```sh
-az acr login --name <REGISTRY_NAME>
-```
-Replace `<REGISTRY_NAME>` with your Azure Container Registry name.
-
-3. Docker Hub or Other Container Registries:
-```sh
-docker login <REGISTRY_URL>
-```
-Enter your username and password when prompted. Replace `<REGISTRY_URL>` with your registry URL, such as `docker.io` for Docker Hub.
-
-
-### Step 4: Build Docker Image with Private/Custom Weights
-
-1. Set Environment Variables
-
-Before building the Docker image, set the relevant environment variables for the image name, version, and weights path:
-```sh
-export IMAGE_NAME="modelsregistry.azurecr.io/phi-3-mini-4k-instruct:0.0.1"
-export VERSION="0.0.1"
-export WEIGHTS_PATH="kaito/phi-3-mini-4k-instruct/weights"
-```
-
-2. Build and Push the Docker Image
-
-Navigate to the KAITO base directory and build the Docker image, ensuring the weights directory is included in the build context:
-```sh
-docker build -t <IMAGE_NAME> --file docker/presets/workspace/models/tfs/Dockerfile --build-arg WEIGHTS_PATH=<WEIGHTS_PATH> --build-arg MODEL_TYPE=text-generation --build-arg VERSION=<VERSION> .
-
-docker push <IMAGE_NAME>
-```
-
-### Step 5: Deploy
-Follow the [Custom Template](../../examples/custom-model-integration/custom-deployment-template.yaml)
\ No newline at end of file
+- **Hugging Face runtime only**: Only HuggingFace runtime is supported for custom models. Inference performance may be slower compared to vLLM runtime if the model supports both.
+- **No multi-node inference**: Distributed inference across multiple nodes is not supported for custom model.
+- **No autoscaling**: KAITO autoscaling relies on metrics exposed by vLLM runtime, they are unavailable in HuggingFace runtime.
+- **No presets**: Users must manually modify the command line in the pod template for parameter changes.
diff --git a/website/versioned_docs/version-v0.5.x/custom-model.md b/website/versioned_docs/version-v0.5.x/custom-model.md
index f6955b8..14b33c5 100644
--- a/website/versioned_docs/version-v0.5.x/custom-model.md
+++ b/website/versioned_docs/version-v0.5.x/custom-model.md
@@ -2,82 +2,21 @@
 title: Custom Model Integration
 ---
 
-## Option 1: Use Pre-Built Docker Image Without Model Weights
-If you want to avoid building a Docker image with model weights, use our pre-built reference image (`ghcr.io/kaito-project/kaito/llm-reference-preset:latest`). This image, built with [Dockerfile.reference](../../../examples/custom-model-integration/Dockerfile.reference), dynamically downloads model weights from HuggingFace at runtime, reducing the need to create and maintain custom images.
+## Using the KAITO base image
 
+The KAITO base image includes both HuggingFace and vLLM runtime libraries along with corresponding FastAPI server scripts. This provides a convenient way to run **any** HuggingFace model using the **HuggingFace** runtime.
 
-- **[Sample Deployment YAML](../../../examples/custom-model-integration/reference-image-deployment.yaml)**
+Note that the vLLM runtime is not supported for arbitrary custom model deployment.
 
+Here is a **[sample deployment YAML](https://github.com/kaito-project/kaito/tree/main/examples/custom-model-integration/custom-model-deployment.yaml)**. To use it:
+1. Specify the HuggingFace model ID in the container command.
+2. For models that require a HuggingFace token to download, users need to add the token to the specified secret.
 
-## Option 2: Build a Custom Docker Image with Model Weights
+The script downloads model weights during server bootstrap, eliminating the need to pre-bake them into the container image.
 
-### Step 1: Clone the Repository
+## Limitations
 
-```sh
-git clone https://github.com/kaito-project/kaito.git
-```
-
-### Step 2: Download Your Private/Custom Model Weights
-
-For example, assuming HuggingFace weights:
-```sh
-git lfs install
-git clone git@hf.co:<MODEL_ID>  # Example: git clone git@hf.co:bigscience/bloom
-# OR
-git clone https://huggingface.co/bigscience/bloom
-```
-
-Alternatively, use curl:
-```
-curl -sSL https://huggingface.co/bigscience/bloom/resolve/main/config.json?download=true -o config.json
-```
-
-More information on downloading models from HuggingFace can be found [here](https://huggingface.co/docs/hub/en/models-downloading).
-
-
-### Step 3: Log In to Your Container Registry
-
-Before pushing the Docker image, ensure you’re logged into the appropriate container registry. Here are general login methods depending on the registry you use:
-
-1. GitHub Container Registry (ghcr.io):
-```sh
-echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin
-```
-Replace CR_PAT with your GitHub Personal Access Token and USERNAME with your GitHub username. This token should have the write:packages and read:packages permissions.
-
-2. Azure Container Registry (ACR): If you're using ACR:
-
-```sh
-az acr login --name <REGISTRY_NAME>
-```
-Replace `<REGISTRY_NAME>` with your Azure Container Registry name.
-
-3. Docker Hub or Other Container Registries:
-```sh
-docker login <REGISTRY_URL>
-```
-Enter your username and password when prompted. Replace `<REGISTRY_URL>` with your registry URL, such as `docker.io` for Docker Hub.
-
-
-### Step 4: Build Docker Image with Private/Custom Weights
-
-1. Set Environment Variables
-
-Before building the Docker image, set the relevant environment variables for the image name, version, and weights path:
-```sh
-export IMAGE_NAME="modelsregistry.azurecr.io/phi-3-mini-4k-instruct:0.0.1"
-export VERSION="0.0.1"
-export WEIGHTS_PATH="kaito/phi-3-mini-4k-instruct/weights"
-```
-
-2. Build and Push the Docker Image
-
-Navigate to the KAITO base directory and build the Docker image, ensuring the weights directory is included in the build context:
-```sh
-docker build -t <IMAGE_NAME> --file docker/presets/workspace/models/tfs/Dockerfile --build-arg WEIGHTS_PATH=<WEIGHTS_PATH> --build-arg MODEL_TYPE=text-generation --build-arg VERSION=<VERSION> .
-
-docker push <IMAGE_NAME>
-```
-
-### Step 5: Deploy
-Follow the [Custom Template](../../../examples/custom-model-integration/custom-deployment-template.yaml)
\ No newline at end of file
+- **Hugging Face runtime only**: Only HuggingFace runtime is supported for custom models. Inference performance may be slower compared to vLLM runtime if the model supports both.
+- **No multi-node inference**: Distributed inference across multiple nodes is not supported for custom model.
+- **No autoscaling**: KAITO autoscaling relies on metrics exposed by vLLM runtime, they are unavailable in HuggingFace runtime.
+- **No presets**: Users must manually modify the command line in the pod template for parameter changes.
diff --git a/website/versioned_docs/version-v0.6.x/custom-model.md b/website/versioned_docs/version-v0.6.x/custom-model.md
index f6955b8..14b33c5 100644
--- a/website/versioned_docs/version-v0.6.x/custom-model.md
+++ b/website/versioned_docs/version-v0.6.x/custom-model.md
@@ -2,82 +2,21 @@
 title: Custom Model Integration
 ---
 
-## Option 1: Use Pre-Built Docker Image Without Model Weights
-If you want to avoid building a Docker image with model weights, use our pre-built reference image (`ghcr.io/kaito-project/kaito/llm-reference-preset:latest`). This image, built with [Dockerfile.reference](../../../examples/custom-model-integration/Dockerfile.reference), dynamically downloads model weights from HuggingFace at runtime, reducing the need to create and maintain custom images.
+## Using the KAITO base image
 
+The KAITO base image includes both HuggingFace and vLLM runtime libraries along with corresponding FastAPI server scripts. This provides a convenient way to run **any** HuggingFace model using the **HuggingFace** runtime.
 
-- **[Sample Deployment YAML](../../../examples/custom-model-integration/reference-image-deployment.yaml)**
+Note that the vLLM runtime is not supported for arbitrary custom model deployment.
 
+Here is a **[sample deployment YAML](https://github.com/kaito-project/kaito/tree/main/examples/custom-model-integration/custom-model-deployment.yaml)**. To use it:
+1. Specify the HuggingFace model ID in the container command.
+2. For models that require a HuggingFace token to download, users need to add the token to the specified secret.
 
-## Option 2: Build a Custom Docker Image with Model Weights
+The script downloads model weights during server bootstrap, eliminating the need to pre-bake them into the container image.
 
-### Step 1: Clone the Repository
+## Limitations
 
-```sh
-git clone https://github.com/kaito-project/kaito.git
-```
-
-### Step 2: Download Your Private/Custom Model Weights
-
-For example, assuming HuggingFace weights:
-```sh
-git lfs install
-git clone git@hf.co:<MODEL_ID>  # Example: git clone git@hf.co:bigscience/bloom
-# OR
-git clone https://huggingface.co/bigscience/bloom
-```
-
-Alternatively, use curl:
-```
-curl -sSL https://huggingface.co/bigscience/bloom/resolve/main/config.json?download=true -o config.json
-```
-
-More information on downloading models from HuggingFace can be found [here](https://huggingface.co/docs/hub/en/models-downloading).
-
-
-### Step 3: Log In to Your Container Registry
-
-Before pushing the Docker image, ensure you’re logged into the appropriate container registry. Here are general login methods depending on the registry you use:
-
-1. GitHub Container Registry (ghcr.io):
-```sh
-echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin
-```
-Replace CR_PAT with your GitHub Personal Access Token and USERNAME with your GitHub username. This token should have the write:packages and read:packages permissions.
-
-2. Azure Container Registry (ACR): If you're using ACR:
-
-```sh
-az acr login --name <REGISTRY_NAME>
-```
-Replace `<REGISTRY_NAME>` with your Azure Container Registry name.
-
-3. Docker Hub or Other Container Registries:
-```sh
-docker login <REGISTRY_URL>
-```
-Enter your username and password when prompted. Replace `<REGISTRY_URL>` with your registry URL, such as `docker.io` for Docker Hub.
-
-
-### Step 4: Build Docker Image with Private/Custom Weights
-
-1. Set Environment Variables
-
-Before building the Docker image, set the relevant environment variables for the image name, version, and weights path:
-```sh
-export IMAGE_NAME="modelsregistry.azurecr.io/phi-3-mini-4k-instruct:0.0.1"
-export VERSION="0.0.1"
-export WEIGHTS_PATH="kaito/phi-3-mini-4k-instruct/weights"
-```
-
-2. Build and Push the Docker Image
-
-Navigate to the KAITO base directory and build the Docker image, ensuring the weights directory is included in the build context:
-```sh
-docker build -t <IMAGE_NAME> --file docker/presets/workspace/models/tfs/Dockerfile --build-arg WEIGHTS_PATH=<WEIGHTS_PATH> --build-arg MODEL_TYPE=text-generation --build-arg VERSION=<VERSION> .
-
-docker push <IMAGE_NAME>
-```
-
-### Step 5: Deploy
-Follow the [Custom Template](../../../examples/custom-model-integration/custom-deployment-template.yaml)
\ No newline at end of file
+- **Hugging Face runtime only**: Only HuggingFace runtime is supported for custom models. Inference performance may be slower compared to vLLM runtime if the model supports both.
+- **No multi-node inference**: Distributed inference across multiple nodes is not supported for custom model.
+- **No autoscaling**: KAITO autoscaling relies on metrics exposed by vLLM runtime, they are unavailable in HuggingFace runtime.
+- **No presets**: Users must manually modify the command line in the pod template for parameter changes.
diff --git a/website/versioned_docs/version-v0.7.x/custom-model.md b/website/versioned_docs/version-v0.7.x/custom-model.md
index f6955b8..14b33c5 100644
--- a/website/versioned_docs/version-v0.7.x/custom-model.md
+++ b/website/versioned_docs/version-v0.7.x/custom-model.md
@@ -2,82 +2,21 @@
 title: Custom Model Integration
 ---
 
-## Option 1: Use Pre-Built Docker Image Without Model Weights
-If you want to avoid building a Docker image with model weights, use our pre-built reference image (`ghcr.io/kaito-project/kaito/llm-reference-preset:latest`). This image, built with [Dockerfile.reference](../../../examples/custom-model-integration/Dockerfile.reference), dynamically downloads model weights from HuggingFace at runtime, reducing the need to create and maintain custom images.
+## Using the KAITO base image
 
+The KAITO base image includes both HuggingFace and vLLM runtime libraries along with corresponding FastAPI server scripts. This provides a convenient way to run **any** HuggingFace model using the **HuggingFace** runtime.
 
-- **[Sample Deployment YAML](../../../examples/custom-model-integration/reference-image-deployment.yaml)**
+Note that the vLLM runtime is not supported for arbitrary custom model deployment.
 
+Here is a **[sample deployment YAML](https://github.com/kaito-project/kaito/tree/main/examples/custom-model-integration/custom-model-deployment.yaml)**. To use it:
+1. Specify the HuggingFace model ID in the container command.
+2. For models that require a HuggingFace token to download, users need to add the token to the specified secret.
 
-## Option 2: Build a Custom Docker Image with Model Weights
+The script downloads model weights during server bootstrap, eliminating the need to pre-bake them into the container image.
 
-### Step 1: Clone the Repository
+## Limitations
 
-```sh
-git clone https://github.com/kaito-project/kaito.git
-```
-
-### Step 2: Download Your Private/Custom Model Weights
-
-For example, assuming HuggingFace weights:
-```sh
-git lfs install
-git clone git@hf.co:<MODEL_ID>  # Example: git clone git@hf.co:bigscience/bloom
-# OR
-git clone https://huggingface.co/bigscience/bloom
-```
-
-Alternatively, use curl:
-```
-curl -sSL https://huggingface.co/bigscience/bloom/resolve/main/config.json?download=true -o config.json
-```
-
-More information on downloading models from HuggingFace can be found [here](https://huggingface.co/docs/hub/en/models-downloading).
-
-
-### Step 3: Log In to Your Container Registry
-
-Before pushing the Docker image, ensure you’re logged into the appropriate container registry. Here are general login methods depending on the registry you use:
-
-1. GitHub Container Registry (ghcr.io):
-```sh
-echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin
-```
-Replace CR_PAT with your GitHub Personal Access Token and USERNAME with your GitHub username. This token should have the write:packages and read:packages permissions.
-
-2. Azure Container Registry (ACR): If you're using ACR:
-
-```sh
-az acr login --name <REGISTRY_NAME>
-```
-Replace `<REGISTRY_NAME>` with your Azure Container Registry name.
-
-3. Docker Hub or Other Container Registries:
-```sh
-docker login <REGISTRY_URL>
-```
-Enter your username and password when prompted. Replace `<REGISTRY_URL>` with your registry URL, such as `docker.io` for Docker Hub.
-
-
-### Step 4: Build Docker Image with Private/Custom Weights
-
-1. Set Environment Variables
-
-Before building the Docker image, set the relevant environment variables for the image name, version, and weights path:
-```sh
-export IMAGE_NAME="modelsregistry.azurecr.io/phi-3-mini-4k-instruct:0.0.1"
-export VERSION="0.0.1"
-export WEIGHTS_PATH="kaito/phi-3-mini-4k-instruct/weights"
-```
-
-2. Build and Push the Docker Image
-
-Navigate to the KAITO base directory and build the Docker image, ensuring the weights directory is included in the build context:
-```sh
-docker build -t <IMAGE_NAME> --file docker/presets/workspace/models/tfs/Dockerfile --build-arg WEIGHTS_PATH=<WEIGHTS_PATH> --build-arg MODEL_TYPE=text-generation --build-arg VERSION=<VERSION> .
-
-docker push <IMAGE_NAME>
-```
-
-### Step 5: Deploy
-Follow the [Custom Template](../../../examples/custom-model-integration/custom-deployment-template.yaml)
\ No newline at end of file
+- **Hugging Face runtime only**: Only HuggingFace runtime is supported for custom models. Inference performance may be slower compared to vLLM runtime if the model supports both.
+- **No multi-node inference**: Distributed inference across multiple nodes is not supported for custom model.
+- **No autoscaling**: KAITO autoscaling relies on metrics exposed by vLLM runtime, they are unavailable in HuggingFace runtime.
+- **No presets**: Users must manually modify the command line in the pod template for parameter changes.



Test patch:
diff --git a/api/v1alpha1/inferenceset_validation_test.go b/api/v1alpha1/inferenceset_validation_test.go
new file mode 100644
index 0000000..cf7ebb3
--- /dev/null
+++ b/api/v1alpha1/inferenceset_validation_test.go
@@ -0,0 +1,151 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package v1alpha1
+
+import (
+	"context"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	admissionregistrationv1 "k8s.io/api/admissionregistration/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+func TestInferenceSet_SupportedVerbs(t *testing.T) {
+	tests := []struct {
+		name     string
+		expected []admissionregistrationv1.OperationType
+	}{
+		{
+			name: "should return Create and Update operations",
+			expected: []admissionregistrationv1.OperationType{
+				admissionregistrationv1.Create,
+				admissionregistrationv1.Update,
+			},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			is := &InferenceSet{}
+			got := is.SupportedVerbs()
+			assert.Equal(t, tt.expected, got)
+			assert.Len(t, got, 2)
+			assert.Contains(t, got, admissionregistrationv1.Create)
+			assert.Contains(t, got, admissionregistrationv1.Update)
+		})
+	}
+}
+
+func TestInferenceSet_Validate(t *testing.T) {
+	tests := []struct {
+		name        string
+		inferencSet *InferenceSet
+		oldIS       *InferenceSet
+		wantErr     bool
+		errField    string
+	}{
+		{
+			name: "valid DNS1123 label name on create",
+			inferencSet: &InferenceSet{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "valid-name",
+					Namespace: "default",
+				},
+			},
+			oldIS:   nil,
+			wantErr: false,
+		},
+		{
+			name: "invalid DNS1123 label name with uppercase",
+			inferencSet: &InferenceSet{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "Invalid-Name",
+					Namespace: "default",
+				},
+			},
+			oldIS:    nil,
+			wantErr:  true,
+			errField: "name",
+		},
+		{
+			name: "invalid DNS1123 label name too long",
+			inferencSet: &InferenceSet{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "this-is-a-very-long-name-that-exceeds-the-maximum-allowed-length-for-dns",
+					Namespace: "default",
+				},
+			},
+			oldIS:    nil,
+			wantErr:  true,
+			errField: "name",
+		},
+		{
+			name: "invalid DNS1123 label name with special chars",
+			inferencSet: &InferenceSet{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "invalid_name",
+					Namespace: "default",
+				},
+			},
+			oldIS:    nil,
+			wantErr:  true,
+			errField: "name",
+		},
+		{
+			name: "valid DNS1123 label name on update",
+			inferencSet: &InferenceSet{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "valid-name",
+					Namespace: "default",
+				},
+			},
+			oldIS: &InferenceSet{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "valid-name",
+					Namespace: "default",
+				},
+			},
+			wantErr: false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			ctx := context.Background()
+			err := tt.inferencSet.Validate(ctx)
+			if tt.wantErr {
+				assert.NotNil(t, err)
+				if tt.errField != "" {
+					assert.Contains(t, err.Error(), tt.errField)
+				}
+			} else {
+				assert.Nil(t, err)
+			}
+		})
+	}
+}
+
+func TestInferenceSet_validateCreate(t *testing.T) {
+	is := &InferenceSet{}
+	err := is.validateCreate()
+	assert.Nil(t, err)
+}
+
+func TestInferenceSet_validateUpdate(t *testing.T) {
+	is := &InferenceSet{}
+	old := &InferenceSet{}
+	err := is.validateUpdate(old)
+	assert.Nil(t, err)
+}
diff --git a/pkg/utils/inferenceset/inferenceset_test.go b/pkg/utils/inferenceset/inferenceset_test.go
new file mode 100644
index 0000000..4722360
--- /dev/null
+++ b/pkg/utils/inferenceset/inferenceset_test.go
@@ -0,0 +1,941 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package inferenceset
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/mock"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/api/meta"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	kaitov1alpha1 "github.com/kaito-project/kaito/api/v1alpha1"
+	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
+	"github.com/kaito-project/kaito/pkg/utils/test"
+)
+
+func TestUpdateStatusConditionIfNotMatch(t *testing.T) {
+	t.Run("Should skip update when condition matches current values", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:       "test-inferenceset",
+				Namespace:  "default",
+				Generation: 1,
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{
+					{
+						Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+						Status:  metav1.ConditionTrue,
+						Reason:  "ResourcesReady",
+						Message: "All resources are ready",
+					},
+				},
+			},
+		}
+
+		ctx := context.Background()
+		err := UpdateStatusConditionIfNotMatch(ctx, mockClient, inferenceset,
+			kaitov1alpha1.ConditionTypeResourceStatus, metav1.ConditionTrue, "ResourcesReady", "All resources are ready")
+
+		assert.NoError(t, err)
+		// No client calls should be made since condition matches
+		mockClient.AssertExpectations(t)
+	})
+
+	t.Run("Should update when condition status differs", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:       "test-inferenceset",
+				Namespace:  "default",
+				Generation: 1,
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{
+					{
+						Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+						Status:  metav1.ConditionFalse,
+						Reason:  "ResourcesNotReady",
+						Message: "Resources are not ready",
+					},
+				},
+			},
+		}
+
+		// Mock the Get call for UpdateInferenceSetStatus
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(2).(*kaitov1alpha1.InferenceSet)
+			*ws = *inferenceset
+		}).Return(nil)
+
+		// Mock the Status().Update call
+		mockClient.StatusMock.On("Update", mock.IsType(context.Background()),
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(1).(*kaitov1alpha1.InferenceSet)
+			// Verify the condition was updated
+			condition := meta.FindStatusCondition(ws.Status.Conditions, string(kaitov1beta1.ConditionTypeResourceStatus))
+			assert.NotNil(t, condition)
+			assert.Equal(t, metav1.ConditionTrue, condition.Status)
+			assert.Equal(t, "ResourcesReady", condition.Reason)
+			assert.Equal(t, "All resources are ready", condition.Message)
+		}).Return(nil)
+
+		ctx := context.Background()
+		err := UpdateStatusConditionIfNotMatch(ctx, mockClient, inferenceset,
+			kaitov1alpha1.ConditionTypeResourceStatus, metav1.ConditionTrue, "ResourcesReady", "All resources are ready")
+
+		assert.NoError(t, err)
+		mockClient.AssertExpectations(t)
+		mockClient.StatusMock.AssertExpectations(t)
+	})
+
+	t.Run("Should update when condition reason differs", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:       "test-inferenceset",
+				Namespace:  "default",
+				Generation: 1,
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{
+					{
+						Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+						Status:  metav1.ConditionTrue,
+						Reason:  "OldReason",
+						Message: "All resources are ready",
+					},
+				},
+			},
+		}
+
+		// Mock the Get call for UpdateInferenceSetStatus
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(2).(*kaitov1alpha1.InferenceSet)
+			*ws = *inferenceset
+		}).Return(nil)
+
+		// Mock the Status().Update call
+		mockClient.StatusMock.On("Update", mock.IsType(context.Background()),
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Return(nil)
+
+		ctx := context.Background()
+		err := UpdateStatusConditionIfNotMatch(ctx, mockClient, inferenceset,
+			kaitov1alpha1.ConditionTypeResourceStatus, metav1.ConditionTrue, "NewReason", "All resources are ready")
+
+		assert.NoError(t, err)
+		mockClient.AssertExpectations(t)
+		mockClient.StatusMock.AssertExpectations(t)
+	})
+
+	t.Run("Should update when condition message differs", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:       "test-inferenceset",
+				Namespace:  "default",
+				Generation: 1,
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{
+					{
+						Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+						Status:  metav1.ConditionTrue,
+						Reason:  "ResourcesReady",
+						Message: "Old message",
+					},
+				},
+			},
+		}
+
+		// Mock the Get call for UpdateInferenceSetStatus
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(2).(*kaitov1alpha1.InferenceSet)
+			*ws = *inferenceset
+		}).Return(nil)
+
+		// Mock the Status().Update call
+		mockClient.StatusMock.On("Update", mock.IsType(context.Background()),
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Return(nil)
+
+		ctx := context.Background()
+		err := UpdateStatusConditionIfNotMatch(ctx, mockClient, inferenceset,
+			kaitov1alpha1.ConditionTypeResourceStatus, metav1.ConditionTrue, "ResourcesReady", "New message")
+
+		assert.NoError(t, err)
+		mockClient.AssertExpectations(t)
+		mockClient.StatusMock.AssertExpectations(t)
+	})
+
+	t.Run("Should add new condition when condition type doesn't exist", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:       "test-inferenceset",
+				Namespace:  "default",
+				Generation: 1,
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{}, // Empty conditions
+			},
+		}
+
+		// Mock the Get call for UpdateInferenceSetStatus
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(2).(*kaitov1alpha1.InferenceSet)
+			*ws = *inferenceset
+		}).Return(nil)
+
+		// Mock the Status().Update call
+		mockClient.StatusMock.On("Update", mock.IsType(context.Background()),
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(1).(*kaitov1alpha1.InferenceSet)
+			// Verify the condition was added
+			condition := meta.FindStatusCondition(ws.Status.Conditions, string(kaitov1beta1.ConditionTypeResourceStatus))
+			assert.NotNil(t, condition)
+			assert.Equal(t, metav1.ConditionTrue, condition.Status)
+			assert.Equal(t, "ResourcesReady", condition.Reason)
+			assert.Equal(t, "All resources are ready", condition.Message)
+		}).Return(nil)
+
+		ctx := context.Background()
+		err := UpdateStatusConditionIfNotMatch(ctx, mockClient, inferenceset,
+			kaitov1alpha1.ConditionTypeResourceStatus, metav1.ConditionTrue, "ResourcesReady", "All resources are ready")
+
+		assert.NoError(t, err)
+		mockClient.AssertExpectations(t)
+		mockClient.StatusMock.AssertExpectations(t)
+	})
+
+	t.Run("Should propagate error from UpdateInferenceSetStatus", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:       "test-inferenceset",
+				Namespace:  "default",
+				Generation: 1,
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{
+					{
+						Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+						Status:  metav1.ConditionFalse,
+						Reason:  "OldReason",
+						Message: "Old message",
+					},
+				},
+			},
+		}
+
+		// Mock the Get call to return an error
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Return(fmt.Errorf("get error"))
+
+		ctx := context.Background()
+		err := UpdateStatusConditionIfNotMatch(ctx, mockClient, inferenceset,
+			kaitov1alpha1.ConditionTypeResourceStatus, metav1.ConditionTrue, "NewReason", "New message")
+
+		assert.Error(t, err)
+		assert.Contains(t, err.Error(), "get error")
+		mockClient.AssertExpectations(t)
+	})
+}
+
+func TestUpdateInferenceSetStatus(t *testing.T) {
+	t.Run("Should successfully update inferenceset status", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:       "test-inferenceset",
+				Namespace:  "default",
+				Generation: 1,
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{},
+			},
+		}
+
+		condition := &metav1.Condition{
+			Type:               string(kaitov1beta1.ConditionTypeResourceStatus),
+			Status:             metav1.ConditionTrue,
+			Reason:             "ResourcesReady",
+			Message:            "All resources are ready",
+			LastTransitionTime: metav1.Now(),
+		}
+
+		// Mock the Get call
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(2).(*kaitov1alpha1.InferenceSet)
+			*ws = *inferenceset
+		}).Return(nil)
+
+		// Mock the Status().Update call
+		mockClient.StatusMock.On("Update", mock.IsType(context.Background()),
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(1).(*kaitov1alpha1.InferenceSet)
+			// Verify the condition was set
+			foundCondition := meta.FindStatusCondition(ws.Status.Conditions, string(kaitov1beta1.ConditionTypeResourceStatus))
+			assert.NotNil(t, foundCondition)
+			assert.Equal(t, condition.Type, foundCondition.Type)
+			assert.Equal(t, condition.Status, foundCondition.Status)
+			assert.Equal(t, condition.Reason, foundCondition.Reason)
+			assert.Equal(t, condition.Message, foundCondition.Message)
+		}).Return(nil)
+
+		ctx := context.Background()
+		key := &client.ObjectKey{Name: "test-inferenceset", Namespace: "default"}
+		err := UpdateInferenceSetStatus(ctx, mockClient, key, func(status *kaitov1alpha1.InferenceSetStatus) error {
+			meta.SetStatusCondition(&status.Conditions, *condition)
+			return nil
+		})
+
+		assert.NoError(t, err)
+		mockClient.AssertExpectations(t)
+		mockClient.StatusMock.AssertExpectations(t)
+	})
+
+	t.Run("Should handle inferenceset not found gracefully", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		condition := &metav1.Condition{
+			Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+			Status:  metav1.ConditionTrue,
+			Reason:  "ResourcesReady",
+			Message: "All resources are ready",
+		}
+
+		// Mock the Get call to return NotFound error
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Return(
+			apierrors.NewNotFound(schema.GroupResource{Group: "kaito.sh", Resource: "inferencesets"}, "test-inferenceset"))
+
+		ctx := context.Background()
+		key := &client.ObjectKey{Name: "test-inferenceset", Namespace: "default"}
+		err := UpdateInferenceSetStatus(ctx, mockClient, key, func(status *kaitov1alpha1.InferenceSetStatus) error {
+			meta.SetStatusCondition(&status.Conditions, *condition)
+			return nil
+		})
+
+		assert.NoError(t, err) // Should not return error for NotFound
+		mockClient.AssertExpectations(t)
+	})
+
+	t.Run("Should return error for other Get failures", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		condition := &metav1.Condition{
+			Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+			Status:  metav1.ConditionTrue,
+			Reason:  "ResourcesReady",
+			Message: "All resources are ready",
+		}
+
+		// Mock the Get call to return a generic error
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Return(fmt.Errorf("network error"))
+
+		ctx := context.Background()
+		key := &client.ObjectKey{Name: "test-inferenceset", Namespace: "default"}
+		err := UpdateInferenceSetStatus(ctx, mockClient, key, func(status *kaitov1alpha1.InferenceSetStatus) error {
+			meta.SetStatusCondition(&status.Conditions, *condition)
+			return nil
+		})
+
+		assert.Error(t, err)
+		assert.Contains(t, err.Error(), "network error")
+		mockClient.AssertExpectations(t)
+	})
+
+	t.Run("Should handle nil condition", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{},
+			},
+		}
+
+		// Mock the Get call
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(2).(*kaitov1alpha1.InferenceSet)
+			*ws = *inferenceset
+		}).Return(nil)
+
+		// Mock the Status().Update call
+		mockClient.StatusMock.On("Update", mock.IsType(context.Background()),
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Return(nil)
+
+		ctx := context.Background()
+		key := &client.ObjectKey{Name: "test-inferenceset", Namespace: "default"}
+		err := UpdateInferenceSetStatus(ctx, mockClient, key, nil)
+
+		assert.NoError(t, err)
+		mockClient.AssertExpectations(t)
+		mockClient.StatusMock.AssertExpectations(t)
+	})
+
+	t.Run("Should retry on retryable errors", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{},
+			},
+		}
+
+		condition := &metav1.Condition{
+			Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+			Status:  metav1.ConditionTrue,
+			Reason:  "ResourcesReady",
+			Message: "All resources are ready",
+		}
+
+		// Mock the Get call (multiple times due to retry)
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(2).(*kaitov1alpha1.InferenceSet)
+			*ws = *inferenceset
+		}).Return(nil)
+
+		// Mock the Status().Update call to fail first with a retryable error, then succeed
+		mockClient.StatusMock.On("Update", mock.IsType(context.Background()),
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Return(
+			apierrors.NewConflict(schema.GroupResource{Group: "kaito.sh", Resource: "inferencesets"}, "test-inferenceset", fmt.Errorf("conflict"))).Once()
+
+		mockClient.StatusMock.On("Update", mock.IsType(context.Background()),
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Return(nil).Once()
+
+		ctx := context.Background()
+		key := &client.ObjectKey{Name: "test-inferenceset", Namespace: "default"}
+		err := UpdateInferenceSetStatus(ctx, mockClient, key, func(status *kaitov1alpha1.InferenceSetStatus) error {
+			meta.SetStatusCondition(&status.Conditions, *condition)
+			return nil
+		})
+
+		assert.NoError(t, err)
+		mockClient.AssertExpectations(t)
+		mockClient.StatusMock.AssertExpectations(t)
+	})
+
+	t.Run("Should return error for non-retryable status update failures", func(t *testing.T) {
+		mockClient := test.NewClient()
+
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{},
+			},
+		}
+
+		condition := &metav1.Condition{
+			Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+			Status:  metav1.ConditionTrue,
+			Reason:  "ResourcesReady",
+			Message: "All resources are ready",
+		}
+
+		// Mock the Get call
+		mockClient.On("Get", mock.IsType(context.Background()),
+			client.ObjectKey{Name: "test-inferenceset", Namespace: "default"},
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Run(func(args mock.Arguments) {
+			ws := args.Get(2).(*kaitov1alpha1.InferenceSet)
+			*ws = *inferenceset
+		}).Return(nil)
+
+		// Mock the Status().Update call to fail with a non-retryable error
+		mockClient.StatusMock.On("Update", mock.IsType(context.Background()),
+			mock.IsType(&kaitov1alpha1.InferenceSet{}), mock.Anything).Return(fmt.Errorf("permanent error"))
+
+		ctx := context.Background()
+		key := &client.ObjectKey{Name: "test-inferenceset", Namespace: "default"}
+		err := UpdateInferenceSetStatus(ctx, mockClient, key, func(status *kaitov1alpha1.InferenceSetStatus) error {
+			meta.SetStatusCondition(&status.Conditions, *condition)
+			return nil
+		})
+
+		assert.Error(t, err)
+		assert.Contains(t, err.Error(), "permanent error")
+		mockClient.AssertExpectations(t)
+		mockClient.StatusMock.AssertExpectations(t)
+	})
+}
+
+func TestComputeInferenceSetHash(t *testing.T) {
+	t.Run("Should generate consistent hash for same InferenceSet", func(t *testing.T) {
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+				Labels: map[string]string{
+					"app": "test",
+				},
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+		}
+
+		hash1 := ComputeInferenceSetHash(inferenceset)
+		hash2 := ComputeInferenceSetHash(inferenceset)
+
+		assert.Equal(t, hash1, hash2)
+		assert.NotEmpty(t, hash1)
+		// SHA256 produces 64 character hex string
+		assert.Len(t, hash1, 64)
+	})
+
+	t.Run("Should generate same hashes for different ObjectMeta", func(t *testing.T) {
+		inferenceset1 := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset-1",
+				Namespace: "default",
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+		}
+
+		inferenceset2 := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset-2",
+				Namespace: "default",
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+		}
+
+		hash1 := ComputeInferenceSetHash(inferenceset1)
+		hash2 := ComputeInferenceSetHash(inferenceset2)
+
+		assert.Equal(t, hash1, hash2)
+	})
+
+	t.Run("Should generate different hashes for different Spec", func(t *testing.T) {
+		inferenceset1 := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+		}
+
+		inferenceset2 := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 5,
+			},
+		}
+
+		hash1 := ComputeInferenceSetHash(inferenceset1)
+		hash2 := ComputeInferenceSetHash(inferenceset2)
+
+		assert.NotEqual(t, hash1, hash2)
+	})
+
+	t.Run("Should ignore Status field changes", func(t *testing.T) {
+		inferenceset1 := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{},
+			},
+		}
+
+		inferenceset2 := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{
+					{
+						Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+						Status:  metav1.ConditionTrue,
+						Reason:  "Ready",
+						Message: "Resources are ready",
+					},
+				},
+			},
+		}
+
+		hash1 := ComputeInferenceSetHash(inferenceset1)
+		hash2 := ComputeInferenceSetHash(inferenceset2)
+
+		assert.Equal(t, hash1, hash2)
+	})
+
+	t.Run("Should handle nil InferenceSet gracefully", func(t *testing.T) {
+		// This test verifies that the function doesn't panic with nil input
+		assert.NotPanics(t, func() {
+			var inferenceset *kaitov1alpha1.InferenceSet
+			hash := ComputeInferenceSetHash(inferenceset)
+			assert.Empty(t, hash)
+		})
+	})
+
+	t.Run("Should generate same hashes for different label values", func(t *testing.T) {
+		inferenceset1 := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+				Labels: map[string]string{
+					"version": "v1",
+				},
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+		}
+
+		inferenceset2 := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+				Labels: map[string]string{
+					"version": "v2",
+				},
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+		}
+
+		hash1 := ComputeInferenceSetHash(inferenceset1)
+		hash2 := ComputeInferenceSetHash(inferenceset2)
+
+		assert.Equal(t, hash1, hash2)
+	})
+
+	t.Run("Should handle empty InferenceSet", func(t *testing.T) {
+		inferenceset := &kaitov1alpha1.InferenceSet{}
+
+		hash := ComputeInferenceSetHash(inferenceset)
+
+		assert.NotEmpty(t, hash)
+		assert.Len(t, hash, 64)
+	})
+}
+func TestMarshalInferenceSetFields(t *testing.T) {
+	t.Run("Should marshal InferenceSet fields successfully", func(t *testing.T) {
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+		}
+
+		jsonData, err := MarshalInferenceSetFields(inferenceset)
+
+		assert.NoError(t, err)
+		assert.NotNil(t, jsonData)
+
+		// Unmarshal to verify the content
+		var result map[string]interface{}
+		err = json.Unmarshal(jsonData, &result)
+		assert.NoError(t, err)
+
+		// Verify Spec content
+		spec, ok := result["Spec"].(map[string]interface{})
+		assert.True(t, ok)
+		assert.Equal(t, float64(3), spec["replicas"]) // JSON unmarshals numbers as float64
+	})
+
+	t.Run("Should return error for nil InferenceSet", func(t *testing.T) {
+		jsonData, err := MarshalInferenceSetFields(nil)
+
+		assert.Error(t, err)
+		assert.Nil(t, jsonData)
+		assert.Equal(t, "InferenceSet object is nil", err.Error())
+	})
+
+	t.Run("Should marshal empty InferenceSet", func(t *testing.T) {
+		inferenceset := &kaitov1alpha1.InferenceSet{}
+
+		jsonData, err := MarshalInferenceSetFields(inferenceset)
+
+		assert.NoError(t, err)
+		assert.NotNil(t, jsonData)
+
+		// Unmarshal to verify the content
+		var result map[string]interface{}
+		err = json.Unmarshal(jsonData, &result)
+		assert.NoError(t, err)
+
+		// Verify the structure
+		assert.Contains(t, result, "Spec")
+		assert.Len(t, result, 1) // Only Spec field should be present
+	})
+
+	t.Run("Should exclude Status field", func(t *testing.T) {
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+			Status: kaitov1alpha1.InferenceSetStatus{
+				Conditions: []metav1.Condition{
+					{
+						Type:    string(kaitov1beta1.ConditionTypeResourceStatus),
+						Status:  metav1.ConditionTrue,
+						Reason:  "Ready",
+						Message: "Resources are ready",
+					},
+				},
+			},
+		}
+
+		jsonData, err := MarshalInferenceSetFields(inferenceset)
+
+		assert.NoError(t, err)
+		assert.NotNil(t, jsonData)
+
+		// Unmarshal to verify the content
+		var result map[string]interface{}
+		err = json.Unmarshal(jsonData, &result)
+		assert.NoError(t, err)
+
+		// Verify Status is not included
+		assert.NotContains(t, result, "Status")
+		assert.Len(t, result, 1) // Only Spec
+	})
+
+	t.Run("Should marshal complex InferenceSet correctly", func(t *testing.T) {
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "complex-inferenceset",
+				Namespace: "production",
+				Labels: map[string]string{
+					"app":     "test",
+					"version": "v1",
+					"env":     "prod",
+				},
+				Annotations: map[string]string{
+					"description": "Test InferenceSet",
+					"owner":       "team-a",
+				},
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 5,
+			},
+		}
+
+		jsonData, err := MarshalInferenceSetFields(inferenceset)
+
+		assert.NoError(t, err)
+		assert.NotNil(t, jsonData)
+
+		// Unmarshal to verify the content
+		var result map[string]interface{}
+		err = json.Unmarshal(jsonData, &result)
+		assert.NoError(t, err)
+	})
+
+	t.Run("Should produce valid JSON output", func(t *testing.T) {
+		inferenceset := &kaitov1alpha1.InferenceSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-inferenceset",
+				Namespace: "default",
+			},
+			Spec: kaitov1alpha1.InferenceSetSpec{
+				Replicas: 3,
+			},
+		}
+
+		jsonData, err := MarshalInferenceSetFields(inferenceset)
+
+		assert.NoError(t, err)
+		assert.NotNil(t, jsonData)
+
+		// Verify it's valid JSON
+		assert.True(t, json.Valid(jsonData))
+	})
+}
+func TestGetWorkspace(t *testing.T) {
+	t.Run("Should return nil for empty workspace name", func(t *testing.T) {
+		workspaceList := &kaitov1beta1.WorkspaceList{
+			Items: []kaitov1beta1.Workspace{
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name: "workspace-1",
+					},
+				},
+			},
+		}
+
+		result := GetWorkspace("", workspaceList)
+		assert.Nil(t, result)
+	})
+
+	t.Run("Should return nil for nil workspace list", func(t *testing.T) {
+		result := GetWorkspace("workspace-1", nil)
+		assert.Nil(t, result)
+	})
+
+	t.Run("Should return workspace when found", func(t *testing.T) {
+		workspaceList := &kaitov1beta1.WorkspaceList{
+			Items: []kaitov1beta1.Workspace{
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "workspace-1",
+						Namespace: "default",
+					},
+				},
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "workspace-2",
+						Namespace: "default",
+					},
+				},
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "workspace-3",
+						Namespace: "default",
+					},
+				},
+			},
+		}
+
+		result := GetWorkspace("workspace-2", workspaceList)
+		assert.NotNil(t, result)
+		assert.Equal(t, "workspace-2", result.Name)
+		assert.Equal(t, "default", result.Namespace)
+	})
+
+	t.Run("Should return nil when workspace not found", func(t *testing.T) {
+		workspaceList := &kaitov1beta1.WorkspaceList{
+			Items: []kaitov1beta1.Workspace{
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name: "workspace-1",
+					},
+				},
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name: "workspace-2",
+					},
+				},
+			},
+		}
+
+		result := GetWorkspace("workspace-3", workspaceList)
+		assert.Nil(t, result)
+	})
+
+	t.Run("Should return nil for empty workspace list", func(t *testing.T) {
+		workspaceList := &kaitov1beta1.WorkspaceList{
+			Items: []kaitov1beta1.Workspace{},
+		}
+
+		result := GetWorkspace("workspace-1", workspaceList)
+		assert.Nil(t, result)
+	})
+
+	t.Run("Should return first matching workspace when duplicates exist", func(t *testing.T) {
+		workspaceList := &kaitov1beta1.WorkspaceList{
+			Items: []kaitov1beta1.Workspace{
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "workspace-1",
+						Namespace: "namespace-1",
+					},
+				},
+				{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "workspace-1",
+						Namespace: "namespace-2",
+					},
+				},
+			},
+		}
+
+		result := GetWorkspace("workspace-1", workspaceList)
+		assert.NotNil(t, result)
+		assert.Equal(t, "workspace-1", result.Name)
+		assert.Equal(t, "namespace-1", result.Namespace)
+	})
+
+	t.Run("Should handle both empty workspace name and nil list", func(t *testing.T) {
+		result := GetWorkspace("", nil)
+		assert.Nil(t, result)
+	})
+}
+
+func TestListWorkspaces(t *testing.T) {
+	t.Run("Should return error for nil InferenceSet", func(t *testing.T) {
+		mockClient := test.NewClient()
+		ctx := context.Background()
+
+		workspaceList, err := ListWorkspaces(ctx, nil, mockClient)
+
+		assert.Error(t, err)
+		assert.Nil(t, workspaceList)
+		assert.Equal(t, "InferenceSet object is nil", err.Error())
+		mockClient.AssertExpectations(t)
+	})
+}
diff --git a/pkg/utils/test/test_utils.go b/pkg/utils/test/test_utils.go
index b3c78d4..a9afceb 100644
--- a/pkg/utils/test/test_utils.go
+++ b/pkg/utils/test/test_utils.go
@@ -471,6 +471,31 @@ var (
 			},
 		},
 	}
+
+	MockInferenceSetWithDeleteOldCR = v1alpha1.InferenceSet{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "testInferenceSet",
+			Namespace: "kaito",
+			Annotations: map[string]string{
+				"inferenceset.kaito.sh/hash":     "1171dc5d15043c92e684c8f06689eb241763a735181fdd2b59c8bd8fd6eecdd4",
+				"inferenceset.kaito.sh/revision": "1",
+			},
+		},
+		Spec: v1alpha1.InferenceSetSpec{
+			Template: v1alpha1.InferenceSetTemplate{
+				Resource: v1alpha1.InferenceSetResourceSpec{
+					InstanceType: "Standard_NC12s_v3",
+				},
+				Inference: v1beta1.InferenceSpec{
+					Preset: &v1beta1.PresetSpec{
+						PresetMeta: v1beta1.PresetMeta{
+							Name: "test-model",
+						},
+					},
+				},
+			},
+		},
+	}
 )
 
 var (
@@ -528,6 +553,30 @@ var (
 			},
 		},
 	}
+
+	MockInferenceSetFailToCreateCR = v1alpha1.InferenceSet{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "testInferenceSet-failedtocreateCR",
+			Namespace: "kaito",
+			Annotations: map[string]string{
+				"inferenceset.kaito.sh/revision": "1",
+			},
+		},
+		Spec: v1alpha1.InferenceSetSpec{
+			Template: v1alpha1.InferenceSetTemplate{
+				Resource: v1alpha1.InferenceSetResourceSpec{
+					InstanceType: "Standard_NC12s_v3",
+				},
+				Inference: v1beta1.InferenceSpec{
+					Preset: &v1beta1.PresetSpec{
+						PresetMeta: v1beta1.PresetMeta{
+							Name: "test-model",
+						},
+					},
+				},
+			},
+		},
+	}
 )
 
 var (
@@ -578,6 +627,30 @@ var (
 			},
 		},
 	}
+
+	MockInferenceSetSuccessful = v1alpha1.InferenceSet{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "testInferenceSet-successful",
+			Namespace: "kaito",
+			Annotations: map[string]string{
+				"inferenceset.kaito.sh/revision": "0",
+			},
+		},
+		Spec: v1alpha1.InferenceSetSpec{
+			Template: v1alpha1.InferenceSetTemplate{
+				Resource: v1alpha1.InferenceSetResourceSpec{
+					InstanceType: "Standard_NC12s_v3",
+				},
+				Inference: v1beta1.InferenceSpec{
+					Preset: &v1beta1.PresetSpec{
+						PresetMeta: v1beta1.PresetMeta{
+							Name: "test-model",
+						},
+					},
+				},
+			},
+		},
+	}
 )
 
 var (
@@ -629,6 +702,31 @@ var (
 			},
 		},
 	}
+
+	MockInferenceSetWithComputeHash = v1alpha1.InferenceSet{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "testInferenceSet",
+			Namespace: "kaito",
+			Annotations: map[string]string{
+				"inferenceset.kaito.sh/hash":     "be5369aea1bec8fc674d900b229888f4b17739e647ca53509953bf2f1f2c0121",
+				"inferenceset.kaito.sh/revision": "1",
+			},
+		},
+		Spec: v1alpha1.InferenceSetSpec{
+			Template: v1alpha1.InferenceSetTemplate{
+				Resource: v1alpha1.InferenceSetResourceSpec{
+					InstanceType: "Standard_NC12s_v3",
+				},
+				Inference: v1beta1.InferenceSpec{
+					Preset: &v1beta1.PresetSpec{
+						PresetMeta: v1beta1.PresetMeta{
+							Name: "test-model",
+						},
+					},
+				},
+			},
+		},
+	}
 )
 
 var (
@@ -681,6 +779,31 @@ var (
 			},
 		},
 	}
+
+	MockInferenceSetUpdateCR = v1alpha1.InferenceSet{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "testInferenceSet",
+			Namespace: "kaito",
+			Annotations: map[string]string{
+				"inferenceset.kaito.sh/hash":     "1171dc5d15043c92e684c8f06689eb241763a735181fdd2b59c8bd8fd6eecdd4",
+				"inferenceset.kaito.sh/revision": "1",
+			},
+		},
+		Spec: v1alpha1.InferenceSetSpec{
+			Template: v1alpha1.InferenceSetTemplate{
+				Resource: v1alpha1.InferenceSetResourceSpec{
+					InstanceType: "Standard_NC12s_v3",
+				},
+				Inference: v1beta1.InferenceSpec{
+					Preset: &v1beta1.PresetSpec{
+						PresetMeta: v1beta1.PresetMeta{
+							Name: "test-model",
+						},
+					},
+				},
+			},
+		},
+	}
 )
 
 var (
@@ -719,6 +842,40 @@ var (
 			},
 		},
 	}
+
+	MockInferenceSetWithUpdatedDeployment = v1alpha1.InferenceSet{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "testInferenceSet",
+			Namespace: "kaito",
+			Annotations: map[string]string{
+				"inferenceset.kaito.sh/hash":     "1171dc5d15043c92e684c8f06689eb241763a735181fdd2b59c8bd8fd6eecdd4",
+				"inferenceset.kaito.sh/revision": "1",
+			},
+		},
+		Spec: v1alpha1.InferenceSetSpec{
+			Template: v1alpha1.InferenceSetTemplate{
+				Resource: v1alpha1.InferenceSetResourceSpec{
+					InstanceType: "Standard_NC12s_v3",
+				},
+				Inference: v1beta1.InferenceSpec{
+					Preset: &v1beta1.PresetSpec{
+						PresetMeta: v1beta1.PresetMeta{
+							Name: "test-model",
+						},
+					},
+					Adapters: []v1beta1.AdapterSpec{
+						{
+							Source: &v1beta1.DataSource{
+								Name:  "Adapter-1",
+								Image: "fake.kaito.com/kaito-image:0.0.1",
+							},
+							Strength: &ValidStrength,
+						},
+					},
+				},
+			},
+		},
+	}
 )
 
 var (
diff --git a/pkg/utils/workspace/workspace_test.go b/pkg/utils/workspace/workspace_test.go
index 343771a..20994ec 100644
--- a/pkg/utils/workspace/workspace_test.go
+++ b/pkg/utils/workspace/workspace_test.go
@@ -16,7 +16,9 @@ package workspace
 import (
 	"context"
 	"fmt"
+	"strings"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/mock"
@@ -511,3 +513,206 @@ func TestUpdateWorkspaceStatus(t *testing.T) {
 		mockClient.StatusMock.AssertExpectations(t)
 	})
 }
+
+func TestGenerateRandomString(t *testing.T) {
+	t.Run("Should return empty string for zero length", func(t *testing.T) {
+		result := generateRandomString(0)
+		assert.Equal(t, "", result)
+	})
+
+	t.Run("Should return empty string for negative length", func(t *testing.T) {
+		result := generateRandomString(-5)
+		assert.Equal(t, "", result)
+	})
+
+	t.Run("Should generate string of correct length", func(t *testing.T) {
+		lengths := []int{1, 5, 10, 20, 50}
+		for _, length := range lengths {
+			result := generateRandomString(length)
+			assert.Len(t, result, length)
+		}
+	})
+
+	t.Run("Should only contain lowercase letters and digits", func(t *testing.T) {
+		result := generateRandomString(100)
+		for _, char := range result {
+			assert.True(t, (char >= 'a' && char <= 'z') || (char >= '0' && char <= '9'),
+				"Character '%c' is not a lowercase letter or digit", char)
+		}
+	})
+
+	t.Run("Should generate different strings on consecutive calls", func(t *testing.T) {
+		// Generate multiple strings and check they're not all the same
+		// Note: There's a very small chance this could fail due to randomness
+		strings := make([]string, 10)
+		for i := range strings {
+			strings[i] = generateRandomString(10)
+			time.Sleep(time.Nanosecond) // Ensure different timestamps
+		}
+
+		// Check that not all strings are identical
+		firstString := strings[0]
+		allSame := true
+		for _, s := range strings[1:] {
+			if s != firstString {
+				allSame = false
+				break
+			}
+		}
+		assert.False(t, allSame, "All generated strings are identical, which is highly unlikely")
+	})
+
+	t.Run("Should handle large length values", func(t *testing.T) {
+		result := generateRandomString(1000)
+		assert.Len(t, result, 1000)
+
+		// Verify all characters are valid
+		for _, char := range result {
+			assert.True(t, (char >= 'a' && char <= 'z') || (char >= '0' && char <= '9'))
+		}
+	})
+
+	t.Run("Should generate string with expected character set distribution", func(t *testing.T) {
+		// Generate a long string to check character distribution
+		result := generateRandomString(10000)
+
+		hasLetter := false
+		hasDigit := false
+
+		for _, char := range result {
+			if char >= 'a' && char <= 'z' {
+				hasLetter = true
+			}
+			if char >= '0' && char <= '9' {
+				hasDigit = true
+			}
+			if hasLetter && hasDigit {
+				break
+			}
+		}
+
+		// With 10000 characters, it's virtually certain to have both letters and digits
+		assert.True(t, hasLetter, "Generated string should contain at least one letter")
+		assert.True(t, hasDigit, "Generated string should contain at least one digit")
+	})
+}
+
+func TestGetWorkspaceNameWithRandomSuffix(t *testing.T) {
+	t.Run("Should generate workspace name with correct format", func(t *testing.T) {
+		baseName := "test-workspace"
+		result := GetWorkspaceNameWithRandomSuffix(baseName)
+
+		// Check that the result starts with the base name
+		assert.True(t, strings.HasPrefix(result, baseName+"-"),
+			"Result should start with base name followed by hyphen")
+
+		// Check total length
+		expectedLength := len(baseName) + 1 + WorkspaceNameSuffixLength // +1 for hyphen
+		assert.Len(t, result, expectedLength)
+
+		// Extract and validate the suffix
+		suffix := result[len(baseName)+1:]
+		assert.Len(t, suffix, WorkspaceNameSuffixLength)
+
+		// Verify suffix contains only lowercase letters and digits
+		for _, char := range suffix {
+			assert.True(t, (char >= 'a' && char <= 'z') || (char >= '0' && char <= '9'),
+				"Suffix character '%c' is not a lowercase letter or digit", char)
+		}
+	})
+
+	t.Run("Should handle empty base name", func(t *testing.T) {
+		baseName := ""
+		result := GetWorkspaceNameWithRandomSuffix(baseName)
+
+		// Should be just hyphen plus random suffix
+		assert.True(t, strings.HasPrefix(result, "-"))
+		assert.Len(t, result, 1+WorkspaceNameSuffixLength)
+
+		// Verify the suffix part
+		suffix := result[1:]
+		assert.Len(t, suffix, WorkspaceNameSuffixLength)
+		for _, char := range suffix {
+			assert.True(t, (char >= 'a' && char <= 'z') || (char >= '0' && char <= '9'))
+		}
+	})
+
+	t.Run("Should handle base name with special characters", func(t *testing.T) {
+		baseNames := []string{
+			"test-workspace-123",
+			"test.workspace",
+			"test_workspace",
+			"test/workspace",
+			"test@workspace",
+		}
+
+		for _, baseName := range baseNames {
+			result := GetWorkspaceNameWithRandomSuffix(baseName)
+			assert.True(t, strings.HasPrefix(result, baseName+"-"))
+			assert.Len(t, result, len(baseName)+1+WorkspaceNameSuffixLength)
+		}
+	})
+
+	t.Run("Should generate different suffixes for same base name", func(t *testing.T) {
+		baseName := "test-workspace"
+		results := make([]string, 10)
+
+		for i := range results {
+			results[i] = GetWorkspaceNameWithRandomSuffix(baseName)
+			time.Sleep(time.Nanosecond) // Ensure different timestamps
+		}
+
+		// Check that not all results are identical
+		firstResult := results[0]
+		allSame := true
+		for _, r := range results[1:] {
+			if r != firstResult {
+				allSame = false
+				break
+			}
+		}
+		assert.False(t, allSame, "Should generate different suffixes for the same base name")
+
+		// All should have the same prefix though
+		for _, r := range results {
+			assert.True(t, strings.HasPrefix(r, baseName+"-"))
+		}
+	})
+
+	t.Run("Should handle very long base name", func(t *testing.T) {
+		baseName := strings.Repeat("a", 1000)
+		result := GetWorkspaceNameWithRandomSuffix(baseName)
+
+		assert.True(t, strings.HasPrefix(result, baseName+"-"))
+		assert.Len(t, result, len(baseName)+1+WorkspaceNameSuffixLength)
+	})
+
+	t.Run("Should handle Unicode characters in base name", func(t *testing.T) {
+		baseName := "..-workspace-.."
+		result := GetWorkspaceNameWithRandomSuffix(baseName)
+
+		assert.True(t, strings.HasPrefix(result, baseName+"-"))
+		// Note: len() counts bytes, not runes for Unicode
+		expectedSuffixStart := len(baseName) + 1
+		suffix := result[expectedSuffixStart:]
+
+		// The suffix length should still be WorkspaceNameSuffixLength
+		assert.Len(t, suffix, WorkspaceNameSuffixLength)
+
+		// Verify suffix contains only ASCII lowercase letters and digits
+		for _, char := range suffix {
+			assert.True(t, (char >= 'a' && char <= 'z') || (char >= '0' && char <= '9'))
+		}
+	})
+
+	t.Run("Should maintain consistent suffix length", func(t *testing.T) {
+		baseNames := []string{"short", "medium-length", "very-long-base-name-for-testing"}
+
+		for _, baseName := range baseNames {
+			result := GetWorkspaceNameWithRandomSuffix(baseName)
+			suffix := result[len(baseName)+1:]
+			assert.Len(t, suffix, WorkspaceNameSuffixLength,
+				"Suffix length should always be %d regardless of base name", WorkspaceNameSuffixLength)
+		}
+	})
+}
diff --git a/pkg/workspace/controllers/inferenceset/inferenceset_controller_test.go b/pkg/workspace/controllers/inferenceset/inferenceset_controller_test.go
new file mode 100644
index 0000000..0792ee4
--- /dev/null
+++ b/pkg/workspace/controllers/inferenceset/inferenceset_controller_test.go
@@ -0,0 +1,244 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package inferenceset
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/mock"
+	appsv1 "k8s.io/api/apps/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	"github.com/kaito-project/kaito/api/v1alpha1"
+	"github.com/kaito-project/kaito/pkg/utils/consts"
+	"github.com/kaito-project/kaito/pkg/utils/test"
+)
+
+func TestInferenceSetSyncControllerRevision(t *testing.T) {
+	testcases := map[string]struct {
+		callMocks     func(c *test.MockClient)
+		inferenceset  v1alpha1.InferenceSet
+		expectedError error
+		verifyCalls   func(c *test.MockClient)
+	}{
+
+		"No new revision needed": {
+			callMocks: func(c *test.MockClient) {
+				c.On("List", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevisionList{}), mock.Anything, mock.Anything).Return(nil)
+				c.On("Get", mock.IsType(context.Background()), mock.Anything, mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).
+					Run(func(args mock.Arguments) {
+						dep := args.Get(2).(*appsv1.ControllerRevision)
+						*dep = appsv1.ControllerRevision{
+							ObjectMeta: v1.ObjectMeta{
+								Annotations: map[string]string{
+									InferenceSetHashAnnotation: "be5369aea1bec8fc674d900b229888f4b17739e647ca53509953bf2f1f2c0121",
+								},
+							},
+							Revision: 1,
+						}
+					}).
+					Return(nil)
+				// Add mock for inferenceset retrieval in updateInferenceSetWithRetry
+				c.On("Get", mock.IsType(context.Background()), mock.Anything, mock.IsType(&v1alpha1.InferenceSet{}), mock.Anything).
+					Run(func(args mock.Arguments) {
+						ws := args.Get(2).(*v1alpha1.InferenceSet)
+						*ws = test.MockInferenceSetWithComputeHash
+					}).
+					Return(nil)
+				c.On("Update", mock.IsType(context.Background()), mock.IsType(&v1alpha1.InferenceSet{}), mock.Anything).
+					Return(nil)
+			},
+			inferenceset:  test.MockInferenceSetWithComputeHash,
+			expectedError: nil,
+			verifyCalls: func(c *test.MockClient) {
+				c.AssertNumberOfCalls(t, "List", 1)
+				c.AssertNumberOfCalls(t, "Create", 0)
+				c.AssertNumberOfCalls(t, "Get", 2) // 1 for ControllerRevision, 1 for InferenceSet
+				c.AssertNumberOfCalls(t, "Delete", 0)
+				c.AssertNumberOfCalls(t, "Update", 1)
+			},
+		},
+
+		"Fail to create ControllerRevision": {
+			callMocks: func(c *test.MockClient) {
+				c.On("List", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevisionList{}), mock.Anything, mock.Anything).Return(nil)
+				c.On("Create", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).Return(errors.New("failed to create ControllerRevision"))
+				c.On("Get", mock.IsType(context.Background()), mock.Anything, mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).
+					Return(apierrors.NewNotFound(appsv1.Resource("ControllerRevision"), test.MockInferenceSetFailToCreateCR.Name))
+			},
+			inferenceset:  test.MockInferenceSetFailToCreateCR,
+			expectedError: errors.New("failed to create new ControllerRevision: failed to create ControllerRevision"),
+			verifyCalls: func(c *test.MockClient) {
+				c.AssertNumberOfCalls(t, "List", 1)
+				c.AssertNumberOfCalls(t, "Create", 1)
+				c.AssertNumberOfCalls(t, "Get", 1)
+				c.AssertNumberOfCalls(t, "Delete", 0)
+				c.AssertNumberOfCalls(t, "Update", 0)
+			},
+		},
+
+		"Successfully create new ControllerRevision": {
+			callMocks: func(c *test.MockClient) {
+				c.On("List", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevisionList{}), mock.Anything, mock.Anything).Return(nil)
+				c.On("Create", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).Return(nil)
+				c.On("Get", mock.IsType(context.Background()), mock.Anything, mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).
+					Return(apierrors.NewNotFound(appsv1.Resource("ControllerRevision"), test.MockInferenceSetFailToCreateCR.Name))
+				// Add mock for inferenceset retrieval in updateInferenceSetWithRetry
+				c.On("Get", mock.IsType(context.Background()), mock.Anything, mock.IsType(&v1alpha1.InferenceSet{}), mock.Anything).
+					Run(func(args mock.Arguments) {
+						ws := args.Get(2).(*v1alpha1.InferenceSet)
+						*ws = test.MockInferenceSetSuccessful
+					}).
+					Return(nil)
+				c.On("Update", mock.IsType(context.Background()), mock.IsType(&v1alpha1.InferenceSet{}), mock.Anything).
+					Return(nil)
+			},
+			inferenceset:  test.MockInferenceSetSuccessful,
+			expectedError: nil,
+			verifyCalls: func(c *test.MockClient) {
+				c.AssertNumberOfCalls(t, "List", 1)
+				c.AssertNumberOfCalls(t, "Create", 1)
+				c.AssertNumberOfCalls(t, "Get", 2) // 1 for ControllerRevision, 1 for InferenceSet
+				c.AssertNumberOfCalls(t, "Delete", 0)
+				c.AssertNumberOfCalls(t, "Update", 1)
+			},
+		},
+
+		"Successfully delete old ControllerRevision": {
+			callMocks: func(c *test.MockClient) {
+				revisions := &appsv1.ControllerRevisionList{}
+				jsonData, _ := json.Marshal(test.MockInferenceSetWithUpdatedDeployment)
+
+				for i := 0; i <= consts.MaxRevisionHistoryLimit; i++ {
+					revision := &appsv1.ControllerRevision{
+						ObjectMeta: v1.ObjectMeta{
+							Name: fmt.Sprintf("revision-%d", i),
+						},
+						Revision: int64(i),
+						Data:     runtime.RawExtension{Raw: jsonData},
+					}
+					revisions.Items = append(revisions.Items, *revision)
+				}
+				relevantMap := c.CreateMapWithType(revisions)
+
+				for _, obj := range revisions.Items {
+					m := obj
+					objKey := client.ObjectKeyFromObject(&m)
+					relevantMap[objKey] = &m
+				}
+				c.On("List", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevisionList{}), mock.Anything, mock.Anything).Return(nil)
+				c.On("Create", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).Return(nil)
+				c.On("Get", mock.IsType(context.Background()), mock.Anything, mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).
+					Return(apierrors.NewNotFound(appsv1.Resource("ControllerRevision"), test.MockInferenceSetFailToCreateCR.Name))
+				c.On("Delete", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).Return(nil)
+				// Add mock for inferenceset retrieval in updateInferenceSetWithRetry
+				c.On("Get", mock.IsType(context.Background()), mock.Anything, mock.IsType(&v1alpha1.InferenceSet{}), mock.Anything).
+					Run(func(args mock.Arguments) {
+						ws := args.Get(2).(*v1alpha1.InferenceSet)
+						*ws = test.MockInferenceSetWithDeleteOldCR
+					}).
+					Return(nil)
+				c.On("Update", mock.IsType(context.Background()), mock.IsType(&v1alpha1.InferenceSet{}), mock.Anything).
+					Return(nil)
+			},
+			inferenceset:  test.MockInferenceSetWithDeleteOldCR,
+			expectedError: nil,
+			verifyCalls: func(c *test.MockClient) {
+				c.AssertNumberOfCalls(t, "List", 1)
+				c.AssertNumberOfCalls(t, "Create", 1)
+				c.AssertNumberOfCalls(t, "Get", 2) // 1 for ControllerRevision, 1 for InferenceSet
+				c.AssertNumberOfCalls(t, "Delete", 1)
+				c.AssertNumberOfCalls(t, "Update", 1)
+			},
+		},
+
+		"Fail to update InferenceSet annotations": {
+			callMocks: func(c *test.MockClient) {
+				revisions := &appsv1.ControllerRevisionList{}
+				jsonData, _ := json.Marshal(test.MockInferenceSetWithUpdatedDeployment)
+
+				for i := 0; i <= consts.MaxRevisionHistoryLimit; i++ {
+					revision := &appsv1.ControllerRevision{
+						ObjectMeta: v1.ObjectMeta{
+							Name: fmt.Sprintf("revision-%d", i),
+						},
+						Revision: int64(i),
+						Data:     runtime.RawExtension{Raw: jsonData},
+					}
+					revisions.Items = append(revisions.Items, *revision)
+				}
+				relevantMap := c.CreateMapWithType(revisions)
+
+				for _, obj := range revisions.Items {
+					m := obj
+					objKey := client.ObjectKeyFromObject(&m)
+					relevantMap[objKey] = &m
+				}
+				c.On("List", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevisionList{}), mock.Anything, mock.Anything).Return(nil)
+				c.On("Create", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).Return(nil)
+				c.On("Get", mock.IsType(context.Background()), mock.Anything, mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).
+					Return(apierrors.NewNotFound(appsv1.Resource("ControllerRevision"), test.MockInferenceSetFailToCreateCR.Name))
+				c.On("Delete", mock.IsType(context.Background()), mock.IsType(&appsv1.ControllerRevision{}), mock.Anything).Return(nil)
+				// Add mock for inferenceset retrieval in updateInferenceSetWithRetry
+				c.On("Get", mock.IsType(context.Background()), mock.Anything, mock.IsType(&v1alpha1.InferenceSet{}), mock.Anything).
+					Run(func(args mock.Arguments) {
+						ws := args.Get(2).(*v1alpha1.InferenceSet)
+						*ws = test.MockInferenceSetUpdateCR
+					}).
+					Return(nil)
+				c.On("Update", mock.IsType(context.Background()), mock.IsType(&v1alpha1.InferenceSet{}), mock.Anything).
+					Return(fmt.Errorf("failed to update InferenceSet annotations"))
+			},
+			inferenceset:  test.MockInferenceSetUpdateCR,
+			expectedError: fmt.Errorf("failed to update InferenceSet annotations: %w", fmt.Errorf("failed to update InferenceSet annotations")),
+			verifyCalls: func(c *test.MockClient) {
+				c.AssertNumberOfCalls(t, "List", 1)
+				c.AssertNumberOfCalls(t, "Create", 1)
+				c.AssertNumberOfCalls(t, "Get", 2) // 1 for ControllerRevision, 1 for InferenceSet
+				c.AssertNumberOfCalls(t, "Delete", 1)
+				c.AssertNumberOfCalls(t, "Update", 1)
+			},
+		},
+	}
+	for k, tc := range testcases {
+		t.Run(k, func(t *testing.T) {
+			mockClient := test.NewClient()
+			tc.callMocks(mockClient)
+
+			reconciler := &InferenceSetReconciler{
+				Client: mockClient,
+				Scheme: test.NewTestScheme(),
+			}
+			ctx := context.Background()
+
+			err := reconciler.syncControllerRevision(ctx, &tc.inferenceset)
+			if tc.expectedError == nil {
+				assert.NoError(t, err)
+			} else {
+				assert.Equal(t, tc.expectedError.Error(), err.Error())
+			}
+			if tc.verifyCalls != nil {
+				tc.verifyCalls(mockClient)
+			}
+		})
+	}
+}
diff --git a/pkg/workspace/controllers/inferenceset/inferenceset_event_handler_test.go b/pkg/workspace/controllers/inferenceset/inferenceset_event_handler_test.go
new file mode 100644
index 0000000..0ab5b58
--- /dev/null
+++ b/pkg/workspace/controllers/inferenceset/inferenceset_event_handler_test.go
@@ -0,0 +1,63 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package inferenceset
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	"github.com/kaito-project/kaito/api/v1beta1"
+	"github.com/kaito-project/kaito/pkg/utils/consts"
+)
+
+func TestGetControllerKeyForWorkspace(t *testing.T) {
+	tests := []struct {
+		name      string
+		workspace *v1beta1.Workspace
+		expected  *client.ObjectKey
+	}{
+		{
+			name: "Workspace with valid label",
+			workspace: &v1beta1.Workspace{
+				ObjectMeta: metav1.ObjectMeta{
+					Namespace: "test-namespace",
+					Labels: map[string]string{
+						consts.WorkspaceCreatedByInferenceSetLabel: "test-name",
+					},
+				},
+			},
+			expected: &client.ObjectKey{Namespace: "test-namespace", Name: "test-name"},
+		},
+		{
+			name: "Workspace without label",
+			workspace: &v1beta1.Workspace{
+				ObjectMeta: metav1.ObjectMeta{
+					Namespace: "test-namespace",
+					Labels:    map[string]string{},
+				},
+			},
+			expected: nil,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := getControllerKeyForWorkspace(tt.workspace)
+			assert.Equal(t, tt.expected, result)
+		})
+	}
+}
diff --git a/pkg/workspace/controllers/inferenceset/metrics_test.go b/pkg/workspace/controllers/inferenceset/metrics_test.go
new file mode 100644
index 0000000..0711065
--- /dev/null
+++ b/pkg/workspace/controllers/inferenceset/metrics_test.go
@@ -0,0 +1,122 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package inferenceset
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	kaitov1alpha1 "github.com/kaito-project/kaito/api/v1alpha1"
+)
+
+func TestDetermineInferenceSetPhase(t *testing.T) {
+	tests := []struct {
+		name     string
+		is       *kaitov1alpha1.InferenceSet
+		expected string
+	}{
+		{
+			name: "inferenceset is deleting",
+			is: &kaitov1alpha1.InferenceSet{
+				Status: kaitov1alpha1.InferenceSetStatus{
+					Conditions: []metav1.Condition{
+						{
+							Type:   string(kaitov1alpha1.InferenceSetConditionTypeDeleting),
+							Status: metav1.ConditionTrue,
+						},
+					},
+				},
+			},
+			expected: "deleting",
+		},
+		{
+			name: "inferenceset ready",
+			is: &kaitov1alpha1.InferenceSet{
+				Status: kaitov1alpha1.InferenceSetStatus{
+					Conditions: []metav1.Condition{
+						{
+							Type:   string(kaitov1alpha1.InferenceSetConditionTypeReady),
+							Status: metav1.ConditionTrue,
+						},
+					},
+				},
+			},
+			expected: "ready",
+		},
+		{
+			name: "inferenceset error",
+			is: &kaitov1alpha1.InferenceSet{
+				Status: kaitov1alpha1.InferenceSetStatus{
+					Conditions: []metav1.Condition{
+						{
+							Type:   string(kaitov1alpha1.InferenceSetConditionTypeReady),
+							Status: metav1.ConditionFalse,
+						},
+					},
+				},
+			},
+			expected: "error",
+		},
+		{
+			name: "inferenceset pending with no conditions",
+			is: &kaitov1alpha1.InferenceSet{
+				Status: kaitov1alpha1.InferenceSetStatus{
+					Conditions: []metav1.Condition{},
+				},
+			},
+			expected: "pending",
+		},
+		{
+			name: "inferenceset pending with unknown condition",
+			is: &kaitov1alpha1.InferenceSet{
+				Status: kaitov1alpha1.InferenceSetStatus{
+					Conditions: []metav1.Condition{
+						{
+							Type:   "UnknownCondition",
+							Status: metav1.ConditionTrue,
+						},
+					},
+				},
+			},
+			expected: "pending",
+		},
+		{
+			name: "deleting takes precedence over ready",
+			is: &kaitov1alpha1.InferenceSet{
+				Status: kaitov1alpha1.InferenceSetStatus{
+					Conditions: []metav1.Condition{
+						{
+							Type:   string(kaitov1alpha1.InferenceSetConditionTypeDeleting),
+							Status: metav1.ConditionTrue,
+						},
+						{
+							Type:   string(kaitov1alpha1.InferenceSetConditionTypeReady),
+							Status: metav1.ConditionTrue,
+						},
+					},
+				},
+			},
+			expected: "deleting",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := determineInferenceSetPhase(tt.is)
+			assert.Equal(t, tt.expected, result)
+		})
+	}
+}
diff --git a/pkg/workspace/controllers/metrics_test.go b/pkg/workspace/controllers/metrics_test.go
new file mode 100644
index 0000000..67e4cdd
--- /dev/null
+++ b/pkg/workspace/controllers/metrics_test.go
@@ -0,0 +1,122 @@
+// Copyright (c) KAITO authors.
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package controllers
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
+)
+
+func TestDetermineWorkspacePhase(t *testing.T) {
+	tests := []struct {
+		name     string
+		ws       *kaitov1beta1.Workspace
+		expected string
+	}{
+		{
+			name: "workspace is deleting",
+			ws: &kaitov1beta1.Workspace{
+				Status: kaitov1beta1.WorkspaceStatus{
+					Conditions: []metav1.Condition{
+						{
+							Type:   string(kaitov1beta1.WorkspaceConditionTypeDeleting),
+							Status: metav1.ConditionTrue,
+						},
+					},
+				},
+			},
+			expected: "deleting",
+		},
+		{
+			name: "workspace succeeded",
+			ws: &kaitov1beta1.Workspace{
+				Status: kaitov1beta1.WorkspaceStatus{
+					Conditions: []metav1.Condition{
+						{
+							Type:   string(kaitov1beta1.WorkspaceConditionTypeSucceeded),
+							Status: metav1.ConditionTrue,
+						},
+					},
+				},
+			},
+			expected: "succeeded",
+		},
+		{
+			name: "workspace error",
+			ws: &kaitov1beta1.Workspace{
+				Status: kaitov1beta1.WorkspaceStatus{
+					Conditions: []metav1.Condition{
+						{
+							Type:   string(kaitov1beta1.WorkspaceConditionTypeSucceeded),
+							Status: metav1.ConditionFalse,
+						},
+					},
+				},
+			},
+			expected: "error",
+		},
+		{
+			name: "workspace pending with no conditions",
+			ws: &kaitov1beta1.Workspace{
+				Status: kaitov1beta1.WorkspaceStatus{
+					Conditions: []metav1.Condition{},
+				},
+			},
+			expected: "pending",
+		},
+		{
+			name: "workspace pending with unknown condition",
+			ws: &kaitov1beta1.Workspace{
+				Status: kaitov1beta1.WorkspaceStatus{
+					Conditions: []metav1.Condition{
+						{
+							Type:   "UnknownCondition",
+							Status: metav1.ConditionTrue,
+						},
+					},
+				},
+			},
+			expected: "pending",
+		},
+		{
+			name: "deleting takes precedence over succeeded",
+			ws: &kaitov1beta1.Workspace{
+				Status: kaitov1beta1.WorkspaceStatus{
+					Conditions: []metav1.Condition{
+						{
+							Type:   string(kaitov1beta1.WorkspaceConditionTypeDeleting),
+							Status: metav1.ConditionTrue,
+						},
+						{
+							Type:   string(kaitov1beta1.WorkspaceConditionTypeSucceeded),
+							Status: metav1.ConditionTrue,
+						},
+					},
+				},
+			},
+			expected: "deleting",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := DetermineWorkspacePhase(tt.ws)
+			assert.Equal(t, tt.expected, result)
+		})
+	}
+}