Repository: openshift/instaslice-operator
Pull request #472: [Updated] Add custom metrics
Author: login: mamy-CS
Labels: name: size/XL
description: Denotes a PR that changes 500-999 lines, ignoring generated files.
name: lgtm
description: Indicates that a PR is ready to be merged.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
Created: 2025-03-04T21:44:33Z | Merged: 2025-03-20T08:33:33Z

Addresses - https://github.com/openshift/instaslice-operator/issues/53
Enable Kubernetes metrics from controllers
And add Custom metrics
- _instaslice_pod_processed_slices_: Pods that are processed with their slice/s allocation
- _instaslice_compatible_profiles_: Profiles compatible with remaining GPU slices in a node and their counts
- _instaslice_total_processed_gpu_slices_ : Number of total processed GPU slices since instaslice controller start time

- [x] Unit tests added for metrics reconcile funcs and calls in instaslice _controller and prometheus_manager

- [x] **Updated**: Successful run of make test, make lint, make test-e2e, and make test-e2e-kind-emulated


Diff:
diff --git a/cmd/controller/main.go b/cmd/controller/main.go
index 878a477a8..4180f1638 100644
--- a/cmd/controller/main.go
+++ b/cmd/controller/main.go
@@ -59,6 +59,12 @@ func init() {
 }
 
 func main() {
+	// Log info before initializing metrics exporter
+	ctrl.Log.Info("Initializing Metrics Exporter.")
+	controller.RegisterMetrics()
+	// Log info after the metrics exporter is initialized
+	ctrl.Log.Info("Metrics Exporter Initialized.")
+
 	var metricsAddr string
 	var enableLeaderElection bool
 	var probeAddr string
diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml
index 2e6975857..13f833d79 100644
--- a/config/manager/manager.yaml
+++ b/config/manager/manager.yaml
@@ -78,6 +78,7 @@ spec:
           capabilities:
             drop:
             - "ALL"
+          runAsUser: 1000
         livenessProbe:
           httpGet:
             path: /healthz
diff --git a/deploy/instaslice-metrics-service.yaml b/deploy/instaslice-metrics-service.yaml
new file mode 100644
index 000000000..f6d3721a9
--- /dev/null
+++ b/deploy/instaslice-metrics-service.yaml
@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: instaslice-metrics
+  namespace: instaslice-system
+  labels:
+    control-plane: controller-manager
+spec:
+  ports:
+    - name: metrics
+      port: 8443
+      protocol: TCP
+      targetPort: 8443
+  selector:
+    control-plane: controller-manager  # Use the correct label here
\ No newline at end of file
diff --git a/deploy/instaslice-servicemonitor.yaml b/deploy/instaslice-servicemonitor.yaml
new file mode 100644
index 000000000..85dbcd42d
--- /dev/null
+++ b/deploy/instaslice-servicemonitor.yaml
@@ -0,0 +1,23 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: instaslice-monitor
+  namespace: instaslice-monitoring
+  labels:
+    release: prometheus  # Label to match Prometheus serviceMonitorSelector
+spec:
+  selector:
+    matchLabels:
+      control-plane: controller-manager # Match labels of the Service exposing metrics
+  namespaceSelector:
+    matchNames:
+      - instaslice-system  # Namespace where the Service resides
+  endpoints:
+    - port: metrics  # Port name exposed in the Service for kube-rbac-proxy
+      interval: 15s
+      path: /metrics
+      scheme: https
+      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token  # Prometheus authentication
+      honorLabels: true
+      tlsConfig:
+        insecureSkipVerify: true  # Set to false if using a valid CA
diff --git a/deploy/prometheus-role.yaml b/deploy/prometheus-role.yaml
new file mode 100644
index 000000000..301fba73f
--- /dev/null
+++ b/deploy/prometheus-role.yaml
@@ -0,0 +1,16 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: prometheus-metrics-reader
+  namespace: instaslice-system
+rules:
+  - apiGroups: [""]
+    resources: ["services", "endpoints", "pods"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["metrics.k8s.io"]
+    resources: ["pods", "nodes"]
+    verbs: ["get", "list"]
+  - apiGroups: ["authorization.k8s.io"]
+    resources: ["subjectaccessreviews"]
+    verbs: ["create"]
+
diff --git a/deploy/prometheus-rolebinding.yaml b/deploy/prometheus-rolebinding.yaml
new file mode 100644
index 000000000..915daeecd
--- /dev/null
+++ b/deploy/prometheus-rolebinding.yaml
@@ -0,0 +1,13 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: prometheus-metrics-binding
+  namespace: instaslice-system
+subjects:
+- kind: ServiceAccount
+  name: prometheus-kube-prometheus-prometheus  # Change this to your Prometheus ServiceAccount
+  namespace: instaslice-monitoring  # Change to Prometheus namespace
+roleRef:
+  kind: Role
+  name: prometheus-metrics-reader
+  apiGroup: rbac.authorization.k8s.io
diff --git a/deploy/prometheus-serviceaccount.yaml b/deploy/prometheus-serviceaccount.yaml
new file mode 100644
index 000000000..91836e79a
--- /dev/null
+++ b/deploy/prometheus-serviceaccount.yaml
@@ -0,0 +1,5 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: prometheus
+  namespace: instaslice-monitoring  # namespace where Prometheus is running
diff --git a/deploy/prometheus.yaml b/deploy/prometheus.yaml
new file mode 100644
index 000000000..08d81ebdb
--- /dev/null
+++ b/deploy/prometheus.yaml
@@ -0,0 +1,36 @@
+alertmanager:
+    enabled: false
+kube-state-metrics:
+    enabled: false
+prometheus-node-exporter:
+    enabled: false
+prometheus-pushgateway:
+    enabled: false
+server:
+    name: instaslice
+    service:
+        enabled: true
+        type: NodePort
+        servicePort: 9090
+    persistentVolume:
+        existingClaim: prometheus-instaslice
+        enabled: false
+    securityContext:
+        runAsUser:
+        runAsNonRoot:
+        runAsGroup:
+        fsGroup:
+extraScrapeConfigs: |
+    - job_name: instaslice-metrics
+      honor_labels: true
+      metrics_path: /metrics
+      scheme: https
+      scrape_interval: 15s
+      static_configs:
+          - targets:
+              - instaslice-metrics.instaslice-system.svc.cluster.local:8443
+      tls_config:
+        insecure_skip_verify: true
+serviceMonitorSelector:
+    matchLabels:
+        release: prometheus
\ No newline at end of file
diff --git a/docs/enable-promethues-metrics.md b/docs/enable-promethues-metrics.md
new file mode 100644
index 000000000..508cc3a5c
--- /dev/null
+++ b/docs/enable-promethues-metrics.md
@@ -0,0 +1,63 @@
+# Enabling Prometheus for Instaslice Monitoring
+
+## Overview
+This document outlines the steps to enable Prometheus monitoring for Instaslice, including setting up Prometheus, configuring metrics collection, and verifying the setup. The following guide follows a **happy path** approach, ensuring a smooth deployment process. Controller failure fault tolerances and additional reliability mechanisms will be added in future work. **Unit tests have been added**, and **end-to-end (E2E) tests for metrics will be incorporated in future work.**
+
+## Available Custom Instaslice Metrics
+Instaslice exposes the following Prometheus metrics:
+
+| Metric Name | Description |
+|-------------|-------------|
+| `instaslice_pod_processed_slices` | Tracks the number of pods that have been processed with their slice allocation. |
+| `instaslice_compatible_profiles` | Displays the profiles compatible with the remaining GPU slices on a node and their counts. |
+| `instaslice_total_processed_gpu_slices` | Counts the total processed GPU slices since the Instaslice controller started. |
+
+## Steps to Deploy Prometheus
+
+1. **Install Prometheus using Helm:**
+   ```sh
+   helm install prometheus -f deploy/prometheus.yaml prometheus-community/kube-prometheus-stack --namespace=instaslice-monitoring --create-namespace
+   ```
+2. **Apply Prometheus role and role binding:**
+   ```sh
+   kubectl apply -f deploy/prometheus-role.yaml
+   kubectl apply -f deploy/prometheus-rolebinding.yaml
+   ```
+3. **Apply the Instaslice ServiceMonitor:**
+   ```sh
+   kubectl apply -f deploy/instaslice-servicemonitor.yaml
+   ```
+
+## Deploy Instaslice and Configure Metrics Service
+
+1. **Apply the Instaslice Metrics Service YAML:**
+   ```sh
+   kubectl apply -f deploy/instaslice-metrics-service.yaml
+   ```
+2. **Check the readiness of Prometheus pods:**
+   ```sh
+   kubectl get pods -n instaslice-monitoring --watch
+   ```
+3. **Retrieve the token to access the service:**
+   ```sh
+   TOKEN=$(kubectl exec -it prometheus-prometheus-kube-prometheus-prometheus-0 -n instaslice-monitoring -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+   ```
+4. **Forward the Instaslice Metrics Service port:**
+   ```sh
+   kubectl port-forward svc/instaslice-metrics 8443:8443 -n instaslice-system
+   ```
+5. **Verify the exposed metrics using curl:**
+   ```sh
+   curl -k -H "Authorization: Bearer $TOKEN" https://localhost:8443/metrics
+   ```
+6. **Forward the Prometheus UI port for monitoring:**
+   ```sh
+   kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n instaslice-monitoring 9090:9090
+   ```
+
+## Future Work
+- **Controller Failure Handling:** Additional fault tolerance mechanisms will be integrated to ensure resilience in case of failures.
+- **E2E Testing:** While unit tests are already in place, future efforts will focus on adding **end-to-end (E2E) tests** for Instaslice metrics.
+
+By following the above steps, Instaslice metrics can be successfully monitored using Prometheus. Future improvements will focus on increasing robustness and automated testing coverage.
+
diff --git a/internal/controller/cache.go b/internal/controller/cache.go
index e84f73398..b306358fa 100644
--- a/internal/controller/cache.go
+++ b/internal/controller/cache.go
@@ -69,13 +69,19 @@ func (r *InstasliceReconciler) updateCacheWithNewAllocation(podUid types.UID, al
 
 // clean allocations that do not exists in spec
 // TODO fix scalability issue, loops over all instaslice objects in the cluster
-func (r *InstasliceReconciler) CleanupOrphanedAllocations(instasliceList *inferencev1alpha1.InstasliceList) {
+func (r *InstasliceReconciler) CleanupOrphanedAllocations(ctx context.Context, instasliceList *inferencev1alpha1.InstasliceList) {
 	var keysToDelete []types.UID
 
 	for uuid := range r.allocationCache {
 		found := false
 		for _, instaslice := range instasliceList.Items {
-			if _, exists := instaslice.Spec.PodAllocationRequests[uuid]; exists {
+			// Fetch latest Instaslice state before updating metrics
+			updatedInstaslice, err := r.getInstasliceObject(ctx, instaslice.Name, instaslice.Namespace)
+			if err != nil {
+				log.FromContext(ctx).Error(err, "Failed to get latest Instaslice object", "instaslice", instaslice.Name)
+				return
+			}
+			if _, exists := updatedInstaslice.Spec.PodAllocationRequests[uuid]; exists {
 				found = true
 				break
 			}
diff --git a/internal/controller/capacity.go b/internal/controller/capacity.go
index a3fe58d50..55e8d0bb2 100644
--- a/internal/controller/capacity.go
+++ b/internal/controller/capacity.go
@@ -62,15 +62,14 @@ func (r *InstasliceReconciler) findNodeAndDeviceForASlice(ctx context.Context, i
 			if updatedInstaSliceObject.Spec.PodAllocationRequests == nil {
 				updatedInstaSliceObject.Spec.PodAllocationRequests = make(map[types.UID]inferencev1alpha1.AllocationRequest)
 			}
-
-			newStart := r.getStartIndexFromAllocationResults(updatedInstaSliceObject, gpuuuid, profileName, pod.UID)
+			gpuAllocatedIndex := r.gpuAllocatedSlices(gpuuuid)
+			newStart := r.getStartIndexFromAllocationResults(updatedInstaSliceObject, profileName, gpuAllocatedIndex, &pod.UID, false)
 			// For example, a newStart of 9 is considered invalid.
 			notValidIndex := int32(9)
 			if newStart == notValidIndex {
 				// Move to next GPU if the index is not valid.
 				continue
 			}
-
 			size, discoveredGiprofile, Ciprofileid, Ciengprofileid := r.extractGpuProfile(updatedInstaSliceObject, profileName)
 			resourceIdentifier := pod.Spec.Containers[0].EnvFrom[0].ConfigMapRef.Name
 
@@ -96,10 +95,10 @@ func (r *InstasliceReconciler) findNodeAndDeviceForASlice(ctx context.Context, i
 			return allocRequest, allocResult, nil
 		}
 	}
-
 	return nil, nil, fmt.Errorf("failed to find allocatable node and gpu")
 }
 
+// sortGPUs returns the sorted gpu IDs stored in the instaslice object
 func sortGPUs(updatedInstaSliceObject *inferencev1alpha1.Instaslice) []string {
 	gpuUUIDs := make([]string, 0, len(updatedInstaSliceObject.Status.NodeResources.NodeGPUs))
 	for _, discoveredGpu := range updatedInstaSliceObject.Status.NodeResources.NodeGPUs {
@@ -109,14 +108,9 @@ func sortGPUs(updatedInstaSliceObject *inferencev1alpha1.Instaslice) []string {
 	return gpuUUIDs
 }
 
-// accounting logic that finds the correct GPU and index where a slice could be placed.
-func (r *InstasliceReconciler) getStartIndexFromAllocationResults(instaslice *inferencev1alpha1.Instaslice, gpuUUID string, profileName string, podUid types.UID) int32 {
+// gpuAllocatedSlices stores already allocated slices in indexes
+func (r *InstasliceReconciler) gpuAllocatedSlices(gpuUUID string) [8]int32 {
 	//TODO: generalize, A100 and H100 have 8 indexes for 3g and 7g and 7 for rest, so go with 8 and we are bounded by
-	allocResult, exists := r.allocationCache[podUid]
-	// allocation already exists in cache
-	if exists {
-		return allocResult.MigPlacement.Start
-	}
 	var gpuAllocatedIndex [8]int32
 	// clean slate init
 	for i := range gpuAllocatedIndex {
@@ -132,6 +126,19 @@ func (r *InstasliceReconciler) getStartIndexFromAllocationResults(instaslice *in
 			}
 		}
 	}
+	return gpuAllocatedIndex
+}
+
+// getStartIndexFromPreparedState finds the correct GPU and index where a slice could be placed.
+func (r *InstasliceReconciler) getStartIndexFromAllocationResults(instaslice *inferencev1alpha1.Instaslice, profileName string, gpuAllocatedIndex [8]int32, podUid *types.UID, simulate bool) int32 {
+	// if actual allocation, check if allocation already exists
+	if !simulate {
+		allocResult, exists := r.allocationCache[*podUid]
+		// allocation already exists in cache
+		if exists {
+			return allocResult.MigPlacement.Start
+		}
+	}
 	// Check if all indices are allocated
 	allAllocated := true
 	for _, allocated := range gpuAllocatedIndex {
@@ -196,7 +203,6 @@ func (r *InstasliceReconciler) getStartIndexFromAllocationResults(instaslice *in
 		}
 
 	}
-
 	return newStart
 }
 
diff --git a/internal/controller/instaslice_controller.go b/internal/controller/instaslice_controller.go
index c5006efb3..299dd155f 100644
--- a/internal/controller/instaslice_controller.go
+++ b/internal/controller/instaslice_controller.go
@@ -41,12 +41,14 @@ import (
 	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/apimachinery/pkg/util/wait"
 	"k8s.io/client-go/kubernetes"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
 	logr "sigs.k8s.io/controller-runtime/pkg/log"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
 )
 
@@ -191,12 +193,12 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 	if pod.Status.Phase == v1.PodFailed && controllerutil.ContainsFinalizer(pod, FinalizerName) {
 		for _, instaslice := range instasliceList.Items {
 			for uuid, allocation := range instaslice.Status.PodAllocationResults {
+				allocRequest := instaslice.Spec.PodAllocationRequests[uuid]
 				if pod.UID == uuid {
 					if allocation.AllocationStatus.AllocationStatusController == inferencev1alpha1.AllocationStatusCreating && allocation.AllocationStatus.AllocationStatusDaemonset == "" {
 						return ctrl.Result{RequeueAfter: Requeue2sDelay}, nil
 					}
 					if allocation.AllocationStatus.AllocationStatusDaemonset == inferencev1alpha1.AllocationStatusCreated || allocation.AllocationStatus.AllocationStatusController == inferencev1alpha1.AllocationStatusUngated {
-						allocRequest := instaslice.Spec.PodAllocationRequests[uuid]
 						resultDeleting, err := r.setInstasliceAllocationToDeleting(ctx, instaslice.Name, &allocation, &allocRequest)
 						if err != nil {
 							return resultDeleting, nil
@@ -210,6 +212,15 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 						if err != nil {
 							return ctrl.Result{}, err
 						}
+						r.CleanupOrphanedAllocations(ctx, &instasliceList)
+						// update DeployedPodTotal Metrics by setting value to 0 as pod allocation is deleted and pod is no loger consuming slices
+						if err = r.UpdateDeployedPodTotalMetrics(string(allocation.Nodename), allocation.GPUUUID, allocRequest.PodRef.Namespace, allocRequest.PodRef.Name, allocRequest.Profile, 0); err != nil {
+							log.Error(err, "Failed to update deployed pod metrics", "nodeName", allocation.Nodename)
+						}
+						//update compatible profiles metrics
+						if err := r.UpdateCompatibleProfilesMetrics(instaslice, instaslice.Name); err != nil {
+							log.Error(err, "Failed to update Compatible Profiles Metrics", "nodeName", instaslice.Name)
+						}
 						// requeue for the finalizer to be removed
 						return ctrl.Result{RequeueAfter: Requeue2sDelay}, nil
 					}
@@ -234,8 +245,8 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 		for _, instaslice := range instasliceList.Items {
 			for uuid, allocation := range instaslice.Status.PodAllocationResults {
 				if uuid == pod.UID {
+					allocRequest := instaslice.Spec.PodAllocationRequests[uuid]
 					if allocation.AllocationStatus.AllocationStatusDaemonset != inferencev1alpha1.AllocationStatusDeleted {
-						allocRequest := instaslice.Spec.PodAllocationRequests[uuid]
 						log.Info("setting status to deleting", "pod", pod.Name)
 						result, err := r.setInstasliceAllocationToDeleting(ctx, instaslice.Name, &allocation, &allocRequest)
 						if err != nil {
@@ -251,6 +262,15 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 						if err != nil {
 							return ctrl.Result{}, err
 						}
+						r.CleanupOrphanedAllocations(ctx, &instasliceList)
+						// update DeployedPodTotal Metrics by setting value to 0 as pod allocation is deleted and pod is no loger consuming slices
+						if err = r.UpdateDeployedPodTotalMetrics(string(allocation.Nodename), allocation.GPUUUID, allocRequest.PodRef.Namespace, allocRequest.PodRef.Name, allocRequest.Profile, 0); err != nil {
+							log.Error(err, "Failed to update deployed pod metrics", "nodeName", allocation.Nodename)
+						}
+						//update compatible profiles metrics
+						if err := r.UpdateCompatibleProfilesMetrics(instaslice, instaslice.Name); err != nil {
+							log.Error(err, "Failed to update Compatible Profiles Metrics", "nodeName", instaslice.Name)
+						}
 						// requeue for the finalizer to be removed
 						return ctrl.Result{RequeueAfter: Requeue2sDelay}, nil
 					}
@@ -276,9 +296,9 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 		// allocation can be in creating or created while the user deletes the pod.
 		for _, instaslice := range instasliceList.Items {
 			for podUuid, allocation := range instaslice.Status.PodAllocationResults {
+				allocRequest := instaslice.Spec.PodAllocationRequests[podUuid]
 				if podUuid == pod.UID && (allocation.AllocationStatus.AllocationStatusDaemonset == inferencev1alpha1.AllocationStatusCreated) {
 					allocation.AllocationStatus.AllocationStatusController = inferencev1alpha1.AllocationStatusDeleting
-					allocRequest := instaslice.Spec.PodAllocationRequests[podUuid]
 					if err := utils.UpdateOrDeleteInstasliceAllocations(ctx, r.Client, instaslice.Name, &allocation, &allocRequest); err != nil {
 						log.Info("unable to set instaslice to state deleted for ungated", "pod", pod.Name)
 						return ctrl.Result{RequeueAfter: 1 * time.Second}, nil
@@ -290,6 +310,15 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 					if err != nil {
 						return ctrl.Result{}, err
 					}
+					r.CleanupOrphanedAllocations(ctx, &instasliceList)
+					// update DeployedPodTotal Metrics by setting value to 0 as pod allocation is deleted and pod is no loger consuming slices
+					if err = r.UpdateDeployedPodTotalMetrics(string(allocation.Nodename), allocation.GPUUUID, allocRequest.PodRef.Namespace, allocRequest.PodRef.Name, allocRequest.Profile, 0); err != nil {
+						log.Error(err, "Failed to update deployed pod metrics", "nodeName", allocation.Nodename)
+					}
+					//update compatible profiles metrics
+					if err := r.UpdateCompatibleProfilesMetrics(instaslice, instaslice.Name); err != nil {
+						log.Error(err, "Failed to update Compatible Profiles Metrics", "nodeName", instaslice.Name)
+					}
 					if controllerutil.RemoveFinalizer(pod, FinalizerName) {
 						if err := r.Update(ctx, pod); err != nil {
 							// requeing immediately as the finalizer removal gets lost
@@ -301,7 +330,6 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 				}
 			}
 		}
-
 		return ctrl.Result{}, nil
 	}
 	// handle graceful termination of pods, wait for about 30 seconds from the time deletiontimestamp is set on the pod
@@ -392,6 +420,16 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 					}
 				}
 			}
+			// Fetch latest Instaslice state before updating metrics
+			updatedInstaslice, err := r.getInstasliceObject(ctx, instaslice.Name, instaslice.Namespace)
+			if err != nil {
+				log.Error(err, "Failed to get latest Instaslice object", "instaslice", instaslice.Name)
+				return ctrl.Result{Requeue: true}, nil
+			}
+			// update compatible profiles metrics
+			if err := r.UpdateCompatibleProfilesMetrics(*updatedInstaslice, instaslice.Name); err != nil {
+				log.Error(err, "Failed to update Compatible Profiles Metrics", "nodeName", updatedInstaslice.Name)
+			}
 		}
 		// pod does not have an allocation yet, make allocation
 		// find the node
@@ -405,7 +443,7 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 				return ctrl.Result{}, err
 			}
 
-			r.CleanupOrphanedAllocations(&instasliceList)
+			r.CleanupOrphanedAllocations(ctx, &instasliceList)
 			for _, instaslice := range instasliceList.Items {
 				// find the GPU on the node and the GPU index where the slice can be created
 				allocRequest, allocResult, err := r.findNodeAndDeviceForASlice(ctx, &instaslice, profileName, policy, pod)
@@ -420,6 +458,15 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 					if err != nil {
 						return ctrl.Result{Requeue: true}, nil
 					}
+					// allocation was successful
+					// update deployed pod total metrics
+					if err := r.UpdateDeployedPodTotalMetrics(string(allocResult.Nodename), allocResult.GPUUUID, allocRequest.PodRef.Namespace, allocRequest.PodRef.Name, allocRequest.Profile, allocResult.MigPlacement.Size); err != nil {
+						log.Error(err, "Failed to update deployed pod metrics (node: %s, namespce: %s, pod: %s): %w", allocResult.Nodename, allocRequest.PodRef.Namespace, allocRequest.PodRef.Name, err)
+					}
+					// update total processed GPU slices metrics
+					if err = r.IncrementTotalProcessedGpuSliceMetrics(instaslice, string(allocResult.Nodename), allocResult.GPUUUID, profileName, pod); err != nil {
+						log.Error(err, "Failed to update total processed GPU slices metric", "nodeName", allocResult.Nodename, "gpuID", allocResult.GPUUUID)
+					}
 					return ctrl.Result{}, nil
 				}
 			}
@@ -434,11 +481,10 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 		}
 
 	}
-
 	return ctrl.Result{}, nil
 }
 
-// create the DaemonSet object
+// createInstaSliceDaemonSet - create the DaemonSet object
 func (r *InstasliceReconciler) createInstaSliceDaemonSet(namespace string) *appsv1.DaemonSet {
 	emulatorMode := r.Config.EmulatorModeEnable
 	instasliceDaemonsetImage := r.Config.DaemonsetImage
@@ -552,6 +598,7 @@ func (*InstasliceReconciler) extractGpuProfile(instaslice *inferencev1alpha1.Ins
 	return size, discoveredGiprofile, Ciprofileid, Ciengprofileid
 }
 
+// isPodSchedulingGated checks if a pod has a scheduling gate and is actively blocked
 func checkIfPodGatedByInstaSlice(pod *v1.Pod) bool {
 	for _, gate := range pod.Spec.SchedulingGates {
 		if gate.Name == GateName {
@@ -596,22 +643,82 @@ func (r *InstasliceReconciler) podMapFunc(ctx context.Context, obj client.Object
 	return requests
 }
 
-// SetupWithManager sets up the controller with the Manager.
+// Initialize Prometheus-compatible profiles metrics when the controller starts
+// Adds a background goroutine that waits for Instaslice objects.
+// Proceeds to setupWithManager(mgr) to start the reconciler
+// Does not block the controller from reconciling
+// UpdateCompatibleProfilesMetrics only updates Prometheus metrics,in-memory and do not persist in etcd
+// TODO: support daemonset fault tolerance and controller fault tolerance (skipping an update for a faster boot)
 func (r *InstasliceReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	restConfig := mgr.GetConfig()
-
 	var err error
 	r.kubeClient, err = kubernetes.NewForConfig(restConfig)
 	if err != nil {
 		return err
 	}
+	mgrAddErr := mgr.Add(manager.RunnableFunc(func(ctx context.Context) error {
+		log := logr.FromContext(ctx)
+		<-mgr.Elected() // Wait for leader election before executing
+		// Retry mechanism to wait for Instaslice objects
+		var instasliceList inferencev1alpha1.InstasliceList
+		retryErr := wait.PollUntilContextTimeout(ctx, 2*time.Second, 60*time.Second, true, func(ctx context.Context) (bool, error) {
+			if err := r.List(ctx, &instasliceList); err != nil {
+				log.Error(err, "Failed to list Instaslice objects, retrying...")
+				return false, nil
+			}
+			if len(instasliceList.Items) > 0 {
+				log.Info("Instaslice objects found", "count", len(instasliceList.Items))
+				return true, nil
+			}
+			log.Info("No Instaslice objects found, waiting...")
+			return false, nil
+		})
+		if retryErr != nil {
+			log.Error(retryErr, "Failed to fetch Instaslice objects after retries")
+			return nil // Do not block the controller from running, meaning reconciler starts in parallel
+		}
+		// Iterate over Instaslices and update Prometheus metrics
+		for _, instaslice := range instasliceList.Items {
+			if err := r.UpdateCompatibleProfilesMetrics(instaslice, instaslice.Name); err != nil {
+				log.Error(err, "Failed to update compatible profiles metrics", "instaslice", instaslice.Name)
+			}
+		}
+		log.Info("Successfully initialized compatible profiles metrics for all Instaslice objects")
+		return nil
+	}))
+
+	if mgrAddErr != nil {
+		return mgrAddErr
+	}
+
+	// Continue with setting up the controller
+	return r.setupWithManager(mgr) // Return error directly for readability
+}
 
-	return ctrl.NewControllerManagedBy(mgr).
+// Enable creation of controller
+func (r *InstasliceReconciler) setupWithManager(mgr ctrl.Manager) error {
+	restConfig := mgr.GetConfig()
+	var err error
+	r.kubeClient, err = kubernetes.NewForConfig(restConfig)
+	if err != nil {
+		return err
+	}
+	err = ctrl.NewControllerManagedBy(mgr).
 		For(&v1.Pod{}).Named("InstaSlice-controller").
 		Watches(&inferencev1alpha1.Instaslice{}, handler.EnqueueRequestsFromMapFunc(r.podMapFunc)).
 		Watches(&v1.Node{},
 			&handler.EnqueueRequestForObject{}).
 		Complete(r)
+
+	if err != nil {
+		log := mgr.GetLogger() // Get logger from the manager
+		log.Error(err, "Failed to set up Instaslice controller")
+		return err
+	}
+
+	log := mgr.GetLogger()
+	log.Info("Successfully set up Instaslice controller")
+	return nil
 }
 
 func (r *InstasliceReconciler) unGatePod(podUpdate *v1.Pod) *v1.Pod {
@@ -699,6 +806,7 @@ func (r *InstasliceReconciler) removeInstasliceAllocation(ctx context.Context, i
 	}
 	return nil
 }
+
 func (r *InstasliceReconciler) setInstasliceAllocationToDeleting(ctx context.Context, instasliceName string, allocResult *inferencev1alpha1.AllocationResult, allocRequest *inferencev1alpha1.AllocationRequest) (ctrl.Result, error) {
 	log := logr.FromContext(ctx)
 	allocResult.AllocationStatus.AllocationStatusController = inferencev1alpha1.AllocationStatusDeleting
diff --git a/internal/controller/instaslice_controller_test.go b/internal/controller/instaslice_controller_test.go
index df475c2ad..31fa3cc86 100644
--- a/internal/controller/instaslice_controller_test.go
+++ b/internal/controller/instaslice_controller_test.go
@@ -18,6 +18,7 @@ package controller
 
 import (
 	"context"
+	"fmt"
 	"testing"
 	"time"
 
@@ -71,7 +72,7 @@ func TestChangesAllocationDeletionAndFinalizer(t *testing.T) {
 				Config: config,
 			}
 
-			podUUID = "test-pod-uuid"
+			podUUID = "test-pod-uuid-1"
 
 			pod = &v1.Pod{
 				ObjectMeta: metav1.ObjectMeta{
@@ -373,7 +374,7 @@ func TestInstasliceReconciler_Reconcile(t *testing.T) {
 				Config: config,
 			}
 
-			podUUID = "test-pod-uuid"
+			podUUID = "test-pod-uuid-3"
 			pod = &v1.Pod{
 				ObjectMeta: metav1.ObjectMeta{
 					Name:      "test-pod",
@@ -972,7 +973,7 @@ func TestFirstFitPolicy_SetAllocationDetails(t *testing.T) {
 		profileName:         "1g.5gb",
 		newStart:            0,
 		size:                1,
-		podUUID:             types.UID("test-pod-uuid"),
+		podUUID:             types.UID("test-pod-uuid-4"),
 		nodename:            types.NodeName("kind-control-plane"),
 		allocationStatus:    inferencev1alpha1.AllocationStatus{AllocationStatusDaemonset: inferencev1alpha1.AllocationStatusCreated},
 		discoveredGiprofile: 0,
@@ -1042,6 +1043,7 @@ func TestFirstFitPolicy_SetAllocationDetails(t *testing.T) {
 	}
 }
 
+// Test `sortGPUs`
 func TestGPUSort(t *testing.T) {
 	Describe("SortGPUs", func() {
 		It("should sort GPU UUIDs in ascending order", func() {
@@ -1064,3 +1066,324 @@ func TestGPUSort(t *testing.T) {
 		})
 	})
 }
+
+// Test `getStartIndexFromPreparedState`
+func TestGetStartIndexFromPreparedState(t *testing.T) {
+	Describe("getStartIndexFromPreparedState", func() {
+		It("should return the correct start index for a profile", func() {
+			instaslice := &inferencev1alpha1.Instaslice{
+				Status: inferencev1alpha1.InstasliceStatus{
+					NodeResources: inferencev1alpha1.DiscoveredNodeResources{
+						MigPlacement: map[string]inferencev1alpha1.Mig{
+							"2g.10gb": {
+								Placements: []inferencev1alpha1.Placement{{Size: 2, Start: 1}},
+							},
+						},
+					},
+				},
+			}
+			gpuAllocatedIndex := [8]int32{1, 0, 0, 0, 0, 0, 0, 0}
+			Expect((&InstasliceReconciler{}).getStartIndexFromAllocationResults(instaslice, "2g.10gb", gpuAllocatedIndex, nil, true)).To(Equal(int32(1)))
+		})
+	})
+}
+
+// Test `calculateProfileFitOnGPU`
+var _ = Describe("calculateProfileFitOnGPU", func() {
+	var (
+		ctx        context.Context
+		fakeClient client.Client
+		r          *InstasliceReconciler
+		instaslice *inferencev1alpha1.Instaslice
+		pod        *v1.Pod
+		podUUID    string
+	)
+
+	BeforeEach(func() {
+		ctx = context.Background()
+		scheme := runtime.NewScheme()
+		Expect(inferencev1alpha1.AddToScheme(scheme)).To(Succeed())
+		Expect(v1.AddToScheme(scheme)).To(Succeed())
+
+		fakeClient = fake.NewClientBuilder().WithScheme(scheme).Build()
+		r = &InstasliceReconciler{
+			Client: fakeClient,
+		}
+		podUUID = "test-pod-uuid-6"
+		pod = &v1.Pod{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-pod",
+				Namespace: InstaSliceOperatorNamespace,
+				UID:       types.UID(podUUID),
+				Finalizers: []string{
+					FinalizerName,
+				},
+			},
+		}
+
+		Expect(fakeClient.Create(ctx, pod)).To(Succeed())
+		instaslice = &inferencev1alpha1.Instaslice{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-instaslice",
+				Namespace: "instaslice-operator",
+			},
+			Status: inferencev1alpha1.InstasliceStatus{
+				NodeResources: inferencev1alpha1.DiscoveredNodeResources{
+					MigPlacement: map[string]inferencev1alpha1.Mig{
+						"4g.20gb": {
+							Placements: []inferencev1alpha1.Placement{{Size: 4, Start: 0}},
+						},
+					},
+				},
+			},
+		}
+
+		Expect(fakeClient.Create(ctx, instaslice)).To(Succeed())
+	})
+
+	It("should return the correct actual slice size allocated", func() {
+		actualSize, err := r.calculateProfileFitOnGPU(instaslice, "4g.20gb", "gpu-1", false, pod)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(actualSize).To(Equal(int32(4)))
+	})
+})
+
+// Test `calculateProfileFitOnGPU for slices fit simulation`
+func TestGetTotalFitForProfileOnGPU(t *testing.T) {
+	var _ = Describe("GetTotalFitForProfileOnGPU", func() {
+		tests := []struct {
+			name          string
+			instasliceObj *inferencev1alpha1.Instaslice
+			profileName   string
+			gpuuid        string
+			remaining     int32
+			expectedFit   int32
+		}{
+			{
+				name: "Zero remaining capacity",
+				instasliceObj: &inferencev1alpha1.Instaslice{
+					Status: inferencev1alpha1.InstasliceStatus{
+						NodeResources: inferencev1alpha1.DiscoveredNodeResources{
+							MigPlacement: map[string]inferencev1alpha1.Mig{
+								"1g.10gb": {
+									Placements: []inferencev1alpha1.Placement{},
+								},
+							},
+						},
+					},
+				},
+				profileName: "1g.10gb",
+				remaining:   0,
+				expectedFit: 0,
+			},
+			{
+				name: "Profile not found",
+				instasliceObj: &inferencev1alpha1.Instaslice{
+					Status: inferencev1alpha1.InstasliceStatus{
+						NodeResources: inferencev1alpha1.DiscoveredNodeResources{
+							MigPlacement: map[string]inferencev1alpha1.Mig{},
+						},
+					},
+				},
+				profileName: "non-existent-profile",
+				remaining:   10,
+				expectedFit: 0,
+			},
+			{
+				name: "Special case: 7g.40gb with exact fit",
+				instasliceObj: &inferencev1alpha1.Instaslice{
+					Status: inferencev1alpha1.InstasliceStatus{
+						NodeResources: inferencev1alpha1.DiscoveredNodeResources{
+							MigPlacement: map[string]inferencev1alpha1.Mig{
+								"7g.40gb": {
+									Placements: []inferencev1alpha1.Placement{{Size: 7}},
+								},
+							},
+						},
+					},
+				},
+				profileName: "7g.40gb",
+				remaining:   7,
+				expectedFit: 0,
+			},
+			{
+				name: "Regular profile fitting multiple times",
+				instasliceObj: &inferencev1alpha1.Instaslice{
+					Status: inferencev1alpha1.InstasliceStatus{
+						NodeResources: inferencev1alpha1.DiscoveredNodeResources{
+							MigPlacement: map[string]inferencev1alpha1.Mig{
+								"3g.20gb": {
+									Placements: []inferencev1alpha1.Placement{{Size: 4}},
+								},
+							},
+						},
+					},
+				},
+				profileName: "3g.20gb",
+				remaining:   9,
+				expectedFit: 1,
+			},
+			{
+				name: "Exact fit scenario",
+				instasliceObj: &inferencev1alpha1.Instaslice{
+					Status: inferencev1alpha1.InstasliceStatus{
+						NodeResources: inferencev1alpha1.DiscoveredNodeResources{
+							MigPlacement: map[string]inferencev1alpha1.Mig{
+								"2g.10gb": {
+									Placements: []inferencev1alpha1.Placement{{Size: 2}},
+								},
+							},
+						},
+					},
+				},
+				profileName: "2g.10gb",
+				remaining:   4,
+				expectedFit: 1,
+			},
+		}
+		for _, tt := range tests {
+			It(tt.name, func() {
+				r := &InstasliceReconciler{}
+				fit, err := r.calculateProfileFitOnGPU(tt.instasliceObj, tt.profileName, tt.gpuuid, true, nil)
+
+				if tt.name == "Profile not found" || tt.name == "Zero remaining capacity" {
+					Expect(err).To(HaveOccurred(), "Expected error for missing profile or empty placement")
+				} else {
+					Expect(err).ToNot(HaveOccurred(), "calculateProfileFitOnGPU should not return an error")
+				}
+				Expect(fit).To(Equal(tt.expectedFit))
+			})
+		}
+
+	})
+}
+
+// Mock client that forces an update error
+type FakeFailingClient struct {
+	client.Client
+	FailOnUpdate bool
+}
+
+func (f *FakeFailingClient) Update(ctx context.Context, obj client.Object, opts ...client.UpdateOption) error {
+	if f.FailOnUpdate {
+		return fmt.Errorf("simulated update failure")
+	}
+	return f.Client.Update(ctx, obj, opts...)
+}
+
+var _ = Describe("Metrics Incrementation", func() {
+	var (
+		ctx        context.Context
+		r          *InstasliceReconciler
+		fakeClient client.Client
+		instaslice *inferencev1alpha1.Instaslice
+		pod        *v1.Pod
+		podUUID    string
+	)
+
+	BeforeEach(func() {
+		ctx = context.TODO()
+
+		scheme := runtime.NewScheme()
+		Expect(inferencev1alpha1.AddToScheme(scheme)).To(Succeed())
+		Expect(v1.AddToScheme(scheme)).To(Succeed())
+
+		fakeClient = fake.NewClientBuilder().
+			WithScheme(scheme).
+			WithStatusSubresource(&inferencev1alpha1.Instaslice{}).
+			Build()
+		config := config.ConfigFromEnvironment()
+
+		r = &InstasliceReconciler{
+			Client: fakeClient,
+			Config: config,
+		}
+
+		podUUID = "test-pod-uuid-2"
+
+		pod = &v1.Pod{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-pod",
+				Namespace: InstaSliceOperatorNamespace,
+				UID:       types.UID(podUUID),
+				Finalizers: []string{
+					FinalizerName,
+				},
+			},
+		}
+
+		Expect(fakeClient.Create(ctx, pod)).To(Succeed())
+
+		instaslice = &inferencev1alpha1.Instaslice{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-instaslice",
+				Namespace: InstaSliceOperatorNamespace,
+			},
+			Spec: inferencev1alpha1.InstasliceSpec{
+				PodAllocationRequests: map[types.UID]inferencev1alpha1.AllocationRequest{
+					types.UID(podUUID): {
+						Profile: "test-profile",
+						PodRef: v1.ObjectReference{
+							Name:      pod.Name,
+							Namespace: InstaSliceOperatorNamespace,
+							UID:       pod.UID,
+						},
+						Resources: v1.ResourceRequirements{},
+					},
+				},
+			},
+			Status: inferencev1alpha1.InstasliceStatus{
+				PodAllocationResults: map[types.UID]inferencev1alpha1.AllocationResult{
+					types.UID(podUUID): {
+						AllocationStatus:            inferencev1alpha1.AllocationStatus{AllocationStatusController: inferencev1alpha1.AllocationStatusCreating},
+						GPUUUID:                     "GPU-12345",
+						Nodename:                    "fake-node",
+						ConfigMapResourceIdentifier: "fake-configmap-uid",
+					},
+				},
+				NodeResources: inferencev1alpha1.DiscoveredNodeResources{
+					MigPlacement: map[string]inferencev1alpha1.Mig{
+						"1g.5gb": {
+							Placements: []inferencev1alpha1.Placement{
+								{Size: 1, Start: 0}, // Example placement
+							},
+						},
+						"1g.10gb": { // Also add the missing profile from the failing test
+							Placements: []inferencev1alpha1.Placement{
+								{Size: 2, Start: 0},
+							},
+						},
+					},
+				},
+			},
+		}
+
+		Expect(fakeClient.Create(ctx, instaslice)).To(Succeed())
+	})
+
+	// Test to prevent double metric incrementation
+	It("should not increment metrics more than once", func() {
+
+		err := r.IncrementTotalProcessedGpuSliceMetrics(*instaslice, "node-1", "gpu-1", "1g.5gb", pod)
+		Expect(err).ToNot(HaveOccurred()) // Should skip incrementation
+	})
+
+	// Validate allocation changes
+	It("should correctly reflect allocation changes in metrics", func() {
+		instaslice.Status.PodAllocationResults["alloc-1"] = inferencev1alpha1.AllocationResult{
+			Nodename: "node-1",
+			GPUUUID:  "gpu-1",
+			MigPlacement: inferencev1alpha1.Placement{
+				Size:  3,
+				Start: 0,
+			},
+		}
+
+		// Check cleanup of incompatible profiles
+		err := r.UpdateCompatibleProfilesMetrics(*instaslice, "node-1")
+		Expect(err).ToNot(HaveOccurred()) // Ensure the function runs without errors
+
+		Expect(instasliceMetrics.compatibleProfiles.WithLabelValues("1g.5gb", "node-1")).NotTo(BeNil())
+		Expect(instasliceMetrics.compatibleProfiles.WithLabelValues("2g.10gb", "node-1")).NotTo(BeNil())
+	})
+})
diff --git a/internal/controller/metrics.go b/internal/controller/metrics.go
new file mode 100644
index 000000000..f53462afe
--- /dev/null
+++ b/internal/controller/metrics.go
@@ -0,0 +1,67 @@
+/*
+Copyright 2025.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+	http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+package controller
+
+import (
+	"fmt"
+
+	inferencev1alpha1 "github.com/openshift/instaslice-operator/api/v1alpha1"
+	v1 "k8s.io/api/core/v1"
+)
+
+// calculateProfileFitOnGPU handles both profile simulation fit and actual allocation size
+// simulate - `true` → simulate fits | `false` → check actual allocation
+func (r *InstasliceReconciler) calculateProfileFitOnGPU(instaslice *inferencev1alpha1.Instaslice, profileName, gpuUUID string, simulate bool, pod *v1.Pod) (int32, error) {
+	// Get the GPU allocation state (already allocated slices)
+	originalAllocatedIndex := r.gpuAllocatedSlices(gpuUUID)
+	// Create a copy of the allocated slots (fixed-size array)
+	var gpuAllocatedIndex [8]int32
+	copy(gpuAllocatedIndex[:], originalAllocatedIndex[:]) // Ensure we don’t modify real allocations
+	// Determine the required slice size for this profile
+	var neededContinuousSlot int32
+	placement, exists := instaslice.Status.NodeResources.MigPlacement[profileName]
+	if !exists || len(placement.Placements) == 0 {
+		return 0, fmt.Errorf("profile %s not found in MigPlacement", profileName)
+	}
+	neededContinuousSlot = placement.Placements[0].Size
+	// If we're checking actual allocation, count and return immediately
+	if !simulate {
+		actualSliceSize := int32(0)
+		startIdx := r.getStartIndexFromAllocationResults(instaslice, profileName, gpuAllocatedIndex, &pod.UID, false)
+		for i := int32(0); i < neededContinuousSlot; i++ {
+			if startIdx+i < int32(len(originalAllocatedIndex)-1) {
+				actualSliceSize++
+			}
+		}
+		return actualSliceSize, nil // Return the **actual** allocated slice count
+	}
+	// If we are simulating, count how many times the profile **could fit**
+	fitCount := int32(0)
+	for i := 0; i < len(originalAllocatedIndex); i++ {
+		startIdx := r.getStartIndexFromAllocationResults(instaslice, profileName, gpuAllocatedIndex, nil, true)
+		// If no valid placement found, break the loop
+		if startIdx == 9 {
+			break
+		}
+		// Simulate allocation by marking the slots
+		for i := int32(0); i < neededContinuousSlot; i++ {
+			if startIdx+i < int32(len(gpuAllocatedIndex)) {
+				gpuAllocatedIndex[startIdx+i] = 1
+			}
+		}
+		fitCount++ // one successful fit
+	}
+	return fitCount, nil // total hypothetical fits
+}
diff --git a/internal/controller/prometheus_manager.go b/internal/controller/prometheus_manager.go
new file mode 100644
index 000000000..6985c6a9e
--- /dev/null
+++ b/internal/controller/prometheus_manager.go
@@ -0,0 +1,104 @@
+/*
+Copyright 2023, 2024.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controller
+
+import (
+	"fmt"
+
+	inferencev1alpha1 "github.com/openshift/instaslice-operator/api/v1alpha1"
+	"github.com/prometheus/client_golang/prometheus"
+	v1 "k8s.io/api/core/v1"
+	"sigs.k8s.io/controller-runtime/pkg/metrics"
+)
+
+type InstasliceMetrics struct {
+	compatibleProfiles *prometheus.GaugeVec
+	processedSlices    *prometheus.GaugeVec
+	deployedPodTotal   *prometheus.GaugeVec
+}
+
+var (
+	instasliceMetrics = &InstasliceMetrics{
+		// deployed pod total
+		deployedPodTotal: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "instaslice_pod_processed_slices",
+			Help: "Pods that are processed with their slice/s allocation.",
+		},
+			[]string{"node", "gpu_id", "namespace", "podname", "profile"}), // Labels: node, GPU ID, namespace, podname, profile
+		// compatible profiles with remaining gpu slices
+		compatibleProfiles: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "instaslice_compatible_profiles",
+			Help: "Profiles compatible with remaining GPU slices in a node and their counts.",
+		},
+			[]string{"profile", "node"}), // Labels: profile, node
+		// total processed slices
+		processedSlices: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "instaslice_total_processed_gpu_slices",
+			Help: "Number of total processed GPU slices since instaslice controller start time.",
+		},
+			[]string{"node", "gpu_id"}), // Labels: node, GPU ID
+	}
+)
+
+// RegisterMetrics registers all Prometheus metrics
+func RegisterMetrics() {
+	// Register custom metrics with the global prometheus registry
+	metrics.Registry.MustRegister(instasliceMetrics.compatibleProfiles, instasliceMetrics.processedSlices, instasliceMetrics.deployedPodTotal)
+}
+
+// UpdateGpuSliceMetrics updates GPU slice allocation metrics
+func (r *InstasliceReconciler) IncrementTotalProcessedGpuSliceMetrics(instasliceObj inferencev1alpha1.Instaslice, nodeName, gpuID string, profile string, pod *v1.Pod) error {
+	processedSlices, err := r.calculateProfileFitOnGPU(&instasliceObj, profile, gpuID, false, pod)
+	if err != nil {
+		return fmt.Errorf("failed to calculate processed GPU slices for profile %s: %w", profile, err)
+	}
+	instasliceMetrics.processedSlices.WithLabelValues(nodeName, gpuID).Add(float64(processedSlices))
+	return nil
+}
+
+// UpdateGpuSliceMetrics updates GPU slice allocation metrics
+func (r *InstasliceReconciler) UpdateDeployedPodTotalMetrics(nodeName, gpuID, namespace, podname, profile string, size int32) error {
+	if namespace == "" && podname == "" && profile == "" {
+		return fmt.Errorf("[UpdateDeployedPodTotalMetrics] : missing required parameters, no prometheus update needed")
+	}
+	instasliceMetrics.deployedPodTotal.WithLabelValues(
+		nodeName, gpuID, namespace, podname, profile).Set(float64(size))
+	return nil
+}
+
+// UpdateCompatibleProfilesMetrics updates metrics based on remaining GPU slices and calculates compatible profiles dynamically
+// TODO: store metrics per gpu and when there is an update, calculate a fit for only one GPU instead of all GPUs on the host
+func (r *InstasliceReconciler) UpdateCompatibleProfilesMetrics(instasliceObj inferencev1alpha1.Instaslice, nodeName string) error {
+	sortedGPUs := sortGPUs(&instasliceObj)
+	// Iterate over each profile
+	for profileName, migPlacement := range instasliceObj.Status.NodeResources.MigPlacement {
+		if len(migPlacement.Placements) > 0 {
+			totalFit := int32(0)
+			// Iterate over all available GPUs
+			for _, gpuID := range sortedGPUs {
+				fit, err := r.calculateProfileFitOnGPU(&instasliceObj, profileName, gpuID, true, nil)
+				if err != nil {
+					return fmt.Errorf("failed to calculate compatible profiles for profile %s: %w", profileName, err)
+				}
+				totalFit += fit
+			}
+			// Update Prometheus metrics
+			instasliceMetrics.compatibleProfiles.WithLabelValues(profileName, nodeName).Set(float64(totalFit))
+		}
+	}
+	return nil
+}
diff --git a/internal/controller/prometheus_manager_test.go b/internal/controller/prometheus_manager_test.go
new file mode 100644
index 000000000..05da08fa3
--- /dev/null
+++ b/internal/controller/prometheus_manager_test.go
@@ -0,0 +1,124 @@
+package controller
+
+import (
+	"context"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	inferencev1alpha1 "github.com/openshift/instaslice-operator/api/v1alpha1"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+)
+
+var _ = Describe("Instaslice Controller", func() {
+	var (
+		ctx        context.Context
+		fakeClient client.Client
+		r          *InstasliceReconciler
+		instaslice *inferencev1alpha1.Instaslice
+		pod        *v1.Pod
+		podUUID    string
+	)
+
+	BeforeEach(func() {
+		ctx = context.Background()
+		scheme := runtime.NewScheme()
+		Expect(inferencev1alpha1.AddToScheme(scheme)).To(Succeed())
+		Expect(v1.AddToScheme(scheme)).To(Succeed())
+
+		fakeClient = fake.NewClientBuilder().WithScheme(scheme).Build()
+
+		r = &InstasliceReconciler{
+			Client: fakeClient,
+		}
+
+		podUUID = "test-pod-uuid"
+
+		pod = &v1.Pod{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-pod",
+				Namespace: InstaSliceOperatorNamespace,
+				UID:       types.UID(podUUID),
+				Finalizers: []string{
+					FinalizerName,
+				},
+			},
+		}
+
+		Expect(fakeClient.Create(ctx, pod)).To(Succeed())
+
+		instaslice = &inferencev1alpha1.Instaslice{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "test-instaslice",
+				Namespace: InstaSliceOperatorNamespace,
+			},
+			Spec: inferencev1alpha1.InstasliceSpec{
+				PodAllocationRequests: map[types.UID]inferencev1alpha1.AllocationRequest{
+					types.UID(podUUID): {
+						Profile: "1g.5gb",
+						PodRef: v1.ObjectReference{
+							Name:      pod.Name,
+							Namespace: InstaSliceOperatorNamespace,
+							UID:       pod.UID,
+						},
+						Resources: v1.ResourceRequirements{},
+					},
+				},
+			},
+			Status: inferencev1alpha1.InstasliceStatus{
+				PodAllocationResults: map[types.UID]inferencev1alpha1.AllocationResult{
+					types.UID(podUUID): {
+						AllocationStatus:            inferencev1alpha1.AllocationStatus{AllocationStatusController: inferencev1alpha1.AllocationStatusCreating},
+						GPUUUID:                     "GPU-12345",
+						Nodename:                    "fake-node",
+						ConfigMapResourceIdentifier: "fake-configmap-uid",
+					},
+				},
+				// ✅ Ensure MigPlacement has the correct profile
+				NodeResources: inferencev1alpha1.DiscoveredNodeResources{
+					MigPlacement: map[string]inferencev1alpha1.Mig{
+						"1g.5gb": {
+							Placements: []inferencev1alpha1.Placement{
+								{Size: 1, Start: 0}, // Example placement
+							},
+						},
+						"1g.10gb": { // Also add the missing profile from the failing test
+							Placements: []inferencev1alpha1.Placement{
+								{Size: 2, Start: 0},
+							},
+						},
+					},
+				},
+			},
+		}
+		Expect(fakeClient.Create(ctx, instaslice)).To(Succeed())
+	})
+
+	// Test IncrementTotalProcessedGpuSliceMetrics
+	It("should increment total processed GPU slice metrics", func() {
+		err := r.IncrementTotalProcessedGpuSliceMetrics(*instaslice, "node-1", "gpu-1", "1g.5gb", pod)
+		Expect(err).ToNot(HaveOccurred())
+	})
+
+	// Test UpdateDeployedPodTotalMetrics
+	It("should update deployed pod total metrics", func() {
+		err := r.UpdateDeployedPodTotalMetrics("node-1", "gpu-1", "namespace-1", "pod-1", "profile-1", 2)
+		Expect(err).ToNot(HaveOccurred())
+	})
+
+	// Test UpdateCompatibleProfilesMetrics
+	It("should update compatible profiles metrics", func() {
+		instaslice := inferencev1alpha1.Instaslice{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "instaslice-1",
+				Namespace: "default",
+			},
+		}
+		err := r.UpdateCompatibleProfilesMetrics(instaslice, "node-1")
+		Expect(err).ToNot(HaveOccurred())
+	})
+})


Reviews:
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T23:50:20Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T23:51:43Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T23:52:26Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T23:54:10Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T23:58:28Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T17:45:31Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T17:50:16Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T18:09:01Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T20:10:10Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T21:02:27Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T22:35:34Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T22:39:56Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T22:42:49Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T22:47:35Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T22:57:49Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T23:07:03Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T23:08:04Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T13:25:35Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T13:31:15Z
user: login: asm582
body: more suggestions
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T13:32:42Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-16T22:09:41Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-16T22:11:29Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-16T22:14:00Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-16T22:14:42Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-16T22:18:54Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-16T22:19:27Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T16:07:44Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T16:09:18Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T16:10:49Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T16:11:20Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T16:13:12Z
user: login: mamy-CS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T16:17:34Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:09:04Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:10:24Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:11:46Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:14:49Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:15:52Z
user: login: asm582
body: Request for changes 
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:17:06Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:23:52Z
user: login: asm582
body: adding more review
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:24:16Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:38:10Z
user: login: asm582
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:41:09Z
user: login: asm582
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T00:33:43Z

Review comments:
diff_hunk: @@ -678,15 +709,22 @@ func (l *RightToLeftPolicy) SetAllocationDetails(profileName string, newStart, s
 	return &inferencev1alpha1.AllocationRequest{}
 }
 
-func (r *InstasliceReconciler) removeInstasliceAllocation(ctx context.Context, instasliceName string, allocation *inferencev1alpha1.AllocationResult) error {
-	if allocation.AllocationStatus.AllocationStatusDaemonset == inferencev1alpha1.AllocationStatusDeleted {
+func (r *InstasliceReconciler) removeInstasliceAllocation(ctx context.Context, instasliceName string, allocResult *inferencev1alpha1.AllocationResult, allocRequest *inferencev1alpha1.AllocationRequest) error {
+	log := logr.FromContext(ctx)
+	if allocResult.AllocationStatus.AllocationStatusDaemonset == inferencev1alpha1.AllocationStatusDeleted {
 		err := utils.UpdateOrDeleteInstasliceAllocations(ctx, r.Client, instasliceName, nil, nil)
 		if err != nil {
 			return err
 		}
 	}
+	// update DeployedPodTotal Metrics by setting value to 0 as pod allocation is deleted
+	if err := r.UpdateDeployedPodTotalMetrics(string(allocResult.Nodename), allocResult.GPUUUID, allocRequest.PodRef.Namespace, allocRequest.PodRef.Name, allocRequest.Profile, 0); err != nil {
path: internal/controller/instaslice_controller.go
user: login: asm582
body: This call should not be in removeInstasliceAllocation method
created_at: 2025-03-04T23:50:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -596,10 +625,12 @@ func (r *InstasliceReconciler) SetupWithManager(mgr ctrl.Manager) error {
 		return err
 	}
 
-	return ctrl.NewControllerManagedBy(mgr).
+	controllerManager := ctrl.NewControllerManagedBy(mgr).
path: internal/controller/instaslice_controller.go
user: login: asm582
body: do we need to assign this to variable?
created_at: 2025-03-04T23:51:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -397,22 +403,38 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 				// Sort by Name in ascending order
 				return instasliceList.Items[i].Name < instasliceList.Items[j].Name
 			})
+
+			var successfulAllocRequest *inferencev1alpha1.AllocationRequest
+			var successfulAllocResult *inferencev1alpha1.AllocationResult
+			var instasliceListItemSuccess inferencev1alpha1.Instaslice
 			for _, instaslice := range instasliceList.Items {
 				// find the GPU on the node and the GPU index where the slice can be created
 				allocRequest, allocResult, err := r.findNodeAndDeviceForASlice(ctx, &instaslice, profileName, policy, pod)
 				if err != nil {
 					continue
 				}
 				podHasNodeAllocation = true
-				if podHasNodeAllocation {
-					err := utils.UpdateOrDeleteInstasliceAllocations(ctx, r.Client, instaslice.Name, allocResult, allocRequest)
-					if err != nil {
-						return ctrl.Result{Requeue: true}, nil
-					}
-					// allocation was successful
-					return ctrl.Result{}, nil
+				successfulAllocRequest = allocRequest
+				successfulAllocResult = allocResult
+				instasliceListItemSuccess = instaslice
path: internal/controller/instaslice_controller.go
user: login: asm582
body: do we need these variables?
created_at: 2025-03-04T23:52:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -191,10 +216,67 @@ func (*InstasliceReconciler) getStartIndexFromPreparedState(instaslice *inferenc
 		}
 
 	}
-
 	return newStart
 }
 
+func (r *InstasliceReconciler) actualProfileSliceSize(instaslice *inferencev1alpha1.Instaslice, profileName, gpuUUID string) int32 {
+	originalAllocatedIndex := gpuAllocatedSlices(instaslice, gpuUUID)
+	gpuAllocatedIndex := make([]int32, len(originalAllocatedIndex))
+	copy(gpuAllocatedIndex, originalAllocatedIndex) // Ensure we're not modifying the original state
+	actualSliceSize := int32(0)
+	startIdx := r.getStartIndexFromPreparedState(instaslice, profileName, gpuAllocatedIndex)
+	// Get required slice size
+	var neededContinousSlot int32
+	for profile, placement := range instaslice.Status.NodeResources.MigPlacement {
+		if profile == profileName {
+			neededContinousSlot = placement.Placements[0].Size
+			break
+		}
+	}
+	// Mark the allocated slots (simulate the placement)
+	for i := int32(0); i < neededContinousSlot; i++ {
+		if startIdx+i < int32(len(gpuAllocatedIndex)-1) {
+			gpuAllocatedIndex[startIdx+i] = 1
+			actualSliceSize++
+		}
+	}
+	return actualSliceSize
+}
+
+// getTotalFitForProfileOnGPU checks how many times a profile can fit on a given GPU.
+func (r *InstasliceReconciler) getTotalFitForProfileOnGPU(instaslice *inferencev1alpha1.Instaslice, profileName, gpuUUID string) int32 {
+	// Create a copy of the allocated GPU slots to simulate placements
+	originalAllocatedIndex := gpuAllocatedSlices(instaslice, gpuUUID)
+	gpuAllocatedIndex := make([]int32, len(originalAllocatedIndex))
+	copy(gpuAllocatedIndex, originalAllocatedIndex) // Ensure we're not modifying the original state
+	fitCount := int32(0)
+	for {
path: internal/controller/capacity.go
user: login: asm582
body: running infinite for loop!
created_at: 2025-03-04T23:54:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -397,22 +403,38 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 				// Sort by Name in ascending order
 				return instasliceList.Items[i].Name < instasliceList.Items[j].Name
 			})
+
+			var successfulAllocRequest *inferencev1alpha1.AllocationRequest
+			var successfulAllocResult *inferencev1alpha1.AllocationResult
+			var instasliceListItemSuccess inferencev1alpha1.Instaslice
 			for _, instaslice := range instasliceList.Items {
 				// find the GPU on the node and the GPU index where the slice can be created
 				allocRequest, allocResult, err := r.findNodeAndDeviceForASlice(ctx, &instaslice, profileName, policy, pod)
 				if err != nil {
 					continue
 				}
 				podHasNodeAllocation = true
-				if podHasNodeAllocation {
-					err := utils.UpdateOrDeleteInstasliceAllocations(ctx, r.Client, instaslice.Name, allocResult, allocRequest)
-					if err != nil {
-						return ctrl.Result{Requeue: true}, nil
-					}
-					// allocation was successful
-					return ctrl.Result{}, nil
+				successfulAllocRequest = allocRequest
+				successfulAllocResult = allocResult
+				instasliceListItemSuccess = instaslice
+				// Break immediately after finding a suitable allocation
+				break
+
+			}
+			if podHasNodeAllocation {
+				err := utils.UpdateOrDeleteInstasliceAllocations(ctx, r.Client, instasliceListItemSuccess.Name, successfulAllocResult, successfulAllocRequest)
+				if err != nil {
+					return ctrl.Result{Requeue: true}, err
+				}
+				// allocation was successful
+				// Update total processed GPU slices metrics
+				// Check if metrics need processing based on ObservedGeneration
+				if err := r.incrementGpuSliceMetricsWithRetry(ctx, &instasliceListItemSuccess, successfulAllocResult, successfulAllocRequest); err != nil {
path: internal/controller/instaslice_controller.go
user: login: asm582
body: is there a risk of stale update?
created_at: 2025-03-04T23:58:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -191,10 +216,67 @@ func (*InstasliceReconciler) getStartIndexFromPreparedState(instaslice *inferenc
 		}
 
 	}
-
 	return newStart
 }
 
+func (r *InstasliceReconciler) actualProfileSliceSize(instaslice *inferencev1alpha1.Instaslice, profileName, gpuUUID string) int32 {
+	originalAllocatedIndex := gpuAllocatedSlices(instaslice, gpuUUID)
+	gpuAllocatedIndex := make([]int32, len(originalAllocatedIndex))
+	copy(gpuAllocatedIndex, originalAllocatedIndex) // Ensure we're not modifying the original state
+	actualSliceSize := int32(0)
+	startIdx := r.getStartIndexFromPreparedState(instaslice, profileName, gpuAllocatedIndex)
+	// Get required slice size
+	var neededContinousSlot int32
+	for profile, placement := range instaslice.Status.NodeResources.MigPlacement {
+		if profile == profileName {
+			neededContinousSlot = placement.Placements[0].Size
+			break
+		}
+	}
+	// Mark the allocated slots (simulate the placement)
+	for i := int32(0); i < neededContinousSlot; i++ {
+		if startIdx+i < int32(len(gpuAllocatedIndex)-1) {
+			gpuAllocatedIndex[startIdx+i] = 1
+			actualSliceSize++
+		}
+	}
+	return actualSliceSize
+}
+
+// getTotalFitForProfileOnGPU checks how many times a profile can fit on a given GPU.
+func (r *InstasliceReconciler) getTotalFitForProfileOnGPU(instaslice *inferencev1alpha1.Instaslice, profileName, gpuUUID string) int32 {
+	// Create a copy of the allocated GPU slots to simulate placements
+	originalAllocatedIndex := gpuAllocatedSlices(instaslice, gpuUUID)
+	gpuAllocatedIndex := make([]int32, len(originalAllocatedIndex))
+	copy(gpuAllocatedIndex, originalAllocatedIndex) // Ensure we're not modifying the original state
+	fitCount := int32(0)
+	for {
path: internal/controller/capacity.go
user: login: mamy-CS
body: The exit condition is when the index is 9, and getStartIndexFromPreparedState always returns 9 when the GPU is full but I have bounded the loop with the max GPU index to be extra safe. 
created_at: 2025-03-06T17:45:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -596,10 +625,12 @@ func (r *InstasliceReconciler) SetupWithManager(mgr ctrl.Manager) error {
 		return err
 	}
 
-	return ctrl.NewControllerManagedBy(mgr).
+	controllerManager := ctrl.NewControllerManagedBy(mgr).
path: internal/controller/instaslice_controller.go
user: login: mamy-CS
body: We don't need to currently, both versions achieve the same thing but I added it to make it easier to debug to inspect controllerManager initially. I reverted it to the original. 
created_at: 2025-03-06T17:50:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -397,22 +403,38 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 				// Sort by Name in ascending order
 				return instasliceList.Items[i].Name < instasliceList.Items[j].Name
 			})
+
+			var successfulAllocRequest *inferencev1alpha1.AllocationRequest
+			var successfulAllocResult *inferencev1alpha1.AllocationResult
+			var instasliceListItemSuccess inferencev1alpha1.Instaslice
 			for _, instaslice := range instasliceList.Items {
 				// find the GPU on the node and the GPU index where the slice can be created
 				allocRequest, allocResult, err := r.findNodeAndDeviceForASlice(ctx, &instaslice, profileName, policy, pod)
 				if err != nil {
 					continue
 				}
 				podHasNodeAllocation = true
-				if podHasNodeAllocation {
-					err := utils.UpdateOrDeleteInstasliceAllocations(ctx, r.Client, instaslice.Name, allocResult, allocRequest)
-					if err != nil {
-						return ctrl.Result{Requeue: true}, nil
-					}
-					// allocation was successful
-					return ctrl.Result{}, nil
+				successfulAllocRequest = allocRequest
+				successfulAllocResult = allocResult
+				instasliceListItemSuccess = instaslice
path: internal/controller/instaslice_controller.go
user: login: mamy-CS
body: Those were used before ObservedGeneration, to break as soon as we found an allocation to avoid unnecessary incrementation during a request. Simplified. 
created_at: 2025-03-06T18:09:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -397,22 +403,38 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 				// Sort by Name in ascending order
 				return instasliceList.Items[i].Name < instasliceList.Items[j].Name
 			})
+
+			var successfulAllocRequest *inferencev1alpha1.AllocationRequest
+			var successfulAllocResult *inferencev1alpha1.AllocationResult
+			var instasliceListItemSuccess inferencev1alpha1.Instaslice
 			for _, instaslice := range instasliceList.Items {
 				// find the GPU on the node and the GPU index where the slice can be created
 				allocRequest, allocResult, err := r.findNodeAndDeviceForASlice(ctx, &instaslice, profileName, policy, pod)
 				if err != nil {
 					continue
 				}
 				podHasNodeAllocation = true
-				if podHasNodeAllocation {
-					err := utils.UpdateOrDeleteInstasliceAllocations(ctx, r.Client, instaslice.Name, allocResult, allocRequest)
-					if err != nil {
-						return ctrl.Result{Requeue: true}, nil
-					}
-					// allocation was successful
-					return ctrl.Result{}, nil
+				successfulAllocRequest = allocRequest
+				successfulAllocResult = allocResult
+				instasliceListItemSuccess = instaslice
+				// Break immediately after finding a suitable allocation
+				break
+
+			}
+			if podHasNodeAllocation {
+				err := utils.UpdateOrDeleteInstasliceAllocations(ctx, r.Client, instasliceListItemSuccess.Name, successfulAllocResult, successfulAllocRequest)
+				if err != nil {
+					return ctrl.Result{Requeue: true}, err
+				}
+				// allocation was successful
+				// Update total processed GPU slices metrics
+				// Check if metrics need processing based on ObservedGeneration
+				if err := r.incrementGpuSliceMetricsWithRetry(ctx, &instasliceListItemSuccess, successfulAllocResult, successfulAllocRequest); err != nil {
path: internal/controller/instaslice_controller.go
user: login: mamy-CS
body: I was being too careful with unnecessary incrementation, I have simplified. I have done significant testing and metrics only updates once during a successful allocation. 
created_at: 2025-03-06T20:10:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -678,15 +709,22 @@ func (l *RightToLeftPolicy) SetAllocationDetails(profileName string, newStart, s
 	return &inferencev1alpha1.AllocationRequest{}
 }
 
-func (r *InstasliceReconciler) removeInstasliceAllocation(ctx context.Context, instasliceName string, allocation *inferencev1alpha1.AllocationResult) error {
-	if allocation.AllocationStatus.AllocationStatusDaemonset == inferencev1alpha1.AllocationStatusDeleted {
+func (r *InstasliceReconciler) removeInstasliceAllocation(ctx context.Context, instasliceName string, allocResult *inferencev1alpha1.AllocationResult, allocRequest *inferencev1alpha1.AllocationRequest) error {
+	log := logr.FromContext(ctx)
+	if allocResult.AllocationStatus.AllocationStatusDaemonset == inferencev1alpha1.AllocationStatusDeleted {
 		err := utils.UpdateOrDeleteInstasliceAllocations(ctx, r.Client, instasliceName, nil, nil)
 		if err != nil {
 			return err
 		}
 	}
+	// update DeployedPodTotal Metrics by setting value to 0 as pod allocation is deleted
+	if err := r.UpdateDeployedPodTotalMetrics(string(allocResult.Nodename), allocResult.GPUUUID, allocRequest.PodRef.Namespace, allocRequest.PodRef.Name, allocRequest.Profile, 0); err != nil {
path: internal/controller/instaslice_controller.go
user: login: mamy-CS
body: I am worried about these 2 issues if I call it outside:
- reconcile timing issue, removeInstasliceAllocation executes as part of the cleanup, if I call UpdateDeployedPodTotalMetrics after, it may miss it.
- pod lookup issue, if removeInstasliceAllocation removes the allocation, and then later I call UpdateDeployedPodTotalMetrics, the allocation may no longer exist in the Instaslice object.
created_at: 2025-03-06T21:02:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,8 +110,8 @@ func sortGPUs(updatedInstaSliceObject *inferencev1alpha1.Instaslice) []string {
 	return gpuUUIDs
 }
 
-// accounting logic that finds the correct GPU and index where a slice could be placed.
-func (*InstasliceReconciler) getStartIndexFromPreparedState(instaslice *inferencev1alpha1.Instaslice, gpuUUID string, profileName string) int32 {
+// gpuAllocatedSlices stores already allocated slices in indexes
+func gpuAllocatedSlices(instaslice *inferencev1alpha1.Instaslice, gpuUUID string) []int32 {
path: internal/controller/capacity.go
user: login: asm582
body: Incorrect accounting: I recommend podallocationresults not be used for calculating allocated slices. This is too late as we already capture intent in podallocationrequests, so podallocationrequests should be used. For accounting logic, I think it is better to do this inside `getStartIndexFromPreparedState` we already have InstaSliceobject passed in it, this could be a function call if the method gets too long
created_at: 2025-03-10T22:26:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+/*
+Copyright 2025.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+	http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+package controller
+
+import (
+	"context"
+	"fmt"
+
+	inferencev1alpha1 "github.com/openshift/instaslice-operator/api/v1alpha1"
+	logr "sigs.k8s.io/controller-runtime/pkg/log"
+)
+
+// updateMetrics - updates UpdateDeployedPodTotalMetrics, UpdateGpuSliceMetrics and UpdateCompatibleProfilesMetrics
+func (r *InstasliceReconciler) updateMetrics(ctx context.Context, instasliceList inferencev1alpha1.InstasliceList) error {
+	log := logr.FromContext(ctx)
+	for _, instaslice := range instasliceList.Items {
+		// Fetch latest Instaslice state before updating metrics
+		updatedInstaslice, err := r.getInstasliceObject(ctx, instaslice.Name, instaslice.Namespace)
+		if err != nil {
+			log.Error(err, "Failed to get latest Instaslice object", "instaslice", instaslice.Name)
+			return err
+		}
+		sortedGPUs := sortGPUs(updatedInstaslice)
+		for _, gpuID := range sortedGPUs {
+			for podUuid, allocation := range updatedInstaslice.Status.PodAllocationResults {
+				if allocation.GPUUUID == gpuID {
+					allocRequest := updatedInstaslice.Spec.PodAllocationRequests[podUuid]
+					nodeName := allocation.Nodename
+					namespace := allocRequest.PodRef.Namespace
+					podname := allocRequest.PodRef.Name
+					profile := allocRequest.Profile
+					freeSlots, usedSlots := getFreeAndUsedSlicesCount(updatedInstaslice, gpuID)
+					// update GpuSliceMetrics
+					if err := r.UpdateGpuSliceMetrics(string(nodeName), gpuID, usedSlots, freeSlots); err != nil {
+						return fmt.Errorf("failed to update GPU slice metrics (nodeName: %s, gpuID: %s): %w", nodeName, gpuID, err)
+					}
+					// update DeployedPodTotalMetrics
+					if err := r.UpdateDeployedPodTotalMetrics(string(nodeName), gpuID, namespace, podname, profile, allocation.MigPlacement.Size); err != nil {
+						return fmt.Errorf("failed to update deployed pod metrics (nodeName: %s): %w", nodeName, err)
+					}
+				}
+			}
+		}
+		// update CompatibleProfilesMetrics
+		if err := r.UpdateCompatibleProfilesMetrics(*updatedInstaslice, updatedInstaslice.Name); err != nil {
+			log.Error(err, "Failed to update Compatible Profiles Metrics", "nodeName", updatedInstaslice.Name)
+			return err
+		}
+	}
+	return nil
+}
+
+// updateMetricsAllSlotsFree - If no allocations exist, update metrics with all slots free
+func (r *InstasliceReconciler) updateMetricsAllSlotsFree(ctx context.Context, instasliceList inferencev1alpha1.InstasliceList) error {
+	log := logr.FromContext(ctx)
+	for _, instaslice := range instasliceList.Items {
+		// Fetch latest Instaslice state before updating metrics
+		updatedInstaslice, err := r.getInstasliceObject(ctx, instaslice.Name, instaslice.Namespace)
+		if err != nil {
+			log.Error(err, "Failed to get latest Instaslice object", "instaslice", instaslice.Name)
+			return err
+		}
+		sortedGPUs := sortGPUs(updatedInstaslice)
+		for _, gpuID := range sortedGPUs {
+			nodeName := updatedInstaslice.Name
+			if len(updatedInstaslice.Spec.PodAllocationRequests) == 0 {
path: internal/controller/metrics.go
user: login: asm582
body: Okay, so we use allocation requests in this code but not in the other. I am curious to know why.
created_at: 2025-03-10T22:30:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,8 +110,8 @@ func sortGPUs(updatedInstaSliceObject *inferencev1alpha1.Instaslice) []string {
 	return gpuUUIDs
 }
 
-// accounting logic that finds the correct GPU and index where a slice could be placed.
-func (*InstasliceReconciler) getStartIndexFromPreparedState(instaslice *inferencev1alpha1.Instaslice, gpuUUID string, profileName string) int32 {
+// gpuAllocatedSlices stores already allocated slices in indexes
+func gpuAllocatedSlices(instaslice *inferencev1alpha1.Instaslice, gpuUUID string) []int32 {
path: internal/controller/capacity.go
user: login: mamy-CS
body: I haven't changed the original logic [here](https://github.com/openshift/instaslice-operator/blob/b4e7e34a36ed19e1e3836f5cf3cad86cf81c11ff/internal/controller/capacity.go#L123C1-L123C70). It uses podallocationresults. 
created_at: 2025-03-10T22:39:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+/*
+Copyright 2025.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+	http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+package controller
+
+import (
+	"context"
+	"fmt"
+
+	inferencev1alpha1 "github.com/openshift/instaslice-operator/api/v1alpha1"
+	logr "sigs.k8s.io/controller-runtime/pkg/log"
+)
+
+// updateMetrics - updates UpdateDeployedPodTotalMetrics, UpdateGpuSliceMetrics and UpdateCompatibleProfilesMetrics
+func (r *InstasliceReconciler) updateMetrics(ctx context.Context, instasliceList inferencev1alpha1.InstasliceList) error {
+	log := logr.FromContext(ctx)
+	for _, instaslice := range instasliceList.Items {
+		// Fetch latest Instaslice state before updating metrics
+		updatedInstaslice, err := r.getInstasliceObject(ctx, instaslice.Name, instaslice.Namespace)
+		if err != nil {
+			log.Error(err, "Failed to get latest Instaslice object", "instaslice", instaslice.Name)
+			return err
+		}
+		sortedGPUs := sortGPUs(updatedInstaslice)
+		for _, gpuID := range sortedGPUs {
+			for podUuid, allocation := range updatedInstaslice.Status.PodAllocationResults {
+				if allocation.GPUUUID == gpuID {
+					allocRequest := updatedInstaslice.Spec.PodAllocationRequests[podUuid]
+					nodeName := allocation.Nodename
+					namespace := allocRequest.PodRef.Namespace
+					podname := allocRequest.PodRef.Name
+					profile := allocRequest.Profile
+					freeSlots, usedSlots := getFreeAndUsedSlicesCount(updatedInstaslice, gpuID)
+					// update GpuSliceMetrics
+					if err := r.UpdateGpuSliceMetrics(string(nodeName), gpuID, usedSlots, freeSlots); err != nil {
+						return fmt.Errorf("failed to update GPU slice metrics (nodeName: %s, gpuID: %s): %w", nodeName, gpuID, err)
+					}
+					// update DeployedPodTotalMetrics
+					if err := r.UpdateDeployedPodTotalMetrics(string(nodeName), gpuID, namespace, podname, profile, allocation.MigPlacement.Size); err != nil {
+						return fmt.Errorf("failed to update deployed pod metrics (nodeName: %s): %w", nodeName, err)
+					}
+				}
+			}
+		}
+		// update CompatibleProfilesMetrics
+		if err := r.UpdateCompatibleProfilesMetrics(*updatedInstaslice, updatedInstaslice.Name); err != nil {
+			log.Error(err, "Failed to update Compatible Profiles Metrics", "nodeName", updatedInstaslice.Name)
+			return err
+		}
+	}
+	return nil
+}
+
+// updateMetricsAllSlotsFree - If no allocations exist, update metrics with all slots free
+func (r *InstasliceReconciler) updateMetricsAllSlotsFree(ctx context.Context, instasliceList inferencev1alpha1.InstasliceList) error {
+	log := logr.FromContext(ctx)
+	for _, instaslice := range instasliceList.Items {
+		// Fetch latest Instaslice state before updating metrics
+		updatedInstaslice, err := r.getInstasliceObject(ctx, instaslice.Name, instaslice.Namespace)
+		if err != nil {
+			log.Error(err, "Failed to get latest Instaslice object", "instaslice", instaslice.Name)
+			return err
+		}
+		sortedGPUs := sortGPUs(updatedInstaslice)
+		for _, gpuID := range sortedGPUs {
+			nodeName := updatedInstaslice.Name
+			if len(updatedInstaslice.Spec.PodAllocationRequests) == 0 {
path: internal/controller/metrics.go
user: login: mamy-CS
body: I haven't changed the other logic, just utilized what was there. But this is the actual metrics logic and the information needed is there in allocation requests
created_at: 2025-03-10T22:42:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -407,12 +419,25 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 				if podHasNodeAllocation {
 					err := utils.UpdateOrDeleteInstasliceAllocations(ctx, r.Client, instaslice.Name, allocResult, allocRequest)
 					if err != nil {
-						return ctrl.Result{Requeue: true}, nil
+						return ctrl.Result{Requeue: true}, err
 					}
 					// allocation was successful
+					// Update total processed GPU slices metrics
+					err = r.IncrementTotalProcessedGpuSliceMetrics(
path: internal/controller/instaslice_controller.go
user: login: asm582
body: should we retry? InstaSlice controller may have already passed the allocation updates as seen in line 420
created_at: 2025-03-10T22:47:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,8 +110,8 @@ func sortGPUs(updatedInstaSliceObject *inferencev1alpha1.Instaslice) []string {
 	return gpuUUIDs
 }
 
-// accounting logic that finds the correct GPU and index where a slice could be placed.
-func (*InstasliceReconciler) getStartIndexFromPreparedState(instaslice *inferencev1alpha1.Instaslice, gpuUUID string, profileName string) int32 {
+// gpuAllocatedSlices stores already allocated slices in indexes
+func gpuAllocatedSlices(instaslice *inferencev1alpha1.Instaslice, gpuUUID string) []int32 {
path: internal/controller/capacity.go
user: login: asm582
body: You are correct; it does not matter much for `getStartIndexFromPreparedState` since it is only one function and one place that drives allocation. However, now we have a second function that is also trying to determine the allocation state, which is called in multiple places. I think a scenario where podallocationrequests exist, but there are no podallocationresults will lead us to do double counting. Wdyt?
created_at: 2025-03-10T22:57:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -264,6 +264,12 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 			}
 			log.Info("finalizer deleted for succeeded ", "pod", pod.Name)
 		}
+		// If no allocations exist, update metrics with all slots free
+		err = r.updateMetricsAllSlotsFree(ctx, instasliceList)
path: internal/controller/instaslice_controller.go
user: login: asm582
body: I think `updateMetricsAllSlotsFree` is called for cases when pods succeeded, can you please confirm what will happen when pods fail,ie logic starting in line 188
created_at: 2025-03-10T23:06:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,8 +110,8 @@ func sortGPUs(updatedInstaSliceObject *inferencev1alpha1.Instaslice) []string {
 	return gpuUUIDs
 }
 
-// accounting logic that finds the correct GPU and index where a slice could be placed.
-func (*InstasliceReconciler) getStartIndexFromPreparedState(instaslice *inferencev1alpha1.Instaslice, gpuUUID string, profileName string) int32 {
+// gpuAllocatedSlices stores already allocated slices in indexes
+func gpuAllocatedSlices(instaslice *inferencev1alpha1.Instaslice, gpuUUID string) []int32 {
path: internal/controller/capacity.go
user: login: mamy-CS
body: If podallocationresults does not exist yet, it does not make sense to update the metrics, as the metrics are intended for slices that are already allocated. That was one of the confusions I had when the API changed and the split happened. 
created_at: 2025-03-10T23:08:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -191,10 +216,68 @@ func (*InstasliceReconciler) getStartIndexFromPreparedState(instaslice *inferenc
 		}
 
 	}
-
 	return newStart
 }
 
+func (r *InstasliceReconciler) actualProfileSliceSize(ctx context.Context, instaslice *inferencev1alpha1.Instaslice, profileName, gpuUUID string) (int32, error) {
path: internal/controller/capacity.go
user: login: asm582
body:  Can we consolidate these two methods `actualProfileSliceSize`,  `getTotalFitForProfileOnGPU` ? 
created_at: 2025-03-11T13:25:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -63,7 +63,8 @@ func (r *InstasliceReconciler) findNodeAndDeviceForASlice(ctx context.Context, i
 				updatedInstaSliceObject.Spec.PodAllocationRequests = make(map[types.UID]inferencev1alpha1.AllocationRequest)
 			}
 
-			newStart := r.getStartIndexFromPreparedState(updatedInstaSliceObject, gpuuuid, profileName)
+			gpuAllocatedIndex := gpuAllocatedSlices(updatedInstaSliceObject, gpuuuid)
+			newStart := r.getStartIndexFromPreparedState(updatedInstaSliceObject, profileName, gpuAllocatedIndex)
path: internal/controller/capacity.go
user: login: asm582
body: I vote for doing minimal changes to the code logic, `getStartIndexFromPreparedState` should already have gpuAllocatedIndex, can we return the updated array from the method? 
created_at: 2025-03-11T13:31:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,85 @@
+/*
+Copyright 2025.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+	http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+package controller
+
+import (
+	"context"
+	"fmt"
+
+	inferencev1alpha1 "github.com/openshift/instaslice-operator/api/v1alpha1"
+	logr "sigs.k8s.io/controller-runtime/pkg/log"
+)
+
+// updateMetrics - updates UpdateDeployedPodTotalMetrics, UpdateGpuSliceMetrics and UpdateCompatibleProfilesMetrics
+func (r *InstasliceReconciler) updateMetrics(ctx context.Context, instasliceList inferencev1alpha1.InstasliceList) error {
+	log := logr.FromContext(ctx)
+	for _, instaslice := range instasliceList.Items {
path: internal/controller/metrics.go
user: login: asm582
body: Why are we updating all InstaSlice objects in the cluster?
created_at: 2025-03-16T22:09:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -93,13 +93,21 @@ func (r *InstasliceReconciler) findNodeAndDeviceForASlice(ctx context.Context, i
 					v1.ResourceMemory: memoryRequest,
 				},
 			)
+			// update deployed pod total metrics
+			if err := r.UpdateDeployedPodTotalMetrics(string(allocResult.Nodename), gpuuuid, allocRequest.PodRef.Namespace, allocRequest.PodRef.Name, allocRequest.Profile, allocResult.MigPlacement.Size); err != nil {
path: internal/controller/capacity.go
user: login: asm582
body: I recommend we update metrics outside this method.
created_at: 2025-03-16T22:11:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -127,6 +136,11 @@ func (*InstasliceReconciler) getStartIndexFromPreparedState(instaslice *inferenc
 			}
 		}
 	}
+	return gpuAllocatedIndex[:]
path: internal/controller/capacity.go
user: login: asm582
body: why do we need to convert the fixed array into a slice?
created_at: 2025-03-16T22:14:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -425,10 +450,15 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 
 	}
 
+	err = r.updateMetrics(ctx, instasliceList)
path: internal/controller/instaslice_controller.go
user: login: asm582
body: I am curious to know why instasliceList is updated.
created_at: 2025-03-16T22:14:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -586,20 +617,73 @@ func (r *InstasliceReconciler) podMapFunc(ctx context.Context, obj client.Object
 	return requests
 }
 
-// SetupWithManager sets up the controller with the Manager.
+// Initialize Prometheus-compatible profiles metrics when the controller starts
+// Adds a background goroutine that waits for Instaslice objects.
+// Proceeds to setupWithManager(mgr) to start the reconciler
+// Does not block the controller from reconciling
+// UpdateCompatibleProfilesMetrics only updates Prometheus metrics,in-memory and do not persist in etcd
 func (r *InstasliceReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	restConfig := mgr.GetConfig()
-
 	var err error
 	r.kubeClient, err = kubernetes.NewForConfig(restConfig)
 	if err != nil {
 		return err
 	}
+	mgrAddErr := mgr.Add(manager.RunnableFunc(func(ctx context.Context) error {
+		log := logr.FromContext(ctx)
+		<-mgr.Elected() // Wait for leader election before executing
+		// Retry mechanism to wait for Instaslice objects
+		var instasliceList inferencev1alpha1.InstasliceList
+		retryErr := wait.PollUntilContextTimeout(ctx, 2*time.Second, 60*time.Second, true, func(ctx context.Context) (bool, error) {
+			if err := r.List(ctx, &instasliceList); err != nil {
+				log.Error(err, "Failed to list Instaslice objects, retrying...")
+				return false, nil
+			}
+			if len(instasliceList.Items) > 0 {
+				log.Info("Instaslice objects found", "count", len(instasliceList.Items))
+				return true, nil
+			}
+			log.Info("No Instaslice objects found, waiting...")
+			return false, nil
+		})
+		if retryErr != nil {
+			log.Error(retryErr, "Failed to fetch Instaslice objects after retries")
+			return nil // Do not block the controller from running, meaning reconciler starts in parallel
+		}
+		// Iterate over Instaslices and update Prometheus metrics
+		for _, instaslice := range instasliceList.Items {
path: internal/controller/instaslice_controller.go
user: login: asm582
body: What would happen when the controller crashes?,  should we wait to update all instaslice objects that already have prometheus metrics or wait for system to catch up?
created_at: 2025-03-16T22:18:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -586,20 +617,73 @@ func (r *InstasliceReconciler) podMapFunc(ctx context.Context, obj client.Object
 	return requests
 }
 
-// SetupWithManager sets up the controller with the Manager.
+// Initialize Prometheus-compatible profiles metrics when the controller starts
+// Adds a background goroutine that waits for Instaslice objects.
+// Proceeds to setupWithManager(mgr) to start the reconciler
+// Does not block the controller from reconciling
+// UpdateCompatibleProfilesMetrics only updates Prometheus metrics,in-memory and do not persist in etcd
 func (r *InstasliceReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	restConfig := mgr.GetConfig()
-
 	var err error
 	r.kubeClient, err = kubernetes.NewForConfig(restConfig)
 	if err != nil {
 		return err
 	}
+	mgrAddErr := mgr.Add(manager.RunnableFunc(func(ctx context.Context) error {
+		log := logr.FromContext(ctx)
+		<-mgr.Elected() // Wait for leader election before executing
+		// Retry mechanism to wait for Instaslice objects
+		var instasliceList inferencev1alpha1.InstasliceList
+		retryErr := wait.PollUntilContextTimeout(ctx, 2*time.Second, 60*time.Second, true, func(ctx context.Context) (bool, error) {
path: internal/controller/instaslice_controller.go
user: login: asm582
body: Please add a TODO: support daemonset fault tolerance 
created_at: 2025-03-16T22:19:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,85 @@
+/*
+Copyright 2025.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+	http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+package controller
+
+import (
+	"context"
+	"fmt"
+
+	inferencev1alpha1 "github.com/openshift/instaslice-operator/api/v1alpha1"
+	logr "sigs.k8s.io/controller-runtime/pkg/log"
+)
+
+// updateMetrics - updates UpdateDeployedPodTotalMetrics, UpdateGpuSliceMetrics and UpdateCompatibleProfilesMetrics
+func (r *InstasliceReconciler) updateMetrics(ctx context.Context, instasliceList inferencev1alpha1.InstasliceList) error {
+	log := logr.FromContext(ctx)
+	for _, instaslice := range instasliceList.Items {
path: internal/controller/metrics.go
user: login: mamy-CS
body: We are not updating the instaslice object, we are just fetching the latest state before updating Prometheus metrics and the reason is, UpdateCompatibleProfilesMetrics metrics store metrics per node and to ensure Metrics Are Always Up-to-Date. 
created_at: 2025-03-17T16:07:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -93,13 +93,21 @@ func (r *InstasliceReconciler) findNodeAndDeviceForASlice(ctx context.Context, i
 					v1.ResourceMemory: memoryRequest,
 				},
 			)
+			// update deployed pod total metrics
+			if err := r.UpdateDeployedPodTotalMetrics(string(allocResult.Nodename), gpuuuid, allocRequest.PodRef.Namespace, allocRequest.PodRef.Name, allocRequest.Profile, allocResult.MigPlacement.Size); err != nil {
path: internal/controller/capacity.go
user: login: mamy-CS
body: I am updating the metrics in that function as we discussed to ensure we are using the up to date instaslice object
created_at: 2025-03-17T16:09:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -127,6 +136,11 @@ func (*InstasliceReconciler) getStartIndexFromPreparedState(instaslice *inferenc
 			}
 		}
 	}
+	return gpuAllocatedIndex[:]
path: internal/controller/capacity.go
user: login: mamy-CS
body: I left a comment there, returning a slice to minimize expense instead of returning copy of the array and also for flexibility fo size later on as we are now generalizing size 8 for all GPUs
created_at: 2025-03-17T16:10:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -425,10 +450,15 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 
 	}
 
+	err = r.updateMetrics(ctx, instasliceList)
path: internal/controller/instaslice_controller.go
user: login: mamy-CS
body: Same comment as above
created_at: 2025-03-17T16:11:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -586,20 +617,73 @@ func (r *InstasliceReconciler) podMapFunc(ctx context.Context, obj client.Object
 	return requests
 }
 
-// SetupWithManager sets up the controller with the Manager.
+// Initialize Prometheus-compatible profiles metrics when the controller starts
+// Adds a background goroutine that waits for Instaslice objects.
+// Proceeds to setupWithManager(mgr) to start the reconciler
+// Does not block the controller from reconciling
+// UpdateCompatibleProfilesMetrics only updates Prometheus metrics,in-memory and do not persist in etcd
 func (r *InstasliceReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	restConfig := mgr.GetConfig()
-
 	var err error
 	r.kubeClient, err = kubernetes.NewForConfig(restConfig)
 	if err != nil {
 		return err
 	}
+	mgrAddErr := mgr.Add(manager.RunnableFunc(func(ctx context.Context) error {
+		log := logr.FromContext(ctx)
+		<-mgr.Elected() // Wait for leader election before executing
+		// Retry mechanism to wait for Instaslice objects
+		var instasliceList inferencev1alpha1.InstasliceList
+		retryErr := wait.PollUntilContextTimeout(ctx, 2*time.Second, 60*time.Second, true, func(ctx context.Context) (bool, error) {
+			if err := r.List(ctx, &instasliceList); err != nil {
+				log.Error(err, "Failed to list Instaslice objects, retrying...")
+				return false, nil
+			}
+			if len(instasliceList.Items) > 0 {
+				log.Info("Instaslice objects found", "count", len(instasliceList.Items))
+				return true, nil
+			}
+			log.Info("No Instaslice objects found, waiting...")
+			return false, nil
+		})
+		if retryErr != nil {
+			log.Error(retryErr, "Failed to fetch Instaslice objects after retries")
+			return nil // Do not block the controller from running, meaning reconciler starts in parallel
+		}
+		// Iterate over Instaslices and update Prometheus metrics
+		for _, instaslice := range instasliceList.Items {
path: internal/controller/instaslice_controller.go
user: login: mamy-CS
body: If the controller crashes as we are updating metrics dynamically for all objects, metrics will catch up when the controller restarts. This was what I was talking about when we were discussing. 
created_at: 2025-03-17T16:13:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -586,20 +617,73 @@ func (r *InstasliceReconciler) podMapFunc(ctx context.Context, obj client.Object
 	return requests
 }
 
-// SetupWithManager sets up the controller with the Manager.
+// Initialize Prometheus-compatible profiles metrics when the controller starts
+// Adds a background goroutine that waits for Instaslice objects.
+// Proceeds to setupWithManager(mgr) to start the reconciler
+// Does not block the controller from reconciling
+// UpdateCompatibleProfilesMetrics only updates Prometheus metrics,in-memory and do not persist in etcd
 func (r *InstasliceReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	restConfig := mgr.GetConfig()
-
 	var err error
 	r.kubeClient, err = kubernetes.NewForConfig(restConfig)
 	if err != nil {
 		return err
 	}
+	mgrAddErr := mgr.Add(manager.RunnableFunc(func(ctx context.Context) error {
+		log := logr.FromContext(ctx)
+		<-mgr.Elected() // Wait for leader election before executing
+		// Retry mechanism to wait for Instaslice objects
+		var instasliceList inferencev1alpha1.InstasliceList
+		retryErr := wait.PollUntilContextTimeout(ctx, 2*time.Second, 60*time.Second, true, func(ctx context.Context) (bool, error) {
path: internal/controller/instaslice_controller.go
user: login: mamy-CS
body: Right now metrics is not dependent on daemonset failures as it updates metrics when controller/ daemonset is healthy. 
created_at: 2025-03-17T16:17:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -93,13 +93,21 @@ func (r *InstasliceReconciler) findNodeAndDeviceForASlice(ctx context.Context, i
 					v1.ResourceMemory: memoryRequest,
 				},
 			)
+			// update deployed pod total metrics
+			if err := r.UpdateDeployedPodTotalMetrics(string(allocResult.Nodename), gpuuuid, allocRequest.PodRef.Namespace, allocRequest.PodRef.Name, allocRequest.Profile, allocResult.MigPlacement.Size); err != nil {
path: internal/controller/capacity.go
user: login: asm582
body: this call has nothing to do with `findNodeAndDeviceForASlice`, it should be called after this method exists.
created_at: 2025-03-17T18:09:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -127,6 +136,11 @@ func (*InstasliceReconciler) getStartIndexFromPreparedState(instaslice *inferenc
 			}
 		}
 	}
+	return gpuAllocatedIndex[:]
path: internal/controller/capacity.go
user: login: asm582
body: I like to have a more readable code than having to optimize for an array of length eight, please return back the array and not the slice
created_at: 2025-03-17T18:10:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -425,10 +450,15 @@ func (r *InstasliceReconciler) Reconcile(ctx context.Context, req ctrl.Request)
 
 	}
 
+	err = r.updateMetrics(ctx, instasliceList)
path: internal/controller/instaslice_controller.go
user: login: asm582
body: Since the controller already knows which instance object it has chosen for a slice, only pass the required instance object, not the list.
created_at: 2025-03-17T18:11:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -586,20 +617,73 @@ func (r *InstasliceReconciler) podMapFunc(ctx context.Context, obj client.Object
 	return requests
 }
 
-// SetupWithManager sets up the controller with the Manager.
+// Initialize Prometheus-compatible profiles metrics when the controller starts
+// Adds a background goroutine that waits for Instaslice objects.
+// Proceeds to setupWithManager(mgr) to start the reconciler
+// Does not block the controller from reconciling
+// UpdateCompatibleProfilesMetrics only updates Prometheus metrics,in-memory and do not persist in etcd
 func (r *InstasliceReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	restConfig := mgr.GetConfig()
-
 	var err error
 	r.kubeClient, err = kubernetes.NewForConfig(restConfig)
 	if err != nil {
 		return err
 	}
+	mgrAddErr := mgr.Add(manager.RunnableFunc(func(ctx context.Context) error {
+		log := logr.FromContext(ctx)
+		<-mgr.Elected() // Wait for leader election before executing
+		// Retry mechanism to wait for Instaslice objects
+		var instasliceList inferencev1alpha1.InstasliceList
+		retryErr := wait.PollUntilContextTimeout(ctx, 2*time.Second, 60*time.Second, true, func(ctx context.Context) (bool, error) {
+			if err := r.List(ctx, &instasliceList); err != nil {
+				log.Error(err, "Failed to list Instaslice objects, retrying...")
+				return false, nil
+			}
+			if len(instasliceList.Items) > 0 {
+				log.Info("Instaslice objects found", "count", len(instasliceList.Items))
+				return true, nil
+			}
+			log.Info("No Instaslice objects found, waiting...")
+			return false, nil
+		})
+		if retryErr != nil {
+			log.Error(retryErr, "Failed to fetch Instaslice objects after retries")
+			return nil // Do not block the controller from running, meaning reconciler starts in parallel
+		}
+		// Iterate over Instaslices and update Prometheus metrics
+		for _, instaslice := range instasliceList.Items {
path: internal/controller/instaslice_controller.go
user: login: asm582
body: I dont understand, if cluster has 10 instaslice objects, your code will re-populate all instaslice objects, all I am asking is if it's worth skipping an update for a faster boot ?
created_at: 2025-03-17T18:14:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -586,20 +617,73 @@ func (r *InstasliceReconciler) podMapFunc(ctx context.Context, obj client.Object
 	return requests
 }
 
-// SetupWithManager sets up the controller with the Manager.
+// Initialize Prometheus-compatible profiles metrics when the controller starts
+// Adds a background goroutine that waits for Instaslice objects.
+// Proceeds to setupWithManager(mgr) to start the reconciler
+// Does not block the controller from reconciling
+// UpdateCompatibleProfilesMetrics only updates Prometheus metrics,in-memory and do not persist in etcd
 func (r *InstasliceReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	restConfig := mgr.GetConfig()
-
 	var err error
 	r.kubeClient, err = kubernetes.NewForConfig(restConfig)
 	if err != nil {
 		return err
 	}
+	mgrAddErr := mgr.Add(manager.RunnableFunc(func(ctx context.Context) error {
+		log := logr.FromContext(ctx)
+		<-mgr.Elected() // Wait for leader election before executing
+		// Retry mechanism to wait for Instaslice objects
+		var instasliceList inferencev1alpha1.InstasliceList
+		retryErr := wait.PollUntilContextTimeout(ctx, 2*time.Second, 60*time.Second, true, func(ctx context.Context) (bool, error) {
path: internal/controller/instaslice_controller.go
user: login: asm582
body: if daemonset fails the code still re-populates those objects, I don't see any filtering
created_at: 2025-03-17T18:15:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+/*
+Copyright 2023, 2024.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controller
+
+import (
+	"fmt"
+
+	inferencev1alpha1 "github.com/openshift/instaslice-operator/api/v1alpha1"
+	"github.com/prometheus/client_golang/prometheus"
+	"sigs.k8s.io/controller-runtime/pkg/metrics"
+)
+
+type InstasliceMetrics struct {
+	compatibleProfiles *prometheus.GaugeVec
+	processedSlices    *prometheus.GaugeVec
+	deployedPodTotal   *prometheus.GaugeVec
+}
+
+var (
+	instasliceMetrics = &InstasliceMetrics{
+		// deployed pod total
+		deployedPodTotal: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "instaslice_pod_processed_slices",
+			Help: "Pods that are processed with their slice/s allocation.",
+		},
+			[]string{"node", "gpu_id", "namespace", "podname", "profile"}), // Labels: node, GPU ID, namespace, podname, profile
+		// compatible profiles with remaining gpu slices
+		compatibleProfiles: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "instaslice_compatible_profiles",
+			Help: "Profiles compatible with remaining GPU slices in a node and their counts.",
+		},
+			[]string{"profile", "node"}), // Labels: profile, node
+		// total processed slices
+		processedSlices: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "instaslice_total_processed_gpu_slices",
+			Help: "Number of total processed GPU slices since instaslice controller start time.",
+		},
+			[]string{"node", "gpu_id"}), // Labels: node, GPU ID
+	}
+)
+
+// RegisterMetrics registers all Prometheus metrics
+func RegisterMetrics() {
+	// Register custom metrics with the global prometheus registry
+	metrics.Registry.MustRegister(instasliceMetrics.compatibleProfiles, instasliceMetrics.processedSlices, instasliceMetrics.deployedPodTotal)
+}
+
+// UpdateGpuSliceMetrics updates GPU slice allocation metrics
+func (r *InstasliceReconciler) IncrementTotalProcessedGpuSliceMetrics(instasliceObj inferencev1alpha1.Instaslice, nodeName, gpuID string, profile string) error {
+	processedSlices, err := r.calculateProfileFitOnGPU(&instasliceObj, profile, gpuID, false)
+	if err != nil {
+		return fmt.Errorf("failed to calculate processed GPU slices for profile %s: %w", profile, err)
+	}
+	instasliceMetrics.processedSlices.WithLabelValues(
+		nodeName, gpuID).Add(float64(processedSlices))
+	return nil
+}
+
+// UpdateGpuSliceMetrics updates GPU slice allocation metrics
+func (r *InstasliceReconciler) UpdateDeployedPodTotalMetrics(nodeName, gpuID, namespace, podname, profile string, size int32) error {
+	if namespace == "" && podname == "" && profile == "" {
+		return fmt.Errorf("[UpdateDeployedPodTotalMetrics] : missing required parameters, no prometheus update needed")
+	}
+	instasliceMetrics.deployedPodTotal.WithLabelValues(
+		nodeName, gpuID, namespace, podname, profile).Set(float64(size))
+	return nil
+}
+
+// UpdateCompatibleProfilesMetrics updates metrics based on remaining GPU slices and calculates compatible profiles dynamically
+func (r *InstasliceReconciler) UpdateCompatibleProfilesMetrics(instasliceObj inferencev1alpha1.Instaslice, nodeName string) error {
+	sortedGPUs := sortGPUs(&instasliceObj)
+	// Iterate over each profile
+	for profileName, migPlacement := range instasliceObj.Status.NodeResources.MigPlacement {
+		if len(migPlacement.Placements) > 0 {
+			totalFit := int32(0)
+			// Iterate over all available GPUs
+			for _, gpuID := range sortedGPUs {
path: internal/controller/prometheus_manager.go
user: login: asm582
body: calculate a fit for only one GPU instead of all GPUs on the host, ie I don't think we need a for loop here.
created_at: 2025-03-17T18:23:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,103 @@
+/*
+Copyright 2023, 2024.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controller
+
+import (
+	"fmt"
+
+	inferencev1alpha1 "github.com/openshift/instaslice-operator/api/v1alpha1"
+	"github.com/prometheus/client_golang/prometheus"
+	"sigs.k8s.io/controller-runtime/pkg/metrics"
+)
+
+type InstasliceMetrics struct {
+	compatibleProfiles *prometheus.GaugeVec
+	processedSlices    *prometheus.GaugeVec
+	deployedPodTotal   *prometheus.GaugeVec
+}
+
+var (
+	instasliceMetrics = &InstasliceMetrics{
+		// deployed pod total
+		deployedPodTotal: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "instaslice_pod_processed_slices",
+			Help: "Pods that are processed with their slice/s allocation.",
+		},
+			[]string{"node", "gpu_id", "namespace", "podname", "profile"}), // Labels: node, GPU ID, namespace, podname, profile
+		// compatible profiles with remaining gpu slices
+		compatibleProfiles: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "instaslice_compatible_profiles",
+			Help: "Profiles compatible with remaining GPU slices in a node and their counts.",
+		},
+			[]string{"profile", "node"}), // Labels: profile, node
+		// total processed slices
+		processedSlices: prometheus.NewGaugeVec(prometheus.GaugeOpts{
+			Name: "instaslice_total_processed_gpu_slices",
+			Help: "Number of total processed GPU slices since instaslice controller start time.",
+		},
+			[]string{"node", "gpu_id"}), // Labels: node, GPU ID
+	}
+)
+
+// RegisterMetrics registers all Prometheus metrics
+func RegisterMetrics() {
+	// Register custom metrics with the global prometheus registry
+	metrics.Registry.MustRegister(instasliceMetrics.compatibleProfiles, instasliceMetrics.processedSlices, instasliceMetrics.deployedPodTotal)
+}
+
+// UpdateGpuSliceMetrics updates GPU slice allocation metrics
+func (r *InstasliceReconciler) IncrementTotalProcessedGpuSliceMetrics(instasliceObj inferencev1alpha1.Instaslice, nodeName, gpuID string, profile string) error {
+	processedSlices, err := r.calculateProfileFitOnGPU(&instasliceObj, profile, gpuID, false)
+	if err != nil {
+		return fmt.Errorf("failed to calculate processed GPU slices for profile %s: %w", profile, err)
+	}
+	instasliceMetrics.processedSlices.WithLabelValues(
+		nodeName, gpuID).Add(float64(processedSlices))
+	return nil
+}
+
+// UpdateGpuSliceMetrics updates GPU slice allocation metrics
+func (r *InstasliceReconciler) UpdateDeployedPodTotalMetrics(nodeName, gpuID, namespace, podname, profile string, size int32) error {
+	if namespace == "" && podname == "" && profile == "" {
+		return fmt.Errorf("[UpdateDeployedPodTotalMetrics] : missing required parameters, no prometheus update needed")
+	}
+	instasliceMetrics.deployedPodTotal.WithLabelValues(
+		nodeName, gpuID, namespace, podname, profile).Set(float64(size))
+	return nil
+}
+
+// UpdateCompatibleProfilesMetrics updates metrics based on remaining GPU slices and calculates compatible profiles dynamically
+func (r *InstasliceReconciler) UpdateCompatibleProfilesMetrics(instasliceObj inferencev1alpha1.Instaslice, nodeName string) error {
+	sortedGPUs := sortGPUs(&instasliceObj)
+	// Iterate over each profile
+	for profileName, migPlacement := range instasliceObj.Status.NodeResources.MigPlacement {
+		if len(migPlacement.Placements) > 0 {
+			totalFit := int32(0)
+			// Iterate over all available GPUs
+			for _, gpuID := range sortedGPUs {
path: internal/controller/prometheus_manager.go
user: login: asm582
body: Please add a TODO
created_at: 2025-03-17T18:38:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -586,20 +617,73 @@ func (r *InstasliceReconciler) podMapFunc(ctx context.Context, obj client.Object
 	return requests
 }
 
-// SetupWithManager sets up the controller with the Manager.
+// Initialize Prometheus-compatible profiles metrics when the controller starts
+// Adds a background goroutine that waits for Instaslice objects.
+// Proceeds to setupWithManager(mgr) to start the reconciler
+// Does not block the controller from reconciling
+// UpdateCompatibleProfilesMetrics only updates Prometheus metrics,in-memory and do not persist in etcd
 func (r *InstasliceReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	restConfig := mgr.GetConfig()
-
 	var err error
 	r.kubeClient, err = kubernetes.NewForConfig(restConfig)
 	if err != nil {
 		return err
 	}
+	mgrAddErr := mgr.Add(manager.RunnableFunc(func(ctx context.Context) error {
+		log := logr.FromContext(ctx)
+		<-mgr.Elected() // Wait for leader election before executing
+		// Retry mechanism to wait for Instaslice objects
+		var instasliceList inferencev1alpha1.InstasliceList
+		retryErr := wait.PollUntilContextTimeout(ctx, 2*time.Second, 60*time.Second, true, func(ctx context.Context) (bool, error) {
path: internal/controller/instaslice_controller.go
user: login: asm582
body: Please add a TODO to figure out skipping logic
created_at: 2025-03-17T18:41:09Z
author_association: CONTRIBUTOR

PR comments:
user: login: mamy-CS
created_at: 2025-03-04T22:05:55Z
body: /hold for final testing
author_association: CONTRIBUTOR
user: login: mamy-CS
created_at: 2025-03-04T22:35:23Z
body: /hold cancel
author_association: CONTRIBUTOR
user: login: mamy-CS
created_at: 2025-03-04T22:36:01Z
body: /assign @mamy-CS 
author_association: CONTRIBUTOR
user: login: asm582
created_at: 2025-03-04T23:46:34Z
body: @mamy-CS Have you tested this on real GPUs?
author_association: CONTRIBUTOR
user: login: asm582
created_at: 2025-03-05T00:21:33Z
body: /hold I think there are a few places in the code that need a discussion on the possibility of stale update(s)
author_association: CONTRIBUTOR
user: login: asm582
created_at: 2025-03-19T16:25:18Z
body: All tests run on a single node. Metrics PR will address TODO's in future PRs. 
/hold cancel
author_association: CONTRIBUTOR
user: login: mamy-CS
created_at: 2025-03-19T22:30:02Z
body: /hold bugs introduced with the new changes in main. 
author_association: CONTRIBUTOR
user: login: mamy-CS
created_at: 2025-03-19T23:48:21Z
body: /hold cancel 
author_association: CONTRIBUTOR
user: login: asm582
created_at: 2025-03-19T23:56:13Z
body: Metrics PR is primarily catered towards KinD deployment and e2e's passes in emulator and non-emulator mode.

/lgtm
author_association: CONTRIBUTOR
user: login: openshift-ci[bot]
created_at: 2025-03-20T00:01:42Z
body: @mamy-CS: all tests **passed!**


[Full PR test history](https://prow.ci.openshift.org/pr-history?org=openshift&repo=instaslice-operator&pr=472). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:mamy-CS).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
author_association: NONE
user: login: asm582
created_at: 2025-03-20T00:15:35Z
body: /approve
author_association: CONTRIBUTOR
user: login: openshift-ci[bot]
created_at: 2025-03-20T00:33:56Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/openshift/instaslice-operator/pull/472#pullrequestreview-2700704405" title="Approved">asm582</a>*, *<a href="https://github.com/openshift/instaslice-operator/pull/472#" title="Author self-approved">mamy-CS</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=openshift%2Finstaslice-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/openshift/instaslice-operator/blob/main/OWNERS)~~ [asm582,mamy-CS]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE