Repository: argoproj-labs/argocd-operator
Pull request #830 (2022-12-16 21:23:00)

feat: Expose instance level metrics
**What type of PR is this?**
> Uncomment only one ` /kind` line, and delete the rest.
> For example, `> /kind bug` would simply become: `/kind bug`

> /kind bug
> /kind chore
> /kind cleanup
> /kind failing-test
/kind enhancement
> /kind documentation
> /kind code-refactoring


**What does this PR do / why we need it**:
This PR exposes new metrics to track the statuses of the argo-cd workloads. It defines the following metrics:
```
argocd_application_controller_status Describes the status of the application controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
argocd_applicationset_controller_status Describes the status of the applicationSet controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
argocd_dex_status Describes the status of the dex workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
argocd_phase Describes the phase of argo-cd instance [2='Pending', 4='Available']
argocd_redis_status Describes the status of the redis workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
argocd_repo_server_status Describes the status of the repo server workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
argocd_server_status Describes the status of the argo-cd server workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
```
It spins up a new metrics server that listens on port `8085` 
workload status reconciliation logic is updated to reflect the status within the newly defined metrics and write them out to the `/metrics` endpoint. Individual workload statuses can be queried by specifying the namespace of the instance that the workload is a part of.

***NOTE:*** this PR assumes there will not be more than 1 argo-cd instance in any namespace as that is an anti-pattern.

This PR should be merged after https://github.com/argoproj-labs/argocd-operator/pull/829


**Have you updated the necessary documentation?**

* [ ] Documentation update is required by this PR.
* [ ] Documentation has been updated.

**Which issue(s) this PR fixes**:

Fixes https://issues.redhat.com/browse/GITOPS-2456

**How to test changes / Special notes to the reviewer**:

1. Deploy operator locally
2. Create argo-cd instance in `argocd` namespace
3. Run a `GET` query against `localhost:8085/metrics` in postman/curl 
4. Verify that the response looks something like this :
```
# HELP argocd_application_controller_status Describes the status of the application controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
# TYPE argocd_application_controller_status gauge
argocd_application_controller_status{namespace="argocd"} 3
# HELP argocd_applicationset_controller_status Describes the status of the applicationSet controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
# TYPE argocd_applicationset_controller_status gauge
argocd_applicationset_controller_status{namespace="argocd"} 0
# HELP argocd_dex_status Describes the status of the dex workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
# TYPE argocd_dex_status gauge
argocd_dex_status{namespace="argocd"} 0
# HELP argocd_phase Describes the phase of argo-cd instance [2='Pending', 4='Available']
# TYPE argocd_phase gauge
argocd_phase{namespace="argocd"} 4
# HELP argocd_redis_status Describes the status of the redis workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
# TYPE argocd_redis_status gauge
argocd_redis_status{namespace="argocd"} 3
# HELP argocd_repo_server_status Describes the status of the repo server workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
# TYPE argocd_repo_server_status gauge
argocd_repo_server_status{namespace="argocd"} 3
# HELP argocd_server_status Describes the status of the argo-cd server workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
# TYPE argocd_server_status gauge
argocd_server_status{namespace="argocd"} 3
```
5. Edit Argo CD CR to enable notifications, and replace appset image by invalid one to create pending status
```
apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  name: example-argocd
spec:
  applicationSet:
    image: quay.io/argoproj/argocd@sha256:8283a9f06033c2377dc61b03daf49
  notifications:
    enabled: true
```
6. Query again and verify that response now contains pending status for appset and running status for notifications controller
```
# HELP argocd_applicationset_controller_status Describes the status of the applicationSet controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
# TYPE argocd_applicationset_controller_status gauge
argocd_applicationset_controller_status{namespace="argocd"} 2
...
# HELP argocd_notifications_controller_status Describes the status of the notifications controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
# TYPE argocd_notifications_controller_status gauge
argocd_notifications_controller_status{namespace="argocd"} 3
```

iam-veeramalla: Do you want to changes the available gauge value to 4 ? and reserve 3 for failed ?

# HELP argocd_phase Describes the phase of argo-cd instance [1='Pending', 3='Available']
iam-veeramalla: Please update the gauge values to `[0='Unknown', 1='Pending', 2='Running', 3='Failure']` to reflect the failure.
iam-veeramalla: We should also consider more granular alerts such as `CrashLoopBackOff`, `ErrImgPull`, `ImagePullBackOff` instead of `Pending` or `Failure`.

I can agree that this will be a good start and my above suggestion can be a follow up PR.
iam-veeramalla: I remember the PR had made some metadata changes to the (0.5.0) CRD and CSV which explains about these metrics. Thanks for removing them from `0.5.0` but should we add them to `0.6.0` ?
jaideepr97: @iam-veeramalla This particular PR doesn't make any changes to CRD/CSV
the changes i had to remove from 0.5.0 bundle were those included in the previous PR to update appset status in the CR (since this branch was based off the branch for that PR) 

Patch:
diff --git a/common/defaults.go b/common/defaults.go
index 7eb0f23b..c95b0ec6 100644
--- a/common/defaults.go
+++ b/common/defaults.go
@@ -237,6 +237,9 @@ const (
 	// ArgoCDDefaultRepoMetricsPort is the default listen port for the Argo CD repo server metrics.
 	ArgoCDDefaultRepoMetricsPort = 8084
 
+	// ArgoCDDefaultInstanceStatusMetricsPort is the default listen port for the Argo CD workload status metrics.
+	ArgoCDDefaultInstanceStatusMetricsPort = 8085
+
 	// ArgoCDDefaultRepoServerPort is the default listen port for the Argo CD repo server.
 	ArgoCDDefaultRepoServerPort = 8081
 
diff --git a/controllers/argocd/metrics.go b/controllers/argocd/metrics.go
new file mode 100644
index 00000000..357b38dc
--- /dev/null
+++ b/controllers/argocd/metrics.go
@@ -0,0 +1,145 @@
+package argocd
+
+import (
+	"fmt"
+	"net/http"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/promauto"
+	"github.com/prometheus/client_golang/prometheus/promhttp"
+)
+
+// workloadState is a named int type that is used to characterize the state (unknown/pending/running/available) a workload is in
+type workloadState int
+
+// numerical representation of workloadStates to encode this information into prometheus
+const (
+	WorkloadUnknownState   workloadState = 0
+	WorkloadFailedState    workloadState = 1
+	WorkloadPendingState   workloadState = 2
+	WorkloadRunningState   workloadState = 3
+	WorkloadAvailableState workloadState = 4
+
+	// MetricsPath is the endpoint to collect instance level metrics
+	MetricsPath = "/metrics"
+)
+
+type WorkloadStatusTrackerMetrics struct {
+	applicationControllerStatus    *prometheus.GaugeVec
+	applicationSetControllerStatus *prometheus.GaugeVec
+	dexStatus                      *prometheus.GaugeVec
+	notificationsControllerStatus  *prometheus.GaugeVec
+	serverStatus                   *prometheus.GaugeVec
+	repoServerStatus               *prometheus.GaugeVec
+	redisStatus                    *prometheus.GaugeVec
+	argoCDPhase                    *prometheus.GaugeVec
+}
+
+var instanceStatusTracker *WorkloadStatusTrackerMetrics
+
+// StartMetricsServer starts a new HTTP server for metrics on given port
+func StartMetricsServer(port int) chan error {
+	errCh := make(chan error)
+	go func() {
+		sm := http.NewServeMux()
+		sm.Handle(MetricsPath, promhttp.Handler())
+		errCh <- http.ListenAndServe(fmt.Sprintf(":%d", port), sm)
+	}()
+	return errCh
+}
+
+// NewWorkloadStatusTrackerMetrics returns a new WorkloadStatustrackerMetrics object
+func NewWorkloadStatusTrackerMetrics() *WorkloadStatusTrackerMetrics {
+	workloadStatusTrackers := &WorkloadStatusTrackerMetrics{}
+
+	workloadStatusTrackers.applicationControllerStatus = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "argocd_application_controller_status",
+		Help: "Describes the status of the application controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']",
+	}, []string{"namespace"})
+
+	workloadStatusTrackers.applicationSetControllerStatus = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "argocd_applicationset_controller_status",
+		Help: "Describes the status of the applicationSet controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']",
+	}, []string{"namespace"})
+
+	workloadStatusTrackers.dexStatus = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "argocd_dex_status",
+		Help: "Describes the status of the dex workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']",
+	}, []string{"namespace"})
+
+	workloadStatusTrackers.notificationsControllerStatus = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "argocd_notifications_controller_status",
+		Help: "Describes the status of the notifications controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']",
+	}, []string{"namespace"})
+
+	workloadStatusTrackers.serverStatus = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "argocd_server_status",
+		Help: "Describes the status of the argo-cd server workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']",
+	}, []string{"namespace"})
+
+	workloadStatusTrackers.repoServerStatus = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "argocd_repo_server_status",
+		Help: "Describes the status of the repo server workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']",
+	}, []string{"namespace"})
+
+	workloadStatusTrackers.redisStatus = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "argocd_redis_status",
+		Help: "Describes the status of the redis workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']",
+	}, []string{"namespace"})
+
+	workloadStatusTrackers.argoCDPhase = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "argocd_phase",
+		Help: "Describes the phase of argo-cd instance [2='Pending', 4='Available']",
+	}, []string{"namespace"})
+
+	return workloadStatusTrackers
+}
+
+// InstanceStatus returns the global instance status tracker object. This stores the various workload status metrics partitioned by instance namespace
+func InstanceStatusTracker() *WorkloadStatusTrackerMetrics {
+	return instanceStatusTracker
+}
+
+// SetApplicationControllerStatus sets the workloadStatus of the application controller workload for the instance present in the supplied namespace
+func (wstm *WorkloadStatusTrackerMetrics) SetApplicationControllerStatus(namespace string, state workloadState) {
+	wstm.applicationControllerStatus.WithLabelValues(namespace).Set(float64(state))
+}
+
+// SetApplicationSetControllerStatus sets the workloadStatus of the applicationSet controller workload for the instance present in the supplied namespace
+func (wstm *WorkloadStatusTrackerMetrics) SetApplicationSetControllerStatus(namespace string, state workloadState) {
+	wstm.applicationSetControllerStatus.WithLabelValues(namespace).Set(float64(state))
+}
+
+// SetDexStatus sets the workloadStatus of the dex workload for the instance present in the supplied namespace
+func (wstm *WorkloadStatusTrackerMetrics) SetDexStatus(namespace string, state workloadState) {
+	wstm.dexStatus.WithLabelValues(namespace).Set(float64(state))
+}
+
+// SetNotificationsControllerStatus sets the workloadStatus of the notifications controller workload for the instance present in the supplied namespace
+func (wstm *WorkloadStatusTrackerMetrics) SetNotificationsControllerStatus(namespace string, state workloadState) {
+	wstm.notificationsControllerStatus.WithLabelValues(namespace).Set(float64(state))
+}
+
+// SetServerStatus sets the workloadStatus of the server workload for the instance present in the supplied namespace
+func (wstm *WorkloadStatusTrackerMetrics) SetServerStatus(namespace string, state workloadState) {
+	wstm.serverStatus.WithLabelValues(namespace).Set(float64(state))
+}
+
+// SetRepoServerStatus sets the workloadStatus of the repo-server controller workload for the instance present in the supplied namespace
+func (wstm *WorkloadStatusTrackerMetrics) SetRepoServerStatus(namespace string, state workloadState) {
+	wstm.repoServerStatus.WithLabelValues(namespace).Set(float64(state))
+}
+
+// SetRedisStatus sets the workloadStatus of the application controller workload for the instance present in the supplied namespace
+func (wstm *WorkloadStatusTrackerMetrics) SetRedisStatus(namespace string, state workloadState) {
+	wstm.redisStatus.WithLabelValues(namespace).Set(float64(state))
+}
+
+// SetArgoCDPhase sets the phase of the instance present in the supplied namespace
+func (wstm *WorkloadStatusTrackerMetrics) SetArgoCDPhase(namespace string, state workloadState) {
+	wstm.argoCDPhase.WithLabelValues(namespace).Set(float64(state))
+}
+
+func init() {
+	instanceStatusTracker = NewWorkloadStatusTrackerMetrics()
+}
diff --git a/controllers/argocd/status.go b/controllers/argocd/status.go
index 0f6a9a56..c3581766 100644
--- a/controllers/argocd/status.go
+++ b/controllers/argocd/status.go
@@ -90,6 +90,16 @@ func (r *ReconcileArgoCD) reconcileStatusApplicationController(cr *argoprojv1a1.
 
 	if cr.Status.ApplicationController != status {
 		cr.Status.ApplicationController = status
+
+		// Update status of applicaiton controller workload for current instance and expose as metric to prometheus
+		if cr.Status.ApplicationController == "Pending" {
+			InstanceStatusTracker().SetApplicationControllerStatus(cr.Namespace, WorkloadPendingState)
+		} else if cr.Status.ApplicationController == "Running" {
+			InstanceStatusTracker().SetApplicationControllerStatus(cr.Namespace, WorkloadRunningState)
+		} else if cr.Status.ApplicationController == "Unknown" {
+			InstanceStatusTracker().SetApplicationControllerStatus(cr.Namespace, WorkloadUnknownState)
+		}
+
 		return r.Client.Status().Update(context.TODO(), cr)
 	}
 	return nil
@@ -112,6 +122,16 @@ func (r *ReconcileArgoCD) reconcileStatusDex(cr *argoprojv1a1.ArgoCD) error {
 
 	if cr.Status.Dex != status {
 		cr.Status.Dex = status
+
+		// Update status of dex workload for current instance and expose as metric to prometheus
+		if cr.Status.Dex == "Pending" {
+			InstanceStatusTracker().SetDexStatus(cr.Namespace, WorkloadPendingState)
+		} else if cr.Status.Dex == "Running" {
+			InstanceStatusTracker().SetDexStatus(cr.Namespace, WorkloadRunningState)
+		} else if cr.Status.Dex == "Unknown" {
+			InstanceStatusTracker().SetDexStatus(cr.Namespace, WorkloadUnknownState)
+		}
+
 		return r.Client.Status().Update(context.TODO(), cr)
 	}
 	return nil
@@ -134,8 +154,19 @@ func (r *ReconcileArgoCD) reconcileStatusApplicationSetController(cr *argoprojv1
 
 	if cr.Status.ApplicationSetController != status {
 		cr.Status.ApplicationSetController = status
+
+		// Update status of applicationSet controller workload for current instance and expose as metric to prometheus
+		if cr.Status.ApplicationSetController == "Pending" {
+			InstanceStatusTracker().SetApplicationSetControllerStatus(cr.Namespace, WorkloadPendingState)
+		} else if cr.Status.ApplicationSetController == "Running" {
+			InstanceStatusTracker().SetApplicationSetControllerStatus(cr.Namespace, WorkloadRunningState)
+		} else if cr.Status.ApplicationSetController == "Unknown" {
+			InstanceStatusTracker().SetApplicationSetControllerStatus(cr.Namespace, WorkloadUnknownState)
+		}
+
 		return r.Client.Status().Update(context.TODO(), cr)
 	}
+
 	return nil
 }
 
@@ -158,8 +189,12 @@ func (r *ReconcileArgoCD) reconcileStatusPhase(cr *argoprojv1a1.ArgoCD) error {
 
 	if cr.Status.ApplicationController == "Running" && cr.Status.Redis == "Running" && cr.Status.Repo == "Running" && cr.Status.Server == "Running" {
 		phase = "Available"
+
+		// Update phase of current instance and expose as metric to prometheus
+		InstanceStatusTracker().SetArgoCDPhase(cr.Namespace, WorkloadAvailableState)
 	} else {
 		phase = "Pending"
+		InstanceStatusTracker().SetArgoCDPhase(cr.Namespace, WorkloadPendingState)
 	}
 
 	if cr.Status.Phase != phase {
@@ -198,6 +233,16 @@ func (r *ReconcileArgoCD) reconcileStatusRedis(cr *argoprojv1a1.ArgoCD) error {
 
 	if cr.Status.Redis != status {
 		cr.Status.Redis = status
+
+		// Update status of redis workload for current instance and expose as metric to prometheus
+		if cr.Status.Redis == "Pending" {
+			InstanceStatusTracker().SetRedisStatus(cr.Namespace, WorkloadPendingState)
+		} else if cr.Status.Redis == "Running" {
+			InstanceStatusTracker().SetRedisStatus(cr.Namespace, WorkloadRunningState)
+		} else if cr.Status.Redis == "Unknown" {
+			InstanceStatusTracker().SetRedisStatus(cr.Namespace, WorkloadUnknownState)
+		}
+
 		return r.Client.Status().Update(context.TODO(), cr)
 	}
 	return nil
@@ -220,6 +265,16 @@ func (r *ReconcileArgoCD) reconcileStatusRepo(cr *argoprojv1a1.ArgoCD) error {
 
 	if cr.Status.Repo != status {
 		cr.Status.Repo = status
+
+		// Update status of repo-server workload for current instance and expose as metric to prometheus
+		if cr.Status.Repo == "Pending" {
+			InstanceStatusTracker().SetRepoServerStatus(cr.Namespace, WorkloadPendingState)
+		} else if cr.Status.Repo == "Running" {
+			InstanceStatusTracker().SetRepoServerStatus(cr.Namespace, WorkloadRunningState)
+		} else if cr.Status.Repo == "Unknown" {
+			InstanceStatusTracker().SetRepoServerStatus(cr.Namespace, WorkloadUnknownState)
+		}
+
 		return r.Client.Status().Update(context.TODO(), cr)
 	}
 	return nil
@@ -243,6 +298,16 @@ func (r *ReconcileArgoCD) reconcileStatusServer(cr *argoprojv1a1.ArgoCD) error {
 
 	if cr.Status.Server != status {
 		cr.Status.Server = status
+
+		// Update status of server workload for current instance and expose as metric to prometheus
+		if cr.Status.Server == "Pending" {
+			InstanceStatusTracker().SetServerStatus(cr.Namespace, WorkloadPendingState)
+		} else if cr.Status.Server == "Running" {
+			InstanceStatusTracker().SetServerStatus(cr.Namespace, WorkloadRunningState)
+		} else if cr.Status.Server == "Unknown" {
+			InstanceStatusTracker().SetServerStatus(cr.Namespace, WorkloadUnknownState)
+		}
+
 		return r.Client.Status().Update(context.TODO(), cr)
 	}
 	return nil
@@ -268,6 +333,15 @@ func (r *ReconcileArgoCD) reconcileStatusNotifications(cr *argoprojv1a1.ArgoCD)
 			cr.Status.NotificationsController = ""
 		} else {
 			cr.Status.NotificationsController = status
+
+			// Update status of notifications controller workload for current instance and expose as metric to prometheus
+			if cr.Status.NotificationsController == "Pending" {
+				InstanceStatusTracker().SetNotificationsControllerStatus(cr.Namespace, WorkloadPendingState)
+			} else if cr.Status.NotificationsController == "Running" {
+				InstanceStatusTracker().SetNotificationsControllerStatus(cr.Namespace, WorkloadRunningState)
+			} else if cr.Status.NotificationsController == "Unknown" {
+				InstanceStatusTracker().SetNotificationsControllerStatus(cr.Namespace, WorkloadUnknownState)
+			}
 		}
 		return r.Client.Status().Update(context.TODO(), cr)
 	}
diff --git a/go.mod b/go.mod
index 67b6489c..f7461287 100644
--- a/go.mod
+++ b/go.mod
@@ -15,6 +15,7 @@ require (
 	github.com/openshift/client-go v0.0.0-20200325131901-f7baeb993edb
 	github.com/operator-framework/operator-sdk v0.18.2
 	github.com/pkg/errors v0.9.1
+	github.com/prometheus/client_golang v1.12.1
 	github.com/sethvargo/go-password v0.2.0
 	github.com/stretchr/testify v1.7.1
 	golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3
@@ -61,7 +62,6 @@ require (
 	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/nxadm/tail v1.4.8 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
-	github.com/prometheus/client_golang v1.12.1 // indirect
 	github.com/prometheus/client_model v0.2.0 // indirect
 	github.com/prometheus/common v0.32.1 // indirect
 	github.com/prometheus/procfs v0.7.3 // indirect
diff --git a/main.go b/main.go
index e3555702..7939425c 100644
--- a/main.go
+++ b/main.go
@@ -202,6 +202,14 @@ func main() {
 		os.Exit(1)
 	}
 
+	// start metrics server and log any errors in channel
+	go func() {
+		msErrCh := argocd.StartMetricsServer(common.ArgoCDDefaultInstanceStatusMetricsPort)
+		if err = <-msErrCh; err != nil {
+			setupLog.Error(err, "metrics server exited with error: %v", err)
+		}
+	}()
+
 	setupLog.Info("starting manager")
 	if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
 		setupLog.Error(err, "problem running manager")


Test patch:
diff --git a/controllers/argocd/metrics_test.go b/controllers/argocd/metrics_test.go
new file mode 100644
index 00000000..ea46ddb2
--- /dev/null
+++ b/controllers/argocd/metrics_test.go
@@ -0,0 +1,226 @@
+package argocd
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	argoprojv1alpha1 "github.com/argoproj-labs/argocd-operator/api/v1alpha1"
+	"github.com/prometheus/client_golang/prometheus/promhttp"
+	"github.com/stretchr/testify/assert"
+	appsv1 "k8s.io/api/apps/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+const (
+
+	// desired exposed metrics for argo-cd instance out of the box
+	desiredDefaultMetricsResponse string = `# HELP argocd_application_controller_status Describes the status of the application controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_application_controller_status gauge
+argocd_application_controller_status{namespace="argocd"} 3
+# HELP argocd_applicationset_controller_status Describes the status of the applicationSet controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_applicationset_controller_status gauge
+argocd_applicationset_controller_status{namespace="argocd"} 0
+# HELP argocd_dex_status Describes the status of the dex workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_dex_status gauge
+argocd_dex_status{namespace="argocd"} 0
+# HELP argocd_phase Describes the phase of argo-cd instance [2='Pending', 4='Available']
+# TYPE argocd_phase gauge
+argocd_phase{namespace="argocd"} 4
+# HELP argocd_redis_status Describes the status of the redis workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_redis_status gauge
+argocd_redis_status{namespace="argocd"} 3
+# HELP argocd_repo_server_status Describes the status of the repo server workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_repo_server_status gauge
+argocd_repo_server_status{namespace="argocd"} 3
+# HELP argocd_server_status Describes the status of the argo-cd server workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_server_status gauge
+argocd_server_status{namespace="argocd"} 3`
+
+	// desired exposed metrics when notifications-controller is enbaled and appset has invalid image
+	desiredMetricsResponse = `# HELP argocd_application_controller_status Describes the status of the application controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_application_controller_status gauge
+argocd_application_controller_status{namespace="argocd"} 3
+# HELP argocd_applicationset_controller_status Describes the status of the applicationSet controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_applicationset_controller_status gauge
+argocd_applicationset_controller_status{namespace="argocd"} 2
+# HELP argocd_dex_status Describes the status of the dex workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_dex_status gauge
+argocd_dex_status{namespace="argocd"} 0
+# HELP argocd_notifications_controller_status Describes the status of the notifications controller workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_notifications_controller_status gauge
+argocd_notifications_controller_status{namespace="argocd"} 3
+# HELP argocd_phase Describes the phase of argo-cd instance [2='Pending', 4='Available']
+# TYPE argocd_phase gauge
+argocd_phase{namespace="argocd"} 4
+# HELP argocd_redis_status Describes the status of the redis workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_redis_status gauge
+argocd_redis_status{namespace="argocd"} 3
+# HELP argocd_repo_server_status Describes the status of the repo server workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_repo_server_status gauge
+argocd_repo_server_status{namespace="argocd"} 3
+# HELP argocd_server_status Describes the status of the argo-cd server workload [0='Unknown', 1='Failed', 2='Pending', 3='Running']
+# TYPE argocd_server_status gauge
+argocd_server_status{namespace="argocd"} 3`
+)
+
+var (
+	desiredReplicas = int32(1)
+)
+
+type testDeploymentSetting struct {
+	name             string
+	deployedReplicas int32
+}
+
+func getTestDeployment(name string, deployedReplicas int32) *appsv1.Deployment {
+	deployment := &appsv1.Deployment{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      name,
+			Namespace: "argocd",
+		},
+		Spec: appsv1.DeploymentSpec{
+			Replicas: &desiredReplicas,
+		},
+		Status: appsv1.DeploymentStatus{
+			ReadyReplicas: deployedReplicas,
+		},
+	}
+	return deployment
+}
+
+func getTestStatefulSet(name string, deployedReplicas int32) *appsv1.StatefulSet {
+	statefulSet := &appsv1.StatefulSet{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      name,
+			Namespace: "argocd",
+		},
+		Spec: appsv1.StatefulSetSpec{
+			Replicas: &desiredReplicas,
+		},
+		Status: appsv1.StatefulSetStatus{
+			ReadyReplicas: deployedReplicas,
+		},
+	}
+	return statefulSet
+}
+
+func TestInstanceStatusMetrics(t *testing.T) {
+
+	tests := []struct {
+		argocd            *argoprojv1alpha1.ArgoCD
+		name              string
+		description       string
+		want              string
+		deploymentSetting []testDeploymentSetting
+	}{
+		{
+			name:        "default",
+			argocd:      makeTestArgoCD(),
+			description: "workload statuses for instance out of the box",
+			want:        desiredDefaultMetricsResponse,
+			deploymentSetting: []testDeploymentSetting{
+				{
+					name:             "argocd-application-controller",
+					deployedReplicas: int32(1),
+				},
+				{
+					name:             "argocd-redis",
+					deployedReplicas: int32(1),
+				},
+				{
+					name:             "argocd-repo-server",
+					deployedReplicas: int32(1),
+				},
+				{
+					name:             "argocd-server",
+					deployedReplicas: int32(1),
+				},
+			},
+		},
+		{
+			name: "modified",
+			argocd: makeTestArgoCD(func(ac *argoprojv1alpha1.ArgoCD) {
+				ac.Spec.Notifications.Enabled = true
+			}),
+			description: "workload statuses for modified instance: notifications-controller is enbaled and appset is pending",
+			want:        desiredMetricsResponse,
+			deploymentSetting: []testDeploymentSetting{
+				{
+					name:             "argocd-application-controller",
+					deployedReplicas: int32(1),
+				},
+				{
+					name:             "argocd-redis",
+					deployedReplicas: int32(1),
+				},
+				{
+					name:             "argocd-repo-server",
+					deployedReplicas: int32(1),
+				},
+				{
+					name:             "argocd-server",
+					deployedReplicas: int32(1),
+				},
+				{
+					name:             "argocd-notifications-controller",
+					deployedReplicas: int32(1),
+				},
+				{
+					name:             "argocd-applicationset-controller",
+					deployedReplicas: int32(0),
+				},
+			},
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			r := makeTestReconciler(t)
+
+			r.Client.Create(context.TODO(), test.argocd)
+			r.Client.Create(context.TODO(), getTestStatefulSet(test.deploymentSetting[0].name, test.deploymentSetting[0].deployedReplicas))
+
+			for _, setting := range test.deploymentSetting[1:] {
+				r.Client.Create(context.TODO(), getTestDeployment(setting.name, setting.deployedReplicas))
+			}
+
+			sm := http.NewServeMux()
+			sm.Handle(MetricsPath, promhttp.Handler())
+
+			err := r.reconcileStatusApplicationController(test.argocd)
+			assert.NoError(t, err)
+			err = r.reconcileStatusDex(test.argocd)
+			assert.NoError(t, err)
+			err = r.reconcileStatusRedis(test.argocd)
+			assert.NoError(t, err)
+			err = r.reconcileStatusRepo(test.argocd)
+			assert.NoError(t, err)
+			err = r.reconcileStatusServer(test.argocd)
+			assert.NoError(t, err)
+			err = r.reconcileStatusNotifications(test.argocd)
+			assert.NoError(t, err)
+			err = r.reconcileStatusApplicationSetController(test.argocd)
+			assert.NoError(t, err)
+
+			err = r.reconcileStatusPhase(test.argocd)
+			assert.NoError(t, err)
+
+			req, err := http.NewRequest("GET", "/metrics", nil)
+			assert.NoError(t, err)
+			rr := httptest.NewRecorder()
+
+			testMetricsServer := &http.Server{
+				Addr:    "localhost:8089",
+				Handler: sm,
+			}
+			testMetricsServer.Handler.ServeHTTP(rr, req)
+			assert.Equal(t, rr.Code, http.StatusOK)
+			body := rr.Body.String()
+
+			assert.True(t, strings.Contains(body, test.want))
+		})
+	}
+}