Chapter 2. Monitoring 2.1. Monitoring with GitOps dashboards You can access a graphical view of GitOps instances with Red Hat OpenShift GitOps monitoring dashboards to observe the behavior and usage of each instance across the cluster. There are three GitOps dashboards available: GitOps Overview : See an overview of all GitOps instances installed on the cluster, including the number of applications, health and sync status, application and sync activity. GitOps Components : View detailed information, such as CPU or memory, for application-controller, repo-server, server, and other GitOps components. GitOps gRPC Services : View metrics related to gRPC service activity between the various components in Red Hat OpenShift GitOps. 2.1.1. Accessing GitOps monitoring dashboards The monitoring dashboards are deployed automatically by the Operator. You can access GitOps monitoring dashboards from the Administrator perspective of the OpenShift Container Platform web console. Note Disabling or changing the content of the dashboards is not supported. Prerequisites You have access to the OpenShift Container Platform web console. The Red Hat OpenShift GitOps Operator is installed in the default namespace, openshift-gitops-operator . The cluster monitoring is enabled on the openshift-gitops-operator namespace. You have installed an Argo CD application in your defined namespace, for example, openshift-gitops . Procedure In the Administrator perspective of the web console, go to Observe Dashboards . From the Dashboard drop-down list, select the desired GitOps dashboard: GitOps (Overview) , GitOps / Components , or GitOps / gRPC Services . Optional: Choose a specific namespace, cluster, and interval from the Namespace , Cluster , and Interval drop-down lists. View the desired GitOps metrics in the GitOps dashboard. 2.2. Monitoring Argo CD instances By default, the Red Hat OpenShift GitOps Operator automatically detects an installed Argo CD instance in your defined namespace, for example, openshift-gitops , and connects it to the monitoring stack of the cluster to provide alerts for out-of-sync applications. 2.2.1. Prerequisites You have access to the cluster with cluster-admin privileges. You have access to the OpenShift Container Platform web console. You have installed the Red Hat OpenShift GitOps Operator in your cluster. You have installed an Argo CD application in your defined namespace, for example, openshift-gitops . 2.2.2. Monitoring Argo CD health using Prometheus metrics You can monitor the health status of an Argo CD application by running Prometheus metrics queries against it. Procedure In the Developer perspective of the web console, select the namespace where your Argo CD application is installed, and navigate to Observe Metrics . From the Select query drop-down list, select Custom query . To check the health status of your Argo CD application, enter the Prometheus Query Language (PromQL) query similar to the following example in the Expression field: Example sum(argocd_app_info{dest_namespace=~"<your_defined_namespace>",health_status!=""}) by (health_status) 1 1 Replace the <your_defined_namespace> variable with the actual name of your defined namespace, for example openshift-gitops . 2.2.3. Disabling automatic scraping of metrics for Argo CD instances By default, the Red Hat OpenShift GitOps Operator automatically scrapes metrics for all Argo CD instances to measure the performance. As a result, the Operator creates the following resources and label to the namespace where the Argo CD instance is installed: gitops-operator-argocd-alerts prometheus rule <argocd_namespace>-read role <argocd_name> , <argocd_name>-repo-server , and <argocd_name>-server service monitors <argocd_namespace>-prometheus-k8s-read-binding role binding openshift.io/cluster-monitoring=true label Scraping metrics for multiple Argo CD instances in a cluster might result in excessive storage usage. As a preventive measure, use the web console's YAML view and configure the ArgoCD custom resource (CR) to disable the automatic scraping of metrics for your Argo CD instance. As a cluster administrator, by disabling metric scraping for individual instances, you can give your users better control, flexibility, and stability to manage their defined namespaces. Procedure Log in to the OpenShift Container Platform web console. In the Administrator perspective of the web console, click Operators Installed Operators . From the Project list, select the project where the user-defined Argo CD instance is installed. Select Red Hat OpenShift GitOps from the installed Operators list and go to the Argo CD tab. Click your user-defined Argo CD instance. Configure the ArgoCD CR of your user-defined Argo CD instance to disable the automatic scraping of metrics: Click the YAML tab and edit the YAML file of the ArgoCD CR. In the ArgoCD CR, set the spec.monitoring.disableMetrics field value to true : Example ArgoCD CR apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: example 1 namespace: spring-petclinic 2 spec: monitoring: disableMetrics: true 1 The name of the user-defined Argo CD instance. 2 The namespace where you want to run the user-defined Argo CD instance. Tip Alternatively, use the following command to disable the automatic scraping of metrics in the Red Hat OpenShift GitOps argocd CLI: Example command USD oc patch argocd example -n spring-petclinic --type='json' -p='[{"op": "replace", "path": "/spec/monitoring/disableMetrics", "value": true}]' Example output argocd.argoproj.io/example patched Verify that the Operator adds the openshift.io/cluster-monitoring=false label to your defined namespace: Go to Administration Namespaces . The Namespaces page displays the created namespaces. Click your defined namespace, go to the YAML tab, and verify that under the metadata.labels section, the openshift.io/cluster-monitoring=false label is added by the Operator. Verify that the Operator deletes the following resources from your defined namespace: Go to Home Search . From the Resources list, select PrometheusRule , Role , RoleBinding , and ServiceMonitors . The Search page displays the selected resources. In the Search page, verify that under the PrometheusRule section, the gitops-operator-argocd-alerts prometheus rule is removed. Under the Roles section, from the Filter list, select Namespace Roles . Verify that the <argocd_namespace>-read role is removed. Under the RoleBindings section, from the Filter list, select Namespace RoleBindings . Verify that the <argocd_namespace>-prometheus-k8s-read-binding role binding is removed. Verify that under the ServiceMonitors section, the <argocd_name> , <argocd_name>-repo-server , and <argocd_name>-server service monitors are removed. Note You can enable the metrics for your instance by modifying the spec.monitoring.disableMetrics field value to false . The Operator then creates the required role, role bindings, and service monitors and adds the openshift.io/cluster-monitoring=true label to your defined namespace. 2.2.4. Additional resources Monitoring Argo CD custom resource workloads 2.3. Monitoring the GitOps Operator performance The Red Hat OpenShift GitOps Operator emits metrics about its performance. With the OpenShift monitoring stack that picks up these metrics, you can monitor and analyze the Operator's performance. The Operator exposes the following metrics, which you can view by using the OpenShift Container Platform web console: Table 2.1. GitOps Operator performance metrics Metric name Type Description active_argocd_instances_total Gauge The total number of active Argo CD instances currently managed by the Operator across the cluster at a given time. active_argocd_instances_by_phase Gauge The number of active Argo CD instances in a given phase, such as pending, or available. active_argocd_instance_reconciliation_count Counter The total number of reconciliations that have occurred for an instance in a given namespace at any given time. controller_runtime_reconcile_time_seconds_per_instance_bucket Counter The number of reconciliation cycles completed under given time durations for an instance. For example, controller_runtime_reconcile_time_seconds_per_instance_bucket{le="0.5"} shows the number of reconciliations that took under 0.5 seconds to complete for a given instance. controller_runtime_reconcile_time_seconds_per_instance_count Counter The total number of reconciliation cycles observed for a given instance. controller_runtime_reconcile_time_seconds_per_instance_sum Counter The total amount of time taken for the observed reconciliations for a given instance. Note Gauge is a value that can go up or down. Counter is a value that can only go up. 2.3.1. Accessing the GitOps Operator metrics You can access the Operator metrics from the Administrator perspective of the OpenShift Container Platform web console to track the performance of the Operator. Prerequisites You have access to the OpenShift Container Platform web console. The Red Hat OpenShift GitOps Operator is installed in the default openshift-gitops-operator namespace. The cluster monitoring is enabled on the openshift-gitops-operator namespace. Procedure In the Administrator perspective of the web console, go to Observe Metrics . Enter the metric in the Expression field. You can choose from the following metrics: active_argocd_instances_total active_argocd_instances_by_phase active_argocd_instance_reconciliation_count controller_runtime_reconcile_time_seconds_per_instance_bucket controller_runtime_reconcile_time_seconds_per_instance_count controller_runtime_reconcile_time_seconds_per_instance_sum (Optional): Filter the metric by its properties. For example, filter the active_argocd_instances_by_phase metric by the Available phase: Example active_argocd_instances_by_phase{phase="Available"} (Optional): Click Add query to enter multiple queries. Click Run queries to enable and observe the GitOps Operator metrics. 2.3.2. Additional resources Installing Red Hat OpenShift GitOps Operator in web console 2.4. Monitoring health information for application resources and deployments The Red Hat OpenShift GitOps Environments page in the Developer perspective of the OpenShift Container Platform web console shows a list of the successful deployments of the application environments, along with links to the revision for each deployment. The Application environments page in the Developer perspective of the OpenShift Container Platform web console displays the health status of the application resources, such as routes, synchronization status, deployment configuration, and deployment history. 2.4.1. Settings for environment labels and annotations This section provides reference settings for environment labels and annotations required to display an environment application in the Environments page, in the Developer perspective of the OpenShift Container Platform web console. Environment labels The environment application manifest must contain labels.openshift.gitops/environment and destination.namespace fields. You must set identical values for the <environment_name> variable and the name of the environment application manifest. Specification of the environment application manifest spec: labels: openshift.gitops/environment: <environment_name> destination: namespace: <environment_name> # ... Example of an environment application manifest apiVersion: argoproj.io/v1beta1 kind: Application metadata: name: dev-env 1 namespace: openshift-gitops spec: labels: openshift.gitops/environment: dev-env destination: namespace: dev-env # ... 1 The name of the environment application manifest. The value set is the same as the value of the <environment_name> variable. Environment annotations The environment namespace manifest must contain the annotations.app.openshift.io/vcs-uri and annotations.app.openshift.io/vcs-ref fields to specify the version controller code source of the application. You must set identical values for the <environment_name> variable and the name of the environment namespace manifest. Specification of the environment namespace manifest apiVersion: v1 kind: Namespace metadata: annotations: app.openshift.io/vcs-uri: <application_source_url> app.openshift.io/vcs-ref: <branch_reference> name: <environment_name> 1 # ... 1 The name of the environment namespace manifest. The value set is the same as the value of the <environment_name> variable. Example of an environment namespace manifest apiVersion: v1 kind: Namespace metadata: annotations: app.openshift.io/vcs-uri: https://example.com/<your_domain>/<your_gitops.git> app.openshift.io/vcs-ref: main labels: argocd.argoproj.io/managed-by: openshift-gitops name: dev-env # ... 2.4.2. Checking health information The Red Hat OpenShift GitOps Operator will install the GitOps backend service in the openshift-gitops namespace. Prerequisites The Red Hat OpenShift GitOps Operator is installed from OperatorHub . Ensure that your applications are synchronized by Argo CD. Procedure Click Environments under the Developer perspective. The Environments page shows the list of applications along with their Environment status . Hover over the icons under the Environment status column to see the synchronization status of all the environments. Click the application name from the list to view the details of a specific application. In the Application environments page, if the Resources section under the Overview tab displays icons, hover over the icons to get status details. A broken heart indicates that resource issues have degraded the application's performance. A yellow yield sign indicates that resource issues have delayed data about the application's health. To view the deployment history of an application, click the Deployment History tab. The page includes details such as the Last deployment , Description (commit message), Environment , Author , and Revision . 2.5. Monitoring Argo CD custom resource workloads With Red Hat OpenShift GitOps, you can monitor the availability of Argo CD custom resource workloads for specific Argo CD instances. By monitoring Argo CD custom resource workloads, you have the latest information about the state of your Argo CD instances by enabling alerts for them. When the component workload pods such as application-controller, repo-server, or server of the corresponding Argo CD instance are unable to come up for certain reasons and there is a drift between the number of ready replicas and the number of desired replicas for a certain period of time, the Operator then triggers the alerts. You can enable and disable the setting for monitoring Argo CD custom resource workloads. 2.5.1. Prerequisites You have access to the cluster as a user with the cluster-admin role. Red Hat OpenShift GitOps is installed in your cluster. The monitoring stack is configured in your cluster in the openshift-monitoring project. In addition, the Argo CD instance is in a namespace that you can monitor through Prometheus. The kube-state-metrics service is running on your cluster. Optional: If you are enabling monitoring for an Argo CD instance already present in a user-defined project, ensure that the monitoring is enabled for user-defined projects in your cluster. Note If you want to enable monitoring for an Argo CD instance in a namespace that is not watched by the default openshift-monitoring stack, for example, any namespace that does not start with openshift-* , then you must enable user workload monitoring in your cluster. This action enables the monitoring stack to pick up the created PrometheusRule. 2.5.2. Enabling Monitoring for Argo CD custom resource workloads By default, the monitoring configuration for Argo CD custom resource workloads is set to false . With Red Hat OpenShift GitOps, you can enable workload monitoring for specific Argo CD instances. As a result, the Operator creates a PrometheusRule object that contains alert rules for all the workloads managed by the specific Argo CD instances. These alert rules trigger the firing of an alert when the replica count of the corresponding component has drifted from the desired state for a certain amount of time. The Operator will not overwrite the changes made to the PrometheusRule object by the users. Procedure Set the .spec.monitoring.enabled field value to true on a given Argo CD instance: Example Argo CD custom resource apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: example-argocd labels: example: repo spec: # ... monitoring: enabled: true # ... Verify whether an alert rule is included in the PrometheusRule created by the Operator: Example alert rule apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: argocd-component-status-alert namespace: openshift-gitops spec: groups: - name: ArgoCDComponentStatus rules: # ... - alert: ApplicationSetControllerNotReady 1 annotations: message: >- applicationSet controller deployment for Argo CD instance in namespace "default" is not running expr: >- kube_statefulset_status_replicas{statefulset="openshift-gitops-application-controller statefulset", namespace="openshift-gitops"} != kube_statefulset_status_replicas_ready{statefulset="openshift-gitops-application-controller statefulset", namespace="openshift-gitops"} for: 1m labels: severity: critical 1 Alert rule in the PrometheusRule that checks whether the workloads created by the Argo CD instances are running as expected. 2.5.3. Disabling Monitoring for Argo CD custom resource workloads You can disable workload monitoring for specific Argo CD instances. Disabling workload monitoring deletes the created PrometheusRule. Procedure Set the .spec.monitoring.enabled field value to false on a given Argo CD instance: Example Argo CD custom resource apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: example-argocd labels: example: repo spec: # ... monitoring: enabled: false # ... 2.5.4. Additional resources Enabling monitoring for user-defined projects

Commands:
sum(argocd_app_info{dest_namespace=~"<your_defined_namespace>",health_status!=""}) by (health_status) 1
apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: example 1 namespace: spring-petclinic 2 spec: monitoring: disableMetrics: true
oc patch argocd example -n spring-petclinic --type='json' -p='[{"op": "replace", "path": "/spec/monitoring/disableMetrics", "value": true}]'
argocd.argoproj.io/example patched
active_argocd_instances_by_phase{phase="Available"}
spec: labels: openshift.gitops/environment: <environment_name> destination: namespace: <environment_name>
apiVersion: argoproj.io/v1beta1 kind: Application metadata: name: dev-env 1 namespace: openshift-gitops spec: labels: openshift.gitops/environment: dev-env destination: namespace: dev-env
apiVersion: v1 kind: Namespace metadata: annotations: app.openshift.io/vcs-uri: <application_source_url> app.openshift.io/vcs-ref: <branch_reference> name: <environment_name> 1
apiVersion: v1 kind: Namespace metadata: annotations: app.openshift.io/vcs-uri: https://example.com/<your_domain>/<your_gitops.git> app.openshift.io/vcs-ref: main labels: argocd.argoproj.io/managed-by: openshift-gitops name: dev-env
apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: example-argocd labels: example: repo spec: # monitoring: enabled: true #
apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: argocd-component-status-alert namespace: openshift-gitops spec: groups: - name: ArgoCDComponentStatus rules: # - alert: ApplicationSetControllerNotReady 1 annotations: message: >- applicationSet controller deployment for Argo CD instance in namespace "default" is not running expr: >- kube_statefulset_status_replicas{statefulset="openshift-gitops-application-controller statefulset", namespace="openshift-gitops"} != kube_statefulset_status_replicas_ready{statefulset="openshift-gitops-application-controller statefulset", namespace="openshift-gitops"} for: 1m labels: severity: critical
apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: example-argocd labels: example: repo spec: # monitoring: enabled: false #