Chapter 4. Sharding clusters across Argo CD Application Controller replicas You can shard clusters across multiple Argo CD Application Controller replicas if the controller is managing too many clusters and uses too much memory. 4.1. Enabling the round-robin sharding algorithm Important The round-robin sharding algorithm is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . By default, the Argo CD Application Controller uses the non-uniform legacy hash-based sharding algorithm to assign clusters to shards. This can result in uneven cluster distribution. You can enable the round-robin sharding algorithm to achieve more equal cluster distribution across all shards. Using the round-robin sharding algorithm in Red Hat OpenShift GitOps provides the following benefits: Ensure more balanced workload distribution Prevent shards from being overloaded or underutilized Optimize the efficiency of computing resources Reduce the risk of bottlenecks Improve overall performance and reliability of the Argo CD system The introduction of alternative sharding algorithms allows for further customization based on specific use cases. You can select the algorithm that best aligns with your deployment needs, which results in greater flexibility and adaptability in diverse operational scenarios. Tip To leverage the benefits of alternative sharding algorithms in GitOps, it is crucial to enable sharding during deployment. 4.1.1. Enabling the round-robin sharding algorithm in the web console You can enable the round-robin sharding algorithm by using the OpenShift Container Platform web console. Prerequisites You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster. You have access to the OpenShift Container Platform web console. You have access to the cluster with cluster-admin privileges. Procedure In the Administrator perspective of the web console, go to Operators Installed Operators . Click Red Hat OpenShift GitOps from the installed operators and go to the Argo CD tab. Click the Argo CD instance where you want to enable the round-robin sharding algorithm, for example, openshift-gitops . Click the YAML tab and edit the YAML file as shown in the following example: Example Argo CD instance with round-robin sharding algorithm enabled apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: openshift-gitops namespace: openshift-gitops spec: controller: sharding: enabled: true 1 replicas: 3 2 env: 3 - name: ARGOCD_CONTROLLER_SHARDING_ALGORITHM value: round-robin logLevel: debug 4 1 Set the sharding.enabled parameter to true to enable sharding. 2 Set the number of replicas to the wanted value, for example, 3 . 3 Set the sharding algorithm to round-robin . 4 Set the log level to debug so that you can verify to which shard each cluster is attached. Click Save . A success notification alert, openshift-gitops has been updated to version <version> , appears. Note If you edit the default openshift-gitops instance, the Managed resource dialog box is displayed. Click Save again to confirm the changes. Verify that the sharding is enabled with round-robin as the sharding algorithm by performing the following steps: Go to Workloads StatefulSets . Select the namespace where you installed the Argo CD instance from the Project drop-down list. Click <instance_name>-application-controller , for example, openshift-gitops-application-controller , and go to the Pods tab. Observe the number of created application controller pods. It should correspond with the number of set replicas. Click on the controller pod you want to examine and go to the Logs tab to view the pod logs. Example controller pod logs snippet time="2023-12-13T09:05:34Z" level=info msg="ArgoCD Application Controller is starting" built="2023-12-01T19:21:49Z" commit=a3vd5c3df52943a6fff6c0rg181fth3248976299 namespace=openshift-gitops version=v2.9.2+c5ea5c4 time="2023-12-13T09:05:34Z" level=info msg="Processing clusters from shard 1" time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" 1 time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" time="2023-12-13T09:05:34Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s" 1 Look for the "Using filter function: round-robin" message. In the log Search field, search for processed by shard to verify that the cluster distribution across shards is even, as shown in the following example. Important Ensure that you set the log level to debug to observe these logs. Example controller pod logs snippet time="2023-12-13T09:05:34Z" level=debug msg="ClustersList has 3 items" time="2023-12-13T09:05:34Z" level=debug msg="Adding cluster with id= and name=in-cluster to cluster's map" time="2023-12-13T09:05:34Z" level=debug msg="Adding cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 and name=in-cluster2 to cluster's map" time="2023-12-13T09:05:34Z" level=debug msg="Adding cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w and name=in-cluster3 to cluster's map" time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id= will be processed by shard 0" 1 time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 will be processed by shard 1" 2 time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w will be processed by shard 2" 3 1 2 3 In this example, 3 clusters are attached consecutively to shard 0, shard 1, and shard 2. Note If the number of clusters "C" is a multiple of the number of shard replicas "R", then each shard must have the same number of assigned clusters "N", which is equal to "C" divided by "R". The example shows 3 clusters and 3 replicas; therefore, each shard has 1 cluster assigned. 4.1.2. Enabling the round-robin sharding algorithm by using the CLI You can enable the round-robin sharding algorithm by using the command-line interface. Prerequisites You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster. You have access to the cluster with cluster-admin privileges. Procedure Enable sharding and set the number of replicas to the wanted value by running the following command: USD oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"sharding":{"enabled":true,"replicas":<value>}}}}' --type=merge Example output argocd.argoproj.io/<argocd_instance> patched Configure the sharding algorithm to round-robin by running the following command: USD oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"env":[{"name":"ARGOCD_CONTROLLER_SHARDING_ALGORITHM","value":"round-robin"}]}}}' --type=merge Example output argocd.argoproj.io/<argocd_instance> patched Verify that the number of Argo CD Application Controller pods corresponds with the number of set replicas by running the following command: USD oc get pods -l app.kubernetes.io/name=<argocd_instance>-application-controller -n <namespace> Example output NAME READY STATUS RESTARTS AGE <argocd_instance>-application-controller-0 1/1 Running 0 11s <argocd_instance>-application-controller-1 1/1 Running 0 32s <argocd_instance>-application-controller-2 1/1 Running 0 22s Verify that the sharding is enabled with round-robin as the sharding algorithm by running the following command: USD oc logs <argocd_application_controller_pod> -n <namespace> Example output snippet time="2023-12-13T09:05:34Z" level=info msg="ArgoCD Application Controller is starting" built="2023-12-01T19:21:49Z" commit=a3vd5c3df52943a6fff6c0rg181fth3248976299 namespace=<namespace> version=v2.9.2+c5ea5c4 time="2023-12-13T09:05:34Z" level=info msg="Processing clusters from shard 1" time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" 1 time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" time="2023-12-13T09:05:34Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s" 1 Look for the "Using filter function: round-robin" message. Verify that the cluster distribution across shards is even by performing the following steps: Set the log level to debug by running the following command: USD oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"logLevel":"debug"}}}' --type=merge Example output argocd.argoproj.io/<argocd_instance> patched View the logs and search for processed by shard to observe to which shard each cluster is attached by running the following command: USD oc logs <argocd_application_controller_pod> -n <namespace> | grep "processed by shard" Example output snippet time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id= will be processed by shard 0" 1 time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 will be processed by shard 1" 2 time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w will be processed by shard 2" 3 1 2 3 In this example, 3 clusters are attached consecutively to shard 0, shard 1, and shard 2. Note If the number of clusters "C" is a multiple of the number of shard replicas "R", then each shard must have the same number of assigned clusters "N", which is equal to "C" divided by "R". The example shows 3 clusters and 3 replicas; therefore, each shard has 1 cluster assigned. 4.2. Enabling dynamic scaling of shards of the Argo CD Application Controller Important Dynamic scaling of shards is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . By default, the Argo CD Application Controller assigns clusters to shards indefinitely. If you are using the round-robin sharding algorithm, this static assignment can result in uneven distribution of shards, particularly when replicas are added or removed. You can enable dynamic scaling of shards to automatically adjust the number of shards based on the number of clusters managed by the Argo CD Application Controller at a given time. This ensures that shards are well-balanced and optimizes the use of compute resources. Note After you enable dynamic scaling, you cannot manually modify the shard count. The system automatically adjusts the number of shards based on the number of clusters managed by the Argo CD Application Controller at a given time. 4.2.1. Enabling dynamic scaling of shards in the web console You can enable dynamic scaling of shards by using the OpenShift Container Platform web console. Prerequisites You have access to the cluster with cluster-admin privileges. You have access to the OpenShift Container Platform web console. You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster. Procedure In the Administator perspective of the OpenShift Container Platform web console, go to Operators Installed Operators . From the the list of Installed Operators , select the Red Hat OpenShift GitOps Operator, and then click the ArgoCD tab. Select the Argo CD instance name for which you want to enable dynamic scaling of shards, for example, openshift-gitops . Click the YAML tab, and then edit and configure the spec.controller.sharding properties as follows: Example Argo CD YAML file with dynamic scaling enabled apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: openshift-gitops namespace: openshift-gitops spec: controller: sharding: dynamicScalingEnabled: true 1 minShards: 1 2 maxShards: 3 3 clustersPerShard: 1 4 1 Set dynamicScalingEnabled to true to enable dynamic scaling. 2 Set minShards to the minimum number of shards that you want to have. The value must be set to 1 or greater. 3 Set maxShards to the maximum number of shards that you want to have. The value must be greater than the value of minShards . 4 Set clustersPerShard to the number of clusters that you want to have per shard. The value must be set to 1 or greater. Click Save . A success notification alert, openshift-gitops has been updated to version <version> , appears. Note If you edit the default openshift-gitops instance, the Managed resource dialog box is displayed. Click Save again to confirm the changes. Verification Verify that sharding is enabled by checking the number of pods in the namespace: Go to Workloads StatefulSets . Select the namespace where the Argo CD instance is deployed from the Project drop-down list, for example, openshift-gitops . Click the name of the StatefulSet object that has the name of the Argo CD instance, for example openshift-gitops-apllication-controller . Click the Pods tab, and then verify that the number of pods is equal to or greater than the value of minShards that you have set in the Argo CD YAML file. 4.2.2. Enabling dynamic scaling of shards by using the CLI You can enable dynamic scaling of shards by using the OpenShift CLI ( oc ). Prerequisites You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster. You have access to the cluster with cluster-admin privileges. Procedure Log in to the cluster by using the oc tool as a user with cluster-admin privileges. Enable dynamic scaling by running the following command: USD oc patch argocd <argocd_instance> -n <namespace> --type=merge --patch='{"spec":{"controller":{"sharding":{"dynamicScalingEnabled":true,"minShards":<value>,"maxShards":<value>,"clustersPerShard":<value>}}}}' Example command USD oc patch argocd openshift-gitops -n openshift-gitops --type=merge --patch='{"spec":{"controller":{"sharding":{"dynamicScalingEnabled":true,"minShards":1,"maxShards":3,"clustersPerShard":1}}}}' 1 1 The example command enables dynamic scaling for the openshift-gitops Argo CD instance in the openshift-gitops namespace, and sets the minimum number of shards to 1 , the maximum number of shards to 3 , and the number of clusters per shard to 1 . The values of minShard and clustersPerShard must be set to 1 or greater. The value of maxShard must be equal to or greater than the value of minShard . Example output argocd.argoproj.io/openshift-gitops patched Verification Check the spec.controller.sharding properties of the Argo CD instance: USD oc get argocd <argocd_instance> -n <namespace> -o jsonpath='{.spec.controller.sharding}' Example command USD oc get argocd openshift-gitops -n openshift-gitops -o jsonpath='{.spec.controller.sharding}' Example output when dynamic scaling of shards is enabled {"dynamicScalingEnabled":true,"minShards":1,"maxShards":3,"clustersPerShard":1} Optional: Verify that dynamic scaling is enabled by checking the configured spec.controller.sharding properties in the configuration YAML file of the Argo CD instance in the OpenShift Container Platform web console. Check the number of Argo CD Application Controller pods: USD oc get pods -n <namespace> -l app.kubernetes.io/name=<argocd_instance>-application-controller Example command USD oc get pods -n openshift-gitops -l app.kubernetes.io/name=openshift-gitops-application-controller Example output NAME READY STATUS RESTARTS AGE openshift-gitops-application-controller-0 1/1 Running 0 2m 1 1 The number of Argo CD Application Controller pods must be greater than or equal to the value of minShard . Additional resources Argo CD custom resource properties Automatically scaling pods with the horizontal pod autoscaler

Commands:
apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: openshift-gitops namespace: openshift-gitops spec: controller: sharding: enabled: true 1 replicas: 3 2 env: 3 - name: ARGOCD_CONTROLLER_SHARDING_ALGORITHM value: round-robin logLevel: debug 4
time="2023-12-13T09:05:34Z" level=info msg="ArgoCD Application Controller is starting" built="2023-12-01T19:21:49Z" commit=a3vd5c3df52943a6fff6c0rg181fth3248976299 namespace=openshift-gitops version=v2.9.2+c5ea5c4 time="2023-12-13T09:05:34Z" level=info msg="Processing clusters from shard 1" time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" 1 time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" time="2023-12-13T09:05:34Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"
time="2023-12-13T09:05:34Z" level=debug msg="ClustersList has 3 items" time="2023-12-13T09:05:34Z" level=debug msg="Adding cluster with id= and name=in-cluster to cluster's map" time="2023-12-13T09:05:34Z" level=debug msg="Adding cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 and name=in-cluster2 to cluster's map" time="2023-12-13T09:05:34Z" level=debug msg="Adding cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w and name=in-cluster3 to cluster's map" time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id= will be processed by shard 0" 1 time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 will be processed by shard 1" 2 time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w will be processed by shard 2" 3
oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"sharding":{"enabled":true,"replicas":<value>}}}}' --type=merge
argocd.argoproj.io/<argocd_instance> patched
oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"env":[{"name":"ARGOCD_CONTROLLER_SHARDING_ALGORITHM","value":"round-robin"}]}}}' --type=merge
argocd.argoproj.io/<argocd_instance> patched
oc get pods -l app.kubernetes.io/name=<argocd_instance>-application-controller -n <namespace>
NAME READY STATUS RESTARTS AGE <argocd_instance>-application-controller-0 1/1 Running 0 11s <argocd_instance>-application-controller-1 1/1 Running 0 32s <argocd_instance>-application-controller-2 1/1 Running 0 22s
oc logs <argocd_application_controller_pod> -n <namespace>
time="2023-12-13T09:05:34Z" level=info msg="ArgoCD Application Controller is starting" built="2023-12-01T19:21:49Z" commit=a3vd5c3df52943a6fff6c0rg181fth3248976299 namespace=<namespace> version=v2.9.2+c5ea5c4 time="2023-12-13T09:05:34Z" level=info msg="Processing clusters from shard 1" time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" 1 time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" time="2023-12-13T09:05:34Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"
oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"logLevel":"debug"}}}' --type=merge
argocd.argoproj.io/<argocd_instance> patched
oc logs <argocd_application_controller_pod> -n <namespace> | grep "processed by shard"
time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id= will be processed by shard 0" 1 time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 will be processed by shard 1" 2 time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w will be processed by shard 2" 3
apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: openshift-gitops namespace: openshift-gitops spec: controller: sharding: dynamicScalingEnabled: true 1 minShards: 1 2 maxShards: 3 3 clustersPerShard: 1 4
oc patch argocd <argocd_instance> -n <namespace> --type=merge --patch='{"spec":{"controller":{"sharding":{"dynamicScalingEnabled":true,"minShards":<value>,"maxShards":<value>,"clustersPerShard":<value>}}}}'
oc patch argocd openshift-gitops -n openshift-gitops --type=merge --patch='{"spec":{"controller":{"sharding":{"dynamicScalingEnabled":true,"minShards":1,"maxShards":3,"clustersPerShard":1}}}}' 1
argocd.argoproj.io/openshift-gitops patched
oc get argocd <argocd_instance> -n <namespace> -o jsonpath='{.spec.controller.sharding}'
oc get argocd openshift-gitops -n openshift-gitops -o jsonpath='{.spec.controller.sharding}'
dynamicScalingEnabled: yes
minShards: 1
maxShards: 3
clustersPerShard: 1
oc get pods -n <namespace> -l app.kubernetes.io/name=<argocd_instance>-application-controller
oc get pods -n openshift-gitops -l app.kubernetes.io/name=openshift-gitops-application-controller
NAME READY STATUS RESTARTS AGE openshift-gitops-application-controller-0 1/1 Running 0 2m 1