Chapter 3. Policy controller advanced configuration You can customize policy controller configurations on your managed clusters by using the ManagedClusterAddOn custom resources. The following ManagedClusterAddOns configure the policy framework, Kubernetes configuration policy controller, and the Certificate policy controller. Required access: Cluster administrator Configure the concurrency of the governance framework Configure the concurrency of the configuration policy controller Configure the rate of requests to the API server Configure debug log Governance metric Verify configuration changes 3.1. Configure the concurrency of the governance framework Configure the concurrency of the governance framework for each managed cluster. To change the default value of 2 , set the policy-evaluation-concurrency annotation with a nonzero integer within quotation marks. Then set the value on the ManagedClusterAddOn object name to governance-policy-framework in the managed cluster namespace of the hub cluster. See the following YAML example where the concurrency is set to 2 on the managed cluster named cluster1 : apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: governance-policy-framework namespace: cluster1 annotations: policy-evaluation-concurrency: "2" spec: installNamespace: open-cluster-management-agent-addon To set the client-qps and client-burst annotations, update the ManagedClusterAddOn resource and define the parameters. See the following YAML example where the queries for each second is set to 30 and the burst is set to 45 on the managed cluster called cluster1 : apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: governance-policy-framework namespace: cluster1 annotations: client-qps: "30" client-burst: "45" spec: installNamespace: open-cluster-management-agent-addon 3.2. Configure the concurrency of the configuration policy controller You can configure the concurrency of the configuration policy controller for each managed cluster to change how many configuration policies it can evaluate at the same time. To change the default value of 2 , set the policy-evaluation-concurrency annotation with a nonzero integer within quotation marks. Then set the value on the ManagedClusterAddOn object name to config-policy-controller in the managed cluster namespace of the hub cluster. Note: Increased concurrency values increase CPU and memory utilization on the config-policy-controller pod, Kubernetes API server, and OpenShift API server. See the following YAML example where the concurrency is set to 5 on the managed cluster named cluster1 : apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: config-policy-controller namespace: cluster1 annotations: policy-evaluation-concurrency: "5" spec: installNamespace: open-cluster-management-agent-addon 3.3. Configure the rate of requests to the API server Configure the rate of requests to the API server that the configuration policy controller makes on each managed cluster. An increased rate improves the responsiveness of the configuration policy controller, which also increases the CPU and memory utilization of the Kubernetes API server and OpenShift API server. By default, the rate of requests scales with the policy-evaluation-concurrency setting and is set to 30 queries for each second (QPS), with a 45 burst value, representing a higher number of requests over short periods of time. You can configure the rate and burst by setting the client-qps and client-burst annotations with nonzero integers within quotation marks. You can set the value on the ManagedClusterAddOn object name to config-policy-controller in the managed cluster namespace of the hub cluster. See the following YAML example where the queries for each second is set to 20 and the burst is set to 100 on the managed cluster called cluster1 : apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: config-policy-controller namespace: cluster1 annotations: client-qps: "20" client-burst: "100" spec: installNamespace: open-cluster-management-agent-addon 3.4. Configure debug log When you configure and collect debug logs for each policy controller, you can adjust the log level. Note: Reducing the volume of debug logs means there is less information displayed from the logs. You can reduce the debug logs emitted by the policy controllers to be display error-only bugs in the logs. To reduce the debug logs, set the debug log value to -1 in the annotation. See what each value represents: -1 : error logs only 0 : informative logs 1 : debug logs 2 : verbose debugging logs To receive the second level of debugging information for the Kubernetes configuration controller, add the log-level annotation with the value of 2 to the ManagedClusterAddOn custom resource. By default, the log-level is set to 0 , which means you receive informative messages. View the following example: apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: config-policy-controller namespace: cluster1 annotations: log-level: "2" spec: installNamespace: open-cluster-management-agent-addon Additionally, for each spec.object-template[] in a ConfigurationPolicy resource, you can set the parameter recordDiff to Log . The difference between the objectDefinition and the object on the managed cluster is logged in the config-policy-controller pod on the managed cluster. View the following example: This ConfigurationPolicy resource with recordDiff: Log : apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: my-config-policy spec: object-templates: - complianceType: musthave recordDiff: Log objectDefinition: apiVersion: v1 kind: ConfigMap metadata: name: my-configmap data: fieldToUpdate: "2" If the ConfigMap resource on the cluster lists fieldToUpdate: "1" , then the diff appears in the config-policy-controller pod logs with the following information: Important: Avoid logging the difference for a secure object. The difference is logged in plain text. 3.5. Governance metric The policy framework exposes metrics that show policy distribution and status. Use the policy_governance_info metric on the hub cluster to view trends and analyze any policy failures. See the following topics for an overview of metrics: 3.5.1. Metric: policy_governance_info The OpenShift Container Platform monitoring component collects the policy_governance_info metric. If you enable observability, the component collects some aggregate data. Note: If you enable observability, enter a query for the metric from the Grafana Explore page. When you create a policy, you are creating a root policy. The framework watches for root policies, Placement resources, and PlacementBindings resources to for information about where to create propagated policies, to distribute the policy to managed clusters. For both root and propagated policies, a metric of 0 is recorded if the policy is compliant, 1 if it is non-compliant, and -1 if it is in an unknown or pending state. The policy_governance_info metric uses the following labels: type : The label values are root or propagated . policy : The name of the associated root policy. policy_namespace : The namespace on the hub cluster where the root policy is defined. cluster_namespace : The namespace for the cluster where the policy is distributed. These labels and values enable queries that can show us many things happening in the cluster that might be difficult to track. Note: If you do not need the metrics, and you have any concerns about performance or security, you can disable the metric collection. Set the DISABLE_REPORT_METRICS environment variable to true in the propagator deployment. You can also add policy_governance_info metric to the observability allowlist as a custom metric. See Adding custom metrics for more details. 3.5.2. Metric: config_policies_evaluation_duration_seconds The config_policies_evaluation_duration_seconds histogram tracks the number of seconds it takes to process all configuration policies that are ready to be evaluated on the cluster. Use the following metrics to query the histogram: config_policies_evaluation_duration_seconds_bucket : The buckets are cumulative and represent seconds with the following possible entries: 1, 3, 9, 10.5, 15, 30, 60, 90, 120, 180, 300, 450, 600, and greater. config_policies_evaluation_duration_seconds_count : The count of all events. config_policies_evaluation_duration_seconds_sum : The sum of all values. Use the config_policies_evaluation_duration_seconds metric to determine if the ConfigurationPolicy evaluationInterval setting needs to be changed for resource intensive policies that do not need frequent evaluation. You can also increase the concurrency at the cost of higher resource utilization on the Kubernetes API server. See Configure the concurrency section for more details. To receive information about the time used to evaluate configuration policies, perform a Prometheus query that resembles the following expression: rate(config_policies_evaluation_duration_seconds_sum[10m])/rate (config_policies_evaluation_duration_seconds_count[10m] The config-policy-controller pod running on managed clusters in the open-cluster-management-agent-addon namespace calculates the metric. The config-policy-controller does not send the metric to observability by default. 3.6. Verify configuration changes When you apply the new configuration with the controller, the ManifestApplied parameter is updated in the ManagedClusterAddOn . That condition timestamp helps verify the configuration correctly. For example, this command can verify when the cert-policy-controller on the local-cluster was updated: You might receive the following output: 3.7. Additional resources See Kubernetes configuration policy controller Return to the Governance topic for more topics. Return to the beginning of this topic, Policy controller advanced configuration .

Commands:
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: governance-policy-framework namespace: cluster1 annotations: policy-evaluation-concurrency: "2" spec: installNamespace: open-cluster-management-agent-addon
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: governance-policy-framework namespace: cluster1 annotations: client-qps: "30" client-burst: "45" spec: installNamespace: open-cluster-management-agent-addon
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: config-policy-controller namespace: cluster1 annotations: policy-evaluation-concurrency: "5" spec: installNamespace: open-cluster-management-agent-addon
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: config-policy-controller namespace: cluster1 annotations: client-qps: "20" client-burst: "100" spec: installNamespace: open-cluster-management-agent-addon
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: config-policy-controller namespace: cluster1 annotations: log-level: "2" spec: installNamespace: open-cluster-management-agent-addon
apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: my-config-policy spec: object-templates: - complianceType: musthave recordDiff: Log objectDefinition: apiVersion: v1 kind: ConfigMap metadata: name: my-configmap data: fieldToUpdate: "2"
Logging the diff: --- default/my-configmap : existing +++ default/my-configmap : updated @@ -2,3 +2,3 @@ data: - fieldToUpdate: "1" + fieldToUpdate: "2" kind: ConfigMap
get -n local-cluster managedclusteraddon cert-policy-controller | grep -B4 'type: ManifestApplied'
- lastTransitionTime: "2023-01-26T15:42:22Z" message: manifests of addon are applied successfully reason: AddonManifestApplied status: "True" type: ManifestApplied