Chapter 1. multicluster engine operator with Red Hat Advanced Cluster Management integration If you are using multicluster engine operator and then you install Red Hat Advanced Cluster Management, you can access more multicluster management features, such as Observability and Policy . For integrated capability, see the following requirements: You need to install Red Hat Advanced Cluster Management. See the Red Hat Advanced Cluster Management Installing and upgrading documentation. See MultiClusterHub advanced configuration for details about Red Hat Advanced Cluster Management after you install. See the following procedures for multicluster engine operator and Red Hat Advanced Cluster Management multicluster management: Discovering multicluster engine operator hosted clusters in Red Hat Advanced Cluster Management Automating import for discovered hosted clusters Automating import for discovered OpenShift Service on AWS clusters Observability integration SiteConfig operator 1.1. Discovering multicluster engine operator hosted clusters in Red Hat Advanced Cluster Management If you have multicluster engine operator clusters that are hosting multiple hosted clusters , you can bring those hosted clusters to a Red Hat Advanced Cluster Management hub cluster to manage with Red Hat Advanced Cluster Management management components, such as Application lifecycle and Governance . Those hosted clusters can be automatically discovered and imported as managed clusters. Note: Since the hosted control planes run on the managed multicluster engine operator cluster nodes, the number of hosted control planes that the cluster can host is determined by the resource availability of managed multicluster engine operator cluster nodes, as well as the number of managed multicluster engine operator clusters. You can add more nodes or managed clusters to host more hosted control planes. Required access: Cluster administrator Prerequisites Configuring Red Hat Advanced Cluster Management to import multicluster engine operator clusters Importing multicluster engine operator manually Discovering hosted clusters from multicluster engine operator 1.1.1. Prerequisites You need one or more multicluster engine operator clusters. You need a Red Hat Advanced Cluster Management cluster that is set as your hub cluster. Install the clusteradm CLI by running the following command: curl -L https://raw.githubusercontent.com/open-cluster-management-io/clusteradm/main/install.sh | bash 1.1.2. Configuring Red Hat Advanced Cluster Management to import multicluster engine operator clusters multicluster engine operator has a local-cluster , which is a hub cluster that is managed. The following default addons are enabled for this local-cluster in the open-cluster-management-agent-addon namespace: cluster-proxy managed-serviceaccount work-manager 1.1.2.1. Configuring add-ons When your multicluster engine operator is imported into Red Hat Advanced Cluster Management, Red Hat Advanced Cluster Management enables the same set of add-ons to manage the multicluster engine operator. Install those add-ons in a different multicluster engine operator namespace so that the multicluster engine operator can self-manage with the local-cluster add-ons while Red Hat Advanced Cluster Management manages multicluster engine operator at the same time. Complete the following procedure: Log in to your Red Hat Advanced Cluster Management with the CLI. Create the addonDeploymentConfig resource to specify a different add-on installation namespace. See the following example where agentInstallNamespace references open-cluster-management-agent-addon-discovery : apiVersion: addon.open-cluster-management.io/v1alpha1 kind: addonDeploymentConfig metadata: name: addon-ns-config namespace: multicluster-engine spec: agentInstallNamespace: open-cluster-management-agent-addon-discovery Run oc apply -f <filename>.yaml to apply the file. Update the existing ClusterManagementAddOn resources for the add-ons so that the add-ons are installed in the open-cluster-management-agent-addon-discovery namespace that is specified in the addonDeploymentConfig resource that you created. See the following example with open-cluster-management-global-set as the namespace: apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ClusterManagementAddOn metadata: name: work-manager spec: addonMeta: displayName: work-manager installStrategy: placements: - name: global namespace: open-cluster-management-global-set rolloutStrategy: type: All type: Placements Add the addonDeploymentConfigs to the ClusterManagementAddOn . See the following example: apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ClusterManagementAddOn metadata: name: work-manager spec: addonMeta: displayName: work-manager installStrategy: placements: - name: global namespace: open-cluster-management-global-set rolloutStrategy: type: All configs: - group: addon.open-cluster-management.io name: addon-ns-config namespace: multicluster-engine resource: addondeploymentconfigs type: Placements Add the addonDeploymentConfig to the managed-serviceaccount . See the following example: apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ClusterManagementAddOn metadata: name: managed-serviceaccount spec: addonMeta: displayName: managed-serviceaccount installStrategy: placements: - name: global namespace: open-cluster-management-global-set rolloutStrategy: type: All configs: - group: addon.open-cluster-management.io name: addon-ns-config namespace: multicluster-engine resource: addondeploymentconfigs type: Placements Add the addondeploymentconfigs value to the ClusterManagementAddOn resource named, cluster-proxy . See the following example: apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ClusterManagementAddOn metadata: name: cluster-proxy spec: addonMeta: displayName: cluster-proxy installStrategy: placements: - name: global namespace: open-cluster-management-global-set rolloutStrategy: type: All configs: - group: addon.open-cluster-management.io name: addon-ns-config namespace: multicluster-engine resource: addondeploymentconfigs type: Placements Run the following command to verify that the add-ons for the Red Hat Advanced Cluster Management local-cluster are re-installed into the namespace that you specified: oc get deployment -n open-cluster-management-agent-addon-discovery See the following output example: NAME READY UP-TO-DATE AVAILABLE AGE cluster-proxy-proxy-agent 1/1 1 1 24h klusterlet-addon-workmgr 1/1 1 1 24h managed-serviceaccount-addon-agent 1/1 1 1 24h 1.1.2.2. Creating a KlusterletConfig resource multicluster engine operator has a local-cluster, which is a hub cluster that is managed. A resource named klusterlet is created for this local-cluster. When your multicluster engine operator is imported into Red Hat Advanced Cluster Management, Red Hat Advanced Cluster Management installs the klusterlet with the same name, klusterlet , to manage the multicluster engine operator. This conflicts with the multicluster engine operator local-cluster klusterlet. You need to create a KlusterletConfig resource that is used by ManagedCluster resources to import multicluster engine operator clusters so that the klusterlet is installed with a different name to avoid the conflict. Complete the following procedure: Create a KlusterletConfig resource using the following example. When this KlusterletConfig resource is referenced in a managed cluster, the value in the spec.installMode.noOperator.postfix field is used as a suffix to the klusterlet name, such as klusterlet-mce-import : kind: KlusterletConfig apiVersion: config.open-cluster-management.io/v1alpha1 metadata: name: mce-import-klusterlet-config spec: installMode: type: noOperator noOperator: postfix: mce-import Run oc apply -f <filename>.yaml to apply the file. 1.1.2.3. Configure for backup and restore Since you installed Red Hat Advanced Cluster Management, you can also use the Backup and restore feature. If the hub cluster is restored in a disaster recovery scenario, the imported multicluster engine operator clusters and hosted clusters are imported to the newer Red Hat Advanced Cluster Management hub cluster. In this scenario, you need to restore the configurations as part of Red Hat Advanced Cluster Management hub cluster restore. Add the backup=true label to enable backup. See the following steps for each add-on: For your addon-ns-config , run the following command: oc label addondeploymentconfig addon-ns-config -n multicluster-engine cluster.open-cluster-management.io/backup=true For your hypershift-addon-deploy-config , run the following command: oc label addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine cluster.open-cluster-management.io/backup=true For your work-manager , run the following command: oc label clustermanagementaddon work-manager cluster.open-cluster-management.io/backup=true For your `cluster-proxy `, run the following command: oc label clustermanagementaddon cluster-proxy cluster.open-cluster-management.io/backup=true For your managed-serviceaccount , run the following command: oc label clustermanagementaddon managed-serviceaccount cluster.open-cluster-management.io/backup=true For your mce-import-klusterlet-config , run the following command: oc label KlusterletConfig mce-import-klusterlet-config cluster.open-cluster-management.io/backup=true 1.1.3. Importing multicluster engine operator manually To manually import an multicluster engine operator cluster from your Red Hat Advanced Cluster Management cluster, complete the following procedure: From your Red Hat Advanced Cluster Management cluster, create a ManagedCluster resource manually to import an multicluster engine operator cluster. See the following file example: apiVersion: cluster.open-cluster-management.io/v1 kind: ManagedCluster metadata: annotations: agent.open-cluster-management.io/klusterlet-config: mce-import-klusterlet-config 1 labels: cloud: auto-detect vendor: auto-detect name: mce-a 2 spec: hubAcceptsClient: true leaseDurationSeconds: 60 1 The mce-import-klusterlet-config annotation references the KlusterletConfig resource that you created in the step to install the Red Hat Advanced Cluster Management klusterlet with a different name in multicluster engine operator. 2 The example imports an multicluster engine operator managed cluster named mce-a . Run oc apply -f <filename>.yaml to apply the file. Create the auto-import-secret secret that references the kubeconfig of the multicluster engine operator cluster. Go to Importing a cluster by using the auto import secret in Importing a managed cluster by using the CLI to add the auto import secret to complete the multicluster engine operator auto-import process. After you create the auto import secret in the multicluster engine operator managed cluster namespace in the Red Hat Advanced Cluster Management cluster, the managed cluster is registered. Run the following command to get the status: oc get managedcluster See following example output with the status and example URLs of managed clusters: NAME HUB ACCEPTED MANAGED CLUSTER URLS JOINED AVAILABLE AGE local-cluster true https://<api.acm-hub.com:port> True True 44h mce-a true https://<api.mce-a.com:port> True True 27s Important: Do not enable any other Red Hat Advanced Cluster Management add-ons for the imported multicluster engine operator. 1.1.4. Discovering hosted clusters After all your multicluster engine operator clusters are imported into Red Hat Advanced Cluster Management, you need to enable the hypershift-addon for those managed multicluster engine operator clusters to discover the hosted clusters. Default add-ons are installed into a different namespace in the procedures. Similarly, you install the hypershift-addon into a different namespace in multicluster engine operator so that the add-ons agent for multicluster engine operator local-cluster and the agent for Red Hat Advanced Cluster Management can work in multicluster engine operator. Important: For all the following commands, replace <managed-cluster-names> with comma-separated managed cluster names for multicluster engine operator. Run the following command to set the agentInstallNamespace namespace of the add-on to open-cluster-management-agent-addon-discovery : oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"agentInstallNamespace":"open-cluster-management-agent-addon-discovery"}}' Run the following command to disable metrics and to disable the HyperShift operator management: oc patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value": "true"},{"name":"disableHOManagement","value": "true"}]}}' Run the following command to enable the hypershift-addon for multicluster engine operator: clusteradm addon enable --names hypershift-addon --clusters <managed-cluster-names> You can get the multicluster engine operator managed cluster names by running the following command in Red Hat Advanced Cluster Management. oc get managedcluster Log into multicluster engine operator clusters and verify that the hypershift-addon is installed in the namespace that you specified. Run the following command: oc get deployment -n open-cluster-management-agent-addon-discovery See the following example output that lists the add-ons: NAME READY UP-TO-DATE AVAILABLE AGE cluster-proxy-proxy-agent 1/1 1 1 24h klusterlet-addon-workmgr 1/1 1 1 24h hypershift-addon-agent 1/1 1 1 24h managed-serviceaccount-addon-agent 1/1 1 1 24h Red Hat Advanced Cluster Management deploys the hypershift-addon , which is the discovery agent that discovers hosted clusters from multicluster engine operator. The agent creates the corresponding DiscoveredCluster custom resource in the multicluster engine operator managed cluster namespace in the Red Hat Advanced Cluster Management hub cluster when the hosted cluster kube-apiserver becomes available. You can view your discovered clusters in the console. Log into hub cluster console and navigate to All Clusters > Infrastructure > Clusters . Find the Discovered clusters tab to view all discovered hosted clusters from multicluster engine operator with type MultiClusterEngineHCP . , visit Automating import for discovered hosted clusters to learn how to automatically import clusters. 1.2. Automating import for discovered hosted clusters Automate the import of hosted clusters by using the DiscoveredCluster resource for faster cluster management, without manually importing individual clusters. When you automatically import a discovered hosted cluster into Red Hat Advanced Cluster Management, all Red Hat Advanced Cluster Management add-ons are enabled so that you can start managing the hosted clusters with the available management tools. The hosted cluster is also auto-imported into multicluster engine operator. Through the multicluster engine operator console, you can manage the hosted cluster lifecycle. However, you cannot manage the hosted cluster lifecycle from the Red Hat Advanced Cluster Management console. Required access: Cluster administrator Prerequisites Configure settings for automatic import Creating the placement definition Binding the import policy to a placement definition 1.2.1. Prerequisites You need Red Hat Advanced Cluster Management installed. See the Red Hat Advanced Cluster Management Installing and upgrading documentation. You need to learn about Policies . See the introduction to Governance in the Red Hat Advanced Cluster Management documentation. 1.2.2. Configuring settings for automatic import Discovered hosted clusters from managed multicluster engine operator clusters are represented in DiscoveredCluster custom resources, which are located in the managed multicluster engine operator cluster namespace in Red Hat Advanced Cluster Management. See the following DiscoveredCluster resource and namespace example: apiVersion: discovery.open-cluster-management.io/v1 kind: DiscoveredCluster metadata: creationTimestamp: "2024-05-30T23:05:39Z" generation: 1 labels: hypershift.open-cluster-management.io/hc-name: hosted-cluster-1 hypershift.open-cluster-management.io/hc-namespace: clusters name: hosted-cluster-1 namespace: mce-1 resourceVersion: "1740725" uid: b4c36dca-a0c4-49f9-9673-f561e601d837 spec: apiUrl: https://a43e6fe6dcef244f8b72c30426fb6ae3-ea3fec7b113c88da.elb.us-west-1.amazonaws.com:6443 cloudProvider: aws creationTimestamp: "2024-05-30T23:02:45Z" credential: {} displayName: mce-1-hosted-cluster-1 importAsManagedCluster: false isManagedCluster: false name: hosted-cluster-1 openshiftVersion: 0.0.0 status: Active type: MultiClusterEngineHCP These discovered hosted clusters are not automatically imported into Red Hat Advanced Cluster Management until the spec.importAsManagedCluster field is set to true . Learn how to use a Red Hat Advanced Cluster Management policy to automatically set this field to true for all type.MultiClusterEngineHCP within DiscoveredCluster resources so that discovered hosted clusters are immediately automatically imported into Red Hat Advanced Cluster Management. Configure your Policy to import all your discovered hosted clusters automatically. Log in to your hub cluster from the CLI to complete the following procedure: Create a YAML file for your DiscoveredCluster custom resource and edit the configuration that is referenced in the following example: apiVersion: policy.open-cluster-management.io/v1 kind: Policy metadata: name: policy-mce-hcp-autoimport namespace: open-cluster-management-global-set annotations: policy.open-cluster-management.io/standards: NIST SP 800-53 policy.open-cluster-management.io/categories: CM Configuration Management policy.open-cluster-management.io/controls: CM-2 Baseline Configuration policy.open-cluster-management.io/description: Discovered clusters that are of type MultiClusterEngineHCP can be automatically imported into ACM as managed clusters. This policy configure those discovered clusters so they are automatically imported. Fine tuning MultiClusterEngineHCP clusters to be automatically imported can be done by configure filters at the configMap or add annotation to the discoverd cluster. spec: disabled: false policy-templates: - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: mce-hcp-autoimport-config spec: object-templates: - complianceType: musthave objectDefinition: apiVersion: v1 kind: ConfigMap metadata: name: discovery-config namespace: open-cluster-management-global-set data: rosa-filter: "" remediationAction: enforce 1 severity: low - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: policy-mce-hcp-autoimport spec: remediationAction: enforce severity: low object-templates-raw: | {{- /* find the MultiClusterEngineHCP DiscoveredClusters */ -}} {{- range USDdc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }} {{- /* Check for the flag that indicates the import should be skipped */ -}} {{- USDskip := "false" -}} {{- range USDkey, USDvalue := USDdc.metadata.annotations }} {{- if and (eq USDkey "discovery.open-cluster-management.io/previously-auto-imported") (eq USDvalue "true") }} {{- USDskip = "true" }} {{- end }} {{- end }} {{- /* if the type is MultiClusterEngineHCP and the status is Active */ -}} {{- if and (eq USDdc.spec.status "Active") (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "mce-hcp-filter") USDdc.spec.displayName) (eq USDdc.spec.type "MultiClusterEngineHCP") (eq USDskip "false") }} - complianceType: musthave objectDefinition: apiVersion: discovery.open-cluster-management.io/v1 kind: DiscoveredCluster metadata: name: {{ USDdc.metadata.name }} namespace: {{ USDdc.metadata.namespace }} spec: importAsManagedCluster: true 2 {{- end }} {{- end }} 1 To enable automatic import, change the spec.remediationAction to enforce . 2 To enable automatic import, change spec.importAsManagedCluster to true . Run oc apply -f <filename>.yaml -n <namespace> to apply the file. 1.2.3. Creating the placement definition You need to create a placement definition that specifies the managed cluster for the policy deployment. Complete the following procedure: Create the Placement definition that selects only the local-cluster , which is a hub cluster that is managed. Use the following YAML sample: apiVersion: cluster.open-cluster-management.io/v1beta1 kind: Placement metadata: name: policy-mce-hcp-autoimport-placement namespace: open-cluster-management-global-set spec: tolerations: - key: cluster.open-cluster-management.io/unreachable operator: Exists - key: cluster.open-cluster-management.io/unavailable operator: Exists clusterSets: - global predicates: - requiredClusterSelector: labelSelector: matchExpressions: - key: local-cluster operator: In values: - "true" Run oc apply -f placement.yaml -n <namespace> , where namespace matches the namespace that you used for the policy that you previously created. 1.2.4. Binding the import policy to a placement definition After you create the policy and the placement, you need to connect the two resources. Complete the following steps: Connect the resources by using a PlacementBinding resource. See the following example where placementRef references the Placement that you created, and subjects references the Policy that you created: apiVersion: policy.open-cluster-management.io/v1 kind: PlacementBinding metadata: name: policy-mce-hcp-autoimport-placement-binding namespace: open-cluster-management-global-set placementRef: name: policy-mce-hcp-autoimport-placement apiGroup: cluster.open-cluster-management.io kind: Placement subjects: - name: policy-mce-hcp-autoimport apiGroup: policy.open-cluster-management.io kind: Policy To verify, run the following command: oc get policies.policy.open-cluster-management.io policy-mce-hcp-autoimport -n <namespace> Important: You can detach a hosted cluster from Red Hat Advanced Cluster Management by using the Detach option in the Red Hat Advanced Cluster Management console, or by removing the corresponding ManagedCluster custom resource from the command line. For best results, detach the managed hosted cluster before destroying the hosted cluster. When a discovered cluster is detached, the following annotation is added to the DiscoveredCluster resource to prevent the policy to import the discovered cluster again. annotations: discovery.open-cluster-management.io/previously-auto-imported: "true" If you want the detached discovered cluster to be reimported, remove this annotation. 1.3. Automating import for discovered OpenShift Service on AWS clusters Automate the import of OpenShift Service on AWS clusters by using Red Hat Advanced Cluster Management policy enforcement for faster cluster management, without manually importing individual clusters. Required access: Cluster administrator Prerequisites Creating the automatic import policy Creating the placement definition Binding the import policy to a placement definition 1.3.1. Prerequisites You need Red Hat Advanced Cluster Management installed. See the Red Hat Advanced Cluster Management Installing and upgrading documentation. You need to learn about Policies . See the introduction to Governance in the Red Hat Advanced Cluster Management documentation. 1.3.2. Creating the automatic import policy The following policy and procedure is an example of how to import all your discovered OpenShift Service on AWS clusters automatically. Log in to your hub cluster from the CLI to complete the following procedure: Create a YAML file with the following example and apply the changes that are referenced: apiVersion: policy.open-cluster-management.io/v1 kind: Policy metadata: name: policy-rosa-autoimport annotations: policy.open-cluster-management.io/standards: NIST SP 800-53 policy.open-cluster-management.io/categories: CM Configuration Management policy.open-cluster-management.io/controls: CM-2 Baseline Configuration policy.open-cluster-management.io/description: OpenShift Service on AWS discovered clusters can be automatically imported into Red Hat Advanced Cluster Management as managed clusters with this policy. You can select and configure those managed clusters so you can import. Configure filters or add an annotation if you do not want all of your OpenShift Service on AWS clusters to be automatically imported. spec: remediationAction: inform 1 disabled: false policy-templates: - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: rosa-autoimport-config spec: object-templates: - complianceType: musthave objectDefinition: apiVersion: v1 kind: ConfigMap metadata: name: discovery-config namespace: open-cluster-management-global-set data: rosa-filter: "" 2 remediationAction: enforce severity: low - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: policy-rosa-autoimport spec: remediationAction: enforce severity: low object-templates-raw: | {{- /* find the ROSA DiscoveredClusters */ -}} {{- range USDdc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }} {{- /* Check for the flag that indicates the import should be skipped */ -}} {{- USDskip := "false" -}} {{- range USDkey, USDvalue := USDdc.metadata.annotations }} {{- if and (eq USDkey "discovery.open-cluster-management.io/previously-auto-imported") (eq USDvalue "true") }} {{- USDskip = "true" }} {{- end }} {{- end }} {{- /* if the type is ROSA and the status is Active */ -}} {{- if and (eq USDdc.spec.status "Active") (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "rosa-filter") USDdc.spec.displayName) (eq USDdc.spec.type "ROSA") (eq USDskip "false") }} - complianceType: musthave objectDefinition: apiVersion: discovery.open-cluster-management.io/v1 kind: DiscoveredCluster metadata: name: {{ USDdc.metadata.name }} namespace: {{ USDdc.metadata.namespace }} spec: importAsManagedCluster: true {{- end }} {{- end }} - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: policy-rosa-managedcluster-status spec: remediationAction: enforce severity: low object-templates-raw: | {{- /* Use the same DiscoveredCluster list to check ManagedCluster status */ -}} {{- range USDdc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }} {{- /* Check for the flag that indicates the import should be skipped */ -}} {{- USDskip := "false" -}} {{- range USDkey, USDvalue := USDdc.metadata.annotations }} {{- if and (eq USDkey "discovery.open-cluster-management.io/previously-auto-imported") (eq USDvalue "true") }} {{- USDskip = "true" }} {{- end }} {{- end }} {{- /* if the type is ROSA and the status is Active */ -}} {{- if and (eq USDdc.spec.status "Active") (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "rosa-filter") USDdc.spec.displayName) (eq USDdc.spec.type "ROSA") (eq USDskip "false") }} - complianceType: musthave objectDefinition: apiVersion: cluster.open-cluster-management.io/v1 kind: ManagedCluster metadata: name: {{ USDdc.spec.displayName }} namespace: {{ USDdc.spec.displayName }} status: conditions: - type: ManagedClusterConditionAvailable status: "True" {{- end }} {{- end }} 1 To enable automatic import, change the spec.remediationAction to enforce . 2 Optional: Specify a value here to select a subset of the matching OpenShift Service on AWS clusters, which are based on discovered cluster names. The rosa-filter has no value by default, so the filter does not restrict cluster names without a subset value. Run oc apply -f <filename>.yaml -n <namespace> to apply the file. 1.3.3. Creating the placement definition You need to create a placement definition that specifies the managed cluster for the policy deployment. Create the placement definition that selects only the local-cluster , which is a hub cluster that is managed. Use the following YAML sample: apiVersion: cluster.open-cluster-management.io/v1beta1 kind: Placement metadata: name: placement-openshift-plus-hub spec: predicates: - requiredClusterSelector: labelSelector: matchExpressions: - key: name operator: In values: - local-cluster Run oc apply -f placement.yaml -n <namespace> , where namespace matches the namespace that you used for the policy that you previously created. 1.3.4. Binding the import policy to a placement definition After you create the policy and the placement, you need to connect the two resources. Connect the resources by using a PlacementBinding . See the following example where placementRef references the Placement that you created, and subjects references the Policy that you created: apiVersion: policy.open-cluster-management.io/v1 kind: PlacementBinding metadata: name: binding-policy-rosa-autoimport placementRef: apiGroup: cluster.open-cluster-management.io kind: Placement name: placement-policy-rosa-autoimport subjects: - apiGroup: policy.open-cluster-management.io kind: Policy name: policy-rosa-autoimport To verify, run the following command: 1.4. Observability integration With the Red Hat Advanced Cluster Management Observability feature, you can view health and utilization of clusters across your fleet. You can install Red Hat Advanced Cluster Management and enable Observability. 1.4.1. Observing hosted control planes After you enable the multicluster-observability pod, you can use Red Hat Advanced Cluster Management Observability Grafana dashboards to view the following information about your hosted control planes: ACM > Hosted Control Planes Overview dashboard to see cluster capacity estimates for hosting hosted control planes, the related cluster resources, and the list and status of existing hosted control planes. For more information, see: Introduction to hosted control planes . ACM > Resources > Hosted Control Plane dashboard that you can access from the Overview page to see the resource utilization of the selected hosted control plane. For more information, see Installing the hosted control planes command-line interface . To enable, see Observability service .

Commands:
curl -L https://raw.githubusercontent.com/open-cluster-management-io/clusteradm/main/install.sh | bash
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: addonDeploymentConfig metadata: name: addon-ns-config namespace: multicluster-engine spec: agentInstallNamespace: open-cluster-management-agent-addon-discovery
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ClusterManagementAddOn metadata: name: work-manager spec: addonMeta: displayName: work-manager installStrategy: placements: - name: global namespace: open-cluster-management-global-set rolloutStrategy: type: All type: Placements
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ClusterManagementAddOn metadata: name: work-manager spec: addonMeta: displayName: work-manager installStrategy: placements: - name: global namespace: open-cluster-management-global-set rolloutStrategy: type: All configs: - group: addon.open-cluster-management.io name: addon-ns-config namespace: multicluster-engine resource: addondeploymentconfigs type: Placements
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ClusterManagementAddOn metadata: name: managed-serviceaccount spec: addonMeta: displayName: managed-serviceaccount installStrategy: placements: - name: global namespace: open-cluster-management-global-set rolloutStrategy: type: All configs: - group: addon.open-cluster-management.io name: addon-ns-config namespace: multicluster-engine resource: addondeploymentconfigs type: Placements
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ClusterManagementAddOn metadata: name: cluster-proxy spec: addonMeta: displayName: cluster-proxy installStrategy: placements: - name: global namespace: open-cluster-management-global-set rolloutStrategy: type: All configs: - group: addon.open-cluster-management.io name: addon-ns-config namespace: multicluster-engine resource: addondeploymentconfigs type: Placements
get deployment -n open-cluster-management-agent-addon-discovery
NAME READY UP-TO-DATE AVAILABLE AGE cluster-proxy-proxy-agent 1/1 1 1 24h klusterlet-addon-workmgr 1/1 1 1 24h managed-serviceaccount-addon-agent 1/1 1 1 24h
kind: KlusterletConfig apiVersion: config.open-cluster-management.io/v1alpha1 metadata: name: mce-import-klusterlet-config spec: installMode: type: noOperator noOperator: postfix: mce-import
label addondeploymentconfig addon-ns-config -n multicluster-engine cluster.open-cluster-management.io/backup=true
label addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine cluster.open-cluster-management.io/backup=true
label clustermanagementaddon work-manager cluster.open-cluster-management.io/backup=true
label clustermanagementaddon cluster-proxy cluster.open-cluster-management.io/backup=true
label clustermanagementaddon managed-serviceaccount cluster.open-cluster-management.io/backup=true
label KlusterletConfig mce-import-klusterlet-config cluster.open-cluster-management.io/backup=true
apiVersion: cluster.open-cluster-management.io/v1 kind: ManagedCluster metadata: annotations: agent.open-cluster-management.io/klusterlet-config: mce-import-klusterlet-config 1 labels: cloud: auto-detect vendor: auto-detect name: mce-a 2 spec: hubAcceptsClient: true leaseDurationSeconds: 60
get managedcluster
NAME HUB ACCEPTED MANAGED CLUSTER URLS JOINED AVAILABLE AGE local-cluster true https://<api.acm-hub.com:port> True True 44h mce-a true https://<api.mce-a.com:port> True True 27s
patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"agentInstallNamespace":"open-cluster-management-agent-addon-discovery"}}'
patch addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine --type=merge -p '{"spec":{"customizedVariables":[{"name":"disableMetrics","value": "true"},{"name":"disableHOManagement","value": "true"}]}}'
clusteradm addon enable --names hypershift-addon --clusters <managed-cluster-names>
get managedcluster
get deployment -n open-cluster-management-agent-addon-discovery
NAME READY UP-TO-DATE AVAILABLE AGE cluster-proxy-proxy-agent 1/1 1 1 24h klusterlet-addon-workmgr 1/1 1 1 24h hypershift-addon-agent 1/1 1 1 24h managed-serviceaccount-addon-agent 1/1 1 1 24h
apiVersion: discovery.open-cluster-management.io/v1 kind: DiscoveredCluster metadata: creationTimestamp: "2024-05-30T23:05:39Z" generation: 1 labels: hypershift.open-cluster-management.io/hc-name: hosted-cluster-1 hypershift.open-cluster-management.io/hc-namespace: clusters name: hosted-cluster-1 namespace: mce-1 resourceVersion: "1740725" uid: b4c36dca-a0c4-49f9-9673-f561e601d837 spec: apiUrl: https://a43e6fe6dcef244f8b72c30426fb6ae3-ea3fec7b113c88da.elb.us-west-1.amazonaws.com:6443 cloudProvider: aws creationTimestamp: "2024-05-30T23:02:45Z" credential: {} displayName: mce-1-hosted-cluster-1 importAsManagedCluster: false isManagedCluster: false name: hosted-cluster-1 openshiftVersion: 0.0.0 status: Active type: MultiClusterEngineHCP
apiVersion: policy.open-cluster-management.io/v1 kind: Policy metadata: name: policy-mce-hcp-autoimport namespace: open-cluster-management-global-set annotations: policy.open-cluster-management.io/standards: NIST SP 800-53 policy.open-cluster-management.io/categories: CM Configuration Management policy.open-cluster-management.io/controls: CM-2 Baseline Configuration policy.open-cluster-management.io/description: Discovered clusters that are of type MultiClusterEngineHCP can be automatically imported into ACM as managed clusters. This policy configure those discovered clusters so they are automatically imported. Fine tuning MultiClusterEngineHCP clusters to be automatically imported can be done by configure filters at the configMap or add annotation to the discoverd cluster. spec: disabled: false policy-templates: - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: mce-hcp-autoimport-config spec: object-templates: - complianceType: musthave objectDefinition: apiVersion: v1 kind: ConfigMap metadata: name: discovery-config namespace: open-cluster-management-global-set data: rosa-filter: "" remediationAction: enforce 1 severity: low - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: policy-mce-hcp-autoimport spec: remediationAction: enforce severity: low object-templates-raw: | {{- /* find the MultiClusterEngineHCP DiscoveredClusters */ -}} {{- range USDdc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }} {{- /* Check for the flag that indicates the import should be skipped */ -}} {{- USDskip := "false" -}} {{- range USDkey, USDvalue := USDdc.metadata.annotations }} {{- if and (eq USDkey "discovery.open-cluster-management.io/previously-auto-imported") (eq USDvalue "true") }} {{- USDskip = "true" }} {{- end }} {{- end }} {{- /* if the type is MultiClusterEngineHCP and the status is Active */ -}} {{- if and (eq USDdc.spec.status "Active") (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "mce-hcp-filter") USDdc.spec.displayName) (eq USDdc.spec.type "MultiClusterEngineHCP") (eq USDskip "false") }} - complianceType: musthave objectDefinition: apiVersion: discovery.open-cluster-management.io/v1 kind: DiscoveredCluster metadata: name: {{ USDdc.metadata.name }} namespace: {{ USDdc.metadata.namespace }} spec: importAsManagedCluster: true 2 {{- end }} {{- end }}
apiVersion: cluster.open-cluster-management.io/v1beta1 kind: Placement metadata: name: policy-mce-hcp-autoimport-placement namespace: open-cluster-management-global-set spec: tolerations: - key: cluster.open-cluster-management.io/unreachable operator: Exists - key: cluster.open-cluster-management.io/unavailable operator: Exists clusterSets: - global predicates: - requiredClusterSelector: labelSelector: matchExpressions: - key: local-cluster operator: In values: - "true"
apiVersion: policy.open-cluster-management.io/v1 kind: PlacementBinding metadata: name: policy-mce-hcp-autoimport-placement-binding namespace: open-cluster-management-global-set placementRef: name: policy-mce-hcp-autoimport-placement apiGroup: cluster.open-cluster-management.io kind: Placement subjects: - name: policy-mce-hcp-autoimport apiGroup: policy.open-cluster-management.io kind: Policy
get policies.policy.open-cluster-management.io policy-mce-hcp-autoimport -n <namespace>
annotations: discovery.open-cluster-management.io/previously-auto-imported: "true"
apiVersion: policy.open-cluster-management.io/v1 kind: Policy metadata: name: policy-rosa-autoimport annotations: policy.open-cluster-management.io/standards: NIST SP 800-53 policy.open-cluster-management.io/categories: CM Configuration Management policy.open-cluster-management.io/controls: CM-2 Baseline Configuration policy.open-cluster-management.io/description: OpenShift Service on AWS discovered clusters can be automatically imported into Red Hat Advanced Cluster Management as managed clusters with this policy. You can select and configure those managed clusters so you can import. Configure filters or add an annotation if you do not want all of your OpenShift Service on AWS clusters to be automatically imported. spec: remediationAction: inform 1 disabled: false policy-templates: - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: rosa-autoimport-config spec: object-templates: - complianceType: musthave objectDefinition: apiVersion: v1 kind: ConfigMap metadata: name: discovery-config namespace: open-cluster-management-global-set data: rosa-filter: "" 2 remediationAction: enforce severity: low - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: policy-rosa-autoimport spec: remediationAction: enforce severity: low object-templates-raw: | {{- /* find the ROSA DiscoveredClusters */ -}} {{- range USDdc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }} {{- /* Check for the flag that indicates the import should be skipped */ -}} {{- USDskip := "false" -}} {{- range USDkey, USDvalue := USDdc.metadata.annotations }} {{- if and (eq USDkey "discovery.open-cluster-management.io/previously-auto-imported") (eq USDvalue "true") }} {{- USDskip = "true" }} {{- end }} {{- end }} {{- /* if the type is ROSA and the status is Active */ -}} {{- if and (eq USDdc.spec.status "Active") (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "rosa-filter") USDdc.spec.displayName) (eq USDdc.spec.type "ROSA") (eq USDskip "false") }} - complianceType: musthave objectDefinition: apiVersion: discovery.open-cluster-management.io/v1 kind: DiscoveredCluster metadata: name: {{ USDdc.metadata.name }} namespace: {{ USDdc.metadata.namespace }} spec: importAsManagedCluster: true {{- end }} {{- end }} - objectDefinition: apiVersion: policy.open-cluster-management.io/v1 kind: ConfigurationPolicy metadata: name: policy-rosa-managedcluster-status spec: remediationAction: enforce severity: low object-templates-raw: | {{- /* Use the same DiscoveredCluster list to check ManagedCluster status */ -}} {{- range USDdc := (lookup "discovery.open-cluster-management.io/v1" "DiscoveredCluster" "" "").items }} {{- /* Check for the flag that indicates the import should be skipped */ -}} {{- USDskip := "false" -}} {{- range USDkey, USDvalue := USDdc.metadata.annotations }} {{- if and (eq USDkey "discovery.open-cluster-management.io/previously-auto-imported") (eq USDvalue "true") }} {{- USDskip = "true" }} {{- end }} {{- end }} {{- /* if the type is ROSA and the status is Active */ -}} {{- if and (eq USDdc.spec.status "Active") (contains (fromConfigMap "open-cluster-management-global-set" "discovery-config" "rosa-filter") USDdc.spec.displayName) (eq USDdc.spec.type "ROSA") (eq USDskip "false") }} - complianceType: musthave objectDefinition: apiVersion: cluster.open-cluster-management.io/v1 kind: ManagedCluster metadata: name: {{ USDdc.spec.displayName }} namespace: {{ USDdc.spec.displayName }} status: conditions: - type: ManagedClusterConditionAvailable status: "True" {{- end }} {{- end }}
apiVersion: cluster.open-cluster-management.io/v1beta1 kind: Placement metadata: name: placement-openshift-plus-hub spec: predicates: - requiredClusterSelector: labelSelector: matchExpressions: - key: name operator: In values: - local-cluster
apiVersion: policy.open-cluster-management.io/v1 kind: PlacementBinding metadata: name: binding-policy-rosa-autoimport placementRef: apiGroup: cluster.open-cluster-management.io kind: Placement name: placement-policy-rosa-autoimport subjects: - apiGroup: policy.open-cluster-management.io kind: Policy name: policy-rosa-autoimport
get policies.policy.open-cluster-management.io policy-rosa-autoimport -n <namespace>