Repository: kubernetes-sigs/cluster-api
Pull request #9390 (2023-09-11 11:32:24)

🌱 observability: move metrics to config and use sidecar in kube-state-metrics
<!-- Thanks for sending a pull request! Here are some tips for you:
    1. If this is your first time, please read our contributor guidelines: https://github.com/kubernetes-sigs/cluster-api/blob/main/CONTRIBUTING.md#contributing-a-patch and developer guide https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/book/src/developer/guide.md

    2. Please add an icon to the title of this PR (see https://sigs.k8s.io/cluster-api/CONTRIBUTING.md#contributing-a-patch), and delete this line and similar ones
    the icon will be either ⚠️ (:warning:, major or breaking changes), ✨ (:sparkles:, feature additions), 🐛 (:bug:, patch and bugfixes), 📖 (:book:, documentation or proposals), or 🌱 (:seedling:, minor or other) 
-->

**What this PR does / why we need it**:

This PR:
* moves the metrics configuration from `hack/observability/kube-state-metrics` to `config/metrics`.
* adds a ClusterRole in `config/metrics/crd-clusterrole.yaml` for the aggregated clusterrole for kube-state-metrics
* adds a `kustomization.yaml` to `config/metrics` which includes the above two
* Changes to the kustomization of kube-state-metrics:
  * Adds a patch to add a sidecar container to kube-state-metrics which
    * periodicly lists configmaps in namespace `observability` which have the label `kube-state-metrics/custom-resource`  set
    * combines the yamls of the configmaps to a single configuration file for kube-state-metrics
    * updates the config file for kube-state-metrics so it reloads the configuration automatically
  * Adds a aggregated clusterrole + clusterrole binding so providers can dynamically add additional rbac rules to allow kube-state-metrics to watch the relevant resources
* Changes to `tilt up`:
  * Adds a configuration option `deploy_kustomizations` which allows to define paths to additional kustomizations which then get deployed to the cluster
    * Example usage: to reference  a kustomization which adds a configuration + clusterrole for additional kube-state-metrics crd configuration.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #

/area observability

k8s-ci-robot: @chrischdi: The label(s) `area/observability` cannot be applied, because the repository doesn't have them.

<details>

In response to [this](https://github.com/kubernetes-sigs/cluster-api/pull/9390):

>
>
>**What this PR does / why we need it**:
>
>This PR:
>* moves the metrics configuration from `hack/observability/kube-state-metrics` to `config/metrics`.
>* adds a ClusterRole in `config/metrics/crd-clusterrole.yaml` for the aggregated clusterrole for kube-state-metrics
>* adds a `kustomization.yaml` to `config/metrics` which includes the above two
>* Changes to the kustomization of kube-state-metrics:
>  * Adds a patch to add a sidecar container to kube-state-metrics which
>    * periodicly lists configmaps in namespace `observability` which have the label `kube-state-metrics/custom-resource`  set
>    * combines the yamls of the configmaps to a single configuration file for kube-state-metrics
>    * updates the config file for kube-state-metrics so it reloads the configuration automatically
>  * Adds a aggregated clusterrole + clusterrole binding so providers can dynamically add additional rbac rules to allow kube-state-metrics to watch the relevant resources
>* Changes to `tilt up`:
>  * Adds a configuration option `deploy_kustomizations` which allows to define paths to additional kustomizations which then get deployed to the cluster
>    * Example usage: to reference  a kustomization which adds a configuration + clusterrole for additional kube-state-metrics crd configuration.
>
>**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
>Fixes #
>
>/area observability


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: This PR is currently missing an area label, which is used to identify the modified component when generating release notes.

Area labels can be added by org members by writing `/area ${COMPONENT}` in a comment

Please see the [labels list](https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md#labels-that-apply-to-kubernetes-sigscluster-api-for-both-issues-and-prs) for possible areas.


<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
**Once this PR has been reviewed and has the lgtm label**, please assign [justinsb](https://github.com/justinsb) for approval. For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcluster-api).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/cluster-api/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["justinsb"]} -->
k8s-ci-robot: @chrischdi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cluster-api-verify-main | aa51a8550be0d520881cc03b294f1b4be2b4377e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/9390/pull-cluster-api-verify-main/1701197001942634496) | true | `/test pull-cluster-api-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=9390). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Achrischdi). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
chrischdi: /area devtools

Patch:
diff --git a/.github/workflows/pr-dependabot.yaml b/.github/workflows/pr-dependabot.yaml
index 7139a63cd..514858f58 100644
--- a/.github/workflows/pr-dependabot.yaml
+++ b/.github/workflows/pr-dependabot.yaml
@@ -27,7 +27,7 @@ jobs:
       uses: actions/setup-go@93397bea11091df50f3d7e59dc26a7711a8bcfbe # tag=v4.1.0
       with:
         go-version: ${{ steps.vars.outputs.go_version }}
-    - uses: actions/cache@704facf57e6136b1bc63b828d79edcd491f0ee84 # tag=v3.3.2
+    - uses: actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 # tag=v3.3.1
       name: Restore go cache
       with:
         path: |
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
index cfa9857d8..20d04c270 100644
--- a/.github/workflows/release.yaml
+++ b/.github/workflows/release.yaml
@@ -22,7 +22,7 @@ jobs:
           fetch-depth: 0
       - name: Get changed files
         id: changed-files
-        uses: tj-actions/changed-files@246636f5fa148b5ad8e65ca4c57b18af3123e5f6 # tag=v39.0.1
+        uses: tj-actions/changed-files@48566bbcc22ceb7c5809ebdd27377309f2c3de8c # tag=v39.0.0
       - name: Get release version
         id: release-version
         run: |
diff --git a/Makefile b/Makefile
index 554819e6d..8b5d4882f 100644
--- a/Makefile
+++ b/Makefile
@@ -175,11 +175,6 @@ GOVULNCHECK_VER := v1.0.0
 GOVULNCHECK := $(abspath $(TOOLS_BIN_DIR)/$(GOVULNCHECK_BIN)-$(GOVULNCHECK_VER))
 GOVULNCHECK_PKG := golang.org/x/vuln/cmd/govulncheck
 
-IMPORT_BOSS_BIN := import-boss
-IMPORT_BOSS_VER := v0.28.1
-IMPORT_BOSS := $(abspath $(TOOLS_BIN_DIR)/$(IMPORT_BOSS_BIN))
-IMPORT_BOSS_PKG := k8s.io/code-generator/cmd/import-boss
-
 CONVERSION_VERIFIER_BIN := conversion-verifier
 CONVERSION_VERIFIER := $(abspath $(TOOLS_BIN_DIR)/$(CONVERSION_VERIFIER_BIN))
 
@@ -554,16 +549,16 @@ generate-e2e-templates-main: $(KUSTOMIZE)
 	$(KUSTOMIZE) build $(INMEMORY_TEMPLATES)/main/cluster-template --load-restrictor LoadRestrictionsNone > $(INMEMORY_TEMPLATES)/main/cluster-template.yaml
 
 .PHONY: generate-metrics-config
-generate-metrics-config: $(ENVSUBST_BIN) ## Generate ./hack/observability/kube-state-metrics/crd-config.yaml
-	OUTPUT_FILE="${OBSERVABILITY_DIR}/kube-state-metrics/crd-config.yaml"; \
-	METRICS_DIR="${OBSERVABILITY_DIR}/kube-state-metrics/metrics"; \
+generate-metrics-config: $(ENVSUBST_BIN) ## Generate ./config/metrics/crd-metrics-config.yaml
+	OUTPUT_FILE="./config/metrics/crd-metrics-config.yaml"; \
+	METRIC_TEMPLATES_DIR="./config/metrics/templates"; \
 	echo "# This file was auto-generated via: make generate-metrics-config" > "$${OUTPUT_FILE}"; \
-	cat "$${METRICS_DIR}/header.yaml" >> "$${OUTPUT_FILE}"; \
+	cat "$${METRIC_TEMPLATES_DIR}/header.yaml" >> "$${OUTPUT_FILE}"; \
 	for resource in clusterclass cluster kubeadmcontrolplane kubeadmconfig machine machinedeployment machinehealthcheck machineset machinepool; do \
-		cat "$${METRICS_DIR}/$${resource}.yaml"; \
-		RESOURCE="$${resource}" ${ENVSUBST_BIN} < "$${METRICS_DIR}/common_metrics.yaml"; \
+		cat "$${METRIC_TEMPLATES_DIR}/$${resource}.yaml"; \
+		RESOURCE="$${resource}" ${ENVSUBST_BIN} < "$${METRIC_TEMPLATES_DIR}/common_metrics.yaml"; \
 		if [[ "$${resource}" != "cluster" ]]; then \
-			cat "$${METRICS_DIR}/owner_metric.yaml"; \
+			cat "$${METRIC_TEMPLATES_DIR}/owner_metric.yaml"; \
 		fi \
 	done >> "$${OUTPUT_FILE}"; \
 
@@ -612,7 +607,7 @@ APIDIFF_OLD_COMMIT ?= $(shell git rev-parse origin/main)
 apidiff: $(GO_APIDIFF) ## Check for API differences
 	$(GO_APIDIFF) $(APIDIFF_OLD_COMMIT) --print-compatible
 
-ALL_VERIFY_CHECKS = licenses boilerplate shellcheck tiltfile modules gen conversions doctoc capi-book-summary diagrams import-restrictions
+ALL_VERIFY_CHECKS = licenses boilerplate shellcheck tiltfile modules gen conversions doctoc capi-book-summary diagrams
 
 .PHONY: verify
 verify: $(addprefix verify-,$(ALL_VERIFY_CHECKS)) lint-dockerfiles ## Run all verify-* targets
@@ -695,10 +690,6 @@ verify-security: ## Verify code and images for vulnerabilities
 		exit 1; \
 	fi
 
-.PHONY: verify-import-restrictions
-verify-import-restrictions: $(IMPORT_BOSS) ## Verify import restrictions with import-boss
-	./hack/verify-import-restrictions.sh
-
 ## --------------------------------------
 ## Binaries
 ## --------------------------------------
@@ -1314,9 +1305,6 @@ $(GOLANGCI_LINT_BIN): $(GOLANGCI_LINT) ## Build a local copy of golangci-lint.
 .PHONY: $(GOVULNCHECK_BIN)
 $(GOVULNCHECK_BIN): $(GOVULNCHECK) ## Build a local copy of govulncheck.
 
-.PHONY: $(IMPORT_BOSS_BIN)
-$(IMPORT_BOSS_BIN): $(IMPORT_BOSS)
-
 $(CONTROLLER_GEN): # Build controller-gen from tools folder.
 	GOBIN=$(TOOLS_BIN_DIR) $(GO_INSTALL) $(CONTROLLER_GEN_PKG) $(CONTROLLER_GEN_BIN) $(CONTROLLER_GEN_VER)
 
@@ -1371,9 +1359,6 @@ $(GOLANGCI_LINT): # Build golangci-lint from tools folder.
 $(GOVULNCHECK): # Build govulncheck.
 	GOBIN=$(TOOLS_BIN_DIR) $(GO_INSTALL) $(GOVULNCHECK_PKG) $(GOVULNCHECK_BIN) $(GOVULNCHECK_VER)
 
-$(IMPORT_BOSS): # Build import-boss
-	GOBIN=$(TOOLS_BIN_DIR) $(GO_INSTALL) $(IMPORT_BOSS_PKG) $(IMPORT_BOSS_BIN) $(IMPORT_BOSS_VER)
-
 ## --------------------------------------
 ## Helpers
 ## --------------------------------------
diff --git a/Tiltfile b/Tiltfile
index 65e60870a..d2090f09d 100644
--- a/Tiltfile
+++ b/Tiltfile
@@ -482,6 +482,19 @@ def deploy_observability():
             objects = ["capi-visualizer:serviceaccount"],
         )
 
+def deploy_kustomizations():
+    for name in settings.get("deploy_kustomizations", []):
+        yaml = read_file("./.tiltbuild/yaml/{}.kustomization.yaml".format(name))
+        k8s_yaml(yaml)
+        objs = decode_yaml_stream(yaml)
+        print("objects")
+        print(find_all_objects_names(objs))
+        k8s_resource(
+            new_name = name,
+            objects = find_all_objects_names(objs),
+            labels = ["kustomization"],
+        )
+
 def prepare_all():
     tools_arg = "--tools kustomize,envsubst,clusterctl "
     tilt_settings_file_arg = "--tilt-settings-file " + tilt_file
@@ -640,6 +653,8 @@ deploy_provider_crds()
 
 deploy_observability()
 
+deploy_kustomizations()
+
 enable_providers()
 
 cluster_templates()
diff --git a/api/.import-restrictions b/api/.import-restrictions
deleted file mode 100644
index f6f10b3ff..000000000
--- a/api/.import-restrictions
+++ /dev/null
@@ -1,5 +0,0 @@
-rules:
-  - selectorRegexp: sigs[.]k8s[.]io/controller-runtime
-    allowedPrefixes:
-      - "sigs.k8s.io/controller-runtime/pkg/conversion"
-    forbiddenPrefixes: []
diff --git a/api/v1alpha4/cluster_types.go b/api/v1alpha4/cluster_types.go
index d4ee0658e..ddf2ab509 100644
--- a/api/v1alpha4/cluster_types.go
+++ b/api/v1alpha4/cluster_types.go
@@ -400,7 +400,7 @@ type ClusterList struct {
 }
 
 func init() {
-	objectTypes = append(objectTypes, &Cluster{}, &ClusterList{})
+	SchemeBuilder.Register(&Cluster{}, &ClusterList{})
 }
 
 // FailureDomains is a slice of FailureDomains.
diff --git a/api/v1alpha4/clusterclass_types.go b/api/v1alpha4/clusterclass_types.go
index 62a6413b7..ddb7276ad 100644
--- a/api/v1alpha4/clusterclass_types.go
+++ b/api/v1alpha4/clusterclass_types.go
@@ -134,5 +134,5 @@ type ClusterClassList struct {
 }
 
 func init() {
-	objectTypes = append(objectTypes, &ClusterClass{}, &ClusterClassList{})
+	SchemeBuilder.Register(&ClusterClass{}, &ClusterClassList{})
 }
diff --git a/api/v1alpha4/groupversion_info.go b/api/v1alpha4/groupversion_info.go
index cd8adb96d..382657c78 100644
--- a/api/v1alpha4/groupversion_info.go
+++ b/api/v1alpha4/groupversion_info.go
@@ -20,9 +20,8 @@ limitations under the License.
 package v1alpha4
 
 import (
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/runtime/schema"
+	"sigs.k8s.io/controller-runtime/pkg/scheme"
 )
 
 var (
@@ -30,19 +29,10 @@ var (
 	GroupVersion = schema.GroupVersion{Group: "cluster.x-k8s.io", Version: "v1alpha4"}
 
 	// SchemeBuilder is used to add go types to the GroupVersionKind scheme.
-	schemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
+	SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}
 
 	// AddToScheme adds the types in this group-version to the given scheme.
-	AddToScheme = schemeBuilder.AddToScheme
+	AddToScheme = SchemeBuilder.AddToScheme
 
-	objectTypes = []runtime.Object{}
-
-	// localSchemeBuilder is used for type conversions.
-	localSchemeBuilder = schemeBuilder
+	localSchemeBuilder = SchemeBuilder.SchemeBuilder
 )
-
-func addKnownTypes(scheme *runtime.Scheme) error {
-	scheme.AddKnownTypes(GroupVersion, objectTypes...)
-	metav1.AddToGroupVersion(scheme, GroupVersion)
-	return nil
-}
diff --git a/api/v1alpha4/machine_types.go b/api/v1alpha4/machine_types.go
index 6c9c7875c..6f5c7ba87 100644
--- a/api/v1alpha4/machine_types.go
+++ b/api/v1alpha4/machine_types.go
@@ -278,5 +278,5 @@ type MachineList struct {
 }
 
 func init() {
-	objectTypes = append(objectTypes, &Machine{}, &MachineList{})
+	SchemeBuilder.Register(&Machine{}, &MachineList{})
 }
diff --git a/api/v1alpha4/machinedeployment_types.go b/api/v1alpha4/machinedeployment_types.go
index dd71006fe..46e70f3f9 100644
--- a/api/v1alpha4/machinedeployment_types.go
+++ b/api/v1alpha4/machinedeployment_types.go
@@ -306,7 +306,7 @@ type MachineDeploymentList struct {
 }
 
 func init() {
-	objectTypes = append(objectTypes, &MachineDeployment{}, &MachineDeploymentList{})
+	SchemeBuilder.Register(&MachineDeployment{}, &MachineDeploymentList{})
 }
 
 // GetConditions returns the set of conditions for the machinedeployment.
diff --git a/api/v1alpha4/machinehealthcheck_types.go b/api/v1alpha4/machinehealthcheck_types.go
index 2933768ca..338970f6b 100644
--- a/api/v1alpha4/machinehealthcheck_types.go
+++ b/api/v1alpha4/machinehealthcheck_types.go
@@ -171,5 +171,5 @@ type MachineHealthCheckList struct {
 }
 
 func init() {
-	objectTypes = append(objectTypes, &MachineHealthCheck{}, &MachineHealthCheckList{})
+	SchemeBuilder.Register(&MachineHealthCheck{}, &MachineHealthCheckList{})
 }
diff --git a/api/v1alpha4/machineset_types.go b/api/v1alpha4/machineset_types.go
index 59c2bb7a1..dbfdffdb5 100644
--- a/api/v1alpha4/machineset_types.go
+++ b/api/v1alpha4/machineset_types.go
@@ -229,7 +229,7 @@ type MachineSetList struct {
 }
 
 func init() {
-	objectTypes = append(objectTypes, &MachineSet{}, &MachineSetList{})
+	SchemeBuilder.Register(&MachineSet{}, &MachineSetList{})
 }
 
 // GetConditions returns the set of conditions for the MachineSet.
diff --git a/api/v1beta1/.import-restrictions b/api/v1beta1/.import-restrictions
deleted file mode 100644
index a2e1dfd08..000000000
--- a/api/v1beta1/.import-restrictions
+++ /dev/null
@@ -1,5 +0,0 @@
-rules:
-  - selectorRegexp: sigs[.]k8s[.]io/controller-runtime
-    allowedPrefixes: []
-    forbiddenPrefixes:
-      - "sigs.k8s.io/controller-runtime"
diff --git a/api/v1beta1/index/.import-restrictions b/api/v1beta1/index/.import-restrictions
deleted file mode 100644
index b3ee9a15d..000000000
--- a/api/v1beta1/index/.import-restrictions
+++ /dev/null
@@ -1,5 +0,0 @@
-rules:
-  - selectorRegexp: sigs[.]k8s[.]io/controller-runtime
-    allowedPrefixes:
-      - "sigs.k8s.io/controller-runtime"
-    forbiddenPrefixes: []
diff --git a/config/metrics/crd-clusterrole.yaml b/config/metrics/crd-clusterrole.yaml
new file mode 100644
index 000000000..bcd5bde16
--- /dev/null
+++ b/config/metrics/crd-clusterrole.yaml
@@ -0,0 +1,38 @@
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: kube-state-metrics-custom-resource-capi
+  labels:
+    kube-state-metrics/aggregate-to-manager: "true"
+rules:
+- apiGroups:
+  - cluster.x-k8s.io
+  resources:
+  - clusterclasses
+  - clusters
+  - machinedeployments
+  - machinepools
+  - machinesets
+  - machines
+  - machinehealthchecks
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - controlplane.cluster.x-k8s.io
+  resources:
+  - kubeadmcontrolplanes
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - bootstrap.cluster.x-k8s.io
+  resources:
+  - kubeadmconfigs
+  verbs:
+  - get
+  - list
+  - watch
diff --git a/hack/observability/kube-state-metrics/crd-config.yaml b/config/metrics/crd-metrics-config.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/crd-config.yaml
rename to config/metrics/crd-metrics-config.yaml
diff --git a/config/metrics/kustomization.yaml b/config/metrics/kustomization.yaml
new file mode 100644
index 000000000..3955c8d0d
--- /dev/null
+++ b/config/metrics/kustomization.yaml
@@ -0,0 +1,13 @@
+resources:
+  - ./crd-clusterrole.yaml
+
+namespace: observability
+
+configMapGenerator:
+- name: kube-state-metrics-crd-config-capi
+  files:
+  - capi.yaml=crd-metrics-config.yaml
+  options:
+    disableNameSuffixHash: true
+    labels:
+      kube-state-metrics/custom-resource: "true"
diff --git a/hack/observability/kube-state-metrics/metrics/README.md b/config/metrics/templates/README.md
similarity index 74%
rename from hack/observability/kube-state-metrics/metrics/README.md
rename to config/metrics/templates/README.md
index 753ba822f..c52a76817 100644
--- a/hack/observability/kube-state-metrics/metrics/README.md
+++ b/config/metrics/templates/README.md
@@ -4,4 +4,4 @@
 
 The make target `generate-metrics-config` is used to generate a single file which contains the Cluster API specific custom resource configuration for kube-state-metrics.
 
-To regenerate the file `../crd-config.yaml`, execute the `make generate-metrics-config` command.
+To regenerate the file `../crd-metrics-config.yaml`, execute the `make generate-metrics-config` command.
diff --git a/hack/observability/kube-state-metrics/metrics/cluster.yaml b/config/metrics/templates/cluster.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/cluster.yaml
rename to config/metrics/templates/cluster.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/clusterclass.yaml b/config/metrics/templates/clusterclass.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/clusterclass.yaml
rename to config/metrics/templates/clusterclass.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/common_metrics.yaml b/config/metrics/templates/common_metrics.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/common_metrics.yaml
rename to config/metrics/templates/common_metrics.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/header.yaml b/config/metrics/templates/header.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/header.yaml
rename to config/metrics/templates/header.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/kubeadmconfig.yaml b/config/metrics/templates/kubeadmconfig.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/kubeadmconfig.yaml
rename to config/metrics/templates/kubeadmconfig.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/kubeadmcontrolplane.yaml b/config/metrics/templates/kubeadmcontrolplane.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/kubeadmcontrolplane.yaml
rename to config/metrics/templates/kubeadmcontrolplane.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/machine.yaml b/config/metrics/templates/machine.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/machine.yaml
rename to config/metrics/templates/machine.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/machinedeployment.yaml b/config/metrics/templates/machinedeployment.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/machinedeployment.yaml
rename to config/metrics/templates/machinedeployment.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/machinehealthcheck.yaml b/config/metrics/templates/machinehealthcheck.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/machinehealthcheck.yaml
rename to config/metrics/templates/machinehealthcheck.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/machinepool.yaml b/config/metrics/templates/machinepool.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/machinepool.yaml
rename to config/metrics/templates/machinepool.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/machineset.yaml b/config/metrics/templates/machineset.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/machineset.yaml
rename to config/metrics/templates/machineset.yaml
diff --git a/hack/observability/kube-state-metrics/metrics/owner_metric.yaml b/config/metrics/templates/owner_metric.yaml
similarity index 100%
rename from hack/observability/kube-state-metrics/metrics/owner_metric.yaml
rename to config/metrics/templates/owner_metric.yaml
diff --git a/docs/book/src/developer/tilt.md b/docs/book/src/developer/tilt.md
index 958fcce2c..40d126270 100644
--- a/docs/book/src/developer/tilt.md
+++ b/docs/book/src/developer/tilt.md
@@ -214,6 +214,14 @@ Supported values are:
 
 \*: Note: the UI will be accessible via a link in the tilt console
 
+**deploy_kustomizations** (map[string]string, default={}): If set, installs the additional kustomizations to the cluster.
+
+Example:
+```yaml
+deploy_kustomizations:
+  capv-metrics: ../cluster-api-provider-vsphere/config/metrics
+```
+
 **debug** (Map{string: Map} default{}): A map of named configurations for the provider. The key is the name of the provider.
 
 Supported settings:
diff --git a/go.mod b/go.mod
index c2b4f4d96..964d55240 100644
--- a/go.mod
+++ b/go.mod
@@ -11,7 +11,7 @@ require (
 	github.com/davecgh/go-spew v1.1.1
 	github.com/docker/distribution v2.8.2+incompatible
 	github.com/drone/envsubst/v2 v2.0.0-20210730161058-179042472c46
-	github.com/evanphx/json-patch/v5 v5.7.0
+	github.com/evanphx/json-patch/v5 v5.6.0
 	github.com/fatih/color v1.15.0
 	github.com/flatcar/container-linux-config-transpiler v0.9.4
 	github.com/flatcar/ignition v0.36.2
@@ -31,7 +31,7 @@ require (
 	github.com/valyala/fastjson v1.6.4
 	go.etcd.io/etcd/api/v3 v3.5.9
 	go.etcd.io/etcd/client/v3 v3.5.9
-	golang.org/x/oauth2 v0.12.0
+	golang.org/x/oauth2 v0.11.0
 	golang.org/x/text v0.13.0
 	gomodules.xyz/jsonpatch/v2 v2.4.0
 	google.golang.org/grpc v1.55.1
@@ -138,12 +138,12 @@ require (
 	go.uber.org/multierr v1.11.0 // indirect
 	go.uber.org/zap v1.25.0 // indirect
 	go4.org v0.0.0-20201209231011-d4a079459e60 // indirect
-	golang.org/x/crypto v0.13.0 // indirect
+	golang.org/x/crypto v0.12.0 // indirect
 	golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e // indirect
-	golang.org/x/net v0.15.0 // indirect
+	golang.org/x/net v0.14.0 // indirect
 	golang.org/x/sync v0.3.0 // indirect
-	golang.org/x/sys v0.12.0 // indirect
-	golang.org/x/term v0.12.0 // indirect
+	golang.org/x/sys v0.11.0 // indirect
+	golang.org/x/term v0.11.0 // indirect
 	golang.org/x/time v0.3.0 // indirect
 	golang.org/x/tools v0.12.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
diff --git a/go.sum b/go.sum
index 019d42400..f64064e2c 100644
--- a/go.sum
+++ b/go.sum
@@ -140,8 +140,8 @@ github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.m
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/evanphx/json-patch v5.6.0+incompatible h1:jBYDEEiFBPxA0v50tFdvOzQQTCvpL6mnFh5mB2/l16U=
 github.com/evanphx/json-patch v5.6.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
-github.com/evanphx/json-patch/v5 v5.7.0 h1:nJqP7uwL84RJInrohHfW0Fx3awjbm8qZeFv0nW9SYGc=
-github.com/evanphx/json-patch/v5 v5.7.0/go.mod h1:VNkHZ/282BpEyt/tObQO8s5CMPmYYq14uClGH4abBuQ=
+github.com/evanphx/json-patch/v5 v5.6.0 h1:b91NhWfaz02IuVxO9faSllyAtNXHMPkC5J8sJCLunww=
+github.com/evanphx/json-patch/v5 v5.6.0/go.mod h1:G79N1coSVB93tBe7j6PhzjmR3/2VvlbKOFpnXhI9Bw4=
 github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d h1:105gxyaGwCFad8crR9dcMQWvV9Hvulu6hwUh4tWPJnM=
 github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d/go.mod h1:ZZMPRZwes7CROmyNKgQzC3XPs6L/G2EJLHddWejkmf4=
 github.com/fatih/camelcase v1.0.0 h1:hxNvNX/xYBp0ovncs8WyWZrOrpBNub/JfaMvbURyft8=
@@ -318,6 +318,7 @@ github.com/imdario/mergo v0.3.13/go.mod h1:4lJ1jqUDcsbIECGy0RUJAXNIhg+6ocWgb1ALK
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
 github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/jonboulle/clockwork v0.2.2 h1:UOGuzwb1PwsrDAObMuhUnj0p5ULPj8V/xJ7Kx9qUBdQ=
@@ -568,8 +569,8 @@ golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/crypto v0.3.0/go.mod h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4=
-golang.org/x/crypto v0.13.0 h1:mvySKfSWJ+UKUii46M40LOvyWfN0s2U+46/jDd0e6Ck=
-golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
+golang.org/x/crypto v0.12.0 h1:tFM/ta59kqch6LlvYnPa0yx5a83cL2nHflFhYKvv9Yk=
+golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -648,8 +649,8 @@ golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96b
 golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY=
-golang.org/x/net v0.15.0 h1:ugBLEUaxABaB5AJqW9enI0ACdci2RUd4eP51NTBvuJ8=
-golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
+golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14=
+golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -659,8 +660,8 @@ golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43/go.mod h1:KelEdhl1UZF7XfJ
 golang.org/x/oauth2 v0.0.0-20201109201403-9fd604954f58/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
-golang.org/x/oauth2 v0.12.0 h1:smVPGxink+n1ZI5pkQa8y6fZT0RW0MgCO5bFpepy4B4=
-golang.org/x/oauth2 v0.12.0/go.mod h1:A74bZ3aGXgCY0qaIC9Ahg6Lglin4AMAco8cIv9baba4=
+golang.org/x/oauth2 v0.11.0 h1:vPL4xzxBM4niKCW6g9whtaWVXTJf1U5e4aZxxFx/gbU=
+golang.org/x/oauth2 v0.11.0/go.mod h1:LdF7O/8bLR/qWK9DrpXmbHLTouvRHK0SgJl0GmDBchk=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -726,14 +727,14 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.12.0 h1:CM0HF96J0hcLAwsHPJZjfdNzs0gftsLfgKt57wWHJ0o=
-golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM=
+golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.0.0-20220526004731-065cf7ba2467/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
-golang.org/x/term v0.12.0 h1:/ZfYdc3zq+q02Rv9vGqTeSItdzZTSNDmfTi0mBAuidU=
-golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
+golang.org/x/term v0.11.0 h1:F9tnn/DA/Im8nCwm+fX+1/eBwi4qFjRT++MhtVC4ZX0=
+golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
diff --git a/hack/observability/kube-state-metrics/chart/kustomization.yaml b/hack/observability/kube-state-metrics/chart/kustomization.yaml
deleted file mode 100644
index 3555c80e7..000000000
--- a/hack/observability/kube-state-metrics/chart/kustomization.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-helmCharts:
-  - name: kube-state-metrics
-    repo: https://prometheus-community.github.io/helm-charts
-    namespace: observability
-    releaseName: kube-state-metrics
-    valuesFile: values.yaml
-    version: 5.12.1
-
-helmGlobals:
-    # Store chart in ".charts" folder instead of "charts".
-    # Otherwise "go mod tidy" picks up dependencies of go files contained in the Helm Chart.
-    # "go mod tidy" ignores folders that begin with ".": https://pkg.go.dev/cmd/go#hdr-Package_lists_and_patterns.
-    chartHome: .charts
diff --git a/hack/observability/kube-state-metrics/chart/values.yaml b/hack/observability/kube-state-metrics/chart/values.yaml
deleted file mode 100644
index 7413a5643..000000000
--- a/hack/observability/kube-state-metrics/chart/values.yaml
+++ /dev/null
@@ -1,53 +0,0 @@
-# Add the CR configuration from the config map.
-volumeMounts:
- - mountPath: /etc/config
-   name: config-volume
-
-volumes:
- - configMap:
-     name: kube-state-metrics-crd-config
-   name: config-volume
-
-extraArgs:
-- "--custom-resource-state-config-file=/etc/config/crd-config.yaml"
-
-rbac:
-  extraRules:
-  - apiGroups:
-    - apiextensions.k8s.io
-    resources:
-    - customresourcedefinitions
-    verbs:
-    - get
-    - list
-    - watch
-  - apiGroups:
-    - cluster.x-k8s.io
-    resources:
-    - clusterclasses
-    - clusters
-    - machinedeployments
-    - machinepools
-    - machinesets
-    - machines
-    - machinehealthchecks
-    verbs:
-    - get
-    - list
-    - watch
-  - apiGroups:
-    - controlplane.cluster.x-k8s.io
-    resources:
-    - kubeadmcontrolplanes
-    verbs:
-    - get
-    - list
-    - watch
-  - apiGroups:
-    - bootstrap.cluster.x-k8s.io
-    resources:
-    - kubeadmconfigs
-    verbs:
-    - get
-    - list
-    - watch
diff --git a/hack/observability/kube-state-metrics/crd-sidecar-cm.yaml b/hack/observability/kube-state-metrics/crd-sidecar-cm.yaml
new file mode 100644
index 000000000..588ddabe0
--- /dev/null
+++ b/hack/observability/kube-state-metrics/crd-sidecar-cm.yaml
@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: kube-state-metrics-crd-sidecar-script
+data:
+  compile.sh: |
+    #!/bin/sh
+    set -x
+    
+    SOURCE_DIR="/tmp"
+    TARGET_FILE="/etc/config/crd-metrics-config.yaml"
+    # This script will use a temporary file to only overwrite the target file once.
+    TARGET_FILE_TMP="${TARGET_FILE}.tmp"
+    # Create header
+    cat << EOF > "${TARGET_FILE_TMP}"
+    kind: CustomResourceStateMetrics
+    spec:
+      resources:
+    EOF
+    # Append custom resource config of all files but remove headers
+    for f in $(ls -1 ${SOURCE_DIR}/*.yaml); do
+      cat $f | grep -v -E -e '^(-|kind: CustomResourceStateMetrics|spec:| +resources:)' \
+        >> "${TARGET_FILE_TMP}"
+    done
+    # Overwrite target file
+    mv "${TARGET_FILE_TMP}" "${TARGET_FILE}"
diff --git a/hack/observability/kube-state-metrics/crd-sidecar-patch.yaml b/hack/observability/kube-state-metrics/crd-sidecar-patch.yaml
new file mode 100644
index 000000000..3690e92f4
--- /dev/null
+++ b/hack/observability/kube-state-metrics/crd-sidecar-patch.yaml
@@ -0,0 +1,50 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: kube-state-metrics
+spec:
+  template:
+    spec:
+      containers:
+      - env:
+        - name: LABEL
+          value: kube-state-metrics/custom-resource
+        - name: FOLDER
+          value: /tmp
+        - name: RESOURCE
+          value: configmap
+        - name: NAMESPACE
+          value: observability
+        - name: SCRIPT
+          value: /script/compile.sh
+        # This image continuously collects config maps with the specified label and
+        # updates the configuration for kube-state-metrics using a script.
+        image: kiwigrid/k8s-sidecar:latest
+        name: crd-sidecar
+        volumeMounts:
+        - mountPath: /etc/config
+          name: config-volume
+        - mountPath: /script
+          name: compile-script
+      initContainers:
+      - command:
+        - /bin/sh
+        - -c
+        - |
+          cat << EOF > "/etc/config/crd-metrics-config.yaml"
+          kind: CustomResourceStateMetrics
+          spec:
+            resources: []
+          EOF
+        # This container initializes an empty configuration for kube-state-metrics
+        # to have a smooth start.
+        image: kiwigrid/k8s-sidecar:latest
+        name: init-crd-config
+        volumeMounts:
+        - mountPath: /etc/config
+          name: config-volume
+      volumes:
+      - configMap:
+          defaultMode: 511
+          name: kube-state-metrics-crd-sidecar-script
+        name: compile-script
\ No newline at end of file
diff --git a/hack/observability/kube-state-metrics/kustomization.yaml b/hack/observability/kube-state-metrics/kustomization.yaml
index cdcfcee5d..e44d34102 100644
--- a/hack/observability/kube-state-metrics/kustomization.yaml
+++ b/hack/observability/kube-state-metrics/kustomization.yaml
@@ -1,15 +1,30 @@
 resources:
   - ../namespace.yaml
-  # The kube-state-metrics helm chart will reference a configmap with name `kube-state-metrics-crd-config`.
-  # The configMapGenerator below will create the configmap and append a hash suffix calculated from its
-  # content to the name. Kustomize will append the suffix hash to all references in the helm chart, but
-  # only when the helm chart content is referenced in "resources".
-  # This would not work if the helm chart is configured in this file via the "helmCharts" option.
-  - ./chart
+  # Aggregated ClusterRole and ClusterRoleBinding for kube-state-metrics crd resources.
+  - rbac-crd-aggregation.yaml
+  # Adds the configmap which contains the script to build the final crd configuration.
+  - crd-sidecar-cm.yaml
+  # Adds the core CAPI crd metrics configmap and CusterRole.
+  - ../../../config/metrics
 
 namespace: observability
 
-configMapGenerator:
-- name: kube-state-metrics-crd-config
-  files:
-  - crd-config.yaml
+helmCharts:
+  - name: kube-state-metrics
+    repo: https://prometheus-community.github.io/helm-charts
+    namespace: observability
+    releaseName: kube-state-metrics
+    valuesFile: values.yaml
+    version: 5.12.1
+
+helmGlobals:
+    # Store chart in ".charts" folder instead of "charts".
+    # Otherwise "go mod tidy" picks up dependencies of go files contained in the Helm Chart.
+    # "go mod tidy" ignores folders that begin with ".": https://pkg.go.dev/cmd/go#hdr-Package_lists_and_patterns.
+    chartHome: .charts
+
+patches:
+# Adds the sidecar container which assembles the final configuration for kube-state-metrics
+- path: crd-sidecar-patch.yaml
+  target:
+    kind: Deployment
diff --git a/hack/observability/kube-state-metrics/rbac-crd-aggregation.yaml b/hack/observability/kube-state-metrics/rbac-crd-aggregation.yaml
new file mode 100644
index 000000000..a7c08eb24
--- /dev/null
+++ b/hack/observability/kube-state-metrics/rbac-crd-aggregation.yaml
@@ -0,0 +1,24 @@
+---
+# ClusterRole to aggregate other ClusterRoles for different Custom Resource Configurations
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: kube-state-metrics-aggregated-role
+aggregationRule:
+  clusterRoleSelectors:
+  - matchLabels:
+      kube-state-metrics/aggregate-to-manager: "true"
+---
+# ClusterRoleBinding for the aggregation role
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: kube-state-metrics-custom-resource-rolebinding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: kube-state-metrics-aggregated-role
+subjects:
+- kind: ServiceAccount
+  name: kube-state-metrics
+  namespace: observability
diff --git a/hack/observability/kube-state-metrics/values.yaml b/hack/observability/kube-state-metrics/values.yaml
new file mode 100644
index 000000000..b9dbfa508
--- /dev/null
+++ b/hack/observability/kube-state-metrics/values.yaml
@@ -0,0 +1,22 @@
+# Add the CR configuration from the config map.
+volumeMounts:
+- mountPath: /etc/config
+  name: config-volume
+
+volumes:
+- emptyDir: {}
+  name: config-volume
+
+extraArgs:
+- "--custom-resource-state-config-file=/etc/config/crd-metrics-config.yaml"
+
+rbac:
+  extraRules:
+  - apiGroups:
+    - apiextensions.k8s.io
+    resources:
+    - customresourcedefinitions
+    verbs:
+    - get
+    - list
+    - watch
diff --git a/hack/tools/go.mod b/hack/tools/go.mod
index 140d5ee74..896cf7593 100644
--- a/hack/tools/go.mod
+++ b/hack/tools/go.mod
@@ -56,7 +56,7 @@ require (
 	github.com/drone/envsubst/v2 v2.0.0-20210730161058-179042472c46 // indirect
 	github.com/emicklei/go-restful/v3 v3.11.0 // indirect
 	github.com/evanphx/json-patch v5.6.0+incompatible // indirect
-	github.com/evanphx/json-patch/v5 v5.7.0 // indirect
+	github.com/evanphx/json-patch/v5 v5.6.0 // indirect
 	github.com/fsnotify/fsnotify v1.6.0 // indirect
 	github.com/go-errors/errors v1.4.2 // indirect
 	github.com/go-logr/logr v1.2.4 // indirect
@@ -111,7 +111,7 @@ require (
 	golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e // indirect
 	golang.org/x/mod v0.12.0 // indirect
 	golang.org/x/net v0.15.0 // indirect
-	golang.org/x/oauth2 v0.12.0 // indirect
+	golang.org/x/oauth2 v0.11.0 // indirect
 	golang.org/x/sync v0.3.0 // indirect
 	golang.org/x/sys v0.12.0 // indirect
 	golang.org/x/term v0.12.0 // indirect
diff --git a/hack/tools/go.sum b/hack/tools/go.sum
index c70d8c7f6..ca8d0d95b 100644
--- a/hack/tools/go.sum
+++ b/hack/tools/go.sum
@@ -140,8 +140,8 @@ github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/evanphx/json-patch v5.6.0+incompatible h1:jBYDEEiFBPxA0v50tFdvOzQQTCvpL6mnFh5mB2/l16U=
 github.com/evanphx/json-patch v5.6.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
-github.com/evanphx/json-patch/v5 v5.7.0 h1:nJqP7uwL84RJInrohHfW0Fx3awjbm8qZeFv0nW9SYGc=
-github.com/evanphx/json-patch/v5 v5.7.0/go.mod h1:VNkHZ/282BpEyt/tObQO8s5CMPmYYq14uClGH4abBuQ=
+github.com/evanphx/json-patch/v5 v5.6.0 h1:b91NhWfaz02IuVxO9faSllyAtNXHMPkC5J8sJCLunww=
+github.com/evanphx/json-patch/v5 v5.6.0/go.mod h1:G79N1coSVB93tBe7j6PhzjmR3/2VvlbKOFpnXhI9Bw4=
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
 github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
 github.com/frankban/quicktest v1.14.4 h1:g2rn0vABPOOXmZUj+vbmUp0lPoXEMuhTpIluN0XL9UY=
@@ -298,6 +298,7 @@ github.com/imdario/mergo v0.3.11/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH
 github.com/imdario/mergo v0.3.13 h1:lFzP57bqS/wsqKssCGmtLAb8A0wKjLGrve2q3PPVcBk=
 github.com/imdario/mergo v0.3.13/go.mod h1:4lJ1jqUDcsbIECGy0RUJAXNIhg+6ocWgb1ALK2O4oXg=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
+github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
 github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
@@ -580,8 +581,8 @@ golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43/go.mod h1:KelEdhl1UZF7XfJ
 golang.org/x/oauth2 v0.0.0-20201109201403-9fd604954f58/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
-golang.org/x/oauth2 v0.12.0 h1:smVPGxink+n1ZI5pkQa8y6fZT0RW0MgCO5bFpepy4B4=
-golang.org/x/oauth2 v0.12.0/go.mod h1:A74bZ3aGXgCY0qaIC9Ahg6Lglin4AMAco8cIv9baba4=
+golang.org/x/oauth2 v0.11.0 h1:vPL4xzxBM4niKCW6g9whtaWVXTJf1U5e4aZxxFx/gbU=
+golang.org/x/oauth2 v0.11.0/go.mod h1:LdF7O/8bLR/qWK9DrpXmbHLTouvRHK0SgJl0GmDBchk=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
diff --git a/hack/tools/internal/tilt-prepare/main.go b/hack/tools/internal/tilt-prepare/main.go
index 5e7f4f7e6..ddd41ee95 100644
--- a/hack/tools/internal/tilt-prepare/main.go
+++ b/hack/tools/internal/tilt-prepare/main.go
@@ -104,13 +104,14 @@ var (
 // Types used to de-serialize the tilt-settings.yaml/json file from the Cluster API repository.
 
 type tiltSettings struct {
-	Debug               map[string]tiltSettingsDebugConfig `json:"debug,omitempty"`
-	ExtraArgs           map[string]tiltSettingsExtraArgs   `json:"extra_args,omitempty"`
-	DeployCertManager   *bool                              `json:"deploy_cert_manager,omitempty"`
-	DeployObservability []string                           `json:"deploy_observability,omitempty"`
-	EnableProviders     []string                           `json:"enable_providers,omitempty"`
-	AllowedContexts     []string                           `json:"allowed_contexts,omitempty"`
-	ProviderRepos       []string                           `json:"provider_repos,omitempty"`
+	Debug                map[string]tiltSettingsDebugConfig `json:"debug,omitempty"`
+	ExtraArgs            map[string]tiltSettingsExtraArgs   `json:"extra_args,omitempty"`
+	DeployCertManager    *bool                              `json:"deploy_cert_manager,omitempty"`
+	DeployObservability  []string                           `json:"deploy_observability,omitempty"`
+	DeployKustomizations map[string]string                  `json:"deploy_kustomizations,omitempty"`
+	EnableProviders      []string                           `json:"enable_providers,omitempty"`
+	AllowedContexts      []string                           `json:"allowed_contexts,omitempty"`
+	ProviderRepos        []string                           `json:"provider_repos,omitempty"`
 }
 
 type tiltSettingsDebugConfig struct {
@@ -308,6 +309,13 @@ func tiltResources(ctx context.Context, ts *tiltSettings) error {
 		)
 	}
 
+	for name, path := range ts.DeployKustomizations {
+		name := fmt.Sprintf("%s.kustomization", name)
+		tasks[name] = sequential(
+			kustomizeTask(path, fmt.Sprintf("%s.yaml", name)),
+		)
+	}
+
 	// Add read configurations from provider repos
 	for _, p := range ts.ProviderRepos {
 		tiltProviderConfigs, err := loadTiltProvider(p)
diff --git a/hack/verify-import-restrictions.sh b/hack/verify-import-restrictions.sh
deleted file mode 100755
index 818503a72..000000000
--- a/hack/verify-import-restrictions.sh
+++ /dev/null
@@ -1,55 +0,0 @@
-#!/usr/bin/env bash
-
-# Copyright 2023 The Kubernetes Authors.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# This script checks import restrictions. The script looks for a file called
-# `.import-restrictions` in each directory, then all imports of the package are
-# checked against each "rule" in the file.
-# Usage: `hack/verify-import-restrictions.sh`.
-
-set -o errexit
-set -o nounset
-set -o pipefail
-
-sub_packages=(
-  "api"
-)
-
-packages=()
-visit() {
-  local count=0
-  for file in "$1"/* ; do
-    if [ -d "$file" ]; then
-      visit "$file"
-    elif [ -f "$file" ]; then
-      ((count += 1))
-    fi
-  done
-  if [ "$count" -gt 0 ]; then
-    # import-boss may not accept directories without any sources
-    packages+=("./$1")
-  fi
-}
-for d in "${sub_packages[@]}"; do
-  visit "$d"
-done
-
-INPUT_DIRS="$(IFS=, ; echo "${packages[*]}")"
-echo "Enforcing imports in source codes under the following directories: ${INPUT_DIRS}"
-
-# Make sure GOPATH is unset to avoid behavior inconsistency
-# as import-boss will go through the sources
-unset GOPATH
-import-boss --include-test-files=true --verify-only --input-dirs "${INPUT_DIRS}"
diff --git a/internal/controllers/topology/cluster/conditions.go b/internal/controllers/topology/cluster/conditions.go
index 44d6f9e10..79ed85b31 100644
--- a/internal/controllers/topology/cluster/conditions.go
+++ b/internal/controllers/topology/cluster/conditions.go
@@ -83,8 +83,7 @@ func (r *Reconciler) reconcileTopologyReconciledCondition(s *scope.Scope, cluste
 				clusterv1.TopologyReconciledCondition,
 				clusterv1.TopologyReconciledClusterClassNotReconciledReason,
 				clusterv1.ConditionSeverityInfo,
-				"ClusterClass not reconciled. If this condition persists please check ClusterClass status. A ClusterClass is reconciled if"+
-					".status.observedGeneration == .metadata.generation is true. If this is not the case either ClusterClass reconciliation failed or the ClusterClass is paused",
+				"ClusterClass not reconciled. If this condition persists please check ClusterClass status.",
 			),
 		)
 		return nil


Test patch:
diff --git a/internal/controllers/topology/cluster/conditions_test.go b/internal/controllers/topology/cluster/conditions_test.go
index 9113f5497..d2b21aa3a 100644
--- a/internal/controllers/topology/cluster/conditions_test.go
+++ b/internal/controllers/topology/cluster/conditions_test.go
@@ -76,11 +76,10 @@ func TestReconcileTopologyReconciledCondition(t *testing.T) {
 					},
 				},
 			},
-			wantConditionStatus: corev1.ConditionFalse,
-			wantConditionReason: clusterv1.TopologyReconciledClusterClassNotReconciledReason,
-			wantConditionMessage: "ClusterClass not reconciled. If this condition persists please check ClusterClass status. A ClusterClass is reconciled if" +
-				".status.observedGeneration == .metadata.generation is true. If this is not the case either ClusterClass reconciliation failed or the ClusterClass is paused",
-			wantErr: false,
+			wantConditionStatus:  corev1.ConditionFalse,
+			wantConditionReason:  clusterv1.TopologyReconciledClusterClassNotReconciledReason,
+			wantConditionMessage: "ClusterClass not reconciled. If this condition persists please check ClusterClass status.",
+			wantErr:              false,
 		},
 		{
 			name:         "should set the condition to false if the there is a blocking hook",
diff --git a/internal/test/envtest/environment.go b/internal/test/envtest/environment.go
index e6028ce1f..6b298414d 100644
--- a/internal/test/envtest/environment.go
+++ b/internal/test/envtest/environment.go
@@ -143,11 +143,11 @@ func Run(ctx context.Context, input RunInput) int {
 
 	if input.MinK8sVersion != "" {
 		if err := version.CheckKubernetesVersion(env.Config, input.MinK8sVersion); err != nil {
-			fmt.Printf("[ERROR] Cannot run tests after failing version check: %v\n", err)
+			fmt.Printf("[IMPORTANT] skipping tests after failing version check: %v\n", err)
 			if err := env.stop(); err != nil {
 				fmt.Println("[WARNING] Failed to stop the test environment")
 			}
-			return 1
+			return 0
 		}
 	}
 
diff --git a/test/go.mod b/test/go.mod
index 72fde0e6e..63da97dc7 100644
--- a/test/go.mod
+++ b/test/go.mod
@@ -9,7 +9,7 @@ require (
 	github.com/docker/docker v24.0.6+incompatible
 	github.com/docker/go-connections v0.4.0
 	github.com/emicklei/go-restful/v3 v3.11.0
-	github.com/evanphx/json-patch/v5 v5.7.0
+	github.com/evanphx/json-patch/v5 v5.6.0
 	github.com/flatcar/ignition v0.36.2
 	github.com/go-logr/logr v1.2.4
 	github.com/google/go-cmp v0.5.9
@@ -135,7 +135,7 @@ require (
 	go.uber.org/zap v1.25.0 // indirect
 	golang.org/x/crypto v0.13.0 // indirect
 	golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e // indirect
-	golang.org/x/oauth2 v0.12.0 // indirect
+	golang.org/x/oauth2 v0.11.0 // indirect
 	golang.org/x/sync v0.3.0 // indirect
 	golang.org/x/sys v0.12.0 // indirect
 	golang.org/x/term v0.12.0 // indirect
diff --git a/test/go.sum b/test/go.sum
index 7bb101f9a..5b297c80d 100644
--- a/test/go.sum
+++ b/test/go.sum
@@ -159,9 +159,8 @@ github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/evanphx/json-patch v5.6.0+incompatible h1:jBYDEEiFBPxA0v50tFdvOzQQTCvpL6mnFh5mB2/l16U=
 github.com/evanphx/json-patch v5.6.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
+github.com/evanphx/json-patch/v5 v5.6.0 h1:b91NhWfaz02IuVxO9faSllyAtNXHMPkC5J8sJCLunww=
 github.com/evanphx/json-patch/v5 v5.6.0/go.mod h1:G79N1coSVB93tBe7j6PhzjmR3/2VvlbKOFpnXhI9Bw4=
-github.com/evanphx/json-patch/v5 v5.7.0 h1:nJqP7uwL84RJInrohHfW0Fx3awjbm8qZeFv0nW9SYGc=
-github.com/evanphx/json-patch/v5 v5.7.0/go.mod h1:VNkHZ/282BpEyt/tObQO8s5CMPmYYq14uClGH4abBuQ=
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
 github.com/felixge/httpsnoop v1.0.3 h1:s/nj+GCswXYzN5v2DpNMuMQYe+0DDwt5WVCU6CWBdXk=
 github.com/felixge/httpsnoop v1.0.3/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
@@ -686,8 +685,8 @@ golang.org/x/oauth2 v0.0.0-20201109201403-9fd604954f58/go.mod h1:KelEdhl1UZF7XfJ
 golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
 golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
-golang.org/x/oauth2 v0.12.0 h1:smVPGxink+n1ZI5pkQa8y6fZT0RW0MgCO5bFpepy4B4=
-golang.org/x/oauth2 v0.12.0/go.mod h1:A74bZ3aGXgCY0qaIC9Ahg6Lglin4AMAco8cIv9baba4=
+golang.org/x/oauth2 v0.11.0 h1:vPL4xzxBM4niKCW6g9whtaWVXTJf1U5e4aZxxFx/gbU=
+golang.org/x/oauth2 v0.11.0/go.mod h1:LdF7O/8bLR/qWK9DrpXmbHLTouvRHK0SgJl0GmDBchk=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=