Repository: giantswarm/cluster
Pull request #478: Support for region-specific container registry mirror configuration via `mirrorsTemplateName` property
Author: login: njuettner
Created: 2025-04-01T11:58:23Z | Merged: 2025-04-16T16:11:42Z

### What does this PR do?

Towards: https://github.com/giantswarm/roadmap/issues/3887

Added `mirrorsTemplateName` property to configure region-specific registry mirrors which can be used by provider charts to change default registry mirror configuration, e.g. China.

This would be integrated by setting a provider specific helm helper function, e.g.:
```yaml
{{- define "awsContainerMirrorRegistry" -}}
{{- if hasPrefix "cn-" (include "aws-region" $ ) -}}
docker.io:
  - endpoint: foo.something.com
  - endpoint: bar.something.com
another.registry.io:
  - endpoint: foo.something.com
  - endpoint: bar.something.com
{{- else -}}
{{- $.Values.global.components.containerd.containerRegistries | toYaml -}}
{{- end -}}
{{- end -}}
```

### Testing

**Also a big fat note on the test**, I currently skipped the rendering of Helm template for `helm/cluster/ci/test-registry-mirrors-values.yaml` on old code which we can change after the merge again. 

When running `helm template -n org-giantswarm -f "old/helm/cluster/ci/ci-values.yaml" -f "${test_file_path}" "old/helm/cluster"` 

It's trying to render the chart from the old/helm/cluster directory, but the custom template test helpers only exist in the current working directory, not in the old code.

- [x] CHANGELOG.md has been updated (if it exists)


Diff:
diff --git a/.github/workflows/zz_generated.diff_helm_render_templates.yaml b/.github/workflows/zz_generated.diff_helm_render_templates.yaml
index 5cb0fb17..77efc664 100644
--- a/.github/workflows/zz_generated.diff_helm_render_templates.yaml
+++ b/.github/workflows/zz_generated.diff_helm_render_templates.yaml
@@ -98,6 +98,12 @@ jobs:
               continue
             fi
 
+            # Skip registry mirrors test for old code since it requires template functions not present there
+            if [[ "${test_file_path}" == *"test-registry-mirrors-values.yaml" ]]; then
+              echo "Skipping registry mirrors test for old code"
+              continue
+            fi
+
             helm template -n org-giantswarm -f "old/helm/${{ github.event.repository.name }}/ci/ci-values.yaml" -f "${test_file_path}" "old/helm/${{ github.event.repository.name }}" > "/tmp/${test_file_path}/render-old.yaml"
           done
       - name: get the diffs
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3ac5315b..11d501c5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Support for region-specific container registry mirror configuration via `mirrorsTemplateName` property.
 - Make `ntpd` configurable.
 
 ## [2.2.0] - 2025-03-18
diff --git a/helm/cluster/.helmignore b/helm/cluster/.helmignore
new file mode 100644
index 00000000..214480f5
--- /dev/null
+++ b/helm/cluster/.helmignore
@@ -0,0 +1,2 @@
+# Ignore Go test binaries
+*.test
diff --git a/helm/cluster/README.md b/helm/cluster/README.md
index 7109bfb1..052f760e 100644
--- a/helm/cluster/README.md
+++ b/helm/cluster/README.md
@@ -891,8 +891,9 @@ Provider-specific properties that can be set by cluster-$provider chart in order
 | `providerIntegration.pauseProperties` | **Pause properties** - A map of property names and their values that will affect setting pause annotation|**Type:** `object`<br/>|
 | `providerIntegration.pauseProperties.*` |**None**|**Types:** `string, number, integer, boolean, null`<br/>|
 | `providerIntegration.provider` | **Provider** - The name of the Cluster API provider. The name here must match the name of the provider in cluster-<provider> app name.|**Type:** `string`<br/>|
-| `providerIntegration.registry` | **Registry** - The registry to use for pulling images. If not set, the default registry is used.|**Type:** `object`<br/>|
-| `providerIntegration.registry.templateName` | **Template name**|**Type:** `string`<br/>|
+| `providerIntegration.registry` | **Registry** - Configuration for container image registries|**Type:** `object`<br/>|
+| `providerIntegration.registry.mirrorsTemplateName` | **Registry Mirrors Configuration Template** - Name of a template that returns the containerd registry mirrors configuration. Defines how different registries (docker.io, gsoci.azurecr.io) should be mirrored to improve pull reliability, especially in regions with network restrictions.|**Type:** `string`<br/>|
+| `providerIntegration.registry.templateName` | **Registry URL Template** - Name of a template that returns the base URL for the container image registry (e.g., 'gsoci.azurecr.io'). Used for the sandbox image and other container images.|**Type:** `string`<br/>|
 | `providerIntegration.resourcesApi` | **Resources API** - Group, version and kind configuration that is required and used by a specific Cluster API provider.|**Type:** `object`<br/>|
 | `providerIntegration.resourcesApi.bastionResourceEnabled` | **Bastion resource enabled** - Flag that indicates if the Bastion resource is enabled and templated. This is meant only for the initial development purposes for the sake of incrementally integrating cluster chart into cluster-$provider apps.|**Type:** `boolean`<br/>**Default:** `true`|
 | `providerIntegration.resourcesApi.cleanupHelmReleaseResourcesEnabled` | **Cleanup HelmRelease hook job resource enabled** - Flag that indicates if the Cleanup HelmRelease Hook Job is enabeld and templated. This is meant only for the initial development purposes for the sake of incrementally integrating cluster chart into cluster-$provider apps.|**Type:** `boolean`<br/>**Default:** `false`|
diff --git a/helm/cluster/ci/test-registry-mirrors-values.yaml b/helm/cluster/ci/test-registry-mirrors-values.yaml
new file mode 100644
index 00000000..43c6fd54
--- /dev/null
+++ b/helm/cluster/ci/test-registry-mirrors-values.yaml
@@ -0,0 +1,49 @@
+global:
+  managementCluster: giantmc
+  release:
+    version: v27.0.0-alpha.1
+  metadata:
+    name: awesome
+    organization: giantswarm
+  connectivity:
+    baseDomain: example.gigantic.io
+  providerSpecific:
+    region: cn-north-1
+providerIntegration:
+  environmentVariables:
+    hostName: COREOS_EC2_HOSTNAME
+    ipv4: COREOS_EC2_IPV4_LOCAL
+  provider: aws
+  resourcesApi:
+    bastionResourceEnabled: false
+    clusterResourceEnabled: true
+    controlPlaneResourceEnabled: true
+    infrastructureCluster:
+      group: infrastructure.cluster.x-k8s.io
+      kind: GiantCluster
+      version: v1beta1
+    machineHealthCheckResourceEnabled: true
+    machinePoolResourcesEnabled: true
+    nodePoolKind: MachinePool
+    infrastructureMachinePool:
+      group: infrastructure.cluster.x-k8s.io
+      version: v1beta1
+      kind: GiantMachinePool
+    helmRepositoryResourcesEnabled: true
+  controlPlane:
+    resources:
+      infrastructureMachineTemplate:
+        group: infrastructure.cluster.x-k8s.io
+        kind: GiantMachineTemplate
+        version: v1beta1
+      infrastructureMachineTemplateSpecTemplateName: cluster.internal.test.controlPlane.machineTemplate.spec
+  workers:
+    defaultNodePools:
+      def00:
+        customNodeLabels:
+        - label=default
+        maxSize: 3
+        minSize: 3
+  registry:
+    templateName: cluster.internal.test.awsContainerImageRegistry
+    mirrorsTemplateName: cluster.internal.test.awsContainerMirrorRegistry
diff --git a/helm/cluster/files/etc/containerd/controlplane-config.toml b/helm/cluster/files/etc/containerd/controlplane-config.toml
index e1c2938d..83d355fd 100644
--- a/helm/cluster/files/etc/containerd/controlplane-config.toml
+++ b/helm/cluster/files/etc/containerd/controlplane-config.toml
@@ -25,7 +25,8 @@ sandbox_image = "{{ include "cluster.image.registry" $ }}/{{ $.Values.providerIn
 
 [plugins."io.containerd.grpc.v1.cri".registry]
   [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
-  {{- range $host, $config := $.Values.global.components.containerd.containerRegistries }}
+  {{- $containerMirrors := include "cluster.container.mirrors" $ | fromYaml }}
+  {{- range $host, $config := $containerMirrors }}
     [plugins."io.containerd.grpc.v1.cri".registry.mirrors."{{$host}}"]
       endpoint = [
         {{- if and $.Values.global.components.containerd.localRegistryCache.enabled (has $host $.Values.global.components.containerd.localRegistryCache.mirroredRegistries) -}}
diff --git a/helm/cluster/files/etc/containerd/workers-config.toml b/helm/cluster/files/etc/containerd/workers-config.toml
index 9d290ef6..4883d7ac 100644
--- a/helm/cluster/files/etc/containerd/workers-config.toml
+++ b/helm/cluster/files/etc/containerd/workers-config.toml
@@ -25,7 +25,8 @@ sandbox_image = "{{ include "cluster.image.registry" $ }}/{{ $.Values.providerIn
 
 [plugins."io.containerd.grpc.v1.cri".registry]
   [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
-  {{- range $host, $config := $.Values.global.components.containerd.containerRegistries }}
+  {{- $containerMirrors := include "cluster.container.mirrors" $ | fromYaml }}
+  {{- range $host, $config := $containerMirrors }}
     [plugins."io.containerd.grpc.v1.cri".registry.mirrors."{{$host}}"]
       endpoint = [
         {{- if and $.Values.global.components.containerd.localRegistryCache.enabled (has $host $.Values.global.components.containerd.localRegistryCache.mirroredRegistries) -}}
diff --git a/helm/cluster/templates/_helpers.tpl b/helm/cluster/templates/_helpers.tpl
index 1071dcd2..dc7a28ac 100644
--- a/helm/cluster/templates/_helpers.tpl
+++ b/helm/cluster/templates/_helpers.tpl
@@ -150,6 +150,15 @@ Where `data` is the data to hash and `global` is the top level scope.
 {{- $registry -}}
 {{- end -}}
 
+{{/* Function to get the container registry mirror configuration based on location */}}
+{{- define "cluster.container.mirrors" -}}
+{{- if $.Values.providerIntegration.registry.mirrorsTemplateName -}}
+  {{- include $.Values.providerIntegration.registry.mirrorsTemplateName $ -}}
+{{- else -}}
+  {{- $.Values.global.components.containerd.containerRegistries | toYaml -}}
+{{- end -}}
+{{- end -}}
+
 {{/*
   cluster.internal.get-provider-integration-values named helper template gets provider-integration Helm values and sets
   them in the $.GiantSwarm.providerIntegration object.
diff --git a/helm/cluster/templates/_test_registry_helpers.tpl b/helm/cluster/templates/_test_registry_helpers.tpl
new file mode 100644
index 00000000..534a015b
--- /dev/null
+++ b/helm/cluster/templates/_test_registry_helpers.tpl
@@ -0,0 +1,21 @@
+{{/* Test helpers for registry configuration */}}
+{{- define "cluster.internal.test.awsContainerImageRegistry" -}}
+{{- if hasPrefix "cn-" $.Values.global.providerSpecific.region -}}
+giantswarm-registry.cn-shanghai.cr.aliyuncs.com
+{{- else -}}
+gsoci.azurecr.io
+{{- end }}
+{{- end }}
+
+{{- define "cluster.internal.test.awsContainerMirrorRegistry" -}}
+{{- if hasPrefix "cn-" $.Values.global.providerSpecific.region -}}
+docker.io:
+- endpoint: giantswarm-registry.cn-shanghai.cr.aliyuncs.com
+- endpoint: gsoci.azurecr.io
+gsoci.azurecr.io:
+- endpoint: giantswarm-registry.cn-shanghai.cr.aliyuncs.com
+- endpoint: gsoci.azurecr.io
+{{- else -}}
+{{- $.Values.global.components.containerd.containerRegistries | toYaml -}}
+{{- end -}}
+{{- end -}}
diff --git a/helm/cluster/tests/components/files/containerd_expected_mirrors_registry.toml b/helm/cluster/tests/components/files/containerd_expected_mirrors_registry.toml
new file mode 100644
index 00000000..faaf418b
--- /dev/null
+++ b/helm/cluster/tests/components/files/containerd_expected_mirrors_registry.toml
@@ -0,0 +1,32 @@
+version = 2
+
+# recommended defaults from https://github.com/containerd/containerd/blob/main/docs/ops.md#base-configuration
+# set containerd as a subreaper on linux when it is not running as PID 1
+subreaper = true
+# set containerd's OOM score
+oom_score = -999
+disabled_plugins = []
+[plugins."io.containerd.runtime.v1.linux"]
+# shim binary name/path
+shim = "containerd-shim"
+# runtime binary name/path
+runtime = "runc"
+# do not use a shim when starting containers, saves on memory but
+# live restore is not supported
+no_shim = false
+
+[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+# setting runc.options unsets parent settings
+runtime_type = "io.containerd.runc.v2"
+[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+SystemdCgroup = true
+[plugins."io.containerd.grpc.v1.cri"]
+sandbox_image = "giantswarm-registry.cn-shanghai.cr.aliyuncs.com/giantswarm/pause:3.9"
+
+[plugins."io.containerd.grpc.v1.cri".registry]
+  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
+    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
+      endpoint = ["https://giantswarm-registry.cn-shanghai.cr.aliyuncs.com","https://gsoci.azurecr.io",]
+    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gsoci.azurecr.io"]
+      endpoint = ["https://zot.giantmc.example.gigantic.io","https://giantswarm-registry.cn-shanghai.cr.aliyuncs.com","https://gsoci.azurecr.io",]
+[plugins."io.containerd.grpc.v1.cri".registry.configs]
diff --git a/helm/cluster/tests/components/test_containerd_config.go b/helm/cluster/tests/components/test_containerd_config.go
index f4127929..6c166605 100644
--- a/helm/cluster/tests/components/test_containerd_config.go
+++ b/helm/cluster/tests/components/test_containerd_config.go
@@ -33,6 +33,9 @@ var (
 
 	//go:embed files/containerd_nodepool_expected_with_cgroupsv1.toml
 	expectedContainerdNodePoolWithCgroupsv1Config string
+
+	//go:embed files/containerd_expected_mirrors_registry.toml
+	expectedContainerdDifferentMirrorsConfig string
 )
 
 var _ = Describe("containerd config", func() {
@@ -59,7 +62,8 @@ var _ = Describe("containerd config", func() {
 		Entry("when only MC Zot is enabled", "test-zot-mc-values.yaml", expectedContainerdZotMcOnlyConfig),
 		Entry("when only local Zot is enabled", "test-zot-only-local-values.yaml", expectedContainerdZotLocalOnlyConfig),
 		Entry("when both local Zot and MC Zot are enabled", "test-zot-mc-and-local-values.yaml", expectedContainerdZotBothMcAndLocalConfig),
-		Entry("when node pools use cgroups v1, control plane always uses cgroups v2", "test-cgroupsv1-values.yaml", expectedContainerdWithoutCgroupsv1Config))
+		Entry("when node pools use cgroups v1, control plane always uses cgroups v2", "test-cgroupsv1-values.yaml", expectedContainerdWithoutCgroupsv1Config),
+		Entry("when registry templates are used", "test-registry-mirrors-values.yaml", expectedContainerdDifferentMirrorsConfig))
 
 	DescribeTable("rendered node pools config file",
 		func(helmValuesFile, nodepoolName, expectedConfig string) {
diff --git a/helm/cluster/values.schema.json b/helm/cluster/values.schema.json
index 485233c5..22693871 100644
--- a/helm/cluster/values.schema.json
+++ b/helm/cluster/values.schema.json
@@ -2843,11 +2843,17 @@
                 "registry": {
                     "type": "object",
                     "title": "Registry",
-                    "description": "The registry to use for pulling images. If not set, the default registry is used.",
+                    "description": "Configuration for container image registries",
                     "properties": {
+                        "mirrorsTemplateName": {
+                            "type": "string",
+                            "title": "Registry Mirrors Configuration Template",
+                            "description": "Name of a template that returns the containerd registry mirrors configuration. Defines how different registries (docker.io, gsoci.azurecr.io) should be mirrored to improve pull reliability, especially in regions with network restrictions."
+                        },
                         "templateName": {
                             "type": "string",
-                            "title": "Template name"
+                            "title": "Registry URL Template",
+                            "description": "Name of a template that returns the base URL for the container image registry (e.g., 'gsoci.azurecr.io'). Used for the sandbox image and other container images."
                         }
                     }
                 },


Reviews:
user: login: AverageMarcus
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T08:02:13Z
user: login: AverageMarcus
body: LGTM but I can't see where one of the templates is used
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T08:03:04Z
user: login: njuettner
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T08:03:08Z
user: login: AverageMarcus
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T08:10:19Z
user: login: njuettner
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T12:49:00Z
user: login: fiunchinho
body: cool!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-15T13:36:46Z

Review comments:
diff_hunk: @@ -0,0 +1,21 @@
+{{/* Test helpers for registry configuration */}}
+{{- define "cluster.internal.test.awsContainerImageRegistry" -}}
path: helm/cluster/templates/_test_registry_helpers.tpl
user: login: AverageMarcus
body: Where is this template used? I'm not seeing it in here. 
created_at: 2025-04-04T08:02:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,21 @@
+{{/* Test helpers for registry configuration */}}
+{{- define "cluster.internal.test.awsContainerImageRegistry" -}}
path: helm/cluster/templates/_test_registry_helpers.tpl
user: login: njuettner
body: It's defined in helm/cluster/ci/test-registry-mirrors-values.yaml
created_at: 2025-04-04T08:03:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,21 @@
+{{/* Test helpers for registry configuration */}}
+{{- define "cluster.internal.test.awsContainerImageRegistry" -}}
path: helm/cluster/templates/_test_registry_helpers.tpl
user: login: AverageMarcus
body: Oh this is test specific? 🤦 
created_at: 2025-04-04T08:10:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,21 @@
+{{/* Test helpers for registry configuration */}}
+{{- define "cluster.internal.test.awsContainerImageRegistry" -}}
path: helm/cluster/templates/_test_registry_helpers.tpl
user: login: njuettner
body: Yes 👍🏻 
created_at: 2025-04-04T12:49:00Z
author_association: MEMBER

PR comments:
user: login: taylorbot
created_at: 2025-04-01T12:07:00Z
body: Hey @njuettner, a test pull request has been created for you in the cluster-aws repo! Go to pull request https://github.com/giantswarm/cluster-aws/pull/1137 in order to test your cluster chart changes on AWS.
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-16T16:10:36Z
body: **There were differences in the rendered Helm template, please check! ⚠️**

<details>
<summary>Output</summary>
<!-- mandatory empty line -->

```
=== Differences when rendered with values file helm/cluster/ci/test-cgroupsv1-values.yaml ===

No difference


=== Differences when rendered with values file helm/cluster/ci/test-registry-mirrors-values.yaml ===
Diff error


=== Differences when rendered with values file helm/cluster/ci/test-required-values.yaml ===

No difference


=== Differences when rendered with values file helm/cluster/ci/test-zot-mc-and-local-values.yaml ===

No difference


=== Differences when rendered with values file helm/cluster/ci/test-zot-mc-values.yaml ===

No difference


=== Differences when rendered with values file helm/cluster/ci/test-zot-only-local-values.yaml ===

No difference
```
</details>
<!-- mandatory empty line -->

author_association: NONE