Repository: kubernetes/kops (go)

Issue:
"kops update cluster" panics while creating JWKS for OIDC
**1. What `kops` version are you running?**

_Client version: 1.24.1 (git-v1.24.1)_

**2. What Kubernetes version are you running?**

Starting with version **1.19.9**, upgrading to version **1.21.14**.

**3. What cloud provider are you using?**

AWS

**4. What commands did you run?  What is the simplest way to reproduce this issue?**

```shell
kops replace --filename=cluster.yaml
kops update cluster --yes
```

**5. What happened after the commands executed?**

It appears that _kops update cluster_ fails when it panics preparing for publishing OIDC Discovery documents to an S3 bucket:
```
W0824 08:36:09.215431   12986 external_access.go:39] KubernetesAPIAccess is empty
I0824 08:36:10.848893   12986 executor.go:111] Tasks: 0 done / 393 total; 110 can run
I0824 08:36:11.805410   12986 executor.go:111] Tasks: 110 done / 393 total; 83 can run
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x108 pc=0x3ec3a7c]

goroutine 1531 [running]:
k8s.io/kops/pkg/model.(*OIDCKeys).Open(0xc0017a32a0?)
        k8s.io/kops/pkg/model/issuerdiscovery.go:134 +0x21c
k8s.io/kops/upup/pkg/fi.CopyResource({0x5c5c140, 0xc00104de60}, {0x5c61d40?, 0xc00108c7c8?})
        k8s.io/kops/upup/pkg/fi/resources.go:85 +0x72
k8s.io/kops/upup/pkg/fi.ResourceAsBytes({0x5c61d40, 0xc00108c7c8})
        k8s.io/kops/upup/pkg/fi/resources.go:112 +0x4c
k8s.io/kops/upup/pkg/fi/fitasks.(*ManagedFile).Render(0x5?, 0x0?, 0xc00168b740?, 0xc000ee9240, 0x2?)
        k8s.io/kops/upup/pkg/fi/fitasks/managedfile.go:154 +0x70
reflect.Value.call({0x4f99d60?, 0xc000ee9240?, 0x4?}, {0x537191a, 0x4}, {0xc000c28c60, 0x4, 0x5c91ab0?})
        reflect/value.go:556 +0x845
reflect.Value.Call({0x4f99d60?, 0xc000ee9240?, 0x53a8fcc?}, {0xc000c28c60, 0x4, 0x4})
        reflect/value.go:339 +0xbf
k8s.io/kops/upup/pkg/fi.(*Context).Render(0xc0011ec0a0, {0x5c633a0?, 0x0}, {0x5c633a0?, 0xc000ee9240}, {0x5c633a0?, 0xc00171f440})
        k8s.io/kops/upup/pkg/fi/context.go:225 +0xf2e
k8s.io/kops/upup/pkg/fi.DefaultDeltaRunMethod({0x5c633a0?, 0xc000ee9240}, 0xc0011ec0a0)
        k8s.io/kops/upup/pkg/fi/default_methods.go:82 +0x46c
k8s.io/kops/upup/pkg/fi/fitasks.(*ManagedFile).Run(0xc0008a2c18?, 0x0?)
        k8s.io/kops/upup/pkg/fi/fitasks/managedfile.go:109 +0x26
k8s.io/kops/upup/pkg/fi.(*executor).forkJoin.func1(0xc00146d8f0, 0x4)
        k8s.io/kops/upup/pkg/fi/executor.go:187 +0x1ea
created by k8s.io/kops/upup/pkg/fi.(*executor).forkJoin
        k8s.io/kops/upup/pkg/fi/executor.go:183 +0x86
```
It appears to be failing on [this line in file _pkg/model/issuerdiscovery.go_](https://sourcegraph.com/github.com/kubernetes/kops@release-1.24/-/blob/pkg/model/issuerdiscovery.go?L134#tab=def), trying to access a public key in memory.

**6. What did you expect to happen?**

_kops update cluster_ would publish all the OIDC Discovery documents to S3, and continue on with the rest of its tasks.

**7. Please provide your cluster manifest.**

<details>
<summary>cluster.yaml file</summary>

```yaml
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  name: my-cluster.example.com
spec:
  additionalSans:
  - api.internal.my-cluster.example.com
  api:
    loadBalancer:
      additionalSecurityGroups:
      - sg-005e2b9c6ffed8582
      class: Network
      crossZoneLoadBalancing: true
      type: Public
  authorization:
    rbac: {}
  certManager:
    enabled: true
    managed: false
  cloudConfig:
    disableSecurityGroupIngress: true
  cloudProvider: aws
  clusterAutoscaler:
    balanceSimilarNodeGroups: true
    enabled: true
  configBase: s3://my-kops-state/my-cluster.example.com
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-us-east-2a
      name: a
    - instanceGroup: master-us-east-2b
      name: b
    - instanceGroup: master-us-east-2c
      name: c
    manager:
      env:
      - name: ETCD_LISTEN_METRICS_URLS
        value: http://0.0.0.0:8081
      - name: ETCD_METRICS
        value: extensive
    name: main
  - etcdMembers:
    - instanceGroup: master-us-east-2a
      name: a
    - instanceGroup: master-us-east-2b
      name: b
    - instanceGroup: master-us-east-2c
      name: c
    manager:
      env:
      - name: ETCD_LISTEN_METRICS_URLS
        value: http://0.0.0.0:8082
      - name: ETCD_METRICS
        value: basic
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    featureGates:
      EphemeralContainers: "true"
  kubeDNS:
    provider: KubeDNS
  kubeProxy:
    enabled: false
  kubelet:
    anonymousAuth: false
    featureGates:
      EphemeralContainers: "true"
    kubeReserved:
      cpu: 750m
      memory: .75Gi
  kubernetesVersion: 1.21.14
  metricsServer:
    enabled: true
  networkCIDR: 10.3.0.0/16
  networkID: vpc-087cd3eb3bf613986
  networking:
    calico:
      bpfEnabled: true
      crossSubnet: true
      encapsulationMode: vxlan
      typhaReplicas: 3
  nonMasqueradeCIDR: 100.64.0.0/10
  serviceAccountIssuerDiscovery:
    discoveryStore: s3://my-kops-oidc-discovery/my-cluster
    enableAWSOIDCProvider: true
  sshAccess:
  - 184.74.210.37/32
  - 184.74.210.38/32
  - 207.141.66.101/32
  - 207.141.66.99/32
  - 212.187.232.28/32
  - 212.187.232.29/32
  - 4.53.131.109/32
  - 4.53.131.110/32
  - 4.71.99.125/32
  - 4.71.99.126/32
  subnets:
  - cidr: 10.3.100.0/22
    id: subnet-0cd20dfb64345dede
    name: utility-us-east-2a
    type: Utility
    zone: us-east-2a
  - cidr: 10.3.104.0/22
    id: subnet-0657e2c2163960a79
    name: utility-us-east-2b
    type: Utility
    zone: us-east-2b
  - cidr: 10.3.108.0/22
    id: subnet-013e44ade2633a1b1
    name: utility-us-east-2c
    type: Utility
    zone: us-east-2c
  - cidr: 10.3.0.0/22
    egress: nat-06a85bf97c4a5b65d
    id: subnet-0ca2f5a3ab50e538e
    name: us-east-2a
    type: Private
    zone: us-east-2a
  - cidr: 10.3.4.0/22
    egress: nat-054d637847b63ea36
    id: subnet-047a72902591ebe60
    name: us-east-2b
    type: Private
    zone: us-east-2b
  - cidr: 10.3.8.0/22
    egress: nat-0df765ca07bb44f0f
    id: subnet-051d2325bcab67fa6
    name: us-east-2c
    type: Private
    zone: us-east-2c
  topology:
    dns:
      type: Public
    masters: private
    nodes: private
  updatePolicy: external
```
</details>

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
  Paste the logs into this report, or in a gist and provide the gist link here.**

Here is the _kops update cluster_ output at verbosity level ten, just before the failure:
```
I0824 08:54:41.457895   13698 executor.go:186] Executing task "MirrorSecrets/mirror-secrets": *fitasks.MirrorSecrets {"Name":"mirror-secrets","Lifecycle":"Sync","MirrorPath":{}}
I0824 08:54:41.461003   13698 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0824 08:54:41.461652   13698 request_logger.go:45] AWS request: iam/GetInstanceProfile
I0824 08:54:41.462143   13698 request_logger.go:45] AWS request: ec2/DescribeSubnets
I0824 08:54:41.462148   13698 request_logger.go:45] AWS request: iam/GetInstanceProfile
I0824 08:54:41.463820   13698 request_logger.go:45] AWS request: iam/ListAttachedRolePolicies
I0824 08:54:41.472058   13698 request_logger.go:45] AWS request: iam/GetRolePolicy
I0824 08:54:41.472136   13698 request_logger.go:45] AWS request: ec2/DescribeSubnets
I0824 08:54:41.472541   13698 s3fs.go:329] Reading file "s3://my-kops-oidc-discovery/my-cluster/openid/v1/jwks"
I0824 08:54:41.472820   13698 request_logger.go:45] AWS request: iam/GetRolePolicy
I0824 08:54:41.473050   13698 request_logger.go:45] AWS request: ec2/DescribeInternetGateways
I0824 08:54:41.473944   13698 request_logger.go:45] AWS request: ec2/DescribeSubnets
I0824 08:54:41.474061   13698 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0824 08:54:41.473806   13698 request_logger.go:45] AWS request: iam/ListAttachedRolePolicies
I0824 08:54:41.473914   13698 request_logger.go:45] AWS request: iam/GetRolePolicy
I0824 08:54:41.510755   13698 request_logger.go:45] AWS request: elasticloadbalancing/DescribeTargetGroups
panic: runtime error: invalid memory address or nil pointer dereference
```

Earlier, I see this pertinent log message:
```
I0824 08:54:41.459055   13698 executor.go:186] Executing task "ManagedFile/keys.json": *fitasks.ManagedFile {"Name":"keys.json","Lifecycle":"Sync","Base":"s3://my-kops-oidc-discovery/my-cluster","Location":"openid/v1/jwks","Contents":{"SigningKey":{"Name":"service-account","alternateNames":null,"Lifecycle":"Sync","Signer":null,"subject":"cn=service-account","issuer":"","type":"ca","oldFormat":false}},"Public":true}
```
Note that at present, the aforementioned S3 bucket exists, but there is no existing object with the path _my-cluster/openid/v1/jwks_.

**9. Anything else do we need to know?**

I have been able to upgrade clusters and activate the "spec.serviceAccountIssuerDiscovery.enableAWSOIDCProvider" field's behavior successfully with earlier versions of _kOps_, which wrote the S3 object as necessary. This version of _kOps_ appears to be failing before it can create this S3 object. _kOps_ _was_ able to create the _my-cluster/.well-known/openid-configuration_ object in the same bucket.

See #13353 for what looks to be an earlier report of a similar defect.

See [the prior discussion in the "kops-users" channel](https://kubernetes.slack.com/archives/C3QUFP0QM/p1661344867276039) of the "Kubernetes" Slack workspace.

/kind bug

Pull request:
get-keypairs: Tolerate key set items without certificates
Allow the _kops get keypairs_ command to consume key sets with old key pair items that lack an associated X.509 certificate. When the command is invoked without the `--distrusted` flag set to true, omit these old items as if they're distrusted. Conversely, when the command is invoked with the `--distrusted` flag set to true, include these items, but omit their details that would be contingent on the nonexistent certificate.

In order to supply only information that is known to be true, treat the following fields in the output as newly optional:
- _issuer_
- _notAfter_
- _notBefore_
- _subject_

With no certificate present, it's not possible to present concrete values for those fields.

Fixes #14174.

Patch:
diff --git a/cmd/kops/get_keypairs.go b/cmd/kops/get_keypairs.go
index 3be0d8c8c6..72dcbed262 100644
--- a/cmd/kops/get_keypairs.go
+++ b/cmd/kops/get_keypairs.go
@@ -89,12 +89,12 @@ type keypairItem struct {
 	ID                string     `json:"id"`
 	DistrustTimestamp *time.Time `json:"distrustTimestamp,omitempty"`
 	IsPrimary         bool       `json:"isPrimary,omitempty"`
-	Subject           string     `json:"subject"`
-	Issuer            string     `json:"issuer"`
+	Subject           string     `json:"subject,omitempty"`
+	Issuer            string     `json:"issuer,omitempty"`
 	AlternateNames    []string   `json:"alternateNames,omitempty"`
 	IsCA              bool       `json:"isCA,omitempty"`
-	NotBefore         time.Time  `json:"notBefore"`
-	NotAfter          time.Time  `json:"notAfter"`
+	NotBefore         *time.Time `json:"notBefore,omitempty"`
+	NotAfter          *time.Time `json:"notAfter,omitempty"`
 	KeyLength         *int       `json:"keyLength,omitempty"`
 	HasPrivateKey     bool       `json:"hasPrivateKey,omitempty"`
 }
@@ -122,30 +122,39 @@ func listKeypairs(keyStore fi.CAStore, names []string, includeDistrusted bool) (
 		}
 
 		for _, item := range keyset.Items {
-			if includeDistrusted || item.DistrustTimestamp == nil {
-				var alternateNames []string
-				alternateNames = append(alternateNames, item.Certificate.Certificate.DNSNames...)
-				alternateNames = append(alternateNames, item.Certificate.Certificate.EmailAddresses...)
-				for _, ip := range item.Certificate.Certificate.IPAddresses {
-					alternateNames = append(alternateNames, ip.String())
-				}
-				sort.Strings(alternateNames)
-
+			if includeDistrusted || (item.DistrustTimestamp == nil && item.Certificate != nil) {
 				keypair := keypairItem{
 					Name:              name,
 					ID:                item.Id,
 					DistrustTimestamp: item.DistrustTimestamp,
 					IsPrimary:         item.Id == keyset.Primary.Id,
-					Subject:           item.Certificate.Subject.String(),
-					Issuer:            item.Certificate.Certificate.Issuer.String(),
-					AlternateNames:    alternateNames,
-					IsCA:              item.Certificate.IsCA,
-					NotBefore:         item.Certificate.Certificate.NotBefore.UTC(),
-					NotAfter:          item.Certificate.Certificate.NotAfter.UTC(),
 					HasPrivateKey:     item.PrivateKey != nil,
 				}
-				if rsaKey, ok := item.Certificate.PublicKey.(*rsa.PublicKey); ok {
-					keypair.KeyLength = fi.Int(rsaKey.N.BitLen())
+				if cert := item.Certificate; cert != nil {
+					keypair.Subject = cert.Subject.String()
+					keypair.Issuer = cert.Certificate.Issuer.String()
+					keypair.IsCA = cert.IsCA
+					{
+						t := cert.Certificate.NotBefore.UTC()
+						keypair.NotBefore = &t
+					}
+					{
+						t := cert.Certificate.NotAfter.UTC()
+						keypair.NotAfter = &t
+					}
+					{
+						var alternateNames []string
+						alternateNames = append(alternateNames, item.Certificate.Certificate.DNSNames...)
+						alternateNames = append(alternateNames, item.Certificate.Certificate.EmailAddresses...)
+						for _, ip := range item.Certificate.Certificate.IPAddresses {
+							alternateNames = append(alternateNames, ip.String())
+						}
+						sort.Strings(alternateNames)
+						keypair.AlternateNames = alternateNames
+					}
+					if rsaKey, ok := cert.PublicKey.(*rsa.PublicKey); ok {
+						keypair.KeyLength = fi.Int(rsaKey.N.BitLen())
+					}
 				}
 				items = append(items, &keypair)
 			}
@@ -196,10 +205,16 @@ func RunGetKeypairs(ctx context.Context, f commandutils.Factory, out io.Writer,
 			return ""
 		})
 		t.AddColumn("ISSUED", func(i *keypairItem) string {
-			return i.NotBefore.Local().Format("2006-01-02")
+			if t := i.NotBefore; t != nil {
+				return t.Local().Format("2006-01-02")
+			}
+			return ""
 		})
 		t.AddColumn("EXPIRES", func(i *keypairItem) string {
-			return i.NotAfter.Local().Format("2006-01-02")
+			if t := i.NotAfter; t != nil {
+				return t.Local().Format("2006-01-02")
+			}
+			return ""
 		})
 		t.AddColumn("PRIMARY", func(i *keypairItem) string {
 			if i.IsPrimary {
diff --git a/cmd/kops/promote_keypair.go b/cmd/kops/promote_keypair.go
index c8e42753f6..4dd6f109db 100644
--- a/cmd/kops/promote_keypair.go
+++ b/cmd/kops/promote_keypair.go
@@ -162,17 +162,17 @@ func promoteKeypair(out io.Writer, name string, keypairID string, keyStore fi.CA
 	}
 
 	if keypairID == "" {
-		highestTrustedId := big.NewInt(0)
+		highestCandidateId := big.NewInt(0)
 		for id, item := range keyset.Items {
-			if item.PrivateKey != nil && item.DistrustTimestamp == nil {
+			if item.PrivateKey != nil && item.DistrustTimestamp == nil && item.Certificate != nil {
 				itemId, ok := big.NewInt(0).SetString(id, 10)
-				if ok && highestTrustedId.Cmp(itemId) < 0 {
-					highestTrustedId = itemId
+				if ok && highestCandidateId.Cmp(itemId) < 0 {
+					highestCandidateId = itemId
 				}
 			}
 		}
 
-		keypairID = highestTrustedId.String()
+		keypairID = highestCandidateId.String()
 		if keypairID == keyset.Primary.Id {
 			fmt.Fprintf(out, "No %s keypair newer than current primary %s\n", name, keypairID)
 			return nil
@@ -184,6 +184,9 @@ func promoteKeypair(out io.Writer, name string, keypairID string, keyStore fi.CA
 		if item.PrivateKey == nil {
 			return fmt.Errorf("keypair has no private key")
 		}
+		if item.Certificate == nil {
+			return fmt.Errorf("keypair has no certificate")
+		}
 	} else {
 		return fmt.Errorf("keypair not found")
 	}
diff --git a/pkg/model/components/awscloudcontrollermanager.go b/pkg/model/components/awscloudcontrollermanager.go
index d90fd19ebf..68cc585216 100644
--- a/pkg/model/components/awscloudcontrollermanager.go
+++ b/pkg/model/components/awscloudcontrollermanager.go
@@ -98,7 +98,7 @@ func (b *AWSCloudControllerManagerOptionsBuilder) BuildOptions(o interface{}) er
 		case 24:
 			eccm.Image = "registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.1"
 		case 25:
-			eccm.Image = "registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1"
+			eccm.Image = "registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0"
 		default:
 			eccm.Image = "registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.1"
 		}
diff --git a/upup/pkg/fi/vfs_castore.go b/upup/pkg/fi/vfs_castore.go
index 43960d02a2..c05a562b61 100644
--- a/upup/pkg/fi/vfs_castore.go
+++ b/upup/pkg/fi/vfs_castore.go
@@ -347,6 +347,9 @@ func (c *VFSCAStore) StoreKeyset(name string, keyset *Keyset) error {
 	if keyset.Items[primaryId].PrivateKey == nil {
 		return fmt.Errorf("keyset's primary id %q must have a private key", primaryId)
 	}
+	if keyset.Items[primaryId].Certificate == nil {
+		return fmt.Errorf("keyset's primary id %q must have a certificate", primaryId)
+	}
 
 	{
 		p := c.buildPrivateKeyPoolPath(name)


Test patch:
diff --git a/tests/integration/update_cluster/karpenter/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/karpenter/data/aws_s3_object_cluster-completed.spec_content
index 2d2b7dcdad..b975638acb 100644
--- a/tests/integration/update_cluster/karpenter/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/karpenter/data/aws_s3_object_cluster-completed.spec_content
@@ -19,7 +19,7 @@ spec:
     clusterCIDR: 100.64.0.0/10
     clusterName: minimal.example.com
     configureCloudRoutes: false
-    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
     leaderElection:
       leaderElect: true
   cloudProvider: aws
diff --git a/tests/integration/update_cluster/karpenter/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content b/tests/integration/update_cluster/karpenter/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
index 1e056b95c3..603851adc8 100644
--- a/tests/integration/update_cluster/karpenter/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
+++ b/tests/integration/update_cluster/karpenter/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
@@ -48,7 +48,7 @@ spec:
           value: arn:aws-test:iam::123456789012:role/aws-cloud-controller-manager.kube-system.sa.minimal.example.com
         - name: AWS_WEB_IDENTITY_TOKEN_FILE
           value: /var/run/secrets/amazonaws.com/token
-        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
         imagePullPolicy: IfNotPresent
         name: aws-cloud-controller-manager
         resources:
diff --git a/tests/integration/update_cluster/karpenter/data/aws_s3_object_minimal.example.com-addons-bootstrap_content b/tests/integration/update_cluster/karpenter/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
index a6ceb00a47..00f8306499 100644
--- a/tests/integration/update_cluster/karpenter/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/karpenter/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
@@ -55,7 +55,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.18
     manifest: aws-cloud-controller.addons.k8s.io/k8s-1.18.yaml
-    manifestHash: 6a2b03cd482dfa68746e47da84351d9a7eae47adb1065cbe760226e0b9cb85ec
+    manifestHash: f87cf8aae411d5d07899144476f0e20f7b3341aaef0eaadfa2895a0b553096f8
     name: aws-cloud-controller.addons.k8s.io
     selector:
       k8s-addon: aws-cloud-controller.addons.k8s.io
diff --git a/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_cluster-completed.spec_content
index 793326ac8c..1060de80cc 100644
--- a/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_cluster-completed.spec_content
@@ -24,7 +24,7 @@ spec:
     clusterCIDR: 172.20.0.0/16
     clusterName: minimal.example.com
     configureCloudRoutes: false
-    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
     leaderElection:
       leaderElect: true
   cloudProvider: aws
diff --git a/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content b/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
index dc671eeab6..9a1152b36b 100644
--- a/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
+++ b/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
@@ -48,7 +48,7 @@ spec:
           value: arn:aws-test:iam::123456789012:role/aws-cloud-controller-manager.kube-system.sa.minimal.example.com
         - name: AWS_WEB_IDENTITY_TOKEN_FILE
           value: /var/run/secrets/amazonaws.com/token
-        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
         imagePullPolicy: IfNotPresent
         name: aws-cloud-controller-manager
         resources:
diff --git a/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_minimal.example.com-addons-bootstrap_content b/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
index d2b23db633..e239cacf55 100644
--- a/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/many-addons-ccm-irsa25/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
@@ -127,7 +127,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.18
     manifest: aws-cloud-controller.addons.k8s.io/k8s-1.18.yaml
-    manifestHash: b3336332893e85ed9c29fd02bc28446db6b55a6bca357a65d9b92c9a5af6f6ed
+    manifestHash: 8a1cb251b05705501c5c5adde70f20c677fc5d1a43da3de5cf6d87171bd991b1
     name: aws-cloud-controller.addons.k8s.io
     selector:
       k8s-addon: aws-cloud-controller.addons.k8s.io
diff --git a/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_cluster-completed.spec_content
index 345501c1f7..0eceeca93e 100644
--- a/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_cluster-completed.spec_content
@@ -19,7 +19,7 @@ spec:
     clusterCIDR: 100.64.0.0/10
     clusterName: minimal.example.com
     configureCloudRoutes: false
-    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
     leaderElection:
       leaderElect: true
   cloudProvider: aws
diff --git a/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content b/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
index 71aaa62b64..11c272d5df 100644
--- a/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
+++ b/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_minimal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
@@ -44,7 +44,7 @@ spec:
         env:
         - name: KUBERNETES_SERVICE_HOST
           value: 127.0.0.1
-        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
         imagePullPolicy: IfNotPresent
         name: aws-cloud-controller-manager
         resources:
diff --git a/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_minimal.example.com-addons-bootstrap_content b/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
index 64aa9918d5..b735524c10 100644
--- a/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/minimal-1.25/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
@@ -55,7 +55,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.18
     manifest: aws-cloud-controller.addons.k8s.io/k8s-1.18.yaml
-    manifestHash: 9c927fe5e4236bc4629228dfbd71848cc838edc24e675b3edd5792c309a7c3fb
+    manifestHash: 7bc69a077859866875d70015364a34c858547bd005885ad1d3eebc701945ba85
     name: aws-cloud-controller.addons.k8s.io
     selector:
       k8s-addon: aws-cloud-controller.addons.k8s.io
diff --git a/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_cluster-completed.spec_content
index 7bba0387b6..52f2f1adc5 100644
--- a/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_cluster-completed.spec_content
@@ -24,7 +24,7 @@ spec:
     clusterCIDR: ::/0
     clusterName: minimal-ipv6.example.com
     configureCloudRoutes: false
-    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
     leaderElection:
       leaderElect: true
   cloudProvider: aws
diff --git a/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content b/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
index f0b6670a98..2d8cc4e7e5 100644
--- a/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
+++ b/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
@@ -44,7 +44,7 @@ spec:
         env:
         - name: KUBERNETES_SERVICE_HOST
           value: 127.0.0.1
-        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
         imagePullPolicy: IfNotPresent
         name: aws-cloud-controller-manager
         resources:
diff --git a/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content b/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content
index 1ac6716eec..4a072d1ee0 100644
--- a/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content
@@ -104,7 +104,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.18
     manifest: aws-cloud-controller.addons.k8s.io/k8s-1.18.yaml
-    manifestHash: 4b59263d0ef1e096598f5b1bc6691a8805f998763d1be9ed0c2cd4759ee487de
+    manifestHash: 9815af144dba1b7625a1d57650dec900e0d7a6d720959e229865e0fd4f266843
     name: aws-cloud-controller.addons.k8s.io
     selector:
       k8s-addon: aws-cloud-controller.addons.k8s.io
diff --git a/tests/integration/update_cluster/privatecalico/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/privatecalico/data/aws_s3_object_cluster-completed.spec_content
index 07be90c630..5db2b439f1 100644
--- a/tests/integration/update_cluster/privatecalico/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/privatecalico/data/aws_s3_object_cluster-completed.spec_content
@@ -21,7 +21,7 @@ spec:
     clusterCIDR: 100.64.0.0/10
     clusterName: privatecalico.example.com
     configureCloudRoutes: false
-    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
     leaderElection:
       leaderElect: true
   cloudProvider: aws
diff --git a/tests/integration/update_cluster/privatecalico/data/aws_s3_object_privatecalico.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content b/tests/integration/update_cluster/privatecalico/data/aws_s3_object_privatecalico.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
index 554a9c4475..4e66e67c0a 100644
--- a/tests/integration/update_cluster/privatecalico/data/aws_s3_object_privatecalico.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
+++ b/tests/integration/update_cluster/privatecalico/data/aws_s3_object_privatecalico.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
@@ -44,7 +44,7 @@ spec:
         env:
         - name: KUBERNETES_SERVICE_HOST
           value: 127.0.0.1
-        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
         imagePullPolicy: IfNotPresent
         name: aws-cloud-controller-manager
         resources:
diff --git a/tests/integration/update_cluster/privatecalico/data/aws_s3_object_privatecalico.example.com-addons-bootstrap_content b/tests/integration/update_cluster/privatecalico/data/aws_s3_object_privatecalico.example.com-addons-bootstrap_content
index e3d26f43d4..cb3ab39bae 100644
--- a/tests/integration/update_cluster/privatecalico/data/aws_s3_object_privatecalico.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/privatecalico/data/aws_s3_object_privatecalico.example.com-addons-bootstrap_content
@@ -104,7 +104,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.18
     manifest: aws-cloud-controller.addons.k8s.io/k8s-1.18.yaml
-    manifestHash: dd0e385882ba62cde842cfd85e681ccbe2a2f6356f9807901af229bd154708d1
+    manifestHash: 9021b3edac0a7bdf0aa5c97e8cad8dc673453d05f561f1a5abbfbc1d49e65b12
     name: aws-cloud-controller.addons.k8s.io
     selector:
       k8s-addon: aws-cloud-controller.addons.k8s.io
diff --git a/tests/integration/update_cluster/privatecanal/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/privatecanal/data/aws_s3_object_cluster-completed.spec_content
index 311bdb53c7..82243ddf73 100644
--- a/tests/integration/update_cluster/privatecanal/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/privatecanal/data/aws_s3_object_cluster-completed.spec_content
@@ -21,7 +21,7 @@ spec:
     clusterCIDR: 100.64.0.0/10
     clusterName: privatecanal.example.com
     configureCloudRoutes: false
-    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
     leaderElection:
       leaderElect: true
   cloudProvider: aws
diff --git a/tests/integration/update_cluster/privatecanal/data/aws_s3_object_privatecanal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content b/tests/integration/update_cluster/privatecanal/data/aws_s3_object_privatecanal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
index b4ce862ba6..9fba4919cd 100644
--- a/tests/integration/update_cluster/privatecanal/data/aws_s3_object_privatecanal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
+++ b/tests/integration/update_cluster/privatecanal/data/aws_s3_object_privatecanal.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
@@ -44,7 +44,7 @@ spec:
         env:
         - name: KUBERNETES_SERVICE_HOST
           value: 127.0.0.1
-        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
         imagePullPolicy: IfNotPresent
         name: aws-cloud-controller-manager
         resources:
diff --git a/tests/integration/update_cluster/privatecanal/data/aws_s3_object_privatecanal.example.com-addons-bootstrap_content b/tests/integration/update_cluster/privatecanal/data/aws_s3_object_privatecanal.example.com-addons-bootstrap_content
index b7902bbada..3dc205fbae 100644
--- a/tests/integration/update_cluster/privatecanal/data/aws_s3_object_privatecanal.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/privatecanal/data/aws_s3_object_privatecanal.example.com-addons-bootstrap_content
@@ -104,7 +104,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.18
     manifest: aws-cloud-controller.addons.k8s.io/k8s-1.18.yaml
-    manifestHash: b896f7a2afd6dc74133fb0a82edbcec2ac1caca1b4d07d2ef732d027dfaddc0e
+    manifestHash: 187850d2efb243c8ec5bb4356046ee2f545016dad4783af8c14d512b47e51b15
     name: aws-cloud-controller.addons.k8s.io
     selector:
       k8s-addon: aws-cloud-controller.addons.k8s.io
diff --git a/tests/integration/update_cluster/privateflannel/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/privateflannel/data/aws_s3_object_cluster-completed.spec_content
index 75533cd90e..8808976b36 100644
--- a/tests/integration/update_cluster/privateflannel/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/privateflannel/data/aws_s3_object_cluster-completed.spec_content
@@ -21,7 +21,7 @@ spec:
     clusterCIDR: 100.64.0.0/10
     clusterName: privateflannel.example.com
     configureCloudRoutes: false
-    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+    image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
     leaderElection:
       leaderElect: true
   cloudProvider: aws
diff --git a/tests/integration/update_cluster/privateflannel/data/aws_s3_object_privateflannel.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content b/tests/integration/update_cluster/privateflannel/data/aws_s3_object_privateflannel.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
index 84532e69fd..581e2fb8ad 100644
--- a/tests/integration/update_cluster/privateflannel/data/aws_s3_object_privateflannel.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
+++ b/tests/integration/update_cluster/privateflannel/data/aws_s3_object_privateflannel.example.com-addons-aws-cloud-controller.addons.k8s.io-k8s-1.18_content
@@ -44,7 +44,7 @@ spec:
         env:
         - name: KUBERNETES_SERVICE_HOST
           value: 127.0.0.1
-        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1
+        image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.0
         imagePullPolicy: IfNotPresent
         name: aws-cloud-controller-manager
         resources:
diff --git a/tests/integration/update_cluster/privateflannel/data/aws_s3_object_privateflannel.example.com-addons-bootstrap_content b/tests/integration/update_cluster/privateflannel/data/aws_s3_object_privateflannel.example.com-addons-bootstrap_content
index c555fee012..86cab41ee1 100644
--- a/tests/integration/update_cluster/privateflannel/data/aws_s3_object_privateflannel.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/privateflannel/data/aws_s3_object_privateflannel.example.com-addons-bootstrap_content
@@ -100,7 +100,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.18
     manifest: aws-cloud-controller.addons.k8s.io/k8s-1.18.yaml
-    manifestHash: e4ec807c43b03174207c88173ba01293fd2d2884c51612a1adefd3f8b2a7580b
+    manifestHash: 2991b9b59bbca24f083a9e6106120a0954d25d47b747c4b785a4f4c341440203
     name: aws-cloud-controller.addons.k8s.io
     selector:
       k8s-addon: aws-cloud-controller.addons.k8s.io