Repository: kubernetes/kops (go)

Issue:
kops not able to create arm64 type Kubernetes cluster
/kind bug

**1. kops version: v1.25.0**

**2. kubectl version: v1.25.2**

**3. cloud provider using: aws**

**4. Commands ran:**
```
export KOPS_STATE_STORE=s3://kops-state-store1
export AWS_ACCESS_KEY_ID=<your-aws-access-key-id>
export AWS_SECRET_ACCESS_KEY=<your-aws-secret-access-key>
export KOPS_CLUSTER_NAME=sample-cluster.k8s.local

kops create cluster --name=$KOPS_CLUSTER_NAME \
    --node-count=1 \
    --node-size=m6g.xlarge \
    --cloud aws \
    --zones us-west-2a \
    --master-size m6g.xlarge \
    --topology private \
    --networking amazon-vpc-routed-eni
```

**5. Output:**
```
I0927 17:12:35.211369   13029 new_cluster.go:1277]  Cloud Provider ID = aws
I0927 17:12:37.318956   13029 subnets.go:185] Assigned CIDR 172.20.32.0/19 to subnet us-west-2a
I0927 17:12:37.319437   13029 subnets.go:199] Assigned CIDR 172.20.0.0/22 to subnet utility-us-west-2a
Previewing changes that will be made:

Error: master-us-west-2a.spec.machineType: Invalid value: "m6g.xlarge": machine type architecture does not match image architecture: "arm64" - "x86_64"
```
and exited after this.

**6. With the earlier version of the kops (v1.24.3), it used to print the list of resources that would be created and other suggestions and later on running the below command would actually create the cluster:**
```
kops update cluster --name ${KOPS_CLUSTER_NAME} --yes --admin
```

**7. Cluster manifest:**

```
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2022-09-27T11:42:47Z"
  name: sample-cluster.k8s.local
spec:
  api:
    loadBalancer:
      class: Classic
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://kops-state-store1/sample-cluster.k8s.local
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-west-2a
      name: a
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-west-2a
      name: a
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  - ::/0
  kubernetesVersion: 1.25.2
  masterPublicName: api.sample-cluster.k8s.local
  networkCIDR: 172.20.0.0/16
  networking:
    amazonvpc: {}
  nonMasqueradeCIDR: 172.20.0.0/16
  sshAccess:
  - 0.0.0.0/0
  - ::/0
  subnets:
  - cidr: 172.20.32.0/19
    name: us-west-2a
    type: Private
    zone: us-west-2a
  - cidr: 172.20.0.0/22
    name: utility-us-west-2a
    type: Utility
    zone: us-west-2a
  topology:
    dns:
      type: Public
    masters: private
    nodes: private

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2022-09-27T11:42:49Z"
  labels:
    kops.k8s.io/cluster: sample-cluster.k8s.local
  name: master-us-west-2a
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20220810
  instanceMetadata:
    httpPutResponseHopLimit: 3
    httpTokens: required
  machineType: m6g.xlarge
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - us-west-2a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2022-09-27T11:42:50Z"
  labels:
    kops.k8s.io/cluster: sample-cluster.k8s.local
  name: nodes-us-west-2a
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20220810
  instanceMetadata:
    httpPutResponseHopLimit: 1
    httpTokens: required
  machineType: m6g.xlarge
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - us-west-2a
```

**8. kops get --name ${KOPS_CLUSTER_NAME} -o yaml -v 10**

```
I0927 17:32:23.305100   14461 factory.go:83] state store s3://kops-state-store1
I0927 17:32:23.306588   14461 s3context.go:345] GOOS="darwin", assuming not running on EC2
I0927 17:32:23.306596   14461 s3context.go:175] defaulting region to "us-east-1"
I0927 17:32:24.303517   14461 s3context.go:215] found bucket in region "us-west-2"
I0927 17:32:24.305024   14461 s3fs.go:329] Reading file "s3://kops-state-store1/sample-cluster.k8s.local/config"
I0927 17:32:25.528165   14461 s3fs.go:366] Listing objects in S3 bucket "kops-state-store1" with prefix "sample-cluster.k8s.local/instancegroup/"
I0927 17:32:25.911170   14461 s3fs.go:394] Listed files in s3://kops-state-store1/sample-cluster.k8s.local/instancegroup: [s3://kops-state-store1/sample-cluster.k8s.local/instancegroup/master-us-west-2a s3://kops-state-store1/sample-cluster.k8s.local/instancegroup/nodes-us-west-2a]
I0927 17:32:25.911281   14461 s3fs.go:329] Reading file "s3://kops-state-store1/sample-cluster.k8s.local/instancegroup/master-us-west-2a"
I0927 17:32:26.220812   14461 s3fs.go:329] Reading file "s3://kops-state-store1/sample-cluster.k8s.local/instancegroup/nodes-us-west-2a"
I0927 17:32:26.561562   14461 s3fs.go:329] Reading file "s3://kops-state-store1/sample-cluster.k8s.local/clusteraddons/default"
```
rest remains same as previous output

**9. This is working fine with kops version v1.24.3. Also I tried running on my mac as well as on ubuntu20, but on both places I am seeing the same error message.**


Pull request:
Add node/master size from create cluster into IG spec
Fixes #14346

Patch:
diff --git a/cmd/kops/create_cluster.go b/cmd/kops/create_cluster.go
index 071eab1eb7..e42db1903d 100644
--- a/cmd/kops/create_cluster.go
+++ b/cmd/kops/create_cluster.go
@@ -63,8 +63,6 @@ type CreateClusterOptions struct {
 	cloudup.NewClusterOptions
 	Yes                  bool
 	Target               string
-	NodeSize             string
-	MasterSize           string
 	MasterVolumeSize     int32
 	NodeVolumeSize       int32
 	ContainerRuntime     string
@@ -544,12 +542,6 @@ func RunCreateCluster(ctx context.Context, f *util.Factory, out io.Writer, c *Cr
 		cluster.Spec.CloudLabels = cloudLabels
 	}
 
-	if c.NodeSize != "" {
-		for _, group := range nodes {
-			group.Spec.MachineType = c.NodeSize
-		}
-	}
-
 	if c.AssociatePublicIP != nil {
 		for _, group := range instanceGroups {
 			group.Spec.AssociatePublicIP = c.AssociatePublicIP
@@ -580,12 +572,6 @@ func RunCreateCluster(ctx context.Context, f *util.Factory, out io.Writer, c *Cr
 		}
 	}
 
-	if c.MasterSize != "" {
-		for _, group := range masters {
-			group.Spec.MachineType = c.MasterSize
-		}
-	}
-
 	if c.MasterVolumeSize != 0 {
 		for _, group := range masters {
 			group.Spec.RootVolumeSize = fi.Int32(c.MasterVolumeSize)
diff --git a/upup/pkg/fi/cloudup/awsup/mock_aws_cloud.go b/upup/pkg/fi/cloudup/awsup/mock_aws_cloud.go
index 240fb94976..a0a2ceab54 100644
--- a/upup/pkg/fi/cloudup/awsup/mock_aws_cloud.go
+++ b/upup/pkg/fi/cloudup/awsup/mock_aws_cloud.go
@@ -358,7 +358,7 @@ func (c *MockAWSCloud) DescribeInstanceType(instanceType string) (*ec2.InstanceT
 				aws.String(ec2.ArchitectureTypeX8664),
 			},
 		}
-	case "a1.large":
+	case "a1.large", "m6g.xlarge":
 		info.ProcessorInfo = &ec2.ProcessorInfo{
 			SupportedArchitectures: []*string{
 				aws.String(ec2.ArchitectureTypeArm64),
diff --git a/upup/pkg/fi/cloudup/new_cluster.go b/upup/pkg/fi/cloudup/new_cluster.go
index 516dea7d47..12a4a1ae38 100644
--- a/upup/pkg/fi/cloudup/new_cluster.go
+++ b/upup/pkg/fi/cloudup/new_cluster.go
@@ -157,6 +157,8 @@ type NewClusterOptions struct {
 	Image       string
 	NodeImage   string
 	MasterImage string
+	MasterSize  string
+	NodeSize    string
 }
 
 func (o *NewClusterOptions) InitDefaults() {
@@ -856,6 +858,7 @@ func setupMasters(opt *NewClusterOptions, cluster *api.Cluster, zoneToSubnetMap
 				}
 			}
 
+			g.Spec.MachineType = opt.MasterSize
 			g.Spec.Image = opt.MasterImage
 
 			masters = append(masters, g)
@@ -983,6 +986,7 @@ func setupNodes(opt *NewClusterOptions, cluster *api.Cluster, zoneToSubnetMap ma
 			}
 		}
 
+		g.Spec.MachineType = opt.NodeSize
 		g.Spec.Image = opt.NodeImage
 
 		nodes = append(nodes, g)
@@ -1461,11 +1465,16 @@ func MachineArchitecture(cloud fi.Cloud, machineType string) (architectures.Arch
 		return architectures.ArchitectureAmd64, nil
 	}
 
+	// Some calls only have AWS initialised at this point and in other cases pass in nil as cloud.
+	if cloud == nil {
+		return architectures.ArchitectureAmd64, nil
+	}
+
 	switch cloud.ProviderID() {
 	case kopsapi.CloudProviderAWS:
 		info, err := cloud.(awsup.AWSCloud).DescribeInstanceType(machineType)
 		if err != nil {
-			return "", fmt.Errorf("error finding instance info for instance type %q: %v", machineType, err)
+			return "", fmt.Errorf("error finding instance info for instance type %q: %w", machineType, err)
 		}
 		if info.ProcessorInfo == nil || len(info.ProcessorInfo.SupportedArchitectures) == 0 {
 			return "", fmt.Errorf("error finding architecture info for instance type %q", machineType)


Test patch:
diff --git a/cmd/kops/create_cluster_integration_test.go b/cmd/kops/create_cluster_integration_test.go
index 16d3fbcfb8..b20f76b1e7 100644
--- a/cmd/kops/create_cluster_integration_test.go
+++ b/cmd/kops/create_cluster_integration_test.go
@@ -53,6 +53,7 @@ func TestCreateClusterMinimal(t *testing.T) {
 	runCreateClusterIntegrationTest(t, "../../tests/integration/create_cluster/minimal-1.24", "v1alpha2")
 	runCreateClusterIntegrationTest(t, "../../tests/integration/create_cluster/minimal-1.25", "v1alpha2")
 	runCreateClusterIntegrationTest(t, "../../tests/integration/create_cluster/minimal-1.26", "v1alpha2")
+	runCreateClusterIntegrationTest(t, "../../tests/integration/create_cluster/minimal-1.26-arm64", "v1alpha2")
 }
 
 // TestCreateClusterHetzner runs kops create cluster minimal.k8s.local --zones fsn1
diff --git a/tests/integration/create_cluster/minimal-1.26-arm64/expected-v1alpha2.yaml b/tests/integration/create_cluster/minimal-1.26-arm64/expected-v1alpha2.yaml
new file mode 100644
index 0000000000..ea3ba143a5
--- /dev/null
+++ b/tests/integration/create_cluster/minimal-1.26-arm64/expected-v1alpha2.yaml
@@ -0,0 +1,97 @@
+apiVersion: kops.k8s.io/v1alpha2
+kind: Cluster
+metadata:
+  creationTimestamp: "2017-01-01T00:00:00Z"
+  name: minimal.example.com
+spec:
+  api:
+    dns: {}
+  authorization:
+    rbac: {}
+  channel: stable
+  cloudProvider: aws
+  configBase: memfs://tests/minimal.example.com
+  etcdClusters:
+  - cpuRequest: 200m
+    etcdMembers:
+    - encryptedVolume: true
+      instanceGroup: master-us-test-1a
+      name: a
+    memoryRequest: 100Mi
+    name: main
+  - cpuRequest: 100m
+    etcdMembers:
+    - encryptedVolume: true
+      instanceGroup: master-us-test-1a
+      name: a
+    memoryRequest: 100Mi
+    name: events
+  iam:
+    allowContainerRegistry: true
+    legacy: false
+  kubelet:
+    anonymousAuth: false
+  kubernetesApiAccess:
+  - 0.0.0.0/0
+  - ::/0
+  kubernetesVersion: v1.26.0
+  masterPublicName: api.minimal.example.com
+  networkCIDR: 172.20.0.0/16
+  networking:
+    cni: {}
+  nonMasqueradeCIDR: 100.64.0.0/10
+  sshAccess:
+  - 0.0.0.0/0
+  - ::/0
+  subnets:
+  - cidr: 172.20.32.0/19
+    name: us-test-1a
+    type: Public
+    zone: us-test-1a
+  topology:
+    dns:
+      type: Public
+    masters: public
+    nodes: public
+
+---
+
+apiVersion: kops.k8s.io/v1alpha2
+kind: InstanceGroup
+metadata:
+  creationTimestamp: "2017-01-01T00:00:00Z"
+  labels:
+    kops.k8s.io/cluster: minimal.example.com
+  name: master-us-test-1a
+spec:
+  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-20220924
+  instanceMetadata:
+    httpPutResponseHopLimit: 3
+    httpTokens: required
+  machineType: m6g.xlarge
+  maxSize: 1
+  minSize: 1
+  role: Master
+  subnets:
+  - us-test-1a
+
+---
+
+apiVersion: kops.k8s.io/v1alpha2
+kind: InstanceGroup
+metadata:
+  creationTimestamp: "2017-01-01T00:00:00Z"
+  labels:
+    kops.k8s.io/cluster: minimal.example.com
+  name: nodes-us-test-1a
+spec:
+  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-20220924
+  instanceMetadata:
+    httpPutResponseHopLimit: 1
+    httpTokens: required
+  machineType: m6g.xlarge
+  maxSize: 1
+  minSize: 1
+  role: Node
+  subnets:
+  - us-test-1a
diff --git a/tests/integration/create_cluster/minimal-1.26-arm64/options.yaml b/tests/integration/create_cluster/minimal-1.26-arm64/options.yaml
new file mode 100644
index 0000000000..d4efb857ec
--- /dev/null
+++ b/tests/integration/create_cluster/minimal-1.26-arm64/options.yaml
@@ -0,0 +1,8 @@
+ClusterName: minimal.example.com
+Zones:
+- us-test-1a
+CloudProvider: aws
+Networking: cni
+KubernetesVersion: v1.26.0
+NodeSize: m6g.xlarge
+MasterSize: m6g.xlarge
\ No newline at end of file
diff --git a/tests/integration/create_cluster/private_gce/expected-v1alpha2.yaml b/tests/integration/create_cluster/private_gce/expected-v1alpha2.yaml
index 0b918c1138..8b35a7d3b4 100644
--- a/tests/integration/create_cluster/private_gce/expected-v1alpha2.yaml
+++ b/tests/integration/create_cluster/private_gce/expected-v1alpha2.yaml
@@ -95,7 +95,7 @@ spec:
   - sg-exampleid3
   - sg-exampleid4
   image: ubuntu-os-cloud/ubuntu-2004-focal-v20220927
-  machineType: e2-medium
+  machineType: e2-standard-2
   maxSize: 1
   minSize: 1
   role: Master
diff --git a/tests/integration/create_cluster/private_gce/options.yaml b/tests/integration/create_cluster/private_gce/options.yaml
index 2ce58af83b..00cc1d34cc 100644
--- a/tests/integration/create_cluster/private_gce/options.yaml
+++ b/tests/integration/create_cluster/private_gce/options.yaml
@@ -15,3 +15,4 @@ KubernetesVersion: v1.21.0
 cloudLabels: "Owner=John Doe,dn=\"cn=John Doe: dc=example dc=com\", foo/bar=fib+baz"
 Project: testproject
 GCEServiceAccount: test-account@testproject.iam.gserviceaccount.com
+MasterSize: e2-standard-2
\ No newline at end of file