Repository: kubernetes/kops (go)

Issue:
Unable to recover from failed cluster deletion
**1. What `kops` version are you running? The command `kops version`, will display
 this information.**

`Version 1.15.0 (git-ecffaff)`
(this is a private fork)

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**

1.15.9

**3. What cloud provider are you using?**

AWS

**4. What commands did you run?  What is the simplest way to reproduce this issue?**

`kops delete cluster --name REDACTED --state REDACTED --yes`

**5. What happened after the commands executed?**

The first time, it failed with:

```
subnet:subnet-08f4c2ba4b71c9492	still has dependencies, will retry
subnet:subnet-015189c75e105f5cc	still has dependencies, will retry
subnet:subnet-0655ae5887a06b155	still has dependencies, will retry
Not all resources deleted; waiting before reattempting deletion
	route-table:rtb-0c47d709931c7f1bd
	route-table:rtb-07f3c8a3d94b134c8
	subnet:subnet-08f4c2ba4b71c9492
	route-table:rtb-09b6a1f0eeb6ae5fe
	subnet:subnet-015189c75e105f5cc
	subnet:subnet-0655ae5887a06b155
subnet:subnet-0655ae5887a06b155	still has dependencies, will retry
subnet:subnet-015189c75e105f5cc	still has dependencies, will retry
subnet:subnet-08f4c2ba4b71c9492	still has dependencies, will retry
Not all resources deleted; waiting before reattempting deletion
	subnet:subnet-015189c75e105f5cc
	route-table:rtb-09b6a1f0eeb6ae5fe
	subnet:subnet-0655ae5887a06b155
	route-table:rtb-0c47d709931c7f1bd
	route-table:rtb-07f3c8a3d94b134c8
	subnet:subnet-08f4c2ba4b71c9492

not making progress deleting resources; giving up
```

The subnets couldn't be deleted due to an apparent transient AWS issue: one of the deleted ELBs was still holding IPs from those subnets. After a weekend, the subnets could be deleted.

After deleting the subnets, reattempting a `kops delete cluster ... --yes` resulted in:

```
error from DescribeNatGateways: NatGatewayNotFound: NAT gateway nat-0a8aa3ba56bb61389 was not found
	status code: 400, request id: 9be9785e-09e2-4e71-9466-2eeb29d5790a
```

**6. What did you expect to happen?**

The second `kops delete cluster` should have continued to delete the cluster and removed the cluster from the state store.

**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
  Paste the logs into this report, or in a gist and provide the gist link here.**

**9. Anything else do we need to know?**

I believe `aws.FindNatGateways()` needs to tolerate this particular error from `c.EC2().DescribeNatGateways()`, treating it as having returned zero NAT gateways.


Pull request:
Treat NatGatewaysNotFound error as already-deleted
Fixes #8424 


Patch:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index c0a94b83f9..3b89a5b7b7 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -2,8 +2,11 @@
 name: build
 
 'on':
-  - push
-  - pull_request
+  push:
+    branches:
+      - master
+  pull_request:
+    branches: ["*"]
 
 env:
   GOPROXY: https://proxy.golang.org
diff --git a/cmd/kops/create_cluster.go b/cmd/kops/create_cluster.go
index 78a698b2ab..803914a3fe 100644
--- a/cmd/kops/create_cluster.go
+++ b/cmd/kops/create_cluster.go
@@ -986,9 +986,6 @@ func RunCreateCluster(ctx context.Context, f *util.Factory, out io.Writer, c *Cr
 		cluster.Spec.Project = c.Project
 	}
 	if api.CloudProviderID(cluster.Spec.CloudProvider) == api.CloudProviderGCE {
-		if cluster.Spec.CloudConfig == nil {
-			cluster.Spec.CloudConfig = &api.CloudConfiguration{}
-		}
 		if cluster.Spec.Project == "" {
 			project, err := gce.DefaultProject()
 			if err != nil {
@@ -1000,13 +997,16 @@ func RunCreateCluster(ctx context.Context, f *util.Factory, out io.Writer, c *Cr
 			}
 			cluster.Spec.Project = project
 		}
-		if c.GCEServiceAccount != "" {
-			klog.Infof("VMs will be configured to use specified Service Account: %v", c.GCEServiceAccount)
-			cluster.Spec.CloudConfig.GCEServiceAccount = c.GCEServiceAccount
-		} else {
+	}
+
+	if c.GCEServiceAccount != "" {
+		klog.Infof("VMs will be configured to use specified Service Account: %v", c.GCEServiceAccount)
+		cluster.Spec.GCEServiceAccount = c.GCEServiceAccount
+	} else {
+		if api.CloudProviderID(cluster.Spec.CloudProvider) == api.CloudProviderGCE {
 			klog.Warning("VMs will be configured to use the GCE default compute Service Account! This is an anti-pattern")
 			klog.Warning("Use a pre-create Service Account with the flag: --gce-service-account=account@projectname.iam.gserviceaccount.com")
-			cluster.Spec.CloudConfig.GCEServiceAccount = "default"
+			cluster.Spec.GCEServiceAccount = "default"
 		}
 	}
 
diff --git a/docs/operations/service_account_token_volumes.md b/docs/operations/service_account_token_volumes.md
index 57257f1454..5d6653cba1 100644
--- a/docs/operations/service_account_token_volumes.md
+++ b/docs/operations/service_account_token_volumes.md
@@ -1,7 +1,7 @@
-Some services, such as Istio and Envoy's Secret Discovery Service (SDS), take advantage of a new feature in Kubernetes 1.12+, [Service Account Token Volume Projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection).
+Some services, such as istio and Envoy's Secrect Discovery Service (SDS), take advantage of a new feature in kubernetes 1.13+, [Service Account Token Volume Projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection).
 
 
-1. In order to enable this feature for Kubernetes 1.12+, add the following config to your cluster spec:
+1. In order to enable this feature for kubernetes 1.12+, add the following config to your cluster spec:
 
 ```
     kubeAPIServer:
diff --git a/k8s/crds/kops.k8s.io_clusters.yaml b/k8s/crds/kops.k8s.io_clusters.yaml
index 001d8bbea8..c7e294e9c7 100644
--- a/k8s/crds/kops.k8s.io_clusters.yaml
+++ b/k8s/crds/kops.k8s.io_clusters.yaml
@@ -195,10 +195,6 @@ spec:
                     type: boolean
                   elbSecurityGroup:
                     type: string
-                  gceServiceAccount:
-                    description: GCEServiceAccount specifies the service account with
-                      which the GCE VM runs
-                    type: string
                   multizone:
                     description: GCE cloud-config options
                     type: boolean
@@ -777,6 +773,10 @@ spec:
                       type: array
                   type: object
                 type: array
+              gceServiceAccount:
+                description: GCEServiceAccount specifies the service account with
+                  which the GCE VM runs
+                type: string
               gossipConfig:
                 description: GossipConfig for the cluster assuming the use of gossip
                   DNS
diff --git a/nodeup/pkg/model/kubelet.go b/nodeup/pkg/model/kubelet.go
index 488dd411e2..fb41444dc8 100644
--- a/nodeup/pkg/model/kubelet.go
+++ b/nodeup/pkg/model/kubelet.go
@@ -141,11 +141,8 @@ func (b *KubeletBuilder) Build(c *fi.ModelBuilderContext) error {
 		})
 	}
 
-	if b.Distribution == distros.DistributionCoreOS {
-		// CoreOS does not ship with socat or conntrack. Install our own (statically linked) version
-		if err := b.addStaticUtils(c); err != nil {
-			return err
-		}
+	if err := b.addStaticUtils(c); err != nil {
+		return err
 	}
 
 	if err := b.addContainerizedMounter(c); err != nil {
@@ -281,6 +278,10 @@ func (b *KubeletBuilder) buildSystemdService() *nodetasks.Service {
 		// We add /opt/kubernetes/bin for our utilities (socat, conntrack)
 		manifest.Set("Service", "Environment", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/kubernetes/bin")
 	}
+	if b.Distribution == distros.DistributionFlatcar {
+		// We add /opt/kubernetes/bin for our utilities (conntrack)
+		manifest.Set("Service", "Environment", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/kubernetes/bin")
+	}
 	manifest.Set("Service", "EnvironmentFile", "/etc/sysconfig/kubelet")
 
 	// @check if we are using bootstrap tokens and file checker
@@ -332,24 +333,52 @@ func (b *KubeletBuilder) buildKubeletConfig() (*kops.KubeletConfigSpec, error) {
 }
 
 func (b *KubeletBuilder) addStaticUtils(c *fi.ModelBuilderContext) error {
-	for _, binary := range []string{"socat", "conntrack"} {
-		assetName := binary
-		assetPath := ""
-		asset, err := b.Assets.Find(assetName, assetPath)
-		if err != nil {
-			return fmt.Errorf("error trying to locate asset %q: %v", assetName, err)
-		}
-		if asset == nil {
-			return fmt.Errorf("unable to locate asset %q", assetName)
+	if b.Distribution == distros.DistributionCoreOS {
+		// CoreOS does not ship with socat or conntrack.  Install our own (statically linked) version
+		// TODO: Extract to common function?
+		for _, binary := range []string{"socat", "conntrack"} {
+			assetName := binary
+			assetPath := ""
+			asset, err := b.Assets.Find(assetName, assetPath)
+			if err != nil {
+				return fmt.Errorf("error trying to locate asset %q: %v", assetName, err)
+			}
+			if asset == nil {
+				return fmt.Errorf("unable to locate asset %q", assetName)
+			}
+
+			t := &nodetasks.File{
+				Path:     "/opt/kubernetes/bin/" + binary,
+				Contents: asset,
+				Type:     nodetasks.FileType_File,
+				Mode:     s("0755"),
+			}
+			c.AddTask(t)
 		}
+	}
 
-		t := &nodetasks.File{
-			Path:     "/opt/kubernetes/bin/" + binary,
-			Contents: asset,
-			Type:     nodetasks.FileType_File,
-			Mode:     s("0755"),
+	if b.Distribution == distros.DistributionFlatcar {
+		// Flatcar does not ship with conntrack.  Install our own (statically linked) version
+		// TODO: Extract to common function?
+		for _, binary := range []string{"conntrack"} {
+			assetName := binary
+			assetPath := ""
+			asset, err := b.Assets.Find(assetName, assetPath)
+			if err != nil {
+				return fmt.Errorf("error trying to locate asset %q: %v", assetName, err)
+			}
+			if asset == nil {
+				return fmt.Errorf("unable to locate asset %q", assetName)
+			}
+
+			t := &nodetasks.File{
+				Path:     "/opt/kubernetes/bin/" + binary,
+				Contents: asset,
+				Type:     nodetasks.FileType_File,
+				Mode:     s("0755"),
+			}
+			c.AddTask(t)
 		}
-		c.AddTask(t)
 	}
 
 	return nil
diff --git a/pkg/apis/kops/cluster.go b/pkg/apis/kops/cluster.go
index 1e5b5eec45..05a5fcd41e 100644
--- a/pkg/apis/kops/cluster.go
+++ b/pkg/apis/kops/cluster.go
@@ -191,6 +191,8 @@ type ClusterSpec struct {
 	SysctlParameters []string `json:"sysctlParameters,omitempty"`
 	// RollingUpdate defines the default rolling-update settings for instance groups
 	RollingUpdate *RollingUpdate `json:"rollingUpdate,omitempty"`
+	// GCEServiceAccount specifies the service account with which the GCE VM runs
+	GCEServiceAccount string `json:"gceServiceAccount,omitempty"`
 }
 
 // NodeAuthorizationSpec is used to node authorization
diff --git a/pkg/apis/kops/componentconfig.go b/pkg/apis/kops/componentconfig.go
index 941cb40a7e..820204f9dd 100644
--- a/pkg/apis/kops/componentconfig.go
+++ b/pkg/apis/kops/componentconfig.go
@@ -709,8 +709,6 @@ type CloudConfiguration struct {
 	Multizone          *bool   `json:"multizone,omitempty"`
 	NodeTags           *string `json:"nodeTags,omitempty"`
 	NodeInstancePrefix *string `json:"nodeInstancePrefix,omitempty"`
-	// GCEServiceAccount specifies the service account with which the GCE VM runs
-	GCEServiceAccount string `json:"gceServiceAccount,omitempty"`
 	// AWS cloud-config options
 	DisableSecurityGroupIngress *bool   `json:"disableSecurityGroupIngress,omitempty"`
 	ElbSecurityGroup            *string `json:"elbSecurityGroup,omitempty"`
diff --git a/pkg/apis/kops/v1alpha2/cluster.go b/pkg/apis/kops/v1alpha2/cluster.go
index ed4bb70d31..53277952ea 100644
--- a/pkg/apis/kops/v1alpha2/cluster.go
+++ b/pkg/apis/kops/v1alpha2/cluster.go
@@ -189,6 +189,8 @@ type ClusterSpec struct {
 	SysctlParameters []string `json:"sysctlParameters,omitempty"`
 	// RollingUpdate defines the default rolling-update settings for instance groups
 	RollingUpdate *RollingUpdate `json:"rollingUpdate,omitempty"`
+	// GCEServiceAccount specifies the service account with which the GCE VM runs
+	GCEServiceAccount string `json:"gceServiceAccount,omitempty"`
 }
 
 // NodeAuthorizationSpec is used to node authorization
diff --git a/pkg/apis/kops/v1alpha2/componentconfig.go b/pkg/apis/kops/v1alpha2/componentconfig.go
index c6e2ebb5e0..2012b2e94d 100644
--- a/pkg/apis/kops/v1alpha2/componentconfig.go
+++ b/pkg/apis/kops/v1alpha2/componentconfig.go
@@ -710,8 +710,6 @@ type CloudConfiguration struct {
 	Multizone          *bool   `json:"multizone,omitempty"`
 	NodeTags           *string `json:"nodeTags,omitempty"`
 	NodeInstancePrefix *string `json:"nodeInstancePrefix,omitempty"`
-	// GCEServiceAccount specifies the service account with which the GCE VM runs
-	GCEServiceAccount string `json:"gceServiceAccount,omitempty"`
 	// AWS cloud-config options
 	DisableSecurityGroupIngress *bool   `json:"disableSecurityGroupIngress,omitempty"`
 	ElbSecurityGroup            *string `json:"elbSecurityGroup,omitempty"`
diff --git a/pkg/apis/kops/v1alpha2/zz_generated.conversion.go b/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
index a8d99cc747..c65963ab81 100644
--- a/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
+++ b/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
@@ -1558,7 +1558,6 @@ func autoConvert_v1alpha2_CloudConfiguration_To_kops_CloudConfiguration(in *Clou
 	out.Multizone = in.Multizone
 	out.NodeTags = in.NodeTags
 	out.NodeInstancePrefix = in.NodeInstancePrefix
-	out.GCEServiceAccount = in.GCEServiceAccount
 	out.DisableSecurityGroupIngress = in.DisableSecurityGroupIngress
 	out.ElbSecurityGroup = in.ElbSecurityGroup
 	out.VSphereUsername = in.VSphereUsername
@@ -1591,7 +1590,6 @@ func autoConvert_kops_CloudConfiguration_To_v1alpha2_CloudConfiguration(in *kops
 	out.Multizone = in.Multizone
 	out.NodeTags = in.NodeTags
 	out.NodeInstancePrefix = in.NodeInstancePrefix
-	out.GCEServiceAccount = in.GCEServiceAccount
 	out.DisableSecurityGroupIngress = in.DisableSecurityGroupIngress
 	out.ElbSecurityGroup = in.ElbSecurityGroup
 	out.VSphereUsername = in.VSphereUsername
@@ -2059,6 +2057,7 @@ func autoConvert_v1alpha2_ClusterSpec_To_kops_ClusterSpec(in *ClusterSpec, out *
 	} else {
 		out.RollingUpdate = nil
 	}
+	out.GCEServiceAccount = in.GCEServiceAccount
 	return nil
 }
 
@@ -2382,6 +2381,7 @@ func autoConvert_kops_ClusterSpec_To_v1alpha2_ClusterSpec(in *kops.ClusterSpec,
 	} else {
 		out.RollingUpdate = nil
 	}
+	out.GCEServiceAccount = in.GCEServiceAccount
 	return nil
 }
 
diff --git a/pkg/model/gcemodel/autoscalinggroup.go b/pkg/model/gcemodel/autoscalinggroup.go
index 990015736f..b36e745ce8 100644
--- a/pkg/model/gcemodel/autoscalinggroup.go
+++ b/pkg/model/gcemodel/autoscalinggroup.go
@@ -139,17 +139,17 @@ func (b *AutoscalingGroupModelBuilder) Build(c *fi.ModelBuilderContext) error {
 				t.CanIPForward = fi.Bool(true)
 			}
 
-			if b.Cluster.Spec.CloudConfig.GCEServiceAccount != "" {
-				klog.Infof("VMs using Service Account: %v", b.Cluster.Spec.CloudConfig.GCEServiceAccount)
+			if b.Cluster.Spec.GCEServiceAccount != "" {
+				klog.Infof("VMs using Service Account: %v", b.Cluster.Spec.GCEServiceAccount)
 				// b.Cluster.Spec.GCEServiceAccount = c.GCEServiceAccount
 			} else {
 				klog.Warning("VMs will be configured to use the GCE default compute Service Account! This is an anti-pattern")
 				klog.Warning("Use a pre-created Service Account with the flag: --gce-service-account=account@projectname.iam.gserviceaccount.com")
-				b.Cluster.Spec.CloudConfig.GCEServiceAccount = "default"
+				b.Cluster.Spec.GCEServiceAccount = "default"
 			}
 
-			klog.Infof("gsa: %v", b.Cluster.Spec.CloudConfig.GCEServiceAccount)
-			t.ServiceAccounts = []string{b.Cluster.Spec.CloudConfig.GCEServiceAccount}
+			klog.Infof("gsa: %v", b.Cluster.Spec.GCEServiceAccount)
+			t.ServiceAccounts = []string{b.Cluster.Spec.GCEServiceAccount}
 			//labels, err := b.CloudTagsForInstanceGroup(ig)
 			//if err != nil {
 			//	return fmt.Errorf("error building cloud tags: %v", err)
diff --git a/pkg/resources/aws/aws.go b/pkg/resources/aws/aws.go
index 7b7f002e3d..edf1b9b2df 100644
--- a/pkg/resources/aws/aws.go
+++ b/pkg/resources/aws/aws.go
@@ -1317,13 +1317,16 @@ func FindNatGateways(cloud fi.Cloud, routeTables map[string]*resources.Resource,
 	}
 
 	var resourceTrackers []*resources.Resource
-	if len(natGatewayIds) != 0 {
-		request := &ec2.DescribeNatGatewaysInput{}
-		for natGatewayId := range natGatewayIds {
-			request.NatGatewayIds = append(request.NatGatewayIds, aws.String(natGatewayId))
+	for natGatewayId := range natGatewayIds {
+		request := &ec2.DescribeNatGatewaysInput{
+			NatGatewayIds: []*string{aws.String(natGatewayId)},
 		}
 		response, err := c.EC2().DescribeNatGateways(request)
 		if err != nil {
+			if awsup.AWSErrorCode(err) == "NatGatewayNotFound" {
+				klog.V(2).Infof("Got NatGatewayNotFound describing NatGateway %s; will treat as already-deleted", natGatewayId)
+				continue
+			}
 			return nil, fmt.Errorf("error from DescribeNatGateways: %v", err)
 		}
 
diff --git a/pkg/validation/validate_cluster.go b/pkg/validation/validate_cluster.go
index 58ac90df61..17ad4bd5ef 100644
--- a/pkg/validation/validate_cluster.go
+++ b/pkg/validation/validate_cluster.go
@@ -166,13 +166,13 @@ func (v *clusterValidatorImpl) Validate() (*ValidationCluster, error) {
 	if err != nil {
 		return nil, err
 	}
-	readyNodes := validation.validateNodes(cloudGroups)
+	validation.validateNodes(cloudGroups)
 
 	if err := validation.collectComponentFailures(ctx, v.k8sClient); err != nil {
 		return nil, fmt.Errorf("cannot get component status for %q: %v", clusterName, err)
 	}
 
-	if err := validation.collectPodFailures(ctx, v.k8sClient, readyNodes); err != nil {
+	if err := validation.collectPodFailures(ctx, v.k8sClient, nodeList.Items); err != nil {
 		return nil, fmt.Errorf("cannot get pod health for %q: %v", clusterName, err)
 	}
 
@@ -199,22 +199,13 @@ func (v *ValidationCluster) collectComponentFailures(ctx context.Context, client
 	return nil
 }
 
-var masterStaticPods = []string{
-	"kube-apiserver",
-	"kube-controller-manager",
-	"kube-scheduler",
-}
-
 func (v *ValidationCluster) collectPodFailures(ctx context.Context, client kubernetes.Interface, nodes []v1.Node) error {
-	masterWithoutPod := map[string]map[string]bool{}
+	masterWithoutManager := map[string]bool{}
 	nodeByAddress := map[string]string{}
 	for _, node := range nodes {
 		labels := node.GetLabels()
 		if labels != nil && labels["kubernetes.io/role"] == "master" {
-			masterWithoutPod[node.Name] = map[string]bool{}
-			for _, pod := range masterStaticPods {
-				masterWithoutPod[node.Name][pod] = true
-			}
+			masterWithoutManager[node.Name] = true
 		}
 		for _, nodeAddress := range node.Status.Addresses {
 			nodeByAddress[nodeAddress.Address] = node.Name
@@ -263,9 +254,9 @@ func (v *ValidationCluster) collectPodFailures(ctx context.Context, client kuber
 
 		}
 
-		app := pod.GetLabels()["k8s-app"]
-		if pod.Namespace == "kube-system" && masterWithoutPod[nodeByAddress[pod.Status.HostIP]][app] {
-			delete(masterWithoutPod[nodeByAddress[pod.Status.HostIP]], app)
+		labels := pod.GetLabels()
+		if pod.Namespace == "kube-system" && labels != nil && labels["k8s-app"] == "kube-controller-manager" {
+			delete(masterWithoutManager, nodeByAddress[pod.Status.HostIP])
 		}
 		return nil
 	})
@@ -273,21 +264,18 @@ func (v *ValidationCluster) collectPodFailures(ctx context.Context, client kuber
 		return fmt.Errorf("error listing Pods: %v", err)
 	}
 
-	for node, nodeMap := range masterWithoutPod {
-		for app := range nodeMap {
-			v.addError(&ValidationError{
-				Kind:    "Node",
-				Name:    node,
-				Message: fmt.Sprintf("master %q is missing %s pod", node, app),
-			})
-		}
+	for node := range masterWithoutManager {
+		v.addError(&ValidationError{
+			Kind:    "Node",
+			Name:    node,
+			Message: fmt.Sprintf("master %q is missing kube-controller-manager pod", node),
+		})
 	}
 
 	return nil
 }
 
-func (v *ValidationCluster) validateNodes(cloudGroups map[string]*cloudinstances.CloudInstanceGroup) []v1.Node {
-	var readyNodes []v1.Node
+func (v *ValidationCluster) validateNodes(cloudGroups map[string]*cloudinstances.CloudInstanceGroup) {
 	for _, cloudGroup := range cloudGroups {
 		var allMembers []*cloudinstances.CloudInstanceGroupMember
 		allMembers = append(allMembers, cloudGroup.Ready...)
@@ -344,9 +332,6 @@ func (v *ValidationCluster) validateNodes(cloudGroups map[string]*cloudinstances
 			}
 
 			ready := isNodeReady(node)
-			if ready {
-				readyNodes = append(readyNodes, *node)
-			}
 
 			if n.Role == "master" {
 				if !ready {
@@ -373,6 +358,4 @@ func (v *ValidationCluster) validateNodes(cloudGroups map[string]*cloudinstances
 			}
 		}
 	}
-
-	return readyNodes
 }


Test patch:
diff --git a/pkg/validation/validate_cluster_test.go b/pkg/validation/validate_cluster_test.go
index 8829148876..fb32e76b21 100644
--- a/pkg/validation/validate_cluster_test.go
+++ b/pkg/validation/validate_cluster_test.go
@@ -374,7 +374,7 @@ func Test_ValidateMasterNotReady(t *testing.T) {
 	}
 }
 
-func Test_ValidateMasterStaticPods(t *testing.T) {
+func Test_ValidateMasterNoKubeControllerManager(t *testing.T) {
 	groups := make(map[string]*cloudinstances.CloudInstanceGroup)
 	groups["node-1"] = &cloudinstances.CloudInstanceGroup{
 		InstanceGroup: &kopsapi.InstanceGroup{
@@ -427,71 +427,37 @@ func Test_ValidateMasterStaticPods(t *testing.T) {
 					},
 				},
 			},
-			{
-				ID: "i-00003",
-				Node: &v1.Node{
-					ObjectMeta: metav1.ObjectMeta{
-						Name:   "master-1c",
-						Labels: map[string]string{"kubernetes.io/role": "master"},
-					},
-					Status: v1.NodeStatus{
-						Conditions: []v1.NodeCondition{
-							{Type: "Ready", Status: v1.ConditionFalse},
-						},
-						Addresses: []v1.NodeAddress{
-							{
-								Address: "9.10.11.12",
-							},
-						},
-					},
-				},
-			},
 		},
 	}
 
-	var podList []map[string]string
-	expectedFailures := []*ValidationError{
-		{
-			Kind:    "Node",
-			Name:    "master-1c",
-			Message: "master \"master-1c\" is not ready",
-		},
-	}
-
-	for i, pod := range []string{
-		"kube-apiserver",
-		"kube-controller-manager",
-		"kube-scheduler",
-	} {
-		podList = append(podList, []map[string]string{
+	v, err := testValidate(t, groups, makePodList(
+		[]map[string]string{
 			{
-				"name":              fmt.Sprintf("pod-a-%d", i),
+				"name":              "pod1",
 				"ready":             "true",
-				"k8s-app":           pod,
+				"k8s-app":           "kube-controller-manager",
 				"phase":             string(v1.PodRunning),
 				"priorityClassName": "system-cluster-critical",
 				"hostip":            "1.2.3.4",
 			},
 			{
-				"name":              fmt.Sprintf("pod-b-%d", i),
+				"name":              "pod2",
 				"namespace":         "other",
 				"ready":             "true",
-				"k8s-app":           pod,
+				"k8s-app":           "kube-controller-manager",
 				"phase":             string(v1.PodRunning),
 				"priorityClassName": "system-cluster-critical",
 				"hostip":            "5.6.7.8",
 			},
-		}...)
-		expectedFailures = append(expectedFailures, &ValidationError{
+		},
+	))
+	require.NoError(t, err)
+	if !assert.Len(t, v.Failures, 1) ||
+		!assert.Equal(t, &ValidationError{
 			Kind:    "Node",
 			Name:    "master-1b",
-			Message: "master \"master-1b\" is missing " + pod + " pod",
-		})
-	}
-
-	v, err := testValidate(t, groups, makePodList(podList))
-	require.NoError(t, err)
-	if !assert.ElementsMatch(t, v.Failures, expectedFailures) {
+			Message: "master \"master-1b\" is missing kube-controller-manager pod",
+		}, v.Failures[0]) {
 		printDebug(t, v)
 	}
 }
diff --git a/tests/integration/create_cluster/gce_byo_sa/expected-v1alpha2.yaml b/tests/integration/create_cluster/gce_byo_sa/expected-v1alpha2.yaml
index 5fc5d09579..2f06093502 100644
--- a/tests/integration/create_cluster/gce_byo_sa/expected-v1alpha2.yaml
+++ b/tests/integration/create_cluster/gce_byo_sa/expected-v1alpha2.yaml
@@ -9,8 +9,6 @@ spec:
   authorization:
     rbac: {}
   channel: stable
-  cloudConfig:
-    gceServiceAccount: test-account@testproject.iam.gserviceaccount.com
   cloudProvider: gce
   configBase: memfs://tests/gce.example.com
   containerRuntime: docker
@@ -27,6 +25,7 @@ spec:
       name: a
     memoryRequest: 100Mi
     name: events
+  gceServiceAccount: test-account@testproject.iam.gserviceaccount.com
   iam:
     allowContainerRegistry: true
     legacy: false
diff --git a/tests/integration/create_cluster/ha_gce/expected-v1alpha2.yaml b/tests/integration/create_cluster/ha_gce/expected-v1alpha2.yaml
index 578d08fe14..0ee09fe0a1 100644
--- a/tests/integration/create_cluster/ha_gce/expected-v1alpha2.yaml
+++ b/tests/integration/create_cluster/ha_gce/expected-v1alpha2.yaml
@@ -9,8 +9,6 @@ spec:
   authorization:
     rbac: {}
   channel: stable
-  cloudConfig:
-    gceServiceAccount: default
   cloudProvider: gce
   configBase: memfs://tests/ha-gce.example.com
   containerRuntime: docker
@@ -35,6 +33,7 @@ spec:
       name: c
     memoryRequest: 100Mi
     name: events
+  gceServiceAccount: default
   iam:
     allowContainerRegistry: true
     legacy: false
diff --git a/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-a-ha-gce-example-com_metadata_startup-script b/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-a-ha-gce-example-com_metadata_startup-script
index c9357b2f05..4d3dfbe99e 100644
--- a/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-a-ha-gce-example-com_metadata_startup-script
+++ b/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-a-ha-gce-example-com_metadata_startup-script
@@ -130,7 +130,6 @@ ensure-install-dir
 
 cat > conf/cluster_spec.yaml << '__EOF_CLUSTER_SPEC'
 cloudConfig:
-  gceServiceAccount: default
   multizone: true
   nodeTags: ha-gce-example-com-k8s-io-role-node
 containerRuntime: docker
diff --git a/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-b-ha-gce-example-com_metadata_startup-script b/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-b-ha-gce-example-com_metadata_startup-script
index ab141c7dee..76ab11e5ce 100644
--- a/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-b-ha-gce-example-com_metadata_startup-script
+++ b/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-b-ha-gce-example-com_metadata_startup-script
@@ -130,7 +130,6 @@ ensure-install-dir
 
 cat > conf/cluster_spec.yaml << '__EOF_CLUSTER_SPEC'
 cloudConfig:
-  gceServiceAccount: default
   multizone: true
   nodeTags: ha-gce-example-com-k8s-io-role-node
 containerRuntime: docker
diff --git a/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-c-ha-gce-example-com_metadata_startup-script b/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-c-ha-gce-example-com_metadata_startup-script
index b81d0b0160..79ad0b17e5 100644
--- a/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-c-ha-gce-example-com_metadata_startup-script
+++ b/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_master-us-test1-c-ha-gce-example-com_metadata_startup-script
@@ -130,7 +130,6 @@ ensure-install-dir
 
 cat > conf/cluster_spec.yaml << '__EOF_CLUSTER_SPEC'
 cloudConfig:
-  gceServiceAccount: default
   multizone: true
   nodeTags: ha-gce-example-com-k8s-io-role-node
 containerRuntime: docker
diff --git a/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_nodes-ha-gce-example-com_metadata_startup-script b/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_nodes-ha-gce-example-com_metadata_startup-script
index 4681498b13..04c0073782 100644
--- a/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_nodes-ha-gce-example-com_metadata_startup-script
+++ b/tests/integration/update_cluster/ha_gce/data/google_compute_instance_template_nodes-ha-gce-example-com_metadata_startup-script
@@ -130,7 +130,6 @@ ensure-install-dir
 
 cat > conf/cluster_spec.yaml << '__EOF_CLUSTER_SPEC'
 cloudConfig:
-  gceServiceAccount: default
   multizone: true
   nodeTags: ha-gce-example-com-k8s-io-role-node
 containerRuntime: docker
diff --git a/tests/integration/update_cluster/minimal_gce/data/google_compute_instance_template_master-us-test1-a-minimal-gce-example-com_metadata_startup-script b/tests/integration/update_cluster/minimal_gce/data/google_compute_instance_template_master-us-test1-a-minimal-gce-example-com_metadata_startup-script
index 64611028bb..7d93c1c2e0 100644
--- a/tests/integration/update_cluster/minimal_gce/data/google_compute_instance_template_master-us-test1-a-minimal-gce-example-com_metadata_startup-script
+++ b/tests/integration/update_cluster/minimal_gce/data/google_compute_instance_template_master-us-test1-a-minimal-gce-example-com_metadata_startup-script
@@ -130,7 +130,6 @@ ensure-install-dir
 
 cat > conf/cluster_spec.yaml << '__EOF_CLUSTER_SPEC'
 cloudConfig:
-  gceServiceAccount: default
   multizone: true
   nodeTags: minimal-gce-example-com-k8s-io-role-node
 containerRuntime: docker
diff --git a/tests/integration/update_cluster/minimal_gce/data/google_compute_instance_template_nodes-minimal-gce-example-com_metadata_startup-script b/tests/integration/update_cluster/minimal_gce/data/google_compute_instance_template_nodes-minimal-gce-example-com_metadata_startup-script
index ced79d8362..d962f17f47 100644
--- a/tests/integration/update_cluster/minimal_gce/data/google_compute_instance_template_nodes-minimal-gce-example-com_metadata_startup-script
+++ b/tests/integration/update_cluster/minimal_gce/data/google_compute_instance_template_nodes-minimal-gce-example-com_metadata_startup-script
@@ -130,7 +130,6 @@ ensure-install-dir
 
 cat > conf/cluster_spec.yaml << '__EOF_CLUSTER_SPEC'
 cloudConfig:
-  gceServiceAccount: default
   multizone: true
   nodeTags: minimal-gce-example-com-k8s-io-role-node
 containerRuntime: docker
diff --git a/util/pkg/vfs/fs_test.go b/util/pkg/vfs/fs_test.go
index e481925db1..5ca6670e3b 100644
--- a/util/pkg/vfs/fs_test.go
+++ b/util/pkg/vfs/fs_test.go
@@ -60,39 +60,3 @@ func TestCreateFile(t *testing.T) {
 		}
 	}
 }
-
-func TestWriteTo(t *testing.T) {
-	var TempDir, _ = ioutil.TempDir("", "test")
-	defer os.Remove(TempDir)
-	tests := []struct {
-		path string
-		data []byte
-	}{
-		{
-			path: path.Join(TempDir, "SubDir", "test1.tmp"),
-			data: []byte("test data\nline 1\r\nline 2"),
-		},
-	}
-	for _, test := range tests {
-		var buf bytes.Buffer
-
-		fspath := NewFSPath(test.path)
-		// Create file
-		err := fspath.CreateFile(bytes.NewReader(test.data), nil)
-		if err != nil {
-			t.Fatalf("Error writing file %s, error: %v", test.path, err)
-		}
-
-		// Write file to buf
-		_, err = fspath.WriteTo(&buf)
-		if err != nil {
-			t.Fatalf("Error reading %s to buf, error: %v", test.path, err)
-		}
-
-		// Check buf content
-		actually_bytes := buf.Bytes()
-		if !bytes.Equal(test.data, actually_bytes) {
-			t.Errorf("Expected %v, actually %v", test.data, actually_bytes)
-		}
-	}
-}