Repository: kubernetes/kops (go)

Issue:
rolling-update: cluster not validated correctly
Using 1.17.0-alpha.1 and kubernetes 1.16.2: trying to update to 1.16.3

```
kops rolling-update cluster --bastion-interval 2m --instance-group bastions,master-zone-1,nodes --validation-timeout 20m --yes

snip

I1210 10:05:51.299467 53 instancegroups.go:357] Waiting for 5s for pods to stabilize after draining.
I1210 10:05:56.299785 53 instancegroups.go:176] deleting node "master-zone-1-1-updatemajorpr-892a8a-k8s-local" from kubernetes
I1210 10:05:56.317191 53 instancegroups.go:302] Stopping instance "36a99cb5-a769-4f26-8b78-b8f91f5408ba", node "master-zone-1-1-updatemajorpr-892a8a-k8s-local", in group "updatemajorpr-892a8a.k8s.local-master-zone-1" (this may take a while).
W1210 10:05:56.317225 53 instance.go:105] This does not work without running kops update cluster --yes in another terminal
I1210 10:05:56.758342 53 instancegroups.go:189] waiting for 15s after terminating instance
I1210 10:06:11.758539 53 instancegroups.go:200] Validating the cluster.
I1210 10:07:02.275601 53 instancegroups.go:261] Cluster did not validate, will try again in "30s" until duration "20m0s" expires: error listing nodes: Get https://10.222.142.60/api/v1/nodes: unexpected EOF.
I1210 10:07:36.134273 53 instancegroups.go:268] Cluster did not pass validation, will try again in "30s" until duration "20m0s" expires: machine "f8d4ab37-f577-4e2a-8bfd-25cc7a4735f0" has not yet joined cluster.
I1210 10:08:06.597076 53 instancegroups.go:268] Cluster did not pass validation, will try again in "30s" until duration "20m0s" expires: machine "f8d4ab37-f577-4e2a-8bfd-25cc7a4735f0" has not yet joined cluster.
I1210 10:08:37.806011 53 instancegroups.go:268] Cluster did not pass validation, will try again in "30s" until duration "20m0s" expires: machine "f8d4ab37-f577-4e2a-8bfd-25cc7a4735f0" has not yet joined cluster.
I1210 10:09:07.741695 53 instancegroups.go:268] Cluster did not pass validation, will try again in "30s" until duration "20m0s" expires: machine "f8d4ab37-f577-4e2a-8bfd-25cc7a4735f0" has not yet joined cluster.
I1210 10:09:36.913556 53 instancegroups.go:268] Cluster did not pass validation, will try again in "30s" until duration "20m0s" expires: machine "f8d4ab37-f577-4e2a-8bfd-25cc7a4735f0" has not yet joined cluster.
I1210 10:10:12.868584 53 instancegroups.go:268] Cluster did not pass validation, will try again in "30s" until duration "20m0s" expires: machine "f8d4ab37-f577-4e2a-8bfd-25cc7a4735f0" has not yet joined cluster.
I1210 10:10:36.515544 53 instancegroups.go:268] Cluster did not pass validation, will try again in "30s" until duration "20m0s" expires: master "master-zone-1-1-updatemajorpr-892a8a-k8s-local" is not ready, kube-system pod "audit-webhook-deployment-blgz2" is pending, kube-system pod "calico-node-6mzhm" is pending, kube-system pod "kops-controller-g4szb" is pending, kube-system pod "openstack-cloud-provider-8pbv4" is pending.
I1210 10:11:06.813311 53 instancegroups.go:268] Cluster did not pass validation, will try again in "30s" until duration "20m0s" expires: kube-system pod "calico-node-6mzhm" is not ready (calico-node).
I1210 10:11:36.556927 53 instancegroups.go:271] Cluster validated.

cluster "updatemajorpr-892a8a.k8s.local" did not pass validation: kube-system pod "kube-controller-manager-master-zone-1-1-updatemajorpr-892a8a-k8s-local" is pending
I1210 10:11:41.741877 1 batch.go:835] error running kops rolling-update cluster --bastion-interval 2m --instance-group bastions,master-zone-1,nodes --validation-timeout 20m --yes
error running: exit status 1
```

So what happens here? It first validates the cluster, but then at the same time services in new master (master-zone-1-1-updatemajorpr-892a8a-k8s-local) is starting. This will lead to rolling update fail

Like it was mentioned in another issue:

- The cluster validated, returning from `validateClusterWithDuration()` from the master-zone-1 instancegroup
- kube-controller-manager on that new node then went into pending state (it started after the master is created)
- The cluster then failed validation from the initial `validateCluster()` from the nodes instancegroup

How we could validate that ALL missing components are in place when running `validateClusterWithDuration()`, or should we skip initial validateCluster if validateClusterWithDuration succeeded previously?

cc @johngmyers I did own issue about this

Pull request description:
test validateCluster twice to make sure it does not flap
fixes #8078 


/kind bug

Patch:
diff --git a/cmd/kops/rollingupdatecluster.go b/cmd/kops/rollingupdatecluster.go
index acb9824940..1df6c67d94 100644
--- a/cmd/kops/rollingupdatecluster.go
+++ b/cmd/kops/rollingupdatecluster.go
@@ -414,7 +414,8 @@ func RunRollingUpdateCluster(f *util.Factory, out io.Writer, options *RollingUpd
 		PostDrainDelay:    options.PostDrainDelay,
 		ValidationTimeout: options.ValidationTimeout,
 		// TODO should we expose this to the UI?
-		ValidateTickDuration: 30 * time.Second,
+		ValidateTickDuration:    30 * time.Second,
+		ValidateSuccessDuration: 10 * time.Second,
 	}
 	return d.RollingUpdate(groups, cluster, list)
 }
diff --git a/docs/advisories/spectre-meltdown-kernel-update.md b/docs/advisories/spectre-meltdown-kernel-update.md
index 54cff42199..a8e77e488e 100644
--- a/docs/advisories/spectre-meltdown-kernel-update.md
+++ b/docs/advisories/spectre-meltdown-kernel-update.md
@@ -94,7 +94,6 @@ running the updated image.
 `kops rolling-update cluster --name $CLUSTER --yes`
 
 ## Resources / Notes
-
 - https://aws.amazon.com/de/security/security-bulletins/AWS-2018-013/
 - https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html
 - https://coreos.com/blog/container-linux-meltdown-patch
diff --git a/docs/boot-sequence.md b/docs/boot-sequence.md
index db36e48472..62654d988e 100644
--- a/docs/boot-sequence.md
+++ b/docs/boot-sequence.md
@@ -98,7 +98,6 @@ means we don't need to manage etcd directly.  We just try to make sure that some
 each volume mounted with etcd running and DNS set correctly.  That is the job of protokube.
 
 Protokube:
-
 * discovers EBS volumes that hold etcd data (using tags)
 * tries to safe_format_and_mount them
 * if successful in mounting the volume, it will write a manifest for etcd into /etc/kubernetes/manifests
diff --git a/docs/calico-v3.md b/docs/calico-v3.md
index 0b3ac6ea75..7e85183ec4 100644
--- a/docs/calico-v3.md
+++ b/docs/calico-v3.md
@@ -5,7 +5,6 @@ the requirements, upgrade process, and configuration to install
 Calico Version 3.
 
 ## Requirements
-
 - The main requirement needed for Calico Version 3 is the etcd v3 API available
   with etcd server version 3.
 - Another requirement is for the Kubernetes version to be a minimum of v1.7.0.
@@ -19,7 +18,6 @@ to remain on Calico V2 or update to etcdv3.
 ## Configuration of a new cluster
 To ensure a new cluster will have Calico Version 3 installed the following
 two configurations options should be set:
-
 - `spec.etcdClusters.etcdMembers[0].Version` (Main cluster) should be
   set to a Version of etcd greater than 3.x or the default version
   needs to be greater than 3.x.
@@ -48,7 +46,6 @@ Assuming your cluster meets the requirements it is possible to upgrade
 your Calico Kops cluster.
 
 A few notes about the upgrade:
-
 - During the first portion of the migration, while the calico-kube-controllers
   pod is running its Init, no new policies will be applied though already
   applied policy will be active.
diff --git a/docs/development/building.md b/docs/development/building.md
index 840f6ff2e0..19a55b4e94 100644
--- a/docs/development/building.md
+++ b/docs/development/building.md
@@ -47,5 +47,5 @@ and then configure your IDE to connect its debugger to port 2345 on localhost.
 
  - Make sure `$GOPATH` is set, and your [workspace](https://golang.org/doc/code.html#Workspaces) is configured.
  - kops will not compile with symlinks in `$GOPATH`. See issue go issue [17451](https://github.com/golang/go/issues/17451) for more information
- - building kops requires go 1.12 or 1.13
+ - building kops requires go 1.8 or 1.9
  - Kops will only compile if the source is checked out in `$GOPATH/src/k8s.io/kops`. If you try to use `$GOPATH/src/github.com/kubernetes/kops` you will run into issues with package imports not working as expected.
diff --git a/docs/development/vendoring.md b/docs/development/vendoring.md
index fbf5034183..1bec549042 100644
--- a/docs/development/vendoring.md
+++ b/docs/development/vendoring.md
@@ -24,9 +24,9 @@ found in the source code. Follow these steps to run the update process:
 2. Run `make gomod` to start the update process. If this step is
 successful, the imported dependency will be added to the `vendor`
 subdirectory.
-3. Commit any changes, including changes to the `vendor` directory,
+1. Commit any changes, including changes to the `vendor` directory,
 `go.mod`, and `go.sum`.
-4. Open a pull request with these changes separately from other work
+1. Open a pull request with these changes separately from other work
 so that it is easier to review.
 
 ## Updating a dependency in the vendor directory (e.g. aws-sdk-go)
@@ -36,5 +36,5 @@ so that it is easier to review.
 3. Review the changes to ensure that they are as intended / trustworthy.
 4. Commit any changes, including changes to the `vendor` directory,
 `go.mod` and `go.sum`.
-5. Open a pull request with these changes separately from other work so that it
+1. Open a pull request with these changes separately from other work so that it
 is easier to review.  Please include any significant changes you observed.
diff --git a/docs/etcd/roadmap.md b/docs/etcd/roadmap.md
index f759af479d..d24149760d 100644
--- a/docs/etcd/roadmap.md
+++ b/docs/etcd/roadmap.md
@@ -17,7 +17,6 @@ tooling into composable tooling that can be upgraded (or even used) separately.
 ## Limitations
 
 The current approach for managing etcd makes certain tasks hard:
-
 * upgrades/downgrades between etcd versions
 * resizing the cluster
 
diff --git a/docs/getting_started/arguments.md b/docs/getting_started/arguments.md
index ab8163d608..70e5ad1f50 100644
--- a/docs/getting_started/arguments.md
+++ b/docs/getting_started/arguments.md
@@ -48,7 +48,6 @@ Examples:
 `cloud-labels` specifies tags for instance groups in AWS. The supported format is a CSV list of key=value pairs.
 Keys and values must not contain embedded commas but they may contain equals signs ('=') as long as the field is
 quoted:
-
 * `--cloud-labels "Project=\"Name=Foo Customer=Acme\",Owner=Jane Doe"` will be parsed as {Project:"Name=Foo Customer=Acme",
 Owner: "Jane Doe"}
 
diff --git a/docs/getting_started/digitalocean.md b/docs/getting_started/digitalocean.md
index adad155c5b..52f644749d 100644
--- a/docs/getting_started/digitalocean.md
+++ b/docs/getting_started/digitalocean.md
@@ -53,7 +53,6 @@ kops delete cluster my-cluster.example.com --yes
 ## Features Still in Development
 
 kops for DigitalOcean currently does not support these features:
-
 * multi master kubernetes clusters
 * rolling update for instance groups
 * multi-region clusters
diff --git a/docs/getting_started/openstack.md b/docs/getting_started/openstack.md
index be23672b5c..05694e8177 100644
--- a/docs/getting_started/openstack.md
+++ b/docs/getting_started/openstack.md
@@ -61,7 +61,6 @@ kops delete cluster my-cluster.k8s.local --yes
 ```
 
 #### Optional flags
-
 * `--os-kubelet-ignore-az=true` Nova and Cinder have different availability zones, more information [Kubernetes docs](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#block-storage)
 * `--os-octavia=true` If Octavia Loadbalancer api should be used instead of old lbaas v2 api.
 * `--os-dns-servers=8.8.8.8,8.8.4.4` You can define dns servers to be used in your cluster if your openstack setup does not have working dnssetup by default
diff --git a/docs/iam_roles.md b/docs/iam_roles.md
index e889c9b6de..e4f89750c3 100644
--- a/docs/iam_roles.md
+++ b/docs/iam_roles.md
@@ -8,7 +8,6 @@ By default Kops creates two IAM roles for the cluster: one for the masters, and
 Work has been done on scoping permissions to the minimum required for a functional Kubernetes Cluster, resulting in a fully revised set of IAM policies for both master & compute nodes.
 
 An example of the new IAM policies can be found here:
-
 - Master Nodes: https://github.com/kubernetes/kops/blob/master/pkg/model/iam/tests/iam_builder_master_strict.json
 - Compute Nodes: https://github.com/kubernetes/kops/blob/master/pkg/model/iam/tests/iam_builder_node_strict.json
 
diff --git a/docs/labels.md b/docs/labels.md
index 5f8ce201c6..62b9988a65 100644
--- a/docs/labels.md
+++ b/docs/labels.md
@@ -19,7 +19,7 @@ An example:
 
 ```
 ...
-spec:
+spec: 
   cloudLabels:
     team: me
     project: ion
@@ -30,7 +30,7 @@ spec:
 
 ```
 ...
-spec:
+spec: 
   cloudLabels:
     team: me
     project: ion
@@ -50,7 +50,7 @@ An example:
 
 ```
 ...
-spec:
+spec: 
   nodeLabels:
     spot: "false"
 ...
diff --git a/docs/operations/addons.md b/docs/operations/addons.md
index 44db4cfbec..5173180e66 100644
--- a/docs/operations/addons.md
+++ b/docs/operations/addons.md
@@ -217,7 +217,6 @@ semver.  The channels tool keeps track of the current version installed (current
 of an annotation on the `kube-system` namespace).
 
 The channel tool updates the installed version when any of the following conditions apply.
-
 * The version declared in the addon manifest is greater then the currently installed version.
 * The version number's match, but the ids are different
 * The version number and ids match, but the hash of the addon's manifest has changed since it was installed.
diff --git a/docs/operations/cluster_upgrades_and_migrations.md b/docs/operations/cluster_upgrades_and_migrations.md
index 0e8283b8a7..7629b2d45d 100644
--- a/docs/operations/cluster_upgrades_and_migrations.md
+++ b/docs/operations/cluster_upgrades_and_migrations.md
@@ -15,7 +15,6 @@ At some point you will almost definitely want to upgrade the Kubernetes version
 Take a snapshot of your EBS volumes; export all your data from kubectl etc.**
 
 Limitations:
-
 * kops splits etcd onto two volumes now: `main` and `events`.  We will keep the `main` data, but you will lose your events history.
 * Doubtless others not yet known - please open issues if you encounter them!
 
@@ -188,7 +187,6 @@ You will also need to release the old ElasticIP manually.
 This method provides zero-downtime when migrating a cluster from `kube-up` to `kops`. It does so by creating a logically separate `kops`-managed cluster in the existing `kube-up` VPC and then swapping the DNS entries (or your reverse proxy's upstream) to point to the new cluster's services.
 
 Limitations:
-
 - If you're using the default networking (`kubenet`), there is a account limit of 50 entries in a VPC's route table. If your cluster contains more than ~25 nodes, this strategy, as-is, will not work.
     + Shifting to a CNI-compatible overlay network like `weave`, `kopeio-vxlan` (`kopeio`), `calico`, `canal`, `romana`, and similar. See the [kops networking docs](../networking.md) for more information.
     + One solution is to gradually shift traffic from one cluster to the other, scaling down the number of nodes on the old cluster, and scaling up the number of nodes on the new cluster.
diff --git a/docs/philosophy.md b/docs/philosophy.md
index 20ab08bab1..2f5b3cd2f9 100644
--- a/docs/philosophy.md
+++ b/docs/philosophy.md
@@ -9,7 +9,6 @@ Each component (kubelet, kube-apiserver...) is explicitly configured: We reuse t
 where we can, and we create additional types for the configuration of additional components.
 
 kops can:
-
 * create a cluster
 * upgrade a cluster
 * reconfigure the components
@@ -33,7 +32,7 @@ There are two primary types:
 
 ## State Store
 
-The API objects are currently stored in an abstraction called a ["state store"](/state.md) has more detail.
+The API objects are currently stored in an abstraction called a "state store".  [state.md](/docs/state.md) has more detail.
 
 ## Configuration inference
 
diff --git a/docs/releases.md b/docs/releases.md
index 6fae032bfe..48e5f0d923 100644
--- a/docs/releases.md
+++ b/docs/releases.md
@@ -15,10 +15,10 @@
 
 #### 3. Release kops to github
 
-* Darwin binary (generated)
-* Linux binary (generated)
-* Source code (zip)
-* Source code (tar.gz)
+*  Darwin binary (generated)
+*  Linux binary (generated)
+*  Source code (zip)
+*  Source code (tar.gz)
 
 #### 4. Release kops to homebrew
 
diff --git a/docs/releases/1.6.1.md b/docs/releases/1.6.1.md
index 224d3df189..8b7446d91b 100644
--- a/docs/releases/1.6.1.md
+++ b/docs/releases/1.6.1.md
@@ -1,15 +1,13 @@
 *Please see [1.6-NOTES.md](1.6-NOTES.md) for known issues*
 
 Features:
-
 * `kops get` can now output a complete cluster spec (thanks @geojaz)
 * `kops create` can set master/node volume size (thanks @matthew-marchetti)
 * Add ability to set cross-subnet mode in Calico (thanks @ottoyiu)
 * Make Weave MTU configurable and configure jumbo frame support for new clusters on AWS (thanks @jordanjennings)
 * Initial support for external-dns project (thanks @sethpollack)
-
+ 
 Fixes:
-
 * Fix calico bootstrapping problems (thanks @ottoyiu, @ozdanborne)
 * Update to latest release of calico (thanks @mad01)
 * Update canal manifests for 1.6 & RBAC (thanks @heschlie)
diff --git a/docs/rotate-secrets.md b/docs/rotate-secrets.md
index d8294003d0..549cca79ac 100644
--- a/docs/rotate-secrets.md
+++ b/docs/rotate-secrets.md
@@ -41,3 +41,6 @@ kubectl delete pods -lk8s-app=kube-dns --namespace=kube-system
 kubectl delete pods -lk8s-app=kube-dns-autoscaler --namespace=kube-system
 pkill -f kube-controller-manager
 ```
+
+
+
diff --git a/docs/run_in_existing_vpc.md b/docs/run_in_existing_vpc.md
index 5fcdfbf55c..e1666633e2 100644
--- a/docs/run_in_existing_vpc.md
+++ b/docs/run_in_existing_vpc.md
@@ -206,7 +206,7 @@ Please note:
 * kops won't create a route-table at all if we're not creating subnets.
 * In the example above the first subnet is using a shared NAT Gateway while the
   second one is using a shared NAT Instance
-
+  
 ### Externally Managed Egress
 
 If you are using an unsupported egress configuration in your VPC, _kops_ can be told to ignore egress by using a configuration like:
diff --git a/docs/single-to-multi-master.md b/docs/single-to-multi-master.md
index 9330bfb850..80b879df91 100644
--- a/docs/single-to-multi-master.md
+++ b/docs/single-to-multi-master.md
@@ -3,7 +3,7 @@
 This document describes how to go from a single-master cluster (created by kops)
 to a multi-master cluster. If you are using etcd-manager you just need to perform some steps of the migration.  
 
-# etcd-manager
+# etcd-manager 
 
 If you are using etcd-manager, just perform the steps in this section. Etcd-manager is default for kops 1.12. Etcd-manager makes the upgrade to multi-master much smoother. 
 
@@ -16,12 +16,12 @@ The list references steps of the next section. To upgrade from a single master t
   - add the masters to your etcd cluster definition (both in section named main and events)
 - Skip Step 3 and 4
 - Now you are ready to update the AWS configuration:
-  - `kops update cluster your-cluster-name`
+  - `kops update cluster your-cluster-name` 
 - AWS will launch two new masters, they will be discovered and then configured by etcd-manager
 - check with `kubectl get nodes` to see everything is ready
 - Cleanup (Step 5) to do a rolling restart of all masters (just in case)
 
-# Etcd without etcd-manager
+# Etcd without etcd-manager 
 
 ## 0 - Warnings
 
diff --git a/docs/upgrade_from_kops_1.6_to_1.7_calico_cidr_migration.md b/docs/upgrade_from_kops_1.6_to_1.7_calico_cidr_migration.md
index 791c8285f6..0284ba2a65 100644
--- a/docs/upgrade_from_kops_1.6_to_1.7_calico_cidr_migration.md
+++ b/docs/upgrade_from_kops_1.6_to_1.7_calico_cidr_migration.md
@@ -11,7 +11,6 @@ The field in the clusterSpec, `.NonMasqueradeCIDR`, captures the IP
 range of the cluster.
 
 Within this IP range, smaller IP ranges are then carved out for:
-
 * Service IPs - as defined as `.serviceClusterIPRange`
 * Pod IPs - as defined as `.kubeControllerManager.clusterCIDR`
 
diff --git a/docs/vsphere-development-status.md b/docs/vsphere-development-status.md
index bbbd8d22f1..b5d6b2e605 100644
--- a/docs/vsphere-development-status.md
+++ b/docs/vsphere-development-status.md
@@ -14,8 +14,8 @@ P2: Rarely occurring issues and features that will bring vSphere support closer
 
 **Notes:**
 
-* Effort estimation includes fix for an issue or implementation for a feature, testing and generating a PR.
-* There are a few issues that are related to startup and base image. If we can resolve "Use PhotonOS for vSphere node template" issue first and replace init-cloud with guestinfo, those issues **might** get resolved automatically. But further investigation is needed and fixed issues will need verifications and testings.
+*   Effort estimation includes fix for an issue or implementation for a feature, testing and generating a PR.
+*   There are a few issues that are related to startup and base image. If we can resolve "Use PhotonOS for vSphere node template" issue first and replace init-cloud with guestinfo, those issues **might** get resolved automatically. But further investigation is needed and fixed issues will need verifications and testings.
 
 |Priority|Task|Type (bug, feature, test|Effort estimate(in days)|Remarks|
 |--- |--- |--- |--- |--- |
@@ -47,10 +47,10 @@ List of all kops commands and how they behave for vSphere cloud provider, as of
 
 # Column explanation
 
-* Command, option and usage example are self-explanatory.
-* vSphere support: whether or not the command is supported for vSphere cloud provider (Yes/No), followed by current status of that command and explanation of any failures.
-* Graceful termination needed: If the command will not supported, does it need additional code to fail gracefully for vSphere provider?
-* Remark: Miscellaneous comments about the command.
+*   Command, option and usage example are self-explanatory.
+*   vSphere support: whether or not the command is supported for vSphere cloud provider (Yes/No), followed by current status of that command and explanation of any failures.
+*   Graceful termination needed: If the command will not supported, does it need additional code to fail gracefully for vSphere provider?
+*   Remark: Miscellaneous comments about the command.
 
 |Command|Option|Usage example|vSphere support|Graceful termination needed (if not fixed)|Remark|
 |--- |--- |--- |--- |--- |--- |
diff --git a/docs/welcome/contributing.md b/docs/welcome/contributing.md
index 023ae2b2de..b337f19220 100644
--- a/docs/welcome/contributing.md
+++ b/docs/welcome/contributing.md
@@ -25,13 +25,13 @@ __Issues__
 
 - Help read and triage issues, assist when possible.
 - Point out issues that are duplicates, out of date, etc.
-    - Even if you don't have tagging permissions, make a note and tag maintainers (`/close`,`/dupe #127`).
+  - Even if you don't have tagging permissions, make a note and tag maintainers (`/close`,`/dupe #127`).
 
 __Pull Requests__
 
 - Read and review the code. Leave comments, questions, and critiques (`/lgtm` ).
 - Download, compile, and run the code and make sure the tests pass (make test).
-    - Also verify that the new feature seems sane, follows best architectural patterns, and includes tests.
+  - Also verify that the new feature seems sane, follows best architectural patterns, and includes tests.
 
 This repository uses the Kubernetes bots.  See a full list of the commands [here](
 https://go.k8s.io/bot-commands).
diff --git a/pkg/instancegroups/instancegroups.go b/pkg/instancegroups/instancegroups.go
index 3b02488d53..0de9c23c8a 100644
--- a/pkg/instancegroups/instancegroups.go
+++ b/pkg/instancegroups/instancegroups.go
@@ -262,8 +262,15 @@ func (r *RollingUpdateInstanceGroup) tryValidateCluster(rollingUpdateData *Rolli
 		klog.Infof("Cluster did not pass validation, will try again in %q until duration %q expires: %s.", tickDuration, duration, strings.Join(messages, ", "))
 		return false
 	} else {
-		klog.Info("Cluster validated.")
-		return true
+		klog.Info("Cluster validated, revalidating to make sure it does not flap.")
+		time.Sleep(rollingUpdateData.ValidateSuccessDuration)
+		result, err = rollingUpdateData.ClusterValidator.Validate()
+		if err == nil && len(result.Failures) == 0 {
+			klog.Info("Cluster validated.")
+			return true
+		}
+		klog.Info("Cluster did not revalidate.")
+		return false
 	}
 }
 
diff --git a/pkg/instancegroups/rollingupdate.go b/pkg/instancegroups/rollingupdate.go
index 7ae9cd27e7..c8cc340ee4 100644
--- a/pkg/instancegroups/rollingupdate.go
+++ b/pkg/instancegroups/rollingupdate.go
@@ -63,6 +63,10 @@ type RollingUpdateCluster struct {
 
 	// ValidateTickDuration is the amount of time to wait between cluster validation attempts
 	ValidateTickDuration time.Duration
+
+	// ValidateSuccessDuration is the amount of time a cluster must continue to validate successfully
+	// before updating the next node
+	ValidateSuccessDuration time.Duration
 }
 
 // RollingUpdate performs a rolling update on a K8s Cluster.


Test patch:
diff --git a/docs/development/testing.md b/docs/development/testing.md
index f953266ba1..ed52665d11 100644
--- a/docs/development/testing.md
+++ b/docs/development/testing.md
@@ -55,7 +55,6 @@ test-infra/jobs/ci-kubernetes-e2e-kops-aws.sh |& tee /tmp/testlog
 ```
 
 This:
-
 * Brings up a cluster using the latest `kops` build from `master` (see below for how to use your current build)
 * Runs the default series of tests (which the Kubernetes team is [also
 running here](https://k8s-testgrid.appspot.com/google-aws#kops-aws)) (see below for how to override the test list)
diff --git a/docs/examples/kops-test-route53-subdomain.md b/docs/examples/kops-test-route53-subdomain.md
index 39e0d6b693..749302c469 100644
--- a/docs/examples/kops-test-route53-subdomain.md
+++ b/docs/examples/kops-test-route53-subdomain.md
@@ -787,7 +787,6 @@ Your cluster mycluster01.kopsclustertest.example.org is ready
 You can see how your cluster scaled up to 3 nodes.
 
 **SCALING RECOMMENDATIONS:**
-
 - Always think ahead. If you want to ensure to have the capability to scale-up to all available zones in the region, ensure to add them to the "--zones=" argument when using the "kops create cluster" command. Example: --zones=us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1e. That will make things simpler later.
 - For the masters, always consider "odd" numbers starting from 3. Like many other cluster, odd numbers starting from "3" are the proper way to create a fully redundant multi-master solution. In the specific case of "kops", you add masters by adding zones to the "--master-zones" argument on "kops create command".
 
diff --git a/docs/examples/kops-tests-private-net-bastion-host.md b/docs/examples/kops-tests-private-net-bastion-host.md
index 641017d944..7a697f460d 100644
--- a/docs/examples/kops-tests-private-net-bastion-host.md
+++ b/docs/examples/kops-tests-private-net-bastion-host.md
@@ -136,7 +136,6 @@ kops create instancegroup bastions --role Bastion --subnet utility-us-east-1a --
 ```
 
 **Explanation of this command:**
-
 - This command will add to our cluster definition a new instance group called "bastions" with the "Bastion" role on the aws subnet "utility-us-east-1a". Note that the "Bastion" role need the first letter to be a capital (Bastion=ok, bastion=not ok).
 - The subnet "utility-us-east-1a" was created when we created our cluster the first time. KOPS add the "utility-" prefix to all subnets created on all specified AZ's. In other words, if we instructed kops to deploy our instances on us-east-1a, use-east-1b and use-east-1c, kops will create the subnets "utility-us-east-1a", "utility-us-east-1b" and "utility-us-east-1c". Because we need to tell kops where to deploy our bastion (or bastions), we need to specify the subnet.
 
@@ -308,7 +307,7 @@ ip-172-20-74-55.ec2.internal    master  True
 Your cluster privatekopscluster.k8s.local is ready
 ```
 
-## MAKING THE BASTION LAYER "HIGHLY AVAILABLE"
+## MAKING THE BASTION LAYER "HIGHLY AVAILABLE".
 
 If for any reason any "legendary monster from the comics" decides to destroy the amazon AZ that contains our bastion, we'll basically be unable to enter to our instances. Let's add some H.A. to our bastion layer and force amazon to deploy additional bastion instances on other availability zones.
 
diff --git a/pkg/instancegroups/rollingupdate_test.go b/pkg/instancegroups/rollingupdate_test.go
index 1518cd515a..027fd6ab10 100644
--- a/pkg/instancegroups/rollingupdate_test.go
+++ b/pkg/instancegroups/rollingupdate_test.go
@@ -46,15 +46,16 @@ func getTestSetup() (*RollingUpdateCluster, awsup.AWSCloud, *kopsapi.Cluster) {
 	cluster.Name = "test.k8s.local"
 
 	c := &RollingUpdateCluster{
-		Cloud:                mockcloud,
-		MasterInterval:       1 * time.Millisecond,
-		NodeInterval:         1 * time.Millisecond,
-		BastionInterval:      1 * time.Millisecond,
-		Force:                false,
-		K8sClient:            k8sClient,
-		ClusterValidator:     &successfulClusterValidator{},
-		FailOnValidate:       true,
-		ValidateTickDuration: 1 * time.Millisecond,
+		Cloud:                   mockcloud,
+		MasterInterval:          1 * time.Millisecond,
+		NodeInterval:            1 * time.Millisecond,
+		BastionInterval:         1 * time.Millisecond,
+		Force:                   false,
+		K8sClient:               k8sClient,
+		ClusterValidator:        &successfulClusterValidator{},
+		FailOnValidate:          true,
+		ValidateTickDuration:    1 * time.Millisecond,
+		ValidateSuccessDuration: 5 * time.Millisecond,
 	}
 
 	return c, mockcloud, cluster