Change:
componentstatuses=$(echo "${cs_status}" | grep -c 'Healthy:') || true healthy=$(echo "${cs_status}" | grep -c 'Healthy:True') || true <del> if ((componentstatuses > healthy)); then </del> <ins> if ((componentstatuses > healthy)) || ((componentstatuses == 0)); then </ins> if ((attempt < 5)); then echo -e "${color_yellow}Cluster not working yet.${color_norm}" attempt=$((attempt+1))

Discussion:
cluster upgrade should maintain a functioning cluster [Feature:ClusterUpgrade] connection refused in version check /sig networking /assign See also
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig network
I took a quickly look at the failures, the root cause is actually the master node is not ready. The following error msg indicates why master is not ready: It is indeed a network related issue.
I am re-assigning this back to you, please delegate. Thanks! cc/
this looks like a CNI related, can you guys take a look?
/assign
One thing I see happening at least in the GCE tests (Because it has the master log, but the GKE tests do not) I0614 04:25:10. 5 ] Node bootstrap-e2e-minion-group-2cz0 has CIDR 10.180.1.0/24, occupying it in CIDR map E0614 04:25:10. 5 ] Error starting "node" F0614 04:25:10. 5 ] error starting controllers: failed to mark cidr as occupied: cidr 10.180.1.0/24 is out the range of cluster cidr 10.100.0.0/14 I see that the test has: cluster/gce/config-"${CLUSTERIPRANGE:-10.100.0.0/14}" And this for the the pod cidr: "--pod-cidr", "10.180.0.0/24", // Assign a fixed CIDR to the node because there is no node controller. Is the above 10.100.0.0/14 supposed to be 10.180/14 ? 10.100/14 is only going to give 10.100 to 10.103 which isn't within the cluster cidr range.
I think I have an idea what happened here.

It looks like this PR changed the cluster CIDR from 10.180/14 to 10.100/14 However I see the the e2e node test assumes the pod cider is 10.180.0/24 Here: And Here: Since I believe we need to keep the 10.100/14 probably the right thing to do now is to fix the e2e node test to pick a pod cidr within 10.100/14
This was green on 6-15, if green again today i'll close this. Thanks for your attention !
should be the one fixing the issue. I will leave this open, until all UpgradeTest on GCE 1.6-1.7 are green. Thanks!
Ok, I thought we still have the issue on this, and assign this to myself. It turns out the test is still running with the old build for the upgrade test. Will continue monitoring this. cc/
fyi
After was merged, the master is up and ready, but the upgrade is still marked as failed. I looked into it this morning, and found controller_manager was running into the crashloop after master was upgraded due to the following error: I believe the issue is that we changed validation test in at which breaks the backward compatibility with 1.6 nodes. cc/ WDYT?
Had a quick discussion with network team The quick fix can be putting all nodes' cidr back 10.180.0.0/24, instead of 10.100.0.0/14 to stop the upgrade craziness. is working on this, and running the upgrade tests locally. Meanwhile, is going to discuss with on the next steps on how to rollout the feature which depends on the new ip range: 10.100.0.0/14
I think the problem now is that we have one version using 10.180.x. (1.6) and one using 10.100x.x. (1.7) And when we upgrade happens the cluster was created with 10.180 but then we upgrade we want 10.100. Can we just set the env var CLUSTERIPRANGE to 10.100 in the test so that the 1.6 cluster comes up with the right range?
In particular I think we could just change this file? To include the CLUSTERIPRANGE.

What you suggested should work theoretically. But since both branches hard-coded with two different ip range in some files, instead of respecting CLUSTERIPRANGE env variable, it won't work now. Since the failure are limited to our tests only, and upgrade test itself relies on the previous branch's latest build, not the real release, we decided to cherry pick fix to 1.6 branch. I closed my revert prs.
Cherrypicking the required changes to 1.6, 1.5:
besides cherrypicking , I believe we also need cherrypick which changes the API validation code; Otherwise, you are going to break 1.6 branch validation path on this. EDITED: Ignore this comment. All changes in are either for node e2e tests, or for validation test itself.
Thanks, looking forward to a green run :) Latest failure (in 1.6 -17 upgrade tests) is as in opening comment:
After cherrypicking those ip_range changes to 1.6 branch, now the master node is successfully upgraded to 1.7, and still accepted nodes running 1.6. The kube-controller-manager is not in crashloop anymore. The below is the output by running kubectl get nodes on the master node with https://k8s-
Also all kube-system pods are running successfully after master is upgraded:
Testing is still running, but I noticed that the cluster upgrade failed in the log at https://k8s- From above comment at , one can see that 1) now master node are up and ready, vs. notready in pre-54 runs 2) kube-controller-manager is running for 3 hour, not in crashloop in pre-54 runs kube-controller-manager-bootstrap-e2e-master 1/1 Running 3 3h 3) kubectl cs returns ok without error. There is no 'kubectl get nodes' failure, no kubect get cs failure. Above error / failure is red-herring, and looks like an autograde test infra issue. cc/
It looks like failed from the e2e test. I don't doubt that it worked from the master, but that's not where the e2e is running. I suspect an auth issue.

When the suite finishes, we can confirm by looking at the api server log for component status access from the kubectl user agent.
I checked kube-, there is no failure from the kubectl user agent, especially right before test claimed UpgradeMaster is failed, /componentstatus from kubectl is ok: I0620 00:23:45. 5 ] GET /api/v1/componentstatuses: (757.782364ms) 200 [[kubectl/v1.7.0 (linux/amd64) kubernetes/] 35.188.93.146:] In test log: https://k8s-, before previous /componentstatuses call, master is upgraded successfully, and all work-nodes are ready: But still complaint cluster is not working: But there is no retry with kubectl get cs which should be called multiple times in validate- and I checked validate- and think there is a problem in the code. The weirdness is that validation- hasn't been touched for a year. Something underneath triggers this change? is sending a quick fix.
Ok, I have more data to say that (along with ) is really buggy. Without any change, the new run of both master and nodes are successfully upgraded to 1.7 in a new run: https://k8s-
This guy is flakey: , Most recent failure about 11pm yesterday, still waiting for another run. Since the test is still failing, this issue should not have been closed - but the bot did it and don't have the authority to re-open. we're skipping some tests because of known API skew failures. Do you want to mark this test as a skip?
/reopen
Let's mark this as flakey and skip it.
we cannot skip this because it is does the real upgrade. If master upgrade failed, the test is red; if the node upgrade failed, the test is red. Sometimes, the script is flaky at the end, the test is red even the entire cluster is upgraded successfully. But we should skip this from 1.5-1.7 upgrade test suites since we only care about kubectl skews tests. There is no need to upgrade node from 1.5-1.7.
So we're going to ignore this in the 15-1.7 tests. Focus on the 1.6-1.7 tests.

Seeing new red in these (after some green) These guys remained green:
Is this related the aggregation API server ready issue?
What's the status of this issue? Should it continue blocking 1.7? If not, please remove the 'approved-for-milestone' label.
is on PTO
After we fixed the subnetwork issue, now upgrade are fine: https://k8s- There is failure for ingress And the engineer indicates it is caused by the infrastructure underneath.
I remove this as the release blocker.
This is failing in gce-1.7-master-upgrade-master: example job: https://k8s- I will apply release blocker and urgency to this. Please advice if it is not the case.
This did occur there, but it doesn't seem to occur often (hasn't happened in that suite since). The major issue with the suite that you linked is the panic you fixed yesterday.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale