Change:
The test job was passing consistently until midday on 6/18, e.g. Then it started failing consistently, e.g: ... with this error: 12:31:41 ... in gke:detect-node-instance-groups() 12:31:42 ERROR: (gcloud.compute.instance-) Some requests did not succeed: 12:31:42 - The resource 'projects/k8s-jkns-e2e-gke-multizone/zones/us-central1-f/instanceGroupManagers/gke-jenkins-e2e-default-pool--grp' was not found 12:31:42 Assigning to and to triage and assign someone to fix.
cc:
Almost certainly caused by . I'm not going to bother rushing in a rollback today since it's not a critical build, but could you fix this up tomorrow
Yeah, that's me. The change to support multiple MIGs in a single zone doesn't quite work. Didn't know of this test variation. Just to confirm, to run this test variation, the additional env vars I should set are (aside from the standard ones): export KUBERNETESPROVIDER=gke export ADDITIONALZONES=us-central1-a,us-central1-b Any other?
-- yet another instance of a PR passing all presubmit tests and breaking other jenkins jobs after being merged.
Can you tell me a bit more how this test is setup? I tried to do a "kube-" with the additional zones and it worked for me. My settings: export KUBERNETESPROVIDER=gke export ZONE=us-central1-f export CLOUDSDKBUCKET=gs://cloud-sdk-testing/ci/staging export CLOUDSDKAPIENDPOINTOVERRIDESCONTAINER=https://test- export ADDITIONAL_ZONES=us-central1-a,us-central1-b
Have you read ?
Oh, sorry, wrong link.
Take a look at and let me know if you need any more info.
The above PR was the one for GCE. The following is the one did for GKE. There are some relatively minor differences:
Hum, I'll have to dig a bit deeper. I tried the exact combination of env vars and my "kube-up" was successful (with the correct number of nodes across the zones). Curious...
Any ETA, I'm getting tired of all the test failure emails :p
I think I found the problem. I have a workaround that'll make it work for now. I'm hoping sometime this afternoon.
The PR merged and gke-multizone Jenkins has gone green. Sorry for the trouble. There's still a larger issue of multi-zone handling in the test runner itself, but I think that's for another day.

Discussion:
# Detect instance group name generated by gke. # <del> # Note that this will only select instance groups in the same zone as the # cluster, meaning that it won't include all groups in a multi-zone cluster. </del> <ins> # Note that the NODE_INSTANCE_GROUPS var will only have instance groups in the # same zone as the cluster, meaning that it won't include all groups in a # multi-zone cluster. The ALL_INSTANCE_GROUP_URLS will contain all the # instance group URLs, which include multi-zone groups. </ins> # # Assumed vars: #   GCLOUD

#   CLUSTER_NAME # Vars set: #   NODE_INSTANCE_GROUPS <ins> #   ALL_INSTANCE_GROUP_URLS </ins> function detect-node-instance-groups { echo "... in gke:detect-node-instance-groups()" >&2 local urls=$("${GCLOUD}" ${CMD_GROUP:-} container clusters describe  --project="${PROJECT}" --zone="${ZONE}"  --format='value(instanceGroupUrls)' "${CLUSTER_NAME}") urls=(${urls//;/ }) <ins> ALL_INSTANCE_GROUP_URLS=${urls[*]} </ins> NODE_INSTANCE_GROUPS=() for url in "${urls[@]:-}"; do <del> NODE_INSTANCE_GROUPS+=("${url##*/}") </del> <ins> local igm_zone=$(expr match ${url} '.*/zones/([a-z0-9-]*)/') if [[ "${igm_zone}" == "${ZONE}" ]]; then NODE_INSTANCE_GROUPS+=("${url##*/}") fi </ins> done }