Change:
We recently switched GKE's master node to GCI image, but not GCE master node. This discrepancy causes a lot of issues for us, especially some prs merged might work on GCE, but totally break GKE. We should get rid of that difference. cc/
/cc (two build cops who have recently suffered the effects of this).
The master image for test is defined here: We want to switch all test jobs? Or just one particular job, e.g., k8s ci testing k8s master head? I am afraid that simply replacing the image in that script will affect all test jobs
I think this is a bad plan unless we continue to test container VM start up somehow. We can't break that while people still use it!
An alternative: Set up another jenkins job dedicated for the purpose and mark it as commit queue blocker.
It may be better to just switch GKE masters back to containerVM for now, then redo the sequence (i.e. switch GCE to GCI, then GKE master to GCI).
if we switch both the test and non-test default from cvm -gci then starting with the 1.3 release people won't be using it (by default), so testing becomes less important.
I don't think we have to. The GKE CI job is already a CQ blocker, but it runs after submit so bad PRs still get merged (though they get identified quickly). What we need is a PR builder that runs tests with GCI, so we can block bad changes in the first place.
Definition of PR builers: cc/
Yes. It could be even simpler, it just has to start up a cluster and tear it down. Just run like 1 fast test.
We've got to do something though, the queue has been blocked for like 3 hours on GKE tests not passing.
I'd still like to run the e2e tests as some tests make assumptions of the OS distro and thus fail on GCI. Can we duplicate (that runs gce-e2e on every PR), replacing ContainerVM with GCI as the master and (or) node image?
It's a bit of a pain to do this, since that job also does a full build and it feels like a major waste to build each PR again. The Jenkins agents probably can't handle that much load, anyway.
at some point we probably need to bite the bullet and set up pipelines so we only build kubernetes once, rather than separately in the unit/integration, e2e, and node e2e jobs. I have no idea how well any of that interacts with the pull request plugin, of course.
Thanks for the information . So looks like the first alternative, running tests with GCI/GKE and block bad changes in the first place, is hard to get done due to some infrastructure challenges (as mentioned in ). It's not impossible though, just not prioritized right now. The second alternative would be, keep running the post-submit tests (the GKE e2e tests), and fix issues as they appear, which is not too bad imo since they run every 30 min and block submit queue on failure. At the end of the day, you cannot prevent 100% of breakages in presubmit tests, and that's why we have to have postsubmit tests to complement.
I think start another jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM is the simplest thing unblock us. The newly created job should only run the basic test suites. After 1.3, I suggested we do
No, this is not acceptable! we've been blocked for 10 out of the last 24 hours due to this! :)
I agree that we need to catch this before merging instead of after merging. The question is how best to do so.
The problem is that the submit-queue puts them in at odd hours, and we don't expect OSS oncall to be watching at 2 AM.
Well, the flip side is you don't have to run all tests (on all providers) so that all your PRs get merged faster. I am not saying I prefer postsubmit tests, it just something you have to face, no matter its gke tests or gce tests. I would prefer that too.
I think the solution to this is to run more tests when the PR is created, but then when the submit queue asks for a retest before merging only run a smoke test. I think that's a good tradeoff.
That's a good topic to talk and I have some ideas about that too, but I'd stop dragging the focus too far. :smile:
I don't see any compelling reasons to have a different master image-type for GCE and GKE -- there are strong benefits to them being identical, as we've observed. We want 1.3+ to run GCI for GKE, therefore I propose we switch both GCE and GKE to use GCI for the master on all 1.3+ tests by default, including PR and post-submit. If there is a testing gap in that situation, because we think it's important to support containerVM for 1.3+ masters on GCE (but not GKE), then add coverage for that gap. I don't have enough context to know whether that's pre-submit or post-submit or both.
(To append: I think post-submit coverage is horrible and would support a P0 of any test tiger team and/or code-yellow exit criteria being that we have pre-submit coverage of all the various test matrix points. I realize that'll be expensive in both time and machine resources, but it'll be less expensive than our current approach and I suspect generate significant velocity benefits.)
This is a decent idea.
is out for vacation, could you please pick up this one? I think we agreed to have a separate jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM. The newly created job should only run the basic test suites. If you need some help, please ask
After talking with and we came to the following agreement: will switch the default base image for master nodes running on GCE from debian container-vm to GCI. will add a jenkins job that runs periodically (post-submit) that will continue to test master nodes running the debian container-vm, initially running against the master branch. will pin this job to the 1.3 release branch once it is cut, and no longer run it against the master branch. will delete the new jenkins job once we have full confidence that we will not roll back the master image from GCI to the debian container-vm, sometime between now and the 1.4 release.
I am going to work on a PR for no. 1. cc/
no. 2 is being in
The build broke again because of this:

Discussion:
# This tarball is used by GCI, Ubuntu Trusty, and CoreOS. KUBE_MANIFESTS_TAR= <del> if [[ "${KUBE_OS_DISTRIBUTION:-}" == "trusty" || "${KUBE_OS_DISTRIBUTION:-}" == "gci" || "${KUBE_OS_DISTRIBUTION:-}" == "coreos" ]]; then </del> <ins> if [[ "${OS_DISTRIBUTION:-}" == "trusty" || "${OS_DISTRIBUTION:-}" == "gci" || "${OS_DISTRIBUTION:-}" == "coreos" ]]; then </ins> KUBE_MANIFESTS_TAR="${KUBE_ROOT}/server/kubernetes-manifests.tar.gz" if [[ ! -f "${KUBE_MANIFESTS_TAR}" ]]; then KUBE_MANIFESTS_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-manifests.tar.gz"

PREEMPTIBLE_MASTER=${PREEMPTIBLE_MASTER:-false} <del> OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION:-debian} MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-container-vm-v20160321} </del> <ins> # Today the cluster startup scripts asssume the master and the nodes use the # same OS distro, thus same set of initialization instructions (e.g., metadata, # startup scripts, etc.). The current workaround is the hack in util.sh that # reloads <os_distro>/helper.sh in the gap between when the master is created # and when the nodes are created. # TODO(#26183): Provide a way to differentiate master OS distro and node OS # distro. OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION:-gci} # For GCI, leaving it blank will auto-select the latest GCI image on the `dev` # channel. MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-} </ins> MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-google-containers} <del> NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-"${MASTER_IMAGE}"} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-"${MASTER_IMAGE_PROJECT}"} </del> <ins> NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-container-vm-v20160321} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} </ins> CONTAINER_RUNTIME=${KUBE_CONTAINER_RUNTIME:-docker} RKT_VERSION=${KUBE_RKT_VERSION:-0.5.5}

PREEMPTIBLE_NODE=${PREEMPTIBLE_NODE:-false} PREEMPTIBLE_MASTER=${PREEMPTIBLE_MASTER:-false} <del> OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION:-debian} MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-container-vm-v20160321} </del> <ins> OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION:-gci} # For GCI, leaving it blank will auto-select the latest GCI image on the `dev` # channel. MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-} </ins> MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-google-containers} <del> NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-"${MASTER_IMAGE}"} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-"${MASTER_IMAGE_PROJECT}"} </del> <ins> NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-container-vm-v20160321} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} </ins> CONTAINER_RUNTIME=${KUBE_CONTAINER_RUNTIME:-docker} RKT_VERSION=${KUBE_RKT_VERSION:-0.5.5}

# Otherwise, we respect whatever set by the user. gci_images=( $(gcloud compute images list --project google-containers  --regexp='gci-dev.*' --format='value(name)') ) <del> if [[ -z "${KUBE_GCE_MASTER_IMAGE:-}" ]]; then </del> <ins> if [[ -z "${MASTER_IMAGE:-}" ]]; then </ins> MASTER_IMAGE="${gci_images[0]}" MASTER_IMAGE_PROJECT="google-containers" fi <del> if [[ -z "${KUBE_GCE_NODE_IMAGE:-}" ]]; then </del> <ins> if [[ -z "${NODE_IMAGE:-}" ]]; then </ins> NODE_IMAGE="${gci_images[0]}" NODE_IMAGE_PROJECT="google-containers" fi

--repo-root="${KUBE_ROOT}"  --node-instance-group="${NODE_INSTANCE_GROUP:-}"  --prefix="${KUBE_GCE_INSTANCE_PREFIX:-e2e}"  <del> ${KUBE_OS_DISTRIBUTION:+"--os-distro=${KUBE_OS_DISTRIBUTION}"}  </del> <ins> ${OS_DISTRIBUTION:+"--os-distro=${OS_DISTRIBUTION}"}  </ins> ${NUM_NODES:+"--num-nodes=${NUM_NODES}"}  ${E2E_CLEAN_START:+"--clean-start=true"}  ${E2E_MIN_STARTUP_PODS:+"--minStartupPods=${E2E_MIN_STARTUP_PODS}"} 

cluster/aws/templates/configure-vm-aws.sh:  env-to-grains "runtime_config" cluster/aws/templates/configure-vm-aws.sh:  kubelet_api_servers: '${KUBELET_APISERVER}' cluster/centos/util.sh:  local node_ip=${node#*@} <ins> cluster/gce/config-default.sh:# reloads <os_distro>/helper.sh in the gap between when the master is created </ins> cluster/gce/configure-vm.sh:  advertise_address: '${EXTERNAL_IP}' cluster/gce/configure-vm.sh:  api_servers: '${KUBERNETES_MASTER_NAME}' cluster/gce/configure-vm.sh:  cloud_config: ${CLOUD_CONFIG}