Change:
/kind bug /priority failing-test /sig cluster-lifecycle testing This test fails: https://k8s- Some clues: waiting for new master timeout: master healthz check failed: etcd is crashloopbacking: I guess the cause may be something associated with etcd version compatibility. any thoughts or suggestions? Thanks! This is also posted in !topic/kubernetes-sig-cluster-lifecycle/c4MW3R5v4v0 /cc
xref: /cc
cc
cc From my understanding, this is expected as etcd can't handle downgrades. I think this has been worked around though in previous releases by avoiding to downgrade etcd on k8s downgrade. That's what we're gonna do in kubeadm at least
Yes, per the etcd , "[automatic] downgrade from this completed state is not possible... Please backup the data directory of all etcd members to make downgrading the cluster possible even after it has been completely upgraded."
thanks! Is there any way to configure the downgrade test not to downgrade etcd? However, this might cause another problem: if we don't downgrade etcd from 3.1.10 to 3.0.17 when downgrading cluster from 1.9 to 1.8, can the 1.8 cluster work well against 3.1.10 etcd since the officially supported etcd version of 1.8 is 3.0.17?
great question. We would probably need to setup tests to actually test it, but I really believe that it should work fine (k8s 1.8 has already etcd client in 3.1.10 version). And also, we don't really have big choice here - that's the only thing we can do/recommend doing.
reasonable. Totally agreed :) So now the concern here is that how we should configure the downgrade test not to downgrade etcd to avoid the job failing. any ideas on the config of this job?
cc
I'm pretty sure we used to do that in the past (when we were doing etcd 2.2.1 -etcd 3.0.17 upgrade in 1.5 -1.6 release (though those might have been manual tests). Assuming that the tests are setup on GCE, it should be enough to set some env variables for the downgrade to pin the downgrade to a particular release: ETCDIMAGE and ETCDVERSION seem to be those ones.
thanks for the info! I will send a PR (touching some e2e/cluster-lifecycle) trying to specify the etcd version 3.1.10 when the cluster downgrade target version is 1.8 but I'm not sure if it's a good solution for this issue.
FYI I'm proposing to unblock the tests from running. A better long-term fix is needed.
I send another workaround I guess it won't conflict with your PR. It may be a starting of a long-term fix.
We need a good release note to warn users that they need to prevent an etcd downgrade if they downgrade k8s. We filled in the warning from 1.6 as a placeholder, but it needs to be updated for this specific issue: We also should add a warning to the upgrade script, similar to this one from 1.6: cc who were involved with the update to 3.1.10 () Can anyone advise on the above?
Looks like we need something along the lines of this for : I'll fill out the google doc for the release notes as well.
The idea you mentioned of checking for and env vars before emitting a warning in sounded useful. is the docker tag of the for image to run etcd from. Each image contains multiple etcd binaries. For example: determine which etcd binary is run. It is passed to as which is used to select the exact etcd binary. If I understood your suggestion right, we could check if and are set to before emitting a warning in and skip the warning if they are set to 3.0.x, since an and etcd upgrade will not occur.
I manually verified that the following works as expected to downgrade from 1.9 to 1.8: Before upgrading, my 1.8 cluster had etcd 3.0.17. After upgrading, my 1.9 cluster had etcd 3.1.10. After downgrading, my 1.8 cluster had etcd 3.1.10 still. I'll work on adding a prompt to if these are unset.
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
The latest downgrade test is running with the config by : Note the But the test still seems to stuck on waiting for new master: . There is no more log so I'm not sure if it's still due to etcd.
also xref
thanks! I didn't notice that comment :)
Let's wait for the next run against :) Is there anyway to stop the current stuck run? It lasts so long (~15h) for each run.
you should be able to reference the variant, which runs more often
Which just had it's first green run. On Dec 12, 2017 19:20, "Sen Lu" <:

Discussion:
fi fi <ins> # Prompt if etcd image/version is unspecified when doing master upgrade. # In e2e tests, we use TEST_ALLOW_IMPLICIT_ETCD_UPGRADE=true to skip this # prompt, simulating the behavior when the user confirms interactively. # All other automated use of this script should explicitly specify a version. if [[ "${master_upgrade}" == "true" ]]; then if [[ -z "${ETCD_IMAGE:-}" && -z "${TEST_ETCD_IMAGE:-}" ]] || [[ -z "${ETCD_VERSION:-}" && -z "${TEST_ETCD_VERSION:-}" ]]; then echo echo "***WARNING***" echo "Upgrading Kubernetes with this script might result in an upgrade to a new etcd version." echo "Some etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path." echo "To pin the etcd version to your current one (e.g. v3.0.17), set the following variables" echo "before running this script:" echo echo "# example: pin to etcd v3.0.17" echo "export ETCD_IMAGE=3.0.17" echo "export ETCD_VERSION=3.0.17" echo echo "Alternatively, if you choose to allow an etcd upgrade that doesn't support downgrade," echo "you might still be able to downgrade Kubernetes by pinning to the newer etcd version." echo "In all cases, it is strongly recommended to have an etcd backup before upgrading." echo if [ -t 0 ] && [ -t 1 ]; then read -p "Continue with default etcd version, which might upgrade etcd? [y/N] " confirm if [[ "${confirm}" != "y" ]]; then exit 1 fi elif [[ "${TEST_ALLOW_IMPLICIT_ETCD_UPGRADE:-}" != "true" ]]; then echo "ETCD_IMAGE and ETCD_VERSION must be specified when run non-interactively." >&2 exit 1 fi fi fi </ins> print-node-version-info "Pre-Upgrade" if [[ "${local_binaries}" == "false" ]]; then

"TEST_ETCD_VERSION="+TestContext.EtcdUpgradeVersion, "STORAGE_BACKEND="+TestContext.EtcdUpgradeStorage, "TEST_ETCD_IMAGE=3.1.10") <ins> } else { // In e2e tests, we skip the confirmation prompt about // implicit etcd upgrades to simulate the user entering "y". env = append(env, "TEST_ALLOW_IMPLICIT_ETCD_UPGRADE=true") </ins> } v := "v" + rawV