Change:
We ran into a lot of issues when etcd crashes. We need tests to ensure all components / daemons are doing the right things etcd is down, and recover later.
Note that, like the existing test, this is a case of a system level that probably can't be parallelized very easily. (Though rolling over etcd while running other tests is a perfectly fine chaos-monkey style test, but it's not very controlled.) cc
As context, this came up at the conference last week. In particular, if etcd crashes or goes unavailable, then recovers, it was reported that some cluster components do not recover gracefully.
We're still looking for a volunteer for this. Any takers? cc: who might be interested.
has accepted
I discussed the test scenario with We came up with the following: up a few pods, so that the system has something to do. a bit, so that they are initialized up 100 watches on the API server (to simulate a large system) if possible, use NewEndpointController (from - which is supposed to be used by all controllers) to simulate real clients. This EndpointController will automatically restart watches. It's interesting to see what happens at the API server when all watches are restarted at the very same time. etcd (ssh onto the machine) bit (?) new pods (maybe don't wait, just retry on error) to see that they are scheduled (we check here only that the scheduler still works). It makes no sense to test everything after etcd is restarted because we would end up repeating all tests. We will just use the fact that tests are run in random order so we hope next tests will find components that didn't recover properly after etcd crashes (or rather after their watches having been terminated). the test
May I suggest a perhaps simpler first step, which does most of the above with minimal effort... Clone an existing e2e test (guestbook and rc are good candidates) Tweak it slightly such that: -- At the point where success would usually be declared, instead of declaring success, ssh into the etcd machine and render etcd temporarily uncontactable (more on how to do that below). -- Now kill all of the Pods in the application (with appropriate retries). We now rely on the replication controller to recover and reconstitute the test application. -- check that the test eventually fulfills success criteria again. I think that the above will be super-simple to get working. Once that is working, you can do the same thing for a much larger scale test (e.g. one of the scalability e2e tests.
Oh, and regarding how to make etcd uncontactable for a while, two suggestions: into the node (use the new ssh utility in test/e2e/utils) and schedule a simple cron job to insert an iptables DROP rule to block all incoming traffic to etcd, sleep for a while (I think it will have to be 5 minutes to trigger stuff), and then remove the iptables rule to allow traffic to flow again. alternatively: into the node and schedule a cron job to repeatedly kill the etcd process for ~5 minutes, and then stop killing it, allowing monit to restart it automatically.
I think your step 1/2 is not really necessary-- there are pods running in the cluster by default. black holeing traffic to etcd seems like a good thing to try too, but I think we should also test killing/restarting it.
agreed. Change "alternatively" to "in addition" above :-)
Ok, how about the guestbook test, with just killing etcd before creating all the objects (maybe several times during 5 minutes). Then I'll add more complicated scenarios. I suggest being careful about modifying the environment (like iptables or cron). If the test crashes in the middle, all following tests will fail. Do we have a cleanup/healthcheck executed procedure before each test? I could insert resetting iptables (or cron) there. (If not, shall we create it?)
Yes, just killing etcd before the test will also work. Regarding modifying, yes, that's why I suggested cron (on the etcd machine), so that the cleanup will happen even if the test code itself fails. I think that's good enough.
/subscribe
fwiw we regularly do this, but have chosen not to make it an e2e due to sec-keys and our etcd cluster resides on a different loc from our master. We could use etcd-discovery service to make it more dynamic, but we would still have the issue of keys.
Is anyone currently working on this?
I emailed earlier today to see if he'd be interested in working on it, but since he'll be out Thu-Fri it would probably be better for you to do it, if you're interested.
please take it as I'm OOO next two days.
Just submitted PR to resolve this. I would appreciate review.
Current e2e tests has two failure cases: network partition and SIGKILL. The network partition runs consistently but the SIGKILL is flaky. I'm investigating further.

Discussion:
<ins> /* Copyright 2015 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/fields" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/wait" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = Describe("Etcd failure", func() { framework := Framework{BaseName: "etcd-failure"} BeforeEach(func() { framework.beforeEach() Expect(RunRC(RCConfig{ Client:    framework.Client, Name:      "baz", Namespace: framework.Namespace.Name, Image:     "nginx", Replicas:  1, })).NotTo(HaveOccurred()) }) AfterEach(framework.afterEach) It("should recover from network partition with master", func() { etcdFailTest( framework, "sudo iptables -A INPUT -p tcp --destination-port 4001 -j DROP", "sudo iptables -D INPUT -p tcp --destination-port 4001 -j DROP", false, ) }) It("should recover from SIGKILL", func() { etcdFailTest( framework, "pgrep etcd | xargs -I {} sudo kill -9 {}", "echo 'do nothing. monit should restart etcd.'", true, ) }) }) func etcdFailTest(framework Framework, failCommand, fixCommand string, repeat bool) { // This test requires SSH, so the provider check should be identical to // those tests. if !providerIs("gce", "gke") { By(fmt.Sprintf("Skippingt test, which is not implemented for %s", testContext.Provider)) return } doEtcdFailure(failCommand, fixCommand, repeat) checkExistingRCRecovers(framework) ServeImageOrFail(framework.Client, "basic", "gcr.io/google_containers/serve_hostname:1.1") } // For this duration, etcd will be failed by executing a failCommand on the master. // If repeat is true, the failCommand will be called at a rate of once per second for // the failure duration. If repeat is false, failCommand will only be called once at the // beginning of the failure duration. After this duration, we execute a fixCommand on the // master and go on to assert that etcd and kubernetes components recover. const etcdFailureDuration = 20 * time.Second func doEtcdFailure(failCommand, fixCommand string, repeat bool) { By("failing etcd") if repeat { stop := make(chan struct{}, 1) go util.Until(func() { defer GinkgoRecover() masterExec(failCommand) }, 1*time.Second, stop) time.Sleep(etcdFailureDuration) stop <- struct{}{} } else { masterExec(failCommand) time.Sleep(etcdFailureDuration) } masterExec(fixCommand) } func masterExec(cmd string) { stdout, stderr, code, err := SSH(cmd, getMasterHost()+":22", testContext.Provider) Expect(err).NotTo(HaveOccurred()) if code != 0 { Failf("master exec command, '%v' failed with exitcode %v: ntstdout: %vntstderr: %v", cmd, code, stdout, stderr) } } func checkExistingRCRecovers(f Framework) { By("assert that the pre-existing replication controller recovers") podClient := f.Client.Pods(f.Namespace.Name) rcSelector := labels.Set{"name": "baz"}.AsSelector() By("deleting pods from existing replication controller") pods, err := podClient.List(rcSelector, fields.Everything()) Expect(err).NotTo(HaveOccurred()) Expect(len(pods.Items) > 0).Should(BeTrue()) for _, pod := range pods.Items { err = podClient.Delete(pod.Name, api.NewDeleteOptions(0)) Expect(err).NotTo(HaveOccurred()) } By("waiting for replication controller to recover") expectNoError(wait.Poll(time.Millisecond*500, time.Second*30, func() (bool, error) { pods, err := podClient.List(rcSelector, fields.Everything()) Expect(err).NotTo(HaveOccurred()) for _, pod := range pods.Items { if api.IsPodReady(&pod) { return true, nil } } return false, nil })) } </ins>