Change:
timeout: 90 emails: '$DEFAULT_RECIPIENTS, beeps@google.com' test-owner: 'beeps' <ins> - 'gce-es-logging': description: 'Run [Feature:Elasticsearch] tests on GCE using the latest successful build.' timeout: 90 emails: '$DEFAULT_RECIPIENTS, mixia@google.com' test-owner: 'mixia' </ins> jobs: - 'kubernetes-e2e-{suffix}'

Discussion:
It failed in gce suite. I honestly have no idea under which team it falls:
Another failure: bumping to p0. - can you find an owner, or at least an appropriate team for this?
owns logging (see team description).
Cannot reproduce failure so far. Leaving it to run overnight...
Still cannot reproduce the exact failure... Increase verbosity...
There were a lot of such failures over the weekend:
Moving this test to flaky (), as it's only 66% stable and is blocking the build quite often: please prioritize this.
This test was passing 100% of the time until Build (Dec 5, 2015 4:13:38 PM). After that, it failed 50% of the time (10 failures in 20 runs). Clearly a PR that merged shortly before the above time introduced the flakyness. That PR should be rolled back.
hmm - I looked into PRs that were merged around that time, and actually nothing looks suspicious to me
This test is 100% successful in flaky test suite so far. I suspect some other e2e test is impacting this one. Adding more sanity checks... (root)/Kubernetes%%20suite/ClusterlevelloggingusingElasticsearchshouldcheckthatlogsfrompodsonallnodesareingestedinto_Elasticsearch/
Let it run in flaky suite overnight. If everything goes well, will unmark it.
I can't follow any of the links but if I could get a bit more information about what is going wrong I could look into the Elasticsearch test if you think that is the issue rather than something else having a bad impact on it.
Thanks for offering to help. Here are the failure symptoms in Jenkins: cluster in red status. I bump into this once with my test cluster. It is cause by es split brain problem. Fix already merged. all log lines from one node. I cannot reproduce this. My guess is the fluentd pod on a node is not working properly. So I validation for fluentd pods. Can you think of anything else?
Can you please point me to the fix for the split brain problem? Is that ? So that works by making sure there are always two master nodes? Is 2 really a good idea for quorum? As suggested here a choice of 2 might not be desirable?

Missing log lines indeed mean a synthetic logging pod did not start for some reason -- without more log information it's hard to work out exactly why this occurred. If I get a moment I will try to do a repo. Thanks.
minimummasternodes set to 2 does not mean having 2 master nodes. It means the es cluster needs to have at least 2 nodes before electing a master. If network partition happened to a es non-master node, it will wait until it could reach out to at least one other node to start electing master. Hence it will prevent the split brain problem. Based on the log in jenkins, all synthetic logging pod ran successfully.
It gets flaky again... I am able to reproduce the problem by running DaemonRestart and Elasticsearch test together. It looks like the DaemonRestart test somehow disrupts the ES cluster. ES cluster gets into bad state, hence failing the es test. Confirmed with This particular test restarts all the containers.
It's a recent regression. Found the cause and will upload a fix soon.
- please monitor this test and close this bug if there won't be any failures in some time.
Test is green for the last 40 builds. Closing the issue.
This test is still marked as at . please send a PR to promote it out of ?
Since is the owner of the issue. I'll let him decide whether the test is ready.
The test has been successful lately. Want to add one more check to improve resilience. I had a pending PR to remove label. Will try to merge it on Monday if everything goes well during the weekend.
I think that is because elasticsearch query return unexpect results. should solve it.
Weirdly hit the same unrelated pr twice, but it looks like the fix just got merged
Still flaking:
what's next on this? Seems to not be resolved...
Moved back into flaky in , is trying for repro/solidifying the test IIUC
Trying to reproduce the problem. It is running successful in flaky suite and my local cluster so far... Will try to run it with other tests together.
Two types of failure: cluster in bad state. This only happens when running with other disruptive tests. Never failed in suite where it is not running with disruptive tests. ES resilience can be improved by log lines in Elasticsearch.

I am able to reproduce this once. It is due to fluentd is hanged. Filed a issue to upstream. Request to mark this test as [Feature] since it depends on two 3rd party components.
The risk here is that ES is a default cluster addon. IMO it's fine not testing it for critical builds, but the risk is we'll ship something that straight crashloops. So if we move it to feature I'd ask that you add a smaller test that verifies basic functionality to a blocking suite (ping es, check that it accepts some logs). Or make it non-default.
Good point. I will work on first. It has to be done anyway. It will also make future ES updates easier. BTW, ES addon is not default for GCE/GKE.
Really? my e2e cluster always has it, why are we testing with it then?
For GCE:
Yeah - another place where test and 'production' configs differ, but I guess we need to test elasticsearch for 'fully opensource' version, thus it's justified...
That's [Feature] then, just like we test flannel. Not something to bring up as an addon on every e2e clusters.
We probably need some suite with all 'important' options (e.g. Features) turned on, on top of dedicated suite for every feature. Ugh...