Change:
for _, field := range fieldsFail { for _, action := range actions { options := &api.ListOptions{FieldSelector: field.AsSelector(), ResourceVersion: "1"} <ins> ctx = context.WithValue(context.WithValue(ctx, "field", field), "action", action) </ins> watcher, err := t.storage.(rest.Watcher).Watch(ctx, options) if err != nil { t.Errorf("unexpected error: %v", err)

Discussion:
https://k8s- Failed: TestWatch {etcd} Previous issues for this test:
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
/cc
are there any known issues on the clientv3 prior to 3.0.8? Just wondering if we need a rev bump.
I can never repro this locally, and the leaks have been fixed.
https://k8s- Failed: TestWatch {etcd}
So it's failing in different spots, but replicasets is the most common occurrence.
xref:
ok, we have a bead on the issue. It will not affect the released bits, but it does affect the UTs.
PR in flight.
What's the default logging level of unit test?
If we looked at the test, they are sort of wrong themselves. For example, It's doing another action on the same key, but it's watching at version "1" again. From my understanding, if we have action "create, delete", I want to watch two actions one by one. But it's watching at "1", now we just watch "create", then another same "create".
Even worse, it could be hiding problems. For example, if there is some event that happened in previous suites, then it will happen to next suites without differentiating and checking the actual values from ResultChan(). In such case, a wrong test might pass.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
- if was supposed to help, it didn't - it is still failing.
has some ideas... and I'm digging too.
https://k8s- Failed: TestWatch {etcd}
Here's my understanding of the scenario: What's happening? A lot of get requests are sent due to prev-value. Note that they are quorum get. Those get request are cancelled due to the wrong logic in tests. What's happening inside etcd is that etcd server is still processing them and trying to send them until grpc fails. These get requests could block further events from being processed.

It gets even worse under heavy loaded environment. What's the problem in the unit test itself? Their logic are wrong themselves. See my previous comments. We need to fix the tests. What's on etcd? I need to stress test this behavior. Then I will coordinate with etcd team to see potentially what could be improved/fixed.
There is just a single etcd instance - so whether this is quorum or not doesn't matter here. Actually - it is a valid usecase that user cancel the request (if I'm not missing anything subtle here). ^
Quorum get affects processing events. The problem is not the cancelling. See my previous comments on the test itself. If we expect to test cancelling requests, then we should have another set of tests for that, and have different checking workflow.
I think the tests watching a "1" expect a new etcd instance each time. Otherwise they would be horribly broken.
https://k8s- Failed: TestWatch {etcd}
I'm still digging into "how" this happens... we fixed a bunch of issues. It's clear the watch is timing out.. sometimes. Every single test fires up it's own client/server pair, each of which is also properly cleaned... now.
share your repro steps?
when digging through latest output across the tests e.g. I'm always seeing a triplet of: But tests always pass, still digging...
This is the problem caused by watching "1". It keeps getting previous events. Meanwhile, watching "1" has some other latency issue. I had done some benchmark that does watch from 1 and latency could spike up to 5 second. I need more time to do more testing on this. But the tests themselves are problematic. We should fix them first. If it doesn't fix the problem, we will try to reproduce it then.
I was assuming your previous comment says that you have changed/fixed the tests and you are still reproducing the problem?
Why is watching from "1" problematic? There shouldn't be any previous events, since we start etcd fresh for every test. , Hongchao Deng wrote:
The tests are problematic. Inside each test there are many test suites not getting a fresh etcd.

Watching "1" will force etcd to query internal database for past events, potentially causing some latency issue on compacted env. With all these said, it's all theories. We still need to do more logging and testing around it to find out real cause.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
for searchability
I see what is saying now re: rewatch @ 1. The tests areas that are failing are in loops that are constantly re-issuing a watch at 1, even though b/c it's in a loop emitting it should not be 1, tests work fine when simply removing entirely. So I'll put together a PR that ups the timeout + removed the ListOption. Although I do think the negative tests should probably be re-written to interleave with the positive tests, to eliminate the time suck.
Thanks. I am very much looking forward to that PR. ;)
Do you have an ETA for the PR to help us figure out if this should be moved to flaky as a stop-gap?
we're still debugging, what doesn't help is that these failures only seem to occur on GCE and no one care repro them outside of that environment.
Frustrating. Are you able to reliable reproduce using the "run until failure" test option?

Merged
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
Still occurring xref:
Yes, just happened 3x in a row here:
Do you think we can create a new etcd instance every time? As mentioned in
won't fix the problem. I do have another workaround for this until etcd fix comes in.
is on