Change:
I got this by running the following against commit : There have been previous attempts to fix this: It appears it's still an issue :cry: We're also seeing it in OpenShift (), although our code hasn't pulled in the fix from quite yet.
cc
What go version are you using? Previously this test has been catching legit bugs, so I'm not sure about the "flaky" moniker.
go 1.5.1. I noticed you had a comment in another issue or PR about this possibly being a bug in Go itself (or did I misread)?
I doubt it's a bug in go. (go1.5 had an unrelated bug, maybe that's what you're thinking of.) We haven't actually flipped 1.5 on so I may sit on this for a bit.
I instrumented this code, and this is being caused by duplicate deletions causing the wait group to decrement below 0. Are duplicate deletions a bug? Or should the wait group be guarded against that?
ping If duplicate deletes are not a bug, I can put together a quick PR to guard against that. If they are a bug, I could try to track it down sometime this week (even thought I'm not that familiar with this code) if no one else is going to look into it.
Duplicate deletes are a bug. (assuming that the test harness isn't actually issuing multiple deletes for the same item.)
That would be great if you could take this, since I won't have time.
So it would be an error in the test if was called multiple times for the same pod? Or should the framework be de-duping that?
IMO, that's a test error. ...If the test is doing that because the framework dup'd an update or an add, then it's a framework error. (adds and updates shouldn't be duped, either) , krousey wrote:
from what I gather, this test is just trying to delete after each pod reaches it's final intended state. This code is deleting on every event, including intermediate ones. In particular, this test case is occasionally causing duplicate deletes to be issued: Sometimes it catches the transition from FROM to ADDMISSED, and then gets ADDMISSED to TO. I think the fix is to tell the test to only issue the delete when the final state is TO. I'll do that and some stress testing to see if it goes away.
3589 runs so far, 10 failures and all 10 failures seem to confirm the above behavior.
Guarding against this: runs so far, 0 failures I will clean this up and send out a fix shortly.
Awesome, thanks! , krousey wrote:
Accidentally left the test case running. Good news:
It's probably safe to say that if there's remaining corner cases, your setup is not going to exercise them! :) , krousey wrote:

Discussion:
pair{FROM, FROM}: true, } <del> pod := func(name, check string) *api.Pod { return &api.Pod{ </del> <ins> pod := func(name, check string, final bool) *api.Pod { p := &api.Pod{ </ins> ObjectMeta: api.ObjectMeta{ Name:   name, Labels: map[string]string{"check": check}, }, } <ins> if final { p.Labels["final"] = "true" } return p } deletePod := func(p *api.Pod) bool { return p.Labels["final"] == "true" </ins> } tests := []func(string){ func(name string) { name = "a-" + name <del> source.Add(pod(name, FROM)) source.Modify(pod(name, TO)) </del> <ins> source.Add(pod(name, FROM, false)) source.Modify(pod(name, TO, true)) </ins> }, func(name string) { name = "b-" + name <del> source.Add(pod(name, FROM)) source.ModifyDropWatch(pod(name, TO)) </del> <ins> source.Add(pod(name, FROM, false)) source.ModifyDropWatch(pod(name, TO, true)) </ins> }, func(name string) { name = "c-" + name <del> source.AddDropWatch(pod(name, FROM)) source.Modify(pod(name, ADD_MISSED)) source.Modify(pod(name, TO)) </del> <ins> source.AddDropWatch(pod(name, FROM, false)) source.Modify(pod(name, ADD_MISSED, false)) source.Modify(pod(name, TO, true)) </ins> }, func(name string) { name = "d-" + name <del> source.Add(pod(name, FROM)) </del> <ins> source.Add(pod(name, FROM, true)) </ins> }, }

if !allowedTransitions[pair{from, to}] { t.Errorf("observed transition %q -> %q for %v", from, to, n.Name) } <del> source.Delete(n) </del> <ins> if deletePod(n) { source.Delete(n) } </ins> }, DeleteFunc: func(obj interface{}) { testDoneWG.Done()