Change:
//modify annotations pod.Annotations["builder"] = "foo" <ins> pod.ResourceVersion = "" // to force update </ins> _, err = f.Client.Pods(f.Namespace.Name).Update(pod) Expect(err).NotTo(HaveOccurred())

Discussion:
In parallel suite:
Can you recreate this? It could be a race or a buffer not flushing.
I'm having a look
Anoter example of the failure: - awesome thanks!
unluckily I cannot seet this page. Thanks anyway
to delegate? (If you don't think it's a node bug, maybe can find an owner?)
Here's our latest conformance log from master with the labels/annotations both failing (mesos/docker cluster): Downward API volume should update labels on modification [Conformance] Downward API volume should update annotations on modification [Conformance]
These failures can be reproduced on master with a single command:
thanks for the logs. I'm investigating the issue using (never worked with so I'm not sure if it's the same or not) Running only one instance of like this --host="127.0.0.1:8080" --provider="local" --ginkgo.v=true --"*on.modification" --kubeconfig="$" basically your command (I guess) I never had a failure. Running the same command in a loop it's different and I managed to reproduce the issue, most of the time the timeout looks to aggressive 15s (while in most of the tests the timeout is something like 5 minutes). since it's seems that this is blocking for several person I'd like to raise the timout (at least to 30seconds). I'm not saying that this is the fix (the old joke about ) but at least it should reduce the problem on busy system.
It looks to me that everytime the fails happens I ran into something like: the same is valid for logs. I tried to in the code (maybe I'm blind) but I couldn't find... Any ideas?
- those lines are coming from this metric: the "kubelet_" at the beginning is "subsystem" in prometheus metrics, that's why you probably didn'g find it
Actually it's printed here:
thanks thanks
We should verify if increasing the timeout: fixed this issue.
Still fails on the mesos/docker cluster on master:
I have been informed that the mesos executor wasn't actually handling annotation/label updates. So these tests were failing legitimately on the mesos/docker cluster.

This will hopefully be fixed by
don't know if makes sense for you but kubelet broke this and we add to fix
Another failure:
:( Sorry I didn't see this... Ok. Having a look again (but today I'm pretty busy)
is not enough. We need also to increase timeout in image which generates the logs sorry for this I didn't realize it when we talked about this thanks for the file!
submitted to increase timout in running image
- great - thanks for looking into it!
I'm reopening it, so that we can monitor if this really fixed the problem. Will close it if we don't observe more errors for a couple of days.
There weren't any failures of this test within last 3 days - closing! Thanks
Saw this flake again in a GKE run:
same test but another issue. we got this: . While the previous issue was a too short timeout (container exited too early for slow envs). would you like to open another issue for this?
Working on this to propose a fix
submitted please have a look
I saw you xref this issue from . I cannot see the relations (and I could be wrong). Mind explain? Let me know if I need to Thanks
- the test you are trying to fix failed in the presubmit of that PR: I didn't review but it's not merged yet, so we don't know if that helps (hopefully yes :))
I think will help on these kind of issues (retrying on the update). thanks for the explanation.
I encountered a failure of this test for a different reason, I can create a separate issue if you think it's unrelated: link:
In the current implementation is performed without retry. implement a retry mechanism similar to
thanks. FYI, Actually we can set the Pod.ResourceVersion="", that will force an update, it would be cleaner than using a Poll.
I'm going to update the PR with what you're suggesting. Thanks