Change:
expected := []*runtimeapi.Container{} state := runtimeapi.ContainerState_CONTAINER_RUNNING <del> var createdAt int64 = 0 </del> <ins> var createdAt int64 = fakeClock.Now().UnixNano() </ins> for i := range configs { // We don't care about the sandbox id; pass a bogus one. sandboxID := fmt.Sprintf("sandboxid%d", i)

Discussion:
https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory.

Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.