Change:
Testgrid: https://k8s- Sample failure log: https://k8s- This test job has always been flaky, but is now failing more often than it's passing. Diagnosing it is difficult, though, because even the "successful" runs have multiple failures in the log, and the reported test failures seem not to be independant failures in the logs, but rather related to multiple failures to deploy pods and storage in the test setup. Does anyone have more insight into what's going on here? This feels more like a google cloud or test config failure than anything else. /kind failing-test /kind flake /sig test-infrastructure /sig gcp /sig apps
/sig testing
also looks like https://k8s- has similar issues, so this is actually not a gke specific problem.
one of the kubelet log has not sure if that's helpful at all... /assign (redirect to get some help from dev-oncall to see if can find any leads here...)
/assign /cc
GitHub didn't allow me to assign the following users: artemvmin. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The suites are failing on both gce and gke, so I don't think it's a gke specific issue. Looking at one occurrence: https://k8s- The first test that started failing was: Because it couldn't schedule the pod. Looking at the scheduler logs, I only see these two lines referring to that pod and nothing else: This is odd because there is usually at least one more log message if scheduling failed or succeeded. This implies that the pod is stuck somewhere in the scheduler and there could potentially be a deadlock. There are no other scheduling logs about other pods after this. An odd pattern that I noticed was that for all the serial runs where a ton of tests fail, the test case that always runs before the subsequent failing tests is the equivalence cache tests. And there's always a line in the like: I wonder if there is some race condition regarding the pod deletion case. I have seen this pattern in the most recent runs where a bunch of tests all fail: https://k8s- https://k8s- https://k8s- https://k8s- The first failing test case is different across all these runs, but the commonality between them all is that the equivalence cache tests run right before it.
can you plz help take a look at this and help triage within scheduling?
fyi, this is pretty easy to repro on a dev cluster on head. I just run the EquivalenceCache tests, and then create any Pod afterwards. The Pod fails to schedule about 50% of the time.
I am investigating it.
I identified the issue and sent a PR to address it. We need to cherrypick the fix to 1.12 as well. Thanks to for the initial investigation and finding reproduction steps!
Thanks a lot and for looking into this and giving a fox PR. /cc needs to be CP'ed into 1.12 as well.
It already got approved. Please create a cherry pick PR for it.

Discussion:
"sync" "k8s.io/api/core/v1" <del> "k8s.io/apimachinery/pkg/util/sets" </del> utilnode "k8s.io/kubernetes/pkg/util/node" "github.com/golang/glog"

// NodeTree is a tree-like data structure that holds node names in each zone. Zone names are // keys to "NodeTree.tree" and values of "NodeTree.tree" are arrays of node names. type NodeTree struct { <del> tree           map[string]*nodeArray // a map from zone (region-zone) to an array of nodes in the zone. zones          []string              // a list of all the zones in the tree (keys) zoneIndex      int exhaustedZones sets.String // set of zones that all of their nodes are returned by next() NumNodes       int mu             sync.RWMutex </del> <ins> tree      map[string]*nodeArray // a map from zone (region-zone) to an array of nodes in the zone. zones     []string              // a list of all the zones in the tree (keys) zoneIndex int NumNodes  int mu        sync.RWMutex </ins> } // nodeArray is a struct that has nodes that are in a zone.

// newNodeTree creates a NodeTree from nodes. func newNodeTree(nodes []*v1.Node) *NodeTree { nt := &NodeTree{ <del> tree:           make(map[string]*nodeArray), exhaustedZones: sets.NewString(), </del> <ins> tree: make(map[string]*nodeArray), </ins> } for _, n := range nodes { nt.AddNode(n)

for _, na := range nt.tree { na.lastIndex = 0 } <del> nt.exhaustedZones = sets.NewString() </del> <ins> nt.zoneIndex = 0 </ins> } // Next returns the name of the next node. NodeTree iterates over zones and in each zone iterates

if len(nt.zones) == 0 { return "" } <ins> numExhaustedZones := 0 </ins> for { if nt.zoneIndex >= len(nt.zones) { nt.zoneIndex = 0 } zone := nt.zones[nt.zoneIndex] nt.zoneIndex++ <del> // We do not check the set of exhausted zones before calling next() on the zone. This ensures </del> <ins> // We do not check the exhausted zones before calling next() on the zone. This ensures </ins> // that if more nodes are added to a zone after it is exhausted, we iterate over the new nodes. nodeName, exhausted := nt.tree[zone].next() if exhausted { <del> nt.exhaustedZones.Insert(zone) if len(nt.exhaustedZones) == len(nt.zones) { // all zones are exhausted. we should reset. </del> <ins> numExhaustedZones++ if numExhaustedZones >= len(nt.zones) { // all zones are exhausted. we should reset. </ins> nt.resetExhausted() } } else {

operations:     []string{"add", "add", "add", "add", "add", "next", "next", "next", "next", "add", "next", "next", "next"}, expectedOutput: []string{"node-4", "node-5", "node-6", "node-7", "node-3", "node-8", "node-4"}, }, <ins> { name:           "remove zone and add new to ensure exhausted is reset correctly", nodesToAdd:     append(allNodes[3:5], allNodes[6:8]...), nodesToRemove:  allNodes[3:5], operations:     []string{"add", "add", "next", "next", "remove", "add", "add", "next", "next", "remove", "next", "next"}, expectedOutput: []string{"node-3", "node-4", "node-6", "node-7", "node-6", "node-7"}, }, </ins> } for _, test := range tests {