Change:
ginkgo.Describe("Granular Checks: Pods", func() { <ins> checkNodeConnectivity := func(config *e2enetwork.NetworkingTestConfig, protocol string, port int) { // breadth first poll to quickly estimate failure. failedPodsByHost := map[string][]*v1.Pod{} // First time, we'll quickly try all pods, breadth first. for _, endpointPod := range config.EndpointPods { framework.Logf("Breadth first check of %v on host %v...", endpointPod.Status.PodIP, endpointPod.Status.HostIP) if err := config.DialFromTestContainer(protocol, endpointPod.Status.PodIP, port, 1, 0, sets.NewString(endpointPod.Name)); err != nil { if _, ok := failedPodsByHost[endpointPod.Status.HostIP]; !ok { failedPodsByHost[endpointPod.Status.HostIP] = []*v1.Pod{} } failedPodsByHost[endpointPod.Status.HostIP] = append(failedPodsByHost[endpointPod.Status.HostIP], endpointPod) framework.Logf("...failed...will try again in next pass") } } errors := []error{} // Second time, we pass through pods more carefully... framework.Logf("Going to retry %v out of %v pods....", len(failedPodsByHost), len(config.EndpointPods)) for host, failedPods := range failedPodsByHost { framework.Logf("Doublechecking %v pods in host %v which werent seen the first time.", len(failedPods), host) for _, endpointPod := range failedPods { framework.Logf("Now attempting to probe pod [[[ %v ]]]", endpointPod.Status.PodIP) if err := config.DialFromTestContainer(protocol, endpointPod.Status.PodIP, port, config.MaxTries, 0, sets.NewString(endpointPod.Name)); err != nil { errors = append(errors, err) } else { framework.Logf("Was able to reach %v on %v ", endpointPod.Status.PodIP, endpointPod.Status.HostIP) } framework.Logf("... Done probing pod [[[ %v ]]]", endpointPod.Status.PodIP) } framework.Logf("succeeded at polling %v out of %v connections", len(config.EndpointPods)-len(errors), len(config.EndpointPods)) } if len(errors) > 0 { framework.Logf("pod polling failure summary:") for _, e := range errors { framework.Logf("Collected error: %v", e) } framework.Failf("failed,  %v out of %v connections failed", len(errors), len(config.EndpointPods)) } } </ins> // Try to hit all endpoints through a test container, retry 5 times, // expect exactly one unique hostname. Each of these endpoints reports // its own hostname.

Discussion:
What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/remove-triage unresolved