Change:
Example failure: Stacktrace: The problem seem to be related to the fact that we're running the test on private cluster (nodes doesn't have external IP address).
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
Given that there are more issues with the NetLB tests than just this particular one, I think we should revert my PR that re-enabled them. ! Wojtek just sent a PR to do that -
But let's leave this issue open - we want to fix and re-enable it again.
The revert merged, but we didn't managed to get it in before the yesterday's run started. Today's run should be green.
Yeah - but that doesn't mean we should close it later - we should fix and reenable this test. The issue should be opened until that test is reenabled.
That's clear, I didn't say anywhere that we should close it
Ok, the tests have been disabled and they don't fail the correctness tests anymore. When it comes to fixing this particular test. I took a cursory glance and I think it is fixable. We should change the code to not expect public IPs on the nodes. And then when poking over http we should do it not from the test but proxy via some container. There is already a code written for that, see /help wanted
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
/remove-triage unresolved
I would like to take this /assign
- thanks! contains brief description on how to fix that
so are we going to access all the nodes via a test container?
Not sure if I fully understand your question, so if you meant something different let me know: to make the test work in large clusters (5k nodes), we can't access all - we may want to do tricks we do in other tests like access min(cluster size, some constant (e.g. 10 or 100) if you meant "via a test container" - yes, I think it's fine to just migrate the test to always use it
Reopening - since we need to reenable this test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Does this issue need an owner? I got a little lost wading through all of the issues/PRs attached to this issue. If I wanted to take up the reins, how do I trigger the 5k node test on a PR or is it automatically triggered?
The test is disabled. I think the crucial part is done now - which is: I can't remember what else left to be done. Unfortunately you can't do that from oss. We only have post-submits exposes as these are too expensive. We can try running it internally to check if it now works...
We reenabled this test and it seem to work. So closing this one. Thanks for all the help on it!

Discussion:
Namespace string } <ins> // NetexecDialResponse represents the response returned by the `netexec` subcommand of `agnhost` type NetexecDialResponse struct { Responses []string `json:"responses"` Errors    []string `json:"errors"` } </ins> // DialFromEndpointContainer executes a curl via kubectl exec in an endpoint container. func (config *NetworkingTestConfig) DialFromEndpointContainer(protocol, targetIP string, targetPort, maxTries, minTries int, expectedEps sets.String) { config.DialFromContainer(protocol, echoHostname, config.EndpointPods[0].Status.PodIP, targetIP, EndpointHTTPPort, targetPort, maxTries, minTries, expectedEps)

return expectedEps } <ins> func makeCURLDialCommand(ipPort, dialCmd, protocol, targetIP string, targetPort int) string { // The current versions of curl included in CentOS and RHEL distros // misinterpret square brackets around IPv6 as globbing, so use the -g // argument to disable globbing to handle the IPv6 case. return fmt.Sprintf("curl -g -q -s 'http://%s/dial?request=%s&protocol=%s&host=%s&port=%d&tries=1'", ipPort, dialCmd, protocol, targetIP, targetPort) } </ins> // DialFromContainer executes a curl via kubectl exec in a test container, // which might then translate to a tcp or udp request based on the protocol // argument in the url.

// pod and confirm it doesn't show up as an endpoint. func (config *NetworkingTestConfig) DialFromContainer(protocol, dialCommand, containerIP, targetIP string, containerHTTPPort, targetPort, maxTries, minTries int, expectedResponses sets.String) { ipPort := net.JoinHostPort(containerIP, strconv.Itoa(containerHTTPPort)) <del> // The current versions of curl included in CentOS and RHEL distros // misinterpret square brackets around IPv6 as globbing, so use the -g // argument to disable globbing to handle the IPv6 case. cmd := fmt.Sprintf("curl -g -q -s 'http://%s/dial?request=%s&protocol=%s&host=%s&port=%d&tries=1'", ipPort, dialCommand, protocol, targetIP, targetPort) </del> <ins> cmd := makeCURLDialCommand(ipPort, dialCommand, protocol, targetIP, targetPort) </ins> responses := sets.NewString() for i := 0; i < maxTries; i++ { <del> stdout, stderr, err := config.f.ExecShellInPodWithFullOutput(config.TestContainerPod.Name, cmd) </del> <ins> resp, err := config.GetResponseFromContainer(protocol, dialCommand, containerIP, targetIP, containerHTTPPort, targetPort) </ins> if err != nil { <del> // A failure to kubectl exec counts as a try, not a hard fail. // Also note that we will keep failing for maxTries in tests where // we confirm unreachability. framework.Logf("Failed to execute %q: %v, stdout: %q, stderr %q", cmd, err, stdout, stderr) } else { var output map[string][]string if err := json.Unmarshal([]byte(stdout), &output); err != nil { framework.Logf("WARNING: Failed to unmarshal curl response. Cmd %v run in %v, output: %s, err: %v", cmd, config.HostTestContainerPod.Name, stdout, err) continue } for _, response := range output["responses"] { trimmed := strings.TrimSpace(response) if trimmed != "" { responses.Insert(trimmed) } </del> <ins> continue } for _, response := range resp.Responses { trimmed := strings.TrimSpace(response) if trimmed != "" { responses.Insert(trimmed) </ins> } } framework.Logf("Waiting for responses: %v", expectedResponses.Difference(responses))

framework.Logf("Failed to execute %q: %v, stdout: %q, stderr: %q", cmd, err, stdout, stderr) } else { framework.Logf("Tries: %d, in try: %d, stdout: %v, stderr: %v, command run in: %#v", tries, i, stdout, stderr, config.HostTestContainerPod) <del> var output map[string][]string </del> <ins> var output NetexecDialResponse </ins> if err := json.Unmarshal([]byte(stdout), &output); err != nil { framework.Logf("WARNING: Failed to unmarshal curl response. Cmd %v run in %v, output: %s, err: %v", cmd, config.HostTestContainerPod.Name, stdout, err) continue } <del> for _, hostName := range output["responses"] { </del> <ins> for _, hostName := range output.Responses { </ins> trimmed := strings.TrimSpace(hostName) if trimmed != "" { eps.Insert(trimmed)

return eps, nil } <ins> // GetResponseFromContainer executes a curl via kubectl exec in a container. func (config *NetworkingTestConfig) GetResponseFromContainer(protocol, dialCommand, containerIP, targetIP string, containerHTTPPort, targetPort int) (NetexecDialResponse, error) { ipPort := net.JoinHostPort(containerIP, strconv.Itoa(containerHTTPPort)) cmd := makeCURLDialCommand(ipPort, dialCommand, protocol, targetIP, targetPort) stdout, stderr, err := config.f.ExecShellInPodWithFullOutput(config.TestContainerPod.Name, cmd) if err != nil { // A failure to kubectl exec counts as a try, not a hard fail. // Also note that we will keep failing for maxTries in tests where // we confirm unreachability. framework.Logf("Failed to execute %q: %v, stdout: %q, stderr: %q", cmd, err, stdout, stderr) return NetexecDialResponse{}, err } var output NetexecDialResponse if err := json.Unmarshal([]byte(stdout), &output); err != nil { framework.Logf("WARNING: Failed to unmarshal curl response. Cmd %v run in %v, output: %s, err: %v", cmd, config.HostTestContainerPod.Name, stdout, err) return NetexecDialResponse{}, err } return output, nil } // GetResponseFromTestContainer executes a curl via kubectl exec in a test container. func (config *NetworkingTestConfig) GetResponseFromTestContainer(protocol, dialCommand, targetIP string, targetPort int) (NetexecDialResponse, error) { return config.GetResponseFromContainer(protocol, dialCommand, config.TestContainerPod.Status.PodIP, targetIP, testContainerHTTPPort, targetPort) } </ins> // DialFromNode executes a tcp or udp request based on protocol via kubectl exec // in a test container running with host networking. // - minTries is the minimum number of curl attempts required before declaring

// GetEndpointNodes returns a map of nodenames:external-ip on which the // endpoints of the Service are running. func (j *TestJig) GetEndpointNodes() (map[string][]string, error) { <del> nodes, err := e2enode.GetBoundedReadySchedulableNodes(j.Client, MaxNodesForEndpointsTests) </del> <ins> nodes, err := j.ListNodesWithEndpoint() </ins> if err != nil { return nil, err } <del> epNodes, err := j.GetEndpointNodeNames() </del> <ins> nodeMap := map[string][]string{} for _, node := range nodes { nodeMap[node.Name] = e2enode.GetAddresses(&node, v1.NodeExternalIP) } return nodeMap, nil } // ListNodesWithEndpoint returns a list of nodes on which the // endpoints of the given Service are running. func (j *TestJig) ListNodesWithEndpoint() ([]v1.Node, error) { nodeNames, err := j.GetEndpointNodeNames() </ins> if err != nil { return nil, err } <del> nodeMap := map[string][]string{} for _, n := range nodes.Items { if epNodes.Has(n.Name) { nodeMap[n.Name] = e2enode.GetAddresses(&n, v1.NodeExternalIP) </del> <ins> ctx := context.TODO() allNodes, err := j.Client.CoreV1().Nodes().List(ctx, metav1.ListOptions{}) if err != nil { return nil, err } epNodes := make([]v1.Node, 0, nodeNames.Len()) for _, node := range allNodes.Items { if nodeNames.Has(node.Name) { epNodes = append(epNodes, node) </ins> } } <del> return nodeMap, nil </del> <ins> return epNodes, nil </ins> } // GetEndpointNodeNames returns a string set of node names on which the

package network import ( <del> "bytes" </del> "context" "encoding/json" "errors"

framework.Failf("Service HealthCheck NodePort still present") } <del> endpointNodeMap, err := jig.GetEndpointNodes() </del> <ins> epNodes, err := jig.ListNodesWithEndpoint() </ins> framework.ExpectNoError(err) <del> noEndpointNodeMap := map[string][]string{} </del> <ins> // map from name of nodes with endpoint to internal ip // it is assumed that there is only a single node with the endpoint endpointNodeMap := make(map[string]string) // map from name of nodes without endpoint to internal ip noEndpointNodeMap := make(map[string]string) for _, node := range epNodes { ips := e2enode.GetAddresses(&node, v1.NodeInternalIP) if len(ips) < 1 { framework.Failf("No internal ip found for node %s", node.Name) } endpointNodeMap[node.Name] = ips[0] } </ins> for _, n := range nodes.Items { <del> if _, ok := endpointNodeMap[n.Name]; ok { continue </del> <ins> ips := e2enode.GetAddresses(&n, v1.NodeInternalIP) if len(ips) < 1 { framework.Failf("No internal ip found for node %s", n.Name) } if _, ok := endpointNodeMap[n.Name]; !ok { noEndpointNodeMap[n.Name] = ips[0] </ins> } <del> noEndpointNodeMap[n.Name] = e2enode.GetAddresses(&n, v1.NodeExternalIP) </del> } <ins> framework.ExpectNotEqual(len(endpointNodeMap), 0) framework.ExpectNotEqual(len(noEndpointNodeMap), 0) </ins> svcTCPPort := int(svc.Spec.Ports[0].Port) svcNodePort := int(svc.Spec.Ports[0].NodePort) ingressIP := e2eservice.GetIngressPoint(&svc.Status.LoadBalancer.Ingress[0]) path := "/clientip" <ins> dialCmd := "clientip" </ins> <del> ginkgo.By(fmt.Sprintf("endpoints present on nodes %v, absent on nodes %v", endpointNodeMap, noEndpointNodeMap)) for nodeName, nodeIPs := range noEndpointNodeMap { ginkgo.By(fmt.Sprintf("Checking %v (%v:%v%v) proxies to endpoints on another node", nodeName, nodeIPs[0], svcNodePort, path)) GetHTTPContent(nodeIPs[0], svcNodePort, e2eservice.KubeProxyLagTimeout, path) } </del> <ins> config := e2enetwork.NewNetworkingTestConfig(f, false, false) </ins> <del> for nodeName, nodeIPs := range endpointNodeMap { ginkgo.By(fmt.Sprintf("checking kube-proxy health check fails on node with endpoint (%s), public IP %s", nodeName, nodeIPs[0])) var body bytes.Buffer pollfn := func() (bool, error) { result := e2enetwork.PokeHTTP(nodeIPs[0], healthCheckNodePort, "/healthz", nil) if result.Code == 0 { </del> <ins> ginkgo.By(fmt.Sprintf("endpoints present on nodes %v, absent on nodes %v", endpointNodeMap, noEndpointNodeMap)) for nodeName, nodeIP := range noEndpointNodeMap { ginkgo.By(fmt.Sprintf("Checking %v (%v:%v/%v) proxies to endpoints on another node", nodeName, nodeIP[0], svcNodePort, dialCmd)) _, err := GetHTTPContentFromTestContainer(config, nodeIP, svcNodePort, e2eservice.KubeProxyLagTimeout, dialCmd) framework.ExpectNoError(err, "Could not reach HTTP service through %v:%v/%v after %v", nodeIP, svcNodePort, dialCmd, e2eservice.KubeProxyLagTimeout) } for nodeName, nodeIP := range endpointNodeMap { ginkgo.By(fmt.Sprintf("checking kube-proxy health check fails on node with endpoint (%s), public IP %s", nodeName, nodeIP)) var body string pollFn := func() (bool, error) { // we expect connection failure here, but not other errors resp, err := config.GetResponseFromTestContainer( "http", "healthz", nodeIP, healthCheckNodePort) if err != nil { return false, nil } if len(resp.Errors) > 0 { </ins> return true, nil } <del> body.Reset() body.Write(result.Body) </del> <ins> if len(resp.Responses) > 0 { body = resp.Responses[0] } </ins> return false, nil } <del> if pollErr := wait.PollImmediate(framework.Poll, e2eservice.TestTimeout, pollfn); pollErr != nil { </del> <ins> if pollErr := wait.PollImmediate(framework.Poll, e2eservice.TestTimeout, pollFn); pollErr != nil { </ins> framework.Failf("Kube-proxy still exposing health check on node %v:%v, after ESIPP was turned off. body %s", <del> nodeName, healthCheckNodePort, body.String()) </del> <ins> nodeName, healthCheckNodePort, body) </ins> } }

return body } <ins> // GetHTTPContentFromTestContainer returns the content of the given url by HTTP via a test container. func GetHTTPContentFromTestContainer(config *e2enetwork.NetworkingTestConfig, host string, port int, timeout time.Duration, dialCmd string) (string, error) { var body string pollFn := func() (bool, error) { resp, err := config.GetResponseFromTestContainer("http", dialCmd, host, port) if err != nil || len(resp.Errors) > 0 || len(resp.Responses) == 0 { return false, nil } body = resp.Responses[0] return true, nil } if pollErr := wait.PollImmediate(framework.Poll, timeout, pollFn); pollErr != nil { return "", pollErr } return body, nil } </ins> // DescribeSvc logs the output of kubectl describe svc for the given namespace func DescribeSvc(ns string) { framework.Logf("nOutput of kubectl describe svc:n")