Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We ran sonobuoy on one of our clusters running on AWS and it reported two failed tests: This happened because the pod employed to conduct the tests wasn't able to reach the service using the node's public IP address: What you expected to happen: The security groups on that cluster don't allow for connections to arbitrary ports through the public IP address. The tests should still succeed because that configuration is seen as a security feature. How to reproduce it (as minimally and precisely as possible): Spin up a Kubernetes cluster on AWS EC2 instances, configure SGs to only allow traffic to the instances from other instances (or from an ELB). Run sonobuoy. Anything else we need to know?: Environment: AWS Kubernetes version (use ): 1.16.8 Cloud provider or hardware configuration: AWS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area conformance /sig network
related:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I am also seeing this issue when running conformance testing. The nodes in our test cluster (where we run conformance tests) have public IP addresses, but nearly all ports all firewalled. We are also running in AWS. These tests should only rely on the internal IP address of the nodes.
/assign
Yes, I have seen this before, that if the nodes have an external address it will use that one, which means the firewall must be open for that. If the nodes do not have an external IP then the test will pass just fine. It's not as much a bug as an inconvenient implementation. But we can treat it as a bug and fix it. The intent of the test is to make sure the service is plumbed properly.
/remove-triage unresolved
Good point. However, we only assign an external IP address to be able to SSH into the host and thus block all other ports.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm seeing the same issue, though I noticed that it happens only on one node in the whole cluster. Even after re-creating a cluster from scratch, the test always fails only on one specific node. However this node is identical to others, but the test succeed on rest of the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
In some user scenarios, the external ip nodeport is for access from APPs outside this cluster/zone. However, in most scenarios, external IP is just for SSH or special use(Ingress Gateway, etc.) Different users have different proposals. Why not disable e2e tests checking on external IP with a flag? Will send a PR for this.
That's not all that useful for those running conformance tests. Would be better to change it to always use the internal IP, we don't need the external access for conformance purposes.
/assign
agree, also most of the network tests now run from the cluster, no need to reach externalIPs please tag me in the PR
So kubernetes applications should not expect to be able to access them with an external node address? FWIW we also require upstream / public internet access in the DNS tests, so needing somewhat unrestricted networking to test the cluster behavior is not new.
I'm not extremely familiar with the conformance tests. Are the DNS tests you are referring to about being able to access public services from inside of the cluster (egress)? Because our problem is that our default Kubernetes deployment doesn't allow ingress other than SSH and we haven't seen any problems with the DNS tests so far when checking our product for conformance. That's why I like approach of disabling that specific test with a flag. Maybe that could also be generalized to let all conformance tests know this is a locked-up cluster with basically no ingress. But that may be a matter of a subsequent PR/issue.
Disabling the tests with a flag is not really an acceptable approach for conformance. The whole point of the conformance program is to validate the Kubernetes implementation against a standard set of tests across all vendors etc. If we don't think access like this is a standard Kubernetes behavior applications should expect anymore, the tests should be removed from conformance and we can consider instead running them just to test the Kubernetes project but not in conformance by moving them to a new test. IMHO we usually only increase the scope of conformance behavior though, not decrease, and it's expected that some users will have more secure clusters that fail conformance (for example we test some privileged behaviors IIRC) ...
After reading comment, I agree that the test should just be removed. However, I'd like to first understand the reasoning behind it being there in the first place. Maybe there's a good reason that such "secured cluster" aren't deemed compliant?!
so, the next step is to review the conformance definition for nodeport and modify the test accordingly Node.Status.Addresses are not a great thing, bad semantics, ... this KEP and this PR summaries perfectly why I personally, would move ExternalIP from Conformance entirely until it has a clear definition
So we should remove it now and then add a new test if needed after a conclusion from the discussion in the KEP. Edited. I may misunderstand. A proposal is to move those external addr NodePort test from conformance to normal e2e test cases In Conformance Case, don't check external ip's NodePort. In e2e test case, check them.
My point is that Conformance folks (cc ) should decide if they want to consider ExternalIPs as part of the conformance definition, I pointed to some issues related to the topic here sig-network will keep testing externalIPs, are part of the API and we need to verify them, Conformance has a narrower scope
The fact that these tests are using ExternalIPs does not need to be in conformance and in fact really shouldn't be. But we still need the tests working against InternalIPs for conformance. It's more of an historical artifact that they use ExternalIPs than an intentional choice. So, agree with that sig-network should test them, but we should remove the use of the external IPs from conformance, and leave in the tests but against the internal ips only. I resolved this, for example, in GKE conformance runs by using a private GKE cluster that does not give the nodes external IPs.

Discussion:
Name      string ID        string Labels    map[string]string <ins> // ExternalIPs should be false for Conformance test // Don't check nodeport on external addrs in conformance test, but in e2e test. ExternalIPs bool </ins> } // NewTestJig allocates and inits a new TestJig.

return testEndpointReachability(externalIP, sp.Port, sp.Protocol, execPod) } <del> func testReachabilityOverNodePorts(nodes *v1.NodeList, sp v1.ServicePort, pod *v1.Pod, clusterIP string) error { </del> <ins> func testReachabilityOverNodePorts(nodes *v1.NodeList, sp v1.ServicePort, pod *v1.Pod, clusterIP string, externalIPs bool) error { </ins> internalAddrs := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) <del> externalAddrs := e2enode.CollectAddresses(nodes, v1.NodeExternalIP) </del> isClusterIPV4 := utilsnet.IsIPv4String(clusterIP) for _, internalAddr := range internalAddrs {

return err } } <del> for _, externalAddr := range externalAddrs { if isClusterIPV4 != utilsnet.IsIPv4String(externalAddr) { framework.Logf("skipping testEndpointReachability() for external adddress %s as it does not match clusterIP (%s) family", externalAddr, clusterIP) continue } err := testEndpointReachability(externalAddr, sp.NodePort, sp.Protocol, pod) if err != nil { return err </del> <ins> if externalIPs { externalAddrs := e2enode.CollectAddresses(nodes, v1.NodeExternalIP) for _, externalAddr := range externalAddrs { if isClusterIPV4 != utilsnet.IsIPv4String(externalAddr) { framework.Logf("skipping testEndpointReachability() for external adddress %s as it does not match clusterIP (%s) family", externalAddr, clusterIP) continue } err := testEndpointReachability(externalAddr, sp.NodePort, sp.Protocol, pod) if err != nil { return err } </ins> } } return nil

if err != nil { return err } <del> err = testReachabilityOverNodePorts(nodes, servicePort, pod, clusterIP) </del> <ins> err = testReachabilityOverNodePorts(nodes, servicePort, pod, clusterIP, j.ExternalIPs) </ins> if err != nil { return err }

ginkgo.By("creating a TCP service " + serviceName + " with type=ClusterIP in namespace " + ns) jig := e2eservice.NewTestJig(cs, ns, serviceName) <ins> jig.ExternalIPs = true </ins> servicePort := 8080 tcpService, err := jig.CreateTCPServiceWithPort(nil, int32(servicePort)) framework.ExpectNoError(err)

ginkgo.By("creating a TCP service " + serviceName + " with type=ClusterIP in namespace " + ns) jig := e2eservice.NewTestJig(cs, ns, serviceName) <ins> jig.ExternalIPs = true </ins> servicePort := 8080 svc, err := jig.CreateTCPServiceWithPort(nil, int32(servicePort)) framework.ExpectNoError(err)

} jig := e2eservice.NewTestJig(cs, ns, serviceName) <ins> jig.ExternalIPs = true </ins> ginkgo.By("creating service " + serviceName + " with type=clusterIP in namespace " + ns) clusterIPService, err := jig.CreateTCPService(func(svc *v1.Service) {

serviceName := "nodeport-update-service" ns := f.Namespace.Name jig := e2eservice.NewTestJig(cs, ns, serviceName) <ins> jig.ExternalIPs = true </ins> ginkgo.By("creating a TCP service " + serviceName + " with type=ClusterIP in namespace " + ns) tcpService, err := jig.CreateTCPService(nil)