Change:
Hello sig-network, We, sig-scalability, are exploring using "private" clusters (i.e. the ones where nodes don't have public IPs) for our large (up to 5K nodes) scalability tests to avoid consuming expensive public IPs. More details in With a few minor modifications we managed to get an almost passing correctness test suite on a private cluster except the 3 sig-network tests which explicitly assume that nodes are reachable over public IPs. Failing tests (): [sig-network] Firewall rule should have correct firewall rules for e2e cluster [sig-network] Loadbalancing: L7 [Slow] Nginx should conform to Ingress spec [sig-network] Services should be able to create a functioning NodePort service Could you take a look at them and advise if it would be possible to rewrite them so they can work both in the "public" and "private" clusters? We'd really appreciate your help assistance here. Let me know if you have any questions. Thanks, Matt
please triage /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
could you take a look?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
After the conversation I had with I'm un-assigning to make it go through the regular triage process. /unassign It's not a super high priority, but would be great if someone from the sig-network team could take a look. This effort could potentially result in releasing over 10K of public IPs.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
want to track this one with the release team as well as it is showing up in /priority critical-urgent /milestone v1.16
could we get someone to look into these issues or get an update of when we might get some news?
Reiterating the mentions to trigger a notification: <detailsIn response to : could we get someone to look into these issues or get an update of when we might get some news? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
should we add a new TAG on tests that does not apply to private clusters?
would adding tags to the tests skip private clusters entirely? I would think ideally private clusters should still have tests... if that's correct who would be responsible for adding more tests?
Hey Folks, sorry I must have missed updates here. I don't think that this should block 1.16 in any way nor it should be marked as priority/critical-urgent. This is an experimental test suite that we run in order to stop using public ips in scalability tests. What's more only 3 out of over 800 tests fail for private clusters and I'm quite confident that these tests don't fail because of a bug in kubernetes, but rather because they are written in a way that assumes that each node will have a public IP. The only action item here is to rewrite the tests and make them compatible with private clusters. I opened the bug to get some insights from sig/network on that as all failing tests belong to them. Removing the milestone and changing the priority, let me know if someone doesn't agree. /priority important-soon /milestone clear
/remove-priority critical-urgent
Thank you for the updates. I understand the test suite is experimental and there is a high level of confidence that the handful of failing sig-network jobs are failing due to the test suite and not to Kubernetes itself but given that this affects the quality of the signal we get from CI signal for this release we want to maintain it in the 1.16 milestone. could you bring this up with (don't see a group for test failures)? /milestone v1.16
friendly ping about this, would you be able to assign the correct person to write tests for this?
Let's bring this up in the sig-network meeting to see if we have any volunteers
Any updates on this issue?

Discussion:
// NginxIngressController manages implementation details of Ingress on Nginx. type NginxIngressController struct { <del> Ns         string rc         *v1.ReplicationController pod        *v1.Pod Client     clientset.Interface externalIP string </del> <ins> Ns     string rc     *v1.ReplicationController pod    *v1.Pod Client clientset.Interface </ins> } // Init initializes the NginxIngressController func (cont *NginxIngressController) Init() { <ins> // Set up a LoadBalancer service in front of nginx ingress controller and pass it via // --publish-service flag (see <IngressManifestPath>/nginx/rc.yaml) to make it work in private // clusters, i.e. clusters where nodes don't have public IPs. framework.Logf("Creating load balancer service for nginx ingress controller") serviceJig := e2eservice.NewTestJig(cont.Client, "nginx-ingress-lb") serviceJig.CreateTCPServiceOrFail(cont.Ns, func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeLoadBalancer svc.Spec.Selector = map[string]string{"k8s-app": "nginx-ingress-lb"} svc.Spec.Ports = []v1.ServicePort{ {Name: "http", Port: 80}, {Name: "https", Port: 443}, {Name: "stats", Port: 18080}} }) svc := serviceJig.WaitForLoadBalancerOrFail(cont.Ns, "nginx-ingress-lb", e2eservice.LoadBalancerCreateTimeoutDefault) serviceJig.SanityCheckService(svc, v1.ServiceTypeLoadBalancer) </ins> read := func(file string) string { return string(testfiles.ReadOrDie(filepath.Join(IngressManifestPath, "nginx", file))) }

framework.Failf("Failed to find nginx ingress controller pods with selector %v", sel) } cont.pod = &pods.Items[0] <del> cont.externalIP, err = framework.GetHostExternalAddress(cont.Client, cont.pod) framework.ExpectNoError(err) framework.Logf("ingress controller running in pod %v on ip %v", cont.pod.Name, cont.externalIP) </del> <ins> framework.Logf("ingress controller running in pod %v", cont.pod.Name) </ins> } func generateBacksideHTTPSIngressSpec(ns string) *networkingv1beta1.Ingress {

args: - /nginx-ingress-controller - --default-backend-service=kube-system/default-http-backend <ins> - --publish-service=$(POD_NAMESPACE)/nginx-ingress-lb </ins>