Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: For these 2 test suites visible in the testgrid at network-gce : http://k8s- http://k8s- the testsuite cannot setup correctly the cluster, due to a firewall issue (I think) See part of the build log below: What you expected to happen: I would think that setup the cluster is part of the process of the test-infra job, not the e2e test itself. Any missing flag or option on the job definition (see below) ? Need guidance on how to find the root cause of this error. How to reproduce it (as minimally and precisely as possible): Just look at the testgrid, it happen every 6 hours. http://k8s- http://k8s- Anything else we need to know?: job definition is: NOTE: this job definition was copied from another one that seems working fine: Environment: Kubernetes version (use ): current (branch master) Cloud provider or hardware configuration: gce
: I finally create this issue to get help. , : any idea on why the firewall rules cannot be set ? Could you indicate how to investigate this issue ? Thanks..
/sig network /sig scalability
I think it's because the name is too long. There's a limit of 64 chars or sth for resource names on GCP. Try changing the --cluster arg to or sth.
proposed a fix here : it is now merge. Need to verify the next run of these testsuites within 0 to 6 hours from now. : Thanks for your advise !
After updating the name of the cluster (shorter). The Cluster is now setup correctly. The PerformanceDNS test itself is failing - unable to create he 10k services. We believe that is an issue on the number of 10k services that k8s cannot support with this configuration of cluster. Next step is to decrease the number of services to 5k - a number we tested locally and know that k8s support.
Can you tell the reason why it's unable to create so many services?
I suspect that the cluster cannot operate correctly with that many services. It can create the services. But the cluster acts undesirably after they are created. In my local tests on a single node cluster when I ramped up the number of services to 10K, I saw the same mode of failure. With 10K services, the cluster is unable to spin up the "util" pod. The "util" pod is required in the test to query the DNS from the context of a pod - if it's not running then the test cannot proceed.
But do you know what exactly is happening? What do you mean by "cluster is unable to spin up the "util" pod"? Is the pod not even being created? Or maybe is it scheduled, but kubelet is not starting it? Having more insight would help debugging it (maybe it's just size of master machine that is too small).
I dont for sure. But... According to the logs, I think the pod is successfully created. But when we try to run DNS queries, it's the container in the pod named "util" that cannot be found - not necessarily the pod itself. So, it seems the pod may be still starting up at the time - perhaps slowed by the number of services present. Maybe if I create the pod before creating the services it will help.
Let's try to understand this better. If it's that creating services is limiting us on the api qps, then starting the pod may potentially be slowed down (due to shortage of qps). Your idea sounds good, or you might alternatively try to slow down the rate of service creation. In general, I believe it shouldn't be a problem to create so many empty services in the cluster.
Please paste link to logs from that cluster (run from testgrid) - so that others can take a look into it.
OK, these are logs of the failure when pod is created after services... creates services OK I believe (this step takes just under 10 minutes). Then it creates the client pod... Looks OK? - client pod is created... So we start querying DNS... But the "util" container is not found, possibly because the pod is not ready yet. Perhaps the helper func that I am using doesn't wait for the pod to be ready.
I looked into kubelet logs - pods was succesfully started but then dies (crashed?) - something seems to be wrong on your side. Logs from here:
It waits for it to be ready, but it dies after that.
OK, thanks but this isnt a pod I have invented for this test. It is already used for testing DNS in other existing tests. So I was expecting it to work. And it does work fine if i reduce the number of services - which I'll go ahead and do.
Let's not down the number of service without understanding why it fails. We need to understand that.
Does anyone know if the 10K limit on services is a hard/enforced limit? I didnt think it was (but thats just an assumption) - but if it is enforced somewhere, and exceeding it causes silent failures, then i'll need to reduce the number of services by a "few" to account for the other already existing services (e.g. the cluster dns service, api, etc...)
It's not a hard limit.
I managed to get this tested locally with 10K services on a single node.... where I create the pod before the services, and it passes. OK if I submit a PR with that change? Or do we want to first try to understand why creating the client pod after services causes a failure?
If you can explain why this works now and didn't work before, I'm fine - otherwise there is a risk that it may e.g. be flaky. We should understand it before blindly changing that.
OK - so I've run it and catched that pod for you. The reason why it is crashlooping is the following log: It just can't initialize itself due to too long argument list. - So I think that the fix should be different. Apparently we are not able to have that many services in a single namespace, because env vars that we provide for all pods (with services from that namespace) are too long in total. So I think what we should do is to: uncover how many services we are able to have in a single namespace (so binary search on number of those should do the job) split those 10k service over multiple namespaces Does that make sense to you? - it's the limitation that recently poped up in our conversations ^^
Thanks Wojtek! And that is why I said we should make use of the experiments dns folks are already doing, to catch that limitation (and we indeed seem to have caught it) :) On Thu, Jun 7, 2018, 10:47 AM Wojciech Tyczynski <:
I'll update that discussion once the value is figured out. Btw - thanks for digging. On Thu, Jun 7, 2018, 11:03 AM Shyam Jeedigunta <: The reason why it is crashlooping is the following log: exec user process caused "argument list too long" It just can't initialize itself due to too long argument list. So I think that the fix should be different. Apparently we are not able to have that many services in a single namespace, because env vars that we provide for all pods (with services from that namespace) are too long in total. So I think what we should do is to: uncover how many services we are able to have in a single namespace (so binary search on number of those should do the job) split those 10k service over multiple namespaces Does that make sense to you? it's the limitation that recently poped up in our conversations ^^ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
FYI - I just run test with 5000 services and it seemed to work fine (I did just a simple thing to reduce number of services, not split them across namespaces). So I think having 2 namespaces should be fine.
But I got it to work with 10k services in one namespace... by creating the client pod before creating the services. On Jun 7, 2018, at 5:47 AM, Wojciech Tyczynski <mailto:: FYI - I just run test with 5000 services and it seemed to work fine (I did just a simple thing to reduce number of services, not split them across namespaces). So I think having 2 namespaces should be fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread
- IIUC that's because you created the pod even before creating the services (so it wasn't passed those service env vars while creation).
OK cool. Seems like the limit would need to be here ... Several of the entries are "per ns" but there isnt one for services. As an unrelated side note: I just noticed that the limit on the number of deployments per ns in the thresholds doc lists 20K, with a long term goal of 10K. Is that a typo? Or is the long term goal to reduce the limit?
Sent for it. Seems so to me (should've probably been 10^5?). cc
Yeah - I think two values are incorrect wrt it: pods/namespace - I think we've never run 15k deployments/namespace - it doesn't make much sense to support more deployments than pods Will send out a PR for it.

Discussion:
const ( parallelCreateServiceWorkers = 1 maxServicesPerCluster        = 10000 <ins> maxServicesPerNamespace      = 5000 </ins> checkServicePercent          = 0.05 )

// answers dns for service - creates the maximum number of services, and then check dns record for one It("Should answer DNS query for maximum number of services per cluster", func() { <del> services := generateServicesInNamespace(f.Namespace.Name, maxServicesPerCluster) </del> <ins> // get integer ceiling of maxServicesPerCluster / maxServicesPerNamespace numNs := (maxServicesPerCluster + maxServicesPerNamespace - 1) / maxServicesPerNamespace var namespaces []string for i := 0; i < numNs; i++ { ns, _ := f.CreateNamespace(f.BaseName, nil) namespaces = append(namespaces, ns.Name) } services := generateServicesInNamespaces(namespaces, maxServicesPerCluster) </ins> createService := func(i int) { defer GinkgoRecover() <del> framework.ExpectNoError(testutils.CreateServiceWithRetries(f.ClientSet, f.Namespace.Name, services[i])) </del> <ins> framework.ExpectNoError(testutils.CreateServiceWithRetries(f.ClientSet, services[i].Namespace, services[i])) </ins> } framework.Logf("Creating %v test services", maxServicesPerCluster) workqueue.Parallelize(parallelCreateServiceWorkers, len(services), createService)

}) }) <del> func generateServicesInNamespace(namespace string, num int) []*v1.Service { </del> <ins> func generateServicesInNamespaces(namespaces []string, num int) []*v1.Service { </ins> services := make([]*v1.Service, num) for i := 0; i < num; i++ { services[i] = &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "svc-" + strconv.Itoa(i), <del> Namespace: namespace, </del> <ins> Namespace: namespaces[i%len(namespaces)], </ins> }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{{