Breaking Kubernetes: How We Broke and Fixed our K8s Cluster
By: Civis Analytics
Tags: Engineering, Software Engineering, Docker, Data Science, Kubernetes

From this load test, it was obvious that Datadog and/or Sumologic were the culprit for the load. We ran a follow-up test where we added back Sumologic and the results were similar, our cluster scaled pretty seamlessly to 700+ nodes. This made us sure that Datadog was the problem.

Other Minor Adjustments

We made two other adjustments to our cluster configuration. The first was we increased the rate limit for non-mutating requests from 400 to 1200 at any given time. Combing through the access logs we saw multiple 429 status codes. We made this adjustment before we discovered the issue with our Datadog Agents. In subsequent load tests, we saw 429 responses drop but no significant change in the memory and cpu profiles for the API servers.

While rate limiting is critical for protecting API servers from load, we kept the change in place because we didn’t see any detrimental changes to the cluster. You can read up on the API server configurations here. We changed the max-requests-inflight option.

Another change we made was increasing the amout of disk IOPS for our etcd containers. In the Open AI blog, they saw a significant drop in disk write latency when they moved their master nodes to use local disk instead of network backed disk. However, Kops was having an issue with generating a correctly sized root partition on local disk backed nodes like the m5d series. Instead, we increased IOPS by increasing the underlying EBS volumes from 20 to 100 GB thus going from 100 to 300 IOPS. However, we didn’t see much improvement. Ultimately, we kept the adjustment since the cost was negligible and it didn’t hurt performance.

Setting Up Datadog For Scale

After discussing with our Datadog reps and combing through our API access logs, we learned that Datadog agents were polling the Kubernetes API for metadata about K8s services. Specifically, the agent would collect the names of all services in the cluster and any pod running on the agent’s node that was fronted by a service would have its metrics tagged by the service name. The offending configuration is DD_KUBERNETES_COLLECT_METADATA_TAGS and is listed in the README. With hundreds of Datadog Agents, one per node, hitting the K8s API every x seconds looking for new services, it was causing unnecessary load on the API. Furthermore, the tags being collected were not useful for our monitoring purposes.

Datadog has two solutions. The first is you can turn off the configuration. The second is the Datadog cluster agent, a new product they released to address scale. The cluster agent acts as a buffer between the API server and the Datadog pods so only the Datadog Cluster agent can talk to the K8s API. Now, instead of n Datadog pods, where n is the number of nodes in the cluster, hitting the API, you only have a single Cluster Agent communicating with the API. Furthermore, we were able to leverage Datadog to scrape Prometheus metrics from our control plane components and remove the overhead of maintaining our own Prometheus set up.

With the Datadog adjustments in place, we reran load tests and saw the cluster easily breeze towards 700+ EC2 instances.

Word of Advice on Auto-Discovery

A core paradigm of Kubernetes is auto-discovery. Kubernetes’ watch stream allow services to watch for changes in the cluster state allowing services to respond to changes on demand without manual intervention from system admins. However, this can come at a cost. With poor design, you can quickly introduce scaling issues with greedy services watching for changes. This is a common pattern we’ve seen with a variety of our DaemonSets.

The Datadog agents were an extreme case, but a couple of our other DaemonSets had similar issues. Kube2iam, a service for access control to EC2 metadata, watched for changes to all pods in the cluster so it could make adjustments to access controls. However, as more pods were added to the cluster, the Kube2iam containers would run out of memory because they were storing all the pods’ metadata in memory. The Fluentd Kubernetes Metadata Filter is a popular plugin that also has the same issue. Linked below is a Github pull request and issue to highlight the problems.

In general, we’ve learned to be extra cautious with the third party solutions we include in our cluster, especially DaemonSets. Often times, the fix is a minor configuration change to reduce the “greediness” of the service, but as we saw with load testing, it isn’t always immediately evident.

Future Work

We still have a lot more work to do to improve our cluster’s scalability. The biggest change we would like to implement is upgrading etcd from 2.2.1 to 3.x. The 3.x versions of etcd come with a slew of scalability upgrades that I won’t go into but are discussed here. We are waiting on Kops to come up to speed on making the migration from etcd 2.x to 3.x safe.

After re-releasing internally, users started to complain about scripts failing because of DNS timeout errors. DNS in Kubernetes is unique in that it needs to resolve canonical names within the cluster first before going out to the rest of the internet. While it was hard for us to pinpoint the exact cause of the issue since we could not reliably reproduce the issue, it almost certainly had to do with scaling our cluster. We put in a mitigating solution which was to change the DNS policy from ClusterFirst to Default for scripts. With ClusterFirst, DNS resolution is attempted internally before going out to the internet. With Default, DNS resolution circumvents internal resolution and goes directly out to the internet significantly reducing latency. This was fine for our use case since we do not allow client scripts to communicate with other workloads running within the cluster. However, in the future we would like to allow client scripts to communicate internally so we’ll have to put in work to solve this issue.

Conclusion

Since these adjustments, we have not had issues with our cluster. In the end, we concluded that our cluster could support 700 nodes. Our biggest takeaway from this experience is to be mindful of the third party resources you utilize in your Kubernetes cluster — many tools are built for standard usage but have not been tested at scale.