A Parable of Pricing, Processors, and Memory
By: Dan Collins
Tags: Engineering, Debugging, Kubernetes, Peak6, Architecture

This shows the pricing system running across all 80 logical CPUs with fairly even distribution and is exactly what we’d expect the behavior to look like. There were a few sighs of relief seeing the expected behavior, but more than a few furrowed brows as we now wanted to know what was different between environments.

Pricing In A Box

I should mention that most of PEAK6’s applications rely on Kubernetes and are containerized. Moving our standalone apps to containers and using an orchestrator like Kubernetes means that we have a common interface and application runtime across all of our systems. We have around 200 nodes across four geographically isolated data centers running around 1,500 applications at any given time. Additionally, as we have systems like pricing and use some low-level hardware features for things like shared memory and multicast, we do end up with some highly tailored Kubernetes configurations. These allow us to continue using the same tooling, monitoring, etc that Kubernetes gives us while fully taking advantage of all the hardware we have available.

Now, in our development environments, we typically launch applications outside of docker as we want to do local testing, debugging, etc, without building images. The down side is, we may run into differences in application behavior depending on whether the application runs locally or within docker. When we tested the pricing system in the development environment as shown above, we tested both natively and using a containerized version of the application. In both cases, the pricing system had a fairly even load distribution across CPUs. So this was another variable we could put to rest, … or so we thought.

7 Out of 5 Whys

Getting stumped while triaging software behavior isn’t a great feeling. If anything, computers are predictable as they only do as instructed. Although, there are many factors that make behavior non-deterministic, emergent, and difficult to understand. Kerninghan’s law, “debugging is twice as hard as writing code in the first place,” carries weight up and down the stack.

At times like these, the best thing you can do is go full detective and review your case notes. Let’s recap what we’ve seen so far:

UAT pricing systems with new hardware show an every-other CPU load distribution

Production pricing systems with existing hardware show an every-other CPU load distribution

In UAT, the behavior is invariant to HyperThreading

In Development pricing systems show even CPU load distribution natively and containerized

It was in reviewing the above, that we realized that in each case we saw every other CPU being utilized, it was always every odd or every even CPU. Why not all CPUs for specific processors? Why not just one processor? This led us to look more at the processor stats where we saw the following:

Processor Stats

Does it jump out at you? Every other CPU either odd or even lines up with NUMA nodes!

NUMA and You

The first thing you need to know about NUMA is that it’s really neat. It’s a way for processors to use local memory as they multitask executing threads across processors on a system. As long as the same workloads end up being scheduled on the same cores, the memory will be locally available and execution time reduced. As an example, in a four-core system running with two threads per core, it might look like the following:

In cases where the scheduler moves a thread across cores and memory is no longer available locally, there is still fairly efficient access to the remote memory within the same NUMA node. In the worst-case scenario, a thread gets scheduled on a processor that has to cross from one NUMA node to another via an “interconnect”, which has the highest latency. On a Linux system, the completely fair scheduler (CFS) will attempt to allocate threads using a local node first. In the case a thread cannot be scheduled locally, the scheduler will migrate the thread, but will not necessarily migrate the memory, as seen below where core 2 reads across the bus remotely for memory that’s available locally for core 3.

Diving deeper, we used the numastat tool to get more info into how NUMA was performing and the trail got hot as we saw the following:

Here the numa_miss and numa_foreign tell us that our pricing system is running almost exclusively on node0. It also explains why threads running non-locally are not executing as efficiently and the workload is unbalanced. If we look at our dev environment, we see a much better profile:

That’s no numa_miss or numa_foreign and almost 1:1 local_node memory use. This gets us to the root of the behavior being inefficient thread scheduling across NUMA-bound processors. However, it doesn’t explain why this was happening.

As a note, I’m simplifying here a bit as there’s more complexity with scheduling algorithms, balancing latency targets, time slicing, etc that can all be tuned and could all be their own posts.

Great Scott! It’s Kubernetes!

Me: Knock, Knock You: Who’s there? Me: Kubelet You: Kubelet who? Me: Kubelet gonna schedule all your containers’ threads randomly

Let’s recap the facts again:

UAT pricing systems with new hardware show an every-other CPU load distribution

Production pricing systems with existing hardware show an every-other CPU load distribution

In UAT, the behavior is invariant to HyperThreading

In Development pricing systems show even CPU load distribution natively and containerized

UAT and Production systems show NUMA misses and CPU imbalances

Development systems show no NUMA misses and no CPU imbalances

The last and final clue to our case came as we compared Kubernetes pod configurations between environments looking for any additional hints for why we’d see a pricing system in development work as expected while a Kubernetes scheduled pricing system in UAT or Production continue to show a CPU imbalance.

As it turns out, our pod configurations for the pricing system did not include any CPU requests in UAT or Production. This seemed benign, but it resulted in Kubelet (Kubernetes worker node orchestrator) assigning only the minimum cpu shares to our pricing system containers. You can see this in source here in the conversion to milli cpu to shares returning MinShares.

In production, we confirmed this by reviewing cpu.shares in the kubepods cgroup:

cpu.shares == 2

More Equal Than Others

Alright, you’re thinking. I get it, there were only two cpu shares defined for these containers, but what does that have to do with NUMA and CPU balancing? Here’s where we go all the way down the rabbit hole.

Kubelet is configured to use the default CPU manager (essentially passing CPU assignment down to the OS(Linux)) Kubelet configures the containers for the pricing system with no CPU limits and two CPU shares Kubelet also configures these containers for burstable quality of service, meaning no guaranteed reservations Linux, seeing the above, divides out available CPU across all CPUs based on any affinities that are set as well as memory stickiness, etc (but the pricing system doesn’t have any) Linux, then divides out all remaining CPU cycles to processes based on CPU shares (but the pricing system comes in at a much lower priority because of the CPU shares it’s been assigned). Threads for the pricing system end up running randomly across CPUs on the system and include performance hits for both migrating threads as well as migrating memory pages across NUMA nodes.

Further evidence of this behavior can be seen by checking out /proc/vmstat which shows us virtual memory statistics including numa_pte_updates and numa_huge_pte_updates. These show that we’re updating memory to be moved within the numa nodes around 19 Billion times and 3,495 for huge pages!

This random allocation also shows up in NUMA memory usage across nodes. Instead of a more equal distribution between each of the NUMA nodes, we end up with one node ~90% allocated while the other is closer to ~60%.

A One-line Solution

Simply adding a resources.requests.cpu: “70” to the pod configuration changes everything. It allows kubelet to properly hint that the pricing system’s threads will need compute, which results in more contiguous processor allocation and better memory locality in NUMA. This one-line change completely changes the pricing system behavior giving us better system saturation across all available resources.

First, we see CPU shares appropriately set at around 72k instead of 2.

cpu.shares == 71680

Reviewing our /proc/vmstat shows a much better profile, with no huge page table updates and ~475MM moves (compared with 19B before!).

Additionally, the memory allocation is better balanced:

Finally, we see a much more even distribution across processors:

We’re rather lucky here as we see the Linux scheduler react nicely to the CPU request. Kubernetes versions continue to add better support for NUMA and CPU topology and assignment. If we were still running into issues after narrowing down CPU shares as the root cause, we could use the finer grained numactl controls or cpuset and memset configurations to continue to tweak the allocations. Thankfully, we don’t have to yet.

Outcomes

After applying our one-line fix for a CPU request, we watched our pricing system performance over the next few days in our UAT environment. The change not only corrected the CPU performance imbalance, but also resulted in about a 3x improvement in calculation speed along with a 75% reduction in overall calculation queuing during market spikes. Heck Yes!

Where to Go From Here

Hopefully you’ve gotten a taste of some of the complexity that comes with maintaining a high-performing pricing system. As we talked a little about CPU features like HyperThreading and affinity, NUMA and memory allocation, and Kubernetes, we also shared how our engineers worked together and triaged this issue from reproducing it to evaluating a fix. If you’re interested in diving into these kinds of problems or working on systems that really scale, check out our open positions. We’re always looking for passionate, curious engineers who like working on hard problems!

References

Helpful links and blogs if you want to dive deeper.