Project: gVisor (Runtime / Container Runtime)
File: 2021-12-02-running-gvisor-in-production-at-scale-in-ant.md

This post was contributed by , a large-scale digital payment platform. Jianfeng and Yong are engineers at Ant Group working on infrastructure systems, and contributors to gVisor. At Ant Group, we are committed to keep online transactions safe and efficient. Continuously improving security for potential system-level attacks is one of many measures. As a container runtime, gVisor provides container-native security without sacrificing resource efficiency. Therefore, it has been on our radar since it was released. However, there have been performance concerns raised by members of and . Users of gVisor tend to bear the extra overhead as the tax of security. But we tend to agree that . In this article, we will present how we identified bottlenecks in gVisor and unblocked large-scale production adoption. Our main focus are the CPU utilization and latency overhead it brings. Small memory footprint is also a valued goal, but not discussed in this blog. As a result of these efforts and community improvements, 70% of our applications running on runsc have <1% overhead; another 25% have <3% overhead. Some of our most valued application are the focus of our optimization, and get even better performance compared with runc. The rest of this blog is organized as follows: * First, we analyze the cost of different syscall paths in gVisor. * Then, a way to profile a whole picture of a instance is proposed to find out if some slow syscall paths are encountered. Some invisible overhead in Go runtime is discussed. * At last, a short summary on performance optimization with some other factors on production adoption. For convenience of discussion, we are targeting KVM-based, or hypervisor-based platforms, unless explicitly stated. is the key design principle of gVisor. In gVisor, different syscalls have different paths, further leading to different cost (orders of magnitude) on latency and CPU consumption. Here are the syscall paths in gVisor. Sentry provides a for its sandboxed processes. Several syscalls are short circuited and implemented in user space. These syscalls cost almost as much as native Linux. But note that the vDSO library is partially implemented. We once noticed some in our environment are not properly terminated in user space. We create some additional implementations to the vDSO, and aim to push these improvements upstream when possible. Most syscalls, e.g., <code>clone(2)</code>, are implemented in Sentry. They are some basic abstractions of a operating system, such as process/thread lifecycle, scheduling, IPC, memory management, etc. These syscalls and all below suffer from a structural cost of syscall interception. The overhead is about 800ns while that of the native syscalls is about 70ns. We'll dig it further below. Syscalls of this kind spend takes about several microseconds, which is competitive to the corresponding native Linux syscalls. Some syscalls, resource related, e.g., read/write, are redirected into the host kernel. Note that gVisor never passes through application syscalls directly into host kernel for functional and security reasons. So comparing to native Linux, time spent in Sentry seems an extra overhead. Another overhead is the way to call a host kernel syscall. Let's use kvm platform of x86_64 as an

After Sentry issues the syscall instruction, if it is in GR0, it first goes to the syscall entrypoint defined in LSTAR, and then halts to HR3 (a vmexit happens here), and exits from a signal handler, and executes syscall instruction again. We can save the "Halt to HR3" by introducing vmcall here, but there's still a syscall trampoline there and the vmexit/vmentry overhead is not trivial. Nevertheless, these overhead is not that significant. For some sentry-contained syscalls in Path 2, although the syscall semantic is terminated in Sentry, it may further introduces one or many unexpected exits to host kernel. It could be a page fault when Sentry runs, and more likely, a schedule event in Go runtime, e.g., M idle/wakeup. An example in hand is that <code>futex(FUETXWAIT)</code> and <code>epollwait(2)</code> could lead to M idle and a further futex call into host kernel if it does not find any runnable Gs. (See the comments in https://go.dev/src/runtime/proc.go for further explanation about the Go scheduler). Other IO-related syscalls, especially security sensitive, go through another layer of protection - Gofer. For such a syscall, it usually involves one or more Sentry/Gofer inter-process communications. Even with the recent optimization that using lisafs to supersede P9, it's still the slowest path which we shall try best to avoid. As shown above, some syscall paths are by-design slow, and should be identified and reduced as much as possible. Let's hold it to the next section, and dig into the details of the structural and implementation-specific cost of syscalls firstly, because the performance of some Sentry-contained syscalls are not good enough. The first kind of cost is the comparatively stable, introduced by syscall interception. It is platform-specific depending on the way to intercept syscalls. And whether this cost matters also depends on the syscall rate of sandboxed applications. Here's the benchmark result on the structural cost of syscall. We got the data on a Intel(R) Xeon(R) CPU E5-2650 v2 platform, using . As we can see, for KVM platform, the syscall interception costs more than 10x than a native Linux syscall. getpid | benchmark (ns) | -- Native | 62 Native-KPTI | 236 runsc-KVM | 830 runsc-ptrace | 6249 \* "Native" stands for using vanilla linux kernel. To understand the structural cost of syscall interception, we did a on kvm platform. According to the analysis, the overhead mainly comes from: KPTI-like CR3 switches: to maintain the address equation of Sentry running in HR3 and GR0, it has to switch CR3 register twice, on each user/kernel switch; Platform's Switch(): Linux is very efficient by just switching to a per-thread kernel stack and calling the corresponding syscall entry function. But in Sentry, each task is represented by a goroutine; before calling into syscall entry functions, it needs to pop the stack to recover the big while loop, i.e., kernel.(*Task).run. Can we save the structural cost of syscall interception? This cost is actually by-design. We can optimize it, for example, avoid allocation and map operations in switch process, but it can not be eliminated. Does the structural cost of syscall interception really matter? It depends on the syscall rate. Most applications in our case have a syscall rate < 200K/sec, and according to flame graphs (which will be described later in this blog), we see 2~3% of samples are in the switch Secondly, most syscalls, except those as simple as <code>getpid(2)</code>, take several microseconds. In proportion, it's not a significant

However, if you have an elephant RPC (which involves many times of DB access), or a service served by a long-snake RPC chain, this brings nontrivial overhead on latency. The other kind of cost is implementation-specific. For example, it involves some heavy malloc operations; or defer is used in some frequent syscall paths (defer is optimized in Go 1.14); what's worse, the application process may trigger a long-path syscall with host kernel or Gofer involved. When we try to do optimization on the gVisor runtime, we need information on the sandboxed applications, POD configurations, and runsc internals. But most people only play either as platform engineer or application engineer. So we need an easier way to understand the whole picture. To quickly understand the whole picture of performance, we need some ways to profile a running gVisor instance. As gVisor sandbox process is essentially a Go process, Go pprof is an existing way: - provides CPU and heap profile through . - provides more internal profile types like synchronization blocking and scheduler latency. Unfortunately, above tools only provide hot-spots in Sentry, instead of the whole picture (how much time spent in GR3 and HR0). And CPU profile relies on the , which may not accurate enough. cannot provide what we need either. It may help to top/record/stat some information in guest with the help of option [--guestkallsyms], but it cannot analyze the call chain (which is not supported in the host kernel, see Linux's perfcallchainkernel). Then we turn to a nice virtual address equation in Sentry: [(GR0 VA) = (HR3 VA)]. This is to make sure any pointers in HR3 can be directly used in GR0. The equation is helpful to solve this problem in the way that we can profile Sentry just as a normal HR3 process with a little hack on kvm. First, as said above, Linux does not support to analyze the call chain of guest. So Change [isinguest] to pretend that it runs in host mode even it's in guest mode. This can be done in ``` int kvmisin_guest(void) { return thiscpuread(current_vcpu) != NULL; return 0; } ``` Secondly, change the process of guest profile. Previously, after PMU counter overflows and triggers a NMI interrupt, vCPU is forced to exit to host, and calls [int $2] immediately for later recording. Now instead of calling [int $2], we shall call do_nmi directly with correct registers (i.e., pt_regs): ``` +void (fn_do_nmi)(struct pt_regs , long); +#define HIGHERHALFCANONICAL_ADDR 0xFFFF800000000000 +void makeptregs(struct kvmvcpu *vcpu, struct ptregs *regs) +{ /* In Sentry GR0, we will use address among * [HIGHERHALFCANONICAL_ADDR, 2^64-1) * when syscall just happens. To avoid conflicting with HR0, * we correct these addresses into HR3 addresses. */ regs->bp =

& ~HIGHERHALFCANONICAL_ADDR; regs->ip = vmcsreadl(GUESTRIP) & ~HIGHERHALFCANONICAL_ADDR; regs->sp = vmcsreadl(GUESTRSP) & ~HIGHERHALFCANONICAL_ADDR; regs->flags = (vmcsreadl(GUESTRFLAGS) & 0xFF) | X86EFLAGSIF | 0x2; regs->cs = USER_CS; regs->ss = USER_DS; +} static void vmxcompleteatomicexit(struct vcpuvmx *vmx) { u32 exitintrinfo; @@ -8943,7 +8965,14 @@ static void vmxcompleteatomicexit(struct vcpuvmx *vmx) / We need to handle NMIs before interrupts are enabled / if (isnmi(exitintr_info)) { kvmbeforehandle_nmi(&vmx->vcpu); asm("int $2"); if (vmcsreadl(GUESTRFLAGS) & X86EFLAGSIF) asm("int $2"); else { struct pt_regs regs; memset((void *)&regs, 0, sizeof(regs)); makeptregs(&vmx->vcpu, &regs); fndonmi(&regs, 0); } kvmafterhandle_nmi(&vmx->vcpu); } } @@ -11881,6 +11927,10 @@ static int init vmx_init(void) } } fndonmi = (void *) kallsymslookupname("do_nmi"); if (!fndonmi) printk(KERNERR "kvm: lookup donmi fail\n"); ``` As shown above, we properly handle samples in GR3 and GR0 trampoline. Firstly, make sure we compile the runsc with symbols not stripped: `bazel build runsc --strip=never` As an example, run below script inside the gVisor container to make it busy: `stress -i 1 -c 1 -m 1` Perf the instance with command: `perf kvm --host --guest record -a -g -e cycles -G <path/to/cgroup> -- sleep 10 >/dev/null` Note we still need to perf the instance with 'perf kvm' and '--guest', because kvm-intel requires this to keep the PMU hardware event enabled in guest mode. Then generate a flame graph using , and we got this . Let's roughly divide it to differentiate GR3 and GR0 like this: Now we can get clear information like: The bottleneck syscall(s): the above flame graph shows <code>sync(2)</code> is a relatively large block of samples. If we cannot avoid them in user space, they are worth time for optimization. Some real cases we found and optimized are: supersede CopyIn/CopyOut with CopyInBytes/CopyOutBytes to avoid reflection; avoid use defer in some frequent syscalls in which case you can say <code>deferreturn()</code> in the flame graph (not needed if you already upgrade to newer Go version). Another optimization is: after we find that append write of shared volume spends a lot of time querying gofer for current file length in the flame graph, we propose to add . If GC is a real problem: we can barely see sample related to GC in this case. But if we do, we can further search <code>mallocgc()</code> to see where the heap allocation is frequent. We can perform a heap profile to see allocated objects. And we can consider adjust , 100% by default, to sacrifice memory for less CPU utilization. We once found that allocating a object > 32 KB also triggers GC, referring to . Percentage of time spent in GR3 app and Sentry: We can determine if it worths to continue the optimization. If most of the samples are in GR3, then we better turn to optimizing the application code instead. Rather large chunk of samples lie in ept violation and <code>fallocate(2)</code> (into HR0). This is caused by frequent memory allocation and free. We can either optimize the application to avoid this, or add a memory buffer layer in memfile management to relieve it. As a short summary, now we have a tool to get a visible graph of what's going on in a running gVisor instance. Unfortunately, we cannot get the details of the application processes in the above flame graph because of the semantic gap. To get a flame graph of the application processes, we have prototyped a way in Sentry. Hopefully, we'll discuss it in later blogs. A visible way is very helpful when we try to optimize a new application on gVisor. However, there's another kind of overhead, invisible like "Dark matter". Sentry inherits timer, scheduler, channel, and heap allocator in Go runtime. While it saves a lot of code to build a kernel, it also introduces some unpleasant overhead. The Go runtime, after all, is designed and massively used for general purpose Go applications. While it's used as a part or the basis of a kernel, we shall be very careful with the implementation and overhead of these syntactic

Unfortunately, we did not find an universal method to identify this kind of overhead. The only way seems to get your hands dirty with Go runtime. We'll show some examples in our use case. It's known that Go (before 1.14) timer suffers from . What's worse, statistics of Sentry syscalls shows that a lot of <code>futex()</code> is introduced by timers (64 timer buckets), and that Sentry syscalls walks a much longer path (redpill), makes it worse. We have two optimizations here: 1. decrease the number of timer buckets, from 64 to 4; 2. decrease the timer precision from ns to ms. You may worry about the decrease of timer precision, but as we see, most of the applications are event-based, and not affected by a coarse grained timer. However, Go changes the implementation of timer in v1.14; how to port this optimization remains an open question. gVisor introduces an extra level of schedule along with the host linux scheduler (usually CFS). A L2 scheduler sometimes brings positive impact as it saves the heavy context switch in the L1 scheduler. We can find many two-level scheduler cases, for example, coroutines, virtual machines, etc. gVisor reuses Go's work-stealing scheduler, which is originally designed for coroutines, as the L2 scheduler. They share the same goal: "We need to balance between keeping enough running worker threads to utilize available hardware parallelism and parking excessive running worker threads to conserve CPU resources and power." -- From . If not properly tuned, the L2 scheduler may leak the schedule pressure to the L1 scheduler. According to G-P-M model of Go, the parallelism is close related to the GOMAXPROCS limit. The upstream gVisor by default uses # of host cores, which leads to a lot of wasted M wake/stop(s). By properly configuring the GOMAXPROCS of a POD of 4/8/16 cores, we find it can save some CPU cycles without worsening the workload latency. To further restrict extra M wake/stop(s), before wakep(), we calculate the # of running Gs and # of running Ps to decide if necessary to wake a M. And we find it's better to firstly steal from the longest local run queue, comparing to previously random-sequential way. Another related optimization is that we find most applications will get back to Sentry very soon, and it's not necessary to handle off its P when it leaves into user space and find an idle P when it gets back. Some optimizations in Go are put . What we learned from the optimization process of gVisor is that digging into Go runtime to understand what's going on there. And it's normal that some ideas work, but some fail. We introduced how we profiled gVisor for production-ready performance. Using this methodology, along with some other aggressive measures, we finally got to run gVisor with an acceptable overhead, and even better than runc in some workloads. We also absorbed a lot of optimization progress in the community, e.g., VFS2. So far, we have deployed more than 100K gVisor instances in the production environment. And it very well supported transactions of . Along with performance, there are also some other important aspects for production adoption. For example, generating a core after a sentry panic is helpful for debugging; a coverage tool is necessary to make sure new changes are properly covered by test cases. We'll leave these topics to later discussions.