Project: gVisor (Runtime / Container Runtime)
File: 2023-04-28-systrap-release.md

We are releasing a new gVisor platform: Systrap. Like the existing ptrace platform, Systrap runs on most Linux machines out of the box without virtualization. Unlike the ptrace platform, its fast . Go try it by adding `--platform=systrap` to the runsc flags. If you want to know more about it, read on. -- gVisor is a security boundary for arbitrary Linux processes. Boundaries do not come for free, and gVisor imposes some performance overhead on sandboxed applications. One of the most fundamental performance challenges with the security model implemented by gVisor is system call interception, which is the focus of this post. To recap on the : gVisor is an application kernel that implements the Linux ABI. This includes system calls, signals, memory management, and more. For example, when a sandboxed application calls , it actually transparently calls into This minimizes the attack surface of the host kernel, because sandboxed programs simply cant make system calls directly to the host in the first place[^1]. This interception happens through an internal layer called the Platform interface, which we have written about in a previous . To handle these interceptions, this interface must also create new address spaces, allocate memory, and create execution contexts to run the workload. gVisor had two platform implementations: KVM and ptrace. The KVM platform uses the kernels KVM functionality to allow the Sentry to act as both guest OS and VMM (Virtual machine monitor). It does system call interception just like a normal virtual machine would. This gives good performance when using bare-metal virtualization, but has a noticeable impact with nested virtualization. The other obvious downside is that it requires support for nested virtualization in the first place, which is not supported by all hardware (such as ARM CPUs) or within some Cloud environments. The ptrace platform was the alternative wherever KVM was not available. It works through the action, which makes the user process hand back execution to the sentry whenever it encounters a system call. This is a clean method to achieve system call interception in any environment, virtualized or not, except that its quite slow. To see how slow, an unrealistic but highly illustrative benchmark to use is the [^2]. This benchmark runs the system call in a tight `while` loop. No useful application has this behavior, so it is not a realistic benchmark, but it is well-suited to measure system call

{:width="100%"} All `getpid` runs have been performed on a GCE n2-standard-4 VM, with the `debian-11-bullseye-v20230306` image. While this benchmark is not applicable to most real-world workloads, just about any workload will generally suffer from high overhead in system call performance. Since running in a virtualized environment is the default state for most cloud users these days, it's important that gVisor performs well in this context. Systrap is the new platform targeting this important use case. Systrap relies on multiple techniques to implement the Platform interface. Like the ptrace platform, Systrap uses Linux's ptrace subsystem to initialize workload executor threads, which are started as child processes of the main gVisor sentry process. Systrap additionally sets a very restrictive seccomp filter, installs a custom signal handler, and allocates chunks of memory shared between user threads and runsc sentry. This shared memory is what serves as the main form of communication between the sentry and sandboxed programs: whenever the sandboxed process attempts to execute a system call, it triggers a `SIGSYS` signal which is handled by our signal handler. The signal handler in turn populates shared memory regions, and requests the sentry to handle the requested system call. This alone proved to be faster than using `PTRACE_SYSEMU`, as demonstrated by the `getpid` benchmark: {:width="100%"} Can we make it even faster? Recall what the main purpose of our signal handler is: to send a request to the sentry via shared memory. To do that, the sandboxed process must first incur the overhead of executing the seccomp filter[^3], and then generating a full signal stack before being able to run the signal handler. What if there was a way to simply have the sandboxed process jump to another user-space function when it wanted to perform a system call? Well, turns out, there is[^4] There is a popular x86 instruction pattern thats used to perform system calls, and it goes a little something like this: `mov sysno, %eax; syscall`. The size of the mov instruction is 5 bytes and the size of the syscall instruction is 2 bytes. Luckily this is just enough space to fit in a `jmp *%gs:offset` instruction. When the signal handler sees this instruction pattern, it signals to the sentry that the original instructions can be replaced with a `jmp` to trampoline code that performs the same function as the regular `SIGSYS` signal

The system call number is not lost, but rather encoded in the offset. The results are even more impressive: {:width="100%"} As mentioned, the `getpid` benchmark is not representative of real-world performance. To get a better picture of the magnitude of improvement, here are some real-world workloads: The measures compilation performance by compiling ; this is a highly system call dependent workload due to needing to do a lot of I/O filesystem operations (gVisors file system overhead is also dependent upon file system isolation it implements, which is something you can learn about ). The runs a multimedia processing tool, to perform video stream encoding/decoding for example; this workload does not require a significant amount of system calls and there are very few userspace to kernel mode switches. The trains a variety of machine learning models on CPU; the system-call usage of this workload is in between compilation and ffmpeg, due to needing to retrieve training and validation data, but the majority of time is still spent just running userspace computations. Finally, the Redis benchmark performs SET RPC calls with 5 concurrent clients, measures the latency that each call takes to execute, and reports the median (scaled by 250,000 to fit the graph's axis); this workload is heavily bounded by system call performance due to high network stack usage. {:width="100%"} Systrap will replace the ptrace platform by September 2023 and become the default. Until then, we are working really hard to make it production-ready, which includes working on additional performance and stability improvements, and making sure we maintain a high bar for security through targeted fuzz-testing for Systrap specifically. In the meantime, we would like gVisor users to try it out, and give us feedback! If you run gVisor using ptrace today (either by specifying `--platform ptrace` or not specifying the `--platform` flag at all), or you use the KVM platform with nested virtualization, switching to Systrap should be a drop-in performance upgrade. All you have to do is specify `--platform systrap` to runsc. If you encounter any issues, please let us know at . <br> <br> -- <!-- mdformat off(Footnotes need to be separated by linebreaks to be rendered) --> several defense-in-depth layers, including a restricted set of seccomp filters. You can find more details here: . Linux, once the process has entered kernel-mode), actually executing the getpid system call itself is very fast, so this benchmark effectively measures single-thread syscall-interception overhead. . time of writing. <!-- mdformat on -->