Project: gVisor (Runtime / Container Runtime)
File: 2023-06-27-directfs.md

Directfs is now the default in runsc. This feature gives gVisors application kernel (the Sentry) secure direct access to the container filesystem, avoiding expensive round trips to the filesystem gofer. Learn more about this feature in the following blog that was on . -- gVisor is used internally at Google to run a variety of services and workloads. One of the challenges we faced while building gVisor was providing remote filesystem access securely to the sandbox. gVisors strict and defense in depth approach assumes that the sandbox may get compromised because it shares the same execution context as the untrusted application. Hence the sandbox cannot be given sensitive keys and credentials to access Google-internal remote filesystems. To address this challenge, we added a trusted filesystem proxy called a "gofer". The gofer runs outside the sandbox, and provides a secure interface for untrusted containers to access such remote filesystems. For architectural simplicity, gofers were also used to serve local filesystems as well as remote. {:width="100%"} When gVisor was as , the same gofer model was copied over to maintain the same security guarantees. runsc was configured to start one gofer process per container which serves the container filesystem to the sandbox over a predetermined protocol (now ). However, a gofer adds a layer of indirection with significant overhead. This gofer model (built for remote filesystems) brings very few advantages for the runsc use-case, where all the filesystems served by the gofer (like rootfs and ) are mounted locally on the host. The gofer directly accesses them using filesystem syscalls. Linux provides some security primitives to effectively isolate local filesystems. These include, , and detached bind mounts[^1]. Directfs is a new filesystem access mode that uses these primitives to expose the container filesystem to the sandbox in a secure manner. The sandboxs view of the filesystem tree is limited to just the container filesystem. The sandbox process is not given access to anything mounted on the broader host filesystem. Even if the sandbox gets compromised, these mechanisms provide additional barriers to prevent broader system compromise. In directfs mode, the gofer still exists as a cooperative process outside the sandbox. As usual, the gofer enters a new mount namespace, sets up appropriate bind mounts to create the container filesystem in a new directory and then s into that directory. Similarly, the sandbox process enters new user and mount namespaces and then s into an empty directory to ensure it cannot access anything via path

But instead of making RPCs to the gofer to access the container filesystem, the sandbox requests the gofer to provide file descriptors to all the mount points via . The sandbox then directly makes file-descriptor-relative syscalls (e.g. , , , etc) to perform filesystem operations. {:width="100%"} Earlier when the gofer performed all filesystem operations, we could deny all these syscalls in the sandbox process using seccomp. But with directfs enabled, the sandbox process's seccomp filters need to allow the usage of these syscalls. Most notably, the sandbox can now make syscalls (which allow path traversal), but with certain restrictions: , and . We also had to give the sandbox the same privileges as the gofer (for example `CAPDACOVERRIDE` and `CAPDACREAD_SEARCH`), so it can perform the same filesystem operations. It is noteworthy that only the trusted gofer provides FDs (of the container filesystem) to the sandbox. The sandbox cannot walk backwards (using '..') or follow a malicious symlink to escape out of the container filesystem. In effect, we've decreased our dependence on the syscall filters to catch bad behavior, but correspondingly increased our dependence on Linux's filesystem isolation protections. Making RPCs to the gofer for every filesystem operation adds a lot of overhead to runsc. Hence, avoiding gofer round trips significantly improves performance. Let's find out what this means for some of our benchmarks. We will run the benchmarks using our newly released on bind mounts (as opposed to rootfs). This would simulate more realistic use cases because bind mounts are extensively used while configuring filesystems in containers. Bind mounts also do not have an overlay (), so all operations go through goferfs / directfs mount. Let's first look at our , which repeatedly calls on a file. {:width="100%"} The `stat(2)` syscall is more than 2x faster! However, since this is not representative of real-world applications, we should not extrapolate these results. So let's look at some . {:width="100%"} We see a 12% reduction in the absolute time to run these workloads and 17% reduction in Ruby load time! The gofer model in runsc was overly restrictive for accessing host files. We were able to leverage existing filesystem isolation mechanisms in Linux to bypass the gofer without compromising security. Directfs significantly improves performance for certain workloads. This is part of our ongoing efforts to improve gVisor performance. You can learn more about gVisor at . You can also use gVisor in with . Happy sandboxing! -- mount(MS_BIND) and then detaching it from the filesystem tree using umount(MNT_DETACH).