Project: gVisor (Runtime / Container Runtime)
File: 2020-09-18-containing-a-real-vulnerability.md

In the previous two posts we talked about gVisor's as well as how those are applied in the . Recently, a new container escape vulnerability () was announced that ties these topics well together. gVisor is to this specific issue, but it provides an interesting case study to continue our exploration of gVisor's security. While gVisor is not immune to vulnerabilities, to minimize the impact and remediate if a vulnerability is found. First, lets describe how the discovered vulnerability works. There are numerous ways one can send and receive bytes over the network with Linux. One of the most performant ways is to use a ring buffer, which is a memory region shared by the application and the kernel. These rings are created by calling with for receiving and for sending packets. The vulnerability is in the code that reads packets when `PACKETRXRING` is enabled. There is another option () that asks the kernel to leave some space in the ring buffer before each packet for anything the application needs, e.g. control structures. When a packet is received, the kernel calculates where to copy the packet to, taking the amount reserved before each packet into consideration. If the amount reserved is large, the kernel performed an incorrect calculation which could cause an overflow leading to an out-of-bounds write of up to 10 bytes, controlled by the attacker. The data in the write is easily controlled using the loopback to send a crafted packet and receiving it using a `PACKETRXRING` with a carefully selected `PACKET_RESERVE` size. ```c static int tpacketrcv(struct skbuff skb, struct net_device dev, struct packettype *pt, struct netdevice *orig_dev) { // ... if (sk->sktype == SOCKDGRAM) { macoff = netoff = TPACKETALIGN(po->tphdrlen) + 16 + po->tp_reserve; } else { unsigned int maclen = skbnetworkoffset(skb); // tp_reserve is unsigned int, netoff is unsigned short. // Addition can overflow netoff netoff = TPACKETALIGN(po->tphdrlen + (maclen < 16 ? 16 : maclen)) + po->tp_reserve; if (po->hasvnethdr) { netoff += sizeof(struct virtionethdr); do_vnet = true; } // Attacker controls netoff and can make macoff be smaller // than sizeof(struct virtionethdr) macoff = netoff - maclen; } // ... // "macoff - sizeof(struct virtionethdr)" can be negative, // resulting in a pointer before h.raw if (do_vnet && virtionethdrfromskb(skb, h.raw + macoff - sizeof(struct virtionethdr), vio_le(), true, 0)) { // ... ``` The capability is required to create the socket above. However, in order to support common debugging tools like `ping` and `tcpdump`, Docker containers, including those created for Kubernetes, are given `CAPNETRAW` by default and thus may be able to trigger this vulnerability to elevate privileges and escape the container. Next, we are going to explore why this vulnerability doesnt work in gVisor, and how gVisor could prevent the escape even if a similar vulnerability existed inside gVisors kernel. gVisor does not implement `PACKETRXRING`, but does support raw sockets which are required for `PACKETRXRING`. Raw sockets are a controversial feature to support in a sandbox environment. While it allows great customizations for essential tools like `ping`, it may allow packets to be written to the network without any

In general, allowing an untrusted application to write crafted packets to the network is a questionable idea and a historical source of vulnerabilities. With that in mind, if `CAPNETRAW` is enabled by default, it would not be secure by default to run untrusted applications. After multiple discussions when raw sockets were first implemented, we decided to disable raw sockets by default, even if `CAPNETRAW` is given to the application. Instead, enabling raw sockets in gVisor requires the admin to set `--net-raw` flag to runsc when configuring the runtime, in addition to requiring the `CAPNETRAW` capability in the application. It comes at the expense that some tools may not work out of the box, but as part of our principle, we felt that it was important for the less secure configuration to be explicit. Since this bug was due to an overflow in the specific Linux implementation of the packet ring, gVisor's raw socket implementation is not affected. However, if there were a vulnerability in gVisor, containers would not be allowed to exploit it by default. As an alternative way to implement this same constraint, Kubernetes allows to be configured to customize requests. Cloud providers can use this to implement more stringent policies. For example, GKE implements an admission controller for gVisor that unless it has been explicitly set in the pod spec. gVisor has its own application kernel, called the Sentry, that is distinct from the host kernel. Just like what you would expect from a kernel, gVisor has a memory management subsystem, virtual file system, and a full network stack. The host network is only used as a transport to carry packets in and out the sandbox[^1]. The loopback interface which is used in the exploit stays completely inside the sandbox, never reaching the host. Therefore, even if the Sentry was vulnerable to the attack, there would be two factors that would prevent a container escape from happening. First, the vulnerability would be limited to the Sentry, and the attacker would compromise only the application kernel, bound by a restricted set of filters, discussed more in depth below. Second, the Sentry is a distinct implementation of the API, written in Go, which provides bounds checking that would have likely prevented access past the bounds of the shared region (e.g. see or , which have similar shared regions). Here, Kubernetes warrants slightly more explanation. gVisor makes pods the unit of isolation and a pod can run multiple containers. In other words, each pod is a gVisor instance, and each container is a set of processes running inside gVisor, isolated via Sentry-internal namespaces like regular containers inside a pod. If there were a vulnerability in gVisor, the privilege escalation would allow a container inside the pod to break out to other containers inside the same pod, but the container still cannot break out of the pod. gVisor follows a that the system should have two layers of protection, and those layers should require different compromises to be broken. We apply this principle by assuming that the Sentry (first layer of defense) . In order to protect the host kernel from a compromised Sentry, we wrap it around many security and isolations features to ensure only the minimal set of functionality from the host kernel is

First, the sandbox runs inside a cgroup that can limit and throttle host resources being used. Second, the sandbox joins empty namespaces, including user and mount, to further isolate from the host. Next, it changes the process root to a read-only directory that contains only `/proc` and nothing else. Then, it executes with the unprivileged user/group ) with all capabilities stripped. Last and most importantly, a seccomp filter is added to tightly restrict what parts of the Linux syscall surface that gVisor is allowed to access. The allowed host surface is a far smaller set of syscalls than the Sentry implements for applications to use. Not only restricting the syscall being called, but also checking that arguments to these syscalls are within the expected set. Dangerous syscalls like <code>execve(2)</code>, <code>open(2)</code>, and <code>socket(2)</code> are prohibited, thus an attacker isnt able to execute binaries or acquire new resources on the host. if there were a vulnerability in gVisor that allowed an attacker to execute code inside the Sentry, the attacker still has extremely limited privileges on the host. In fact, a compromised Sentry is much more restricted than a non-compromised regular container. For CVE-2020-14386 in particular, the attack would be blocked by more than one security layer: non-privileged user, no capability, and seccomp filters. Although the surface is drastically reduced, there is still a chance that there is a vulnerability in one of the allowed syscalls. Thats why its important to keep the surface small and carefully consider what syscalls are allowed. You can find the full set of allowed syscalls . Another possible attack vector is resources that are present in the Sentry, like open file descriptors. The Sentry has file descriptors that an attacker could potentially use, such as log files, platform files (e.g. `/dev/kvm`), an RPC endpoint that allows external communication with the Sentry, and a Netstack endpoint that connects the sandbox to the network. The Netstack endpoint in particular is a concern because it gives direct access to the network. Its an `AF_PACKET` socket that allows arbitrary L2 packets to be written to the network. In the normal case, Netstack assembles packets that go out the network, giving the container control over only the payload. But if the Sentry is compromised, an attacker can craft packets to the network. In many ways this is similar to anyone sending random packets over the internet, but still this is a place where the host kernel surface exposed is larger than we would like it to be. Security comes with many tradeoffs that are often hard to make, such as the decision to disable raw sockets by default. However, these tradeoffs have served us well, and we've found them to have paid off over time. CVE-2020-14386 offers great insight into how multiple layers of protection can be effective against such an attack. We cannot guarantee that a container escape will never happen in gVisor, but we do our best to make it as hard as we possibly can. If you have not tried gVisor yet, its easier than you think. Just follow the steps . <br> <br> -- them to local containers or send them out the NIC. The packet will be handled by many switches, routers, proxies, servers, etc. along the way, which may be subject to their own vulnerabilities.