Project: gVisor (Runtime / Container Runtime)
File: gpu.md

[TOC] gVisor adds a layer of security to your AI/ML applications or other CUDA workloads while adding negligible overhead. By running these applications in a sandboxed environment, you can isolate your host system from potential vulnerabilities in AI code. This is crucial for handling sensitive data or deploying untrusted AI workloads. gVisor supports running most CUDA applications on preselected versions of . To achieve this, gVisor implements a proxy driver inside the sandbox, henceforth referred to as `nvproxy`. `nvproxy` proxies the application's interactions with NVIDIA's driver on the host. It provides access to NVIDIA GPU-specific devices to the sandboxed application. The CUDA application can run unmodified inside the sandbox and interact transparently with these devices. The `runsc` flag `--nvproxy` must be specified to enable GPU support. gVisor supports GPUs in the following environments. The is packaged as part of the . This runtime is just a shim and delegates all commands to the configured low level runtime (which defaults to `runc`). To use gVisor, specify `runsc` as the low level runtime in `/etc/nvidia-container-runtime/config.toml` and then run CUDA containers with `nvidia-container-runtime`. NOTE: gVisor currently only supports . The alternative, , is not yet supported. The "legacy" mode of `nvidia-container-runtime` is directly compatible with the `--gpus` flag implemented by the docker CLI. So with Docker, `runsc` can be used directly (without having to go through `nvidia-container-runtime`). ``` $ docker run --runtime=runsc --gpus=all --rm -it nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8 [Vector addition of 50000 elements] Copy input data from the host memory to the CUDA device CUDA kernel launch with 196 blocks of 256 threads Copy output data from the CUDA device to the host memory Test PASSED Done ``` uses a different GPU container stack than NVIDIA's. GKE has (which is different from ). GKE's plugin modifies the container spec in a different way than the above-mentioned methods. NOTE: `nvproxy` does not have integration support for `k8s-device-plugin` yet. So k8s environments other than GKE might not be supported. gVisor supports a wide range of CUDA workloads, including PyTorch and various generative models like LLMs. Check out . gVisor undergoes continuous tests to ensure this functionality remains robust. of gVisor across different CUDA workloads helps discover and address potential compatibility or performance issues in `nvproxy`. `nvproxy` is a passthrough driver that forwards `ioctl(2)` calls made to NVIDIA devices by the containerized application directly to the host NVIDIA driver. This forwarding is straightforward: `ioctl` parameters are copied from the application's address space to the sentry's address space, and then a host `ioctl` syscall is made. `ioctl`s are passed through with minimal intervention; `nvproxy` does not emulate NVIDIA kernel-mode driver (KMD) logic. This design translates to minimal overhead for GPU operations, ensuring that GPU bound workloads experience negligible performance impact. However, the presence of pointers and file descriptors within some `ioctl` structs forces `nvproxy` to perform appropriate translations. This requires `nvproxy` to be aware of the KMD's ABI, specifically the layout of `ioctl` structs. The challenge is compounded by the lack of ABI stability guarantees in NVIDIA's KMD, meaning `ioctl` definitions can change arbitrarily between

While the NVIDIA installer ensures matching KMD and user-mode driver (UMD) component versions, a single gVisor version might be used with multiple NVIDIA drivers. As a result, `nvproxy` must understand the ABI for each supported driver version, necessitating internal versioning logic for `ioctl`s. As a result, `nvproxy` has the following limitations: Supports selected GPU models. Supports selected NVIDIA driver versions. Supports selected NVIDIA device files. Supports selected `ioctl`s on each device file. gVisor currently supports NVIDIA GPUs: T4, L4, A100, A10G and H100. Please if you want support for another GPU model. The range of driver versions supported by `nvproxy` directly aligns with those available within GKE. As GKE incorporates newer drivers, `nvproxy` will extend support accordingly. Conversely, to manage versioning complexity, `nvproxy` will drop support for drivers removed from GKE. This strategy ensures a streamlined process and avoids unbounded growth in `nvproxy`'s versioning. To see what drivers a given `runsc` version supports, run: ``` $ runsc nvproxy list-supported-drivers ``` gVisor only exposes `/dev/nvidiactl`, `/dev/nvidia-uvm` and `/dev/nvidia#`. Some unsupported NVIDIA device files are: `/dev/nvidia-caps/*`: Controls `nvidia-capabilities`, which is mainly used by Multi-instance GPUs (MIGs). `/dev/nvidia-drm`: Plugs into Linux's Direct Rendering Manager (DRM) subsystem. `/dev/nvidia-modeset`: Enables `DRIVER_MODESET` capability in `nvidia-drm` devices. To minimize maintenance overhead across supported driver versions, the set of supported NVIDIA device `ioctl`s is intentionally limited. This set was generated by running a large number of CUDA workloads in gVisor. As `nvproxy` is adapted to more use cases, this set will continue to evolve. Currently, `nvproxy` focuses on supporting compute workloads (like CUDA). Graphics and video capabilities are not yet supported due to missing `ioctl`s. If your GPU compute workload fails with gVisor, please note that some `ioctl` commands might still be unimplemented. Please to describe about your use case. If a missing `ioctl` implementation is the problem, then the will contain warnings with prefix `nvproxy: unknown *`. While CUDA support enables important use cases for gVisor, it is important for users to understand the security model around the use of GPUs in sandboxes. In short, while gVisor will protect the host from the sandboxed application, NVIDIA driver updates must be part of any security plan with or without gVisor. First, a short discussion on . gVisor protects the host from sandboxed applications by providing several layers of defense. The layers most relevant to this discussion are the redirection of application syscalls to the gVisor sandbox and use of on gVisor sandboxes. gVisor uses a "platform" to tell the host kernel to reroute system calls to the sandbox process, known as the sentry. The sentry implements a syscall table, which services all application syscalls. The Sentry may make syscalls to the host kernel if it needs them to fulfill the application syscall, but it doesn't merely pass an application syscall to the host kernel. On sandbox boot, seccomp filters are applied to the sandbox. Seccomp filters applied to the sandbox constrain the set of syscalls that it can make to the host kernel, blocking access to most host kernel vulnerabilities even if the sandbox becomes

For example, is mitigated because gVisor itself handles the syscalls required to use namespaces and capabilities, so the application is using gVisor's implementation, not the host kernel's. For a compromised sandbox, the syscalls required to exploit the vulnerability are blocked by seccomp filters. In addition, seccomp-bpf filters can filter by argument names allowing us to allowlist granularly by `ioctl(2)` arguments. `ioctl(2)` is a source of many bugs in any kernel due to the complexity of its implementation. As of writing, gVisor does by argument for things like terminal support. For example, is mitigated by gVisor because the application would use gVisor's implementation of `ioctl(2)`. For a compromised sentry, `ioctl(2)` calls with the needed arguments are not in the seccomp filter allowlist, blocking the attacker from making the call. gVisor also mitigates similar vulnerabilities that come with device drivers (). Recall that "nvproxy" allows applications to directly interact with supported ioctls defined in the NVIDIA driver. gVisor's seccomp filter rules are modified such that `ioctl(2)` calls can be made . The allowlisted rules aligned with each . This approach is similar to the allowlisted ioctls for terminal support described above. This allows gVisor to retain the vast majority of its protection for the host while allowing access to GPUs. All of the above CVEs remain mitigated even when "nvproxy" is used. However, gVisor is much less effective at mitigating vulnerabilities within the NVIDIA GPU drivers themselves, because gVisor passes through calls to be handled by the kernel module. If there is a vulnerability in a given driver for a given GPU `ioctl` (read feature) that gVisor passes through, then gVisor will also be vulnerable. If the vulnerability is in an unimplemented feature, gVisor will block the required calls with seccomp filters. In addition, gVisor doesn't introduce any additional hardware-level isolation beyond that which is configured by by the NVIDIA kernel-mode driver. There is no validation of things like DMA buffers. The only checks are done in seccomp-bpf rules to ensure `ioctl(2)` calls are made on supported and allowlisted `ioctl`s. Therefore, it is imperative that users update NVIDIA drivers in a timely manner with or without gVisor. To see the latest drivers gVisor supports, you can run the following with your runsc release: ``` $ runsc nvproxy list-supported-drivers ``` Alternatively you can view the or download it and run: ``` $ make run TARGETS=runsc:runsc ARGS="nvproxy list-supported-drivers" ``` While gVisor doesn't protect against all NVIDIA driver vulnerabilities, it does protect against a large set of general vulnerabilities in Linux. Applications don't just use GPUs, they use them as a part of a larger application that may include third party libraries. For example, Tensorflow that every application does. Designing and implementing an application with security in mind is hard and in the emerging AI space, security is often overlooked in favor of getting to market fast. There are also many services that allow users to run external users' code on the vendor's infrastructure. gVisor is well suited as part of a larger security plan for these and other use cases.