Project: gVisor (Runtime / Container Runtime)
File: nvidia_driver_proxy.md

Status as of 2023-06-23: Under review Allow applications running within gVisor sandboxes to use CUDA on GPUs by providing implementations of Nvidia GPU kernel driver files that proxy ioctls to their host equivalents. Non-goals: Provide additional isolation of, or multiplexing between, GPU workloads beyond that provided by the driver and hardware. Support use of GPUs for graphics rendering. gVisor executes unmodified Linux applications in a sandboxed environment. Application system calls are intercepted by gVisor and handled by (in essence) a Go implementation of the Linux kernel called the sentry, which in turn executes as a sandboxed userspace process running on a Linux host. gVisor can execute application code via a variety of mechanisms, referred to as "platforms". Most platforms can broadly be divided into process-based (ptrace, systrap) and KVM-based (kvm). Process-based platforms execute application code in sandboxed host processes, and establish application memory mappings by invoking the `mmap` syscall from application process context; sentry and application processes share a file descriptor (FD) table, allowing application `mmap` to use sentry FDs. KVM-based platforms execute application code in the guest userspace of a virtual machine, and establish application memory mappings by establishing mappings in the sentry's address space, then forwarding those mappings into the guest physical address space using KVM memslots and finally setting guest page table entries to point to the relevant guest physical addresses. provides code for preparing a container for GPU use, and serves as a useful reference for the environment that applications using GPUs expect. In particular, contains a helpful list of relevant filesystem paths, while is the primary entry point into container configuration. Of these paths, `/dev/nvidiactl`, `/dev/nvidia#` (per-device, numbering from 0), `/dev/nvidia-uvm`, and `/proc/driver/nvidia/params` are kernel-driver-backed and known to be required. Most "control" interactions between applications and the driver consist of invocations of the `ioctl` syscall on `/dev/nvidiactl`, `/dev/nvidia#`, or `/dev/nvidia-uvm`. Application data generally does not flow through ioctls; instead, applications access driver-provided memory mappings. `/proc/driver/nvidia/params` is informational and read-only. `/dev/nvidiactl` and `/dev/nvidia#` are backed by the same `struct fileoperations nvfrontend_fops` in kernel module `nvidia.ko`, rooted in `kernel-open/nvidia` in the . The top-level `ioctl` implementation for both, `kernel-open/nvidia/nv.c:nvidia_ioctl()`, handles a small number of ioctl commands but delegates the majority to the "resource manager" (RM) subsystem, `src/nvidia/arch/nvalloc/unix/src/escape.c:RmIoctl()`. Both functions constrain most commands to either `/dev/nvidiactl` or `/dev/nvidia#`, as indicated by the presence of the `NVCTLDEVICEONLY` or `NVACTUALDEVICEONLY` macros respectively. `/dev/nvidia-uvm` is implemented in kernel module `nvidia-uvm.ko`, rooted in `kernel-open/nvidia-uvm` in the OSS driver source; its `ioctl` implementation is `kernel-open/nvidia-uvm/uvm.c:uvm_ioctl()`. The driver API models a collection of objects, using numeric handles as references (akin to the relationship between file descriptions and file descriptors). Objects are instances of classes, which exist in a C++-like inheritance hierarchy that is implemented in C via code generation; for example, the `RsResource` class inherits from the `Object` class, which is the hierarchy's root. Objects exist in a tree of parent-child relationships, defined by methods on the `Object` class. API-accessed objects are most frequently created by invocations of `ioctl(NVESCRM_ALLOC)`, which is parameterized by `hClass`. `src/nvidia/src/kernel/rmapi/resource_list.h` specifies the mapping from `hClass` to instantiated ("internal") class, as well as the type of the pointee of `NVOS21PARAMETERS::pAllocParms` or `NVOS64PARAMETERS::pAllocParms` which the object's constructor takes as input ("alloc param

Most application ioctls to GPU drivers can be proxied straightforwardly by the sentry: The sentry copies the ioctl's parameter struct, and the transitive closure of structs it points to, from application to sentry memory; reissues the ioctl to the host, passing pointers in the sentry's address space rather than the application's; and copies updated fields (or whole structs for simplicity) back to application memory. Below we consider complications to this basic idea. GPUs are equipped with "device" memory that is much faster for the GPU to access than "system" memory (as used by CPUs). CUDA supports two basic memory models: `cudaMalloc()` allocates device memory, which is not generally usable by the CPU; instead `cudaMemcpy()` is used to copy between system and device memory. `cudaMallocManaged()` allocates "unified memory", which can be used by both CPU and GPU. `nvidia-uvm.ko` backs mappings returned by `cudaMallocManaged()`, migrating pages from system to device memory on GPU page faults and from device to system memory on CPU page faults. We cannot implement UVM by substituting a sentry-controlled buffer and copying to/from UVM-controlled memory mappings "on demand", since GPU-side demand is driven by GPU page faults which the sentry cannot intercept directly; instead, we must map `/dev/nvidia-uvm` into application address spaces as in native execution. UVM requires that the virtual addresses of all mappings of `nvidia-uvm` match their respective mapped file offset, which in conjunction with the FD uniquely identify a shared memory segment[^cite-uvm-mmap]. Since this constraint also applies to sentry mappings of `nvidia-uvm`, if an application happens to request a mapping of `nvidia-uvm` at a virtual address that overlaps with an existing sentry memory mapping, then `memmap.File.MapInternal()` is unimplementable. On KVM-based platforms, this means that we cannot implement the application mapping, since `MapInternal` is a required step to propagating the mapping into application address spaces. On process-based platforms, this only means that we cannot support e.g. `read(2)` syscalls targeting UVM memory; if this is required, we can perform buffered copies from/to UVM memory using `ioctl(UVMTOOLSREAD/WRITEPROCESSMEMORY)`, at the cost of requiring `MapInternal` users to explicitly indicate fill/flush points before/after I/O. The extent to which applications use `cudaMallocManaged()` is unclear; use of `cudaMalloc()` and explicit copying appears to predominate in performance-sensitive code. PyTorch contains one non-test use of `cudaMallocManaged()`[^cite-pytorch-uvm], but it is not immediately clear what circumstances cause the containing function to be invoked. Tensorflow does not appear to use `cudaMallocManaged()` outside of test code. For both `cudaMalloc()` and "control plane" purposes, applications using CUDA map some device memory into application address spaces, as follows: The application opens a new `/dev/nvidiactl` or `/dev/nvidia#` FD, depending on the memory being mapped. The application invokes `ioctl(NVESCRMMAPMEMORY)` on an existing `/dev/nvidiactl` FD, passing the new FD as an ioctl parameter (`nvioctlnvos33parameterswith_fd::fd`). This ioctl stores information for the mapping in the new FD (`nvlinuxfileprivatet::mmap_context`), but does not modify the application's address space. The application invokes `mmap` on the new FD to actually establish the mapping into its address

Conveniently, it is apparently permissible for the `ioctl` in step 2 to be invoked from a different process than the `mmap` in step 3, so no gVisor changes are required to support this pattern in general; we can invoke the `ioctl` in the sentry and implement `mmap` as usual. However, mappings of device memory often need to disable or constrain processor caching for correct behavior. In modern x86 processors, caching behavior is specified by page table entry flags[^cite-sdm-pat]. On process-based platforms, application page tables are defined by the host kernel, whose `mmap` will choose the correct caching behavior by delegating to the driver's implementation. On KVM-based platforms, the sentry maintains guest page tables and consequently must set caching behavior correctly. Caching behavior for mappings obtained as described above is decided during `NVESCRMMAPMEMORY`, by the "method `RsResource::resMap`" for the driver object specified by ioctl parameter `NVOS33_PARAMETERS::hMemory`. In most cases, this eventually results in a call to on an associated `Memory` object. Caching behavior thus depends on the logic of that function and the `MEMORY_DESCRIPTOR` associated with the `Memory` object, which is typically determined during object creation. Therefore, to support KVM-based platforms, the sentry could track allocated driver objects and emulate the driver's logic to determine appropriate caching behavior. Alternatively, could we replicate the caching behavior of the host kernel's mapping in the sentry's address space (in `vmareastruct::vmpageprot`)? There is no apparent way for userspace to obtain this information, so this would necessitate a Linux kernel patch or upstream change. `ioctl(NVESCRMALLOCMEMORY, hClass=NV01MEMORYSYSTEMOSDESCRIPTOR)` and `ioctl(NVESCRMVIDHEAP_CONTROL, function=NVOS32FUNCTIONALLOCOSDESCRIPTOR)` create `OsDescMem` objects, which are `Memory` objects backed by application anonymous memory. The ioctls treat `NVOS02PARAMETERS::pMemory` or `NVOS32PARAMETERS::data.AllocOsDesc.descriptor` respectively as an application virtual address and call Linux's `pinuserpages()` or `getuserpages()` to get `struct page` pointers representing pages starting at that address[^cite-osdesc-rmapi]. Pins are held on those pages for the lifetime of the `OsDescMem` object. The proxy driver will need to replicate this behavior in the sentry, though doing so should not require major changes outside of the driver. When one of these ioctls is invoked by an application: Invoke `mmap` to create a temporary `PROT_NONE` mapping in the sentry's address space of the size passed by the application. Call `mm.MemoryManager.Pin()` to acquire file-page references on the given application memory. Call `memmap.File.MapInternal()` to get sentry mappings of pinned file-pages. Use `mremap(oldsize=0, flags=MREMAPFIXED)` to replicate mappings returned by `MapInternal()` into the temporary mapping, resulting in a virtually-contiguous sentry mapping of the application-specified address range. Invoke the host ioctl using the sentry mapping. `munmap` the temporary mapping, which is no longer required after the host ioctl. Hold the file-page references returned by `mm.MemoryManager.Pin()` until an application ioctl is observed freeing the corresponding `OsDescMem`, then call `mm.Unpin()`. Since ioctl parameter structs must be copied into the sentry in order to proxy them, gVisor implicitly restrict the set of application requests to those that are explicitly implemented. We can impose additional restrictions based on parameter values in order to further reduce attack surface, although possibly at the cost of reduced development velocity; introducing new restrictions after launch is difficult due to the risk of regressing existing users. Intuitively, limiting the scope of our support to GPU compute should allow us to narrow API usage to that of the CUDA runtime. [Nvidia GPU driver CVEs are published in moderately large batches every ~3-4 months](https://www.nvidia.com/en-us/security/), but insufficient information regarding these CVEs is available for us to determine how many of these vulnerabilities we could mitigate via parameter

By default, the driver prevents a `/dev/nvidiactl` FD from using objects created by other `/dev/nvidiactl` FDs[^cite-rm-validate], providing driver-level resource isolation between applications. Since we need to track at least a subset of object allocations for OS-described memory, and possibly for determining memory caching type, we can optionally track all objects and further constrain ioctls to using valid object handles if driver-level isolation is believed inadequate. While `seccomp-bpf` filters allow us to limit the set of ioctl requests that the sentry can make, they cannot filter based on ioctl parameters passed via memory such as allocation `hClass`, `NVESCRM_CONTROL` command, or `NVESCRMVIDHEAP_CONTROL` function, limiting the extent to which they can protect the host from a compromised sentry. The contains code to configure an unstarted container based on , [invoking `nvidia-container-cli` from `libnvidia-container` (described above) to do most of the actual work](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/arch-overview.html). It is used ubiquitously for this purpose, including by the . The simplest way for `runsc` to obtain Nvidia Container Toolkit's behavior is obviously to use it, either by invoking `nvidia-container-runtime-hook` or by using the Toolkit's code (which is written in Go) directly. However, filesystem modifications made to the container's `/dev` and `/proc` directories on the host will not be application-visible since `runsc` necessarily injects sentry `devtmpfs` and `procfs` mounts at these locations, requiring that `runsc` internally replicate the effects of `libnvidia-container` in these directories. Note that host filesystem modifications are still necessary, since the sentry itself needs access to relevant host device files and MIG capabilities. Conversely, we can attempt to emulate the behavior of `nvidia-container-toolkit` and `libnvidia-container` within `runsc`; however, note that `libnvidia-container` executes `ldconfig` to regenerate the container's runtime linker cache after mounting the driver's shared libraries into the container[^cite-nvc-ldcache_update], which is more difficult if said mounts exist within the sentry's VFS rather than on the host. When running on the proprietary kernel driver, applications invoke `ioctl(NVESCRM_CONTROL)` commands that do not appear to exist in the OSS driver. The OSS driver lacks support for GPU virtualization[^cite-oss-vgpu]; however, Google Compute Engine (GCE) GPUs are exposed to VMs in passthrough mode[^cite-oss-gce], and Container-Optimized OS (COS) switched to the OSS driver in Milestone 105[^cite-oss-cos], suggesting that OSS-driver-only support may be sufficient. If support for the proprietary driver is required, we can request documentation from Nvidia. Nvidia requires that the kernel and userspace components of the driver match versions[^cite-abi-readme], and does not guarantee kernel ABI stability[^cite-abi-discuss], so we may need to support multiple ABI versions in the proxy. It is not immediately clear if this will be a problem in practice. To simplify the initial implementation, we will focus immediate efforts on process-based platforms and defer support for KVM-based platforms to future work. In the sentry: Add structure and constant definitions from the Nvidia open-source kernel driver to new package `//pkg/abi/nvidia`. Implement the proxy driver under `//pkg/sentry/devices/nvproxy`, initially comprising `FileDescriptionImpl` implementations proxying `/dev/nvidiactl`, `/dev/nvidia#`, and `/dev/nvidia-uvm`. `/proc/driver/nvidia/params` can probably be (optionally) read once during startup and implemented as a static file in the sentry. Each ioctl command and object class is associated with its own parameters type and logic; thus, each needs to be implemented

We can generate lists of required commands/classes by running representative applications under on a variety of GPUs; a list derived from a minimal CUDA workload run on a single VM follows below. The proxy driver itself should also log unimplemented commands/classes for iterative development. For the most part, known-required commands/classes should be implementable incrementally and in parallel. Concurrently, at the API level, i.e. within `//runsc`: Add an option to enable Nvidia GPU support. When this option is enabled, and `runsc` detects that GPU support is requested by the container, it enables the proxy driver (by calling `nvproxy.Register(vfsObj)`) and configures the container consistently with `nvidia-container-toolkit` and `libnvidia-container`. Since setting the wrong caching behavior for device memory mappings will fail in unpredictable ways, `runsc` must ensure that GPU support cannot be enabled when an unsupported platform is selected. To support Nvidia Multi-Process Service (MPS), we need: Support for `SCM_CREDENTIALS` on host Unix domain sockets; already implemented as part of previous MPS investigation, but not merged. Optional pass-through of `statfs::f_type` through `fsimpl/gofer`; needed for a runsc bind mount of the host's `/dev/shm`, through which MPS shares memory; previously hacked in (optionality not implemented). Features required to support Nvidia Persistence Daemon and Nvidia Fabric Manager are currently unknown, but these are not believed to be critical, and we may choose to deliberately deny access to them (and/or MPS) to reduce attack surface. , so it is not clear that granting MPS access to sandboxed containers is safe. Implementation notes: Each application `open` of `/dev/nvidictl`, `/dev/nvidia#`, or `/dev/nvidia-uvm` must be backed by a distinct host FD. Furthermore, the proxy driver cannot go through sentry VFS to obtain this FD since doing so would recursively attempt to open the proxy driver. Instead, we must allow the proxy driver to invoke host `openat`, and ensure that the mount namespace in which the sentry executes contains the required device special files. `/dev/nvidia-uvm` FDs may need to be `UVM_INITIALIZE`d with `UVMINITFLAGSMULTIPROCESSSHARINGMODE` to be used from both sentry and application processes[^cite-uvm-vaspacemm_enabled]. Known-used `nvidia.ko` ioctls: `NVESCCHECKVERSIONSTR`, `NVESCSYSPARAMS`, `NVESCCARDINFO`, `NVESCNUMA_INFO`, `NVESCREGISTERFD`, `NVESCRMALLOC`, `NVESCRMALLOCMEMORY`, `NVESCRMALLOCOSEVENT`, `NVESCRMCONTROL`, `NVESCRM_FREE`, `NVESCRMMAPMEMORY`, `NVESCRMVIDHEAP_CONTROL`, `NVESCRMDUPOBJECT`, `NVESCRMUPDATEDEVICEMAPPINGINFO` `NVESCRM_CONTROL` is essentially another level of ioctls. Known-used `NVOS54PARAMETERS::cmd`: `NV0000CTRLCMDSYSTEMGETBUILD_VERSION`, `NV0000CTRLCMDCLIENTSETINHERITEDSHARE_POLICY`, `NV0000CTRLCMDSYSTEMGETFABRICSTATUS`, `NV0000CTRLCMDGPUGETPROBEDIDS`, `NV0000CTRLCMDSYNCGPUBOOSTGROUP_INFO`, `NV0000CTRLCMDGPUATTACHIDS`, `NV0000CTRLCMDGPUGETID_INFO`, `NV0000CTRLCMDGPUGETATTACHEDIDS`, `NV2080CTRLCMDGPUGETACTIVEPARTITION_IDS`, `NV2080CTRLCMDGPUGETGIDINFO`, `NV0080CTRLCMDGPUGETVIRTUALIZATIONMODE`, `NV2080CTRLCMDFBGETINFO`, `NV2080CTRLCMDGPUGETINFO`, `NV0080CTRLCMDMCGETARCHINFO`, `NV2080CTRLCMDBUSGET_INFO`, `NV2080CTRLCMDBUSGETPCIINFO`, `NV2080CTRLCMDBUSGETPCIBAR_INFO`, `NV2080CTRLCMDGPUQUERYECCSTATUS`, `NV0080CTRLFIFOGETCAPS`, `NV0080CTRLCMDGPUGETCLASSLIST`, `NV2080CTRLCMDGPUGETENGINES`, `NV2080CTRLCMDGPUGETSIMULATIONINFO`, `NV0000CTRLCMDGPUGETMEMOPENABLE`, `NV2080CTRLCMDGRGET_INFO`, `NV2080CTRLCMDGRGETGPCMASK`, `NV2080CTRLCMDGRGETTPCMASK`, `NV2080CTRLCMDGRGETCAPSV2`, `NV2080CTRLCMDCEGET_CAPS`, `NV2080CTRLCMDGPUGETCOMPUTEPOLICY_CONFIG`, `NV2080CTRLCMDGRGETGLOBALSMORDER`, `NV0080CTRLCMDFBGETCAPS`, `NV0000CTRLCMDCLIENTGETADDRSPACE_TYPE`, `NV2080CTRLCMDGSPGET_FEATURES`, `NV2080CTRLCMDGPUGETSHORTNAME_STRING`, `NV2080CTRLCMDGPUGETNAMESTRING`, `NV2080CTRLCMDGPUQUERYCOMPUTEMODE_RULES`, `NV2080CTRLCMDRCRELEASEWATCHDOGREQUESTS`, `NV2080CTRLCMDRCSOFTDISABLEWATCHDOG`, `NV2080CTRLCMDNVLINKGETNVLINKSTATUS`, `NV2080CTRLCMDRCGETWATCHDOGINFO`, `NV2080CTRLCMDPERFBOOST`, `NV0080CTRLCMDFIFOGETCHANNELLIST`, `NVC36FCTRLGETCLASS_ENGINEID`, `NVC36FCTRLCMDGPFIFOGETWORKSUBMIT_TOKEN`, `NV2080CTRLCMDGRGETCTXBUFFERSIZE`, `NVA06FCTRLCMDGPFIFO_SCHEDULE` Known-used `NVOS54_PARAMETERS::cmd` that are apparently unimplemented and may be proprietary-driver-only (or just well-hidden?): 0x20800159, 0x20800161, 0x20801001, 0x20801009, 0x2080100a, 0x20802016, 0x20802084, 0x503c0102, 0x90e60102 Known-used `nvidia-uvm.ko` ioctls: `UVM_INITIALIZE`, `UVMPAGEABLEMEMACCESS`, `UVMREGISTERGPU`, `UVMCREATERANGEGROUP`, `UVMREGISTERGPUVASPACE`, `UVMCREATEEXTERNALRANGE`, `UVMMAPEXTERNALALLOCATION`, `UVMREGISTER_CHANNEL`, `UVMALLOCSEMAPHOREPOOL`, `UVMVALIDATEVARANGE` Known-used `NVESCRM_ALLOC` `hClass`, i.e. allocated object classes: `NV01ROOTCLIENT`, `MPSCOMPUTE`, `NV01DEVICE0`, `NV20SUBDEVICE_0`, `TURINGUSERMODEA`, `FERMIVASPACEA`, `NV50THIRDPARTY_P2P`, `FERMICONTEXTSHAREA`, `TURINGCHANNELGPFIFOA`, `TURINGCOMPUTEA`, `TURINGDMACOPYA`, `NV01EVENTOSEVENT`, `KEPLERCHANNELGROUP_A` guarantees whatsoever, and even API compatibility breaks occasionally." - https://github.com/NVIDIA/open-gpu-kernel-modules/discussions/157#discussioncomment-2757388 kernel modules, version 530.41.03. ... Note that the kernel modules built here must be used with GSP firmware and user-space NVIDIA GPU driver components from a corresponding 530.41.03 driver release." - https://github.com/NVIDIA/open-gpu-kernel-modules/blob/6dd092ddb7c165fb1ec48b937fa6b33daa37f9c1/README.md => (OSS)." - https://cloud.google.com/container-optimized-os/docs/release-notes/m105#cos-beta-105-17412-1-2vsmilestone101. Also see b/235364591, go/cos-oss-gpu. passthrough mode so that your VMs have direct control over the GPUs and their associated memory." - https://cloud.google.com/compute/docs/gpus virtualization, neither as a host nor a guest." - https://github.com/NVIDIA/open-gpu-kernel-modules/discussions/157#discussioncomment-2752052 => . `APISECURITYINFO::clientOSInfo` is set by . Both `PDBPROPSYSVALIDATECLIENT_HANDLE` and `PDBPROPSYSVALIDATECLIENTHANDLESTRICT` are enabled by default by .