# Enabling RDMA and GPUs in Rootless Kubernetes for Accelerated HPC and AI Applications

Lise Jolicoeur, Vanessa Sochat, François Diakhaté, Daniel Milroy

► To cite this version:

Lise Jolicoeur, Vanessa Sochat, François Diakhaté, Daniel Milroy. Enabling RDMA and GPUs in Rootless Kubernetes for Accelerated HPC and AI Applications. VHPC 2025 - 20th Workshop on Virtualization in High-Performance Cloud Computing (held in conjunction with Europ-Par 2025 - the European Conference on Parallel and Distributed Computing), Aug 2025, Dresden, Germany. hal-05236403

HAL Id: hal-05236403

https://inria.hal.science/hal-05236403v1

Submitted on 2 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Enabling RDMA and GPUs in Rootless
Kubernetes for Accelerated HPC and AI
Applications

Lise Jolicoeur (✉)¹,², Vanessa Sochat³, François Diakhaté¹, and Daniel Milroy³

¹ CEA, DAM, DIF, Arpajon 91297, France firstname.lastname@cea.fr

² University of Bordeaux, INRIA, CNRS, Bordeaux INP, LaBRI, UMR 5800, F-33400, Talence, France

³ Lawrence Livermore National Laboratory

**Résumé** HPC workflows increasingly integrate diverse technologies such as artificial intelligence (AI), machine learning (ML), data analytics, databases, and web services. Orchestrators like Kubernetes have been designed to facilitate deploying these heterogeneous workloads in cloud environments. Allowing Kubernetes to be launched and managed as a resource on HPC clusters would facilitate the deployment of modern workflows in HPC environments. To enable the deployment of Kubernetes by unprivileged HPC users, we evaluate the usability of a rootless version of Kubernetes, Kubernetes. We analyze synthetic benchmarks as well as HPC and ML proxy apps to evaluate the overhead of Kubernetes for HPC/ML workloads deployed on high performance networks and GPUs. While the results show that applications running in Kubernetes can take advantage of InfiniBand networks and NVIDA GPUs, some benchmarks incur measurable overheads at scale which warrant further investigation.

**Keywords:** HPC · Kubernetes · RDMA · GPU.

# 1 Introduction

High Performance Computing (HPC) has historically been an environment fine tuned for specific types of scientific workflows, such as large-scale simulations, with little flexibility for accommodating diverse computing needs. Primarily characterized by performance, applications developed for HPC environments are usually tightly integrated with the underlying hardware to maximize resource efficiency. Recently, we have witnessed the arrival of complex and composite workflows that require diverse components diverging from the typical use cases of HPC. Traditional HPC environments struggle to support complex workflows' needs for cloud-native applications and services. Workflows such as MuMMI[1] and AMS[4] couple typical HPC scientific workloads with machine learning (ML) components that drive simulations and interpret data on the fly to optimize or steer future calculations. Workflows like MuMMI and AMS can also depend on

4. https://github.com/CURRENT/ams
databases or registries to store intermediate results or share data, and require
communication facilitating software such as message brokers.

Enabling these new workflows requires capabilities like reproducibility, portability and flexibility. Containers have been central for enabling these capabilities since Docker popularized portable application packaging and isolated runtime environments. In a recent survey of HPC system administrators, RSEs and scientists, 80% of respondents indicated using containers weekly [2]. Containers are essential for cloud deployment using Kubernetes, one of the most popular open-source platform for automating the deployment, scaling, and management of containerized applications. Orchestration of workloads on HPC with Kubernetes would be a significant step forward toward enabling capabilities for complex heterogeneous workflows.

**Related Work** Kubernetes has a wide range of capabilities for deploying cloud native applications. Modern tools like Kubeflow<fnref n="5" /> enable deploying large scale AI/ML workflows in a simple and portable way. Workflows may involve parallel processing to handle large amounts of data and distribute calculation steps. Abstractions such as JobSet<fnref n="6" /> have emerged to facilitate the deployment of distributed workloads. Distributed applications also require close physical task placement to optimize communications, which requires scheduling plugins or meta-schedulers to change the default behavior of Kubernetes. Many projects have been created to enhance Kubernetes scheduling capabilities, such as Kqueue<fnref n="7" /> that adds fairshare and other smart queuing capabilities. We can also replicate an HPC environment within Kubernetes with projects such as the Flux Operator[3] that deploys the Flux Framework within a Kubernetes allocation. Similar projects exist for deploying Slurm[4][5]. This method enables users to use the same submission format for the HPC application they are porting while enabling them to use Kubernetes tools for the rest of the workflow.

*Problem statement* Emerging use cases from complex scientific workflows need cloud-native capabilities that are not well supported on HPC clusters. The lack of access to highly contended resources like GPUs, the cost of the cloud, and the availability of on-premises HPC resources encourage running complex workflows on HPC. Supporting needed Kubernetes capabilities on HPC clusters integrates cloud and HPC environments, allowing users to harness on-premises resources and improve the portability and ease of deployment of their workflows. Such a converged cloud and HPC environment enables declarative deployment and management of complex workflows, automating and simplifying their execution.

We are therefore interested in allowing users to deploy Kubernetes clusters on demand, directly on HPC resources that have been allocated for them through HPC workload managers. This approach optimizes resource utilization by enabling dynamic partitioning of resources, as opposed to statically dividing resources at the system level. Having access to a full-fledged Kubernetes cluster

<footnote n="5">5. https://www.kubeflow.org/</footnote>

<footnote n="6">6. https://github.com/kubernetes-sigs/jobset</footnote>

<footnote n="7">7. https://kqueue.sigs.k8s.io/</footnote>
makes it easier for users to use Kubernetes native tools and deploy workloads that are not strictly HPC such as AI/ML inference or applications deployed as microservices. However, deploying Kubernetes on HPC resources does not come without challenges. In particular, Kubernetes typically requires some components to execute as root, notably in order to create the network that links the container to the rest of the network. This is incompatible with the fact that users on HPC clusters are non-privileged.

A project called Kubernetes[6] allows a non-privileged user to deploy Kubernetes within rootless Docker managed by containerd, Docker or Podman. While Kubernetes can support HPC use-cases[7], its userspace networking strategy is problematic, with recorded bandwidth performance 30 times lower than regular Ethernet communications[8]. One aspect remains unexplored: enabling RDMA (Remote Direct Memory Access) over high speed interconnect and GPUs are two important components for optimizing execution. RDMA over high speed interconnect is widely used for MPI communications, and workloads increasingly rely on GPUs to optimize their calculations. For this reason, it is essential to determine if workloads running in Kubernetes can leverage these accelerators and to assess the potential performance overhead.

*Contributions* In order to determine if Kubernetes is usable in an HPC environment, we need to show how we can enable the targeted accelerators, RDMA and GPUs in the Kubernetes containers. We also need to evaluate the overhead of running HPC applications and detect potential bottlenecks before moving to HPC resources. To achieve that, we test 3 HPC applications and a machine learning benchmark to evaluate usability and performance of RDMA and GPUs in Kubernetes.

# 2 Methods

## 2.1 Execution environment

*Overview* We ran our tests in an on-premises virtualized cluster for the CPU benchmarks and on Microsoft Azure virtual machines for ML benchmarks to have access to GPUs and a managed Kubernetes solution for comparison.

Kubernetes is an open-source platform for automating the deployment, scaling, and management of containerized applications. Now a flagship project of the Cloud Native Computing Foundation (CNCF), Kubernetes has become the de facto standard for container orchestration. Its declarative workload specification format supports a wide range of use cases, from lightweight microservices to complex workflows that require autoscaling and parallel processing. Kubernetes needs root privilege for the Node components, specifically the kubelet that runs on each node of the cluster. The kubelet launches and manages the containers through a container runtime. The container runtime needs root privilege in order to setup the namespaces and the networking. For networking, a common action is the creation of a virtual ethernet ('veth') interface that links the container
network namespace to the host and enables the transmission of messages without
additional copies. Because we cannot carry out these actions as unprivileged
users, we cannot directly deploy Kubernetes on HPC resources.

*Usernames* [6] is a project that allows the deployment of Kubernetes without root privileges directly on a host. Kubernetes documents multiple technologies and methodologies to deploy rootless Kubernetes components, but all of use the same underlying technology : user namespaces. A user namespace provides a mechanism for unprivileged users to act with root-level permissions within the namespace’s scope, while maintaining isolation from the host system. In practice, this means an unprivileged user can perform certain privileged operations related to the filesystem and the network within the confines of the namespace. The mechanism for deploying Kubernetes on unprivileged hosts is thus to deploy it in rootless containers with the user namespace enabled. Kubernetes components can run with root privileges within their containers, while still being executed as unprivileged users on the host system.

The rootless container in which Kubernetes is launched still needs to send and receive packets using the host network interfaces without the ability to link a veth to the host network namespace. Multiple projects were created for this use case, but the most widespread one is slirp4netns<fnref n="8" />. This project enables connecting two network namespaces by leveraging TAP devices, which are virtual ethernet interfaces used to create a userspace network bridge<fnref n="9" />. This technique involves copying and emulating the IP stack in user space packets in user space, which means the latency and bandwidth of container communications can be greatly impacted. In [8], the throughput recorded with iperf3 is 17Gbps for a regular rootful setup, and about 500Mbps for the slirp4netns based setup. In the context of HPC, Ethernet typically only accounts for a small number of communications, primarily during the setup of the execution environment for MPI based programs. The bulk of high-performance communication is instead handled by OS bypassing mechanisms such as RDMA. RDMA is a communication mode that allows data to be transferred directly from the memory of one node to another with minimal involvement of the operating system and CPU, providing low-latency, high-throughput data transfers over high-performance network fabrics like InfiniBand. For this reason, we do not directly study the impact of Ethernet communications but rather look at overall networking overhead for HPC applications that are expected to primarily use RDMA.

*Hardware* For CPU applications we use on-premises AMD EPYC ‘Rome’ nodes with 128 cores, 256GB of memory and a Mellanox InfiniBand ConnectX-6 HDR100 HCA. For GPU experiments we use the Microsoft Azure NDv2 series instances with 8 NVIDIA Tesla V100 NVLINK-connected GPUs with 32GB of GPU memory, InfiniBand EDR HCAs, Intel Xeon Platinum 8168 with 40 vCPUs, and 672 GB of memory.

<footnote n="8">8. https://github.com/rootless-containers/slirp4netns</footnote>

<footnote n="9">9. https://docs.kernel.org/networking/tuntap.html</footnote>
*Software* For the purpose of the study, we created a base virtual machine (VM) image and Docker container image replicating the same software environment. An identical base image allows us to compare the results of the benchmarks in both environments without having to factor in potential differences in software version that can have a direct impact on performance. For the base operating system of the VMs, we chose a recent version of Ubuntu, 24.04 TLS. We install OpenMPI 4.1.2 with support for UCX 1.17. UCX is a communication layer that exposes a set of communication primitives which will enable the use of RDMA by the HPC applications. We disabled the use of devx in all environments, both in Singularity and in Kubernetes, due to unexplained crashes in some Kubernetes benchmarks. To launch HPC applications, we deploy the Flux Framework both on the VMs, and in Kubernetes through the Flux Operator[3]. VM and container images are equipped with the Flux scheduler in version 0.40.0 with PMIx support. To launch containers on the cloud virtual machines through Flux, we use Singularity version 4.0.1.

*RDMA and GPUs* To enable RDMA and GPUs in the containers, we need to make sure we have access to the interfaces and have the right drivers. For RDMA, the interfaces located in /dev and /sys are automatically mounted by Singularity. They can be mounted in Kubernetes containers by defining a privileged security context that gives it access to host devices. This works as long as the container has a working configuration for RDMA that includes the libraries for using RDMA on Infiniband such as libibverbs. We use the Nvidia Kubernetes Device Plugin in version 0.15.0¹⁰ to expose GPU devices to Pods. We leverage the Container Device Interface and NVIDIA container toolkit to expose GPUs to each rootless Kubernetes Docker container (kubelet). The process and final configuration for enabling GPUs in Kubernetes are described in [9].

*Experiments* For CPU experiments, we run 3 applications described below : OSU, LAMMPS and AMG. We scale up to 32 nodes and use AMD EPYC Rome instances. We compare Kubernetes to execution in Singularity containers. For GPU experiments, we run PyTorch with the MNIST dataset on up to 4 nodes (32 GPUs) using ND40rs instances. We compare the performance of this training orchestrated in a Kubernetes cluster and in the managed Kubernetes solution provided by Microsoft Azure, AKS. The AKS cluster is deployed with the same instances, VM images and configuration as the Kubernetes setup. The Kubernetes instance runs in rootful mode with default configuration and Infiniband support. Table 1 summarizes the experiments we will be carrying out.

## 2.2 Applications

*OSU benchmarks* The OSU benchmarks<fnref n="11" /> are a suite of synthetic micro benchmarks that evaluate the performance of different categories of MPI communications over the network. We consider both point to point and collective operations.

<footnote n="10">10. https://github.com/NVIDIA/k8s-device-plugin</footnote>

<footnote n="11">11. http://mvapich.cse.ohio-state.edu/benchmarks/</footnote>
<table><caption>TABLE 1 – Summary of the experiments that are carried out to evaluate the networking performance of Kubernetes.</caption><thead><tr><th>Type</th><th>Platform</th><th>Comparison</th><th>Benchmarks</th></tr></thead><tbody><tr><td>CPU</td><td>VMs on HPC cluster</td><td>Singularity/Usernetes</td><td>OSU, LAMMPS, AMG</td></tr><tr><td>GPU</td><td>Microsoft Azure</td><td>AKS/Usernetes</td><td>Pytorch MNIST</td></tr></tbody></table>

Point-to-point communication involves direct message exchange between two processes or hosts, providing two fundamental performance metrics: bandwidth and latency. Collective operations involve groups of processes for operations such as synchronization or data movement, and can be used to evaluate how well a system scales and performs under realistic, multi-process workloads. We use the Allreduce collective, which combines values generated from all processes and distributes the results back to all processes, to assess latency for communication across multiple processes. For point to points benchmarks, we select two nodes and run each of latency and bandwidth 20 times. This allows us to evaluate the overhead without accounting for potential differences in performance between pairs of nodes. For the Allreduce benchmark, we run 40 iterations to account for higher variability of the results.

We decided to evaluate 3 HPC proxy apps that have different characteristics and execution time. The parameters used for each application were chosen to run it optimally while taking into account system limitations such as available memory.

AMG is a "parallel algebraic multigrid solver for linear systems arising from problems on unstructured grids"<fnref n="12" />. It distributes work across all nodes by subdividing the grid into chunks of equal size from the given problem size. We chose to run problem 2 with a problem size of 256x256x128 for all job sizes. We leverage both MPI and OpenMP by having each task use 3 threads on 3 cores. We use the FOM as reported in the output of the job, which is expressed as $\frac{nnz\_AP}{SetupPhaseTime+3*SolvePhaseTime}$.

LAMMPS (Large-scale Atomic/Molecular Massively Parallel Simulator) is a proxy application to model the reaction of atoms, first calculating the force, energy, and pressure during chemical reactions[10]. We used the ReaxFF package for this study and used the millions of atom steps per second (m-atom steps) as the FOM. We use a problem size of 64x64x32 (x, y, z respectively) for all job sizes.

PyTorch provides a framework to run deep learning benchmarks<fnref n="13" />. We will use MNIST, a widely used dataset of handwritten digits commonly employed for

<footnote n="12">12. https://github.com/LLNL/AMG2023</footnote>

<footnote n="13">13. https://pytorch.org</footnote>
evaluating and training deep learning model, to perform distributed training using the NCCL backend for multi GPU and multi node communications. We execute 4 epochs with a batch size of 128.

# 3 Results

## 3.1 OSU benchmarks

Results for latency and bandwidth for point-to-point tests are shown in Figures 1 and 2. In Figure 1, we observe identical median latency between the two environments, with more variability in Kubernetes. The latency is around 1.35 microseconds, which is expected for RDMA over InfiniBand on the cluster. For bandwidth (Figure 2) we also observe identical medians for both environments, even if the Kubernetes results present more variability. We will investigate the cause of this variability in future work.

<figure><img src="image_2.png" /><figcaption>FIGURE 1 – osu_latency with 20 iterations over the same 2 nodes.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>FIGURE 2 – osu_bw with 20 iterations over the same 2 nodes.</figcaption></figure>

The results of OSU Allreduce in Figure 3 show a noticeable overhead for Kubernetes. Singularity has a consistently lower latency: 10% lower at 8 nodes, 15% lower at 16 nodes, 35% lower at 32 nodes. We speculate that this overhead could be caused by system noise introduced by the Kubernetes daemons. This hypothesis is supported by the results shown in Figure 4 in which we compare the performance of Singularity before launching Kubernetes to the performance of Singularity once the Kubernetes daemons (kubelet, api-server, etc.) have been launched and run alongside the execution of the benchmarks. We can see that the latency increases by 10% at 8 nodes, 18% at 16 nodes and 26% at 32 nodes when the Kubernetes daemons are running in the background. From these results, we conclude that Kubernetes daemons, have a non-negligible impact on performance of workloads running alongside Kubernetes and assume that they have a similar impact on applications running inside Kubernetes. Further investigation is required to determine if the behavior of these daemons can be tuned to reduce this impact.
<figure><img src="image_4.png" /><figcaption>FIGURE 3 – AllReduce latency for a message size of 64 bytes.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>FIGURE 4 – AllReduce latency for a message size of 64 bytes in Singularity while Kubernetes is running compared to when Kubernetes is not running.</figcaption></figure>

## 3.2 HPC applications

AMG The average FOM over 5 runs for each job size is reported in Figure 6. AMG shows little variability at execution for both Kubernetes and Singularity, with near identical averages between the two environments at all sizes.

LAMMPS is shown in Figure 5. Similarly to AMG, the average FOM (Total CG Mflops) follows the same trend for the two environments. However, a larger gap starts to appear at 16 nodes and grows to an 11% overhead at 32 nodes for Kubernetes. At scale, the performance gap between LAMMPS running in Kubernetes and Singularity may continue to grow. We hypothesize that the degradation of performance is due to the Kubernetes daemons as we have seen that their impact is non-negligible for OSU Allreduce. Further testing to understand the cause of this overhead is future work.

<figure><img src="image_6.png" /><figcaption>FIGURE 5 – LAMMPS: 128 tasks per node, problem size 64x32x32.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>FIGURE 6 – AMG 2023: OMP_NUM_THREADS=3, 3 cores per task, problem size 256x256x128</figcaption></figure>
## 3.3 GPU applications

The results for the total runtime for PyTorch training runs are shown in Figure 7. The total execution time is recorded from the Python program to get a precise measurement that excludes the time spent waiting for the workers to connect back to the leader. Execution time requires coordination between nodes, and thus reflects efficiency of communication. We also look at accuracy and loss measurements for each epoch, which represent respectively the percentage of correct and wrong predictions by the model. Accuracy and loss are comparable between Singularity and Kubernetes for all epochs. For example, at 4 nodes for the fourth epoch we have 80.712% (+/- 0.04) accuracy and 53.13% (+/- 0.04) loss on average on AKS; for Kubernetes, we have 80.718% (+/- 0.04) accuracy and 53.15% (+/- 0.03) loss. We observe a 3% maximum overhead for the execution time in Kubernetes as compared to AKS. Determining the cause of the overhead is future work.

<figure><img src="image_8.png" /><figcaption>FIGURE 7 – Execution time of PyTorch training with MNIST dataset.</figcaption></figure>

# 4 Discussion and Future Work

A similar study will have to be conducted when deploying Kubernetes on bare metal HPC clusters to confirm and further investigate the overhead we observed for some HPC applications in Kubernetes. In particular, we need to precisely measure the impact of the Kubernetes daemons on performance for applications running in Kubernetes. The amount of OS noise introduced by these daemons should be quantified using benchmarks such as osnoise [11] or NetGauge [12]. It would be interesting to profile the Kubernetes daemons to better understand
their CPU usage patterns and determine if they can be tuned to minimize their CPU utilization when idle.

While HPC applications frequently rely on RDMA for low-latency and high-bandwidth communications, TCP communications are important for basic exchanges with services and applications, so it is important to reduce the overhead of the userspace IP networking. To address this issue, [8] presents a solution to bypass the rootless components that slow down the network called bypass4netns. With this mechanism, we can achieve 30x faster throughput, close to the rootful network performance. We can also run some of the benchmarks with this enabled, notably OSU Allreduce and LAMMPS, to assess whether TCP communications are causing the performance gap.

# 5 Conclusion

This solution allows Kubernetes to be used as a tool for users to either better integrate with or to deploy modern workflows directly on HPC clusters, where resources can be easier to access. The performance results for the benchmarks we selected indicate good performance overall, with some measured overhead that requires further investigating. OSU point-to-point benchmarks show no significant overhead, but the extra variability in the Kubernetes environment should be investigated in follow-up work. Allreduce allowed us to detect that Kubernetes daemons can impact performance. AMG shows no significant overhead, but we measure overhead in LAMMPS as the number of nodes increases. The cause of the increasing overhead could be due to system noise introduced by Kubernetes daemons, but a complete root-cause analysis requires further testing at larger scale. However, as large scale MPI applications are well supported by traditional HPC resource managers, they are not the main target of this work. The execution of the GPU application in Kubernetes shows no significant difference in the quality of the predictions but shows a small overhead for the execution time of the training. It is important to conduct a similar study with Kubernetes deployed on a bare metal on-premise HPC cluster to confirm the usability and further study performance in this specific context.

**Acknowledgments.** This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 and was supported by the LLNL-ISCP Program Project No. 25-COMP-003 (LLNL-CONF-2005961).

**Disclosure of Interests.** The authors have no competing interests to declare that are relevant to the content of this article.

# Références

1. Di Natale et al., A massively parallel infrastructure for adaptive multiscale simulations : modeling RAS initiation pathway for cancer (2019), https://doi.org/10.1145/3295500.3356197
2. The HPC Container Community Survey, https://supercontainers.github.io/hpc-containers-survey/2024/two-thousand-twenty-four/, last accessed 2025/04/17

3. Sochat V., Culquicondor A., Ojea A. and Milroy D., The Flux Operator. F1000Research 2024, 13:203 https://doi.org/https://doi.org/10.12688/f1000research.147989.1

4. SUNK (Slurm on Kubernetes), https://docs.coreweave.com/docs/products/sunk, last accessed 2025/04/17

5. Slinky: Bridging Slurm and Kubernetes, https://www.schedmd.com/slinky/why-slinky/, last accessed 2025/04/23

6. Kubernetes: Kubernetes without the root privileges (Generation 2), https://github.com/rootless-containers/usernetes, last accessed 2025/04/17

7. Sochat V., Fox D. and Milroy D., HPC Alongside User-space Kubernetes (2024), https://doi.org/https://arxiv.org/abs/2406.06995

8. Matsumoto N., Suda A., bypass4netns: Accelerating TCP/IP Communications in Rootless Containers. arXiv (2024). https://doi.org/10.48550/arXiv.2402.00365

9. Sochat V., Rootless User-Space Kubernetes with GPU (2025), https://vsoch.github.io/2025/rootless-usernetes-gpu/, last accessed 2025/07/06

10. A. P. Thompson et al., LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales, Comp Phys Comm, 271 (2022) 10817, https://doi.org/10.1016/j.cpc.2021.108171.

11. OSNOISE Tracer, https://docs.kernel.org/trace/osnoise-tracer.html, last accessed 2025/07/07

12. Netgauge - Operating System Noise Measurement, https://htor.inf.ethz.ch/research/netgauge/osnoise/, last accessed 2025/07/07