Accelerators(FPGA/GPU) Chaining to Efficiently Handle Large AI/ML Workloads in K8s
Channel: CNCF [Cloud Native Computing Foundation]
Date: 21/03/2024

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon North America in Salt Lake City from November 12 - 15, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io

Accelerators(FPGA/GPU) Chaining to Efficiently Handle Large AI/ML Workloads in K8s - Sampath Priyankara, Nippon Telegraph and Telephone Corporation & Masataka Sonoda, Fujitsu Limited

Large AI/ML workloads in production require tightly coupled multiple accelerators such as FPGA, GPU, TPU, or ASICs in chains to achieve its service level performance goals such as low latency, and low power consumption. Accelerators in these chains are using direct data transfer between devices which can eliminate unnecessary memory copies, decrease CPU overheads, and reduce latency, resulting in significant performance improvements. K8s is used as a backend for most AI/ML workload deploying frameworks. However, current K8s do not have the functionality to orchestrate or configure such device chains. In this talk, we present how we extend the K8s with Custom Resource-based resource architecture and operators to orchestrate and configure such device chains in a Kubernetes-native way. We will also demonstrate how easy to deploy such workloads and discuss our future work with Dynamic Resource Allocation (DRA) support and CNI extensions.