Building Serverless AI Apps with Spin and WebAssembly - Matt Butcher & Radu Matei, Fermyon
Channel: CNCF [Cloud Native Computing Foundation]
Date: 2024-03-20

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon North America in Salt Lake City from November 12 - 15, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io

Building Serverless AI Apps with Spin and WebAssembly - Matt Butcher & Radu Matei, Fermyon

Spin is an open source tool for building serverless WebAssembly apps that can scale from zero to thousands. One of the virtues of WebAssembly is its platform neutrality. Spin apps can be compiled once and run on a variety of OSes, CPU architectures, and even GPUs. This means developers can locally build AI applications using whatever GPU architecture is available, and then deploy into a cluster that has a completely different architecture. In this talk, we’ll build a simple AI inferencing app using the LLaMa2 Chat LLM, test it locally, and then deploy it into a variety of environments, including Docker Desktop and a Kubernetes cluster with Wasm support. We’ll see how the same app runs in each environment, and analyze the performance characteristics of each. Then we’ll talk about the nuances of scheduling GPUs in a clustered environment, and see how Spin’s fine-grained GPU scheduling can boost GPU utilization across multiple apps.