WasmEdge, portable and lightweight runtime for AI/LLM workloads | Project Lightning Talk
Channel: CNCF [Cloud Native Computing Foundation]
Date: 2024-03-26

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon North America in Salt Lake City from November 12 - 15, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io

WasmEdge, portable and lightweight runtime for AI/LLM workloads | Project Lightning Talk

There are two major challenges in running modern AI / LLM workloads — the Python-based toolchains are too complex and heavyweight, and the applications are not portable. Those challenges are exacerbated in heterogeneous cloud and edge environments with different kinds of CPUs and GPUs on resource constrained devices. Wasm provides a standard-based abstraction layer to separate application bytecode from underlying hardware and drivers, while achieving the optimal performance. In this talk, we will discuss how WasmEdge enables developers to create lightweight and portable AI / LLM apps in Rust. Once compiled to WasmEdge, the bytecode app runs without modification on your own devices, edge IoT devices, and cloud servers, which further allows them to be managed by k8s and container tools.