Mastering LLM Delivery in Private Clouds: A Journey to Seamless Dep... Autumn Moulder & Marwan Ahmed
Channel: CNCF [Cloud Native Computing Foundation]
Date: 22/11/2023

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon Europe in Paris from March 19-22, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io

Mastering LLM Delivery in Private Clouds: A Journey to Seamless Deployments with Kubernetes and OCI - Autumn Moulder & Marwan Ahmed, Cohere

Deploying LLMs is challenging. This talk is a case study in how cloud native technologies, specifically Kubernetes and OCI artifacts, simplifies private LLM deployments. Allowing teams to run models in their infrastructure solves significant data governance & security challenges. However, it is still difficult to efficiently share large artifacts between model developers and model consumers. Autumn and Marwan share how open standards unblocked challenges and simplified LLM delivery. First, we explore how Kubernetes made it possible to rapidly deliver a highly portable, cloud-native inference stack. Second, OCI Artifacts have been underutilized as a delivery mechanism for artifacts beyond container images. We explore how we achieved significant efficiency gains by reducing duplicate storage, increasing download speed, and minimizing governance overhead. Walk away learning how you can leverage Kubernetes and OCI in your MLOps journey.