Gen-AI at Scale: Simplifying Orchestration of Healthcare...- Selvi Kadirvel & Christopher Nuland
Channel: CNCF [Cloud Native Computing Foundation]
Date: 20/03/2024

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon North America in Salt Lake City from November 12 - 15, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io

Gen-AI at Scale: Simplifying Orchestration of Healthcare Applications Across Multi-Cluster Environme - Selvi Kadirvel, Elotl & Christopher Nuland, Red Hat

Running Gen-AI applications at scale is operationally complex - it requires the management of application workloads; infrastructure services, data pipelines and datasets across a multi-cluster, multi-cloud environment. In this talk we describe how to seamlessly operate healthcare workloads using multi-cluster orchestration to deploy across a fleet of Kubernetes clusters. These orchestrators allow scheduling workloads dynamically and flexibly across fleets simplifying operations and improving reliability. We use Weaviate, a vector DB with integrations to OpenAI’s LLMs to store, index, retrieve, and query healthcare documents. KEDA, an event-based autoscaler, enables scaling on demand and ensures cluster resources are used efficiently when consuming data to be indexed by Weaviate. We will demonstrate day-0 and day-2 operations: application and infra deployment, curation of data and workload migration (lift-and-shift) across clusters when new data sets become available.