Kube of Thoughts – Scaling Generative AI Models with Kubernetes and Inference Decisi... Asheesh Goja
Channel: CNCF [Cloud Native Computing Foundation]
Date: 22/11/2023

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon Europe in Paris from March 19-22, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io

Kube of Thoughts – Scaling Generative AI Models with Kubernetes and Inference Decision Trees - Asheesh Goja, Amazon Web Services

Generative AI technologies such as LLMs are accelerating the pace of innovation. However, using such models come at a steep computational cost and is a major barrier to entry for many businesses and researchers. This talk proposes a novel approach to improve LLM efficiency, for complex reasoning tasks, by composing containerized LLMs as an inference decision tree called Tree Of Thoughts (ToT) and orchestrating it using various K8S ecosystem projects. ToT is an inference framework in combinatorial problem space, where each node represents a specific train of thought. This approach improves inference efficiency by refining prompts from the initial inputs through ‘conversations’ between models. All this sounds abstract, so using CNCF projects such as Operator Framework, Redis, Neo4j, and KEDA you will learn how to build a concrete ToT implementation. With real time visualization of graphs of intermediate LLM prompts and thoughts, you will see how ToT on Kubernetes enhances the computational efficiency of LLM inferencing.