Beyond LLMs: Compounds Systems  Agents  and Whole AI Products
Source: tai_blog

A Framework for Building Great AI Products   The other day  I found myself reflecting on a classic concept that I was taught in business school  Maslows hierarchy of needs  a simple but powerful framework for understanding human motivation  with basic physiological needs at the foundation (air  food  water  shelter  sleep  clothing ) and the pursuit of self-actualization at the pinnacle.   This got me thinking  in the world of tech (especially AI) and products  what is an equivalent? I mean  users always have needs  and needs in the product vary significantly subject to the use case and the problem being solved  but a spectrum definitely exists. Is there a model or a framework we can use to identify what constitutes the right product for customers and what customers would expect of the product? Luckily  Geoffrey Moores Crossing the Chasm provides some answers. In his book  Moore references Levitts Whole Product Model  and goes further to simplify by introducing the Simplified Whole Product Model.   In this post  we will internalize Moores model  expand it  and show how it can be applied specifically to AI products (applies to any product as well). Well dive into the trade-offs inherent in building AI applications and illustrate these concepts with real-world examples.   My goal is that after you read this post  you should have a mental model and a framework for building great/usable AI products  which would help you not only think about the technology  but also how it fits in the big picture.   Thanks for reading The Technomist! Subscribe for free to receive new posts and support my work.   The Whole Product Primer (Plus its Descendants)The Whole Product model revolves around the idea that a core/generic product must be complemented by additional services and interfaces (aka enablers) making up the Whole Product which should provide a solution to the customers problem and to address their needs.   In Geoffery Moores book  the core/generic product is defined as the fundamental offering or technology that a company produces  which may not be sufficient to fully solve the customers problem or meet their needs.   This is where the outer ring comes into play. It represents the whole (expected) product  which is divided into sectors. This outer ring encompasses all the additional elements that customers expect or require to make the core product fully functional and valuable to them  lets call them the enablers.   The Adapted (Simplified) Whole Product ModelIn the tech industry  companies often prefer to build upon existing open-source projects or technologies rather than developing everything from scratch. These companies focus on adding unique value through layers of customization  support  consulting services  integrations  and proprietary patterns  creating a whole product that is more than the sum of its parts.   Furthermore  any successful technology is bound to become commoditized over time  a strategy we often see in tech employed by competitors who gain from doing so  forcing value into higher layers in the value chain (which they usually have thus wanting to commoditize). Recognizing this  companies need to continually innovate and differentiate their offerings to maintain a competitive edge (related  see a previous post on AI market dynamics and what companies in the space focus their efforts on).   Therefore  lets adapt the simplified whole product model with two key adjustments. First  well shift from fixed sectors to a more modular  petal-like structure. This reflects the interconnected yet distinct components that comprise the whole product layer. Second  well introduce a new layer above the whole product layer  called the differentiated product layer. This layer will highlight the unique value propositions that set companies and their products apart  showcasing how they create the most value for their customers.   To be more concrete  lets show how this can be applied to Slack for example (this is just for illustration purposes  the real differentiators could very well be very different).   In addition to representing the products enablers differently using petal-like modular components  we added a new layer to highlight the differentiators. In the example above and in the case of Slack  enablers could be threads  Slack Connect  the workflow builder  and/or Slack AI.   We are very close to being done here with the adaptations  so we will add one last thing to our new framework. In addition to the differentiated layer  we would like to model customizability for products. I.e.  one customers whole product may not be the same for another. I.e.  not all customers desire exactly the same features  so its important to cater based on customers constraints/needs. For example  generically  some customers value safety/security over cost  others might value speed  etc.   Lets continue the slack example. Slack might have different customers to cater for. Enterprise customers  use it mainly as a means for company-wide communication  in that case  the focus will be security and compliance with the companys communication policy  leading to:   Prioritized Enablers: Enterprise-grade security  granular permissions  compliance features (e.g.  data retention policies)Emphasized Differentiators: Slack Connect for secure external collaboration  integration with enterprise security toolsAnother use-case  focus area might be on developers  and Slack being part of their dev/test workflows. In that case  the focus will be on developer productivity and collaboration  leading to:   Prioritized Enablers: Integrations with development tools (e.g.  GitHub  Jira)  code snippets  powerful searchEmphasized Differentiators: Workflow Builder for automating tasks  Slack AI for code suggestions and knowledge retrievalThe takeaway here is that versatility can be a core differentiator on its own because it allows for tailored product experiences. Another way to look at it is that the constraint being imposed defines the core value proposition of the product and how it is shaped to best serve and differentiate in a particular space.   In our example  Slack can tailor its offering to different customer segments  highlighting the features and capabilities that are most relevant to each group. This customization not only enhances the user experience but also strengthens Slacks value proposition in a competitive market.   Towards Whole AI Products (aka Systems)Hopefully  you have a handle on the adapted simplified whole product framework by now. Next  we will focus on using the framework and mapping it to the super exciting world of AI applications.   Key Ingredients to Building AI ApplicationsBefore the mapping  lets do a quick primer on the core ingredients of AI products and applications (a sample not an exhaustive list). We will cover the key ideas  but we wont delve into the technical intricacies. For that  there are many resources available  some of which I will be referencing as we go for further reading.   LLMs AND/OR SLMsIn a previous post  I introduced the model product possibilities frontier  a framework for studying the tradeoffs and use cases of large language models (LLMs) and Small Language Models (SLMs)  which I will not be repeating here for brevity. That said  the choice of which models and their size to use is a key ingredient for building generative AI applications and products.   Here are a few considerations/questions to ask yourself when reasoning about the tradeoffs:   What are your most favorable constraints? Is it speed  quality  cost  etc?What about privacy? Do you value data staying in-house (Small models are easier/cheaper to deploy  train  and serve on-premise)How are you going to evaluate the performance of your AI applications that make use of these models?Is a smaller model easier to test and evaluate (think about the specificity as truth vs the versatility of LLMs which introduces more variability/hallucination  and thus makes it harder to test)While we did not call it out explicitly  large or small models can be fine-tuned and aligned. This is covered in greater detail in this post.   Retrieval Augmented Generation (RAG)Id say 2023 was the year of RAG. We went from naive RAG to Advanced RAG. I liked naive tbh  it communicated simplicity  but well  these days advanced is perceived as better  something we are yet to fix  but thats a different story U+1F642. This paper provides more details.   RAG workflows are comprised of many moving pieces and optimizations. The goal is to retrieve the best content to augment the context for LLMs (text generation) with necessary information. In that case  LLMs become curators rather than innovators/generators of sorts (they shape the retrieval results and make them relatable as an output to a user but are not the source of knowledge themselves). To give you an idea of the moving pieces involved with RAG  here is a rough brain dump (feel free to surf the mindmap as you please  I will not enumerate the details here for brevity).   When considering RAG for building AI applications  some questions come to mind around tradeoffs and decisions  usually between RAG  long context  and Fine-tuning. Again  we wont cover details  but here are a set of questions that you can ask to inform your decision.   Does the application require access to external data sources to provide accurate and up-to-date responses (RAG usually makes sense if data freshness is important  especially since language models are point-in-time trained)?Is it crucial for the model to adapt its behavior  writing style  or domain-specific knowledge to match specific requirements (RAG does not customize behavior  fine-tuning would make sense if behavior customization is a goal)?How critical is it to minimize the risk of the model generating false or fabricated information (hallucinations)?How much labeled training data is available for fine-tuning? Does it adequately represent the target domain and tasks?How frequently does the underlying data change? How important is it for the model to have access to the latest information?Is it important to understand the reasoning behind the models responses and trace them back to specific data sources?How important is minimizing computational costs for your project or organization?Do your typical queries require multi-step reasoning (complex queries or simple questions)?How important is the ability to scale your solution to handle a large number of queries?Finally  here is a short guide I created to help you make informed decisions about RAG/Fine-tuning if you wish to use it:   For more information  check the below papers  which I found very useful in understanding the differences and the trade-offs:   [2407.16833] Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach[2401.08406] RAG vs Fine-tuning: Pipelines  Tradeoffs  and a Case Study on Agriculture[2312.05934] Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMsRAG today has become synonymous with building AI applications in some contexts. Whats clear is that RAG is not one component  its a system comprised of many moving pieces with levers to turn on/off for what makes sense most subject to context and use-case.   Agents ft. Agentic OR Agentless!In addition to the model (LLM/SLM)  RAG  there is the notion of agents and agentic workflows (also agentless to counter U+1F642). While this is again not going to be a deep-dive  lets cover the basics.   What are agents  what is agentic behavior  and why agentless sometimes?   The notion of agents is not new. Agents have existed for decades (see this for examples)  they are officially called Intelligent agents. Below is the definition of an Intelligent Agent.   In intelligence and artificial intelligence  an intelligent agent (IA) is an agent acting in an intelligent manner. It perceives its environment  takes actions autonomously in order to achieve goals  and may improve its performance with learning or acquiring knowledge. An intelligent agent may be simple or complex: A thermostat or other control system is considered an example of an intelligent agent  as is a human being  as is any system that meets the definition  such as a firm  a state  or a biome.[1]   Whats changed is that with the advent of LLMs is that agents got a capability boost  from symbolic  rule-based  predefined simple actions with low autonomy (see the history post for more details  you may be reminded of expert systems) to being able to understand and generate natural language  learn and adapt across diverse domains  and perform complex  autonomous actions. In todays context  An agent is a software entity possessing autonomy  goal-oriented behavior allowing it to operate and generalize cross-domains and take complex actions.   Agentic behavior  in this context  refers to an agents ability to operate independently  make decisions aligned with its objectives  and execute actions (potentially with tools/functions-calling ) to achieve those goals. The level of agency can vary based on factors like the complexity of the environment  the agents goals  and the degree of user supervision required. More agentic systems can operate autonomously in intricate environments  pursue complex objectives  and utilize advanced techniques such as planning and tool use   Finally  there is the notion of flow-engineered / AGENTLESS which relies on determinism and only interfaces with language models for specific clarifying actions  in a sense similar to intelligent agents of the past  with the exception of having access to external intelligence capable of better identifying areas where the predefined action could be taken.   To simplify your life  Ive included this visual below (higher resolution here) to help you build a clearer mental picture of agents/agentic.   Other componentsBesides agents  RAG  the models  there are multiple other ingredients that go into building an AI applications  going through each and every one is out of scope for this post  but here is a non-exhaustive list for reference:   Data Pipeline: System for collecting and processing data  think extractions  transformation.Knowledge Base: where the processed knowledge/data is stored.User Interface: Web or app interface for users.Query/prompt Cache: avoid unnecessary query round-trips which can greatly reduce costs.APIs: To interface with other systems.Infrastructure: an important component that is usually overlooked  where to host the model/app  how to scale it  etc.Observability: be able to log  monitor  trace an AI application.Model Gateways: to interface between the user-query and its destination. Along the way  it makes sure the query is authenticated/authorized  masked/audited for sensitive content (e.g.  PII)  and finally routed to the best model to serve the query (best here is dependent on the use-case  see this post)<Many more>As I was writing this  I came across this blog post  which discusses the technical details of some of the most used components for AI applications.   Compounds AI SystemsYou have come a long way brave reader  the end is near  and you shall be rewarded. So far we have been separately covering important components and ingredients that are key to the making of AI applications  but what makes the interconnection of these components towards achieving a shared goal? A system!   A system is a group of interacting or interrelated elements that act according to a set of rules to form a unified whole   Zaharia et. al recently introduced the notion of Compound AI Systems. In their post  they define it as:   A system that tackles AI tasks using multiple interacting components  including multiple calls to models  retrievers  or external tools. In contrast  an AI Model is simply a statistical model  e.g.  a Transformer that predicts the next token in text.   The authors also emphasize the complexity of designing AI systems:   While compound AI systems can offer clear benefits  the art of designing  optimizing  and operating them is still emerging. On the surface  an AI system is a combination of traditional software and AI models  but there are many interesting design questions. For example  should the overall control logic be written in traditional code (e.g.  Python code that calls an LLM)  or should it be driven by an AI model (e.g. LLM agents that call external tools)? Likewise  in a compound system  where should a developer invest resources  for example  in a RAG pipeline  is it better to spend more FLOPS on the retriever or the LLM  or even to call an LLM multiple times.   In their post  they showcase a table of AI systems  and the components they are composed of. Additionally  they highlight the need for optimization across the chosen components to build reliable AI systems. Below we extract the components mentioned in the post and categorize them into Ops (i.e.  operations)  Tools  Context/Knowledge  and models.   If you remember in the previous section  we covered similar components and more as ingredients to build AI applications. The takeaway here is that building reliable AI applications takes a system not a singleton component. I.e.  the whole is more than the sum of the parts   Another way to visualize it is to consider a dashboard looking like a cockpit with all knobs needed to build your AI application  here is an example of what that could look like:   Without abstraction  youd have to configure all these knobs manually (i.e.  youd have to understand what each of these means). Nowadays  there exist many frameworks to do the orchestration which to a good extent abstracts away some if not all these details. Is that a good thing? I will let you decide. My take? It can be a good thing if you are experimenting  learning  but if reliability  performance  and security are concerns (and they should be)  youd still have to understand what all these knobs mean before you pick up automation/orchestration tooling. Think of it this way  do pilots just take on their license without understanding what each and every knob in their cockpit means? I would guess not! But when they do  they can auto-pilot if they choose to because at any point they CAN switch back to pilot-mode and turn on the right knobs to fly the plane safely.   Thanks for reading The Technomist! Subscribe for free to receive new posts and support my work.   From Compound Systems to Whole AI ProductsNow that we understand the key ingredients needed to build AI applications and compound AI systems  which is the technical pattern we will use to describe the components of an AI application and how they intermingle  lets go ahead and map that back to our adapted simplified whole product framework.   Note: While having a technical system encapsulating the main function(s) of the product is great  shipping and building whole products take more time/effort (to execute) than the technical parts.   As you can see in the diagram above (higher resolution here)  we took the components of compound AI as we categorized them in the previous section  and mapped them to the generic/core (right in the middle)  and the whole product layer comprised of one or more enablers. You may notice that we left out the differentiated product layer  thats intentional. We will cover that in a coming section. What about the constraints? Lets model them as well well.   The constraints will heavily depend on the use-case  I used Enterprise here as an example. For enterprise AI use-cases  safety  and reliability are important concerns. Using the constraints  we put emphasis on specific parts of the whole product  highlighting key enablers. In that case we chose legal  ops  gateway  and UX.   Different use-cases will place different emphasis on the whole product  resulting in some layers being more important than others. Some use-cases even simplify the whole product by losing unneeded enablers  making the whole product leaner and more directed towards solving the problem/use case at hand.   Defensibility AND Compound MOATSPreviously we took a tour to compare and contrast the current AI Market Landscape. We showed how companies that have a mission to better something other than just the model might have better odds in surviving in a competitive market (I.e.  AI as an enabler vs. AI as the core product). We have also shown how companies are releasing open-source language models  which increases competitiveness and commoditizes the model layer completely making it pertinent for startups and companies to see defensibility through differentiation  i.e.  what is the companys MOAT?   For defensibility  lets summarize the most prominent strategies:   Having strong using communities and strong user engagement.Transitioning from Foundational Models to Purpose-Based ApproachesBuilding Layers of Value Beyond the ModelDifferentiating at Various Layers of the AI StackLets briefly get into each.   Fostering a strong community and high user engagement: This involves cultivating a rapidly growing user base  harnessing the power of network effects  and creating a vibrant community that engages users across different generations. I.e.  Who will use my product  what value to I provide beyond just the model  and why do I have a community in the first place?Transitioning from general foundational models to purpose-built applications: By focusing on specific user needs and problems  companies can tailor their AI solutions to provide more value and differentiate themselves in the market using existing business models  E.g.  I already a social network  I make good money from Ads  how can I add more value to the existing community by incorporating AI?Building layers of value beyond the model: Invest in research to continually improve models and applications  leverage proprietary data (data as moat) for enhanced performance (after all garbage in  garbage out  gold in  gold out)  and continuously refine products based on user feedback. By building a loyal customer base and offering unique value propositions  companies can establish a strong competitive advantage.Differentiate by focusing various layers of the AI stack: This can involve developing superior AI models or smaller niche models (focusing on a tiny use-case but beating anyone else at doing it)  providing scalable and efficient AI infrastructure  or creating user-friendly interfaces and seamless integrations (a GPT store  for example?). Each layer presents an opportunity for differentiation and can contribute to a companys overall defensibility.These are just but some strategies that can be used to build moats  it is rarely a single component  its the sum of multiple to make a better whole defensible product. Compound MOATs  are the way! The last strategy is the one with lowest chances of surviving alone  so Id consider at least two of the above strategies to start differentiating. Some questions to ask:   What processes do you have in place to ensure that AI models are being leveraged as enablers rather than being treated as end products?What strategies are you employing to rapidly grow your user base  create network effects  and foster a sense of community?What investments are you making in research  data  product refinements  and customer acquisition to build layers of value?What resources are you allocating to differentiate your company at the model layer  infrastructure layer  or application layerHow are you evaluating and prioritizing potential areas of differentiation to ensure a sustainable competitive advantage?Adding The Differentiated Product LayerAlright alright alright  now that we understand moats/defensibility strategies  how do we model them back into our framework?! Using any (or additional) defensibility strategies to differentiate  additional components are added to the differentiated product layer in the model. In that case  we added strong community  integration with partners  a store/marketplace  innovations at the application layer (value above the model)  and unique data. This layers makes a companys set of Compound MOATs  which are also what create brand differentiation  loyalty  retention  etc.   AI Whole Products in PracticeIts 2024  almost two years after the release of ChatGPT  almost 70 years after the perceptron  the first manifestation of neural networks (see this post for more details)  and ~40 years after the creation of expert systems which was the closest Applied AI could get. In the post  I go into the details of why expert systems did not pan out (and partially led to an AI winter)  but for brevity  it was a consumption gap  what we had back then in terms of compute  community  and technology was a far cry from where we are today.   With LLMs showing a glimpse of what can be achieved with natural language  and with the maturity of predictive AI and deep neural networks  applied AI is a reality now more than ever. In this section  we show hope AI applications are built using compound AI systems in the wild. There are many sources of knowledge about applications of AI that can be found on the internet. I chose to use the Federal AI use-case inventory to extract some examples use-cases  followed by a real case of how Uber and OpenAI make use of compound AI systems to build whole AI products and map them to our adapted simplified whole product framework.   Federal AI Use-Cases ExamplesBelow is the breakdown for 6 example use-cases from the inventory after we have applied the framework (use the codes to find them in the inventory).   Note: Higher resolution of the image below can be found here.   Example 1: TowerScout (HHS-00222023)Problem: Identifying potential sources of Legionnaires Disease outbreaks during investigations.   Constraints: Accuracy  speed of detection  ability to process aerial imagery.   Core Product: Object detection and image classification models trained to recognize cooling towers.   Enablers:   Data Pipeline: System to acquire  process  and store aerial imagery.Knowledge Base: Geographic data on building locations  potential water sources.Tools: Image annotation tools  model training infrastructure  visualization software (GIS).Differentiated Product Layer:   Integration: Direct integration with CDC outbreak investigation workflows and databases.Unique Data: Access to CDCs epidemiological data for model training and validation.Example 2: USDA Cropland Data Layer (USDA-00262023)Problem: Classifying crop types and land use for agricultural monitoring and statistics.   Constraints: Accuracy  national coverage  consistency over time  ability to handle satellite data.   Core Product: Machine learning algorithms (likely Random Forest) trained to classify crops from satellite imagery.   Enablers:   Data Pipeline: System to acquire  process  and store multi-temporal satellite imagery.Knowledge Base: Ground truth data from farm surveys  historical crop patterns  weather data.Tools: Image processing software  model training infrastructure  geospatial analysis tools.Differentiated Product Layer:   Long-Term Data: Historical CDL data provides valuable insights into agricultural trends.Public Availability: Open access to CDL data makes it widely used by researchers and policymakers.Example 3: Human Resource Apprentice (OPM-00002023)Problem: Time-consuming and potentially subjective evaluation of applicant qualifications in government hiring.Constraints: Accuracy  fairness  ability to process applicant resumes and job descriptions  explainability.Core Product: AI model (NLP and potentially ranking algorithms) trained on data from previous hiring decisions.Enablers:   Data Pipeline: System to acquire and process applicant data from applications and resumes.Knowledge Base: Job descriptions  qualification requirements  competency frameworks.Tools: NLP libraries  model training infrastructure  user interface for HR specialists.Differentiated Product Layer:   Bias Mitigation: Robust testing and evaluation for fairness and adverse impact mitigation.Explainability: Ability for the system to provide clear rationale for applicant rankings.Example 4: HaMLET (Harnessing Machine Learning to Eliminate Tuberculosis)  HHS-00232023 (CDC)Problem: Improving the accuracy and efficiency of overseas health screenings for immigrants and refugees  specifically for tuberculosis.   Constraints: Accuracy  speed (high throughput)  ability to process chest x-rays  potential resource limitations in overseas settings.   Core Product: Computer vision models trained to detect TB from chest x-rays.   Enablers:   Data Pipeline: System for acquiring  digitizing  and storing chest x-rays.Knowledge Base: Large  labeled dataset of chest x-rays with confirmed TB diagnoses.Tools: Image annotation tools  model training infrastructure  potentially lightweight deployment for use on less powerful devices.Differentiated Product Layer:   Public Health Impact: Potential to significantly reduce TB transmission and improve global health outcomes.Resource Efficiency: Automating screening can reduce the need for specialized personnel  making it more feasible in resource-constrained settings.Example 5: RelativityOne (DHS-00262023  Dept. of Homeland Security)Problem: Inefficient and time-consuming document review in litigation  FOIA requests  and other legal processes involving large volumes of documents.   Constraints: Accuracy  speed  ability to handle diverse document formats  legal and ethical considerations around data privacy and access.   Core Product: A document review platform using machine learning techniques (continuous active learning  clustering).   Enablers:   Data Pipeline: System for ingesting  processing  and indexing large volumes of documents.Knowledge Base: Legal frameworks  case law  and other relevant information for model training.Tools: Text extraction and analysis tools  user interface for legal professionals to review and manage documents and results.Differentiated Product Layer:   Enhanced Efficiency: Significantly reduces the time and resources required for document review.Improved Accuracy: ML models can identify relevant documents and patterns that humans might miss.Compliance and Security: Strong focus on data security and compliance with legal and ethical requirements.Example 6: Cybersecurity Threat Detection (HHS-00152023  ASPR)Problem: Effectively analyzing the massive volume of cybersecurity threat data to identify and respond to real threats.   Constraints: Speed  accuracy  ability to handle diverse data sources  evolving nature of cyber threats.   Core Product: AI and ML models trained to detect anomalies and malicious activity in network traffic and other security data.   Enablers:   Data Pipeline: Real-time data ingestion from various security tools (firewalls  intrusion detection systems  etc.)Knowledge Base: Databases of known threats  attack patterns  and vulnerabilities.Tools: Data visualization and analysis tools  security orchestration and automation platforms for incident response.Differentiated Product Layer:   Proactive Threat Detection: AI models can identify emerging threats and zero-day attacks that traditional rule-based systems might miss.Automated Response: AI can automate incident response actions  such as quarantining infected devices  to contain threats faster.Companies & ProductsBeyond the federal AI use-cases  let us apply the framework to products released out in the open by well-known companies and startups. We will be covering Uber  and OpenAI.   Ubers Michael AngeloRecently  I came across this post and this post covering Ubers journey in developing and refining their AI platform  Michelangelo  over the past 8 years. According to the posts  Michelangelo plays a critical role in powering nearly every aspect of Ubers operations  from core functions like ETA prediction and ride matching to fraud detection and customer support. Additionally  since 2023  Uber has been building various internal generative AI applications and platforms to provide a good foundation for building those applications (see this post on how to build platforms for more details). Here is a distribution of their generative AI use-cases/goals:   With that in mind  lets apply our adapted whole product framework to Ubers internal AI use-case with Michaelangelo and building an AI platform.   Problem: Lack of a standardized and scalable system for developing  deploying  and managing ML across Ubers diverse business needs with tiering/prioritization.   Goal: Harness the power of both traditional ML and LLMs to improve core operations (ETA  pricing)  enhance user experiences (customer support  app features)  and boost internal productivity.   Constraints:   Scale: Managing massive data volume and real-time prediction demands of a global user base.Latency: Delivering low-latency predictions for time-sensitive applications.Security & Privacy: Protecting user data  particularly PII  especially when using external LLMs.Collaboration: Supporting efficient workflows for diverse teams of data scientists  ML engineers  and application developers.Adaptability: Rapidly evolving to integrate new AI/ML technologies and adapt to the changing landscape.Cost-Effectiveness: Managing the computational expenses of large-scale AI  optimizing where possible.Core Product: Fine-tuned / Custom self-hosted LLMs tailored for Ubers internal use-cases.Enablers:   Data Pipeline: System for collecting and processing data  think extractions  transformation.Palette: Feature store for managing  sharing  and accessing features across Uber.Data Processing & Prep: Tools for collecting  cleaning  and transforming data for both traditional ML and LLMs.Knowledge Integration: Connecting LLMs to knowledge bases  APIs  and Uber-specific data sources for grounding and context.Tools (part of enablers):   Development: Michelangelo Studio (MA Studio) for UI-based workflows; Canvas for code-driven development  version control  and CI/CD.Training: Horovod  Ray  Spark  support for TensorFlow and PyTorch; specialized tools for LLM fine-tuning and optimization.Serving: Triton Inference Server  Michelangelos real-time prediction service (OPS).Monitoring: Model Excellence Score (MES) for quality assessment  feature monitoring  SLA integration  and LLM performance tracking.Gateways: Ubers Specialized Gateways such as (GenAI  CO Inference) abstracting complexities and providing easier access to AI capabilities.User Interfaces: Michelangelo Studio: Unified UI for managing ML workflows.Legal & Operations (part of enablers):   Security & Compliance: PII redaction  access controls  bias detection  and mechanisms for ensuring responsible AI usage.Cost Management: Tracking LLM usage  setting budgets  and implementing cost optimization strategies.Model Versioning & Artifact Management: Ensuring reproducibility  tracking experiments  and managing model deployments.Differentiated Product Layer:   Scale and Operational Efficiency: Michelangelo and its integrated gateways are built to handle the complexities of AI/ML at Ubers global scale.Internal Platform Expertise: Ubers AI platform team has deep knowledge of the companys unique data  business needs  and engineering environment.Focus on Developer Experience: Tools like MA Studio and Canvas  combined with the abstraction layers of gateways  prioritize developer productivity and ease of use.Hybrid Approach: Combining traditional ML and LLMs through a unified architecture allows Uber to address a wider range of use cases.If you have noticed  and in the mapping we have done so far for Michael Angelo  the whole product is the platform. Its what enables developers to build products that customers love  take their mobile application for example. I have discussed platforms as products or products of the platforms in more length in this post. Feel free to take a refresher trip if you are looking for more details on the distinction.   OpenAIs ChatGPTBy now you most likely have used a variant of ChatGPT  what you have not seen is whats running under the hood to allow you to use the interface exposed and get the chat experience you get. Below is a diagram from an OpenAI talk about what the platform looks like under the hood and what it takes to run ChatGPT and expose to the world.   To get more visibility  lets apply the adapted whole product framework to ChatGPT :   Problem: How to providing accessible  versatile  and powerful AI assistance for a wide range of tasks and queries.   Constraints:   Safety and ethical considerationsScalability to handle massive user demandAccuracy and reliability of outputsCost-effectiveness of compute resourcesCore Product: Large Language Models (GPT series)   Enablers:   Context/Knowledge: Fine-tuning datasets for specific tasks and safety alignmentTool-use: ChatGPT  DALL-E  and Codex for code generation and understandingUX: the ChatGPT web interface + the Mobile appOps (part of enablers):   Scalable infrastructure for model training and inferenceMonitoring and logging systemsUser feedback collection and analysisDifferentiated Product Layer:   GPT Store: Marketplace for custom GPTs created by users and organizationsStrong Community and User Engagement: Rapidly growing user base for ChatGPT as well as an active developer community using OpenAI API (in a sense its become the standard)Continuous Model Improvements: Regular updates (e.g.  GPT-3 to GPT-4) and Integration capabilities with other tools and platformsState-of-the-Art Performance: Leading performance in various language tasksUnique Data and Feedback Loop: Massive web-scraped dataset for pre-training  vast amounts of user interaction data for model improvement.Innovation at Application Layer: ChatGPT plugins ecosystem  Realistic Voice with imitation  Assistant API for creating AI agentStrategic Partnerships: Microsoft partnership for exclusive access to GPT models increasing distribution blast radius to all Azure users.Infrastructure: Access to large-scale infrastructure and compute (partially enabled by the Microsoft partnership as well)The (Adapted) Market Development Life CycleSo far we have been traveling across the lands of the adapted simplified whole product framework. Along the way  we have also covered some real examples to demonstrate how the framework is (or can be) used. It wouldnt be a whole product framework adaptation if we didnt adapt it to Moores Market Development Life Cycle model though.   Note: higher resolution of the image below can be found here.   It all starts with a Generic (core) Product  a barebones model appealing to innovators/techies who prioritize core functionality. If you would pick an open-source LLM (maybe fine-tuned to solve a specific problem?) and just put it to the test  that would be an example of a core/generic product (the enabling technology which is at the heart of making a future whole product possible). Innovators here are tinkering with the tech that you seemingly are building your product around (or that you might have built yourself). Questions they might ask here: how does it fair  do we need it  do we have better alternatives  would we require additional support (skill/knowledge?)  do they (your company) have it? youll neeed to make sure you have answers to those questions.   To cross to the Early Adopters and their desire for somewhat practical solutions  your product should find a way to meet the expectations (aka the Expected Product)  for the problem your customer is trying to solve  what are some of the key enablers you made sure to add to create a Minimum Viable Product (MVP)? Here you must have started to target a specific niche  and started to provide enough enablers in the product that it solves 80% of their use-case (they might be willing to help because now they SEE the value of what you are offering). At this stage  relationships and feedback matter.   Now its the moment of truth  to cross the chasm to the early majority. This stage often makes or breaks your product/value prop. You will have to navigate a tradeoff: maintain the speed and innovation that attracted early adopters while at the same time also addressing the reliability/demands to make this product Whole. Make no mistake  the likelihood of others doing the same is high at this stage  but you will need to cross here anyways. Examples of enablers at this stage:   An efficient pipeline for data acquisition  processing  and storage. Think of Ubers Michelangelo platform  with its specialized data management tools like Palette.User-friendly interfaces  efficient model training infrastructure  observability (think compound systems here  and tailor to constraints). Using our Ubers example  think Michelangelo Studio and their AI gateway (AuthN/Z  routing  etc).Knowledge Integration  connecting the AI to relevant knowledge bases (RAG maybe)  well-defined APIs  and domain-specific data sources to enhance its capabilities.Once you do cross  know you have augmented your product just enough to make it whole  welcome to the land of the pragmatists  and congratulations  you have an augmented whole product with well-defined key-enablers that solve the customers problem.   You are not done though! Now you get a chance to tell the world why you are different  ruffle your feathers  and be ready to differentiate  welcome to the Differentiated Product layer. At this stage  youll need to focus on highlighting your unique value proposition and solidify your maots. Examples here could:   Foster an active community around the product (if you have that already  you might be a winner) and encourage user contributions/feedback. Both Slack and OpenAI have cultivated vibrant communities around their products (there are different ways to do that  but thats not the topic of this post maybe more on this later).Collaborate with key partners to expand reach  access valuable resources  and enhance the products capabilities. For example  OpenAIs partnership with Microsoft exemplifies this  granting them access to compute and distribution Leverage unique datasets  if you have a community  you likely also have data unique to your products/services (with consent of course I hope). Develop and customize your models  and refine your core optimizatoions to create a competitive edge. Ubers Michelangelo leverages their vast ride-sharing data and internal expertise to optimize AI for their specific business needs.As you move through the stages  youll notice how the products complexity increases  natural and reflects the evolving needs and expectations of each customer segment/use-case. The visual above hopefully acts as a guide/framework to highlight the importance adapting your AI product strategy accordingly to achieve success in each phase of the lifecycle. Failing to adapt will leave you behind  while successfully listening and continuously building/iterating can give your company and your product a boost into a temporarily blue-ocean (we will talk about that later) where you excel for what you do.   Putting it All Together: Building Whole AI ProductsYou MADE IT! By now  you understand what it takes to build whole AI products! Lets quickly recap.   In this post  we went together on a journey that started from classic business principles like Maslows hierarchy of needs to the world of compound AI systems AND how they map and transform into whole AI products. Weve explored the critical components of successful AI products and applications  adapting Moores Simplified Whole Product Model along the way  and finally fitted our new framework into Moores infamous Model Development Lifecycle framework (again with some adaptations/opinions). Here are some take-aways from our journey:   Its Not Just About the Model: While LLMs and SLMs are powerful (open-source or not)  they are just one ingredient in the recipe for a successful AI product. And yes open source unlocks many potential benefits (out of scope)  but it does NOT mean it rivals whole products!Compound AI Systems make a good pattern/foundation for whole AI products: The true power of AI is unleashed when you combine models  data pipelines  knowledge bases  retrieval mechanisms (like RAG)  agents  user interfaces  and robust infrastructure (and more) into a cohesive system that works well with the defined constraints.Differentiation is key: In a rapidly evolving AI landscape  establishing a moat (see above) is essential for long-term success. Focus on building strong communities  transitioning to purpose-built applications  creating value beyond the model  and differentiating at various layers of the AI stack. Compound MOATs (read above) are the way to go!Constraints Shape Your Product: Clearly define the problem youre solving and the specific constraints of your target audience. These constraints will guide your choices regarding the core product  enablers  and even the differentiators.The Adapted Whole Product Framework Provides a Roadmap: By considering each layer of the framework  the generic/core product  enablers  constraints  and differentiated product layer  you can develop a complete understanding of what constitutes a valuable and defensible AI product.Building AI products is not a one-size-fits-all endeavor. The examples from the Fed-AI use-case inventory  Ubers Michaelangelo  or OpenAIs ChatGPT (some of many examples in the wild) highlight the different approaches and strategies companies/institutions are employing today to build AI products and applications. By focusing on user needs  and continuously innovating/iterating/discovering  you can navigate the uncertainties of the AI landscape and create AI products that truly deliver on their promise.   With all that said and done  now Its Your Turn  friend:   Think about an AI product you are working on or envisioning. Use the adapted simplified whole product framework and the guiding questions posed throughout this post to analyze its strengths  weaknesses  and opportunities for differentiation. Remember  building successful AI products requires building a perspective that goes beyond just the technology itself  remember the whole is greater than the sum of its parts  so make sure how you connect the parts resonates will with your brand  mission  and strategy.   Thanks for reading The Technomist! Subscribe for free to receive new posts and support my work.   Thats it! If you want to collaborate  co-write  or chat  reach on LinkedIn. I look forward to hearing from you!   If you like the article and would like to support me  make sure to:U+1F44F Clap awayU+1F449 Follow on MediumU+1F514 Subscribe to The Technomist NewsletterU+1F514 Follow on: LinkedIn U+007C Twitter U+007C GitHub