Built-In AI Web APIs Will Enable A New Generation Of AI Startups
Source: tai_blog

AI models are getting bigger and better by the day.   Asking what the best frontier AI model is is akin to asking what the best vehicle is. The question is ill-posed and any answer cant be much more than an approximation.   However  frontier AI models are showing a few clear trends :   Converging PerformanceIncreasing Training Costs For Marginal ImprovementsNote that I am referring to Frontier Models as large-scale machine-learning models that exceed the capabilities currently present in the most advanced existing models  and can perform a wide variety of tasks.  Frontier Model Forum.   Converging PerformanceThe latest and highest-scoring frontier models are almost indistinguishable from the general user's point of view.   If users were to test the top frontier models in each category  most likely they wouldnt be able to tell them apart.   Just the three metrics above show a variety of models that could be considered the best in their category. The top is getting crowded and this chart still does not include the recent models   OpenLM by AppleLlama 3.1 by Meta (Try it on HuggingFace  or through IBM no-code RAG solution)Mistral Large 2 by Mistral (Try on la Plateforme under the name mistral-large-2407  HuggingFace  or Vertex)Increasing Training Costs For Marginal ImprovementsThe cost of training frontier AI models keeps rising while improvements are marginal. According to an analysis from Stanford Universitys 2024 Artificial Intelligence Index Report  OpenAIs GPT-4 used an estimated $78 million worth of compute to train  while Googles Gemini Ultra cost $191 million for compute.   Googles Gemini Ultra costs 2.5x OpenAIs GPT-4  but it is not near 2.5x better in General Ability  Reasoning  or Coding skills. We could argue  however  that Gemini has an incredibly larger context than GPT-4 (almost 8x).   On the other side  cloning and approximating existing models seems to be fairly cheap as shown by Stanford researchers who have cloned ChatGPT for just $600.   You can find Stanford Alpaca: An Instruction-following LLaMA Model on GitHub  In a preliminary human evaluation  we found that the Alpaca 7B model behaves similarly to the text-davinci-003 model. For context  GPT-3 175B (davinci) costs about $4 million.   Slowdown In Frontier ModelsOverall  these trends will clash with   High energy requirements (cost and availability) Semiconductor chips constraints  Demand continues strong for H100  with H200 and Blackwell ramps through the second half of the year. Expects demand to exceed supply well into next year (Benchmark  July 12  2024).Lack of training data.Will it make commercial sense for companies to continue training larger and larger models?   Clearly  the gap between closed-source and open-weight models is closing fast if not already closed.   Built-In AI On-DeviceCompanies started releasing physical products with built-in AI.   Samsungs Galaxy AI is an example. The Galaxy AI demos at Unpacked showed the evolution of current Galaxy AI features  with a big focus on live translation features.   Skype has been doing that for years  but Samsungs Galaxy AI reduces friction since it is built-in.   I guess live translation can be useful in some not-so-daily cases (calling a hotel for a vacation?) but Sketch to Image (which lets users generate images using AI) is more of a toy than a real selling point  in my opinion.   Everything else on Galaxy AI is more or less boosting whatever is already available in the Android ecosystem. Apple and other device manufacturers will catch up quickly and effortlessly.   So  whats the point of built-in AI?   Built-In AI In The Browser And AppsAs for Samsungs Galaxy AI  built-in AI in the browser and apps has a few undebatable benefits over using models online:   Virtually Zero CostsFaster Response TimeOffline availabilityLocal processing of sensitive dataVirtually Zero CostsThe cost of running a built-in model is virtually zero. Most likely  we will end up in a hybrid scenario where running models on-device will be free but accessing more powerful capabilities online will have a price.   Startups might capitalize on this possibility to offload the computational costs of running AI to the users.   Telecommunication companies are well positioned to offer AI packages to allow using premium AI features from different providers seamlessly.   Faster Response TimeBuilt-in AI models will have a faster response time.   This is not because models online are slow but because internet connections might be unstable and reduce the user-perceived response time.   Where access to 5G is not an issue  response time might be less of a concern. Furthermore  as online models are becoming faster  we might want to throttle the response time to resemble human interactions.   I dont think generic chatbot startups will have an easy life in this environment. However  startups capitalizing on private expert models might thrive for specific uses and needs.   See Moshi: the first voice-enabled AI. Very hyped launch but when I tested it I was a bit underwhelmed. Maybe my connection wasnt great or their servers were overloaded  anyway great work from the team  and I will try it again.   Unveiling of Moshi: the first voice-enabled AI openly accessible to all.In just 6 months  with a team of 8  the Kyutai research lab developed from scratch an AI model with unprecedented vocalwww.youtube.com   Offline AvailabilityThis is inherently connected to spotty internet connections or no internet connection.   Built-in AI models grant a whole new level of interactions with websites and applications even when offline or with unstable connectivity.   Local processing of sensitive dataEspecially interesting for privacy. Your interactions with on-device models could be entirely offline and private but you can bet this wont be by default across all device manufacturers.   Extrapolating how the mobile phone industry is addressing privacy concerns nowadays  I wouldnt be surprised to see Apple offering a built-in model (on the line of OpenELM  maybe?) that is strict about privacy. But what about other apps and the browser?   Disregarding the OS    Mobile Apps: Should app stores (Apple App Store  Google Play  Amazon Appstore  etc) gate-keep this? Probably they will end up doing it  because you dont want harmful trash on your store  right?Browser: Chrome's built-in AI will be used on iOS  whats Apple's take on it?Until then  cooperation and integrations seem an interesting model  although not very privacy-friendly.   Built-In AI  Small Expert ModelsAll the benefits from built-in AI depend on the possibility of running small expert models on-device.   And BigTech is going there as well.   Microsoft offers the Phi-3 family of models with Phi-3 Mini running on 3B parameters.   Google goes even further by proposing AI Web APIs  web platform APIs and browser features designed to integrate AI models  including large language models (LLMs)  directly into the browser.   In simple terms  developers will be able to create applications using small AI models right on your device as they now use geolocation and Bluetooth on your device  for example.   This is interesting and it opens a lot of possibilities and concerns.   Will Chrome only run Google models? With Edge running Microsoft models  Safari running OpenAI until they get OpenLM ready  and Firefox running something truly open-source  maybe OpenLM again?   Or will developers get to pick and choose the model they want to run on their applications and websites?   What are the incentives and costs for the various stakeholders in the ecosystem?   I dont have a clear answer to these questions yet but the trend seems clear.   Some customers may only need small models  some will need big models and many are going to want to combine both in a variety of ways  Tiny but mighty