2023 AI Readiness Report

2

Scale

Zeitgeist: 2023 AI Readiness Report3Introduction
Table ofContents
AI Year in Review

04

AI Adoption Trends

16

Top Use Cases By Industryii

22

Insurance

24

Retail & eCommerce

25

Financial Services

26

Logistics & Supply Chain

28

ML Lifecycle

30

Working withFoundation Models

31

Data Challenges

33

Data Best Practices

35

Model Evaluation

37

Conclusion

40

About Scale

42

Methodology

44

Scale

At Scale, our mission is to accelerate
the development of AI applications.
That’s why we are excited to
introduce the 2nd edition of Scale
Zeitgeist: AI Readiness Report, asurvey of more than 1,600 executives
and ML practitioners to uncoverwhat’s working, what’s not, and thebest practices for organizations to
deploy AI for real business impact.
Over the past several months, hype about Generative AI has flooded popular discourse. Seemingly
overnight, every CEO began adding Generative
AI to their business strategy. Many influencers
and leaders have opposing views of Generative AI:
some are worried about highly competent AI systems causing mass job displacement, while others
find these models are not enterprise-ready due
to privacy and security concerns. We are here to
get past the hype and help you separate the signal
from the noise on what it really takes to adopt
Generative AI to help you solve your most important business challenges.

Zeitgeist: 2023 AI Readiness Report

Despite the widespread interest in GenerativeAI, with 72% of survey respondents planning toincrease their AI investments this year, we foundthat large enterprises, small companies, and governments are still uncertain about how to adopt
this technology. Executives are quickly realizingthat you can’t take these models off the shelf andexpect to get a unique business advantage – youneed to train them to perform for your specificbusiness needs, with your proprietary data.
We designed this report to explore the real impactof Generative AI, how every industry can adopt
AI to accelerate innovation, and the best practices
companies can follow to maximize their investment in the technology. The results of this reportshow that with the right foundation in place for
implementing AI, employee productivity increases, customer experience improves, and revenue
and profitability grow.
Our goal is to continue to shed light on the realities of what it takes to unlock AI for every business. The AI landscape is moving fast, but we hopethat this research helps demystify the changes thatare underway and sheds light on what businesses
need to take full advantage of this moment.

01

The next 2-3 years of AI aredefinitely going to define thecoming 2-3 decades of the world.
For those in technology: youlive a lifetime for a moment like
this—don’t waste it.
There are decades where nothing
happens, and weeks where
decades happen.
—Alexandr Wang
FOUNDER & CEO, SCALE

02

Scale

Zeitgeist: 2023 AI Readiness Report

03

TO P USE CASES BY IND U STRYAI YEAR IN REVIEWAI Year in
Review
Generative AI
takes a giant leap
Giant leaps in the capabilities of Generative AI
defined 2022. In mid-2022, the image diffusion
models Dall-E 2 (OpenAI) and Stable Diffusion (Stability AI) captured headlines. Many
new startups rushed to build their businesses
on top of these powerful image-creation tools.
In late November, OpenAI released the large
language model GPT-3.5 and the chat interface
ChatGPT which quickly became one of themost impactful technology launches ever. Key
to the impact of this model was the use of Reinforcement Learning with Human Feedback(RLHF), a technique to align model performance with human intent.

04

Scale

Vast amounts of data are collectedfrom a variety of sources, primarily
data from the internet. A model istrained on GPUs, producing a basemodel that is highly capable, but
not aligned to human preferences.

The base model is then fine-tuned
via Reinforcement Learning from
Human Feedback (RLHF) toalign them more closely to humanpreferences and in the case of LLMs
interact in a conversational or“chat” format.

In March 2023, OpenAI launched GPT-4, themost capable large language model ever created. Also aligned with RLHF, GPT-4 impressively
passed many exams designed to test humancapability in the 90th percentile or higher, including several AP exams and the bar exam. In
December 2022, Anthropic announced a closed
beta of Claude, an LLM with chat capabilities
similar to ChatGPT, and an approach to humanalignment based on “constitutional AI,” where
human oversight is provided through a listof rules or principles. In March 2023, Google
released Bard, its conversational AI based on
Google’s LaMDA model. Several other companies are also building large language models, including AI21 labs, Carper AI, Stability
AI, and Cohere. Domain-specific models will

Zeitgeist: 2023 AI Readiness Report

Generative models are thenable to produce unlimited andinfinitely creative imagery, engage
in conversations with users,
summarize documents, and write
code.

also be developed over the coming year, like
BloombergGPT, a model purpose-built for financial use cases. A steadily growing number ofthese models will be trained and released overthe next year.
Generative models are being integrated into
Google Workspace and Microsoft Office, enabling massive productivity gains for business
users. These tools help you write the first draftof a document, generate complete presentations from a single prompt, or automaticallyanalyze and visualize financial data in a spreadsheet. Enterprise software platforms like Salesforce are enabling analysts and executives to gain
new perspectives on data using Generative AI.

05

AI YEAR IN REVIEWAI YEAR IN REVIEWThe significantly improved
capabilities of generative models
in 2022 enormously impacted
businesses’ AI strategies, with 65%
either accelerating their existing
strategies or creating an AI strategy for the first time.

“I think that for this decade, what we’rereally going to see is these tools proliferating.
They’re going to be everywhere. They’regoing to be baked into every company. I
think it’s kind of like the internet transition
If you’re a company, what’s your internet
strategy? And here we are today, where we
don’t even talk about internet strategies. It’s
just so integral to every business. It’s not a
separate part of your business that you canpick or choose whether you’re going to haveit. And I think that AI is going to be muchthe same.”

While most respondents (60%)
are experimenting with generativemodels or plan on working withthem in the next year, only 21%
have these models in production.
Companies see the potential ofgenerative models to improvetheir business, but getting theminto production is challenging.
To unlock the power of their data
and take full advantage of thesemodels, companies need machinelearning expertise, fine-tuning
infrastructure, and the resources
to perform RLHF at scale.

—GREG BROCKMAN,PRESIDENT & CO-FOUNDER, OPENAI(TransformX October 2022)

65%

06

either accelerated their existing
strategies or created an AI strategy
for the first time

Scale

Zeitgeist: 2023 AI Readiness Report07

AI YEAR IN REVIEWAI YEAR IN REVIEW

Most companies don’t have the necessary resources
or mandate to create their own generative models, so they must rely on third parties. Of thosecompanies that plan on working with generativemodels, the vast majority are looking to leverage
open-source generative models (41%) or Cloud API
generative models (37%), while very few are looking
to build their own generative models (22%).

61% of companies are looking to
AI to help enhance the customer experience, 56% to improve
operational efficiency, and 50%
to increase profitability. Focusing
on customer-centricity benefits
organizations immensely, withimproved customer goodwill inthe short term and greater profitability in the long term.

Furthermore, 28% are exclusively using open-source
models, while 26% use cloud APIs (commerciallyavailable models such as Anthropic’s Claude,
OpenAI’s GPT-4, and Cohere’s Command), and only
15% are exclusively building their own.

89% of companies adopting AI benefit
from the ability to develop new products or services, 78% from enhanced
customer experiences, and 76% frombetter collaboration across business
functions. These companies also see
improved organizational process
efficiency and profitability. Despite
the positive outcomes for AI adopters,even greater outcomes are possible ascompanies accelerate their AI strategies
and increase their investments in AI.

There are multiple factors thatenterprises must weigh when deciding
on their Generative AI infrastructure,including their in-house machinelearning expertise, budget, security requirements, and need for domain-specific capabilities. Leveraging a cloudprovider is the easiest and fastest
path to obtain generative capabilities,
but it comes with higher security
risk, less control over the underlying
models, lower quality performance atdomain-specific tasks, and can be expensive. Open-source models provide
more control and are cheaper, butthey require more in-house expertise
to deploy and fine-tune. Companies
looking to build their own generativemodels benefit from greater control
but incur greater costs from datacollection, compute, and hiring machinelearning experts to train and deploy them.

08

An organization’s goals also shape theeffectiveness of its AI implementation.
Those companies that list shareholder/investor demand as the top goal for AI
adoption also show the poorest results
for customer experience, revenue, andprofitability. To ensure success with an
AI implementation, organizations must
avoid implementing AI for the sake of
implementing AI, but instead, ensure
the goals of an AI implementation arealigned with company priorities andthat AI is a good solution for a given
problem.

Scale

Zeitgeist: 2023 AI Readiness Report09

AI YEAR IN REVIEWAI YEAR IN REVIEW

A New
Era

Models Are Increasing in Size

Generative models are alreadytransforming how we create art,
understand our world, and conductbusiness.
Large language models help us write content
such as blogs, emails, or ad copy more quickly
and creatively. They summarize long-form
content so that we can quickly understand themost critical information from reports andnews articles. Diffusion models streamline
marketing workflows, enabling marketers
to generate unlimited and infinitely creative
product imagery. Developers use LLMs to write
code more efficiently and help them quickly
identify and fix bugs. Advanced chatbots enablebusinesses to improve their customer service
at a lower cost. Finally, organizations areunlocking the power of their knowledge bases
by customizing LLMs with their proprietarydata to perform better on tasks unique to theirbusiness.

Over time, generative models have become morecapable as they’ve increased in size. Model size istypically determined by its training dataset size
measured in tokens (parts of words) or by its
number of parameters (the number of values themodel can change as it learns).
• BERT (2018) was 3.7B tokens and 240 million parameters.
• GPT-2 (2019) was 9.5B tokens and 1.5 billion parameters.
• GPT-3 (2020) has 499B tokens and 175 billion parameters.
• PaLM (2022) was 780B tokens and 540 billion parameters.

We will now look at a few key terms and trends
essential to understanding this new era ofGenerative AI.

10

As these models scale in size, they become
increasingly capable, providing more incentive
for companies to build applications on top.
Generative models are now more widely available as many large model developers provide
APIs or make them open-source, and companies are quickly adopting these large models to

Scale

Zeitgeist: 2023 AI Readiness Reporttheir specific business use cases.
Generative models are trained on a large
amount of internet data, making them competent generalists. These models can write poetry,solve logic puzzles, and identify bugs in code.
While generative models are great generalists,they are poor specialists when solving problems
outside of their data distribution. Since a significant portion of data is proprietary to individualorganizations, base large language models are
not well adapted to these specific domains.
To improve performance on the specific tasksof, say, an insurance company, an eCommercecompany, or a logistics company, these models
must be fine-tuned and aligned to excel at thoseparticular tasks and provide responses that are
useful to customers and employees.

11

AI YEAR IN REVIEW

Reinforcement Learning from Human
Feedback (RLHF)
Though Reinforcement Learning from Human Feedback (RLHF) is not new to the research community,
in 2022, it catapulted in popularity as it was a criticalingredient in the success of ChatGPT.
Instead of attempting to write a loss function withwhich to train a model, RLHF involves soliciting
feedback from human users and training a reward
model on that feedback. This human-defined reward
model is then used to train a base model. This also
allows training on much more data since the human
feedback is mimicked by the reward model, so thedataset size is now only constrained by how many
prompts you can create.
RLHF tuning results in models better aligned to human preferences, producing more detailed and factual
responses.
RLHF also defines the “personality” and “mood”
of the model, making it more helpful, friendly, andfactual than the base model would be otherwise.
This means we get responses from the model thatfeel more human and less like talking to a machine.
RLHF is a critical component to the success ofrecent LLMs and is also critical to ensuring thatenterprises using Generative AI get model responses
that align with their policies and brands.

12

AI YEAR IN REVIEWChatGPT

Prompt Engineer

ChatGPT is a large language model that has been
tuned specifically for the task of conversational text
generation. ChatGPT was trained with RLHF anddata in dialogue formats to enable it to act as a conversational chatbot.

2022 also saw the proliferation of a new role for machine learning teams, the “prompt engineer.” While
generative models are competent at generating
the desired output for business use cases, the rightprompt is required to optimize the model’s effectiveness. Prompt engineers carefully craft natural
language inputs to get more consistent and reliable
responses from models so that the model outputs
can then be used in business applications. Insteadof writing an SQL query, these engineers craft finely
honed natural language prompts.

ChatGPT quickly became one of the most impactful
product launches of all time, reaching 1 million users
in just five days and currently sits at over 100 million
users.
ChatGPT was initially launched with GPT-3.5 but
now also includes GPT-4 for ChatGPT plus subscribers. These models are highly capable of question
answering, content generation, and summarization.
While these models provide more robust, informative, and creative responses than their predecessors, the real breakthrough for adoption was theirability to hold conversations with humans. The
ability to interact with the models in an intuitive
way increased the accessibility of the models so thatanyone can use them.

Scale

For example, when integrating applications with
LLMs, the varied responses from the model canbreak the integration if they are not formatted properly. Say you are creating an application dealing withfinancial data. A user’s input may be related to theQ4 earnings of a particular company (see graphic,right).
Prompt engineers help model responses to solve
business challenges with greater accuracy, efficiency,and quality. They also ensure that responses alignwith an organization’s brand guidelines and voice.
They are also critical at finding vulnerabilities inmodels by using prompt injection techniques andhelping to resolve those vulnerabilities before an
employee or customer does. This role is highly
valuable in ensuring organizations’ successful implementation of generative models.

Zeitgeist: 2023 AI Readiness Report

13

AI YEAR IN REVIEWAI YEAR IN REVIEW

What to Expect in 2023

“One of the ways that I describe Stable
Diffusion is as a generative searchengine. You don’t need to use image
search anymore cause you can justmake the image. By putting this in apipeline at the right place and having
human-in-the-loop interactions like thework that you do at Scale AI, understanding how humans do that at scale,and having these engines, it will allow
us to have even better experiences that
understand what we want.”

Increased investment in AI

Proprietary data will unlock
the power of generative models

As Generative AI is now more capable and widely
available, companies are quickly incorporating it
into their operations. 72% of companies will significantly increase their investment in AI each year forthe next three years.

On their own, base generative models are valuable
tools. Paired with a business’s proprietary data,
they become strong differentiators, improving thecustomer experience, product development, andprofitability.

Increasing capabilities of generativemodelsMany organizations are now building their own large
generative models. These models are being incorporated into search engines and paired with othertools, including internal knowledge bases, to become
powerful business tools. These models will also
become multimodal, meaning that they will be able
to consume and generate text, images, and video,
making them even more useful than they are today.
You can upload a product catalog with both images
and text to a multimodal model, and it will recognize specific products, write product descriptions,fill in missing attributes, and generate new images toenrich your product catalog automatically.

— E M A D M O S TA Q U E ,F O U N D E R & C E O , S TA B I L I T Y A I

for our models to start using tools
out in the world. Giving them accessto knowledge bases, giving themaccess to search engines like Google
or Bing, and augmenting them withthat knowledge as a resource so thatinstead of them having to memorize
all of these facts, they can makereference to a live, updated, knowledge
base. I think that’s going to be super(TransformX October 2022)

Widespread accessibility of generativemodelsMuch like the cloud, widely accessible generativemodels represent a paradigm shift for companies.
Incorporating this type of AI will quickly become thestatus quo, and those who are slow to adopt will beleft behind.

14“I’m most excited about...the ability

Scale

Zeitgeist: 2023 AI Readiness Report

impactful.”—AIDAN GOMEZ,CEO, COHERE(TransformX October 2022)

15

Adoption Trends

AI Adoption

16

Scale

Business leaders have identified that AI is criticalto the future of their companies and are looking
to adopt it as quickly and with as much impact
as possible. We examine this trend and provide
insights on best practices.

72%

59% of companies view AI as critical or highly
critical to their business in the next year, and
69% in the next three years. The increasing
capabilities and availability of Generative AI will
accelerate AI adoption.

As companies view AI as more critical to the future success of their business, they are increasing
AI investments over the next three years. 72% ofcompanies plan to increase their investment in AI
each year for the next three years.

Zeitgeist: 2023 AI Readiness Reportof companies are looking toincrease their investment in AI
each year for the next three years

17What outcomes are achieved by companies that adopt AI?
As mentioned previously, companies adopting AI are seeing
positive outcomes from improved customer experiences,
the ability to develop new products or services and improveexisting products, and improved
collaboration across business
functions.

Companies are spending less time and effort on
traditional computer vision applications and instead
focusing on LLMs and Generative AI. Of companies making significant investments in AI, 52% areinvesting heavily in LLMs, 36% in generative visualmodels, and 30% in computer vision applications.
With the recently popularized capabilities of LLMs,
companies have rapidly shifted their AI strategies toharness the power of Generative AI.

18

Of companies making
significant investments in AI:

52%
36%
30%

Across the board, companies
adopting AI are achieving positive outcomes in almost every
category. Like any technology
program, the success of AI
programs depends on the ability
to implement AI and align implementation efforts with measurable organizational goals.

are in investing heavily
in LLMs“I really believe that we are at a transformative moment today where ML is moving at an incredible
are in investing heavily
in generative visualmodelsspeed and problems that were thought to be too complex to solve with computers a few years ago,
are now being solved by applying machine learning. So we have this great opportunity. If machinelearning becomes more accessible, the world will move faster, our economy will move faster, science
will move faster.”are in investing incomputer vision models— F R A N C O I S C H O L L E T,
AI ENGINEER AND RESEARCHER, GOOGLE(TransformX October 2022)

Scale

Zeitgeist: 2023 AI Readiness Report

19Which resources do companies feel they have enough of inorder to successfully implement their AI strategies? Which
resources are they lacking?
Companies that view AI as critical to their businessindicate they have the executive support, strategy and vision, and budget they need to succeed in
implementing their company’s AI strategy. However,these companies generally lack the necessary expertise, software, and tools required to achieve success.

While leaders have identified the need to adopt AI,
the execution of these strategies is difficult, nuanced, and heavily dependent on expertise. The field
is moving so quickly that it is difficult to keep up
with the pace of advancement. Highly talented people with expertise in Generative AI are simply notavailable to most organizations. Similarly, selecting,standardizing, and updating the software and tools
associated with Generative AI, MLOps, and even
DevOps is challenging for companies without dedicated teams to keep up with these changes as therequisite tech stacks are constantly evolving.

“As a product of this shortage in AI talent,
most businesses are missing out on a huge
opportunity to integrate this tech into
products and into their developer’s workflows.
Consumers are missing out on products thathave more magical, intuitive, and smart
experiences. The start of the fix comes withthe product folks making the decision about
what’s prioritized, looking at what they cando, understanding where the technology istoday, and where they could insert it, and thenbuilding it. I think we need to start making AI a
standard piece of every single product. I don’t
think consumers are going to tolerate dumbproducts anymore. We need to make themmuch, much smarter.”—AIDAN GOMEZ,CEO, COHERE(TransformX October 2022)

20

Scale

Zeitgeist: 2023 AI Readiness Report21

TO P USE CASES BY IND U STRYTO P USE CASES BY INDUSTRY

AI Adoption
by Industry
Every industry is looking toincrease its AI budgets over the
next three years. Those that top
the list are:

80%

79%

77%

75%

74%

Logistics &Supply Chain

Healthcare &Life Sciences

22

Insurance

While all industries are increasing their AI budgets,each industry has unique use cases. These range frominsurance companies looking to reduce claim processing
times to eCommerce companies implementing customer
service chatbots. We examine how a few key industries
are adopting AI in 2023.

Financial
Services

Retail &eCommerce

Scale

Zeitgeist: 2023 AI Readiness Report

23

TO P USE CASES BY IND U STRYTO P USE CASES BY INDUSTRY

Insurance

Retail and eCommerce

Insurance companies look to AI
to help them improve customer
experience and improve operational efficiency.

Retail and eCommerce companies
look to AI to help them growrevenue, improve the customer
experience, and increase operational efficiency.

To help achieve these goals,
insurance companies are looking
to adopt AI to improve claims
processing, fraud detection, andrisk assessment/underwriting.

To help achieve these goals,
retail and eCommerce companies
are adopting AI to improve thecustomer experience with morecapable chatbots. They also want
to improve operational efficiency
with more productive content
and marketing operations built on
AI-generated product imagery anddescriptions. These companies arealso enhancing their operational
efficiency with better forecasting,purchasing, pricing, and inventory
management. Retail and eCommerce companies are not focusing
on adopting AI to directly growrevenue but are indirectly looking
to influence revenue growthwith increased operational
efficiency and an improved
customer experience.

For claims processing in particular, insurance companies believethat AI can help reduce timeto process claims and reduce
processing errors which will
result in a better experience for
their customers and improved
operational efficiency.

24

Scale

Zeitgeist: 2023 AI Readiness Report25

TO P USE CASES BY IND U STRYTO P USE CASES BY INDUSTRY

Financial Services

Financial Services

Financial services companies
look to AI to help them enhance
the customer experience, growrevenue, and increase operational
efficiency.

Content summarization includes
summarizing data sources suchas financial statements, historical data, news, and social media.
Trend detection is applying AI
to data to help identify patterns
humans are otherwise ill-equipped
to detect.

To help achieve these goals,
financial services companies arelooking to adopt AI to improve
investment research, fraud detection, customer-facing process
automation, and to power personalized chatbots.

Financial services companies areusing financial statements, historical market data, and 3rd party
data in their investment models.
Fewer are relying on social media
content and geospatial/satellite
imagery.

For investment research in particular, financial services companies
are applying AI to summarize
content, detect trends, and classify topics to improve investment
decisions, resulting in increased
revenue and improved operational efficiency. By improving theirinvestment decisions, these organizations will indirectly improvethe customer experience throughpresumably higher returns.

26

Scale

Zeitgeist: 2023 AI Readiness Report27

TO P USE CASES BY IND U STRYTO P USE CASES BY INDUSTRY

Logistics and Supply Chain

Logistics and Supply Chain

Logistics and supply chain companies adopt AI to help them
improve operational efficiency,
improve customer experience, andgrow revenue.

For route planning, logistics andsupply chain companies believe AI
can help improve efficiency, reduce
costs, improve delivery accuracy,
and reduce shipping times. This
directly translates to improved
operational efficiency and a better
customer experience while indirectly translating to revenue growth.

To help achieve these goals,
logistics and supply chain companies are looking to adopt AI for
better inventory management anddemand forecasting, improved
route optimization, to deploy autonomous vehicles, and improve
document processing throughput
and quality. These tools directly
impact operational efficiency,
which has downstream impacts on
the overall customer experience,with reliable delivery and fewer
delays.

For document processing, logistics
and supply chain companies look
to AI to help them with information processing, document classification, and compliance. Thisapplication is strictly dedicated toincreasing operational efficiency
and reducing costs.
Logistics and supply chain companies process a lot of paperwork.
To improve operational efficiency,
this paperwork must be processed as quickly and accurately
as possible. Logistics documents,
such as bills of lading, commercial
invoices, and packing lists are fullof critical information required toclear shipments past customs andonto warehouses for distribution.
Traditional OCR applications require the creation of templates foreach type of document, which isinfeasible and inefficient for global
logistics companies. Instead, thesecompanies rely on machine-learning-powered document processing, which requires no templates
and still processes the documents
at over over 95% accuracy.

For inventory management anddemand forecasting, logistics
and supply chain companies are
adopting AI to help reduce costs,
improve customer satisfaction,and improve forecast accuracy.

28

Scale

Zeitgeist: 2023 AI Readiness Report29

ML Lifecycle

Working with Foundation Models
As of late 2022, the most widely used foundation models were
BERT, GPT-3, Stable Diffusion, BLOOM, and T5/FLAN.
However, this landscape is quickly changing as more and more powerful generative models are being developed.*

BERT plays a critical but quieter role in many organizations today, providing natural language understanding capabilities at a significantly reduced costcompared to larger models such as GPT-3.
However, this trend is shifting as BERT is not a generative model, so its use cases are limited compared

to state-of-the-art models like GPT-4. The computerequired to run these more sophisticated models will
become cheaper over time, and companies will havemore third-party tools and expertise available to help
integrate these larger models into their operations.
Open-source models such as LLaMA are already being optimized to run on consumer laptops.

HELP IMPROVE THESE INSIGHTS

We’d appreciate your help to improve these insights! Given the rapid progress of Generative AI and thefrequent introduction of new models, we’re curious to learn which models are now the most widelyadopted. Please take a moment to fill out the survey.

*(ChatGPT/GPT-3.5/GPT-4 were not included in this survey as survey collection began in late 2022 before these models were launched)

30

Scale

Zeitgeist: 2023 AI Readiness Report31Data Challenges

As previously discussed, of those companies thatplan on working with generative models, the vast
majority are looking to leverage open-source
generative models (41%) or cloud API generativemodels (37%), while very few are looking to build
their own generative models (22%).
The majority of companies using Cloud API LLMs
are using OpenAI (64%), followed by AI21labs(26%) and Cohere (26%). Google and Anthropic
recently launched their own LLMs, and several
other companies expect to launch their own models in 2023.
To get the most value out of large generativemodels, many enterprises will need to fine-tune
foundation models using their proprietary data
and knowledge bases. The biggest challenges
for fine-tuning foundation models are acquiring
training data and the necessary ML infrastructure.
Organizations working with generative models
find it challenging and resource-intensive to finetune their models in-house. The most effective
techniques, like RLHF, require humans to apply
feedback to model outputs, custom software, andspecialized skill sets to ensure high quality and limit human biases in models.

32

64%
26%
26%

Data quality remains the most challenging part of acquiring training data, closely
followed by data collection. The challenges
for acquiring training data are nearly identical to last year’s survey findings.
Curating data and annotation quality arestill the top challenges for companies
preparing training data for models. The
biggest change YOY was that it takes toolong to get labeled data (typically greaterthan one month), with 21% of respondents
citing this as their #1 challenge, up fromonly 13% a year ago. Companies are looking
to move faster to label their data, but it isdifficult to keep up.

Scale

Zeitgeist: 2023 AI Readiness Report33Data Best Practices
The majority of respondents continue to have problems with
their training data.

Companies that invest in good data annotation infrastructure
can deploy new models, retrain existing ones, and deploy theminto production faster.

Most respondents continue to face problems with
the quality of their training data. Data noise remains
the number one issue (64%), followed by data bias(50%), and domain gaps (43%).

Our data show that there seems to be a correlation between teams that get annotated data fasterand teams that deploy new models to production
faster and are able to update existing models more
frequently.

34

Scale

Zeitgeist: 2023 AI Readiness Report35Model Evaluation

Respondents that only label their data manually mostfrequently rank labeling quality as their number one
challenge in preparing training data, while those thatleverage human-in-the-loop and automated labelingrank this challenge less frequently. 60% of respondents leveraging manual labeling in some capacity

36

rank labeling quality as their number one challenge,
compared to 47% using automated labeling and only
44% using human-in-the-loop labeling. The combination of automated labeling plus human-in-the-loop
is recommended as a best practice as it nearly always
outpaces the accuracy and efficiency of either alone.

Scale

While most of the report has focused on RLHF
and Generative AI, companies apply machinelearning in many different ways, from object detection to recommendation systems. One criticalcomponent of these production ML systems is
how companies evaluate and monitor their performance.

Zeitgeist: 2023 AI Readiness Report

Just as we found in 2022, measuring the business
impact of models remains a challenge, especiallyfor startups or very small companies (those withfewer than 250 employees). These companies
rely more on aggregated model metrics as they
are building products and a customer base, so thebusiness impact is difficult to measure. However,small to medium size organizations (500-9,999
employees) are measuring business impact morethan they were even one year ago, at about 73%
now (compared to 55% in last year’s survey).

37

Larger companies take longer to identify issues in models

Model Evaluation Methods vs. Time to Deployment

As in last year’s report, smaller organizations canusually identify issues with their models quickly, inless than one week. Larger companies (those with5,000 employees or more) may be more likely tomeasure business impact but are more likely thansmaller ones to take longer (one month or more) to
identify issues with their models.

As we found in last year’s report, ML teams that
identify issues fastest with their models are most
likely to use A/B testing when deploying models.

These larger companies are operating complex systems at scale, and the issues they face are less likely
to cause immediate business impact, so they are notcaught as quickly as startups or small companies,
which are operating simpler systems and closely
monitoring model metrics. Additionally, these larger
companies typically have a larger customer base,so their users will hit on more edge cases than thesmaller companies. For these reasons, it is important that as companies scale, they continually refine
their MLOps practices, use data curation tools that
can help them identify edge cases, and monitor theirmodels while continuing to measure business impact.

Aggregate metrics are a useful baseline, but as enterprises develop a more robust modeling practice,tools such as “shadow deployments,” ensembling,
A/B testing, and even scenario tests can help validate
models in challenging edge-case scenarios or rareclasses.

While we have covered best practices for evaluating models, we must note that for the first time,
the performance of Generative AI models is nearly
impossible to evaluate automatically. This is because
there are many ways for the model to respond

38

Scale

Zeitgeist: 2023 AI Readiness Report

Although small, agile teams at smaller companies
may find failure modes, problems in their models, or
problems in their data earlier than teams at large enterprises, their validation, testing, and deployment
strategies are typically less sophisticated. Thus, withsimpler models solving more uniform problems forcustomers and clients, it’s easier to spot failures.
When the system grows to include a large market
or even a large engineering staff, complexity andtechnical debt begin to grow. At scale, scenario tests
become essential, and even then, it may take longer
to detect issues in a more complex system.

appropriately, and human judgment is required toevaluate correctness. That means any sound model
evaluation, whether done by a generative modelbuilder or an enterprise, will require human-in-theloop validation and verification on an ongoing basis.

39

Conclusion
Generative AI is rapidly transforming
the world, and businesses needto understand how to adopt thistechnology quickly or get left behind.

“Large generative models are alreadygiving people a productivity boost—
we’ve seen how these systems help
people write, code, learn, and more.
We expect the capabilities of these

The most significant AI and ML readiness trend has
been the enormous impact of Generative AI on companies, large and small, across all industries. Whilethe 2022 AI Readiness Report focused on companies
with in-house machine learning expertise, this year’s
report examined how all companies can adopt AI.
This change in focus reflects the zeitgeist: dramatic
improvements in the capabilities of Generative AI toaccelerate innovation and transform every business.
We found that many companies plan to work withor experiment with foundation models, but manylack the expertise and tools to get these models
into production. Most companies are adopting AI toenhance the customer experience, optimize operational efficiency, or improve profitability. Generativemodels will become increasingly more useful andwidely accessible, making them essential to everyorganization’s business strategy with a similar impact to the internet. Early adopters of AI are seeing
the improved ability to develop new products orservices, enhanced customer experience, and bettercollaboration across business functions, in addition
to improved revenue and profitability.

40

Enterprises widely use older non-generative models
like BERT, but have realized they must adopt moregenerative models to stay ahead of the competition.
Companies that fine-tune foundation models findtheir most significant challenges are acquiring training data, ML infrastructure, and comparing experiments across different models. Human evaluation
has replaced benchmarks as the de-facto method toanalyze large generative models and determine how
they will work in a specific enterprise. Enterprises
and governments need to leverage their unique data
to unlock the full potential of generative models.
At Scale, our mission is to accelerate the development of AI applications to power the most ambitious
AI projects in the world. To support that mission, weare excited to share the results of the Scale Zeitgeist:
AI Readiness Report with you. We will continue toshed light on what it really takes to adopt AI and help
you separate the signal from the noise.

Scalemodels to rapidly improve, possibly
beyond our imagination. If we canlearn how to safely integrate AI
into businesses by creating helpful,harmless, and honest systems, it
could have a transformative effecton the economy and industries as weknow them.”—JA R E D K A P L A N
C H I E F S C I E N T I S T, A N T H R O P I C

Zeitgeist: 2023 AI Readiness Report41AboutScale
At Scale, our mission is toaccelerate the development ofAI applications. We believe thatto make the best models, youneed the best data.
The Scale Enterprise Generative AI Platform leverages your enterprise data tocustomize powerful base generative models
to safely unlock the value of AI. The Scale
Data Engine consists of all the tools andfeatures you need to collect, curate andannotate high-quality data, in addition torobust tools to evaluate and optimize yourmodels. Scale powers the most advanced
LLMs and generative models in the world
through world-class RLHF, data generation,model evaluation, safety, and alignment.
scale.com

42

Zeitgeist: 2023 AI Readiness Report43Methodology

This survey was conducted online within the
United States by Scale AI from December 15, 2022,
to January 25, 2023. We received 2,909 responses
from ML practitioners (e.g., ML engineers, data
scientists, development operations, etc.) andleaders involved with AI in their companies. Afterdata cleaning and filtering out those who indicatedthey are not involved with AI or ML projects and/or are not familiar with any steps of the ML development lifecycle, the dataset consisted of 1,699
respondents. We examined the data as follows:
When asked to describe their level of seniority intheir organizations, over one-third of respondents(37%) reported they are an individual contributor, nearly one quarter (25%) said they function
as a team lead, 33% are a department head or
executive, and 4% are owners. Most come fromsmall companies with fewer than 500 employees(39%) or large companies with more than 10,000
employees (21%).

sciences (4%), media/entertainment/hospitality(3%), manufacturing (2%), and other (6%).
Many respondents (31%) represent organizations
that are advanced in their AI/ML adoption—they
have multiple models deployed to production andare regularly retrained. About 18% are slightly lessadvanced—they have multiple models deployed
to production—while 8% have only one model
deployed to production, 12% are developing theirfirst model, and 11% are only evaluating use cases.

24% of respondents represent financial services/
insurance, 22% represent the software/Internet/
telecommunications industry, followed by retailand eCommerce (13%), logistics and supply chain(9%), education (7%), business and customer services (6%), automotive (5%), healthcare and life

44

Scale

Zeitgeist: 2023 AI Readiness Report45