December 2023

Ref: 2200j-23

SBD Explores: Next-Gen Virtual Assistant in Car

The revolution of in-car experience withnew artificial intelligence engine
10-minute InsightOEMs and suppliers are rapidly developing next-generation incar virtual assistants powered by advanced Large Language
Models (LLMs). These new virtual assistants aim to deliver anenhanced in-vehicle experience that is more seamless.
However, implementation is important, or the virtualassistant’s capabilities will fall short of users’ expectations.
In this edition of SBD Explores, we delve into theopportunities and challenges in creating these sophisticatedvirtual assistants, examining their implications for bothconsumers and OEMs.

Target audience
StrategyMarketing/ Product Planning

Focus market(s)Global1

What is happening?
OEMs working on next-gen virtual assistants
ChatGPT-powered
Beta programs

Announced interest in
ChatGPT in-vehicle

Non-ChatGPT powered assistants

Suppliers working on new AI engines

Key takeaway
The new wave of generative artificial
intelligence (GenAI) and large language
models (LLM) have had an immediate
impact on in-car virtual assistants. WithGenAI, OEMs can integrate additional
services and embed their brand DNA
more deeply into vehicles. GenAI canalso deliver more natural speech.
• Several OEMs are actively exploring theintegration of ChatGPT into their vehicles.
Currently, Mercedes-Benz and DS
Automobiles are running beta programs
that are open to vehicle owners. The DS
program is capped at 20,000
participants. GM and Stellantis haveannounced interest in incorporating
ChatGPT into their vehicles’ virtualassistants.
• Other OEMs, especially in the Chinese
market where ChatGPT isn't available,
are focusing on developing their own inhouse GenAI model.

Leading tech-giants in AI• Suppliers of traditional virtual assistant
are working to incorporate the latest LLM
technology into their forthcoming
products.
• Tech-giants such as Amazon and Google
are repositioning their virtual assistantproduct teams, aiming to substitute theircurrent AI engines with new ones
powered by LLMs.
2

Why does it matter?
Key takeaway

06

01

Vehicle Services• Smart maintenancerecommendations• Service log tracking• Service scheduling

Mobility Assistant• Navigation• Fuel or charging
station locator

0205

Eco Systems• Make reservations• Pay for parking
and tolls• Purchase products
and services

Virtual assistants will be able
to support the owner via
direct requests and indirect
prompts. Key areas that
virtual assistants can support
are shown in the diagram.

04
Smart Home
• Control lights• Control doors andlocks

03Entertainment• Media controls• Lighting controls• In-car meetings• This innovation transforms the driving
experience by offering natural and precise
conversation at a human level. The
advanced intelligence of LLMs not only
increases usage of virtual assistants invehicles, but also builds consumer trust.
• These new types of assistant are more thanjust a gateway to complex vehicle settings.
They serve as a hub that seamlessly
integrates services like table reservations,ticket bookings, and payments. This ability
opens opportunities for OEMs to create newrevenue streams by incorporating moreservices.

Vehicle Control• Climate control• Smart summon• Vehicle security
management

VirtualAssistant

The next-generation of virtual assistants,
powered by Large Language Models(LLMs), will be a key feature of tomorrow'svehicles. They will be able to impact manydifferent systems within the vehicle.

Well implemented virtualassistants will be fun andconvenient to use.
Consumers will look to use
virtual assistants more as
they develop further.

• With time, the new generation of assistants
will evolve into a trusted agent,
representing both the car and the OEM. It
will learn to understand user behaviour andoffer timely and proactive support.
• While integrating LLMs into mobile devices
like smartphones remains a challenge,
today's smart vehicles are ideal hosts forLLMs. Equipped with the necessary
computing capacity, sensors, connections,and data, they are well-prepared to carrytrue intelligence.
3Where next?
Foundations
Core pieces of technologythat are needed tosupport virtual assistants

Wellness
Sensing

Foundational pieces provide thefirst levels needed to construct
a virtual assistant. From there,capabilities can be defined, andfeatures added.

Robustness

Human-like
Voice

Work Offline

Voice/Face
Biometrics

Multi-Seat
Communication

Personalization

Just Talk

Model
HardwareSurrounding
Awareness

Knowledge
Data

OTA Updates

Capabilities
Specifications that candifferentiate OEMs
from one another.

VehicleKnowledgeTrustworthy

Privacy
Protection

Security &Resilience

To build an engine for a next-gen virtualassistant, four core foundations, amodel, hardware, data, and knowledge
are required. Capabilities build on thesecore elements to allow for features.
• Model – The future intelligence in cars
isn't just a voice assistant, but a digitalagent capable of perceiving the world in
various ways. Therefore, a multi-modal
model is essential. To ensure smooth ondevice inference, the model needs to beoptimized and compressed to fit into aCar system on chip.
• Hardware – This includes an AI chipwith sufficient computing power that canoperate with lower power consumption,e.g. Qualcomm Snapdragon 8 Gen 3.
Hardware also encompasses sensors likecameras, microphone arrays, and motion
detectors for enhanced perception.
• Data – A “smart” model must be trainedusing a large amount of data (corpus)
and fine-tuned with domain-specific data,such as vehicle knowledge. Accumulating
this data might take years.

SafetyBiasManagement

Key takeaway

FeaturesSuccessful virtualassistants will havethese features built-in.

• Knowledge – OEMs can either purchase,build, or co-develop the system withpartners. In any case, intensive
knowledge in UX, vehicle domain
expertise, data science, and ML
engineering is required.
4What to watch out for?
Key takeaway

ActionPhone callInternet search
Settle payment
Reasoning
Route planning•••••

Avatar Control• Speaker, andmicrophone array• Camera• Motion detector• Radar• Exterior sensors

ASRTTS

Dialogue
ManagementSensor Layer

User –Interact
via voice,touch, andgestureCore AIModel

Screen touch
GestureMotionEye positionFacial expression

Human knowledge
Vehicle knowledge
Operation manual
Emergency service
Roadside assistanceInsurance
information• Driver information••••••Non-Voice Data
Processing•••••

KnowledgeTools•••••

Vehicle setting
Vehicle command
Volume control
Door control
Ambience light

The architecture of future virtual assistantsystems is complex. The key components
are shown below:• Multi-modal model is required for futuresystems as the virtual assistant should not
only understand voice commands but alsointerpret non-voice cues such as touch,gestures, and facial expressions. This will
give a more comprehensive interactionexperience.
• Virtual avatars should be used to enhanceuser engagement and to help build
familiarity with the system. They will be afocus of OEMs as they look to create aunique brand experience.
• Knowledge module is one of the key
components that OEMs can use todifferentiate from others. The more
information a virtual assistant has at its
disposal, the more use cases it cansupport.
• Action modules will grow substantially with
the introduction of LLMs and multi-modal
models as virtual assistants develop.
• Dialogue management is a necessary
module for privacy protection and bias
management. User privacy will remain alarge concern for owners as virtualassistants develop.
5How should you react?

1
2
3

Prioritize

Prioritize key virtual assistantfeatures that can benefit theconsumer directly and differentiate
from other brands.

Fine-tune

Choose correct GenAI model andspend most effort on fine-tuning
the model using domain-specificdata.

IterateAcquaint team with the machinelearning update cycle and iteratively
enhance both core model and its
downstream applications.

Authors

Michael Graf

Andy Qiu

Research Analyst

Domain Principal

Related SBD Reports
Ref: 2200e

SBD Explores
AI AutomotiveApplications

Ref:201
Innovation
Guide

Related SBD Consultancy
 Cyber security assessment Technology forecasting UX and CX
 AI strategic advisory

Interested in finding out more?

Most of our work is helping clients go deeper into new challenges andopportunities through custom projects. If you would like to discuss recent
projects that we’ve completed relating to Automotive AI applications,please contact us.
6