#635

In-Car HMI UX Evaluation& Benchmarking

UX
TABLE OF CONTENTS

EVALUATION

Executive Summary
Introduction to this ReportIn-car HMI UX
Evaluations

Features and Functionality
Execution
Perceived Quality

Mercedes-Benz E-ClassADAS Domain

Infotainment Domain
Navigation Domain

In this edition of the In-Car HMI UX Evaluation and Benchmarking series, the UX team is testing the
Mercedes-Benz E-Class.

Voice Recognition Domain
Connected Features Domain
Convenience Domain

RELATED SBD REPORTS
619 – UX Benchmarking Series
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s UX Team evaluates the infotainment
user experience of over 40 vehicles.
806 – ADAS HMI Evaluations
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s Autonomous Car Team evaluates the
ADAS performance and usability of over 20
vehicles.

COVERAGE

FREQUENCY

PUBLICATION FORMAT

PAGES

6
NA

CHINA

EUROPEANNUALLYQUARTERLYCARS PER YEARPD F

POWERPOINT

EXCEL

ONLINE

150+

#635/ In-Car HMI UX Evaluation & Benchmarking

Scoring> Features and functionality: > Reliability/stability:
evaluating whether theevaluating the repeatedsolutions provide featuresusability and whether thethat customers expect, needusers can have a similarand solve problems (or(positive) experience eachprovide a wow factor).
time.
> Usability:
evaluating whether the> Perceived quality:
features available are easy
evaluating the potential
to learn and use. This
perception in quality of theconsiders areas such as
HMI components and howergonomics, legibility,
this contributes to the overallusability characteristics andcustomer experience.
how the system implements
the various features.

This research is useful for

PRODUCT PLANNERS

M A R K E T I NG

C- S UITEDo I have access?

USER E XPE R IENCEE NGI NE E R S

Portal access

Request a quote for
In-Car HMI UX Evaluation & Benchmarking Series
Mercedes-Benz E-ClassIn-Car HMI UX Evaluation & Benchmarking
Mercedes-Benz E-Class

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – Mercedes-Benz E-Class
Introduction≫

5▪ Report introduction

The basics≫

22

Features and functionality≫▪ Aim of report▪ Overview▪ Scope of report▪ Delight and performance features▪ 2024 vehicles▪ Hygiene and navigation▪ UX principles▪ SBD experience▪ ADAS feature set▪ What’s new in vehicle UX?

▪ Methodology▪ Comparison of past vehicles▪ Report structure▪ IoT integration, music, entertainment
and information

Bird’s-eye view≫

7▪ 645 Hands-free driving▪ OTA, FaaS + SW-defined features guide

Analysis≫▪ The cockpit guide▪ SUS scores by domain▪ Health and wellbeing in automotive▪ Kano analysis▪ Digital cockpit & infotainment forecast▪ Positive and negative implementations▪ CX recommendations

Executive summary≫▪ Key takeaway 1▪ Key takeaway 2▪ Key takeaway 3▪ Ranking▪ Scoring16▪ Potential improvements

29

Execution≫▪ Overview▪ System scoring▪ UX laws▪ System performance▪ Stability▪ Ergonomics▪ Legibility and readability

36

42

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – Mercedes-Benz E-Class
Perceived quality≫

62

Navigation domain≫

105

Convenience domain≫▪ Introduction▪ Summary▪ UX heuristics▪ Definitions▪ UX heuristics▪ HVAC summary▪ Evaluation results▪ Navigation specific scoring▪ HUD summary▪ Highlights▪ Key positive and negatives

ADAS domain≫

70

Voice recognition domain≫▪ Summary▪ Summary▪ UX Heuristics▪ Functionality▪ SAE L0

▪ Recognition▪ SAE L1&2

▪ SUS score▪ SBD UX principles

Infotainment domain≫87▪ Command structure▪ Summary▪ Localization▪ UX heuristics▪ Level of integration

Future outlook≫
126

136

145

▪ Introduction▪ UX principle groups

Next steps≫

150

Contact us≫

159▪ Key positive and negative

Total number of pages - 159

Introduction

Introduction

Report Introduction
Welcome to the 2024 HMI benchmarking report series. This report has been created toprovide a fair, unbiased and objective view of the latest in-vehicle HMI solutions in the US,
European, and Chinese markets. Evaluations are carried out by SBD usability experts witha deep understanding of CASE domains such as the Connected Car and ADAS & autonomy
domains.
SBD supports clients throughout the development of new HMI and products from a
relatively simple companion app to a more complex multi-domain infotainment solution.
The methodologies used in these reports take into account many years of experience with
consumer testing and custom client projects to provide a fair and, as much as possible,
objective methodology.
All viewpoints and analysis within the report are aimed defining areas of concern througha data driven approach. This report aims to benchmark and score solutions whilst also
being able to provide actionable recommendations to design and development teams.

Please note that due to the ever-evolving automotive technology market, SBD updates it’s
methodology each year, but does not update scores from the previous years. Therefore,
please assume a slight drop in scores for both user experience (UX) and functionality fromthe previous year.

Intuitive

7

Consistency

Supportive

Brevity

Flexibility

Depth

Presentation

SectionContent

Birds Eye View

An overview of the key findings from SBD’s various CX related andadjacent reports.

Executive
Summary

Presents key highlights and conclusions from the report.

The Basics

What do you need to know about SBD’s CX evaluation
methodology?

AnalysisAnalysis of report findings by SBD experts.

Features andfunctionality
Execution
Perceived quality

Overview of key features and functionality by domain.
Assess success of implementation and overall execution of various
system elements.
Scoring and analysis of interior perceived quality levels.

ADAS domain

Highlight and analysis of key positive and negative points within
the ADAS domain.

Infotainment
domainHighlight and analysis of key positive and negative points within
the infotainment domain.

Navigation domain

Highlight and analysis of key positive and negative points within
the navigation domain.

Voice recognition
domainSummary and scoring of various aspects of the voice recognition
system.

Convenience
domainSummary of various convenience focussed features.

Future Outlook

Seven UX principles are considered against drivers and barriers
into the future of this reports test vehicle.

Next StepsCan SBD help you with any unanswered questions?
7

The Basics

Aim of this report▪ Features and functionality: evaluating whether the solutions provide
features that customers expect & need, and solve problems (or provide a
wow factor)▪ Usability: evaluating whether the features available are easy to learn anduse. This considers areas such as ergonomics, legibility, usability
characteristics and how the system implements the various features

Brand perception

Consumer testing

One of the core goals of these studies is to provide a true indication of what
the final customer experience of each solution could be. To do thisevaluations are focused on providing scoring and analysis in the following
areas:▪ Reliability/stability: evaluating the repeated usability and whether theusers can have a similar (positive) experience each time
▪ Perceived quality: evaluating the potential perception in quality of the
HMI components and how this contributes to the overall customer
experience“Do I have a positive emotional attachment to thebrand?”

Aesthetics and engagement“Do I have a positive emotional attachment to thesolution?”

EMOTIONAL
APPEAL

Repeated usability

Expert testing(the focus of this report)

RELIABILITY“Can I have a similar experience each time?”

Ergonomics, usability characteristics &execution“Are the features available easy to learn
and use?”USABILITY

Domain functionality &prioritisation“Does the system provide features I
want and need to use?” i.e. “Does it
solve problems for me?”

8

BRAND
APPEALFUNCTIONALITY

SBD’s view on the hierarchy of needs for CX benchmarking
8The Basics

Scope of report: focus on in-car HMI evaluations
The scope of evaluations in this report are constrained to the in-car HMI
experience, in both static and dynamic conditions. One notable element isdriver distraction which SBD covers at only a high level in this study as
carrying out a full driver distraction evaluation requires biometrics test
equipment to ensure the collected data is unbiased and objective.

Prepurchase

Purchase

Activation

In-car HMI

Driverdistraction

IoT

A full evaluation of the end-to-end customer experience is not within scope of
this report, but it is something which SBD has many years experience in fromboth a consumer and expert perspective. Other areas such as the companion
app, online portal and in-home smart devices are not in scope as they aredefined as “out of car” experiences.
Within the vehicle, any HMI element the user interacts with is evaluatedincluding steering wheel controls, touch screen displays, voice control, HUDs
and digital keys. The features and services on offer have been broadlygrouped into the following domains (or test areas):▪ ADAS domain▪ Infotainment domain▪ Navigation domain▪ Voice recognition domain▪ Convenience domain
99The Basics

2024 vehicle list
SBD has chosen nine cars to evaluate in 2024, based on two selection categories. New/interesting UX focuses on systems with to never-seen-before
features or functionality, or the implementation of a solution that has previously been a challenge or pain-point for end-users. New mass-market UX
includes vehicles in segments that are sold in high numbers and are entering a new generation of UI for that vehicle. While we make best efforts toadhere to the chosen cars and schedule, the last two years have seen release dates slipping significantly, so it may be necessary to makesubstitutions.
Group A
Cars tested

Priority target vehicles for 2024

BMW X1

Mercedes-Benz E-Class▪ First BMW to be released
with iDrive9▪ Android based system▪ US market test

▪ All-new MBUX Superscreen▪ Unique features, apps andgames▪ German market test

Hyundai Kona/Kona Electric▪ 12.3-inch cluster andcentral display▪ New Bluelink+ services▪ US market test

Volvo EX90
▪ Volvo Pilot Assist▪ 14.5-inch central display▪ EU or US market test

Ford F-150 Lightning▪ BlueCruise hands free
▪ EV truck▪ US market test

Group B
Other vehicles being considered for 2024

Xiaomi SU7▪ Xiaomi’s first vehicle offering▪ New vehicle from CE company▪ Xiaomi Pilot MAX▪ Chinese market test
10

Audi Q6 e-tron▪ All-new “Digital Stage”
infotainment system▪ AR HUD integration▪ EU market test

Acura ZDX
▪ Google built-in▪ AcuraWatch 360+ withhands free cruise▪ US market test

Avatr 12▪
▪
▪
▪

Harmony 4.0 OS
Huawei ADS 2.0 (ADAS)
Innovative displays
Chinese market test
10

The Basics

SBD experience through years of testing in-car solutions
Over the last ten years SBD has evaluated over 100 solutions from a Connected Car or ADAS perspective for our public report series (many more for privateclient evaluations). This current report series is an evolution of both test methodologies to provide a holistic view of in-car HMI. Furthermore, custom
evaluations methodologies used across the globe for SBD clients have been included where applicable to enhance to overall approach.

11
11

The Basics

One page methodology overview
One of SBD’s core goals of this report is to be as objective, fair
and as transparent as possible. To achieve this, various
methodologies are used throughout the testing to evaluate
different areas of the solution in various conditions.
These methodologies are a mix of different types of tests:▪ Objective tests: where the value provided is not influenced
by a tester’s viewpoint e.g. response time
▪ Subjective tests: the test score is based on the expert
testers’ viewpoints e.g. task ease of use▪ Task-based: evaluations carried out based on a predefinedtask list e.g. navigate to a pizza restaurant near location X▪ Freeform: random free testing by the tester with no clear
pre-defined task list. This allows the testers flexibility to dig
deeper into various parts of a solution when needed▪ Scoring range: ranges and definitions of how to score atest element e.g. poor depth and accuracy score = theresults provided are not in line with what is reasonably
expected by the user

Type of tests
Test area

Objective

First impressions

Subjective










Dynamic tasks









Random freeNavigation specific tests







Voice recognition







Performance & response

System Usability Scale
(SUS)





UX heuristics

Legibility & readability

Perceived Quality (PQ)


































































Ergonomics

Misuse/failures





ADAS

Dynamic

▪ Dynamic: tests are carried out when the vehicle is moving
in various road conditions and locations e.g.
motorways/highways, cities, villages, country roads etc.

Static

ExecutionFor a detailed discussion and presentation of SBD’s
12methodology
please contact us.

Scoringrange

Final SBD UX score

This document does not provide a detailed description of themethodology and this page serves to provide an overview of
the approach.

Freeform

Static tasks▪ Static: tests are carried out when the vehicle is not moving▪ Misuse/failures: carried out to evaluate the stability of thesolution in unusual conditions e.g. repeatedly pressing thevoice command button

Task based









12

The Basics

Report structure and how to interpret certain data sets
Throughout the testing and evaluation process SBD uses multiple methodologies to align to the situation and test area. Outputs from these evaluations can be broadly grouped into thefollowing three types of report outputs - SBD’s goal with these options is to ensure understanding of the results are as clear and fair as possible.

UX & functionality score

Modified Kano feature analysis

Subjective & objective scores

Final usability scored based on a 100-point scale withsolutions scoring less than 40 defined as “not fit for purpose”
with major user complaints expected and score above 80
defined as “exceptional”.

Features plotted against three lines based on their implementationand satisfaction levels:▪ Delight features: “wow” features likely to provide highsatisfaction even with poor implementation▪ Performance features: as the level of implementation increases
so does the customer satisfaction▪ Hygiene features: poor implementation provides low
satisfaction, but good implementation may not provide positive
satisfaction as it can be considered as expected functionality

Scoring across multiple areas through subjective scoring withpre-defined ranges, definitions, and comparison to past
results.

Functionality score based on eight core feature set areas:delight features, performance features, hygiene features,
13
navigation
feature, ADAS, IoT integration, music,
entertainment and info features.

Objective scoring generally based on a pass/fail criteria ortime-based considerations. SBD attempts to minimize
subjectivity as much as possible with results aimed at being
fair and reasonable with a minimal level of bias.
13

Example slides fromthe report

Executive Summary

Key Takeaway 1

User given the choice to adopt ‘Zero-layer’ theme

Giving users the choiceand flexibility rightfrom the start•

The ‘Zero-layer’ theme intends to minimize menu depth and levels throughout thesystem by allowing users to interact with frequently used features and apps from the top
menu level. The ‘Zero-layer’ suggests varying features an shortcuts based on past user
behavior and habits.

•

When setting up a profile for the first time, the user is presented with the choice toselect the ‘Zero-layer’ theme or select the ‘Classic’ theme. Progression beyond this point
cannot be made without choosing one of the options. Each option is clearly depictedwith call outs showing the properties of each.

•This level of personalization and choice early on in the user journey allows the selectionof a theme that best meets the needs and preferences of the user. Alternatively, if thesystem had simply defaulted to the ‘Classic’ theme and left the user to discover analternate theme on their own, the uptake of ‘Zero-layer’ is likely to be much lower.

15

Users given the choice between ‘Zero-layer’ and ‘Classic’‘Zero-layer’ theme
15Features & Functionality

Delight and Performance features

Significant wow factor throughout, issues with performance features•

An abundance of delight features are integrated into this vehicle. Thesefeatures are likely to create a wow-factor for users and create a lasting
positive impression of the brand.

•

Verbal lane guidance given during active route guidance is often too late or
too early, resulting in the driver making mistakes and entering the wrong
lane.

•

Some of the features include a passenger display, ambient lighting that canreact to music, digital headlights with animations, Superscreen integration,
games, selfie camera and 4D sound.

•

Steering wheel controls are cumbersome to operate accurately andintimidating for new users.

•

POI data can sometimes be out of date while certain POI sub-categories
can be difficult to navigate to.

Abundance of delight features
When turning on or off the car, a series of animations are played through theheadlights on the road ahead. These can include a wave of three-pointed stars
and the text Mercedes-Benz. The 4D sound feature utilizes small actuators in
the front seats to accentuate the bass from media to create a more immersive
experience. Users can take photos and videos and save to external memorycars using the built-in camera on top of the dashboard. Ambient lighting has adiverse range of colors and themes to choose from. It can be set to react to
16 music if desired.

Cumbersome steering wheel controls
The capacitive touch nature of the steering wheel controls mean the risk ofmisoperation is high. The button can be unintentionally triggered by the driverpalm or thumb when turning the steering wheel.
In addition, the set of button dedicated to operating the central display are
difficult to use and often lead to frustration. Trying to use these buttons whiledriving is distracting and adds to the cognitive load of the driver.

16

ADAS Domain

SAE L0 - SBD viewpoint – System usage

Warning is good, but could be improved using existing elements
Category

System usage

SystemRCTA
Once the system has been turned ON via the central display, activation is
automatic when reverse gear is chosen by the driver.

SBD
viewpoint

Cross traffic warnings are provided for both the rear of the vehicle and the frontof the vehicle. If the system senses that a collision with a detected obstacle
passing by is imminent, harsh automatic braking is applied to mitigate contact.
The visual warning shown in the central display is overlaid over the rear camera
view and is directional in nature. The RCTA warning icon is shown on the left or
right of the image depending on the direction of the approaching object. The
audio warning provided is clear and distinctive from other audio warnings usedthroughout ADAS.

Some level of differentiation is provided between vehicles and pedestrians, withdifferent warning messages provided for each.
UX impact

Minor Positive

SBD
viewpoint

The side mirrors are not utilized to show visual warnings for RCTA. Considering
this is a potential location that users may be looking when reversing, utilizing the
BSM icon in the mirror may provide an additional level of warning to grab theusers attention.

UX impact
17

Minor Negative

Visual and audio warning, along with automatic braking

BSM warning in mirror not utilized for RCTA
17

Infotainment Domain

Positive points

Highly customizable instrument cluster

The instrument cluster offer various layouts and themes to meet theusers' preferences.
• Five main layouts can be chosen in the cluster using the steering
wheel buttons. These are Sport, Standard, Navigation, Assistanceand Vehicle Information. Each layout prioritizes different types of
information and uses different color schemes.
SBD
viewpoint• Within the Standard cluster display, various information can be
toggled through in the center of the cluster. These include, trip
computer, media, navigation, eco display, consumption andattention assist.
Having various layouts and information available to be displayed
within each allows the user to tailor the system to meet their needs
and preferences while prioritizing information that is important tothem.

UX impact
18

Major negative

Minor negative

Minor positive

Multiple cluster layouts

Major positive18

Infotainment Domain

Negative points

Steering wheel button implementation presents high risk of misoperation
The shape, layout and use of capacitive touch makes the steering
wheel buttons difficult to use.

• The steering wheel button design uses four panels with multiple
capacitive touch buttons across each panel relating to different
systems. The upper two panels on each side of the steering wheel
have the same icons but in reverse layout. This is visuallyconfusing and take some time to understand the differingfunctionality.

SBD
viewpoint• The capacitive touch nature of the buttons paired with the need topress some of the buttons make for a confusing experience. Some
controls require a swipe such as SET+/SET- and some require atap.
• Misoperation of these buttons is frequent when holding thesteering wheel. The driver thumb can often graze the outer most
buttons and trigger a response in the cluster or central display.
• The shape of the buttons presents issues with heavy reflection of
the curved surface from the glass roof and windshield. Fingerprint
build-up is also noticeable but does not present as much of anissue as on the central display.

Misoperation via thumb or palm

The buttons layout and capacitive touch technology make for control
panel that is initially intimidating for a new user. Frequent
misoperation is not only frustrating but also presents a safety risk
considering the nature of some of the controls positioned on the out
most edge.

UX impact
19

Major negative

Minor negative

Minor positive

Major positive
19

Navigation Domain

Positive points

Good integration of sharing route and location

The user can opt to share route and destination information to mobile
devices via SMS.

• User can share their journey by pressing the share option in thecentral display. Options include sharing a current location,
destination, route or estimated time of arrival.
• When sharing live tracking via SMS, the recipient will receive a link
to a website from which they can track live journey progress.
SBD
viewpoint• When sharing a journey, a number of options are provided on how
long the information is shared before. For example, the user canchoose from share once, share until arrival at destination, sharefor 15 minutes or share for one hour.
Giving the user the option to share location and journey information
allows them to keep contacts outside of the vehicle updated onprogress and arrival times. Not only is this a convivence and safetyenhancement but it also prevents users from using their phone toshare the same information, potentially while driving.

UX impact
20

Major negative

Minor negative

Minor positive

Journey sharing options

Major positive20

Convenience

HUD summary

Head-up Display (HUD) summary (1/2)
The HUD in the 2024 Mercedes-Benz E-Class uses a projection method to display information on thewindshield. The user can choose to customize the layout and content based on their own preferences.
Layout options include Minimal, Sport and Standard View. Settings including height, brightness andlayout can be adjusted.

Content Options
DomainConsistency between the HUD and the cluster is mostly good with only a few slight differences. The
HUD does not display a constant LKA status icon like the cluster. Color conventions are followed
correctly and can be understood easily.

Turn by turn
Lane guidance

While the user has the option to display information and status from various systems in the HUD, thelayout can become overwhelming and cluttered, especially when route guidance is active. Informationsuch as the road name, ETA and distance to next direction all feel squashed in the top right handcorner of the HUD.
Overall, the HUD is consistent in its display and provides a good level of information for ADAS andnavigation without the need for the driver to take their eyes away from the road. However, at times
this rich level of information can become distracting and overwhelming, especially at night.

Content

Map view
Navigation

Street nameDistance to current command
ETA
Distance remaining
ADAS icons
Speed limit

ADAS

Current speed
Set speed
FAPA information

21

Active route guidance in HUDSport display
21

Request price forthe full reportContact Us

Contact SBD Automotive

Do you have any questions?
If you have any questions or feedback about thisresearch report or SBD Automotive’s consulting services,
you can email us at info@sbdautomotive.com or discusswith your local account manager below.

info@sbdautomotive.com

Book a meeting

USA

Garren Carr
North America
garrencarr@sbdautomotive.com+1 734 619 7969

23

UK

Germany

Luigi Bisbiglia
UK, South & West Europe
luigibisbiglia@sbdautomotive.com+44 1908 305102

Andrea Sroczynski
Germany, North & East Europe
andreasroczynski@sbdautomotive.com+49 211 9753153-1

India

China

Japan

SBD China Sales TeamChina
salesChina@sbdautomotive.com+86 18516653761

SBD Japan Sales TeamJapan, South Korea & Australia
postbox@sbdautomotive.com+81 52 253 6201
23