#635

In-Car HMI UX Evaluation& Benchmarking

UX
TABLE OF CONTENTS

EVALUATION

Executive Summary
Introduction to this ReportIn-car HMI UX
Evaluations

Features and Functionality
Execution
Perceived Quality

BMW X1ADAS Domain

Infotainment Domain

In this edition of the In-Car HMI UX Evaluation and Benchmarking series, the UX team is testing the
BMW X1. Evaluations are carried out by SBD usability experts with a deep understanding of CASE
domains such as the Connected Car and ADAS & autonomy domains.

Navigation Domain
Voice Recognition Domain
Connected Features Domain

The 2024 BMW X1 is the first BMW model launched with the new iDrive 9 operating system (OS). The
new iDrive 9 system is based on Android OS, which is the same for BMW’s previous models with theiDrive 8.5 system. However, SBD found some surprising issues during the evaluation that weren’t major
issues with previous BMW’s models in the past.

Convenience Domain

RELATED SBD REPORTS
619 – UX Benchmarking Series
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s UX Team evaluates the infotainment
user experience of over 40 vehicles.
806 – ADAS HMI Evaluations
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s Autonomous Car Team evaluates the
ADAS performance and usability of over 20
vehicles.

COVERAGE

FREQUENCY

PUBLICATION FORMAT

PAGES

9
NA

CHINA

EUROPEANNUALLYQUARTERLYCARS PER YEARPD F

POWERPOINT

EXCEL

ONLINE

170+

#635/ In-Car HMI UX Evaluation & Benchmarking

Scoring> Features and functionality: > Reliability/stability:
evaluating whether theevaluating the repeatedsolutions provide featuresusability and whether thethat customers expect, needusers can have a similarand solve problems (or(positive) experience eachprovide a wow factor).
time.
> Usability:
evaluating whether the> Perceived quality:
features available are easy
evaluating the potential
to learn and use. This
perception in quality of theconsiders areas such as
HMI components and howergonomics, legibility,
this contributes to the overallusability characteristics andcustomer experience.
how the system implements
the various features.

This research is useful for

PRODUCT PLANNERS

M A R K E T I NG

C- S UITEDo I have access?

USER E XPE R IENCEE NGI NE E R S

Portal access

Request a quote for
In-Car HMI UX Evaluation & Benchmarking Series
BMW X1

In-Car HMI UX Evaluation & Benchmarking
BMW X1

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – BMW X1
Introduction≫

5▪ Report introduction

The basics≫

22

Features and functionality≫▪ Aim of report▪ Overview▪ Scope of report▪ Delight and performance features▪ 2024 vehicles▪ Hygiene and navigation▪ UX principles▪ SBD experience▪ ADAS feature set▪ What’s new in vehicle UX?

▪ Methodology▪ Comparison of past vehicles▪ Report structure▪ IoT integration, music, entertainment
and information

Bird’s-eye view≫

7▪ 645 Hands-free driving▪ OTA, FaaS + SW-defined features guide

Analysis≫▪ The cockpit guide▪ SUS scores by domain▪ Health and wellbeing in automotive▪ Kano analysis▪ Digital cockpit & infotainment forecast▪ Positive and negative implementations▪ CX recommendations

Executive summary≫▪ Key takeaway 1▪ Key takeaway 2▪ Key takeaway 3▪ Ranking▪ Scoring16▪ Potential improvements

29

Execution≫▪ Overview▪ System scoring▪ UX laws▪ System performance▪ Stability▪ Ergonomics▪ Legibility and readability

36

42

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – BMW X1
Perceived quality≫

72

Navigation domain≫120

Convenience domain≫▪ Introduction▪ Summary▪ UX heuristics▪ Definitions▪ UX heuristics▪ HVAC summary▪ Evaluation results▪ Navigation specific scoring▪ HUD summary▪ Highlights▪ Key positive and negatives

ADAS domain≫

80

Voice recognition domain≫▪ Summary▪ Summary▪ UX Heuristics▪ Functionality▪ SAE L0

▪ Recognition▪ SAE L1&2

▪ SUS score▪ SBD UX principles

Infotainment domain≫98▪ Command structure▪ Summary▪ Localization▪ UX heuristics▪ Level of integration

Future outlook≫
142

154

160

▪ Introduction▪ UX principle groups

Next steps≫

165

Contact us≫174▪ Key positive and negative

Customer Feedback
Provide your feedback to
SBD regarding this report

Total number of pages - 174

Introduction

Introduction

Report Introduction
Welcome to the 2024 HMI benchmarking report series. This report has been created toprovide a fair, unbiased and objective view of the latest in-vehicle HMI solutions in the US,
European, and Chinese markets. Evaluations are carried out by SBD usability experts witha deep understanding of CASE domains such as the Connected Car and ADAS & autonomy
domains.
SBD supports clients throughout the development of new HMI and products from a
relatively simple companion app to a more complex multi-domain infotainment solution.
The methodologies used in these reports take into account many years of experience with
consumer testing and custom client projects to provide a fair and, as much as possible,
objective methodology.
All viewpoints and analysis within the report are aimed defining areas of concern througha data driven approach. This report aims to benchmark and score solutions whilst also
being able to provide actionable recommendations to design and development teams.

Please note that due to the ever-evolving automotive technology market, SBD updates it’s
methodology each year, but does not update scores from the previous years. Therefore,
please assume a slight drop in scores for both user experience (UX) and functionality fromthe previous year.

Intuitive

7

Consistency

Supportive

Brevity

Flexibility

Depth

Presentation

SectionContent

Birds Eye View

An overview of the key findings from SBD’s various CX related andadjacent reports.

Executive
Summary

Presents key highlights and conclusions from the report.

The Basics

What do you need to know about SBD’s CX evaluation
methodology?

AnalysisAnalysis of report findings by SBD experts.

Features andfunctionality
Execution
Perceived quality

Overview of key features and functionality by domain.
Assess success of implementation and overall execution of various
system elements.
Scoring and analysis of interior perceived quality levels.

ADAS domain

Highlight and analysis of key positive and negative points within
the ADAS domain.

Infotainment
domainHighlight and analysis of key positive and negative points within
the infotainment domain.

Navigation domain

Highlight and analysis of key positive and negative points within
the navigation domain.

Voice recognition
domainSummary and scoring of various aspects of the voice recognition
system.

Convenience
domainSummary of various convenience focussed features.

Future Outlook

Seven UX principles are considered against drivers and barriers
into the future of this reports test vehicle.

Next StepsCan SBD help you with any unanswered questions?
7

The Basics

Aim of this report▪ Features and functionality: evaluating whether the solutions provide
features that customers expect & need, and solve problems (or provide a
wow factor)▪ Usability: evaluating whether the features available are easy to learn anduse. This considers areas such as ergonomics, legibility, usability
characteristics and how the system implements the various features

Brand perception

Consumer testing

One of the core goals of these studies is to provide a true indication of what
the final customer experience of each solution could be. To do thisevaluations are focused on providing scoring and analysis in the following
areas:▪ Reliability/stability: evaluating the repeated usability and whether theusers can have a similar (positive) experience each time
▪ Perceived quality: evaluating the potential perception in quality of the
HMI components and how this contributes to the overall customer
experience“Do I have a positive emotional attachment to thebrand?”

Aesthetics and engagement“Do I have a positive emotional attachment to thesolution?”

EMOTIONAL
APPEAL

Repeated usability

Expert testing(the focus of this report)

RELIABILITY“Can I have a similar experience each time?”

Ergonomics, usability characteristics &execution“Are the features available easy to learn
and use?”USABILITY

Domain functionality &prioritisation“Does the system provide features I
want and need to use?” i.e. “Does it
solve problems for me?”

8

BRAND
APPEALFUNCTIONALITY

SBD’s view on the hierarchy of needs for CX benchmarking
8The Basics

Scope of report: focus on in-car HMI evaluations
The scope of evaluations in this report are constrained to the in-car HMI
experience, in both static and dynamic conditions. One notable element isdriver distraction which SBD covers at only a high level in this study as
carrying out a full driver distraction evaluation requires biometrics test
equipment to ensure the collected data is unbiased and objective.

Prepurchase

Purchase

Activation

In-car HMI

Driverdistraction

IoT

A full evaluation of the end-to-end customer experience is not within scope of
this report, but it is something which SBD has many years experience in fromboth a consumer and expert perspective. Other areas such as the companion
app, online portal and in-home smart devices are not in scope as they aredefined as “out of car” experiences.
Within the vehicle, any HMI element the user interacts with is evaluatedincluding steering wheel controls, touch screen displays, voice control, HUDs
and digital keys. The features and services on offer have been broadlygrouped into the following domains (or test areas):▪ ADAS domain▪ Infotainment domain▪ Navigation domain▪ Voice recognition domain▪ Convenience domain
99The Basics

2024 vehicle list
SBD has chosen nine cars to evaluate in 2024, based on two selection categories. New/interesting UX focuses on systems with to never-seen-before
features or functionality, or the implementation of a solution that has previously been a challenge or pain-point for end-users. New mass-market UX
includes vehicles in segments that are sold in high numbers and are entering a new generation of UI for that vehicle. While we make best efforts toadhere to the chosen cars and schedule, the last two years have seen release dates slipping significantly, so it may be necessary to makesubstitutions.
Group A
Cars tested

BMW X1

Priority target vehicles for 2024

Mercedes-Benz
E-Class

Hyundai Kona/Kona
Electric

Volvo EX90

Ford F-150
LightningGroup B
Other vehicles being considered for 2024

Li Auto MEGA

Audi Q6 e-tron

Acura ZDX

Avatr 1210
10

The Basics

SBD experience through years of testing in-car solutions
Over the last ten years SBD has evaluated over 100 solutions from a Connected Car or ADAS perspective for our public report series (many more for privateclient evaluations). This current report series is an evolution of both test methodologies to provide a holistic view of in-car HMI. Furthermore, custom
evaluations methodologies used across the globe for SBD clients have been included where applicable to enhance to overall approach.

11
11

The Basics

One page methodology overview
One of SBD’s core goals of this report is to be as objective, fair
and as transparent as possible. To achieve this, various
methodologies are used throughout the testing to evaluate
different areas of the solution in various conditions.
These methodologies are a mix of different types of tests:▪ Objective tests: where the value provided is not influenced
by a tester’s viewpoint e.g. response time
▪ Subjective tests: the test score is based on the expert
testers’ viewpoints e.g. task ease of use▪ Task-based: evaluations carried out based on a predefinedtask list e.g. navigate to a pizza restaurant near location X▪ Freeform: random free testing by the tester with no clear
pre-defined task list. This allows the testers flexibility to dig
deeper into various parts of a solution when needed▪ Scoring range: ranges and definitions of how to score atest element e.g. poor depth and accuracy score = theresults provided are not in line with what is reasonably
expected by the user

Type of tests
Test area

Objective

First impressions

Subjective
P

P

P

P

P

Dynamic tasks

P

P

P

P

Random freeP

P

Navigation specific tests

P

P

P

Voice recognition

P

P

P

Performance & response

P

System Usability Scale
(SUS)

P

P

UX heuristics

P

P

Legibility & readability

P

Perceived Quality (PQ)

P

P

P
P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

Ergonomics

P

Misuse/failures

P

P

ADAS

Dynamic

P

P▪ Dynamic: tests are carried out when the vehicle is moving
in various road conditions and locations e.g.
motorways/highways, cities, villages, country roads etc.

StaticP

P
P

ExecutionPPP

For a detailed discussion and presentation of SBD’s
12methodology
please contact us.

Scoringrange

PFinal SBD UX score

This document does not provide a detailed description of themethodology and this page serves to provide an overview of
the approach.

Freeform

Static tasks▪ Static: tests are carried out when the vehicle is not moving▪ Misuse/failures: carried out to evaluate the stability of thesolution in unusual conditions e.g. repeatedly pressing thevoice command button

Task based

P

P

P

P

12

The Basics

Report structure and how to interpret certain data sets
Throughout the testing and evaluation process SBD uses multiple methodologies to align to the situation and test area. Outputs from these evaluations can be broadly grouped into thefollowing three types of report outputs - SBD’s goal with these options is to ensure understanding of the results are as clear and fair as possible.

UX & functionality score

Modified Kano feature analysis

Subjective & objective scores

Final usability scored based on a 100-point scale withsolutions scoring less than 40 defined as “not fit for purpose”
with major user complaints expected and score above 80
defined as “exceptional”.

Features plotted against three lines based on their implementationand satisfaction levels:▪ Delight features: “wow” features likely to provide highsatisfaction even with poor implementation▪ Performance features: as the level of implementation increases
so does the customer satisfaction▪ Hygiene features: poor implementation provides low
satisfaction, but good implementation may not provide positive
satisfaction as it can be considered as expected functionality

Scoring across multiple areas through subjective scoring withpre-defined ranges, definitions, and comparison to past
results.

Functionality score based on eight core feature set areas:delight features, performance features, hygiene features,
13
navigation
feature, ADAS, IoT integration, music,
entertainment and info features.

Objective scoring generally based on a pass/fail criteria ortime-based considerations. SBD attempts to minimize
subjectivity as much as possible with results aimed at being
fair and reasonable with a minimal level of bias.
13

Example slides fromthe reportAnalysis

Significant ability to customize and configure

In-vehicle app store

Significant personalization

Integration of live data

The inclusion of the BMW ConnectedDrive store, accessible
via the head unit, allows users to expand the featureoffering within their vehicle by downloading games andother entertainment & information features direct to thevehicle. The app store is easy to navigate and will befamiliar to users of other consumer electronic devices in theway it operates. An app store and the expansion of featuresets
15it provides is likely to contribute towards longer
consumer engagement and satisfaction with the vehicle.

A significant level of personalization is possible allowing theuser to customize the system to meet their own
preferences. ‘My Modes’ gives access to curated themeschanging the interior lighting, sounds and other elements.
User habits are learned over time allowing the vehicle to
make proactive suggestions unique to the user profile.
ADAS settings can be tied to user profiles allowing users toadjust and save their own personal settings.

Live data is integrated successfully throughout the navigation
system. During route guidance, the user is alerted to various
hazards including construction, obstructions and developing
traffic situations ahead utilizing cloud sourced data. Other
data integration includes on-street parking locations and POI
review data provided by Yelp.

15

Analysis

System stability issues and lacking expected intuitiveness

Poor level of stability

Complicated system architecture

Complex steering wheel controls

Throughout one week of testing, the observed level ofsystem stability is generally poor. A total of 13 bugs are
identified ranging from minor, less frequent occurrences tocritical issues. A system struggling with stability has anegative impact on the overall user experience andperception of quality. One issue of stability required a soft
reset, something that many users will not expect to have toperform.
16

A disjointed information architecture is partly caused bymain menu buttons located in different areas. Just off theleft-hand side of the display are permanent buttons forMedia, Telephone, Navigation and a back button aredisplayed using words. However, along the bottom of thedisplay built into the HVAC bar are buttons for HVAC,
Homepage and App screen. These buttons use icons insteadof words. This disjointed appearance is visually jarring andcan cause difficulty in navigating the system.

The integration of steering wheel controls to control the HUD
and/or cluster has the potential to confuse and frustrateusers. With the HUD off, the right-hand side steering wheel
controls interact with the cluster. With the HUD turned on,they interact with the HUD. Dual use for buttons always
holds the potential to confuse users and this is no exception.
It takes longer than expected to understand this
implementation.
16

Executionⅼ

UX Laws

UX Laws (1/4)
Law name

Law definition

Fitt’s law

The time to acquire a target is afunction of the distance to andsize of the target

Are there any small buttons
which hinder their usability?

Concerns

Hick’s law

The time it takes to make adecision increases with thenumber and complexity ofchoices

Are there any instances where
the HMI in the vehicle offers toomuch choice, and makes it
difficult to make a decision?

Concerns

Jakob’s law

Users will transfer expectations
they have built around one
familiar product to another thatappears similar

Are there any instances where
the HMI in the vehicle uses anoperating principle that is unlike
similar products available today?
If so, does this lack of familiarity
make the HMI difficult to use?

Concerns

In the navigation screen, the location of the search bar is at the bottom of
the screen, which is unfamiliar to navigation apps on a phone and othernavigation systems in other vehicles, which is cause for mistouches andfrustration by the user. There is also no direct tune input on the radio screenin the central display.

The average person can only
keep 7 (plus or minus 2) items intheir working memory

Throughout user journeys, areusers required to remember alarge number of things tocomplete that journey? An easy
example of this is in menu
architectures which are deep andrequire the user to memorize thelocation of a particular setting orfeature.

Acceptable

Some ADAS settings, like distance control / gap distance, require a larger
number of steps to remember, but overall, most tasks can be completed witha reasonable number of steps.

Concerns

There are a few instances where there are unnecessary steps required tocomplete a task. There is no button on the steering wheel controls that offers
just one touch to toggle between presets / favorites for radio / media. Somesteering wheel buttons control a menu on the HUD or instrument cluster
requiring the driver to look at the display and take multiple steps to complete
the task. Some ADAS settings, like distance control / gap distance, require
unnecessary steps. Some HVAC and radio tasks also require someunnecessary steps.

Miller’s law

Occam’s Razor

17

Complexity: The path along theuser journey should be simplified
to the point where no additional
choices or steps in that journey
can be removed

What to check

Has the complexity of a user
journey had all unnecessary steps
removed?
Or are there a number of
unnecessary inputs required
before the task is complete?

Score

SBD comments & reasoning
On the central display, the temperature (+/-) controls, the HVAC fan touch
button, the apps button and the media/radio sources button are all small andthe touch target isn't always clearly defined, making it difficult to use,
especially while driving.
The apps screen can be overwhelming at times, making it difficult to findwhat the user needs. The layout of the radio screen with the long list of radio
stations makes it difficult to make a decision. Navigating through ADAS
settings is complex as well, making it hard to find the setting the user islooking for.

17

ExecutionⅼStability issues

System does not give the driver a chance to speak

5. Voice Recognition

On multiple occasions, the system fails to give the driverchance to speak.
When pressing the voice button on the steering wheel orsaying ‘Hey BMW’, the voice recognition system instantly
opens with “Please try again” or “Try it once more”. The
user can speak over the top of this message but isn’t
always recognized by the system.

Frequency

Low

MediumHigh

Severity

Minor

Major

Critical18
18Perceived quality

Visual highlights

Perceived Quality: Visual
Level 1VisualHarmony/alignment: The cluster and the central display areintegrated into one surface, elevating the visual of the cockpit. Other
components such as surfaces, trim, and controls are consistent withentry level luxury vehicles.

Shortcut buttons
integrated into
surface lack haptic
feedback

Geometric: Some steering wheel buttons are too small and haveunclear icons/text. On the central display, some buttons lacked easily
identifiable touch targets. Some visual shapes for the screen on thecentral display, such as buttons, were inconsistent (i.e. size, shape,
location).

SBD
viewpoint

Spatial harmony: The app menu within the central display appears
cluttered and can be overwhelming. However, the hard buttons
throughout the cabin are well spaced and do not appear to be
cramped.
Branding: “M” badge for the vehicle model tested is branded on thebottom of the steering wheel. Overall HMI is sharp and dynamic,aligning with the ethos of the BMW brand.
Output HMI: Key output HMI factors include lightbars on thesteering wheel to show relevant ADAS status/warnings and voiceassistant animations that appear on the HUD, cluster, and central
display.

Ambient lighting
present within thecabin

Cluttered app
menu on thecentral display
with long scroll list

Interior lighting: Ambient lighting with many color choices is built
into the vehicle trim with a thin illuminating bar that wraps aroundthe cockpit. Other buttons are illuminated well in low light instances.

Level 2 scoring
Harmony/alignment

Geometric

Spatialharmony

Branding

Output HMIInterior
lighting

Good

Fair

Fair

Good

Good

Good
19
19

ADAS Domain

SAE L1&2 - Key positive and negative points

System usage:• Headway can only be adjusted via the settings menu in the central display.

Major

Key positive and negative points - ACC
System turn ON:• System turn ON controls located conveniently on steering wheel.
System usage:• New upcoming (higher/lower) speed limit is suggested by the system.

Negative

Positive

System turn ON:• Icon and text on ‘mode’ button and ON button does not match that found in thecluster or relate to the system well.
• One button is shared between multiple systems (ACC, PD, Speed Limiter).

System turn ON:• Activation is automatic when turned ON if conditions are met.
• Reaction when turned ON is acceptable (text label provided when switching from PD
or speed limiter to ACC)

System usage:• Detail lost from ACC display when alternative content is chosen in cluster.
• Headway is only shown with assistance display chosen.

System usage:• ADAS display in cluster differentiates between cars and trucks.
• Good graphics shown in cluster and HUD when assistance display is chosen.
Vehicle ahead, set speed and headway are shown.
• When activated, ACC set speed and icon turn green.

20

Minor

System turn OFF:• Only minimal reaction when the system is turned OFF.
• Press and hold action to turn OFF the system is not intuitive.

System cancellation/turn OFF:• When cancelled using the brake pedal or button, correct color logic is used.

20

Infotainment Domain

Positive points

Good integration of Android Auto and Apple CarPlay

Wireless Android Auto and Apple CarPlay are available in the BMW
X1.

Apple Carplay button in native system toggles to Apple CarPlay• With a mobile device paired, a toggle setting in the phone screen
can be used to enable/disable Android Auto and Apple CarPlay.
Buttons for Bluetooth media and Android Auto/Apple CarPlay inthe phone screen can also be used to swap between the broughtin and native infotainment systems.
SBD
viewpoint• When Apple CarPlay or Android Auto are active, telephonefunctions and media playback are only available through thosesystems, not the native system.

Home button in Apple CarPlay toggles to native system• When enabled, the home menu button becomes a toggle switch
between the native infotainment system and Android Auto/Apple
CarPlay.
• The climate controls are still present at the bottom of the central
display when Android Auto and Apple CarPlay are in use.
Owners can easily swap between the systems with one button press,
leading to a seamless integration of these services which enhances
the overall user experience.

UX impact
21

Major negative

Minor negative

Minor positive

Major positive

Toggle setting and buttons to enable/disable Apple CarPlay (or Android Auto if
set up)21

Request price forthe full reportContact Us

Contact SBD Automotive

Do you have any questions?
If you have any questions or feedback about thisresearch report or SBD Automotive’s consulting services,
you can email us at info@sbdautomotive.com or discusswith your local account manager below.

info@sbdautomotive.com

Book a meeting

USA

Garren Carr
North America
garrencarr@sbdautomotive.com+1 734 619 7969

23

UK

Germany

Luigi Bisbiglia
UK, South & West Europe
luigibisbiglia@sbdautomotive.com+44 1908 305102

Andrea Sroczynski
Germany, North & East Europe
andreasroczynski@sbdautomotive.com+49 211 9753153-1

India

China

Japan

SBD China Sales TeamChina
salesChina@sbdautomotive.com+86 18516653761

SBD Japan Sales TeamJapan, South Korea & Australia
postbox@sbdautomotive.com+81 52 253 6201
23