#635

In-Car HMI UX Evaluation& Benchmarking

UX
TABLE OF CONTENTS

EVALUATION

Executive Summary
Introduction to this ReportIn-car HMI UX
Evaluations

Features and Functionality
Execution
Perceived Quality

Xiaomi SU7

ADAS Domain

Infotainment Domain
Navigation Domain

In this edition of the In-Car HMI UX Evaluation and Benchmarking series, the UX team is testing the
Xiaomi SU7.

Voice Recognition Domain
Connected Features Domain
Convenience Domain

RELATED SBD REPORTS
619 – UX Benchmarking Series
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s UX Team evaluates the infotainment
user experience of over 40 vehicles.
806 – ADAS HMI Evaluations
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s Autonomous Car Team evaluates the
ADAS performance and usability of over 20
vehicles.

COVERAGE

FREQUENCY

PUBLICATION FORMAT

PAGES

6
NA

CHINA

EUROPEANNUALLYQUARTERLYCARS PER YEARPD F

POWERPOINT

EXCEL

ONLINE

150+

#635/ In-Car HMI UX Evaluation & Benchmarking

Scoring> Features and functionality: > Reliability/stability:
evaluating whether theevaluating the repeatedsolutions provide featuresusability and whether thethat customers expect, needusers can have a similarand solve problems (or(positive) experience eachprovide a wow factor).
time.
> Usability:
evaluating whether the> Perceived quality:
features available are easy
evaluating the potential
to learn and use. This
perception in quality of theconsiders areas such as
HMI components and howergonomics, legibility,
this contributes to the overallusability characteristics andcustomer experience.
how the system implements
the various features.

This research is useful for

PRODUCT PLANNERS

Do I have access?

USER E XPE R IENCE

Portal access

M A R K E T I NG

C- S UITEE NGI NE E R S

Portal access

Request a quote for
In-Car HMI UX Evaluation & Benchmarking Series
Xiaomi SU7

May 2024

635-24(24c)

In-Car HMI UX Evaluation & Benchmarking
Xiaomi SU7

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – Xiaomi SU7
Introduction≫

5▪ Report introduction

The basics≫

23

Features and functionality≫▪ Aim of report▪ Overview▪ Scope of report▪ Delight and performance features▪ 2024 vehicles▪ Hygiene and navigation▪ UX principles▪ SBD experience▪ ADAS feature set▪ What’s new in vehicle UX?

▪ Methodology▪ Comparison of past vehicles▪ Report structure▪ IoT integration, music, entertainment
and information

Bird’s-eye view≫

7▪ 645 Hands-free driving▪ OTA, FaaS + SW-defined features guide

Analysis≫▪ The cockpit guide▪ SUS scores by domain▪ Health and wellbeing in automotive▪ Kano analysis▪ Digital cockpit & infotainment forecast▪ Positive and negative implementations▪ CX recommendations

Executive summary≫▪ Key takeaway 1▪ Key takeaway 2▪ Key takeaway 3▪ Ranking▪ Scoring16▪ Potential improvements

30

Execution≫▪ Overview▪ System scoring▪ UX laws▪ System performance▪ Stability▪ Ergonomics▪ Legibility and readability

37

43

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – Xiaomi SU7
Perceived quality≫

61

Navigation domain≫

107

Convenience domain≫▪ Introduction▪ Summary▪ UX heuristics▪ Definitions▪ UX heuristics▪ HVAC summary▪ Evaluation results▪ Navigation specific scoring▪ HUD summary▪ Highlights▪ Key positive and negatives

ADAS domain≫

69

Voice recognition domain≫▪ Summary▪ Summary▪ UX Heuristics▪ Functionality▪ SAE L0

▪ Recognition▪ SAE L1&2

▪ SUS score▪ SBD UX principles

Infotainment domain≫91▪ Command structure▪ Summary▪ Localization▪ UX heuristics▪ Level of integration

134▪ Sentinel mode
▪ Phone reminder124▪ Rear seat display▪ Dynamic seat

Future outlook≫

145▪ Introduction▪ UX principle groups

Next steps≫

150

Contact us≫

159▪ Key positive and negative

Total number of pages - 159

Introduction

Introduction

Report Introduction
Welcome to the 2024 HMI benchmarking report series. This report has been created toprovide a fair, unbiased and objective view of the latest in-vehicle HMI solutions in the US,
European, and Chinese markets. Evaluations are carried out by SBD usability experts witha deep understanding of CASE domains such as the Connected Car and ADAS & autonomy
domains.
SBD supports clients throughout the development of new HMI and products from a
relatively simple companion app to a more complex multi-domain infotainment solution.
The methodologies used in these reports take into account many years of experience with
consumer testing and custom client projects to provide a fair and, as much as possible,
objective methodology.
All viewpoints and analysis within the report are aimed defining areas of concern througha data driven approach. This report aims to benchmark and score solutions whilst also
being able to provide actionable recommendations to design and development teams.

Please note that due to the ever-evolving automotive technology market, SBD updates it’s
methodology each year, but does not update scores from the previous years. Therefore,
please assume a slight drop in scores for both user experience (UX) and functionality fromthe previous year.

Intuitive

7

Consistency

Supportive

Brevity

Flexibility

Depth

Presentation

SectionContent

Birds Eye View

An overview of the key findings from SBD’s various CX related andadjacent reports.

Executive
Summary

Presents key highlights and conclusions from the report.

The Basics

What do you need to know about SBD’s CX evaluation
methodology?

AnalysisAnalysis of report findings by SBD experts.

Features andfunctionality
Execution
Perceived quality

Overview of key features and functionality by domain.
Assess success of implementation and overall execution of various
system elements.
Scoring and analysis of interior perceived quality levels.

ADAS domain

Highlight and analysis of key positive and negative points within
the ADAS domain.

Infotainment
domainHighlight and analysis of key positive and negative points within
the infotainment domain.

Navigation domain

Highlight and analysis of key positive and negative points within
the navigation domain.

Voice recognition
domainSummary and scoring of various aspects of the voice recognition
system.

Convenience
domainSummary of various convenience focussed features.

Future Outlook

Seven UX principles are considered against drivers and barriers
into the future of this reports test vehicle.

Next StepsCan SBD help you with any unanswered questions?
7

The Basics

Aim of this report▪ Features and functionality: evaluating whether the solutions provide
features that customers expect & need, and solve problems (or provide a
wow factor)▪ Usability: evaluating whether the features available are easy to learn anduse. This considers areas such as ergonomics, legibility, usability
characteristics and how the system implements the various features

Brand perception

Consumer testing

One of the core goals of these studies is to provide a true indication of what
the final customer experience of each solution could be. To do thisevaluations are focused on providing scoring and analysis in the following
areas:▪ Reliability/stability: evaluating the repeated usability and whether theusers can have a similar (positive) experience each time
▪ Perceived quality: evaluating the potential perception in quality of the
HMI components and how this contributes to the overall customer
experience“Do I have a positive emotional attachment to thebrand?”

Aesthetics and engagement“Do I have a positive emotional attachment to thesolution?”

EMOTIONAL
APPEAL

Repeated usability

Expert testing(the focus of this report)

RELIABILITY“Can I have a similar experience each time?”

Ergonomics, usability characteristics &execution“Are the features available easy to learn
and use?”USABILITY

Domain functionality &prioritisation“Does the system provide features I
want and need to use?” i.e. “Does it
solve problems for me?”

8

BRAND
APPEALFUNCTIONALITY

SBD’s view on the hierarchy of needs for CX benchmarking
8The Basics

Scope of report: focus on in-car HMI evaluations
The scope of evaluations in this report are constrained to the in-car HMI
experience, in both static and dynamic conditions. One notable element isdriver distraction which SBD covers at only a high level in this study as
carrying out a full driver distraction evaluation requires biometrics test
equipment to ensure the collected data is unbiased and objective.

Prepurchase

Purchase

Activation

In-car HMI

Driverdistraction

IoT

A full evaluation of the end-to-end customer experience is not within scope of
this report, but it is something which SBD has many years experience in fromboth a consumer and expert perspective. Other areas such as the companion
app, online portal and in-home smart devices are not in scope as they aredefined as “out of car” experiences.
Within the vehicle, any HMI element the user interacts with is evaluatedincluding steering wheel controls, touch screen displays, voice control, HUDs
and digital keys. The features and services on offer have been broadlygrouped into the following domains (or test areas):▪ ADAS domain▪ Infotainment domain▪ Navigation domain▪ Voice recognition domain▪ Convenience domain
99The Basics

2024 vehicle list
SBD has chosen nine cars to evaluate in 2024, based on two selection categories. New/interesting UX focuses on systems with to never-seen-before
features or functionality, or the implementation of a solution that has previously been a challenge or pain-point for end-users. New mass-market UX
includes vehicles in segments that are sold in high numbers and are entering a new generation of UI for that vehicle. While we make best efforts toadhere to the chosen cars and schedule, the last two years have seen release dates slipping significantly, so it may be necessary to makesubstitutions.
Group A
Cars tested

Priority target vehicles for 2024

BMW X1

Mercedes-Benz E-Class▪ First BMW to be released
with iDrive9▪ Android based system▪ US market test

▪ All-new MBUX Superscreen▪ Unique features, apps andgames▪ German market test

Lincoln Nautilus▪
▪
▪
▪

BlueCruise hands free
Digital Experience
All-new infotainment system
US market test

Hyundai Kona/Kona Electric

Volvo EX90

▪ 12.3-inch cluster andcentral display▪ New Bluelink+ services▪ US market test

▪ Volvo Pilot Assist▪ 14.5-inch central display▪ EU or US market test

Group B
Cars tested

Xiaomi SU7▪ Xiaomi’s first vehicle offering▪ New vehicle from CE company▪ Xiaomi Pilot MAX▪ Chinese market test
10

Other vehicles being considered for 2024

Audi Q6 e-tron▪ All-new “Digital Stage”
infotainment system▪ AR HUD integration▪ EU market test

Acura ZDX
▪ Google built-in▪ AcuraWatch 360+ withhands free cruise▪ US market test

Avatr 12▪
▪
▪
▪

Harmony 4.0 OS
Huawei ADS 2.0 (ADAS)
Innovative displays
Chinese market test
10

The Basics

SBD experience through years of testing in-car solutions
Over the last ten years SBD has evaluated over 100 solutions from a Connected Car or ADAS perspective for our public report series (many more for privateclient evaluations). This current report series is an evolution of both test methodologies to provide a holistic view of in-car HMI. Furthermore, custom
evaluations methodologies used across the globe for SBD clients have been included where applicable to enhance to overall approach.

11
11

The Basics

One page methodology overview
One of SBD’s core goals of this report is to be as objective, fair
and as transparent as possible. To achieve this, various
methodologies are used throughout the testing to evaluate
different areas of the solution in various conditions.
These methodologies are a mix of different types of tests:▪ Objective tests: where the value provided is not influenced
by a tester’s viewpoint e.g. response time
▪ Subjective tests: the test score is based on the expert
testers’ viewpoints e.g. task ease of use▪ Task-based: evaluations carried out based on a predefinedtask list e.g. navigate to a pizza restaurant near location X▪ Freeform: random free testing by the tester with no clear
pre-defined task list. This allows the testers flexibility to dig
deeper into various parts of a solution when needed▪ Scoring range: ranges and definitions of how to score atest element e.g. poor depth and accuracy score = theresults provided are not in line with what is reasonably
expected by the user

Type of tests
Test area

Objective

First impressions

Subjective










Dynamic tasks









Random freeNavigation specific tests







Voice recognition







Performance & response

System Usability Scale
(SUS)





UX heuristics

Legibility & readability

Perceived Quality (PQ)


































































Ergonomics

Misuse/failures





ADAS

Dynamic

▪ Dynamic: tests are carried out when the vehicle is moving
in various road conditions and locations e.g.
motorways/highways, cities, villages, country roads etc.

Static

ExecutionFor a detailed discussion and presentation of SBD’s
12methodology
please contact us.

Scoringrange

Final SBD UX score

This document does not provide a detailed description of themethodology and this page serves to provide an overview of
the approach.

Freeform

Static tasks▪ Static: tests are carried out when the vehicle is not moving▪ Misuse/failures: carried out to evaluate the stability of thesolution in unusual conditions e.g. repeatedly pressing thevoice command button

Task based









12

The Basics

Report structure and how to interpret certain data sets
Throughout the testing and evaluation process SBD uses multiple methodologies to align to the situation and test area. Outputs from these evaluations can be broadly grouped into thefollowing three types of report outputs - SBD’s goal with these options is to ensure understanding of the results are as clear and fair as possible.

UX & functionality score

Modified Kano feature analysis

Subjective & objective scores

Final usability scored based on a 100-point scale withsolutions scoring less than 40 defined as “not fit for purpose”
with major user complaints expected and score above 80
defined as “exceptional”.

Features plotted against three lines based on their implementationand satisfaction levels:▪ Delight features: “wow” features likely to provide highsatisfaction even with poor implementation▪ Performance features: as the level of implementation increases
so does the customer satisfaction▪ Hygiene features: poor implementation provides low
satisfaction, but good implementation may not provide positive
satisfaction as it can be considered as expected functionality

Scoring across multiple areas through subjective scoring withpre-defined ranges, definitions, and comparison to past
results.

Functionality score based on eight core feature set areas:delight features, performance features, hygiene features,
13
navigation
feature, ADAS, IoT integration, music,
entertainment and info features.

Objective scoring generally based on a pass/fail criteria ortime-based considerations. SBD attempts to minimize
subjectivity as much as possible with results aimed at being
fair and reasonable with a minimal level of bias.
13

Example slides fromthe reportAnalysis

Impressive feature set, largely stable system

Car to X Synergy

LLM (Large Language Model)

Good level of stability and performance

As part of the IoT integration feature set, many car-tohome and home-to-car interactions are possible. For
support for the control of smart home devices is offeredincluding smart door locks, homes camera and lights. Geo
fencing scenarios can be created to run certain homeroutines. Home-to-car synergies include doorbell vision via
the central display and the ability to control many vehicle
settings
from a Xiaomi smart speaker. This level of15
connection adds to user convenience levels.

The large language model used to support the voicerecognition system allows for a more human likeinteraction, which many users are likely to enjoy using. The
integration of the LLM means a very high success rate atcompleting tasks quickly and accurately. It is noted that thedepth of results and data is very good, especially whencompared to other system that do not employ an LLM.

During testing, only one instance of instability is noted. This
low level of encountered instability not only has a positiveeffect on user experience and brand perception as a whole,
but also has a significant effect on the overall score for thisvehicle. Demonstrating a largely stable system may be linked
to Xiaomi’s experience in the CE world. Performance is also
considered fast, regarding either cold or warm boot, to routecalculation times.
15

Features & Functionality

Hygiene & Navigation

Hygiene features lacking, intuitive navigation features•

Users are not able to save radio presets or station favorites in the radio menu for
FM/AM stations.

•Traffic signal countdown feature integrated into the map display during active route
guidance.

•

Users are restricted on how they can search for radio station and cannot search forradio stations by radio number.

•

Automatic charge point route calculation plots a route to incorporate multiple
charge points on the way to a given destination.

•

Input method types are limited and several common input methods in China are
not accepted.

•

Limited POI subcategories make it difficult to search for more specific types of POI
like restaurant cuisines.

Not able to save favourite FM/AM radio stations

Traffic light countdown feature

The most notable missing hygiene feature from this system is the lack of ability to save
FM/AM radio station favorites or presets for easy and quick recall later. This issomething that nearly all users are likely to expect as a hygiene feature in all modern
vehicles. Even though FM/AM station may not be as widely used in China, thisfunctionality should still be made available. The absence of this feature will likely lead tofrustration and disappointment among some users.

An effective and accurate red-light countdown feature is integrated into the navigation
map display. This feature communicates the remaining time left at a red light andshows extra information on amber and green lights during active route guidance. Not
only is this feature an added convenience elements, but it may also contribute toenhanced levels of safety. However, it is noted that the display of this feature is small
and compact, making it hard for drivers to see while the vehicle is in motion. It also
adds to a sense of visual clutter experienced throughout the wider HMI.

16

16

Perceived quality

Visual highlights

Perceived Quality: Visual
Level 1Minimal cabin
design dominatedbig large central
display

VisualHarmony/alignment: Most of the controls and buttons in the cabinfollow a similar design language and ensure a sense of visual
harmony. However, the central display is not so well integrated anddominates the cabin. Physical buttons are minimalist, following thesame design philosophy as the cabin.

Geometric: All buttons and controls have shape that is appropriate
for the intended function.

SBD
viewpoint

Spatial harmony: The lower center console features tab switches
which are clear and uncluttered. The steering wheel contain two
extra small spokes with a button each. This does cornubite to visual
clutter but is not considered a major issue.

Minor visual clutter
from extra
steering wheel
control spokes

Branding: Branding is in line with what would be expected from a CE
device manufacturer. Crisp and clean lines are dominated by a large
screen with only minimal visual branding.
Output HMI: Output HMI throughout the cabin is consistent andfollows the accepted norms and conventions.
Interior lighting: Ambient lighting is visually impressive and can be
set to react to music. Button panels and displays are well lit both indaytime and nighttime conditions.

Level 2 scoring
Harmony/alignment

Good

Geometric

Spatialharmony

Branding

Output HMIInterior
lighting

Good

Excellent

Excellent

Good

Good

*Below expectation for segment

Appropriately
shaped buttons forintended function

17
17

ADAS Domain

SAE L0 - SBD viewpoint – System usage

Good warning in the central display, missing in the side mirrors
Category

System usage

SystemRCTA
Once the system is turned ON, it will stay in standby mode and be active as soon
as reverse gear is selected.

SBD
viewpoint

Both the surround view camera display and ADAS display are integrated wellwith RCTA HMI. If the system senses an obstacle passing behind the vehicle witha chance of imminent collision, the system will deploy rapid & strong braking(when set to Warning and Braking mode). A large pop-up warning is shown inthe central display at the same time. If in surround view camera display, theedge of impact will be displayed in the bird’s eye view in red. If in the ADAS
display, the target obstacle will be marked in red. The audio warning is loud,clear and distinctive from other audio warnings used throughout ADAS.
If the ADAS display is being shown, multiple types of traffic elements will berendered (pedestrians, two/three-wheelers, motorcycles and different types ofcars).

UX impact

Minor Positive

SBD
viewpoint

No warning lights are used on the side exterior mirrors for RCTA. Since these are
the locations that drivers are likely to be looking at when reversing, an alert
light/icon could be implemented to provide an extra level of alert when reversing.

UX impact
18

Minor Negative

RCTA UI integration in different displays (surround view/ADAS display)

No warning in mirror

18

Infotainment Domain

Positive points

Good level of UI customization

The system supports customization of several UI components on the
homepage.
• When searching for an address on the navigation widget, thekeyboard will automatically pop-up. Users can drag the keyboard
to move its position. This prevents information from being
obscured by the keyboard and also improves the ease of typing forboth the driver and passenger.
SBD
viewpoint

Drag to adjust the position of keyboard• Users can use the control bars on the left, right or top of eachwidget to adjust its size and position in the central display. This
allows driver to bring more commonly used features closer to thedriver's side of the vehicle.
• Users can display the mapped apps on a mobile device in the formof widgets or full-screen on the homepage (e.g., Drag the control
bar on the right side of the mapping screen to enable a full screen
display).
Being able to arrange the homepage and keyboard layout allows theuser to tailor the system to meet their needs and preferences while
prioritizing information that is important to them.

Drag to show the Baidu Map app in full-screen

UX impact
19

Major negative

Minor negative

Minor positive

Major positive
19

Navigation Domain

Negative points

Occasional strange lane guidanceOn occasion, the system will only recommend one possible lane when
in fact multi lanes can be taken to travel in the same direction.
• As shown in the image to the right, all 6 lanes have free traffic
flow. At that time, the middle 4 lanes (2nd to 5th) are the lanes
that can go straight. The recommended lane by the system is thesecond-to-last (5th) one.
SBD
viewpoint• However, the logic for its recommendation is not clear. If therecommendation is based on the level of congestion in each lane,
the third lane is the best based on this logic. If it is based on the
logic of being closer to the next turn, the second lane would be thebest.
While the recommended lane is not actually incorrect, the systemdoes not seem to be following any intelligent logic as to they onlythat lane is being recommended and not the other possible lanes in
addition. This could result in a confusing situation for the driver.
Recommendation logic for tidal flow is a little bit confusing

UX impact
20

Major negative

Minor negative

Minor positive

Major positive20

Convenience

Phone reminder

Phone reminder when getting off the car
Xiaomi offers a phone reminder feature when the phone is located on the wirelesscharging pad:
• This feature reminds the user that their phone is still in the vehicle (including
phones charged on the wireless charging panel and phones charged on the phone
stand) when the user opens the door and leaves the vehicle.

• A phone reminder pop-up message will be shown on the central screen ”Detectedthat your cell phone is still in the car”.
• A voice reminder will also be played “Your cell phone is still in the car”.
This feature prevents users from forgetting important items when in a hurry. As thephone stand is an optional accessory, this value-added feature also helps to promote
the purchase for accessories and realize the purpose of revenue generation.

Cell phone stand with wireless

21 charging (optional: RMB 149)

wireless charging panel (standard equipment)

Open the door

Phone reminder pop-up message on the central screen
21Request price forthe full reportContact Us

Contact SBD Automotive

Do you have any questions?
If you have any questions or feedback about thisresearch report or SBD Automotive’s consulting services,
you can email us at info@sbdautomotive.com or discusswith your local account manager below.
Envelope with solid fill

info@sbdautomotive.com
Book a meetingBook a meeting

Logo
Description automatically generated

23

USA

Garren Carr
North America
garrencarr@sbdautomotive.com+1 734 619 7969

UK

Germany

Luigi Bisbiglia
UK, South & West Europe
luigibisbiglia@sbdautomotive.com+44 1908 305102

Andrea Sroczynski
Germany, North & East Europe
andreasroczynski@sbdautomotive.com+49 211 9753153-1

India

China

Japan

SBD China Sales TeamChina
salesChina@sbdautomotive.com+86 18516653761

SBD Japan Sales TeamJapan, South Korea & Australia
postbox@sbdautomotive.com+81 52 253 6201
23