#635

TABLE OF CONTENTS
Introduction to this Report

In-Car HMI UX Evaluation& Benchmarking

UX
EVALUATION

Birds Eye View
Executive Summary
The basics

In-car HMI UX
Evaluations

Analysis
Features and Functionality
Execution

Lincoln Nautilus

Perceived Quality

ADAS Domain
Infotainment Domain

In this edition of the In-Car HMI UX Evaluation and Benchmarking series, the UX team is testing the
Lincoln Nautilus.

Navigation Domain
Voice Recognition Domain
Convenience Domain
Next Steps

RELATED SBD REPORTS
619 – UX Benchmarking Series
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s UX Team evaluates the infotainment
user experience of over 40 vehicles.
806 – ADAS HMI Evaluations
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s Autonomous Car Team evaluates the
ADAS performance and usability of over 20
vehicles.

COVERAGE

FREQUENCY

PUBLICATION FORMAT

PAGES

9
NA

CHINA

EUROPEANNUALLYQUARTERLYCARS PER YEARPD F

POWERPOINT

EXCEL

ONLINE

150+

#635/ In-Car HMI UX Evaluation & Benchmarking

Scoring> Features and functionality: > Reliability/stability:
evaluating whether theevaluating the repeatedsolutions provide featuresusability and whether thethat customers expect, needusers can have a similarand solve problems (or(positive) experience eachprovide a wow factor).
time.
> Usability:
evaluating whether the> Perceived quality:
features available are easy
evaluating the potential
to learn and use. This
perception in quality of theconsiders areas such as
HMI components and howergonomics, legibility,
this contributes to the overallusability characteristics andcustomer experience.
how the system implements
the various features.

This research is useful for

PRODUCT PLANNERS

Do I have access?

USER E XPE R IENCE

Portal access

M A R K E T I NG

C- S UITEE NGI NE E R S

Portal access

Request a quote for
In-Car HMI UX Evaluation & Benchmarking Series
Lincoln Nautilus

In-Car HMI UX Evaluation & Benchmarking
Lincoln Nautilus

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – Lincoln Nautilus
Introduction≫

5▪ Report introduction

The basics≫

21

Features and functionality≫▪ Aim of report▪ Overview▪ Scope of report▪ Delight and performance features▪ 2024 vehicles▪ Hygiene and navigation▪ UX principles▪ SBD experience▪ ADAS feature set▪ What’s new in vehicle UX?

▪ Methodology▪ 645 Hands-free driving▪ Report structure▪ IoT integration, music, entertainment
and information

Bird’s-eye view≫

7▪ OTA, FaaS + SW-defined features guide▪ The cockpit guide

Analysis≫▪ Health and wellbeing in automotive▪ SUS scores by domain▪ Digital cockpit & infotainment forecast▪ Kano analysis▪ Positive and negative implementations

Executive summary≫▪ Key takeaway 1▪ Key takeaway 2▪ Key takeaway 3▪ Ranking▪ Scoring

15▪ CX recommendations▪ Potential improvements

28

Execution≫▪ Overview▪ System scoring▪ UX laws▪ System performance▪ Stability▪ Ergonomics▪ Legibility and readability

35

41

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – Lincoln Nautilus
Perceived quality≫

74

Navigation domain≫

112

Convenience domain≫▪ Introduction▪ Summary▪ UX heuristics▪ Definitions▪ UX heuristics▪ HVAC summary▪ Evaluation results▪ Navigation specific scoring▪ HUD summary▪ Highlights▪ Key positive and negatives

ADAS domain≫

82

Voice recognition domain≫▪ Summary▪ Summary▪ UX Heuristics▪ Functionality▪ SAE L0

▪ Recognition▪ SAE L1&2

▪ SUS score▪ SBD UX principles

Infotainment domain≫99▪ Command structure▪ Summary▪ Localization▪ UX heuristics▪ Level of integration

Future outlook≫
128

139

145

▪ Introduction▪ UX principle groups

Next steps≫

150

Contact us≫

159▪ Key positive and negative

Total number of pages - 159

Introduction

Introduction

Report Introduction
Welcome to the 2024 HMI benchmarking report series. This report has been created toprovide a fair, unbiased and objective view of the latest in-vehicle HMI solutions in the US,
European, and Chinese markets. Evaluations are carried out by SBD usability experts witha deep understanding of CASE domains such as the Connected Car and ADAS & autonomy
domains.
SBD supports clients throughout the development of new HMI and products from a
relatively simple companion app to a more complex multi-domain infotainment solution.
The methodologies used in these reports take into account many years of experience with
consumer testing and custom client projects to provide a fair and, as much as possible,
objective methodology.
All viewpoints and analysis within the report are aimed defining areas of concern througha data driven approach. This report aims to benchmark and score solutions while also
being able to provide actionable recommendations to design and development teams.
Please note that due to the ever-evolving automotive technology market, SBD updates it’s
methodology each year, but does not update scores from the previous years. Therefore,
please assume a slight drop in scores for both user experience (UX) and functionality fromthe previous year.

Intuitive

7

Consistency

Supportive

Brevity

Flexibility

Depth

Presentation

SectionContent

Birds Eye View

An overview of the key findings from SBD’s various CX related andadjacent reports.

Executive
Summary

Presents key highlights and conclusions from the report.

The Basics

What do you need to know about SBD’s CX evaluation
methodology?

AnalysisAnalysis of report findings by SBD experts.

Features andfunctionality
Execution
Perceived quality

Overview of key features and functionality by domain.
Assess success of implementation and overall execution of various
system elements.
Scoring and analysis of interior perceived quality levels.

ADAS domain

Highlight and analysis of key positive and negative points within
the ADAS domain.

Infotainment
domainHighlight and analysis of key positive and negative points within
the infotainment domain.

Navigation domain

Highlight and analysis of key positive and negative points within
the navigation domain.

Voice recognition
domainSummary and scoring of various aspects of the voice recognition
system.

Convenience
domainSummary of various convenience focussed features.

Future Outlook

Seven UX principles are considered against drivers and barriers
into the future of this reports test vehicle.

Next StepsCan SBD help you with any unanswered questions?
7

The Basics

Aim of this report▪ Features and functionality: evaluating whether the solutions provide
features that customers expect & need, and solve problems (or provide a
wow factor)▪ Usability: evaluating whether the features available are easy to learn anduse. This considers areas such as ergonomics, legibility, usability
characteristics and how the system implements the various features

Brand perception

Consumer testing

One of the core goals of these studies is to provide a true indication of what
the final customer experience of each solution could be. To do thisevaluations are focused on providing scoring and analysis in the following
areas:▪ Reliability/stability: evaluating the repeated usability and whether theusers can have a similar (positive) experience each time
▪ Perceived quality: evaluating the potential perception in quality of the
HMI components and how this contributes to the overall customer
experience“Do I have a positive emotional attachment to thebrand?”

Aesthetics and engagement“Do I have a positive emotional attachment to thesolution?”

EMOTIONAL
APPEAL

Repeated usability

Expert testing(the focus of this report)

RELIABILITY“Can I have a similar experience each time?”

Ergonomics, usability characteristics &execution“Are the features available easy to learn
and use?”USABILITY

Domain functionality &prioritisation“Does the system provide features I
want and need to use?” i.e. “Does it
solve problems for me?”

8

BRAND
APPEALFUNCTIONALITY

SBD’s view on the hierarchy of needs for CX benchmarking
8The Basics

Scope of report: focus on in-car HMI evaluations
The scope of evaluations in this report are constrained to the in-car HMI
experience, in both static and dynamic conditions. One notable element isdriver distraction which SBD covers at only a high level in this study as
carrying out a full driver distraction evaluation requires biometrics test
equipment to ensure the collected data is unbiased and objective.

Prepurchase

Purchase

Activation

In-car HMI

Driverdistraction

IoT

A full evaluation of the end-to-end customer experience is not within scope of
this report, but it is something which SBD has many years experience in fromboth a consumer and expert perspective. Other areas such as the companion
app, online portal and in-home smart devices are not in scope as they aredefined as “out of car” experiences.
Within the vehicle, any HMI element the user interacts with is evaluatedincluding steering wheel controls, touch screen displays, voice control, HUDs
and digital keys. The features and services on offer have been broadlygrouped into the following domains (or test areas):▪ ADAS domain▪ Infotainment domain▪ Navigation domain▪ Voice recognition domain▪ Convenience domain
99The Basics

2024 vehicle list
SBD has chosen nine cars to evaluate in 2024, based on two selection categories. New/interesting UX focuses on systems with to never-seen-before
features or functionality, or the implementation of a solution that has previously been a challenge or pain-point for end-users. New mass-market UX
includes vehicles in segments that are sold in high numbers and are entering a new generation of UI for that vehicle. While we make best efforts toadhere to the chosen cars and schedule, the last two years have seen release dates slipping significantly, so it may be necessary to makesubstitutions.
Group A
Cars tested

BMW X1

Mercedes-Benz E-Class▪ First BMW to be released
with iDrive9▪ Android based system▪ US market test

▪ All-new MBUX Superscreen▪ Unique features, apps andgames▪ German market test

Priority target vehicles for 2024

Lincoln Nautilus▪
▪
▪
▪

BlueCruise hands free
Digital Experience
All-new infotainment system
US market test

Hyundai Kona/Kona Electric

Volvo EX90

▪ 12.3-inch cluster andcentral display▪ New Bluelink+ services▪ US market test

▪ Volvo Pilot Assist▪ 14.5-inch central display▪ EU or US market test

Group B
Cars tested

Xiaomi SU7▪ Xiaomi’s first vehicle offering▪ New vehicle from CE company▪ Xiaomi Pilot MAX▪ Chinese market test
10

Other vehicles being considered for 2024

Audi Q6 e-tron▪ All-new “Digital Stage”
infotainment system▪ AR HUD integration▪ EU market test

Acura ZDX
▪ Google Built-in▪ AcuraWatch 360+ withhands free cruise▪ US market test

Avatr 12▪
▪
▪
▪

Harmony 4.0 OS
Huawei ADS 2.0 (ADAS)
Innovative displays
Chinese market test
10

The Basics

SBD experience through years of testing in-car solutions
Over the last ten years SBD has evaluated over 100 solutions from a Connected Car or ADAS perspective for our public report series (many more for privateclient evaluations). This current report series is an evolution of both test methodologies to provide a holistic view of in-car HMI. Furthermore, custom
evaluations methodologies used across the globe for SBD clients have been included where applicable to enhance to overall approach.

11
11

The Basics

One page methodology overview
One of SBD’s core goals of this report is to be as objective, fair
and as transparent as possible. To achieve this, various
methodologies are used throughout the testing to evaluate
different areas of the solution in various conditions.
These methodologies are a mix of different types of tests:▪ Objective tests: where the value provided is not influenced
by a tester’s viewpoint e.g. response time
▪ Subjective tests: the test score is based on the expert
testers’ viewpoints e.g. task ease of use▪ Task-based: evaluations carried out based on a predefinedtask list e.g. navigate to a pizza restaurant near location X▪ Freeform: random free testing by the tester with no clear
pre-defined task list. This allows the testers flexibility to dig
deeper into various parts of a solution when needed▪ Scoring range: ranges and definitions of how to score atest element e.g. poor depth and accuracy score = theresults provided are not in line with what is reasonably
expected by the user

Type of tests
Test area

Objective

First impressions

Subjective










Dynamic tasks









Random freeNavigation specific tests







Voice recognition







Performance & response

System Usability Scale
(SUS)





UX heuristics

Legibility & readability

Perceived Quality (PQ)


































































Ergonomics

Misuse/failures





ADAS

Dynamic

▪ Dynamic: tests are carried out when the vehicle is moving
in various road conditions and locations e.g.
motorways/highways, cities, villages, country roads etc.

Static

ExecutionFor a detailed discussion and presentation of SBD’s
12methodology
please contact us.

Scoringrange

Final SBD UX score

This document does not provide a detailed description of themethodology and this page serves to provide an overview of
the approach.

Freeform

Static tasks▪ Static: tests are carried out when the vehicle is not moving▪ Misuse/failures: carried out to evaluate the stability of thesolution in unusual conditions e.g. repeatedly pressing thevoice command button

Task based









12

The Basics

Report structure and how to interpret certain data sets
Throughout the testing and evaluation process SBD uses multiple methodologies to align to the situation and test area. Outputs from these evaluations can be broadly grouped into thefollowing three types of report outputs - SBD’s goal with these options is to ensure understanding of the results are as clear and fair as possible.

UX & functionality score

Modified Kano feature analysis

Subjective & objective scores

Final usability scored based on a 100-point scale withsolutions scoring less than 40 defined as “not fit for purpose”
with major user complaints expected and score above 80
defined as “exceptional”.

Features plotted against three lines based on their implementationand satisfaction levels:▪ Delight features: “wow” features likely to provide highsatisfaction even with poor implementation▪ Performance features: as the level of implementation increases
so does the customer satisfaction▪ Hygiene features: poor implementation provides low
satisfaction, but good implementation may not provide positive
satisfaction as it can be considered as expected functionality

Scoring across multiple areas through subjective scoring withpre-defined ranges, definitions, and comparison to past
results.

Functionality score based on eight core feature set areas:delight features, performance features, hygiene features,
13
navigation
feature, ADAS, IoT integration, music,
entertainment and info features.

Objective scoring generally based on a pass/fail criteria ortime-based considerations. SBD attempts to minimize
subjectivity as much as possible with results aimed at being
fair and reasonable with a minimal level of bias.
13

Example slides fromthe report

Executive Summary

Key Takeaway 1

Overall, a holistic system

Good consistency
across displays andthroughout the cabin•

Typically when comparing consistency between displays in a cockpit, like the instrument
cluster to the central display or the head-up display (HUD) to the instrument cluster,
there are some obvious differences between displays noted for the same feature. Making
it obvious that the displays were designed by two different design teams. However, thisis not the case for the Lincoln Nautilus when comparing consistency between thepanoramic display and the central display.

•

Most of the features for the Nautilus are mostly consistent in look and feel between thepanoramic display and the central display, giving the impression that both displays were
designed by the same team, or the two teams did really well communicating with eachother during the design process. For example, radio and media, and navigation are allvery similar between the displays to provide one cohesive system. Things like Android
Auto and Apple CarPlay integration are also well thought out and integrated to support
this.

•

Along with the displays, there’s consistency throughout the rest of the cockpit withthings like material and build quality. Making sure the core features are well thought out
and integrated consistently across displays goes a long way with consumers and helps toprovide an overall positive user experience. This also helps to elevate impressions ofoverall perceived quality of the vehicle.

15

Good consistency between displays overall

Apple CarPlay integration well thought out for both displays
15

Features & Functionality

Delight and Performance features

Few delight features, serious performance feature issues•

Panoramic display offers visually impressive delight feature as well as bringing
information up closer to the driver’s line of sight.

•

Changing functionality of steering wheel controls impact multiple systems
including ADAS.

•

Configurable fragrance box allows drivers the ability to personalize the fragrance
within the cabin.

•

Route guidance issues include the system guiding down closed roads andindicating illegal turn maneuvers.

•

User can download and use conferencing apps within the vehicle including Webex.

•

The voice recognition system fails to take into account other input methods
such as touch input.

Panoramic display dominates cabin

Unlabeled steering wheel buttons

The panoramic display stretches from pillar-to-pillar and offers an impressive visualelement to the cabin. While the left-hand side houses the cluster and cannot be
customized, the right-hand side does allow for a limited amount of customization. In
addition to being something that many users are likely to get a wow-factor from, it also
brings information up closer to the driver line of sight while driving, minimizing theglance down angle to obtain important information.

The lack of labelling and changing functionality of steering wheel buttons is one of the
key negative issues regarding the entire HMI. This causes confusion and high risk ofmisoperation while driving. Not only is this frustrating but it is also likely to negatively
impact confidence levels of new users. Providing intuitive icons and labelling would
allow users to glance at the buttons and understand their functionality immediately
rather than having to touch them first in order to bring up a legend in the cluster
screen.

1616Perceived quality

Visual highlights

Perceived Quality: Visual
Level 1VisualHarmony/alignment: The main negative aspect that disrupts thevisual harmony is the obvious line down the middle of the panoramic
display.

Obviousseparation line
down center ofpanoramic display

Geometric: Slight concern surrounding the circular volume dial on
the lower center console. The shape and location suggests it may be
a gear selector, when in fact it controls volume. In addition, the gear
selection buttons are not in shape and position that would suggest
their functionality. Also, the buttons on each side of the volume dialgive the impression that their toggle switches, when in fact, they'rejust push buttons.
SBD
viewpoint

Spatial harmony: Some elements of the on-screen HMI including
the media screen and HVAC bar can appear cluttered and squished.
Some of this clutter is cause for small touch targets, making it
difficult to select/adjust, especially while driving.
Branding: Branding is very minimal throughout the cabin and the
HMI, but the visual appearance of the HMI seems to match the brandidentity and ethos of Lincoln and the car.
Output HMI: Unique integration and usage of ambient lighting on
the inside of the doors in which it changes color based on whether
the door is open or closed.

Interior lighting: Very nice settings offering for ambient lighting,many different color choices offered, and many different
intensity/brightness levels offered. Good illumination of buttons anddial in the dark.

Unique ambient
lighting
implementation

Level 2 scoring
Harmony/alignment

Geometric

Spatialharmony

Branding

Output HMIInterior
lighting

Fair

Fair

Fair

Good

Good

Good
17
17

ADAS Domain

SAE L0 - SBD viewpoint – System usage

Good directional warning, but could utilize mirrors too
Category

System usage

SystemRCTA

Once the system has been turned ON via the central display, activation is
automatic when reverse gear is chosen by the driver.

SBD
viewpoint

Infotainment and other volume is reduced when an RCTA warning is given inorder to prioritize critical information for the driver. While the volume is notcompletely muted, this is a good addition to information prioritization. The visual
warning provided in the panoramic display is directional providing extra context
for the driver. A clear and concise warning message is also shown. When thesystem detects a collision is imminent, automatic braking is applied without theneed for driver intervention. This proactive feature helps to avoid collision thatthe driver may not be quick enough to mitigate themselves.

UX impact

Minor Positive

SBD
viewpoint

When detecting obstacles behind the vehicle, no obvious distinction between
vehicles and pedestrians is given. While this isn’t essential, it does help toprovide extra context for the driver in a warning situation. The color used on the
directional graphic warning is red, while the color used on the warning icon isorange. The mismatch could be confusing and suggest that two different system
are active and providing a warning.

RCTA event warning in panoramic display

RCTA does not utilize any visual warning in the side mirrors. This is a potential
missed opportunity to enhance the level of visual warning in an area that theuser is likely to be looking while reversing.
UX impact
18

Minor Negative

No use of warning in mirror

18

Infotainment Domain

Positive points

Good integration of Apple CarPlay/Android Auto with panoramic display

Wireless Apple CarPlay and Android Auto are offered for the
infotainment system.
• A new shortcut button appears on the left-side controls whichallow users to quickly swap between the native and phone systems.

SBD
viewpoint• When navigation is set, the map on the panoramic display willswap from Google Maps to Apple Maps and show the navigation
when CarPlay is connected. When Android Automotive is used, thepanoramic display will keep the Google Maps.

• When playing media while using Apple CarPlay or Android Auto,the media screen on the panoramic display will indicate it isplaying audio from Apple CarPlay or Android Auto, as opposed toBluetooth.
Having common third-party solutions well integrated into the system
is a plus for consumers. Allowing the user to choose which systemthey want to use (native/Apple CarPlay/Android Auto) provides
flexibility.
Apple Maps appears in place of Google Maps on the panoramic display

UX impact
19

Major negative

Minor negative

Minor positive

Major positive
19

Navigation Domain

Negative points

Many inaccuracies experienced with navigation

During testing, there were inaccuracies observed in the navigation
system. These included the following points.
• While navigating through downtown Detroit, twice the systemprovided route guidance that included illegal turns. One wasmarked by a no left turn sign at an intersection and the second
was an illegal U-turn where one was not possible to be made.
SBD
viewpoint• Occasional instances were recorded where the lane guidance did
not reflect ground truth (i.e. 5 lanes shown when the road was
only 4 wide).

Road is 4 lanes wide including turn lanes, not 5 according to lane guidance• One instance was recorded where navigation set the destination
far away from the actual desired building (Henry Ford Museum).
These inaccuracies, especially illegal turns, is a cause of concern.
Most competent drivers would be able to deal with these issues, butusers would expect more out of a premium vehicle system.

Arrow indicatesvehicle direction of
travel. Lane intended
for left-hand turnonto brush street,not U-turn

UX impact
20

Major negative

Minor negative

Minor positive

Major positive

Navigation suggests a U-turn where it is not possible
20

Voice recognition

Recognition

SBD recognition score: C
Recognition

Recognition results are based on the rate of utterance recognition of the solution.
Currently the solution is scored as “C” overall for performance.

100%▪ At 55% overall positive recognition score, the system is rated as “C+” forrecognition. With unsupported use cases removed from the recognition stack,the rate of accuracy increases to 89%, showing a significant improvement to
the recognition rate.

90%▪ When looking at all possible SBD use cases, most recognition failures occur
within inferred meaning, phone and vehicle control use cases.

80%▪ HVAC and Navigation use cases demonstrate a particularly strong rate of
positive recognition with the majority of use cases passing successfully.
▪ A small number of second positives are noted within media use cases andcalendar use cases within integration tasks. Within media, this is due to thesystem giving a choice of sources to play from when asking for a specificartist. Once the user has replied with a preferred source, the system will play
the requested artist.

Fail
Incorrect Result

Fail2nd positive

70%
Incorrect Result

60%

50%

2nd positive

Positive

40%

30%
Positive
20%

10%

21

0%

All possible
SBD use cases

System-supported
use cases only

21

Request price forthe full reportContact Us

Contact SBD Automotive

Do you have any questions?
If you have any questions or feedback about thisresearch report or SBD Automotive’s consulting services,
you can email us at info@sbdautomotive.com or discusswith your local account manager below.
Envelope with solid fill

info@sbdautomotive.com
Book a meetingBook a meeting

Logo
Description automatically generated

23

USA

Garren Carr
North America
garrencarr@sbdautomotive.com+1 734 619 7969

UK

Germany

Luigi Bisbiglia
UK, South & West Europe
luigibisbiglia@sbdautomotive.com+44 1908 305102

Andrea Sroczynski
Germany, North & East Europe
andreasroczynski@sbdautomotive.com+49 211 9753153-1

India

China

Japan

SBD China Sales TeamChina
salesChina@sbdautomotive.com+86 18516653761

SBD Japan Sales TeamJapan, South Korea & Australia
postbox@sbdautomotive.com+81 52 253 6201
23