#635

In-Car HMI UX Evaluation& Benchmarking

UX
TABLE OF CONTENTS

EVALUATION

Executive Summary
Introduction to this ReportIn-car HMI UX
Evaluations

Features and Functionality
Execution
Perceived Quality

Fisker Ocean

ADAS Domain

Infotainment Domain

In this edition, the UX team is testing the Fisker Ocean.

Navigation Domain
Voice Recognition Domain

The Fisker Ocean returns a disappointing user experience score, finishing lowest among all vehicles tested
in 2023 so far and only achieves a B- for functionality, finishing equal to the second lowest vehicle tested in
2023. This scoring is based on the current offering at the time of testing (August 2023).

Connected Features Domain
Convenience Domain

RELATED SBD REPORTS
619 – UX Benchmarking Series
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s UX Team evaluates the infotainment
user experience of over 40 vehicles.
806 – ADAS HMI Evaluations
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s Autonomous Car Team evaluates the
ADAS performance and usability of over 20
vehicles.

COVERAGE

FREQUENCY

PUBLICATION FORMAT

PAGES

6
NA

CHINA

EUROPEANNUALLYQUARTERLYCARS PER YEARPD F

POWERPOINT

EXCEL

ONLINE

167

#635/ In-Car HMI UX Evaluation & Benchmarking

Scoring> Features and functionality: > Reliability/stability:
evaluating whether theevaluating the repeatedsolutions provide featuresusability and whether thethat customers expect, needusers can have a similarand solve problems (or(positive) experience eachprovide a wow factor).
time.
> Usability:
evaluating whether the> Perceived quality:
features available are easy
evaluating the potential
to learn and use. This
perception in quality of theconsiders areas such as
HMI components and howergonomics, legibility,
this contributes to the overallusability characteristics andcustomer experience.
how the system implements
the various features.

This research is useful for

PRODUCT PLANNERS

M A R K E T I NG

C- S UITEDo I have access?

USER E XPE R IENCEE NGI NE E R S

Portal access

Request a quote for
In-Car HMI UX Evaluation & Benchmarking Series
Fisker OceanSeptember 2023

In-Car HMI UX Evaluation & Benchmarking
Fisker Ocean

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – Fisker Ocean
Executive Summary≫5

Features and functionality≫

22

Perceived quality≫▪ Overall Score▪ Overview▪ Introduction▪ SUS score by domain▪ Delight and performance features▪ Definitions▪ Kano analysis▪ Hygiene and navigation▪ Evaluation results▪ Positive and negative implementations▪ ADAS feature set▪ Highlights▪ Driver distraction▪ IoT integration, music, entertainment
and information▪ PQ Overview▪ CX recommendations

Execution≫▪ Potential improvements

Introduction to this report≫▪ Aim of report▪ Scope of report▪ 2023 vehicles▪ SBD experience▪ Methodology▪ Report structure▪ Overview

15

ADAS domain≫
28

63

71

▪ Summary▪ Failure modes▪ Misuse▪ System scoring▪ UX Heuristics▪ UX laws▪ SAE L0

▪ System performance▪ SAE L1&2

▪ Reliability▪ Ergonomics

Infotainment domain≫▪ Legibility and readability▪ Summary▪ UX heuristics▪ Key positive and negative95

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – Fisker Ocean
Navigation domain≫

111

Connected features domain≫▪ Summary▪ Summary▪ UX heuristics▪ Key positive and negatives

143▪ Navigation specific scoring

Convenience domain≫▪ Key positive and negatives

151▪ UX heuristics

Voice recognition domain≫▪ Summary▪ Functionality

133▪ HVAC summary▪ HUD summary▪ User profile summary▪ Performance▪ Stability▪ SUS score▪ SBD UX principles▪ Command structure

Next steps≫

158

Contact us≫167▪ Localization▪ Integration▪ Amazon Alexa integration

Total number of pages - 167

Introduction

Introduction

Aim of this report

One of the core goals of these studies is to provide a true indication of what
the final customer experience of each solution could be. To do thisevaluations are focused on providing scoring and analysis in the following
areas:▪ Features and functionality: evaluating whether the solutions provide
features that customers expect & need, and solve problems (or provide a
wow factor)

Brand perception

Consumer testing

Welcome to the 2023 HMI benchmarking report series. This report has beencreated to provide a fair, unbiased and objective view of the latest in-vehicle
HMI solutions in the US, European, and Japanese markets. Evaluations arecarried out by SBD usability experts with a deep understanding of CASE
domains such as the Connected Car and ADAS & autonomy domains.

▪ Usability: evaluating whether the features available are easy to learn anduse. This considers areas such as ergonomics, legibility, usability
characteristics and how the system implements the various features▪ Perceived quality: evaluating the potential perception in quality of the
HMI components and how this contributes to the overall customer
experience
SBD supports clients throughout the development of new HMI and products
from a relatively simple companion app to a more complex multi-domain
infotainment solution. The methodologies used in these reports take into
account many years of experience with consumer testing and custom client
projects to provide a fair and, as much as possible, objective methodology.
All viewpoints and analysis within the report are aimed defining areas ofconcern through a data driven approach. This report aims to benchmark andscore solutions whilst also being able to provide actionable recommendations
to design and development teams.

Please note that due to the ever-evolving automotive technology market,
SBD updates it’s methodology each year, but does not update scores from
7 previous years. Therefore, please assume a slight drop in scores for boththeuser experience (UX) and functionality from the previous year.

BRAND
APPEAL

Aesthetics and engagement“Do I have a positive emotional attachment to thesolution?”

EMOTIONAL
APPEAL

Repeated usability
RELIABILITY

Expert testing(the focus of this report)▪ Reliability/stability: evaluating the repeated usability and whether theusers can have a similar (positive) experience each time

“Do I have a positive emotional attachment to thebrand?”“Can I have a similar experience each time?”

Ergonomics, usability characteristics &execution“Are the features available easy to learn
and use?”USABILITY

Domain functionality &prioritisation“Does the system provide features I
want and need to use?” i.e. “Does it
solve problems for me?”FUNCTIONALITY

SBD’s view on the hierarchy of needs for CX benchmarking
7

Introduction

Scope of report: focus on in-car HMI evaluations
The scope of evaluations in this report are constrained to the in-car HMI
experience, in both static and dynamic conditions. One notable element isdriver distraction which SBD covers at only a high level in this study as
carrying out a full driver distraction evaluation requires biometrics test
equipment to ensure the collected data is unbiased and objective.

Prepurchase

Purchase

Activation

In-car HMI

Driverdistraction

IoT

A full evaluation of the end-to-end customer experience is not within scope of
this report, but it is something which SBD has many years experience in fromboth a consumer and expert perspective. Other areas such as the companion
app, online portal and in-home smart devices are not in scope as they aredefined as “out of car” experiences.
Within the vehicle, any HMI element the user interacts with is evaluatedincluding steering wheel controls, touch screen displays, voice control, HUDs
and digital keys. The features and services on offer have been broadlygrouped into the following domains (or test areas):▪ ADAS domain▪ Infotainment domain▪ Navigation domain▪ Voice recognition domain▪ Connected services domain▪ 8Convenience domain
8Introduction

2023 vehicle list
SBD has chosen six cars to evaluate in 2023, based on two selection categories. New/interesting UX focuses on systems with to never-seen-before
features or functionality, or the implementation of a solution that has previously been a challenge or pain-point for end-users. New mass-market UX
includes vehicles in segments that are sold in high numbers and are entering a new generation of UI for that vehicle. While we make best efforts toadhere to the chosen cars and schedule, the last two years have seen release dates slipping significantly, so it may be necessary to makesubstitutions.

Cars tested

Jeep Grand
Wagoneer

Nissan Ariya▪ Passenger display▪ ProPilot Assist▪ Amazon Fire TV▪ Unique HMI andinterior▪ Active Drive Assist▪ US market test

▪ EV SUV▪ German Market Test

BYD Seal▪ Unique Home Screen
HMI

▪ EV
▪ Chinese market test

Planned cars for 2023

Peugeot 408

Fisker Ocean▪ Drive Assist 2.0▪ EV SUV▪ 3D instrument cluster▪ Sustainable materials▪ German market test

▪ Large central display▪ German market test

MG4
▪ Chinese-owned brandsold in EU market▪ MG Pilot ADAS
▪ Live services▪ EU market test99Introduction

SBD experience through years of testing in-car solutions
Over the last ten years SBD has evaluated over 100 solutions from a Connected Car or ADAS perspective for our public report series (many more for privateclient evaluations). This current report series is an evolution of both test methodologies to provide a holistic view of in-car HMI. Furthermore, custom
evaluations methodologies used across the globe for SBD clients have been included where applicable to enhance to overall approach.

1010Introduction

One page methodology overview
One of SBD’s core goals of this report is to be as objective, fair
and as transparent as possible. To achieve this, various
methodologies are used throughout the testing to evaluate
different areas of the solution in various conditions.
These methodologies are a mix of different types of tests:▪ Objective tests: where the value provided is not influenced
by a tester’s viewpoint e.g. response time
▪ Subjective tests: the test score is based on the expert
testers’ viewpoints e.g. task ease of use▪ Task-based: evaluations carried out based on a predefinedtask list e.g. navigate to a pizza restaurant near location X▪ Freeform: random free testing by the tester with no clear
pre-defined task list. This allows the testers flexibility to dig
deeper into various parts of a solution when needed▪ Scoring range: ranges and definitions of how to score atest element e.g. poor depth and accuracy score = theresults provided are not in line with what is reasonably
expected by the user

Type of tests
Test area

Objective

First impressions

Subjective
P

P

P

P

P

Dynamic tasks

P

P

P

P

Random freeP

P

Navigation specific tests

P

P

P

Voice recognition

P

P

P

Performance & response

P

System Usability Scale
(SUS)

P

P

UX heuristics

P

P

Legibility & readability

P

Perceived Quality (PQ)

P

P

P
P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

Ergonomics

P

Misuse/failures

P

P

ADAS

Dynamic

P

P▪ Dynamic: tests are carried out when the vehicle is moving
in various road conditions and locations e.g.
motorways/highways, cities, villages, country roads etc.

StaticP

P
P

ExecutionPPP

For a detailed discussion and presentation of SBD’s
11
methodology
please contact us.

Scoringrange

PFinal SBD UX score

This document does not provide a detailed description of themethodology and this page serves to provide an overview of
the approach.

Freeform

Static tasks▪ Static: tests are carried out when the vehicle is not moving▪ Misuse/failures: carried out to evaluate the stability of thesolution in unusual conditions e.g. repeatedly pressing thevoice command button

Task based

P

P

P

P

11

Introduction

Report structure and how to interpret certain data sets
Throughout the testing and evaluation process SBD uses multiple methodologies to align to the situation and test area. Outputs from these evaluations can be broadly grouped into thefollowing three types of report outputs - SBD’s goal with these options is to ensure understanding of the results are as clear and fair as possible.

UX & functionality score

Modified Kano feature analysis

Subjective & objective scores

Final usability scored based on a 100-point scale withsolutions scoring less than 40 defined as “not fit for purpose”
with major user complaints expected and score above 80
defined as “exceptional”.

Features plotted against three lines based on their implementationand satisfaction levels:▪ Delight features: “wow” features likely to provide highsatisfaction even with poor implementation▪ Performance features: as the level of implementation increases
so does the customer satisfaction▪ Hygiene features: poor implementation provides low
satisfaction, but good implementation may not provide positive
satisfaction as it can be considered as expected functionality

Scoring across multiple areas through subjective scoring withpre-defined ranges, definitions, and comparison to past
results.

Functionality score based on eight core feature set areas:delight features, performance features, hygiene features,
12
navigation
feature, ADAS, IoT integration, music,
entertainment and info features.

Objective scoring generally based on a pass/fail criteria ortime-based considerations. SBD attempts to minimize
subjectivity as much as possible with results aimed at being
fair and reasonable with a minimal level of bias.
12

Example slides fromthe reportFeatures & Functionality

Delight and Performance features

Impressive delight features, lack of flexibility and filtering•

The Fisker is likely to provide high levels of satisfaction via its numerous delightfeatures integrated into the vehicle.

•

Slow route calculation times and lag when interacting with the map affect theoverall impression of the system.

•

Major delight features include Hollywood Mode, California Mode, Boost Mode andsolar panel roof.

••

Other minor delight features that are till likely to impress many users include underseat storage, rear LCD HVAC controls and fold out/pull out trays for both driver andpassenger.

Filtering of charge point location and searching for POI in specific locations
other than the current location is not optimized and is likely to become
frustrating.

•

The lack of ability to customize menus and button layouts prevents the systemfrom being truly flexible and allowing the user to tailor the HMI to their needs.

Hollywood Mode

Lack of filtering ability for charge points

By pressing and holding the center button on the HVAC bar below the central display,
the screen rotates to landscape orientation which is referred to as Hollywood Mode.
This gives access to all entertainment apps and allows users to stream movies andwatch videos while the vehicle is stationary. This mode is not possible when the vehicle
is in motion. As soon as another gear other than Park is selected, the screen rotatesback to portrait orientation.

The only obvious parameters for filtering charge points when searching using thenavigation system is by charge speed. Only two speeds are able to be selected asfilters. User can filter by brand but this must be typed out in full using the search bar.
This limited filtering is likely to become frustrating for many users and may result inusers switching to external devices to find the desired chargers.

14
14

ExecutionⅼReliability issues

Unresponsive navigation interface

6. Navigation

On one occasion, the entire navigation interface
including controls and buttons become unresponsive.
Other buttons around the navigation window responded
correctly to user input. A soft software reset was
required.

Frequency

Low

MediumHigh

Severity

Minor

Major

Critical15
15

ExecutionⅼErgonomics highlights

Key highlight

Steering wheel buttons easy to operate
The buttons on the steering wheel are large and prevent misoperation
The buttons located on the steering wheel panels are very large andobvious and can be operated easily. This reduces chances of
misoperation and allows for operation while driving without the need totake eyes off the road.
Additionally, many users will be able to operate these buttons while still
keep their elbow on the center arm rest and door armrest.

Large steering wheel buttons

16
16

ExecutionⅼLegibility & Readability

Legibility: Contrast

Contrast is an issue in central display
Navigation zoom and re-center buttons present poor contrast

The navigation zoom buttons and re-center button in the central display
are colored in a light gray against a very similar light gray map coloring.
This results in the button blending into the background and the black
text becoming confused with map labelling. This contrast scored“Significant risk” with 1.60:1. As these buttons are likely to be usedwhile driving, it scores an “Unacceptable” acceptability rating.
The instrument cluster poses no risk with good contrast due to lightcolored text on a dominantly dark background. The instrument cluster
scores “Low/no risk”.

Contrast concerns in central display

Display

Worst text contrast ratio

Final Rating

Central display

1.60:1Significant risk

Instrument cluster

9.99:1

Low/no risk

Severity / Driver distraction
RatingAcceptability Rating

Central display

Significant risk

Unacceptable

Instrument cluster

Low/no risk

Acceptable

Display17
17Perceived quality

Tactile highlights

Perceived Quality: Tactile
Level 1

TactileStiffness & looseness: The screen can wobble when slightpressure is applied. It does not feel secure. Trim bar across thedashboard feels loose and bends under slight pulling and pushing.
Door handles have a flimsy tactile quality. Steering wheel buttons
have wobble to the entire panel. The center arm rest has a large
amount of unintended movement and lower center console plastic
components move around more than expected.

Trim bar wobbles
and flexes easily

Force feedback: Steering wheel buttons feel soft and welldamped. HVAC button bar has a harsh clicking feel with nodamping. Steering wheel scroll wheels have massive indents which
add a perception of quality.
SBD viewpoint

Material quality: The door handle has the lowest tactile quality. It
gives the impression of being hollow. Lower center console has aninconsistent material quality to everything else. Upper dash andsuede material give premium touch and appearance. The HVAC
buttons can be pulled upwards and to increase values. (clearly notintended)Material harmony: Material quality varies throughout cabin withno real logic. Four different materials come into close visualcontact and can create a messy appearance.
Geometric & Positioning: The geometric and chunky appearance
and tactile quality of buttons, especially around the steering wheel
match the rugged yet minimal aesthetic of the vehicle.

Suede dashboard

Door handle below
expected quality

Level 2 scoring
Stiffness &looseness

Force feedback

Material quality

Material
harmony

Geometric &positioning

Fair

Fair

Fair

Fair

Good
18
18

ADAS Domain

SAE L0 - SBD viewpoint – System usage

Good level of warning, lack of differentiation for LKA/LDW
Category
System

SBD viewpoint

UX impact

SBD viewpoint

19

LKA/LDW
LKA/LDW warnings are presented in a permanent
ADAS display in the top portion of the central
display. An icon is also presented in the cluster
during warnings. The ADAS display is permanentand does not change based on other information
shown in the central display.
A good level of warning is shown in the ADAS
display during an event with the correspondinglane line illuminating and flashing orange alongwith a clear and distinctive audible warning. The
LKA/LDW ISO icon is used to show an additional
visual warning in the cluster. In the settings
screen, the user can choose to adopt the‘recommended settings’ for a quick setup or
customize them to their preferences.
Minor Positive
The warning icon shown in the cluster does notdepict which side of the lane is being crossed andoffers very little detail. The audio warnings given
are not prioritized over infotainment volume andcan become lost if the infotainment volume is high.
Locating the main LKA/LDW warnings in thecentral display means users must look right (Left
Hand Drive) to be informed of a lane departure
happening on the left. This causes disorientation.
No lane tracking status is provided in any of thedisplays to show system status. The difference in
HMI between LKA and LDW is so subtle that it
could easily be missed. When LDW is chosen andpermanent icon is show in the cluster with slightdifference to that of the LKA warning icon.

System Usage
BSM

RCTA

Three visual warnings are given for BSM in three
separate locations throughout the HMI. While thewarnings are noticeable, they are distracting whenpurely notifying of a vehicle’s presence. When thevehicle is below the minimum speed for operation,no level of indication is given in the HMI to informthe user of this.

Minor Negative
The ISO BSM icon is used in the side mirrors aspart of the visual warning. This helps to link thewarning to the system clearly. When a vehicle isdetected in the blind spot, the icon in the mirror,
wave in the central display and segment in thecluster illuminate orange. When indicating into thepath of this vehicle all the graphics begin to flash
and an audio alert is given.
Orange propagating waves shown behind vehicle
graphic in central display when vehicle detected.
Audio warnings are prioritized over infotainment
volume and can be clearly heard in all scenarios. A
door opening warning utilizes the BSM system toprovide extra alerts when the vehicle is stationary
and a vehicle is approaching. In the settings
screen, the user can choose to adopt the‘recommended settings’ for a quick setup or

19

Infotainment Domain

Positive points

Permanent shortcut bar enables quick access

The side bar menu provides quick access for various features.
• A permanent shortcuts bar is shown down the driver’s side of thecentral display, or along the bottom when in landscape orientation.
This shortcuts bar provides quick access to all of the main featuresand helps to create a shallow menu structure throughout thesystem.
SBD
viewpoint• The side bar menu provides permanent access to the home screen
from any page within the central display. Being positioned down
the driver’s side of the screen, the driver can access and operate
this menu bar without moving their hand too far away from thesteering wheel.
The permanent menu bar is a good way of anchoring quick access tothe commonly used features within the system. One improvement
could be to make this bar customizable in order to add even more
flexibility and personalization.

Permanent shortcuts bar

UX impact
20

Major negative

Minor negative

Minor positive

Major positive20

Connected features

Negative points

No IoT offering

No IoT integration is offered as part of the system yet.
• IoT integration is severely lacking as part of the feature offering inthis system. No supplier of IoT such as Google or Amazon has
been integrated into the system.

SBD
viewpoint• The lack of IoT means the user is unable to interact with any
smart home devices or automated smart systems outside of thevehicle.
• Amazon Alexa is expected to be introduced in a future update,however at the time of testing this feature was not present.
While IoT integration in vehicles is still in relevant infancy, it is
becoming ever more popular and common. OEMs should consider
this in order to not get left behind in terms of in-cabin featureofferings.

UX impact
21

Major negative

Minor negative

Minor positive

No IoT offering

Major positive21

Request price forthe full reportContact Us

Contact SBD Automotive

Do you have any questions?
If you have any questions or feedback about thisresearch report or SBD Automotive’s consulting services,
you can email us at info@sbdautomotive.com or discusswith your local account manager below.

info@sbdautomotive.com

Book a meeting

USA

Garren Carr
North America
garrencarr@sbdautomotive.com+1 734 619 7969

23

UK

Germany

Luigi Bisbiglia
UK, South & West Europe
luigibisbiglia@sbdautomotive.com+44 1908 305102

Andrea Sroczynski
Germany, North & East Europe
andreasroczynski@sbdautomotive.com+49 211 9753153-1

India

China

Japan

SBD China Sales TeamChina
salesChina@sbdautomotive.com+86 18516653761

SBD Japan Sales TeamJapan, South Korea & Australia
postbox@sbdautomotive.com+81 52 253 6201
23