#635

In-Car HMI UX Evaluation& Benchmarking

UX
TABLE OF CONTENTS

EVALUATION

Executive Summary
Introduction to this ReportIn-car HMI UX
Evaluations

Features and Functionality
Execution
Perceived Quality

Jeep Grand Wagoneer

ADAS Domain

Infotainment Domain

In this edition, the UX team is testing the Jeep Grand Wagoneer.

Navigation Domain
Voice Recognition Domain

Overall, the Jeep Grand Wagoneer impresses with its comprehensive feature offerings including a highly
connected navigation system, entertainment and information features and numerous delight features
meaning it scores highly for SBD functionality score. However, the system is pulled down in terms of user
experience score due to a high number of critical or major bugs experienced during testing and concerns
regarding overall legibility.

Connected Features Domain
Convenience Domain
Support Areas

RELATED SBD REPORTS
619 – UX Benchmarking Series
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s UX Team evaluates the infotainment
user experience of over 40 vehicles.
806 – ADAS HMI Evaluations
A precursor to the new In-Car HMI UX
Evaluation & Benchmarking Series, SBD
Automotive’s Autonomous Car Team evaluates the
ADAS performance and usability of over 20
vehicles.

COVERAGE

FREQUENCY

PUBLICATION FORMAT

PAGES

6
NA

CHINA

EUROPEANNUALLYQUARTERLYCARS PER YEARPD F

POWERPOINT

EXCEL

ONLINE

175

#635/ In-Car HMI UX Evaluation & Benchmarking

Scoring> Features and functionality: > Reliability/stability:
evaluating whether theevaluating the repeatedsolutions provide featuresusability and whether thethat customers expect, needusers can have a similarand solve problems (or(positive) experience eachprovide a wow factor).
time.
> Usability:
evaluating whether the> Perceived quality:
features available are easy
evaluating the potential
to learn and use. This
perception in quality of theconsiders areas such as
HMI components and howergonomics, legibility,
this contributes to the overallusability characteristics andcustomer experience.
how the system implements
the various features.

This research is useful for

PRODUCT PLANNERS

M A R K E T I NG

C- S UITEDo I have access?

USER E XPE R IENCEE NGI NE E R S

Portal access

Request a quote for
In-Car HMI UX Evaluation & Benchmarking Series

Jeep Grand Wagoneer

March 2023

In-Car HMI UX Evaluation & Benchmarking
Jeep Grand Wagoneer

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – Jeep Grand Wagoneer
Executive Summary≫5

Features and functionality≫

22

Perceived quality≫▪ Overall Score▪ Overview▪ Introduction▪ SUS score by domain▪ Delight and performance features▪ Definitions▪ Kano analysis▪ Hygiene and navigation▪ Evaluation results▪ Positive and negative implementations▪ ADAS feature set▪ Highlights▪ Driver distraction▪ IoT integration, music, entertainment
and information▪ PQ Overview▪ CX recommendations

Execution≫▪ Potential improvements

Introduction to this report≫▪ Aim of report▪ Scope of report▪ 2023 vehicles▪ SBD experience▪ Methodology▪ Report structure▪ Overview

15

ADAS domain≫
28

63

71

▪ Summary▪ Failure modes▪ Misuse▪ System scoring▪ UX Heuristics▪ UX laws▪ SAE L0

▪ System performance▪ SAE L1&2

▪ Reliability▪ Ergonomics

Infotainment domain≫▪ Legibility and readability▪ Summary▪ UX heuristics▪ Key positive and negative96

Contents
635 - In-Car HMI UX Evaluation & Benchmarking – Jeep Grand Wagoneer
Navigation domain≫

117

Connected features domain≫▪ Summary▪ Summary▪ UX heuristics▪ Key positive and negatives

147▪ Navigation specific scoring

Convenience domain≫▪ Key positive and negatives

155▪ UX heuristics

Voice recognition domain≫▪ Summary▪ Functionality▪ Performance▪ Stability▪ SUS score▪ SBD UX principles▪ Command structure▪ Localization▪ Integration

136▪ HVAC summary▪ HUD summary▪ Rear seat entertainment summary▪ Massage seat summary▪ User profile summary▪ Relax mode summary

Support areas≫

166

Contact us≫

175▪ Amazon Alexa integration

Total number of pages - 175

Introduction

Introduction

Aim of this report

One of the core goals of these studies is to provide a true indication of what
the final customer experience of each solution could be. To do thisevaluations are focused on providing scoring and analysis in the following
areas:▪ Features and functionality: evaluating whether the solutions provide
features that customers expect & need, and solve problems (or provide a
wow factor)

Brand perception

Consumer testing

Welcome to the 2023 HMI benchmarking report series. This report has beencreated to provide a fair, unbiased and objective view of the latest in-vehicle
HMI solutions in the US, European, and Japanese markets. Evaluations arecarried out by SBD usability experts with a deep understanding of CASE
domains such as the Connected Car and ADAS & autonomy domains.

▪ Usability: evaluating whether the features available are easy to learn anduse. This considers areas such as ergonomics, legibility, usability
characteristics and how the system implements the various features▪ Perceived quality: evaluating the potential perception in quality of the
HMI components and how this contributes to the overall customer
experience
SBD supports clients throughout the development of new HMI and products
from a relatively simple companion app to a more complex multi-domain
infotainment solution. The methodologies used in these reports take into
account many years of experience with consumer testing and custom client
projects to provide a fair and, as much as possible, objective methodology.
All viewpoints and analysis within the report are aimed defining areas ofconcern through a data driven approach. This report aims to benchmark andscore solutions whilst also being able to provide actionable recommendations
to design and development teams.

Please note that due to the ever-evolving automotive technology market,
SBD updates it’s methodology each year, but does not update scores from
7 previous years. Therefore, please assume a slight drop in scores for boththeuser experience (UX) and functionality from the previous year.

BRAND
APPEAL

Aesthetics and engagement“Do I have a positive emotional attachment to thesolution?”

EMOTIONAL
APPEAL

Repeated usability
RELIABILITY

Expert testing(the focus of this report)▪ Reliability/stability: evaluating the repeated usability and whether theusers can have a similar (positive) experience each time

“Do I have a positive emotional attachment to thebrand?”“Can I have a similar experience each time?”

Ergonomics, usability characteristics &execution“Are the features available easy to learn
and use?”USABILITY

Domain functionality &prioritisation“Does the system provide features I
want and need to use?” i.e. “Does it
solve problems for me?”FUNCTIONALITY

SBD’s view on the hierarchy of needs for CX benchmarking
7

Introduction

Scope of report: focus on in-car HMI evaluations
The scope of evaluations in this report are constrained to the in-car HMI
experience, in both static and dynamic conditions. One notable element isdriver distraction which SBD covers at only a high level in this study as
carrying out a full driver distraction evaluation requires biometrics test
equipment to ensure the collected data is unbiased and objective.

Prepurchase

Purchase

Activation

In-car HMI

Driverdistraction

IoT

A full evaluation of the end-to-end customer experience is not within scope of
this report, but it is something which SBD has many years experience in fromboth a consumer and expert perspective. Other areas such as the companion
app, online portal and in-home smart devices are not in scope as they aredefined as “out of car” experiences.
Within the vehicle, any HMI element the user interacts with is evaluatedincluding steering wheel controls, touch screen displays, voice control, HUDs
and digital keys. The features and services on offer have been broadlygrouped into the following domains (or test areas):▪ ADAS domain▪ Infotainment domain▪ Navigation domain▪ Voice recognition domain▪ Connected services domain▪ 8Convenience domain
8Introduction

2023 vehicle list
SBD has chosen six cars to evaluate in 2023, based on two selection categories. New/interesting UX focuses on systems with to never-seen-before
features or functionality, or the implementation of a solution that has previously been a challenge or pain-point for end-users. New mass-market UX
includes vehicles in segments that are sold in high numbers and are entering a new generation of UI for that vehicle. While we make best efforts toadhere to the chosen cars and schedule, the last two years have seen release dates slipping significantly, so it may be necessary to makesubstitutions.

Cars tested

Jeep Grand
Wagoneer

Planned cars for 2023

Nissan Ariya▪ Passenger display▪ ProPilot Assist▪ Amazon Fire TV▪ Unique HMI andinterior▪ Active Drive Assist▪ US market test

▪ EV SUV▪ German Market Test

BYD Seal

BMW X1▪ Unique Home Screen
HMI▪ Expected iDrive 9
debut

▪ EV

▪ German market test

▪ Chinese market test

Fisker Ocean

Ford F-150
Lightning▪ EV SUV▪ Android-based OS▪ Sustainable materials▪ EV Truck▪ Large central display▪ US market test

▪ US market test99Introduction

SBD experience through years of testing in-car solutions
Over the last ten years SBD has evaluated over 100 solutions from a Connected Car or ADAS perspective for our public report series (many more for privateclient evaluations). This current report series is an evolution of both test methodologies to provide a holistic view of in-car HMI. Furthermore, custom
evaluations methodologies used across the globe for SBD clients have been included where applicable to enhance to overall approach.

1010Introduction

One page methodology overview
One of SBD’s core goals of this report is to be as objective, fair
and as transparent as possible. To achieve this, various
methodologies are used throughout the testing to evaluate
different areas of the solution in various conditions.
These methodologies are a mix of different types of tests:▪ Objective tests: where the value provided is not influenced
by a tester’s viewpoint e.g. response time
▪ Subjective tests: the test score is based on the expert
testers’ viewpoints e.g. task ease of use▪ Task-based: evaluations carried out based on a predefinedtask list e.g. navigate to a pizza restaurant near location X▪ Freeform: random free testing by the tester with no clear
pre-defined task list. This allows the testers flexibility to dig
deeper into various parts of a solution when needed▪ Scoring range: ranges and definitions of how to score atest element e.g. poor depth and accuracy score = theresults provided are not in line with what is reasonably
expected by the user

Type of tests
Test area

Objective

First impressions

Subjective
P

P

P

P

P

Dynamic tasks

P

P

P

P

Random freeP

P

Navigation specific tests

P

P

P

Voice recognition

P

P

P

Performance & response

P

System Usability Scale
(SUS)

P

P

UX heuristics

P

P

Legibility & readability

P

Perceived Quality (PQ)

P

P

P
P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

P

Ergonomics

P

Misuse/failures

P

P

ADAS

Dynamic

P

P▪ Dynamic: tests are carried out when the vehicle is moving
in various road conditions and locations e.g.
motorways/highways, cities, villages, country roads etc.

StaticP

P
P

ExecutionPPP

For a detailed discussion and presentation of SBD’s
11
methodology
please contact us.

Scoringrange

PFinal SBD UX score

This document does not provide a detailed description of themethodology and this page serves to provide an overview of
the approach.

Freeform

Static tasks▪ Static: tests are carried out when the vehicle is not moving▪ Misuse/failures: carried out to evaluate the stability of thesolution in unusual conditions e.g. repeatedly pressing thevoice command button

Task based

P

P

P

P

11

Example slides fromthe reportFeatures & Functionality

Delight and Performance features

Many delight features, minor performance feature issues•

The system features a large selection of delight features that are expected to wow
the majority of users.

•

Voice recognition can be inconsistent in understanding and interpreting
commands correctly.

•

Minor delight features include a McIntosh sound system, rear seat entertainment
with Amazon Fire TV, rear touch screen HVAC control, massage seats, drop-down
step and Fam Cam (interior passenger camera) from which front passengers can
individually select rear seats to monitor.

•

Visual inconsistency between cluster and HUD, especially regarding navigationcommands.

•

Some ADAS such as ACC lacks sufficient user support in stop and go traffic
situations.

Passenger display is a major wow-factor

Voice recognition struggles with some commands

One of the major delight features is a passenger display which is visible to thepassenger but cannot be viewed by the driver to minimize distraction. Passengers canwatch Amazon Fire TV, plan and send routes to the navigation system in the central
display and connect external devices such as mobile devices, headphones and otherdevices via HDMI.

The voice recognition system fails some simple use cases. For example, the system fails
to successfully complete some media-based commands such as ‘pause’, ‘shuffle’ and‘repeat all tracks’. This is likely to fall short of most users' expectations for a premium
vehicle. Most navigation-based performance tasks are also not possible, often being
misunderstood by the system or completely rejected.

13
13

Features & Functionality

Hygiene & Navigation

Inconsistent audio prioritization for some systems, POIs lack detail•

Audio prioritization for ADAS is very good, however it is insufficient for navigation
which struggles to prioritize voice commands over infotainment volume.

•

The navigation system is mostly well implemented with accurate ETAs from start tofinish and efficient routing.

•Phone SMS messaging is often unstable, working for a certain period before
becoming unavailable.

•

Verbal commands are delivered in a timely manner and include important
information such as street name and direction.

•

Volume control from the steering wheel is not possible. This is something that mostusers would likely notice is missing. Source selection is only possible when the
media screen is selected in the cluster.

Audio prioritization for some features, but no others

POI icons can sometimes be inaccurate and illogicalSome systems including ADAS have complete audio prioritization. When an ADAS audio
alert is given, all infotainment volume is completely muted to allow for full prioritization.
However, for navigation some level of prioritization is given but when infotainment
volume is loud, voice commands still struggle to be heard. Ideally, audio prioritization
should be consistent across all elements of the system.

When navigating to the airport, the system showed a POI icon that clearly depicted an‘aircraft’. However, on arrival the POI icon now used a gate image. This icon meant‘gated access’ which was clear inaccurate. When navigating past an airport, a ‘Shellfuel’ icon is shown nearby on the map, when in reality there is no fuel station nearby
accessible to the public.

14
14

Features & Functionality

ADAS feature set

Good status communication, some poor graphical implementation
SBD tested six ADAS on the 2022 Jeep Grand Wagoneer. These were Fully Automatic Park Assist (FAPA), Rear Cross Traffic Alert (RCTA), Lane Departure Warning/Lane Keeping Assist(LKA/LKA), Blind Spot Monitoring (BSM), Adaptive Cruise Control (ACC) and Piloted Driving (PD).
Overall, most systems provided good visual feedback in the cluster and excellent audio prioritization. However some systems were let down by basic visual warning in the side mirrors andcentral display while others lacked the required level of user support. Use of correct and clear iconography was also lacking throughout most systems. Multiple ADAS views are available inthe cluster and all maintain basic system status and warning displays.

The system does not
automatically scan at
low speeds, The user
must press the
button to begin
scanning. Step-bystep instruction isclear, however FAPA
graphics fall well
below the expected
standard.

RCTA activation is
automatic when reversegear is engaged.
However, the system isbundled with BSM andcannot be turned ON or
OFF independently. The
audio warning isprioritized but the visual
warning falls short of the
expected level.

LKA provides lane tracking
status clearly and iscommunicated via themain ADAS display, HUD
and repeater icon toensure it is always shown.
The correct color logic isgiven for warnings;
however, some confusing
button labelling prevents itfrom being as intuitive as
it could be.

BSM operates mostly as
expected, withprioritized audio
warnings whenindicating into the pathof detected vehicles.
The system could be
improved by utilizing the
ISO icon in the side
mirrors.

ACC is controlled via
buttons on the steering
wheel. A good level ofvisual feedback is givenin the cluster and HUD
using correct color logic.
However, the system is
let down by a poor levelof support in stop & gotraffic.

Overall PD provides goodvisual support in thecluster and HUD withclear graphics. Status isclearly communicated at
all times. Hands-off
warning is clear androbust, however it fails
to cancel after threeattempts of warning.

15
15

ExecutionⅼReliability issues

Bluetooth pairing issues

1. Phone

On multiple occasions, the user was unable to pair amobile device to the system via Bluetooth. Even after
following the correct on-screen instructions, the mobile
device displayed a message indicating the system could
not pair successfully. The user was forced to restart theBluetooth pairing process by turning off and on
Bluetooth on the mobile device.

Frequency

Low

MediumHigh

Severity

Minor

Major

Critical16
16

ExecutionⅼLegibility & Readability

Display reading distance & text size

Mixed results across HMI
Instrument cluster text size is too small
In the central display, text such as that for temperature readout is 3.8mm inheight. This is above the target size of 3.5mm and scores “Low/no risk”.
In the instrument cluster, text for speed sign recognition is 2.3mm in size. This
falls below the target size of 4.1mm but still scores an acceptability rating of‘Acceptable’ as the icon is distinctive enough to be recognisable as a speed limit
without always having to read the text. As a result it scores “Emerging risk”.

Central display text is well above target size

Display

Reading
distance

Text size

Target size

Final Rating

Central display

69cm

3.8mm

3.5mm

Low/no risk

Instrument cluster

71cm

2.3mm

4.1mm

Emerging risk

Severity / Driver distraction
RatingAcceptability Rating

Central display

Low/no risk

Acceptable

Instrument cluster

Significant risk

Acceptable

Display17
17Perceived quality

Tactile highlights

Perceived Quality: Tactile
Level 1

TactileStiffness & looseness: All fixtures, fittings and surfaces demonstrate
a good level of rigidity and solidity including regular touchpoints andsurfaces lower down within the cabin not regularly felt by users.
Force feedback: Good quality feedback is felt from most buttons andknobs, however the level of haptic feedback is inconsistent across the
HMI. Lower console buttons provide subtle haptic feedback (LKA,
FAPA etc). The memory seat buttons on the door card have harshhaptic feedback that vibrates the entire panel. No feedback isprovided for seat heaters/ventilation buttons. As a result misoperation
is frequent.

SBD viewpoint

Material quality: Most surfaces and materials give the expected
sensation of high quality including real wood paneling with inlaid
branding and a real metal gear selector. High gloss, black plastic isused in the cabin and feels cheap and often results in misoperation
due to fingerprint build up.

Harsh force
feedback for door
mounted seat
controls causes
vibration noises.

Solid indents forvolumeadjustment

Material harmony: Material quality remains high throughout thecabin, low down on the door cards and rear seat rows. Turn signal
stalks feel cheap and appear to be from a lower segment vehicle,
however this is only a minor point. Hard, gloss black plastic aroundthe central display feels cheap, collects fingerprints and dust easily
and is inconsistent throughout the cabin.
Geometric & Positioning: All switches and buttons look and feelsolid. This is fitting for the cars aesthetic and overall brand appeal.

Level 2 scoring
Stiffness &looseness

Force feedback

Material quality

Material
harmony

Geometric &positioning

Good

Fair

Fair

Good

Good

High gloss black
plastic feels cheap
and is inconsistentthroughout cabin.

18
18

Connected features

Positive points

Amazon services offering

The system features multiple services provided by Amazon. These
include Alexa voice assistant and Amazon Fire TV. These services canbe accessed with an Amazon account and when connected to a Wi-Fi
hotspot connection.
• When enabled and connected to a Wi-Fi source, Alexa can beinteracted with by either using a wake-word or by pressing thesteering wheel button. Alexa can complete more integration use
cases compared to the native system when enabled.
SBD
viewpoint• Amazon Fire TV can be accessed via the central display, passenger
display and two rear seat entertainment displays. The rear seatentertainment displays can be interacted with using touch or via
dedicated remote control.
Amazon Alexa provides an effective alternative to the native voicerecognition system. Alexa can successfully complete some use casesthat the native system fails, however the user must go through theprocess of enabling Alexa, connecting to the hotspot and signing into
their account to use it. Amazon Fire TV provides an entertainment
features that is likely to be popular with most users and provide adegree of wow factor.

UX impact
19

Major negative

Minor negative

Minor positive

Amazon services

Major positive19

Request price forthe full reportContact Us

Contact SBD Automotive

Do you have any questions?
If you have any questions or feedback about thisresearch report or SBD Automotive’s consulting services,
you can email us at info@sbdautomotive.com or discusswith your local account manager below.

info@sbdautomotive.com

Book a meeting

USA

Garren Carr
North America
garrencarr@sbdautomotive.com+1 734 619 7969

21

UK

Germany

Luigi Bisbiglia
UK, South & West Europe
luigibisbiglia@sbdautomotive.com+44 1908 305102

Andrea Sroczynski
Germany, North & East Europe
andreasroczynski@sbdautomotive.com+49 211 9753153-1

India

China

Japan

SBD China Sales TeamChina
salesChina@sbdautomotive.com+86 18516653761

SBD Japan Sales TeamJapan, South Korea & Australia
postbox@sbdautomotive.com+81 52 253 6201
21