Measuring Voice UX Qantitatively: A Rapid Review
Measuring Voice UX Qantitatively
Katie Seaborn

Tokyo Institute of Technology
seaborn.k.aa@m.titech.ac.jp

ABSTRACT
Computer voice is experiencing a renaissance through the growing
popularity of voice-based interfaces, agents, and environments. Yet,
how to measure the user experience (UX) of voice-based systems
remains an open and urgent question, especially given that theirform factors and interaction styles tend to be non-visual, intangible,
and often considered disembodied or “body-less.” As a frst step, we
surveyed the ACM and IEEE literatures to determine which quantitative measures and measurements have been deemed importantfor voice UX. Our fndings show that there is little consensus, evenwith similar situations and systems, as well as an overreliance on lab
work and unvalidated scales. In response, we ofer two high-leveldescriptive frameworks for guiding future research, developing
standardized instruments, and informing ongoing review work.
Our work highlights the current strengths and weaknesses of voice
UX research and charts a path towards measuring voice UX in amore comprehensive way.

CCS CONCEPTS• Human-centered computing → Human computer interaction(HCI); Human computer interaction (HCI); Interaction devices;
Sound-based input / output.

KEYWORDS
Voice user interfaces, voice agents, voice assistants, voice user experience
ACM Reference Format:
Katie Seaborn and Jacqueline Urakami. 2021. Measuring Voice UX Quantitatively: A Rapid Review: Measuring Voice UX Quantitatively. In CHI
Conference on Human Factors in Computing Systems Extended Abstracts (CHI
’21 Extended Abstracts), May 08–13, 2021, Yokohama, Japan. ACM, New York,NY, USA, 8 pages. https://doi.org/10.1145/3411763.3451712

1

INTRODUCTION

Voice-based interfaces, agents, environments, and displays use natural and intuitive forms of interaction inspired by human-humancommunication. These technologies are not new, with several
decades of history to be found in the feld of human-computer
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proft or commercial advantage and that copies bear this notice and the full citationon the frst page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
CHI ’21 Extended Abstracts, May 08–13, 2021, Yokohama, Japan© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-8095-9/21/05. . . $15.00
https://doi.org/10.1145/3411763.3451712

Jacqueline Urakami

Tokyo Institute of Technology
urakami.j.aa@m.titech.ac.jp
interaction (HCI) and related disciplines [40, 48]. Yet, modern advances in consumer devices and infrastructure, artifcial intelligence(AI) and speech recognition, and natural language synthesizers
have driven the widespread uptake of such systems in daily life.
Smartphone-based voice assistants (VAs) like Apple’s Siri and homesmart speaker systems like Amazon’s Echo with voice assistantAlexa are well-known consumer examples. Cutting-edge technologies like Google Duplex [34] are pushing the current standards toan extreme, where it is almost impossible to distinguish a computer
voice from that of a real person. This recent and rapid progresshas triggered a renewed interest in research on “the voice of themachine” [9, 45]. In the CHI conference alone, there was a 2.46-foldincrease in voice-related publications between 2015 and 2020.
The technical quality of voice recognition and voice expression
is just one aspect of voice-based interaction. Decades of work on
interactive voice response (IVR) systems, computer voice, conversational agents, and social robots has shown that user perceptionof voice plays a signifcant role [42]. People are afected by theirpre-existing attitudes and behaviors towards computers but also
tend to react to humanlike features of computer voice as they would
react to people, without realizing it [31, 40, 42]. Moreover, there has
been a growing recognition that social and psychological factors,
including afect and emotion, trust, credibility, rapport, and therelational context, need to be considered [17, 36]. This indicates ashift in focus from functionality and usability to user experience(UX).
The question of how to measure voice UX has subsequently
been raised. A round table at CHI 2018 highlighted the need forunderstanding current practice as well as what approaches andmethods should be used [22]. Clark et al. [9] reviewed the stateof the art on speech user interfaces, covering research design, approaches to evaluation, and more. They noted that measurement
work on speech interface UX is limited, in terms of where and withwhom as well as with respect to validity and reliability. Seaborn
et al. [45] noted similar issues for computer-based agents that use
voice, calling for unifcation and standardization of quantitative
measures. Kocaballi et al. [26] surveyed the conversational userinterface (CUI) literature on measuring UX. They found a lack ofconsensus on what CUI UX is and how it is measured, leadingthem to develop a high-level unifying framework. Brüggemeier et
al. [6] recognized a gap in measuring the UX of commercial voiceassistants and then evaluated the validity of several instruments
for Amazon Alexa. Nowacki et al. [43] proposed a set of heuristics
for voice usability and UX, updating previous guidelines in light ofchanging technologies and conceptualizations of experience. Taken
together, this body of work highlights raising interest in voice UX,
as well as challenges that need to be addressed.

CHI ’21 Extended Abstracts, May 08–13, 2021, Yokohama, Japan

In continuation of this line of work, and with a view to answering the call made by the CHI 2018 panel, we pinpointed a gap at
the intersections: measurement of voice UX. Kocaballi et al. [26]reviewed CUIs, which are not necessarily voice-based: other modalities, notably text, are used. Brüggemeier et al. [6] narrowly focusedon voice assistants, excluding other “bodyless” voice-based possibilities. Nowacki et al. [43] focused on heuristics and guidelines
rather than measurement. Clark et al. [9] and Seaborn et al. [45]each provided high-level reviews on approaches to the design andevaluation of speech interfaces—which may not have voice output—
and computer agent voice—which may not have voice input. Here,we complement and extend this body of work by focusing on voice
explicitly and comprehensively, especially audible and nonvisual,
intangible form factors.
Underlying this work is also a recognition that UX itself is noteasily defned. Indeed, lack of defnitional and operational consensus has been a running problem [1, 18, 26, 28, 29]. Subsequently,
despite the infux of voice UX research and a growing interest inmeasurement, there is yet no systematic overview of what is being
done or how to do it. But such a review is necessary to achieveconsensus and advance the feld, as well as to develop reliable andvalid measures, improve the quality of research and practice, andincrease the comparability of studies.
In this paper, we provide an urgently needed state of the art
to direct and support current and ongoing research. Using a gold
standard rapid review process, we present a comprehensive description of how voice UX is being conceptualized and measured
in HCI. We cover a variety of technologies that rely on voice as
the primary or sole means of interaction, ranging from agents tointerfaces to environments and displays. As yet, there does notseem to be consensus on a catch-all term for such systems. Here,
we use voice UX, defning it as verbal and auditory interaction witha computer-based system that uses voice-only input, voice-only output, and/or voice input-output. As this is a late-breaking work, we
restrict our focus to quantitative measures, which are common in
HCI [50] and allow for ease of aggregation and comparison. We
also restricted our databases to ACM and IEEE, major publishers
of HCI work. We asked: How is voice UX being measured quantitively? We focused on three key sub-questions related to research
design. These were: RQ1. What factors are being experimentallymanipulated as independent variables (IVs)? RQ2. What measures arebeing used to evaluate dependent variables (DVs?) And RQ3. Whatis the relationship between the IVs and DVs? In this work, we are
pre-emptively responding to the raising wave of studies on voice
UX and increasing calls for an answer to the question of how tomeasure it. Our main contributions are threefold: (i) a descriptionof the state of the art on quantitative voice UX measurement, (ii)
an indication of emerging consensus and disagreement, as well as
gaps and opportunities; and (iii) two frameworks for categorizing
IVs and DVs in voice UX research that may be used to guide future
research. This work acts a frst step towards more comprehensive
survey work that includes qualitative voice UX. We expect thiswork to help researchers in deciding which measures to choose andprovoke initiatives on merging similar measures or developing newones based on consensus.

Katie Seaborn and Jacqueline Urakami

2

METHODS

We conducted a rapid review of the literature following the
Cochrane gold standard protocol [15]. Rapid reviews are used togenerate a quick yet systematic descriptive and narrative summaryof a focused area of study [12, 14, 23]. Importantly, they are conducted when a timely assessment is needed. As per the Cochrane
protocol, we used the PRISMA checklist [35], adapted for HCI
work1 , to structure or process and reporting. Our fow diagram isin the Supplementary Materials. Two researchers performed thereview. The frst author conducted the initial database searches anddata extractions. Both authors then categorized the data, analyzed
the data, and wrote up the results. The protocol for this study was
registered (OSF # 423pq) before data extraction on December 10th ,20202 .

2.1

Eligibility Criteria

We included full user studies involving at least one form of voiceonly or primarily voice-based user interface, agent, display, environment, or some other interactive system. This included voiceassistants, voice user interfaces, interactive voice response systems,
conversational user interfaces, smart vehicles, and so on. We included studies that used at least one quantitative measure. Pilot
studies, proposals, protocols, technical reports, literature reviews
and surveys, and grey literature were not included. Papers werealso excluded if there were not enough details to understand themeasure and/or measurement. Only English papers were included.

2.2

Information Sources, Search Queries, andStudy Selection

The ACM Digital Library and IEEE Xplore databases were queriedon November 12th, 2020. All queries included the following keywords: “voice user interface,” “voice assistant,” “smart speaker,” “interactive voice response.” Given the amount of technical work in
the IEEE database, index terms (“Publication Topics”) were used in
an attempt to flter results: “human computer interaction,” “userexperience,” “human factors.” The * qualifer was used to accountfor pluralization and diferent grammatical forms for each keyword.
Manual additions from the paper citations and previous survey
work were added. Papers were then screened independently by two
raters for inclusion based on the eligibility criteria, each taking arandom half of the set. A fnal 29 papers were included.

2.3

Data Collection, Items, and Analysis

Two researchers decided on the items for extraction based on the
research question. They included the following, as stated in the
pre-extraction protocol registration: description of studies (study
type, research design, setting, agent type, agent device, participant
demographics); IVs; DVs (measurements, subjective/objective, response format, validation). Each independently extracted metadatafrom half of the papers. The frst author then generated descriptive
statistics. Then, they conducted two inductive thematic analyses[5, 16] to create categorical frameworks for each of the IV and DV
data. First, they independently developed basic frameworks, then1 Some language and items related to the medical feld (e.g., structured summaries)
were modifed or excluded.
2 https://osf.io/423pq

Measuring Voice UX Qantitatively: A Rapid Review

CHI ’21 Extended Abstracts, May 08–13, 2021, Yokohama, Japan

Table 1: IV framework with context of study and examples.
Category and Description

Subcategory

Setting

ExampleSources

Form Factor
Manipulation of the form, style, andtype of voice system

None

Lab,
Field

Comparison of diferent devices; voice-only(disembodied systems) vs. embodied systems (e.g.,
robots)[37]; [51];[33]; [13];[10]; [46]

Lab
Lab

[10]; [30][49]; [11]

Lab,
Field

Masculine vs. feminine voices
Prestigious vs. regional accents
American vs. Swedish accents
Emotional expressions in graphical displays;emotionality of voice (energetic vs. subdued)
Personality type (e.g., extrovert vs. introvert);
diferent characters (e.g., friendly vs. hostile)

Lab
Lab
Lab

Male and female participantsComparing extrovert/introvert participants
Driver emotion (happy/upset)

[30]
[32]
[41]

Lab

Verbal abuse style; response style; information style(e.g., low warning vs. high warning with details);
conversational fllers; conversation style matching
Environment (e.g., living room vs. kitchen, vs.
home ofce); baseline task; level of privacy;
information content
Simple commands; advanced instructions[8]; [38];[21]; [19][39]; [20];[53]; [47],[25][39]; [20]Voice Characteristics
Manipulating sound as well as
Genderpsychological characteristics of the voiceAccent
outputEmotions
Personality
User Characteristics
Comparing diferent groups of users or Gendermanipulating afective state of the user Personality
EmotionConversation
Style
Manipulating the voice system’s
response style, environment of use, or
language style
Context

Interaction Modality
Manipulating how users can interact
with the voice system

Lab

Lab

NaturalLab

ModalityLab

Manipulating input modalities (e.g., voice vs. text
vs. query form)

Query FormLab

Words vs. gaze activation

combined them through discussion, and then applied them to all
29 papers. Two rounds of coding were conducted for inter-rater
reliability to be achieved, with Kappa values of .80+ per category.
Only categories that achieved consensus were included. The second author then decided on the fnal attributions of themes wherediscrepancies in ratings existed.

3

RESULTS

Descriptive results on the study information are presented. Then,a visual timeline featuring the categories from the frameworks
based on the thematic analyses of the IVs and DVs are presented.
The tables for these are provided in their own sections with patterns highlighted and described. Finally, a matrix of the IV and DV
categories is given.

3.1

Description of Studies

The surveyed papers reported on a variety of experiments and user
studies (31 total). 66% (21) used a between-subjects design, and 34%(11) used within. 77% (24) were lab-based, 3 were in the feld, and 4
were other (e.g., questionnaires). 59% (17) involved voice assistants,
with the rest involving conversational agents (5), computer voice (2),in-car assistants (2), and three others (website, chatbot, and speech[47]; [41][52]; [32];[4][33]; [38];[7]; [11][7]interface). Devices included smart speakers (38% or 11), computer
speakers (17% or 5), an unknown speaker (2), smartphones (4), andone tablet. 1807 participants were involved across all studies: 667
men, 497 women, and two of another gender; a t-test did not fnd
a signifcant diference between the number of men and women
included across studies, t = .82, p = .42.

3.2

IVs and IV Framework

The framework resulting from the thematic analysis of the IVs is in
Table 1. Relevant metadata and examples are provided along withdescriptions for each category. Typical manipulations were varying
the physical form of the voice system(s), vocal and social characteristics of the voice, conversation style, and interaction modality.
Table 1 shows that each major category applies to several (3+)
studies. Emotions and personality have received most attention for
psychological factors. Manipulation of conversation and interaction
modality were limited. While most are not unique to voice UX, one
category stands out: Query Form. This refers to the use of a “wakeword” or verbal query that triggers the start of an exchange withvoice-based systems, especially smart speakers and voice assistantson smartphones, like Siri and Alexa. It applies to “bodyless” andever-present voice systems, where there is not necessarily a visualor physical interface to begin or conduct interactions. The table also

CHI ’21 Extended Abstracts, May 08–13, 2021, Yokohama, Japan

Katie Seaborn and Jacqueline Urakami

Table 2: DV framework with measures/measurement grouped by type.
Category

Subcategories

Type

Measure and/or Measurement

Usability

None

Obj

System task performance [39]; Exercise Behavior [37]; Gaze [16]; Language production(e.g., lexical complexity, adaptation) [51]; Interaction time [27]; Driving performance[38]; Disclosure (user responses) [53]; Accidents [41]Continued use [33]; Satisfaction [33]; System efectiveness [24]; Tone clarity [8];Satisfaction [3]; System Usability Scale (SUS) [6, 44]; Speech User Interface Service
Quality Questionnaire (SUISQ-R) [6]; Document feedback [38]; Voice understandability[10]; Perception of rapport [46]; Group decision performance [46]; Voice performance[11]; MeCue questionnaire [4]Interaction behavior (e.g., frequency and time) [37]; Emotional engagement (facial
expressions) [47]Cognitive engagement [47]; Time spent talking to virtual passenger [41]; Stimulation(UEQ modules) [4]Attentional allocation [27]; Situational awareness [38]; Driver attention [41]Perceived attention [41]None
Mental workload [51]; User burden [13]; Driving activity load index [4]Facial expressions (Afdex Software Development Kit – Afdex SDK) [4]Negative impact [13]; User beneft [13]; Moral emotions (e.g., guilt) [8]; Emotion
manipulation check [41]Interaction behavior (e.g. gaze frequency, conversation turns) [27]Social presence [32, 33]; Intimacy [25, 33]; Closeness (Subjective closeness index) [53];Perceived power [46]; Conformity in decision-making [30]; Helpfulness [25];Self-validation [25]Trust (investment game) [49]
Trust [7, 25, 33]; Trustworthiness [30, 46]; Safety [38]None
Likeability [4, 7, 8, 21, 24]; Voice attractiveness [30]; Voice liking [11]; Information
liking [11]; Attractiveness (UEQ modules) [4]; Desirability (Marlowe-Crowne Social
Desirability Scale - MCSDS) [10]NonePreference (self-reported) [33, 38, 46]; Smile-o-meter [52]None
Acceptance (self-reported) [7]; Acceptance Scale [4]; Familiarity [11]None
Wiggins’s interpersonal adjective scales [32]; Big Five Inventory [4, 19]
None
Anthropomorphism [8, 46]; Human-likeness [21]; Voice gender [30]None
Perceived Intelligence [8, 21, 46]
None
Human-likeness [13]; Attractiveness diferences [6]; Self-efciency [7]; Interaction
questionnaire [19]; Godspeed questionnaire [19]Subj

Engagement

None

Obj
Subj

Cognition

Attention
Workload

Obj
Subj
Obj
Subj
Obj
Subj

Afect

None

Sociality

None

Obj
Subj

Attitudes

Trust

Obj
Subj
Obj
Subj

Likeability

Preference
Acceptance
Perception of Voice Personality
Anthropomorphism
Intelligence
Aggregate UX

None

Obj
Subj
Obj
Subj
Obj
Subj
Obj
Subj
Obj
Subj
Obj
Subj

illustrates how most categories have relied on lab-based studies.
Only Form Factor and Voice Characteristics involved feld work.

3.3

DVs and DV Framework

A variety of DVs were identifed. Many studies used Usability measures, a traditional approach to measuring UX. Measurements of
Cognition, such as Workload and Attention, also have a long tradition in ergonomics. Aside from these, we also found measurementsof Afect, Sociality, and Engagement, signaling a shift to social

factors. We also found instruments that measured several diferent
aspects of experience together, which we call Aggregate UX.
Most measurements were self-report scales that used a Likert
scale response format (49), although these varied from 5-point (35%or 17) to 7-point (39% or 19) to 9-point (3) to 10-point (20% or 10). 49%(43) were created by the researchers, 44% (39) were existing scales,and seven were modifcations of these. 48% (42) were validated,while 37% were not, and 12 could not be determined. 14 papers
reported validation; most (12) used Cronbach’s alpha (M = .84, SDMeasuring Voice UX Qantitatively: A Rapid Review

CHI ’21 Extended Abstracts, May 08–13, 2021, Yokohama, Japan

Table 3: Matrix of the categories from the IV and DV frameworks. Circles indicate an intersection. Dots indicate no intersection.

Aggregate UX

Perceptionof Voice
Likeability

Preference

Acceptance

Personality

Intelligence

Anthropomorphism

⃝
⃝
•
•
⃝
⃝
⃝
•
•
⃝
•
⃝
•

Attitudes

•
•
•
⃝
⃝
•
•
⃝
⃝
•
•
⃝
•

Trust

⃝
•
•
•
⃝
•
•
•
•
•
•
•
•

Sociality

⃝
•
•
⃝
•
•
•
⃝
⃝
•
•
•
•

Afect
Workload

⃝
•
•
⃝
⃝
•
•
⃝
•
⃝
•
⃝
•

Cognition

Engagement

⃝
⃝
•
⃝
⃝
•
•
⃝
⃝
⃝
•
⃝
⃝

Attention

Usability

Dependent Variables (with subcategories)

⃝
⃝
⃝
•
⃝
⃝
•
•
⃝
⃝
⃝
⃝
•

⃝
⃝
⃝
•
⃝
⃝
•
•
⃝
⃝
⃝
•
•

⃝
•
•
•
⃝
⃝
•
•
⃝
⃝
•
•
•

•
•
•
•
⃝
•
•
•
•
•
⃝
•
•

•
•
•
•
⃝
•
⃝
•
⃝
•
•
•
•

⃝
•
•
•
•
•
•
•
⃝
•
•
•
•

⃝
⃝
•
•
•
⃝
•
•
⃝
•
•
•
•

Independent Variables
Form Factor
Voice
Characteristics
User
Characteristics
Conversation
Interaction
Modality

None
GenderAccentEmotion
Personality
Gender
Personality
EmotionStyle
Context
NaturalModality
Query Form

= .09), one used test-retest reliability (M = .85, SD = .09), and oneused Spearman’s rho (.78).
Table 2 presents the framework resulting from the thematic
analysis of the DVs. A large number of measures and measurements fall under similar categories, notably usability (21). There are
more subjective measures than objective (50 vs. 16); one category(i.e.., Aggregate UX) and several subcategories (e.g., Attitudes >Likeability, Cognition > Workload, Attitudes > Acceptance, etc.)
have no objective measures. While most measures (as labelled)
and measurements (as described) appear to have been stand-alone
or unique, some were apparently used across several studies, including trust/trustworthiness (5 studies), likeability (5 studies), andpreference (4 studies).

3.4

IV x DV Matrix

A matrix table was generated to compare, contrast, and contextualize the DV and IV frameworks against each other; see Table 3.
The matrix shows that the DV categories of Usability, Engagement,
Sociality, Trust, and Likeability are well-represented across the IV
categories. Some gaps and intersections are notable. Cognition was
not used for Conversation or Interaction Modality, even though
it is generally considered relevant to these factors. Perceptions
of Voice Intelligence was not manipulated or considered a factor
in Conversation. Acceptance is a predictor for long-term use andshould be examined. Aggregate UX, in contrast to Usability, hasonly been considered with respect to Form Factor and Conversation.

⃝
•
•
•
•
•
•
•
⃝
⃝
⃝
•
•

However, Personality DVs and IVs overlap, indicating research
design congruence.
Figure 1 presents two timelines for the IV and DV frameworks, showing how interest in categories have waxed and waned
over time. Both indicate a peripatetic exploration of factors from
2000-2010, with a drop-of until a rapid increase starting in2017. Figure 1a shows how Voice Characteristics was focusedon earlier but has lost focus since 2010. Figure 1b shows a renewed interest in Usability and Attitudes, and a new interest in
Aggregate UX.

4DISCUSSION

Our rapid, systematic review of the voice UX literature on
quantitative measures revealed a lack of consensus as well as
several strengths and weaknesses. Many studies focused on theefects of “form factor” using usability measures. This represents
the large variety of existing agent types and devices as well as asolid foundation on well-established usability work. An emerging
focus on measuring psychological and social aspects shows thatbasic usability criteria might be satisfed by existing systems.
Indeed, voice interaction is more than simple voice recognition;
the feld is starting to consider the context of use and importanceof afective factors, trust, and sociality, as refected in the recent
panel [22] and related survey work [26]. Yet, there is a strong
dependence on subjective measures based on self-reports. We
need to develop objective measures (e.g., behavioral measures)

CHI ’21 Extended Abstracts, May 08–13, 2021, Yokohama, Japan

Katie Seaborn and Jacqueline Urakami

Figure 1: Timelines for (a) the IV framework and (b) the DV framework.
to support these fndings and validate subjective measurements.
Additionally, while voice systems have a large range of applications,most studies were done in the lab. Relatedly, we did not fnd asingle longitudinal study in our data set. Yet, we know that user
expectation and attitudes can change with repeated use, and so
this needs to be addressed. Overall, there was an extensive—weargue overly extensive—variety of measures for our DV categories.
Our original goal was to present an overview of measures, but we
had to give up on this idea because of this lack of agreement how
to measure what. We now turn to what we can do about this stateof afairs.

4.1

Next Steps and Future Work

This work marks a frst step towards understanding the state of art
on measuring voice UX and crafting a path forward. We ofer foursuggestions on what needs to be done next to address the gaps andimprove the feld:• Start using the same measures and measurements for thesame DVs, in lab and in feld.
• Operationalize measures using theory about specifc concepts (e.g., is rapport about trust?) and then merge measuresthat are highly related or equivalent.
• Create and validate a standardized aggregate UX tool forvoice UX. Similar to how the Godspeed questionnaire forhuman-robot interaction was developed [2], use the IV and
DV frameworks as a guide based on consensus.
• Develop objective measures where there are gaps as well asones that complement subjective measures.
The next stage of this work will address its limitations by taking
a comprehensive approach, including general databases (such as

Scopus) and qualitative work. Future work will also summarize
voice UX fndings across studies.

5

CONCLUSION

We have systematically described the state of the art in quantitatively measuring voice UX. We have unearthed the strengths andweaknesses in this body of work, showing a lack of consensus. Weurge researchers to consider our roadmap for improving the qualityof voice UX research, especially in terms of standardization.

ACKNOWLEDGMENTS
We thank the reviewers for their positive and useful feedback.

REFERENCES[1] Javier A. Bargas-Avila and Kasper Hornbæk. 2011. Old wine in new bottles ornovel challenges: A critical analysis of empirical studies of user experience. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems,
2689-2698. https://doi.org/10.1145/1978942.1979336[2] Christoph Bartneck, Dana Kulić, Elizabeth Croft, and Susana Zoghbi. 2009. Measurement instruments for the anthropomorphism, animacy, likeability, perceived
intelligence, and perceived safety of robots. International Journal of Social Robotics
1, 1, 71-81. https://doi.org/10.1007/s12369-008-0001-3[3] Marta Bortoli, Marco Furini, Silvia Mirri, Manuela Montangero, and Catia Prandi.
2020. Conversational interfaces for a smart campus: A case study. In Proceedingsof the International Conference on Advanced Visual Interfaces (AVI ’20). https://doi.org/10.1145/3399715.3399914[4] Michael Braun, Anja Mainz, Ronee Chadowitz, Bastian Pfeging, and Florian Alt.
2019. At your service: Designing voice assistant personalities to improve automotive user interfaces. In Proceedings of the 2019 CHI Conference on Human Factors in
Computing Systems (CHI ’19), 40:1-40:11. https://doi.org/10.1145/3290605.3300270[5] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2, 77-101. https://doi.org/10.1191/
1478088706qp063oa[6] Birgit Brüggemeier, Michael Breiter, Miriam Kurz, and Johanna Schiwy. 2020.
User experience of Alexa when controlling music: Comparison of face and construct validity of four questionnaires. In Proceedings of the 2nd Conference on
Conversational User Interfaces (CUI ’20). https://doi.org/10.1145/3405755.3406122

Measuring Voice UX Qantitatively: A Rapid Review[7] Laura Burbach, Patrick Halbach, Nils Plettenberg, Johannes Nakayama, Martina Ziefe, and André Calero Valdez. 2019. “Hey, Siri”, “Ok, Google”, “Alexa”.
Acceptance-relevant factors of virtual voice-assistants. In 2019 IEEE International Professional Communication Conference (ProComm) (ProComm ’19), 101-111.
https://doi.org/10.1109/ProComm.2019.00025[8] Hyojin Chin, Lebogang Wame Molef, and Mun Yong Yi. 2020. Empathy is all you
need: How a conversational agent should respond to verbal abuse. In Proceedingsof the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20),
1-13. https://doi.org/10.1145/3313831.3376461[9] Leigh Clark, Philip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl,
Jens Edlund, Matthew Aylett, João Cabral, Cosmin Munteanu, Justin Edwards,
and Benjamin R Cowan. 2019. The state of speech in HCI: Trends, themes andchallenges. Interacting with Computers 31, 4, 349-371. https://doi.org/10.1093/iwc/
iwz016[10] Charles R. Crowell, Michael Villanoy, Matthias Scheutzz, and Paul Schermerhornz.
2009. Gendered voice and robot entities: Perceptions and reactions of male andfemale subjects. In Proceedings of the 2009 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS 2009), 3735-3741. https://doi.org/10.1109/
IROS.2009.5354204[11] Nils Dahlbäck, QianYing Wang, Cliford Nass, and Jenny Alwin. 2007. Similarity ismore important than expertise: Accent efects in speech interfaces. In Proceedingsof the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07),
1553-1556. https://doi.org/10.1145/1240624.1240859[12] Maureen Dobbins. 2017. Rapid Review Guidebook: Steps for Conducting a
Rapid Review. National Collaborating Centre for Methods and Tools, Hamilton,
ON. Retrieved from https://www.nccmt.ca/capacity-development/rapid-reviewguidebook[13] Philip R. Doyle, Justin Edwards, Odile Dumbleton, Leigh Clark, and Benjamin
R. Cowan. 2019. Mapping perceptions of humanness in intelligent personal
assistant interaction. In Proceedings of the 21st International Conference on HumanComputer Interaction with Mobile Devices and Services (MobileHCI ’19). https://doi.org/10.1145/3338286.3340116[14] Rebecca Ganann, Donna Ciliska, and Helen Thomas. 2010. Expediting systematic
reviews: Methods and implications of rapid reviews. Implementation Science 5, 1,
56. https://doi.org/10.1186/1748-5908-5-56[15] Chantelle Garritty, Gerald Gartlehner, Barbara Nussbaumer-Streit, Valerie J. King,
Candyce Hamel, Chris Kamel, Lisa Afengruber, and Adrienne Stevens. 2021.
Cochrane Rapid Reviews Methods Group ofers evidence-informed guidance
to conduct rapid reviews. Journal of Clinical Epidemiology 130, 13-22. https://doi.org/10.1016/j.jclinepi.2020.10.007[16] Greg Guest, Kathleen M. MacQueen, and Emily E. Namey. 2012. Applied Thematic
Analysis. Sage, Thousand Oaks, CA.
[17] Andrea L. Guzman and Seth C. Lewis. 2019. Artifcial intelligence and communication: A Human–Machine Communication research agenda. New Media &Society, 1461444819858691. https://doi.org/10.1177/1461444819858691[18] Marc Hassenzahl and Noam Tractinsky. 2006. User experience – a research
agenda. Behaviour & information technology 25, 2, 91-97.
[19] Rens Hoegen, Deepali Aneja, Daniel McDuf, and Mary Czerwinski. 2019. An
end-to-end conversational style matching agent. In Proceedings of the 19th ACM
International Conference on Intelligent Virtual Agents (IVA ’19), 111-118. https://doi.org/10.1145/3308532.3329473[20] Razan Jaber, Donald McMillan, Jordi Solsona Belenguer, and Barry Brown. 2019.
Patterns of gaze in speech agent interaction. In Proceedings of the 1st International
Conference on Conversational User Interfaces - CUI ’19 (the 1st International
Conference), 1-10. https://doi.org/10.1145/3342775.3342791[21] Yuin Jeong, Juho Lee, and Younah Kang. 2019. Exploring efects of conversationalfllers on user perception of conversational agents. In Extended Abstracts of the2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19), 1-6.
https://doi.org/10.1145/3290607.3312913[22] Joseph “Jofsh” Kaye, Joel Fischer, Jason Hong, Frank R. Bentley, Cosmin
Munteanu, Alexis Hiniker, Janice Y. Tsai, and Tawfq Ammari. 2018. Panel:
Voice assistants, UX design and research. In Extended Abstracts of the 2018
CHI Conference on Human Factors in Computing Systems (CHI EA ’18), 1-5.
https://doi.org/10.1145/3170427.3186323[23] Sara Khangura, Kristin Konnyu, Rob Cushman, Jeremy Grimshaw, and David
Moher. 2012. Evidence summaries: The evolution of a rapid review approach.
Systematic Reviews 1, 1, 10. https://doi.org/10.1186/2046-4053-1-10[24] Johannes Kiesel, Arefeh Bahrami, Benno Stein, Avishek Anand, and Matthias
Hagen. 2019. Clarifying false memories in voice-based search. In Proceedings ofthe 2019 Conference on Human Information Interaction and Retrieval (CHIIR ’19),
331-335. https://doi.org/10.1145/3295750.3298961[25] Songhyun Kim, Junseok Goh, and Soojin Jun. 2018. The use of voice input toinduce human communication with banking chatbots. In Companion of the 2018
ACM/IEEE International Conference on Human-Robot Interaction (HRI Companion’18), 151-152. https://doi.org/10.1145/3173386.3176970[26] Ahmet Baki Kocaballi, Liliana Laranjo, and Enrico Coiera. 2019. Understanding
and measuring user experience in conversational interfaces. Interacting with
Computers 31, 2, 192-207. https://doi.org/10.1093/iwc/iwz015

CHI ’21 Extended Abstracts, May 08–13, 2021, Yokohama, Japan[27] Dimosthenis Kontogiorgos, Andre Pereira, Olle Andersson, Marco Koivisto, Elena
Gonzalez Rabal, Ville Vartiainen, and Joakim Gustafson. 2019. The efects ofanthropomorphism and non-verbal social behaviour in virtual assistants. In
Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents(IVA ’19), 133-140. https://doi.org/10.1145/3308532.3329466[28] Carine Lallemand, Guillaume Gronier, and Vincent Koenig. 2015. User experience:
A concept without consensus? Exploring practitioners’ perspectives through aninternational survey. Computers in Human Behavior 43, 35-48. https://doi.org/10.
1016/j.chb.2014.10.048[29] Efe Lai-Chong Law, Virpi Roto, Marc Hassenzahl, Arnold P.O.S. Vermeeren,
and Joke Kort. 2009. Understanding, scoping and defning user experience: A
survey approach. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’09), 719-728. https://doi.org/10.1145/1518701.1518813[30] Eun Ju Lee, Cliford Nass, and Scott Brave. 2000. Can computer-generated speech
have gender?: An experimental test of gender stereotype. In Proceedings of the
CHI’00 Extended Abstracts on Human factors in Computing Systems (CHI EA ’00),
289-290. https://doi.org/10.1145/633292.633461[31] Jong-Eun Roselyn Lee and Cliford I. Nass. 2010. Trust in computers: The
Computers-Are-Social-Actors (CASA) paradigm and trustworthiness perception in human-computer communication. Trust and Technology in a Ubiquitous Modern Environment: Theoretical and Methodological Perspectives, 1-15.
https://doi.org/10.4018/978-1-61520-901-9.ch001[32] Kwan Min Lee and Cliford Nass. 2003. Designing social presence of social actors
in human computer interaction. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (CHI ’03), 289-296. https://doi.org/10.1145/
642611.642662[33] Sunok Lee, Minji Cho, and Sangsu Lee. 2020. What if conversational agents
became invisible? Comparing users’ mental models according to physical entity
of AI speaker. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3. https://doi.org/10.1145/3411840[34] Yaniv Leviathan. 2018. Google Duplex: An AI System for Accomplishing RealWorld Tasks Over the Phone. Google AI Blog. Retrieved January 8, 2021 from http://ai.googleblog.com/2018/05/duplex-ai-system-for-natural-conversation.html[35] Alessandro Liberati, Douglas G. Altman, Jennifer Tetzlaf, Cynthia Mulrow, Peter
C. Gøtzsche, John P. A. Ioannidis, Mike Clarke, P. J. Devereaux, Jos Kleijnen, andDavid Moher. 2009. The PRISMA statement for reporting systematic reviews andmeta-analyses of studies that evaluate healthcare interventions: Explanation andelaboration. PLoS Medicine 6, 7. https://doi.org/10.1371/journal.pmed.1000100[36] Jessica Lindblom and Rebecca Andreasson. 2016. Current challenges for UX evaluation of human-robot interaction. In Advances in ergonomics of manufacturing:
Managing the enterprise of the future. Springer, 267-277.
[37] Yuhan Luo, Bongshin Lee, and Eun Kyoung Choe. 2020. TandemTrack: Shaping
consistent exercise experience by complementing a mobile app with a smart
speaker. In Proceedings of the 2020 CHI Conference on Human Factors in Computing
Systems (CHI ’20), 1-13. https://doi.org/10.1145/3313831.3376616[38] Nikolas Martelaro, Jaime Teevan, and Shamsi T. Iqbal. 2019. An exploration of
speech-based productivity support in the car. In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems (CHI ’19), 1-12. https://doi.
org/10.1145/3290605.3300494[39] Donald McMillan, Barry Brown, Ikkaku Kawaguchi, Razan Jaber, Jordi Solsona
Belenguer, and Hideaki Kuzuoka. 2019. Designing with gaze: Tama – a gazeactivated smart-speaker. Proc. ACM Hum.-Comput. Interact. 3, CSCW. https://doi.
org/10.1145/3359278[40] Cliford Ivar Nass and Scott Brave. 2005. Wired for Speech: How Voice Activates
and Advances the Human-Computer Relationship. MIT Press, Cambridge, MA.
[41] Cliford Nass, Ing-Marie Jonsson, Helen Harris, Ben Reaves, Jack Endo, Scott
Brave, and Leila Takayama. 2005. Improving automotive safety by pairing driveremotion and car voice emotion. In CHI ’05 Extended Abstracts on Human Factors
in Computing Systems (CHI EA ’05), 1973-1976. https://doi.org/10.1145/1056808.
1057070[42] Cliford Nass, Jonathan Steuer, and Ellen R. Tauber. 1994. Computers are social
actors. In Proceedings of the SIGCHI conference on Human factors in computing
systems (CHI ’94), 72-78. https://doi.org/10.1145/191666.191703[43] Caroline Nowacki, Anna Gordeeva, and Anne-Hélène Lizé. 2020. Improving theusability of voice user interfaces: A new set of ergonomic criteria. In Design, User
Experience, and Usability. Design for Contemporary Interactive Environments (HCII
2020), 117-133. https://doi.org/10.1007/978-3-030-49760-6_8[44] Debajyoti Pal, Chonlameth Arpnikanondt, Suree Funilkul, and Vijayakumar
Varadarajan. 2019. User experience with smart voice assistants: The accent
perspective. In 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT ’19), 1-6. https://doi.org/10.1109/
ICCCNT45670.2019.8944754[45] Katie Seaborn, Norihisa Paul Miyake, Peter Pennefather, and Mihoko OtakeMatsuura. 2021. Voice in human-agent interaction: A survey. ACM Computing
Surveys (CSUR), in press. https://doi.org/10.1145/3386867[46] Ameneh Shamekhi, Q. Vera Liao, Dakuo Wang, Rachel K. E. Bellamy, and Thomas
Erickson. 2018. Face value? Exploring the efects of embodiment for a groupfacilitation agent. In Proceedings of the 2018 CHI Conference on Human Factors

CHI ’21 Extended Abstracts, May 08–13, 2021, Yokohama, Japanin Computing Systems (CHI ’18), 391:1-391:13. https://doi.org/10.1145/3173574.
3173965[47] Yang Shi, Xin Yan, Xiaojuan Ma, Yongqi Lou, and Nan Cao. 2018. Designingemotional expressions of conversational states for voice assistants: Modality andengagement. In Extended Abstracts of the 2018 CHI Conference on Human Factors
in Computing Systems (CHI ’18), 1-6. https://doi.org/10.1145/3170427.3188560[48] Carl R. Strathmeyer. 1990. Voice in computing: An overview of available technologies. Computer 23, 8, 10-15. https://doi.org/10.1109/2.56866[49] Ilaria Torre, Jeremy Goslin, Laurence White, and Debora Zanatto. 2018. Trustin artifcial voices: A “congruency efect” of frst impressions and behaviouralexperience. In Proceedings of the 2018 Technology, Mind, and Society Conference(TechMindSociety ’18), Article No. 40. https://doi.org/10.1145/3183654.3183691[50] Tom Tullis and Bill Albert. 2008. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufmann, Burlington, MA.

Katie Seaborn and Jacqueline Urakami[51] Yunhan Wu, Justin Edwards, Orla Cooney, Anna Bleakley, Philip R. Doyle,
Leigh Clark, Daniel Rough, and Benjamin R. Cowan. 2020. Mental workload
and language production in non-native speaker IPA interaction. In Proceedings of the 2nd Conference on Conversational User Interfaces (CUI ’20). https://doi.org/10.1145/3405755.3406118[52] Svetlana Yarosh, Stryker Thompson, Kathleen Watson, Alice Chase, Ashwin
Senthilkumar, Ye Yuan, and A. J. Bernheim Brush. 2018. Children asking questions:
Speech interface reformulations and personifcation preferences. In Proceedingsof the 17th ACM Conference on Interaction Design and Children (IDC ’18), 300-312.
https://doi.org/10.1145/3202185.3202207[53] Qian Yu, Tonya Nguyen, Soravis Prakkamakul, and Niloufar Salehi. 2019. “I almost
fell in love with a machine”: Speaking with computers afects self-disclosure. In
Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing
Systems (CHI EA ’19), 1-6. https://doi.org/10.1145/3290607.3312918