Qualitative Approaches to Voice UX
KATIE SEABORN
Tokyo Institute of Technology, Meguro-ku, Japan, seaborn.k.aa@m.titech.ac.jp

JACQUELINE URAKAMI
Kyocera Corporation, Tokyo, Japan, jacqueline.urakami.ke@kyocera.jp

PETER PENNEFATHER
University of Toronto, Toronto, Canada, p.pennefather@gmail.com

NORIHISA P. MIYAKERIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan, norihisa.miyake@alum.riken.jp
Voice is a natural mode of expression offered by modern computer-based systems. Qualitative perspectives on voicebased user experiences (voice UX) offer rich descriptions of complex interactions that numbers alone cannot fully
represent. We conducted a systematic review of the literature on qualitative approaches to voice UX, capturing the natureof this body of work in a systematic map and offering a qualitative synthesis of findings. We highlight the benefits ofqualitative methods for voice UX research, identify opportunities for increasing rigour in methods and outcomes, anddistill patterns of experience across a diversity of devices and modes of qualitative praxis.
CCS CONCEPTS • Human-centered computing~Human computer interaction (HCI)~Interaction devices~Sound-based

i put / output • Hu a -ce tered co puti g~Hu a co puter i teractio HCI ~HCI desig a d evaluatio
ethods •Human-centered computing~Human computer interaction (HCI)~Interaction paradigms~Natural language interfaces
Additional Keywords and Phrases: Voice user experience (voice UX), voice interaction, computer voice, voice userinterface (VUI), conversational user interface (CUI), interactive voice response (IVR), voice assistant, speech-based
interaction, speech synthesis, voice perception, vocalics, embodied conversational agent (ECA), qualitative research

1

INTRODUCTION

Voice is ubiquitous in human communication, expression, and embodiment [79,85]. This has inspired vested
efforts in crafting technologies and conducting research on voice interaction, speech recognition, and vocal
synthesis in computer systems. Modern handheld devices and virtual assistants demonstrate an increasing
prevalence of, or at least a desire for, two-way voice-based interaction [17,24,139]. Global sales of suchagents and devices as well as smart speakers indicate commercial and consumer interest. For instance,
Amazon shipped 40.1 million smart speaker units in 2021 alone, with 60% of the market share being
younger people aged 18-241. Within research and industrial R&D spaces, decades of work has pushed foradvances in voice-enabled speech interfaces and environments. Notable examples include work on natural1

https://www.statista.com/topics/4748/smart-speakers/

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first
page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit ispermitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM 0360-0300/2024/04-ART
http://dx.doi.org/10.1145/3658666
ACM Comput. Surv.

language processing (NLP) in machine learning (ML) fields [22]; efforts to replicate realistic human-like
vocals in text-to-speech (TTS) systems [16]; and a move towards natural forms of expression during voice
and speech interaction, perhaps best represented by embodied conversational agents (ECAs) [19] andconversational user interfaces (CUIs) [102]. Application areas range from interfaces for Internet of Things(IoT) systems, smart cars that speak, accessibility for blind and low-vision users, and also hands-freecontexts of use [26,139]. Suffice to say, ―visions‖ of voice interaction, advances in technology, and ongoingcommercial viability have drawn the attention of researchers across a range of technical fields of study to thequestion of what voice user experiences (UX) modern forms of voice interaction in computer-based systems
can, and should, provide.
Exploring voice UX, indeed any form of UX, requires multiple forms of inquiry. UX as a human factor
and research construct is not easily reduced to a series of variables or best represented by numbers[55,56,82,86]. Experience is by definition qualitative, subjective, impressionistic, and encompassing. It canbe difficult to describe, complex, and can even shift over time for one individual. This is not to say thatquantitative approaches to voice UX are invalid. Indeed, previous literature surveys and systematic reviews[4,17,24,139,141,153] have fruitfully mapped out various modes of quantitative inquiry, measures andinstruments, and generalized (or generalizable) findings on the UX that computer voice, speech interaction,and conversational formats can offer. What is needed, as pointed out in the limitations of recent reviews[24,139], is a greater understanding of the qualitative side. This means not only the qualities of experiencethat people can have with voice and speech in computer systems, but also what qualitative and mixed
methodologies have been most rewarding, all towards generating new questions, understanding quantitative
results, and setting the stage for theory development [32]. This may be especially important for those of us incomputer science and engineering [78] who may not be exposed to these forms of inquiry, understand their
purpose, or recognize their value.
To this end, we have conducted a systematic review of the literature on qualitative approaches to voice
UX. Our objectives were twofold. First, we aimed to generate a critical reflective systematic map describing
the nature of qualitative approaches to voice UX. Second, we aimed to conduct a qualitative evidence
synthesis of the qualitative findings from those studies about voice UX. Our research questions were
threefold. For the first objective, we sought to map out the state of affairs in qualitative work on voice UX.
We asked: RQ1. What are the characteristics of the research and qualitative approaches used?
(Methodology) Our second objective was to understand why qualitative inquiry is used for voice UX andwhat characteristics of voice UX inspire, or potentially require, qualitative approaches. We asked: RQ2.
What reason(s) did the researchers provide for taking a qualitative approach to evaluating voice UX?
(Epistemology) For the third objective, we sought to demonstrate the value of using qualitative approaches tounderstand voice UX, aiming for a synthesis of findings, where possible, as well as highlighting thosefindings derived through qualitative inquiry. We asked: RQ3. What findings on voice UX have thesequalitative approaches generated? (Knowledge) These questions generated a rich tapestry of methodologicaldescriptions and knowledge synthesis on qualitative approaches to the study of voice UX, our main
contribution. We hope that this work demonstrates the value of taking on a qualitative lens and will inspire
further qualitative and mixed forms of inquiry within voice UX research.

ACM Comput. Surv.

2

METHODS

We conducted a systematic review based on the Cochrane standard [60]. Cochrane is a worldwide group of
professionals who have developed standardized methods and procedures for conducting systematic review
work that has the goal of comprehensively gathering and analyzing the available materials on a given subject
or within a field of study [53]. While focused on the health and medical domains, the Cochrane standard istranslatable to other areas [53] and often used in the field of human-computer interaction (HCI) [131], underwhich voice UX research and practice falls. The method requires the use of a standard procedure in review
protocol and reporting. For this, we followed a modified version 2 of the Preferred Reporting Items for
Systematic Reviews and Meta-Analyses (PRISMA) 2020 Statement3, which is recommended by Cochrane[111]. The PRISMA statement is made up of a flow diagram representing the review process and a 27-item
checklist that covers all action and reporting items considered foundational to the Cochrane systematic
review method, e.g., eligibility criteria, search strategies, etc. As a tool, it supports customizations to enable
use in other disciplines, like HCI. We start by defining terms and operationalizing key concepts. We thenreport on our protocol in detail. This protocol was registered before data collection at OSF 4 on June 11th,2021.
2.1 Definitions and Frameworks

We recognize the interdisciplinary nature of certain terms and concepts used in this work. For clarity, wedefine these terms and cite the conceptualizations we relied upon in our work.
2.1.1 Voice and Speech.

Voice is defined as ―an expressive aural medium of communication‖ premised in sound [139]. Voice refers
to the use of sounds to express utterances, comprising vocalizations, verbalizations, and other forms ofauditory content (output), that are heard and interpreted by the sender and any recipients (input), including
non-humans [47,79,80]. In humans, voice is typically generated through vibrations produced by the vocal
folds found in the larynx that are then altered by features of the vocal tract, including the throat, nasal cavity,and so on. Voice can also be generated by machine-synthesized vibrations. While voice is often used tocommunicate words, i.e., the act of verbalizing, this is not always the case; babbling, coughing, whistling,backchanneling, vocal pauses and fillers, and other non- and pseudo-linguistic vocalizations are also mattersof voice. Speech, while often used interchangeably with voice, is a distinct concept [104,116] that can referto verbalizations as well as more technically to the state of voice after leaving the body. In this survey, we
focus on work that explicitly targeted or assessed voice as the ―how‖ of sound-based communication.

2

Our deviations were due to the qualitative nature of the studies included and disciplinary differences between the
medical and health sciences and computer science and engineering. For example, most publications do not use structured
abstracts.
3

http://www.prisma-statement.org

4

https://osf.io/c6y8z

ACM Comput. Surv.

2.1.2 Vocalics and Paralanguage.

Vocalics, also called paralanguage, refers to the perceived nonverbal paralinguistic and prosodic qualities ofvoice [122]. Nonverbal paralinguistic qualities include tone of voice, loudness, pitch, and timbre (or voice
quality). Nonverbal prosodic qualities include rhythm over the course of a vocalization, intonation as
variation in pitch, and stress or emphasis of certain sounds over others. These properties vary and intersect toproduce higher-level expressions indicative of emotion and affective states, social information (including
personality, dialect, and gender), and biological states (including health and age) [80,85]. Given its
importance to voice, we paid special attention to vocalics in the work surveyed.
2.1.3 Utterances and Conversation.

An utterance is an auditory message [112] or the ―what‖ of voice-based communication [124]. Utterances
can be linguistic, i.e., word- and sentence-based, or non-linguistic, i.e., gibberish, vocal bursts [14].
Conversation involves interactive rounds of utterances between at least two actors [25,112] but may or may
not involve voice, as soundless, text-based CUIs and chatbots demonstrate. In this survey, we included
technologies that used any kind of utterance but excluded technologies that did not have voice input and/or
output.
2.1.4 Artificial Intelligence (AI), Agents, Interfaces, Systems, and Spaces.

Voice is being embedded in a large array of interactive systems and technologies. At present, we do not havesufficient evidence to meaningfully distinguish the UX that these systems offer in terms of voice [139];
indeed, one motivation of this survey is to gear research in that direction. We therefore included all options.
We begin with definitions for these options to distinguish common underlying technologies and modes
voice-based of interaction. Artificial intelligence (AI) is a broad and murky term, due to its prevalence acrossvarious technologies as well as ongoing debate about what ―intelligence‖ should or can mean, for computers
and in general [108]. A more useful framing may be agent, referring to any person or thing that can takeaction in some world, virtual or real. Agents are embodied in the world, acting on and within it, potentiallywith other agents or on its own, typically through some combination of sensors and actuators [100,114]. Yet,
not all voice-based computer technologies can be classified as agents. Agents are distinct from the user,
implying a relinquishing of user control [147]. Interactions and control mechanisms can be found in objects,
e.g., voice-based refrigerators, interfaces, e.g., voice-controlled vehicle operation, and spaces, e.g., smart
homes. There are many ―bodies,‖ possible and uncharted [139]. We have taken a platform-neutral approachin this survey, including work that involves voice even when the computer is embedded and potentiallyunapparent for the user.
2.1.5 Voice User Interfaces (VUIs), User Experience (UX), and Voice UX.

Voice user interfaces (VUIs), also called speech interfaces, are computer-based technologies that use voice
as the primary or sole modality during interaction with people [26,141]. Specifically, they are ―what a person
interacts with‖ [26:5, our emphasis], be it an interface, agent, display, environment, or a voice of unknown
source and potentially ―bodiless‖ [139]. Such systems are characterized by voice interaction or "verbal andauditory interaction with a computer-based system that uses voice-only input, voice-only output, and/or
voice input-output" [141:2]. As such, VUIs typically employ speech recognition systems and synthesized

ACM Comput. Surv.

speech to enable spoken language interactions through the auditory modality [26]. However, VUIs can be
one-way displays, delivering content through words or other sound-based phenomena, and/or recipients ofhuman speech or other inputs, auditory or otherwise. We must also recognize that many VUI systems areprototypes that involve a human operator, such as in Wizard of Oz systems where a human controls the TTS
or even speaks on behalf of the VUI, while the user interacts unawares [24,63,120].
Interacting with VUIs and other voice systems gives rise to user experience (UX). The International
Organization for Standardization (ISO) defines UX as a ―user‘s perceptions and responses that result fromthe use and/or anticipated use of a system, product or service‖ (ISO 9241) [66]. Voice user experience (voiceUX) then refers to the forms of UX that arise during voice interaction with a voice system [141]. Notably,
UX is subjective, an experiential construct that comprises affective responses and emotions, attitudes andbeliefs, preferences and perceptions, and psychological and behavioural responses within and outside of theexperience [66]. UX occurs regardless of the intentions or awareness of the designers, developers,
researchers, and/or providers of the system. Whether evaluated or not, UX happens for the user in a given
context featuring interactions with a system [66]. In general, UX is a complex phenomenon and factor of
study [55,56,82,86] involving subjective and objective modes of inquiry, quantitative measures andqualitative descriptions, and patterns of attitudes and behaviours that may shift over time and differ widely
among individuals. Previous work (e.g., [24,139,141]) has reviewed quantitative approaches to studying
voice UX. The current review adds to this prior work by addressing the thus far understudied qualitative side.
We only included papers that provided insights on the UX of voice systems, excluding, for example,
technical reports and system test papers.
2.1.6 Qualitative Research.

Qualitative approaches to research focus on the qualities of experience. Where quantitative approaches
involve, as a baseline, some form of measurement resulting in numerical data, qualitative approaches areconcerned with collecting non-quantitative or non-quantifiable data that describes, often deeply, the nature ofexperience [29–32]. As Pluye and Hong [117] characterize it, the qualitative is about ―the power of stories‖
while the quantitative is about ―the power of numbers.‖ Both forms of data are empirical in nature but are
distinguished by the data collection tool and data types used. Specifically, non-numerical data in the form ofwords, sound, images, video, and so on, and non-quantifiable data representing qualities of an experiencerather than their frequency and intensity. Furthermore, qualitative methods tend not to be concerned withmaking generalizable claims or prescribing a fixed view of reality, while quantitative methods often takesuch a positivist approach and aim for generalizations about people. Finally, qualitative research often takes
place in situ so as to fully capture the experience in its natural context [35]. This is not a requirement of
quantitative forms of research, which often take place in artificial settings, i.e., labs. These differences andthe potential for each approach to enhance and inform the other have led to the uptake of mixed methods
research [29,30]. Mixed methodologies, as the name implies, mix quantitative and qualitative methodologies
and/or data collection and/or data in some way. The ways in and extent to which the mixture can happen
varies widely. Nevertheless, it is common in HCI [161], as well as in multidisciplinary fields of study
generally [29,31]. As such, we include both nominally qualitative and mixed methods research in thisreview, zoning in on the qualitative aspects in cases involving the latter.

ACM Comput. Surv.

Qualitative research can be broadly categorized in a typology of five approaches: narrative,
phenomenology, grounded theory, ethnography, and case study [32]. We describe each below:
 In narrative research, the subject of study and/or method is in the form of a story, typically comprising
events in chronological order and focused on one person’s lived experiences [27]. For example, anarrative study on voice UX could follow the path of use by an older adult in the home over several
seasons, capturing novelty effects, onboarding, errors and surprises, relationship-building and breaking, trust development, and so on.
 In contrast, phenomenology focuses on generating a common understanding or “composite description”[32:45] of a specific phenomenon experienced by multiple people. Important is acknowledging andbracketing out the researcher/s’ own understanding and experiences of the phenomenon anddiscussing philosophical assumptions about the fruits of the research, especially how each person’s
subjective experience may differ and how these experiences may or may not inform objective reality. A
voice project attempting to create a gender-neutral voice, for instance, would consider how individuals
perceive the genderedness of the voice or its lack, as well as how the researcher’s own framing of“neutrality” may not match that of participants.
 Grounded theory, rather than being merely descriptive as with phenomenology, attempts to generate a
new theory, or general explanation, of a process or (inter)action-based phenomenon that is “grounded”
in the data collected from multiple people [155]. Typically, data collection and analysis occur
synchronously and cyclically, with the researcher/s comparing and contrasting accounts of experienceas the process plays out. For example, grounded theory may be used to generate a description of how
varying the frequency or pitch of a voice that speaks gibberish is dependent on the underlying language
upon which the gibberish is based, requiring several cycles through different languages with the same
participants.
 In ethnography, the researcher/s centre their inquiry on a specific and identifiable cultural group orcommunity whose members directly share an experience of the phenomenon rather than experiencingthat phenomenon separately [9]. For example, a voice-enabled smart refrigerator could be given to
people in an office space who may use it in different ways together and individually, at certain points inthe day or week. Often, the researcher/s enter into the same spaces and even the daily lives of thegroup so as to capture shared patterns of behaviour and social organization in line with apredetermined theory. Continuing the example, a researcher might attempt to map out how the smart
refrigerator fits into theories of planned behavior, observing how attitudes and social norms shift overtime in response to the presence of the new “voice” in the room.
 Finally, case studies focus on a slice of experience: a “thick” description of a bounded entity, which
could range from an individual to a community to a problem to a relationship and so on [174]. A casestudy of voice UX could, for instance, focus on the installation of a voice-based agent in place of a
receptionist at a popular local hospital, and people’s first reactions to it. Importantly, the case is strictly
defined by specific parameters (“bounded”) and followed over time at a particular site. A case isgenerally chosen as a representative means of better understanding and/or illustrating a complex,
unique, and/or multifaceted phenomenon.

ACM Comput. Surv.

Importantly, these categories of qualitative inquiry are not exhaustive nor restrictive. HCI research that
employs qualitative forms of inquiry can be liminal: involving a range of data collection, data forms, anddata analysis that may not fit into typical research patterns. For example, Blandford [8] coined the term semistructured qualitative studies to describe a common form of qualitative engagement in HCI that tends to use
observation and interviews that are then systematically coded. Moreover, critical methodologists operating
within and beyond HCI [8,28], notably Rode [129], have pointed out that qualitative approaches undertaken
in HCI, even under the same name, may not be employed in the same way as in other disciplines. Finally, forthese and other reasons, researchers may not label their approach explicitly, opting for a more generalmoniker, especially ―qualitative study.‖ As such, we use this five-part typology to describe the qualitativeorientations of this body of work, focusing instead on the forms of data, information, and knowledge that
may be usefully generated through such a variety of qualitative frames on voice UX.
Additionally, we do the same for mixed methods studies, harnessing Greene, Caracelli, and Graham‘s[50] five-part typology to describe the ways in which qualitative forms of inquiry were included. According
to Greene, Caracelli, and Graham [50], methods may be mixed so as to triangulate for consensus,
complement to enhance one or the other, develop the other form of inquiry in a follow-up study, initiate newquestions or provocations that may buck against convention, and/or expand the scope of inquiry in a multipart fashion. Where possible, we categorize the research as described by the researchers. However, in theabsence of labels and to avoid prescription, we employ this five-part typology as a way of understanding
mixed methods orientations to voice UX research. These framings thus represent a form of meta-level
qualitative data that may be used to spark new forms of inquiry in future research.
2.2 Eligibility Criteria

Eligibility criteria were developed to identify voice UX research that involved qualitative modes of inquiry.
Inclusion criteria were: Full studies, including user studies, experiments, interviews, field studies, etc.;
qualitative approach/es used; voice UX as a main subject of study evaluated with a voice system (as defined
above); and published to a peer-reviewed academic venue. Exclusion criteria were: Voice interaction notevaluated (even if the system has a voice or uses voice interaction); not enough detail reported to understandthe study procedure; not studies, e.g., literature surveys, or not full studies, e.g., posters, proposals, small
pilots; studies where only quantitative work was reported; inaccessible work (including after attempts tocontact author/s directly); and work not in English, Japanese, German, or French, the languages known by
the researchers.
2.3 Information Sources and Search Strategies

General academic databases, specifically Web of Science, Scopus, EBSCOhost 5 , and PsychInfo, anddisciplinary academic databases, specifically ACM Digital Library and IEEE Xplore, were included in the
search conducted on Thursday September 16th, 2021; see the spreadsheet on OSF 6 for the full search

5Note that we were unable to export records from EBSCOhost.

6

https://osf.io/eb8n6

ACM Comput. Surv.

strategies per information source. We also manually added relevant papers from the data set provided by
Seaborn et al. [139].
2.4 Selection and Data Collection Process and Data Items

One researcher screened all abstracts, and a second researcher independently reviewed the papers marked forrejection. Then, four researchers independently reviewed the full text of about a quarter of these papers.
Next, one researcher reviewed the papers flagged for rejection. Then, three researchers extracted the data,with the lead researcher taking on half of the papers and the rest divided equally. Extractions were checkedby the lead researcher. If a paper was flagged for removal, all authors were notified beforehand. At eachstage, the papers were presented in a random order. When disagreements occurred, the researchers discussedwith each other and/or one of the researchers not involved until consensus was reached. Data items were:participant demographics, user groups, agent types and enabling technologies, algorithms, contexts of use,theories used, qualitative approach, qualitative data collection methods, instruments, inquiry methods,
research purpose/s, reasons for using qualitative approaches, and qualitative findings.
2.5 Data Analysis and Synthesis Methods

We generated descriptive statistics, including mean (M), median (MD), interquartile ranges (IQR), standard
deviations (SD), counts, and percentages, as well as inferential statistics (such as t-tests) for quantitative(such as gender data) or quantifiable (such as theme frequencies) data. Missing data was counted as its own
variable. For the qualitative data, a reflexive thematic analysis approach was used [12] out of recognition of
the diverse qualitative nature of our data set. Three researchers coded each set of data. The research purpose
data was coded in a combined inductive-deductive way, inspired by the thematic structures from previous
research on quantitative voice UX [24,139,141]. For example, we attempted to apply the dependent variable
themes from Seaborn and Urakami [141], which included usability, engagement, cognition, affect, sociality,
attitudes, perceptions of voice, and aggregate UX; however, not all of these categories applied to our data set.
The themes that emerged were then used to structure the research findings data. A deductive approach wastaken for the qualitative reasons data; one researcher used the frameworks provided by Creswell and Poth[32] for the qualitative studies and Greene, Caracelli, and Graham [50] for the mixed method studies, as
outlined in Creswell and Plano Clark [31]. Finally, an inductive approach was taken for the instrument data,
which was diverse and not tied to a theory or paradigm. In recognition of this, two researchers synchronously
and collaboratively coded the data to develop a shared representation of its utility. For all analyses, codes
were grouped into themes and theme frequency counts were generated. We did not conduct risk of bias
assessments or assess certainty/confidence in reporting due to the qualitative and mixed methods nature ofthis body of work, as well as its variety, as assessment may be difficult or inappropriate, e.g., if the authors
are philosophically opposed to measures of agreement [96].

ACM Comput. Surv.

3

RESULTS

From an initial 609 results across six databases, 66 items were selected for inclusion; refer to our PRISMA
flow diagram, available on OSF7. Note that counts refer to papers unless otherwise specified. The full dataset
is available on OSF8.
3.1 Methodology: Characteristics of the Research and Qualitative Approaches (RQ1)
3.1.1 Participant Demographics and User Groups.

Publications spanned 18 years, from 2003 to 2021, but most are clustered in the last five years (Figure 1). A
total of 3080 people participated across all studies. The average number of participants per study was 46.7
(SD=108.1, MD=21, IQR=35.5). Breaking this down by qualitative inquiry method, we find an average of
36 (SD=34.3, MD=20, IQR=31) for interviews, 243 (SD=423.6, MD=40, IQR=231) for questionnaires(which could be large-scale surveys or questionnaires deployed in a study, hence the large range), 44(SD=38.6, M=38, IQR=14) for focus groups, 20 (SD=11.3, MD=20, IQR=8) for workshops, 32 (SD=23.5,
MD=21, IQR=33.3) for observational studies, and 20 (SD=9.2, MD=19, IQR=3) for transcript analyses; notethat counts overlap because one study may have used multiple modes of inquiry. 1251 women (42.4%) and
1317 men (44.6%) as well as three with no gender identified were reported; no one of another gender identity
was reported. A t-test did not find a statistically significant difference in the number of men and women,
indicating balanced representation among men and women, t(2567) = .063, p = .47. However, the gender of
512 participants was not reported, so this result is tentative. Most participants were drawn from the general
population (24 or 36.4%) or constituted a special group (21 or 31.8%). Eleven studies (16.7%) included older
adults, ten (15.2%) included children, and nine (13.6%) included students. Two studies did not specify theuser group. Six studies focused on caregivers and/or mothers, four on blind/low-vision/visually impaired
folks, three on those with or were parents of those with HIV, three on employees, two on roboticists ordesigners, and two on people with dementia.

7

https://osf.io/3arz9

8

https://osf.io/yhwrs

ACM Comput. Surv.

Figure 1: Papers employing qualitative inquiry to explore voice UX over time.
3.1.2 Voice Systems and Enabling Technologies.

Note that many studies used more than one system, so count totals may vary. Most voice systems (Figure 2,left) were smartphone-based (19 or 28.8%), general phones (17 or 25.8%), and robots (15 or 22.7%). Other
voice systems included virtual characters (6 or 9.1%), computers, i.e., computer voice through speakers (5 or
7.6%), or some other computer-based system, such as custom prototypes (7 or 10.6%). Notably, four studies(6.1%) did not report on a type of voice system, due to the nature of the studies as attitudinal, explorative,
and/or participatory. Specific technologies included interactive voice response systems (16 or 24.2%), smart
things, especially speakers (13 or 19.7%) but also cars (2 or 3%) and watches (2 or 3%), and the NAO robot(2 or 3%). TTS‘s included Amazon Alexa (12 or 18.2%), Google Home (6 or 9.1%), and NAO (3 or 4.5%).
Voice-enabling technologies included TTS‘s and recordings (Figure 2, right). Most studies did not report the
TTS used, if any (16 or 24.2%) while fourteen (21.2%) used another TTS, i.e., one-offs, and five (7.6%) useda custom TTS. Five studies (7.6%) used pre-recorded speech and one used non-speech sounds. Overall, onethird (22) studies looked at human voices, one-third (22) at synthetic voices, and one-third (22) did not reporton the voice. Two studies used animal voices and two did not specify the voice type. While 29 studies(43.9%) did not specify the underlying intelligence, six (9.1%) used Google, notably DialogFlow, nine
(13.6%) used Amazon Web Services (AWS), eight (12.1%) used a custom system, and six (9.1%) used
another platform, such as NaoQi.
3.1.3 Contexts of Use.

Most studies explored an instruction context (24 or 36.4%), conversation (15 or 22.7%), or a service context(14 or 21.2%). Others employed a game (6 or 9.1%), a driving context (4 or 6.1%), or another context (8 or
12.1%), such as controlling a smart lock, disclosures of social behaviour, and folding a cloth.

ACM Comput. Surv.

Figure 2. Voice systems (left) and voice-enabling technologies (right).

3.1.4 Purpose for the Research.

We derived four high-level themes that describe the reasons behind conducting qualitative research on voice
UX (refer to Table 1 for a summary; the full table is available on OSF 9 ). Purposes expressed by the
researchers were experiential (63 or 95.5%), targeting usability (30 or 45.5%) and/or embodiment (33 or50%), attitudinal (45 or 68.2%), communal (41 or 62.1%), including specific populations (30 or 45.5%) andsocial factors (11 or 16.7%), and vocal, or reasons deemed of particular relevance, if not exclusive to voice
UX (17 or 25.8%). This last category, which may be especially pertinent here, included: conversational
styles [128], response styles [177], verbal behaviour [125], communicative signals [136], voice input/output(voice input/output) [166], disclosures [93], long-turns in conversation [33], living noise [70], source
orientation [51], conversation expectations [25], emotion in conversation [83], features of dialogue [121],
verbal prompts [1], synthetic versus real voices [81], conversational exchange and repair [39], and ―rich‖communication [3].
3.1.5 Theoretical Frameworks and Conceptual Models.

Many studies (27 or 40.9%) did not report on theory. Six (9.1%) used the Technology Acceptance Model(TAM) [167], four (6.1%) referred to the Computers Are Social Actors (CASA) paradigm [87,106,107], andtwo referenced the Uncanny Valley model [101]. Others were found once: the Big 5 theory of personality[71], the circumplex arousal-valence model of emotion [133], interpersonal communication theory [77],theory of mind, the integrative model of organizational trust [94], personification and the three-factor model ofanthropomorphism [43], rapport [158], hyperarticulation [152], source orientation [51], peer instruction [95], thecircumplex arousal-valence model [133], interpersonal communication theory [132], trouble-source repair [110], theoryof common ground [23], media equation theory [105], cognitive load theory [157], consequential communication [142],
para-social relationships theory [62], social response theory [105], extended privacy calculus model [36], and thestereotype content model [46].

9https://osf.io/qup5j

ACM Comput. Surv.

Table 1: Thematic framework of research purposes
Theme

Sub-Theme

Description

ExamplesCitations

Experiential

Usability

Relates to ease of use, effectiveness,and efficiency, including
accessibility, alternative modalities,
performance, and obstacles.
Relates to how the agent was or
could be embodied in form factor
and interaction, including(multi)modality, humanlikeness,
and nonverbals.

―explored general UX‖ …
―how older adults use thesystem‖ … ―effect onadherence behaviour‖ …
―explored 3 types of displays
from voice to visual‖ …
―expressiveness in speech
patterns‖ … ―sought toidentify gaze behaviours‖[1,11,13,18,21,33,39,42,52,
61,73,74,98,99,103,123,128,130,156,159,163,168,169,1
73][1,3,6,25,34,37,54,61,74,81,
83,91,113,119,121,125,136,
143,144,146,154,156,171,1
73,176,177]

Embodiment

Attitudinaln/aRelates to attitudes and ideas
expressed about the system,
including acceptability, mental
models, and trust.

―understand feelings of trust‖… ―how appropriate kids
thought the agent was‖ …
―understand perceived
persona‖[1,5,6,10,13,25,34,37,40,42,
51,52,54,57,64,70,73,81,88,
90,91,93,99,113,115,121,12
3,125,126,130,134,136,144,
154,159,162,163,168,171]

Socialn/a

Purposes were specialized bypopulation and social factors.

―rural mothers‖ … ―looked atfamily use‖ … ―

Vocal

n/a

Relates to factors particular or
exclusive to voice UX.

―hands free design‖ …
―explore long-turns inconversation‖[5,6,10,13,18,37–
40,42,52,54,57,61,64,70,72,
73,93,98,99,115,121,123,12
6,130,134,143–
145,154,156,159,162,163,1
66,172,176][1,3,7,25,33,39,51,70,81,83,
93,121,125,128,136,166,177]

3.1.6 Inquiry Methods and Modes of Qualitative Inquiry.

Most studies (39 or 59.1%) relied on a qualitative approach. A further 21 (31.8%) used a mixed methods
approach, while six (9.1%) generated qualitative data using quantitative methods. The most frequent mode ofinquiry was interviews (43 or 65.2%). Other common modes included observation (26 or 39.4%) and fieldstudies or ―in the wild‖ research (27 or 40.9%). Others included focus groups (7 or 10.6%), questionnaires
and surveys (7 or 10.6%), transcript analysis (5 or 7.6%) and design workshops (2 or 3%). Techniques used
once included diaries [70] and drawing [88]. Most studies evaluated participants' direct interaction with avoice system (34 or 51.5%). Many studies were conducted in-field rather than in the lab (24 or 36.4%) whileothers used Wizard of Oz regardless of the study context (13 or 19.7%). Ten (15.2%) of studies used a
scenario approach, with six (9.1%) using video and four (6.1%) using audio. Six (9.1%) did not describe indetail. A further five (7.6%) used another method, such as conducting a simulation, analyzing drawing, andsourcing transcripts.
3.1.7 Qualitative Instruments.

While a range of inquiry methods and modes of inquiry were used, many researchers employed questions togather qualitative data. We found this to be the case in 46 (69.7%) papers. We then inductively identified 331
codes across four themes and eight sub-themes bearing on how and why qualitative instruments were used.
Table 2 represents the distribution of these alongside the overarching qualitative approaches deployed andACM Comput. Surv.

the citations touching on each of the categories identified; the full data set with tool extractions is available
on OSF.10 Usage refers to current or previous usage (12 codes or 3.6%). Phenomenal impressions refer tointeractive experiences (95 codes or 28.7%), agent impressions (42 codes or 12%.7), and attitudes towards
the voice system (59 codes or 17.8%). Methodology includes feedback elicitation, co-design strategies, tool
validation, data mixing, and type of respondent, such as the user, a co-designer, or a system-manager (96codes or 29%). We finally considered lack of rigor in reporting, via ambiguous or missing details (26 codes
or 8.2%). For example, Strathmann, Szczuka, and Krämer [154] wrote of asking ―63 questions‖ on ―thetransparency of the voice assistant‘s functionality, the intensity of the relationship built, as well as thecommunication behavior towards the device" (p. 6). While they provided an example question for eachcategory, e.g., ―Would you tell Google/Alexa a secret?,‖ they did not specify an instrument or otherwise
provide all questions used. These gaps and diversity across the corpus of papers makes it difficult togeneralize best practices. The paucity of rigorous studies highlights the opportunities for development ofqualitative tools for voice UX.
3.2 Epistemology: Reasons for a Qualitative Approach to Researching Voice UX (RQ2)

Researchers provided a range of reasons behind employing a qualitative approach and the specific qualitative
approach taken. Table 3 presents the methodological orientations based on our thematic analysis using thetypologies from Creswell and Poth [32] and Greene, Caracelli, and Graham [50]. Note that while about onequarter (17 or 25.8%) of researchers provided explicit reasons and orientations to qualitative inquiry, agreater portion (30 or 45.5%) were less exact, such as by not describing with enough detail or not referring to
specific qualitative typology. A further 19 (28.8%) did not provide any reason or explanation of theirorientation to inquiry, simply stating that a qualitative approach was used or merely reporting on the use of aqualitative means of inquiry, such as an interview.
Of the qualitative-only work, none used a narrative orientation. Most took a phenomenological (21 or
31.8%) or ethnographic approach (17 or 25.8%), five used grounded theory (7.6%), and one [42] was a casestudy. Of the mixed methods research, most used qualitative inquiry in a complementary way (15 or 22.7%)
or for triangulation with the quantitative data (8 or 12.1%), while others sought to expand the range ofinquiry (4 or 6.1%) or had another reason for mixing methods (2 or 3%), specifically as a corrective response
to failures in quantitative procedures [18] and based on prior recommendations in the literature [5]. None had
developmental or initiation-oriented reasons for mixing methods.

10

https://osf.io/dj6cb

ACM Comput. Surv.

Table 2. Thematic framework of qualitative research instruments
Theme

Sub-Theme

Description

ExamplesCitations

Qualitative and Mixed
Methods Approaches

Usage

n/a

Previous or current
usage patterns.

―we asked aboutfamiliarity withvoice systems‖[6,7,13,34,52,57,113,136,
156,166,169]

Phenomenology,
Grounded Theory,
Ethnography,
Complementary

Phenomenal
Impressions

Interactive
Experience

General usability and
UX as well as for theagent, obstacles andbarriers, context of use,and social context.

―overall impressionof the agent‘sfacilitation‖[1,3,6,7,10,13,21,25,34,37
Phenomenology,,42,51,52,54,57,61,70,72,
Grounded Theory,
75,81,90,99,113,115,121,
Ethnography, Case
123,128,130,136,145,146, Study, Triangulation,
154,156,159,162,163,166,
Complementary,
168,169,171]
Expansion

Agent
Impressions

Communicative ability
and competency,
CASA factors (or―identity‖ factors), andbody factors.

―impressions of theagent‖ … ―are theytalking to a phoneor assistant‖[3,5,7,10,13,21,34,51,57,7
2,72–
75,81,90,99,113,115,121,
123,126,128,136,146,154,
156,162,163,168,171]

Attitudes

Utility and usefulness,
acceptability andwillingness-to-use,
preference, and trustand security.

―is the VA useful‖… ―do you haveany privacy
concerns while
using it‖

[5,6,10,13,21,25,34,37,52,
Phenomenology,
Ethnography, Grounded
54,57,61,70,72–
74,81,90,99,115,121,123,
Theory,Complementary,
126,130,136,145,146,156,
Expansion
159,163,166,168,169,171]

Feedback
Elicitation

Information-gatheringthat does not raise to
the level of co-design.

―suggestions toimprove‖[3,6,10,13,21,25,34,61,70,
Phenomenology,
73,74,81,90,98,99,113,11 Ethnography, Grounded
Theory,5,121,145,159,162,163,16
Complementary,
8,169]
Expansion

Co-Design

Users involved as codesigners.

―what features to
add‖[3,10,52,61,72,73,75,98,9
9,145,146,156,169]

Tool
Validation

Focus is on validating a
research tool.

―how did you use
the tools to explain
your view‖[75,90,136]

Data Mixing

Varieties of qualitative
and/or quantitative datacollected together.

n/a[5,37,42,72,73,90,99,126,
130,166]

Ethnography, Case
Study, Triangulation,
Complementary

Type of
RespondentUser, user-adjacentstakeholder, experts,
providers.

―We theninterviewed parentsof child users‖

n/a

Phenomenology,
Ethnography, Grounded
Theory, Case Study,
Complementary,
Expansionn/aDetails about the
instrument missing,such that they cannot
be extracted or
reproduced.

n/aMethodology

Ambiguous
Reporting

Phenomenology,
Grounded Theory,
Ethnography,
Triangulation,
Complementary,
Expansion

Phenomenology,
Ethnography,
Complementary
Phenomenology[1,3,6,7,13,21,25,37,42,51
Phenomenology,,52,54,61,70,72,75,98,113
Grounded Theory,,126,130,146,154,156,166
Ethnography, Case
Study, Triangulation,,168,171,177]
Complementary

The reasons provided were diverse and hard to categorize with more specific themes. Moreover, the levelof depth and detail varied widely. Bailey et al. [5] followed a recommendation by other researchers to use
qualitative methods. Ellway et al. [42] went into great detail, framing their approach as one informed byinterpretivism in information system fields and a desire to answer ―how‖ questions. They further explained

ACM Comput. Surv.

that they chose one, rather than multiple, cases for three reasons relating to complexity of the experience, themultiple actors and roles involved in the call centre, and the ability of the method to allow the researchers tofocus on how calls were passed between teams, a key factor in their research. Damen et al. [34] andShamekhi et al. [146] explained their use of a complementary mixed methods approach as a means ofgenerating greater insight into the quantitative data analyses. Bentley et al. [7], working with a large base of1000 commands, sought to identify patterns within and among teams and team members, effectively
combining phenomenology and grounded theory. Lopatovska and Williams [91] aimed to understand how
people personified Amazon Alexa, a qualitative data form that may have similar features across people,
especially with the focus on one agent. Similarly, Lee, Kim, and Lee [88] conducted a drawing study on
several agents but also several different people. Robb et al. [128] noted that they could focus on topics ofinterest while also shifting focus to emergent participant perspectives. Sayago [134] recognized that being aparticipant and observer in the classroom would allow for more naturalistic observations. Sezgin et al. [145]wished to better understand the quantitative data previously collected and how it might be used in futuresystems. Reasons were diverse but practical, focusing on the merits of the methods.
Table 3: Studies categorized by orientation to qualitative inquiry
ApproachTypology

Counts (%)

Qualitative

Narrative

n/a

n/a

Phenomenology

21 (31.8%)[1,3,7,21,25,38,39,51,61,75,83,88,91,103,121,136,143,154,169,172,176]

Grounded Theory
Ethnography

5 (7.6%)17 (25.8%)[38,51,70,103,128][6,10,13,54,70,72,73,98,123,128,130,134,144,159,162,163,168]

Mixed Methods

Citations

Case Study

One

[42]

Triangulation

8 (12.1%)[33,37,40,64,99,113,171,177]

Complementary

15 (22.7%)[5,11,34,52,57,74,81,90,93,119,126,145,146,156,166]

Development

n/a

n/a

Initiation
Expansionn/a
4 (6.1%)n/a[74,115,125,173]

Other

2 (3%)[5,18]

3.3 Knowledge: Qualitative Findings on Voice UX (RQ3)

We now turn to what knowledge has been garnered through qualitative approaches to evaluating voice UX.
We organize the findings by domain of study to highlight the state of knowledge for specific areas of study,
application domains, or topics of focus. Each subsection represents a domain of study: blind and low-vision
users and accessibility; teaching and instructional contexts; driving and navigation; well-being andtherapeutic contexts; health information contexts; multi-user contexts; user mental models ofanthropomorphism and social identity; user mental models of embodiment and form factors; and vocal
factors. Although diverse, within each domain of study we can summarize the findings by way of a series ofvoice UX factors, described in the following high-level framework: Experiential factors, which include:o Usability refers to how the agent, interface, space, or system performed for the user at the task oractivity, including accessibility and obstacles. Usability, as an ISO standard, is a basic part of evaluating

ACM Comput. Surv.

UX in general [67] and refers to meeting user needs and desires for effectiveness, efficiency, andsatisfaction.
o Agent embodiment refers to how the agent is situated within, interacts with, and is influenced by thecontext of use, including the environment and other agents, human or otherwise, within it [100].
Embodiment typically refers to the “body” of the voice UX system, including sensory modalities,
enabling technologies and devices, form factors and morphologies, notably level of anthropomorphismor humanlikeness, and expressiveness. When it comes to voice UX, vocalics may occur alongsidevisual and tangible modes of expression, including nonverbal behaviours and gestures [139,165].
 Attitudinal factors, which include:o Acceptability refers to the attitudes that people have about the voice system, from adoption topreferences, liking and disliking, feelings of satisfaction and appropriateness, and general usefulness
for the system in their lives. Acceptability may be especially important for voice as a new and evolving
mode of interaction and system expression for most people [167].
o Trust and privacy are entwined attitudinal factors that have a long history in agent-based and generalsystem work involving computation [68,87,92,164]. Trust refers to opening up to others with the
expectation of a reliable and honest exchange, while privacy refers to closing off from others or
restricting access in some way. Subfactors include rapport, safety, security, reliance, abuse, betrayals
and being tricked, and so on.
 Social factors or sociality, which refer to the social features and relations involved in or implicated byvoice UX. Voice is essentially about communication, a social act likely to have social repercussions,especially in terms of creating feelings of social connectedness among people.
 Vocal factors that refer to voice-specific and potentially voice-exclusive forms of interactions andeffects.
We further contextualize this framework with respect to specific qualitative modes of inquiry in Table 4. For
now, we consider what we know (and perhaps do not yet know) by domain of voice UX research. This
is not an exhaustive list of domains; rather, it represents the largest clusters of related work in the
corpus of work surveyed.
3.3.1 Blind and Low-Vision Users and Accessibility.

Voice may be particularly relevant to blind/low vision users. Voice-only modes of interaction were found tobe beneficial and enabling, even when compared to non-visual alternatives, such as tangible interaction[18,98]. Metatla et al. [98] conducted a series of co-design workshops with students, educators, support staff,
and local authorities, a novel approach to identifying stakeholder-sensitive insights on designing voicesystems for school contexts. Vashistha et al. [166] identified voice interaction as an essential mode ofaccessing and expressing oneself on social media. Visually impaired older adults appreciated the notion of a
medical voice agent, even though the implementation was not as smooth or humanlike as desired [54].
Indeed, they had mixed reactions to the medical voice agent [54]. Speed and pronunciation of medication
names were key issues, as a desire for a smaller device as well as direct interaction with people rather thantechnology. Most of these issues may be resolved with advances in the underlying technologies. In sum,voice UX offers a compelling and accessible experience for people with impairments related vision, even if it
is not perfect yet.

ACM Comput. Surv.

3.3.2 Teaching and Instructional Contexts.

Voice systems have been deployed as teaching aids across a variety of contexts, from helping older adults
how to use an electronic programme guide [18] to learning how to use a text editor [171] to simply being acommon mode of engagement with smart speakers, i.e., information seeking [7,134,144]. Others havefocused on the ability of the technology to enable an educational experience compared to human standards interms of task performance [40,136], notably as ―peers‖ or ―companions‖ [40,134] or even ―supporters‖ or―sponsors‖ in the context of behaviour change [42,73,126]. In learning contexts, trust has been a focal point,
notably achieved by way of vulnerability through disclosures [93] and characters speaking in the local
language [64,72]. In general, voice-based engagement was preferred over other options, with notablecaveats. Device trade-offs may be important to consider for new learners. Militello et al. [99], for instance,found that phones were easy to learn but prone to speech recognition errors, while the more accurate smart
speakers were too expensive. Multimodality might also be important for learners and learning contexts, withmany participants expressing a desire for visuals or even tactility to add voice-based engagement[98,99,145,173], perhaps especially when the voice is coming from a robot [40]. Children may be a special
category of student in need of special care because of the linguistic engagement offered by many voice-based
systems. Du et al. [39] found that Alexa was too erudite during learning games and parents had to intervene.
In contrast, Dönmez et al. [38] found that children bonded to a robotic tutor that was playful, provided clear
responses and constant feedback, although it is unclear if this was eased by the robotic embodiment of theagent. On the other hand, Bailey et al. [5] found that low-literacy adult learners preferred the interactive
voice response system to events involving other people, perhaps because of social stigma around literacy.
In sum, voice appears to be an acceptable and sometimes powerful mode of engagement in instructionalsettings with a variety of learners. Still, we should consider the inclusion of visual or other modalities,
choose the enabling device and persona based on the socioeconomic background of the learners, and gear the
linguistic ability of the voice to the learner, especially when they are children, low literacy, or not fluent in
the system language.
3.3.3 Driving and Navigation.

Voice interaction within vehicles and navigation contexts is an emerging area of study that can involve lab
and field work, or some combination of the two, for experimental control or safety‘s sake. The surveyed
work raised certain factors as particularly relevant, if not crucial, for these contexts. The clear benefit ofvoice interaction in these contexts is the hands-free potential of voice interfaces. What form voiceinteractions can and should take have been explored and illuminated. Perrin et al. [113] found that
understandability in the case of navigational information was best for voice feedback compared to visual,auditory, and tactile feedback modalities, even though auditory feedback reaction time was longest. Large etal. [83] found that input commands were performed in a command-based style. Others have considered thepsychological and cognitive dimensions of voice UX. Kim et al. [74] found that voice-task demand was
related to driver interruptibility and adaptive behavior such as reducing speed in a driving task. This may be
attributed to the increased cognitive load caused by voice-only interaction. Braun et al. [11] found that theaffective qualities of the voice assistant were beneficial for expressing empathy with the driver andmediating their mood, especially negative states. In sum, the qualitative findings on voice UX in driving andnavigation contexts highlight the usability of the auditory modality and its flexibility for the task, as well as

ACM Comput. Surv.

provide support for voice against driver distraction and road rage. Voice in driving and navigation contexts
may be best designed around short commands that do not induce additional cognitive load or distractions on
the driver and personas that express empathy in a way that adapts to driver mood.
3.3.4 Well-being and Therapeutic Contexts.

Voice UX in well-being and therapeutic contexts is an emerging domain of study. Voice-based interaction intherapeutic contexts can improve mood and relaxation for pain relief [123]. Emotional voice assistants that
empathize with the user improved negative emotional states and were rated positively [11]. Voice was also
perceived as having a calming effect and acted as social support that helped with smoking cessation [126].
Clearly, voice interaction can supplement therapeutic activities. Other work has brought attention to factorsof sociality and anthropomorphism of the voice system. In Huang and Li [64], ―role play‖ with characters in
the local Cambodian language built trust. Pou-Prom et al. [121] found that human voices were moreengaging and that the Wizard of Oz prototype, being driven by a human operator, allowed for greater
flexibility in responses, which was highlighted as particularly well-received by participants compared to theautonomous, inflexible version. Other research highlights opportunities for interaction design work. For
instance, participants in a study on therapy adherence monitoring desired to edit inputs with voice [52].
Altogether, voice can be a calming and empathic supplementary feature, but must otherwise be designed forthe particular well-being activity or therapeutic context, with the needs of the care recipient in mind.
3.3.5 Health Information Contexts.

Most work has been focused on health information purveyance, especially by way of interactive voice
response systems in low-literary and low-resources areas of the world. Key factors include expectations ofsystem performance, especially length of call (shorter is better), authenticity, repetition, effort andengagement, social influence and sharing of devices, and fear of making mistakes or unintentional disclosure[5,13,42,52,130]. Voice was a better fit than text for low literacy folks [64] and blind/low-vision folks [156].
Interactive voice response systems were also preferred over SMS in mHealth applications [130]. Still,
Militelo et al. [99] found that a visual display improved satisfaction with the voice system. Social identity
may play a role, as using the voice of a real doctor led to increased perceptions of usefulness and trust forpeople living with HIV [72]. Social context may also be crucial for usability in community [52] and familial[145] contexts; at the least, we must recognize that people may share their voice system with others or use it
together on purpose. Overall, the voice modality was superior to other options from usability and acceptanceperspectives, but, given the possible sensitivity of health information, caution must be paid when multiple
users (or bystanders) may be involved.
3.3.6 Multi-User Contexts.

Privacy concerns [34,40,130,159] are paramount in multi-user and public contexts. Trajkova and MartinHammond [159] and Adaimi et al. [1] reported hesitation to use VUIs in the presence of others or public,
shared spaces. The social robot Fribo was tested in a four-week field study, which showed that frequent realworld social interactions through activity sharing lessened the feeling of privacy intrusion [70]. A voicesystem sharing ―living noise‖ and daily activity information with close friends improved feelings of social
connectedness between them [70]. Using multiple voices to generate the impression of multiple agents

ACM Comput. Surv.

increased the sense of being in a team, even though a single agent was quicker and easier to remember [176].
Age must also be considered; for example, appropriate speech patterns in a voice system for children were
found to position the VA as a near-peer [6]. Relatedly, Haberer et al. [52] found that parents of HIV-positive
children shared their phones with each other and potentially other people, indicating that we should be
careful about relying on a ―solo user‖ model even with ―personal devices.‖ The ways in which voice UX
plays out in multi-user and multi-agent contexts may be a key trajectory in future research. In short, voicesystems may be used socially or affected by the social context, with longer-term engagements easing users
away from privacy concerns—for good or ill.
3.3.7 User Mental Models of Anthropomorphism and Social Identity.

Voice appears to be crucial for establishing an agent‘s social identity and humanlikeness. Even when weknow better, we often cannot help but attribute social and human-like characteristics to the machine [115].
People seem to prefer natural, human-like voices over artificial agents [81,121]. The use of natural speech
can lead to anthropomorphizing [154]. With Alexa, users tended to use polite expressions such as ―thankyou‖ and ―please,‖ similar to natural conversations [91]. Kim et al. [75] identified personality traits thatpeople desire in voice agents: empathizing, trustworthiness, submissiveness, and smart-yet-modest, as well
as distinctive traits that express individuality and neutral traits, such as adapting to the preferences of theuser. Pitardi and Marriott [115] showed that perceptions of privacy and trust were formed based on the brandand producer behind the VA (e.g., Alexa being made by Amazon). Perceived privacy may thus depend not
only on the design of the VA but also on the brand image connected to it. Personalization of the voiceinteraction might increase acceptability. For example, Doering et al. [37] reported that users wanted to berecognized by the system, and the personalized version was preferred in a study by Kassavou and Sutton[73]. However, individual factors seem to play a role; for example, a system with a personality was
perceived well by some but not by others [90]. Altogether, people have a tendency to anthropomorphize,
especially when a human feature like voice is present, and we must take care to understand what mental
models of agency and social identity this invokes—and then whether or not to modify properties of the voice
to shift those models for the user, task, and/or context.
3.3.8 User Mental Models of Embodiment and Form Factors.

The physicality of the system in terms of embodiment and form factor seems to be important for optimal UX.
However, whether physicality is advantageous or not seems to depend on its quality and the purpose of theinteraction. Beneteau et al. [6] noted that close physical and social proximity helped to instill trust in thesystem. However, Pollman et al. [119] found that a robot‘s body motions were perceived as being confusing
and concluded that these should be avoided if they cannot be improved. Similarly, Shamekhi et al. [146]found that the value of having a VUI with a face depended on the type of assistance that the system provided.
Doering et al. [37] noted that users found it easier to accept the voice over the body of their humanoid mobile
shopping robot. Furthermore, Damen and Toh [34] studied the role of voice gender, concluding that it wasmore important ―what‖ something was said than ―how‖ (e.g., gendered voice). Guzman [51] explored what
the ―source‖ of the voice was perceived to be, finding that most identified the source as the VA itself or the
enabling device. Relatedly, participants in Pitardi and Marriott [115] were aware that Amazon, rather thanACM Comput. Surv.

the agent per se, was ―listening in‖ during interactions with Alexa. Lee et al. [88] found that participants
often included the speaker form factor within the agent design when drawing the ―body‖ of voice-based CAs.
The voice system‘s physical attributes might also be key to the perception of robots as socially present
agents [40]. Edwards et al. [40] reported that observable nonverbal codes were important to convey affectalongside voice. Seo et al. [143] identified a range of nonverbal cues accompanying voice during rapportbuilding and rapport-breaking scenarios with an industrial robot. Voice interaction supported verbal and nonverbal rapport-building, resulting in compliments but also in terse speech and even silence, indicating that
the VA was treated like a social agent [143]. Ho et al. [61] showed that the combination of movements andverbal commands of a robot increased its social presence. Additionally, grade of assistance by way of voiceseems to be another factor [171]. Physicality may also offer communicative modalities through nonverbal
cues [165]. Nevertheless, this may not be universal or accessible for blind/low-vision user groups.
In sum, the presence or absence of a ―body‖ for the voice and that body‘s form factor can haveramifications for trust, understandability, applicability to the task and context, and acceptance. A variety of―bodies‖ are available or imaginable against the range of task and contexts applicable to voice interaction.
Determining ideal—and not-so-ideal—intersections is an undertaking that requires a mix of qualitative andquantitative approaches.
3.3.9 Vocal Factors.

One of our aims was to determine whether and how voice UX is distinguished from UX in general. Many ofthese are related or adjacent to conversation and speech, but not always. For example, Schreitter and Krenn[136] identified characteristics of dialogues by examining how people verbally teach a robot, including
backchanneling, like ―ah,‖ sentence fragments, repair statements, contractions, repetitions, interruptions, theuse of time markers like "now,‖ and verbal references to the environment. In the overall corpus of work
surveyed, highlights relevant to voice include: backchanneling [136] prosody, including rhythm and monotony [57] humanlike versus synthetic voice qualities [81]
 vocal expressions of im/politeness [125]Speech-level features and utterances (speech content) appear to intersect with voice in important ways, aswell. For instance, Martelaro et al. [93] found that a combination of vulnerable speech content andparalinguistics led to a greater number of personal disclosures from participants. Highlights from thesurveyed work relevant to speech include: dialogue-level features [121,136,177]
 time, including length of exchanges [162], long-turns [33], and time of day [7] disclosures [93]
 reassurance [128]

Embodiment remains a key issue for voice systems, with several factors of embodiment unearthed or
perhaps best explored with qualitative methods. Key factors in embodiment include:

ACM Comput. Surv.

 source orientation [51]
 richness of communication through supplementary modalities [3]Qualitative voice UX work also revealed several topics relevant to interaction styles. Highlights include: hands-free interaction [99,128]
 multi-tasking via other modalities [1,99,128,144]
 hands-free interaction [1] enabling multitasking [99] (intersection)
 voice enabled by voice [166]Communication maintenance also arose as a key topic. We hazard to characterize this topic as―relational,‖ given that systems do not yet have the capacity for partner modelling, empathy, emotion, orother hallmarks of independent social intelligence. Still, voice UX often involves conversational formats andactivities, and maintaining communication in the short and long terms, especially when problems arise orwhen users misbehave, is of ongoing concern. For example, Rehm [125] found that, when placed in asituation where the voice-based agent was deceptive, some participants responded with verbal abuse andother impolite, if not unacceptable (at least among people) behaviours. We highlight:
 habit- and relationship-building [40] and future expectations for the relationship with the system [25,83]
 negative and possibly abusive reactions [125] conversation exchange and repair [39] missed, unintentional, and unfamiliar intents and prompts [103,169]
 other issues with voice-based system performance (e.g., errors, missed activation) [40,42,103,169]

This likely does not represent the full range of voice UX factors that qualitative research can reveal. But itshows how qualitative inquiry can unearth and richly describe factors that may otherwise be missed.
4

DISCUSSION

Qualitative approaches are rapidly being taken up, as indicated by the spread of the work over time and therecent upward swing (Figure 1). This echoes the ―vocaloid shift‖ recently identified for quantitative work on
voice UX [139]. What does the qualitative side add? In Table 4, we summarize the relationship between
factors of voice UX and the qualitative approaches used in the corpus of work so far. We now turn to making
sense of the state of affairs, focusing on the patterns of experience and notable phenomena highlighted bythis body of work. We discuss the value of qualitative approaches for voice UX in light of researchers‘
reasons, orientations to qualitative inquiry, the techniques employed, and the outcomes so far mapped out.
We finally turn to the weaknesses, gaps, and opportunities identified through this review and how to address
them in future qualitative work on voice UX.

ACM Comput. Surv.

Table 4: Summary of relationship between voice UX factors and qualitative approaches
Voice UX Factors
Usability

Experiential
Agent Embodiment

Attitudinal
Acceptability
Privacy &TrustSocialVocal

Qualitative
Narrative
Phenomenology[1,21,39,61,103,16
9][1,3,25,61,70,75,83,
88,91,121,136,143,
154,176][1,70,154][38,39,61,70,121 [1,3,7,25,5,143,154,172,17 1,83,121]
6]

Grounded Theory[103,128]

[70]

[70]

[70]

[70]

[7,51,128]

Ethnography[13,73,98,123,128,
130,159,163,168][6,54,144,159][6,13,54,73,130,144,162,163,168]

[70]

[128]

Case Study[42][6,10,13,54,72,7
3,98,123,130,13
4,144,159,162,1
63]
[42]

Mixed Methods
Triangulation[33,99][37,40,113,119,171,
177][37,40,64,99,113,17
1][37,64,99][33,177][11,52,74,126,156][34,81,146][5,52,81,90,93,126][34][5,52,93,126,145,156,166][57,93]

Expansion[173][125,173]

[115]

[115]

Other

[18]

Complementary[42]

Development
Initiation[18]

4.1 Why Qualitative Inquiry? The Value for Voice UX

Voice is a qualitative medium, and voice interaction takes advantage of its qualities in technology used bypeople. It should then come as no surprise that most of the reasons behind exploring voice systems
qualitatively appear to match the reasons for exploring voice quantitatively [24,139,141]. Yet, qualitative
approaches to voice aim for different forms of outcomes and raise new perspectives. Voice characteristics,
linguistic utterances and conversation, non-linguistic utterances and vocal sounds, and even singing … these
are all non-numerical forms of expression that have qualitative implications for HCI. We summarize thevalue of qualitative inquiry for voice UX across phenomenological, methodological, and ethical factors in
Table 5. As our findings show, qualitative inquiry can reveal as well as deeply illustrate specific features ofvoice UX. But the voice is only one factor. The eyes-free and hands-free interaction opportunities, forinstance, have implications for multi-tasking and people with disabilities, while also raising challenges forinteraction design. How can we leverage voice input for storage or word processing, especially when wewish to review and revise our recordings? Voice is ephemeral. Adding supplementary modalities, such as
text and visuals, can fix some problems but introduce others. Suddenly the power of an audio-only mode ofinteraction is disrupted.

ACM Comput. Surv.

Table 5: Summary of factors revealed by qualitative inquiry for voice UX
Voice UX Factors
PhenomenologicalExperiential

Qualitative Factors of Voice UX
Eyes-free interaction; hands-free interaction; ephemerality of voice input/output; trade-offs with
supplementary modalities and multi-modality; longitudinal UX and long-term engagement; richness andcomplexity of real contexts of use; anthropomorphism; role of the body; diversity in individual responses

Attitudinal

Mental models, esp. over time and with changes to the voice or underlying technologies; orientations to andrelationships with the system; relative awareness of trust and privacy issues, esp. with voice cloning andvoice impersonation

SocialExperience of people with disabilities and marginalized populations; multiple users at the same or different
times; peripheral users who may unintentionally ―listen in‖

Vocal

Vocalics and voice characteristics; utterances and conversation; non-linguistic utterances; vocalizations andvocal sounds

Methodological

Mixed methods; overreliance on a few approaches to qualitative inquiry and data collection; limited tasks
and activities

EthicalTrust and over-trust; privacy and security, esp. data storage, transcripts, who has access or can ―overhear‖;voice cloning and impersonation; task designed as one-way prescription; decision-making around changing
technologies that people have become familiar with and reliant on; disruptions caused by the presence ormismatch between voice and body; handling disclosures of sensitive information

As the corpus of work reviewed here shows, qualitative research often takes place within real-worldcontexts over long spans of time, unlike quantitative work. While this can lead to more ecologically valid
findings [20], it also sparks new lines of inquiry related to time, notably: how expectations for and mental models of the device evolve whether and how time of day or day of the week matters [149] whether a relationship with the system comes into being, how, and what its nature is over time

What if the voice system changes—what if a new version comes out? If the default voice is made more
realistic or is re-voiced, does it become more acceptable because anthropomorphism is ideal? Or will it berejected, just as Sir. Stephen Hawking rejected an ―updated voice‖ for his voice generator?11 We can begin
to map out not just the quantitative ―if‖ but also the qualitative ―why‖ and ―how‖ to such questions.
Qualitative inquiry was often found deployed hand-in-hand with quantitative approaches: the mixed
methods format. Yet, the majority of the work employed the qualitative side in similar and potentiallylimited ways:
 Most qualitative approaches served as complements to the main quantitative strand
 Most employed interviews as a technique of data collection, echoing the notion of a practical, positivistleaning semi-structured take to qualitative inquiry [8]A few others, however, approached qualitative inquiry in other ways that could be accelerated:11

https://www.npr.org/2018/03/27/597390626/an-engineers-quest-to-save-stephen-hawkings-voice

ACM Comput. Surv.

 Taking on expansive approaches, where the qualitative component acts as a provocation or corrective
measure Taking advantage of the vocal format, particularly conversational scripts that emerge during interaction

Indeed, deriving new intersections between qualitative and quantitative forms of data and metadata may
be necessary to understand the relationship between the voice and other matters of embodiment. For
instance, we are now able, with some degree of accuracy, to mock up uncanny visions of the face of thevoice [109]12. Machine learning is making these ―vocal deepfakes‖ possible: virtual clones of voices that cantrick human ears by reproducing the qualitative features of a person‘s voice [15]. While several works in thisreview considered matters of security and privacy, highlighting the notion of unintentional third-party use,e.g., overhearing, transcript access, sharing phones, few explored the user identification front. Are people
aware that their voice could be used to reveal who they are? This is not a matter of deriving identity fromwhat are largely social constructs. We are living in a world of big data, where we freely offer our personal
bits and bytes to the world, often without realizing the implications [84] or preserving control [2]. We can
expand the qualitative orchestra, in concert with quantitative methods, for methodological harmony anddeeper engagement with such emerging matters of great concern in voice UX.
A critical look at the findings reveals some patterns that deserve interrogation, which we highlight asfollows:
 Much of the work deployed voice as a means of compliance and prescription through one-way lectures,e.g., [5,42,72,130,145,163,173]. This was especially the case in medical interactive voice response
systems. From a technical perspective, it is not hard to imagine why this may be common. Displaying
information, including through an audio channel, is far easier than developing an interactive experience,
which typically requires the use of advanced techniques like NLP. Yet, this becomes an issue of powerand calls into question the descriptor of “interactive.” Deeper inquiry into the body of the voice is warranted. Robotic bodies, for instance, may not benecessary. As recent work in HRI has shown, a physical form factor may even introduce newchallenges, both technically and in terms of optimizing UX, and especially when not designed to matchthe voice [140,160].
 When it comes to embodiment, phenomenological approaches with a mixed methods component that
triangulates quantitative and qualitative data about the body may be ideal (refer to Table 4).
 Voice systems thought to be “bodiless,” such as smart speakers, may not be interpreted that way by
everyone, leading to different assumptions about source and who may be “listening in,” as some of thework included in this survey found [70,98,145,159,163,168]. This creates a tension when considering thedesires of many to have a more personalized experience with voice-based agents. How can we enable
personalization and recognition of the user while also ensuring privacy and security?

12

https://speech2face.github.io

ACM Comput. Surv.

 Voice interaction with multiple people can introduce new shifts in the perceived embodiment of thedevice. When merely a purveyor of human-human communication, does the voice system fold into thebackground? Or does it remain a third party, however sidelined?
 Social identity can play a role. Voice agents that use a real human voice known to the user/s may build
trust and help overcome initial barriers to use, as Joshi et al. [72] found. But what happens when thevoice breaks down socially rather than technically [6,40,70,115]? Also, does experience with the“impersonator” over time go on to influence future experiences with the real deal? The influence may goboth ways. Qualitative research can bring these multiple strands of experience together against trust
and privacy factors (refer to Table 4).
 People tend to prefer anthropomorphic forms of embodiment, not just in terms of voice but also bodyand behaviour. This has implications when people treat these voice agents in negative ways … or whenan agent treats the user poorly, as Rehm explored [125], even if it is a technical glitch. Designing agentsthat are vulnerable, as in Martelaro et al. [93], may lead to more disclosures, but what are therepercussions for the person? Do they realize what they are disclosing and to whom? Are roboticists,
researchers, companies, governments, and other stakeholders willing and prepared to handle sensitive
information?

Reflexivity is a powerful method of self-discipline and self-discovery often employed in qualitative
research [129]. We can take up this method to shift the scope of inquiry beyond the user to otherstakeholders who must also grapple with voice UX and highlight ethical issues on the peripheries. This will
require drawing from the literatures outside of computer science and engineering. Biology and neuroscience
have explored the possible evolutionary emergence of human conversation for the purpose of creating
persuasive narratives so that interlocutors can collaborate and develop shared mental models. The selective
advantage offered by grammatically complex codes may have driven the shift away from the simpler speechof our primate ancestors [44]. Whether ideas are abstract or concrete, an affective dimension appears to playa role across human languages [170]. Voices can also be used to signal social identity and reinforce social
norms [76], a topic that needs greater exploration in voice UX. Efforts such as harmonization and co-creation
with intelligent agents that use voice but potentially other auditory modalities, including music and nonvocal sounds, are on the horizon [175]. This is where a breadth of work from the humanities and social
sciences can enter technical fields and drive new questions and collaborations [48,148]. Indeed, there is alarge base of literature that is qualitative in nature. For example, we may be inspired by new, critical,
qualitative analyses on the racialization of voice and sound, reflecting on our tendency to design around andresearch acousmatic questions about machine identity by way of the voices being embedded in agentic andanthropomorphic form factors that shape our reactions and interactions [41]. Future work could craft a
research agenda for cross-pollination across different fields of study that have already produced bodies ofwork on voice, even if not on computer-based interactive voice systems specifically. There is much we canlearn here.
4.2 Opportunities for Increasing Rigour and ComprehensivenessQualitative approaches to understanding voice UX add value to the quantitative work and have the potential
to generate special insights and knowledge. Yet, our review unearthed several weaknesses and gaps that

ACM Comput. Surv.

should be addressed. We outline the major points of concern, and opportunities, for increasing rigour inqualitative approaches to voice UX:
4.2.1 Report All Demographics.

The ―who‖ of qualitative work is often as important or more so than the ―what.‖ Yet, the ―who‖ was at times
underreported or not reported at all. For instance, gender demographics of 512 participants across 15 papers,
almost one-quarter of the work (22.7%), was not made available. This is not simply a matter ofmethodological rigour, but a matter of inclusion. As critical scholarship has pointed out, human variables
such as gender, age, and race are not only captured for data analysis or even to understand the phenomenon
under study, but also to ensure accountability, fairness, and diversity [49,135]. Reporting on thesedemographics is also important for comparison and consensus-building, even to understand the extent towhich generalizations that are repeated within one population may not hold across others. We recognize thatit may not always be appropriate to ask about sensitive demographics. Qualitative work can be intimate, anddemographics reporting in studies involving small numbers of individuals could risk unwanted and/or
unethical identification. Still, a statement on why certain demographics were not captured would be ideal.
Even if one takes the position that qualitative work is not generalizable, we must consider the case of
mixed methods, which makes up a significant portion of the work. Is voice UX research WEIRD, with user
populations drawn from Western, Educated, Industrial, Rich, and Democratic nations [58,59,89,97,137]? We
cannot answer such questions without sufficient reporting of demographics. Additionally, it is surprising,
given recent work in HCI spaces (e.g., [69,151]), that there were no participants who identified outside of thegender binary. Researchers can refer to recent work on how to best capture such multifarious variables inaccurate and inclusive ways. Gender is particularly tricky because it is often not defined by or forparticipants [138,150] and is used interchangeably with sex and sometimes sexuality [65]. Gender as a social
construct can shift over time and is culturally sensitive. It can be difficult for some people to answer, as they
may be unsure of their gender identity, or it may be risky for them to volunteer that information. Response
options for gender should aim to distinguish it from sex and allow for a range of responses that are not forced[150]. Otherwise, we may generate invalid data and cause discomfort or worse in participants.
4.2.2 Fully Describe the Technology and Voice(s).

While the basic type of technology, one that uses voice for interaction, was generally well-reported, there
were notably gaps when it came to the specifics of the systems under study. For instance, it was not clear in
one-third of cases whether a natural, human voice was used or a synthetic one, and to what degree the voice―sounded‖ synthetic or ―real.‖ Almost one-quarter did not report on the TTS used, possibly because theresearchers believed that it was implied by the system. It may also have been the voice of the researchers
themselves, i.e., Wizard of Oz, which is not always disclosed. Additionally, the intelligence underlying thevoice was not reported almost half of the time (43.9%), possibly for similar reasons as above. Yet, what thesystem can do and how it behaves with the people is the crux of HCI research. If Wizard of Oz or simulated
intelligence was used, or if only part of the system was automated, this should be reported.
Investigating ideal interactions is important for designing the future of voice UX. Still, we need to
distinguish between what is possible now and what is desired for then. This will also open up pathways forcomparing different voice systems and forms of voice interaction. Notably, not many works compared

ACM Comput. Surv.

systems, even commercial platforms like Amazon Alexa and Google Home. Qualitative work may notnecessarily allow for comparative research designs or aim for generalizability. Nevertheless, some do, andwe should not forget about mixed methods forms of inquiry. Full reporting may allow for more fruitful forms
of inquiry or post hoc consensus building in follow-up review work.
4.2.3 Clearly State the Qualitative Approach and Epistemological Assumptions.

Most papers (49 or 74.2%) did not explicitly label or describe the qualitative approach undertaken. This
limited our ability to categorize the work for this review, but it also limits our understanding of themethodology and epistemological underpinnings of the researchers. Epistemology, or our position on whatknowledge is and how much we can have access to it, is in some sense a matter of personal philosophy. At
the same time, our epistemological stance has clear implications for research [29,32], which is a social
practice. A common and pertinent challenge in negotiating qualitative and quantitative research, especially ina mixed methods frame [29], is positivism versus everything else. Positivism in research is characterized bysystematic processes involving precise and replicable measurement of a phenomenon resulting in aquantifiable outcome to which statistical analyses can be applied and a generalizable outcome obtained. The
generalizability of the outcome tends to be the highlight of this approach. Qualitative research, on the other
hand, is rarely characterized as positivist. Some argue that qualitative outcomes tend not to be generalizable
in practice; others argue that they cannot be generalizable and that is not the aim to begin with; and stillothers argue for consensus and synthesis when appropriate and possible [32,45,118]. For instance, Polit andBeck describe the goal of most qualitative research as being to ―provide a rich, contextualized understandingof human experience through the intensive study of particular cases‖ [118:1452, our emphasis]. They cautionthat we must be careful when generalizing within as well as between qualitative and quantitative traditions.
Creswell and Poth [32], on the other hand, argue for elucidation and transparency (and some degree ofpragmatism). We do not necessarily argue for or against treating qualitative outcomes one way or the other
here. Instead, we wish to call attention to the benefits of, as a first step, clearly describing one‘s position, sothat others can make informed choices and fruitful debate premised in stated positions can then ensue.
4.2.4 Explore and Synchronize Novel and Underused Qualitative Approaches and Techniques.

We identified several forms of qualitative and mixed methods inquiry that appear to be underused, despite
their potential value for voice UX. Narrative studies, for instance, are based in story–and what is aninteraction, especially one that takes place over time, but a story? What is a conversation with a voice-based
system but a tale stripped down to its most common form of communicative exchange? The building blocksof narrative approaches have already started to be explored, notably diaries, dialogues and transcripts, design
outcomes from workshops and jams, and drawing procedures. We highlight these promising techniques in
Figure 3. The outcomes could be used to form a narrative or the techniques themselves could be employedby participants to construct their own narratives about the voice-based systems they use (or may wish to use).
Indeed, we may be on the cusp of a narrative revolution in qualitative approaches to voice UX. For instance,
Ringfort-Felner et al. [127] explored design fiction as a means of mapping out conversations with voiceassistants. Similarly, the case study format was all but unused. Yet, a deep dive into a particular device or
unusual use case with a range of data collection techniques could be instructive for the present and future ofvoice UX.

ACM Comput. Surv.

Grounded theory was also underused, but we are not confident that this was truly the case. Many studies
relied on common techniques used in grounded theory in terms of data collection and analysis, especiallythematic analysis. Yet, they did not refer to grounded theory (in label or reference) and/or did not seem totreat the thematic framework as a theory in the grounded theory sense. This could point to what Blandford[8] has described as a semi-structured approach to qualitative inquiry: harnessing methods from otherdisciplines in a pragmatic fashion to better understand HCI phenomena in a structured way, or at least in asomewhat structured way. The ―semi‖ nature of this approach disrupts expectations for epistemologicalstance, deployment, analysis, and triangulation. Importantly, those who take such an approach should
faithfully record what they have done and how they have drawn their conclusions so that others ―cancomprehend the journey from an initial question to a conclusion, assess its validity and generalizability, andbuild on the research in an informed way‖ [8:3]. Blandford suggests that this is in some part a naming
problem, i.e., there may not be a label for the specific approach used due to deviations from the original
methodologies, the pragmatic taking-up of methods from other fields, the mixing of methods and techniques(―bricolage‖), and so on. Nevertheless, we urge researchers to clearly mark their work and refer tofoundational material and methodological guidance on the methods used whenever possible. Labels can beused even in the case of mixing and deviating from established protocols, e.g., ―We used grounded theory
with these changes …‖ Recognition, acknowledgement, and transparency are key.

Figure 3: Four emerging techniques for exploring voice UX qualitatively

On the mixed methodology side, two forms of ―mixing‖ were not found: development and initiation. The
dearth of work taking a development approach may be due to the nascency of voice UX research generally.
We may find that, in time, the results of qualitative work may be used to inform the design of future studies,including quantitative measures, instruments, and research questions. The lack of orientations towards
initiation in mixing methods is harder to explain. Yet, the goal is a bit counterintuitive: rather than seeking
stable patterns, meaning, and consensus, initiation-based approaches aim for provocations and unearthing
contradictions that cannot be easily explained–or found.

ACM Comput. Surv.

We highlight these frames for future research. Indeed, there is great opportunity to mix the qualitative andquantitative in new and provocative ways. Where possible, we should aim to use the same tools, and validate
these tools, when investigating a particular phenomenon of study or measure of experience. Otherwise, we
may not be able to determine if any change is due to our tools or our representation of reality as researchers
and/or users. A 10-year retrospective on a single tool could be enlightening.
4.3 Ask the Same Questions About the Same Notions? Towards a Mixed Methods Future

Knowledge creation is a multifaceted, social activity that benefits from a plurality of perspectives andresearch activities. Qualitative research is often valued for its ability to find something new rather thanconfirm what may already be known. Indeed, as our synthesis of findings has shown, the qualitative researchon voice UX to date has illuminated an array of phenomena primed for further inquiry–whether qualitative,
quantitative, or some combination of the two. Moreover, future synthesis on a given phenomenon would
benefit from shared foundations. Specifically, we mean theoretical or conceptual bases and researchquestions. Similar to a recent quantitative survey [139], nearly half of studies did not reference theory or
employ a theoretical model (46% versus 40.9%). Qualitative research is often theory-generating rather thantheory-confirming, which could explain this. Still, only five studies (7.6%) were explicitly oriented this way,i.e., employed a grounded theory approach. It is possible that more of this work could be used in a theorygenerating way based not on the qualitative orientation but rather on data analysis methods used, especiallythematic analysis. Follow-up work could trace how the results of such qualitative work has subsequently
been used to guide theorizing, confirmatory studies, or further exploration … and then use it. Employing
different methods to explore the questions derived from qualitative inquiry and/or the same questions in a
multi-methods manner may be effective for discovery activities but also would make for ease of comparison
between experimental and non-experimental survey work at a high level in future. We do not suggest
limiting research questions per se, but rather strive for a combination of novelty and stability in forms ofinquiry, with a view to present and future comprehensiveness. Finally, we did not have the scope in thispaper to survey and compare the qualitative and quantitative literatures. Future work can draw on and update
existing open data sets on these literatures and carry out these analyses.
5

CONCLUSION

Voice UX represents a modern form of engagement with computer systems inspired by one of humanity‘slongest standing and perhaps essential, forms of self-expression and communication. Experiences with voiceinteraction are qualitative, and while quantifiable, these subjective, non-numerical engagements deserve
appropriate inquiry. We have mapped out the state of affairs on this topic, demonstrating the what and howof qualitative approaches to understanding voice UX across a range of computer-based agents, interfaces,
environments, and systems–as well as the why. We have highlighted pertinent findings, from human factors
especially relevant to voice UX, general patterns in UX across the corpus of research so far, and trajectories
for future research and theory-building. Now that we have mapped out the what, why, and how, we
encourage voice UX researchers to join us in considering and justifying the where and when of incorporating
qualitative forms of inquiry into our work.

ACM Comput. Surv.

ACKNOWLEDGMENTS
We thank the editor and reviewers for their service. This work was funded by a Japan Society for the Promotion of
Science (JSPS) Grant-in-Aid for Early-Career Scientists (KAKENHI Wakate) (Grant no. 21K18005).

REFERENCES[1]

Rebecca Adaimi, Ka Tai Ho, and Edison Thomaz. 2020. Usability of a hands-free voice Input interface for ecological momentary assessment. In
2020
IEEE
International
Conferenceon
Pervasive
ComputingandCommunications
Workshops,
2020.
https://doi.org/10.1109/PerComWorkshops48775.2020.9156226[2]Mark Andrejevic. 2014. Big data, big questions| The big data divide. Int. J. Commun. 8, 0 (June 2014), 17.

[3]

Mirzel Avdic, Nicolai Marquardt, Yvonne Rogers, and Jo Vermeulen. 2021. Machine body language: Expressing a smart speaker‘s activity withintelligible physical motion. In Designing Interactive Systems Conference 2021 (DIS ’21), 2021, New York, NY, USA. Association for Computing
Machinery, New York, NY, USA, 1403–1418. https://doi.org/10.1145/3461778.3462031[4]

Ameer A. Badr and Alia K. Abdul-Hassan. 2020. A review on voice-based interface for human-robot interaction. Iraqi J. Electr. Electron. Eng. 16, 2(2020). Retrieved May 28, 2021 from https://www.iasj.net/iasj/article/195932[5]

Angela N. Bailey, K.J. Porter, J.L. Hill, Y. Chen, Paul A. Estabrooks, and J.M. Zoellner. 2016. The impact of health literacy on rural adults‘satisfaction with a multi-component intervention to reduce sugar-sweetened beverage intake. Health Educ. Res. 31, 4 (2016), 492–508.
https://doi.org/10.1093/her/cyw024[6]Erin Beneteau, Yini Guan, Olivia K. Richards, Mingrui Ray Zhang, Julie A. Kientz, Jason Yip, and Alexis Hiniker. 2020. Assumptions checked:
How families learn about and use the Echo Dot. Proc ACM Interact Mob Wearable Ubiquitous Technol 4, 1 (March 2020).
https://doi.org/10.1145/3380993[7]Frank Bentley, Chris Luvogt, Max Silverman, Rushani Wirasinghe, Brooke White, and Danielle Lottridge. 2018. Understanding the long-term use ofsmart speaker assistants. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 3 (September 2018), 91:1-91:24.
https://doi.org/10.1145/3264901[8]Ann E. Blandford. 2013. Semi-structured qualitative studies. In The Encyclopedia of Human-Computer Interaction (2nd ed.), Mads Soegaard and
Rikke Friis Dam (eds.). Interaction Design Foundation, Aarhus, Denmark. Retrieved from http://www.interaction- design.org/encyclopedia/semistructured_qualitative_studies.html[9]Jeannette Blomberg, Mark Burrell, and Greg Guest. 2003. An ethnographic approach to design. In The Human-Computer Interaction Handbook:
Fundamentals, Evolving Technologies and Emerging Applications, J.A. Jacko and A. Sears (eds.). Lawrence Erlbaum Associates, Inc., Mahwah, NJ.

[10] Hannah Louise Bradwell, Katie Jane Edwards, Rhona Winnington, Serge Thill, and Ray B. Jones. 2019. Companion robots for older people:
Importance of user-centred design demonstrated through observations and focus groups comparing preferences of older people and roboticists inSouth West England. BMJ Open 9, 9 (2019). https://doi.org/10.1136/bmjopen-2019-032468[11] Michael Braun, Jonas Schubert, Bastian Pfleging, and Florian Alt. 2019. Improving driver emotions with affective strategies. Multimodal Technol.
Interact. 3, 1 (2019). https://doi.org/10.3390/mti3010021[12] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qual. Res. Psychol. 3, 2 (2006), 77–101.
https://doi.org/10.1191/1478088706qp063oa[13] J. Brinkel, P. Dako-Gyeke, A. Kramer, J. May, and J.N. Fobil. 2017. An investigation of users‘ attitudes, requirements and willingness to use mobile
phone-based interactive voice response systems for seeking healthcare in Ghana: a qualitative study. PUBLIC Health 144, (March 2017), 125–133.
https://doi.org/10.1016/j.puhe.2016.11.017[14] Jeffrey A. Brooks, Panagiotis Tzirakis, Alice Baird, Lauren Kim, Michael Opara, Xia Fang, Dacher Keltner, Maria Monroy, Rebecca Corona, Jacob
Metrick, and Alan S. Cowen. 2023. Deep learning reveals what vocal bursts express in different cultures. Nat. Hum. Behav. 7, 2 (February 2023),
240–250. https://doi.org/10.1038/s41562-022-01489-2[15] Matt Burgess. The race to hide your voice. Wired. Retrieved June 6, 2022 from https://www.wired.com/story/voice-recognition-privacy-speechchanger/[16] Julia Cambre, Jessica Colnago, Jim Maddock, Janice Tsai, and Jofish Kaye. 2020. Choice of voices: A large-scale evaluation of text-to-speech voice
quality for long-form content. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), April 21, 2020,
Honolulu, HI, USA. Association for Computing Machinery, Honolulu, HI, USA, 1–13. https://doi.org/10.1145/3313831.3376789[17] Julia Cambre and Chinmay Kulkarni. 2019. One voice fits all? Social implications and research challenges of designing voices for smart devices.
Proc. ACM Hum.-Comput. Interact. 3, CSCW (November 2019), 223:1-223:19. https://doi.org/10.1145/3359325[18] Alex Carmichael, Helen Petrie, Fraser Hamilton, and Jonathan Freeman. 2003. The vista project: Broadening access to digital TV electronic
programme guides. PsychNology J. 1, 3 (2003), 229–241.
[19] Justine Cassell, Joseph Sullivan, Scott Prevost, and Elizabeth Churchill (Eds.). 2000. Embodied Conversational Agents. MIT Press, Cambridge, MA.
[20] Alan Chamberlain and Andy Crabtree. 2020. Research ‗In the Wild.‘ In Into the Wild: Beyond the Design Research Lab, Alan Chamberlain andAndy Crabtree (eds.). Springer International Publishing, Cham, 1–6. https://doi.org/10.1007/978-3-030-18020-1_1[21] Janghee Cho and Emilee Rader. 2020. The role of conversational grounding in supporting symbiosis between people and digital assistants. Proc.
ACM Hum.-Comput. Interact. 4, CSCW1 (2020). https://doi.org/10.1145/3392838[22] K. R. Chowdhary. 2020. Natural Language Processing. In Fundamentals of Artificial Intelligence, K.R. Chowdhary (ed.). Springer India, New Delhi,

ACM Comput. Surv.

603–649. https://doi.org/10.1007/978-81-322-3972-7_19[23] Herbert H. Clark. 1996. Using Language. Cambridge University Press.
[24] Leigh Clark, Philip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl, Jens Edlund, Matthew Aylett, João Cabral, Cosmin Munteanu, Justin
Edwards, and Benjamin R Cowan. 2019. The state of speech in HCI: Trends, themes and challenges. Interact. Comput. 31, 4 (June 2019), 349–371.
https://doi.org/10.1093/iwc/iwz016[25] Leigh Clark, Cosmin Munteanu, Vincent Wade, Benjamin R. Cowan, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards,
Brendan Spillane, Emer Gilmartin, and Christine Murad. 2019. What Makes a Good Conversation?: Challenges in Designing Truly Conversational
Agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19 (CHI ’19), 2019, Glasgow, Scotland Uk.
ACM Press, Glasgow, Scotland Uk, 1–12. https://doi.org/10.1145/3290605.3300705[26] Michael Harris Cohen, James P. Giangola, and Jennifer Balogh. 2004. Voice User Interface Design. Addison-Wesley Professional.
[27] Michael F. Connelly and Jean D. Clandinin. 1990. Stories of experience and narrative inquiry. Educ. Res. 19, 5 (June 1990), 2–14.
https://doi.org/10.3102/0013189X019005002[28] Andrew Crabtree, Tom Rodden, Peter Tolmie, and Graham Button. 2009. Ethnography considered harmful. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI ’09), April 04, 2009, New York, NY, USA. Association for Computing Machinery, New
York, NY, USA, 879–888. https://doi.org/10.1145/1518701.1518835[29] John W. Creswell. 2008. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (3rd ed.). Sage, Thousand Oaks, CA.
[30] John W. Creswell and Vicki L. Plano Clark. 2017. Designing and Conducting Mixed Methods Research (3rd ed.). SAGE Publications, Newbury
Park, CA. Retrieved from https://us.sagepub.com/en-us/nam/designing-and-conducting-mixed-methods-research/book241842[31] John W. Creswell and Vicki L. Plano Clark. 2018. Choosing a mixed method design. In Designing and Conducting Mixed Methods Research (3rded.). SAGE Publications, 53–106. Retrieved May 19, 2022 from https://us.sagepub.com/sites/default/files/upm-binaries/35066_Chapter3.pdf[32] John W. Creswell and Cheryl N. Poth. 2016. Qualitative Inquiry and Research Design: Choosing Among Five Approaches (4th ed.). Sage, Thousand
Oaks, CA.
[33] Katharina Cyra and Karola Pitsch. 2017. Dealing with ‗long turns‘ produced by users of an assistive system: How missing uptake and recipiency leadto turn increments. In 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), September 28, 2017.
329–334. https://doi.org/10.1109/ROMAN.2017.8172322[34] Nicole B. Damen and Christine A. Toh. 2018. Designing for trust: Understanding the role of agent gender and location on user perceptions of trust in
home automation. J. Mech. Des. 141, 6 (2018), MD-18-1340. https://doi.org/10.1115/1.4042223[35] Norman K. Denzin and Yvonna S. Lincoln (Eds.). 1994. Handbook of Qualitative Research. Sage Publications, Thousand Oaks, CA, US.
[36] Tamara Dinev and Paul Hart. 2006. An extended privacy calculus model for e-commerce transactions. Inf. Syst. Res. 17, 1 (March 2006), 61–80.
https://doi.org/10.1287/isre.1060.0080[37] Nicola Doering, Sandra Poeschl, Horst-Michael Gross, Andreas Bley, Christian Martin, and Hans-Joachim Boehme. 2015. User-centered design andevaluation of a mobile shopping robot. Int. J. Soc. Robot. 7, 2 (April 2015), 203–225. https://doi.org/10.1007/s12369-014-0257-8[38] Yasemin Dönmez, Naz A. G. Z. Börekçi, and Mathieu Gielen. 2018. ―Wow he is talking!‖: A study on intrinsic motivations for child-robotic toy
interactions. In Proceedings of the 17th ACM Conference on Interaction Design and Children (IDC ’18), 2018, New York, NY, USA. Association
for Computing Machinery, New York, NY, USA, 349–356. https://doi.org/10.1145/3202185.3202756[39] Yao Du, Kerri Zhang, Sruthi Ramabadran, and Yusa Liu. 2021. ―Alexa, what is that sound?‖ A video analysis of child-agent communication fromtwo Amazon Alexa games. In Proceedings of the 2021 International Conference on Interaction Design and Children, 2021. 513–520.
https://doi.org/10.1145/3459990.3465195[40] Bosede I. Edwards, Idris O. Muniru, Nosiba Khougali, Adrian D. Cheok, and R. Prada. 2019. A physically embodied robot teacher (PERT) as a
facilitator for peer learning. 2019. https://doi.org/10.1109/FIE.2018.8658445[41] Nina Sun Eidsheim. 2019. The Race of Sound: Listening, Timbre, and Vocality in African American Music. Duke University Press.
https://doi.org/10.1215/9781478090359[42] Ben Ellway. 2016. Design vs practice: How problematic call routing and rerouting restructures the call centre service system. Int. J. Oper. Prod.
Manag. 36, 4 (2016), 408–428. https://doi.org/10.1108/IJOPM-11-2013-0487[43] Nicholas Epley, Adam Waytz, and John T. Cacioppo. 2007. On seeing human: A three-factor theory of anthropomorphism. Psychol. Rev. 114, 4(2007), 864.
[44] Francesco Ferretti and Ines Adornetti. 2021. Persuasive conversation as a new form of communication in Homo sapiens. Philos. Trans. R. Soc. B
Biol. Sci. 376, 1824 (March 2021), 20200196. https://doi.org/10.1098/rstb.2020.0196[45] William A. Firestone. 1993. Alternative arguments for generalizing from data as applied to qualitative research. Educ. Res. 22, 4 (May 1993), 16–23.
https://doi.org/10.3102/0013189X022004016[46] Susan T. Fiske, Amy J. C. Cuddy, and Peter Glick. 2007. Universal dimensions of social cognition: Warmth and competence. Trends Cogn. Sci. 11, 2(February 2007), 77–83. https://doi.org/10.1016/j.tics.2006.11.005[47] W. Tecumseh Fitch. 2017. Empirical approaches to the study of language evolution. Psychon. Bull. Rev. 24, 1 (2017), 3–33.
https://doi.org/10.3758/s13423-017-1236-5[48] Sascha Frühholz and Pascal Belin. 2018. The Oxford Handbook of Voice Perception. Oxford University Press.
[49] Maartje de Graaf, Giulia Perugia, Eduard Fosch-Villaronga, Angelica Lim, Frank Broz, Elaine Schaertl Short, and Mark Neerincx. 2022. Inclusive
HRI: Equity and diversity in design, application, methods, and community. In Proceedings of the 2022 ACM/IEEE International Conference on

ACM Comput. Surv.

Human-Robot Interaction (HRI ’22), 2022, Sapporo, Hokkaido, Japan. IEEE Press, 1247–1249.
[50] Jennifer C. Greene, Valerie J. Caracelli, and Wendy F. Graham. 1989. Toward a conceptual framework for mixed-method evaluation designs. Educ.
Eval. Policy Anal. 11, 3 (September 1989), 255–274. https://doi.org/10.3102/01623737011003255[51] Andrea L. Guzman. 2019. Voices in and of the machine: Source orientation toward mobile virtual assistants. Comput. Hum. Behav. 90, (2019), 343–
350. https://doi.org/10.1016/j.chb.2018.08.009[52] Jessica E. Haberer, Julius Kiwanuka, Denis Nansera, Ira B. Wilson, and David R. Bangsberg. 2010. Challenges in using mobile phones for collectionof antiretroviral therapy adherence data in a resource-limited setting. AIDS Behav. 14, 6 (2010), 1294–1301. https://doi.org/10.1007/s10461-0109720-1[53] Neal R. Haddaway and Gary S. Bilotta. 2016. Systematic reviews: Separating fact from fiction. Environ. Int. 92–93, (July 2016), 578–584.
https://doi.org/10.1016/j.envint.2015.07.011[54] Marja Harjumaa, Minna Isomursu, Salla Muuraiskangas, and Antti Konttila. 2011. HearMe: A touch-to-speech UI for medicine identification. 2011.
85–92. https://doi.org/10.4108/icst.pervasivehealth.2011.246120[55] Marc Hassenzahl. 2008. User experience (UX): Towards an experiential perspective on product quality. In Proceedings of the 20th International
Conference of the Association Francophone d’Interaction Homme-Machine, 2008, Metz, France. ACM, Metz, France, 11–15.
[56] Marc Hassenzahl and Noam Tractinsky. 2006. User experience – a research agenda. Behav. Inf. Technol. 25, 2 (2006), 91–97.
https://doi.org/10.1080/01449290500330331[57] Zachary Henkel, Cindy L. Bethel, John Kelly, Alexis Jones, K. Stives, Zach Buchanan, Deborah K. Eakin, David C. May, and Melinda Pilkinton.
2017. He can read your mind: Perceptions of a character-guessing robot. In 2017 26th IEEE International Symposium on Robot and Human
Interactive Communication (RO-MAN), September 28, 2017. 242–247. https://doi.org/10.1109/ROMAN.2017.8172309[58] Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behav. Brain Sci. 33, 2–3 (June 2010), 61–83.
https://doi.org/10.1017/S0140525X0999152X[59] Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. Most people are not WEIRD. Nature 466, 7302 (July 2010), 29–29.
https://doi.org/10.1038/466029a[60] Julian P. T. Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J. Page, and Vivian A. Welch (Eds.). 2019.
Cochrane Handbook for Systematic Reviews of Interventions (2nd ed.). John Wiley & Sons.
[61] Hui-Ru Ho, Bengisu Cagiltay, Nathan Thomas White, Edward M. Hubbard, and Bilge Mutlu. 2021. RoboMath: Designing a learning companion
robot to support children‘s numerical skills. In Interaction Design and Children (IDC ’21), 2021, New York, NY, USA. Association for Computing
Machinery, New York, NY, USA, 283–293. https://doi.org/10.1145/3459990.3460709[62] Donald Horton and R. Richard Wohl. 1956. Mass communication and para-social interaction. Psychiatry 19, 3 (August 1956), 215–229.
https://doi.org/10.1080/00332747.1956.11023049[63] Siying Hu, Hen Chen Yen, Ziwei Yu, Mingjian Zhao, Katie Seaborn, and Can Liu. 2023. Wizundry: A cooperative Wizard of Oz platform forsimulating future speech-based interfaces with multiple Wizards. Proc. ACM Hum.-Comput. Interact. 7, CSCW1 (April 2023), 115:1-115:34.
https://doi.org/10.1145/3579591[64] Shan Huang and Mu Li. 2017. Piloting a mHealth intervention to improve newborn care awareness among rural Cambodian mothers: A feasibility
study. BMC Pregnancy Childbirth 17, 1 (2017). https://doi.org/10.1186/s12884-017-1541-z[65] Janet Shibley Hyde, Rebecca S. Bigler, Daphna Joel, Charlotte Chucky Tate, and Sari M. van Anders. 2019. The future of sex and gender inpsychology: Five challenges to the gender binary. Am. Psychol. 74, 2 (2019), 171–193. https://doi.org/10.1037/amp0000307[66] International Organization for Standardization. 1997. ISO 9241-1:1997: Ergonomic requirements for office work with visual display terminals(VDTs) — Part 1: General introduction. Retrieved April 23, 2023 from https://www.iso.org/standard/21922.html[67] ISO. 2002. ISO/TR 16982:2002: Ergonomics of human-system interaction -- Usability methods supporting human-centred design.
[68] Brett W. Israelsen and Nisar R. Ahmed. 2019. ―Dave... I can assure you... that it‘s going to be all right...‖ A definition, case for, and survey ofalgorithmic assurances in human-autonomy trust relationships. ACM Comput. Surv. 51, 6 (January 2019), 113:1-113:37.
https://doi.org/10.1145/3267338[69] Samantha Jaroszewski, Danielle Lottridge, Oliver L. Haimson, and Katie Quehl. 2018. ―Genderfluid‖ or ―attack helicopter‖: Responsible HCI
research practice with non-binary gender variation in online communities. In Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems (CHI ’18), April 21, 2018, Montreal QC. ACM, Montreal QC, 1–15. https://doi.org/10.1145/3173574.3173881[70] Kwangmin Jeong, Jihyun Sung, Hae-Sung Lee, Aram Kim, Hyemi Kim, Chanmi Park, Yuin Jeong, JeeHang Lee, and Jinwoo Kim. 2018. Fribo: A
social networking robot for increasing social connectedness through sharing daily home activities from living noise data. In Proceedings of the 2018
ACM/IEEE International Conference on Human-Robot Interaction (HRI ’18), 2018, New York, NY, USA. Association for Computing Machinery,
New York, NY, USA, 114–122. https://doi.org/10.1145/3171221.3171254[71] Oliver P. John and Sanjay Srivastava. 1999. The Big Five trait taxonomy: History, measurement, and theoretical perspectives. Handb. Personal.
Theory Res. 2, 1999 (1999), 102–138.
[72] Anirudha Joshi, Mandar Rane, Debjani Roy, Nagraj Emmadi, Padma Srinivasan, N. Kumarasamy, Sanjay Pujari, Davidson Solomon, Rashmi
Rodrigues, D.G. Saple, Kamalika Sen, Els Veldeman, and Romain Rutten. 2014. Supporting treatment of people living with HIV / AIDS in resource
limited settings with IVRs. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), 2014, New York, NY,
USA. Association for Computing Machinery, New York, NY, USA, 1595–1604. https://doi.org/10.1145/2556288.2557236[73] Aikaterini Kassavou and Stephen Sutton. 2017. Reasons for non-adherence to cardiometabolic medications, and acceptability of an interactive voice
response intervention in patients with hypertension and type 2 diabetes in primary care: A qualitative study. BMJ Open 7, 8 (2017).

ACM Comput. Surv.

https://doi.org/10.1136/bmjopen-2016-015597[74] Auk Kim, Jung-Mi Park, and Uichin Lee. 2020. Interruptibility for in-vehicle multitasking: Influence of voice task demands and adaptive behaviors.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 1 (2020). https://doi.org/10.1145/3381009[75] Hankyung Kim, Dong Yoon Koh, Gaeun Lee, Jung-Mi Park, and Youn-kyung Lim. 2019. Designing personalities of conversational agents. In
Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19), May 02, 2019, New York, NY, USA.
Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312887[76] Kati Kish Bar-On and Ehud Lamm. 2023. The interplay of social identity and norm psychology in the evolution of human groups. Philos. Trans. R.
Soc. B Biol. Sci. 378, 1872 (January 2023), 20210412. https://doi.org/10.1098/rstb.2021.0412[77] Mark L. Knapp, John P. Caughlin, and Anita L. Vangelisti. 2014. Interpersonal communication and human relationships (7th ed.). Pearson.
[78] Mirka Koro-Ljungberg and Elliot P. Douglas. 2008. State of qualitative research in engineering education: Meta-analysis of JEE articles, 2005–2006.
J. Eng. Educ. 97, 2 (2008), 163–175. https://doi.org/10.1002/j.2168-9830.2008.tb00965.x[79] Jody Kreiman, Gerratt R. Bruce, Gail B. Kempster, Andrew Erman, and Gerald S. Berke. 1993. Perceptual evaluation of voice quality. J. SpeechLang. Hear. Res. 36, 1 (February 1993), 21–40. https://doi.org/10.1044/jshr.3601.21[80] Jody Kreiman, Diana Vanlancker-Sidtis, and Bruce R. Gerratt. 2003. Defining and measuring voice quality. In Proceedings of the 2003 conference
on Voice Quality: Functions, Analysis and Synthesis, 2003, Geneva, Switzerland. ISCA, Geneva, Switzerland, 115–120. Retrieved December 17,
2019 from https://www.isca-speech.org/archive_open/voqual03/voq3_115.html[81] Katharina Kühne, Martin H. Fischer, and Yuefang Zhou. 2020. The human takes it all: Humanlike synthesized voices are perceived as less eerie andmore likable. Evidence from a subjective ratings study. Front. Neurorobotics 14, (2020). https://doi.org/10.3389/fnbot.2020.593732[82] Carine Lallemand, Guillaume Gronier, and Vincent Koenig. 2015. User experience: A concept without consensus? Exploring practitioners‘
perspectives through an international survey. Comput. Hum. Behav. 43, (February 2015), 35–48. https://doi.org/10.1016/j.chb.2014.10.048[83] David R. Large, Leigh Clark, Gary Burnett, Kyle Harrington, Jacob Luton, Peter Thomas, and Pete Bennett. 2019. ―It‘s small talk, Jim, but not as weknow it.‖: Engendering trust through human-agent conversation in an autonomous, self-driving car. In Proceedings of the 1st International
Conference on Conversational User Interfaces (CUI ’19), August 22, 2019, New York, NY, USA. Association for Computing Machinery, New
York, NY, USA, 1–7. https://doi.org/10.1145/3342775.3342789[84] Stefan Larsson, Anders Jensen-Urstad, and Fredrik Heintz. 2021. Notified but unaware: Third-party tracking online. Crit. Anal. Law 8, 1 (April
2021), 101–120.
[85] John Laver. 1980. The Phonetic Description of Voice Quality. Cambridge University Press, Cambridge, UK.
[86] Effie Lai-Chong Law, Virpi Roto, Marc Hassenzahl, Arnold P.O.S. Vermeeren, and Joke Kort. 2009. Understanding, scoping and defining user
experience: A survey approach. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09), April 04, 2009, New
York, NY. ACM, New York, NY, 719–728. https://doi.org/10.1145/1518701.1518813[87] Jong-Eun Roselyn Lee and Clifford I. Nass. 2010. Trust in computers: The Computers-Are-Social-Actors (CASA) paradigm and trustworthinessperception in human-computer communication. Trust Technol. Ubiquitous Mod. Environ. Theor. Methodol. Perspect. (2010), 1–15.
https://doi.org/10.4018/978-1-61520-901-9.ch001[88] Sunok Lee, Sungbae Kim, and Sangsu Lee. 2019. ―What does your agent look like?‖: A drawing study to understand users‘ perceived persona of
conversational agent. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19), 2019, New York,NY. ACM, New York, NY, 1–6. https://doi.org/10.1145/3290607.3312796[89] Sebastian Linxen, Christian Sturm, Florian Brühlmann, Vincent Cassau, Klaus Opwis, and Katharina Reinecke. 2021. How WEIRD is CHI? In
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21), May 06, 2021, New York, NY, USA. Association for
Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3411764.3445488[90] Irene Lopatovska, Elena Korshakova, Diedre Brown, Yiqiao Li, Jie Min, Amber Pasiak, and Kaige Zheng. 2021. User perceptions of an intelligent
personal assistant‘s personality: The role of interaction context. In Proceedings of the 2021 Conference on Human Information Interaction and
Retrieval (CHIIR ’21), 2021, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 15–25.
https://doi.org/10.1145/3406522.3446018[91] Irene Lopatovska and Harriet Williams. 2018. Personification of the Amazon Alexa: BFF or a mindless companion. In Proceedings of the 2018
Conference on Human Information Interaction & Retrieval (CHIIR ’18), March 01, 2018, New York, NY, USA. Association for Computing
Machinery, New York, NY, USA, 265–268. https://doi.org/10.1145/3176349.3176868[92] Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In 11th Australasian Conference on Information Systems, 2000,
Australia. Citeseer, Australia, 6–8.
[93] Nikolas Martelaro, Victoria C. Nneji, Wendy Ju, and Pamela Hinds. 2016. Tell me more: Designing HRI to encourage more trust, disclosure, andcompanionship. In The Eleventh ACM/IEEE International Conference on Human Robot Interaction (HRI ’16), 2016, Christchurch, New Zealand.
IEEE Press, Christchurch, New Zealand, 577. https://doi.org/10.1109/HRI.2016.7451750[94] Roger C. Mayer, James H. Davis, and F. David Schoorman. 1995. An integrative model of organizational trust. Acad. Manage. Rev. 20, 3 (1995),
709–734. https://doi.org/10.2307/258792[95] Eric Mazur. 1997. Peer instruction: Getting students to think in class.
https://doi.org/10.1063/1.53199

AIP Conf. Proc. 399, 1 (March 1997), 981–988.

[96] Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for
CSCW and HCI practice. Proc. ACM Hum.-Comput. Interact. 3, CSCW (November 2019), 72:1-72:23. https://doi.org/10.1145/3359174[97] Michael Meadon and David Spurrett. 2010. It‘s not just the subjects - there are too many WEIRD researchers. Behav. Brain Sci. 33, 2–3 (June 2010),

ACM Comput. Surv.

104–105. https://doi.org/10.1017/S0140525X10000208[98] Oussama Metatla, Alison Oldfield, Taimur Ahmed, Antonis Vafeas, and Sunny Miglani. 2019. Voice user interfaces in schools: Co-designing forinclusion with visually-impaired and sighted pupils. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, May 02,
2019, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 1–15. Retrieved November 7, 2021 fromhttps://doi.org/10.1145/3290605.3300608[99] Lisa Militello, Emre Sezgin, Yungui Huang, and Simon Lin. 2021. Delivering perinatal health information via a voice interactive app (SMILE):
Mixed methods feasibility study. JMIR Form. Res. 5, 3 (2021). https://doi.org/10.2196/18240[100] Blanca Miller and David Feil-Seifer. 2016. Embodiment, situatedness, and morphology for humanoid robots interacting with people. In Humanoid
Robotics: A Reference, Ambarish Goswami and Prahlad Vadakkepat (eds.). Springer Netherlands, Dordrecht, 1–23. https://doi.org/10.1007/978-94007-7194-9_130-1[101] Masahiro Mori, Karl F. MacDorman, and Norri Kageki. 2012. The uncanny valley [from the Field]. IEEE Robot. Autom. Mag. 19, 2 (June 2012), 98–
100. https://doi.org/10.1109/MRA.2012.2192811[102] Cosmin Munteanu, Gerald Penn, and Christine Murad. 2021. Conversational voice user interfaces: Connecting engineering fundamentals to design
considerations. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA ’21), May 08, 2021, New
York, NY, USA. Association for Computing Machinery, New York, NY, USA, 1–3. https://doi.org/10.1145/3411763.3445008[103] Chelsea Myers, Anushay Furqan, Jessica Nebolsky, Karina Caro, and Jichen Zhu. 2018. Patterns for how users overcome obstacles in voice userinterfaces. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), April 19, 2018, New York, NY, USA.
Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/3173574.3173580[104] Clifford Ivar Nass and Scott Brave. 2005. Wired for Speech: How Voice Activates and Advances the Human-Computer Relationship. MIT Press,
Cambridge, MA.
[105] Clifford Nass and Youngme Moon. 2000. Machines and mindlessness: Social responses to computers. J. Soc. Issues 56, 1 (2000), 81–103.
https://doi.org/10.1111/0022-4537.00153[106] Clifford Nass, Youngme Moon, and Nancy Green. 1997. Are machines gender neutral? Gender-stereotypic responses to computers with voices. J.
Appl. Soc. Psychol. 27, 10 (1997), 864–876. https://doi.org/10.1111/j.1559-1816.1997.tb00275.x[107] Clifford Nass, Jonathan Steuer, and Ellen R. Tauber. 1994. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors
in computing systems, 1994, Boston, MA. ACM, Boston, MA, 72–78. https://doi.org/10.1145/191666.191703[108] Nils J. Nilsson. 2009. The quest for artificial intelligence. Cambridge University Press.
[109] Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T. Freeman, Michael Rubinstein, and Wojciech Matusik. 2019. Speech2Face:
Learning the face behind a voice. In CVPR 2019, 2019, Long Beach, CA. Long Beach, CA, 7539–7548. Retrieved June 10, 2022 fromhttps://openaccess.thecvf.com/content_CVPR_2019/html/Oh_Speech2Face_Learning_the_Face_Behind_a_Voice_CVPR_2019_paper.html[110] J. B. Orange, Rosemary B. Lubinski, and D. Jeffery Higginbotham. 1996. Conversational repair by individuals with dementia of the Alzheimer‘s
type. J. Speech Lang. Hear. Res. 39, 4 (August 1996), 881–895. https://doi.org/10.1044/jshr.3904.881[111] Matthew J. Page, Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, Jennifer
M. Tetzlaff, Elie A. Akl, Sue E. Brennan, Roger Chou, Julie Glanville, Jeremy M. Grimshaw, Asbjørn Hróbjartsson, Manoj M. Lalu, Tianjing Li,
Elizabeth W. Loder, Evan Mayo-Wilson, Steve McDonald, Luke A. McGuinness, Lesley A. Stewart, James Thomas, Andrea C. Tricco, Vivian A.
Welch, Penny Whiting, and David Moher. 2021. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372,
(March 2021), n71. https://doi.org/10.1136/bmj.n71[112] Paul Pangaro and Hugh Dubberly. 2014. What is Conversation? How Can We Design for Effective Conversation? In What is Conversation? How
Can We Design for Effective Conversation? Birkhäuser, 144–159. https://doi.org/10.1515/9783038212843.144[113] Xavier Perrin, Ricardo Chavarriaga, Céline Ray, Roland Siegwart, and José del R. Millán. 2008. A comparative psychophysical and EEG study ofdifferent feedback modalities for HRI. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (HRI ’08), 2008,
New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 41–48. https://doi.org/10.1145/1349822.1349829[114] Rolf Pfeifer and Christian Scheier. 2001. Understanding Intelligence. MIT Press, Cambridge, MA.
[115] Valentina Pitardi and Hannah R. Marriott. 2021. Alexa, she‘s not human but… Unveiling the drivers of consumers‘ trust in voice-based artificial
intelligence. Psychol. Mark. 38, 4 (2021), 626–642. https://doi.org/10.1002/mar.21457[116] Jeff Pittam. 1994. Voice in Social Interaction: An Interdisciplinary Approach. Sage, Thousand Oaks, CA.
[117] Pierre Pluye and Quan Nha Hong. 2014. Combining the power of stories and the power of numbers: Mixed methods research and mixed studies
reviews. Annu. Rev. Public Health 35, 1 (2014), 29–45. https://doi.org/10.1146/annurev-publhealth-032013-182440[118] Denise F. Polit and Cheryl Tatano Beck. 2010. Generalization in quantitative and qualitative research: Myths and strategies. Int. J. Nurs. Stud. 47, 11(November 2010), 1451–1458. https://doi.org/10.1016/j.ijnurstu.2010.06.004[119] Kathrin Pollmann, Christopher Ruff, Kevin Vetter, and Gottfried Zimmermann. 2020. Robot vs. voice assistant: Is playing with Pepper more fun thanplaying with Alexa? In Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’20), 2020, New York, NY.
ACM, New York, NY, 395–397. https://doi.org/10.1145/3371382.3378251[120] Martin Porcheron, Joel E. Fischer, and Michel Valstar. 2020. NottReal: A Tool for Voice-based Wizard of Oz studies. In Proceedings of the 2nd
Conference on Conversational User Interfaces (CUI ’20), July 22, 2020, New York, NY, USA. Association for Computing Machinery, New York,NY, USA, 1–3. https://doi.org/10.1145/3405755.3406168[121] Chloé Pou-Prom, Stefania Raimondo, and Frank Rudzicz. 2020. A conversational robot for older adults with Alzheimer‘s disease. J Hum-Robot
Interact 9, 3 (May 2020). https://doi.org/10.1145/3380785

ACM Comput. Surv.

[122] Fernando Poyatos. 1993. Paralanguage: A Linguistic and Interdisciplinary Approach to Interactive Speech and Sounds. John Benjamins Publishing
Company, Amsterdam/Philadelphia.
[123] Lihui Pu, Wendy Moyle, and Cindy Jones. 2020. How people with dementia perceive a therapeutic robot called PARO in relation to their pain andmood: A qualitative study. J. Clin. Nurs. 29, 3–4 (2020), 437–446. https://doi.org/10.1111/jocn.15104[124] Robin Read and Tony Belpaeme. 2012. How to use non-linguistic utterances to convey emotion in child-robot interaction. In 2012 7th ACM/IEEEInternational Conference on Human-Robot Interaction (HRI), March 2012. 219–220. https://doi.org/10.1145/2157689.2157764[125] Matthias Rehm. 2008. ‘She is just stupid‘--Analyzing user-agent interactions in emotional game situations. Interact. Comput. 20, 3 (May 2008), 311–
325. https://doi.org/10.1016/j.intcom.2008.02.005[126] Nancy A. Rigotti, Yuchiao Chang, Lisa C. Rosenfeld, Sandra J. Japuntich, Elyse R. Park, Hilary A. Tindle, Douglas E. Levy, Zachary Z. Reid,
Joanna Streck, Timothy Gomperts, Jennifer H.K. Kelley, and Daniel E. Singer. 2017. Interactive voice response calls to promote smoking cessation
after hospital discharge: Pooled analysis of two randomized clinical trials. J. Gen. Intern. Med. 32, 9 (2017), 1005–1013.
https://doi.org/10.1007/s11606-017-4085-z[127] Ronda Ringfort-Felner, Matthias Laschke, Shadan Sadeghian, and Marc Hassenzahl. 2022. Kiro: A design fiction to explore social conversation withvoice assistants. Proc. ACM Hum.-Comput. Interact. 6, GROUP (January 2022), 33:1-33:21. https://doi.org/10.1145/3492852[128] David A. Robb, José Lopes, Stefano Padilla, Atanas Laskov, Francisco J. Chiyah Garcia, Xingkun Liu, Jonatan Scharff Willners, Nicolas Valeyrie,
Katrin Lohan, David Lane, Pedro Patron, Yvan Petillot, Mike J. Chantler, and Helen Hastie. 2019. Exploring interaction with remote autonomous
systems using conversational agents. In Proceedings of the 2019 on Designing Interactive Systems Conference (DIS ’19), 2019, New York, NY,
USA. Association for Computing Machinery, New York, NY, USA, 1543–1556. https://doi.org/10.1145/3322276.3322318[129] Jennifer A. Rode. 2011. Reflexivity in digital anthropology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems,
2011, Vancouver, BC. ACM, Vancouver, BC, 123–132. https://doi.org/10.1145/1978942.1978961[130] Rashmi Rodrigues, S. Poongulali, Kavitha Balaji, Salla Atkins, Per Ashorn, and Ayesha De Costa. 2015. ―The phone reminder is important, but willothers get to know about my illness?‖ Patient perceptions of an mHealth antiretroviral treatment support intervention in the HIVIND trial in South
India. BMJ Open 5, 11 (2015). https://doi.org/10.1136/bmjopen-2015-007574[131] Katja Rogers and Katie Seaborn. 2023. The systematic review-lution: A manifesto to promote rigour and inclusivity in research synthesis. In
Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA ’23), April 19, 2023, New York, NY, USA.
Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3544549.3582733[132] Rebecca B. Rubin, Elizabeth M. Perse, and Carole A. Barbato. 1988. Conceptualization and measurement of interpersonal communication motives.
Hum. Commun. Res. 14, 4 (June 1988), 602–628. https://doi.org/10.1111/j.1468-2958.1988.tb00169.x[133] James A. Russell. 1980. A circumplex model of affect. J. Pers. Soc. Psychol. 39, 6 (1980), 1161–1178. https://doi.org/10.1037/h0077714[134] Sergio Sayago. 2020. Voice assistants as learning companions: An initial exploration with computer science students. In Extended Abstracts of the
22nd International Conference on Human-Computer Interaction with Mobile Devices and Services: Expanding the Horizon of Mobile Interaction,
2020. 1–3. https://doi.org/10.1145/3406324.3410707[135] Ari Schlesinger, W. Keith Edwards, and Rebecca E. Grinter. 2017. Intersectional HCI: Engaging identity through gender, race, and class. In
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), May 02, 2017, Denver, Colorado. ACM, Denver,
Colorado, 5412–5427. https://doi.org/10.1145/3025453.3025766[136] Stephanie Schreitter and Brigitte Krenn. 2014. Exploring inter- and intra-speaker variability in multi-modal task descriptions. In The 23rd IEEE
International Symposium on Robot and Human Interactive Communication, August 2014. 43–48. https://doi.org/10.1109/ROMAN.2014.6926228[137] Katie Seaborn, Giulia Barbareschi, and Shruti Chandra. 2023. Not only WEIRD but ―uncanny‖? A systematic review of diversity in human-robot
interaction research. Int. J. Soc. Robot. (March 2023). https://doi.org/10.1007/s12369-023-00968-4[138] Katie Seaborn and Alexa Frank. 2022. What pronouns for Pepper? A critical review of gender/ing in research. In Proceedings of the 2022 CHI
Conference on Human Factors in Computing Systems (CHI ’22), 2022, New Orleans, Louisiana, USA. ACM, New Orleans, Louisiana, USA, 1–15.
https://doi.org/10.1145/3491102.3501996[139] Katie Seaborn, Norihisa Paul Miyake, Peter Pennefather, and Mihoko Otake-Matsuura. 2021. Voice in human-agent interaction: A survey. ACM
Comput. Surv. CSUR 54, 4 (May 2021), 1–43. https://doi.org/10.1145/3386867[140] Katie Seaborn, Takuya Sekiguchi, Seiki Tokunaga, Norihisa P. Miyake, and Mihoko Otake-Matsuura. 2022. Voice over body? Older adults‘
reactions to robot and voice assistant facilitators of group conversation. Int. J. Soc. Robot. 15, (November 2022), 143–163.
https://doi.org/10.1007/s12369-022-00925-7[141] Katie Seaborn and Jacqueline Urakami. 2021. Measuring voice UX quantitatively: A rapid review. In Extended Abstracts of the 2021 CHI
Conference on Human Factors in Computing Systems, 2021, Yokohama, Japan. ACM, Yokohama, Japan, No. 416.
https://doi.org/10.1145/3411763.3451712[142] Leon D. Segal. 1995. Designing Team Workstations: The Choreography of Teamwork. In Local Applications of the Ecological Approach To HumanMachine Systems. CRC Press.
[143] Stela H. Seo, Keelin Griffin, James E. Young, Andrea Bunt, Susan Prentice, and Verónica Loureiro-Rodríguez. 2018. Investigating people‘s rapport
building and hindering behaviors when working with a collaborative robot. Int. J. Soc. Robot. 10, 1 (2018), 147–161. https://doi.org/10.1007/s12369017-0441-8[144] Uttara Seshu, Homam A. Khan, Manish Bhardwaj, C. Sangeetha, G. Aarthi, Sujit John, R. Thara, and Vijaya Raghavan. 2021. A qualitative study on
the use of mobile-based intervention for perinatal depression among perinatal mothers in rural Bihar, India. Int. J. Soc. Psychiatry 67, 5 (2021), 467–
471. https://doi.org/10.1177/0020764020966003

ACM Comput. Surv.

[145] Emre Sezgin, Garey Noritz, Simon Lin, and Yungui Huang. 2021. Feasibility of a voice-enabled medical diary app (SpeakHealth) for caregivers ofchildren with special health care needs and health care providers: Mixed methods study. JMIR Form. Res. 5, 5 (2021). https://doi.org/10.2196/25503[146] Ameneh Shamekhi, Q. Vera Liao, Dakuo Wang, Rachel K. E. Bellamy, and Thomas Erickson. 2018. Face value? Exploring the effects ofembodiment for a group facilitation agent. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), 2018,
New York, NY, USA. ACM, New York, NY, USA, 391:1-391:13. https://doi.org/10.1145/3173574.3173965[147] Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. Interactions 4, 6 (November 1997), 42–61.
https://doi.org/10.1145/267505.267514[148] Kai Siedenburg, Charalampos Saitis, Stephen McAdams, Arthur N. Popper, and Richard R. Fay (Eds.). 2019. Timbre: Acoustics, Perception, andCognition. Springer International Publishing, Cham. https://doi.org/10.1007/978-3-030-14832-4[149] Frank J. Sileo and Wesley A. Kayson. 1988. When will annoying phone calls be listened to?: Effects of sex, tone of voice, and time of day. Psychol.
Rep. 62, 2 (April 1988), 351–355. https://doi.org/10.2466/pr0.1988.62.2.351[150] Katta Spiel, Oliver L. Haimson, and Danielle Lottridge. 2019. How to do better with gender on surveys: A guide for HCI researchers. Interactions
26, 4 (June 2019), 62–65. https://doi.org/10.1145/3338283[151] Katta Spiel, Os Keyes, and Pınar Barlas. 2019. Patching gender: Non-binary utopias in HCI. In Extended Abstracts of the 2019 CHI Conference on
Human Factors in Computing Systems (CHI EA ’19), May 02, 2019, New York, NY. ACM, New York, NY, 1–11.
https://doi.org/10.1145/3290607.3310425[152] Amanda J. Stent, Marie K. Huffman, and Susan E. Brennan. 2008. Adapting speaking after evidence of misrecognition: Local and global
hyperarticulation. Speech Commun. 50, 3 (March 2008), 163–178. https://doi.org/10.1016/j.specom.2007.07.005[153] Brodrick Stigall, Jenny Waycott, Steven Baker, and Kelly Caine. 2019. Older adults‘ perception and use of voice user interfaces: A preliminary
review of the computing literature. In Proceedings of the 31st Australian Conference on Human-Computer-Interaction (OZCHI’19), December 02,
2019, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 423–427. https://doi.org/10.1145/3369457.3369506[154] Clara Strathmann, Jessica Szczuka, and Nicole Krämer. 2020. She talks to me as if she were alive: Assessing the social reactions and perceptions ofchildren toward voice assistants and their appraisal of the appropriateness of these reactions. In Proceedings of the 20th ACM International
Conference on Intelligent Virtual Agents (IVA ’20), 2020, New York, NY, USA. Association for Computing Machinery, New York, NY, USA.
https://doi.org/10.1145/3383652.3423906[155] Anselm Strauss and Juliet M. Corbin. 1990. Basics of qualitative research: Grounded theory procedures and techniques. Sage Publications, Inc,
Thousand Oaks, CA.
[156] Saiganesh Swaminathan, Indrani Medhi Thies, Devansh Mehta, Edward Cutrell, Amit Sharma, and William Thies. 2019. Learn2Earn: Using mobile
airtime incentives to bolster public awareness campaigns. Proc ACM Hum-Comput Interact 3, CSCW (November 2019).
https://doi.org/10.1145/3359151[157] John Sweller. 2011. CHAPTER TWO - Cognitive Load Theory. In Psychology of Learning and Motivation, Jose P. Mestre and Brian H. Ross (eds.).
Academic Press, 37–76. https://doi.org/10.1016/B978-0-12-387691-1.00002-8[158] Linda Tickle-Degnen and Robert Rosenthal. 1990. The nature of rapport and its nonverbal correlates. Psychol. Inq. 1, 4 (October 1990), 285–293.
https://doi.org/10.1207/s15327965pli0104_1[159] Milka Trajkova and Aqueasha Martin-Hammond. 2020. ―Alexa is a toy‖: Exploring older adults‘ reasons for using, limiting, and abandoning Echo.
In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), April 21, 2020, Honolulu, HI, USA. ACM,
Honolulu, HI, USA, 1–13. https://doi.org/10.1145/3313831.3376760[160] Christiana Tsiourti, Astrid Weiss, Katarzyna Wac, and Markus Vincze. 2019. Multimodal integration of emotional signals from voice, body, andcontext: Effects of (in)congruence on emotion recognition and attitudes towards robots. Int. J. Soc. Robot. 11, 4 (August 2019), 555–573.
https://doi.org/10.1007/s12369-019-00524-z[161] Koen van Turnhout, Arthur Bennis, Sabine Craenmehr, Robert Holwerda, Marjolein Jacobs, Ralph Niels, Lambert Zaad, Stijn Hoppenbrouwers,
Dick Lenior, and René Bakker. 2014. Design patterns for mixed-method research in HCI. In Proceedings of the 8th Nordic Conference on HumanComputer Interaction: Fun, Fast, Foundational (NordiCHI ’14), October 26, 2014, New York, NY, USA. Association for Computing Machinery,
New York, NY, USA, 361–370. https://doi.org/10.1145/2639189.2639220[162] Raymond Tweheyo, Hannah Selig, Dustin G. Gibson, George William Pariyo, and Elizeus Rutebemberwa. 2020. User perceptions and experiencesof an interactive voice response mobile phone survey pilot in Uganda: Qualitative study. JMIR Form. Res. 4, 12 (2020).
https://doi.org/10.2196/21671[163] Adelline Twimukye, Agnes Bwanika Naggirinya, Rosalind Parkes-Ratanshi, Ronnie Kasirye, Agnes Kiragga, Barbara Castelnuovo, Jacob Wasswa,
Maria Sarah Nabaggala, Elly Katabira, Mohammed Lamorde, and Rachel Lisa King. 2021. Acceptability of a mobile phone support tool (Call for
Life Uganda) for promoting adherence to antiretroviral therapy among young adults in a randomized controlled trial: Exploratory qualitative study.
JMIR MHealth UHealth 9, 6 (2021). https://doi.org/10.2196/17418[164] Takane Ueno, Yuto Sawa, Yeongdae Kim, Jacqueline Urakami, Hiroki Oura, and Katie Seaborn. 2022. Trust in human-AI interaction: Scoping out
models, measures, and methods. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA ’22), 2022,
New Orleans, LA. ACM, New Orleans, LA, 1–7. https://doi.org/10.1145/3491101.3519772[165] Jacqueline Urakami and Katie Seaborn. 2023. Nonverbal cues in human-robot interaction: A communication studies perspective. ACM Trans. Hum.Robot Interact. 12, 2 (March 2023), 1–21. https://doi.org/10.1145/3570169[166] Aditya Vashistha, Edward Cutrell, Nicola Dell, and Richard Anderson. 2015. Social media platforms for low-income blind people in India. 2015.
259–272. https://doi.org/10.1145/2700648.2809858[167] Viswanath Venkatesh. 2000. Determinants of perceived ease of use: Integrating control, intrinsic motivation, and emotion into the technology

ACM Comput. Surv.

acceptance model. Inf. Syst. Res. 11, 4 (December 2000), 342–365. https://doi.org/10.1287/isre.11.4.342.11872[168] Alexandra Voit, Jasmin Niess, Caroline Eckerth, Maike Ernst, Henrike Weingärtner, and Paweł W. Woźniak. 2020. ―It‘s not a romantic
relationship‖: Stories of adoption and abandonment of smart speakers at home. In 19th International Conference on Mobile and Ubiquitous
Multimedia (MUM 2020), November 22, 2020, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 71–82.
https://doi.org/10.1145/3428361.3428469[169] Philip Weber and Thomas Ludwig. 2020. (Non-)Interacting with conversational agents: Perceptions and motivations of using chatbots and voiceassistants. 2020. 321–331. https://doi.org/10.1145/3404983.3405513[170] Bodo Winter. 2022. Abstract concepts and emotion: Cross-linguistic evidence and arguments against affective embodiment. Philos. Trans. R. Soc. B
Biol. Sci. 378, 1870 (December 2022), 20210368. https://doi.org/10.1098/rstb.2021.0368[171] Jun Xiao, John Stasko, and Richard Catrambone. 2004. An empirical study of the effect of agent competence on user performance and perception.
2004.
178–185.
Retrievedfromhttps://www.scopus.com/inward/record.uri?eid=2-s2.04544302018&partnerID=40&md5=1d7baca8283406341f1c8356f149ce89[172] Ying Xu and Mark Warschauer. 2020. Exploring young children‘s engagement in joint reading with a conversational agent. In Proceedings of theInteraction Design and Children Conference (IDC ’20), June 21, 2020, New York, NY, USA. Association for Computing Machinery, New York,NY, USA, 216–228. https://doi.org/10.1145/3392063.3394417[173] Min Yin and Shumin Zhai. 2006. The benefits of augmenting telephone voice menu navigation with visual browsing and search. In Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems (CHI ’06), April 22, 2006, New York, NY, USA. ACM, New York, NY, USA,319–328. https://doi.org/10.1145/1124772.1124821[174] Robert K. Yin. 2009. Case Study Research: Design and Methods. SAGE, Thousand Oaks, CA.
[175] Asterios Zacharakis, Maximos Kaliakatsos-Papakostas, Stamatia Kalaitzidou, and Emilios Cambouropoulos. 2021. Evaluating human-computer cocreative processes in music: A case study on the CHAMELEON melodic harmonizer. Front. Psychol. 12, (2021). Retrieved April 25, 2023 fromhttps://www.frontiersin.org/articles/10.3389/fpsyg.2021.603752[176] Nima Zargham, Michael Bonfert, Georg Volkmar, Robert Porzel, and Rainer Malaka. 2020. Smells like team spirit: Investigating the player
experience with multiple interlocutors in a VR game. In Extended Abstracts of the 2020 Annual Symposium on Computer-Human Interaction in Play(CHI PLAY ’20), 2020, New York, NY, USA. Association for Computing Machinery, New York, NY, USA, 408–412.
https://doi.org/10.1145/3383668.3419884[177] Chenyang Zhang, Ronggang Zhou, Yaping Zhang, Yanyan Sun, Liming Zou, and Min Zhao. 2020. How to design the expression ways of
conversational agents based on affective experience. In Proceedings of Human-Computer Interaction. Multimodal and Natural Interaction (Lecture
Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)), 2020. 320.
https://doi.org/10.1007/978-3-030-49062-1_21

ACM Comput. Surv.