Work-in-Progress

CHI 2015, Crossings, Seoul, Korea

Is Interpretation of Artificial Subtle
Expressions Language-Independent?
Comparison among Japanese, German,
Portuguese, and Mandarin Chinese
Takanori Komatsu

Seiji Yamada

Meiji University

National Institute of Informatics

4-21-1 Nakano, Nakano

2-1-2 Hitotsubashi, Chiyoda,

Tokyo 165-8525, JapanTokyo 101-8430, Japantkomat@meiji.ac.jp

seiji@nii.ac.jp

Rui Prada

Kotaro Funakoshi

INESC-ID

Honda Research Institute Japan

Av. Prof. Aníbal Cavaco Silva,

8-1 Honcho, Wako,
Porto Salvo 2744-016, Portugal Saitama 351-0188, Japan.
rui.prada@tecnico.ulisboa.pt
funakoshi@jp.honda-ri.com
Kazuki Kobayashi
Shinshu University
4-7-1 Wakasato, Nagano,
Nagano 380-8553, Japan.
kby@shinshu-u.ac.jp

Mikio Nakano
Honda Research Institute Japan
8-1 Honcho, Wako,
Saitama 351-0188, Japan.
nakano@jp.honda-ri.com

Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for third-party components of this work must be honored. For all otheruses, contact the Owner/Author.
Copyright is held by the owner/author(s).
CHI'15 Extended Abstracts, Apr 18-23, 2015, Seoul, Republic of Korea
ACM 978-1-4503-3146-3/15/04.
http://dx.doi.org/10.1145/2702613.2732710

Abstract
Up until now, several studies have shown that a speechinterface system giving verbal suggestions with beeping
sounds that decrease in pitch conveyed a low system
confidence level to users intuitively, and these beeping
sounds were named “artificial subtle expressions”(ASEs). However, all participants in these studies were
only Japanese, so if the participants’ mother tongue has
different sensitivity to variations in pitch compared withJapanese, the interpretations of the ASEs might bedifferent. We then investigated whether the ASEs areinterpreted in the same way as with Japaneseregardless of the users’ mother tongues; specifically wefocused on three language categories in traditional
phonological typology. We conducted a web-based
experiment to investigate whether the ways speakersof German, Portuguese (stress accent language),
Mandarin Chinese (tone language) and Japanese (pitchaccent language) interpret the ASEs are different or
not. The results of this experiment showed that theways of interpreting did not differ, so this suggests thatthese ways are language-independent.

Author Keywords
Artificial subtle expressions (ASEs); tone language;
pitch accent language; stress accent language

2175

Work-in-Progress

CHI 2015, Crossings, Seoul, Korea

ACM Classification Keywords
H.5.2. User Interfaces: Evaluation/methodology; J.4.
Social and behavioral sciences: Psychology.

showed that suggestions made with decreasing ASEs
conveyed a low system confidence level to users [11].

Introduction
Although there is little hope that speech interface
systems will ever be perfectly reliable [2,6], people’sinteraction with such non-perfect systems has only
been analyzed sparsely. Recently, some studies havebeen focusing on displaying a system’s confidence level
to users, and these studies have shown that it isactually effective for various aspects of interaction
between humans and systems [3-5,9,12,14]. For
example, Antifakos et al. [1] showed that users adapt
to a system easily if the system’s confidence isdisplayed on a computer screen. Horvitz and Barry [8]proposed a context-aware system that can estimate theexpected value of revealed information to enhancecomputer displays for time-critical applications.
Therefore, expressing a system’s confidence to users is
an important requirement for user interfaces. Most ofthese studies used actual human-like expressions to
express their confidence level to users, e.g., speechsounds from speakers or character information on
computer displays.
In contrast with the above approaches, Komatsu et al.
[10] proposed using artificial subtle expressions (ASEs)
as machine-like expressions used to convey a system’s
confidence level to users. Specifically, they proposed
two simple beeping sounds used as ASEs: a flat sound(flat ASE) and a sound with a decreasing pitch(decreasing ASE). These ASEs were added 0.2 seconds
after the system’s verbal suggestions. They thenFigure 1. Artificial Subtle Expressions (ASEs)

These experiments were conducted only in Japan, andall of the participants were Japanese, so there is somepossibility that the sensitivity of the participants in theirmother tongue to variations in pitch might affect theways ASEs are interpreted; for example, if theparticipants have different sensitivity to variations inpitch compared with Japanese, the interpretations ofdecreasing ASEs might be different. In terms of thesensitivity to such pitch variation, the three language
categories shown in Table 1 are proposed in traditional
phonological typology [7,11,13,15]. “Pitch accent
language” means that variations in pitch can be used todifferentiate words, “tone language” means thatdifferent tones change the meaning of the words (moresensitive to variations in pitch compared with pitchaccent language), and “stress accent language” meansthat only the variations in power are used forexpressing an accent (less sensitive compared withpitch accent languages).

2176

Work-in-Progress

Type

Example oflanguages

Tone language
Pitch accent
languageStress accent
language

Mandarin Chinese, Thai,
Vietnamese
Japanese, Swedish
English, German,
Spanish, Portuguese

Table 1. Language categorization inphonological typology

CHI 2015, Crossings, Seoul, Korea

Therefore, to investigate whether the ASEs areinterpreted in the same way as with Japanese (pitchaccent language) regardless of the users’ mothertongues, it is necessary to investigate how the ASEs
are interpreted by participants whose mother tongues
are categorized as the remaining two language types,
e.g., tone language and stress accent language.
Specifically, we focused on Mandarin Chinese as thetone language and German and Portuguese as thestress accent language (Table 1). The reason that we
focused on two stress accent languages is the different
positions of stress in a word. In general, German has astrong stress on the first syllable, while Portuguese
does on the second or third last syllable [7].
We thus conducted a web-based experiment toinvestigate whether the ways speakers of German,
Portuguese, Mandarin Chinese, and Japanese interpret
ASEs are different or not. This investigation would
clarify whether these ways are language-independent
or not and will determine whether the ASEs can beapplied to various kinds of speech interface systems
regardless of the users’ spoken language or mothertongues, while most current techniques for speechinterface systems are obviously language-dependent.

ExperimentSetting
We conducted a web-based experiment to investigate
the effects of the participants’ mother tongue on theirinterpretation of the ASEs. We used a “driving treasurehunting” video game as an experimental environment(Figure 2). In this game, the game image scrolls
forward on a straight road as if the participant is driving
a car with a navigation system and with small threemounds of dirt appearing along the way. A coin is

inside one of the three mounds, while the other two
mounds contain nothing. The game ends after theparticipants encounter 24 sets of mounds (24 trials).

Figure 2. Driving treasure hunting video gameThe purpose for the participants is to get as many coins
as possible. The location of the coin among the threemounds is randomly assigned. In each trial, thenavigation system to the left of the driver seat (circled
in the left image of Figure 2) told them which mound it
expected the coin to be in by using speech with theASEs. The participants could feely accept or reject thenavigation system’s suggestions. In each trial, evenafter the participants selected one mound among thethree, they were not told whether the selected moundhad the coin or not (only a question mark appearing
from the opened treasure box is displayed, as shown in
the right image of Figure 2). The participants were
informed of their total numbers of coins only after theyfinished all 24 trials.
Stimuli
The navigation system used speech with the ASEs totell participants the expected position of the coin. We

2177

Work-in-Progress

CHI 2015, Crossings, Seoul, Korea

prepared three different pieces of speech and two
different ASEs. This means that the system could
present six different stimuli for the participants. The
navigation system used English speech sounds tosuggest to the participants the expected location of thecoin, that is, “number one,” “number two,” and“number three.” These speech sounds were generated
by AT&T Natural Voices1.

Figure 3. Speech waveform of verbal suggestion “number
one” with decreasing ASE

0.2 seconds after the end of the speech sounds, one of
the two ASEs were played (Figure 3). These two ASEs
were triangle wave sounds 0.5 seconds in duration, but
their inflection patterns of pitch were different; that is,
one was a flat ASE (onset fundamental frequency (F0):250 Hz and end F0: 250 Hz), and the other was adecreasing ASE (onset F0: 250 Hz and end F0: 100
Hz). The former studies [10] already showed thatsuggestions made with decreasing ASEs conveyed a low
system confidence level to users. The interval between
the suggestions and the ASEs (0.2 seconds) and the1

http://www2.research.att.com/~ttsweb/tts/demo.php

duration of the ASEs (0.5 seconds) was the same as in
the former studies.
Participants
117 volunteers (73 men and 44 women; 20 – 45 years
old, mean age of 24.91) participated. These
participants voluntary responded to a call forparticipants from the authors. Out of the 117
participants, 44 participants’ mother tongue wasPortuguese (Portuguese group, nationality: Portugal),
26 was Japanese (Japanese group, nationality:
Japanese), 23 was German (German group, nationality:
Germany, Austria, and Switzerland), and 24 was
Mandarin Chinese (Chinese group, nationality: China).
All the instructions in the experimental system were
written in English so that all the participants could
understand these English descriptions. Therefore, there
was some possibility that the participants would react
to the given suggestions as if their mother tongue were
English, but the verbal suggestions used are quite
simple English phrases like “number one”, and Komatsu[10] reported that participants interpret the meaningsof ASEs intuitively and unconsciously, so we assumed
that their interpretation of ASEs was not significantly
affected by their English skills but purely by theirmother tongue.
Procedure
We used a web-based experiment system to record theparticipants’ behavior in regard to selecting whichmound contained the coin depending on the given
suggestions. First, the system displayed a consent form
and instructions for the experiment. These instructions
never mentioned or explained the ASEs to theparticipants. Before starting the game, the participants

2178

Work-in-Progress

CHI 2015, Crossings, Seoul, Korea

were asked to listen to a test sound via speakers or
headphones and to adjust the sound volume to acomfortable level. Afterward, they played the driving
treasure hunting video game. Among the 24 trials, thesystem expressed all 6 stimuli 4 times (the flat ASEs 12times and the decreasing ASEs 12 times). The order inwhich the stimuli were expressed was randomized.

German, and Mandarin Chinese interpreted the ASEs
did not differ from each other, so this result suggeststhat the ways are language-independent.

RESULTS
Language

12 flat ASEs

Portuguese

3.27(n=44)(SD=3.03)

Japanese(n=26)
German(n=23)
Chinese(n=24)

12 dec ASEs
6.45 (3.73)

3.62 (3.22)

6.27 (4.01)

3.69 (3.17)

5.83 (3.78)

3.67 (2.93)

6.13 (2.88)

Table 2. Rejection counts of flat anddecreasing (dec) ASEs for four
participant groups

To investigate the effects of the different ASEs on theparticipants’ behavior in terms of how often they
accepted or rejected the system’s suggestions in eachgroup, we then counted the rejection count, indicating
how many of system’s suggestions the participants
actually rejected. The average rejection counts of the
12 flat ASEs and 12 decreasing ASEs for the four
participant groups are summed in Table 2.
These rejection counts were analyzed by using a 2 × 4
mixed plan ANOVA (within-participant independent
variable: types of ASEs, flat/decreasing, betweenparticipant independent variable: language group,
Portuguese/German/Japanese/Chinese, dependent
variable: rejection counts). The results of the ANOVA
showed that there was no significant differences in theinteraction effect [F(3,114) = 0.44, n.s.] and in themain effect of the between-participant independent
variable (four language groups) [F(3,114) = 0.02,
n.s.], but there was a significant difference between thewithin-participant independent variables (two ASEs)[F(1,114) = 44.02, p < .01, effect size: η2 = 0.13](Figure 4). To sum up, the system’s suggestions withthe decreasing ASEs showed significantly higher
rejection counts compared with those with the flat
ASEs, regardless of the participants’ mother tongue.
Therefore, the ways speakers of Portuguese, Japanese,

Figure 4. Rejection counts for flat and decreasing ASEs andobserved significant differences among these factors

Discussions and Conclusion
This paper succeeded in clearly showing that the waysof interpreting ASEs did not differ among theparticipants whose mother tongues are a tone language(Mandarin Chinese), pitch accent language (Japanese),and stress accent language (Portuguese and German).
This result suggests that the ways are languageindependent, while most techniques for speechinterface systems are obviously language-dependent.
We believe that ASEs can then be applied to various
kinds of speech interface systems, e.g., car navigation
systems or speech recognition systems in mobile
devices, regardless of the users’ mother tongues or
spoken languages.

2179

Work-in-Progress

CHI 2015, Crossings, Seoul, Korea

However, we have to tackle several consecutive
experiments in order to conclude that the ways arelanguage-independent. One is conducting experimentsby recruiting participants whose mother tongues are
English, Spanish, Hindi, and Arabic because these
populations total nearly one billion, and the other isconducting an experiment with those who cannot
understand English at all because our experimental
system was implemented in English and theparticipants in this study can understand English as aforeign language. Thus, we could not perfectly exclude
the effects of English in the results of this paper. We
believe the results of these consecutive studies would
compensate for the effectiveness of the ASEs and willlead to the above strong conclusions.

Computer-Human Interaction 11, 4, ACM Press (2004),
329-356.
[6] Higashinaka, R., Sudoh, L., and Nakano, M.
Incorporating Discourse Features into Confidence
Scoring of Intention Recognition Results in Spoken
Dialogue Systems, Speech Communication 48, 3-4(2006), 417-436.
[7] Hirst, D., and Di Cristo, A. (Eds). Intonation
systems: A survey of twenty languages, Cambridge
University Press, 1998.
[8] Horvitz, E., and Barry, M. Display of information fortime-critical decision making, In Proc. 11th Conf. on
Uncertainty in Artificial Intelligence, Morgan Kaufmann(1995), 296-305.
[9] Horvitz, E. Principles of mixed-initiative userinterfaces, In Proc. CHI’99, ACM Press (1999), 159166.

References[1] Antifakos, S., Kern, N., Shiele, B., andSchwaninger, A. Towards Improving Trust in Context
Aware Systems by Displaying System Confidence, In
Proc. MobileHCI’05, ACM Press (2005), 9-14.

[10] Komatsu, T., Yamada, S., Kobayashi, K.,
Funakoshi, K. and Nakano, M. Artificial Subtle
Expressions: Intuitive Notification Methodology for
Artifacts, In Proc. CHI’10, ACM Press (2010), 19411944. 14.

[2] Bellotti, V. and Edwards, K. Intelligibility andaccountability: Human considerations in context-awaresystems, Human-Computer Interaction 16, 2 (2001),
193-212.

[11] McCawley, J. D. What is a tone language, In
Fromkin, V (Eds), Tone: a linguistic survey, 113-131,
Academic Press, 1978.

[3] Benzeghibaa, M., De Moria, R., Derooa, O., DupontS., Erbesa, T., Jouveta, D., Fissorea, F., Lafacea, P.,
Mertinsa, A., Risa, C., Rosea, R., Tyagia, V., andWellekensa, C. Automatic speech recognition andspeech variability: A review, Speech Communication
49, 10-11 (2007), 763-786.
[4] Cai, H. and Lin, Y. Tuning Trust Using Cognitive
Cues for Better Human-Machine Collaboration, In Proc.
HFES2010, pp. 2437-2441(5).
[5] Feng, J., and Sears, A. Using Confidence Scores to
Improve Hands-Free Speech Based Navigation inContinuous Dictation Systems, ACM Transactions on[12] Ogawa, A., and Nakamura, A. Joint estimation ofconfidence and error causes in speech recognition,
Speech Communication 54, 9 (2012), 1014-1028.
[13] O’Grady, W., Dobrovolsky, M., and Aronoff, M.
Contemporary Linguistics (Third Edition), St. Martin’s
Press, 1997[14] Parasuraman, R. Human use and abuse ofAutomation, In M. Mouloua and J. Koonce (Eds.),
Human-Automation Interaction: Research and Practice.
Erlbaum Associates, 1997.
[15] Van der Hulst, H., and Smith, N (Eds).
Autosegmental studies in pitch accent, Foris, 1998.

2180