Acoustics 08 Paris

The Effect of Learning on Listening to Ultra-Fast Speech
T. Nishimotoa , Y. Kariyab and T. WatanabebaGraduate School of Information Science and Technology, The University of Tokyo, 7-3-1
Hongo, Bunkyo-ku, 113-8656 Tokyo, Japanb
Department of Communication, Tokyo Woman’s Christian University, 2-6-1 Zenpukuzi,
Suginami-ku, 167-8585 Tokyo, Japannishi@hil.t.u-tokyo.ac.jp6119

Acoustics 08 Paris
We investigated the intelligibility of ultra-fast speech which may be used for screen reader for persons withvisual disability. The subjects were 35 women who are university students and are not visually/hearing
impaired. They were divided into four groups and they listened to 150 words with the speed of approximately 20 morae/sec. The vocabulary contained the tasks of high and low familiarity words, and theorders of tasks were diﬀerent by the groups. Four morae Japanese words from the FW03 database wereused as the vocabulary of the recall test. As a result, signiﬁcant learning eﬀect was observed in cases
where the subject listened the high familiarity words in succession. This indicates that the learning eﬀect
to the ultra-fast speech is promoted when the mental lexical access is easy. We also investigated themental workloads of the listening task using the NASA-TLX method. As the results, signiﬁcantly highworkload scores were observed at the listening of low familiarity words. The results also suggested thatthe mental workload decreases when the subject was aware that the mental lexical access was diﬃcult.

1

Introductionout that the task performances of elderly persons andyoung students are almost same. The learning eﬀects
of the elderly persons are not signiﬁcant in either case,though those of the young students are signiﬁcant forseveral weeks.
On the basis of past experiment, it is necessary to
carry out experiments for more general vocabulary. This
paper investigates the eﬀects of word familiarity of thestimulus in this learning eﬀect. We also examine themental workload during listening. The process of learning of subject can be investigated in detail by measuringmental workload, because the subject may increases themental workload during the active learning. Even if listener was able to hear the fast talking voice, it is undesirable for listener when the mental workload increases.

It is important for persons with visual disability to use
PCs and/or Internet with voice, which allows real-timecommunication and gives the chance of taking part tosocial activities. Screen-readers help the persons withvisual disability to access the Web and read or writee-mails. To make such systems easy to use, we must
consider how quickly and accurately the users understand information by listening to the voice of the Textto-Speech (TTS) systems. Watanabe [1] investigated
how the PC users with visual disability in Japan are setting the voice of screen readers. He reported that mostof the users are using TTS with the maximum reading
rate that the softwares can read with. In many cases, it
was double the speed of the normal speaking rate.
Asakawa et al. [2, 3] created rapidly-spoken Japanese
speech sentences by using the time-stretch/compressionfunction of the CoolEdit audio processing software andshortened recorded human voices linearly on the timeaxis. Then the stimuli were evaluated by subjects withvisual disability who are skilled users of screen reading
softwares. According to their work, the suitable speed
and the highest speed are deﬁned respectively as the
speaking rate for which listeners can understand a sentence suﬃciently, and the speaking rate for which listeners can understand approximately 50% of the words
in the sentence. They report results which show thatskilled listeners of synthesized speech assess the suitablespeed (recall rate = 90%) as 1100-1170 morae/minuteand the highest speed (recall rate = 50%) as 1400-1500
morae/minute, respectively. Most of the commercial
text-to-speech engines could not produce voices at such
fast speaking rates, so their work suggested a way to
improve non-visual user interfaces for people with visual disability.
We investigated the intelligibility of Japanese Textto-Speech systems at fast speaking rates, using fourdigit random numbers as the vocabulary of the recall
test[4]. The results showed that the learning eﬀect wassigniﬁcant in the early stage of the trials and the effect sustained for several weeks. We also investigatedthe eﬀects of age-related eﬀects on the same task[5].
Elderly persons can recall fast speech at some levels.
However, their average recall rates are lower than theyoung university students and the individual diﬀerences
are signiﬁcant. The results of this tasks we consider areaﬀected by the diﬃculty of auditory perception itself
and the diﬃculty of recall the numbers in the correct
order. Leaving out the eﬀect of latter factor, it turned

2

Mental Workload Measurement

Both physiological methods and objective evaluation techniques (e.g. dual task method) are well-known techniques to measure mental workload. In comparison withthese techniques, we can easily carry out subjective assessment. Subject may be aware of diﬀerence of small
workload that we cannot compare in objective evaluation. In such a case, more sensitive result is provided bysubjective assessment. However, these results provided
from introspectiveness of subject may not be reliable.
NASA-TLX[6, 7] is a subjective technique to evaluate mental workload with the following factors;• Mental demand• Physical demand• Temporal pressure• Eﬀort• Frustration level• Performance
At ﬁrst, the subject evaluates importance of eachfactor by doing the rankings. Then the subject accomplishes the work, followed by the evaluations of the workload for each factor. In our study, the subject inputs
value from 0 to 100 by operation of scroll bar for eachfactor. Based on the ranking, we calculate weighted
mean workload score (WWL). Magnitude of the mostimportant factor and the most unimportant factor is 6
and 1 respectively.

6120

Acoustics 08 Paris

Figure 1: The NASA-TLX main menu with
explanations of workload factors.
Figure 2: The evaluation of workload factors.

3

Familiarity-Controlled Words• FH : Word familiarity is from 7.0 to 5.5.

FW03[8] is an audio data set for word intelligibility test.
Vocabulary of this data is made based on Japanese wordfamiliarity database. The word familiarity data weremade by work to give value of familiarity with hands
in 80,000 words of Japanese dictionary. The words of
4 morae which accent were controlled were chosen forevery 4 groups corresponding to diﬀerent familiarity. 20
groups of 50 words were made in consideration of balance of phoneme. This vocabulary was read aloud by foreach 2 speaker of man and woman, and it was recorded.
The speed to read is controlled in this recording. Calibration of level is accomplished for this audio data.

4

Preliminary Experiment

4.1

Experimental Setup• FL : Word familiarity is from 2.5 to 1.0.
Conditions about speech rate are as follows:• S1 : Speech ﬁles that are included in FW03.
• S2 : Speech ﬁles that we changed to double speed
using speech editing software Adobe Audition 2.0.
The audio ﬁles are 48KHz sample rate, 16bit monaural
speech ﬁles of 1 male speaker.
The task consists of FH-S1, FH-S2, FL-S1 and FLS2. Each task consists of 50 words which are chosen inconsideration of phoneme balance. All the words consistof 4 morae. Order of the words in each task is random.
Average speed of S1 and S2 are 5.3 morae/sec and 10.8
morae/sec, respectively. The speech rate does not havesigniﬁcant diﬀerence between each group of FH and FL.
Subjects were 11 undergraduate students (women),and their native language was Japanese. They had neither visual impairment nor hearing impairment. They
have not listened to the speech of the task.
We used a PC and a headphone for each subject.
They listened to the speech played at intervals of 10
seconds and ﬁlled in answer sheets with the words thatthey heard. To avoid the eﬀect of order, we divided thesubjects to one group which listened to the fast speechearlier and another group which listened to the ordinary
speech earlier.

In the ﬁrst experiment, we verify appropriateness ofevaluating speech listening task with NASA-TLX. Furthermore, we verify that word familiarity aﬀects both
intelligibility and mental workload in this experiment.
We developed NASA-TLX software for Windows.
Figure1 is screen of menu and explanation of each factor,
and Figure2 is screen of evaluation of each factor. We
designed it to make subject be conscious of magnitude
of value between tasks. In other words, in evaluation ofseveral tasks, we designed screen so that subject could
refer to past evaluation value. In explanation of factor,
we supplemented the following explanations:• The highest performance means that you can listen to all words and write down correctly.

4.2

Results

The result of 10 subjects were analyzed. A subject who
did inappropriate answer was excluded. Figure 3 shows
the distribution of word intelligibility and WWL in eachtask of the subjects. Figure 4 shows the distribution ofword intelligibility and N-WWL (Normalized WWL).
In this normalization, we transformed the average andstandard deviation of WWL of each subject to 50 and
10, respectively. Table 1 shows Average(SD) of Intelligibility, WWL and Normalized WWL. The t-test (two• Mental demand includes the workload to listen tosounds.
• Physical demand includes the workload to listen
to sounds and to write characters.
We evaluated speech included in FW03 database.
Conditions with regard to word familiarity are as follows.

6121

Acoustics 08 Paris
Table 1: Average(SD) of Intelligibility, WWL and
N-WWL (Exp.1)
Task
Intelligibility
WWL
N-WWL
FH-S1
99.8(0.6)
47.9(14.6) 39.7(3.7)
FH-S2
96.8(2.2)
53.3(13.0) 48.3(5.0)
FL-S1
97.4(1.6)
53.1(12.9) 47.1(4.2)
FL-S2
65.8(7.5)
63.5(12.9) 64.9(2.4)

Intelligibility Score, %

100

80

60

40

Table 2: Experimental setup of Exp.2.
Group Trial 1 Trial 2 Trial 3
L-L-L FL-V1 FL-V2 FL-V3
H-H-L FH-V4 FH-V5 FL-V3
L-L-H FL-V1 FL-V2 FH-V6
H-H-H FH-V4 VH-V5 FH-V6

FH-S1
FH-S2
FL-S1
FL-S2

20

0
0

20

40

60

80

100

Weighted Workload (WWL)other words we use FH and FL in this experiment. We
prepared three sets of vocabulary at each level of wordfamiliarity (i.e. vocaburary V1-V3 for familiarity FL,
and V4-V6 for FH, respectively). The reason is we pay
our attention to learning eﬀect for voice, and to disturb
learning for vocabulary.
The subjects were 35 undergraduate students (women).
We divided them into 4 groups. Table 2 shows setup of
the experiment. They listened to the speech played atintervals of 16 seconds. The subjects input words thatthey heard as alphabets (Roman characters) with keyboard of the PC.

Figure 3: WWL and intelligibility score (Exp.1)

Intelligibility Score, %

100

80

60

40

5.2

FH-S1
FH-S2
FL-S1
FL-S2

20

We wanted to verify the following hypotheses throughthis experiment:
1. Learning eﬀect in this task would consist of (a)learning for task performance itself, (b) learning
for acoustic stimulus as bottom-up information,(c) learning in regard to mental lexical access as
top-down information.
Under the condition that continues listening towords of low word familiarity, learning in regard
to mental lexical access would be disturbed.
In contrast, under the condition that continues listening to word of high word familiarity, learning of
mental lexical access would be promoted. Therefore we would observe remarkable learning eﬀect
in HHH condition in comparison with LLL condition.

0
0

20

Hypotheses

40

60

80

100

Normalized WWL

Figure 4: N-WWL and intelligibility score (Exp.1)
tailed distribution, level of signiﬁcance = 5%) among
the groups showed signiﬁcant diﬀerence in all combinations of the groups, except the case between FH-S2 andFL-S1.
These results showed appropriateness of our experimental method. In addition, it was suggested that we
could normalize individual variations using the adjusted
standard deviation score of WWL.

5

Experiment on Learning Eﬀects

5.1

Experimental Setup

2. Under the condition that can utilize much topdown information, there becomes little mental workload. Subjects listening to speech of high word familiarity ﬁrst notice that top-down information isavailable.
Therefore, as for subjects listening to speech of
low word familiarity after speech of high word familiarity, remarkable increase of mental workload
will be seen.
In contrast, subjects listening to speech of low
word familiarity ﬁrst will know that top-down information is not available. Therefore, as for them,In the second experiment, we examine relations of learning eﬀect in listening of fast speech and word familiarity.
The rate of speech which we used in this experiment isfour times faster than the original speech. Condition
about word familiarity is the same as experiment 1. In

6122

Acoustics 08 Paris100

Table 3: Average (SD) of intelligibility, % (Exp.2)
Group
Trial 1
Trial 2
Trial 3
L-L-L 56.2 (8.4) 65.5 (6.1) 66.4 (6.8)
H-H-L 77.7 (5.5) 80.9 (3.1) 63.4 (7.3)
L-L-H 57.6 (6.5) 64.3 (5.4) 85.0 (5.8)
H-H-H 74.7 (8.8) 81.5 (6.3) 86.4 (5.2)

Intelligibility Score, %80

Table 4: Average (SD) of N-WWL (Exp.2)
Group
Trial 1
Trial 2
Trial 3
L-L-L 48.5 (11.2) 52.2 (7.9) 49.3 (10.2)
H-H-L
45.3 (9.2) 44.6 (6.6) 60.1 (4.2)
L-L-H
49.0 (6.0) 55.9 (8.6) 45.1 (11.4)
H-H-H 54.9 (10.8) 52.9 (7.7) 42.2 (5.6)

T3 (L)T2 (L)60
T1 (L)

40

20

0
0

20

40

60

80

100

N-WWL

their mental workload will not reduce when they
will listen to speech of high word familiarity laterbecause we do not use top-down information intentionally.

Figure 5: N-WWL and intelligibility (L-L-L of Exp.2)
100

5.3

Results

We counted the results of this experiment with respect
to each morae which constituted the word. Table 3 and4 show the average (SD) of intelligibility (%) and Normalized WWL (N-WWL), respectively. Figure 5, 6, 7
and 8 show the average intelligibility and N-WWL ofeach trial for the group L-L-L, H-H-L, L-L-H and H-HH, respectively. The vertical and horizontal bars depict
the standard deviations.
Results of t-test of two tailed distribution concerning
word intelligibility are as follows:

T2 (H)T1 (H)

Intelligibility Score, %80

T3 (L)

60

40

20• In result of group L - L - L, signiﬁcant diﬀerence
was seen only between T1–T2 (p=0.004).

0• In case of H-H-H, signiﬁcant diﬀerences were seen
both between T1–T2 (p=0.015) and T2–T3 (p=0.014).
• In case of H-H-L, signiﬁcant diﬀerences were seen
both between T1–T2 (p=0.021) and T2–T3 (p=0.000).
The latter diﬀerence can be assumed that the learning eﬀect was canceled out by the eﬀect of low
word familiarity.
• In case of L-L-H, signiﬁcant diﬀerence were seen
both between T1–T2 (p=0.001) and T2–T3 (p=0.000).
The latter diﬀerence can be assumed as the resultof the learning eﬀect and the eﬀect of high wordfamiliarity.
Results of t-test of N-WWL are as follows:0• In case of H-H-H, however, signiﬁcant diﬀerence
was observed only between T2–T3 (p=0.010).
• In case of H-H-L, signiﬁcant diﬀerences were also
seen between T1–T3 (p=0.011) and T2–T3 (p=0.000),
respectively.

6123

40

60

80

100

N-WWL

Figure 6: N-WWL and intelligibility (H-H-L of Exp.2)
These results support the hypothesis described in
section 5.2. It cannot be said that we can listen to the
speech of high word familiarity well after we get used tothe speech of low word familiarity. We were not able toverify the opposite eﬀect; i.e. the exercise with speechof high word familiarity would be eﬀective for speech of
low word familiarity.

6• In cases of L-L-L and L-L-H, no signiﬁcant diﬀerence was observed.

20

Conclusion

We investigated learning eﬀect in listening of fast speechwith NASA-TLX. We investigated in particular eﬀectof word familiarity using speech database FW03. Important ﬁndings that we obtained in this research are
as follows. Particularly remarkable learning eﬀect wasprovided by continuing listening to speech of high wordfamiliarity. Mental workload was not aﬀected by diﬃculty of task. Mental workload increased by carrying

Acoustics 08 Paris

Acknowledgments

100

40

We developed the Japanese version of NASA-TLX software in reference to a version developed by Dr. Kazumitsu Shinohara of Osaka University. In the second experiment, we used a high-quality speech rate conversion
program developed by Dr. Nobutaka Ono of The University of Tokyo. This work was partially supportedby the Ministry of Education, Culture, Sports, Scienceand Technology (Japan), Grant-in-Aid for Scientiﬁc Research, #16091210 (2004) and #17700173 (2005).

20

References

T3 (H)

Intelligibility Score, %80
T2 (L)60

T1 (L)[1] T. Watanabe, ”A Study on Voice Setting of Screen
Readers for Visually-Impaired PC Users,” The IEICE Transactions on Information and Systems,
Pt.1, Vol.J88-D-I, No.8, pp.1257-1260, Aug 2005
(in Japanese).

0
0

20

40

60

80

100

N-WWL

Figure 7: N-WWL and intelligibility (L-L-H of Exp.2)[2] C. Asakawa, H. Takagi, S. Ino, T. Ifukube, ”Maximum listening speeds for the blind,” Proceedings
Conference of International Community for Auditory Display 2003, pp. 276-279, 2003.

100
T3 (H)

Intelligibility Score, %[3] C. Asakawa, H. Takagi, S. Ino, T. Ifukube, ”The
Optimal and Maximum Listening Rates in Presenting Speech Information to the Blind,” Journal of
Human Interface Society, Vol.7, No.1, pp.105–111,
2005 (in Japanese).

T2 (H)80

T1 (H)

60[4] T. Nishimoto, S. Sako, S. Sagayama, K. Ohshima,
K. Oda, T. Watanabe: ”Eﬀect of Learning on Listening to Ultra-Fast Synthesized Speech,” Proceedings of the 28th IEEE Engineering in Medicine andBiology Society Annual International Conference(EMBC2006), pp.5691-5694, New York, Sep 2006.

40

20[5] T. Nishimoto, T. Watanabe: ”The Learning Eﬀects
and Age-Related Eﬀects on Listening to Ultra-Fast
Speech,” Proceedings of Human Interface 2007,
3134, pp.937-942, Sep 2007 (in Japanese).

0
0

20

40

60

80

100

N-WWL

[6] S. G. Hart, L. E. Staveland: “Development ofNASA-TLX (Task Load Index): Results of empirical and theoretical research,” in P. A. Hancock and
N. Meshkati (Eds.) Human Mental Workload, Amsterdam, North Holland Press (1998).

Figure 8: N-WWL and intelligibility (H-H-H of Exp.2)
out task with less top-down information.
Our future work includes the following topics:• Examination of the eﬀect of learning in longer period.

[7] S. Haga: The Theory and Measurement of Mental Workload, Japan Publication Service, 2001 (inJapanese).

• Examination of diﬀerence of the eﬀect by age andgender.

[8] S. Amano, T. Kondo, S. Sakamoto, Y. Suzuki:
Japanese Speech Dataset for Familiarity-Controlled
Spoken-Word Intelligibility Test (FW03), NII
Speech Resources Consortium (2006).

• Eﬀects of smaller changes of speaking rate andword familiarity.
• Eﬀects of instructions to subjects in the experiments.
• Applications to speech synthesis technology andlanguage study education.

6124