Effect of Learning on Listening to Ultra-Fast Synthesized Speech
Takuya Nishimoto, Shinji Sako, Shigeki Sagayama, Kazue Ohshima, Koichi Oda and Takayuki Watanabe

Abstract— A text-to-speech synthesizer that would produce
easily understandable voices at very fast speaking rates isexpected to help persons with visual disability to acquire
information effectively with screen reading softwares. We investigated the intelligibility of Japanese Text-to-Speech systems
at fast speaking rates, using four-digit random numbers as thevocabulary of the recall test.
We also studied the fast and intelligible text-to-speech engine, using HMM-based synthesizer with the corpus with fast
speaking rate. As the results, the statistical models trained withthe fast speaking corpus was effective. The learning effect wassignificant in the early stage of the trials and the effect sustained
for several weeks.

I. INTRODUCTION
According to a survey by the Japanese Cabinet Office in
2001, there are around 306,000 persons with visual disability
in Japan. Those who can use Braille among them are lessthan 10%. Braille is not commonly used because it isespecially difficult for elderly people with visual disability tolearn it. On the other hand, more than 10,000 persons withvisual disability use personal computers (PC), according tothe same survey.
It is important for persons with visual disability to use
PCs and/or Internet with voice, which allows real-timecommunication and gives the chance of taking part to social
activities. Recently, screen-readers for the Japanese versionof Microsoft Windows have been developed, which help thepersons with visual disability to access the Web and read orwrite e-mails. They are sold at low prices and do not require
any additional hardware.
To make such systems easy to use, we must consider how
quickly and accurately the users understand information bylistening to the voice of the Text-to-Speech (TTS) systems.
Watanabe [1] investigated how the PC users with visualdisability in Japan are setting the voice of screen readers.
He reported that most of the users are using TTS with themaximum reading rate that the softwares can read with. In
many cases, it was double the speed of the normal speaking
rate.
The aim of our research is to improve the voice quality ofthe TTS for the screen readers. We must therefore evaluate
objectively the various factors of TTS. For example, whendeveloping better speaker models for TTS, it is important toprove that the technique is effective. In this paper, we focus
Takuya Nishimoto, Shinji Sako, Shigeki Sagayama are with theGraduate School of Information Science and Technology, The University of Tokyo, 7-3-1, Hongo, Bunkyo-ku, Tokyo, 113-8656, JAPANnishi@hil.t.u-tokyo.ac.jp
Kazue Ohshima, Koichi Oda and Takayuki Watanabe are with theDepartment of Communication, Tokyo Woman’s Christian University, 2-6-1Zenpukuzi, Suginami-ku, Tokyo, 167-8585, JAPANon the relationship between the reading rate of TTS and theintelligibility (recall rate). Using the method we propose, one
can expect to be able to choose the TTS engine or the speakermodel which is the most intelligible, in other words, the one
through which users with visual disability can understandinformation via fast-speaking voice in the shortest time.
There are many related works which propose different
evaluation methods of speech synthesis systems’ performance. For example, the evaluation method standardizedby JEITA [2] consists of tree parts: the correctness ofpronunciations, the intelligibility and the overall judgment.
Our research, however, focuses on the word intelligibility
test. The familiarity of the words used may have an effect on
the performance of the recall test. We thus chose to use fourdigit random numbers as the vocabulary of the recall test, as
one can assume that the familiarity to random numbers does
not vary much among the users. Very few works investigatedthe learning effect for synthesized voices, but the users may
adapt to the voice and performance then increase during thesessions of the recall tests. We assumed that this was also thecase for users of TTS, and that individual variations of thislearning effect could not be disregarded. We thus investigated
how the users can learn to listen to fast speaking synthesizedvoice.

II. R ELATED W ORKS
A. Listening Speed for the Persons with Visual Disability
In a prior work, Asakawa et al. [3], [4] created rapidlyspoken Japanese speech sentences by using the timestretch/compression function of the CoolEdit audio processing software and shortened recorded human voices linearlyon the time axis. Then the stimuli were evaluated by subjects
with visual disability who are skilled users of screen reading
softwares. According to their work, the suitable speed and thehighest speed are defined respectively as the speaking rate
for which listeners can understand a sentence sufficiently,
and the speaking rate for which listeners can understandapproximately 50% of the words in the sentence. They
report results which show that skilled listeners of synthesized
speech assess the suitable speed (recall rate = 90%) as 11001170 morae/minute and the highest speed (recall rate =50%) as 1400-1500 morae/minute, respectively. Most of thecommercial text-to-speech engines could not produce voices
at such fast speaking rates, so their work suggested a way
to improve non-visual user interfaces for people with visualdisability.

B. HMM-based Speech Synthesis
In our study, GalateaTalk [5], [6] and HTS (HMM-BasedSpeech Synthesis System) [7] are used as the Japanese TTS
system. GalateaTalk, the first open-source Japanese TTS,
consists of a text analyzer and a waveform generation engine.
The text analyzer decomposes arbitrary Japanese input texts
containing Kanji, Kana, alphabetic and numeric characters,
and optionally embedded tags specifying the speaking style,linguistic information including pronunciation, accent type,part of speech, etc., partly utilizing ChaSen [8] and newly
developed dictionaries for Japanese morphological analysis.
The waveform generation engine, derived from the corresponding part of HTS, is a HMM(Hidden Markov Models)based speech synthesizer. The speech spectrum is consideredas the observation output of the states of an HMM, in thesame way as in speech recognition systems. Additionally,two HMMs are respectively used to model F0 and theduration of each phoneme, i.e. the number of frames in eachstate.
We created rapidly-speaking synthesized voices by using
the HMM-based speech synthesizer and rapidly-speaking
statistical models [9], which are trained from a corpus
of rapidly-speaking human voices. We investigated more
precisely the durations of phonemes of the statistical models
in [12]. The system can produce voices with a speed fasterthan 1500 morae/minute (i.e. 25 morae/second). The durationof each segment of the phoneme can be generated using themeans and variances trained by the corpus. When the target
speaking rate is higher than the speaking rate of the voicesof the corpus, if the variance of the duration is large, thenumber of frames corresponding to the segment is highlydecreased, and if the variance of the duration is small, it isless decreased. One expects that the use of HMM-based TTS
and rapidly-speaking statistical model enables to produce
intelligible and fast-speaking TTS.
C. Perception of Speech and Speaker
Legge et. al. [13] investigated the learning of unfamiliar
voices. In their experiment, subjects listened to a series ofrecorded voice samples obtained from unfamiliar speakers
and were then given a two-alternative forced-choice recognition test. According to their results, voice learning wasinferior to face learning.
Palmeri et. al. [14] suggested that detailed information
about a speaker’s voice is retained in long-term episodic
memory representations of spoken words. In their work,
recognition memory for spoken words was investigated witha continuous recognition memory task. Independent variables were the number of intervening words between initialand subsequent presentations of a word, the total numberof speakers in the stimulus set, and whether words wererepeated by the same voice or a different one.
Nygaard et. al. [15] suggested that speech perception may
involve speaker-contingent processes. The results of theirexperiments showed that the ability to identify a speaker’s
voice improved intelligibility of novel words produced bythat speaker.

TABLE ID URATION OF THE S YNTHESIZED S PEECHSpeakerNormalFastDuration of sentence(ms)
2090
1325

Speed(morae/min)402
634

III. F IRST E XPERIMENT
A. Procedure
We performed a preliminary experiment to show that therecall rates decrease at fast speaking rates, and that subjects
can learn to listen to fast speaking voices during the tests [9].
We created two corpora from the voice of a male professional narrator. The normal-speed corpus is a set ofspeech utterances corresponding to 503 phonetically balanced Japanese sentences. The fast-speed corpus is a set ofspeech utterances corresponding to 100 sentences which area subset of the 503 previous sentences.
Both corpora were used to create statistical speaker models
for GalateaTalk TTS engine. In the model, each phoneme
consists of five states. Table I shows the duration and thespeed of the speech synthesized using the speaker models
on the Japanese test sentence “Bango-wa ichi ni san yondesu” (The number is one two three four). The duration ofeach phoneme of this synthesized speech is as follows:• Normal: b[70] a[95] N[100] g[65] o[100] o[80] w[95]a[100] i[95] ch[105] i[90] n[55] i[95] s[100] a[90]N[100] y[85] o[85] N[100] d[45] e[95] s[165] U[80],• Fast: b[45] a[70] N[60] g[45] o[45] o[70] w[45] a[50]i[60] ch[65] i[40] n[50] i[60] s[75] a[65] N[65] y[70]o[55] N[65] d[30] e[55] s[80] U[60].
It turned out that the TTS with the fast speaker model
could speak approximately 1.6 times faster than the TTS
with the normal speaker model. We also confirmed that theduration is quantized with the frame shift time (5ms). The
frame is the unit of generation and processing of speech
signals inside the waveform generation engine.
We used two configurations of TTS for testing. The Fast-1
uses the statistical information of the Fast corpus to produce
the duration, F0 and the spectrum. The Fast-2 uses thestatistical information of the Fast corpus to produce theduration, and the statistical information of the Normal corpus
to produce F0 and the spectrum. Both conditions use thesame duration model, therefore the speaking rates of theirsynthesized voices are exactly the same.
The subjects were three women who are university students and are not visually/hearing impaired.
The stimuli are four-digit random numbers embedded in
the sentence as described above. The subjects used Windows
PC to listen to the stimuli and input the answer. Headphones(Sony MDR-CD480) were used during the tests. The subjects
were indicated that if there were some digits which they
could not hear, question marks were available for input
instead of the numbers, such as ’1 ? 3 0.’
The speaking rates we used were 0.3, 0.4, 0.5, 0.6 and0.7, these values indicating the ratio of the duration of theFig. 1.

Average score for each speed in the first experiment.

synthesized speech to the duration of the original speech.
The durations of each segment of the phonemes were notgenerated by proportional allotment, but generated in consideration of the means and the variances of the statisticalmodel.
The trial T1, consisting of 200 recall tests, was performed,
followed by the first training session, consisting of 50
training tasks, and the trial T2, which is the same as T1.
During the training task, the subjects listen to a fast-speedvoice, then input the answer, and finally listen to the correct
answer in the form of the corresponding normal-speed voice.
The trial T3, the second training session and the trial T4
were conducted three days after in the same way.
B. Results
Figure 1 shows the average score of the recall task
in the first experiment. The recall rate decreases for fast
speaking rates, therefore proving the appropriateness of ourexperiments. The rates of Fast1 and Fast2 are very similar,
showing that the difference of spectrum has little impact onthe speech quality if the durations are the same.
Figure 2 shows the learning effect in the first experiment.
The results indicate that all the subjects learned to listen tothe voice very quickly, and that the effect sustained for atleast several days.
IV. S ECOND E XPERIMENT
A. Procedure
We performed a second experiment to investigate theindividual variations of the learning effect [10], [11].
The experiment was conducted over three weeks. Nineteen
female university students participated in the sessions of thefirst week. Five of them also participated in the sessions ofthe second and third week succeedingly. The data of sixteen
people was used for analysis. These sixteen people were
not visually/hearing impaired, and four persons among them
participated in all sessions.
The stimuli are four-digit random numbers embedded in
the same sentence as in the first experiment. The subjects

Fig. 2. Learning effect in the first experiment (s1-s3:subjects, all:average).

used Windows PC to listen to the stimuli and input the answer. Headphones (Panasonic RP-H750-S) were used during
the tests. Question marks were available for input.
In this experiment, the trial T1, consisting of 200 recall
tests, was performed, followed by the first training session,consisting of 25 training tasks, and the trial T2. The trial T1
consisted of the iterations of a set of 50 tasks, and the trial
T2 consisted of the iterations of the set of the same tasks in
the reverse order.
The waveform generation engine was changed from
GalateaTalk to the original HTS, and the frame shift time waschanged from 5ms to 2ms. To produce the duration, F0 andthe spectrum, the Fast and the Normal speaker models used
the statistical information of the Fast and Normal corpora,respectively.
In order to improve speech quality, we introduced a
restriction on the duration of each state in a phoneme,
which must not be 0ms. Therefore, the speaking rates ofthe synthesized voices were not the same as the target rates
which we indicated to the engine.
B. Results
Figure 3 shows the average recall rates in the secondexperiment. The results of T11-T14 and T21-T24 correspondto the iterations of the task set in the trial T1 and T2,respectively. Significant difference between T11 and T12 wasobserved . The F-test statistic value was F (2, 30) = 30.93
at the significance level of p < 0.05.
The results indicate that the learning effect is significant
only in the early stage of the trials.
Figure 4 shows the average recall rates in the trial T1 andT2 in the second experiment. One can see that the recall
rates of the Fast model surpasses that of the Normal model,
regardless of the learning effect.
Figure 5 shows the evolution on the three weeks ofexperiments of the average recall rates of four subjects. Onecan see that the learning effect sustains for three weeks.
Though the number of subjects was limited, we investigated the individual variations of the learning effect. The
variation of the individual recall rate is especially large inFig. 3. Recall rates in the first week of the second experiment. The bars
represent the distributions.

Fig. 5. Evolution of the recall rates during the three weeks of the secondexperiment. The bars represent the distributions.

in-Aid for Scientific Research on Priority Areas, #16091210,
2004.
R EFERENCES

Fig. 4.

Recall rates in trials T11 and T24 of the second experiment.

the sessions of the first two weeks. In the sessions of thethird week, however, the variation decreases,as can be seenin Figure 5. The performance of a subject whose recall rate
was lower than the other subjects in the first week came close
to that of the others in the third week.
V. CONCLUSION
In this article, we investigated evaluation methods ofJapanese TTS at fast speaking rates using a recall test ofrandom numbers. We showed that the proposed method iseffective for comparing the intelligibility of TTS.
Future work includes improvement of our HMM-based
speech synthesizer and the rapidly-speaking statistical models, evaluation by subjects with visual disability, evaluation
using a vocabulary other than the random numbers, andapplications to voice interface systems/spoken dialog systems
for the Web and home electric appliances [16].
VI. ACKNOWLEDGMENTS
This work was partially supported by the Ministry ofEducation, Culture, Sports, Science and Technology, Grant-[1] T. Watanabe, ”A Study on Voice Setting of Screen Readers for
Visually-Impaired PC Users,” The IEICE Transactions on Informationand Systems, Pt.1, Vol.J88-D-I, No.8, pp.1257-1260, Aug 2005 (inJapanese).
[2] Japan Electronics and Information Technology Industries Association(JEITA), Speech Synthesis System Performance Evaluation Methods,JEITA IT-4001, Feb 2003 (in Japanese).
[3] C. Asakawa, H. Takagi, S. Ino, T. Ifukube, ”Maximum listening speeds
for the blind,” Proceedings Conference of International Community forAuditory Display 2003, pp. 276-279, 2003.
[4] C. Asakawa, H. Takagi, S. Ino, T. Ifukube, ”The Optimal and Maximum Listening Rates in Presenting Speech Information to the Blind,”
Journal of Human Interface Society, Vol.7, No.1, pp.105–111, 2005
(in Japanese).
[5] http://hil.t.u-tokyo.ac.jp/∼galatea/[6] http://www.astem.or.jp/istc/[7] http://hts.ics.nitech.ac.jp/[8] http://chasen.aist-nara.ac.jp/[9] T. Nishimoto, S. Sako, S. Sagayama, K. Oda, T. Watanabe, ”Evaluationof text-to-speech synthesizers at fast speaking rates,” IEICE Technical
Report, WIT2005-5, pp.23-28, May 2005 (in Japanese).
[10] T. Nishimoto, S. Sako, S. Sagayama, K. Ohshima, K. Oda, T. Watanabe, ”Listener’s familiarizations to text-to-speech synthesizers at fastspeaking rates,” Proc. ASJ Autumn Conf., 3-6-14, pp.355-356, Sep
2005 (in Japanese).
[11] K. Ohshima, T. Nishimoto, T. Watanabe, ”Relationship between oral
comprehension and hearing experience of fast TTS for the visuallydisabled,” IEICE Technical Report, WIT2005-43/SP2005-81, pp.1924, Oct 2005 (in Japanese).
[12] S. Sako, T. Nishimoto, S. Sagayama, ”A study on rapid speechsynthesis using HMM-based speech synthesis technique,” Proc. ASJAutumn Conf., 3-6-15, pp.357-358, Sep 2005 (in Japanese).
[13] G.E. Legge, C. Grosmann, C.M. Pieper. “Learning unfamiliar voices,”
Journal of Experimental Psychology: Learning, Memory and Cognition, 10, 298-303, 1984.
[14] T.J. Palmeri, S.D. Goldinger, D.B. Pisoni, ”Episodic encoding ofvoice attributes and recognition memory for spoken words,” Journalof Experimental Psychology: Learning Memory and Cognition, 19,pp.309–328, 1993.
[15] L.C. Nygaard, M.S. Sommers, D.B. Pisoni, ”Speech perception asa talker-contingent process,” Psychological Science, Vol. 5, No. 1,pp.42–46, Jan 1994.
[16] T. Watanabe, M. Yasumura, K. Oda, T. Nishimoto, ”Basic researchon speech recognition for the visually impaired and its application tovoice interaction,” IEICE Technical Report, WIT2004-74, pp.7-12,
Mar 2005 (in Japanese).