# Developing a Child Friendly Text-to-Speech System

Authors: Agnes Jacob; P. Mythili
Journal: Advances in Human-Computer Interaction
 (2008)

---

## Abstract

This paper discusses the implementation details of a child friendly, good quality, English text-to-speech (TTS) system that is phoneme-based, concatenative, easy to set up and use with little memory. Direct waveform concatenation and linear prediction coding (LPC) are used. Most existing TTS systems are unit-selection based, which use standard speech databases available in neutral adult voices. Here reduced memory is achieved by the concatenation of phonemes and by replacing phonetic wave files with their LPC coefficients. Linguistic analysis was used to reduce the algorithmic complexity instead of signal processing techniques. Sufficient degree of customization and generalization catering to the needs of the child user had been included through the provision for vocabulary and voice selection to suit the requisites of the child. Prosody had also been incorporated. This inexpensive TTS system was implemented in MATLAB, with the synthesis presented by means of a graphical user interface (GUI), thus making it child friendly. This can be used not only as an interesting language learning aid for the normal child but it also serves as a speech aid to the vocally disabled child. The quality of the synthesized speech was evaluated using the mean opinion score (MOS).


---

## Body

## 1. Introduction

There
are  various  critical factors  to be considered while designing a TTS system that
will produce intelligible speech. Any TTS should be appealing to the child user
whether it is used as a language learning aid or as a vocal aid. Children
find learning more fun when their typed inputs are mapped to vocalized outputs. The first crucial step in
the design of any concatenative TTS system is to select the most appropriate
units or segments that result in smooth concatenation. This involves a tradeoff
between longer and shorter units. In the case of shorter units, such as the
phonemes, less memory is required. But, the sample collection and labeling
procedures become more complex. The number of segments and the time required to
cover the language increase steadily from word to the phoneme. In addition to
being part of computationally manageable inventory of items, the synthesis
segments chosen should capture all the transient and transitional information.
The latter had been emphasized throughout this work, which in turn contributed
to the smooth concatenation of speech segments in this TTS.Even though speech is analog, phonemes are
discrete. Inclusive of allophones, phonemes are less than hundred and are
mainly the vowels, diphthongs, and consonants [1]. These allophones can be
concatenated to produce smooth utterances without enormous computational effort
compared to concatenation from the basic set of just 44 phonemes. Although
phoneme appears to be an attractive linguistic unit for speech synthesis
because of its limited number, most efforts [2] to string them together have
failed. Pronunciation of phonemes depends on contextual effects, speaker's
characteristics, and emotions. During continuous speech, the articulator
movements depend on the preceding and the following phonemes. This causes some
variations on how the individual phoneme is pronounced which lead to
spontaneous variations in phoneme quality that is often known as coarticulation
[3]. Thus, as per available facts [4], phoneme-sized building blocks were found
to be unsatisfactory as synthesis segments because of the coarticulatory
effects of the adjacent sounds. Further,
one of the problems associated with segmenting words and storing the excised
phonemes is the preservation of the characteristics of the sound, which is
present at the transitions at the beginning and end of the segment. The
characteristic sound of these transitions could be lost from both segments if
they are smoothed together using signal processing techniques, resulting in a
loss of naturalness in the utterance [5]. Hence,  a major challenge in this
method is that the boundaries between the phonemes correspond to areas that are
acoustically volatile. Speech synthesized with phonemes as units is
intelligible when each phoneme is represented by several allophones in the
segment database. Different emotions and speaker characteristics could be
implemented with such a database.Figure1 shows the major steps for
implementing this work by the two methods. In the first method, a smooth,
direct waveform concatenation of phoneme segments had been done, maximizing the
speech quality in terms of naturalness and intelligibility. In the second
method, LPC had been used to reduce memory requirements. A comparison of the
above two methods, in terms of performance as well as the resources used, had
been done using MOS. It was found that
both methods gave a good quality TTS for children, which are not overloaded by
too many analytical aspects and can be simulated in a short period compared to
any other TTS. Sufficient care of the various linguistics aspects ensured
natural sounding speech, inclusive of emotions.Figure 1
Overview of the TTS implementation.


## 2. Text/Vocabulary Selection

Selecting
the vocabulary of the TTS is often referred to as fixing the target, since each
of the utterances, for which this TTS system is designed, is called target and
it is the speech output corresponding to a phonemic input. In the text selection phase, the target vocabulary as
well as pitch, amplitude, and duration or speed of utterance was chosen. These
varied styles correspond to different emotions. Being an experimental work with
phonemes, the target was chosen to be a limited vocabulary of 635 different
words covered by thirty-eight different phonemes and their allophones.
Restriction of the targets words to a specific domain was seen to give better
performance, since ultimately, the prosody is limited to the extent embedded in
the recorded speech from which the segments are excised. Moreover, limiting the
variety of utterances to one’s anticipated needs can help one to provide
sufficient samples of the segments without unnecessarily increasing the memory
requirements of the database. This TTS had been initially designed to meet the
minimal vocal requirements of a vocally impaired child. Alternately, it can also be modified to suit the language learning
requirements of a normal child.


## 3. Text-to-Phoneme Conversion

The
input text should be valid in the sense that it should belong to the set of
target words. An error message is displayed in the graphical user interface
(GUI), for cases falling outside the predetermined vocabulary (set of fixed
inputs for which the speech database is designed). This project uses an event-driven
front-end, which is simulated by means of a MATLAB program to transcribe input
target words (the decided vocabulary) into their corresponding different
phonemic forms. The international
phonetic transcription (IPA) symbols are used. A dictionary-based approach has
been used here for text-to-phoneme conversion. The pronunciation   dictionary
is initially designed in the context of the 635 different, typed input words
only. The algorithmic complexity in the TTS design is considerably reduced due
to this dictionary-based approach. The program stores words along with their
correct pronunciation. This method is accurate and quick. The transcriptions
are obtained from the sixteenth edition [6] of the English Pronouncing Dictionary. The pronouncing dictionary of this TTS provides essential
information
such as pronunciation of proper names and variant pronunciation than is usual
in a general dictionary. Therefore, the primary aim of this TTS dictionary is
to list pronunciation likely to be used by learners such as one that reflects
the regional accent.After considerable linguistics study and
based on the findings from the literature survey, certain words are selected
for the text corpus (to be recorded). These words are constituted by the
phonemes of desired interest (present in the target words). Choosing specimen
words for recording, followed by segmenting of these word recordings to extract
the desired phonemes is the next
crucial task and is related to linguistics.


## 4. Design of the Text Corpus

Careful design of the text corpus is an essential
prerequisite to speech database design and involves the preparation of an
inventory of words, which have the same phonemic constituents (along with
contexts) of the target words. Care is taken to include three to five examples
of words with the same allophone and these words are recorded. Even though the
simplest approach is to add data to the speech database till the quality does
not improve anymore, in all practical cases, there is a tradeoff between the
quality and quantity. Each of the constituent phoneme segments of the target
words is examined, and care is taken to select and record words having these
very features. Examples of certain “design rules” formulated on the basis of
the linguistics study are given below [7]. These rules are also followed when
selecting segments for concatenation as follows.(1)
A vowel preceding a voiced
consonant in the same syllable tends to be longer than the same vowel   preceding a voiceless consonant.(2)
The length of long vowels and
diphthongs is very much reduced when they occur in syllables closed by
consonants such as /p, t, k, s, h/.(3)
The consonant letter y can act as a
vowel and as a consonant.(4)
The length of a phoneme is the
least when it is in the middle of the word and maximum at the end. The use of
stressed words served to increase the duration of the phonemes by more than
20%. Since content words (words that are important for their meaning, e.g.,
nouns, adjectives, adverbs) are stressed, the text corpus to be recorded is
constituted mostly by content words.Based on the above criteria, a table was
prepared with each row containing a word to be synthesized, its phonemic
constituents, and examples of words with these constituent phonemes in varied
positions in the word. A sample is shown in Table1 above. The database is
designed to offer sufficient coverage of the units to make sure that an
arbitrary input sentence can be synthesized with more or less
homogeneous quality.Table 1
The design process.
Word (target)TranscriptionWords to recordDescriptionEyeaIBuy, why, bideB∣aI, w∣aI, b|aI|dDiphthong. greater
duration on first sound a: cases favored
are word final
or followed by voiced consonant.


## 5. Phoneme Database Development

A series of preliminary
stages have  to be fulfilled before the synthesizer can produce its first
utterance. The database of WAV
files is obtained by recording the natural voicing of the
targets. A sampling frequency of 22.05 kHz was used to make the synthesized
voice sound more pleasant. An amplitude resolution of 16 bits was used [8]. The
recordings were done in male and female voices. Repeated segmentation of these
speech files was done to excise phonemes/allophones from the speech database.
As the output quality of any concatenative speech synthesizer relies heavily on
the accuracy of segment boundaries in the speech database [9], manual method of
segmentation was used.In this work, the
coarticulatory effect was put to good use by excising phonemes from different
environments (surrounding phonemes), adding to the variability and naturalness
of the database. These allophone segments were also stored as WAV files after
appropriate labelling.


## 6. Concatenation

Relevant
literature cites that concatenating and modifying the prosody of speech units
without introducing audible artifacts are difficult [3]. In this work, this
problem was overcame by appropriate linguistic design of the text corpus and
careful preparation of the speech database. Moreover, the acoustic inventory
used consists of a rich storage of needed allophones rather than phonemes.
Several linguistic rules have been closely followed. For instance, we apply
that vowels are key components [2] determining the synthesized voice quality.
After manually editing the WAV files and trying out the direct waveform concatenation to identify the
right constituent segments for any word in the predetermined vocabulary, the
appropriate segments are concatenated programmatically to yield the synthesized
speech. Sentences could also be synthesized with the prosody corresponding to those
embedded in the segments [10]. Sentences made from segments of longer
duration give rise to slow utterances and correspond to sad emotions, while
sentences from short duration segments give rise to fast utterances
corresponding to any happy, energetic person. These varied styles could be
chosen using tags in the text entry. Allowing the choice between male and
female voices provided an additional degree of customization [11]. A female voice, sampled at 22.05 kHz, was
played back at a reduced sampling frequency of 17.5 kHz in order to produce a
distinct male voice. Such voice conversion attempts proved to be amusing to
children, instilled in them curiosity about the mechanism of speech production
and improved their intellectual ability. This TTS system is a brilliant way
to expose interested students to the basics of phonetics and motivates them to
setup any TTS of their choice.


### 6.1. Parametric Representation
of Speech

The second
method uses the linear predictive coded (LPC) speech [3], wherein wave files
are replaced by parametric models. In the LPC method of resynthesis, the
voicing, pitch, gain, and LPC parameters were found for the down-sampled
versions of the above constituent wave files, and speech was resynthesized. The
source filter model [12] of speech production used here hypothesizes that an
acoustic speech signal can be seen as a source signal (the glottal source, or
noise generated at a constriction in the vocal tract), filtered with the
resonances in the cavities of the vocal tract, downstream from the glottis, or
the constriction. The LPC model was used for the prosody modification as it
explicitly separates the pitch of a signal from its spectral envelope. Speech
is parameterized by an amplitude control, voiced/voiceless flag, Fundamental
Frequency (F0), and filter coefficients at a small interval. The F0 is the physical aspect of speech
corresponding to perceived pitch. As a periodic signal, voiced speech has
spectra consisting of harmonics of the fundamental frequency of vocal fold
vibration. The loudness control is determined from the power of speech at the
time frame of analysis. The concatenative approach to speech synthesis requires
that speech samples should
be stored in some parametric representation that will be suitable for
connecting the segments and changing the signal characteristics like loudness and F0. F0 extraction
algorithms determine the voicing of speech as well as the fundamental
frequency.The LPC analysis is done for the same
candidate WAV files
used in the direct waveform concatenation. The pitch detection function
(program) developed in MATLAB
takes the speech audio signal and divides it into 30 milliseconds
frames, over which speech is quasistationary. These overlapping frames start
every 15 milliseconds and were further Hamming windowed to avoid distortion [3]. The function returns the pitch value in hertz for voiced frames, whereas it returns a zero for unvoiced frames. Monotone speech was
produced from the synthesizer by replacing the pitch
signal calculated by the function with
a vector of constant values. The constant values, that were selected, were 100 Hz and 380 Hz, which correspond to male and child voices, respectively.The Synthesis
function returns the reconstructed audio. Using the voicing, pitch, gain, and
LPC coefficients, each frame was synthesized. These were then put together to
form the synthesized speech signal. The initial 30 milliseconds
signal was created based on the pitch information. If the pitch is zero, the
frame is unvoiced. This means that the 30 milliseconds signal needs to be
composed of white noise. White noise is
noise with a flat spectrum (uniform power spectral density) over the entire
frequency range of interest. The term “white” is used in analogy with white
light, which is a superposition of all visible spectral components.Unvoiced
excitation is usually modelled as such a white noise limited to the bandwidth
of speech [3]. In this implementation, MATLAB
function “randn” was used for producing white noise. For nonzero pitch, a 30 milliseconds
signal was created with pulses at the pitch frequency. These initial signals
were filtered using the gain and filter coefficients and then connected
together in overlapping frames, for smooth transition from one frame to the
next.


### 6.2. MOS Evaluation

Evaluating synthetic speech formally is difficult as there
are many complex factors, dealing with intelligibility, naturalness, and the
flexibility to simulate different voices and speaking rates. Due to lack of
suitable standards for comparison, objective methods could not be used in this
work. Hence, evaluating synthetic speech output was almost exclusively a
subjective process. Certain subjective tests such as the dynamic rhyme test
(DRT) are not realistic for practical application [3]. Therefore, mean opinion score (MOS) [13] has
been used to evaluate the quality of this TTS, mainly  in terms of intelligibility and naturalness.
We have used the five level scales given in Table 2 as they are easy and
provide some instant, explicit information.Table 2
Scales used in MOS.
RatingMOS1Bad2Poor3Fair4Good5ExcellentAn MOS rating
greater than 4 indicates good quality. Any rating between 3.5 and 4 indicates
that the utterance possesses telephonic communication quality.
Ten volunteers without any known hearing disabilities participated in
the MOS evaluation of the outputs of both phases. The listeners were all
nonnative speakers of English. As is
required, none were experts in TTS. There were five teachers (one of them well
versed in linguistics), two 8-year-old kids (who are used to synthetic voices),
two doctors, and 1 person who had recovered from a voice loss recently. The
participants were briefed about the project.


## 6.1. Parametric Representation
of Speech

The second
method uses the linear predictive coded (LPC) speech [3], wherein wave files
are replaced by parametric models. In the LPC method of resynthesis, the
voicing, pitch, gain, and LPC parameters were found for the down-sampled
versions of the above constituent wave files, and speech was resynthesized. The
source filter model [12] of speech production used here hypothesizes that an
acoustic speech signal can be seen as a source signal (the glottal source, or
noise generated at a constriction in the vocal tract), filtered with the
resonances in the cavities of the vocal tract, downstream from the glottis, or
the constriction. The LPC model was used for the prosody modification as it
explicitly separates the pitch of a signal from its spectral envelope. Speech
is parameterized by an amplitude control, voiced/voiceless flag, Fundamental
Frequency (F0), and filter coefficients at a small interval. The F0 is the physical aspect of speech
corresponding to perceived pitch. As a periodic signal, voiced speech has
spectra consisting of harmonics of the fundamental frequency of vocal fold
vibration. The loudness control is determined from the power of speech at the
time frame of analysis. The concatenative approach to speech synthesis requires
that speech samples should
be stored in some parametric representation that will be suitable for
connecting the segments and changing the signal characteristics like loudness and F0. F0 extraction
algorithms determine the voicing of speech as well as the fundamental
frequency.The LPC analysis is done for the same
candidate WAV files
used in the direct waveform concatenation. The pitch detection function
(program) developed in MATLAB
takes the speech audio signal and divides it into 30 milliseconds
frames, over which speech is quasistationary. These overlapping frames start
every 15 milliseconds and were further Hamming windowed to avoid distortion [3]. The function returns the pitch value in hertz for voiced frames, whereas it returns a zero for unvoiced frames. Monotone speech was
produced from the synthesizer by replacing the pitch
signal calculated by the function with
a vector of constant values. The constant values, that were selected, were 100 Hz and 380 Hz, which correspond to male and child voices, respectively.The Synthesis
function returns the reconstructed audio. Using the voicing, pitch, gain, and
LPC coefficients, each frame was synthesized. These were then put together to
form the synthesized speech signal. The initial 30 milliseconds
signal was created based on the pitch information. If the pitch is zero, the
frame is unvoiced. This means that the 30 milliseconds signal needs to be
composed of white noise. White noise is
noise with a flat spectrum (uniform power spectral density) over the entire
frequency range of interest. The term “white” is used in analogy with white
light, which is a superposition of all visible spectral components.Unvoiced
excitation is usually modelled as such a white noise limited to the bandwidth
of speech [3]. In this implementation, MATLAB
function “randn” was used for producing white noise. For nonzero pitch, a 30 milliseconds
signal was created with pulses at the pitch frequency. These initial signals
were filtered using the gain and filter coefficients and then connected
together in overlapping frames, for smooth transition from one frame to the
next.


## 6.2. MOS Evaluation

Evaluating synthetic speech formally is difficult as there
are many complex factors, dealing with intelligibility, naturalness, and the
flexibility to simulate different voices and speaking rates. Due to lack of
suitable standards for comparison, objective methods could not be used in this
work. Hence, evaluating synthetic speech output was almost exclusively a
subjective process. Certain subjective tests such as the dynamic rhyme test
(DRT) are not realistic for practical application [3]. Therefore, mean opinion score (MOS) [13] has
been used to evaluate the quality of this TTS, mainly  in terms of intelligibility and naturalness.
We have used the five level scales given in Table 2 as they are easy and
provide some instant, explicit information.Table 2
Scales used in MOS.
RatingMOS1Bad2Poor3Fair4Good5ExcellentAn MOS rating
greater than 4 indicates good quality. Any rating between 3.5 and 4 indicates
that the utterance possesses telephonic communication quality.
Ten volunteers without any known hearing disabilities participated in
the MOS evaluation of the outputs of both phases. The listeners were all
nonnative speakers of English. As is
required, none were experts in TTS. There were five teachers (one of them well
versed in linguistics), two 8-year-old kids (who are used to synthetic voices),
two doctors, and 1 person who had recovered from a voice loss recently. The
participants were briefed about the project.


## 7. Results and Discussion

Synthesis
of polysyllabic words was done without any difficulty. Each
participant randomly selected and listened to 25 words. The average MOS rating
for each stimulus (TTS utterance) was later calculated and tabulated. As these
tests were aimed at assessing the segmental as well as the suprasegmental
characteristics, [14] the volunteers took further listening tests of a prepared
list of 20 sentences and were asked to rate these using MOS. A sample is given
in Table 3. The MOS tests were conducted individually for each listener.
After hearing a test stimulus, the listener indicated his/her rating on a 5-point
scale. The tests for both methods were administered at a stretch.Table 3
Sample MOS for sentences.
No.Test sentencesMOS rating1Where are you going?4.52Please leave me alone.4A sample of the MOS [13]
evaluation of direct waveform concatenation of phonemes is as given below in
Table 4. The average MOS ratings indicated that the speech output of the TTS
is of good/“toll” quality.Table 4
MOS for WAV concatenation.
No.WordsMOS1Fine52Bill4.93Queue4However, as
expected, results of the MOS evaluation of the LPC-based, parametric,
concatenative TTS indicate that no words are rated as excellent. Listeners
unanimously stated that the intelligibility of words increased considerably
when these were embedded in sentence utterances rather than in isolation.The memory
comparison given in Table5 highlights
the advantage of incorporating the LPC
model. There is considerable saving of memory though at the expense of the
quality of the synthesized speech. The average memory gain factor was found to
be 8.96, thus justifying the use of LPC as a means to achieve database
compression. This is a significant achievement compared to the vast memory
requirements of conventional unit selection-based methods.Table 5
Sample comparison of memory for LPC and WAV file
storage.
WordConstituent
files(X)
WAV  file storage (bytes)(Y)
LPC parametric storage (bytes)(X)/(Y) gain factorSOS01SJ8 O1O82400027608.7Additionally, it has been found that the target vocabulary
could be generalized in the sense that using the database of phonemes suitable
to produce the predefined vocabulary, many other words could also be produced
by suitable concatenation. Hence, this implementation is efficient.As
such, it can be implemented by any person with a basic knowledge in linguistics
and programming. Since all can use the
same basic program, this can, therefore, be self-implemented by students with
minimal guidance from a tutor. Since children are often more receptive to
certain voices like those of their teacher or parent, the database can be
recorded in any of their preferred voices for better and enjoyable learning.
The vocabulary also can be chosen as per the preferences of the child. This TTS in one’s own voice or in any other
preferred voice motivates the child learner to try out new words. The provision
to record user’s own voice and compare it with the TTS utterance provides
sufficient motivation for the child learner to expand his vocabulary; if one
keeps track of correct utterances of TTS and assigns appropriate scores to the
user. Thus word
building exercises can be made more interactive and amusing,
as only
correct words will be vocalized, whereby children can feel out the word and get
immense satisfaction with each complete utterance.Unlike unit
selection concatenative systems [15] which make use of varied speech units and
mostly engineering techniques like cost optimization and signal processing,
this TTS implementation minimized its algorithmic complexity primarily by
incorporating appropriate linguistic aspects like coarticulation and a
carefully designed database to ensure smooth concatenation. Further reduction
in algorithmic complexity was achieved by using table lookup methods for the
grapheme to phoneme conversion. Though there are various speech aids for the
vocally handicapped, any person who once possessed the ability to speak would
normally prefer to use his/her own voice compared to any other robotic voice.
Hence, this will also help children facing the risk of an impending vocal
impairment due to some illness. Besides, this sort of speech synthesizer
requires no additional expense for a person with a good computer along with
speakers. With the proliferation of laptops and notebook computers, mobility is
also not a problem.


## 8. Conclusion

In this paper, the implementation
details of a child friendly phoneme-based concatenative TTS, with sufficient
degree of customization and which uses linguistic analysis to circumvent most
of the problems of existing concatenative systems, have been presented. The use
of a dictionary-based approach for text-to-phoneme conversion along with a
tailored speech database helps to avoid all algorithmic complexities and
concatenation mismatches, characteristic of existing TTS systems. While
conventional unit selection-based TTS requires hundreds of megabytes of memory,
this TTS required only hundreds of kilobytes of memory.Voice conversion feature has been incorporated
in this TTS using the LPC method with provision for varying the voice quality
over a wide range by varying the F0 values in the synthesis stage. Another
feature of this work is that it was implemented using female voice, whereas
most of the successful LPC-based TTSs have been implemented in male voice. This
TTS further has add-on facility in that new words can be synthesized, after
adding these words, their transcription, and constituent wave file names to
their respective databases. The prosody of the utterance can be designed to
vary depending on the nature of the recordings in the speech database  from which the phoneme segments are excised.
Thus, a simple, flexible, and efficient TTS that can be user defined has been
setup with minimum resources to serve multiple purposes. Though this had been
developed for English, it can be suitably modified for any other language. This
TTS was found to be a successful vocal aid/language learning aid as the users
were able to get a real feel of phonemes, the most basic speech units. The
learning environment can be conditioned to any particular accent by using an
appropriate combination of database and pronunciation dictionary. Alternately,
content specific learning too can be encouraged implicitly. By suitable design
of the TTS vocabulary and database, a child can be familiarized with all common
terms associated with any specific topic. Thus, such a TTS helps the child user
get acquainted with the regular as well as any other selective vocabulary.



---

*Source: 597971-2008-09-30.xml*