ISCA Archive

International Workshop on
Spoken Language Translation 2012
Hong Kong, December 6-7, 2012

http://www.isca-speech.org/archive

A Method for Translation of Paralinguistic Information
Takatomo Kano, Sakriani Sakti, Shinnosuke Takamichi, Graham Neubig,
Tomoki Toda, Satoshi Nakamura
Graduate School of Information Science, Nara Institute of Science and Technology
Abstracttion system that not only translates linguistic information, butalso paralinguistic speech information between source andtarget utterances. Our ﬁnal goal is to allow the user to speak
a foreign language like a native speaker by recognizing the
input acoustic features (F0, duration, power, spectrum etc.)
so that we can adequately reconstruct these details in the target language.
From the many different possible paralinguistic featuresto handle, in this paper we chose duration and power. We
propose a method that can translate these paralinguistic features from the input speech to the output speech in continuous space. In this method, we extract features at the levelof Hidden Markov Model (HMM) states, and use linear regression to translate them to the duration and power of HMM
states of the output speech. We perform experiments that use
this technique to translate paralinguistic features and reconstruct the input speech’s paralinguistic information, particularly emphasis, in output speech.
We evaluate the proposed method by recording parallelemphasized utterances and using this corpus to train and test
our paralinguistic translation model. We measure the emphasis recognition rate and intensity by objective and subjective
assessment, and ﬁnd that the proposed paralinguistic translation method is effective in translating this paralinguistic information.

This paper is concerned with speech-to-speech translationthat is sensitive to paralinguistic information. From the manydifferent possible paralinguistic features to handle, in this paper we chose duration and power as a ﬁrst step, proposing amethod that can translate these features from input speech tothe output speech in continuous space. This is done in a simple and language-independent fashion by training a regression model that maps source language duration and power information into the target language. We evaluate the proposed
method on a digit translation task and show that paralinguistic information in input speech appears in output speech, andthat this information can be used by target language speakers
to detect emphasis.

1. Introduction
In human communication, speakers use many different varieties of information to convey their thoughts and emotions.
For example, great speakers enthrall their listeners by not
only the contents of the speech but also their zealous voice
and conﬁdent looks. This paralinguistic information is not a
factor in written communication, but in spoken communication it has great importance. These acoustic and visual cues
transmit additional information that cannot be expressed inwords. Even if the context is the same, if the intonation andfacial expression are different an utterance can take an entirely different meaning [1, 2].
However, the most commonly used speech translation
model is the cascaded approach, which treats Automatic
Speech Recognition (ASR), Machine Translation (MT) and
Text-to-Speech (TTS) as black boxes, and uses words as thebasic unit for information sharing between these three components. There are several major limitations of this approach.
For example, it is widely known that errors in the ASR
stage can propagate throughout the translation process, andconsidering several hypotheses during the MT stage can improve accuracy of the system as a whole [3]. Another lessnoted limitation, which is the focus of this paper, is thatthe input of ASR contains rich prosody information, but
the words output by ASR have lost all prosody information.
Thus, information sharing between the ASR, MT, and TTS
modules is weak, and after ASR source-side acoustic details
are lost (for example: speech rhythm, emphasis, or emotion).
In our research we explore a speech-to-speech transla-

2. Conventional Speech-to-Speech Translation
Conventionally, speech to speech translation is composed of
ASR, MT, and TTS. First, ASR ﬁnds the best source language sentence E given the speech signal S,
Ê = arg max P (E|S).
E(1)

Second, MT ﬁnds the best target language sentence J given
the sentence E,
Ĵ = arg max P (J|Ê).
J(2)Finally, TTS ﬁnds ﬁnds the best target language speech parameter vector sequence C given the sentence Ĵ,Ĉ = arg max P (O|Ĵ)(3)subject to O = MC,

(4)

C

158
The 9th International Workshop on Spoken Language Translation
Hong Kong, December 6th-7th, 2012

where O is a joint static and dynamic feature vector sequenceof the target speech parameters and M is a transformation
matrix from the static feature vector sequence into the joint
static and dynamic feature vector sequence.

3.1. Speech Recognition
The ﬁrst step of the process uses ASR to recognize the lexicaland paralinguistic features of the input speech. This can be
represented formally as

It should be noted that in the ASR step here we are translating speech S, which is full of rich acoustic and prosodic
cues, into a simple discrete string of words E. As a result,in conventional systems all of the acoustic features of speech
are lost during recognition, as shown in Figure 1. These features include the gender of the speaker, emotion, emphasis,and rhythm. In the TTS stage, acoustic parameters are generated from the target sentence and training speech only, whichindicates that they will reﬂect no feature of the input speech.

Ê, X̂ = arg max P (E, X|S),
E,X

(5)

where S indicates the input speech, E indicates the words
included in the utterance and X indicates paralinguistic features of the words in E.
In order to recognize this information, we construct aword-based HMM acoustic model. The acoustic model istrained with audio recordings of speech and the corresponding transcriptions E using the standard Baum-Welch algorithm. Once we have created our model, we perform simple
speech recognition using the HMM acoustic model and a language model that assigns a uniform probability to all digits.
Viterbi decoding can be used to ﬁnd E.
Finally we can decide the duration and power vector
xi of each word ei . The duration component of the vector is chosen based on the time spent in each state of the
HMM acoustic model in the path found by the Viterbi algorithm. For example, if word ei is represented by the acoustic model A, the duration component will be a vector withlength equal to the number of HMM states representing ei
in A, with each element being an integer representing thenumber of frames emitted by each state. The power component of the vector is chosen in the same way, and we takethe mean value of each feature over frames that are aligned
to the same state of the acoustic model. We express poweras [power, Δpower, ΔΔpower] and join these features together as a super vector to control power in the translation
step.
3.2. Lexical Translation
Lexical translation is deﬁned as ﬁnding the best translation Jof sentence E.

Figure 1: Conventional speech to speech translation modelĴ = arg max P (J|E),
J

3. Acoustic Feature Translation Model(6)

where J indicates the target language sentence and E indicates the recognized source language sentence. Generally
we can use a statistical machine translation tool like Moses[4], to obtain this translation in standard translation tasks.
However in this paper we have chosen a simple number-tonumber translation task so we can simply write one-to-one
lexical translation rules with no loss in accuracy.

In order to resolve this problem of lost acoustic information,
we propose a method to translate paralinguistic features ofthe source speech into the target language. Our proposed
method consists of three parts: word recognition and feature extraction with ASR, lexical and paralinguistic translation with MT and linear regression respectively, and speechsynthesis with TTS. While this is the same general architecture as traditional speech translation systems, we add an additional model to translate not only lexical information but
also two types of paralinguistic information: duration andpower. In this paper, in order to focus speciﬁcally on paralinguistic translation we chose a simple, small-vocabulary
lexical MT task: number-to-number translation.

3.3. Paralinguistic Translation
Paralinguistic translation converts the source-side durationand mean power vector X into the target-side duration andmean power vector Y according to the following equationŶ = arg max P (Y|X).
Y

159
The 9th International Workshop on Spoken Language Translation
Hong Kong, December 6th-7th, 2012(7)

training data by minimize root mean squared error (RMSE)
with a regularization termŴe,j = arg minNWei ,ji n=1||y ∗ n − y n ||2 + α||Wei ,ji ||2 , (10)where N is the number of training samples, n is the id of eachtraining sample, y ∗ is target language reference word duration and power vector, and α is a hyper-parameter for the regularization term to prevent over-ﬁtting.1 This maximization
can be solved efﬁciently in closed form using simple matrix
operations.
3.4. Speech Synthesis
In the TTS part of the system we use an HMM-based speechsynthesis system [5], and reﬂect the duration and power information of the target word paralinguistic information vector onto the output speech. The output speech parameter
vector sequence C = [c1 , · · · , cT ] is determined by maximizing the target HMM likelihood function given the target
word duration and power vector Ŷ and the target language
sentence Ĵ as follows:

Figure 2: Overview of paralinguistic translation
In particular, we control duration and power of each word
using a source-side duration and power super vector xi =

[x1 , · · · , xNx ] and a target-side duration and power super
vector yi = y1 , · · · , yNy . In these vectors Nx represents the number of HMM states on the source side andNy represents the number of HMM states on the target side.
indicates transposition. The sentence duration and power
vector consists of the concatenation of the word duration andpower vectors such that Y = [y1 , · · · , yi , · · · , yI ] where I
is the length of the sentence. In this work, to simplify ourtranslation task, we assume that duration and power translation of each word pair is independent from that of otherwords, allowing us to ﬁnd the optimal Y using the following
equation:Ŷ = arg maxY

P (yi |xi ).

(11)

subject to O = MC,

(12)

C

where O is a joint static and dynamic feature vector sequenceof the target speech parameters and M is a transformation
matrix from the static feature vector sequence into the joint
static and dynamic feature vector sequence.
While TTS generally uses phoneme-based HMM models, we instead used a word based HMM to maintain the consistency of feature extraction and translation. In this task the
vocabulary is small, so we construct an independent context
model.

4. Evaluation
4.1. Experimental Setting(8)i

We examine the effectiveness of the proposed method
through English-Japanese speech-to-speech translation experiments. In these experiments we assume the use ofspeech-to-speech translation in a situation where the speakeris attempting to reserve a ticket by phone in a different language. When the listener accidentally makes a mistake when
listening to the ticket number, the speaker re-speaks, emphasizing the place where the listener has made the mistake. In
this situation, if we can translate the paralinguistic information, particularly emphasis, this will provide useful information to the listener about where the mistake is. This information will not be present with linguistic information only.

The word-to-word acoustic translation probability
P (yi |xi ) can be deﬁned with any function, but in this work
we choose to use linear regression, which indicates that yi is
distributed according to a normal distribution
P (yi |xi ) = N (yi ; Wei ,ji xi , S)Ĉ = arg max P (O|Ĵ, Ŷ)(9)

where x is 1x and Wei ,ji is a regression matrix (including a bias) deﬁning a linear transformation expressing
the relationship in duration and power between ei and ji . An
important point here is how to construct regression matrices
for each of the words we want to translate. In order to do so,
we optimize each regression matrix on the translation model

1 We chose α to be 10 based on preliminary tests but the value had little
effect on subjective results.

160
The 9th International Workshop on Spoken Language Translation
Hong Kong, December 6th-7th, 2012

power output by the proposed method is more similar to thereference than the baseline over all eleven categories, indicating the proposed method is objectively more accurate intranslating duration and power.

In order to simulate this situation, we recorded a bilingual
speech corpus where an English-Japanese bilingual speakeremphasizes one word during speech in a string of digits.
The lexical content to be spoken was 500 sentences from the
AURORA2 data set, chosen to be word balanced by greedy
search [6]. The training set is 445 utterances and the test set
is 55 utterances, graded by 3 evaluators. We plan to makethis data freely available by the publication of this paper.
Before the experiments, we analyzed the recorded
speech’s emphasis. We found several inclinations of emphasized segments such as shifts in duration and power. For
example there are often long silences before or after emphasized words, and the emphasized word itself becomes longer
and louder.
We further used this data to build an English-Japanese
speech translation system that include our proposed paralinguistic translation model. We used the AURORA2 8440 utterance bilingual speech corpus to train the ASR module.
Speech signals were sampled at 8kHz with utterances from55 males and 55 females. We set the number of HMM states
per word in the ASR acoustic model to 16, the shift length to
5ms, and other various settings for ASR to follow [7]. For thetranslation model we use 445 utterances of speech from ourrecorded corpus for training and hold out the remainder for
testing. As the recognition and translation tasks are simple
are simple , the ASR and MT models achieved 100% accuracy on every sentence in the test set. For TTS, we use thesame 445 utterances for training an independent context synthesis model. In this case, the speech signals were sampled
at 16kHz. The shift length and HMM states are identical to
the setting for ASR.
In the evaluation, we compare the baseline and two proposed models shown below:

Training sentences
Word error rate
HMM states

8440
0
16

Table 1: Setting of ASR
Training utterances
Test utterances
Regularization term

445
55
10

Table 2: Setting of paralinguistic translation
Training utterances
HMM states

445
16

Table 3: Setting of TTS
As a subjective evaluation we asked native speakers ofJapanese to evaluate how well emphasis was translated into
the target language. The ﬁrst experiment asked the evaluators to attempt to recognize the identities and positions of
the emphasized words in the output speech. The overview
of the result for the word and emphasis recognition rates isshown in Figure 5. We can see that both of the proposedsystems show a clear improvement in the emphasis recognition rate over the baseline. Subjectively the evaluators foundthat there is a clear difference in the duration and power ofthe words. In the proposed model where only duration wastranslated, many testers said emphasis was possible to recognize, but sometime it was not so clear and they were confused. When we also translate power, emphasis became moreclear and some examples of emphasis that only depended on
power were also able to be recognized. When we examined
the remaining errors, we noticed that even when mistakes
were made, mistakenly recognized positions tended to be directly before or after the correct word, instead of being in an
entirely different part of the utterance.
The second experiment asked the evaluators to subjectively judge the strength of emphasis, graded with the following three degrees.

Baseline: traditional lexical translation model only
Duration: Paralinguistic translation of duration only
Duration + Power: Paralinguistic translation of duration andpower
The word translation result is the same between both models,
but the proposed model has more information than the baseline model with regards to duration and power. In addition,
we use naturally spoken speech as an oracle output. We evaluate both varieties of output speech with respect to how wellthey represent emphasis.
4.2. Experimental Results
We ﬁrst perform an objective assessment of the translation
accuracy of duration and power, the results of which arefound in Figure 3 and Figure 4. For each of the nine digits
plus “oh” and “zero,” we compared the difference between
the proposed and baseline duration and power and the reference speech duration and power in terms of RMSE. Fromthese results, we can see that the target speech duration and

1: not emphasized
2: slightly emphasized
3: emphasized
The overview of the experiment regarding the strength of emphasis is shown in Figure 6. This ﬁgure shows that there161
The 9th International Workshop on Spoken Language Translation
Hong Kong, December 6th-7th, 2012

Figure 5: Prediction rate

Figure 3: Root mean squared error rate (RMSE) between thereference target duration and the system output for each digit

Figure 6: Degree of emphasis[9, 10].
While the above methods consider paralinguistic information to boost translation accuracy, as we mentioned before, there is more to speech translation than just the accuracyof the target sentence. It is also necessary to consider otherfeatures such as the speaker’s facial and prosodic expressions to fully convey all of the information included in natural speech. There is some research that considers translatingthese expressions and improves speech translation quality inother ways that cannot be measured by BLEU. For example
some work focuses on mouth shape and uses this information
to translate speaker emotion from source to target [1, 11]. On
the other hand, [2] focus on the input speech’s prosody, extracting F0 from the source speech at the sentence level andclustering accent groups. These are then translated into target
side accent groups. V. Kumar et al consider the prosody inencoded as factors in the Moses translation engine to convey
prosody from source to target [12].
In our work, we also focus on source speech paralinguistic features, but unlike previous work we extract them andtranslate to target paralinguistic features directly and in continuous space. In this framework, we need two translation
models. One for word-to-word lexical translation, and another for paralinguistic translation. We train a paralinguistictranslation model with linear regression for each word pair.
This allows for relatively simple, language-independent implementation and is more appropriate for continuous features
such as duration and power.

Figure 4: Root mean squared error rate (RMSE) between thereference target power and the system output for each digit
is a signiﬁcant improvement in the subjective perception ofstrength of emphasis as well. Particularly, when we analyzed
the result we found two interesting trends between durationtranslation and duration and power translation. Particularly,
the former method was often labeled with a score of 2 indicating that the duration is not sufﬁcient to represent emphasis clearly. However, duration+power almost always scored3 and can be recognized as the position of emphasis. Thismeans that in English-Japanese speech translation, speech’s
power is an important factor to convey emphasis.

5. Related Works
There have been several studies demonstrating improved
speech translation performance by utilizing paralinguistic information of source side speech. For example, [8] focuses
on using the input speech’s acoustic information to improvetranslation accuracy. They try to explore a tight coupling of
ASR and MT for speech translation, sharing information on
the phone level to boost translation accuracy as measured by
BLEU score. Other related works focus on using speech intonation to reduce translation ambiguity on the target side162
The 9th International Workshop on Spoken Language Translation
Hong Kong, December 6th-7th, 2012

6. Conclusion

speech database,” in Proceedings of the 15th International Congress of Phonetic Sciences. ICPhS, 2003.

In this paper we proposed a method to translate duration andpower information for speech-to-speech translation. Experimental results showed that duration and power information
in input speech appears in output speech, and that this information can be used by target language speakers to detectemphasis.
In future work we plan to expand beyond the easy lexicaltranslation task in the current paper to a more general translation task. Our next step is to expand our method to work withphrase-based machine translation. Phrase-based SMT handles non-monotonicity, insertions, and deletions naturally,
and we are currently in the process devising methods to deal
with the expand vocabulary in paralinguistic translation. In
addition, traditional speech-to-speech translation, the ASR
and TTS systems generally use phoneme-based HMM acoustic models. And it will be necessary to change our wordbased ASR and TTS to phoneme-based systems to improvetheir performance on open-domain tasks. Finally, while welimited our study to duration and power, we plan to expand toother acoustic features such as F0, which play an importantpart in other language pairs, and also paralinguistic featuresother than emphasis.

[7] H. G. Hirsh and D. Pearce, “The AURORA experimental framework for the performance evaluations ofspeech recognition systems under noisy conditions,” in
ISCA ITRW ASR2000, 2000.
[8] J. Jiang, Z. Ahmed, J. Carson-Berndsen, P. Cahill, andA. Way, “Phonetic representation- based speech translation,” in Proceedings of Machine Translation Summit
13, 2011.
[9] T. Takezawa, T. Morimoto, Y. Sagisaka, N. Campbell,
H. Iida, F. Sugaya, A. Yokoo, and S. Yamamoto, “AJapanese-to-English speech translation system: ATRMATRIX,” in In Proceedings of the 5th International
Conference on Spoken Language Processing. ICSLP,
1998.
[10] W. Wahlster, “Robust translation of spontaneous
speech: a multi-engine approach,” in IJCAI’01 Proceedings of the 17th international joint conference on
Artiﬁcial intelligence - Vol 2. IJCAI, 2001.
[11] S. Morishima and S. Nakamura, “Multimodal translation system using texture mapped Lip-Sync images
for video mail and automatic dubbing applications.”
EURASIP, 2004.

7. Acknowledgment
Part of this work was supported by JSPS KAKENHI Grant
Number 24240032.

[12] V. Kumar, S. Bangalore, and S. Narayanan, “Enriching machine-mediated speech-to-speech translation using contextual information,” 2011.

8. References[1] S. Ogata, T. Misawa, S. Nakamura, and S. Morishima,“Multi-modal translation system by using automatic facial image tracking and model- based lip synchronization,” in ACM SIGGRAPH2001 Conference Abstracts
and Applications,Sketch and Applications. Siggraph,
2001.
[2] P. D. Agero, J. Adell, and A. Bonafonte, “Prosody
generation for speech-to-speech translation,” in In Proceedings of ICASSP. ICASSP, 2006.
[3] H. Ney, “Speech translation: coupling of recognition
and translation,” in Proceedings of Acoustics, Speech,
and Signal Processing. IEEE Int. Conf, 1999.
[4] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,
M. Federico, N. Bertoldi, B. Cowan, W. Shen,
C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin,
and E. Herbst, “Moses: Open source toolkit for statistical machine translation,” in Proceedings of ACL, 2007.
[5] H. Zen, K. Tokuda, and A. W. Black, “Statistical parametric speech synthesis.”
Speech Communication,
2009.
[6] J. Zhang and S. Nakamura, “An efﬁcient algorithm
to search for a minimum sentence set for collecting

163
The 9th International Workshop on Spoken Language Translation
Hong Kong, December 6th-7th, 2012