INTERSPEECH 2013

Generalizing Continuous-space Translationof Paralinguistic Information
Takatomo Kano, Shinnosuke Takamichi, Sakriani Sakti,
Graham Neubig, Tomoki Toda, Satoshi Nakamura
Graduate School of Information Science, Nara Institute of Science and Technology{takatomo-k,shinnosuke-t,ssakti,neubig,tomoki,s-nakamura}@is.naist.jp

Abstract

In this work we expand the paralinguistic translation modelto adapt to more general tasks by training a single model thatis applicable to all words using neural networks. There are two
merits to using neural networks. First, neural network possess
sufficient power to express difficult regression problems suchas translation of acoustic features for multiple words. Second,neural network can be expanded with features expressing additional information such as the input word and translated word,
the position of both words, parts of speech, and so on.
We evaluate the proposed method by using parallel emphasized utterances to train and test our paralinguistic translation
model. We measure the emphasis recognition rate and intensity by objective and subjective assessment, and find that the
proposed generalized paralinguistic translation method is justas effective in translating this paralinguistic information as theprevious word-dependent approach.

In previous work, we proposed a model for speech-to-speechtranslation that is sensitive to paralinguistic information such as
duration and power of spoken words [1]. This model uses linear regression to map source acoustic features to target acoustic
features directly and in continuous space. However, while themodel is effective, it faces scalability issues as a single model
must be trained for every word, which makes it difficult to generalize to words for which we do not have parallel speech. In
this work we first demonstrate that simply training a linear regression model on all words is not sufficient to express paralinguistic translation. We next describe a neural network modelthat has sufficient expressive power to perform paralinguistictranslation with a single model. We evaluate the proposed
method on a digit translation task and show that we achieve
similar results with a single neural network-based model as previous work did using word-dependent models.
Index Terms: speech translation, paralinguistic information,
linear regression, neural network

2. Related Work
There have been several studies demonstrating improved speechtranslation performance by utilizing paralinguistic information.
[2] focus on the input speech’s prosody, extracting F0 from thesource speech at the sentence level and clustering accent groups.
These are then translated into target side accent groups. [3]consider the prosody encoded as factors in a factored translation
model [4] to convey prosody from source to target.
In our previous work [1], we also focus on source speechparalinguistic features, but unlike previous work we extract
them and translate to target paralinguistic features directly
and in continuous space. This allows for relatively simple,
language-independent implementation and is more appropriate
for continuous features such as duration and power. However,this requires a separate model for each word, a restriction we
try to lift in this work.

1. Introduction
Speech-to-speech translation (S2ST) consists of automatic
speech recognition (ASR), machine translation (MT), and text
to speech (TTS) components. The most commonly used speechtranslation model uses words as the basic unit for information
sharing between these three components, but there are several
major limitations of this approach. For example, in human communication, speakers use many different varieties of information to convey their thoughts and emotions. This paralinguistic information is not a factor in written communication, but
in spoken communication it has great importance. However,this information is not translated in standard speech translation
systems. For example, even if the input of ASR contains richprosody information conveying emphasis or emotion, the words
output by TTS will be given the canonical prosody for the input
text, not reflecting these traits.
In previous work we proposed a method that overcomes thisproblem by translating paralinguistic information [1]. In thismethod we constructed a regression matrix for each word in
the vocabulary to map its acoustic features. However, in thisframework it is difficult to model for large vocabulary tasks, as itis necessary to construct separate models that represent acoustic
feature mappings for individual word. This is because simple
linear regression lacks the expressive power to map multiple
words in a single model. In this paper we demonstrate that, as
expected, making a single regression matrix mapping all words’
acoustic features barely exceeds a baseline of not translating
paralinguistic information at all.

Copyright © 2013 ISCA

3. Continuous-Space Paralinguistic
Translation
We first briefly overview the model using linear regression to
perform translation of acoustic information. In order to focus
specifically on paralinguistic translation we describe the modelin the context of a small-vocabulary lexical S2ST task: digittranslation.
3.1. Speech Recognition
The first step of the process uses ASR to recognize the lexical and paralinguistic features of the input speech. This can be2614

25- 29 August 2013, Lyon, Francerepresented formally as
Ê, X̂ = argmax P (E, X|S),

point here is how to construct regression matrices for each of
the word pairs ⟨e, j⟩ we want to translate. In order to do so, we
optimize each regression matrix on the translation model training data for ⟨e, j⟩ by minimize root mean squared error (RMSE)
with a regularization term

(1)

E,X

where S indicates the input speech, E indicates the words included in the utterance and X indicates paralinguistic featuresof the words in E. In order to recognize this information, we
perform speech recognition using an HMM acoustic model anda language model that assigns a uniform probability to all digits.
Viterbi decoding can be used to find Ê.
Using these recognition results we decide the duration andpower vector xi of each word ei . The duration componentof the vector is chosen based on the time spent in each stateof the HMM in the Viterbi path. The power component of
the vector is chosen in a similar way, and by taking the mean
power value of each feature over frames that are aligned tothe same state of the acoustic model. We express power as[power, ∆power, ∆∆power] and join these features together
as a super vector to control power in the translation step.

Ŵ e,j = argmin
W e,j

In the TTS part of the system we use an HMM-based speechsynthesis system [6], and reflect the duration and power information of the target word paralinguistic information vector
onto the output speech. The output speech parameter vector sequence C = [c1 , · · · , cT ]⊤ is determined by maximizing thetarget HMM likelihood function given the target word durationand power vector Ŷ and the target language sentence Jˆ as follows:(2)

J

where J indicates the target language sentence and E indicatesthe recognized source language sentence. Generally we can use
a statistical machine translation tool like Moses [5], to obtain
this translation in standard translation tasks, but for digit translation we can simply write one-to-one lexical translation ruleswith no loss in accuracy.

ˆ Ŷ )Ĉ = argmax P (O|J,(7)

C

subject to O = M C,

(8)

where O is a joint static and dynamic feature vector sequenceof the target speech parameters and M is a transformation matrix from the static feature vector sequence into the joint staticand dynamic feature vector sequence. While HMM TTS generally uses phoneme-based models, we instead used a wordbased HMM to maintain the consistency of feature extraction
and translation. In this task the vocabulary is small, so we construct an independent context model.

3.3. Paralinguistic Translation
Paralinguistic translation converts the source-side acoustic feature vector X into the target-side acoustic feature vector Y according to the following equation(3)

Y

In particular, we control duration and power of each word
using a source-side duration and power super vector xi =[x1 , · · · , xNx ]⊤ and a target-side duration and power super[]⊤
vector yi = y1 , · · · , yNy . In these vectors Nx represents
the number of HMM states on the source side and Ny represents the number of HMM states on the target side. ⊤ indicatestransposition. The sentence duration and power vector consistsof the concatenation of the word duration and power vectors
such that Y = [y1 , · · · , yi , · · · , yI ] where I is the length of
the sentence. We can assume that duration and power translation of each word pair is independent from that of other words,
allowing us to find the optimal Y using the following equation:∏Ŷ = argmax
P (yi |xi ).
(4)
Y

(6)

n=1

3.4. Speech Synthesis

Lexical translation finds the best translation J of sentence E.

Ŷ = argmax P (Y |X).

||y ∗ n − y n ||2 + α||W e,j ||2 ,where N is the number of training samples for the word pair,n is the ID of each training sample, y ∗ is target language
reference word duration and power vector, and α is a hyperparameter for the regularization term to prevent over-fitting.1
This maximization can be solved in closed form using simple
matrix operations.

3.2. Lexical TranslationJˆ = argmax P (J |E),N∑

4. Generalizing Paralinguistic Translation
In this section we describe two ways to generalize to a single
model for all words in the vocabulary: global linear regression
models and global neural network models.
4.1. Global Linear Regression Models
In the previous section, we described a method that requires thetraining of a regression matrix for each word pair ⟨e, j⟩. The
simplest way to generalize this model is by not training a separate model for each word, but a global model for all words in the
vocabulary. This can be done by changing the word-dependent
regression matrix W e,j into a single global regression matrix
W and training the matrix over all samples in the corpus. However, this model can be expected to not be expressive enough toperform paralinguistic translation properly. For example, themapping of duration from a one-syllable word to another onesyllable word, and from a one-syllable word to a two-syllable
word would vary greatly, but the linear regression model only
has the power to perform the same mapping for each word.

i

The word-to-word acoustic translation probability P (yi |xi ) isdefined according to linear regression matrix that indicates that
yi is distributed according to a normal distribution
P (yi |xi ) = N (yi ; W ei ,ji x′i , S)(5)[ ⊤ ]⊤′
where x is 1x
and W ei ,ji is a regression matrix (including a bias) defining a linear transformation expressing the relationship in duration and power between ei and ji . An important

1 We chose α to be 10 based on preliminary tests but the value had
little effect on subjective results.

2615

ASR
Training sentences
HMM states
MT
Training utterances
Test utterances
Neural net structure
TTS
Training utterances
HMM states

8440
16
445
55
29/25/16
445
16
Figure 1: RMSE between the reference and system duration

Table 1: Experimental Settings

where the speaker is attempting to reserve a ticket by phone in adifferent language. When the listener makes a mistake when listening to the ticket number, the speaker re-speaks, emphasizing
the mistaken number. In this situation, if we can translate theparalinguistic information, particularly emphasis, this will provide useful information to the listener about where the mistake
is. In order to simulate this situation, we recorded a bilingual
speech corpus where an English-Japanese bilingual speaker emphasizes one word during speech in a string of digits. The lexical content to be spoken was 500 sentences from the AURORA2
data set, chosen to be word balanced by greedy search [7]. The
training set is 445 utterances and the test set is 55 utterances.2
We further used this data to build an English-Japanese
speech translation system that include our proposed paralinguistic translation model. We used the AURORA2 8440 utterance
bilingual speech corpus to train the ASR module. Speech signals were sampled at 8kHz with utterances from 55 males and
55 females. We set the number of HMM states per word in the
ASR acoustic model to 16, the shift length to 5ms, and othervarious settings for ASR to follow [8][9].
We selected 500 balanced sentence from the 8440 utterances of training data, and divide the utterances into 445 utterances for training and 55 utterances for testing paralinguistic translation. As the utterances were spoken in a noise-freeenvironment with a high-quality close-talking mic, the speakerspoke slowly and clearly, and the utterances are included in the
training data for ASR (although with different speakers), we
achieved a 100% word accuracy on ASR.3 For TTS, we use thesame 445 utterances for training an independent context synthesis model. In this case, the speech signals were sampled at
16kHz. The shift length and HMM states are identical to thesetting for ASR.
In the evaluation, we compare the following two baselines:

4.2. Global Neural Network Models
As a solution to the problem of the lack of expressivity in linear regression, we propose a global method for paralinguistictranslation using neural networks. Neural networks have higher
expressive power due to their ability to handle non-linear mappings, and are thus an ideal candidate for the task. In addition,
they allow for adding features for many different types of information following ASR, MT and TTS’s common practice, suchas word ID vectors, word position, left and right words of input
and target words, part of speech, the number of syllables, accent
types, etc. This information is known to be useful in TTS [6],so we can likely improve estimation of the output duration andpower vector in translation as well.
In this research, we prepare a feed forward neural networkthat proposes the best output word acoustic feature vector Ŷgiven input word acoustic feature vector X. As additional features, we also add a binary vector with the ID of the present
word set to 1, and the position of the output word. In this work,
because the task is simple we just use this simple feature set,but this could be expanded easily for more complicated tasks.
For the sake of simplicity in this formulation we show an
example with the word acoustic feature vector only. First, we
set each input unit ιi equal to the input vector value:ιi = xi .

(9)The hidden units πj are calculated according to the input-hidden
unit weight matrix W h :
πj =

1
∑ h ,
1 + exp(−α i wij
ιi )

(10)

where α is gradient of sigmoid function. The output units ψkand final acoustic feature output yk are set as∑ oψk =wjk πj(11)

None No translation of paralinguistic information
EachLR Linear regression with a model for each word

j

yk

=

ψk ,

with three global models of paralinguistic translation:(12)

where W o is hidden-output unit weight matrix. As an optimization criterion we use minimization of RMSE, which isachieved through simple back propagation.

AllLR A single linear regression model trained on all words

5. Evaluation

In addition, we use naturally spoken speech as an oracle output.

AllNN A single neural network model trained on all words
AllNN -ID The AllNN model without additional features

2 Freely available at http://www.phontron.com/pcbeu

5.1. Experimental Setting

3 This simplifies our analysis as we do not need to consider the cases

We examine the effectiveness of the proposed method throughEnglish-Japanese S2ST experiments, summarized in Table 4.2.
In these experiments we assume the use of S2ST in a situation

where ASR makes errors, but it will be interesting to investigate theeffect of paralinguistic translation on erroneously recognized data in
the future.

2616

Figure 2: RMSE between the reference and system power

Figure 4: Prediction rate

Figure 3: Duration RMSE for each number of hidden units

Figure 5: Subjective degree of emphasis

5.2. Automatic Evaluation

line. There is no significant difference between previous workand this work, indicating that the neural network learned a paralinguistic information mapping that allows listeners to identify
emphasis effectively.
The second experiment asked the evaluators to subjectively
judge the strength of emphasis with the following three degrees:

We first perform an objective assessment of the translation accuracy of duration and power, the results of which are found in
Figure 1 and Figure 2. We compared the difference between thesystem duration and power and the reference speech durationand power in terms of RMSE.
From these results, we can see that the AllLR model is noteffective at mapping duration and power information, achieving
results largely equal to the baseline. The AllNN model without linguistic information does slightly better but still falls wellshort of the EachLR baseline. Finally, AllNN is able to effectively model translation of paralinguistic information, althoughaccuracy of power lags slightly behind that of duration.
We also show the relationship between the number of NN
hidden units and RMSE of duration in 3 (the graph for power
was similar). It can be seen that RMSE continues to decrease
as we add more units, but with diminishing returns after 25 hidden units. When comparing the number of free parameters in
the EachLR model of previous work (17*16*11=2992) and the
AllNN model with 25 hidden units (28*25+25*16=1100), it canbe seen that we were able to significantly decrease the numberof parameters with little change in accuracy.

1: not emphasized
2: slightly emphasized
3: emphasized
The overview of the experiment regarding the strength of emphasis is shown in Figure 5. This figure shows that all systems
show a significant improvement in the subjective perception ofstrength of emphasis. There seems to be a slight subjective preference towards EachLR when power is considered, reflecting
the slightly larger RMSE found in the automatic evaluation.

6. Conclusion
In this paper we proposed a generalized model to translate duration and power information for S2ST. Experimental results
showed that quality is similar to previous work, but without theneed to create separate models for each word pair.
In future work we plan to expand beyond the digit translation task to a more general translation task with full sentences.
The difficulty here is the procurement of parallel corpora withsimilar paralinguistic information for large-vocabulary translation tasks. We are currently considering possibilities including
simultaneous interpretation corpora and movie dubs. Anotheravenue for future work is to expand to other acoustic features
such as F0, which play an important part in the translation offull sentences. We would also like to empirically compare andcontrast the output of our method to actual interpreter speech.
Acknowledgment: Part of this work was supported by
JSPS KAKENHI Grant Number 24240032.

5.3. Perception Tests
As an evaluation we asked native speakers of Japanese to evaluate how well emphasis was translated into the target language
for the baseline, oracle, and EachLR and AllNN models whentranslating duration or duration+power.
The first experiment asked the evaluators to attempt to recognize the identities and positions of the emphasized words inthe output speech. The overview of the result for the word andemphasis recognition rates is shown in Figure 4. We can seethat all of the paralinguistic translation systems show a clear
improvement in the emphasis recognition rate over the base-2617

7. References[1] T. Kano, S. Sakti, S. Takamichi, G. Neubig, T. Toda, and S. Nakamura, “A method for translation of paralinguistic information,” inProceedings of IWSLT, 2012.
[2] P. D. Agüero, J. Adell, and A. Bonafonte, “Prosody generation forspeech-to-speech translation,” in Proceedings of ICASSP, 2006.
[3] V. Kumar, S. Bangalore, and S. Narayanan, “Enriching machinemediated speech-to-speech translation using contextual information,” Computer Speech and Language, 2011.
[4] P. Koehn and H. Hoang, “Factored translation models,” in Proceedings of EMNLP, 2007, pp. 868–876.
[5] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico,
N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer,
O. Bojar, A. Constantin, and E. Herbst, “Moses: Open source
toolkit for statistical machine translation,” in Proceedings of ACL,
2007.
[6] H. Zen, K. Tokuda, and A. W. Black, “Statistical parametric speechsynthesis,” Speech Communication, 2009.
[7] J. Zhang and S. Nakamura, “An efficient algorithm to search fora minimum sentence set for collecting speech database,” in Proceedings of the 15th International Congress of Phonetic Sciences,
2003.
[8] H. G. Hirsh and D. Pearce, “The AURORA experimental framework for the performance evaluations of speech recognition systems under noisy conditions,” in ISCA ITRW ASR2000, 2000.
[9] R. Leonard, “A database for speaker independent digit recognition,” in Proceedings of ICASSP, 1984.

2618