IMPROVED STATISTICAL MODELS FOR SMT-BASED SPEAKING STYLE
TRANSFORMATION
Graham Neubig, Yuya Akita, Shinsuke Mori, Tatsuya Kawahara
Kyoto University, School of Informatics
Sakyo-ku, Kyoto, 606-8501, Japan
ABSTRACT
Automatic speech recognition (ASR) results contain not only
ASR errors, but also disﬂuencies and colloquial expressions
that must be corrected to create readable transcripts. We
take the approach of statistical machine translation (SMT) to“translate” from ASR results into transcript-style text. We introduce two novel modeling techniques in this framework: acontext-dependent translation model, which allows for usageof context to accurately model translation probabilities, andlog-linear interpolation of conditional and joint probabilities,
which allows for frequently observed translation patterns to
be given higher priority. The system is implemented using
weighted ﬁnite state transducers (WFST). On an evaluation
using ASR results and manual transcripts of meetings ofthe Japanese Diet (national congress), the proposed methods
showed a signiﬁcant increase in accuracy over traditionalmodeling techniques.
Index Terms— speaking style transformation, log-linearmodels, weighted ﬁnite state transducers
1. INTRODUCTION
The task of automatic speech recognition (ASR) is conventionally modeled as ﬁnding the verbatim utterance V given
the acoustic signal X. A statistical model is created, and adecoder searches for utterance V̂ that maximizes P (V |X).
However, ASR results often contain disﬂuencies, redundantor colloquial expressions, and dropped words, in addition toASR errors. These phenomena must be corrected in order tocreate readable, natural transcripts from ASR results.
A reasonably large body of work has been conducted
on correcting these phenomena automatically, with a particular focus on disﬂuency deletion and punctuation insertion [1, 2, 3]. However, in addition to disﬂuency deletionand punctuation insertion, human editors make a number ofother edits, including correction of colloquial expressions
and dropped words. Handling these phenomena is particularly important in formal settings such as public speeches andcongressional meetings, where disﬂuencies and repairs areless frequent than in conversational speech, but the resulting
transcript must be grammatically and stylistically correct.

978-1-4244-4296-6/10/$25.00 ©2010 IEEE

5206

Previous research on handling the arbitrary transformations necessary for creation of formal transcripts has usedtechniques from statistical machine translation (SMT), treating verbatim and clean transcripts as different languages and“translating” between them. Shitaoka et al [4] presented anoisy-channel model for speaking style transformation (SST).
We expanded this model through a weighted ﬁnite state transducer (WFST) implementation and the introduction of a variety of features in a log-linear framework [5].
This paper addresses enhancement of the translation
model for ﬁnite-state SMT-based SST. First, we review thetraditional noisy-channel and a context-dependent joint probability model previously proposed for ﬁnite-state machinetranslation [6]. Then, we propose two improvements tothe existing methods. The ﬁrst method allows for use of
a context-dependent translation model in the noisy-channelframework by transforming context-dependent joint probabilities into conditional probabilities. The second method
allows greater emphasis to be placed on frequent translation
patterns by log-linearly interpolating the joint and conditional
probability models. An evaluation is performed on committee meetings from the Japanese Diet (national congress),
and the proposed methods show signiﬁcant improvements inaccuracy for both manual transcripts and ASR results.

2. MODELS FOR SPEAKING STYLE
TRANSFORMATION (SST)
2.1. Noisy-Channel Modeling
SMT-based SST transforms an actual utterance (or ASR results) V into transcript-style text W by creating a statisticalmodel for P (W |V ), and searching for the Ŵ that maximizes
P (W |V ) for any given V . A parallel corpus of aligned sentences is used to estimate the parameters of the model. Because the size of available parallel corpora is often dwarfed
by the size of available clean transcripts, Bayes’ law is used todecompose P (W |V ) into the translation model (TM) probability Pt (V |W ) and language model (LM) probability Pl (W )Ŵ = argmaxPt (V |W )Pl (W ).
W(1)

ICASSP 2010

While the TM must be trained on a parallel corpus (indicatedby the subscript t), the LM can utilize a larger body of cleantranscripts (subscript l). Models decomposed in this mannerare often called noisy-channel models, and are used in mostprevious research on SST [1, 3, 4].
Sentence TM probabilities can be approximated as theproduct of word TM probabilities to simplify parameter estimationPt (V |W ) ≈Pt (vi |wi ).
(2)iWord TM probabilities are determined using maximum likelihood estimation.
To handle insertions and deletions, the empty string  istreated as a word in the vocabulary, and probabilities Pt (v|)and Pt (|w) are calculated. To handle one-to-many substitutions (e.g. “don’t” → “do not”), common multi-word phrases
are treated as individual vocabulary words. The segmentation of the target sentence into these words is determined bya unigram segmentation model.

Ŵ = argmaxPt (W, V )W

is ensured to give the same result as Equation (1).
GIATI models the joint probability by assuming alignments are monotonic, an assumption that generally holds in
the SST task as there is little non-monotonic permutation.
The source sentence V = v1 , . . . , vk and target sentence
W = w1 , . . . , wk are represented as a string of symbols
Γ = γ1 , . . . , γk , where γi = vi , wi . Using these monotonic alignments, a smoothed n-gram model is trained overa corpus of Γ strings. Pt (W, V ) is approximated using thefollowing equation:
Pt (γi |γi−n+1 , . . . , γi−1 ).

Pt (vi |v1 , . . . , vi−1 , w1 , . . . , wk , )i=1=k

Pt (vi |γ1 , . . . , γi−1 , wi , . . . , wk ).

i=1

Further, we assume that vi does not depend on any w greaterthan wi , and use an n-order Markov model to limit the lengthof the considered history:
Pt (V |W ) ≈kPt (vi |γi−n+1 , . . . , γi−1 , wi ).

(3)

i=1

2.3. Context-Dependent Translation Modeling
While joint probability models provide an effective way tohandle context, they also leave no room for use of large-scale
non-parallel data through the LM probability Pl (W ). We propose a technique for approximating a context-dependent TM
probability from GIATI probabilities. This allows for the creation of a model that can both consider context when choosing

5207

(4)

i=1

Equation (4) can further be transformed intok
Pt (γi |γi−n+1 , . . . , γi−1 )i=1While word translation probabilities were assumed to be independent in the previous section, in many cases the translation
probability is actually highly context dependent (e.g. whether“like” is a ﬁller or function word, etc.). One method for expressing context directly in the TM is the GIATI method [6].
GIATI skips the step of noisy-channel decomposition and directly models the joint probability Pt (W, V ). By limiting thesearch space so that V is the source sentence,k

Pt (V |W ) =k

Pt (V |W ) ≈2.2. Joint Probability Modeling

Pt (W, V ) = Pt (Γ) ≈translation probabilities, and use non-parallel data to compensate for sparsity in the parallel corpus.
We ﬁrst note that Pt (V |W ) can be modeled sequentially:

Pt (wi |γi−n+1 , . . . , γi−1 ).

(5)The denominator of this equation is the sum of the n-gramprobabilities for γj where wj = wi
Pt (wi |γi−n+1 , . . . , γi−1 ) =Pt (γj |γi−n+1 , . . . , γi−1 ).

(6)γj ∈{γ̃:w̃=wi }Because the numerator of Equation (5) and each elementin the sum of Equation (6) have the same form as the n-gramprobabilities in Equation (3), Pt (V |W ) can be estimated using the n-gram probabilities obtained by the GIATI method.
This context-dependent model for Pt (V |W ) can be usedalong with LM probability Pl (W ) in Equation (1).
2.4. Log-Linear Interpolation with Joint Probabilities
While the conditional model has the advantage of allowing
the usage of non-parallel text, it lacks a model of overall translation pattern frequency. For example, if there is a pattern γxwith counts ct (γx ) = 100, ct (wx ) = 1000, and a pattern γywith counts ct (γy ) = 1, ct (wy ) = 10, both will be given thesame probability
Pt (vx |wx ) = Pt (vy |wy ) = 0.1
even though the less frequent γy may simply be the resultof semi-random variance in sparse training data. Infrequent
patterns are particularly unreliable when dealing with ASR
data, which is highly inconsistent.
While Pt (vx |wx ) and Pt (vy |wy ) are equal, Pt (γx ) willbe 100 times larger than Pt (γy ). Thus, the joint probability can be used as a source of information about translation
pattern frequency. We propose a model M (W, V ) that uses

Table 1. Size of the test set, and number of transformations
necessary for the manual transcriptions and ASR results.
Turns
Words
CommasPeriods
Fillers
DeletionsNon-ﬁllers
SubstitutionsInsertions

Manual
22,52024,450
4,954
4,584

1,023
300,059
20,629
7,196
ASR
19,46842,10528,503
11,332

Manual Transcripts (18.62% Unedited)Model
LL 1-gram 2-gram 3-gram
Noisy
6.51%
5.33%
5.32%
Noisy LL
5.99%
5.15%
5.13%
Joint
9.89%
4.70%
4.60%
Noisy+Joint
5.81%
4.12% 4.05%



log-linear interpolation [7] to combine the LM, TM, and joint
probabilities, thus capturing this frequency information
M (W, V ) =λ1 logPl (W ) + λ2 logPt (V |W ) + λ3 logPt (W, V )Table 2. Each model, whether it is log-linear (LL), and its
WER for each TM order. Italics are statistically signiﬁcantly
different from the baseline.

(7)

Note that while setting λ3 = 0 is an extension to thenaive noisy-channel model in Equation (1), setting λ2 = 0
and interpolating only the ﬁrst and third elements is neither theoretically correct nor practical. From the theoreticalstandpoint, combining Pl (W ) and Pt (W, V ) can in no way
derive P (W |V ), the posterior function that we are trying
to optimize. Practically, a model created in this way overaggressively deletes words, resulting in accuracy no better
than the standard models. It is for this reason that the conditional model introduced in the previous section is necessary,even when interpolating with the joint probability.
3. EXPERIMENTAL EVALUATION
3.1. Experimental Setup
The proposed system was trained and tested on a corpus ofcommittee meetings of the Japanese Diet (national congress).
With the ofﬁcial Diet transcripts as the ﬁnal target, separate
tests were conducted on both manually-created verbatim transcripts and ASR results as input. Punctuation was treated thesame as any other word in the translation model, but a symbol
indicating pauses of greater than 200ms was included in theASR output to provide information for punctuation insertion.
ASR was performed with a system dedicated to this task [8],and an ASR WER of 17.1% was achieved. A summary of thetest set can be found in Table 1.
A corpus of 158M words of ofﬁcial Diet transcripts wasused to train the LM. A smaller 2.83M word parallel corpus
was used for training of the TM. Log-linear weights weretuned on a set of held-out data consisting of 66.3k words.
3.2. Training/Decoding
The LM was a Kneser-Ney smoothed 3-gram. This conﬁguration was used for all noisy-channel models regardless of theTM order.

5208

ASR Results (36.10% Unedited)Model
LL 1-gram 2-gram 3-gram
Noisy
21.83% 21.00% 21.09%
Noisy LL
21.63% 20.97% 21.09%
Joint
28.61% 22.62% 21.98%
Noisy+Joint
21.32% 20.04% 20.03%

The TM n-grams were also Kneser-Ney smoothed, andorders 1-3 were tested (4-grams were inferior to 3-grams forall models). For the system using manual transcripts as input, a parallel corpus of verbatim and ofﬁcial transcripts wasused as TM training data. Likewise, when using ASR results
as input, a corpus of ASR results and ofﬁcial transcripts wasused for training the TM1 . Word alignment was performedby ﬁrst aligning words to minimize edit distance, after which
words in non-matching sections were aligned using the EM
algorithm to minimize the entropy of the joint TM.
The TM, LM, joint, and segmentation models were each
expressed as separate WFSTs, and were composed into anoverall model using the OpenFst toolkit [9]. Decoding wasperformed using a beam-search WFST decoder, Kyfd2 . Loglinear weights were trained using the minimum error rate
training tool included in the Moses SMT toolkit [10].
3.3. Effect of Translation Models
Table 2 shows results using four separate models: the noisychannel model of Equation (5), a noisy-channel model withseparate log-linear weights for the TM and LM (Equation (7)with λ3 ﬁxed at 0), the joint probability model of Equation(3), and the noisy-channel/joint model of Equation (7). Because the 1-gram noisy-channel model is equivalent to traditional noisy-channel models, it is used as a baseline.
The 3-gram noisy-channel/joint model performed best,achieving a WER of 4.05% over manual transcriptions and20.03% over ASR results. This is a statistically signiﬁcant
gain over the baseline values (6.51% and 21.83% respectively) according to the two-proportion z-test at 99% conﬁdence. The proposed context-dependent TM made a large
1 A system trained with manual transcripts performed ≈ 3% absolute
WER worse, largely because models trained on ASR results are better at
inserting punctuation, as well as correcting homonyms and ASR errors.
2 http://www.phontron.com/kyfd

model that interpolates joint and noisy-channel probabilities
displays both good performance on small data and continuing
improvment, achieving a low WER over all data sizes.
Second, accuracy continues to improve for the joint models even at 2.32M words, indicating that it may be useful tocollect more data. We plan to create more ASR results fortraining the SST system with ASR input.
Finally, for most models, the error rate decreases rapidly
for corpus sizes under 17k words, and more slowly after 17kwords. This suggests that common patterns like ﬁller deletions are learned after 17k words, and any additional data afterthat helps train more difﬁcult context-dependent patterns.
4. CONCLUSION

Fig. 1. Effect of corpus size for manual transcripts.

This paper presented techniques to model context and translation pattern frequency for SMT-based SST. A system using both of these techniques showed a statistically signiﬁcant
improvement over a traditional noisy-channel model for bothmanual transcripts and ASR results. A promising future research direction is the integration of the SST module with aWFST-based recognition engine to ﬁnd globally optimal output given acoustic features. We also plan to combine the features introduced in [5] with the translation models presented
here and investigate their mutual effect.
5. REFERENCES

Fig. 2. Effect of corpus size for ASR results.
contribution, resulting in signiﬁcant gains for both ASR andmanual transcripts.
The simple joint probability models (excluding the contextindependent 1-gram) performed well on the manual transcripts, but could not outperform the baseline on ASR results.
This is because it encountered sparsity issues over the inconsistent data, and was not able to utilize the LM trainedwith large amounts of data as a fallback. However, when
the joint and noisy-channel models were interpolated, furthergains were observed, particularly for the 2-gram and 3-grammodels where the pattern frequency information provided by
the joint probability helped relieve sparseness problems.
3.4. Effect of Corpus SizeWe also performed an investigation on the effect of the sizeof the parallel corpus used in TM training. The results aresummarized in Fig. 1 and 2.
First, while noisy-channel models perform better thanjoint models for small parallel corpora (due to the use of large
amounts of clean data), joint models improve at a faster rate
as the amount of data increases, even surpassing the noisychannel model in the manual transcipts of Figure 1. The

5209[1] M. Honal and T. Schultz, “Correction of disﬂuencies in spontaneous speech using a noisy-channel approach,” in Proc. EuroSpeech2003, 2003, pp. 2781–2784.
[2] Y. Liu, E. Shriberg, A. Stolcke, D. Hillard, M. Ostendorf, andM. Harper, “Enriching speech recognition with automatic detection of sentence boundaries and disﬂuencies,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 14,
no. 5, pp. 1526–1540, 2006.
[3] S. Maskey, B. Zhou, and Y. Gao, “A phrase-level machinetranslation approach for disﬂuency detection using weighted
ﬁnite state transducers,” in Proc. InterSpeech2006, 2006, pp.
749–752.
[4] K. Shitaoka, H. Nanjo, and T. Kawahara, “Automatic transformation of lecture transcription into document style using
statistical framework,” in Proc. InterSpeech2004, 2004, pp.
2169–2172.
[5] G. Neubig, S. Mori, and T. Kawahara, “A WFST-based loglinear framework for speaking-style transformation,” in Proc.
InterSpeech2009, 2009, pp. 1495–1498.
[6] F. Casacuberta and E. Vidal, “Machine translation with inferred stochastic ﬁnite-state transducers,” Computational Linguistics, vol. 30, no. 2, pp. 205–225, 2004.
[7] F. J. Och and H. Ney, “Discriminative training and maximum
entropy models for statistical machine translation,” in Proc.
ACL02, 2002, pp. 295–302.
[8] Y. Akita, M. Mimura, and T. Kawahara, “Automatic transcription system for meetings of the Japanese national congress,” inProc. InterSpeech2009, 2009, pp. 84–87.
[9] C. Allauzen, M. Riley, J. Schalkwyk, W. Skut, and M. Mohri,“OpenFst: a general and efﬁcient weighted ﬁnite-state transducer library,” in Proc. CIAA ’07, 2007, pp. 11–23.
[10] P. Koehn et al., “Moses: Open source toolkit for statisticalmachine translation,” in Proc. ACL07, 2007, pp. 177–180.