Improved Statistical Models for SMT-Based Speaking Style Transformation

Improved Statistical Models for
SMT-Based Speaking Style Transformation
Graham Neubig, Yuya Akita, Shinsuke Mori, Tatsuya Kawahara
School of Informatics, Kyoto University, Japan1

Improved Statistical Models for SMT-Based Speaking Style Transformation

1. Overview ofSpeaking-Style Transformation2

Improved Statistical Models for SMT-Based Speaking Style Transformation

Speaking Style Transformation (SST)●●V●W

ASR is generally modeled to find the verbatim
utterance V given acoustic features X
In many cases verbatim speech is difficult to read:ya know when I was asked earlier about uh the issue ofcoal uh you under my plan uh of a cap and trade system ...

In order to create usable transcripts from ASR results,
it is necessary to transform V into clean text W
When I was asked earlier about the issue of coal under myplan of a cap and trade system, ...
3

Improved Statistical Models for SMT-Based Speaking Style Transformation

Previous Research●Detection-Based Approaches
Focus on deletion of fillers, repeats, and repairs, as wellas insertion of punctuation● Modeled using noisy-channel models [Honal & Schultz03, Maskey et al. 06], HMMs, and CRFs [Liu et al. 06]●●SMT-Based Approaches
Treat spoken and written language as different
languages, and “translate” between them
● Proposed by [Shitaoka et al. 04] and implemented
using WFSTs and log-linear models in [Neubig et al. 09]● Is able to handle colloquial expression correction,insertion of dropped words (important for formal settings)4●

Improved Statistical Models for SMT-Based Speaking Style Transformation

Research Summary●Propose two enhancements of the statistical model
for finite-state SMT-based SSTIncorporation of context in a noisy channel model bytransforming context-sensitive joint probabilities toconditional probabilities●Allowing greater emphasis on frequent patterns by
log-linearly interpolating joint and conditional probability
models●●

Evaluation of the proposed methods on both verbatim
transcripts and ASR output for the Japanese Diet(national congress)
5Improved Statistical Models for SMT-Based Speaking Style Transformation

2. Noisy-Channel and Joint-Probability
Models for SMT

6

Improved Statistical Models for SMT-Based Speaking Style Transformation

Noisy Channel Model●Statistical models for SST attempt to maximize P W ∣V 

●Training requires a parallel corpus of W and V
It is generally easier to acquire a large volume of cleantranscripts (W) than a parallel corpus (W and V)● Bayes' law is used to decompose the probabilities● =argmax P W ∣V W
W=argmax P t V∣W  P l W WTranslation Model (TM)
●Language Model (LM)P l W  is estimated using an n-gram (3-gram) model

7

Improved Statistical Models for SMT-Based Speaking Style Transformation

Probability Estimation for the TM●P t V∣W  is difficult to estimate for the whole sentence
Assume that the word TM probabilities are independent● Set the sentence TM probability equal to the product ofthe word TM probabilities●P t V∣W ≈∏ P t  v i∣w i i●However, the word TM probabilities are actually notcontext independent
Pt(like| ε )I like told him that I really like his new hairstyle.
Pt(like| ε, H1 ) (large)

Pt(like| ε, H2 ) (small)8

Improved Statistical Models for SMT-Based Speaking Style Transformation

Joint Probability Model[Casacuberta & Vidal 2004]●The joint probability model is an alternative to the noisychannel model for speech translation

 =argmax P t W ,V WW●Sentences are aligned into matching words or phrases
V= ironna e- koto de chumon tsukeru to desu ne ...
W= iroiro na
koto de chumon o tsukeru to...

●A sequence Γ of word/phrase pairs is created
Γ= ironnna/iroiro_na e-/ε koto/koto de/dechumon/chumon ε/o tsukeru/tsukeru to/to desu/ε ne/ε9Improved Statistical Models for SMT-Based Speaking Style TransformationJoint Probability Model (2)●The probability of Γ is estimated using a smoothed ngram model trained on Γ stringsKP t W ,V =P t  ≈∏ k =1 P t  k∣●●k−1k− n1Context information is contained in the joint probabilityHowever, this probability can only be trained onparallel text (an LM probability cannot be used)argmax P t W ∣V ≠argmax P t W ,V  P l W W●W

It is desirable to have a context-sensitive model that10
can be used with a language modelImproved Statistical Models for SMT-Based Speaking Style Transformation

3. A Context-Sensitive
Translation Model11

Improved Statistical Models for SMT-Based Speaking Style Transformation

Context-Sensitive
Conditional Probability●It is possible to model the conditional (TM) probability
from right-to-left, similarly to the joint probability
kP t V∣W =∏ i =1 P t v i∣v 1 ,  , v i −1 , w 1 ,  , w k k=∏i =1 P t  v i∣1 , , i−1 , w i , , w k Context Information

Prediction Unit v i −2 v i−1 v i v i1 v i 2 
 w i −2 w i−1 w i w i1 w i 2 
12

Improved Statistical Models for SMT-Based Speaking Style Transformation

Independence Assumptions●To simplify the model, we make two assumptions●

Assume that word probabilities rely only on precedingwordskP t V∣W ≈∏ i =1 P t v i∣1 , , i−1 , w i ●Limit the history lengthkP t V∣W ≈∏i =1 P t v i∣i− n1 ,  , i −1 , w i  v i −2 v i−1 v i v i1 v i 2 
 w i −2 w i−1 w i w i1 w i 2 13

Improved Statistical Models for SMT-Based Speaking Style Transformation

Calculating Conditional Probabilities
from Joint Probabilities●It is possible to decompose this equation into itsnumerator and denominator
P t  i∣i −n1 , , i −1 P t v i∣i −n1 , , i −1 , w i =P t  w i∣i −n1 , , i −1 ●The numerator is equal to the joint n-gram probability,
while the denominator can be marginalizedP t v i∣i −n1 , , i −1 , w i =P t i∣i− n1 ,  , i −1∑∈{  : 〈 v , w i 〉}●P t  ∣
 i −n1 , , i−1 This conditional probability uses context informationand can be combined with a language model
14

Improved Statistical Models for SMT-Based Speaking Style Transformation

Training the Proposed Model
Clean Corpus

Parallel Corpus
Clean(W)
会議録
Transcripts (W)

Clean
Transcripts(W)Verbatim Transcripts
or ASR Results (V)Train

PPt W∣VW ,V 
Joint Prob.

CalculateTrain

P t V∣W P ContextW∣V P l W P W∣VLM Sensitive TM

Noisy-Channel
Model15

Improved Statistical Models for SMT-Based Speaking Style Transformation

Log-Linear Interpolation
with the Joint Probability●The joint probability contains information about patternfrequency not present in the conditional probabilityc(γ1) = 100c(w1) = 1000●●c(γ2) = 1c(w2) = 10

Pt(v1|w1) = Pt(v2|w2)
Pt(γ1) ≠ Pt(γ2)

High-frequency patterns are more reliable
The strong points of both models can be utilized
through log-linear interpolation
Noisy-Channel Model

Joint Probabilitylog  P W ∣V ∝ 1 log P t V∣W 2 log P l W 3 log P t V , W 16Improved Statistical Models for SMT-Based Speaking Style Transformation

Training the Proposed Model
Clean Corpus

Parallel Corpus
Clean(W)
会議録
Transcripts (W)

Clean
Transcripts(W)

Verbatim Transcripts
or ASR Results (V)PPt W∣VW ,V 

Train

Train

Joint Prob.

Calculate3P t V∣W P ContextW∣V 

P l W P W∣VLM 2Sensitive TM

Log-LinearModel

117

Improved Statistical Models for SMT-Based Speaking Style Transformation

4. Evaluation18

Improved Statistical Models for SMT-Based Speaking Style Transformation

Experimental Setup●Verbatim transcripts and ASR output of meetings fromthe Japanese Diet were used as a target
Data Type●Size

Time PeriodLM Training

158M

1/1999 - 8/2007

TM Training

2.31M

1/2003 - 10/2006

Weight Training 66.3k

10/2006-12/2006

Testing

10/2007300k

TM training:

Verbatim system: Verbatim transcripts and clean text
● ASR system: ASR output and clean text
● Baseline: noisy channel, 3-gram LM, 1-gram TM
19●

Improved Statistical Models for SMT-Based Speaking Style Transformation

Effect of Translation Models (Verbatim Transcripts)●

4 models were comparedA) The context-sensitive noisy-channel modelB) A with log-linear interpolation of the LM and TMC) The joint-probability modelD) B and C log-linearly interpolated●

Evaluated using edit distance from the clean transcript(WER), with no editing, the WER was 18.62%

ModelLL
★

5.33%
5.15%

5.32%
5.13%

★

9.89%
5.81%

4.70%
4.12%

4.60%
20
4.05%

C. Joint Probability (Joint)
D. B+C (Noisy+Joint LL)

1-gram 2-gram 3-gram
6.51%
5.99%

A. Noisy-Channel (Noisy)
B. Noisy-Channel (Noisy LL)

TM n-gram orderImproved Statistical Models for SMT-Based Speaking Style Transformation

Effect of Translation Models (ASR Output)●●The WER between ASR output and verbatim
transcripts (ASR WER) was 17.10%
ASR output and clean transcripts was 36.10%
ModelLL

A. Noisy-Channel (Noisy)
B. Noisy-Channel (Noisy LL)★

C. Joint Probability (Joint)
D. B+C (Noisy+Joint LL)●★

TM n-gram Order

1-gram 2-gram 3-gram
21.83% 21.00%

21.09%

21.63% 20.97%

21.09%

28.61% 22.62%

21.98%

21.32% 20.04% 20.03%

The noisy-channel model was more effective than the21
joint-probability model for ASR output

Improved Statistical Models for SMT-Based Speaking Style Transformation

Comparison with Phrase-Based SMT(New Results)●The proposed techniques were also compared withMoses, a popular system for phrase-based SMT
ModelBaseline
Noisy LL (2-gram or 3-gram)
Noisy+Joint (2-gram or 3-gram)
Moses●Verbatim
WER
6.51%
5.13%

ASR
WER
21.83%
20.97%

4.05%
5.45%

20.03%
20.97%

Noisy LL is able to achieve performance as good orbetter than Moses, while Noisy+Joint greatly
22outperforms itImproved Statistical Models for SMT-Based Speaking Style Transformation

Effect of Corpus Size(Verbatim Transcripts)●The noisy-channel model is more effective with small data
sizes, but the joint model improves rapidly
8.0%

Baseline
Noisy LL 3-gram
Joint 3-gram
Noisy/Joint 3-gram

7.5%

WER (%)

7.0%
6.5%
6.0%
5.5%
5.0%
4.5%
4.0%

3.2k

4.9k

8.7k

17k

35k

66k

143k

287k

573k

1.14M 2.32M

Words in TM Training Data
●Combining both allows for greater accuracy at all sizes

23

Improved Statistical Models for SMT-Based Speaking Style Transformation

Conclusion●●●●We proposed two improved statistical models for SMTbased SST
The proposed methods showed a significant
improvement over the baseline for verbatim transcripts
and ASR results
Models transforming ASR output can be trained
without using verbatim transcripts
A promising future direction is tight coupling with a
WFST-based ASR decoder24

Improved Statistical Models for SMT-Based Speaking Style Transformation

Thank you for listening.

25

Improved Statistical Models for SMT-Based Speaking Style Transformation

Target Phenomena●●●

Deletion of Extraneous Words: These include fillers(“um”), context-dependent deletions (“like”), repeats
Colloquial Expressions: Expressions used in speech but
less in writing (“ya'know”→“you know”, “ironna” → “iroiro-na”)

Insertion of Words and Punctuation: Words are omitted
in speech, but not in writing (“[did you] talk to the boss?”,“chumon [o] tsukeru”)
various
いろんなironna

ahh things byあーa-

いろいろ なiroironasub●

fillorderことkotoでde注文
chu-monことkotoでde注文
chu-mon-obj

makeifit isつける と です ね …
tsuk eru to desu ne
を つける とo tsuk eru toins…
non-fill

Other Phenomena: order reversal, repairs, fragments

26

Improved Statistical Models for SMT-Based Speaking Style Transformation

Effect of Corpus Size (ASR Results)
Baseline
Noisy LL 3-gram
Joint 3-gram
Noisy/Joint 3-gram

25.5%

WER (%)

24.5%

23.5%

22.5%

21.5%

20.5%

19.5%
3.2k

4.9k

8.7k

17k

35k

66k

143k

Words in TM Training Data

287k

573k

1.14M

2.32M27

Improved Statistical Models for SMT-Based Speaking Style Transformation

Accuracy by Transformation Type (Verbatim Transcript)
100%

Noisy LL 1
Joint 3
Noisy LL 3
Noisy+Joint LL 3

90%80%70%

F-Measure60%
50%
40%30%20%10%0%

Fillers

Deletions

Insertions

Substitutions

CommasPeriods

28

Improved Statistical Models for SMT-Based Speaking Style Transformation

Accuracy by Transformation Type (ASR Output)
Noisy LL 1
Joint 3
Noisy LL 3
Noisy+Joint LL 3

100%

90%

80%

70%

F-Measure

60%

50%

40%

30%20%

10%0%

Fillers

Deletions

Insertions

Substitutions

CommasPeriods

29