A WFST-based Log-linear Framework for Speaking-style Transformation
Graham Neubig, Shinsuke Mori, Tatsuya Kawahara
Graduate School of Informatics, Kyoto University
Sakyo-ku, Kyoto 606-8501, Japan
Abstract

recognition system. Shitaoka et al [13] showed that a noisychannel machine translation model can be used to transformverbatim transcripts into clean transcripts.
We present a system that uses SMT techniques to handle
deletions, insertions, and substitutions simultaneously. We utilize a log-linear framework to combine non-independent features that provide benefits over a simple noisy-channel basedmodel. All features are implemented using weighted finite statetransducers (WFSTs) to provide flexibility in adding or removing features, and to enable integration with other WFST-based
systems. The model was trained and evaluated on a corpus ofmeetings of the National Diet (Congress) of Japan.
In the remainder of the paper, Section 2 describes the taskof speaking style transformation in more detail. Section 3 givesa description of log-linear models and the training of parameters. Section 4 presents various features that may be usefulto the task. Section 5 gives a brief overview of WFSTs anddescribes the implementation of the search and features in the
WFST framework. Section 6 gives experimental results and ananalysis of the effect of each feature. Section 7 concludes withpossible future directions.

When attempting to make transcripts from automatic speech
recognition results, disfluency deletion, transformation of colloquial expressions, and insertion of dropped words must be
performed to ensure that the final product is clean transcriptstyle text. This paper introduces a system for the automatictransformation of the spoken word to transcript-style languagethat enables not only deletion of disfluencies, but also substitutions of colloquial expressions and insertion of dropped words.
A number of potentially useful features are combined in a loglinear probabilistic framework, and the utility of each is examined. The system is implemented using weighted finite statetransducers (WFSTs) to allow for easy combination of featuresand integration with other WFST-based systems. On evaluation, the best system achieved a 5.37% word error rate, a 5.49%
absolute gain over a rule-based baseline and a 1.54% absolute
gain over a simple noisy-channel model.
Index Terms: speaking style transformation, disfluency detection, weighted finite state transducers, log-linear model

1. Introduction

2. Task definition

Conventional ASR systems have been designed to faithfully decode what is uttered V , given an acoustic observation X, andthus were formulated to find the hypothesis which maximizes
the posterior probability P (V |X). In spontaneous speech, however, this scheme does not necessarily lead to intelligible transcripts as there are a number of disfluent phenomena, redundantor colloquial expressions, and incomplete sentences.
When professional human stenographers transcribe speech,
they correct these phenomena to improve transcript readability.
An ideal transcription system should have the ability to do so as
well. It has been shown that this sort of transformation of ASR
results is useful not only for human readers, but also for language model adaptation [1, 4] and down-the-line applications
such as statistical machine translation (SMT) [12].
In this context, a number of works have been conductedon the detection of fillers, false-starts, and repairs. These
adopted approaches such as noisy-channel modeling [3], CRFs
[6], and weighted finite state transducers [8] (WFSTs). However, in more formal settings such as public speeches, congressional meetings, or lectures given to large audiences, disfluencies such as false starts and repairs are less frequent. On theother hand, human stenographers often must perform correctionof colloquial or ungrammatical expressions to make a formal,
document-style text. This is particularly true of languages such
as Japanese where the disconnect between spoken and written
speech is large [13].
Here, we focus on the task of transformation of Japaneseverbatim text to transcript-style text. Previously, Hori et al [4]demonstrated that non-probabilistic transformation rules can be
useful for language model adaptation in a WFST-based speech

Copyright © 2009 ISCA

One of the major driving forces behind the development of systems to transform verbatim transcripts into clean transcripts was
the DARPA EARS program’s meta-data extraction (MDE) task[7]. The MDE task, broadly speaking, handled the deletion of
extraneous words and insertion of sentence boundaries. This is,
as shown in Figure 1, generally all that is necessary to transforma native speaker’s English into transcript form.
However, these models are a limited subclass of a broader
class of possible speaking style transformation models. In thispaper, we focus on another speaking-style transformation taskthat requires not only deletions, but also insertions and substitutions: transcription of spontaneous Japanese. We broadly divide
the corrections that are required to transform spoken Japanese
into written Japanese into four categories:• Filler Deletions: Japanese contains a number of words
that are used exclusively, or nearly exclusively as fillers,and must be deleted.
• Non-filler Deletions: Other than fillers, there are a number of deletions that must be performed. These include
repeats, repairs, and words that are only treated as fillers
in certain contexts, such as ”desu ne,” which is used bothas a filler and an actual word meaning ”it is, isn’t it?”• Substitutions: The majority of substitutions are colloquial expressions. The large disconnect between spoken Japanese and written Japanese makes it necessary
to transform these colloquial expressions into formal expressions when creating transcripts.

1495

6- 10 September, Brighton UKthat is

like

uh

del

del

that isnot a problem
not a problem

posterior probability P (W |V ).
ins

Ŵ = argmaxP (W |V ).

(1)

W

Figure 1: An English sentence requiring only word deletions
and punctuation insertion
various
いろんな
ironnaahh things by
あーa-いろいろ な
iroiro
na
sub

order

こと
koto

で
de

注文
chu-mon

こと
koto

で
de

注文
chu-mon

fill

-obj

make

if

According to Bayes’ law, the following maximization is equivalent to Equation (1):it isŴ = argmaxP (V |W )P (W ).

つける と です ね …
tsukeru to desu ne
を つける とo tsukeru toinsThis is often referred to as the noisy-channel approach. It
is popular in the ASR and SMT communities because it allows the division of the modeling into two separate tasks. For
speaking style transformation, the translation model probability
P (V |W ) must be trained on a parallel corpus of verbatim andtranscript-style sentences, but the language model probability
P (W ) can be trained using a much larger set of transcript-stylesentences with no verbatim counterparts.
Previous works such as Honal et al [3] and Shitaoka et al[13] have used this noisy channel model to correct deletions,substitutions, and insertions and create clean transcriptions.

…
non-fill

Figure 2: A Japanese phrase meaning ”If you make orders forvarious things...” requiring four kinds of corrections• Insertions: Japanese spoken language allows the omitting of words such as particles (preposition-like function
words). However, this is not permitted in written language, so omitted particles must be restored in a transcript.

3.2. Log-linear modelingAn alternative approach to noisy-channel modeling that has
proven useful in SMT is the log-linear model[11]. A simple
log-linear model is achieved by taking the logarithm of thenoisy-channel model, and multiplying each log probability bya weight as shown in equation (3)The proposed system was trained and tested on a corpus ofcommittee meetings of the National Diet (Congress) of Japan.
A large corpus of 3.62M sentences of official Diet transcripts
from 1999-2007 was used to train a language model. A smallerset (56.2K sentences) of aligned verbatim and clean transcripts
was used to train translation probabilities. Parameter tuning wasdone on a 974 sentence set of held-out data.

Ŵ = argmax [α log(P (V |W )) + β log(P (W ))] .

SpokenWords
Transcript
Fillers
DeletionsNon-fillers
Substitutions
Insertions
Word Error %26,663
20,143
4,936
4,622

(3)

W

In the case that α and β are both equal to 1, the maximizationof Equation (3) is equivalent to the maximization of Equation
2. This implies that log-linear models of this form are moregeneral than simple noisy channel models, and thus can be expected to perform as well or better as long as the weights arechosen appropriately.
In addition, unlike normal noisy-channel models, log-linear
models need make no assumptions about the independence offeatures, allowing the addition of an arbitrary number of possibly non-independent additional features [11].

Table 1: Test set details
Sentences

(2)

W

7,181
341,400
299,216(47.3%)(35.7%)(8.8%)(8.2%)
16.40%

Ŵ = argmax [αf1 (V, W ) + βf2 (V, W ) + . . .] .

Testing was done on a set of meetings from 2007 that occured after all those in the training data. Statistics on the test
corpus are shown in Table 1. It should be noted that while filler
deletions are the most frequent correction at 47.3%, non-filler
deletions, substitutions, and insertions account for 52.7% of therequired corrections. This clearly indicates the necessity for
an intelligent model to transform Japanese verbatim text intotranscript-style text.

W

3.3. Weight training
When using log-linear models with non-independent features,
it is important to train the weights of the various features usedin the model. The method we chose here is minimum error
rate training (MERT), which attempts to find values that minimize some error measure over held-out data [10]. Weights weretrained to minimize WER over the held-out data set using the
MERT tool included in open source SMT software Moses [5].

3. Noisy-channel and log-linear models
3.1. Noisy-channel modeling

4. Feature selection

In statistical models for applications such as ASR and statisticalmachine translation (SMT), the problem is defined as finding
the most likely output given the input. For the transformationfrom spoken to transcript style, the input is a verbatim sentence
V , and the output is a transcript-style sentence W . Thus, thetask is to find the transcript-style sentence Ŵ , with the maximal

As mentioned in the previous section, one of the benefits oflog-linear models is that they allow for the easy combination of
non-independent features. This section gives a summary of thevarious features that we attempted and the motivation for using
them.

1496

4.1. Initial setup

Table 2: The weights for each feature: language model (LM),c(v, w) (CWV), c(w) (CW), transformation groups (TG), deletions (D), substitutions (S), insertions (I), and fillers (F)The initial setup is a noisy-channel model with only the language model and translation model log probabilities as features.
P (W ) is modeled using a standard 3-gram language modelsmoothed using the Kneser-Ney method. P (V |W ) is modeledby assuming that each separate word transformation (vi → wi )is independent, and taking the product of the probabilities.
Y
P (vi |wi )P (V |W ) =

LM
1.00

=

c(vi , wi )/c(wi ).

CW
-1.00

TG
1.34

D
0.00

S
0.26

I
-2.67

F
-0.46

allowing for the definition of weighted relations between two
strings. In this case we use these weights to represent log probabilities.
Efficient algorithms for operations such as composition(X ◦ Y ), intersection (X ∩ Y ), determinization (det(X)), andminimization (min(X)) over WFSTs exist, allowing for creation of a single more complex model out of multiple WFSTs
representing simpler models. In the proposed system, all features are implemented as single WFSTs and combined using
composition or intersection.
Furthermore, after a single speaking-style transformation
WFST SST that transforms spoken word V to transcript-styletext W has been created, it can be seamlessly integrated withother WFST-based models. By composing with an ASR modelthat transforms acoustic observations X into verbatim text V , amodel ASR ◦ SST that directly transforms acoustic observations into transcript-style text can be created. Conversely, composing SST with a language model LM that has been trainedon transcript-style text results in a speaking-style adapted language model SST ◦ LM .
The decoder used for the experiments was written using
the open source toolkit OpenFst [2]. Small modifications weremade to allow for the storing of multiple feature scores forlater use in MERT, and beam-search style trimming to limit the
search space.

i

Y

CWV
1.76

(4)

i

The probability is calculated using the number of times vi wastranslated into wi and the total number of times wi occurs. For
insertions and deletions the empty string  is aligned with the
input and output respectively, and P (|wi ) and P (vi |) are estimated. When a single word was aligned with multiple words,
the multiple-word string was treated as one word.
4.2. Filler dictionary
The first additional feature was motivated by the fact that manyof the deleted words were common fillers. A 23-word filler list
was created, and a fixed score was added every time one of thefillers was deleted. This feature is expected to make the system
work as well or better than the simple filler-removal baseline.
4.3. Transformation group penalty
Because there is a tendency for transformed areas to appear ingroups (strings of deleted fillers, etc.), it makes sense to add
a penalty that contributes to the cohesion of these groups. By
adding a fixed penalty for each group of words translated, we
can assure that in a phrase like ”I uh don’t um do,” insteadof creating two translation groups by deleting the underlined
words in ”I uh don’t um do,” the system will prefer to delete asingle block as in ”I uh don’t um do.”

6. Results and evaluation
6.1. Baseline

4.4. Transformation type penalty

Experiments were performed on the data described in Section
2. A simple rule-based method that deletes all words occurring in the list of 23 common fillers was used as a baseline.
The baseline achieved an 89.90% precision and 100% recall on
filler deletions and performed no other edits. The WER after
filler deletions was 10.86%, compared with 16.40% when nodeletions were performed.

There is a general trend that some types of errors are more difficult than others to correct. In particular, deletions tend to be easier than substitutions. In order to take this fact into account, we
introduce a separate penalty for each type of correction (deletion, substitution, and insertion), allowing the weight training
algorithim to find an appropriate balance between precision andrecall for each type.

6.2. Results

4.5. Decomposed translation model

The model that performed best on the test data set was the modelthat included all of the features listed in Section 4. It achieved a
5.37% error rate, which is a 5.49% absolute improvement overthe filler-removed baseline.
The system was able to successfully perform 658 separate
varieties of corrections, 328 of which occurred in the test corpus
only once. This demonstrates a clear advantage of the proposedsystem over a rule-based system, which would require handcrafted rules for each type of correction.
In addition, the proposed model achieved a 1.54% absolute
improvement over the simple noisy-channel model with equaltranslation model and language model weights. This is dueto the fact that the proposed system made, in general, fewer
but more accurate corrections than the noisy channel model.
In particular, the proposed system did much better at inserting
dropped particles and determining whether non-filler deletion

As in equation (4), the word transformation probability P (v|w)
is computed from the frequencies c(v, w) and c(w). By treatingthese two as separate elements it is possible to provide a separate weight for each. If c(v, w) is assigned a higher weight thanc(w) by the weight training process, more frequently-observed
transformations will be given precedence over less common
transformations, which can be expected to increase the precision.

5. WFST implementation
We implemented the transformation model in the weighted finite state transducer (WFST) framework [9]. Finite state transducers are finite automata with transitions labeled with input
and output symbols. WFSTs also assign a weight to transitions,

1497

Table 3: Performance of the model with all features, the effect of removing each feature, the noisy channel model, and the baseline

All Features Used
-Filler Dictionary-Type Penalty-Translation Groups-Decomposed Model
Noisy Channel Model
Filler-removed Baseline

WER [%]
5.37
5.46
5.62
5.67
5.74
6.91
10.86

Fillers [%]
Prec.
Rec.
99.12 97.58
99.22 97.49
98.83 97.95
98.94 96.69
98.24 98.87
98.74 97.53
89.90 100.0

Other Deletions [%]
Prec.
Rec.
85.02
52.03
79.85
54.68
83.88
51.05
79.68
52.65
75.02
55.94
63.36
59.70
-

Substitutions [%]
Prec.
Rec.
88.53
56.00
84.43
57.80
85.46
57.52
88.13
54.32
83.78
54.09
68.17
60.43
-

Insertions [%]Prec. Recall
87.47 54.39
82.47 56.10
85.97 44.81
82.06 56.82
80.77 52.51
65.38 55.78-

8. Acknowledgements

candidates should actually be deleted or not.

The authors would like to kindly thank Markus Dreyer for providing access to his code implementing expectation semi-rings
in OpenFst.

6.3. Feature effects
The weights assigned to each feature in the model containing
all the features are shown in Table 2. In order to examine whichof the features were contributing to the gains in accuracy, weremoved one feature at a time from the optimal model and examined the change in performance of each type of correction.
The performance of the systems with one feature removed isshown in Table 3, with the filler-deleted baseline and the unweighted noisy-channel models provided for reference. The difference between the best model and all other models (excluding
the model with only the filler dictionary removed) is statistically significantly according to the two-proportions z-test witha confidence level of 1%.
The feature that contributed the most to the gain in accuracy was the decomposed translation model. By increasing theweight on the joint frequency c(v, w), frequent transformations
were given precedence over uncommon transformations. This
resulted in a large gain in precision across the board at the costof a small drop in recall.
The translation group penalty feature also had a relatively
large effect on accuracy. This was the feature that contributed
most to determining whether words that had the potential to be
fillers should actually be deleted or not. It also had an effect ofpreventing borderline transformations, increasing the precisionand reducing the recall in all four transformation categories.
The transformation type penalty also made a significant
contribution. It added a large negative penalty (bonus) to insertions, increasing recall by nearly 10 points, and increased
precision of substitutions at the cost of recall.
The filler dictionary showed a small (statistically insignificant) gain, slightly increasing the recall of both filler and nonfiller deletions and raising precision in the other two transformation types.

9. References[1] Y. Akita and T. Kawahara. Topic-independent speaking-styletransformation of language model for spontaneous speech recognition. In Proc. ICASSP2007, pages 33–36, 2007.
[2] C. Allauzen, M. Riley, J. Schalkwyk, W. Skut, and M. Mohri.
OpenFst: a general and efficient weighted finite-state transducer
library. In Proc. CIAA ’07, pages 11–23, 2007.
[3] M. Honal and T. Schultz. Correction of disfluencies in spontaneous speech using a noisy-channel approach. In Proc. EuroSpeech2003, pages 2781–2784, Geneva, Switzerland, 2003.
[4] T. Hori, D. Willett, and Y. Minami. Language model adaptation using wfst-based speaking-style translation. In Proc.
ICASSP2003, pages 228–231, 2003.
[5] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico,
N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer,
O. Bojar, A. Constantin, and E. Herbst. Moses: Open Source
Toolkit for Statistical Machine Translation. In Proc. ACL07, pages
177–180, Prague, Czech Republic, June 2007. Association for
Computational Linguistics.
[6] Y. Liu, E. Shriberg, A. Stolcke, D. Hillard, M. Ostendorf, and
M. Harper. Enriching speech recognition with automatic detection of sentence boundaries and disfluencies. IEEE Transactions
on Audio, Speech, and Language Processing, pages 1526–1540,
2006.
[7] Y. Liu, E. Shriberg, A. Stolcke, B. Peskin, J. Ang, D. Hillard,
M. Ostendorf, M. Tomalin, P. Woodland, and M. Harper.
Structural metadata research in the EARS program. In Proc.
ICASSP2005, pages 957–960, 2005.
[8] S. Maskey, B. Zhou, and Y. Gao. A phrase-level machine translation approach for disfluency detection using weighted finite statetransducers. In Proc. InterSpeech2006, pages 749–752, 2006.
[9] M. Mohri. Finite-state transducers in language and speech processing. Computational Linguistics, 23(2):269–311, 1997.

7. Conclusion and future directions[10] F. J. Och. Minimum error rate training in statistical machine translation. In Proc. ACL03, pages 160–167, 2003.

The results of the evaluation demonstrate that using probabilistic methods have clear advantages over simple deletion of
fillers when transforming verbatim text into transcript-style text.
Proper selection and weight training of additional features leads
to further increases in transformation accuracy. Because thedescribed features are implemented in the WFST framework,
the translation model can be incorporated directly with WFSTbased ASR systems or be used for language model adaptation.
The ultimate goal of the proposed system must be to function on ASR output. ASR output presents a new set of challenges, including the insertion of punctuation and handling ofincorrectly recognized words.

[11] F. J. Och and H. Ney. Discriminative training and maximum entropy models for statistical machine translation. In Proc. ACL02,
pages 295–302, Morristown, NJ, USA, 2002. Association for
Computational Linguistics.
[12] S. Rao, I. Lane, and T. Schultz. Improving spoken language translation by automatic disfluency removal: Evidence from conversational speech transcripts. In Machine Translation Summit XI,
pages 177–180, Copenhagen, Denmark, September 2007.
[13] K. Shitaoka, H. Nanjo, and T. Kawahara. Automatic transformation of lecture transcription into document style using statisticalframework. In Proc. InterSpeech2004, pages 2169–2172, 2004.

1498