Learning a Language Model from Continuous Speech

Learning a Language Model from
Continuous Speech
Graham Neubig, Masato Mimura,
Shinsuke Mori, Tatsuya Kawahara
School of Informatics, Kyoto University, Japan1

Learning a Language Model from Continuous Speech

1. Outline

2

Learning a Language Model from Continuous Speech

Training of a Speech Recongition System
Text Corpus
Speechthis is the song that never ends it justgoes on and on my friends and if you
started singing it not knowing what it was
you'll just keep singing it forever justbecause this is the song that never ends
it just goes on and on my friends and if
you started singing it not knowing what it
was you'll just keep singing it forever justbecause this is the song that never ends
it just goes on and on my friends and if
you started singing it not knowing what it
was you'll just keep singing it forever justbecause this is the song that never ends
it just goes on and on my friends and if...

Training

Decoder

SpeechTranscription
this is the song that never ends it justgoes on and on my friends and if you
started singing it not knowing what it was
you'll just keep singing it forever justbecause this is the song that never ends
it just goes on and on my friends and if...

Language
Model

Training

Acoustic
Model
3Learning a Language Model from Continuous Speech

Training of a Speech Recongition System
Text Corpus
Speechthis is the song that never ends it justgoes on and on my friends and if you
started singing it not knowing what it was
you'll just keep singing it forever justbecause this is the song that never ends
it just goes on and on my friends and if
you started singing it not knowing what it
was you'll just keep singing it forever justbecause this is the song that never ends
it just goes on and on my friends and if
you started singing it not knowing what it
was you'll just keep singing it forever justbecause this is the song that never ends
it just goes on and on my friends and if...

Training

Decoder

SpeechTranscription
this is the song that never ends it justgoes on and on my friends and if you
started singing it not knowing what it was
you'll just keep singing it forever justbecause this is the song that never ends
it just goes on and on my friends and if...

Language
Model

Training

Acoustic
Model
4Learning a Language Model from Continuous Speech

Training of a Speech Recongition System
SpeechTraining

Decoder

SpeechTranscription
this is the song that never ends it justgoes on and on my friends and if you
started singing it not knowing what it was
you'll just keep singing it forever justbecause this is the song that never ends
it just goes on and on my friends and if...

Language
Model

Training

Acoustic
Model
5Learning a Language Model from Continuous Speech

Why Learn a Language
Model from Speech?
●

A straightforward way to handle spoken language●●

Fillers, colloquial expressions, and pronunciation
variants are included in the model

A way to learn models for resource-poor languages
LMs can be learned even for languages with nodigitized text
● Use with language-independent acoustic models?
[Schultz & Waibel 01]●●

Semi-supervised Learning●

Learn a model from newspaper text, update it withspoken expressions or new vocabulary from speech

6

Learning a Language Model from Continuous Speech

Our Research●

Goal: Learn a LM using no text●

Two problems:
Word boundaries are not clear → use unsupervised word
segmentation● Acoustic ambiguity→Use a phoneme lattice to absorb
acoustic model errors●●

Method: Apply a Bayesian word segmentation method[Mochihashi+ 09] to phoneme lattices●●

Implementation using weighted finite state transducers (WFST)

Result: An LM learned from continuous speech was able to7
significantly reduce the ASR phoneme error rate on test data

Learning a Language Model from Continuous Speech

Previous Research●

Learning words from speech
Using audio/visual data and techniques such as MMI or
MDL, learn grounded words [Roy+ 02, Taguchi+ 09]● Find similar audio segments using dynamic time
warping and acoustic similarity scores [Park+ 08]●●

Learning language models from speech

Use standard LM learning techniques on 1-best AM
results [de Marcken 95, Gorin+ 99]● Multigram model from acoustic lattices [Driesen+ 08]● No research learning n-gram LMs with acoustic uncertainty●●

Most work handles small vocabulary (infant directedspeech, digit recognition)

8

Learning a Language Model from Continuous Speech

2. Unsupervised word segmentation

9

Learning a Language Model from Continuous SpeechLM-based Supervised
Word Segmentation●●

Training: Use corpus W that is annotated with wordboundaries to train model G
Decoding: for character sequence x, treat all word
sequences w as possible candidates●

The probability of a candidate is proportional to its LM
probability

x=iam

Language
Model GP(w=iam; G)P(w=i am; G)P(w=ia m; G)P(w=i a m; G)
10

Learning a Language Model from Continuous Speech

LM-Based Unsupervised
Word Segmentation●●

Estimate an unobserved word sequence W of
unsegmented corpus X, train language model G over W
We desire a model that is highly expressive, but simple
Likelihood P(W|G) prefers expressive (complex) models● Add a prior P(G) that prefers simple models● Find a model with high joint probability
P(G,W)=P(G)P(W|G)●

Simple Model

Ideal Model

Complex model

P(G) highP(W|G) low
P(G)P(W|G) low

P(G) mid
P(W|G) mid
P(G)P(W|G) mid

P(G) low
P(W|G) high
11
P(G)P(W|G) low

Learning a Language Model from Continuous Speech

Hierarchical Pitman-Yor Language
Model (HPYLM) [Teh 06]●●

An n-gram language model based on non-parametric
Bayesian statistics
Has a number of attractive traits
Language model smoothing is realized through prior P(G)● Parameters can be learned using Gibbs sampling●

PY(Ha, d3, Θ3) ~

Hca …

Hba

Hab

Ha

Hb
Hε

… ~ PY(H , d , Θ )
b
3
3

Hdb
…

~ PY(Hε, d2, Θ2)~ PY(Hbase, d1, Θ1)12

Learning a Language Model from Continuous Speech

Unsupervised Word Segmentation
using HPYLMs [Mochihashi+ 09]●

The model G is separated into a word-based language
model LM and a character-based spelling model SM●

Words and spellings are connected in a probabilistic
framework (unknown words can be modeled)
i am in chiba now

PLM(i|
) PLM(am|i) PLM(in|am) PLM(<unk>|in) PLM(now|<unk>) PLM(
|now)
PSM(c|
) PSM(h|c) PSM(i|h) PSM(b|i) PSM(a|b) PSM(
|a)●

It is possible to sample word boundaries using atechnique called forward-filtering/backward-sampling
Can be used with any (non-cyclic) finite-state automaton
13● Very similar to the forward-backward algorithm for HMMs●

Learning a Language Model from Continuous Speech

Forward Filtering●

Forward filtering is identical to the forward step in theforward-backward algorithmp(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

forward filtering
add forward probabilities in order14

Learning a Language Model from Continuous Speech

Forward Filtering●

Forward filtering is identical to the forward step in theforward-backward algorithmp(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

forward filtering
add forward probabilities in orderf(s0) = 1
15

Learning a Language Model from Continuous Speech

Forward Filtering●

Forward filtering is identical to the forward step in theforward-backward algorithmp(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

forward filtering
add forward probabilities in orderf(s0) = 1
f(s1) = p(s1|s0)*f(s0)16Learning a Language Model from Continuous Speech

Forward Filtering●

Forward filtering is identical to the forward step in theforward-backward algorithmp(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

forward filtering
add forward probabilities in orderf(s0) = 1
f(s1) = p(s1|s0)*f(s0)f(s2) = p(s2|s0)*f(s0)17Learning a Language Model from Continuous Speech

Forward Filtering●

Forward filtering is identical to the forward step in theforward-backward algorithmp(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

forward filtering
add forward probabilities in orderf(s0) = 1
f(s1) = p(s1|s0)*f(s0)f(s2) = p(s2|s0)*f(s0)f(s3) = p(s3|s1)*f(s1) + p(s3|s2)*f(s2)18Learning a Language Model from Continuous Speech

Forward Filtering●

Forward filtering is identical to the forward step in theforward-backward algorithmp(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

forward filtering
add forward probabilities in orderf(s0) = 1
f(s1) = p(s1|s0)*f(s0)f(s2) = p(s2|s0)*f(s0)f(s3) = p(s3|s1)*f(s1) + p(s3|s2)*f(s2)f(s4) = p(s4|s1)*f(s1) + p(s4|s2)*f(s2)19Learning a Language Model from Continuous Speech

Forward Filtering●

Forward filtering is identical to the forward step in theforward-backward algorithmp(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

forward filtering
add forward probabilities in orderf(s0) = 1
f(s1) = p(s1|s0)*f(s0)f(s2) = p(s2|s0)*f(s0)f(s3) = p(s3|s1)*f(s1) + p(s3|s2)*f(s2)f(s4) = p(s4|s1)*f(s1) + p(s4|s2)*f(s2)f(s5) = p(s5|s3)*f(s3) + p(s5|s4)*f(s4)20Learning a Language Model from Continuous Speech

Backward Sampling●

Backward sampling samples a path, starting at the
final state, using the edge and forward probabilities
p(s1|s0)s1p(s3|s1)s0p(s2|s0)s2s3p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

backward sampling
sample edges from the final statee(s5→x)p(x=s3) ∝ p(s5|s3)*f(s3)p(x=s4) ∝ p(s5|s4)*f(s4)21Learning a Language Model from Continuous Speech

Backward Sampling●

Backward sampling samples a path, starting at the
final state, using the edge and forward probabilities
p(s1|s0)s1p(s3|s1)s0p(s2|s0)s2s3p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

backward sampling
sample edges from the final statee(s5→x)p(x=s3) ∝ p(s5|s3)*f(s3)p(x=s4) ∝ p(s5|s4)*f(s4)22Learning a Language Model from Continuous Speech

Backward Sampling●

Backward sampling samples a path, starting at the
final state, using the edge and forward probabilities
p(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

backward sampling
sample edges from the final statee(s3→x)p(x=s1) ∝ p(s3|s1)*f(s1)p(x=s2) ∝ p(s3|s2)*f(s2)23Learning a Language Model from Continuous Speech

Backward Sampling●

Backward sampling samples a path, starting at the
final state, using the edge and forward probabilities
p(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

backward sampling
sample edges from the final state24

Learning a Language Model from Continuous Speech

Backward Sampling●

Backward sampling samples a path, starting at the
final state, using the edge and forward probabilities
p(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

backward sampling
sample edges from the final state25

Learning a Language Model from Continuous Speech

Backward Sampling●

Backward sampling samples a path, starting at the
final state, using the edge and forward probabilities
p(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

backward sampling
sample edges from the final state

26

Learning a Language Model from Continuous Speech

3. WFST Implementation andLearning from Speech27

Learning a Language Model from Continuous Speech

Generating Word Segmentation
Candidates with WFSTs●●

We propose a simple way to generate word segmentation
candidates using WFSTs

The WFSTs are quite similar to those used in ASR

Input X

Dictionary L

LM + SM

m/ε

i/i
a/ε

a/a

ε/amw

Next

m/m
m/mc

28

Learning a Language Model from Continuous Speech

A Language Model WFST
for Word Segmentation
w2:p(w2|w1)●

w1 w :p(w )11ε:p(FB|w1)

w2

w2:p(w2)

ε:p(FB|w2)

LM

Express both theLanguage Model LM
and Spelling Model
SM as a single WFSTεε:p(FB)
ε:p(
|c1)

c1:p(c1|
)
ε:p(FB|c1)

ε:p(FB|
)

ε:p(
|c2)c1

c2:p(c2)c2

c1:p(c1)εSMε:p(FB|c2)
29

Learning a Language Model from Continuous Speech

A Language Model WFST
for Word Segmentation
w2:p(w2|w1)●

w1 w :p(w )11ε:p(FB|w1)

w2

w2:p(w2)

ε:p(FB|w2)

LM

Express both theLanguage Model LM
and Spelling Model
SM as a single WFSTεε:p(FB)
ε:p(
|c1)

c1:p(c1|
)
ε:p(FB|c1)

ε:p(FB|
)

The key is theweighted edges
connecting thetwo models

ε:p(
|c2)c1

c2:p(c2)c2

c1:p(c1)εSMε:p(FB|c2)
30

Learning a Language Model from Continuous Speech

Word Segmentation
Candidates as a WFST●

Vocabulary “i, a, am”, unigram model
m/am ｗ :PL(am)a/εi/i ｗ :PL(i)PL(FB) ε/
:
PS(
)i/ic:PS(i)a/a ｗ :PL(a)PL(FB)

ε/
:
PS(
)a/ac:PS(a)PL(FB) ε/
:
PS(
)ε/
:
PL(
)m/mc:PS(m)31Learning a Language Model from Continuous Speech

Word Segmentation
Candidates as a WFST●

Vocabulary “i, a, am”, unigram model
m/am ｗ :PL(am)a/εi/i ｗ :PL(i)PL(FB) ε/
:
PS(
)i/ic:PS(i)a/a ｗ :PL(a)PL(FB)

ε/
:
PS(
)a/ac:PS(a)PL(FB) ε/
:
PS(
)ε/
:
PL(
)m/mc:PS(m)32Learning a Language Model from Continuous Speech

Word Segmentation
Candidates as a WFST●

Vocabulary “i, a, am”, unigram model
m/am ｗ :PL(am)a/εi/i ｗ :PL(i)PL(FB) ε/
:
PS(
)i/ic:PS(i)a/a ｗ :PL(a)PL(FB)

ε/
:
PS(
)a/ac:PS(a)PL(FB) ε/
:
PS(
)ε/
:
PL(
)m/mc:PS(m)33Learning a Language Model from Continuous Speech

Word Segmentation
Candidates as a WFST●

Vocabulary “i, a, am”, unigram model
m/am ｗ :PL(am)a/εi/i ｗ :PL(i)PL(FB) ε/
:
PS(
)i/ic:PS(i)a/a ｗ :PL(a)PL(FB)

ε/
:
PS(
)a/ac:PS(a)PL(FB) ε/
:
PS(
)ε/
:
PL(
)m/mc:PS(m)34Learning a Language Model from Continuous Speech

Adaptation to Speech●

When using WFSTs, adaptation to speech is simple●

Replace input X with a HMM-based acoustic model●●

Forward-filtering ＝ creation of a recognition lattice

However, full expansion using HMMs is impossible●

Instead, we use a trimmed phoneme lattice withacoustic model scores

Text Xia

Speech X
mi/PAM(i)e/PAM(e)a/PAM(a)y/PAM(y)m/PAM(m)a/PAM(a)35

Learning a Language Model from Continuous Speech

Learning from Text,
Learning from Speech
Text

SpeechInput

Character String

Phoneme LatticeTechnique

WFST Composition,
Sampling

WFST Composition,
Sampling

Probability

P(W|G)P(G)(LM Likelihood, Prior)

P(X|W)P(W|G)P(G)(AM, LM Likelihoods,
Prior)

Samples

Segmentation, LM

Phoneme String for
Each Utterance,
Segmentation, LM 36

Learning a Language Model from Continuous Speech

4. Evaluation

37

Learning a Language Model from Continuous Speech

Experimental Setting●

Target: Speech from meetings of the Japanese diet
Fluent, large-vocabulary speech● Actual vocabulary size is 2858 words●●

Data preparation: triphone acoustic model
PER: one-best 34.2%, oracle 8.1%
● Used syllable lattices, not phoneme lattices (due torequirements of the decoder)● 8-117 minutes of training data, 27 minutes of test data●●

Evaluation standard:●

Phoneme error rate over the test data using language
model learned from training speech38

Learning a Language Model from Continuous Speech

PER Results

AM Only 34.2%

Phoneme Error Rate

28.00%

Proposed 3-gram
Proposed 2-gram
Proposed 1-gram

27.50%
27.00%
26.50%
26.00%
25.50%
25.00%

0

7.9

16.1

31.1

58.7

116.7

Size of Training Data (Minutes)●●

An LM learned from continuous speech reduced the PER
by 8.92%
3-gram is better than 1-gram: learned contextual info

39

Learning a Language Model from Continuous Speech

Other Training Methods
29.00%
28.50%

Phoneme Error Rate

28.00%
27.50%

1-best & syllable 3-gram

27.00%

1-best & unsupervised seg.

26.50%
26.00%

lattice & unsupervised seg(proposed method)

25.50%
25.00%

manual transcription,
segmentation

24.50%
24.00%
0.0

7.9

16.1

31.1

58.7

Size of Training Data (Minutes)

116.7
40

Learning a Language Model from Continuous Speech

5. Conclusion

41

Learning a Language Model from Continuous Speech

Conclusion●

We demonstrated that it is possible to learn alanguage model from continuous speech
Released open source
http://www.phontron.com/latticelm●

A number of potential applications
Learning language models and dictionaries forresource-poor languages● Elegant handling of spoken language● Semi-supervised learning●

42

Learning a Language Model from Continuous Speech

Thank You43

Learning a Language Model from Continuous Speech

Extra Slides

44

Learning a Language Model from Continuous Speech

Vocabulary/Model Complexity
1-gram2-gram3-gramGold
Standard
3-gramVocabulary

4480

1351

708

2858

Average Word
Length (Syl.)

2.03

1.37

1.18

1.73

Language Model
States

4480

16150

38759

34073

Spelling Model
States

9624

3869

2426

8378
45

Learning a Language Model from Continuous Speech

Words learned
Particles
word

noEnglishpossessive

ni positional
to andColloquial Expressions# (rank)
1052 (1)word

830 (2)685 (5)yu:

Englishsay (colloq)# (rank)
324 (19)e:filler202 (28)

desune discourse marker

94 (65)rimasukeredomo, mo:shiage, yu:fu:ni

Subwords
word

ka
to:

Englishparticle,
subwordsubwordsai subword

Content/Function Words# (rank)
713 (3)204 (27)
94 (65)

word

koto

Englishthing# (rank)
189 (32)

omo

think (stem)

56 (109)

hanashi speak

23 (242)46

jo:kyo:, kangae, chi:ki, toki, shiteki

Learning a Language Model from Continuous Speech

Experimental Setup (2)●Training:
8-117 minutes of continuous speech as training data
● 0.5-20 second utterances● Flat priors on hyperparameters, little influence● 20 samples burn-in, 50 LM samples●●

Testing:
27 minutes of speech separate from the training data
● Lattice rescoring (not speech recognition)● Viterbi phoneme strings for each LM sample combinedusing ROVER●

47

Learning a Language Model from Continuous SpeechInteresting Pronunciation Variants●

nippon (Japan) → nippo:nLearned with a long vowel not in the transcription● Extra emphasis is put on the name of the country,
particularly when using nippon instead of nihon●●shiteorimasu (is doing)→ shitorimasu●●

N → nothing●●

There are many places where the speakers skip vowels
Many word-final Ns are not recognized by the AM

Perhaps taking these into account would improve AM
training?
48

Learning a Language Model from Continuous Speech

Entropy Evaluation●

Gain over 1-best is much lower, why?
Different pronunciations than the transcription
shiteorimasu→shitorimasu● Large effect on entropy, small on PER●

7Per-Syllable Entropy

6.5

1-best & syllable 3-gram

6

1-best & unsupervised seg.

5.5

lattice & unsupervised seg(proposed method)

5
4.5
4

0

7.9

16.1

31.1

58.7

Minutes of Training Data

116.7

manual transcription,
segmentation
49

Learning a Language Model from Continuous Speech

Future Work: Grounding●

The model learns a segmented phoneme string●

For transcription, use actual text
Grounding with a grapheme string without
pronunciations (subtitles?)● In semi-supervised learning, phonetic pronunciations of
unknown words is often sufficient●●

For dialog, use semantic grounding●

Use a robot with cameras, match images to words

50

Learning a Language Model from Continuous Speech

Future Work: Integration with HMM
Currently working on lattices, direct integration with
HMM will give better results (for both training, testing)
28.00%

Proposed 3-gram
Oracle 3-gram

27.50%

Phoneme Error Rate●

Transcript 3-gram

27.00%
26.50%
26.00%
25.50%
25.00%
24.50%
24.00%
0.0

7.9

16.1

31.1

Minutes of Training Data

58.7

116.7
51

Learning a Language Model from Continuous Speech

Future Work: Implementation
●

Speed
●

Expanding FST lattice and forward filtering take a fairamount of time–●

0.5-1 times real time

Several ways for improvement
Perform beam-search trimming during forward filtering– Parallel sampling–●

Open-source
Will be made open-source pending code clean-up● Goal: mid-September●

52

Learning a Language Model from Continuous Speech

Formal Modeling●

For text word segmentation, P(X|W) = 1, but forspeech this is not the case●

Our new objective is the joint probability of the modeland acoustic features
P(X,W,G)=P(X|W)P(W|G)P(G)Acoustic Model●

Language Model

Prior

Use an acoustic model scaling factor
P(X,W,G)=P(X|W)λP(W|G)P(G)●53
Set to .2 (values between .1-.2 produced similar results)

Learning a Language Model from Continuous Speech

Weighted Finite State Transducers(WFSTs)●Finite state automata with input/output/weight
A
a/s:2

B
b/t:1

s/x:0.5
t/y:3●

Define weighted relations over strings●●

If weights are probabilities, probabilistic relations

Transducers combined through composition
A○B
a/x:2.5b/y:4
54

Learning a Language Model from Continuous Speech

Connecting Edges in Detail
ε:p(FB)ε:p(
|c2)

ε

c2



●

To the SM from the basestate●

ε:p(
|c1)●

Equal to the probability ofgenerating a symbol fromthe base distribution

c1

In HPYLM, n-grams with an
unknown word as wi-1 areequal to base probabilities*
P w i∣w i −2 ,UNK=P w i∣UNK=P w i ●

OK to make edges from the
SM only to base state* technically not true if the same word
appears twice in a single sentence55Learning a Language Model from Continuous Speech

Difference from Mochihashi's Method
Mochihashi

NeubigSpelling Model∞-gram+Character 3-gram
Poisson Distribution
Explicit Length Limit No Length Limit

Implementation

Algorithmic
Faster?

WFST-Based
Simpler?, Lattice Possible

Worst-Case
Complexity

O(MLn)
M=Sentence length
L=Max word length
n=n-gram lengthO(Mn+1)

Expected
Complexity

O(MLn)

O(kM+E)
E=Number of existing
word n-grams

56