Graham Neubig – Non-parametric Bayesian Statistics

Non-parametric Bayesian
Statistics

Graham Neubig
2011-12-22

1Graham Neubig – Non-parametric Bayesian Statistics

Overview●

About Bayesian Non-parametrics
Basic theory● Inference using sampling● Learning an HMM with sampling● From the finite HMM to the infinite HMM● Recent developments (in sampling and modeling)● Applications to speech and language processing●●

Focus on unsupervised learning for discrete
distributions
2

Graham Neubig – Non-parametric Bayesian Statistics

Non-parametricBayes

The number of parameters
Put a prior on theis not decided in advance parameters and consider(i.e. infinite)their distribution3

Graham Neubig – Non-parametric Bayesian Statistics

Types of Statistical Models

Maximum
Likelihood
BayesianParametric
Bayesian
Nonparametric

Prior on# ofParameters Parameters(Classes)No
FiniteDiscrete
Distribution

Continuous
Distribution

Multinomial

Gaussian

Yes

FiniteMultinomial+ Gaussian+
Dirichlet
Gaussian
Prior
Prior

Yes

Infinite

Multinomial+ Gaussian
Dirichlet
Process
Process
Covered Here

4

Graham Neubig – Non-parametric Bayesian Statistics

Bayesian Basics

5Graham Neubig – Non-parametric Bayesian Statistics

Maximum Likelihood (ML)●We have an observed sample
X=1245214414●

Gather counts C={c 1, c 2, c 3, c 4, c 5 }={3,2,0,4,1}●

Divide counts to get probabilities
P  x =i =ci∑i c i
multinomial

P  x =
 ={0.3,0.2, 0 ,0.4, 0.1}
6Graham Neubig – Non-parametric Bayesian Statistics

Bayesian Inference●

ML is weak against sparse data●

Don't actually know parameters
we could have
 ={0.3,0.2, 0 ,0.4, 0.1}

if c  x ={3,2,0,4,1}

or we could have
 ={0.35,0.05, 0.05,0.35, 0.2}●

Bayesian statistics don't pick one probability●

Use the expectation insteadP  x =i =∫  P  ∣X d i7Graham Neubig – Non-parametric Bayesian Statistics

Calculating Parameter Distributions●

Decompose with Bayes' law
likelihood

P ∣X =prior

P  X∣ P ∫ P  X∣ P  d regularization coefficient●

likelihood easily calculated according to the model●prior chosen according belief about probable values●

regularization requires difficult integration...
●… but conjugate priors make things easier
8Graham Neubig – Non-parametric Bayesian Statistics

Conjugate Priors●

Definition: Product of likelihood and prior takes thesame form as the priorMultinomial Likelihood * Dirichlet Prior = Dirichlet Posterior
Gaussian Likelihood * Gaussian Prior = Gaussian Posterior
Same●

Because the form is known, no need to take theintegral to regularize

9Graham Neubig – Non-parametric Bayesian Statistics

Dirichlet Distribution/Process●

Assigns probabilities to multinomial distributions
e.g.

●

P {0.3, 0.2,0.01, 0.4, 0.09}=0.000512
P {0.35, 0.05,0.05, 0.35,0.2 }=0.0000963

Defined over the space of proper probability
distributions {1 ,, n }∀  0 i 1
i●n

∑i =1 i =1

Dirichlet process is a generalization of distribution●

Can assign probabilities to infinite spaces
10Graham Neubig – Non-parametric Bayesian Statistics

Dirichlet Process (DP)
●

Eq.

n
1
P

P   ;  ,P base = ∏i =1  iZbase x =i −1

α is the “concentration parameter,” larger value means
more data needed to diverge from prior● P
is the “base measure,” expectation of θbase●

Way of writing in
Dirichlet distribution

i = P base  x =i Way of writing in
Dirichlet processn●∏i =1   P base  x =i RegularizationZ=ncoefficient:  ∑ i=1  P base  x =i (Γ=gamma function)

11

Graham Neubig – Non-parametric Bayesian Statistics

Examples of Probability Densities
α = 10Pbase =
{0.6,0.2,0.2}

α=9Pbase =
{0.22,0.33,0.44}

α = 15
Pbase =
{0.2,0.47,0.33}

α = 14
Pbase =
{0.43,0.14,0.43}
12From Wikipedia

Graham Neubig – Non-parametric Bayesian Statistics

Why is the Dirichlet Conjugate?
Likelihood is product of multinomial probabilities
Data: x 1 =1, x 2 =5, x 3 =2, x 4 =5
P  X∣=p x =1∣  p  x =5∣  p  x =2∣  p  x =5∣ = 1  5  2  5●●

Combine multiple instances into a single countc  x =i ={1, 1, 0, 0, 2}n2
5c  x =i iP  X∣=1  2  = ∏i =1 ●Take product of likelihood and priorn∏i=1 c  x =i i∗

1
Z priorn∏i=1 

i −1
i

→

1
Z postn∏i =1 c  x = i i −1
i13

Graham Neubig – Non-parametric Bayesian Statistics

Expectation of θ in the DP●

When N=2

1

1  −1  −1
E [ 1 ]=∫0  1  1  2 d  1
Z
1 1 
 −1
= ∫0  1 1− 1 
d 1
Z
1

1

2

1

u=

du=1 dv =1− 1 

2 −1

2

2

1

2

Integration by Parts
1
1

1


1
= [− 1 1−1  / 2 ]0 −
Z
1 1

 −1
−1− 1  / 2∗1  1 d 1
∫
0
Z

1 −1
1

d 1

v =−1− 1  /22∫ u dv =u v −∫ v dud 1

 1 1 1  −1

=0

1−

d 1
∫
1
1
0
2 Z
1
1
= E [ 2 ]= 1−E [ 1 ]
2
2
1

2

1
E [ 1 ]=1  214

Graham Neubig – Non-parametric Bayesian Statistics

Multi-Dimensional Expectation
 P base  x =i 
E [ i ] === P base  x =i n
∑i =1 i
i●

Posterior distribution for multinomial with DP prior:
1

P  x =i =∫0  i
Observed
Counts●1
Z postnc  x =i i −1
i∏i=1 c  x =i ∗P base  x =i =c  ・

Same as additive smoothing

Base Measure
Concentration
Parameter
15

Graham Neubig – Non-parametric Bayesian Statistics

Marginal Probability●

Calculate prob. of observed data using the chain ruleX = 1 2 1 3 1 α=1 Pbase(x=1,2,3,4) = .25
c = { 0, 0, 0, 0 }01∗.25
P  x 1=1==.25
01c = { 1, 0, 0, 0 }01∗.25
P  x 2 =2∣x 1 =
=.125
11
c = { 1, 1, 0, 0 }11∗.25
P  x 3 =1∣x 1,2 =
=.417
21

c  x i ∗P base  x i P  x i =c  ・ c = { 2, 1, 0, 0 }P  x 4 =3∣x 1,2,3 =

01∗.25=.063
31

c = { 2, 1, 1, 0 }P  x 5 =1∣x 1,2,3,4 =

21∗.25=.45
41

Marginal Probability
P(X) = .25*.125*.417*.063*.45
16

Graham Neubig – Non-parametric Bayesian Statistics

Chinese Restaurant Process●

Way of expressing DP and other stochastic processes●

Chinese restaurant with infinite number of tables●

Each customer enters restaurant and takes action:
P sits at table i ∝ c i P sits at a new table∝ ●

When the first customer sits at a table, choose thefood served there according to Pbase
X = 1 2 1 3 1 α=1 N=4
1

2

1

3

…
17

Graham Neubig – Non-parametric Bayesian Statistics

Sampling Basics

18

Graham Neubig – Non-parametric Bayesian Statistics

Sampling Basics●

Generate a sample from probability distribution:

Distribution: P(Noun)=0.5 P(Verb)=0.3 P(Preposition)=0.2
Sample: Verb Verb Prep. Noun Noun Prep. Noun Verb Verb Noun …
●Count the samples and calculate probabilities

P(Noun)= 4/10 = 0.4, P(Verb)= 4/10 = 0.4, P(Preposition) = 2/10 = 0.2

More samples = better approximation
1

Probability

●

0.8

Noun
Verb
Prep.

0.6
0.4
0.2
0
1E+00 1E+01 1E+02 1E+03 1E+04 1E+05 1E+06
Samples19

Graham Neubig – Non-parametric Bayesian Statistics

Actual Algorithm
SampleOne(probs[])z = sum(probs)
remaining = rand(z)for each i in 1:probs.size
remaining -= probs[i]if remaining <= 0
return i

Calculate sum of probs
Generate number from
uniform distribution over [0,z)Iterate over all probabilities
Subtract current prob. value
If smaller than zero, returncurrent index as answer

Bug check, beware of overflow!
20Graham Neubig – Non-parametric Bayesian Statistics

Gibbs Sampling●

Want to sample a 2-variable distribution P(A,B)… but cannot sample directly from P(A,B)● … but can sample from P(A|B) and P(B|A)●●●

Gibbs sampling samples variables one-by-one torecover true distribution
Each iteration:
Leave A fixed, sample B from P(B|A)
Leave B fixed, sample A from P(A|B)

21

Graham Neubig – Non-parametric Bayesian Statistics

Example of Gibbs Sampling●●

Parent A and child B are shopping, what sex?
P(Mother|Daughter) = 5/6 = 0.833
P(Mother|Son) = 5/8 = 0.625
P(Daughter|Mother) = 2/3 = 0.667
P(Daughter|Father) = 2/5 = 0.4
Original state: Mother/Daughter
Sample P(Mother|Daughter)=0.833, chose Mother
Sample P(Daughter|Mother)=0.667, chose Sonc(Mother, Son)++Sample P(Mother|Son)=0.625, chose Mother
Sample P(Daughter|Mother)=0.667, chose Daughterc(Mother, Daughter)++
22…

Graham Neubig – Non-parametric Bayesian Statistics

Try it Out:
Probability

1
0.8
0.6

Moth/Daugh
Moth/Son
Fath/Daugh
Fath/Son

0.4
0.2
0
1E+00

1E+02

1E+04

1E+06

Number of Samples●

In this case, we can confirm this result by hand
23

Graham Neubig – Non-parametric Bayesian Statistics

Learning a Hidden Markov Model
Part-of-Speech Tagger
with Sampling24

Graham Neubig – Non-parametric Bayesian Statistics

Unsupervised Learning●

Observed Training Data X●●

Hidden Variables Y●●

e.g.: A corpus of natural language text
e.g.: States of the HMM = Parts of Speech of words

Unobserved Parameters θ●Generally probabilities

25

Graham Neubig – Non-parametric Bayesian Statistics

Task: Unsupervised POS Induction●

Input: Collection of word strings X
the boats row in a row●

Output: Collection of clusters Y
1

2

3

4

1

2

1→Determiner 2→Noun 3→Verb 4→Preposition
the boats row in a row
Det N
V P Det N

26

Graham Neubig – Non-parametric Bayesian Statistics

Model: HMM
Variables Y correspond to hidden states● State transition probability: P y ∣yTii−1 = T , y , y● Generate each word from a hidden state
P E  x i∣y i = E , y , x● Word emission probability:●ii

PT(1|0)0

PT(2|1)

PT(3|2)

i

…

1

2

3

4

1

2

the

boats

row

in

a

row

PE(the|1)i −1PE(boats|2) PE(row|3)

…

0

27

Graham Neubig – Non-parametric Bayesian Statistics

Sampling the HMM●

Initialize Y randomly●

Sample each element of Y using Gibbs sampling
sample this only
0

1

2

3

4

1

2

the

boats

row

in

a

row

0

28

Graham Neubig – Non-parametric Bayesian Statistics

Sampling the HMM
Markov blanket
0

●

1

2

3

4

1

2

the

boats

row

in

a

row

0

Probabilities affected by a single tag●

Transition from previous tag: PT(yi|yi-1)●

Transition to next tag:PT(yi+1|yi)●

Emission probability:

PE(xi|yi)●Sample the tag value according to these probabilities●

All variables that have effect are “Markov blanket”

29Graham Neubig – Non-parametric Bayesian Statistics

Calculating HMM Probabilities
with DP Priors●

Transition probability:c y i−1 y i T ∗P baseT  y i 
P T y i∣y i−1 =c y i−1 T●

Emission probability:c  y i , x i E ∗P baseE  x i 
P E  x i∣y i =c  y i E

30

Graham Neubig – Non-parametric Bayesian Statistics

Sampling Algorithm for One Tag
SampleTag(yi)c(yi-1 yi)--; c(yi yi+1)--; c(yi→xi)--

Subtract current
tag counts
Calculate all possible
tag probabilities

for each tag in S (all POS tags)
p[tag]=PE(tag|yi-1)*PE(yi+1|tag)*PT(xi|tag)yi = SampleOne(p)Choose a new tagc(yi-1 yi)++; c(yi yi+1)++; c(yi→xi)++

Add the newtag counts

31Graham Neubig – Non-parametric Bayesian Statistics

Sampling Algorithm for All Tags
SampleCorpus()
initialize Y randomly

Randomly initialize tagsfor N iterations

For N iterations

for each yi in the corpus
SampleTag(yi)save parameters
average parameters

Sample all the tags

Save sample of θ
Average parameters θ32Graham Neubig – Non-parametric Bayesian Statistics

Choosing Hyperparameters●

Must choose α properly to get desired effect●

Small α(<0.1) creates sparse distributions
–

●

●

●

If we want each word to have one POS tag, we can set
αE of the emission distribution Pe to be small

Most distributions are sparse, so often α is set small

Best to confirm through experiments
Can also give hyperparameters a prior and sample
them as well33

Graham Neubig – Non-parametric Bayesian Statistics

From the Finite HMM
to the Infinite HMM

34

Graham Neubig – Non-parametric Bayesian Statistics

Base Measure and Dimensionality
Using a uniform distribution as the base measure

Probability

6 Parts of Speech
0.2
0.1
0

1

2

3

4

5

6

POS Number

20 Parts of Speech
Probability

●

0.06
0.04
0.02
0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

POS Number

35

Graham Neubig – Non-parametric Bayesian Statistics

In the Limit...
0.02
0.01
0.01
0

1 Million Parts of Speech
Probability

Probability

100 Parts of Speech

5 13 21 29 37 45 53 61 69 77 85 93
1 9 17 25 33 41 49 57 65 73 81 89 97

POS Number●

1.50E-06
1.00E-06
5.00E-07
0.00E+00

130000 370000 610000 850000
10000 250000 490000 730000 970000

POS Number

As the number of POSs goes to infinity●

Probabilities of each POS Pbase goes to zero●But total probability of Pbase is the samec  y i −1 y i ∗P base y i P  y i∣y i −1 =c  y i−1 N=numberof POSsNlim ∑i =1

N ∞

1
=1
36
N

Graham Neubig – Non-parametric Bayesian Statistics

Finite HMM and Infinite HMM●

Finite HMM
Probability of emitting POS yi (after yi-1)c  y i −1 y i ∗P base y i P  y i∣y i −1 =c  y i−1 ●Infinite HMM
Probability of omitting
existing POS yi (after yi-1)
Probability of omitting
new POS (after yi-1)c  y i −1 y i P  y i∣y i −1 =c  y i −1 P  y i =new∣y i −1 =c  y i −1 
37Graham Neubig – Non-parametric Bayesian Statistics

Example●Assume c(yi-1=1 yi=1)=1 c(yi-1=1 yi=2)=1

When there are 2 possible POSs
P  y i =1∣y i−1 =1=1∗1 /2
1∗1/2
P  y i =2∣y i −1 =1=
2
2
∗0
P  y i ≠1,2∣y i −1 =1=2When there are 20 possible POSs
P  y i =1∣y i−1 =1=1∗1 /20
1∗1/20
P  y i =2∣y i −1 =1=22∗18 /20
P  y i ≠1,2∣y i −1 =1=2When there are infinite possible POSs
P  y i =1∣y i−1 =1=1∗1 /∞1∗1/∞P  y i =2∣y i −1 =1=
2
2
∗1
P  y i ≠1,2∣y i −1 =1=238

Graham Neubig – Non-parametric Bayesian Statistics

Sampling Algorithm
SampleTag(yi)c(yi-1 yi)--; c(yi yi+1)--; c(yi→xi)--

Remove counts for
current tag
Calculate existing POS
probabilities

for each tag in S (possible POSs)
p[tag]=PE(tag|yi-1)*PE(yi+1|tag)*PT(xi|tag)
p[|S|+1]=PE(new|yi-1)*PE(yi+1|new)*PT(xi|new)Calculate new POS
probability

yi = SampleOne(p)Pick a single valuec(yi-1 yi)++; c(yi yi+1)++; c(yi→xi)++

Add the new counts
39Graham Neubig – Non-parametric Bayesian Statistics

Non-Uniform Base Measures●●

Previous slides assumed uniform base measures, but
this is not required
Example: Language model unknown word modelc  word∗P base  word
P  word=c  word●Split each word into characters, give some probability
to all words:
P base  word=P len  4 P char  w P char o P char r  P char d●Probability is not equal, but gives some probability to40each member of an infinite collectionGraham Neubig – Non-parametric Bayesian Statistics

Implementation Tips●

Zero count classes remain → wasted memory●

When new classes are made, re-use class numbersc(y1)=5 c(y2)=0 c(y3)=1●●

Dumb: c(y1)=5 c(y2)=0 c(y3)=1 c(y4)=1
Smart: c(y1)=5 c(y2)=1 c(y3)=1

When c(y)=0, probability of revival becomes 0This model doesn't do well with new POSs
New POSs can only appear after 1 type of POS● Can fix this with hierarchical model●Transition Prob.
POS Prob.

P T y i∣y i−1 =DP  ,P T  y i 
P T y i =DP  , P base  y i 

41

Graham Neubig – Non-parametric Bayesian Statistics

Debugging●

Unit tests! Unit tests! Unit tests!
●

●

●

●

●

Remove bugs in implementation, and conceptualization

Create fail-safe function for adding/subtracting counts,
terminate if count goes below zero
When program finishes, remove all samples and makesure the counts are exactly zero
The likelihood will not always go up, but if itconsistently goes down something is probably wrong
Set the random seed to a single value (srand)42

Graham Neubig – Non-parametric Bayesian Statistics

Recent Topics

43

Graham Neubig – Non-parametric Bayesian Statistics

Block Sampling●

Often hidden variables depend on each-other strongly
depends sampling depends

0

●

●

●

1

2

3

4

1

2

the

boats

row

in

a

row

0

For example, variables close in time and space
Block sampling samples multiple hidden variables at atime, considering dependence
HMMs use forward filtering/backward sampling●

Context free grammars, etc. also possible

44

Graham Neubig – Non-parametric Bayesian Statistics

Forward Filtering●

forward-filtering adds up probabilities starting from an
initial statep(s1|s0)s1

s0p(s2|s0)s2p(s3|s1)s3

p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

forward filtering
calculate forward probabilities ff(s0) = 1
f(s1) = p(s1|s0)*f(s0)f(s2) = p(s2|s0)*f(s0)f(s3) = p(s3|s1)*f(s1) + p(s3|s2)*f(s2)f(s4) = p(s4|s1)*f(s1) + p(s4|s2)*f(s2)f(s5) = p(s5|s3)*f(s3) + p(s5|s4)*f(s4)45Graham Neubig – Non-parametric Bayesian Statistics

Backward Sampling●

Backward sampling starts at the acceptance state andsamples edges in backwards orderp(s1|s0)s1p(s3|s1)s0p(s2|s0)s2s3p(s5|s3)p(s3|s2)p(s4|s1)s5

s4p(s5|s4)p(s4|s2)

backward sampling
considers edge probs and forward probse(s5→x)p(x=s3) ∝ p(s5|s3)*f(s3)p(x=s4) ∝ p(s5|s4)*f(s4)e(s3→x)p(x=s1) ∝ p(s3|s1)*f(s1)p(x=s2) ∝ p(s3|s2)*f(s2)46Graham Neubig – Non-parametric Bayesian Statistics

Type-Based Sampling●

Sample variables that have the same Markov blanket
at once
0

0

1

2

3

4

1

2

the

boats

row

in

a

row

2

5

3

4

1

2

he

will

jump

in

the

pool

0

0

47
●

Here, the Markov blanket is “3,in,1”

Graham Neubig – Non-parametric Bayesian Statistics

Type-base Sampling●

Models based on Dirichlet distributions tend to assign
same tag to similar values (rich-gets-richer)Good for modeling: Induces consistent, compact model● Bad for inference: Creates “valleys” in posterior prob●

We are on the right side
●The left side has moreprobability, but requires
several variable changes●Possible to escape, buttakes a very long time
●

Prob.

48

# of x=1 Values

Graham Neubig – Non-parametric Bayesian Statistics

Type-based Sampling●

For each type, sample thenumber of instances x=1●●“x=1” has one instanceMarkov blankets are
identical, probabilities arealso●

Can set one instance to
x=1 randomly, all others to
x=2

Prob.

# of x=1 Values

49

Graham Neubig – Non-parametric Bayesian Statistics

Hierarchical Models●

Multiple levels using the hierarchical Dirichlet process
P(yi|y =1)
i-1

P(yi|y =2)
i-1

P(yi|y =3)
i-1…

Pbase(yi)Transition prob:
Shared base measure:c  y i −1 y i ∗P base y i P  y i∣y i −1 =c  y i−1 c base  y i ∗1 /N
P base  y i =c base  ・ 
50

Graham Neubig – Non-parametric Bayesian Statistics

Counting cbase●

1

Use the Chinese restaurant process
yi-1 = 1

yi-1 = 2

yi = 1 2 1 3 1

yi = 1 4 2 2 4

2

1

3

…

1

4

3

4

2

…

base
1
●

2

…

Add customers to top level for each data point, add
customers to bottom level for each table in top level51

Graham Neubig – Non-parametric Bayesian Statistics

Pitman-Yor Process●

Similar to Dirichlet process, but adds table discount dc  x i −d∗t  x i d∗t  ・ ∗P base  x i P  x i =c  ・ 

1

2

1

3

…

3−d∗2d∗4∗0.25
P  x i =1=
5
●

●

Similar to absolute discounting for language models
Able to model power-law distributions, which arecommon in language

52

Graham Neubig – Non-parametric Bayesian Statistics

Examples from
Speech and Language Processing53

Graham Neubig – Non-parametric Bayesian Statistics

Topic Models●

Latent Dirichlet Allocation (LDA) [Blei+ 03]

Collection ofDocuments
Generate a multinomial
topic distribution (with a
Dirichlet prior)
Generate each word's
topic from the topic dist.
Generate each wordfrom the topic's word dist●●

this is a document
this is a document
this is a document
this is a document

this is a document
this is a document
this is a document
this is a document

this is a document
this is a document
this is a document
this is a document

Poli. Enter. Sport Econ. Soci. Science{ 0.4, 0.05, 0.3, 0.2, 0.01, 0.04}
1

1

4

3

3

3

Bill Clinton buys the Detroit Tigers

Infinite topic models [Teh+ 06]
Applications to computer vision, document clustering, 54
language modeling (e.g.: [Heidel+ 07] ）Graham Neubig – Non-parametric Bayesian Statistics

Language Models●

Hierarchical Pitman-Yor language model [Teh 06]
bi-gram
uni-gram

P(wi|w =1)
i-1

P(wi|w =2)
i-1

P(wi|w =3)
i-1…

Pbase(wi)Improvements to modeling accuracy by using PitmanYor process● Similar accuracy to Kneser-Ney●●

Used in speech recognition [Huang&Renals 07]
55

Graham Neubig – Non-parametric Bayesian Statistics

Unsupervised Word Segmentation●

Generate word sequences from 1-gram or 2-grammodels [Goldwater+ 09]
これ は 単語 でSamplingは

単P( 単語 )
or

or

これ
●

す

語

で

す

P( 単 )P( 語 )Improvements using block sampling and Pitman-Yor
language model [Mochihashi+ 09]

56

Graham Neubig – Non-parametric Bayesian Statistics

Learning a Language Model from
Continuous Speech●

Use Pitman-Yor language model to learn language modeland word dictionary from speech [Neubig+ 10]
Acoustic
Model
Speech●●Learning
Phoneme Lattice

SpokenLanguage
Model

Use forward filtering-backward sampling over phoneme
lattices
Can be used for:Learning models for languages with no written text
● Learning models faithful to spoken language●

57

Graham Neubig – Non-parametric Bayesian Statistics

Learning Various Typesof Linguistic Information●

POS using infinite HMM [Beal+ 02]●

CFG [Johnson+ 07] and infinite CFG [Liang+ 07]●●

Word and phrase alignment for machine translation[DeNero+ 08, Blunsom+ 09, Neubig+ 11]
Non-parametric extension of unsupervised semantic
parsing [Poon+ 09, Titov+ 11]

58

Graham Neubig – Non-parametric Bayesian Statistics

References
●

●

●

●

●

●

●

●

M.J. Beal, Z. Ghahramani, and C.E. Rasmussen. 2002. The infinite hidden Markov model.
Proceedings of the 16th Annual Conference on Neural Information Processing Systems, 1:577–
584.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of
Machine Learning Research, 3:993–1022.
Phil Blunsom, Trevor Cohn, Chris Dyer, and Miles Osborne. 2009. A Gibbs sampler for phrasal
synchronous grammar induction. In Proceedings of the 47th Annual Meeting of the Association
for Computational Linguistics, pages 782–790.
John DeNero, Alex Bouchard-Cˆot´e, and Dan Klein. 2008. Sampling alignment structure undera Bayesian translation model. In Proceedings of the Conference on Empirical Methods in
Natural Language Processing, pages 314–323.
Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. 2009. A Bayesian framework forword segmentation: Exploring the effects of context. Cognition, 112(1):21–54.
A. Heidel, H. Chang, and L. Lee. 2007. Language model adaptation using latent Dirichlet
allocation and an efficient topic inference algorithm. In Proceedings of the 8th Annual
Conference of the International Speech Communication Association (InterSpeech).
S. Huang and S. Renals. 2007. Hierarchical Pitman-Yor language models for ASR in meetings.
In Proceedings of the 2007 IEEE Automatic Speech Recognition and Understanding Workshop,
pages 124–129.
Mark Johnson, Thomas Griffiths, and Sharon Goldwater. 2007. Bayesian inference for PCFGs
via Markov chain Monte Carlo. In Proceedings of the Human Language Technologies: The
59
Annual Conference of the North American Chapter of the Association for Computational
Linguistics, pages 139–146

Graham Neubig – Non-parametric Bayesian Statistics

References
●

●

●

●

●

●

●

●

P. Liang, S. Petrov, M. Jordan, and D. Klein. 2007. The infinite PCFG using hierarchical Dirichlet
processes. In Proceedings of the Conference on Empirical Methods in Natural Language
Processing, pages 688–697.
Daichi Mochihashi, Takeshi Yamada, and Naonori Ueda. 2009. Bayesian unsupervised word
segmentation with nested Pitman-Yor modeling. In Proceedings of the 47th Annual Meeting of
the Association for Computational Linguistics.
Graham Neubig, Masato Mimura, Shinsuke Mori, and Tatsuya Kawahara. 2010. Learning alanguage model from continuous speech. In Proceedings of the 11th Annual Conference of theInternational Speech Communication Association (InterSpeech), Makuhari, Japan, 9.
Graham Neubig, Taro Watanabe, Eiichiro Sumita, Shinsuke Mori, and Tatsuya Kawahara. 2011.
An unsupervised model for joint phrase alignment and extraction. In Proceedings of the 49th
Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, USA, 6.
H. Poon and P. Domingos. 2009. Unsupervised semantic parsing. In Proceedings of theConference on Empirical Methods in Natural Language Processing, pages 1–10.
Y.W. Teh, M.I. Jordan, M.J. Beal, and D.M. Blei. 2006. Hierarchical dirichlet processes. Journalof the American Statistical Association, 101(476):1566–1581.
Yee Whye Teh. 2006. A Bayesian interpretation of interpolated Kneser-Ney. Technical report,
School of Computing, National Univ. of Singapore.
Ivan Titov and Alexandre Klementiev. 2011. A Bayesian model for unsupervised semantic
parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational
Linguistics.

60