Scale-Invariant Infinite Hierarchical Topic Model
Shusei Eshima
Department of Government, Harvard University
shuseieshima@g.harvard.edu

Scale-Invariant Infinite Hierarchical Topic Model (ihLDA)▶ Topic models: summarize, annotate, and categorize documents for a human reader▶ Hierarchical topic models: learning hierarchicallatent topic organization, especially for a large
number of topics (say, > 1000) [1, 2, 3]
Existing models: fragmented tree structures▶ Expected probabilities of topics decay exponentially along the depth of tree
▶ Heuristic rules to update topics▶ Restricting the tree structure (e.g., truncating thedepth to three levels)

Contributions
1. Adjusting the probability scale by considering thesize of the parent topic → less fragmentation
2. Extending the tree-structured stick-breaking process (TSSB) [4] → variety of applications
3. Efficiently drawing an infinite topic tree for eachdocument from a base infinite tree in a hierarchical
Bayesian fashion

Document-Topic Distribution:▶ We apply the hierarchical Dirichlet process separately to the verticaland horizontal probabilities.
ÍÎ𝜈𝝐 ∼ Be(𝑎𝜏𝝐˜ , 𝑎(1− 𝜿 Í⪯ 𝝐˜ 𝜏𝜿 )) where 𝜏𝝐 = 𝜈𝝐𝜿≺𝝐 (1−𝜈𝜿 )𝜓𝝐 𝑘 ∼ Be(𝑏𝜙𝝐˜ 𝑘 , 𝑏(1− 𝑘𝑗=1 𝜙𝝐˜ 𝑗 ))Topic-Word Distribution:▶ The hierarchical Pitman-Yor process [5]▶ The semantic similarity between a parent topic
and its children▶ Increasing the specificity as the tree deepens
Inference:▶ Gibbs sampling, Retrospective sampling + Binary search, Slice sampling

Proposed:
L1: said year would also people
L2: said people mobile technology phone
L3: said software site user mailL2: said would government people law
L3: tax said government would budget
L3: labour election said party blair
L2: film said best award year
L3: music band song year album
L3: game dvd film year sony

TSSB is a crucial building block of recent neural
hierarchical topic models [2, 3]. Let 𝜿 ≺ 𝝐 indicatethat 𝜿 is an ancestor of 𝝐,𝜋 𝝐 = 𝜈𝝐(1−𝜈𝜿 ) ·𝜿≺𝝐𝜙𝜿 , 𝜙𝝐 𝑘 = 𝜓𝝐 𝑘𝜿 ⪯𝝐𝑘−1Ö(1 − 𝜓𝝐 𝑗 )𝑗=1𝜈𝝐 ∼ Be(1, 𝛼0 ) , 𝜓𝝐 ∼ Be(1, 𝛾0 ) .

▶ The first term: the probability of stopping at thetopic 𝝐 vertically.
▶ The next product terms: to passing ancestors of 𝝐
while horizontally stopping at 𝝐 and its ancestors.
▶ Vertical and horizontal probabilities of stopping
follow Beta distribution.
The scale-invariant TSSB rescales 𝛾0 ,

...

...

...

Comparing Top Words from Three Models (max level = 3 for comparison)

Scale-Invariant TSSBÖ...

Data Generation Process: 𝝅 (𝑑) := a TSSB for a doc 𝑑.
e.
1. Draw a base TSSB 𝝅
2. Draw topic-word distributions 𝐻𝝐 from the HPY fore.
each topic in 𝝅
3. Draw a document-topic distribution for each document𝑑, 𝝅 (𝑑) ∼ HTSSB(e𝝅 ).
4. For each word position 𝑖 in a document 𝑑, draw a topic,𝑧 𝑑𝑖 ∼ 𝝅 (𝑑) and draw a word, 𝑤 𝑑𝑖 ∼ 𝐻 𝑧𝑑𝑖 .

Probabilistic: nCRP [1]
L1: said year one time would
L2: said year also would company
L3: film show magic would child
L3: film indian star india actor
L3: film dvd effect extra man
L3: film harry potter dvd warner
L2: best award film actor actress
L2: film star story life singer
L2: film star movie actress alsoExperiments

Neural: TSNTM [2]
L1: said show year also would
L2: said year game world time
L3: england first game ireland win
L3: said labour blair party election
L3: said would people law gov.
L3: said would gov. election tax
L3: said would tax gov. election
L3: said would tax gov. election
L3: said would tax gov. election

Topic-Word Probabilities (BBC, L6)

Data:▶ BBC News: 2,225 documents in five topic areasfrom the BBC news website▶ 20News: a collection of 18,828 posts from 20
USENET newsgroups▶ Wikipedia: 50,153 English articles randomly
sampled from ten main categories

Topic−word probability

Problem StatementÖDaichi Mochihashi
The Institute of Statistical Mathematics
daichi@ism.ac.jp

Setup: Comparing topics with at least 100 assigned
words (no truncation in parentheses)

system

0.03
0.02
0.01
0.00

digitalsony

government

electionjohnsonwinplayerronaldoFirst branch
Third branch
Fourth branch

0.03
0.02
0.01
0.00
0.03
0.02
0.01
0.00
12

3

456

12345612

3456

Level in the estimated topic tree𝜓𝝐 ∼ Be(1, 𝜙𝝐 ′ 𝛾0 ).
Using the horizontal breaking proportion of a parent
topic, 𝜙𝝐 ′ , to draw a relative stick length for its child
topic → A larger break if the stick to break is shorter.
The expected probabilty of horizontal break at ℓ is
E[𝜙 | ℓ] ≈ 1/(2𝛾 + 1/E[𝜙 | ℓ − 1]) for ℓ ≥ 2. It wasℓoriginally E[𝜙 | ℓ] ≈ 1/(2𝛾 +1) .
SI-TSSB has a slower decay.

Expected probability

0.4

φ[1] = 0.4

Model
Scale−invariant TSSB
Original TSSB

0.30.2 φ[1] = 0.20.1Top Words of Selected Topics from the Estimated Topic Tree (BBC, L6)
said year would also people

film award bestyear said
best film award
actor british

said company
year firm also

said people could
one systemgame technology
mobile music digital

said chip makemachine computer

film angel france
actor young

service people phonemobile broadband

mini mac said
apple computer

dvd game sony
psp nintendo

said law government
minister would
said labour election
party blair

lord labour reform
minister speaker

robot car ministreet produced

said year finalwin firstgame player teamclub league

immigration asylum
state measure argued

minister mayor ken
journalist apologise

sport winter breakplayer league

club deal playcontract season

think apologise mayor
reporter johnsonmusic said award
artist year

sound mobile
song audio
bestengland madrid
great sport stadium…

song music bestyear british
music court
industry
sharing peer
beckham real owenbernabeu ronaldo

φ[1] = 0.1

0.0
1

2

3

4

5

6

7

8

910

Evaluation Metrics

Level

Model

References

Max
Lvl.
3ihLDA[1] D. M. Blei, T. L. Griffiths, M. I. Jordan, and J. B. Tenenbaum,“Hierarchical topic models and the nested Chinese restaurantprocess,” in NIPS, 2003, pp. 17–24.
[2] M. Isonuma, J. Mori, D. Bollegala, and I. Sakata, “Treestructured neural topic model,” in ACL, 2020, pp. 800–806.
[3] Z. Chen, C. Ding, Z. Zhang, Y. Rao, and H. Xie, “Treestructured topic modeling with nonparametric neural variational inference,” in ACL, 2021, pp. 2343–2353.
[4] R. P. Adams, Z. Ghahramani, and M. I. Jordan, “Treestructured stick breaking for hierarchical data,” in NIPS, 2010,
pp. 19–27.
[5] Y. W. Teh, “A Bayesian interpretation of interpolated KneserNey,” NUS School of Computing, Tech. Rep., 2006.

≥4nCRPrCRP
TSNTM
nTSNTM

3Tree Diversity (↑)
BBC 20News Wiki

Topic Uniqueness (↑)
BBC 20News Wiki

Average Overlap (↓)
BBC 20News Wiki

# of Topics
BBC 20News Wiki

2.24
(2.24)
2.53(2.54)
1.92
0.15

2.88
(2.86)
2.88(2.80)
2.16–

2.63(2.49)
2.50(2.51)––

0.60(0.60)
0.55(0.49)
0.36
0.01

0.82
(0.80)
0.76(0.51)
0.32–

0.66(0.63)
0.65(0.63)––

0.28(0.28)
0.26(0.30)
0.03
0.53

0.11(0.14)
0.12(0.38)
0.02–

0.16(0.19)
0.15(0.16)––

38(38)
85(134)517
278

27(31)
67(203)2108–

17(18)
73(101)
–
–

1.98
2.11

2.54
2.57

2.47
2.34

0.43
0.46

0.80
0.68

0.64
0.60

0.26
0.09

0.09
0.01

0.06
0.02

2268

41
81

44111

▶ Tree Diversity: uniqueness of topics while considering the importance of the parent topics▶ Topic Uniqueness [3]: uniqueness of all topics▶ Average Overlap [3]: the average repetition rate of the top words (less overlap does not necessarily mean betterinterpretability)▶ The results without truncation are shown in parentheses for the proposed model.