BAT-Chain: Bayesian-Aware Transport Chain for Topic Hierarchies Discovery
Authors: Dongsheng Wang
He Zhao
Dan Dan Guo
Xinyang Liu
Miaoge Li
Bo Chen
Mingyuan Zhou
Venue: Submitted to ICLR 2023 (2023)
Keywords: Topic modeling
hierarchical representation
optimal transport
conditional transport
concept learning

Abstract: Topic modeling has been an important tool for text analysis. Originally, topics discovered by a model are usually assumed to be independent. However, as a semantic representation of a concept, a topic is naturally related to others, which motivates the development of learning hierarchical topic structure. Most existing Bayesian models are designed to learn hierarchical structure, but they need non-trivial posterior inference. Although the recent transport-based topic models bypass the posterior inference, none of them considers deep topic structures. In this paper, we interpret document as its word embeddings and propose a novel Bayesian-aware transport chain to discover multi-level topic structures, where each layer learns a set of topic embeddings and the document hierarchical representations are defined as a series of empirical distributions according to the topic proportions and corresponding topic embeddings. To fit such hierarchies, we develop an upward-downward optimizing strategy under the recent conditional transport theory, where document information is first transported via the upward path, and then its hierarchical representations are refined by the downward path under the Bayesian perspective. Extensive experiments on text corpora show that our approach enjoys superior modeling accuracy and interpretability. Moreover, we also conduct experiments on learning hierarchical visual topics from images, which demonstrate the adaptability and flexibility of our method.

Under review as a conference paper at ICLR 2023
BT-CHAIN: BIDIRECTIONAL TRANSPORT CHAIN FOR TOPIC HIERARCHIES DISCOVERY
Anonymous authors Paper under double-blind review
ABSTRACT
Topic modeling has been an important tool for text analysis. Originally, topics discovered by a model are usually assumed to be independent. However, as a semantic representation of a concept, a topic is naturally related to others, which motivates the development of learning hierarchical topic structure. Most existing Bayesian models are designed to learn hierarchical structure, but they need non- trivial posterior inference. Although the recent transport-based topic models bypass the posterior inference, none them considers deep topic structures. paper, we interpret the document its word embeddings and propose novel bidirectional transport chain to discover multi-level topic structures, where each layer learns set of topic embeddings and the document hierarchical representations are defined as a series of empirical distributions according to the topic proportions and corresponding topic embeddings. To fit such hierarchies, we develop an upward-downward optimizing strategy under the recent conditional transport theory, where document information first transported via the upward path, and then its hierarchical representations are refined according to the adjacent upper and lower layers in layer-wise manner via the downward path. Extensive experiments on text corpora show that our approach enjoys superior modeling accuracy and interpretability. Moreover, we also conduct experiments on learning hierarchical visual topics from images, which demonstrate the adaptability and flexibility of our method.
this
INTRODUCTION
Topic models (TMs) like latent Dirichlet allocation (LDA) (Blei al. 2003). Poisson factor analysis (PFA) (Zhou et al.
2012). and their various extensions (Teh et al.) 2006 Hoffman et al. 2010 Blei
2012; Zhou et al. 2016) are family of popular techniques for discovering the hidden semantic structure from collection of documents in an unsupervised manner. In addition to learning shallow topics, mining the potential hierarchical topic structures has obtained much research effort since the hierarchies are ubiquitous in big text corpora (Meng al. 2020 Lee et al. (022) and can be applied to a wide range of applications (Grimmer 2010 Zhang et al.) Guo et al. 2020).
Hierarchical Bayesian probabilistic models have been commonly used to learn topic structures Blei
et al. 2010 Paisley et al. 2014| Gan et al.) 2015 Henao et al., 2015; Zhou et al. 2016), where a hierarchy of topies learned and the topics the higher layers serve the priors of the topics in the
lower layers. Despite the success of Bayesian models in topic structure mining, most of them employ
Bayesian posterior inference optimize their parameters (e.g., Markov Chain Monte Carlo (MCMC) and Variational Inference (VI)), which is usually non-trivial to derive and can be less flexible and efficient for big text corpora (Zhang et al. 2018). Recent developments
Autoencoding Variational Inference (AVI) (Kingma & Welling 2013 Rezende et al.) 2014) provide stronger inference tools for Bayesian models and have inspired several neural topic models (Zhang et al., 2018 Duan et al.) 2021a), resulting in improved efficiency and flexibility. However, applying AVI to neural topic models still has some limitations or concerns. First, the estimation of variational posterior always needs trade-off between accuracy and efficiency an asymptotically exact method (Salimans et al.) 2015). Besides, the latent distributions are required to be reparameterizable and KL divergence is expected to be analytical, both of which are hard to meet for topic models since they usually depend on Dirichlet distribution or the Gamma distribution (Blei et al., 2003) Zhou et al. 2015). Another concern comes from likelihood maximization, in which the inference of topic structure relies word

---

Under review as a conference paper at ICLR 2023
co-occurrence patterns within document. This has been recently found to give poor quality topics in case of little evidence of co-occurrences. such as corpus with a small number of documents or containing short context (Huynh et al.] 2020 Wang et al. 2022). This concern is even more acute in hierarchies learning where more topics and their correlations need to be inferred (Meng et al. (020). Several existing studies have targeted to incorporate meta knowledge to improve topic representation. The source of side information may come from various fields, including knowledge graph (Xie ct al.
2013 Duan al. 202Tb), pre-trained language model (Bianchi et al. 202T Meng et al. 2022) and
word embeddings (Wang et al. 2022 Duan et al. 2021a).
Another notable tendency developed recently is the conditional transport (CT) theory (Zheng & Zhou
202Ia). It provides an efficient tool to measure the distance between two probability distributions and has been employed in numerous machine learning problems, such as domain adaptation, generative model, and document representation (Zheng et al., [021]] Tanwisuth et al. [02T]] Wang et al. (022). The CT distance is defined by the bidirectional (forward and backward) transport cost between the source and target distributions, allowing the two distributions not to share the same support. Moreover, the CT distance can be unbiasedly approximated with the discrete empirical distributions, making amenable to stochastic gradient descent-basse optimization. |Wang al. (2022) first introduced CT into topic modeling by minimizing the transport cost between the word and topic space, resulting in better topic quality and document representation. The similar idea is shared with recent optimal transport-basee methods (Kusner et al., 2015: Huynh et al. 2020 Zhhao et al., (202T). However, they all focus on single-layer topic discovery, ignoring multi-level topic dependencies.
This paper goes beyond hierarchical Bayesian models for topic structure learning and aims to discover topic hierarchies based on the conditional transport between distributions. To formulate topic structure learning as a transport problem, we first provide a hierarchical, distributional view of topic modeling, where each layer of topic hierarchy learns set of topics presented as embedding vectors. Moreover, the to-be-learned topics share the same word embedding space and are organized in taxonomy where the upper-level topics are more general while the lower-level topies are more specific (Zhang et al. 2018). In detail, we view each document as an empirical distribution of word embeddings and consider that a document can also be presented by the topic embeddings (Wang ct al.] 2022) at each layer. Those hierarchical empirical distributions have different supports but share semantic consistency across topical levels. With this view, we propose to learn topic hierarchies with a Bidirectional Transport chain (BT-chain) where a document's 's topic distributions in two adjacent layers are learned by being pushed close to each other in terms of the CT loss. This results in a more flexible and efficient method than VAI-based NTMs, while keeping the interpretability of Bayesian models.
With different mechanism from previous hierarchical topic models, the proposed BT-chain is straightforward and novel approach for topic structure learning, which can be flexibly integrated with deep neural networks. To achieve an efficient and end-to-end training algorithm, an upward- downward optimizing strategy is developed carefully, which first warms up the empirical distributions by transporting the input via the bottom-to-tu ttttom-to-topppottt path, and then applies the backward layer- wise refinement by considering the bidirectional information stream from the Bayesian perspective. The main contributions in this paper are as follows: (1) We view the hierarchical topic modeling from a new perspective of multi-layer conditional transport, which facilities us to develop novel bidirectional transport chain for topic structure learning. (2) To effectively and efficiently implement the proposed method, we propose upward -dwwawardd training algorithm for BT-chain with proper amortizations and strategy. (3) We conduct extensive experiments on text corpora to show that our approach enjoys superior modeling accuracy and interpretability compared with the state-of-the- the-art hierarchical topic models. To extend the application of topic modeling, we also apply it on learning hierarchical visual topics from images, which shows interesting visualizations.
2 BACKGROUND
In this section, we recap the background of transport distance between two discrete distributions. Let us consider two discrete probability distributions and g E P(X) on space X & R Em_n", u162,, and q Σ/=1 uy0gg. where x; and yy are two points in the arbitrary same space X. E Σn and E E", the simplex of Rp" and Ihm denotes two probability values of the discrete mm
states satisfying Σn u; = I and プド Uj = 1. 8, refers to a point mass located at coordinate HR
2

---

Under review as a conference paper at ICLR 2023
To measure distance between such two discrete distributions, the optimal transport (OT) dis- tance (Villani 2009) between p and is formulated as an optimization problem: OT(p, g) = T1 u, T"1"
(., Zi, CijCij.
The minimum of the transport plan
TE T is taken over u v) with element tay, defined as the set of all possible joint probability measures и the whole space IR" IR"", with the marginals constraints. I m is the dimensional vector of ones. Gjj = c(xi, Myy 2 is the transport cost between the two points Ty and yj defined by an arbitrary cost function c(-).
More recently, the demand on efficient computation and bidirectional asymmetric transport promote the development of conditional transport (CT) Zheng Zhou, 202Ib), which can be applied to quantify the difference between discrete empirical distributions in various applications (Zheng et al. 2021 Tanwisuth al. 202T, Wang et 2022). Specifically, given the above source and target distributions p and g. the CT cost is defined with a bidirectional distribution ceaobbutoo-toodiiiee to-distribution transport, where a forward CT measures the transport cost from the source to the target and a backward CT reverses the transport direction. Therefore, the CT problem can be defined as:
Et 1es + Σ
T Jecss);
CT(p,g)
min TT
where E's in T acts as the transport probability (the navigator) from the source point T; to the target 7
point y,: Z.j = u1 En-= eye "aleregg hence T m Similarly, we have the reversed transport
eye dyliciyy))
probability: T Up Σg uue - (opoe)) and T1n v. The distance function dø( Vj)
use "alyer
parameterized with can be implemented by deep neural networks to measure the semantic similarity between two points, making CT amenable to stochastic gradient descent-basse optimization.
3 BT-CHAIN: BIDIRECTIONAL TRANSPORT CHAIN
We introduce the details of the proposed method, including a distributional view of multi-layer document representation, construction of BT-chain, and its upward-downwer training algorithm.
3.1 HIERARCHICAL, DISCRETE DISTRIBUTIONAL REPRESENTATIONS OF DOCUMENTS
Given a collection of corpora with J documents and V distinct tokens, conventional TMs usually represent the jth document as dimensional Bag-of- -fordd vector æ , E R.Y where Tju indicates the frequency of the v-th word in document (Blei et al., 2003 Dieng al. 2020). With æ, and the word embedding matrix E XV where H is the embedding dimension, we represent each document as an empirical distribution P; in the word embedding space:
P3 = Σ Ejuden with e, E R.
where ④; E ΣVin is the normalization of æz and e, is the embedding of the e-th word in the vocabulary, ie., the v-th column of E. Notably that Pú in Eq. [ not only contains the word co-oceurrence patterns but also considers word semantic information, which has been proven useful for high-quality topic learning (Dieng et al., 2020 Meng et al. 2022 but is often ignored by conventional TMs.
This paper aims to learn L layers of topics in topic hicrarchy, each of which contains K₁ topics for satisfying K(+1 < K₁ < Ki-1. It means that in the higher layers, there are fewer yet more abstract topics. Different from conventional TMs that assume topic the distribution over words, view topics as the continuous vectors that lie in the same semantic space of words. Thus, each layer in (1) Ki
BT-chain is associated with set of topic embeddings {ax Jk=1. 1,..,, L. Together with the topic proportion of j-th document
(²)) E Ki that denotes the topical weights over the K, topics, we derive the topical distributional representation of the j-th document in layer / as:
Ki
2 =
kj a
with
a t
E H
1= 1,,,..L
(2)
k=1
3

---

Under review as a conference paper at ICLR 2023
C
Forward CT
Backward CT
The directed graphical model of BT-chain. The adjacent layers connected via the bidirectional CT, and g of BT--chin links Q!--1), t Q(++1) via the spward-downwen training strategy, guaranteeing their semantic consistency. The solid dots and triangles denote word and topic embeddings in the emhedding space, and different color means different layers.
Figure 1:
where : 0 / 1 is the normalized topic proportions of document j I-th layer that ensures the simplex constraint. We follow previous hierarchical topie models (Duan al. (202aa] and employ an encoder network fw parameterized by to infer the multi-layer topic proportions: (0 = fxz)) fw implemented by several stacked fully connected layers and the details can be found in Appendix. B
To summarize, the proposed BT-chain views the j-th document series empirical distributions over the word and topic embeddings: (0, MEo With (0) E, g Φ₁ Ko = V, we have
2000 = P; P, which denotes the word-level observed data. Note that those empirical distributions are all defined by the word/topic weights together with the corresponding semantic vectors. They are formulated in similar form and share semantic consistency but with different topical supports.
3.2 LINK HH VIA BT-CHAIN
Ho
Given the above distributional view of documents, we develop BT-chain to learn hierarchical doc- ument distributions } Ji=1
where each layer in BT-chain is linked with its adjacent layers via the conditional transport and corresponds to learning of topic embeddings and topic proportions. Fig. shows the overview BT-chain. Specifically, in the topic hierarchy = L... 1, the current layer is connected with its lower layer / 1 and higher layer 1 + (the uppermost layer L is only connected to the lower layer L 1). Note that Q Q." Q capture the semanties of the same document, where Q focuses on more detailed semantics while Qf(+11) attends more general concepts. 0.00 thus acts here as an intermediate node that integrates information from both directions and learns semantically smooth topics. To meet such properties, it is natural to push Q Q/-1) and Q!(+1)) close as possible. poses question on how to define the closeness between two discrete distributions with different supports. Although recent studies have provided several transport-based alternatives to measure such distance (Cuturi 2013 Yurochkin al. 2019 Hu et al.)
2021 Zhao et al. 202T). most of them focus on shallow transport problems and ignore the deep case. To this end, our proposed BT-chain derives a layer-wise distance with the recent CT technique and expresses the transport loss in [-th layer as:
((0) (O of+") = =CT (o,
+ +CT (0 (o ofth)..
(3)
((1) links the adjacent three layers via the two CT costs guarantee the current layer Q/"² semantically close of and Q (1+1)) Thus for hierarchy with T. layers, have (C()) each of which strengthens the learning of its local g0)) by considering the neighborhood information and they together ensure the semantic consistency of the transport chain. Moreover, we minimize the above loss (!) K;
in terms of O) which by definition consists of 600) and {a 1k=1- The detailed parameterizations and implementations are shown in Section of the Appendix and our designed training strategy for the transport chain is described in Section
4

---

Under review as a conference paper at ICLR 2023
Algorithm Training algorithm for our proposed BTT-chiin
Input:
documents, pre-trained word embeddiiggs E, topic list [Kiffn. hyperparameter B. Initialize: topic embeddings {ax (1) K1.d km1,l=1. of the inference network.
for iter 1.2.3,... do
Sample batch of documents and get (2 15=1 with Eq.
Get ) )- j=1_1=1 by the encoder network f.
Upward warming-up
for / 1,,,, 1,2,..., do
Fix (1-1) and compute L = Σラー CT (of-1),09))
Update {ax ]k=1 and w with stochastic gradients of L end for
Downward refining
for ! L 1, ... do
Fix ((t+1)) and (
Compute L ش== CT (0h-")
CT (0
Update {a { [k=1 and w with stochastic gradients of Ldown
end for end
As suggested in the previous works ZZao al. 2021). add regularization term based on the cross entropy for the CT loss in Eq. 3
( = -T
1 0 - _ log ( (2(++10/(+t)
(4)
where 4 (((11)) Softmax y 0 () (1()) RK each row of which is a (1) 8 is the trade-off hyperparameter that balances the weight of the cross entropy regularization. $ acts as the
(I+1)
"decoder" that decodes 0 to layer and the cross entropy measures how close the decoded one to its "target" 0²" In the training, we replace the CT distances in Eq. (3) with the regularized ones.
3.3
UPWARD-DOWNWARE TRAINING ALGORITHM FOR BT-CHAIN
Given the training documents and pre- ree-trained word embeddings E, aim to learn topic embeddings KILL
{ax 1=1,111 and the encoder network that infers the topic proportions { At layer the loss Eq. (3) consists of two CT distances that connect layer / with the lower and higher layers. Simultaneously minimizing such two CT distances can be difficult, we propose a layer-wise upward-downward training algorithm, which consists two steps in each training iteration.
Upward Warming-up In this step, we start with the learning of Q.") given the data 2000 in word embedding space, by minimizing CE (o))) ( i.e., the first CT distance in Eq. (3). After 0.0))
learned, we it and then learn g²(²) by minimizing CT (0))) -Q(2)) this way, we transport
the information from the observed word layer to the uppermost topic layer Q.(7) step by step.
Downward refining In the upward training step, the lower layer / - 1 is fixed and treated the data for its higher layer I, i.e., the message flows in the bottom-up direction. This does not consider how the higher layer / affects its lower layer / - 1. Therefore, after the upward step, we further introduce learn Q,4-1), we fix its two adjacent layers Q)2) and Q((-2) and minimize (()), i.e., both terms in Eq. (3). Similarly, we can update the distributions until we reach layer 1. this way, the message both layers I and
the downward step, where we start with the second uppermost layer
I 1 will flow to layer I. The training algorithm is outlined in Algorithm
3.4 DISCUSSIONS
Topic hierarchies learned by BT-chain BT-chain aims to leam topic hierarchy where the topics in the higher layers are more general and abstract than those in the lower layers. How BT-chain achieves this can be interpreted from information compression abstraction perspective. Specifically, in
5

---

Under review as a conference paper at ICLR 2023
layer O, the data 2,00 lies in a Ko-dimensional word embedding space, which is a sparse representation of the semantics of document j. Q lies in Ki-dimensional topic embedding space, where Ki < Ko- By pushing Q."h) Q500 as close as possible in terms of the CT cost, BT-chain forces 2 to compress abstract the information in Q.0) with a denser representation. Similar things happen in the higher layers, as K₁ > K2 > > Kr. Moreover, as all the topics are presented as embeddings, the topic correlations between the topics of two layers can be simply obtained by
the distances between the topic embeddings, i.e., T0') in Eq. (4). Therefore, the topic correlations in BT--hain can be interpreted in the same way as conventional hierarchical TMs (Blei et al. 2010
Zhou et al. 2016).
Bayesian Flavor of BT-chain Many deep topic models are implemented based on hierarchical Bayesian probabilistic models (Blei et al. 2010; Paisley et al.) 2014 Gan al., 2015 Henao et al.) 2013 Zhou et al. 2016 Zhao et al.) 2018) 1). where the topics in the higher layers serve as the priors
of the topies the lower layers. In these Bayesian models, according to Bayes' theorem, a topic's posterior distribution consists of two terms: the data distribution (i.e., the word information in document) and prior distribution (i.e., the topics in the higher layer). Although BT-chain is not a Bayesian model, it is interesting to interpret our method with Bayesian flavor. For example, the value of Q.2") is learned according to 2,00 and prior, respectively. Instead of learning the model with Bayes' theorem, we optimize B BTT-haii by minimizing the CT costs between the distributions, avoiding non-trivial Bayesian posterior inference.
where 2,00 and Q,(²) can be viewed as data and
RELATED WORK
Topic structure learning There a surge of research interest in capturing the correlations among topics and generating topic structures. For example, there are many models based hierarchical Bayesian prior such as the Dirichlet process (DP) and Chinese Restaurant Process (CRP), including hLDA (Grilfiths et al. 2003), nCRP (Blei al. 2010) and nHDP (Paisley et al. 2014). Li & McCallum (206)) propose the Pachinko Allocation Model (PAM) to model the co-occurrences of topics via a directed acyclic graph. hPAM (Mimno et al. 2007) is built on PA.M and represents the topic hierarchical structure through the Dirichlet-multiomiia parameters of the internal node distributions. More recently, various hierarchical extensions of Poisson factor analysis (PFA) (Zhou
et al. 2012) have been proposed, including DPFA (Gan et al. 2015). DPFM (Henao et al., 2015).
GBN iZhou et al. (0166) and DirBN (Zhao et al. 2018). Zhang al. (0118) develop Weibull hybrid autoencoding inference (WHAI) for GBN and Duan et al. (2021a) introduce word embedding into GBN and design the SawETM Connection (SC) to explore the relationship between topics. Although both WHAI and SawETM are the most related works to ours, which are Bayesian generative models and learned by maximizing the evidence lower bound (ELBO), our proposed model views deep topic modeling as cross-layer transport problem.
Topic models based on transport Using transport between distributions is recent trend in topic modeling. Xu et al. (2018) propose distilled Wasserstein learning (DWL) where the distance between topics is achieved by the OT between their word distributions built on the embedding- based underlying distance. The OT based LDA (OTLDA) (Huynh et al. 2020) shares similar idea but aims minimize the OT distance between documents and topics in the vocabulary space. Yurochkin et al.) (2019) view the documents as the distributions over topics, those topics themselves are modeled as the distribution over words and introduce the hierarchical OT as the meta-distance between documents. However, it not a topic model but method using topic models compute document distances. Based on topic embeddings, Zhao et al. (202T) propose the Neural Sinkhorn Topic Model (NSTM) that optimizes the OT distance between the normalized BoW vector and the topic proportions, where the cost matrix is calculated by the word and topic embeddings. In Wang et al. (2022). We'Te views each document as a set of mixtures of word embeddings and a set of mixtures of topic embeddings and employs the conditional transport (CT) cost to quantify the difference between those two sets. Compared with the above shallow topic models, our proposed model aims to learn hicrarchical document representations and topic structures.
EXPERIMENTS
We in this section conduct comprehensive experiments on various datasets and compare the proposed BT-chain with different baseline methods to illustrate its superior performance.

---

Under review as a conference paper at ICLR 2023
20NG(TC)
WB(TC)
DP(TC)
RCV2(TC)
0.12
0.15
0.15
0.1
0.1
0.1
0.06
0.04
0.05
0.00
0.05
0.02
3
20NG(TD)
WB(TD)
DP(TD)
RCV2(TD)
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
2
TTcchain-UU
Figure 2: Topic Coherence (TC) and Topic Diversity (TD) of hicrarchical topic models the four datascts.
Table Comparison of K-Means clustering purity (km-Purityy and NMI (km-NMI) for various methods. The number in brackets after the method indicates the total number of topic layers L. The best and second best scores of cach datasct are highlighted in boldface and with an underline, respectively.
Method
Am-Purity( (%)
km-NMI %)
WS
RCV2
DP
20NG(6)
WS
RCV2
DP
20NG(6)
LDA-Gibbs
46.4110.
52.410.4
60.8 =0.5
59.210066
25.1
38.2:00.5
$4.7 +0.3
32.4 #0.4
DVAE
26.6:11.55
52.61:12
61.
64.6
10.8
31.31009
50.8 #0.6
29.8
ETM
32.942.3
50.210.6
63.1 115
62.6 +2.
12.3122.3
30.311.0
53.2
29.3 11.5
NSTM
42.14:0.6
53.81110
20.2 =0.7
62.6+11.
17.4 4.066
36.810.3
6.63-10.11
31.1 #1.
WeTe
60.81002
62.910.5
77.1 110
68.5 +0.2
34.9110.4
42.8:1033
63.7.10.4
36.3 10.2
WHAK()
50.8 +0.2
59.0 +0.1
65.1 10.11
60.4 +0.3
25.8 +0022
40.0 +.0.1
$3.0 +0.3
29.6 10.1
SawETM(1)
34.4
62.7 20.4
-02
70.1 +0.2
10.7 10.6
1003
$4.0
38.4 10.2
BT-chain( (OOrs))
61.2 +02
62.4 +0.3
77.5 +0.3
70.4 +0.1
35.4 +0.4
44.3 +0.2
66.2 +.2
39.7 +0.1
WHAK()
52.1 10.4
60.5 10.2
66.9 10.1
60.8 10.1
29.9
40.0 10022
55.2 10.3
30.2 10.1
SawFIM(3)
43.7 +0.8
64,2 +0.5
70.1 +0.1
72.1 +0.3
21.9 +0.7
45.2 +0.4
58.9 +0.2
42.7 +0.1
BT-chain(3)(Ourr)
62.7-10.3
63.7 40.3
78.0 0.2
73.3 0.1
35.9 0.4
44.6 4.033
66.6 1.11
43.9 10.1
Datasets We conduct the experiments on four widely used benchmark corpus: 20 News Group (20NG), Web Snippets (WS) (Phan et al. 22008) DBpedia (DP) (Lehmann et al. 2015) and Reuters Corpus Volume 2 (RCV2) (Lewis et al. 2004). These datasets have very different characteristics in terms of document length, the number of documents, and vocabulary size, where DP and RCV2 large-scale datasets, WS and DP consist of short documents. We pre-process those datasets following WeTe (Wang al.) 2022), e.g., we tokenize and clean text by excluding standard stop words and low-frequency words. The statistics of the datasets are summarized at Table. C. of Appendix.
Baselines and settings We compare BT-chain against conventional and advanced topic models, including (1) Single-layer baselines: Collapsed Gibbs Sampling LDA (L.DA-Gibbs) as described in (Griffiths & Steyvers] 20044); Neural topic models, such as Dirichlet VAE (DVAE) (Burkhardt Kramer (019) and embedded topic model (ETM) (Dieng et al.) 2020). the former is VAE based topic model and ETM is the first NTM that introduces word embeddings; NTMs with transport, Neural Sinkhorn Topic model (NSTM) (Zhao et al.] (021) and (WeTee (Wang et al.) 2022), both of them use transport distance as the loss function: (2) Hierarchical topic models: WHAI Zhhang et al. 2018) and STwETM (Duan et al. 202 they are hierarchical generative model based on Gamma belief network and are compared as our multi- -uyti-layer baselines. Besides the above baselines, also propose variant of BT-chain that only uses the upward warming-up up path to train the topic embeddings and the encoder. which we name BT-chain-U U. For all baselines, we use the default parameters given with the source code the best settings reported in their paper. For the models
7

---

Under review as a conference paper at ICLR 2023
Table 2: Topic coherence (TC) and topic diversity (TD) results of single-layer methods on four datasets.
Corpus
metries
LDA-Gibbs
DVAE
ETM
NSTM
WeTe
BT-chain
TC
0.058
0.013
0.019
0.103
0.101
0.103
20NG(6)
TD
0.631
0.661
0.551
0.647
0.591
0.664
TC
0.097
-0.032
0.067
0.118
0.143
0.152
RCV2
TD
0.641
0.730
0,502
0.639
0.760
0.743
TC
0.074
0.065
0.053
0.104
0.108
0.125
DP
TD
0.684
0.645
0.583
0.715
0.763
0.790
TC
0.071
0.078
0.003
0.122
0.123
0.135
WS
TD
0.760
0.540
0.585
0.940
0.764
0.904
that work with word embeddings, including ETM, NSTM, WeTe, and our BT-chain, we use the pre-trained Glo vectors for a fair comparison. We summarize those baselines at Table. C. 2
Evaluation metries Notably, our work aims mine high-quality topic structures, where evaluation metrics about topics are our main focus. Though perplexity is a common-used metric for generative topic model. recent studies suggest that might not be an appropriate measure of the topic quality (Chang et al. (209).. Besides, CT based topic models (e.g., NSTM and WeTe) are learned by minimizing the transport cost instead of maximizing the log-likelihood, which is important to achieve better perplexity results. Therefore, we put more attention on following metrics. We use Topic Coherence (TC) and Topic Diversity (TD) to evaluate the learned topics from both the interpretability and diversity aspects. In detail, TC is the average Normalized Pointwise Mutual Information (NPMI) over the top 10 words each topic which is highly correlated to human judgment. TD is calculated by the average percentage of unique words in the top 25 words of all topics. The higher the better for both TC and TD. Besides the topic quality, we also report Purity and Normalized Mutual Information (NMI) (Schütze et al. 2008) clustering tasks to measure the performance of document representation. We first train the model on the training datasets and infer the topic proportions on the testing documents. Given the collection of topic proportions, we apply KMeans predict the document label. The clustering number is set as 20 for 20NG, WS, and DP, while 52 for the RCV2 dataset. For 20NG dataset, we follow WeTe and use its six super categories at the first level as the ground truth, and denote it as 20NG(6).
Implementation details of BT-chain As discussed above, the proposed model takes the bag of features 0,00 Σ Ku Jom
0 8 (0)0) as its input. For corpus, Ox (0) is the word embeddings in the
000) is the normalized TF-IDF vector. We set as and the topic numbers each layer for HTMs in the baselines and BT-chain K = [100, 64, 32] for a fair comparison. We set K 100 for all the single-layer models. We use the Adam optimizer with learning rate 0.001, batch size 500. All experiments are performed on the same machine and our model is implemented in PyTorch
vocabulary,
Quantitative results For all the methods, we run the algorithm five times with different random seeds, and report the mean and standard deviation on the clustering task, while choosing the best results for topic quality. We fix the hyperparameter B = 50 for all the datasets and also report the effect of various settings on the document clustering in the Appendix. C.4 Table reeports the km- am-Purity and km-NMI on four corpora. The first group is the single-layer topic models, including LDA, NTMs, and transport based models, and the last two groups are hierarchical topic models (HTMs) with a single layer and three layers respectively. We find that i) In general, our proposed BT-chain outperforms the others in most cases, especially on the short WS and DP corpus. We attribute this to the empirical distribution representation with word/topic embeddings. ii) The deep models achieve better performance on both Purity and NMI than the shallow models. This is mainly because their hierarchical document representations can provide more comprehensive information which one of motivations. üii) Our single-layes BT-chain achieves better scores than WeTe, which shows the efficiency of introducing the word weights at layer 0, c.g., the normalized TF-IDF vector, rather than treating them according to the word frequency in We'e.
To fully compare the qualities of learned topics. Table 2 and Figure (where x-axis denotes the topic layers.) report the topic coherence (TC) and topic diversity (TD) for single-layer models and deep models respectively. Overall, we observe that the topics learned from our BT-chain has a better TC and TD, especially for the short corpora. This meets with the observation on the clustering task and
8

---

Under review as a conference paper at ICLR 2023
9-8 tspical bu: eds. fieees, zag. gii, zet, taie
laye: medical is, pleskian soexe, coappter, atkile
144h lipic leyet
45-th lopk loyer geo ds ward ocliala, hallis clanca chankall
52-t kojer medica,, pobeats, aeet aborditee, phizoopher
tapa liyes saftaae rosezzs, dezaaa, beoks, medied. applicaton systen
len, infurmtte, ang, gin. onde, ab, corputrr
14th kyu sanace. Ihrard, tadrodogs ingjazring
17-th toplc hatentry: gaznay orginatee
65 th tople balt, fiee artrggeee apendiies bppaleis sitmnl ridis
23 rpkc eyt, doci beb. daganed pukert.dine
56 th spie spirion, Say, real iniss lelens love. chiidaa
2-B tspi windors, file, sufteae, iisk poggrn diive sysken
sugery
Figure 3: Topic hicrarchies Iearned from 20NG (left) and minilmageNNet (right). The links between two adjacent layers are obtained according to the semantic similarities of topics (e.g. + in Eq
again proves the efficiency of the introduced word/topic semantics. Besides, our three-layer BT-chain outperforms all the other HTMs in most cases, which benefits from the new view of topic hierarchies discovery from the transport theory. The proposed BTT-hain guarantees semantic consistency by minimizing the total transport cost via (()) 1, L 1, resulting in high-quality topic structures. Finally, our BT-chain consistently achieves higher scores than BT-chain- on all datasets high layers. Note that BT BT--hain-UU only uses the upward path to transport messages from word space to higher-level topic space. This demonstrates the efficiency of our downward training strategy.
Qualitative results To visualize the learned topic structures, we show the topic hierarchies on 20NG in Fig. (left), where it can be observed that topics in higher layers are mixtures of semantically close topics in the lower levels. Recalling that BT-chain receives set of feature embeddings as its input, this broadens the scope of application of BTT-chain beyond text corpora. Here we conduct experiments on minilmageNet and visualize the learned concept hierarchy in Fig. (right). MinilmageNeet contains a total number of 100 classes with 600 images in each class, which are extracted from the ImageNet dataset (Russakovsky et al., 2015). To obtain the empirical distribution for image data, adopt Transformer (Rigotti et al. (2022) to obtain the bag of features for images, where the j-th image is first divided evenly into N patches 2j = E 1
N beny. and each patch aligns to M concepts via the cross-attention e,, = >M anmoe... where en, E Rd is the patch embeddings, and Omm is the attention weights, Ce R M is the concept embedding matrix. We average all patches and thus the final 2)
(0) for image can be expressed as: 4/00 > ( En anm )6 Once trained on image data, we visualize the images assigned to the learned visual topics righht) where we can observe interesting semantic relations between visual concepts. For example, the bottom layer contains the clear and concrete concepts, e.g., the regular textures, animal eyes, small objects in pure background, and on. The second layer contains more complex semanties: the facial part of animal, and the small objects in textured background. It interesting to see that the topic of warships in the second layer is closely related to the topic of occans and the topic of missiles in the bottom layer. Similar observation can be found in the topic hierarchy of documents (Fig. leftt, where we list the most related words of each topic. We report qualitative results and time complexity analysis in Appendix C.3 and D
CONCLUSION
In this paper, we present a bidirectional transport chain (BT-chain) for hierarchical representation. BT-chain views each document as a series of empirical distributions over word/topic embeddings, each of which consists of topic proportions and the corresponding embedding vectors. With the fact that those multi-layer representations share semantic consistency of the same document, we propose an effective upward-downward layer-wise training algorithm to learn topic hierarchies and document representations based on the conditional transport theory. Our framework can be viewed as a hierarchical extension to the recently developed topic models based on transport. Extensive experi- ments text corpora show that our approach enjoys superior modeling accuracy and interpretability. Moreover, we also have conducted experiments on learning hierarchical visual topics from images, which demonstrate the adaptability and flexibility of our method.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Federico Bianchi, Silvia Terragni, and Dirk Hovy. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. Association for Computational Linguistics, 2021.
David Blei. Probabilistic topic models. Communications of the ACM, 55(4):77-84, 2012.
David M Blei, Andrew Y Ng. and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):933-102) 2003.
David Blei, Thomas Griffiths, and Michael I Jordan. The nested chinese restaurant process and
Bayesian nonparametric inference of topic hierarchies. Journal of the ACM (JACM), 57(2):1-30, 2010.
Sophie Burkhardt and Stefan Kramer. Decoupling sparsity and smoothness in the dirichlet variational autoencoder topic model. Mach. Learn. Res., 20(131):1 2019.
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber, and David Blei. Reading tea leaves: How humans interpret topic models. In Advances in neural information processing systems, pp. 288-2980 2009.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NIPS, pp. 2390, 2013.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling embedding spaces. Transac- 2020.
fions of the Association for Computational Linguisties,
Zhibin Duan, Dongsheng Wang, Bo Chen, Chaojie Wang, Wenchao Chen, Yewen Li, Jie Ren, and Mingyuan Zhou. Sawtooth factorial topic embeddings guided gamma belief network. In International Conference on Machine Learning, pp. 2903-2913. PMLR, 2021a.
Zhibin Duan, Yi Xu, Bo Chen, Chaojie Wang, Mingyuan Zhou, et al. Topicnet: Semantic graph- guided topic discovery. Advances in Neural Information Processing Systems, 34:547-559, 2021b.
Zhe Gan, Changyou Chen, Ricardo Henao, David Carlson, and Lawrence Carin. Scalable deep Poisson factor analysis for topic modeling. In International Conference on Machine Learning, pp. 1823-1832. PMLR, 2015.
Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. Hierarchical topic models and the nested chinese restaurant process. Advances in neural information processing systems, 16, 2003.
Thomas L Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National academy of Sciences, 101(suppl 1):5228-5222 2004.
Justin Grimmer. A bayesian hierarchical topic model for political texts: Measuring expressed agendas in senate press releases. Political Analysis, 18(1): 2010.
Dandan Guo, Bo Chen, Ruiying Lu, and Mingyuan Zhou. Recurrent hierarchical topic-guided mn for language generation. In International conference machine learning, pp. 3810-3821. PMLR, 2020.
Ricardo Henao, Zhe Gan, James Lu, and Lawrence Carin. Deep Poisson factor modeling. Advances in Neural Information Processing Systems, 28, 2015.
Matthew Hoffman, Francis Bach, and David Blei. Online learning for latent dirichlet allocation. advances in neural information processing systems, 23, 2010.
Yuqing Hu, Vincent Gripon, and Stéphane Pateux. Leveraging the feature distribution in transfer- based few-shot learning. In Artificial Neural Networks and Machine Learning ICANN 2021, 2021.
Viet Huynh, He Zhao, and Dinh Phung. Otlda: A geometry-awarc optimal transport approach for topic modeling. Advances in Neural Information Processing Systems, 13:1817--18882. 2020.
10

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. ICLR, 2013.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International conference on machine learning, pp. 957-966. PMLR, 2015.
Dongha Lee, Jiaming Shen, Seongku Kang, Susik Yoon, Jiawei Han, and Hwanjo Yu. Taxocom:
Topic taxonomy completion with hierarchical discovery of novel topic clusters. WWW '22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 29, 2022. ACM, 2022.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sehastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia-a large- scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195, 2015.
D. D. Lewis, Y. Yang, T. G. Rose, and Li. Rev1: new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361-397, 2004.
Wei Li and Andrew McCallum. Pachinko allocation: Dag-structured mixture models of topic correlations. In Proceedings of the 23rd international conference on Machine learning, pp. 577-584, 2006.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Yu Zhang, Chao Zhang, and Jiawei Han. Hierarchical topic mining via joint spherical tree and text embedding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 1908-1917. 2020.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Yu Zhang, and Jiawei Han. Topic discovery via latent space clustering of pretrained language model representations. pp. 3143-3152. ACM, 2022.
David Mimno, Wei Li, and Andrew McCallum. Mixtures of hierarchical topics with pachinko allocation. In Proceedings of the 24th international conference on Machine learning, pp. 633-640, 2007.
John Paisley, Chong Wang, David M Blei, and Michael Jordan. Nested hierarchical dirichlet processes. IEEE transactions on pattern analysis and machine intelligence, 37(2):256 2700 2014.
Xuan Hieu Phan, Le-Minh Nguyen, and Susumu Horiguchi. Learning to classify short and sparse text & web with hidden topics from large-scale data collections. In Proceedings of the 17th international conference World Wide Web, pp. 91-100, 2008
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, pp. 1278-1286, 2014.
Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind, and Paolo Scotton. Attention- based interpretability with concept transformers. International Conference on Learning Repre- sentations, 2022. URL https:://// httppss//peeeeeeiii....... .nw. wttp://wwwwee//onniieeeee
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Ini. J. Comput. Vis., 115())211-2555 2015.
Tim Salimans, Diederik Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. In International conference on machine learning, pp. PMLR, 2015.
Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction to information retrieval, volume 39. Cambridge University Press Cambridge, 2008
Korawat Tanwisuth, Xinjie Fan, Huangjie Zheng, Shujian Zhang, Hao Zhang. Bo Chen, and Mingyuan Zhou. A prototype-oriented framework for unsupervised domain adaptation. In NeurIPS 2021: Neural Information Processing Systems, Dec. 2021. (the first three authors contributed equally).
Yee Whye Tch, Michael I Jordan, Matthew Beal, and David M Blei. Hicrarchical dirichlet processes. Journal of the american statistical association, 101(476): 1566--1115666588 2006.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.

---

Under review as a conference paper at ICLR 2023
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng, Korawat Tanwisuth, Bo Chen, and Mingyuan Zhou, Representing mixtures of word embeddings with mixtures of topic embeddings. International Conference on Learning Representations, 2022. URL https
In ICLR 2022: //openreview.oo nww. httes/ooommiidd--ee -tt.ee/ewe...e//eerei-i------- TYM IFU
Pengtao Xie, Diyi Yang, and Eric Xing. Incorporating word correlation knowledge into topic modeling. In Proceedings of the 2015 conference of the north American chapter of the association for computational linguistics: human language technologies, pp. 725-734,001
Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. Distilled wasserstein learning for word embedding and topic modeling. Advances in Neural Information Processing Systems, 31, 2018.
Mikhail Yurochkin, Sebastian Claici, Edward Chien, Farzaneh Mirzazadeh, and Justin M Solomon. Hierarchical optimal transport for document representation. Advances in Neural Information Processing Systems, 32, 2019.
Hao Zhang, Bo Chen, Long Tian, Zhengjue Wang, and Mingyuan Zhou. Variational hetero-encoder randomized gans for joint image-text modeling. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAI: weibull hybrid autoencoding inference for deep topic modeling. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httpt: /attps::/openneeveee.. wtwp://wwweett/toommdddddd---
RW
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. arXiv preprint arXiv:1509.01620 2015.
He Zhao, Lan Du, Wray Buntine, and Mingyuan Zhou. Dirichlet belief networks for topic structure learning. Advances in neural information processing systems, 31, 2018.
He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray L. Buntine, Neural topic model via optimal transport. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. httpttttt///setitee [d=0os98K9Lv
net for Forum?
Huangjie Zheng and Mingyuan Zhou. Act: Asymptotic conditional transport. Advances in Neural Information Processing Systems, 2021a.
Huangjie Zheng and Mingyuan Zhou, Exploiting chain rule and Bayes' theorem to compare proba- bility distributions. Advances in Neural Information Processing Systems, 34, 2021b.
Huangjie Zheng, Xu Chen, Jiangchao Yao, Hongxia Yang, Chunyuan Li, Ya Zhang, Hao Zhang, Ivor Tsang, Jingren Zhou, and Mingyuan Zhou Contrastive conditional transport for representation learning, arXiv preprint arXiv:2105.03746 2021.
Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. Beta-negative binomial process and poisson factor analysis. In Artificial Intelligence and Statistics, pp. 1462-1474 PMLR, 2012.
Mingyuan Zhou, Yulai Cong, and Bo Chen. The Poisson gamma belief network. Advances in Neural Information Processing Systems, 28, 2015.
Mingyuan Zhou, Yulai Cong, and Bo Chen. Augmentable gamma belief networks. The Journal of Machine Learning Research, 17(1):66565550 5699, 2016.
12

---

Under review as a conference paper at ICLR 2023
A DETAILED DERIVATION OF CT IN BT-CHAIN
C
( o ) measures the transport cost of two discrete empirical distributions
(heng & Zhou
2021b), which consists of a forward CT that constructs navigator to transport p) to Pg pl++) and backward CT that reverse the transport direction:
CT 2 0 2 = cी oft1i) + +CT (044",091
Specifically, the forward CT tries to minimize the expected transport cost from a set of Ki embeddings to set of Ki+1 embeddings:
(1+1) 0(+1)s s (ak O
сt E t -a
(akaaa a az u+1) (+)) (attt Le (al
(aaaaa.att)))) a.tti)]. (al+1) 1
((+)))
(1)
Σ LK= R' s(a.) a9 (t1)
Keg 0(+1)
where we specify transport cost c(-) of two embedding vectors with the inner product: a caak (1) og ) = exp(-ar (1)"" akk ((+1), e.g., the closer the two vectors are, the smaller the point-to- point transport cost. The navigator TT-) is the conditional probability of given embedding aj being transported to embedding oi ((+1), which is determined by both the topic weights 0 and the (1)
(I)
similarity score s-) of the source and target topics. Therefore, it would be easier to transport ag to
O' (I+1) if ox (!+1) describes more popular topic that semantically closer to ogg in the embedding
(1)
((+t)))=exp)) (1) 0 (1+1)) = exxp(a O* (++)) to reduce space. We here also use the inner product to define s(a)k) the computational cost, although other possible choices.
(t)"
Similarly, we can derive the backward CT by exchanging the source set and the target set:
0 ax .. (1)
(1+1)
CT- E ((+1) ( a (a( a (a
(1+1) aaaa (aff ((())((tt)))
Σx
g (1+1) ;
My ( .00
B INFERENCE NETWORK FOR TOPIC PROPORTIONS
To guarantee the sparse property of the topic proportion 0 and as suggested in Wang al. 2022
we leverage a Hierarchical Weibull Reparameterization Encoder (HWRE) infer 0 from the information of the previous layers:
000) Weibull (k 2 (9) 2 SoftPlus (D(h))), =
- (n (n)-)))))
(5)
where h.00) æy and g implemented with neural network, Weibull(k; 1) is the Weibull distribution, which is reparameterizable (Zhang et al., 2018); Drawing s Weibull(ks, equivalent maping. 8 (o-(og(( e))2/k Uniform(0,, 1). The SoftPlus applies log(1 + exp(-) nonlincarity to ensure the positive Weibull shape and scale parameters.
C DATASETS AND FURTHER EXPERIMENTS
C.1 DATASETS
Our experiments are conducted on four widely benchmark text datasets including 20 News Group (20NG), Web Snippets (WS) (Phan et al. 2008). DBpedia (DP) (Lehmann et al., 2015), Reuters Corpus Volume 2 (RCV2) (Lewis et al. 2004) and one additional image dataset minilmageNet (Vinyals et al. 2016). For the text datasets, WS and DP are short documents, RCV2 and DP are large scale datasets. Since RCV2 owns multiple labels for one document, follow the previous work and remove documents containing multiple labels in the second level, resulting in 1.M documents. For the image dataset, minilmageNet is subset of ImageNet (Russakovsky al. 2015) dataset.
13

---

Under review as a conference paper at ICLR 2023
20N 20 Newsgroups consists of newsgroups post including 18,846 articles. We remove stop words and words with document frequency less than 100 times. We also ignore documents that contain only word from the corpus. We follow We'e of (Wang et al.) 2022) and use the 6 super-categories as 20NG's ground truth and denote it as 20NG(6) in the clustering task.
wS: Web Snippets is a short corpus that contains 12,237 web search snippets with categories. There are 10,052 words in the vocabulary and the average length of snippet is 15.
DF DBpedia extracted from Wikipedia pages. We follow the pre-pro pre-processing process in Zhang et al. (2013), where the fields we used for this dataset contain title and abstract of each Wikipedia article.
RCV1 The original Reuters Corpus Volume 2 dataset consists of 804,414 documents. We here left documents that only contains single label at the second topic level, resulting in 0.15M documents totally, whose vocabulary size is 7282 and average length is 85.
minilmageNet + minilmageNNt a subset randomly sampled from ImageNet. In total, there are 100 classes with 600 samples of 84 x 84 color images per class. These 100 classes are divided into 64, and 20 classes respectively for training, validation and testing.
summary of text corpora statistics is shown in Table
Table C.
Statistics of the datasets
Number of docs
Vocabulary size(V)
average length
Number of labels
20NG
18,864
22,636
108
DP
449,665
9,835
22
14
WS
12,337
10,052
15
RCV2
150,737
7,282
85
52
C.2
BASELINES
We summary the baseline methods at Table. C.
Model
VAE-based
HTM
transport-based
Word embedding
2004) LDA Griffiths & Steyvers,
DVAE
Burrhardt & Kramer
2019
ETM
Dieng et al.
2020)
NSTM (Zhao al.
2021
We'Te
Wang et al.
2022
WHAI
Zhang et
2018
SawETM
(Duan et al.
2021a
BT-chain
Table C. 2: Summaries of baseline models. HTM denotes the hierarchical topic model.
C.3 MORE VISUALIZATIONS ON TEXT AND IMAGES
Similar to the main paper, we provide more visualizations of the learned topic hierarchies in Fig. and Fig.
http://qwoneccom// com/ http./////nooo////e/errrruuuu
https://enn wikipsdhia.rrg/wii//iMiiii Page
https:/meeeinsttgollaaaaaarntttt
14

---

Under review as a conference paper at ICLR 2023
dhiige kom snnp kanna Ohioke Entser
Jeebo ca add
Madb bear
Wub
31 andid low rec Nec. del relcsigo
th at prddac intart un longe
fla lode exrel sis Lebvr toxm
Lt Biigle urtp Ace sagg
13- liz rives
674b codls
Uike 197 reacies
pablid, gate peacal
peotlet adl deniless poord.
baseaanc bbilyyoodd
dac -tal
wels grvv
portdaas pam 19.n
artatic
Figure C. Topic hierarchies learned from minilmageNNe (left) and DP (right).
wwll gload hastome, Jeurtcan ckita
The fimdt prodeot pablid, paroan,
IE Syye
Skibiap bove
38 ts seic toge oompiny faaadd mnesene
16 tei loyer mina lic Nic.nnew a.mon cen
Branca ntc. hadget colars
gooth pelcs tude
14:11 bettics
31 spp
1 tapt werld
24hh [eaxel. cndtt
35 licc ong zeeots lanv kenikks
30 mera, the, meres feans Diary. daasmday.
Londser
Jasm 20
made tased
narme
feme:
wepi bater
Figure C. 2: Topic hierarchies learned from minilmageNet (left) and WS (right).
20N0081/Prity)
60
70
08
54
WS(NMM)
ROV2/NM)
DP/NMO
20NO(0)/MM)
35
64
30
22
53
32
100
20
60
100
23
100
20
100
Figure C. 3: Clustering results with various |3.
C.4 HYPERPARAMETER SENSITIVITY
We fixed the hyperparameter 3 = 50.0 in all previous experiments for fair comparison. Here we report the document clustering results (Purity and NMI) with different hyperparameter settings on the four corpus in Fig. C. 3 Note that 3 controls the weight of the cross entropy in Eq.6 in the mainscript. The regularized loss degenerates the CT loss when = 0. From Fig. C. we find that the regularization term helps the document representations, as there is a significant improvement from = = 10; Besides, one can get better results than those reported in our experiments by fine-tuning 3 for each dataset.
D TIME COMPLEXITY ANALYSIS
The core computational module in BT-chain is the regularized CT loss between two adjacent layers e (Q))) Q ((())). mainly contains the cost matrix, which has time complexity of O ((KKK)11) (Ki is the number of topics at layer 1), the bidirectional transport plan æ, which has time complexity of O(6K, We note that both of them have linear complexity over the product of the number topics at adjacent layers. For layer 0, we compute words within the target document rather than
15

---

Under review as a conference paper at ICLR 2023
all words in vocabulary, e.g. Ku N, Ny is the number of words in document N2 < V. Thus the proposed BTT-chain has an acceptable time complexity.
E METRICS
In our experiment, report Topic Coherence (TC) and Topic Diversity (TD) to evaluate the learned topics from both the interpretability and diversity aspects. Given a reference corpus, TC measures the semantic relevance in the most significant words (top 10 words in our case) of topic, which is computed by the Normalized Pointwise Mutual Information (NPMI) over the selected words each topic Dieng al. (2020)
f wy) log p(w;, mzz) p(w))p(wy)) /1 logp(ws; wy)]
where p(u; [1; wy) the probability words wy and wy co-occurring in document and p(w;) the marginal probability of word wi, and both of them are estimated with empirical counts. Those models owing higher topic coherence are more interpretable topic models. TD measures how diverse the learned topics are. We define TD with the percentage of the unique word in the top 25 words of all topics Zhao et al. (202T). that closes to indicates redundant topies; that closes means more diverse topics.
We also report Purity and Normalized Mutual Information (NMI) (Schüitze et al. 2008 on clustering tasks to measure the performance of document representation. To compute Purity, each cluster is assigned to the class which most frequent in the cluster, and then the accuracy of this assignment measured by counting the number of correctly assigned documents and dividing by the total number of documents:
Purity(5,C) 1
Σ N
max(ss 8% 3
where (s., SK) the of clusters, and sets of documents in cluster k and class respectively. Higher purity means higher matching between S and C. NMI is related to the information theory, which can calculated as:
CJ is the of classes. 8 and ej
2T(S, C)
NMI(S,C)
H(S) + H(C)
where I(S, C) is the mutual information of S and C, (()) is the entropy S.
16


Reviews:
[{"review_id": "wkGCulytsO", "reviewer": "Reviewer_G6X6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes BAT-Chain, a hierarchical topic model leveraging multi-layer conditional transport (CT) theory to seize topic structures. The proposal is inspired by previous application of CT to the single-layer topic modeling.  The authors conduct extensive experiments in both textual and visual settings to showcase the merits of their method.", "summary_of_the_review": "A novel approach is proposed to tackle the multi-layer topic discovery problem. The elucidation of the CT framework is rational and comprehensible. However, there is some degree of vagueness in the motivation and hypothetical discussion. Moreover, the qualitative analysis is also not convincing, which drives me to my overall judgement.", "strength_and_weaknesses": "**Strengths**: \n\nThe authors introduce a novel solution for discovering topic hierarchy. The paper is generally smooth to follow, and the experiments can prove the prowess of conditional transport for hierarchical topic modeling.\n\n**Weaknesses**: \n\nFrom my perspective, there exist several drawbacks in the Introduction, Methodology, and Experiment sections:\n\n(1) In the introduction section, I cannot see clearly why reparameterization and KL divergence hurt AVI, especially if AVI and CT are adapted to hierarchical topic modeling, and what merits does CT provide to address the advantages of AVI?\n\n(2) The motivation of CT for hierarchical topic modeling is quite obscure. The introduction only explained that CT was applied because previous single-layer topic modeling had not continued to explore its usage for hierarchical circumstances.\n\n(3) The methodology section mentioned that ``A TOPIC in layer 1 is expected to capture more general information than A WORD in layer 0``. Comparing a topic with a word sounds unnatural to me. Such interpretation somehow makes the hypothetical basis for CT topic hierarchy less persuasive. \n\n(4) The desiderata of the proposed framework is that the upper layers will capture more general topics than the lower layers. Nevertheless, Figure 3 does not clearly indicate the aforementioned relationship. In contrast, the illustrated upper topics have more tendency to display semantic proximity to the lower ones, instead of generalization relation. Additionally, the topics appear to be incoherent as well. For example, why do topic 52 and 24 comprise ``philosopher`` and ``medical``, respectively?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_vkVeDsD5nm", "reviewer": "Reviewer_8ZrW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a hierarchical topic modeling approach based on conditional transport, combining previous research on hierarchical topic models with recent research on conditional transport. The idea is that documents are represented by discrete distributions over word embeddings, topics at each level of the hierarchy are also represented via embeddings, and similarity between topics at different levels of the hierarchy is enforced via a conditional transport cost between each document's distribution-over-embedding representations at adjacent layers/levels of the hierarchy. ", "summary_of_the_review": "The proposed CT-based hierarchical topic model is an interesting combination of previous and recent ideas into a new sophisticated model with nice and potentially useful properties. While the work tackles a worthy problem and proposes an elegant formulation, the training algorithm is heuristic in nature and its ability to optimize the stated objective function is not theoretically motivated. It is likely that a modification of the proposed method could be on a firmer theoretical footing (see the review above). A couple of other claims in the paper are unsupported as well (E.g. the \"Bayesian\" interpretation, perplexity metric is not appropriate). Overall, the research is promising but it needs a bit more work before it is ready for publication.", "strength_and_weaknesses": "Strengths:\n\n- The proposed formulation is sensible and elegant, combining ideas from two previously disparate research directions in a clever way to develop a sophisticated model that leverages the advantages of both.\n\n- The conditional transport approach is expected to provide computational advantages over the previous Bayesian probabilistic modeling approaches.\n\n- The application of the method to image data as well as text was quite interesting.\n\n- It's nice that the method can leverage TFIDF data as input instead of bag of words data. I have not seen topic models that do this. It probably improved performance. (It would however be worth doing a comparison to bag of words to show this, and this was not done in the present manuscript.)\n\nWeaknesses:\n\n- While the modeling formulation is quite compelling, the proposed training algorithm is not well justified. Two update steps are proposed: an \"upward warming up\" used in an initial phase and a \"downward refining\" step (more accurate?). The upward phase ignores half of the terms in the loss function, so is very approximate. The downward phase updates one layer at a time, leveraging the approximate solutions from the layer below computed in the first pass. The layers are not revisited, so every update depends on the approximate first-pass solution from the layer below. This may be a practical approach, but it is clearly not optimal.\n\nPotentially, a better approach might be to perform stochastic gradient updates where the stochasticity is over the loss terms from all of the layers together. I.e., in each stochastic gradient iteration, choose a random document j and a random layer l and take a stochastic gradient step updating Equation 3 with the chosen j and l. This algorithm is a stochastic gradient optimizer for the overall loss function, which is easier to justify than the proposed algorithm.\n\n- The \"Bayesian flavor\" interpretation of the method is a bit of a stretch. I see no substantive connection to Bayesian updating, as the paper claims.\n\n- Perplexity results are not given as it is stated the metric is \"unsuitable for CT based topic models.\" Why is it unsuitable? Is it because CT-based topic models look bad under this metric? (This isn't a deal-breaker since the TC and TD results were good, and these metrics are arguably more important. But there needs to be a better explanation of it.)\n\n- Minor point: A \"BAT-chain-U\" baseline is mentioned but I do not see results for it in any of the tables.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZzjkyQDnTy", "reviewer": "Reviewer_hEFb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a topic model that discovers a topic hierarchy and works with word embeddings. Inference in the model is done by using conditional transport. The inference is also linked to the Bayesian inference. ", "summary_of_the_review": "I believe the question about code similarity would be resolved (there is nothing wrong in using the existing code as a base), then the only real issues left would be overselling of the Bayesian link, which is fixable by downplaying it in the text or elaborating on it to be more solid, and this lack of excitement about the method, but this is not really an issue and may be just me. If to look at the facts, this is a solid piece of work, extending the existing topic modeling approach for discovering topic hierarchies with rather extensive empirical study to support this extension. ", "strength_and_weaknesses": "**Strengths**:\n* Tackling topic hierarchy discovery\n* Rather extensive experiments\n* Easy to follow text\n\n**Weaknesess**:\n* (minor) Polishing is required (some examples below) but should be doable during the rebuttal stage\n* (minor) Although there is nothing apparently wrong with the paper, it doesn’t excite too much. Topic models is a well-researched area. The paper takes the previously proposed settings with word embeddings and topics living in the same space as embeddings with inference based on conditional transport. The paper then extends these ideas to discover topic hierarchy. I can’t say there is not enough novelty in this, but it is just not exciting as looks like a kind of natural extension of those previous settings – it is probably easy to say when it has been done already in this paper, but still. The paper does make an attempt on bringing something else by discussing the link with Bayesian learning – which is a great way to “spice” things up. However, and this is a separate weakness on its own, this link to Bayesian learning discussion is a little bit weak. It would be great to see it more elaborated, with probably some illustrative examples. The whole link is currently based on the fact that in Bayesian learning we have likelihood from the data (i.e., words) and prior from topics of the higher levels and in the proposed model training of a topic on each level is done by transferring the information from below (which is words at the end) and above (which is topics of the higher levels). \n* The link to Bayesian learning is not convincing or elaborated enough (see details above). I believe the paper should either elaborate on this or downplay this link in the text (it is even used in the title which is not justified in the text). \n\n**Question (which may lead to weaknesses)**:\n\nThe provided code looks very similar to the code for the WeTe model, moreover, it is even called WeTe. First, WeTe initial code should be acknowledged. Second, could the authors please briefly discuss in their rebuttal what are the main differences in the code? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject