Meta-Complementing the Semantics of Short Texts in Neural Topic Models
Authors: Delvin Ce Zhang
Hady W. Lauw
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural topic model
short text
graph neural networks
semantic complement

Abstract: Topic models infer latent topic distributions based on observed word co-occurrences in a text corpus. While typically a corpus contains documents of variable lengths, most previous topic models treat documents of different lengths uniformly, assuming that each document is sufficiently informative. However, shorter documents may have only a few word co-occurrences, resulting in inferior topic quality.  Some other previous works assume that all documents are short, and leverage external auxiliary data, e.g., pretrained word embeddings and document connectivity. Orthogonal to existing works, we remedy this problem within the corpus itself by proposing a Meta-Complement Topic Model, which improves topic quality of short texts by transferring the semantic knowledge learned on long documents to complement semantically limited short texts. As a self-contained module, our framework is agnostic to auxiliary data and can be further improved by flexibly integrating them into our framework. Specifically, when incorporating document connectivity, we further extend our framework to complement documents with limited edges. Experiments demonstrate the advantage of our framework.


Meta-Complementing the Semanties of Short Texts in Neural Topic Models
Delvin Ce Zhang
School of Computing and Information Systems
Singapore Management University
Singapore 178902 cezhang- 2018@amu.... www.mm.du..s...........e........
Hady W. Lauw
School of Computing and Information Systems
Singapore Management University
Singapore 178902 hadyrlaur@smmu edu. sg
Abstract
Topic models infer latent topic distributions based on observed word co-occursences in a text corpus. While typically a corpus contains documents of variable lengths, most previous topic models treat documents of different lengths uniformly, assum- ing that each document is sufficiently informative. However, shorter documents may have only a few word co-occurrences, resulting in inferior topic quality. Some other previous works assume that all documents short, and leverage external auxiliary data, e.g., pretrained word embeddings and document connectivity. Or- thogonal to existing works, we remedy this problem within the corpus itself by proposing a Meta-Complement Topic Model, which improves topic quality short texts by transferring the semantic knowledge learned on long documents to complement semantically limited short texts. As a self-contained module, our framework is agnostic to auxiliary data and can be further improved by flexibly integrating them into our framework. Specifically, when incorporating document connectivity, further extend framework complement documents with limited edges. Experiments demonstrate the advantage of our framework.
Introduction
Much of the data on the Web can be represented as text documents. Topic models help to understand the main themes within documents, i.c., each document is represented by a topic distribution, and each topic is interpreted by its key words. The quality of topic distribution of each document depends sufficient word co-oceurrences. However, many real-word corpora contain documents of variable lengths. Academic papers vary from journal manuscripts to conference papers to extended abstracts. News articles could be headlines, short full articles, detailed commentaries. Fig. T (a) illustrates an academic paper corpus where the distribution of document lengths exhibits long-tail tail distribution. Despite variable lengths (with different degrees of sufficiency of word co-occurrences), existing works, e.g., ProdLDA (11 and GATON (37], treat documents uniformly, resulting in inferior topic quality for short texts. Evidentially. Fig. (b)) presents document classification accuracy on four subsets of corpus with descending lengths. Accuracies gradually drop as the length decreases. The inferior topic quality of short texts limits the overall performance topic model.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

bocumant kouth
100
GATON
observed content protein, dlsa, ma gene, cell.
Inleeling semanties prediction d Team
ma clone
observed content
200
1on
Predict
protenn ocil
50
complementary
content
2250 2000 by
0-35 23-30 spbut 75-200
musroon nerve
doc (ong)
(short)
lonight
(x) Document lengths
(b) ClassiFication accuracy
Semantics transfer and complement
Figure I: Illustration of (a) paper corpus with various-length documents, (b) classification accuracy on four subsets of the corpus by descending length, and (c) semantic transfer and complement.
Challenges. Most existing topic models optimize the learning process by averaging the generative losses of different documents, without paying special attention to semantically limited short texts. few studies, e.g., OTLDA (15), take weighted summation of losses based on document lengths, they compute the weights by dividing the length of each document by the length of the whole corpus, which further deemphasizes the importance of short text modeling. Thus, we seek to improve short text topic modeling within variable-length corpus, without hurting topic quality of long documents.
To mitigate the scarcity of word co-occurrences in short texts, some works leverage auxiliary knowledge to enhance topic modeling. ETM [8] exploits pretrained word embeddings [27] [2] to capture word similarities. RTM [5] constructs document network, e.g., paper citation network, to aggregate topics of connected documents. However, they rely on the availability of auxiliary data.
Approach. We propose Meta- Meta-Commlemeet Topic Model (MCTM). Since corpus contains variable lengths of documents, we are motivated to learn the transferable semantic knowledge on long docu- ments, and complement the semantics-scarce short texts and enhance the latter's topic distributions. As illustrated by Fig. [[c.. we learn a function on long document A with abundant content, and use it to predict missing semantics (red box) for short document B complement its content for more expressive topic distribution. Since meta-learning [10] is emerging to improve model performance with few labeled observations, but no one explores its design in topic modeling. We are thus motivated to integrate it and optimize the proposed MCTM by meta-leaming objective.
Orthogonal existing works relying on auxiliary data, our framework is self-contained, assuming only in-corpus information, which offers new direction to improve short text topic modeling. When auxiliary data are available, our framework can be further improved by flexibly incorporating them. In particular, when incorporating document network structure, we discover that document degrees also exhibit similar long-tail distribution, i.e., some structure-abundant documents link sufficient neighbors as auxiliary, while others contain scarce links. Besides textual semantic complement at Fig. Te), we extend MCTM to further complement structural semantics on the document network.
Contributions. Our contributions are as follows. First, we propose MCTM, which learns how to complement textual semantics by semantic knowledge transfer. Second, we derive two alternatives to implement missing semantics prediction function to capture document similarities. Third, although agnostic to auxiliary data, MCTM can also flexibly integrate them to further improve the performance. We demonstrate our adaptability by modeling pretrained word embeddings and document networks. For the latter, we extend MCTM to further complement structural semantics. Fourth, extensive experiments verify the effectiveness of MCTM.
2 Problem Formulation and Preliminaries
We are given a corpus of documents D = [d. Each document E R vocabulary space V. la " wev du is the length of document d where d is the word count of w in d. When word embeddings are available, we have H = (h } wey where h is the embedding of word w. Documents may link to others in document network G = {D,E), with documents and network connectivity & = {Ciy {Ciy] -.j=1. If document& d links to d', Cd,d' 1, otherwise ea,a' = 0. (d) is the set of neighbors of d. We consider an undirected network, ed, = Ed' ,d. Corpus D contains documents of variable lengths (1, [ We introduce hyperparameter L as the threshold where short documents are those with fewer observed words, Datont = {d, < L.). and long documents are defined symmetrically, Diong = (d. 14, 2 L). We consider L as predefined hyperparameter and leave other designs future work. See Table I for summary of math notations.
a vector in the

---

Table Summary of math notations.
Notation
Description
D
a corpus of N documents, D = Jd
d
a document representation in the word space, d E R vocabulary
La
length of document d
a set of pretrained word embeddings pretrained word embedding of word a document network
b
g
E
a set of links connecting documents the neighbor set of document the subset of long documents in corpus D
(d)
D Diong
short
the subset of short documents corpus D the task of document d, i.e., generating observed words of document d a set of support words of document a set of query words of document d a collection of parameters of encoder fo topic distribution of document d, Z E RRK number of topics
Ta
Sa
Qd
0
Z.
K
m,
missing semantics of document
p
a collection of parameters of semantics prediction function 9μ
Swaa))
a set of support neighbors of document a set of query neighbors of document d
Qx(())
Given variable-lengee corpus D (as well word embeddings H. and network & if observed) as input, we aim to output topic distributions for documents where the topic quality of short documents improved. without hurting long documents. Note that goal is not to allow short texts to reach the performance of long documents, but to improve short text topic modeling as much as possible.
MetanLeernnnng Meta-learning [10] optimizes globally shared parameters, a.k.a. prior knowledge, over meta-training tasks, so as rapidly adapt the model to previously unseen meta-testing tasks with only few observed data. Since topic models generally learn topics by content generative process, here we consider generating observed words for document corresponds to a training document d and consists of support set and query set, Ta = (Sa, Qa). Each set contains randomly sampled words from document d, such that support words and query words are mutually exclusive, Sa Qa @ and Sy U Qa d. Meta Meta-training has two steps:
task Ta- A meta-training task Ta
Local update. Given topic model fo with parameter fo is first updated from the globally shared parameter 0 to document-specifie local parameter 0₂ w.r.t. loss on d's support words Sa.
0₂ = - aV eeV,((0,5)) Sa) where Ta = (Sa, Qa) E Tur
meta-learning rate, V gradient, is loss function, Ti a set of meta-training tasks (documents).
2. Global update. After obtaining 0g for each document d, we compute the loss on query words C(Oa: Qa). Together with other training tasks, optimize the globally shared parameter 0.
0"=
Σ C(Oa, 2x) min Σ L(O - (Vo[oVoL(O,Sa)) Qa). TAET.
TaET.
8" the new globally shared parameter and will replace at Eq. D for the next iteration. After convergence, the final global parameter 6* can easily be adapted to meta- meta-testing tasks.
Meta-testing tasks The unseen test documents. All the observed words support words, Ta Sa = d. During meta-testing, topic model fo- with optimized global parameter 0* updated w.r.t. Sa by Eq. I and obtain 0- The topic distribution of testing document is inferred by z, Soy (d).
3 Methodology
We introduce Meta- Ceta-Complement Topic-Model (MCTM) at Fig. 2 Below we elaborate three compo- nents, graph convolutional encoder, missing semantics prediction, and meta-learning optimization.

---

dlssx
preaatedd word
errbekiing
Oz
poopo
Suppor Agee (6.82)
pruned sharte!
prandd saarta
quen
dooumers
O foo
Sunyiaifleedd teifthord,
decument redworre
Creny less minyy ((0%,.)0.oz) Ra) 160 Metsseening y for beth docoment and
Graph supvolutional bope
Ib scmantics prediction with
peodution Couseriig
Figure 2: Model architecture of Meta Meta-Coomlemmnttin Topic Model, MCTM.
3.1 Graph Convolutional Topic Encoding
We follow GATON 37 and first present a graph convolutional encoder fo (Fig. aa),, which projects documents to K-dimensional topic distributions. We defer the discussion short text modeling to the following subsections. Given a corpus D, considering documents and words as vertices, we construct bipartite graph, the links represent word occurrences in the documents. Since documents and words preserve heterogeneous feature spaces, we project them to the same topic space by learnable matrix,
(1+1) w ((+1) 2d
(1+1) W
((+1), 2 where d D, w E V.
(3)
I is the (-th convolutional layer. zz and z are inputs, i..., Bag-of- Bfrros-Words Words and one-hot respectively.
To differentiate the importance of words, we evaluate attention between document d and its observed support words at Eq. 4 and aggregate words at Eq. where [-|1-] is concatenation.
ad softmax ((an((b(+))
11+1)T [Z
where w E S
(4)
(1+1) tanh z
11+1)
Σ
ad 2 ))
(5)
wESa
Symmetrically, we modify Eq. and obtain aw,d; i.e. the attention between word w and documents
it appears in. After symmetric aggregation at Eq. 5 we obtain ((+1) for word w. So far, complete the convolution from I-th to (1 1)-th layer. For simplicity, we summarize aggregation at Eq. by
(++1) AGG(W((+D))
(((+1) Zm lwo E Sa).
Zm
(1+1) AGG( Zu (((+1)
Ad Sa).
(6)
We repeat Eq. 6 for maximum layers and obtain K didiensiionll topics Za L)) for document
and Z., for word The complete encoder is Eq, 7 0 is the set of all encoding parameters.
(L)
Zá Zw (z) (1=0) (!=0))d ld E D, w V).
3.2 Missing Semantics Prediction with Contrastive Learning
short document with few words leaves some content poorly described, resulting in incomplete topic distribution zy. As toy example, document B at Fig. [[c) contains limited words, e.g., gene and clone, and leaves other contents, e.g., protein and cell, uncovered. We aim complement the topics of short documents. For a document d, regardless of long or short, we complement its semantics by
z + Ma.
(8)
We name my R. missing semantics of document d. If d is long document with complete semantics, ma is a zero vector. A function 9μ predicts missing semantics at Eq. 9 with topic distributions of d and its support words as inputs. We will elaborate the design of function gn shortly.
m 9 (zd, Zu E Sa).
Contrastive Learning. Long documents contain relatively more sufficient word co-occurrences than short documents. Thus, we learn missing semantics prediction function In, on long documents, and then transfer the learned semantic knowledge complement short documents. On one hand, a long document d with enough content does not need semantic complement. Thus we have below constraint
+ 0 where d E Diong.
(10)

---

On the other hand, although we aim to transfer the semantic knowledge from long to short documents, there does not exist one-to-one correspondence in corpus D. As a result, we may transfer semanties of a long document (e.g., machine learning concepts) to a short one describing completely distinct content (e.g., biology). To overcome this limitation, we introduce another constraint with contrastive objective. For long document d with support words Sa. we randomly hide proportion of words to mimic a short document, denoted as d' with remaining observed words Sy C Sa and length by < L. For both long text d and its short version d', we predict their missing semantics by Eq. 9 and obtain ma and mdy respectively. ma should complement the previously hidden semantics, i.c., Sa Sa,
L m
+ Zd
+
my
Zi
- Zu
where d de Diong-
(11)
Together with above Eq. 10 which forces z = Zd + my to approach Zg. Eq. acctually has
Z ma
Zel
a - Zd
Zel'
where E DDunn.
(12)
To summarize, we arrive at the following constraint loss.
Leon (d) = - -log or (cos(ccmmmm))) Zd - z2))) where d E Diong.
(13)
(a)) = 1+exp(-x) is sigmoid function, and cos(;,) :) is cosine similarity. We here use cosine similarity, mainly due its superior performance on our datasets. Besides cosine, other similarity metries, such as inner product and Euclidean distance, are also possible, depending on different datasets.
Missing Semanties Prediction. We now define the function of missing semantics prediction 9p- Given topic distributions of a document and its observed support words as input, a desirable function should aggregate them and output single missing semantics vector. We propose two alternatives.
1. GNN function. The first is to implement 9r using a graph neural network, see Fig. ().
ma = I (Za; Zw w E Sa) = AGG U2zw| E Sa).
(14)
A corpus D contains documents with diverse themes. For example, some documents discuss machine learning, while others describe biology. However, the assumption of corpus-lever level shared parameter U, and U2 is not flcxible to model diverse documents for missing semantics prediction, since different documents may have distinet optimal parameters, which are sometimes even in opposing direction. As a result, the predicted missing semantics my centers its mass around the most frequent topics and leaves other distinet topics uncovered. We seek to personalize Uj and U2 for each document d to recover the distinct missing semantics. Formally, we introduce function Φ. which transforms U, and U2 to doecment-specific parameters Ua,1 and Ua,2 2 by scaling and shifting. Taking U, as example,
Ua,1 = d(U1, Zd, Zu w E Sa) U₁ O [(Ta + x] + [(Ba)x k]]
(15)
Yaa and Bg are document-specific vectors for scaling and shifting shared parameter U1. ② is element- wise product. [xx)x x K] is a matrix with K identical column vector is a vector of ones, ensuring the scaling matrix centers around one. U2.2 is similarly defined. Eq. alloww each document d to have its own parameters, while all documents still share the common knowledge. Similar documents scale and shift U₃ and U2 to similar directions. Different documents push them to distinct directions.
We define scaling Ya and shifting Ba. parameterized by topics of document d and its support words.
Ya tanh (W, za + W W,sss), Ba = tanh (W azd W Wy
(16)
Sss [Sa] Ewes, Zhe is the average of d's words. In summary, we use Eq. 14 to predict d's missing semantics, except that shared parameters U, and U2 are replaced by d-specitic ones, U2 and U222-
1
Clustering function. We propose an alternative method by semantic clustering to recover distinct missing semantics, Fig. c))). Documents with similar content fall into related clusters, while unique documents belong to different ones. If we assign each cluster a set of parameters for missing semantics prediction, similar documents would recover their own distinct topics. Specifically, we introduce R centroids {c, /2 each corresponding to one cluster. Given topic distributions of document and its support words, we first evaluate the assignment probability between document d and each cluster by
exp(
1bo
c,11
Pr(r) = softmax
2 ho c, (1))
(17)
R pd
exp( lho C,- 113)
5

---

ho = Iz [s, is the concatenation of Zu and ES.. We then assign parameters to each cluster,
h1 Σ Pr(r) x ReLU(W, + b,).
(18)
Therefore, related documents obtain similar clustering probabilities and missing semanties.
Above process is a flat clustering. Our function can be extended to multiple clustering layers. Each layer s consists of R(*) centroids. After obtaining the output from previous layer, h, we repeat Eq. 18 by replacing hn with - and obtain the output of the current layer s, i..., hg For maximum layers, we get my = hg. We leave adaptive learning of number of clusters R as future work.
3.3 Probabilistic Decoding with Meta-Learning Optimization
After semantic complement, we obtain deD at Sec. 3.2 We use Zy [Zw, ZZwz' ...; Zuppl
R. x/V| represent topic-wor word distribution, each column z topic distribution of a word w, and each row is to the distribution of a topic over the vocabulary. As in previous topic models [ [1], we generate the observed support words S₃ by ds, a (Zyzz). Compared to the ground-truth support words ds.. we follow [37 and obtain generative loss Lgen l|ds, dsallă. However, this loss requires inefficient computation over the whole vocabulary. We instead use negative sampling (27).
M
Lgen(() = Σ [(dm - drr)²
+
Σ 11
Ew Prn (dw - dw))
(19)
wESa
M is the number of negative samples, and Pr, (wr noise distribution over vocabulary. du (zzTz..). d,, 1 if E Sa, otherwise d = 0. In addition, if d a long document d Diong we also created its corresponding pruned short version d' with subset of support words Sa C Sa at Sec.
3.2 Although we do not observe the complete support words for d', its complete topic distribution Z after semantic complement at Eq. 8 should be able to generate the complete support words Sg- Therefore, together with semantic complement constraint at Eq, 13 we arrive at the complete loss.
[(d) (den(d)) + Loon + Areg (0)
(20)
where (d) = Egen(()) (d) + I[d E Diongg) Agen
Leon = I(d E EDionn DDoong Leea (d).
Egen (d) is generative loss, consisting of document d in the corpus E D = Diong U Dahont and the corresponding pruned short version d' (if d Drong). I[d E Phwng) 1 if d E Diong otherwise O. freg (0) is L.2 regularizer for encoding parameters 0. Agen and Aveg are hyperparameters.
Optimization. Finally, with the objective of meta- meta-learning at Sec. 2 we reach the optimization:
1. Local update. Given topic encoder fo defined at Sec. 3.1 optimize encoding parameter 0 w.r.t. the generative loss Ege((d) on support words Sa by Eq. and obtain 0₂ for each document.
2. Global update. With encoding parameter 0d compute the overall loss C(d) query words Qs and optimize all parameters & {6, p}. including encoder parameters and parameters μ of missing semantics prediction function 9μ- aj and 02 are local and global learning rate, respectively.
h- min EEED C(10a. 11, Qa)
x = - a2 V EEED C(O aj VeLyon (10,))) 11,, Sa); Qa). (21)
With new parameter t" = {0-, 1 we substitute it for + for the next iteration. In contrast to previous topic models that generate the content of given documents only, further create the short version of long documents for semantic complement, and jointly optimize them. See supplementary materials for learning algorithm.
3.4 Extensions with Auxiliary Data
MCTM with Pretrained Word Embeddings. Pretrained word embeddings 27 29 encode word similarity. As in previous works 7 45), we incorporate them into topic-word distribution Zy = [ to improve topic modeling. We introduce topic embedding [tx) and evaluate cosine similarity between topic k and word w by cos(t,, h..). We then combine with topic-word distribution by
2k, ( cos(te, (t. h) and obtain topic-word distribution Z l2h for decoding.

---

Table Dataset statisties.
Avg. #words/doc
Std.Dev. of #words/doc
Name
#Documents
#Links
Vocabulary
#Labels
ML
2,947
8,146
5,814
7
66.7
34.0
PL
2.449
7,274
5,066
9
66.0
36.9
HEP-TH 20,151
234,193
5,001
N.A.
48.4
22.9
Web
116,544
309,499
5,021
N.A.
34.1
70.0
MCTM with Document Network. A document network (e.g., citation network) reveals semantic similarities between connected documents (cited papers discuss related research). A document's degree or number of links exhibits a long-tail distribution. Some link to many neighbors, others to few. Previously focus was on textual semantic complement. We extend MCTM to model structural semantic complement for link-scarce documents. We consider generating both observed words and neighbors as a task Ta. As for words, we split the neighbors /N (dd) of a document d into support and query neighbors Ta {Sa, Sxria), Qa, Qv(a)) Sz and Swid) denote support words and neighbors, respectively, and ditto for query sets. We correspondingly extend three modeling components.
1. Encoding. Previously, we inferred topic distribution Zu of document d by its textual words at Eq. Here, we extend this process by designing a structural convolutional module.
(1+1) K d
AGG(W
(I+1)) (1)
Kdd W Kid
((++ (1)
ld' E S
(22)
3
The topic distribution from encoder is Za 1 (Za + Ka). with both texts Sa and structure Sy(d)-
Semantics complement. For long document d. in addition to randomly hiding some words, we also drop some neighbors and create pruned version d'. Now the missing semanties m. should contain both textual and structural information. For GNN function 9p. we extend Eq. by
m. 9r (Zu, Zu Z ww E Sa, d' SN(I)) AGG La, 2d Ua,3Za E Sa, d' Sx(d)).
(23)
Scaling has extra input, Y/a tanh W,za a + W,Zs. + W ) ditto for shifting. [Syray] EdeEssia za'- For Clustering 9p. we extend input by h [za 2ss [ssce))
3. Decoding with meta-learming. In addition to generating support words using complemented zjz we also generate support neighbors. The generative loss similar to Eq. 19 exceppt that i) we replace
d with ed,d', the ground-truth link between d and d'; üi êd. a (z ked Finally, meta-learning learns how accurately predict missing semantics my for both textual and structural complement.
Experiments
The goal of experiments is to evaluate if our model MCTM can improve short text topic modeling through evaluative tasks, e.g., document classification, link prediction, topic analysis.
Datasets. Since our model is flexible to incorporate auxiliary data, we rely on four datasets with textual documents, auxiliary word embeddings, and auxiliary network links for experiments. Cora 24 is corpus of academic papers with citations as links. We created two independent datasets, Machine Learning (ML) and Programming Language (PL). In addition, HEP-TH (21] a corpus of
Physics papers with their citations. Web 20 is a Web page hyperlink network where each page is news article, and the hyperlinks connect related articles. See Table 2 for details.
Since our model has three variants, i.c., MCTM with plain texts, with auxiliary word embeddings, and with auxiliary document networks, we correspondingly compare three categories of baselines. i) Topic models with plain texts, ProdLDA BII. WLDA (28,, and GATON [37]. They model all documents uniformly without dealing with short texts. We compare to them and show the advantage of MCTM on improving short texts. ii) Topic models with word embeddings, ETM g and NSTM 45 Since our model is built on top of GATON, we also compare to GATON with word embeddings. denoted as GATON+ WE. By comparing to them, we verify the effectiveness of semantic complement meta-learning to further improve topic quality. ii) Topic models with document networks, RTM [5), Adjaccent-Encode [40), LANTM (35), and GATON+DN, which is
Baselines.
7

---

Table 3: Classification accuracy (in percentage) four subsets of test set with descending length. Best baselines are underlined. We show improvement of MCTM (G) over GATON and best baseline.
Carger
Mahd
DoWzE
0355
FL
S77ETH
Medch
MLBA
11 34M67
313339
S18-22
143222
2:2:11
174213
400233
25:231
(4mh butt
MI MUTNER
SESTER
113-2
11a8 1140
639
949"
1575 15775
1830 1839"
1229 12.5%
1125
11.29
335"
1329 1929
1A35 1309
084133
WALDD
418128
479111
C4119
with seheddiigs
GAIONIW
234333
302233
401823
MCTMANBF (C)
011-11
611-24
0.1-11
531615
687122
515-15
ieprrviee
ITM
542322
129236
T18:13
50A334
5136111
1872335
3.5230
524134
426622
369467
146-14
134:21
4
01:8111
53 136019
018415
018631
13
Medels
with networda
GALONNUN
0232122
367333
713-31
612382
302241
5632200
53 632220
00:2311
010812
44::33
MCTISSEN DO
133337
863139
137-111
118:24
021111
738:10
113:443
758:38
713139
(81:22
22.9 13.49*
1339"
1329 "19""
3093" 15.15*
34090 15.20*
1829 16.15"
15777
13,92 1409
35,75P 2005*
the extension of GATON with document networks. For document network scenario, we include graph embedding model, meta-tail2 (23), which uses meta-learning to improve nodes with low degrees, but is not topie model and ignores variable lengths of node attributes, i.e., texts.
We set L = 2 convolutional layers. Agen = 2 and A = 0.05. Number of negative samples M = and number of semantic clusters R = 5. L is the median length of the corpus. Local and global learning rates are 01 = 0.001 and ag = 0.0005. We use 300D Glove embeddings. We experiment with 5 independent runs, report mean and std.dev. All the experiments were done Linux server with a Tesla K80 GPU with 11411MBB
4.1 Quantitative Evaluation
Documents from the same category discuss related topics, As in LDA []. we conduct classification to evaluate topic quality. We split 80% documents for training (10% are for validation). Labels are not involved during training. After convergence, we train a kNN classifier
Document Classification.
(k: = 5) 2 with training documents and predict the labels of test documents. We set 64 topics. We compare our models within each category of baselines and report classification accuracy at Table 3 We split the test set into four subsets with descending document length and report the result of both overall test and each subset. 0-25% at Table means the subset with the longest 25% test documents. MCTM (G) and MCTM (C) denote our model with GNN and Clustering function, respectively, We use 4igh
to represent statistically significant improvement with paired I-test at 0.05 significance level.
Our models significantly outperform baselines within each category. We outperform GATON, the best baseline in the plain text and word embedding category, since textual semantic complement improves short texts, and the overall test set is also improved. MCTM (G) performs slightly better than MCTM (C), potentially because GNN recognizes importance of words with attention, while the Clustering function takes simple average. To show our models indeed improve short text quality, we present the improvement of MCTM (G) over both GATON and the best baseline. Our performance generally improves more as the length decreases, which verifies the advantage of semantic complement.
Topic Coherence. Each row of topic-word distribution Zy E K is the distri- bution of one topic over the vocabulary, and the key words of this topic correspond to the highest values on this row. As in ProdLDA (II, we evaluate the coherence of key words by Google Web IT 5-gram Version 1 (9) with NPMI as metric. Table (left) summarizes the results. Toric-word distribution Zy is model parameter and is separate from document length, thus we can not report results of different lengths. LANTM cannot run on large dataset Web. Meta-tail2vec is not topic model, thus excluded. Overall, our models outperform baselines on ML and PL. and are competitive with the best baseline on HEP-TH and Web. This indicates that our models at least do not hurt topic coherence, but can significantly improve other tasks, e.g., classification. Compared to GATON, models significantly improve it, verifying the
Table 5: Topic interpretability.
Topic
Key words of MCTM (G)
variance, prohability, generalize, covariance, approximation non-genetic, ma, stinulu-response, epistasis, mismatch
Topic Key words of MCTM (C)
scalability, muliiprocessor, obviate, compute, algorithm sphere, tungent, three-dimensioonll, vector, geometrical
8

---

Table 4: Topic coherence NPMI (left, in percentage) and perplexity (right) K = 64.
Topic Cobgrence NPMI
Perplexity
Calegory
Model
PT.
HEP-TH
HEP-TH
PredLDA
6.31022
9420.
10.320.
16.2114
7.1910:00
1.2110300
7.7210.00
833.00.00
Models with
WIDD
9.7+0.2
11.6+0.1
13.7-0.4
23.9-.8
18.90+0.73
19.57+0.30
44.31+0.18
45.22+0.00
GATON
9.920.9
84215
8.9115
48+1.1
9.64:0.27
9.17=0.10
8,79:0.57
8.5200.12
plain text
MCTM (G)
10,0+1,4
12,1-122
1377-17
135522
381+0.24
3.00-0.51
3.98+0.29
3.27=0041
(CTM(C)
9.942.0
12.02.8
13.2=2.1
16.11111
3.76
3.63:0.20
4.12:0.30
3.1300.1
ETM
5,540.1
7.740.2
7.2:0.
16.4+0.7
8.67+0.00
8.52+0.00
8,51+0.00
8.5200.00
Models with
NSTM
16.021.0
18.620.6
18.2=0.5
27.910.6
8.4610.00
8.34.0000
8.39
8.0200000
word
GATON-WE
16.1414
12.9:155
16.9-111
12.4+1.1
5.50:0.21
5,5640.48
8.36:-0.22
7.98-0.02
embeddings
(C
12.
19,1228
0.17
3.820.12
MCTM-WE (C)
168+1.1
10.3
18.7-0.9
23.8+1100
4.62:40.16
4.62400.24
3.50+-0.17
0.04
RTM
7.340.22
8.911..5
6.6220.3
18.0.0.4
8.07:10.01
7.93.10.11
8.04.00000
8.9610.13
Models
Adj-Fne
8.4+0.4
10.5++011
6,4004
7.2+0.5
7.41 +0.0+
7.34+0.13
7.45+0.19
7.68+00.00
with document networks
LANTM
9.91122
104=1.5
N.A.
8.63:0000
8.48:0.00
8.5020.00
GATON+DN
10.3++07
10,7++111
9.7-08
7.7+2.0
8.58++0.22
8,43++0.00
8.33+0.01
8:3++0.02
MCTMADN (G)
119E220
11.1200
10.6117
15.5111
4.07 4.0720222
4.121031
339910.40
MCTM+DN (C)
116115
10.3114
1.441.55
17001111
0.36
3.63:10.46
4.1310.23
3.75
Table 6: Link prediction (in percentage) on overall test set and the shorter half of the test set. Best baselines are underlined. We show the improvement of MCTM (G) over GATON and best baseline.
Model
ML
Uvenll
PL
HEP TII
Overal
Web
Short
65.
099:0011
75.3:00.1 78.5+0.2
Adj-Fin
88. 1-01+0.22
86.2+0.3
796++0.
23/8+0.2
88.9+0.1
884-0.1
82.740.1
LANTM
76.5+1.2
76.1+116
739=000
70.51111
86.6+00
852-000
N.A.
N.A.
GATON-DN
74.2.00.4
716610.
71.6::088
65.810066
90.11002
19.1:12
74.3:102
71,71003
mesa
69.7422.3
66.5+115
68.7+177
64.3+115
N.A
N.A
N.A.
N.A.
MCTMADN
91.5:0.1
90.6:008
03.9:022
R34:0.1
SI.3
MCTM-DN
92.7.10.7
91.21088
90.2.00.9
92.510.33
92441044
80.6.002
77,510..
improveent
26.652 6.75
28.458 6.6%
27.95 15.0%
37.75 22.85*
4.3%8 4.2%%
5.4% 5.494
12.29 0.8%
12,38 2,58
advantage of semantic complement. To understand what topics our models capture, we randomly present two topics with top-5 key words at Table MCTM (G) captures statistics and computational genetics, while MCTM (C) reveals scalability and geometric learning.
we evaluate perplexity. Since perplexity is exponential and varies much w.r.t. its power, we report its power, log Pr(Pen) (lower is better). Table 4 (right) reveals that our models generate high likelihood to
Perplexity. Topic model should generalize to unseen documents. Following 3
Lne Day fa unseen documents, which we attribute to semantic complement meta-learning module.
Link Prediction. A good model should infer similar topics for potentially linked documents. Since we model document network as auxiliary data, we follow RTM [5] and predict links. As in [40, the probability of a link is p(ea.a)) and compare with the ground-truth links with AUC as metric. Since only the third category (network models) incorporate links, we mainly compare the MCTM+DN version these network baselines. Table 66 indicates that our models predict links more accurately than baselines. The comparison to network baselines demonstrates the effectiveness of both textual and structural semantic complement.
exp(-I|zj z (l2) We predict the links within test documents
4.2
Model Analysis
To better understand our model, we conduct model analysis here.
Effect of Semantic Complement. To see if semantic complement helps short texts, we remove it from the complete model and present classification accuracy at Table 7 (left). We show the plain text results, and put the auxiliary data version in supplementary. Models do better with semantic complement than without. The accuracy on short subset declines more than the overall test set, which reveals that semantic complement improves short texts, and removing it hurts short documents more.
Effect of Meta-Learning, To analyze if meta- veta-learning benefits the optimization with a few observed words, we replace it with the commonly used stochastic gradient descend. Table (middle) shows that result drops more on short subset than on the overall test set, which verifies that meta-learning is good at optimization with only few observed words and improves short text modeling. We further remove both semantic complement and meta-learning, and report the result at Table 7 (right), which presents the worst accuracy. This observation further verifies that both components are important.

---

Table 7: Effect of semantic complement and meta-learning on document classification on ML.
Effect Semanric Complement
Effect Meta-I Learnnn
Effect of hoth
Model
Overall test set
Short subset
Overall tost sct
Short subset
Test
Short
with
without
with
without
with
without
with
without
without
without
MCTM (G)
6..1
58.21124
60.1 1,9
50.8
67.1
65.911.3
60.1
57.0.3.55
52.114.1
44,51.5.5
(decline)
(13.3%*)
(15.5%
(5.2%%*)
(22.4%-9
(26.0%+*
MCTM (C)
66.9114
56.7=3.1
1..9+188
49,4=2.9
66.9114
65.6+1.0 (1.9%)
58.118
46,924.8 52,5+2.4 56.142.4
(decline)
(15.2%*)
(16.1%%9)
(4.8%
(21.5%8-)
(20.4%*)
Effect of Scaling and Shifting, The GNN version of our model uses recover distinct topics. To test usefulness, we disregard and summarize classification accuracy on ML at Table 8 (left). Removing scaling and shifting leads to worse performance, since can not personalize shared parameters to each document to recover its distinct missing topics for complement.
method to
Effect of Clustering, We set the number of clusters R all documents share the same
Table 8: Effect of scaling-and-shiiting and clustering.
parameters for missing semantics prediction with no clustering. Table [8] rright) concludes that clustering is helpful to share common semantics for related documents and distin- guish documents of different clusters.
Scaling and Shifting
Rffect Clustering
Model
with without
with
without
MCTM
67.1:2.1 65.7=2.9
66.9
6..721.
MCTM+WE
66.8+2.0
65.9+1.0
6..0+1.4*
62.4+1.9
MCTM+DN
83.311.77
82.4#14
83.0:1..
81.011.3
5 Related Work
Traditional topic models are graphical models 3 13 Recent ones are neural models [26].
mainly
variants of VAE [17]. ProdLDA [31 and DVAE 4 analyze Dirichlet prior. WHAI 43
looks into
Gamma prior. The variational divergence metric also attracts attention. WLDA 28
and WAE B2
apply mean maximum discrepancy and generative adversarial network 12).
KATE
6
increases
topic sparsity. More recently, topic models are built with graph neural networks [8]
e.g.. GATON
37.. GraphBTM [46), etc. They average losses of all documents regardless of lengths. MCTM
designs semantic complement to predict missing semantics for short texts to improve topic quality.
To alleviate scarce word co-occurrences of short texts, existing works leverage auxiliary data, such as word embeddings [27 [29]. Graphical models include GLDA (7). GPUMM (22] LCTM (14),
MetaLDA [44) WDL [36, and OTLDA [15]. Neural models include ETM g and NSTM
45
Another category improves short text modeling with document networks, e-g.. citation network. RTM
[5] and NetPLSA [25] are graphical models. Neural models include NRTM []]. MajaccettEEncode
40 LANTM [B5), NetDTM
42 SemiVN 41. In contrast, our semantic complement module
agnostic to auxiliary data.
Here, also briefly review meta-learning works. One category metric-based, e-g., ProtoNet 30) and MatchingNet (33), which learn a metric function over tasks. mandient-hased meta-learning works
optimize model parameters for quick adaptation new tasks [10 16
38 39
6 Conclusion
We improve short text topic modeling with semantic complement meta-learning. We complement the semantics for short documents by contrastive learning and design alternatives for missing semantics prediction. Meta-learning helps to optimize and predict the missing semantics. Experiments on document classification, topic coherence, perplexity, and link prediction verify the effectiveness of our model. One limitation is to assume a variable-length corpus with both long and short documents for semantic transfer. We also assume the content is truthful. If the corpus is infiltrated by fake news, those may appear in some topics. A future work is exploring online learning to speed up training.
Acknowledgement
This research/project supported by the National Research Foundation, Singapore under its AI Singa- pore Programme (AISG Award No: AISG2-RP-2021 -020). Hady W. Lauw gratefully acknowledges the support by the Lee Kong Chian Fellowship awarded by Singapore Management University.
10

---

References
[1] Haoli Bai, Zhuangbin Chen, Michael R Lyu, Irwin King, and Zenglin Xu. Neural relational topie models for scientific article analysis. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 27-36, 2018.
[2] Christopher M Bishop. Pattern recognition and machine learning. springer. 2006.
[3] David M Blei, Andrew Y Ng. and Michael Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 1-299-1022,, 2003.
[4] Sophie Burkhardt and Stefan Kramer. Decoupling sparsity and smoothness in the dirichlet variational autoencoder topic model. Journal of Machine Learning Research, 20(131):1-27, 2019.
[5] Jonathan Chang and David Blei. Relational topic models for document networks. In Artificial intelligence and statistics, pages 81-88. PMLR, 2009.
[6] Yu Chen and Mohammed Zaki. Kate: K-competitive autoencoder for text. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 85-94, 2017.
[7] Rajarshi Das, Manzil Zaheer, and Chris Dyer. Gaussian Ida for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume Long Papers), pages 795-804, 2015.
[8] Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439-453, 2020.
[9] Stefan Evert. Google web It 5-grams made easy (but not for the computer). In Proceedings of the NAACL HLT 2010 Sixth Web as Corpus Workshop, pages 32 40, 2010.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126-1 135. PMLR, 2017.
[11] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-leaming. Advances in neural information processing systems, 31, 2018.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[13] Thomas Hofmann. Probabilistic latent semantic indexing. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 50-57, 1999.
[14] Weihua Hu and jun'ichi Tsujii. A latent concept topic model for robust topic inference using word embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 380-386, 2016.
[15] Viet Huynh, He Zhao, and Dinh Phung Otlda: A geometry-aware optimal transport approach for topic modeling. Advances in Neural Information Processing Systems, 33:18573-18582. 2020.
[16] Taesup Kim, Jaesik Yoon, Ousmane Dia. Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. arXiv preprint arXiv:1806.03855. 2018
[17] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[18] Thomas N Kipf and Max Welling. Seni-supervised classification with graph convolutional networks. ICLR, 2016.

---

[19] Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, pages 2927-2936. PMLR, 2018.
[20] Jure Leskovec, Lars Backstrom, and Jon Kleinberg. Meme-tracking and the dynamies of the news cycle. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 497-506, 2009.
[21] Jure Leskovec, Jon Kleinberg, and Christos Faloutsos. Graphs over time: densification laws, shrinking diameters and possible explanations. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pages 177-187, 2005.
[22] Chenliang Li, Haoran Wang, Zhiqian Zhang, Aixin Sun, and Zongyang Ma. Topic modeling for short texts with auxiliary word embeddings. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 165-174, 2016.
[23] Zemin Liu, Wentao Zhang, Yuan Fang, Xinming Zhang, and Steven CH Hoi. Towards locality- aware meta-leariing of tail node embeddings networks. In Proceedings of the 29th ACM International Conference on Informaton & Knowledge Management, pages 975-984, 2020.
[24] Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 3(2):127-163,
2000.
[25] Qiaozhu Mei, Deng Cai, Duo Zhang, and ChengXiang Zhai. Topic modeling with network regularization. In Proceedings of the 17th international conference on World Wide Web, pages 101-110, 2008
[26] Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In International conference on machine learning, pages 1727-1736. PMLR, 2016.
[27] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 2013.
[28] Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. Topic modeling with wasserstein autoencoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6345 , 2019.
[29] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages
2014.
[30] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[31] Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. ICLR, 2017.
[32] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. arXiv preprint arXiv:1711.01530 2017.
[33] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 29, 2016.
[34] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph Lim. Multimodal model-agnostic meta-learning via task-aware modulation. Advances in Neural Information Processing Systems, 32, 2019.
[35] Yiming Wang, Ximing Li, and Jihong Ouyang. Layer-assisted neural topic modeling over document networks. In International Joint Conference on Artificial Intelligence, pages 3148- 3 315,, 2021.
12

---

[36] Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin, Distilled wasserstein learning for word embedding and topic modeling, Advances Neural Information Processing Systems, 31, 2018.
[37] Liang Yang, Fan Wu, Junhua Gu, Chuan Wang, Xiaochun Cao, Di Jin, and Yuanfang Guo. Graph attention topic modeling network. In Proceedings of The Web Conference 2020, pages 144-154, 2020.
[38] Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning In International Conference on Machine Learning, pages 7045-7054. PMLR, 2019.
[39] Huaxiu Yao, Xian Wu, Zhiqiang Tao, Yaliang Li, Bolin Ding, Ruirui Li, and Zhenhui Li. Automated relational meta-learning. In International Conference on Learning Representations, 2019.
[40] Ce Zhang and Hady W Lauw. Topic modeling document networks with adjacent-encoder. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6737-6745, 2020.
[41] Delvin Ce Zhang and Hady W Lauw. Semi-supervised semantic visualization for networked documents. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 762-778. Springer, 2021.
[42] Delvin Ce Zhang and Hady W Lauw. Dynamic topic models for temporal document networks. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 26281-26223 PMLR, 17-23 Jul 2022.
[43] Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. Whai: Weibull hybrid autoencoding inference for deep topic modeling. ICLR, 2018.
[44] He Zhao, Lan Du, Wray Buntine, and Gang Liu. Metalda: A topic model that efficiently incorporates meta information. In 2017 IEEE International Conference on Data Mining (ICDM), pages 335-644. IEEE, 2017.
[45] He Zhao, Phung, V Huynh, Le, and W Buntine, Neural topic model via optimal transport, 2020.
[46] Qile Zhu, Zheng Feng, and Xiaolin Li. Graphbtm: Graph enhanced autoencoded variational inference for biterm topic model. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We aim to design a neural topic model to improve short text topic modeling with semantic complement meta-learning The Methodology section and Experiments section reflect these claims.
(b) Did you describe the limitations of your work? [Yes] See Section g
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We do not have theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] We do not have theoretical results.
3. If you experiments...
13

---

(a) Did include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We include the source code, datasets, and readme file for reproducibility.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specify all the training details at Section
(c) Did you report error bars (c.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Table 3-8
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We correctly cite the papers that provide datasets and code of baseline models for experiments.
(b) Did you mention the license of the assets? [N/A]
(C) Did you include any new assets either in the supplemental material or as a URL? [N/A] All the datasets and baseline models are publicly available.
(d) Did you discuss whether and how consent using/curating? [N/A] All the datasets and baseline models are publicly available. (c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The data we are using does not contain personally identifiable information or offensive content.
obtained from people whose data you' 're
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We did not use crowdsourcing or conduct reseatch with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] We did reseatch with human subjects.
crowdsourcing or conduct
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We did not use crowdsourcing or conduct reseatch with human subjects.
14


Reviews:
[{"review_id": "y5GCdnpuoL-", "reviewer": "Reviewer_yv6N", "summary": "A number of issues this study are handled are listed as follows.\n1. Most of the previous topic models treated the document with various length uniformly.\n2. Short documents may only have few words co-occurrences which will make the model hard to identify the corresponding topics.\n3. Previous solutions dealt with the short documents require external auxiliary data.", "questions": "The proposed method has clear motivation that it wants to treat short documents and the long documents uniquely. Therefore, more semantics can be extracted from the short documents to increase model performance in classification and topic coherence. Meta-learning is also introduced to deal with the limited data. However, the learning criterions used in this work is unclear and the organization is not well adjusted. The figure only delivers limited information. Since the learning process is quite complicated, it is suggested to provide Algorithm.", "limitations": "The novelty is not sufficiently justified.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n1. The proposed method showed good performances in their evaluation.\n2. Ablation study was conducted.\n\nCons\n1. Content organization was not well adjusted. Figure only showed the limited information and there is no Algorithm. Therefore, it was quite hard for the readers to understand the learning process.\n2. The methodology was not well-explained:\n  (1) In the contrastive loss, “What is the meaning of contrastive in the loss function?, because there are no positive negative pairs” and “What is the intuition of using cosine similiraity as an input for softmax function?”\n  (2) In the generative loss, \" what is the function of m?\", It just described as number of negative samples, but what is the role of m in the equation?\n3. The proposed method was not evaluated on the dataset commonly used in the mentioned previous works, which is 20newsgroup. Therefore, it’s hard to say that the proposed method could outperform the previous solutions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "n6fiuu0SAgj", "reviewer": "Reviewer_hYpp", "summary": "This paper proposes a meta-complement topic model (MCTM) to improve the quality of discovered topics in short texts. The key idea is to transfer the semantic knowledge learned on long documents to short texts via meta-learning. Some previous studies rely on external data, such as pre-trained word embeddings and document citation networks, to mitigate the scarcity of word co-occurrences in short texts. In contrast, this work proposes to solve this problem within the corpus itself. The authors also show how to generalize their framework when word embeddings or citation networks are available. Experiments on 4 datasets demonstrate the effectiveness of MCTM in text classification, link prediction, topic coherence, and perplexity.", "questions": "- Could you conduct t-tests in your result tables and report whether your improvement over baselines/ablation versions is statistically significant or not?", "limitations": "The authors mention the technical limitations in Section 6.\n\nThey also mention the potential negative societal impact of their work in Section 6. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The idea of using meta-learning to deal with short texts in topic modeling is novel and intuitive from my perspective. According to Section 3.4, it is also straightforward to extend the model when external signals are available, indicating the flexibility of MCTM.\n\n+ The empirical study is comprehensive. Besides the comparison of common topic modeling metrics (i.e., NPMI and perplexity), the authors also explore downstream tasks such as document classification and link prediction (given that the model can be extended to utilize document links).\n\n+ Ablation studies demonstrate the contribution of different components in model design.\n\nOverall, this is a solid submission in my view. However, I still need to explain the following minor concerns.\n\nWeaknesses:\n- Significance tests are missing in all result tables. It is unclear whether your improvement over baselines/ablation versions is statistically significant or not. In fact, in Tables 3 and 7, some gaps are quite subtle. Please conduct t-tests and report p-values. (Also, please order all tables according to their number. For example, Table 1 should be presented before Table 2.)\n\n- For the evaluation of topic modeling, the adopted metrics are NPMI and perplexity. However, a recent study [1] shows that such metrics may not align well with human judgment. I wonder if more human annotations can be included (e.g., the intrusion test) when evaluating topic qualities.\n\n[1] Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence. NeurIPS'21.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PD6HwXjmm18", "reviewer": "Reviewer_zq8q", "summary": "In this paper, the authors address the limitation of previous topic models in modeling short texts. Their analysis demonstrates a decreasing classification accuracy with decreasing text lengths, due to the lack of word co-occurrences in short texts. They propose to complement the semantics of short text, by training a \"missing semantic prediction\" model on long text. Specifically, a proportion of a long text is masked and the missing semantic prediction model is expected to output a complementary representation (\"missing semantics\"), which adds to the representation of masked long text to reconstruct the representation of the full long text. The authors propose two configurations of the missing semantic prediction model, which are based on GNN and clustering respectively, and GNN seems to be a slightly better choice. It can also be extended to leverage external data, such as pre-trained word embeddings or document networks for further improvement. Experimental results show significant improvement in document classification, topic coherence, perplexity, and link prediction, for both long and short texts.", "questions": "1. The authors choose to complement document representations. However, it might be worth exploring a textual level complement, such as using a language model to generate extra sentences on short texts for richer semantics.\n2. Is the constraint in formula (10) met? In other words, will the model still complement relatively long texts, or keep them unchanged?\n3. It is quite surprising to observe a large improvement in long texts (0-25%) classification as well. What could be the reason for improvement? Will there be such an improvement if we take a look at more extremely long texts (0-5%)?", "limitations": "The authors did not include a discussion on limitations or ethical impact. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well motivated with quantitative analysis to demonstrate the disadvantage of previous topic models in handling short texts.\n2. The method is novel in complementing the semantics of short text to overcome the lack of word co-occurrences in short texts.\n3. Overall, the paper is well written with a comprehensive explanation of the method and formulations.\n4. The experimental results show significant improvement, and the ablation studies are sufficient to justify the design choices.\n\nWeaknesses:\n1. Some important hyperparameter details are missing, such as the threshold for defining short text and long text, the percentage of masking long text, negative sampling rate, etc. \n2. The explanation of the meta-learning part is unclear. How do you optimize formulas 1 and 2? Also, it might be better to merge the paragraphs on meta-learning at the end of Section 2 and Section 3.3 for easier understanding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Jcj4Ngdf3q", "reviewer": "Reviewer_kUTk", "summary": "This paper focuses on improving topic modeling performance on short documents. Short documents have fewer word co-occurrences that results in worse topic quality. The paper assumes a corpus of documents of various lengths and leverage the longer document to complement the semantics of the short text. They intend to improve the performance on short text without hurting the performance on the longer text. Their model can also work with auxiliary data such as pre-trained word embeddings and the document connectivity structure. Experiments demonstrate improvement of classification accuracy on Cora-ML and Cora-PL, with shorter documents benefit more. Topic coherence and perplexity is evaluated and reported on Cora-ML, Cora-PL, HEP-TH, and Web. The proposed models outperform other methods on Cora-ML and Cora-PL and are similar to baseline methods on HEP-TH and Web.", "questions": "1. If I read correctly, in Table 2, it seems MCTM with plain texts is performing better than MCTM with auxiliary word embeddings. I'm wondering if there is any explanation of why auxiliary information (word embeddings) is not helping. (The presentation of Table 2 could be improved by adding the description of each group (Topic models with plain text, etc.)\n\n2. I'm wondering if Table 6 can be re-organized and adding the number of without both semantic complement and meta-learning. It seems the effect of semantic complement is bigger.\n\n3. I wonder if semantic complement could be viewed as a prediction/generation problem (given z_d', predict m_d') and utilize the pre-trained large language model for such prediction.", "limitations": "I'm assuming that if the length of the document is not uniform across the topics, for example, certain topics only contains short documents, it might be a problem for the proposed method. I'm wondering if the author has done analysis regarding this. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses a focused problem: how to improve the performance on short document when doing topic modeling, which relies on the co-occurrence of words. The semantics transfer and complement seems to have good improvement on the performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept