JOINTLY LEARNING TOPIC SPECIFIC WORD AND DOCUMENT EMBEDDING
Authors: Farid Uddin
Zuping Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: Language modeling ·Document embedding ·Natural language processing ·Machine learning

Abstract: Document embedding generally ignores underlying topics, which fails to capture polysemous terms that can mislead to improper thematic representation. Moreover, embedding a new document during the test process needs a complex and expensive inference method. Some models first learn word embeddings and later learn underlying topics using a clustering algorithm for document representation; those methods miss the mutual interaction between the two paradigms. To this point, we propose a novel document-embedding method by weighted averaging of jointly learning topic-specific word embeddings called TDE: Topical Document Embedding, which efficiently captures syntactic and semantic properties by utilizing three levels of knowledge -i.e., word, topic, and document. TDE obtains document vectors on the fly simultaneously during the jointly learning process of the topical word embeddings. Experiments demonstrate better topical word embeddings using document vector as a global context and better document classification results on the obtained document embeddings by the proposed method over the recent related models.

Under review as a conference paper at ICLR 2022
JOINTLY LEARNING TOPIC SPECIFIC WORD AND DOC- UMENT EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
Document embedding generally ignores underlying topics, which fails to capture polysemous terms that can mislead improper thematic representation. More- over, embedding a new document during the test process needs a complex and expensive inference method. Some models first learn word embeddings and later learn underlying topics using a clustering algorithm for document representation; those methods miss the mutual interaction between the two paradigms. To this point, propose novel document-embeddiin method by weighted averaging of jointly learning topic-specific word embeddings called TDE: Topical Docu- ment Embedding, which efficiently captures syntactic and semantic properties by utilizing three levels knowledge -i..., word, topic, and document. TDE obtains document vectors the fly simultaneously during the jointly learning process of the topical word embeddings. Experiments demonstrate better topical word embeddings using document vector as a global context and better document clas- sification results on the obtained document embeddings by the proposed method over the recent related models.
INTRODUCTION
In recent years the neural network- based methods have shown great success in various NLP-related tasks (Zhao et al. 2021). Tow popular complementary natural language processing methods are word2vec (Mikolov et al. 2013a b) and Latent Dirichlet Allocation (LDA)(Blei al. (203);; but executing a model in two-step lacks mutual interaction between the two paradigms and fails to cap- ture mutual reinforcement between themselves. Doc2vec (Le & Mikolov 2014) generalizes the idea of vector representation for documents as stand-alone method. Though it is expensive to infer new document vectors during the test process, and regardless of assigning PV each word in the document as a single topic, it outperforms BOW (Zhang et al.] 2010) and LDA (Fan et al. (0088) in various text understanding tasks. Encouraged by word2vec and Doc2vec, Niu et al. (2015) intro- duced Topic2 It incorporates the LDA topic model into NPLM (Bengio et al.) 2003). However, Topie2 Vec cannot capture the polysomic characteristics of terms as LDA models topic occurrence almost independently without capturing topic correlations (He et al. (017). The purpose of con- ducting this research is to introduce an efficient multi topical document representation with flowing objectives:
Firstly, learn prohabilistic topical word embeddings using document vector as a global context where each word can part of different underlying topics of the document instead of just single paragraph vector. Thus, avail to capture polysemous terms.
Secondly, embedding documents by simply averaging the topical word embeddings instead of imposing expensive inference method to learn the document embedding separately.
Unlike clustering- based document modeling with the ability capture multi-sense of words (Gupta jet al. 2019), here the challenge is to learn polysemous words a jointly learning framework for topic-specific word and document embedding. Capturing polysemous terms is vital because the same word around different context words can represent different themes (Liu et al. 2020) and without noticing them mislead to unexpected thematic representation of document. The Jointly Learning Word Embeddings and Latent Topics (Shi et al.) 2017)) comes to the rescue for addressing the polysemous word problem, which learns different topical word embeddings using

---

Under review as a conference paper at ICLR 2022
Skip-gram gram Topical word Embedding (STE) framework. STE gets the term distributions of the topics and the topic distributions of the document using PLSA(Hofman 2013) like mixture weighting for each data point. However, STE doesn't 't deal with document embedding in its jointly learning process.
In this research, propose novel document embedding framework called TDE: Topical Doeu- ment Embedding. TDE learns topical word and document embeddings jointly. It incorporates the STE-diff framework (Shi et al. 2017) and follows the simple averaging of word embedding with corruption mechanism for document embedding introduced by Chen (2017)
2 MOTIVATION AND RELATED WORK
Huang et al. (2012) proved that combining local and global context performs better for predict- ing target words in the neural fetwork-basee language modeling, but this model doesn't I support topic-specific word embeddings. [iu et al. (2015) proposed topic-specifie word embeddings where document embeddings get by aggregating over all topical-word embeddings of each word in the document. |rrora al. (017) extend embedding to the sentence level by smooth inverse frequency (SIF) weighted averaging of the words in sentence. However, this model is lack of addressing mul- tiple sense of words. Moody (2116) introduced hybrid model called Ida2vec, which suffers from uncorrelated topics due to the incorporation of LDA. For word sense disambiguation with correlated topic modeling, Chaplot Salakhutdinov (2018) replaced the LDA topic proportions by synset pro- portions for document. Other than LDA topic modeling, AAroa et al. (2018) have introduced Gaussian random walk model for word sense disambiguation. This work can address polysemous word representation but doesn't cover document-level embeddings. Mekala al. (2016) introduced the Gaussian mixture model- -ased soft clustering method called SCDV, which incorporates both the topic and word embeddings. However, SCDV not suitable for multisentence documents. Hence Luo et al. (2019) proposed P-SIF, which is fit for large documents with multiple sentences while addressing the underlying topics as well. Though P-SIF overcomes SCDV's shortcomings, it is still two-step process.
From the motivation of the above advantages and drawbacks, we introduce novel document mod- cling capable of learning document embedding simultancously during the jointly learning process of tppic-specific word embeddings. This model creates a pivot by concatenating a document vector and center word vector to predict the context word vector. Here the document vector works as mem- ory backup for missing information. The output document embeddings produced by this model is more expressive it represents each document into a higher dimensional embedding space similar to P-SIF.
3 MODEL DESCRIPTION
Inspired by the Skip-gram Topical word Embedding (STH) (Shi et al. 2017) and the simplicity of the Document Vector through Corruption(Chen] (017)). we design a novel document embedding model called TDE: Topical Document Embedding. In contrast to Vec2 TDE constructs global context (document vector) as a weighted averaging of the randomly selected topic-specific word embeddings. Unlike STE, TDE uses generating function, which predicts context words by given the topics and pivot. Pivot constructs by concatenating global context (document vector) and center word (Figure[). The probability of predicting a context word depends on the inner product of the topic-specific embedding of that word (context word) and the pivot, which ensures the participation of underlying word, topic, and document levels knowledge.
3.1 TOPICAL DOCUMENT EMBEDDING
A matrix D represents the documents in the corpus, D d1, dg...., dn of size n. Each document d contains variable-length sequence of words W = W1, ..2, WT,, Each word in the vocabulary V of size of the corpus is generally associated with an input projection matrix Ue R of size and an output projection matrix V'e_R"x R of size h. The input projection matrix represents the
input to the hidden space, whereas the output projection matrix represents the hidden to output space. Matrix ' has the same dimension as matrix U but a transpose matrix. The column u of matrix U is the input vector of word and the column uhe of matrix VT is the output vector of word w.
2

---

Under review as a conference paper at ICLR 2022
Context words
Topics
W++1
W t++
W t+n
Z1
Context
Center
Zp We +
x
Zz
Wt+i
Wt
Concatenation (pivot)
Underlying topics
x
Z
T
Document vector (averuge)
Figure 1:
Predicting context words by giving pivots and underlying topics of the document.
Figure 2: Topics assignment of polyse- mous words.
TDE follows unbiased mask-out/drop-oou corruption mechanism in each update of its stochastic process for randomly select topic-specific word embeddings to create the global context. Under corruption mechanism, each dimension of the original document d overwrites with the probability and sets the uncorrupted dimensions (1-q) times of their original values to make the corruption unbiased. The corruption mechanism reduces word-level noise in the document by forcing the embeddings of common and non-discriminative ones close to zero. This corruption mechanism leads TDE to speed up the training process by significantly pruning the number of parameters for the backpropagation mechanism, document vectors represented by only embeddings of remaining words. The resultant global context is thus as below:
O,
with probability q
Fd
(1)
otherwise
I-q²
Σ Fd
Document vector (global contert)
where is the length of the document (number of words) and the document vector is the weighted average of the words in a document, weighted by the corruption mechanism mentioned above.
The probability of observing the context word Wi+i given the center word wy as well as the global context and the topics z4+i and 2 for context and center word respectively as
exp (Ve (V Eeti (Um,, 21 + +UZz))
p(we+ Wes 2445, 21, 2)
(3)
Em'cs eA eap VT :044 (Um,, + + +Ua))
where A is the vocabulary of the whole corpus and input projection matrix constructs as U Uw., + 1. U.s..
The denominator in equation (3) is generally intractable due to the term EmicA As such, the computational cost is proportional to the size of A. deal with this intractability requires an effi- cient approximation method. TDE employs the approximation with the negative sampling (Mikolov) (ct al. 2013a] Chen 2017). TDE, the probability of predicting a context word W document d depends the pivot ū and the topic distributions of the document. The pivot constructed by concatenating the center word w, and the global context ă as-
Pivot = wíã = wt + ă
(4)
So, for every skip-gram (w,, ui wr-i), where the topic assignment Z+4s of we++ is independent of the topic assignment of w, we have
p(we+i |W,ē, d)
Σ
Σ
p(w44s wyē, 2441 21) p(ze41s z4 (d)
#=1 s1t
(5)
Σ
Σ
plwe+e I 244s, 2)) p(zsld)p(zetr
3

---

Under review as a conference paper at ICLR 2022
Following similar analogy STE-diff, equation (5) assumes that the context word and the center word may come from the same topic or may come from two different topics as well without im- posing any hard constraint (Figure 2). Here, multiple meanings of word depend their contexts instead of directly & topics disjoint subspace partitioning as commonly does in clustering- topic modeling(Gupie common shared space. Instead of the free flow of scattered words in the embedding space, different topical word embeddings stay close together document-wise rather than topic-wise gathering. Cre- ating pivots by concatenating the center word with the document vector provides extra guidance as global context with a complete thematic sense of the document. Thus, TDE represents more con- sistent topical word embeddings over STE-diff. In TDE, the compositionality of words encodes in the learned embeddings; as a result, the requirement of heuristic weighting for test documents no more required; each output document vector is represented simply as an average of the topic-specific word embeddings. Given the learned projection matrix ww.s, TDE forms document embedding dvec as below:
al., 20T6). In TDE, word embeddings from different topics co-exist in
duec B +==1T
Eve,
Uws
(6)
where T' is the length (number of words) of the document and i represents the underlying topics of the document.
In equation (6), the term e>_1 represents topic-wise concatenation. In this case, words in the same topic are just simple average as they represent the same thematic meaning. However, words in different topics carry different themes. Thus, concatenate over topics increases the expressive power of a document. This strategy represents documents into a higher k X s dimension, where is the underlying topics, and s is the dimension of the word embeddings.
3.1.1 ALGORITHM DESIGN
The inference framework of TDE (Algorithm I is similar to STE-diff, based on the EM algorithm with negative sampling (Shi et al. 2017). In each iteration, updates the word embeddings first, then based on updated word embeddings, updates the document embedding (glohal context) ă. Finally, evaluate the topic distribution p( d) of each document. To update word embeddings, iterates over each skip-gram (w,, Wx+s) using negative sampling and evaluate the posterior topic distribution. So, the estimation of the probability of predicting the context word wt+i given pivot w.ã and topic assignment denoted by p(we+il Wzd, 2) as follows:
log (eeeelerr logo
VT V (Uw,, = - [UAs))
+ Σ Ewayp logo (-ViT - x (Uw..:=+ + Una )
_ T
where (xx) = 1+ezp(-z) and Wi a negative instance, which is sampled from the unigram distri- bution p()) over terms in the vocabulary and has value similar to Gupta et al. (2019)'s 's findings. The overall training objective is how well the model can predict surrounding words, which for each document d, given sequence of words W1, Wi,-..., WT, then the log-likelihood La is defin- ing as
1
La Σ
Σ
logp(wers |wre, d) =
(8)
-eSise; $50
where e is the size of the training windows. The overall liceliidd C is then
C C- Ls
The gradients of the objective function to evaluate log p(w++ wwz d) for each term concerning the input and the output projection matrix U and VT respectively can formulate as follows:
ac
= (Ew - -o(V2 W. Um) V2... p(z |d, we w²
(10)
U w. R,
4

---

Under review as a conference paper at ICLR 2022
Oc
(Eor - ui'z Unit, x)) Uw. a, p(= d, wri, w')
(11)
V
where
Ex/ we {
1, if w in the contert window of
(12)
0, otherwise
In TDE, the update rule of U and VT maximizes pl p(wt+s/we. 2). thus log / 2) for each skip-gram (wy, We++) instead of maximizing
W/, 2) as STE does. This observation shows both of the models flow the same procedure. STE has proved capable of discovering good topics and identifying high-quality word embedding vectors. Thus, TDE inherits all the promi- nences; moreover, it keeps Doc2 simplicity for document embeddings and extends reliability by filling up essential information gaps using global document context. When concerning topics for the embedding process, the generally used hierarchical Softmax Negative sampling methods are incapable of obtaining the emheddings efficiently. The EM-negative sampling Algorithm thus comes to the rescue.
In the E-step, the posterior topic distribution for each skip-gram (w,, wi-s) in document d evaluate using the Bayes rule:
p(z d, w.s. Wh+s) = p(we+i 1 w. P( |d)
(13)
p(wes [2, w,
In the M-step, given the posterior topic distribution, now time to update the topic-specific word embeddings and the topic distribution of the document by maximizing the following Q function:
Tx
Q
Σ
Σ
Ex(= dd weē, wete) log(p(= [d)pwers wxk))
t=1 eSise 440
Σ
Σ
n(d, We, Wi+i) Ep(a wrē, we+s) [log(= |d) + log(p(ww++ 12, wx)))]
Wess) Pa
(14)
Where Pa is the set of skip-gram in document d and ndd Wa, wx++) denotes the number of skip- gram (we, we+s) in document d. For each document d, the prior topic distribution p(> dd) can be updated using the Lagrange multi- plier while keeping the normalization constraints i.e. >. |d) 1.
Σ E(werweted Pa n(d, we, W1+:)p(2 d, wiē, wx+s)
p(2 d)
(15)
E(ma c P3 n(d, W2, Wc++)
The EM negative sampling algorithm in TDE follows a similar workflow as in STE. TDE can eval- uate the likelihood estimation p(we+= 2, wyx) for each Siip-gram (w), w(++)) As result, each topic represents by the ranked list of the bi-grams, similar to STE. As the document vector is learn- ing simultaneously with word embeddings by averaging the word vectors, the time complexity of p(We+s =, wxã) is thus |A A| where AA| is the size of vocabulary in the document and K is the number of topics, which is the same as STE.
For each word w a given new document d', the posterior topic distribution infers by considering the topic distribution the context words w. and the document d' while keeping the values of the learned input and output projection matrices U and VT respectively unchanged. As instead of the center word w itself, we use pivot wzē. Therefore, using the Bayes rule:
p(2 wu Wey d') x log(z dd) + log p(w, 2 wē, d')
(16)
where we is the set of the context words of word w, and wi the pivot created by concatenating global context ž with the center word vector w. The log-likelihood term log Dog((ws (w. z, wä,, d') can define as the sum of log bgg((w/ wē, 2), and log(z d') the corresponding prior probability. The
probability log W. wē, 2) can be computed using E
The output document vector creates directly using 6g without imposing an inference method for the document separately.
5

---

Under review as a conference paper at ICLR 2022
Algorithm TDE using EM negative sampling
1: Initialize Input and Output projection matrix U and V respectively and the topic distribution of the document p22 dd).
2: for out_iter = 1 to Mar_Out.te Satt__tt_iterr do
3: for each document d in do
4: for each word in U do
5: Create document vector (global context) & using Eq and Eq
6: end for
7: for each skip-gram (wy, ((+i) in do
8:
Sample negative instances from the distribution P.
9:
Construct pivot w,ã using Eq[4
Update likelihood function p We Wxē, 2) using Eq 7 and posterior topic distribution
2l d, wtē, Wt++) using Eq 13
11:
for in.iter = 1 to Maz in iiter do
Update projection matrix U and VT using gradient decent method with Eq 10 and Eq [esppeevsttt
13:
end for end for
14:
15:
Update topic distribution p(> (d) using Eq end for
16:
17:
end for
/* Creating document vectors using test dataset */
18: for each rest_document d' in D' do
19: for each word w in updated Uw. do
Construct the output document vector dvee using Eq 21: end for
end for
Table I: Document classification using linear SVM (libliner-2.30) classifier
Models
Number of Train docs
Number of Test does
Accuracy (%)
Error rate (%)
50,000
50,000
75.00
25.00
TDE
100,000 (all)
100,000 (all)
87.50
12.50
50,000
50,000
72.29
27.71
Dec2VecC
100,000 (all)
100,000 (all)
87.32
12.68
4 EXPERIMENTS AND ANALYSIS
The purpose of this research is to implement simple but effective document embedding method. This section is dedicated to demonstrate detailed analysis of the experiments and obtained results. To analyze document embeddings obtained by the proposed model (TDE), we use a linear SVM classifier to check the performance the document classification problem. We also extend our analysis to topical word embeddings efficiency check we proposed to utilize document vector as a global context in the process of jointly learning word-topic embeddings.
4.1 DOCUMENT CLASSIFICATION FOR DOCUMENT EMBEDDING EVALUATION
Doc2VecC doesn't consider topic-specific words in its document embeddings; it is thus in the same embedding space as the word. On the other hand, TDE represents documents into higher (number of clusters x number of word embeddings We use subset of the Wikipedia dump (Chen,2013) which contains over 300,000 Wikipedia pages 100 categories. representation. For both models, we set embedding size to 400, window size to 10, negative sam- pling to 8, subsampling frequent words to 10e-4, number of threads to 10, sentence sample to 0.1, and choose oblinear-2.330
fewadddingssaaattuieoooe features) dimensional space.
our experiments both TDE and Doc2VecC learn 100,000 unique documents
linear SVM classifier (Fan et al. 2008). for TDE set number of topics to 10, inner and outer iteration to 15. Table [able[[shows both TDE and Doc2 VecC (base model) perform better classification seen data than unseen data. For TDE and Dec2. gave the

---

Under review as a conference paper at ICLR 2022
first 10,000-ddcument vectors train set; and the last 50,000 document vectors as test set, where TDE gives 75% accuracy with a 25% error rate and Doc2VecC exhibits 72.29% 72.29% accuracy with 27.71% error rate. When provide all 100,000 document vectors for both train and test, TDE gives 87.5% accuracy with a 12.5% error rate, and Doc2VeeC shows 87.32% accuracy with a 12.68% emorr rate. In T highlighted cells represent better scores obtained by TDF.
4.2 WORD EMBEDDING EVALUATION
For word embedding evaluation, we use Stanford Contextual Word Similarity (SCWS) data set (Huang al. 2012). SCWS dataset contains 2003 word pairs; most of them are monosemous words, only 2wo-word pairs containing the same term within a word pair among them only 50 have a similarity score less than 5.0. That indicates small number of challenging polysemous words in the 20Newsgrup
the SCWS dataset. To obtain word embeddings for both STE-diff and TDE, dataset to train the models. For both models, we set embedding size to 400, number topics to 20, window size 10, negative sampling 8, subsampling frequent words to 10e-4, number of threads to 10, both inner and outer iteration to 15, and minimum word count to 5. We take 1693 word pairs under consideration from the SCWS dataset (commonly found in the 20Newsgrup and SCWS datasets) for Spearman's is rank correlation coefficient measures between models scores (achieved by TDE and STE-diff) and the human judgments of similarity scores in the SCWS dataset. For each pair of words, pairwise cosine similarity uses for the word similarity evaluation of the base (STE- diff) and the PTDD)sed(TDE) model; for both models, we use two variants of similarity scoring- AgvSimC and MaxSimC (Shi et al. 2017). Table 2 exhibits obtained scores by the base and the
Table 2: Spearman correlation (p x 100)
Model
Similarity Metrics
p 100
Table
PMI scores
topic coherence
MaxSimC
39.22
Model
T=5
T=10
T=15
T=20
TDE
AvgSimC
39.04
TDE STE-diff
-69.36
-331.74
-729.65
-139..66
MaxSimC 31.56
-85.23
-374.84
-875.77
-1599.42
STE-diff
AvgSimC
33.28
proposed model under the conditions mentioned above. According to Table 2 TDE obtains almost similar scores for MaxSimC (39.22) and the AvgSimC (39.04) whereas, in STE-diff, MaxSimC gets 31.56 and AvgSimC gets 33.28. Highlighted cells indicate better scores in the table obtained by TDE.
4.3 WORD EMBEDDING VISUALIZATION
To visualize the interaction between words and latent topics in the embedding space, we use the Wikipedia dump from April 2010 (Shaoul 2010), similar to STE. For both base (STE-diff) and pro- posed (TDE) models, we set embedding size to 400, number of topics to 10, window size to 10, negative sampling 8, subsampling frequent words 10e-4, number of threads to 10, the inner and the outer iterations to 15. For visualization in Fig 3 and Fig 4 we use the t-SNE algorithm (Van der Maaten & Hinton 2008), where each node represents topic-specific word vector. For both models, we set perplexity to 40, n_iteration to 2500, and random state to 23. We consider the word party with two different meanings political party and social gathering as a polysemous word for visualization. In Fig 3, party#9 is close to government#9 indicating political party but party#7 is far from governmentiis, our extensive analysis found party#O, party#1, p2rty#2, pa and p9 contain governmenti99 and governmentiO within topmost 100 similar words, whereas party#3,, party#4, p5 #7, party#8 are close to word summers#5, indicating gathering and celebration. These words contain summer#5 in the topmost 50 similar words.
In Fig 4 It seems party and government exist little scattered in the embedding space for STE-diff than TDE; it indicates learning better embeddings by TDE over STE-diff. Our extensive experi- ment found, only #6 contains government#6 within the topmost 100 similar words, whereas summer#4 exists in the most 150 similar words of p4rty#4 only.
7

---

Under review as a conference paper at ICLR 2022
Topic
-10
Topic
topic
Topic
Topic
-15
Topic
Topic
Topic
Topic
-20
-10
Figure 3: TDE polysemous word em- bedding visualization.
Figure 4: STE-diff polysemous word embedding visualization.
Table 4: TDE topics generating bigrams for 20Newsgrups dataset
god heaven
image files
mr president national security
jesus died
open file
believe god
file format
national institute
jesus god
file server
president clinton
jesus christ
file manager
clinton administration
father son
image processing
senior administration
father jehovah
file size
national center
god father
file stream output file
press conference
jehovah elohim
bill clinton
christ god
postscript file
press release
alt.atheism
comp.graphics
talk.politics mise
soc.religion.christinn
crmppossmsswindowss
aalk.politics.gue
talk.
comp.sys.ibre ....i.pp..aarriee pe.hardware
aakkpolitics.mider
comp.sy crnp.sys.m.. www.smac...aaaa..
comp. windwws
4.4 TOPIC COHERENCE
In TDE, pointwise mutual information (PMI) scores are using to evaluate topic coherence as PMI scores strongly correlate with human annotations of topic coherence (Newman al.] 2010 Lau
et al. 2014). For this experiment, we use the 20Newsgrup dataset. The parameter setting for both models base (STE-diff) and proposed (TDE) are the same as mentioned in section 4.2. Table exhibits overall PMI scores for different numbers of top words (T) under 20 topics. The highlighted cells in the table indicate higher coherence, as we obtained negative coherence scores so, the smaller better.
Topics in TDE can be interpreted by bigrams words as shown in the publication (Shi et al,, 2017)) for STE-same. In our experiments, found for STE-diff, each topic represents bigrams randomly
8

---

Under review as a conference paper at ICLR 2022
Table 5: Multi-class document classification for word embedding evaluation.
Model
Accuracy
Precision
Recall
F-measure
PV
75.4
74.9
74.3
74.3
STE-same 80.4
80.3
80.4
80.2
TDE 80.5
80.5
80.5
80.3
STE-diff
82.9
82.5
82.3
82.5
from various categories without thematic grouping of bigrams, whereas TDE maintains thematic gathering of bigrams. Table 4 represents three topical representations of bigram words with their source categories for TDE. The first column represents religion-related topic, the second is related to the computer-related topic, and the third is related to politics.
4.5 DOCUMENT CLASSIFICATION FOR WORD EMBEDDING EVALUATION
In this experiment, we create document vectors by TF-IDF weighted averaging each word em- beddings under every document, using equation (17). For both models, the base (STE-diff) and the proposed (TDE), we use 20Newsgroup dataset with the same parameter settings mentioned in section 4.2.
du tfidf E(( Iw C),
(17)
where Un.z is the learned input projection matrix, represents context words, wi represents pivots (concatenating center words with the global document context T), and ifidff the TF-IDF score of w in the document.
For document classification in this section, we use several standard evaluation metrics for the multi- class document classification tasks, c.g., Accuracy, Precision, Recall, and F-measure for LinearSVM classifier. We compare scores obtained by TDE with the obtained scores by PV, STE-diff, and STE- same models published by (hhi et al. (2017). Table 5 represents obtained scores using evaluation metrics in the classification task, where TDE beats STE-same and PV, but STE-diff performs better in this case. TDE achieves better topic coherence than STE-diff, but STE-diff achieves better doc- ument classification, which we see a similar analogy between STE-same and STE-diff in Shi et al. 2017 work.
CONCLUSION
TDE:
Topical Document Embedding is jointly learning word-topic and document embeddings method, which learns document embedding on the fly simultancously with the jointly learning pro- cess of the topic-specific word embeddings by taking document vector as global context. As result, TDE captures better syntactic and semantic properties by utilizing word, topic, and document- level information. poses the simplest form of document embedding without imposing complex inference methods for document vectors. As such, each document vector is created by a simple weighted averaging of the topic-specific word embeddings, which is effective regardless of doeu- ment size. Moreover, using the mask-out/drop-onn corruption mechanism enhances learning speed and ensures quality embeddings as well. Our extensive experimental findings reveal that besides the document embeddings, the proposed joint learning method also captures better topical word embeddings over the base model.
REPRODUCIBILITY STATEMENT
Model source code written C programming language, to run the code need to type ./go.sh in the terminal (optional: for creating make file type make CFLAG="-Im -pthread -03 -Wall -funroll- loops"). Model parameters can be reset in this file (e.g., training dataset). All datasets used in this research available in preprocessed form (use ready for the model) in the data folder of the package (Model sourceCode Datasets). The model output should be reproduced effortlessly a I Linux machine with GCC C++ compiler installed.

---

Under review as a conference paper at ICLR 2022
For model evaluaiton, we have added evaluation source code files ste iintalize.iyyb.... The evaluation procedure requires python-2.7, numpy-1.16.1, scikit-leam-0 0.17.. For convenience, also included obtained evaluation results and detail analysis for embedding visualization and pearsons correlation the obtained results subfolder for both base and proposed models.
and
REFERENCES
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. simple but tough-to-beat bascline for sentence embeddings. ICLR, 2017.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic struc- ture of word senses, with applications to polysemy. Transactions of the Association for Compu- tational Linguistics, 6:483-495, 2018.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The journal of machine learning research, 3:1137-1155 2003.
David M Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022, 2003.
DevendkKnowledge-base word sense disambiguation using topic modelsa Singh Chaplot and Rus- lan Salakhutdinov. Knowledge-berd word sense disambiguation using topic model. In Thirty- Second AAAI Conference on Artificial Intelligence, 2018.
Minmin Chen. Efficient vector representation for documents through corruption. ICLR, 2017.
Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. Liblinear: A library for large linear classification. the Journal of machine Learning research, 9:1871-1874, 2008.
Vivek Gupta, Harish Karnick, Ashendra Bansal, and Pradhuman Jhala. Product classification in e-commerce using distributional semantics. 17Xiv:6606.00883, 2016.
Vivek Gupta, Ankit Saw, Pegah Nokhiz, Harshit Gupta, and Partha Talukdar. Improving document classification with multi-sense embeddings. arXiv:1911.07913 2019.
Junxian He, Zhiting Hu, Taylor Birk-Kiikp,,,icc, Ying Huang, and Eric P Xing. Efficient correlated topic modeling with topic embedding. Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 223-233,20 2017.
Thomas Hofmann. Probabilistic latent semantic analysis. 1301.66300..700 2013.
Eric H Huang, Richard Socher, Christopher Manning, and Andrew Y Ng. Improving word rep-
resentations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, volume 1, pp. 873-882, 2012.
Jey Han Lau, David Newman, and Timothy Baldwin. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 530-539, 2014.
Quoe Le and Tomas Mikolov. Distributed representations of sentences and documents. In Interna- tional conference on machine learning, pp. 1188-1196 PMLR, 2014.
Qi Liu, Matt J Kusner, and Phil Blunsom. A survey on contextual embeddings. arXiv:2003.07278, 2020.
Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Topical word embeddings. In Twenty- ninth AAAI conference on artificial intelligence, pp. 2418-2424, 2015.
Junyu Luo. Min Yang, Ying Shen, Qiang Qu, and Haixia Chai. Learning document embeddings with crossword prediction. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33(01), pp. 9993-9994, 2019.
10

---

Under review as a conference paper at ICLR 2022
Dheeraj Mekala, Vivek Gupta, Bhargavi Paranjape, and Harish Karnick. Scdv: Sparse composite document vectors using soft clustering over distributional representations. arXiv:1612.0677, 2016.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv:1301 1301.3811, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 3111-3119, 2013b.
Christopher E Moody. Mixing dirichlet topic models and word embeddings make Ida2vec. arXiv:1605.02019 2016.
David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic coherence, In Human language technologies: The 2010 anmual conference of the North American chapter of the association for computational linguistics, 100-108, 2010.
Liqiang Niu, Xinyu Dai, Jianbing Zhang, and Jiajun Chen. Topic2vec: learning distributed repre- sentations of topics. In 2015 International conference on asian language processing (IALP), pp- 193-196. IEEE, 2015.
Cyrus Shaoul. The westbury lab wikipedia corpus. Edmonton, AB: University of Alberta, 131, 2010.
Bei Shi, Wai Lam, Shoaib Jameel, Steven Schockaert, and Kwun Ping Lai. Jointly learning word embeddings and latent topics. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 375-384, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Yin Zhang, Rong Jin, and Zhi-Hua Zhou. Understanding bag-of-words model: statistical frame- work. International Journal of Machine Learning and Cybernetics, 1(1-4):43-5, 2010.
He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. Neural topic model via optimal transport. ICLR, 2021.


Reviews:
[{"review_id": "kRJdtJqLdb", "reviewer": "Reviewer_Nesz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths\n1) This paper is well written and easy to read.\n2) TDE follows the previous work and aims to learn three different level embeddings.\n3) This paper shows visualization of the interaction between words and latent topics in the embedding space.\n\nweaknesses\n1) Approaches are straightforward and lack originality. This approach is simple combination of  Doc2Vec and STE.\n2) Baselines are old and weak in now.\n3) The experimental results reported are validated on a single dataset,\nand no human evaluation and error analysis.", "summary_of_the_paper": "This paper proposes Topical Document Embedding, TDE,  to obtain document vectors on during the jointly learning process of the topical word embeddings,\nwhere it captures syntactic and semantic properties by utilizing three levels of knowledge (i.e., word, topic, and document).\nExperiments demonstrate better topical word embeddings using document vector and better document classification results on the obtained document embeddings by the proposed method over the recent related models.", "summary_of_the_review": "Now, the mainstream of representation learning is attention based models or Transformer based models, \nso we would like to see a comparison with these and discussion.\nFor example, we use BERT and then get a similar expression from [CLS] token.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "_ggUiv4vhXA", "reviewer": "Reviewer_a6Hd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ introduced a single framework that jointly learns topically-aware word embeddings and document vectors where topic-embeddings are learning using global content and each word embedding can be part of different underlying topics of the document. On top, document vectors are computed by averaging topic word embeddings. \n+ clear problem statement \n+ paper is well written  \n+ sound qualitative evaluation of word embedding visualization \n\nWeaknesses:\n- missing empirical comparison to related works in compositional models [1, 2, 3] of jointly learning topic and word embeddings \n- experimental setup is unclear for document classification task: How is classification task performed? First LM-based objective then using the document representation into classification?  \n- related works section is weak, missing several related works [1, 2, 3]\n- quantitative evaluation is limited (missing key baselines and requires additional data sets in comparative analysis) \n- Document classification is performed on a single (large) data set.  Would be interesting to validate the methodology using at least 2-3 datasets and in sparse data settings. \n- Document classification should be further evaluated on standard data sets for example 20newsgroups, Reuters, TMN, etc.\n- Empirical evaluation is incomplete for document and word embedding evaluation: missing comparison with several strong and most related baselines such as [1, 2, 3] etc. that jointly learns topic and word embeddings\n- please include a case study and/or comparison/differences to related works [1, 2, 3] in terms of:\n\t- jointly learning topic and word embeddings in a single framework \n\t- captures both the syntactic and semantic linguistic structure \n\t- combines both local and global contextual information from word, topic and document\n\t- incorporates both: topic and word embedding spaces in language modeling based training objective (loss function)\n\nQuestions:\n1. Since the Document vector is computed by averaging its word vectors, therefor the averaging scheme may not consider the syntactic structure e.g. word ordering. However, the methods such as [1, 2] employs LSTM to compute document vectors and globsl context considering linguistic structures.\n2. In equation 3, does the global context \\~{x} include w_{i+1} ? Unclear. \n3. In qualitative evaluation of topics, why are the topics of bigram words only? Please also include qualitative topics analysis of unigram words?\n\nMissing references:\n1. Gupta et al., 2019. textTOvec: DEEP CONTEXTUALIZED NEURAL AUTOREGRESSIVE TOPIC MODELS OF LANGUAGE WITH DISTRIBUTED COMPOSITIONAL PRIOR. ICLR 2019.\n2. Lau et al., 2017. Topically driven neural language model.  ACL 2017.\n3. Dieng et al., 2016. TopicRNN: A recurrent neural network with long-range semantic dependency.", "summary_of_the_paper": "This paper focuses on learning document embeddings, presenting a topic-document embedding (TDE) method employing syntactic and semantic properties by jointly learning topic and word embedding in a single framework. The proposed TDE approach follows corruption mechanism to create the global context and randomly select topic-specific word embeddings in learning document vectors, thus employing word, topic and document information in learning document vectors.\n\nThe experimental results have shown improved performed in terms of document classification, topic coherence and word embedding similarity evaluations.", "summary_of_the_review": "- quantitative evaluation is weak \n- missing related works and quantitative comparison \n- incremental work", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Rj1rLCTQi7I", "reviewer": "Reviewer_PLvo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed Skip-gram model contains two major parts that are different from the standard Skip-Gram model:\n\n* One is to assume each word has a different representation under each individual topic. In such a way, the topic assignment of a word can be factored in the computation of the skip-gram probability. In other words, the probability of a context word will depend on not only the centre word but also their topic assignments. However, this idea is from the Skip-gram topical embeddings by Shi et al, 2017. \n\n* The other is the use of a document vector as the global context, which originates from Chen 2017. The proposed model simply injects this context vector into the Skip-gram topical word embedding.  If one compares eq (4) in Shi et al, 2017 and eq (3) of this paper, the only difference is the term $\\frac{1}{T} U_{\\tilde{x}_d}$, constructed similarly to Eq (2) of Chen 2017. This simple modification makes the learning algorithm naturally follow the algorithm design discussed in Section 3.2 of Shi et al, 2017.\n\nThere is nothing wrong with combining the idea of two models together. However, it makes one have doubt about the innovation of this work unless the proposed model is very effective in terms of performance, which is not really the case.\n\nIn regards to the experiments, the proposed model is compared mainly compared with the models in Shi et al, 2017. The experimental methodology follows closely that used by Shi et al, 2017, i.e. document classification, topic coherence, word embedding. The classification performance is only marginally better than STE-same, much worse than STE-diff. Meanwhile, the paper claims that the proposed model is capable of dealing with polysemy, which however is only insufficiently demonstrated by Figures 3 and 4.\n\nThe writing of this paper could be further improved, particularly those notations.\n\nBesides, I have the following questions\n* Should the two project matrices be in $\\mathcal{R}^{k \\times s}$?  If that is the case, most likely, then it column is not a word embedding. I understand if you follow the notation of Chen 2017, then the column should be word embeddings, However, the equations tell us that U has a topic dimension. Let me know if I was wrong.\n* By Eq (4), do the authors mean that the skip-gram probability conditions on both $w_t$ and $\\tilde{x}$? And presumably, the first probability of the left term in Eq (5) is computed based on Eq (3). \n* Eq (1): $\\tilde{x}$ is scalar. The sum in Eq (2) will be scalar, won't it? According to Chen 2017,  Eq(1) will generate a word masking vector for  a document. The masking vector is then used to generate the document representation via averaging. Right?\n\n\n", "summary_of_the_paper": "Jointly learning word embeddings and document embeddings while taking into account topical information is somewhat interesting. The paper presents an improved skip-gram model where each individual word is associated with two matrices (e.g., the input projection matrix and the output projection matrix) which is designed to capture the topical aspect of words. In other words, each word is associated with K topic embeddings. While computing the predictive probability, the proposed model further considers a global context, i.e, a document embedding computed with the idea of document corruption. Basically, this paper simply combines the work of Shi et al. 2017 and Chen 2017, Both the innovation and experimental improvement are mariginal.", "summary_of_the_review": "As discussed above, the proposed method simply inject the idea of the global context in Chen 2017 into the Skip-gram topical embeddings by Shi et al, 2017, which is very incremental. Even regardless of that, the performance gain of the proposed model is very marginal, even worse than STM-diff. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "O27v9ETPQXs", "reviewer": "Reviewer_iEk8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The proposed TDE method is just a simple mix of the ideas of Doc2VecC proposed by (Chen 2017) and STE proposed by (Shi et al., 2017). \n\n- Although the paper clearly explains the motivation and the related work section has a good collection of references on conventional word embeddings and document embeddings, most references (including two baselines) are publications of 4 or 5 years ago and completely misses recent major progresses of contextual sentence/document embeddings since ELMO and BERT.\n\n- What's the advantage of the proposed TDE over the recent contextual embedding methods for representation of sentences and documents? \n\n-  Both Doc2VecC (Chen 2017) and STE (Shi et al., 2017) conducted extensive experiments on the quality of the embeddings and performance on the downstream application of document classification. But this paper only presents very limited experiments and is lack of clear interpretation of experimental results. For example, there are only 2 base lines (STE and Doc2VecC) studied in the experiments, but why STE is still missing Table 1, and Doc2VecC is missing in Table 5? What's the key observations and insights from Table 4?\n\n- It seems there is lack of basic check and proof reading\n    - Typos: for example, \"Tow popular complementary\" in Page 1 and \"flowing objectives\" in Page 1\n    - error on major equation: for example: I guess “d/1-q” should be \"x_d/1-q\" in equation (1) \n    - Missing definition: for example, \"PV\" in Page 1 and other pages (I guess \"PV\" means \"paragraph vector\")\n    - Poor Notations and writings, hard to read and follow, for example, section 3.1\n", "summary_of_the_paper": "- This paper presents a document-embedding method called Topical Document Embedding (TDE) which utilizes Document Vector through Corruption (Doc2VecC) proposed by (Chen 2017) as the basic framework and combines the idea of Skip-gram Topical word Embedding (STE) proposed by (Shi et al., 2017) for further enhancement.\n\n- The authors claimed that the proposed TDE demonstrates better topical word embeddings using document vector as a global context and better document classification results on the obtained document embeddings.\n", "summary_of_the_review": "In general, this paper has not been carefully checked and well organized, and the experimental evaluations are not solid. This paper needs some major updates for future improvements.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strength
* The paper is relatively clearly written.
* A new method is proposed.

Weakness
* The evaluation is weak.  The experimental setup is not clear enough.  More quantitative evaluation is necessary. There are strong and new baselines that need to be compared with.
* Relation with existing work needs to be more clearly described.
* The novelty of the work is limited.  It is a combination of existing methods.
* Justification of the proposed method needs to be provided.
* The writing of the paper can be improved.
Decision: Reject