Year: 2021

Abstract:
Document embedding generally ignores underlying topics, which fails to capture polysemous terms that can mislead to improper thematic representation. Moreover, embedding a new document during the test process needs a complex and expensive inference method. Some models first learn word embeddings and later learn underlying topics using a clustering algorithm for document representation; those methods miss the mutual interaction between the two paradigms. To this point, we propose a novel document-embedding method by weighted averaging of jointly learning topic-specific word embeddings called TDE: Topical Document Embedding, which efficiently captures syntactic and semantic properties by utilizing three levels of knowledge -i.e., word, topic, and document. TDE obtains document vectors on the fly simultaneously during the jointly learning process of the topical word embeddings. Experiments demonstrate better topical word embeddings using document vector as a global context and better document classification results on the obtained document embeddings by the proposed method over the recent related models.

heading: 1 INTRODUCTION
text: In recent years the neural network-based methods have shown great success in various NLP-related tasks (Zhao et al., 2021). Tow popular complementary natural language processing methods are word2vec (Mikolov et al., 2013a;b) and Latent Dirichlet Allocation (LDA)(Blei et al., 2003); but executing a model in two-step lacks mutual interaction between the two paradigms and fails to capture mutual reinforcement between themselves. Doc2vec (Le & Mikolov, 2014) generalizes the idea of vector representation for documents as a stand-alone method. Though it is expensive to infer new document vectors during the test process, and regardless of assigning PV to each word in the document as a single topic, it outperforms BOW (Zhang et al., 2010) and LDA (Fan et al., 2008) in various text understanding tasks. Encouraged by word2vec and Doc2vec, Niu et al. (2015) introduced Topic2Vec. It incorporates the LDA topic model into NPLM (Bengio et al., 2003). However, Topic2Vec cannot capture the polysomic characteristics of terms as LDA models topic occurrence almost independently without capturing topic correlations (He et al., 2017). The purpose of conducting this research is to introduce an efficient multi topical document representation with flowing objectives:
• Firstly, learn probabilistic topical word embeddings using document vector as a global context where each word can be part of different underlying topics of the document instead of just a single paragraph vector. Thus, avail to capture polysemous terms. • Secondly, embedding documents by simply averaging the topical word embeddings instead
of imposing an expensive inference method to learn the document embedding separately.
Unlike clustering-based document modeling with the ability to capture multi-sense of words (Gupta et al., 2019), here the challenge is to learn polysemous words in a jointly learning framework for topic-specific word and document embedding. Capturing polysemous terms is vital because the same word around different context words can represent different themes (Liu et al., 2020) and without noticing them can mislead to an unexpected thematic representation of a document. The Jointly Learning Word Embeddings and Latent Topics (Shi et al., 2017) comes to the rescue for addressing the polysemous word problem, which learns different topical word embeddings using
Skip-gram Topical word Embedding (STE) framework. STE gets the term distributions of the topics and the topic distributions of the document using PLSA(Hofmann, 2013) like mixture weighting for each data point. However, STE doesn’t deal with document embedding in its jointly learning process. In this research, we propose a novel document embedding framework called TDE: Topical Document Embedding. TDE learns topical word and document embeddings jointly. It incorporates the STE-diff framework (Shi et al., 2017) and follows the simple averaging of word embedding with a corruption mechanism for document embedding introduced by Chen (2017).

heading: 2 MOTIVATION AND RELATED WORK
text: Huang et al. (2012) proved that combining local and global context performs better for predicting target words in the neural network-based language modeling, but this model doesn’t support topic-specific word embeddings. Liu et al. (2015) proposed topic-specific word embeddings where document embeddings get by aggregating over all topical-word embeddings of each word in the document. Arora et al. (2017) extend embedding to the sentence level by smooth inverse frequency (SIF) weighted averaging of the words in a sentence. However, this model is lack of addressing multiple sense of words. Moody (2016) introduced a hybrid model called lda2vec, which suffers from uncorrelated topics due to the incorporation of LDA. For word sense disambiguation with correlated topic modeling, Chaplot & Salakhutdinov (2018) replaced the LDA topic proportions by synset proportions for a document. Other than LDA topic modeling, Arora et al. (2018) have introduced a Gaussian random walk model for word sense disambiguation. This work can address polysemous word representation but doesn’t cover document-level embeddings. Mekala et al. (2016) introduced the Gaussian mixture model-based soft clustering method called SCDV, which incorporates both the topic and word embeddings. However, SCDV is not suitable for multisentence documents. Hence Luo et al. (2019) proposed P-SIF, which is fit for large documents with multiple sentences while addressing the underlying topics as well. Though P-SIF overcomes SCDV’s shortcomings, it is still a two-step process. From the motivation of the above advantages and drawbacks, we introduce a novel document modeling capable of learning document embedding simultaneously during the jointly learning process of topic-specific word embeddings. This model creates a pivot by concatenating a document vector and center word vector to predict the context word vector. Here the document vector works as a memory backup for missing information. The output document embeddings produced by this model is more expressive as it represents each document into a higher dimensional embedding space similar to P-SIF.

heading: 3 MODEL DESCRIPTION
text: Inspired by the Skip-gram Topical word Embedding (STE) (Shi et al., 2017) and the simplicity of the Document Vector through Corruption(Chen, 2017), we design a novel document embedding model called TDE: Topical Document Embedding. In contrast to Doc2VecC, TDE constructs global context (document vector) as a weighted averaging of the randomly selected topic-specific word embeddings. Unlike STE, TDE uses a generating function, which predicts context words by given the topics and pivot. Pivot constructs by concatenating global context (document vector) and center word (Figure1). The probability of predicting a context word depends on the inner product of the topic-specific embedding of that word (context word) and the pivot, which ensures the participation of underlying word, topic, and document levels knowledge.

heading: 3.1 TOPICAL DOCUMENT EMBEDDING
text: A matrix D represents the documents in the corpus, D = d1, d2, . . . , dn of size n. Each document d contains a variable-length sequence of wordsW = w1, w2, . . . , wTd . Each wordw in the vocabulary V of size v of the corpus is generally associated with an input projection matrix U Rh×v of size h and an output projection matrix V T Rv×h of size h. The input projection matrix represents the input to the hidden space, whereas the output projection matrix represents the hidden to output space. Matrix V T has the same dimension as matrix U but a transpose matrix. The column uw of matrix U is the input vector of word w , and the column vw of matrix V T is the output vector of word w.
TDE follows an unbiased mask-out/drop-out corruption mechanism in each update of its stochastic process for randomly select topic-specific word embeddings to create the global context. Under corruption mechanism, each dimension of the original document d overwrites with the probability q and sets the uncorrupted dimensions 1(1−q) times of their original values to make the corruption unbiased. The corruption mechanism reduces word-level noise in the document by forcing the embeddings of common and non-discriminative ones close to zero. This corruption mechanism leads TDE to speed up the training process by significantly pruning the number of parameters for the backpropagation mechanism, document vectors represented by only embeddings of remaining words. The resultant global context is thus as below:
x̃d =  0, with probability q
d 1−q , otherwise
(1)
Document vector (global context) x̃ = 1
T
∑ x̃d (2)
where T is the length of the document (number of words) and the document vector x̃ is the weighted average of the words in a document, weighted by the corruption mechanism mentioned above. The probability of observing the context word wt+i given the center word wt as well as the global context x̃ and the topics zt+i and zt for context and center word respectively as
p(wt+i |wt, zt+i, zt, x̃) = exp
( V Twt+i, zt+i . (Uwt, zt + 1 T Ux̃d) ) ∑ w′ Λ exp ( V T w′ , zt+i . (Uwt, zt + 1 T Ux̃d)
) (3) where Λ is the vocabulary of the whole corpus and input projection matrix constructs as U = Uwt, zt + 1 T Ux̃d .
The denominator in equation (3) is generally intractable due to the term ∑ w′ Λ. As such, the computational cost is proportional to the size of Λ. To deal with this intractability requires an efficient approximation method. TDE employs the approximation with the negative sampling (Mikolov et al., 2013a; Chen, 2017). In TDE, the probability of predicting a context word wt+i in document d depends on the pivot wtx̃ and the topic distributions of the document. The pivot constructed by concatenating the center word wt and the global context x̃ as-
Pivot = wtx̃ = wt + x̃ (4)
So, for every skip-gram 〈wt, wt+i〉, where the topic assignment zt+i of wt+i is independent of the topic assignment zt of wt we have :
p(wt+i |wtx̃, d) = k∑
zt=1 k∑ zt+i=1 p(wt+i | wtx̃, zt+i, zt) p(zt+i, zt |d)
= k∑ zt=1 k∑ zt+i=1 p(wt+i |wtx̃, zt+i, zt) p(zt|d)p(zt+i |d)
(5)
Following a similar analogy as STE-diff, equation (5) assumes that the context word and the center word may come from the same topic or may come from two different topics as well without imposing any hard constraint (Figure 2). Here, multiple meanings of a word depend on their contexts instead of directly k topics disjoint subspace partitioning as commonly does in clustering-based topic modeling(Gupta et al., 2016). In TDE, word embeddings from different topics co-exist in a common shared space. Instead of the free flow of scattered words in the embedding space, different topical word embeddings stay close together document-wise rather than topic-wise gathering. Creating pivots by concatenating the center word with the document vector provides extra guidance as a global context with a complete thematic sense of the document. Thus, TDE represents more consistent topical word embeddings over STE-diff. In TDE, the compositionality of words encodes in the learned embeddings; as a result, the requirement of heuristic weighting for test documents is no more required; each output document vector is represented simply as an average of the topic-specific word embeddings. Given the learned projection matrix Uw,z , TDE forms document embedding dvec as below:
dvec = ⊕kz=1 1
T ∑ w D Uw,z (6)
where T is the length (number of words) of the document and k represents the underlying topics of the document. In equation (6), the term ⊕kz=1 represents topic-wise concatenation. In this case, words in the same topic are just simple average as they represent the same thematic meaning. However, words in different topics carry different themes. Thus, concatenate over topics increases the expressive power of a document. This strategy represents documents into a higher k × s dimension, where k is the underlying topics, and s is the dimension of the word embeddings.

heading: 3.1.1 ALGORITHM DESIGN
text: The inference framework of TDE (Algorithm 1) is similar to STE-diff, based on the EM algorithm with negative sampling (Shi et al., 2017). In each iteration, it updates the word embeddings first, then based on updated word embeddings, updates the document embedding (global context) x̃. Finally, evaluate the topic distribution p(z | d) of each document. To update word embeddings, iterates over each skip-gram 〈wt, wt+i〉 using negative sampling and evaluate the posterior topic distribution. So, the estimation of the probability of predicting the context word wt+i given pivot wtx̃ and topic assignment z denoted by p(wt+i |wtx̃, z) as follows:
log p(wt+i|wtx̃, z) ∝ log σ ( V Twt+i, z . (Uwt, z + 1
T Ux̃d) ) +
n∑ i Ewi∼p [ log σ ( −V Twi, z . (Uwt, z + 1 T Ux̃d )] (7)
where σ(x) = 11+exp(−x) and wi is a negative instance, which is sampled from the unigram distribution p(.) over terms in the vocabulary and has a value similar to Gupta et al. (2019)’s findings. The overall training objective is how well the model can predict surrounding words, which is for each document d, given a sequence of words w1, w1, . . . ., wTd , then the log-likelihood Ld is defining as
Ld = Td∑ t=1 ∑ −c≤i≤c; i 6=0 log p(wt+i |wtx̃, d) = 0 (8)
where c is the size of the training windows. The overall log-likelihood L is then
L = ∑ d Ld (9)
The gradients of the objective function to evaluate log p(wt+i |wtx̃, d) for each term concerning the input and the output projection matrix U and V T respectively can formulate as follows:
∂L Uwt x̃, z
= − ( ξw′wt − σ(V T w′ ,z Uwtx̃, z) ) . V T w′ ,z . p(z | d, wtx̃, w ′ ) (10)
∂L V T w′ ,z
= − ( ξw′wt − σ(V T w′ ,z Uwtx̃, z) ) . Uwt x̃,z. p(z | d, wtx̃, w ′ ) (11)
where
ξw′wt =  1, if w ′ in the contextwindow of wt
0, otherwise (12)
In TDE, the update rule of U and V T maximizes p(wt+i|wtx̃, z), thus log p(wt+i|wtx̃, z) for each skip-gram 〈wt, wt+i〉 instead of maximizing log p(wt+i |wt, z) as STE does. This observation shows both of the models flow the same procedure. STE has proved capable of discovering good topics and identifying high-quality word embedding vectors. Thus, TDE inherits all the prominences; moreover, it keeps Doc2VecC’s simplicity for document embeddings and extends reliability by filling up essential information gaps using global document context. When concerning topics for the embedding process, the generally used hierarchical Softmax or Negative sampling methods are incapable of obtaining the embeddings efficiently. The EM-negative sampling Algorithm thus comes to the rescue. In the E-step, the posterior topic distribution for each skip-gram 〈wt, wt+i〉 in document d evaluate using the Bayes rule:
p(z ′ k | d,wtx̃, wt+i) = p(wt+i |z
′ k, wtx̃)p(z ′ k | d)∑ z p(wt+i |z, wix̃)p(z | d)
(13)
In the M-step, given the posterior topic distribution, it is now time to update the topic-specific word embeddings and the topic distribution of the document by maximizing the following Q function: Q = ∑ d Td∑ t=1 ∑ −c≤i≤c; i 6=0 ∑ z p(z | d,wtx̃, wt+i) log(p(z | d)p(wt+i | z, wtx̃))
= ∑ d ∑ {wt, wt+i} Pd n(d, wt, wt+i) ∑ z p(z | d, wtx̃, wt+i) [log(z | d) + log(p(wt+i | z, wtx̃))]
(14)
Where Pd is the set of skip-gram in document d and n(d, wt, wt+i) denotes the number of skipgram 〈wt, wt+i〉 in document d. For each document d, the prior topic distribution p(z | d) can be updated using the Lagrange multiplier while keeping the normalization constraints i.e. ∑ z p(z | d) = 1.
p(z | d) = ∑ {wt,wt+i} Pd n(d, wt, wt+i)p(z | d, wtx̃, wt+i)∑
{wt,wt+i} Pd n(d, wt, wt+i) (15)
The EM negative sampling algorithm in TDE follows a similar workflow as in STE. TDE can evaluate the likelihood estimation p(wt+i | z, wtx̃) for each skip-gram 〈wt, wt+i〉. As a result, each topic represents by the ranked list of the bi-grams, similar to STE. As the document vector is learning simultaneously with word embeddings by averaging the word vectors, the time complexity of p(wt+i | z, wtx̃) is thus |Λ|2×K , where |Λ| is the size of vocabulary in the document and K is the number of topics, which is the same as STE. For each word w in a given new document d ′ , the posterior topic distribution infers by considering the topic distribution of the context words wc and the document d ′ , while keeping the values of the learned input and output projection matrices U and V T respectively unchanged. As instead of the center word w itself, we use pivot wx̃. Therefore, using the Bayes rule:
p(z |wx̃, wc, d ′ ) ∝ log(z | d ′ ) + log p(wc | z, wx̃, d ′ ) (16)
where wc is the set of the context words of word w, and wx̃ is the pivot created by concatenating global context x̃ with the center word vector w. The log-likelihood term log p(wc | z, wx̃, d ′ ) can define as the sum of log p(wc |wx̃, z), and log(z|d ′ ) is the corresponding prior probability. The probability log p(wc |wx̃, z) can be computed using Eq.7 The output document vector creates directly using Eq.6 without imposing an inference method for the document separately.
Algorithm 1 TDE using EM negative sampling 1: Initialize Input and Output projection matrix U and V T respectively and the topic distribution
of the document p(z | d). 2: for out iter = 1 to Max Out iter do 3: for each document d in D do 4: for each word w in U do 5: Create document vector (global context) x̃ using Eq.1 and Eq.2 6: end for 7: for each skip-gram 〈wt, wt+i〉 in d do 8: Sample negative instances from the distribution P. 9: Construct pivot wtx̃ using Eq.4
10: Update likelihood function p(wt+i |wtx̃, z) using Eq.7 and posterior topic distribution p(z ′
k | d, wtx̃, wt+i) using Eq.13 11: for in iter = 1 to Max in iter do 12: Update projection matrix U and V T using gradient decent method with Eq.10 and Eq.11 respectively. 13: end for 14: end for 15: Update topic distribution p(z | d) using Eq.15 16: end for 17: end for
/* Creating document vectors using test dataset */ 18: for each test document d ′ in D ′ do 19: for each word w in updated Uw,z do 20: Construct the output document vector dvec using Eq.6 21: end for 22: end for

heading: 4 EXPERIMENTS AND ANALYSIS
text: The purpose of this research is to implement a simple but effective document embedding method. This section is dedicated to demonstrate a detailed analysis of the experiments and obtained results. To analyze document embeddings obtained by the proposed model (TDE), we use a linear SVM classifier to check the performance of the document classification problem. We also extend our analysis to topical word embeddings efficiency check as we proposed to utilize document vector as a global context in the process of jointly learning word-topic embeddings.

heading: 4.1 DOCUMENT CLASSIFICATION FOR DOCUMENT EMBEDDING EVALUATION
text: Doc2VecC doesn’t consider topic-specific words in its document embeddings; it is thus in the same embedding space as the word. On the other hand, TDE represents documents into higher (number of clusters× number of word embeddings features) dimensional space. We use a subset of the Wikipedia dump (Chen, 2017), which contains over 300,000 Wikipedia pages in 100 categories. In our experiments both TDE and Doc2VecC learn 100,000 unique documents representation. For both models, we set embedding size to 400, window size to 10, negative sampling to 8, subsampling frequent words to 10e-4, number of threads to 10, sentence sample to 0.1, and choose liblinear-2.30 as a linear SVM classifier (Fan et al., 2008), for TDE we set number of topics to 10, inner and outer iteration to 15. Table 1 shows both TDE and Doc2VecC (base model) perform better classification on seen data than unseen data. For TDE and Doc2VecC, we gave the
first 50,000-document vectors as a train set; and the last 50,000 document vectors as a test set, where TDE gives 75% accuracy with a 25% error rate and Doc2VecC exhibits 72.29% accuracy with a 27.71% error rate. When we provide all 100,000 document vectors for both train and test, TDE gives 87.5% accuracy with a 12.5% error rate, and Doc2VecC shows 87.32% accuracy with a 12.68% error rate. In Table 1, highlighted cells represent better scores obtained by TDE.

heading: 4.2 WORD EMBEDDING EVALUATION
text: For word embedding evaluation, we use Stanford Contextual Word Similarity (SCWS) data set (Huang et al., 2012). SCWS dataset contains 2003 word pairs; most of them are monosemous words, only 241-word pairs containing the same term within a word pair among them only 50 have a similarity score less than 5.0. That indicates a small number of challenging polysemous words in the SCWS dataset. To obtain word embeddings for both STE-diff and TDE, we use the 20Newsgrup dataset to train the models. For both models, we set embedding size to 400, number of topics to 20, window size to 10, negative sampling to 8, subsampling frequent words to 10e-4, number of threads to 10, both inner and outer iteration to 15, and minimum word count to 5. We take 1693 word pairs under consideration from the SCWS dataset (commonly found in the 20Newsgrup and SCWS datasets) for Spearman’s rank correlation coefficient measures between models scores (achieved by TDE and STE-diff) and the human judgments of similarity scores in the SCWS dataset. For each pair of words, pairwise cosine similarity uses for the word similarity evaluation of the base (STEdiff) and the proposed(TDE) model; for both models, we use two variants of similarity scoringAgvSimC and MaxSimC (Shi et al., 2017). Table 2 exhibits obtained scores by the base and the
Table 2: Spearman correlation (ρ× 100)
Model Similarity Metrics ρ× 100
TDE MaxSimC 39.22AvgSimC 39.04 STE-diff MaxSimC 31.56AvgSimC 33.28
Table 3: PMI scores as topic coherence
Model T=5 T=10 T=15 T=20 TDE -69.36 -331.74 -729.65 -1349.46 STE-diff -85.23 -374.84 -875.77 -1599.42
proposed model under the conditions mentioned above. According to Table 2, TDE obtains almost similar scores for MaxSimC (39.22) and the AvgSimC (39.04) whereas, in STE-diff, MaxSimC gets 31.56 and AvgSimC gets 33.28. Highlighted cells indicate better scores in the table obtained by TDE.

heading: 4.3 WORD EMBEDDING VISUALIZATION
text: To visualize the interaction between words and latent topics in the embedding space, we use the Wikipedia dump from April 2010 (Shaoul, 2010), similar to STE. For both base (STE-diff) and proposed (TDE) models, we set embedding size to 400, number of topics to 10, window size to 10, negative sampling to 8, subsampling frequent words to 10e-4, number of threads to 10, the inner and the outer iterations to 15. For visualization in Fig 3 and Fig 4, we use the t-SNE algorithm (Van der Maaten & Hinton, 2008), where each node represents a topic-specific word vector. For both models, we set perplexity to 40, n iteration to 2500, and random state to 23. We consider the word party with two different meanings political party and social gathering as a polysemous word for visualization. In Fig 3, party#9 is close to government#9 indicating political party but party#7 is far from government#9. our extensive analysis found party#0, party#1, party#2, party#6, and party#9 contain government#9 and government#0 within topmost 100 similar words, whereas party#3, party#4, party#5, party#7, party#8 are close to word summers#5, indicating a gathering and celebration. These words contain summer#5 in the topmost 50 similar words. In Fig 4, It seems party and government exist little scattered in the embedding space for STE-diff than TDE; it indicates learning better embeddings by TDE over STE-diff. Our extensive experiment found, only party#6 contains government#6 within the topmost 100 similar words, whereas summer#4 exists in the most 150 similar words of party#4 only.

heading: 4.4 TOPIC COHERENCE
text: In TDE, pointwise mutual information (PMI) scores are using to evaluate topic coherence as PMI scores strongly correlate with human annotations of topic coherence (Newman et al., 2010; Lau et al., 2014). For this experiment, we use the 20Newsgrup dataset. The parameter setting for both models base (STE-diff) and proposed (TDE) are the same as mentioned in section 4.2. Table 3 exhibits overall PMI scores for different numbers of top words (T) under 20 topics. The highlighted cells in the table indicate higher coherence, as we obtained negative coherence scores so, the smaller is better. Topics in TDE can be interpreted by bigrams words as shown in the publication (Shi et al., 2017) for STE-same. In our experiments, we found for STE-diff, each topic represents bigrams randomly
from various categories without thematic grouping of bigrams, whereas TDE maintains thematic gathering of bigrams. Table 4 represents three topical representations of bigram words with their source categories for TDE. The first column represents a religion-related topic, the second is related to the computer-related topic, and the third is related to politics.

heading: 4.5 DOCUMENT CLASSIFICATION FOR WORD EMBEDDING EVALUATION
text: In this experiment, we create document vectors by TF-IDF weighted averaging of each word embeddings under every document, using equation (17). For both models, the base (STE-diff) and the proposed (TDE), we use the 20Newsgroup dataset with the same parameter settings mentioned in section 4.2.
dw = tfidfw × k∑ z=1 p(z |wx̃, c).Uw, z (17)
where Uw,z is the learned input projection matrix, c represents context words, wx̃ represents pivots (concatenating center words w with the global document context x̃), and tfidfw is the TF-IDF score of w in the document. For document classification in this section, we use several standard evaluation metrics for the multiclass document classification tasks, e.g., Accuracy, Precision, Recall, and F-measure for LinearSVM classifier. We compare scores obtained by TDE with the obtained scores by PV, STE-diff, and STEsame models published by Shi et al. (2017). Table 5 represents obtained scores using evaluation metrics in the classification task, where TDE beats STE-same and PV, but STE-diff performs better in this case. TDE achieves better topic coherence than STE-diff, but STE-diff achieves better document classification, which we see a similar analogy between STE-same and STE-diff in Shi et al. (2017)’s work.

heading: 5 CONCLUSION
text: TDE: Topical Document Embedding is a jointly learning word-topic and document embeddings method, which learns document embedding on the fly simultaneously with the jointly learning process of the topic-specific word embeddings by taking document vector as a global context. As a result, TDE captures better syntactic and semantic properties by utilizing word, topic, and documentlevel information. It poses the simplest form of document embedding without imposing complex inference methods for document vectors. As such, each document vector is created by a simple weighted averaging of the topic-specific word embeddings, which is effective regardless of document size. Moreover, using the mask-out/drop-out corruption mechanism enhances learning speed and ensures quality embeddings as well. Our extensive experimental findings reveal that besides the document embeddings, the proposed joint learning method also captures better topical word embeddings over the base model.

heading: 6 REPRODUCIBILITY STATEMENT
text: Model source code written in C programming language, to run the code need to type ./go.sh in the terminal (optional: for creating make file type make CFLAG=”-lm -pthread -O3 -Wall -funrollloops”) . Model parameters can be reset in this file (e.g., training dataset). All datasets used in this research are available in preprocessed form (use ready for the model) in the data folder of the package (Model sourceCode Datasets). The model output should be reproduced effortlessly in a Linux machine with GCC C++ compiler installed.
For model evaluaiton, we have added evaluation source code files ste d2vc scws similarity.py and visualize.ipynb. The evaluation procedure requires python-2.7, numpy-1.16.1, scikit-learn-0.17. For convenience, we also included obtained evaluation results and detail analysis for embedding visualization and pearsonś correlation in the obtained results subfolder for both base and proposed models.

Review summaries:
This paper focuses on learning document embeddings, presenting a topic-document embedding (TDE) method employing syntactic and semantic properties by jointly learning topic and word embedding in a single framework. The proposed TDE approach follows corruption mechanism to create the global context and randomly select topic-specific word embeddings in learning document vectors, thus employing word, topic and document information in learning document vectors.  The experimental results have shown improved performed in terms of document classification, topic coherence and word embedding similarity evaluations.

Jointly learning word embeddings and document embeddings while taking into account topical information is somewhat interesting. The paper presents an improved skip-gram model where each individual word is associated with two matrices (e.g., the input projection matrix and the output projection matrix) which is designed to capture the topical aspect of words. In other words, each word is associated with K topic embeddings. While computing the predictive probability, the proposed model further considers a global context, i.e, a document embedding computed with the idea of document corruption. Basically, this paper simply combines the work of Shi et al. 2017 and Chen 2017, Both the innovation and experimental improvement are mariginal.