Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes a combined topic modeling and word embedding method.  The idea is to define word and topic embeddings in the same space, represent documents based on both of them, and use a transport-based method to train these two representations to be similar to each other.  The final objective function combines a "conditional transport" objective with a log-likelihood term, and is optimized via Adam.  Experiments find that the method achieves good topic coherence scores and document clustering metrics compared to baselines.

main_review: Strong points:

- While the combination of topic models and word embeddings has received substantial study, it remains an interesting problem.  An optimal transport (or conditional transport) approach such as the one proposed is a promising direction.

- The ability to initialize with pre-trained word embeddings and fine-tune the model is valuable (though this paper is not the first to do this).

- Experimental results versus a range of baseline methods are encouraging.

- The qualitative results on embeddings of words and topics were nice to see.

Weak points:

- My biggest complaint with this paper is that the necessity and motivation for the proposed method is unclear.  While combining topic models and word embeddings is a worthy problem, and transport methods are an interesting potential solution, this paper is far from the first to study the intersection of topic models and word embeddings.  This is already a crowded space which includes the cited papers, such as the ETM and neural topic models, and several others as well.  The present manuscript doesn't make a compelling argument that the proposed method resolves urgent problems in this area.  Furthermore, the proposed method is stated then studied without a principled justification. It reads like the authors wanted to try this method, so they did, rather than that there was a particular reason or mathematical justification for why the method is designed the way that it is.

- The manuscript was quite hard to follow at times, partly because the lack of clearly stated justification for the proposed approach leads to substantial effort to understand what exactly the authors are doing.  The use of non-standard notation in some instances was a contributing factor in this issue.

- Transformer-based language models such as BERT have become the state of the art for representation learning in natural language processing in recent times.  This work should ideally compare with BERT, etc, or justify why it does not do so.  It also needs to discuss the merits of the proposed approach versus transformer-based language models.

- Since the objective function (Equation 13) combines the conditional transport objective with a likelihood objective, both an approach using conditional transport alone and an approach using likelihood optimization alone should be compared to as baselines.

- Separately encoding transport costs c(w, \phi) and distances d(w, \phi), which are "related but also with differences" (pg 4), seems inelegant. 

- If speed advantages are claimed (cf. pg 6, "more efficient learning"), this should be demonstrated with rigorous experiments.

- Hyperparameter tuning could have been done more systematically for the experiments, rather than using fixed values for most experiments, followed by a brief sensitivity test.

- The training algorithm could have been explained more clearly, and in the paper itself instead of in the appendix.


Additional feedback / minor suggestions:

Pg 6, Equation 6, it looks like there is an issue in the last step, which doesn't clearly follow from the previous step.  I think there is a missing term on the top and bottom here corresponding to P_j(W_{ji}).  Compare to Equation 9, which has \theta terms on the top and bottom of the last equation in the line.

Pg. 10 (references) - "Latent dirichlet allocation" needs a capital "D."

Pg. 12 (Appendix), "the training algorithm of our WeTe is shown in Algorithm 1" needs a capital T.

A couple of highly relevant missing references:
K. Keya, Y. Papanikolaou, and J. R. Foulds. Neural embedding allocation: Distributed representations of topic models. ArXiv preprint arXiv:1909.04702 [cs.CL], 2019.

J. R. Foulds. Mixed Membership Word Embeddings for Computational Social Science. Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS), 2018.
summary_of_the_review: While the proposed method has potential and there are some positive aspects of the work, the manuscript is not ready for publication due to deficiencies in several aspects, including: motivation of the need for the general approach, justification of the specific choices in the approach, other clarity issues, relationship to BERT and transformer-based language models, and certain aspects of the experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Jointly learning latent topic representations with word embeddings has caught some attention in the past. The reason why researchers choose this route is to exploit the complementary advantage of both these models, for instance, to improve the performance of the model on short texts. In this paper, the authors study a new topic model where topics and words are encoded in the same embedding space. Usually, in the traditional topic modelling literature, the posterior inference is realised using algorithms such as Gibbs sampling or collapsed Gibbs sampling; however, in this paper, since the authors exploit complementary knowledge between the two models, the end-to-end model is jointly trained the document-specific discrete latent topic distribution is made to appear as close as possible to the discrete distribution over words in the document. A probabilistic bidirectional transport model is developed that measures the difference between the discrete distributions in the embedding space. The authors also discuss several key advantages of the model.

Experiments are then followed with topic diversity and coherence measures as evaluation metrics which are widely used in the literature. Qualitatively, the authors show some example results from their dataset in the form of a t-SNE plot.
main_review: Strengths:
1. The authors develop a novel model to jointly model latent topics and word embeddings. While the model is novel, there are already existing works that have jointly modelled latent topics and word embeddings exploiting their complementary advantages. The authors also present the derivation of the model and the key equations that demonstrate how the model works.
2. The authors have also tried to intertwine their model with the experiments. As a result, the authors have conducted some experiments to demonstrate that the model improves upon some existing methods.

There are several ways in which this paper can be improved:
1. The authors can strengthen their experiments by experimenting against some downstream applications such as document classification. Unsure of why the authors did not cite this work that has used document classification as one of the experiments (https://dl.acm.org/doi/10.1145/3077136.3080806).
2. Relating to the work cited in (1) above, it would be nice to have some cooperation on how the proposed work differs from the work that is already published by Bei et al. There are some follow-up works too since this work was published and the authors are requested to have a go at them too. Where Bei's work would be crucial is mainly in comparing against the run-time performance of the models where Bei's work is a much simpler version of a model that jointly learns latent topics and embeddings. Besides that, the effectiveness of the model could be compared as a result.
3. Correcting some grammar issues, e.g., facilitates instead of facilities, latter instead of later; authors can use the PDF search feature to find such words.
4. The authors could consider theoretically comparing their embeddings with those static embeddings obtained using Skipgram or GloVe. Or, do we expect that the embeddings that the authors learn using their model are different in property from traditional static embeddings such as GloVe or Skipgram.
summary_of_the_review: While there are strengths in terms of the model novelty, there is some weakness too. The paper can be further improved from various angles such as strengthening the experiments section by comprising with some closely relevant works including conducting some downstream application analysis such as document classification performance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The aim of the paper is to revisit topic modeling through the lens of word embeddings and using a bidirectional version optimal transport. TIn this paper each document is considered as a set of embeddings words, each topic is seen as a mixture of embedding living in the same space as words embeddings
The aim is to minimize the distance of the topic distribution and the words distribution for each documents
main_review: Strength of the paper : the point of view is relatively original, since it provides a new framework for topic modelling. It relies on a version of bidirectional optimal transport which is quite origineal and embedd words and topics in the same space. The relevance of the model is assess by numerical experiments comparing in terms of perplexity this approach with others ones more classical as vanilla LDA, OTLDA...

Weaknesses : the idea of mixing OT and LDA is not so new, even if bidirectional OT has not been used yet. In addition, this version of bidirectional OT is just a symmetrisation, which is not so original
summary_of_the_review: The paper mixes OT and LDA in a rather original way. Nevertheless, this idea is not new and has been already proposed in other papers
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new topic modeling framework called WeTe: each document in text corpus is represented as a bag of word embeddings vectors, and each topic is modeled as an embedding vector in the shared word embedding space. WeTe minimizes the expected difference between those two sets over all documents. A bidirectional transport-based method is proposed to learn the topic embeddings as well as topic proportions for documents efficiently. Extensive experiments on news and web pages show that the proposed model outperforms competitive methods for both deriving high-quality latent topics and generating better document representations for clustering tasks. 
main_review: Strengths:
1. A diversity of datasets including web pages and news articles are used in the evaluation, and experiments show the superiority of the proposed topic modeling method. 
2. The algorithm is reasonable and has the advantage that it can learn word embeddings and topic embeddings from scratch. 
3. The paper is well written and easy to follow. The authors provide concrete figures and tables to show the results of experiments. 

Weaknesses:
1. Given a few recent works on learning dense word vectors jointly with latent document-level mixtures of topic vectors, the novelty of this paper is not significant. For example, there's a missing reference:  Moody, Christopher E. "Mixing dirichlet topic models and word embeddings to make lda2vec." arXiv preprint arXiv:1605.02019 (2016). 

2. The experiment results shown in this paper do not include t-test results such as p-values. We cannot tell how significant the performance of the proposed algorithm is, compared to other state-of-the-art methods. 

3. To show the interpretability of the resulting topics, it'd be great if the authors can provide qualitative analysis such as in Chang, Jonathan, et al. "Reading tea leaves: How humans interpret topic models." Advances in neural information processing systems. 2009.

4. For comparison of time/space complexities, it'd be great if the authors can provide a detailed complexity analysis. 



summary_of_the_review: The idea of learning latent topics for text documents using bag-of-words embeddings is not quite new given that a few previous papers describe similar methods. However, it's still empirically useful to introduce concrete optimization methods to learn such a model. This paper presents extensive experiments on news and web pages to show the superiority of the proposed method, compared to other state-of-the-art methods.
review_title: null

Meta-review: This is a borderline paper on the well researched theme of Topic models.
The strongest point of the paper is that it proposes a new topic modelling framework where both word and topic embeddings live in the same space.
 It then appeals to optimal transport theory to do the necessary training using SGD. However, this is not the first paper to examine
Topic models and Optimal Transport theory. Several papers[1,2,3] in the recent past have started investigating this line of research.  In the rebuttal phase, the author(s) justify the choice of state of the art methodologies and also discuss the key conceptual difference between 
existing literature and the submitted one. The major difference seems to 
they approach the problem differently leading to better quality topics(as measured by several metrics) and computational efficiency---existing state of the art requires more complicated iterations whereas proposed approach works with SGD. 
The manuscript, if accepted, needs to be updated considerably to reflect some of these aspects. 




[1] Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. Distilled wasserstein learning for wordembedding and topic modeling. In NeurIPS2018.

[2] Viet Huynh, He Zhao, and Dinh Phung. OTLDA: A geometry-aware optimal transport approach fortopic modeling. In NeurIPS2020.


[3] He Zhao, D Phung, V Huynh, T Le, and W Buntine. Neural topic model via optimal transport, 2020
Decision: Accept (Poster)