Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I am unimpressed with the quality of writing and presentation, to begin with. There are numerous grammatical errors and typos that make the paper a very difficult read. The presentation also follows an inequitable pattern where the backgrounds and related works are overemphasized and the actual contribution of the paper seems very limited. In its current form, this paper is not ready for publication in ICLR.

The idea of representing a document as an average of the embeddings of the words is a rather crude idea. Paragraph2vec and many of its derivatives have shown significant improvements with document modelling. The perplexity improvements are nice to have, but I would have liked to see the embeddings being applied to some supervised problems to assess their utilities. 

There are quite a few computationally expensive normalization terms. I am curious to understand how these summations do not slow the training process down without further approximations. The authors may present some computational complexity measures to convince readers about the practical applications of the proposed models.

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper proposes a neural topic model that aim to discover topics by minimizing a version of the PLSA loss. According to PLSA, a document is presented as a mixture of topics, while a topic is a probability distribution over words, with documents and words assumed independent given topics. Thanks to this assumption, each of these probability distributions (word|topic, topic|document, and word|document) can essentially be expressed as a matrix multiplication of the other two, and EM is usually adopted for the optimization. This paper proposes to embed these relationships in a neural network and then optimize the model using SGD.

I believe the paper should be rejected because: 1) most aspects of this paper are a little dated 2) novelty is little 3) experimental section is very limited and unconvincing.

To elaborate on the experimental section:
- Only LDA has been presented as baseline. There's plenty of neural topic models to compare against (you mentioned some in your related work section) but no comparison with any of those is presented. If the concern is their training time on large datasets, they should be at least presented as comparison for the smaller datasets. For the large datasets there's other approaches that would scale and should be presented as baselines: 1) train on a sample of the dataset 2) co-occurrence based topic methods on sliding windows of text are extremely fast (eg see "A Biterm Topic Model", "A Practical Algorithm for Topic Modeling with Provable Guarantees", and "A Reduction for Efficient LDA Topic Reconstruction" which could fit your scenario with large datasets where topics most likely have small overlap with each other and are almost separable by anchor words.)
- Even regarding just LDA: what hyper-parameters \alpha and \beta did you set for LDA? Tuning \beta to a small value might have an impact for large datasets.
- Metrics: only perplexity is presented and metrics but it's well known that perplexity on its own is quite limited and often is not correlated to human judgment. Consider adding topic coherence measures as well.
- The section on continuous document embeddings is confusing and the explanation should be improved and the formalism tightened.


Other (did not impact the score):
- Biases: you're adding biases to your probability estimation equations. This is not in line  with the PLSA assumption. What happens if no biases are used?

The paper has several typos and grammatical errors, e.g.:
- page 2, L#1: networks -> network
- page 4, sec 3.2: set unobserved -> set of unobserved
- page 5, sec 5: pratise -> practice
- several places: it's -> its

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: First, some minor issues.  I didn't understand equation (3).  It seems to be a variant of equation (4), and seems to be in disagreement with equation (6).  Might be better if the equation was just dropped.  For equation (9), you should have brackets "()" around the argument to the exp.

Second, in terms of comparisons, the paper lacks adequate related work.  Some non-parametric but non-neural
models not implemented in GPUs substantially beat LDA, and will run on all the big data sets you list, though perhaps
not quickly!   There has also been a number of neural and hybrid topic models developed.  
DocNADE and LLA (Zaheer etal), for instance, work very well in PPL.  Then there are many new deep topic models.  Some use the amortised inference that you adopt in section 5.    Some incorporate word embeddings or document metadata to
further improve performance metrics.  Note some of the earlier ICLR/NeurIPS papers with deep models didn't
do extensive comparative empirical testing, so may not work well against DocNADE or more recent algorithms.

In terms of related work, topic models is a bit of a mine-field because there is a huge amount of work in
a huge number of venues, and few authors do a good job of covering related work.  What you have listed are mainly the
older works.  Recent work also includes Poisson Matrix Factorisation and its variants, as well as hierarchical
variants of LDA, much better than the 2004 paper you mention.

To do the coherence comparisons, easiest way is to use the Palmetto software.
You can also evaluate models by using them as features in a classification task.

It was interesting that you only did one layer for your networks, i.e.,  equations (4)-(6).  Why was this?
I would have liked to have seen the impact of more layers. However, your model is remarkably simple 
so if it works well, that is good.

Anyway, the experimental evaluation shows good results on all three datasets for your models, but its hard to be sure
since you only have one comparison, an old LDA, and nothing recent.  So promising work, but
related work and experimental work need to be improved.


review_title: Official Blind Review #1

Meta-review: This paper presents a neural topic model with the goal of improving topic discovery with a PLSA loss. Reviewers point out major limitations including the following:

1) Empirical comparison is done only with LDA when there are many newer models that perform much better.
2) Related work section is incomplete, especially for the newer models.
3) Writing is unclear in many parts of the paper.

For these reasons, I recommend that the authors make major improvements to the paper before resubmitting to another venue.
Decision: Reject