Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This is an emergency review.

This work proposes a novel method to use pre-trained topic embeddings and pre-trained word embeddings obtained from various corpora in the transfer learning framework. 

Their model architecture is based on DocNADE, unsupervised neural-network based topic model, and the authors propose two strategies to use pre-trained topic embeddings and pre-trained word vectors.
1) Addition of a weighted sum of pre-trained word embeddings and the hidden vector of DocNADE.
2) L2-Regularization term between topic embedding of DocNADE and pre-trained topic embeddings. They propose to align these two embeddings by multiplying align matrix "A" to the topic embedding of DocNADE.

They show the transfer learning performance of their model on various source/target domain datasets, including medical target corpora, and verify that their model outperforms on a short text and small document collection.

Strengths.
1. Comparison with the data augmentation baseline shows the performance gain is not only from bigger training data. Even though comparison with the naive baseline (data augmentation) seems too obvious, I think the results clearly show their claim about the importance of using transfer learning in neural topic modeling domain.
2. As the first approach that introduces a novel transfer learning framework with pre-trained topic embeddings, they show tons of experimental results with various datasets and metrics to show the specification of their method. Their experimental setting is well designed.

Weaknesses and comments:
Their method to combine pre-trained word embeddings and pre-trained topic embeddings is too simple. Since this is the first approach to use topic embedding in the transfer learning field, the simplicity of the proposed method is somewhat necessary. However, a weighted sum of pre-trained topic/word vectors seems not enough to transfer multisource knowledge. For instance, word vectors obtained from individual training processes do not share embedding vector space. As you apply the alignment method to topic embeddings from various sources, you should align word embeddings too.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper proposes a multi-source and multi-view transfer learning for neural topic modelling with the pre-trained topic and word embedding. The method is based on NEURAL AUTOREGRESSIVE TOPIC MODELs --- DocNADE (Larochelle&Lauly,2012). DocNADE learns topics using language modelling framework. DocNADEe (Gupta et al., 2019) extended DocNADE by incorporating word embeddings, the approach the authors described as a single source extension of the existing method.

In this paper, the proposed method adds a regularizer term to the DocNADE loss function to minimize the overall loss whereas keeping the existing single-source extension. The authors claimed that incorporating the regularizer will facilitate learning the (latent) topic features in the trainable parameters simultaneously and inherit relevant topical features from each of the source domains and generate meaningful representations for the target domain. The analysis and evaluation were presented to show the effectiveness of the proposed method. However, the results are not significantly improved than the based line model DocNADE. 

Overall, the paper is written well. However, it is not clear to me that the improved results are resulted due to multi-source multi-view transfer learning or for the better leaning of the single-source model due to the incorporation of the regularizer. 



review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: On the basis of existing topic modelling approaches, the authors apply a transfer learning approach to incorporate additional knowledge to topic models, using both word embeddings and topic models. The underlying idea is that topic models contain a global view that differs on a thematic level, while word embeddings contain a local, immediate contextual view. The combination of both local and global view transfer to enhance a topic model is the main contribution of this paper, especially when using multiple sources (therefore the title: multi-source multi-view transfer).
Given a document collection, DocNADE is used to generate the topic-word matrix. In the local view transfer step, the pre-trained WordPool is used, from which knowledge is transferred on the target document. The global view transfer is done by transferring knowledge from the pre-trained TopicPool to the target. As described in Algorithm 1 in the paper, both Word- and TopicPool are jointly used in the transfer learning process. 
For evaluation, three different measures are taken into account: Perplexity, Topic Coherence and Precision (Information Retrieval). In comparison to a DocNADE only approach, all values are better in the settings that use the transfer learning approach. Compared to DocNADE + word embeddings, the results are competitive as well. In both experiments, the multi-source setting evaluates best overall.

In conclusion, the paper shows that exploiting multiple sources and views in transfer learning leads to an overall improvement in the given tasks. The main contribution is the usage topic models in a transfer learning framework. Additionally the use of multi-source word embeddings is novel too, especially in the joint setting with the topic model transfer. The paper shows how the DocNADE approach is enhanced to make use of both local and global view transfer and how this enhancement leads to improved performance on various related tasks. 
Still, the overall contribution is mostly in combining existing methods and can be judged as rather incremental.

Minor note: A small mistake has been found in Table 5. The best perplexity value in the first column is not the bold 638, but the 630 in the local-view transfer setting.

Edit after rebuttal: In my review I did not value the contribution of the transfer learning approach enough. So, when also considering the extensive evaluation I am now leaning towards accept.
review_title: Official Blind Review #2

Meta-review: This paper presents a transfer learning framework in neural topic modeling. Authors claim and reviewers agree that this view of transfer learning in the realm of topic modeling is novel.

However, after much deliberation and discussion among the reviewers, we conclude that this paper does not contribute sufficient novelty in terms of the method. Also, reviewers find the experiments and results not sufficiently convincing.

I sincerely thank the authors for submitting to ICLR and hope to see a revised paper in a future venue.
Decision: Reject