Task: Language Identification
Input (Spanish, English): This work combines a LDA-type topic model with a RNN and models this by having an additive effect on the predictive distribution via the topic parameters. A variational auto-encoder is used to infer the topic distribution for a given piece of text and the RNN is trained as a RNNLM. The last hidden state of the RNNLM and the topic parameters are then concatenated to use as a feature representation.  The paper is well written and easy to understand. Using the topic as an additive effect on the vocabulary allows for easy inference but intuitively I would expect the topic to affect the dynamics too, e.g. the state of the RNN. The results on using this model as a feature extractor for IMDB are quite strong. Is the RNN fine-tuned on the labelled IMDB data? However, the results for PTB are weaker. From the original paper, an ensemble of 2 LSTMs is able to match the topicRNN score. This method of jointly modelling topics and a language model seems effective and relatively easy to implement.  Finally, the IMDB result is no longer state of the art since this result appeared in May (Miyato et al., Adversarial Training Methods for Semi-Supervised Text Classification).  Some questions: How important is the stop word modelling? What do the results look like if l_t = 0.5 for all t?  It seems surprising that the RNN was more effective than the LSTM. Was gradient clipping tried in the topicLSTM case? Do GRUs also fail to work?  It is also unfortunate that the model requires a stop-word list. Is the link in footnote 4 the one that is used in the experiments?  Does factoring out the topics in this way allow the RNN to scale better with more neurons? How reasonable does the topic distribution look for individual documents? How peaked do they tend to be? Can you show some examples of the inferred distribution? The topics look odd for IMDB with the top word of two of the topics being the same: 'campbell'. It would be interesting to compare these topics with those inferred by LDA on the same datasets.  Minor comments: Below figure 2: GHz -> GB \Gamma is not defined.
Output (English): en