Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors argue in this paper that due to the limited rank of the  context-to-vocabulary logit matrix in the currently used version of the softmax output layer, it is not able to capture the full complexity of language. As a result, they propose to use a mixture of softmax output layers instead where the mixing probabilities are context-dependent, which allows to obtain a full rank logit matrix in complexity linear in the number of mixture components (here 15). This leads to improvements in the word-level perplexities of the PTB and wikitext2 data sets, and Switchboard BLEU scores.

The question of the expressiveness of the softmax layer, as well as its suitability for word-level prediction, is indeed an important one which has received too little attention. This makes a lot of the questions asked in this paper extremely relevant to the field. However, it is unclear that the rank of the logit matrix is the right quantity to consider. For example, it is easy to describe a rank D NxM matrix where up to 2^D lines have max values at different indices. Further, the first two "observations" in Section 2.2 would be more accurately described as "intuitions" of the authors. As they write themselves "there is no evidence showing that semantic meanings are fully linearly correlated." Why then try to link "meanings" to basis vectors for the rows of A?

To be clear, the proposed model is undoubtedly more expressive than a regular softmax, and although it does come at a substantial computational cost (a back-of-the envelope calculation tells us that computing 15 components of 280d MoS takes the same number of operations as one with dimension 1084 = sqrt (280*280*15)), it apparently manages not to drastically increase overfitting, which is significant.

Unfortunately, this is only tested on relatively small data sets, up to 2M tokens and a vocabulary of size 30K for language modeling. They do constitute a good starting place to test a model, but given the importance of regularization on those specific tasks, it is difficult to predict how the MoS would behave if more training data were available, and if one could e.g. simply try a 1084 dimension embedding for the softmax without having to worry about overfitting.

Another important missing experiment would consist in varying the number of mixture components (this could very well be done on WikiText2). This could help validate the hypothesis: how does the estimated rank vary with the number of components? How about the performance and pairwise KL divergence? 

This paper offers a promising direction for language modeling research, but would require more justification, or at least a more developed experimental section.

Pros:
- Important starting question
- Thought-provoking approach
- Experimental gains on small data sets

Cons:
- The link between the intuition and reality of the gains is not obvious
- Experiments limited to small data sets, some obvious questions remain
review_title: The authors identify an issue with current prediction models and propose a solution in the form of a mixture of softmax. The idea is interesting, but would require more justification and experimental validation.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Language models are important components to many NLP tasks. The current state-of-the-art language models are based on recurrent neural networks which compute the probability of a word given all previous words using a softmax function over a linear function of the RNN's hidden state. This paper argues the softmax is not expressive enough and proposes to use a more flexible mixture of softmaxes. The use of a mixture of softmaxes is motivated from a theoretical point of view by translating language modeling into matrix factorization.

Pros:
--The paper is very well written and easy to follow. The ideas build up on each other in an intuitive way.
--The idea behind the paper is novel: translating language modeling into a matrix factorization problem is new as far as I know.
--The maths is very rigorous.
--The experiment section is thorough.

Cons:
--To claim SOTA all models need to be given the same capacity (same number of parameters). In Table 2 the baselines have a lower capacity. This is an unfair comparison
--I suspect the proposed approach is slower than the baselines. There is no mention of computational cost. Reporting that would help interpret the numbers. 

The SOTA claim might not hold if baselines are given the same capacity. But regardless of this, the paper has very strong contributions and deserves acceptance at ICLR.
review_title: nice idea

final_score: rating: 7: Good paper, accept
review_content: The authors has addressed my concerns, so I raised my rating. 

The paper is grounded on a solid theoretical motivation and the analysis is sound and quite interesting.

There are no results on large corpora such as 1 billion tokens benchmark corpus, or at least medium level corpus with 50 million tokens. The corpora the authors choose are quite small, the variance of the estimates are high, and similar conclusions might not be valid on a large corpus. 

[1] provides the results of character level language models on Enwik8 dataset, which shows regularization doesn't have much effect and needs less tuning. Results on this data might be more convincing.

The results of MOS is very good, but the computation complexity is much higher than other baselines. In the experiments, the embedding dimension of MOS is slightly smaller, but the number of mixture is 15. This will make it less usable, I think it's necessary to provide the training time comparison.

Finally experiments on machine translation or speech recognition should be done and to see what improvements the proposed method could bring for BLEU or WER. 

[1] Melis, Gábor, Chris Dyer, and Phil Blunsom. "On the state of the art of evaluation in neural language models." arXiv preprint arXiv:1707.05589 (2017).

[2] Joris Pelemans, Noam Shazeer, Ciprian Chelba, Sparse Non-negative Matrix Language Modeling,  Transactions of the Association for Computational Linguistics, vol. 4 (2016), pp. 329-342

[3] Shazeer et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017

review_title: The paper is grounded on a solid theoretical motivation and the analysis is sound and quite interesting. 

Meta-review: Viewing language modeling as a matrix factorization problem, the authors argue that the low rank of word embeddings used by such models limits their expressivity and show that replacing the softmax in such models with a mixture of softmaxes provides an effective way of overcoming this bottleneck. This is an interesting and well-executed paper that provides potentially important insight. It would be good to at least mention prior work related to the language modeling as matrix factorization perspective (e.g. Levy & Goldberg, 2014).
Decision: Accept (Oral)