Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Paper Strengths: 
-- Elegant use of MoE for expanding model capacity and enabling training large models necessary for exploiting  very large datasets in a computationally feasible manner

-- The effective batch size for training the MoE drastically increased also

-- Interesting experimental results on the effects of increasing the number of MoEs, which is expected.


Paper Weaknesses:

--- there are many different ways of increasing model capacity to enable the exploitation of very large datasets; it would be very nice to discuss  the use of MoE and other alternatives in terms of computational efficiency and other factors.

review_title: Elegant use of MoE for expanding model capacity, but it would be very nice to discuss MoE alternatives in terms of computational efficiency and other factors.

final_score: rating: null
review_content: Can you explain the difference between Importance and Load as per-expert utilization measures?Why you think we need both terms in the loss function?

Can you motivate the definition of P(x,i) and give an intuitive explanation? I'm not sure I understand what does kth_excluding term do here.
review_title: Difference between Importance and Load

final_score: rating: null
review_content: Can you justify the use of Mixture-of-Experts in LM and MT? 
review_title: Use of huge Mixture-of-Experts 

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a method for significantly increasing the number of parameters in a single layer while keeping computation in par with (or even less than) current SOTA models. The idea is based on using a large mixture of experts (MoE) (i.e. small networks), where only a few of them are adaptively activated via a gating network. While the idea seems intuitive, the main novelty in the paper is in designing the gating network which is encouraged to achieve two objectives: utilizing all available experts (aka importance), and distributing computation fairly across them (aka load). 
Additionally, the paper introduces two techniques for increasing the batch-size passed to each expert, and hence maximizing parallelization in GPUs.
Experiments applying the proposed approach on RNNs in language modelling task show that it can beat SOTA results with significantly less computation, which is a result of selectively using much more parameters. Results on machine translation show that a model with more than 30x number of parameters can beat SOTA while incurring half of the effective computation.

I have the several comments on the paper:
- I believe that the authors can do a better job in their presentation. The paper currently is at 11 pages (which is too long in my opinion), but I find that Section 3.2 (the crux of the paper) needs better motivation and intuitive explanation. For example, equation 8 deserves more description than currently devoted to it. Additional space can be easily regained by moving details in the experiments section (e.g. architecture and training details) to the appendix for the curious readers. Experiment section can be better organized by finishing on experiment completely before moving to the other one. There are also some glitches in the writing, e.g. the end of Section 3.1. 
- The paper is missing some important references in conditional computation (e.g. https://arxiv.org/pdf/1308.3432.pdf) which deal with very similar issues in deep learning.
- One very important lesson from the conditional computation literature is that while we can in theory incur much less computation, in practice (especially with the current GPU architectures) the actual time does not match the theory. This can be due to inefficient branching in GPUs. It would be nice if the paper includes a discussion of how their model (and perhaps implementation) deal with this problem, and why it scales well in practice.
- Table 1 and Table 3 contain repetitive information, and I think they should be combined in one (maybe moving Table 3 to appendix). One thing I do not understand is how does the number of ops/timestep relate to the training time. This also related to the pervious comment.

review_title: null

final_score: rating: 7: Good paper, accept
review_content: This paper describes a method for greatly expanding network model size (in terms of number of stored parameters) in the context of a recurrent net, by applying a Mixture of Experts between recurrent net layers that is shared between all time steps.  By process features from all timesteps at the same time, the effective batch size to the MoE is increased by a factor of the number of steps in the model; thus even for sparsely assigned experts, each expert can be used on a large enough sub-batch of inputs to remain computationally efficient.  Another second technique that redistributes elements within a distributed model is also described, further increasing per-expert batch sizes.

Experiments are performed on language modeling and machine translation tasks, showing significant gains by increasing the number of experts, compared to both SoA as well as explicitly computationally-matched baseline systems.

An area that falls a bit short is in presenting plots or statistics on the real computational load and system behavior.  While two loss terms were employed to balance the use of experts, these are not explored in the experiments section.  It would have been nice to see the effects of these more, along with the effects of increasing effective batch sizes, e.g. measurements of the losses over the course of training, compared to the counts/histogram distributions of per-expert batch sizes.

Overall I think this is a well-described system that achieves good results, using a nifty placement for the MoE that can overcome what otherwise might be a disadvantage for sparse computation.



Small comment:
I like Fig 3, but it's not entirely clear whether datapoints coincide between left and right plots.  The H-H line has 3 points on left but 5 on the right?  Also would be nice if the colors matched between corresponding lines.
review_title: Nice use of MoE with good results

final_score: rating: null
review_content: An additional baseline comparison, which I don't see yet, may be to replace the MoE with a simple 2- or so layer network with similar number of executed parameters or connections as the MoE (but fewer total parameters).  E.g., to use the k in the top-k experts (plus the gating net) to find the number of executed params/connections and create a slightly deeper, k layer net matching this number of connections (e.g., k layers each with the same 512x512 matrix size), that replaces the MoE layer.  The total number of stored params is fewer here, but the number of executed params and connections should be about the same, and it is a deeper net doing the mapping.

review_title: Multilayer mapping net baseline?

Meta-review: The paper uses mixtures of experts to increase the capacity of deep networks, and describes the implementation of such a model on a cluster of GPUs. The proposed mixture model achieves strong performances in language modeling and machine translation.
Decision: Accept (Poster)