Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper propose an adaptive dropout strategy for class logits. They learn a distribution q(z | x, y) that randomly throw class logits. By doing so they ensemble predictions of the models between different set of classes, and focuses on more difficult discrimination tasks. They learn the dropout distribution by variational inference with concrete relaxation. 

Overall I think this is a good paper. The technique sounds, the presentation is clear and I have not seen similar paper elsewhere (not 100% sure about the originality of the work though). 

Pro:
* General algorithm

Con:
* The experiment is a little weak. Only on CIFAR100 the proposed approach is much better than other approaches. I would like to see the results on more datasets. Maybe should also compare with more dropout algorithms, such as DropConnect and MaxOut.
review_title: Adaptively zero out class logits based on the input

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros
- The proposed model is a nice way of multiplicatively combining two features :
  one which determines which classes to pay attention to, and other that
provides useful features for discrimination.

- The adaptive component seems to provide improvements for small dataset sizes
  and large number of classes.

Cons
- "One can easily see that if o_t(x; w) = 0, then class t becomes neutral in the
  classification and the gradients are not back-propagated from it." : This does
not seem to be true. Even if the logits are zero, the class would have a
non-zero probability and would receive gradients. Do the authors mean
exp(o_t(x;w)) = 0 ?

- Related to the above, it should be clarified what is meant by dropping a
  class. Is its logit set to zero or -\infty ? Excluding a class from the
softmax is equivalent to having a logit of -\infty, not zero. However, from the
equations in the paper it seems that the logit is set to zero. This would not
result in excluding the unit. The overall effect would just be to raise the
magnitude of logits across the entire softmax.

- It seems that the model benefits from at least two separate effects - one is
  the attention mechanism provided by the sigmoids, and the other is the
stochasticity during training. Presently, it is not clear if only one of the
components is providing most of the benefits, or if both things are useful. It
would be great to compare this model to a non-stochastic one which just has the
multiplicative effects applied in a deterministic way (during both training and
testing).

- The objective of the attention mechanism that sets the dropout mask seems to
  be the same as the primary objective of classifying the input, and the
attention mechanism is prevented from solving the task by adding an extra
entropy regularization. It would be useful to explain more why this is needed.
Would it not be fine if the attention mechanism did a perfect job of selecting
the class ?

Quality
The paper makes relevant comparisons and is overall well-motivated. However,
some aspects of the paper can be improved by adding more explanations.

Clarity
Some crucial aspects of the paper are unclear as mentioned above.

Originality
The main contribution of the paper is similar to multiplicative gating. The
added stochasticity and the model ensembling interpretation is probably novel.
However, experiments are insufficient to determine whether it is this novelty
that contributes to improved performance or just the gating.

Significance
This paper makes incremental improvements and would be of moderate interest to
the machine learning community.

Typos :
- In Eq 3, the numerator has z_t. Should that be z_y ?
- In Eq 5, the denominator has z_y. Should that be z_t ?
review_title: Needs more clarity and a deterministic baseline

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper discusses dropping out the pre-softmax logits in an adaptive manner. This isn't a huge conceptual leap given previous work, for instance that of Ba and Frey 2013 or the sequence of papers by Gal and his coauthors on variational interprations of dropout. In the spirit of the latter series of papers on variational dropout there is a derivation of this algorithm using ideas from variational inference. The variational approximation is a bit odd in that it doesn't have any variational parameters, and indeed a further regulariser in equation (14) is needed to give the desired behaviour. A fairly small, but consistent improvement on the base model and other similar ideas is reported in Table 1. I would have liked to have seen results on ImageNet. I don't find (the too small) Figure 2 to be compelling evidence that "our dropmax effectively prevents
overfiting by converging to much lower test loss". The test loss in question looks like a noisy version of the base test loss with a slightly lower mean. There are grammatical errors throughout the paper at a higher rate than would normally be found in a successful submission at this stage. Figure 3 illustrates the idea nicely. Which of the MNIST models from Table 1 was used?

review_title:  A relevant idea, but not especially innovative and not brilliantly carried out.

Meta-review: This paper proposes a general regularization algorithm which builds on the dropout idea. This is a very significant topic. The overall motivation is good, but the specific design choices are less well motivated over, for example, ad-hoc choices. Some concerns remain after the post-rebuttal discussion with the reviewers: the improvement is incremental in terms of concepts and methodology, the clarity needs to be improved and the experiments are somehow weak.
In summary, the main idea and research direction is interesting, but the attempted generality of the algorithm and the significance of the area call for a more clear and convincing presentation.

Decision: Invite to Workshop Track