Improving Deep Neural Networks with Probabilistic Maxout Units
Authors: Jost Tobias Springenberg, Martin Riedmiller
ICLR 2014

Abstract: We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).


Peer review:
[Review]:
The paper introduces a generalization of the maxout unit, called probout. The output of a maxout unit is defined as the maximum of a set of linear filter responses. The output of a probout unit is sampled from a softmax defined on the linear responses. For vanishing temperature this turns into the maxout response.

While the idea is probably not revolutionary, it seems reasonable and it seems to work fairly well on the datasets tried in the paper. 

It is a bit unfortunate that unlike for dropout/maxout, there does not seem to be a closed-form, deterministic activation function at test time that works well. At least the authors did not find any.

Instead they propose to average multiple outputs. This makes probout networks much slower at test time than a maxout network. It also puts into perspective the improved classification results over maxout. It seems unlikely that the common practice of halving weights at test time is exactly the optimal way of making predictions for a model trained with dropout. And it is conceivable that some kind of model averaging will be able to improve performance for those networks, too.

It is interesting that probout units with group size two tend to yield filter pairs in quadrature relationship, and much more clearly so than maxout. In that respect they behave similar to average pooling units. It would be interesting to investigate this further in future work.