Improving Deep Neural Networks with Probabilistic Maxout Units
Authors: Jost Tobias Springenberg, Martin Riedmiller
ICLR 2014

Abstract: We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).


Peer review:
[Review]:
For fig 3b, what sampling distribution did you use for the maxout baseline? Are you sampling from the distribution defined in eqn 3? Why is this a meaningful thing to evaluate for maxout, which hasn't been trained to know you're going to sample in that particular way? It seems like it would be more fair to sample different dropout masks, since each of the maxout subnetworks have actually been trained to do the classification task. It's not very surprising to find that a neural net that has been trained to do task X performs better than a neural net that has not been trained to do task X.