Improving Deep Neural Networks with Probabilistic Maxout Units
Authors: Jost Tobias Springenberg, Martin Riedmiller
ICLR 2014

Abstract: We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).


Peer review:
[Review]:
For figure 4, I think it's important to note that your evaluation method only returns a low number if *all* of the units in a layer are invariant to the studied transformation. If the layer has a factored representation that represents the studied transformation with one set of units and represents other properties of the input with another, disjoint set of units, there will still be a large cosine difference between the representation of two transformed versions of the input, because the change in the portion of the representation that corresponds to the studied property will result in a change of the normalization constant and thus a change in all of the code elements. I think it would make more sense to normalize each unit separately based on a statistic such as its standard deviation across the training set, and then plot histograms showing how much each normalized unit changes as you vary the input. This way you still control for the possibility that the models operate on different scales, but you can also tell if the representation is factored or not.