Improving Deep Neural Networks with Probabilistic Maxout Units
Authors: Jost Tobias Springenberg, Martin Riedmiller
ICLR 2014

Abstract: We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).


Peer review:
[Review]:
Authors propose replacing max operation of maxout with probabilistic version, same what Zeiler did for spatial max-pooling in 'stochastic pooling' paper. 

For inference, they run the network 50 times using the same sampling procedure, and average the outputs to get probabilities.

They add a 'temperature' parameter to allow to interpolate between maxout and 'uniform random' sampling, and find that annealing the temperature helps.

Also the analyze the per-layer optimal setting of the temperature and found that stochasticity is most important in first 2 convolutional layers, whereas in the last 2 layers, using probout did not give any advantage over maxout.

There are a few minor corrections, but overall this is a solid submission with high relevance for ICLR.

Issues:

- Minor style issue: use 	ext or mbox for 'softmax' and 'multinomial' inside formula

- Table 3: could add 2.16% error rate obtained by another ICLR submission -- 'Multi-digit Number Recognition from Street View
Imagery using Deep Convolutional Neural Networks'

- There's an explanation for why first layers benefit the most from stochasticity -- how stochasticity 'pulls the units in the subspace closer together.' This is unclear to me, and I would recommend expanding/explaining this view.