Improving Deep Neural Networks with Probabilistic Maxout Units
Authors: Jost Tobias Springenberg, Martin Riedmiller
ICLR 2014

Abstract: We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).


Peer review:
[Review]:
As mentioned in a previous comment I want to report back with
preliminary results from the hyperparameter search I am conducting. 
Essentially I am optimizing over all notable parameters except the
number of units in each layer - in order to fix the model size 
(More specifically this includes: learning rate,
momentum, pooling shape, pooling stride, size of the convolutional
kernels). 

My preliminary results suggest that on CIFAR-10 the best probout
network trained without data augmentation can achieve an error rate of <= 10.7% .
During the hyperparameter search I however also became aware that the
parameter search carried out for the original maxout paper was
probably  not very exhaustive, as improved performance can also
be achieved with a vanilla maxout model. 
While the difference in performance between probout and maxout appears
to be observable for all hyperparameter settings, the best maxout model 
I obtained so far achieves 10.92% error on CIFAR-10 without data
augmentation.

Interestingly, the best hyperparameter settings found so far
are much closer to the parameters that seem to be used in the 'Network
in Network' paper (also submitted to ICLR) than the original settings
used in the maxout paper (and maxout seems to perform on par
with 'Network in Network' when a fully connected layer is used after the convolutional
layers). I will cross-link these results in the discussion to that paper. 

I will include the results into the paper as soon as the full
parameter search is finished and disclose the parameters found.

It would be interesting if Ian could jump in with a short comment on
how exactly he fixed hyperparameters for his original work.