Improving Deep Neural Networks with Probabilistic Maxout Units
Authors: Jost Tobias Springenberg, Martin Riedmiller
ICLR 2014

Abstract: We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).


Peer review:
[Review]:
First of all we want to thank both reviewers for their detailed comments. 
Before giving a more elaborate response we want to mention that we have incorporated your suggestions in  a new version of the paper which should appear on arxiv as of Feb 19.

To both reviewers:
As already acknowledged in the paper we agree that our proposed method comes with an attached high computational cost during inference. While this cost, as you mentioned, can be seen as an important practical drawback we believe that explorative research towards novel stochastic regularization techniques (for which an efficient inference procedure is not immediately available) still constitutes a worthwhile research direction from which new, more efficient, regularization techniques could be obtained in the future.

In addition to this increase in computational cost you point out that the improvement over maxout achieved by our approach is minor/non-significant. Although this is acknowledged at the end of the introduction and in the discussion, we have now additionally rephrased the experiments section to reflect this fact more clearly. We want to reiterate that since both methods are tightly coupled - in both their motivation and computational properties as well as our parameter choices - we believe that our results still provide an interesting contribution and can serve as a starting point for future research on the impact of including subspace pooling in deep neural networks. 

To reviewer 2 (Anonymous 4eb4):

We agree that the best hyperparameters for both maxout and probout cannot generally be assumed to coincide. However, as a full hyperparameter search for both methods on all datasets requires significant computational resources we decided to stick to the original parameters in an attempt to make a comparison that is at worst biased towards the maxout results (as we were concerned to not make an unfair comparison to previous results). We are currently running a parameter search on both CIFAR-10 and SVHN in a similar manner as you suggested and will include the results in an additional updated version in the coming days. 

As you point out, our investigation of the invariance properties of the network in the original paper is only a qualitative one. We performed an additional quantitative analysis in the new version of the paper, comparing invariance properties of maxout and probout networks in a manner similar to [1,2].

[1] Koray Kavukcuoglu, Marc'Aurelio Ranzato, Rob Fergus, and Yann LeCun, 'Learning Invariant Features through Topographic Filter Maps', in Proc. International Conference on Computer Vision and Pattern Recognition (CVPR'09), 2009.
[2] Visualizing and Understanding Convolutional Networks M.D. Zeiler, R. Fergus Arxiv 1311.2901 (Nov 28, 2013)