Improving Deep Neural Networks with Probabilistic Maxout Units
Authors: Jost Tobias Springenberg, Martin Riedmiller
ICLR 2014

Abstract: We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).


Peer review:
[Review]:
This manuscript extends the recently proposed “maxout” scheme for neural networks by making the linear subspace pooling stochastic, i.e. by parameterizing the probability of activating each filter as the softmax of the filter responses and then sampling from the resulting discrete distribution. Experimental results are presented on CIFAR10/CIFAR100/SVHN and contrasted with the original maxout work.

Novelty: low
Quality: low to medium

Pros:
- The idea is somewhat interesting and worth trying, and the manuscript itself is generally well-written
- Experiments and hyperparameter choices are generally described in detail, including software packages used
- Attempts a fair-handed comparison between the method and the one they are building off (though see below)

Cons:
- The benchmark results are quite lackluster: the improvements are of questionable statistical significance (see below)
- This questionable gain comes at a >=10x increase in computational cost. 
- The experimental comparisons have several shortcomings (not all of them strictly advantageous to the proposed method, however -- see below).
- The abstract mentions enhancing invariance properties as the goal but there are no attempts to quantify the degree of learned invariance as has been examined in the literature. See, for example, Goodfellow et al’s “Measuring Invariances in Deep Networks” from NIPS 2009 for one such attempt. 

Detailed comments:

The procedure doesn’t yield a single deterministic model at test time, which may be a significant practical drawback as compared with conventional maxout or relu networks.

Still, a more significant drawback to the proposed method is that the test time computation involves several forward propagations through the entire network in order to have a noticeable (but statistically negligible) advantage over maxout. Figure 3b suggests that, on a relatively simple dataset, 10 or more fprops per example may be necessary to yield a significant advantage over the maxout baseline. This is in addition to the cost additional cost incurred by sampling pseudorandom variates as part of the inference process.

A quick computation of a confidence interval (based on the confidence interval of a Bernoulli parameter, i.e. the probability of classifying an example incorrectly) for both the results reported in Goodfellow et al (2013) and this work reveals that the confidence intervals can be seen to overlap significantly:

- CIFAR10 Maxout (no augmentation): 11.68% +/- 0.63%, Probout: 11.35% +/- 0.62%
- CIFAR100 Maxout: 38.57% +/- 0.95%, Probout: 38.14% +/- 0.95%
- SVHN Maxout: 2.47% +/- 0.19%, Probout: 2.39% +/- 0.19%

The same comparison between maxout and the competitors reported in the original work yields non-overlapping confidence intervals for all tasks above. The original maxout work does not achieve a statistically significant improvement over the existing state of the art for CIFAR10, and these authors report no improvement, suggesting the task is sufficiently well regularized by the data augmentation that their complementary regularization does not help.

Hyperparameter search: while the authors reused the exact same architectures and other hyperparameters employed by the original maxout manuscript in an attempt to be fair-handed, I believe this is, in truth, a mistake that disadvantages their method in the comparison. Certain hyperparameters, critically the learning rate and momentum, will be very sensitive to changes in the learning dynamics such as those introduced by the stochastic generalization of the activation function. Even the optimal network architecture may not be the same. The way I would suggest approaching this is with a randomized hyperparameter search (even one in the neighbourhood of the  original settings) wherein the same hyperparameters are tried for both methods, and each point in the shared hyperparameter space is further optimized via randomized search over the hyperparameters specific to probout. This gives the method in question a fairer shot by not insisting that the optimal hyperparameters for maxout be the same as those for probout (there is no a priori reason that they should be).

The choice of temperature schedules also seems like an area that should be further explored. It seems odd that increasing the temperature during training should help (the paper does not specify the length of the linear (inverse) decrease period, this should be noted). What is the intuition for why this helps? How could one validate these intuitions experimentally?

The claim that the stochastic procedure “prevents [filters] from being unused by the network” is dubious. Section 8.2 of the original maxout paper suggests that dropout SGD training alone is remarkably effective in this respect. This claim should be quantitatively investigated and verified if it is to be made at all.

Finally, the paper motivates around probout units learning better invariances, but no attempt is made at quantitatively validating this claim. As it stands, there is a qualitative assessment of the first layer convolutional filters learned, noting that they appear to resemble easily recognizable transformations & quadrature pairs moreso than those learned by vanilla maxout, but I find this somewhat unconvincing. Just because the invariances learned by vanilla maxout are not always obvious to the human practitioner does not mean they are not useful invariances for the model to encode.