Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks
Authors: Andrew Davis, Itamar Arel
ICLR 2014

Abstract: Scalability properties of deep neural networks raise key research questions, particularly as the problems considered become larger and more challenging. This paper expands on the idea of conditional computation introduced by Bengio, et. al., where the nodes of a deep network are augmented by a set of gating units that determine when a node should be calculated. By factorizing the weight matrix into a low-rank approximation, an estimation of the sign of the pre-nonlinearity activation can be efficiently obtained. For networks using rectified-linear hidden units, this implies that the computation of a hidden unit with an estimated negative pre-nonlinearity can be ommitted altogether, as its value will become zero when nonlinearity is applied. For sparse neural networks, this can result in considerable speed gains. Experimental results using the MNIST and SVHN data sets with a fully-connected deep neural network demonstrate the performance robustness of the proposed scheme with respect to the error introduced by the conditional computation process.


Peer review:
[Review]:
Summary of contributions: Proposes a specific means to achieve Bengio's goal of 'conditional computation'. The specific mechanism is to use the sign of approximate activations to predict which activations are zero, so that the full-rank values of rectified linear units don't need to be computed for units that are probably 0. 

Novelty: moderate
Quality: low

Pros:
-Demonstrates that the sign approximation method slightly improves test error.
Cons:
-The main purpose of the work is to reduce computational cost, but computational cost is never quantified
-It's not obvious that predicting which units are zero will lead to the ability to reduce computation cost
-If computational cost can in fact be reduced, it's not clear that the cost of the predictor is less than the cost that its use could remove
-The baselines that are improved upon are not competitive
-It's not clear that the hyper parameters were chosen in a way that makes the comparisons fair
-Some of the commentary is misleading or questionable

Detailed comments:

pg 1
I don't think you can necessarily conclude that the computational requirements to train a large net decreased between the publication of [12] and the publication of [4]. The purpose of [12] was largely to demonstrate how many machines their system could use. At the ICML poster session where [12] was first presented, I heard someone else ask Quoc Le if the number of machines was actually necessary to get the results they had, and he said the high number of machines was not necessary, they just wanted to claim credit for networking that many machines to train a neural net. It's likely that most of the savings in [4] relative to [12] are just due to the fact that most of the computation in [12] was wasted.

pg 2

Section 2.1

I don't think you can conclude that redundancy of parameters implies redundancy of activations. For example, two units with the same weight vector have very redundant parameters, but changing just their bias unit can give them very different activations. Presumably nets would not benefit much from increased size if the units were truly redundant.

Simply being able to predict which units in a net are going to have zero activation doesn't necessarily mean that you will get computational savings. You need the cost of predicting which units are zero to be cheap enough that on average running both the predictor net and the predicted subset of the main net is cheaper than running just the main net. You also need some mechanism for cheaply running subsets of the main net. Doing sparse matrix operations with an arbitrary, dynamically updated sparsity mask is not significantly more efficient than just doing dense operations, to my knowledge.

I think you're missing the fundamental idea of conditional computation, at least as described in [2] and [3]. Bengio is not advocating figuring out cheaper ways of computing exactly the same function we already compute. He's advocating figuring out how to compute only some of the features that are most relevant to processing a particular input. What you're doing is more of a software engineering optimization where you figure out which features are zero. Bengio is advocating something more ambitious, figuring out which features are unnecessary even if they are nonzero. Another issue with your approach is that you're trying to individually predict which features are nonzero. This means you need to be able to have the predictor make an explicit decision for each of them individually, so your runtime is going to have asymptotic cost which is at least linear in the number of units. Bengio is advocating making predictions about entire groups of units at the same time. If done right, this has the chance of having cost only logarithmic in the number of features in the model, or, to phrase it more excitingly, being able to run a model that has exponentially more features than runtime and memory requirements.

Section 2.2
I'm not sure sparsity of the representation is the correct explanation for the performance of the rectifier nets in [7]. The sparse rectifier nets in [7] have sparse activations, but they have equally sparse gradients. i.e., the gradient through a unit is zero for an example if and only if the activation is zero for that unit. This paper ( http://arxiv.org/pdf/1302.4389v4.pdf ) shows that similar nets can perform well if they have sparse gradients but non-sparse activations. See especially Fig 2. of that paper. So sparse gradients may be more important than sparse activations.


pg 3

Your presentation is confusing because you refer the reader to fig 3.1 before defining S, but S is used in the caption of this figure.

Usually one does not use italic letters when denoting the 'sgn' function, since italic letters are used for algebraic variables.

pg 4

Section 3.3

I don't understand the paragraph on dropout and sparsity. The most problematic sentence is 'During training, the sparsity of the network is likely less than p for each minibatch.' Does 'more sparsity' mean fewer active units? So less sparsity means more active units? If we set p to zero, this implies that every unit is likely active. But that is the default SGD training algorithm, in which we certainly know that not all the units are active. Also, are you counting units zeroed by dropout as contributing to the sparsity? What about units that are zeroed by their input being negative? Why should their be any straightforward and simple relation between the dropout p and the number of units zeroed by their inputs being negative?


Section 3.4

1 is probably a pretty high value for the initial biases. Was dropout applied to the input as well as the hidden layers? If so, p = 0.5 is probably too high for use on the input.


pg 5

Table 4.1: Where did these hyper parameters come from? Why should the same hyper parameters be used for all conditions? How can we know that your 'control' curve in Fig 4.1 is a good baseline to improve over? If you hand-tuned your hyper parameters to work well with the S predictor net active, then it's not very surprising the control would perform worse. I'm not saying you intentionally did this, but without a description of your methodology for obtaining the hyperparameters, it seems likely that you implicitly did this while trying to get your new method to work.

It seems like your experiments are bypassing the most important part of the paper, which is to demonstrate a speedup. I don't see anything about timing experiments here. I suspect you can't actually get a speedup by this approach due to the difficulty of leveraging a speedup from sparsity if the sparsity pattern is dynamic rather than fixed, also due to the difficulty of leveraging sparsity on GPU if there is not a regular structure to the sparsity pattern. Moreover, as far as I can tell, you don't quantify the cost of frequently running SVD during training.

Are your nets actually large enough to be interesting? The bigger of the two has about three million parameters and doesn't seem to be able to fit the SVHN dataset all that well. Conditional computation is mostly interesting for its ability to push the frontier of network capacity, so I'm not sure the scenario you've experimented on is really interesting from the perspective of evaluating a conditional computation method.

pg 6

Similar criticism for MNIST, except here I'm confident that your baseline control system is not good. With dropout you should be able to get down to < 1% test error.