Understanding Deep Architectures using a Recursive Convolutional Network
Authors: David Eigen, Jason Rolfe, Rob Fergus, Yann LeCun
ICLR 2014

Abstract: Convolutional neural network models have recently been shown to achieve excellent performance on challenging recognition benchmarks. However, like many deep models, there is little guidance on how the architecture of the model should be selected. Important hyper-parameters such as the degree of parameter sharing, number of layers, units per layer, and overall number of parameters must be selected manually through trial-and-error. To address this, we introduce a novel type of recursive neural network that is convolutional in nature. Its similarity to standard convolutional models allows us to tease apart the important architectural factors that influence performance. We find that for a given parameter budget, deeper models are preferred over shallow ones, and models with more parameters are preferred to those with fewer. Surprisingly and perhaps counterintuitively, we find that performance is independent of the number of units, so long as the network depth and number of parameters is held constant. This suggests that, computational efficiency considerations aside, parameter sharing within deep networks may not be so beneficial as previously supposed.


Peer review:
[Review]:
An analysis of deep networks is presented that tries to determine the effect of various convnet parameters on final classification performance.  A novel convnet architecture is proposed that ties weights across layers.  This enables clever types of analysis not possible with other architectures.  In particular, the number of layers can be increased without increasing the number of parameters, thus allowing the authors to determine whether the number of layers is important or whether the number of parameters is important independently.  (Normally, these are confounding factors that are hard to judge separately.)  Several experiments are proposed that independently vary the number of maps, number of parameters, and number of layers.  It is reported that while the number of maps appears to be irrelevant in this setup, the number of layers and number of parameters are very important [more is better!]

This is a pretty unorthodox but interesting form of analysis.  I think it is worth highlighting that “number of layers” experiment, since I didn’t see immediately how you could do this with another tying strategy.  The results confirm our intuitions about important parameters, but also suggest that perhaps weight-tying spatially is one place for improvement.

I am a bit worried that there are caveats to this analysis that could be better analyzed.  For example, section 3.2.2 shows the “untied” system working better than “tied” in 3.2.2, but could this have more to do with the finicky nature of recurrent models (e.g., failure to find a good minimum) than the number of parameters?  Section 3.2.3 might implicitly address this:  for fixed number of layers and parameters, the tied model performs about the same.  If it had performed worse again, we might have been tricked into thinking that the number of maps mattered, when it could have implied that the tied model itself was a worse performer.  A bit more clarity about the potential caveats of this analysis and the implications of each experiment would help.

One experiment I was surprised not to see:  holding M and P fixed, compare a tied model with L layers to an untied model with fewer layers [to keep P constant].  This is apparently not what is done in 3.2.1, but might help address my concern above.

Pros:

Clever, novel analysis of interplay of deep network characteristics and their effect on classification performance.

Useful rules of thumb that may benefit future work.  Appears to confirm widely-held intuitions about depth and scale of models.

Cons:

The analysis method might be introducing effects that are not clear (e.g., the effect of using recurrence on the optimization problem).

Hard to know how these results will transfer to more typical convnets that use max pooling, LCN, etc.

Other:

In much of the analysis I thought it may be more useful to consider training error as the main metric for comparing networks.  At this point, being able to achieve low training error is the main goal of fiddling with the model size, etc. and testing error/generalization is governed by a different bag of tricks [synthetic data, Mechanical Turk, unsupervised learning, extensive cross-validation, bagging, etc.]