Understanding Deep Architectures using a Recursive Convolutional Network
Authors: David Eigen, Jason Rolfe, Rob Fergus, Yann LeCun
ICLR 2014

Abstract: Convolutional neural network models have recently been shown to achieve excellent performance on challenging recognition benchmarks. However, like many deep models, there is little guidance on how the architecture of the model should be selected. Important hyper-parameters such as the degree of parameter sharing, number of layers, units per layer, and overall number of parameters must be selected manually through trial-and-error. To address this, we introduce a novel type of recursive neural network that is convolutional in nature. Its similarity to standard convolutional models allows us to tease apart the important architectural factors that influence performance. We find that for a given parameter budget, deeper models are preferred over shallow ones, and models with more parameters are preferred to those with fewer. Surprisingly and perhaps counterintuitively, we find that performance is independent of the number of units, so long as the network depth and number of parameters is held constant. This suggests that, computational efficiency considerations aside, parameter sharing within deep networks may not be so beneficial as previously supposed.


Peer review:
[Review]:
This paper analyzes the effect of different hyper-parameters for a convolutional neural network: the number of convolutional layers (L), the number of feature maps (M), the total number of free parameters in the model (P).  The main challenge is the tight relation between these three hyper-parameters. To study the effect of each factor independently, a recurrent architecture is proposed where weights are tied between different convolutional layers so that the number of layers can be varied without changing the total number of parameters. Pooling is only applied to the very first layer but is not applied in any of the tied layers on top.

While It is important to see experimental papers that offers analysis of the effect of different design parameters for neural networks like this one, weight tying across different convolutional layers is a bit artificial for this task (you scan the whole image at once in each layer). 

The main take home message of this paper is that varying the number of feature maps is not important given that the number of free parameters and model depth are held constant. It is important to add experiments/arguments that show if the same conclusion holds when pooling is used for example. 

General comments:
- What are the kind of features learned by the tied network vs the untied (normal) one? Does the network work around tying by dedicating some features to be used most of the time per specific hidden layer but not by the others? (as if it is working in an untied regime but with smaller number of feature maps per layer).

- Don’t think table 1 is needed because the paper is not aiming at achieving the best ever accuracy but rather exploring different factors affecting performance.

- Regarding writing, the paper gets repetitive at points, for example, the information in table 2 is stated in a paragraph in page 7. The same conclusions are stated in the same way multiple times.

- In page 8 “We then compared their classification performance after training as described in Section 2.2.” Is there something specific you mean by “after training”?