Understanding Deep Architectures using a Recursive Convolutional Network
Authors: David Eigen, Jason Rolfe, Rob Fergus, Yann LeCun
ICLR 2014

Abstract: Convolutional neural network models have recently been shown to achieve excellent performance on challenging recognition benchmarks. However, like many deep models, there is little guidance on how the architecture of the model should be selected. Important hyper-parameters such as the degree of parameter sharing, number of layers, units per layer, and overall number of parameters must be selected manually through trial-and-error. To address this, we introduce a novel type of recursive neural network that is convolutional in nature. Its similarity to standard convolutional models allows us to tease apart the important architectural factors that influence performance. We find that for a given parameter budget, deeper models are preferred over shallow ones, and models with more parameters are preferred to those with fewer. Surprisingly and perhaps counterintuitively, we find that performance is independent of the number of units, so long as the network depth and number of parameters is held constant. This suggests that, computational efficiency considerations aside, parameter sharing within deep networks may not be so beneficial as previously supposed.


Peer review:
[Review]:
I like that this paper attempts to disentangle the effects of different parameters.  However, I find the argument that 
more parameters should be spent on more layers somewhat unconvincing.  Taking this to the extreme suggests that every layer should have only 1 feature map, whereas you only go as low as 32.  I don't think overfitting is the only reason a network with M=32,L=16 would (probably) outperform a model with M=1,L=512. 



Some specific comments:


1.1:
1st sentence: you say 'recursive networks' but the refernces you give are for reCURRENT networks.  I would explicitly disambiguate the two, as they are easy to confuse.

Socher et al [24] and [23] are not the same group or researchers.  Saying 'more recently [23], they' implies they are. 


3.1 'For SVHN, we used between 32 and 256 feature maps and between 1 and 8 layers beyond the ﬁrst, also incrementing by powers of 2' -- I would match syntax with the previous sentence for readability. ('For SVHN, we used M = 32,64,128, 256 and L = 1,2,4,8')

'That we were able to train networks at
these large depths is due to the fact that we initialize all W^l_m to the identity' - I feel this sentence could use some explaining

'(except once they go beyond 4 layers for CIFAR-10, at which
point they overﬁt, as indicated by the still-decreasing training error).' - actually, the training error goes up as well from M=32, L=8 to M=32, L=16.  


3.2.1
'For both CIFAR-10 and SVHN,
performance increases as the number of layers increases, although there is an upward tick at 8 layers
in the CIFAR-10 curves due to overﬁtting of the model.' - It looks like this is not true for 32 features on CIFAR-10.


Why not run linear regressions on the results in figures 6 and 7 to quantify the effects and show how significant/unsignificant these factors are?