Understanding Deep Architectures using a Recursive Convolutional Network
Authors: David Eigen, Jason Rolfe, Rob Fergus, Yann LeCun
ICLR 2014

Abstract: Convolutional neural network models have recently been shown to achieve excellent performance on challenging recognition benchmarks. However, like many deep models, there is little guidance on how the architecture of the model should be selected. Important hyper-parameters such as the degree of parameter sharing, number of layers, units per layer, and overall number of parameters must be selected manually through trial-and-error. To address this, we introduce a novel type of recursive neural network that is convolutional in nature. Its similarity to standard convolutional models allows us to tease apart the important architectural factors that influence performance. We find that for a given parameter budget, deeper models are preferred over shallow ones, and models with more parameters are preferred to those with fewer. Surprisingly and perhaps counterintuitively, we find that performance is independent of the number of units, so long as the network depth and number of parameters is held constant. This suggests that, computational efficiency considerations aside, parameter sharing within deep networks may not be so beneficial as previously supposed.


Peer review:
[Review]:
I understand that M=1 still has 64 units, but I don't see why you would need 17,379 layers, although I haven't thought about it much.  I think it would actually be nice to include in the paper an equation for the total number of parameters (given # of layers, kernels, tied/untied) of a model with the structure you are using.

My point here, though, is not that you should try to test these extreme cases.  Rather, I am using it as a thought experiment to make an argument against (my understanding of) your interpretation of the results you present.  

It appears to me that you are claiming that the ONLY reason performance decreases sometimes in deeper models (compared with shallower models with the same # of parameters) is because of overfitting.  But I think this is incorrect, for two reasons.  The M=1 thought experiment is one reason that demonstrates the intuition of my belief, the other (empirical) reason is that the training error increases from M=32, L=8 to M=32, L=16 (not just test error).  If the increase in test error in this case were due to overfitting, I would expect the training error to increase as well.  Indeed, in the write-up, you state that the decreasing training error demonstrates that overfitting is taking place '(except once they go beyond 4 layers for CIFAR-10, at which point they overfit, as indicated by the still-decreasing training error)', but since the training error is NOT still decreasing (for all cases), this interpretation does not appear correct to me.