Understanding Deep Architectures using a Recursive Convolutional Network
Authors: David Eigen, Jason Rolfe, Rob Fergus, Yann LeCun
ICLR 2014

Abstract: Convolutional neural network models have recently been shown to achieve excellent performance on challenging recognition benchmarks. However, like many deep models, there is little guidance on how the architecture of the model should be selected. Important hyper-parameters such as the degree of parameter sharing, number of layers, units per layer, and overall number of parameters must be selected manually through trial-and-error. To address this, we introduce a novel type of recursive neural network that is convolutional in nature. Its similarity to standard convolutional models allows us to tease apart the important architectural factors that influence performance. We find that for a given parameter budget, deeper models are preferred over shallow ones, and models with more parameters are preferred to those with fewer. Surprisingly and perhaps counterintuitively, we find that performance is independent of the number of units, so long as the network depth and number of parameters is held constant. This suggests that, computational efficiency considerations aside, parameter sharing within deep networks may not be so beneficial as previously supposed.


Peer review:
[Review]:
Summary of contributions: Studies the effect of # of parameters, # of layers, and # of units on convolutional net performance. Uses recurrence to run nets that have e.g. more layers but not more parameters in order to distinguish the effects of these three properties.

Novelty: moderate
Quality: low

Pros:
	-Nice empirical demonstration that more parameters helps.
Cons:
	-Does not study dropout. I think dropout is really important for this kind of paper, because dropout has a strong effect on the optimal model size. Also, dropout is crucial part of the state of the art system on both CIFAR-10 and SVHN, so it seems out of place for a paper on how to set hyperparameters to good performance out of a neural net to disregard one of the most important techniques for getting good performance.
	-Insufficient tuning of hyperparameters.
	-Support for the claims in the abstract seems weak, with many experiments going against the claims
	-The stated goal is to provide guidance for how to set hyperparameters so that practitioners don’t have to resort to trial and error. But I don’t really see anything here that prevents that. For example, Fig 4a shows standard U-shaped curves for the # of layers hyperparameter. The paper says “adding more layers tends to increase performance” but this is only true on the left side of the U! The whole point of trial and error is to figure out where the bottom of the U is, and this paper completely ignores that.
	-The kind of parameter tying considered in this paper is not one that is typically used in practice, at least not for this kind of problem. The conclusions are therefore not all that helpful. i.e., the authors introduce a new form of parameter tying, and then show it isn’t useful. We don’t need to publish that conclusion, because no one is using this useless form of parameter tying anyway.
	-The authors don’t investigate the effect of the tiling range of tiled convolution, which is a form of control on the degree of parameter sharing that people actually use. It’d be much more interesting to study this form of parameter sharing. (This paper feels a bit like it started off as a “new methods” paper advocating convolutional recurrence, and then when the new method didn’t perform well, the authors tried to salvage it as an “empirical investigation” paper, but the empirical investigation part isn’t really targeted at the methods that would be most useful to study)

Detailed comments:

1.1 Related work:

	You should also mention “Multi-Prediction Deep Boltzmann Machines”, Goodfellow et al 2013. This paper uses recurrent networks on the image datasets MNIST and NORB. Like DrSAE, it is discriminative. It may be interpreted as a form of autoencoder, like the methods you mention in the second paragraph.

	
2 Approach:
	Your approach is definitely not the first to use recurrence and convolution in the same model. It’s probably worth discussing similarities and differences to Honglak Lee’s convolutional DBN. He describes performing mean field inference in this model. The mean field computations are essentially forward prop in a convolutional recurrent architecture, but the connectivity is different than in yours, since each update reads from two layers, and some of the weight matrices are constrained to be the transpose of each other rather than being constrained to be equal to each other.

	It’s also probably worth discussing how you handle the boundaries of the image, since this has a strong effect on the performance of a convolutional net. Since you say all the layers have the same size, I’m assuming you implicitly pad the hidden layer with zeros when running convolution so that the output of the discrete convolution operation has the same size as the input.

	
2.1 Instantiation on CIFAR-10 and SVHN

	I don’t know what it means to put the word “same” in quotes. I’m assuming this refers to the zero padding that I described above, but it’s worth clarifying.

2.2
	I think it’s fairly disappointing that you don’t train with dropout.
	How did you choose this one fixed learning rate and momentum value? How do you know it doesn’t bias the results? For example, if you find that deeper models are better, are you really finding that deeper models are better in general, or are you just finding that deeper models are more compatible with this specific learning rate and momentum setting?
	It seems especially important to tune the learning rate in this work because varying the amount of parameter sharing implies varying the number of gradient terms that affect each parameter. The speed at which the parameters move is probably much higher for the nets with many recurrent steps than it is for the nets with no recurrence.

3.1
	“That we were able to train networks at these large depths is due to the fact that we initialize all W to the identity” -> it’s not obvious to me that it should be hard to train convolutional rectifier networks at most of these depths. For example, Google’s house number transcription paper submitted to this conference at the same time trains a 12 layer mostly convolutional network with no mention of network depth posing a challenge or requiring special initialization. The maxout paper reports difficulty training a 7 layer rectifier net on MNIST, but that was densely connected, not convolutional. Was it only difficult to train the recurrent nets, or also the untied ones? This is important to explain, since if the recurrent nets are significantly harder to optimize, that affects the interpretation of your results.
	Are the higher layer weights for all of the networks initialized to the identity, or only the ones with tied parameters? Is it literally identity or identity times some scalar? If it’s literally identity rather than identity times some scalar, it might be too hard for SGD to shrink the initial weights and learn a different more interesting function. Have you tried other initializations that don’t impose such a strong hand-designed constraint, such as James Martens’ sparse initialization, where each hidden units gets exactly k randomly chosen non-zero incoming weights? This initialization scheme is also described as making it easier to train deep or recurrent nets, and it seems to me like it doesn’t trap the recurrent layer as being a fairly useless memory layer that mostly functions to echo its input.

	“Likewise, for any given combination of feature maps and layers, the untied model outperforms the tied one, since it has more parameters.” I don’t agree with the claim that the untied model performs better because it has more parameters. This would make sense if the tied model was in the underfitting regime. But you have already said in the same paragraph that many of the tied models are in the overfitting regime. If you look at fig 2. there are several points where both the tied and untied model have 0 training error and the tied model has higher validation set error. If the correct story here is overfitting due to too many parameters, then the untied model should do worse. I suspect what’s going on here is something like the identity initialization being a bad prior, so that you fit the training set in a way that doesn’t generalize well, or maybe just your choice of a single momentum and learning rate setting for all experiments ended up benefiting the untied model somehow. For example, as I said above, the recurrent nets will generally have larger gradients on each parameter, so maybe the high learning rate makes the recurrent net adapt too much to the first few minibatches it sees. 

	Fig 2
		In the abstract you say “for a given parameter budget, deeper models are preferred over shallow ones.” It would be nice if on the plot on the left you evaluted points along the parameter budget contour lines instead of points on a grid, since the grid points don’t always hit the contour lines. As is, it’s hard to evaluate the claim from the abstract. However, I don’t see a lot of support for it. The best test error you get is toward the bottom right: 0.160 at the rightmost point in the second row from the bottom. Of course, this is the only point on that parameter budget contour, so it may just be winning because of its cost. However, if I look for the point with the most depth, I see one with 0.240 near the 2^18 contour line. At the bottom right of this contour line, the shallow but wide model gets 0.205.
		Overall, here is my summary of all your contour lines:
			2^16: only one point on it
			2^17: Contradicts claim
			2^18: Contradicts claim
			2^19: Contradicts claim
			2^20: Supports claim (sort of, points aren’t that close to contour line)
			2^21: Supports claim (sort of, points aren’t that close to contour line)
			2^22: Supports claim (sort of, points aren’t that close to contour line)
		So it seems to me that this plot contradicts the claim from the abstract at least as much as it supports it.

	Right figure:
		This supports the claim in your abstract.

	Table 1: While it does make sense to compare *these* experiments against methods that don’t use dropout or data augmentation, I don’t think it makes sense for these to be your only experiments. I think the case for excluding data augmentation from consideration is getting very weak. There is now a lot of commercial interest in using neural nets on very large datasets. Augmentation of small datasets provides a nice low-cost proxy for exploring this regime.
As far as I know, the main reasons for not considering data augmentation are 1) data augmentation requires knowledge of the data domain, in this case that the input is an image and the output is invariant to shifts in the input. But you are already exploiting exactly that same knowledge by using a convolutional net and spatial pooling. 2) gaining improvements in performance by improving data augmentation techniques distracts attention from improving machine learning methods and focuses it on these more basic engineering tricks. But I’m not asking you to engineer new data augmentation methods here; you can just use exactly the same augmentation as many previous authors have already used on CIFAR-10 and SVHN.
I don’t think there is any valid case at all for excluding stochastic regularization from consideration. It doesn’t require any knowledge of the data domain and it is absolutely a machine learning technique rather than just an engineering trick. Moreover, it is computationally very cheap, and state of the art across the board. By refusing to study stochastic regularization you are essentially insisting on studying obsolete methods. The only regime in which stochastic regularization is not universally superior to deterministic backprop is in the extremely large data domain, which as academics you probably don’t have access to and you are also actively avoiding by not using data augmentation.

	Fig 2 and 3 in general: I understand it’s too expensive to extensively cross-validate every point on these plots, but I think it’d be good to pick maybe 4 points for each plot (maybe the upper-left and lower-right of two different contour lines) and run around 10 experiments each for those 4 points. Overall that is 80 training runs, which I think is totally reasonable. The current plots are somewhat interesting but it’s hard to have much confidence that the trends they indicate are real. Obtaining higher confidence estimates of the real value of a small number of points would help a lot to confirm that the trends are actually caused by the # of feature maps and depth rather than compatibility with a fixed optimization scheme.

Section 4:
	I don’t think the “received wisdom” is that more depth is always better, just that the optimal depth is usually greater than 1.
	You say your experiments show that more depth is better for a fixed parameter budget, but doesn’t Fig 2. (right) contradict this?