Venue: ICLR 2017 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The authors have grouped recent work in convolutional neural network design (specifically with respect to image classification) to identify core design principles guiding the field at large. The 14 principles they produce (along with associated references) include a number of useful and correct observations that would be an asset to anyone unfamiliar with the field. The authors explore a number of architectures on CIFAR-10 and CIFAR-100 guided by these principles.

The authors have collected a quality set of references on the subject and grouped them well which is valuable for young researchers. Clearly the authors explored a many of architectural changes as part of their experiments and publicly available code base is always nice.

Overall the writing seems to jump around a bit and the motivations behind some design principles feel lost in the confusion. For example, "Design Pattern 4: Increase Symmetry argues for architectural symmetry as a sign of beauty and quality" is presented as one of 14 core design principles without any further justification. Similarly "Design Pattern 6: Over-train includes any training method where the network is trained on a harder problem than necessary to improve generalization performance of inference" is presented in the middle of a paragraph with no supporting references or further explanation.

The experimental portion of this paper feels scattered with many different approaches being presented based on subsets of the design principles. In general, these approaches either are minor modifications of existing networks (different FractalNet pooling strategies) or are novel architectures that do not perform well. The exception being the Fractal-of-Fractal network which achieves slightly improved accuracy but also introduces many more network parameters (increased capacity) over the original FractalNet.


Preliminary rating:
It is a useful and perhaps noble task to collect and distill research from many sources to find patterns (and perhaps gaps) in the state of a field; however, some of the patterns presented do not seem well developed and include principles that are poorly explained. Furthermore, the innovative architectures motivated by the design principles either fall short or achieve slightly better accuracy by introducing many more parameters (Fractal-Of-Fractal networks). For a paper addressing the topic of higher level design trends, I would appreciate additional rigorous experimentation around each principle rather than novel architectures being presented.
review_title: Review

final_score: rating: 3: Clear rejection
review_content: The paper formulates a number of rules for designing convolutional neural network architectures for image processing and computer vision problems. Essentially, it reads like a review paper about modern CNN architectures. It also proposes a few new architectural ideas inspired by these rules. These are experimentally evaluated on CIFAR-10 and CIFAR-100, but seem to achieve relatively poor performance on these datasets (Table 1), so their merit is unclear to me.

I'm not sure if such a collection of rules extracted from prior work warrants publication as a research paper. It is not a bad idea to try and summarise some of these observations now that CNNs have been the model of choice for computer vision tasks for a few years, and such a summary could be useful for newcomers. However, a lot of it seems to boil down to common sense (e.g. #1, #3, #7, #11). The rest of it might be more suited for an "introduction to training CNNs" course / blog post. It also seems to be a bit skewed towards recent work that was fairly incremental (e.g. a lot of attention is given to the flurry of ResNet variants).

The paper states that "it is universal in all convolutional neural networks that the activations are downsampled and the number of channels increased from the input to the final layer", which is wrong. We already discussed this previously re: my question about design pattern 5, but I think the answer that was given ("the nature of design patterns is that they only apply some of the time") does not excuse making such sweeping claims. This should probably be removed.

"We feel that normalization puts all the layer's input samples on more equal footing, which allows backprop to train more effectively" (section 3.2, 2nd paragraph) is very vague language that has many possible interpretations and should probably be clarified. It also seems odd to start this sentence with "we feel", as this doesn't seem like the kind of thing one should have an opinion about. Such claims should be corroborated by experiments and measurements. There are several other instances of this issue across the paper.

The connection between Taylor series and the proposed Taylor Series Networks seems very tenuous and I don't think the name is appropriate. The resulting function is not even a polynomial as all the terms represent different functions -- f(x) + g(x)**2 + h(x)**3 + ... is not a particularly interesting object, it is just a nonlinear function of x.

Overall, the paper reads like a collection of thoughts and ideas that are not very well delineated, and the experimental results are unconvincing.
review_title: null

final_score: rating: null
review_content: Design pattern 5: while I'm aware that the scope of the paper is image classification only, even within this domain there are tasks where pyramidal shape is not necessarily desirable, e.g. semantic segmentation (which is a pixelwise classification problem). Could the authors explain whether they intended to include such tasks, and otherwise could they more clearly define which types of image classification tasks are considered and which aren't?

Design pattern 6: only one sentence of the paper discusses this, but it's not clear to me what is meant by "training a network on a harder problem than necessary to improve generalisation performance". Does this encompass any form of regularisation? Because I would argue that some forms of regularisation actually make the training problem easier (i.e. improve convergence as well as have a regularising effect). Could the authors clarify this?

Compared to the others, design pattern 14 seems oddly specific: most people treat maxout as yet another nonlinearity. Could the authors comment on why they treat maxout more as an essential design element rather than a choice of nonlinearity?

Freeze-path: why is this expected to be a useful technique? As long as a given branch is still used in the forward pass, not updating its weights has no effect whatsoever on the updates in the rest of the network. If anything it seems like it would lead to underfitting. I think this warrants a better justification.
review_title: a few questions

final_score: rating: null
review_content: The accuracies are presented as the mean of test accuracy of the networks every 500 iterations for the final 3000 of training. This seems non-standard, why report results in this way? Could you also provide the standard deviation of these results? 


review_title: Experimental Results

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors take on the task of figuring out a set of design patterns for current deep architectures - namely themes that are recurring in the literature.  If one may say so, a distributed representation of deep architectures. 

There are two aspects of the paper that I particularly valued: firstly, the excellent review of recent works, which made me realize how many things I have been missing myself.  Secondly, the "community service" aspect of helping someone who starts figure out the "coordinate system" for deep architectures - this could potentially be more important than introducing yet-another trick of the trade, as most other submissions may do.

However I think this work is still half-done, and even though working on this project is a great idea, the authors do not yet do it properly. 

Firstly, I am not too sure how the choice of these 14 patterns was made. Maxout for instance (pattern 14) is one of the many nonlinearities (PreLU, ReLU, ...) and I do not see how it stands on the same grounds as something as general as "3 Strive for simplicity".

Similarly some of the patterns are as vague as "Increase symmetry" and are backed up by statements such as "we noted a special degree of elegance in the FractalNet". I do not see how this leads to a design pattern that can be applied to a new architecture - or if it applies to anything other than the FractalNet. 

Some other patterns are phrased with weird names "7 Cover the problem space" - which I guess stands for dataset augmentation; or "6 over-train" which is not backed up by a single reference. Unless the authors relate it to regularization (text preceding "overtrain"), which then has no connection to the description of "over-train" provided by the authors ("training a network on a harder problem to improve generalization"). If "harder problem" means one where one adds an additional term (i.e. the regularizer), the authors are doing harm to the unexperienced reader, confusing "regularization" with something that sounds like "overfitting" (i.e. the exact opposite).

Furthermore, the extensions proposed in Section 4 seem a bit off tune - in particular I could not figure out 
-how the Taylor Series networks stem from any of the design patterns proposed in the rest of the paper. 
-whether the text between 4.1 and 4.1.1 is another of the architecture innovations (and if yes, why it is not in the 4.1.2, or 4.1.0) 
-and, most importantly, how these design patterns would be deployed in practice to think of a new network. 

To be more concrete, the authors mention that they propose the "freeze-drop-path" variant from "symmetry considerations" to "drop-path". 
Is this an application of the "increase symmetry" pattern? How would "freeze-drop-path" be more symmetric that "drop-path"?
 Can this be expressed concretely, or is it some intuitive guess? If the second, it is not really part of applying a pattern, in my understanding. If the first, this is missing. 


What I would have appreciated more (and would like to see in a revised version) would have been a table of "design patterns" on one axis, "Deep network" on another, and a breakdown of which network applies which design pattern. 

A big part of the previous work is also covered in cryptic language - some minimal explanation of what is taking place in the alternative works would be useful. 


  

review_title: interesting direction - but not really solid

Meta-review: The authors agree with the reviewers that this manuscript is not yet ready.
Decision: Reject