Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper attempts to build a taxonomy for regularization techniques employed in deep learning. The authors categorize existing works related to regularization into five big categories, including data, model architecture, regularization term, error term and optimization. Subgroups are identified based on certain attributes. 

The paper is written as a survey paper on literatures related to regularization in deep learning. The five top-level categories are quite obvious.  The authors organize works belonging to the first three categories into three big tables, and summarizing the key point of each one using one-liners to provide an overview for readers. While it is a worthy effort, I am not sure it offers much value to readers. Also there is a mix of trainability and regularization. Some of the works were proposed to address trainability issues instead of regularization, for example, densenet, and some of the initialization techniques. 

The authors try to group items in each category into sub-groups according to certain attributes, however,  little explanation on how and why these attributes are identified was provided. For example, in table 1, what kind of information does the transformation space or phase provide in terms of helping readers choosing a particular data transformation / augmentation technique. At the end of section 3, the authors claim that dropout, BN are close to each other. Please elaborate on this point. 

The authors offer some recommendation on how to choose or combine different regularization techniques at the end of the paper. However, it is not clear from reading the paper where these insights came from. 
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper is unusual in that it is more of a review than contributing novel knowledge. It considers a taxonomy of all the ways that machine learning (mostly deep learning) methods can achieve a form of regularization. 

Unfortunately, it starts with a definition of regularization ('making the model generalize better') which I believe misses the point which was made in Goodfellow et al 2016 ('intend to improve test error but not necessarily training error'), i.e., that we would like to separate as much as possible the regularization effects from the optimization effect. Indeed, under the definition proposed here, any improvement in the optimizer could be considered like a regularizer, so long as we are not in the overfitting regime. That does not sound right to me.

There are several places where the authors make TOO STRONG STATEMENTS, taking for truth what are simply beliefs with no strong supporting evidence (at least published). This is not good for a review and when making recommendations.

The other weakness I estimate in this paper is that I did not get a sense that the taxonomy really helped us (me at least) to get insight into the different mentions being cited. Besides the obvious proposal to combine ideas to write new papers (but we did not need that paper to figure that out) I did not find much meat in the 'future directions' section.

However, I that except in a few places the understand of the field displayed by the authors is pretty good and, with correction, could serve as a useful reference for students of deep learning. The recommendations were reasonable although lacking empirical support (or pointers to the literature), so I would take them somewhat carefully, more as the current 'group think' than ground truth.

Finally, here a few minor points which could be fixed.

Eq. 1: in typical DL, minimization is approximate, not exact, so the proposed formalism does not reflect reality.

Eq. 4: in many cases, the noise is not added (e.g. dropout), so that should be clarified there.

page 3, first bullet of 'Effect on the data representation': not clear, may want to give translations as an example of such transformations,.

page 8, activation functions: the ReLU is actually older than the cited papers, it was used by computational neuroscientists a long time ago. Jarrett 2009 did not use the ReLU but an absolute-value rectifier and it was Glorot 2011 who showed that the ReLU really kicked ass for deeper networks. Nair 2010 used the ReLU in a very different context (RBMs), not really feedforward multi-layer networks where it shines now.
In that same section (and probably elsewhere) there are TOO STRONG STATEMENTS, e.g., the "facts" mentioned are not facts but merely folk belief, as far as I know, and I would like to see well-done supporting evidence before treating those as facts. Note that approximating the sigmoid precisely would require many ReLUs!

page 8: it is not clear how multi-task learning fits under the 'architecture' formalism provided at the beginning of section 4.

section 7 (page 10): there is earlier work on the connection between early stopping and L2 regularization, at least dating back to Ronan Collobert's PhD thesis (with neural nets), probably earlier for linear systems.
review_title: Review of 'Regularization for DL: a taxonomy'

final_score: rating: 4: Ok but not good enough - rejection
review_content: The main aim of ICLR conference, at least as it is written on its website, is to provide new results on theories, methods and algorithms, supporting further breakthroughs in AI and DL.

In this respect the authors of the paper claim that their “systematic approach enables the discovery of new, improved regularization methods by combining the best properties of the existing ones.”

However, the authors did not provide any discoveries concerning new approaches to regularisation supporting this claim. Thus, the main contribution of the paper is that the authors made a review and performed classification of available regularisation methods. So, the paper is in fact a survey paper, which is more appropriate for full-scale journals. The work, developed by the authors, is really big. However, I am not sure it will bring a lot of benefits for readers except those who need review for some reports, introductions in PhD thesis, etc.

Although the authors mentioned some approaches to combine different regularisations, they did not performed any experiments supporting their ideas.

Thus, I think that
- the paper is well written in general,
- it can be improved (by taking into account several important comments from the Reviewer 2) and served as a review paper in some appropriate journal,
- the paper is not suited for ICLR proceedings due to reasons, mentioned above.
review_title: interetsing review but not appropriate for ICLR proceedings

Meta-review: The paper is a well-written review of regularization approaches in deep learning.
It does not offer novel approaches or novel insight with empirically demonstrated usefulness
=> ICLR is not the appropriate venue for it.
Decision: Reject