Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary of the paper
---------------------------
The paper addresses the issue of online optimization of hyper-parameters customary involved in deep architectures learning.  The covered framework is limited to regularization parameters. These hyper-parameters, noted $\lambda$, are updated along the training of model parameters $\theta$ by relying on the generalization performance (validation error). The paper proposes a dynamical system including the dynamical update of $\theta$ and the update of the gradient $y$, derivative of $\theta$ w.r.t. to the hyper-parameters. The main contribution of the paper is to propose a way to re-initialize $y$ at each update of $\lambda$ and a clipping procedure of $y$ in order to maintain the stability of the dynamical system. Experimental evaluations on synthetic or real datasets are conducted to show the effectiveness of the approach.

Comments
-------------
- The materials of the paper sometimes may be quite not easy to follow. Nevertheless the paper is quite well written.
- The main contributions of the paper can be seen as an incremental version of (Franceschi et al, 2017) based on the proposal in (Luketina et al., 2016). As such the impact of the contributions appears rather limited even though the experimental results show a better stability of the method compared to competitors.
- One motivation of the approach is to fix the slow convergence of the method in (Franceschi et al, 2017). The paper will gain in quality if a theoretical analysis of the speed-up brought by the proposed approach is discussed.
- The goal of the paper is to address automatically the learning of regularization parameters. Unfortunately, Algorithm 1 involves several other hyper-parameters (namely clipping factor $r$, constant $c$ or $\eta$) which choices are not clearly discussed. It turns that the paper trades a set of hyper-parameters for another one which tuning may be tedious. This fact weakens the scope of the online hyper-parameter optimization approach.
- It may be helpful to indicate the standard deviations of the experimental results.
review_title: Online Hyper-Parameter Optimization

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary of paper:

This work proposes an extension to an existing method (Franceschi 2017) to optimize regularization hyperparameters. Their method claims increased stability in contrast to the existing one.

Summary of review:

This is an incremental change of an existing method. This is acceptable as long as the incremental change significantly improves results or the paper presents some convincing theoretical arguments. I did not find either to be the case. The theoretical arguments are interesting but lacking in rigor. The proposed method introduces hyper-hyperparameters which may be hard to tune. The experiments are small scale and it is unclear how much the method improves random grid search. For these reasons, I cannot recommend this paper for acceptance.

Comments:
1. Paper should cite Domke 2012 in related work section.
2. Should state and verify conditions for application of implicit function theorem on page 2.
3. Fix notation on page 3. Dot is used on the right hand side to indicate an argument but not left hand side for equation after "with respect to \lambda".
4. I would like to see more explanation for the figure in Appendix A. What specific optimization is being depicted? This figure could be moved into the paper's main body with some additional clarification.
5. I did not understand the paragraph beginning with "This poor estimation". Is this just a restatement of the previous paragraph, which concluded convergence will be slow if \eta is too small?
6. I do understand the notation used in equation (8) on page 4. Are <, > meant to denote less than/greater than or something else?
7. Discussion of weight decay on page 5 seems tangential to main point of the paper. Could be reduced to a sentence or two.
8. I would like to see some experimental verification that the proposed method significantly reduces the dropout gradient variance (page 6), if the authors claim that tuning dropout probabilities is an area they succeed where others don't.
9. Experiments are unconvincing. First, only one hyperparameter is being optimized and random search/grid search are sufficient for this. Second, it is unclear how close the proposed method is to finding the optimal regularization parameter \lambda. All one can conclude is that it performs slightly better than grid search with a small number of runs. I would have preferred to see an extensive grid search done to find the best possible \lambda, then seen how well the proposed method does compared to this.
10. I would have liked to see a plot of how the value of lambda changes throughout optimization. If one can initialize lambda arbitrarily and have this method find the optimal lambda, that is more impressive than a method that works simply because of a fortunate initialization.


Typos:
1. Optimization -> optimize (bottom of page 2)
2. Should be a period after sentence starting "Several algorithms" on page 2.
3. In algorithm box on page 5, enable_projection is never used. Seems like warmup_time should also be an input to the algorithm. 

review_title: Needs more work

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
# Summary of paper
The paper proposes an algorithm for hyperparameter optimization that can be seen as an extension of Franceschi 2017 were some estimates are warm restarted to increase the stability of the method. 

# Summary of review
I find the contribution to be incremental, and the validation weak. Furthermore, the paper discusses the algorithm using hand-waiving arguments and lacks the rigor that I would consider necessary on an optimization-based contribution. None of my comments are fatal, but together with the incremental contribution I'm inclined as of this revision towards marginal reject. 

# Detailed comments

1. The distinction between parameters and hyperparameters (section 3) should be revised. First, the definition of parameters should not include the word parameters. Second, it is not clear what "parameters of the regularization" means. Typically, the regularization depends on both hyperparameters and parameters. The real distinction between parameters and parameters is how they are estimated: hyperparameters cannot be estimated from the same dataset as the parameters as this would lead to overfitting and so need to be estimated using a different criterion, but both are "begin learnt", just from different datasets.

2. In Section 3.1, credit for the approach of computing the hypergradient by backpropagating through the training procedure is attributed to Maclaurin 2015. This is not correct. This approach was first proposed in Domke 2012 and refined by Maclaurin 2015 (as correctly mentioned in Maclaurin 2015).

3. Some quantities are not correctly specified. I should not need to guess from the context or related literature what the quantities refer to. theta_K for example is undefined (although I could understand its meaning from the context) and sometimes used with arguments, sometimes without (i.e., both theta_K(lambda, theta_0) and theta_K are used).

4. The hypothesis are not correctly specified. Many of the results used require smoothness of the second derivative (e.g., the implicit function theorem) but these are nowhere stated.

5. The algorithm introduces too many hyper-hyperparameters, although the authors do acknowledge this. While I do believe that projecting into a compact domain is necessary (see Pedregosa 2016 assumption A3), the other parameters should ideally be relaxed or estimated from the evolution of the algorithm.

# Minor

missing . after "hypergradient exactly".

"we could optimization the hyperparam-" (typo)

References:
 Justin  Domke.    Generic  methods  for  optimization-based modeling.  In
International Conference on Artificial Intelligence and Statistics, 2012.

review_title: Interesting idea, weak execution

Meta-review: This paper presents an update to the method of Franceschi 2017 to optimize regularization hyperparameters, to improve stability.  However, the theoretical story isn't so clear, and the results aren't much of an improvement.  Overall, the presentation and development of the idea needs work.
Decision: Reject