Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper shows a simple method for predicting the performance that neural networks will achieve with a given architecture, hyperparameters, and based on an initial part of the learning curve.
The method assumes that it is possible to first execute 100 evaluations up to the total number of epochs.
From these 100 evaluations (with different hyperparameters / architectures), the final performance y_T is collected. Then, based on an arbitrary prefix of epochs y_{1:t}, a model can be learned to predict y_T.
There are T different models, one for each prefix y_{1:t} of length t. The type of model used is counterintuitive for me; why use a SVR model? Especially since uncertainty estimates are required, a Gaussian process would be the obvious choice. 

The predictions in Section 3 appear to be very good, and it is nice to see the ablation study.

Section 4 fails to mention that its use of performance prediction for early stopping follows exactly that of Domhan et al (2015) and that this is not a contribution of this paper; this feels a bit disingenious and should be fixed.
The section should also emphasize that the models discussed in this paper are only applicable for early stopping in cases where the function evaluation budget N is much larger than 100.
The emphasis on the computational demand of 1-3 minutes for LCE seems like a red herring: MetaQNN trained 2700 networks in 100 GPU days, i.e., about 1 network per GPU hour. It trained 20 epochs for the studied case of CIFAR, so 1-3 minutes per epoch on the CPU can be implemented with zero overhead while the network is training on the GPU. Therefore, the following sentence seems sensational without substance: "Therefore, on a full meta-modeling experiment involving thousands of neural network configurations, our method could be faster by several orders of magnitude as compared to LCE based on current implementations."

The experiment on fast Hyperband is very nice at first glance, but the longer I think about it the more questions I have. During the rebuttal I would ask the authors to extend f-Hyperband all the way to the right in Figure 6 (left) and particularly in Figure 6 (right). Especially in Figure 6 (right), the original Hyperband algorithm ends up higher than f-Hyperband. The question this leaves open is whether f-Hyperband would reach the same performance when continued or not. 
I would also request the paper not to casually mention the 7x speedup that can be found in the appendix, without quantifying this. This is only possible for a large number of 40 Hyperband iterations, and in the interesting cases of the first few iterations speedups are very small. Also, do the simulated speedup results in the appendix account for potentially stopping a new best configuration, or do they simply count how much computational time is saved, without looking at performance? The latter would of course be extremely misleading and should be fixed. I am looking forward to a clarification in the rebuttal period. 
For relating properly to the literatue, the experiment for speeding up Hyperband should also mention previous methods for speeding up Hyperband by a model (I only know one by the authors' reference Klein et al (2017)).

Overall, this paper appears very interesting. The proposed technique has some limitations, but in some settings it seems very useful. I am looking forward to the reply to my questions above; my final score will depend on these.

Typos / Details: 
- The range of the coefficient of determination is from 0 to 1. Table 1 probably reports 100 * R^2? Please fix the description.
- I did not see Table 1 referenced in the text.
- Page 3: "more computationally and" -> "more computationally efficient and"
- Page 3: "for performing final" -> "for predicting final"


Points in favor of the paper:
- Simple method
- Good prediction results
- Useful possible applications identified

Points against the paper:
- Methodological advances are limited / unmotivated choice of model
- Limited applicability to settings where >> 100 configurations can be run fully
- Possibly inflated results reported for Hyperband experiment
review_title: Strong performance predictions possible for budgets >> 100 configurations

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper explores the use of simple models for predicting the final
validation performance of a neural network, from intermediate values
during training.  It uses support vector regression to show that a
relatively small number of samples of hyperparameters, architectures,
and validation time series can lead to reasonable predictions of
eventual performance.  The paper performs a modest evaluation of such
simple models and shows surprisingly good r-squared values.  The
resulting simple prediction framework is then used for early stopping,
in particular within the Hyperband hyperparameter search algorithm.

There's a lot that I like about this paper, in particular the ablation
study to examine which pieces matter, and the evaluation of a couple
of simple models.  Ultimately, however, I felt like the paper was
somewhat unsatisfying as it left open a large number of obvious
questions and comparisons:

- The use of the time series is the main novelty.  In the AP, HP and
  AP+HP cases of Table 2, it is essentially the same predictive setup
  of SMAC, BO, and other approaches that are trying to model the map
  from these choices to out-of-sample performance.  Doesn't the good
  performance without TS on, e.g., ResNets in Table 2 imply that the
  Deep ResNets subfigure in Figure 3 should start out at 80+?

- In light of the time series aspect being the main contribution, a
  really obvious question is: what does it learn about the time
  series?  The linear models do very well, which means it should be
  possible to look at the magnitude of the weights.  Are there any
  surprising long-range dependencies?  The fact that LastSeenValue
  doesn't do as well as a linear model on TS alone would seem to
  indicate that there are higher order autoregressive coefficients.
  That's surprising and the kind of thing that a scientific
  investigation here should try to uncover; it's a shame to just put
  up a table of numbers and not offer any analysis of why this works.

- In Table 1 the linear SVM uniformly outperforms the RBF SVM, so why
  use the RBF version?

- Given that the paper seeks to use uncertainty in estimates and the
  entire regression setup could be trivially made Bayesian with no
  significant computational cost over a kernelized SVM or OLS,
  especially if you're doing LOOCV to estimate uncertainty in the
  frequentist models.  Why not include Bayesian linear regression and
  Gaussian process regression as baselines?

- Since the model gets information from the AP and HP before doing any
  iterations, why not go on and use that to help select candidates?

- I don't understand how speedup is being computed in Figure 4.

- I'd like a more explicit accounting of whether 0.00006 seconds vs
  0.024 seconds is something we should care about in this kind of
  work, when the steps can take minutes or hours on a GPU.

- How useful is r-squared as a measure of performance in this setting?
  My experience has been that most of the search space has very poor
  performance and the objective is to find the small regions that work
  well.

Minor things:

- y' (prime) gets overloaded in Section 3.1 as a derivative and then
  in Section 4 as a partial learning curve.

- "... is more computationally and ..."

- "... our results for performing final ..."

review_title: Some Interesting Results, but Unsatisfying Analysis

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes the use of an ensemble of regression SVM models to predict the performance curve of deep neural networks. This can be used to determine which model should be trained (further). The authors compare their method, named Sequential Regression Models (SRM) in the paper, to previously proposed methods such as BNN, LCE and LastSeenValue and claim that their method has higher accuracy and less time complexity than the others. They also use SRM in combination with a neural network meta-modeling method and a hyperparameter optimization one and show that it can decrease the running time in these approaches to find the optimized parameters.

Pros: The paper is proposing a simple yet effective method to predict accuracy. Using SVM for regression in order to do accuracy curve prediction was for me an obvious approach, I was surprised to see that no one has attempted this before. Using features sur as time-series (TS), Architecture Parameters (AP) and Hyperparameters (HP) is appropriate, and the study of the effect of these features on the performance has some value. Joining SRM with MetaQNN is interesting as the method is a computation hog that can benefit from such refinement. The overall structure of the paper is appropriate. The literature review seems to cover and categorize well the field.

Cons: I found the paper difficult to read. In particular, the SRM method, which is the core of the paper, is not described properly, I am not able to make sense of the description provided in Sec. 3.1. The paper is not talking about the weaknesses of the method at all. The practicability of the method can be controversial, the number of attempts require to build the (meta-)training set of runs can be huge and lead to something that would be much more costful that letting the runs going on for more iterations. 

Questions:
1. The approach of sequential regression SVM is not explained properly. Nothing was given about the combination weights of the method. How is the ensemble of (1-T) training models trained to predict the f(T)?
2.  SRM needs to gather training samples which are 100 accuracy curves for T-1 epochs. This is the big challenge of SRM because training different variations of a deep neural networks to T-1 epochs can be a very time consuming process. Therefore, SRM has huge preparing training dataset time complexity that is not mentioned in the paper. The other methods use only the first epochs of considered deep neural network to guess about its curve shape for epoch T. These methods are time consuming in prediction time. The authors compare only the prediction time of SRM with them which is really fast. By the way still, SRM is interesting method if it can be trained once and then be used for different datasets without retraining. Authors should show these results for SRM. 
3. Discussing about the robustness of SRM for different depth is interesting and I suggest to prepare more results to show the robustness of SRM to violation of different hyperparameters. 
4. There is no report of results on huge datasets like big Imagenet which takes a lot of time for deep training and we need automatic advance stopping algorithms to tune the hyper parameters of our model on it.
5. In Table 2 and Figure 3 the results are reported with percentage of using the learning curve. To be more informative they should be reported by number of epochs, in addition or not to percentage.
6. In section 4, the authors talk about estimating the model uncertainty in the stopping point and propose a way to estimate it. But we cannot find any experimental results that is related to the effectiveness of proposed method and considered assumptions.

There are also some  typos. In section 3.3 part Ablation Study on Features Sets, line 5, the sentence should be “Ap are more important than HP”.

review_title: Good simple idea, paper hard to read, lack explanations

Meta-review: The paper proposes to use simple regression models for predicting the accuracy of a neural network based on its initial training curve, architecture, and hyper-parameters; this can be used for speeding up architecture search. While this is an interesting direction and the presented experiments look quite encouraging, the paper would benefit from more evaluation, as suggested by reviewers, especially within state-of-the-art architecture search frameworks and/or large datasets.
Decision: Invite to Workshop Track