Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper addresses the problem of predicting learning curves. The key difference from prior work is that (1) the authors learn a neural network that generalizes across hyperparameter settings and (2) the authors use a Bayesian neural network with SGHMC. 
The authors demonstrate that the proposed approach is effective on extrapolating partially observed curves as well as predicting unobserved learning curves on various architectures (FC, CNN, LR and VAE). This seems very promising for Bayesian optimization, I'd love to see an experiment that evaluates the relative advantage of this proposed method :)

Have you thought about ways to handle learning rate decays? Perhaps you could run the algorithm on a random subset of data and extrapolate from that?

I was thinking of other evaluation measures in addition to MSE and LL. In practice, we care about the most promising run. Would it make sense to evaluate how accurately each method identified the best run?

Minor comments:

Fonts are too small and almost illegible on my hard copy. Please increase the font size for legends and axes in the figures.

Fig 6: not all figures seem to have six lines. Are the lines overlapping in some cases?
review_title: Nice paper on predicting learning curves using Bayesian NNs

final_score: rating: null
review_content: How robust is your algorithm to choice of architecture and hyper-parameters of the Bayesian neural network? Some sensitivity analysis would be nice. 

Have you tried using other predictors (e.g. random forests) instead of Bayesian neural networks?


review_title: stability to hyperparameter settings?

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new Bayesian neural network architecture for predicting the values of learning curves during the training of machine learning models. This is an exploratory paper, in that the ultimate goal is to use this method in a Bayesian optimization system, but for now the experiments are limited to assessing the quality of the predictions. This builds on previous work in Domhan, 2015, however in this work the model incorporates information from all tested hyperparameter settings rather than just extrapolating from a single learning curve. This paper also explores two MCMC methods for inference: SGLD and SGHMC, but I couldn’t tell if either of these were tested in Domhan, 2015 as well.

The performance seems overall positive, particularly in the initial phase of each curve where there is very little information. In this case, as expected, sharing knowledge across curves helps. One regime which did not seem to be tested, but might be very informative, is when some curves in the training set have been mostly, or fully observed. This might be a case where sharing information really helps.

Something that concerns me about this approach is the timing. The authors stated that to train the network takes about 20-60 seconds. In the worst case, with 100 epochs, this results in a little over 1.5 hours spent training the Bayesian network. This is a non-trivial fraction of the several hours it takes to train the model being tuned.

The Bayesian network makes many separate predictions, as shown in Figure 2. It would be interesting to see how accurate some of these individual pieces are. For example, did you bound the asymptotic value of the learning curve, since you mostly predicted accuracy? If not, did the value tend to lie in [0,1]?

Below are some minor questions/comments.

Figure 1 axes should read “validation accuracy”
Figure 6 can you describe LastSeenValue (although it seems self-explanatory, it’s good to be explicit) in the bottom left figure, and why isn’t it used anywhere else as a baseline?
Figure 7 and Table 1 are you predicting just the final value of the curves? Or every value along each curve, conditioned on the previous values?
Why do you only use 5 basis functions? Does this sufficiently capture all of the flexibility of these learning curves? Would more basis functions help or hurt?

review_title: Very interesting model for predicting learning curves

final_score: rating: null
review_content: What is the cost of training the network and inference compared to the cost of running an additional iteration of training? Is the network prone to overfitting in the early stages when limited data is available? How did you tune the hyperparameters of the learning curve prediction network?
review_title: Computational cost and overfitting

final_score: rating: 7: Good paper, accept
review_content: This paper is about using Bayesian neural networks to model learning curves (that arise from training ML algorithms). The application is hyper-parameter optimization: if we can model the learning curve, we can terminate bad runs early and save time. The paper builds on existing work that used parametric learning curves. Here, the parameters of these learning curves form the last layer of a Bayesian neural network. This seems like a totally sensible idea. 

I think the main strength of this paper is that it addresses an actual need. Based on my personal experience, there is high demand for a working system to do early termination in hyperparameter optimization. What I'd like to know, which I wish I'd asked during pre-review questions, is whether the authors plan to release their code. Do you? I sincerely hope so, because I think the code would be a significant part of the paper's contribution, since the nature of the paper is more practical than conceptual.

The experiments in the paper seem thorough but the results are a bit underwhelming. I'm less interested in the part about whether the learning curves are actually modeled well, and more interested in the impact on hyperparameter optimization. I was hoping to see BIG speedups as a result of using this method, but I am left feeling unsure how big the speedup really is. Instead of "objective function vs. iterations" I would be more interested in the inverse plot: number of iterations needed to get to a fixed objective function value. Since what I'm really interested in is how much time I can save. Ideally there would also be some mention of real time as sometimes these hyperparameter optimization methods are themselves so slow that they end up being unusable. 

Finally, one figure that I feel is missing is a histogram of termination times over different runs. This would provide me with more intuition than all the other figures. Because it would tell me, what fraction of runs are being terminated early. And, how early? Right now I have no sense of this, except that at least *some* runs are clearly being terminated early, since this is neccessary for the proposed method to outperform other methods.

Overall, I think this paper merits acceptance because it is a solid effort on an interesting problem. The progress is fairly incremental but I can live with that.
review_title: A solid paper addressing a very common problem

final_score: rating: null
review_content: I'd be interested to know more about the basis functions in Figure 3 of the paper. For example, what is "vapor pressure" -- is it one of your basis functions? Are these basis functions described somewhere? And, unlike the other ones in Fig 3, vapor pressure can become negative -- does this have any implications (good or bad)?

Also, aside from vapor pressure, it seems that the other basis functions are monotonically decreasing. Is this true? And, if so, will they have trouble modeling real learning curves that are not monotonic?

On another note, I just want to make sure I understand things correctly. I asked myself, "why do we still need the basis functions at all instead of just using the BNN for everything?" I assume the answer is that we want to enforce these characteristic decaying shapes and the BNN might not do this (as in Figure 5, left-hand side, green curve). Am I understanding correctly?

Finally, I couldn't find any mention/discussion of the right-hand panels of Figure 6. Could you please clarify their significance?
review_title: basis functions in figure 3

Meta-review: The reviewers agreed that this is a good paper that proposes an interesting approach to modeling training curves. The approach is well motivated in terms of surrogate based (e.g. Bayesian) optimization. They are convinced that a great model of training curves could be used to extrapolate in the future, greatly expediting hyperparameter search methods through early stopping. None of the reviewers championed the paper, however. They all stated that the paper just did not go far enough in showing that the method is really useful in practice. While the authors seem to have added some interesting additional results to this effect, it was a little too late for the reviewers to take into account. 
 
 It seems like this paper would benefit from some added experiments as per the authors' suggestions. Incorporating this within a Bayesian optimization methodology is certainly non-trivial (e.g. may require rethinking the modeling and acquisition function) but could be very impactful. The overall pros and cons are as follows:
 
 Pros:
 - Proposes a neat model for extrapolating training curves
 - Experiments show that the model can extrapolate quite well
 - It addresses a strong limitation of current hyperparameter optimization techniques
 
 Cons
 - The reviewers were underwhelmed with the experimental analysis
 - The paper did not push the ideas far enough to demonstrate the effectiveness of the approach

Overall, the PCs have established that, despite some of its weaknesses, this paper deserved to appear at the conference.
Decision: Accept (Poster)