Title: Learning Curve Prediction with Bayesian Neural Networks
Authors: Aaron Klein
Stefan Falkner
Jost Tobias Springenberg
Frank Hutter
Venue: ICLR 2017
Keywords: Deep learning
Applications

Abstract: Different neural network architectures, hyperparameters and training protocols lead to different performances as a function of time.
Human experts routinely inspect the resulting learning curves to quickly terminate runs with poor hyperparameter settings and thereby considerably speed up manual hyperparameter optimization. Exploiting the same information in automatic Bayesian hyperparameter optimization requires a probabilistic model of learning curves across hyperparameter settings. Here, we study the use of Bayesian neural networks for this purpose and improve their performance by a specialized learning curve layer.

TL;DR: We present a general probabilistic method based on Bayesian neural networks to predit learning curves of iterative machine learning methods.

Review discussion:
I'd be interested to know more about the basis functions in Figure 3 of the paper.
For example, what is "vapor pressure" -- is it one of your basis functions? Are these basis functions described somewhere? And, unlike the other ones in Fig 3, vapor pressure can become negative -- does this have any implications (good or bad)?
Also, aside from vapor pressure, it seems that the other basis functions are monotonically decreasing. Is this true? And, if so, will they have trouble modeling real learning curves that are not monotonic?
On another note, I just want to make sure I understand things correctly. I asked myself, "why do we still need the basis functions at all instead of just using the BNN for everything?" I assume the answer is that we want to enforce these characteristic decaying shapes and the BNN might not do this (as in Figure 5, left-hand side, green curve). Am I understanding correctly?
Finally, I couldn't find any mention/discussion of the right-hand panels of Figure 6. Could you please clarify their significance?
Thanks for your suggestion, we now included a list of the formulas for the
basis functions in Appendix B. Yes, the vapor pressure basis function does
indeed not necessarily increase monotonically; we chose it because this
also allows us to model learning curves of configurations that start to
overfit (i.e. non-monotonic).
Our approach does not have trouble with non-monotonic learning curves. As it
can be seen in Figure 1 many curves in our dataset are actually
non-monotonic and our model is still able to capture their general trend.
About why we need the basis functions: Yes exactly. One can think of
this basis function as an additional prior that we use to improve the
predictions of our model
We will add a more detailed discussion about Figure 6 in the next
revision of the paper.
Thanks again for your constructive feedback.
What is the cost of training the network and inference compared to the cost of running an additional iteration of training? Is the network prone to overfitting in the early stages when limited data is available? How did you tune the hyperparameters of the learning curve prediction network?
Training our network is relatively cheap (~20 to 60 seconds on a CPU ) while training some of those benchmarks can take several hours.
Bayesian neural networks are naturally quite robust to overfitting and we did not encounter any overfitting so far even with very few points.
So far we did not tune the hyperparameter of our model as we found the default hyperparameter by Springenberg et al to work quite well.
But we will add a more detailed analysis (see answer above)
How robust is your algorithm to choice of architecture and hyper-parameters of the Bayesian neural network? Some sensitivity analysis would be nice.
Have you tried using other predictors (e.g. random forests) instead of Bayesian neural networks?
We found that our model as well as vanilla Bayesian neural networks in general to be quite insensitive to its hyperparameters such as the number of units or the step size.
However, we will apply Bayesian optimization to optimize them and add some plots generated by the Fanova toolkit (https://github.com/automl/fanova) to gain more insights.
We also considered other methods such as random forest however those are known to extrapolate poorly which is the main challenge for this task.
But we will compare to more baselines.
This paper proposes a new Bayesian neural network architecture for predicting the values of learning curves during the training of machine learning models. This is an exploratory paper, in that the ultimate goal is to use this method in a Bayesian optimization system, but for now the experiments are limited to assessing the quality of the predictions. This builds on previous work in Domhan, 2015, however in this work the model incorporates information from all tested hyperparameter settings rather than just extrapolating from a single learning curve. This paper also explores two MCMC methods for inference: SGLD and SGHMC, but I couldn’t tell if either of these were tested in Domhan, 2015 as well.
The performance seems overall positive, particularly in the initial phase of each curve where there is very little information. In this case, as expected, sharing knowledge across curves helps. One regime which did not seem to be tested, but might be very informative, is when some curves in the training set have been mostly, or fully observed. This might be a case where sharing information really helps.
Something that concerns me about this approach is the timing. The authors stated that to train the network takes about 20-60 seconds. In the worst case, with 100 epochs, this results in a little over 1.5 hours spent training the Bayesian network. This is a non-trivial fraction of the several hours it takes to train the model being tuned.
The Bayesian network makes many separate predictions, as shown in Figure 2. It would be interesting to see how accurate some of these individual pieces are.
For example, did you bound the asymptotic value of the learning curve, since you mostly predicted accuracy? If not, did the value tend to lie in [0,1]?
Below are some minor questions/comments.
Figure 1 axes should read “validation accuracy”
Figure 6 can you describe LastSeenValue (although it seems self-explanatory, it’s good to be explicit) in the bottom left figure, and why isn’t it used anywhere else as a baseline?
Figure 7 and Table 1 are you predicting just the final value of the curves? Or every value along each curve, conditioned on the previous values?
Why do you only use 5 basis functions? Does this sufficiently capture all of the flexibility of these learning curves? Would more basis functions help or hurt?
Many thanks for your review. We fixed the axis label, and added a more detailed description of LastSeenValue and why it is not used in all experiments.
Domhan et al. did not study SGLD or SGHMC, but used a general purpose, gradient-free MCMC sampler. To efficiently sample the weights of the network, a sampler that can leverage a noisy gradient is crucial. We’ve now emphasized that difference in the paper.
We agree with the reviewer that the overheads of fitting the BNN from scratch at every epoch would be quite high and could quickly become non-negligible. One option for avoiding this high overhead is to use approximate updates of the BNN as new data becomes available and only refit the model from scratch periodically. The option we chose in our exemplary hyperparameter optimization experiment is even simpler: we only refit the model after every internal iteration of Hyperband, yielding a method that has almost no overhead and already leads to large speedups over both Hyperband and Bayesian optimization.
Of course, this solution still leaves performance potential untapped, and in future work we aim to explore methods that refit & exploit the model in more fine-grained steps; we expect further improvements from doing so.
We bound the prediction of the asymptotic value by using a sigmoid function as output. Intuitively this should help since we mostly model validation accuracy which is limited to be in [0, 1].
However, even if other metrics are optimized (such as the lower bound of the likelihood in the Variational Auto-Encoder case) it is typically straightforward to scale them to lie in [0, 1].
For Figure 7 and Table 1, all models predicted unseen learning curves completely. The resulting MSE and ALL capture the model’s ability to describe whole learning curves from beginning to end. This part is complementary to the extrapolation of partially observed curves. At this point, we do not draw sample learning curves from our model which would require to condition the prediction on the performance in previous epochs. This would be an integral part when using our model in look-ahead methods, which could be an interesting direction for future research.
We tried different numbers of basis function and found that often just a small set of diverse basis functions is sufficient to achieve good quality. Just increasing the number of basis function would rather hurt as it also increases the number of parameters and thus makes the regression problem harder for the neural network.
This paper is about using Bayesian neural networks to model learning curves (that arise from training ML algorithms). The application is hyper-parameter optimization: if we can model the learning curve, we can terminate bad runs early and save time. The paper builds on existing work that used parametric learning curves. Here, the parameters of these learning curves form the last layer of a Bayesian neural network. This seems like a totally sensible idea.
I think the main strength of this paper is that it addresses an actual need. Based on my personal experience, there is high demand for a working system to do early termination in hyperparameter optimization. What I'd like to know, which I wish I'd asked during pre-review questions, is whether the authors plan to release their code. Do you? I sincerely hope so, because I think the code would be a significant part of the paper's contribution, since the nature of the paper is more practical than conceptual.
The experiments in the paper seem thorough but the results are a bit underwhelming. I'm less interested in the part about whether the learning curves are actually modeled well, and more interested in the impact on hyperparameter optimization. I was hoping to see BIG speedups as a result of using this method, but I am left feeling unsure how big the speedup really is.
Instead of "objective function vs. iterations" I would be more interested in the inverse plot: number of iterations needed to get to a fixed objective function value. Since what I'm really interested in is how much time I can save. Ideally there would also be some mention of real time as sometimes these hyperparameter optimization methods are themselves so slow that they end up being unusable.
Finally, one figure that I feel is missing is a histogram of termination times over different runs. This would provide me with more intuition than all the other figures. Because it would tell me, what fraction of runs are being terminated early. And, how early? Right now I have no sense of this, except that at least *some* runs are clearly being terminated early, since this is neccessary for the proposed method to outperform other methods.
Overall, I think this paper merits acceptance because it is a solid effort on an interesting problem. The progress is fairly incremental but I can live with that.
Thank you for your review and your insightful comments.
First of all, yes we will make all our code as well as our datasets publicly available soon (well in time before the conference).
We agree that next to showing that our model is able to accurately model learning curves, it is also important to show its impact on speeding up hyperparameter optimization. In the new Section 3.4 of the revised version of the paper, we report results on integrating it into the Hyperband method (in a nutshell, this runs very many hyperparameter settings for a short time, then drops half, continue running, drops half, etc; here, we use our model to decide which ones to drop).
Even though the function evaluations in our experiments are relatively cheap for deep learning experiments (a few hours at most), and even though the time for training our model is not negligible, these experiments show substantial speedups over Bayesian optimization for finding solutions very close to optimal. We expect that for more expensive tasks the speedups would be even larger. We also note that this is just a simple first example of how our model can be used to speed up hyperparameter optimization and we consider further ways to deploy our model as future research.
The paper addresses the problem of predicting learning curves. The key difference from prior work is that (1) the authors learn a neural network that generalizes across hyperparameter settings and (2) the authors use a Bayesian neural network with SGHMC.
The authors demonstrate that the proposed approach is effective on extrapolating partially observed curves as well as predicting unobserved learning curves on various architectures (FC, CNN, LR and VAE). This seems very promising for Bayesian optimization, I'd love to see an experiment that evaluates the relative advantage of this proposed method :)
Have you thought about ways to handle learning rate decays? Perhaps you could run the algorithm on a random subset of data and extrapolate from that?
I was thinking of other evaluation measures in addition to MSE and LL.
In practice, we care about the most promising run. Would it make sense to evaluate how accurately each method identified the best run?
Minor comments:
Fonts are too small and almost illegible on my hard copy. Please increase the font size for legends and axes in the figures.
Fig 6: not all figures seem to have six lines. Are the lines overlapping in some cases?
Thanks for your positive and constructive feedback. To give more intuition of the relative advantages of our proposed method we added a new experiment in the revised version of the paper. In this experiment, we incorporate our model in the recently developed hyperparameter optimization method Hyperband and show that it yields substantial speedups over Bayesian optimization.
One remark about the handling of learning rate decays: In fact our model is already able to handle learning rate decays, as it can be seen in the FCNet example, which uses an exponentially decaying learning rate schedule. In general our model should also be able to handle other decaying strategies such as step functions if it gets the hyperparameters of the strategy as an input (e.g. number of steps after which a decay happens).
About the number of lines in Figure 6: LastSeenValue doesn’t provide a predictive variance, and hence cannot be used to compute a likelihood.
Therefore, the right panels only have 5 curves.

Meta-review:
role: PC
data: comment: The reviewers agreed that this is a good paper that proposes an interesting approach to modeling training curves. The approach is well motivated in terms of surrogate based (e.g. Bayesian) optimization. They are convinced that a great model of training curves could be used to extrapolate in the future, greatly expediting hyperparameter search methods through early stopping. None of the reviewers championed the paper, however. They all stated that the paper just did not go far enough in showing that the method is really useful in practice. While the authors seem to have added some interesting additional results to this effect, it was a little too late for the reviewers to take into account. 
 
 It seems like this paper would benefit from some added experiments as per the authors' suggestions. Incorporating this within a Bayesian optimization methodology is certainly non-trivial (e.g. may require rethinking the modeling and acquisition function) but could be very impactful. The overall pros and cons are as follows:
 
 Pros:
 - Proposes a neat model for extrapolating training curves
 - Experiments show that the model can extrapolate quite well
 - It addresses a strong limitation of current hyperparameter optimization techniques
 
 Cons
 - The reviewers were underwhelmed with the experimental analysis
 - The paper did not push the ideas far enough to demonstrate the effectiveness of the approach

Overall, the PCs have established that, despite some of its weaknesses, this paper deserved to appear at the conference.

Decision: Accept (Poster)