Forecasting Deep Learning Dynamics with Applications to Hyperparameter Tuning
Authors: Piotr Kozakowski
Łukasz Kaiser
Afroz Mohiuddin
Venue:  (2020)

Abstract: Well-performing deep learning models have enormous impact, but getting them
to perform well is complicated, as the model architecture must be chosen and a
number of hyperparameters tuned. This requires experimentation, which is timeconsuming and costly. We propose to address the problem of hyperparameter
tuning by learning to forecast the training behaviour of deep learning architectures.
Concretely, we introduce a forecasting model that, given a hyperparameter schedule
(e.g., learning rate, weight decay) and a history of training observations (such as
loss and accuracy), predicts how the training will continue. Naturally, forecasting
is much faster and less expensive than running actual deep learning experiments.
The main question we study is whether the forecasting model is good enough to be
of use - can it indeed replace real experiments? We answer this affirmatively in two
ways. For one, we show that the forecasted curves are close to real ones. On the
practical side, we apply our forecaster to learn hyperparameter tuning policies. We
experiment on a version of ResNet on CIFAR10 and on Transformer in a language
modeling task. The policies learned using our forecaster match or exceed the ones
learned in real experiments and in one case even the default schedules discovered
by researchers. We study the learning rate schedules created using the forecaster
are find that they are not only effective, but also lead to interesting insights.

Under review as a conference paper at ICLR 2020
FORECASTING DEEP LEARNING DYNAMICS WITH APPLICATIONS TO HYPERPARAMETER TUNING
Anonymous authors Paper under double-bling review
ABSTRACT
Well-perforning deep learning models have enormous impact, but getting them to perform well complicated, as the model architecture must be chosen and a number of hyperparameters tuned. This requires experimentation, which is time- consuming and costly. We propose to address the problem of hyperparameter tuning by learning forecast the training behaviour of deep learning architectures. Concretely, we introduce a forecasting model that, given a hyperparameter schedule (e.g., learning rate, weight decay) and a history of training observations (such as loss and accuracy), predicts how the training will continue. Naturally, forecasting is much faster and less expensive than running actual deep learning experiments.
The main question we study is whether the forecasting model good enough to be of use can it indeed replace real experiments? We answer this affirmatively in two ways. For one, we show that the forecasted curves are close to real ones. On the practical side, apply our forecaster learn hyperparameter tuning policies. We experiment on version of ResNet on CIFAR10 and on Transformer in a language modeling task. The policies learned using our forecaster match or exceed the ones learned in real experiments and in one case even the default schedules discovered by researchers. We study the learning rate schedules created using the forecaster are find that they not only effective, but also lead to interesting insights.
INTRODUCTION
Machine learning researchers working with deep neural networks spend time looking at training curves and asking themselves the question: How would the results change if I modified some hyperparameters? They run many experiments and, with time, develop better understanding of how modifications in various hyperparameters affect the learning dynamics of our models.
In this work, we attack this problem of understanding deep learning dynamics with deep learning itself. To this end, we collect a data-set of training curves of deep learning models with diverse set of hyperparameter schedules. Next, we train an autoregressive deep model to predict the training curves conditioned on the schedule.
More concretely, our model observes the setting of small number of hyperparameters, such as learning rate, weight decay, or dropout rate. Then, it predicts small number of values that a researcher would usu- ally look at, such as training and validation loss and accuracy. We predict how these values will change a few hundred training steps later. Then, the model gets new settings for the next few hundred training steps, and predicts the next values. The model is autoregres- sive, so uses the history of its own predictions from previous time-steps predict the next ones.
1.4
We study how to do time-series prediction with au- toregressive models even in the presence of stochastic behaviour. We study different losses and introduce loss weight scheme that allows use Transformer
Figure 1: Real and predicted evaluation accu- racy curves, see text the left for details.

---

Under review as a conference paper at ICLR 2020
X
0.10
110
0.01 011
NUMBERS
5
2/
5
S
SYMBOLS
6
Figure 2: Discretization procedure where Ię is the discretized sequence. it represented in fixed- precision encoding with 2 base-8 digits per number, so each symbol s; corresponds 3 bits of precision. Weights of symbols are decayed with parameter 3 according to their significance, so the more significant digit has weight and the other has weight
models for stochastic time-series predictions. We show that the new method we introduce outperforms baselines and is able to faithfully approximate the learning dynamics deep learning models, as shown in the examples in Figure
Having a model of learning dynamics is interesting in its own right, as it allows to study the training process much faster, without running full experiments. But it also opens up the possibility to learn hyperparameter schedules by using reinforcement learning in the model. We experiment with this and validate forecaster: the policies learned using reinforcement learning and the forecaster work well when training real models. Running the forecaster is of course much faster than running real experiments, so learning policy in the forecaster thousands of times faster.
TIME-SERIES MODELING WITH TRANSFORMER
Transformer (Vaswani et al. prediction and transduction. Its variants achieve state-of-the-ar results in machine translation and language modeling (Shoeybi composed almost entirely of self-attention layers, with no recurrent connections. This allows the model consider the entire sequence when predicting the next token, while being very efficient to train using GPUs, due to the massive paralleism allowed by attention layers.
2017 is autoregressive neural architecture for discrete sequence
al. (2019), RRafford al. (2019). The architecture is notably
In this work, show that Transformer can also be used to predict time series concretely, training curves of deep learning models. One obvious way to apply Transformer to time-series prediction would be try to directly predict the next value in the sequence with L2 loss. We show that this is not good strategy when sequences can be stochastic and introduce a discretization technique that reduces the problem the standard discrete sequence prediction setting and that yields good results.
2.1 DISCRETIZATION
In order to model time series using Transformer, represent each number in the sequence in fixed-precision, base-k encoding. We concatenate the consecutive base-k digits corresponding to each number, starting from the most significant one. To predict multiple time series time, arrange the representations of single element of each time series in fixed order, and then concatenate the representations of each time step, as shown at Figure This way can model the joint probability distribution of multiple time series using single model. We train the Transformer decoder to predict consecutive symbols in this discrete sequence autoregressively. We use cross-entropy as the loss function, weighted with exponential decay according to the significance of each digit:
L(l,p) Σ ((....))))
where denotes the cross-entropy, 4; are the true digits, p: is the model prediction and indexes digits in the fixed precision, base-k encoding of number, starting from the most significant one.
2.2 RESULTS ON SYNTHETIC DATA
We present an experiment on synthetic data, demonstrating the ability of Transformer to model a distribution time series. The experiment is simple by design, devised that it is easy to measure
2

---

Under review as a conference paper at ICLR 2020
Halepas Fetindte
19 80
86
Figure 3: Synthetic curves generated by discrete Transformer decoder with weighted cross-entrops loss together with the histogram of estimated a values. In the generation process used uniform distribution and the histogram approximates it well.
the diversity of data sampled by the model, while retaining high-level similarity with the real-world data. Diversity in our case crucial because, as described in the next section, the learned model is used to provide experience for training reinforcement learning agent. We want this experience to be as diverse as in the real environment, so the agent can learn to be robust to various future scenarios.
We train Transformer dataset of synthetically generated curves caleulated using the formula:
32 = 1 +0(00,0²)
where is integer from the interval [1, N] and is scale of the noise, set 0.01. The dataset is designed to mimic training accuracy curves, starting from 0 and converging to in the limit. Rate of convergence is controlled by a, sampled uniformly from the interval [0.5, 5] for each curve.
To measure the diversity of the generated curves, we estimate the parameter a from curve predicted by the model, by averaging over pointwise estimates along the curve:
N-1
-1
N
1=0
then visualize the distribution in histogram. As shown Figure 3 Transformer is able to generate diverse and convincingly-looking curves.
As baseline, we consider modeling the curves without discretization, using the Transformer decoder without embedding and predicting sequence of real numbers, training the model with L2 loss. During inference, we add Gaussian noise with variance o² to the prediction at each step, to introduce stochasticity. As shown at Figure 4 the curves generated this way tend collapse to single shape.
Predicted serves
Heltegumn of estin sihles
Figure 4: Synthetic curves generated by continuous Transformer decoder with L2 loss together with the histogram of values that shows collapse.
HYPERPARAMETER TUNING USING REINFORCEMENT LEARNING
As is standard in reinforcement learning, frame the problem of controlling model hyperparameters during training as a cartially-obseraiie Markov decision process. Each transition the MDP corresponds steps of training followed by evaluation held-out set of 5% of the training
3

---

Under review as a conference paper at ICLR 2020
data. Observations are the current values of training and evaluation accuracy and loss. Note that the agent does not have access to full state of the environment, because that would encompass the current values of all parameters of the model, which is intractable an input for the agent. Rewards are differences between the values of an optimized metric between two timesteps, so that the cumulative reward is equal the final value of that metric. In all experiments optimize for validation accuracy.
Actions are relative changes in each controlled hyperparameter. Instead of predicting continuous actions using a parametric policy, we use a simple discretization scheme. Discretization of the action space allows the policy distribution to be multi-modal and has been empirically shown to improve training of reinforcement learning agents in terms of average performance, stability and robustness to hyperparameters (Tang & Agrawal 2019). Concretely, we predict the change independently for each hyperparameter, out discrete set of values (-50%, -20%, - 5%, O, +5%, +5%,, + 100%). The action space conveys the intuition that hyperparameter schedule should be approximately continuous in the log space, while allowing for both significant changes in the hyperparameters between timesteps, and for more fine-grained control. Opposite values of relative change cancel each other out, so that random walk in the action space has a median relative change of approximately for every hyperparameter.
3.1 MODEL-FREE APPROACH
As basic model-free approach to solving this problem, we use the Proximal Policy Optimization algorithm (Schulman et al. 2017). PPO is widely-used policy gradient method, notable for being stable and easy to implement, while significantly improving upon the sample-efficieng, of vanilla policy gradient. It allows to perform multiple gradient updates based on the same amount of collected experience by optimizing a surrogate objective, ensuring that the policy distribution does not drift too far from the original one.
We set the discount factor y to 1 to get unbiased estimate of the expected return. This does not cause the return to diverge, as the rewards are bounded and the trajectory length is constant. We calculate advantages with Generalized Advantage Estimation (Schulman et al. (0155) with À 0.95.
As policy network, we use the decoder part of Transformer, without the embedding, so it accepts continuous input. The network has two heads, one predicting the distribution other the value (an estimate of the future return). We add to the PPO objective the L. loss of the value head with weight 1 and an entropy bonus with weight 0.1.
actions, and the
3.2 MODEL-BASED APPROACH
As more sample-efficient, model-based approach, use Simulated Policy Learning (SimPLe. Kaiser et al. (2017)) The algorithm consists of three phases, repeated a loop, as shown in Figure 5| First, a of trajectories is collected, either using the current policy network (which is randomly initialized) or using an external data-set. Then a predictive model of the environment is trained on the collected data. After that, the policy is trained using PPO in the environment simulated by the model. The improved policy is used to collect new data, and the loop continues.
We use variant of SimPLe where we start the loop from training the model on set of pre-collected data, and then resume the standard SimPLe algorithm, collecting new data in each iteration.
SCTION
POLICY
To be able to use the Transformer decoder as an en- vironment model, we discretize the entire history of observations and actions and concatenate their rep- resentations into one long sequence 01 02a2...0n Observations are discretized as described in subsec- [ion 2.1 Actions are already discrete, so we just rewrite them using one discrete symbol per each con- trolled hyperparameter. We train the model to predict just the observation symbols, masking out the loss terms corresponding action symbols. During infer-
T
ENMRONMENT MODEL
Figure 5: SimPLe: π is the policy, e the en- vironment model, and I' the set of collected trajectories.
4

---

Under review as a conference paper at ICLR 2020
ence, we calculate the reward based on the difference of an appropriate metric in two consecutive predicted observations.
For the policy network, we use the same architecture as for the environment model, except for the action and value prediction heads. This lets us pre-initialize the policy parameters with those of the trained environment model, which significantly accelerates the learning process. We conjecture that the attention heads in the policy network needs to attend to similar positions in the sequence as the attention heads in the environment model, so a policy network with parameters initialized in such way needs to learn mostly the action and value prediction heads, which is much faster.
4 RELATED WORK
Time-Series Modeling with Deep Learning. Time series forecasting has been traditionally studied using statistical models like ETS (Hyndman et al., (008) and ARIMA (Box & Jenkins] 1994). There are a number of works that use feed forward neural networks (FFNNs) for time series forecasting, see a survey by Zhang et al. (9988). However, FFNNs break the input series into consecutive fixed size input windows, so the temporal order is ignored within each input window and every new input is considered in isolation. It is also not uncommon to use a hybrid approach of using FFNNs along with ARIMA or ETS as in Khashei & Bijari 1201T) and Faruk (2010).
Recurrent Neural Networks (RNNs) are a more natural fit for sequence prediction tasks and have gained popularity for natural language processing tasks. They started to be applied to time-series later and were found to be competitive for point forecasts in a univariate context (Hewamalage et al.) 2019)).
Recently Convolutional Neural Networks (CNNs) have also been used for time series forecasting, especially for capturing long-term dependencies, see yan den Oord et al. (2016) and Lai et al. (201 8).
Other approaches have used Deep Belief Networks and Stacked Denoising Autoencoders to predict temperature and traffic flow (Romeu et al. 2013 Lv et al. (013).. Outside of language and generating audio, Transformers (Vaswani et al.) 2017) have not been widely used for time-series prediction.
Reinforcement Learning. DQN (Mnih al. 2013) bootstrapped the field of Deep Reinforcement Learning by training a CNN on raw input pixels to predict the value of future rewards. Further work that built top of DQN includes Double-DQN by van Hasselt et al. (016). Dueling-DQN Wang ee al. (2016) etc. On the other hand, pure policy optimization techniques like TRPO (Schulman et al. 2015a) and PPO (Schulman et al. 2017) directly represent the policy and optimize the policy parameters. PPO is quite simple to implement and works well on a variety of tasks: Atari, MuJoCo etc. A downside of pure policy gradient algorithms using on-policy learning is the large number of environment interactions needed to achieve satisfactory performance (sample complexity).
Model-based Reinforcement Learning. All RL methods mentioned above suffer from the require- ment of a large number of interactions with the environment. This can be very costly, as in our case. The idea to use a model of the environment instead of the true one has been explored for a long time. For example, HHollnd et (2018) use a variant of Dyna Sutton (199T) to learn model of the environment and generate experience for policy training in the context of Atari games. Outside of games, model-basee reinforcement learning has been investigated at length for applications such as robotics Deisenroth et al. (2013). Though most of such works do not use image observations, several recent works have incorporated images into real-world robotic control (Finn al. 2016 Finn & Levine 2016 Babaeizadeh et al. 2017 Ebert et al. 2017: Piergiovanni et al.) 2018 Paxton et al. 2018 Rybkin et | [2018] Ebert et al. 2018) and simulated Watter et al. (2015): Hafner et al. (2018).
Hyperparameter Tuning. Black-box hyperparameter tuning is extremely popular in industry and
academia, examples include Google Vizier (Golovin et al., 2017). Hyperopt (Bergstra & Bengio 2012). Spearmint (Snoek et al. 2012). These approaches assume the existence of metric on validation set. Sequential Model-Basee Optimization (SMBO). (Hutter et al.] (01T) is a family of methods that builds a model of the validation metric with hyperparameters as input, this model is usually trained on the previous trials of the hyperparameters. A large category of SMBO algorithms is Bayesian Optimization that builds probabilistic model of the above function.A detailed survey about Bayesian Optimization can be found at (Sahriari et al. (2016).
As method of accelerating hyperparameter optimization, Baker et al. (2117) predict the training curve of a network being trained using a deterministic model, and use for early stopping. We improve upon this work by modeling the stochasticity of training curves and by allowing the hyperparameters
5

---

Under review as a conference paper at ICLR 2020
Transformer
Transformer
03-
0.1
SemPiLe
SimFiLe
PPO Tuman
03
PFD
0.0
Rumma
20000
30000
Transformer Treebank
Wide ResNet CIFAR-10
0.5
0.0
0.6
PPD
SimPLu
PPO human
SmiLe
0.00
human
7500 12500
2000
8200
10000
Figure 6: Test accuracy plots for all considered tasks. Each plot shows the minimum, mean and maximum accuracy aggregated over 4 runs of each experiment.
to change over time. Furthermore, we jointly model the training and validation, accuracy and loss curves, while the aforementioned work restricted modeling one curve.
Similar to our work, Domhan et al. (2015) and Klein et al. (0117) learn probabilistic models of training curves and use them to accelerate hyperparameter search. Both methods use handcrafted basis of nonlinear functions of shapes similar to the training curves being modelled. In contrast, our method does not make any assumptions about the shape of the modelled curves, enabling to capture wide range of shapes and not requiring the domain knowledge needed to choose such a basis.
Population Based Training of Neural Networks (Jaderberg al., 2017) another approach that jointly optimizes the parameters and hyperparameters of a network. It starts off with population of agents, that after every few training iterations compare their fitness and mutate themselves using traditional evolutionary techniques (copying, mutation etc) and resume training from the next iteration onward, yielding an adaptive hyperparameter schedule that is similar to our work.
Also similar to work, Xu et al. (2019) learn to tune hyperparameters during model training using reinforcement learning, but they only use a model-free approach and restrict themselves to controlling the learning rate.
Related to our work, Goldblum et al.] (2019) draw interesting conclusions about the interaction between learning and weight decay rate. They also introduce an alternative weight decay, which they find to work better in practice.
EXPERIMENTAL RESULTS
In the experiments, we compare the policy learned using SimPLe with the one learned using model- free PPO and with a manually-tuned baseline. We tune the learning rate, weight decay rate and various other architecture-spedee specific hyperparameters. In all experiments, the manually-tune baseline changes the learning rate over the course of training and keeps the other hyperparameters constant. This tuning approach, while being simple, is a common practice in the field, as it is infeasible to devise schedules for hyperparameters manually. The authors of state-of-the-art methods in the considered tasks typically do not report scheduling other parameters than the learning rate (Shocybi ctt al. 22019), Radford et al. (2019). Hu et al. (2017)).
For all experiments, we first gather data-set of 10K training runs of a given architecture given task, while varying the hyperparameters using PPO on 128 parallel environments. That means there 128 concurrent workers training the network in each epoch, after which the policy gets updated and another batch of 128 workers start training. We run parallel runs of this, each for 20 epochs, resulting in 128* 20 10240 training curves.

---

Under review as a conference paper at ICLR 2020
Architecture and task Transformer on LMIB Transformer on WMT EN -> DE
SimPLe
PPO
Human
35.9%
30.2%
35%
59.9%
49.5%
60
Transformer Penn Treebank
23.4%
19.2%
23.2%
Wide ResNet on CIFAR-10
91.6%
91.2%
90%
Table 1: Final test accuracies for all considered tasks, averaged over 4 runs for each experiment.
On this dataset, we train the forecasting model a 3-layer Transformer with the loss described in Section 2.1 This forecaster fits a holdout set curves quite well, see SeeuFe T] We run 10 epochs of SimPLe to train the policy, in each one gathering 128 additional training curves used to fine-tune the forecaster. In total, we use 10240 + 10x * 128 11520 training curves for each experiment.
We evaluate the learned policies by letting them tune hyperparameters for the corresponding tasks and report the accuracy on the test set Table Figure 6). We use temperature 0.3 to sample from policies during evaluation. The resulting policies are significantly better than the ones obtained using actual training curves, as shown in Figure 6) which validates our claim that the forecaster a useful model of deep learning dynamics.
5.1 TRANSFORMER ON LANGUAGE MODELING TASKS
In these experiments we tune the Transformer 3 language modelling datasets: LMIB, WMT English -> German translation and Penn Treebank. We frame the translation task as a language modeling problem- we concatenate the input and output string with special symbol in the middle: inpul # ouipul and only use the output string calculate the loss.
For all tasks we use 6-layer Transformer. In addition to learning rate and weight decay, we also tune 3 dropout parameters: the attention dropout, the dropout in the middle of feed-forwart layers and the dropout at the final residual connection a layer. Not only are there different dropouts, we tune them separately for the first layer, last layer, and all other layers in the middle (2-5). The manually-tuned baseline adjusts the learning rate according to an inverse-square root schedule with a linear warmup, commonly used in the field, and keeps the other hyperparameters constant.
As shown in Figure 6, the SimPLe policy consistently matches or exceeds the performance of the manually-tun -tuneddd bascline. In all tasks, the SimPLe-tuned model converges significantly faster. The difference gradually decreases during training, with the final performance either better (LM1B) or approximately the same (WMT, PTB) the baseline. The final accuracies are shown in detail in Table I] The SimPLe training process is also remarkably stable, yielding similar results across runs of the experiment. We attribute it to the pre-initialization of the policy parameters, which provides good starting point for policy learning.
kaaiing
drapout attartion layeri
dissout 'oudfarward layer2
drusout fral layer2
01337
03353
WHiGh
dropaut attanzin lapar E2.53
dispott feudforward laver (2.5)
dicoout Small layar $2.5
00082200
aressu embsadin
dmppout afterbon Invert
aocoutt wedtowaed tnyent
drnocur teal Inyert
2.3
66
D4
PPO Turao
00
Figure 7: Learned and manually-tuned hyperparameter schedules for Transformer LM1B.
7

---

Under review as a conference paper at ICLR 2020
weight, detatddecyayaaaee
learning rate
momentum
10
0.000000
08-
00000006
03
06-
0,000004
92
04
0.000022
01
02
00000000
60
00
Figure 8: Learned and manually- -tuned hyperparameter schedules for Wide ResNet CIFAR-10.
In contrast, the PPO training process is quite unstable. The best run of the experiment achieves similar performance as SimPLe on LMIB and WMT, but the mean is significantly worse. Curiously, the PPO-tuned models also converge faster than the manually-tunet ones.
Discussion. While the above results attest to the quality of the forecaster, can we draw any insights from the learned policies? We show example of learned hyperparameter schedules for LMIB in Figure 7 The SimPLe policy clearly employs some interesting strategies. For tuning the learning rate, follows well-established pattern of warmup and gradual decay. Dropouts up to the maximum of 0.9 at varying rates, effectively stopping the learning process for those layers. Interestingly, dropouts of attention layers are increased slower and more gradually than the feedforward layers. The weight decay rate is kept high for a while in the beginning of training and then is quickly decreased. This behavior is also followed by the PPO policy.
ramped
5.2 WIDE RESNET ON CIFAR-10
In this experiment we tune Wide ResNet (Zagoruyko & Komodakis 2016) on the CIFAR-10 image classification task. The tuned hyperparameters include the learning rate, weight decay rate and momentum mass. Manually-tume baseline adjusts the learning rate according step- wise decrease schedule, commonly used in the field, and keeps the other hyperparameters constant.
In Figure 6 we can see that both the SimPLe and PPO policies outperform the manually-tune tuned baseline by large margin. The stability of the PPO training is much higher than in the Transformer experiments, which might be attributed to the smaller action space.
Discussion. As shown in Figure 8 the SimPLe policy uses a significantly lower learning rate and higher momentum rate than both the human baseline and PPO policy. Weight decay rate controlled by both SimPL.e and PPO follows a similar pattern as in the LM1 task, which suggests that in those tasks weight decay is useful mostly in the initial stage of training.
6 CONCLUSIONS
Modeling the learning dynamics of deep models is a complex problem, but modification of the Transformer model with the discretization strategy we developed in Section [2.T]].. yiildds good results in many cases. The predicted curves not only have low 1.2 distance and look close to the real ones, as seen in Figure [ but can also be used in to create parameter schedules.
Learning parameter scheduling policies using the forecaster is much more efficient than when running real experiments. A single inference of the forecaster, at batch 128, takes less than minute even on a single GPU. Training the models on the other hand takes at least hour and one needs 128 GPUs for that. So it is at least 60 128 = 7680 times more efficient to operate in simulation. We show experimentally that the forecasting model is indeed good enough to be used in this way.
One question that we leave for future work how to effectively gather data for the forecaster model. Having lot of training curves from varied models would allow pre-training the Transformer forecaster before using task-specific data. Pre-training with large data-sets has worked very well, e.g., in the context of BERT and other NLP tasks, so consider it promising direction in this
To pre-train the forecaster we will need to scale-up the effort gathering data. Luckily, this can be a community effort as many people training models and could benefit from better hyperparameter
8

---

Under review as a conference paper at ICLR 2020
schedules. In preparation for this we have already released the code for our experiments open- source and we are gathering feedback as we extend its upplicabilityy.
Finally, aside from the applications hyperparameter tuming, forecaster can be applied to any time-series prediction task. We have not experimented with other tasks than the synthetic one, but results in Section 2.1 are very encouraging and let believe that Transformer-bann basee models can achieve good results in many time-series prediction tasks.
REFERENCES
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. ICLR, 2017.
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Practical neural network performance prediction for early stopping. CoRR, 1iss/705.1182.. 2017. URL /ttL:tttp://aaxx...orr////
1705. 10823
James Bergstra and Yoshua Bengio. Random search for pppparppprrmmeteertppiiiiin J. Mach. Learn. Res., 13:281-305, 2012. URL (ttt:t/:://dl acm. wwt.m/mwggg.c.aaaa..oo..cco.m//.
George Edward Pelham Box and Gwilym M. Jenkins. Time Series Analysis: Forecasting and Control.
Prentice Hall PTR. Upper Saddle River, NJ, USA, 3rd edition, 1994. ISBN 0130607746.
Mare Peter Deisenroth, Gerhard Neumann, and Jan Peters. survey on policy search for robotics. Foundations and Trends in Robotics, 2(1-2), 2013.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In I/CAI, 2015.
Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. CoRR, 0oss/170.,05668, 2017.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.000:: 2018
Durdu Ömer Faruk. A hybrid neural network and ARIMA model for water quality time series prediction. Eng. Appl. of AI, 23(4):586-594, 2010. doi: 10. wwtpp:.engappaa.2000.0099
URL
https: orggorgggg.o........ooo...o ///. http:/0...111..e.e..... http:///...ggppp...oo.. 2009. 09. 015
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. CoRR, abs/1610.00696, 2016.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation, ICRA 2016, Stockholm, Sweden, May 16-21, 2016, pp. 512-519, 2016. doi: 10. 109/ 1t///11.11110000 11 2016 6. 7487173
ICRA.2016.7487177 CRL/https://do..oom
Micah Goldblum, Jonas Geiping, Avi Schwarzschild, Michael Moeller, and Tom Goldstein. Truth backpropaganda? an empirical investigation of deep learning theory. CoRR, abs/1910.00339, 2019.
URL httthh/tp///axx/
org/ abs/19111 00359
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: service for black-box optimization. In Proceedings the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 17, 2017, pp. 1487-1495, ACM, 2017. ISBN 978- 978- 4-53--44--8-88--4 doi: 10.1145/3097983. 3098043. URL 1ww.://///...................../
3098044
Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. CoRR, abs/1811.0455) 1.04551, 2018
I Links the repositories omitted here to preserve anonymity.

---

Under review as a conference paper at ICLR 2020
Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Recurrent neural networks for time series forecasting: Current status and future directions, 2019.
G. Zacharias Holland, Erik Talvitie, and Michael Bowling. The effect of planning shape on dyna-
style planning in high-dimensional state spaces. CoRR, abs/1806.01825. 2018. URL http:
7 //arxiv. bwwxxi..rrg/bb//B8000
01825
Jie Hu, Li Shen, and Gang Sun. Squeeze-and networks. CoRR, abs/ fbss//79.00107,,, 2017. htgg/bbs/170) 1000000000000
URL UR.http: http::ttp///rrtt
Frank Hutter, Holger H. Hoos, and Kevin Leyton- Jeyton-Brown. Sequential model-based optimization for general algorithm configuration. Carlos A. Coello Coello (ed.), Learning and Intelligent Optimization 5th International Conference, LION 5, Rome, Italy, January 17-21, 201 I. Selected Papers, volume 6683 of Lecture Notes in Computer Science, pp. 507-523. Springer, 2011. ISBN doi: 10..0007/988-3--222255566
978-3-642-2556333 1007 978 642 25566- 3 40
3-642-25566-61 3112-255566--4440 URL
Robin Hyndman, Anne B. Koehler, Keith Ord, and Ralph D. Snyder. Forecasting with Exponential Smoothing The State Space Approuch: Robin Hyndman. Springer- Verlag Berlin Heidelberg, 2008. URL httpt::p://// http:/wwwwpp.gg..e.... com/ hptbbooo////35407777777
Max Jaderberg. Valentin Dalibard, Simon Osindero, Wojeiech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals. Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. CoRR, abs/1711 111111111996666 2017.
URL httt:://pxx/xii org/ abs/1711 000171111199811 09846
Eukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, and Konrad Czechowski Dumitru Erhan Chelsea Finn Piotr Kozakowski Sergey Levine Afroz Mohiuddin Ryan Sepassi George Tucker Henryk Michalewski Blazej Osinski, Roy H Campbell. Model-based reinforcemmnt learning for atari. CoRR, aww/aa/s.a... 900000000444.... 2017. htt.:httpp//axx..orrr//////
00374
Mehdi Khashei and Mehdi Bijari. novel hybridization of artificial neural networks and arima models for time series forecasting. Appl. Soft Comput., 1102):2664 March 2011. 1. ISSN 1568- 4946. doi:
doi. URL http:///xxoooo.r////////o// 1010/
201 0, 10. 015
Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httttts:/opeeeeeve///e
net/
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In The 4 International ACM SIGIR Conference on Research &#38; Development in Information Retrieval, SIGIR '18 pp. 95-104, New York, NY, USA, 2018. ACM. ISBN 978-1-4503 1550----5----5---2 http: acm. 1teg/.0..145/320997888 3210006
doi: 10. 10.11455320997833200000
URL
Yisheng Lv. Yanjie Duan, Wenwen Kang. Zhengxi Li, and Fei- Yue Wang. Traffic flow prediction with big data: A deep learning approach. /EEE Trans. Intelligent Transportation Systems, 16 doi: 10.1109/11TS22214444666 2014..3566.... URL /ttp://////o....00..0000100
(2):865 8733, 2015.
TITS. 2014 0.f.234556 2345663
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
httttp:://axxv..rrgg/
5602
Chris Paxton, Yotam Barnoy, Kapil D. Katyal, Raman Arora, and Gregory D. Hager. Visual robot task planning. CoRR, abs/1 804.804.0000 00062, 2018.
A. J. Piergiovanni, Alan Wu, and Michael S. Ryoo. Learning real-world robot policies by dreaming. CoRR, foss/1805.077 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
10

---

Under review as a conference paper at ICLR 2020
Pablo Romeu, Francisco Zamora-Mattn Martinez, Paloma Botella- and Juan Pardo. Time- series forecasting of indoor temperature using pre-trained deep neural networks. In Valeri Mladenov, Petia D. Koprinkova- Hrrittvv,, Günther Palm, Alessandro E. P. Villa, Bruno Ap- pollini, and Nikola Kasabov (eds.), Artificial Neural Networks and Machine Learning- ICANN 2013 23rd International Conference on Artificial Neural Networks, Sofia, Bulgaria, Septem- ber 10-13, 2013. Proceedings, volume 8131 1 of Lecture Notes in Computer Science, pp. 451- 458. Springer, 2013. ISBN 978-3-642-44777--7 doi: 10. https: /ww.
URL
Oleh Rybkin, Karl Pertsch, Andrew Jaegle, Konstantinos G. Derpanis, and Kostas Daniilidis. Unsuper- vised learning of sensorimotor affordances by stochastic future prediction. CoRR, 090.00655 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning. ICMI., pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. CoRR, abs/ 506.02438, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06320 2017.
Bobak Shahriari, Kevin Swersky, Ziyu Wang. Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):
148-175, 2016. doi: 10. 1 10//PP00..205552444411. URL
JPROC 2015 2494218
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Imi Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08055 2019. htRphtt:: / waw.arivvvv.///.////.aa org/ abs 1909.9. 08053
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Peter L. Bartlett, Fernando C. N. Pereira, Christo- pher J. C. Burges, Léon Bottou, and Kilian Q. Weinherger (eds.), Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Pro- cessing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States., pp. 2960-2968, 2012.
nips wttpppprrccppppprr/cpppppppp.eee hat.aa/wwo..ii.iiooooooooaaaeeea
1552-practical--avve
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull. 160-116, July 1991.
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for pollcy optimization. CoRR, abs/1901.10500 2019.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. CoRR, (bbs//09..0399, 2016 URL/ttt:://axxi> wwt.:bg///aao.ooooooo..ooo.. 16099, 03499
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Dale Schuurmans and Michael P. Wellman (eds.), Proceedings of the Thirtieth AAA/ Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pp. 2094- 2100. AAAI Press, 2016. ISBN 978-1-57735--660 URL http: httt:://www..o http://wwwaa....... awwi. orgg ocs/ Bitlexx.pp/AAAA/AAAAA http://AAAAAAAAAaapaap//e///////
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5998-6008, 6008, 2017. URL http:://appe: n ps. cc watsscc/ppaprr/// / 81 attentionn a
you-

---

Under review as a conference paper at ICLR 2020
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference Machine Learning, ICMI, 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp.
AaoriaaFlorna Balcan and
JMr.R.rrg 2016. URL http::/prrccedd ngs
mlr. wwwpseeesss.... /48 wwtps///v//wwaggl151 html
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin A. Riedmiller. Embed to control: locally linear latent dynamics model for control from raw im- ages. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 2746-2754, 2015. URL http: http:://ppppr...iiss..cooo wwwpssssspppccccppppp.ppppepp
Zhen Xu, Andrew M. Dai, Jonas Kemp, and Luke Metz. Learning an adaptive learning rate schedule. CoRR, 099//999.0972,, 2019. URLhttp://arxiy> http:///awxxv....aa///////a///// 09712
org/
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016, 2016.
Guoqiang Zhang, B. Eddy Patuwo, and Michael Y. Hu. Forecasting with artificial neural networks::
The state the art. International Journal of Forecasting. 14(1):35 62, 1998. ISSN 0169-2070. doi: http:://ooooooooooo0oo0000000000 httt:p::: www. http://wwwwwwwwwwcccec//c/
com/ http://caaaaa/arrr///////////172
12


Reviews:
[{"review_id": "rye5LcLAYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed to train a network with training curves and corresponding parameters, and use policy search to find optimal parameter to replace hundreds or thousands of training in real case scenario, and it is clearly much faster using the trained network to infer parameters, instead of tuning the network manually.\n\nThe first point would be: what's the meaning of synthetically generating training curves other than proving that transformer achieves good performance in modeling discrete distribution? Most practical problems would not have the same distribution as the previously gathered public dataset, thus the data is not representative, and synthetic training curves just does not make sence.\n\nThe cited paper 'Learning an adaptive learning rate schedule' does not appear online.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyeR04XiuS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work focuses on learning a good policy for hyperparameters schedulers, for example learning rate or weight decay, using reinforcement learning. The main contributions include 1) a discretization on the learning curves such that transformer can be applied to predict the them; 2) an empirical evaluations using the predicted learning curves to train the policy. \n\nThe main novelties are two folds. On the methodology side, using predicted learning curves instead of real ones can speed up training significantly. On the technical side, the author presented a discretization step to use transformer for learning curve predictions. The results are mixed, we see slightly advantage over human baseline on one task but worse in the other. Human baseline does not need any training! On the writing part, it would be nice to provide more context for both transformer, Proximal Policy Optimization and Simulated Policy Learning to make the paper more self-complete.\n\nI like the directions using surrogate to speed up HPO in general but I feel the learning curve prediction part can be improved. There are already some works, not using deep learning method, for example the following:\n\n* Baker, Bowen, et al. \"Accelerating neural architecture search using performance prediction.\" arXiv preprint arXiv:1705.10823 (2017).\n* Domhan, Tobias, Jost Tobias Springenberg, and Frank Hutter. \"Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves.\" Twenty-Fourth International Joint Conference on Artificial Intelligence. 2015.\n\nWhy these methods are not considered in the beginning? In my opinion, transformer is good for modeling long term dependency and concurrent predictions which is not necessarily the case for learning curves. How does the transformer based method comparing to others? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hyxs5tB0FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the possibility of learning a model to predict the training behaviour of deep learning architectures from hyperparameter information and a history of training observations. The model can then be used by researchers or a reinforcement learning agent to make better hyperparameter choices. The paper first adapts the Transformer model to be suitable to this prediction task by introducing a discretization scheme that prevents the transformer decoder's predictions from collapsing to a single curve. Next, the problem is formalized as a partially-observable MDP with a discrete action set, and PPO and SimPLe are introduced. The proposed model-based method is compared against a human and a model-free baseline training a Wide ResNet on CIFAR-10. The model-based method achieves better validation error than the other baselines that use actual data. Next, the method is compared against a human and a model-free baseline training Transformer models on the Penn Treebank dataset. While the human achieves the best performance at the end of the run, the proposed method appears to learn more quickly than the others and finishes with performance comparable to the model-free baseline.\n\nCurrently I lean towards accepting this paper for publication, despite a few issues. It asks an interesting question: can we learn a model of the training dynamics to avoid actually having to do the training? This could potentially prevent a lot of unnecessary computation and also lead to better-performing models. It then shows some experimental evidence suggesting that this is possible.\n\nMost importantly, I would like to see a measure of variance/uncertainty like confidence intervals included in the results; otherwise it's impossible to assess whether the results are likely to be significant or not. Other questions:\n1. In the PTB experiment, it looks like the human only adapts the learning rate and leaves the rest of the hyperparameters alone. Why was this policy used as the baseline? It seems extremely basic and unlikely to truly lead to optimal performance.\n2. Why were more baselines from the related work not included? I understand the experiments are a proof of concept, but it would be nice to get a feeling for what some of the other methods do.\n3. How do PPO and SimPLe handle partial observability? Is it principled to apply them to partially-observable environments?\n4. Why not use continuous actions with a parameterized policy (e.g. Gaussian)?\n5. Is it reasonable to assume that the learning dynamics of all deep learning architectures are similar enough that a model trained on one set of deep learning architectures and problems will generalize to new architectures and problems?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper trains a transformer to extrapolate learning curves, and uses this in a model-based RL framework to automatically tune hyperparameters. This might be a good approach, but it's hard to know because the experiments don't include direct comparisons against existing hyperparameter optimization/adaptation techniques (either the ones based on extrapolating training curves, or standard ones like BayesOpt or PBT). The presentation is also fairly informal, and it's not clear if a reader would be able to reproduce the results. Overall, I think there's significant cleanup and additional experiments needed before publication in ICLR.

Decision: Reject