Learning with Random Learning Rates.

Abstract: Hyperparameter tuning is a bothersome step in the training of deep learning mod- els. One of the most sensitive hyperparameters is the learning rate of the gradient descent. We present the All Learning Rates At Once (Alrao) optimization method for neural networks: each unit or feature in the network gets its own learning rate sampled from a random distribution spanning several orders of magnitude. This comes at practically no computational cost. Perhaps surprisingly, stochastic gra- dient descent (SGD) with Alrao performs close to SGD with an optimally tuned learning rate, for various architectures and problems. Alrao could save time when testing deep learning models: a range of models could be quickly assessed with Alrao, and the most promising models could then be trained more extensively. This text comes with a PyTorch implementation of the method, which can be plugged on an existing PyTorch model.

Reviews and author responses:

POST REBUTTAL: I think the paper is decent, there are some significant downsides to the method but it could constitute a first step towards a more mature learning-rate-free method. However, in its current state the paper is left with some gaping holes in its experiment section. The authors tried to add experiments on Imagenet, but these experiments apparently didn't finish before the end of the rebuttal period. For that reason, the paper probably should not be accepted for publication (even if the authors manage to finish running these experiments, we would not have a chance to review these results). 

------
Original review
------

In this paper, the authors present a method for training deep networks with randomly sampled feature-wise learning rates, removing the need for fixed learning rates and their tuning. The method is shown to perform comparatively to SGD with a learning rate roughly optimized with regards to validation performance. The method applies to the most popular types of deep learning architectures, which includes fully connected layers, convolutional layers and recurrent cells. 

Quality: The paper is of a decent quality in general, I noticed no glaring omissions while reading the paper. However, I do worry that the method provides little gain for a lot of work. It is becoming more and more easy to tune the learning rate of deep learning models with strategies such as early stopping, and this method comes at a high cost for models with a big final layer. 

Clarity: The paper is well written, but the reader is often (too often?) sent to the Appendix, which is itself ordered in a strange way (e.g., the first reference to the Appendix in the paper refers to Appendix F?). If some sections of the Appendix are not needed, I would remove them. 

Originality: The work is original in the approach, i.e. randomization as a way to get rid of learning rates is a novel method. However, there was one work presented last year at NIPS which concerns itself with the same problem, which is getting rid of learning rates:

“Training Deep Networks without Learning Rates Through Coin Betting” by Francesco Orabona and Tatiana Tommasi, NIPS, 2017.

They don’t compare on the same methods and the same datasets, but I think the authors should be aware of this work and perhaps compare themselves with it. The work takes a very different approach to solve the problem so I don’t think it’s an issue for this paper. 

Significance: I think the work is important, in that it adds another tool to solve the learning rate problem. I would not say it is likely to have a very high impact, because it involves a lot of work, for little benefit. Furthermore, the cost of reproducing multiple times the last layer of the network will be prohibitive in many cases for NLP. 

The method feels ad-hoc in many respects, and there are no guarantees that it would work any better than Adam does on pathological cases. Perhaps some mathematical analysis on simpler problems would help make the contributions stronger. 

The authors state that the learning rate range has little impact on performance, yet it still has enough impact to justify tuning it for different models and datasets (on CIFAR it is 10^-5 to 10^1, on Pennbank it is 10^-3 to 10^2). I would tend to agree that the alrao method is more robust to the choice of learning rate than plain SGD, however the fact of the matter is that there are still parameters to tune. 

Figure 5. also seems to suggest that the range is important, although the models were not trained until the end, so it is not clear.

Some additional comments:

Nitpicking: In Section 2, most sub-sections (or paragraphs titles?) have the name of the method in them. That’s redundant. Instead of “Alrao principle”, “Alrao update”, etc., just write “Principle.”, “Update.”.

Is there a justification for using the same learning rate for all weights in an LSTM unit? 

I believe there is a mistake in Equation 2. The denominator should be log(\eta_{max}) - log(\eta_{min})

[second paragraph on page 4.] Once again nitpicking for the sake of clarity: “For each classifier Cθ cl j, we set a learning rate log eta_j = …” this reads as if the learning rate would be set to log eta_j, but you probably mean you will set the learning rate to eta_j = exp(...).

Figure 5b in the appendix does not specify which curve has which learning rate interval. 


The authors have put in an effort to answer the earlier questions, and acknowledged the aspects which cannot be directly addressed at this point. The draft/appendix has been updated to include new material and technical results. While the work and ideas are still in somewhat early stages, it will be good to see the ideas (of using random learning rates) get discussed more widely -- so I am updating my score to be above the threshold. 


This work proposes an optimization method called All Learning Rate
At Once (Alrao) for hyper-parameter tuning in neural networks.
Instead of using a fixed learning rate, Alrao assigns the learning
rate for each neuron by randomly sampling from a log-uniform
distribution while training neural networks. The neurons with
proper learning rate will be well trained, which makes the whole
network eventually converge. The proposed method achieves
performance close to the SGD with well-tuned learning rate on the
experiments of image classification and text prediction.


#Pros:

-- The use of randomly sampled learning rate for deep learning
models is novel and easy to implement. It can become a good
approximation of using SGD with the optimal learning rate.

-- The paper is well-written and easy to follow. The proposed
method is illustrated in a clear way.

-- The experiments are solid, and the performance on three
different architectures are shown for comparison. According to the
experiments, the proposed method is not sensitive to the
hyper-parameter \eta_{min} and \eta_{max}.

#Cons:

-- The authors have not given any theoretical convergence analysis
on the proposed method.

-- Out of all four experiments, the proposed method only
outperforms Adam once, which does not look like strong support.

-- Alrao achieves good performance with SGD, but not with Adam.
Also, there are no experimental results on Alrao with other
optimization methods.

#Detailed comments:

(1) I understand that Alrao will be more efficient compared to
applying SGD with different learning rate, but will it be more
efficient compared to Adam? No clear clarification or experimental
results have been shown in the paper.

(2) The units with proper learning rate could learn well and
construct good subnetworks. I am wondering if the units with "bad"
(too small or too large) learning rate might give a bad influence
on the convergence or performance of the whole network.

(3) The experimental setting is not clear, such as, how the input
normalized, how data augmentation is used in the training phase,
and what are the depth, width and other settings for all three
architectures.

(4) The explanation on the influence of using random learning rate
in the final layer is not clear to me.

(5) Several small comments regarding writing:
    (a) Is the final classifier layer denoted as $C_{\theta^c}$ or  $C_{\theta^{cl}}$ in the third paragraph of "Definitions and notations"?
    (b) In algorithm 1, what is the stop criteria for the do while? The "Convergence ?" in the while condition is confusing.
    (c) Is the learning curve in Figure 2 from one run or is it the average of all runs? Are the results consistent for each run? How about the learning curves for VGG19 and LSTM, do they have similar learning curves with the two architectures in Figure 2?
    (d) For Figure 3, it will be easier to compare the performance on the training and test set, if the color bars for the two figures share the same range.


Thank you for your comments and for insisting on more substantial
experimental validation.  This comment spurred us to perform a series of
experiments on ImageNet with several architectures, all confirming our
previous observations (see general comment above).

We hope these additional experiments may mitigate your opinion that "related
to the experimental evaluation of the method. I find the experimental
evidence for the effectiveness of Alrao insufficient."

Indeed the text offers no experimental use case related to architecture
search. As a result, following your comment we have decided to
de-emphasize architecture search as a motivation in the text. Instead, we
provide more experiments to insist on the robustness of the method and
its ability to provide results in a single run.

About Alrao versus setting per-weight random learning rates: our design
is based on a theoretical argument. Indeed, with per-weight learning
rates, *every* unit would have some incoming weights with large learning
rate. So, every unit would be at risk of divergence or quick saturation
(eg for sigmoid activations). With per-unit learning rates, units with
too-large learning rates may saturate and "die", but units with suitable
learning rates will be able to propagate information to subsequent
layers. Hopefully a sub-network made of units with suitable learning
rates will emerge.

About additional time in practice: the per-iteration or per-epoch
computational overhead of Alrao (a few seconds) is negligible compared to
the computation time of each epoch (approximately an hour) in our Imagenet experiments.
The number of epochs for convergence may be slightly worse than SGD
but varies depending on the setup, see Figs 2 and 5.


Thank you for your compliments on the importance, originality and clarity
of the paper, and for pointing out ways to improve the paper (eg
theoretical analysis).

About possible gains from the method: It is true that "It is becoming more
and more easy to tune the learning rate of deep learning models".
However, the early stopping method still assumes we can launch several
copies of the model: this is not the case in an online setting, for
instance.  Future applications may require on-the-spot fine-tuning or
transfer, for instance to adapt to a specific end-user's characteristics,
without off-line retraining.

The added ImageNet experiments (see general comment above) provide more
arguments for the benefits of the method, notably its robustness compared
to Adam.

Thank you for pointing out Orabona-Tommasi 2017. This is a very
intriguing paper, but it seems to us that doing away with gradients
altogether is a bold step which requires extensive testing,
while Alrao stays closer to the well-tested SGD principle.
Orabona-Tommasi's method has still to be tested for standard
architectures (VGG, GoogleNet, etc.). We have added this reference in the
related work.

About Alrao needing "a lot of work": We have provided a generic Alrao
implementation, so the additional work to incorporate Alrao into an
existing model is just a call to that library (for standard
architectures).

About the cost of the last layer for NLP: this is true. However, this
problem is already present in many NLP applications even without Alrao,
and over the years a number of strategies have been developed to reduce
substantially the last layer's parameter burden for large vocabulary
sizes (see several references in Jozefowicz 2016).

About a mathematical analysis on a simple problem: see the new Theorem
1 in Appendix B. Namely, for logistic regression (or any convex
function with fixed pre-classifier), with eta_min small enough, then
Alrao eventually reaches small loss. Going beyond that simplified
situation would already require a general analysis of the precise
SGD dynamics of neural networks with a single learning rate.

About the impact of the learning rate range, and using [10^-5:10] for
CIFAR versus [10^-3:10^2] for PennTreeBank: Fig 3 clearly shows that
using [10^-3;10^2] for CIFAR provides very similar results to [10^-5:10].
So we could have used the same interval. Our general prescription is to
use for Alrao the range of learning rates that would have classically
been tested via grid search. Generally, recurrent networks tend to
require larger learning rates than convolutional networks: we do not
forbid the use of such expert knowledge in Alrao, and this is why we
initially tested [10^-3:10^2] for LSTMs.  These intervals are just the
first that came to mind, and we did not tweak them to get better results.
(Partly, the difference is just because these experiments were run by two
different coauthors).

Why use the same learning rate for all parts of an LSTM unit: The
intuition is similar to fixing the learning rate per unit rather than per
weight. We would rather have a mixture of fully-functioning units and
non-functioning units, than the presence of too-large weights inside
every unit, which may screw up all units.

Other minor comments: thank you for pointing out these mistakes, they have
been fixed.

Thank you for your balanced review with pros and cons. Let us answer each
con and detailed comment in turn.

#Cons:

No theoretical argument for convergence: The newly included theorem in Appendix B
states that for a simple cases such as logistic regression, with eta_min
small enough, then Alrao eventually reaches the optimal loss.  (Going
beyond that simplified situation would already require a general analysis
of the precise dynamics of neural networks with a single SGD learning rate,
which is an open problem.)

Beating Adam only once: We have introduced experiments on ImageNet with
three models. They show that Adam's behavior is less consistent than
Alrao (including a model where Adam does not even start to learn). Thus,
the Alrao/Adam comparison on CIFAR is not an isolated occurrence.

Alrao working only with SGD: We suspect a bad interaction between Alrao's
learning rate mechanism, and the adaptive learning rates used in
optimizers fancier than SGD. We acknowledge this limitation; we have
edited the text to introduce Alrao on SGD by default, not as a generic
idea (though the tests with Alrao+Adam have been kept).

#Detailed comments:

(1) Efficiency compared to Adam: Please see the general comment above and
the new ImageNet experiments.

(2) Influence of neurons with "bad" learning rates, too small or too
large. 1/ Too small learning rates: they will result in neurons not
changing much from their initialization.  Such neurons effectively behave
like fixed random features. Our results in Appendix H show that including random
features does not hurt and indeed can improve training. 2/ Too large
learning rates: indeed we expect Alrao to work only if some mechanism
prevents bad neurons from having an ubbounded influence on subsequent
layers. For instance, with sigmoid activation functions, the worse that
can happen is that a neuron gets stuck to activities 0 or 1 that are
decorrelated from the desired output signal. In that case, the subsequent
layers have to learn to just ignore these 0/1 activities, but the
network will not diverge. With ReLU and BatchNorm, activations keep a
bounded variance thanks to the BatchNorm, and the worse that can happen
is that activities become N(0,1) variables decorrelated from the output.
The effect would be comparable to noisy neurons, which can be ignored by
subsequent layers.

(3) Details of architectures, experimental setup, data preprocessing: now
added in Appendix D.

(4) Why a different mechanism on the last layer. On the last layer, each
neuron corresponds directly to a class/label. If assigning one rate per class,
an unlucky class could get a very small learning rate and never learn anything
at all, or a large learning rate and immediately diverge. (For inner layers,
this is not a problem as a layer contains many neurons and the subsequent layers
can learn to listen to those neurons which have a good learning rate.)
To give another example, imagine that we are working on a regression problem,
with a unique output neuron: then, choosing a fixed learning rate at random for
this unique output neuron is clearly a bad idea. Our current method keeps
several copies of the output, with several learning rates.

(5) (a) and (b): fixed.

(5) (c) Fig 2a plots a single run, Fig 2b plots an average of 3 runs. The
results are very consistent between runs, as shown by the standard deviations in Table 1.
The curves for VGG19 do look similar; we can add them in an appendix if you deem it necessary.

(5) (d) Color code in Fig 3: After deliberation we have decided to keep a
per-figure color code, which allows for better comparison of relative
performance inside each figure. (The scales for train error and for test
error can be quite different.)


We would like to thank the reviewers for their insightful questions and remarks, which spurred us towards 
including more substantial experimental support.

Additionally, a major concern of the reviewers seems to be that Alrao
would not be that useful in practice.  Let us point out that in all our
experiments with Alrao-SGD, not a single run failed to learn.  For us,
Alrao's tuning-free robustness and its ability to provide results in a
single run are major arguments in its favor.

The new version uploaded today adds to the text:
- a series of experiments on ImageNet
- a theoretical analysis of Alrao in a simple case (convex function, fixed preclassifier) 
- answers to the various remarks (detailed below)

The new Imagenet experiments involve three standard models (AlexNet,
ResNet50, DenseNet121). They largely confirm the robustness of Alrao,
with results comparable to best-adjusted SGD (and better in one
instance). [The table contains "?" and best bounds so far where some
experiments are still running. They will be completed in a few days.]

On the same models, default Adam has mitigated results: on one model it
does not learn anything at all, and on the other two it first reaches
very good performance then diverges shortly thereafter.

We answer below to the more specific comments of each review.

In this paper the authors propose a method called “All learning rates at once” (Alrao) which aims to save the time needed to tune learning rate for DNN models testing. The method sets individual learning rate to each feature in each layer of a network using the values sampled from truncated log-uniform distribution. The only cost of the method is the creation of several branches of the classifier layer. Each of the branches is trained with a predefined learning rate value, and the final predictions are obtained by model averaging. In the presented experiments Alrao demonstrates performance comparable to SGD with optimal learning rate and more stable results compared to Adam. The authors indicate limitations of Alrao caused by the overhead in the final layer which complicates the application of the method for models with large classifier layer.

Overall, the paper is written clearly and organized well. However, Equation (2) needs to be corrected. The denominator in the normalizing constant of log-uniform distribution should be \log\eta_{max} - \log\eta_{min}.

My main concern is related to the experimental evaluation of the method. I find the experimental evidence for the effectiveness of Alrao insufficient. As the authors propose to employ the method to quickly evaluate models and select best models to further training it would be beneficial to have more results in order to ensure that the method is reliable in this setting. Other demonstrations which would show possibly that the method enhances performance of architecture search methods may emphasize significance of the proposed method. Also, more experiments comparing Alrao against sampling learning rates per weight are needed. Given the current results, it is still unclear whether the proposed method performs better. Finally, I recommend to include comments explaining how much more time is needed in practice to train model with Alrao compared to SGD training.

Meta-review: The paper proposes a new optimization approach for neural nets where, instead
of a fixed learning rate (often hard to tune), there is one learning rate per
unit, randomly sampled from a distribution. Reviewers think the idea is
novel, original and simple. Overall, reviewers found the experiments
unconvincing enough in practice. I found the paper really borderline,
and decided to side with the reviewers in rejecting the paper.

Decision: rejected-papers