Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work contributes a analysis of the dynamics of neural network training using a new quantity called the activation pattern temperature, that measures the amount of (ReLU) neuron activations that switch from 0 to positive, or positive to 0.

They measure the APT during training of typical CNNs using different learning rate schedules and show that using these schedulers neural networks typically undergo a transition from a high temperature regime to a small temperature one.

They then design a learning rate scheduler that adapts the learning rate so that the temperature of the optimization dynamics during training follows a predefined curve -- in this case it decreases linearly from 3.17 (why?) to 0.
main_review: I enjoyed the originality and clarity of this paper. There are however a few points that can be improved, and I am not exactly convinced that introducing the APT gives a useful tool to analyze training dynamics, or in other word that this contribution is significant enough for a publication at NeurIPS.

(Major) things that can be clarified

1. What new insight can APT give that other quantities in the literature cannot?
2. Additionally, how can we use the APT to design better neural networks?
3. How did you come up with the formula in eq. 5? 
4. line 255, you write "we interpolate between the new and old learning rate" => Can you make this statement more precise? At first read it looks that you can anticipate on the future (i.e. the new) optimal values for mu?
5. Are you using momentum in the experiments in figure 2? If so, how does it affect the actual step size in parameter space after applying momentum?
6. In Figure 3., you are using momentum. It seems that in this case, the actual APT is not fully determined by the learning rate, but also by the additional momentum term in the update. Can you please discuss how momentum interacts with the APT?

(Minor) questions relating to log scales in plots:
 1. I question the choice of a log scale in Fig. 1 right for the APT plot. Aren't we looking at the log of a quantity that is already a log (the temperature)?
 2. On the same figure, I find it a little bit weird to use a log scale for the validation error, whereas I have always seen linear scales in other works.
 3. Most of the change in accuracy typically occurs during a few training epochs in the beginning, can you maybe reflect that by plotting epochs on a log scale in Figure 2 e.g.?
 4. In Figure 3, can you use a log scale for the learning rate, as it typically ranges over several orders of magnitude?
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper attempts to understand the change of activation patterns, meaning the post-ReLu sign flip before and after one parameter update, for the same batch of input data. A measure, activation pattern temperature (APT), is then defined on a layer as the self-information of the event that an activation pattern has not changed. The author uses APT to interpret different training methods and training dynamics. Finally, based on the observation that existing learning rate schedule adheres to a linear decrease of temperature, the author propose ActCooLR, a LR scheduler that was tested on a variety of training tasks.

main_review: The paper is elegantly written, very engaging throughout. One impression I get after the first read is that it is more about the temperature concept, aka, APT, than the learning rate scheduler ActCooLR. But the title seems to suggest otherwise. One suggestion would be to modify the title to truthfully reflect the content.

Possibly one remark that would be raised by reviewers, and all readers in general, is that the eventually devised LR scheduler doesn't really work -- on all the experiments it didn't show a superiority over other contemporary methods. But I am a strong believer that the merit of a scientific work does not solely depend on the positivity of results. And I do believe the APT method, the relationship between LR and APT, and the empirical analysis provide great insight to the community that's actively trying to uncover the mystery of training dynamics.
limitations_and_societal_impact: I strongly suggest rewriting the title (and maybe the abstract and part of the introduction) to reflect that this paper is more about the notion of APT, and how it can be an investigatory tool to uncover unknown bits of model training, than it is about a brand new, game-changing LR scheduler (too bad it's not). And I encourage the author to further explore APT as a tool to study other aspects of learning, not just LRs.

The major limitation, as far as I can see, is that ActCooLR doesn't really perform better than other hand designed schedulers. I do not overly discredit the overall scientific value of this work because of that, but it remains a major drawback of this work.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper introduces a measure (Activation Pattern Temperature, APT) of the rate of change in the function modelled by a neural network (with ReLU non-linearities) as training progresses. 

This measure is then used to set a learning-rate schedule by regulating the APT, an adaptive learning rate schedule denominated ActCooLR.

The paper compares the performance achieved by ActCooLR to a standard step-wise decaying learning rate and to a schedule called 1-cycle.  ActCooLR did not obtain better results than the alternatives.
main_review: The main idea presented in the paper, measuring the rate of change to a different linear regime (per linear-ReLU layer) is original and interesting. However,  the paper does not yet provide enough theoretical insight or compelling practical methods to warrant  being presented to a wide audience. The method, as presented, is only applicable to networks with piece-wise-linear non-linearities, and the adaptive learning-rate introduced does not achieve better results than the alternatives.

The paper starts with a good introduction to related work and the problem of hyperparameter scheduling. But the quality of the exposition deteriorates later on in the paper.  For example:
 * The reasons why the APT measure is denominated a temperature is not clearly explained given that it is introduced as a log-probability in equation 4. 
* There are several hints in the paper that the authors will connect this measure to temperatures in simulated annealing but this connection is never explicitly presented.
* Why is APT calculated per layer, and not for the whole network or per unit? This is not even discussed in the main paper.
* The sequences of $\theta$ and $M$ are mentioned to be Markov chains, but this is not necessary for the rest of the exposition. Same when in line 193 the authors mention there are hidden variables responsible for the course of APT, this is rather mysterious and not necessary for the exposition.
* The figure captions are incomplete and it is necessary to read the main text to understand what is being presented. Sometimes the axis labels used are difficult to understand (e.g. Figure 1 for the validation errors, why use $10^{-0.1} - 10^{-1}$ instead of 0.8 - 0.1? 
* They introduce Hypothesis 1, and then they support it by saying it fits in practice. This gives a veneer of formality that is not really necessary. Simply saying they are fitting the APT vs learning rate function with a logistic would be clearer.

Finally, ActCooLR does not obtain better final performance than the alternatives (which are simpler to implement and more general).

I would like to end encouraging the authors to pursue the ideas further, as they are interesting but the paper is not yet conference-ready. Some of the ideas left for future work in the paper, like using a different learning-rates per layer, may push the final accuracy and make it a practical alternative for applications.
limitations_and_societal_impact: The authors haver addressed the limitations and potential negative societal impact adequately.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This papers aims to explore the impact of learning rate schedules in neural network training, through   a newly proposed measure denoted the “activation pattern temperature” which examines the probability that ReLU activations will change between the 0 and linear regions. Thus this measure aims to examine the changes in non-linear behavior of the network. Several learning rate schedules are examined through this lens, and a learning rate schedule based on reducing this temperature is proposed and examined.
main_review: The idea of defining and estimating notion of temperature and linking it to the learning rate is interesting, and the text provides several experiments that explore the development of the temperature throughout the training of several networks. I think there are several aspects of the text that would benefit from further insight in order to understand the potential utility of the temperature measure.

How the APT is computed is not clear from the “computation” paragraph, could the authors provide an algorithm?

Except the first few epochs, the APT seems to track the LR very closely (as seen in Fig. 1) and the APT distribution seems to change very little (Fig. 2). Are the changes in APT distributions in early training understood or can they be explored more, and can these changes give insights into the early training process? Beyond that early training, APT seems very closely related to LR, what additional insights about the training development can I get from APT over LR? How does the APT relate to the gradient norms? Or how does it affect parameter trajectories?

What is the benefit of using an APT scheduler instead of a LR scheduler? It would seem to require the same amount of schedule design, but more computation to eventually determine the LR is needed with APT. How does designing temperature schedule, i.e. a schedule damping of non-linear network changes, make the training process easier, it seems as abstract as an LR schedule?

How do you know when you are in a chaotic vs non-chaotic non-linear regime, and how can a reader see that controlling APT better navigates these regimes?

Table 1 - why are different method comparisons used for ever experiment? It makes it challenging to compare methods across experiments.

The authors state that only tracking the nonlinear behavior is enough to provide the information needed for step-size control. It was not clear to me how the conclusion was drawn. Are there toy experiments that could show this?

------------------------------
After Author response:

I believe the authors have provided detailed answers to all of my questions, and I believe these answers are sufficient and provide more clarification for me. I believe that ensuring this information is clearly discussed in the paper would be an important update for the text. I think the paper would benefit from a deeper mathematical / theoretical exploration of the ideas presented, but the current empirical analysis is still of value. I have update my scores accordingly.
limitations_and_societal_impact: The authors clearly discuss limitations and broader impact
review_title: null

Meta-review: In this paper the authors study the rate of sign flips of the RELU units (or more precisely their inputs) during training, and design a learning rate schedule which tries to achieve a linear decrease in this rate over iterations (which was seen empirically for other well-performing LR schedules).

The reviewers found the main idea of studying sign flips for RELUs during optimization to be original and interesting. Their main complaints are that the theoretical explanations for the method are missing or vague, and the experimental results aren't very exciting. Since this method doesn't actually outperform the best LR schedules, its utility comes down to whether it truly offers a simplifying explanation to the problem of tuning LR schedules, and whether this allows it to generalize in non-trivial/surprising ways to new problems. Unfortunately, the reviewers didn't really seem convinced by the explanations, and the experiments were limited to very standard problems that were of the same sort as those used to help design the method in the first place.
Decision: Reject