Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes forward-mode differentiation with sharing (FDS), which aims to combine the advantages of derivative-free HPO and gradient-based HPO. Memory scaling issues and gradient degradation issues motivate this work. The paper talks about a new method on the gradient-based path of HPO. According to the experiments of the paper, this new method has the potential to have major impact in the HPO area with further evolution. 
main_review: The motivation and the idea of the paper are both good. The language is smooth and the structure is clear.
There are still a few weaknesses of the paper.
First, in line 308 "Note also that we are not interested in finding the architecture with the best performance, but rather in finding the best hyperparameters given an architecture". The work aims at HPO rather than structure searching, but to show one HPO method really works, more experiments on a variety of datasets and with different structures of NNs are necessary. 
Second, also extends the first issue, to utilize one HPO method in AutoML on NNs, the co-working with structure searching is inevitable. It is recommended to talk about the advantages and disadvantages of co-working, compared to existing HPO methods.
limitations_and_societal_impact: The most vital limitation of the work is discussed in line 349-351 by the authors.
It's unlikely for the work to have any potential negative societal impact.
So I believe the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes methods to allow stable gradient-based hyperparameter optimization. The first proposed method is to make use of forward-mode differentiation instead of reverse-mode, allowing differentiating through long horizons with constant memory cost. The second proposed method is to share hyperparameters across time, so that the hyperparameter value is constant for a number of steps, instead of having different values every step. The third, rather hidden, proposed method is a sign-based optimizer for training the hyperparameters.
main_review: Overall, this paper explores a couple simple ideas that seem to work well. Forward-mode AD is a nice solution to the memory problem of taking gradients through long horizons, but I’m not convinced there is a need for deriving update equations (such as Eq 3) on a case-by-base basis. Reducing the number of hyperparameters (by sharing them across time steps) is a convincing method to handle hypergradient stochasticity, but this seems like a separate contribution and should probably be ablated against greedy methods (i.e. apply alternative methods to a piecewise constant hyperparameter schedule).

Questions:

Is there a reason why the method is restricted to SGD + momentum? Forward-mode autodiff should be applicable to all kinds of optimizers. Note that you can simulate forward-mode autodiff from reverse-mode autodiff (see https://j-towns.github.io/2017/06/12/A-new-trick.html and https://pytorch.org/docs/stable/generated/torch.autograd.functional.jvp.html?highlight=jvp#torch.autograd.functional.jvp), and JAX natively supports forward-mode autodiff along with just-in-time compilation. I imagine that the Jacobian dtheta_t / dtheta_{t-1} can be quite sparse and that a manual construction of this matrix removes the possibility of automatic memory and compute optimizations.

What is the empirical memory cost of using FDS, compared to alternative methods?

A single step may be too short, but what if you took H to be somewhere between 1 and T? What does the trade-off curve look like between memory cost and validation error, for both reverse-mode and forward-mode differentiation? In essence, this can help answer (i) how non-greedy is better than a short-but-not-too-short horizon, and (ii) an empirical memory cost comparison between reverse-mode AD vs. this manual forward-mode AD.

Is the sign-based hypergradient optimizer used for all experiments, or only those in 5.4? Was this used for the baseline methods as well? It's not clear to me how much this contributes to the experiments. Some ablation experiments would be good.


Minor:

In section 4.2, H is mentioned but not defined. Is this referring to A from the previous sections? 
limitations_and_societal_impact: More clarification regarding the improvements for each individual contribution could be warranted.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a long-horizon hyperparameter-optimization through sharing hyperparameters in combination with forward-mode differentiation. 
main_review: The paper is nicely written, the results are well presented and thoroughly tested. 
The novelty of the paper is limited, nevertheless, the overall presentation, as well as the theoretical insights, make the paper valuable to the community. 
I find it intriguing that hyperparameters, in this case, the learning rates, tend to follow commonly used hand-crafted schedules.
In general, but that also applies to related work, I am curious about the effect of the validation set size. 
Can you comment on that / provide numbers? It would be interesting to see the effects of that e.g. the changes of the presented learning rate curves. 
limitations_and_societal_impact: Have the authors adequately addressed the limitations and potential negative societal impact of their work? If not, please include constructive suggestions for improvement: Yes.
review_title: null

Meta-review: All reviewers agree on acceptance. This is a straightforward decision. The authors should use the feedback given by the reviewers to improve the paper for its camera-ready version.
Decision: Accept (Poster)