Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes using second-order optimizers (K-FAC) to train large-scale Pre-Trained Models (PTMs). The key idea is to only apply the second-order update on adapter layers to make the storage and computation of the gradient covariances in K-FAC feasible in large PTMs. To further stabilize the training, the authors introduce a clipping scheme to control the size of the K-FAC update. Empirically, the proposed method can adapt additional layers in PTMs with faster convergence and improved performance compared to the AdamW baseline. 
strength_and_weaknesses: Strengths:
- The general idea of only applying second-order optimizers to adapter layers in Pre-Trained Models (PTMs) is interesting. 
- The paper is generally clearly-written and well-motivated. The topic of the paper is relevant to the ICLR community. 

Weaknesses:
- The justifications for gradient clipping (pre-clipping and post-clipping) are non-theoretical and weak. Moreover, the authors do not mention the existing works on damping and clipping, which I further address below. 
- Algorithm 1 contains several undefined variables and functions, and it isn't easy to understand the exact algorithm. 
- The empirical results in the paper are not convincing due to two reasons: (1) the authors do not include the results of (a) no clipping, (b) only pre-clipping, (c) only post-clipping, and (d) the proposed modification, and (2) the search space for hyperparameters such as learning rate seem to be much larger for K-FAC and I believe that the comparison to weakly tuned AdamW is unfair. 
clarity,_quality,_novelty_and_reproducibility: 1. Originality
- While the general idea of applying K-FAC to adapter parameters in training Pre-Trained Models (PTMs) is interesting, the authors do not sufficiently describe existing works on damping and clipping in Section 4. 
2. Clarity & Quality
- The paper is generally clearly written. However, I found the arguments in Section 4 (gradient clipping) to be weak and non-theoretical. 
- I believe that the paper has some technical issues, which I addressed in the additional comments section.
- Moreover, the experiments can be improved, which I also addressed in the additional comments section. 
3. Reproducibility
- The paper does not provide the code and the implementation is not described in detail. It would be challenging to reproduce the experiments.
4. Additional Comments
- The authors, after Eqn. 6, mention that they do not distinguish Newton’s method and natural gradient descent. However, I feel that this distinction is important and the paper needs correctly distinguish between Newton's update and natural gradient descent. Moreover, do the authors intend to say $\nabla^2_{\theta} L(\theta) \approx \hat{\mathbf{F}}(\theta)$?
- As opposed to the authors’ claims, K-FAC can compute the gradient covariances from samples from predictive distribution and can be seen as an approximation to the true Fisher instead of empirical Fisher. To be more specific, $\mathbf{G}_i$ should not be the covariance of the actual gradients seen during training. These are important distinctions in second-order methods [1] and must be carefully described and implemented. 
- Intuitively, it makes sense to control the (parameter-wise) step size of the K-FAC update, as it can make a very large step in low curvature directions. As a result, the authors propose gradient-clipping on natural gradient descent update. However, the authors do not mention previous works on damping [4] and clipping [2, 3]. 
- I believe that the arguments in Section 4.3 are weak and do not sufficiently explain the need for both pre-clipping and post-clipping. In Algorithm 1, what are other required quantities $\mathbf{h}$ and $\mathbf{r}$? At the moment, I cannot understand algorithm 1 due to missing definitions. Couldn’t these be merged into a single clip after the transformation? 
- As a follow-up question, the authors do not show which clipping improves the result in Table 2. Moreover, it would be helpful if the authors present the results without clipping to show improvements with the proposed modification.  
- In Table 2, what are the previously published results? It is not referenced in the paper. Moreover, the results of the validation loss are not shown in Figure 5 (and Figure 6).
- What were the search spaces of K-FAC? In Appendix B, the authors mention the rough range for the learning rate, clipping scale, and damping factor, but do not mention how many samples were used for the grid search. Is the comparison fair with AdamW? Looking at Table 3 and Table 4, it seems that the search space for K-FAC was larger compared to AdamW.
- The authors mention that K-FAC is more vulnerable to the choice of hyperparameters. It would helpful if the authors conduct an ablation study empirically showing this sensitivity. 
5. Minor Comments
- In the first line of the introduction, “Pre-trained models (PTMs)” → “Pre-Trained Models (PTMs).” Furthermore, I believe that the writing in the introduction can be improved.
- There needs to be a mention that Hessian is assumed to be invertible (strictly convex) in Eqn. 2.
- After Eqn. 7, the authors mention that the tunable approach accounts for roughly 0.5% ~ 8% of the parameters, but I believe that this linearly scales with the bottleneck dimension. By choosing higher $r$, couldn’t we bound the additional parameters? My comment also applies to Eqn. 8. 
- For the last paragraph in Section 3, are the authors claims’ based on the KFAC or the full Hessian matrix? 
- In Figure 2, why is there a fluctuation in memory usage? Is it due to the memory stored for a forward and a backward pass? 
- The $e$ notation in the Appendix seems weird to me. 

[1] Kunstner, F., Hennig, P., & Balles, L. (2019). Limitations of the empirical Fisher approximation for natural gradient descent. Advances in neural information processing systems, 32.

[2] Ba, J., Grosse, R., & Martens, J. (2016). Distributed second-order optimization using Kronecker-factored approximations.

[3] Koroko, A., Anciaux-Sedastrian, A., Gharbia, I., Garès, V., Haddou, M., & Tran, Q. H. (2022). Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decomposition. arXiv preprint arXiv:2201.10285.

[4] Martens, J., & Grosse, R. (2015, June). Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning (pp. 2408-2417). PMLR.

summary_of_the_review: While I believe that the key idea proposed in the paper is interesting, the paper has critical weaknesses in both technical and empirical aspects. At the moment, I recommend a score of 3 (reject).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper exploits a trend of low-rank finetuning or finetuning tiny number of parameters by applying a second order optimizer instead of typically used AdamW.
strength_and_weaknesses: Strengths
- The general idea of leveraging expensive (but more effective) optimizers in the case of finetuning small number of parameters seems novel and practically useful.

Weaknesses
- It is unclear if tuning the learning rate more heavily may be able to subsume post-clipping, it would be interesting plot how many iterations clipping is actually applied. If it is applied in all iterations, then it may be corrected by just tuning the learning rate more thoroughly.
- It would also be interesting to apply clipping to the baseline AdamW method. The authors discuss a hand-wavy rationale for not doing it, but it does not seem satisfactory and I would be interested in well-tuned empirical results on clipping AdamW update.
- Another interesting experiment would be to run adamw longer and see if it can recover performance with Newton steps.
clarity,_quality,_novelty_and_reproducibility: Presentation issues:
- The paper seems to be written quite informally, for instance "The mere combination of second-order optimization and parameter-efficient tuning is still far from training smoothly" or "clips the gradients as soon as the gradients are figured out". The paper's presentation can improve quite a bit if the authors would make their writing precise.
- Section 5.2: typo "NewtwonClip"

Reproducibility:
- I was unable to find # of training steps for the experiments in the appendix. I think it is useful to report it.
- The authors mention that "The search space for learning rate, Newton-step clipping scale, and damping factor are 1e − 2 ∼ 0.5, 0.1 ∼ 2.0 and 1e − 2 ∼ 1e − 6, respectively". To me, this is not specific enough. Please report either exact discrete points which were tested using a grid search or mention more details about an automatic tuner.

summary_of_the_review: While the paper proposes an interesting idea of leveraging potentially more effective optimizers in the case of finetuning small set of parameters, the paper's contribution of clipping Newton step poses several unresolved questions and not quite fully supported through evidence.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper makes two observations regarding training large language models:
1. During fine-tuning large transformers, several authors have previously shown that it is necessary to train a small number of parameters. This paper proposes using K-FAC to fine tune such models. The authors say that second order methods are very expensive for full training, but since fine-tuning involves few parameters, K-FAC is tractable, and achieves better results compared to first order methods.
2. Gradient clipping is a common method to prevent training blowups due to large gradients. The authors show that for K-FAC, this is not sufficient due to preconditioning, instead clipping should be done both before and after preconditioning.

The paper contains several experiments showing that a network fine-tuned with K-FAC achieves slightly better results than with AdamW.
strength_and_weaknesses: The main ideas in the paper are very easy to understand and well explained. The experimental section is good, with some caveats.

K-FAC is a carefully designed algorithm with extensive theoretical justification. How does NewtonClip affect this theory? Does the convergence rate change? Does the bias introduced by NewtonClip mess the independence assumptions?

Given that second order methods are more expensive computationally, the authors should include a comparison of the runtime and memory cost of AdamW with NewtonClip. Also, please state the results of training with K-FAC only (without clipping).

The statement "In spite of that amelioration, second-order optimization still requires at least N^2 ∼ N^3 order of storage space and computing operations" is not really correct for K-FAC --- in fact the storage is at most 2mN, where m is the largest dimension of any layer, so this is much less than N^2. Similarly, compute operations are at most m^2 N. Still large, but not N^3.

Page 7: NewtwonClip --> NewtonClip
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written.

My main concern is around novelty. The key ideas of fine-tuning with a small number of parameters, and clipping gradients are well studied in the literature. So the two new ideas here are --- using K-FAC instead of AdamW, and clipping gradients before and after preconditioning.
Neither are sufficiently novel to constitute an ICLR paper.
summary_of_the_review: The paper is currently not strong enough for ICLR. It can be strengthened by addressing some of the theoretical questions above, and slight enhancements to the experimental section.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This study applies second-order optimization (K-FAC) in parameter-efficient tuning (with adapters and LoRA) of Transformer models. Measurements show that the memory cost of second-order optimization can be relatively small due to the small number of trainable parameters, suggesting a second-order method is more suitable in this setting. This study proposes "Newton-step clipping," in which gradient clipping is applied after preconditioning, and shows that the training with K-FAC is stable. In the parameter-efficient tuning of the RoBERTa large model, K-FAC achieves lower training loss than AdamW for a given number of epochs.
strength_and_weaknesses: Strengths
- Second-order optimization methods, which have computational and memory overhead, are suitable for parameter-efficient tuning. The numbers of the tunable parameters and the memory overhead of a second-order optimization method (I assume it is K-FAC) in fine-tuning pre-trained Transformer models presented in this work (Table1) help highlight this point.

Weaknesses
- The technical significance is limited.
    - With parameter-efficient tuning, the training time could already be short (this is what parameter-efficient tuning is for), even with first-order optimization. The low computational and memory cost of second-order optimization is attractive, but (as I also describe below “the validity of the comparison”) it is questionable whether the benefit of the second-order information is worth the cost in parameter-efficient tuning. 
    - The proposed “Newton-step clipping” merely shifts the timing (from pre- to post-preconditioning) at which gradient clipping is applied. Post-preconditioning clipping of K-FAC gradient based on the KL-divergence of the model’s predictive distribution before and after the update (rather than the Euclidean norm of the update as in the gradient clipping) has already been proposed (https://jimmylba.github.io/papers/nsync.pdf). How one should determine the timing (pre- or post-) and "distance" (e.g., KL-divergence, Euclidean norm) is an interesting question, but that is not studied in this work.

- The validity of the comparison is questionable.
    - It is not clear whether the number of epochs shown in Figure 5, which depends on the task, is reasonable since the timing at which training stops (e.g., when training loss reaches a threshold, a fixed number of epochs) is not stated. In this case, there appears to be room for AdamW to achieve better performance when the number of epochs is further increased, and its difference from K-FAC could be more negligible.
    - Furthermore, there is no mention of how the model is selected for evaluation (e.g., using a validation set) or how the training set, validation set, and test set are partitioned, so it is doubtful that hyperparameters are tuned on the test set. (Just because training loss converges faster, as shown in Figure 5, does not necessarily mean that the model generalizes well.) 
    - It is unclear whether K-FAC is "faster" than AdamW since there is no comparison regarding the wall-clock time for training. K-FAC is not necessarily faster than AdamW in wall-clock time (= number of steps x time per step) because of the per-step computational overhead compared to first-order optimization methods.
    - Finally, optimization methods should be compared not only for best results but also for their sensitivity to the choice of hyperparameters. It is difficult to consider an optimization method practical for other tasks if it is too sensitive. 

Other comments
- Table1: what are the “first-order optimization” and “second-order optimization”? A reader can guess they are SGD (or AdamW) and K-FAC, but it is unclear.
- Eq (19) 2nd line: I believe it should be (a \times g)(a \times g)^T. The shape of each matrix and vector should be described to avoid confusion.
- p6 the last paragraph, “derivative-based optimization”: I believe this should be “gradient-based optimization.”
clarity,_quality,_novelty_and_reproducibility: As I mentioned in “Weaknesses,” some experiment details are unclear. This raises questions about the quality of the comparison and makes it difficult to reproduce the results. The technical novelty of this work seems limited. 

summary_of_the_review: The use of second-order information in parameter-efficient tuning is more reasonable than in full-parameter tuning, but its practical benefit is not clearly demonstrated in this study. The novelty of the proposed clipping procedure is limited, and its justification is more of an intuitive interpretation than a "theoretical" one. Based on the above, I do not believe that this study at this time is ready for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to train large models by combining two previously proposed ideas: K-FAC optimization and parameter-efficient tuning. In addition, it proposes to replace (or combine) gradient clipping with clipping of the final update.

On one hand, this paper is a nice demonstration that second-order optimization techniques can scale to 3-billion-parameter models. On the other hand, the reviewers feel (and I agree) that the novelty is fairly limited, as the proposed method combines two existing ideas in a relatively straightforward way. The clipping trick is fairly incremental, and various reviewers pointed out prior occurrences in the literature. The question, then, is whether the experimental results are compelling enough by themselves to justify acceptance; i.e., should they persuade us that the combination of PBT and K-FAC is more powerful than we might have naively supposed?  There was a fair amount of discussion between reviewers and authors about the details of the experimental validation, but at the end my impression is that the evidence is ambiguous. I hope the authors take the reviewers' feedback into account for resubmission.

justification_for_why_not_higher_score: See above.
Decision: Reject