CAME: Confidence-guided Adaptive Memory Efficient Optimization
Year: 2023

Abstract:
Adaptive gradient methods, such as Adam and LAMB, have demonstrated excellent performance in the training of large language models. Nevertheless, the need for adaptivity requires maintaining second-moment estimates of the per-parameter gradients, which entails a high cost of extra memory overheads. To solve this problem, several memory-efficient optimizers (e.g., Adafactor) have been proposed to obtain a drastic reduction in auxiliary memory usage, but with a performance penalty. In this paper, we first study a confidence-guided strategy to reduce the instability of existing memory efficient optimizers. Based on this strategy, we propose CAME to simultaneously achieve two goals: fast convergence as in traditional adaptive methods, and low memory usage as in memory-efficient methods. Extensive experiments demonstrate the training stability and superior performance of CAME across various NLP tasks such as BERT and GPT-2 training. Notably, for BERT pre-training on the large batch size of 32,768, our proposed optimizer attains faster convergence and higher accuracy compared with the Adam optimizer. The implementation of CAME is publicly available1.

heading: 1 Introduction
text: Robust training of large language models (LLMs) often relies on adaptive gradient-based optimization methods (Li et al., 2022; Kingma and Ba, 2015; Zhuang et al., 2020). Through the use of cumulative second-order statistics, these methods adapt the per-parameter learning rate and demonstrate superior convergence speed during the training process of LLMs. However, the remarkable performance of adaptive methods incurs an extra cost of memory usage indeed. For example, Adam requires to preserve the first moment estimate and second
*Work was done when Yang Luo was an intern at Huawei Noah’s Ark Lab.
1https://github.com/huawei-noah/Pretrained-LanguageModel/tree/master/CAME
raw moment estimate of each gradient in order to tune the learning rate for each parameter, which inevitably triples the memory usage concerning the optimizer states. Besides, with the growing size of the model, LLMs are becoming increasingly expensive in terms of memory, and the limitation of memory is gradually emerging as a main bottleneck for training LLMs.
Many existing memory-efficient optimizers attempt to store second-order statistics with sublinear memory requirement while retaining the exceptional convergence property of adaptivity (Shazeer and Stern, 2018; Anil et al., 2019). Adafactor optimizer achieves remarkable memory cost reduction by applying the non-negative matrix factorization algorithm (Lee and Seung, 2000) to factorize the accumulator matrix for squared gradients into two rank-1 factors as shown in Figure 1, where the memory requirement for the original matrix V decreases from O(nm) to O(n + m). Whereas, it is observed that Adafactor suffers a performance degradation in the training of large language models universally compared with conventional adaptive gradient-based optimization methods. The reason for this phenomenon is Adafactor inevitably introduces some errors that cause instability in training deep networks due to the operation of nonnegative matrix factorization.
4442
In addition, in the case of large-batch training that aims to accelerate the training of deep neural networks, the memory consumption of each machine (GPU/TPU) is much higher than general batch size training, which further imposes a grave constraint on the performance of the trained model. In comparison to standard training tasks, largebatch training presents more challenges for optimizers. Empirically, when the mini-batch size increases after a certain point (e.g. 1024), the test accuracy of the converged solution decreases significantly compared with the baseline (He et al., 2021). To our knowledge, there is currently no work related to memory-efficient optimizers for large-batch training.
Motivated by these challenges, we firstly study a confidence-guided strategy catered to alleviate the instability of Adafactor by calculating the confidence of the generated update at each training step. On the basis of the adaptation strategy, we propose a novel CAME optimizer that saves nearly the same memory footprint as existing memoryefficient optimizers while attaining faster convergence and superior generalization performance. To further assess the scalability of our proposed algorithm, we consider an additional challenging experiment - performing large-batch training on BERT using CAME optimizer.
Contributions of our paper can be summarized in the following:
• Inspired by training instability of Adafactor, we explore a confidence-guided strategy centered on the existing error in the raw updates of Adafactor for parameters of large language models.
• In light of the dedicated strategy, we propose a novel optimization algorithm, CAME, for achieving faster convergence and less performance degradation catered at memoryefficient optimization. We further investigate the effect of the proposed memory-efficient optimization algorithm in large-batch training settings.
• We demonstrate the powerful performance of CAME with extensive NLP experiments: CAME shows faster convergence and better generalization capability than Adam in BERT pre-training task with two different batch sizes (32k and 8k); in the training of GPT-2 model
and T5 model, CAME achieves fast convergence speed as Adam without degrading of performance. Notably, in the large-batch training of the BERT model, CAME obtains comparable validation accuracy with LAMB using around 15% less memory usage.

heading: 2 Related Work
text: Memory Efficient Adaptive Optimization Memory efficient optimizers maintain the benefits of standard per-parameter adaptivity while significantly reducing memory footprint. Adafactor (Shazeer and Stern, 2018) proposes to reconstruct a low-rank approximation of the exponentially smoothed accumulator at each training step that is optimal with respect to the generalized KullbackLeibler divergence. SM3 (Anil et al., 2019) divides the elements in the second-order gradient matrix into sets by the observed similarity of the elements, and each item in the generated approximation matrix is the minimum of the maximum value of each set in which it is located. The methods mentioned above behave poorly in the training of large language models and converge slowly, which raises a significant challenge for memory-efficient optimization methods.
Large Batch Training A large-batch training scheme is preferred in distributed machine learning because of its ability to increase parallelism by enhancing large-scale cluster utilization. It has seen growing interest in recent years in large-batch training (Liu et al., 2022; Li et al., 2021; Huo et al., 2021). In particular, a layer-wise adaptive learning rate algorithm LARS (You et al., 2017a) is proposed to scale the batch size to 32k for ResNet50. Based on LARS, LAMB optimizer (You et al., 2019) can finish the BERT training in 76 minutes through TPU v3 Pod. Despite the success of these approaches for BERT models, the much larger batch size highly boosts the GPU usage which is prohibitively expensive and inaccessible to most researchers.
Moreover, training with a large batch size incurs additional challenges (Hoffer et al., 2017; Keskar et al., 2016). Large-batch training is prone to converge to sharp local minima, since the number of interactions will decrease when the batch size is increased if the number of epochs is fixed, which causes a wide gap in generalization of the model(Keskar et al., 2016). Traditional methods seek to narrow the generalization gap by carefully
tuning hyperparameters, such as learning rate, momentum, and label smoothing, to narrow the generalization gap (Goyal et al., 2017a; Shallue et al., 2018; You et al., 2017b). Yet there have been few attempts to reduce memory usage in large-batch training, and the underlying challenge remains unclear.

heading: 3 Method
text: In this section, we firstly provide a brief description of the Adafactor optimizer and discuss the errors contained in the update of Adafactor (erroneous update). We further study a confidence-guided strategy and introduce the proposed CAME in detail in light of the strategy.

heading: 3.1 An overview of Adafactor
text: The L(θ) ∈ R represents the loss function that we plan to minimize, where θ ∈ Rn×m is the parameter of the model. gt is the gradient at step t, η is the learning rate, rt and ct are the exponential moving average of two low-rank factors for the second moments of the gradient. ϵ1 is a small regularization constants and ut is the current approximate update.
In the training of large language models, Adafactor is required to apply momentum to ensure the convergence (Chowdhery et al., 2022), and the corresponding pseudocode is illustrated in Algorithm 1. The problem setting is as follows. Assume that we aim to minimize the expected value of an objective function f(θ). At each training step, we receive the loss derived from a mini-batch of data, and calculate the gradient gt of the function based on the previous parameters. Subsequently, we update the exponential running averages of two factors for second moments of the gradient rt and ct, compute approximations for the second moments of the gradient vt, and adjust the generated update (ut) when RMS(ut) surpasses a specific threshold value d as in:
ût = ut
max(1, RMS(ut)/d) (1)
where RMS(ut) refers to the root-mean-square calculation of the components of ut. Finally, the first moment of the adjusted update mt is utilized to update the parameter, resulting in a new iteration θt. The optimization continues until the parameters converge and returns the final iteration θT as our approximate solution.
Adafactor derives an effective solution for nonnegative matrix factorization in the special case
of rank-1 factors, which obtains the minimal Kullback–Leibler divergence (Lee and Seung) between the matrix V and the approximated matrix WH . The formulation of the solution is as follows, in which 1m = (1, ..., 1) ∈ Rm represents a column vector of m ones:
W = V 1m, H = 1TnV
1TnV 1m . (2)
It should be noted that Adafactor stores only the moving averages of these factors rather than the entire matrix V , yielding considerable memory savings and requiring memory usage proportional to O(n+m) instead of O(nm).

heading: 3.2 Erroneous Update
text: The non-negative matrix factorization operation in Adafactor will inevitably incur erroneous update in the training of deep neural networks. As shown in Figure 2, Adafactor always converge slower than Adam due to the existing error in calculated updates, which further limits the application scenarios of memory-efficient optimizers.
As shown in Figure 3, two scenarios demonstrate how two types of erroneous updates are supposed to be handled in the ideal case. In Figure 3(a), the difference between the momentum of updates mt and the current update ut is large, illustrating that the historical experience for the update of original Adafactor contains high level of errors that will inevitably influence the stability of the training process. If we utilize the raw mt to take an optimization step, the direction of optimization will deviate increasingly from the desired direction, which is reflected by the slow convergence and performance
Algorithm 1: Adafactor Optimizer Input: Initial parameters θ0, learning rate η, momentum of update m0, r0, c0, step t, regularization
constant ϵ1, exponential moving average parameters β1, β2, clipping threshold d while θt not converge do
Compute gt = ∇f(θt−1) rt = β2rt−1 + (1− β2)(g2t + ϵ11n1Tm)1m ct = β2ct−1 + (1− β2)1Tn (g2t + ϵ11n1Tm) vt = rtct/1 T nrt
ut = gt/ √ vt ût = ut/max(1, RMS(ut)/d) mt = β1mt−1 + (1− β1)ût θt = θt−1 − ηmt
end
degradation of existing memory-efficient optimizers. By contrast, when the difference between mt and ut is small as shown in Figure 3(b), the momentum mt is stable with limited errors and high confidence therefore a large optimization step is required with the updating direction close to mt.
Inspired by the erroneous update that is universal in existing memory-efficient optimizers, we firstly consider an efficient approach to decrease the side effect caused by insecure updating. Given mt and ut, we take the residual between them as the instability in the preserved momentum and set generated instability as the denominator of original mt to more adaptively take an update step. Following
is the formulation of the adjusted update u′, where ϵ is the regularization constant:
u′t = mt√
(mt − ut)2 + ϵ (3)
Extending further on the plain method, we propose a confidence-guided strategy that enables selfadjusted updates by taking the confidence of the raw update of Adafactor into consideration. The intuition behind the proposed strategy is to calculate the residual between the exponential moving average (EMA) of the update and the current update, which represents the deviation of the approximated update. The larger the deviation of the EMA value from the current generated update, the wider the error EMA of update contains, resulting in a lower level of confidence in the EMA of update. Obviously, we expect the optimizer to take a small update when it incorporates huge error (a large residual from the present update), while updating parameters more when the optimization process is stable (involved error of EMA is limited).
Specifically, the EMA of update mt is directly used to take an update step in Adafactor, while in our proposed strategy, mt is divided by √ Ut, where Ut is the calculated instability matrix. Therefore, 1√ Ut
is the confidence in the observation: viewing mt as the prediction of the update, if mt deviates greatly from ut (Ut is large), which indicates a weak confidence in mt, the optimizer performs a small optimization step; if ut closely matches mt, we have solid confidence in mt, and correspondingly take a large optimization step.

heading: 3.3 CAME Algorithm
text: Based on the proposed confidence-guided strategy, we develop a brand-new variant of memory-
Algorithm 2: CAME Optimizer Input: Initial parameters θ0, learning rate η, momentum of update m0 = 0, r0 = 0, c0 = 0, step t
= 0, regularization constants ϵ1, ϵ2, exponential moving average parameters β1, β2, β3, clipping threshold d
while θt not converge do Compute gt = ∇f(θt−1) rt = β2rt−1 + (1− β2)(g2t + ϵ11n1Tm)1m ct = β2ct−1 + (1− β2)1Tn (g2t + ϵ11n1Tm) vt = rtct/1 T nrt
ut = gt/ √ vt ût = ut/max(1, RMS(ut)/d) mt = β1mt−1 + (1− β1)ût Ut = (ût −mt)2 Rt = β3Rt−1 + (1− β3)(Ut + ϵ21n1Tm)1m Ct = β3Ct−1 + (1− β3)1Tn (Ut + ϵ21n1Tm) St = RtCt/1 T nRt
θt = θt−1 − η√Stmt end
efficient optimization methods with faster convergence. Our proposed CAME optimization method successfully obtains the same rate of convergence as prevailing first-order optimization algorithms (e.g., Adam) and with almost equal memory cost to available memory-efficient optimizers (e.g., Adafactor). The pseudocode of CAME algorithm is specified in Algorithm 2.
By calculating Ut at each training step, we employ non-negative matrix factorization on the instability matrix Ut following (Shazeer and Stern, 2018) where the generalized Kullback-Leibler divergence between V and WH is minimal. With Ut factorized into Rt and Ct, it is sufficient to store only the moving averages of these factors rather than the full matrix Ut, thus saving considerable memory footprint.
We simply validate intuitions and the corresponding example is shown in Figure 4, in which the proposed CAME reaches the optimal point much faster than Adafactor. Learning rate is 10−3 for all optimizers. In the example, we set the parameters of CAME to be the same as the default in Adafactor, β1 = 0.9, β2 = 0.999 and set extra β3 = 0.9999 for CAME.

heading: 4 Experiments
text: In this section, we present extensive comparisons with existing optimizers on training tasks of three important large language models: BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2018a) and T5
(Raffel et al., 2022).

heading: 4.1 Setup
text: Dataset We perform experiments on the BookCorpus (Radford et al., 2018a) and English Wikipedia with 800M and 2.5B words respectively. Furthermore, we focus on the GLUE benchmark (Peters et al., 2018), SQuAD v1.1 dataset (Rajpurkar et al., 2016) and SQuAD v2.0 dataset (Rajpurkar et al., 2018) to demonstrate the performance of pre-trained BERT models with CAME optimizer.
Model We evaluate the efficiency of our proposed CAME on three trending large language models: BERT, GPT-2 and T5. We further test the performance of CAME for large-batch training
with BERT-Large. Compared methods The main baselines comprise two widely-used optimizers: classic optimizer Adam and memory-efficient optimizer Adafactor. With regard to large-batch training, LAMB optimizer is additionally considered when setting baselines.
Implementation Detail We implement our optimization algorithm in Pytorch (Paszke et al., 2019). The parameters β1 and β2 in Algorithm 2 are set as 0.9 and 0.999 respectively, and we search for optimal β3 among {0.9, 0.99, 0.999, 0.9999, 0.99999}. We use 8 Tesla V-100 GPUs and set ϵ1, ϵ2 as 10−30, 10−16 in all experiments with gradient accumulation and model parallelism. Besids, we set η as 2 × 10−4, 6 × 10−4, 3 × 10−4 for BERT-Large (32K), GPT-2, T5 training and apply learning rate warmup scheduling (Goyal et al., 2017b) to avoid divergence due to the large learning rate, by starting with a smaller learning rate η and gradually increasing to the large learning rate η. To make sure we are comparing with solid baselines, we use grid search to tune the hyperparameters for Adafactor, Adam and LAMB. We further improve the performance of large-batch training by applying Mixup (Zhang et al., 2017) to scale the batch size up to 32,768.

heading: 4.2 BERT Training
text: We firstly present empirical results in the training task of BERT model to evaluate the performance of our proposed CAME optimizer, focusing on its larger variant, BERT-Large, which has 340M parameters in all. Following the default setting, we pre-train the BERT-Large model (L = 24, H = 1024) with a sequence length of 128 on 8 Tesla
V-100 GPUs. The experiments were implemented with the code from NVIDIA 2 and mainly include two types of batch sizes: 8k and 32k, one of which represents the widely used setting for pre-training BERT and the other denotes the training scenario under large-batch training. The empirical results are presented in Figure 5 and Figure 6. As illustrated in Figure 5, CAME achieves a significant improvement compared with Adam and Adafactor. To be specific, CAME (66.5%) increases validation accuracy at with an increment 3.4% in comparison to Adafactor (63.1%) using same number of training steps (20k). Apart from Adafactor, our proposed CAME achieves better performance than Adam in the pre-training of BERT-Large model with a huge reduction of memory cost.
To evaluate the performance of our proposed CAME for large-batch training, we scale the batch size for BERT-Large training to 32,768 on Wikipedia dataset. As illustrated in Figure 6, CAME consistently reaches a more remarkable improvement compared with Adafactor. We notice that the accuracy of CAME on BERT-Large pretraining is 68.0%, which is highly over the original Adafactor (61.9%) with same number of training steps. In addition, CAME reaches comparable accuracy with only half the training steps required for Adafactor. With batch size getting larger from 8k to 32k, CAME brings more enhancements to the training of BERT-Large in comparison with Adam and Adafactor. Compared with LAMB in large-batch training, CAME saves a high-level of
2https://github.com/NVIDIA/ DeepLearningExamples
memory footprint with slight training performance degradation.
Memory Usage Comparison We set batch size to 1 to measure the memory usage of each optimizer more efficiently. As shown in Table 1, the two optimizers (Adam and LAMB) frequently employed for training large language models consume the highest amount of memory usage. Meanwhile, our proposed CAME optimizer exhibits a reduced memory footprint over the existing SM3 memory-efficient optimizer. As a consequence of our confidence-guided strategy in CAME, there is no doubt that CAME will introduce an increased memory footprint in comparison with Adafactor. However, the extra memory footprint incurred of CAME is almost negligible (1%) with a substantial performance improvement.
For further demonstration of the memory saving effect of CAME, we expand BERT model to BERT-4B with 4 billion weights using the scaling method of GPT-3 (Brown et al., 2020). We set the mini-batch size to 64 and the accumulation steps to 16 in this experiment. In Figure 7, we train BERT4B with three different optimizers using PyTorch framework. As a result, CAME can save 47% memory footprint about optimizer states compared with Baseline (Adam) when the weights number of a model get to 4 billion.

heading: 4.3 Downstream Tasks
text: We select a representative set of downstream tasks to further demonstrate the performance of BERT models pre-trained by our proposed CAME. In this part we adopt BERT-Base model for the fine-tuning task and follow the originally published BERT-
Base results in (Devlin et al., 2019) and (Liu et al., 2019) as the main baseline. The learning rate is tuned on the dev set for each setting and each task is fine-tuned for three epochs.
We compare the end-task performance of BERTBase with the baseline on typical downstream tasks and the empirical results are presented in Table 2. The experimental results demonstrate the efficiency of our proposed CAME optimizer by showing that BERT-Base model trained with CAME on two batch sizes both achieve comparable performance to the baseline with less memory cost. In particular, we observe that BERT-Base model trained with large batch (32k) presents no performance degradation and even attains higher evaluation metrics scores on some downstream tasks. Specifically, the BERT-Base model trained on CAME improves on average by 0.5 across five metrics compared to the baseline, proving the feasibility of CAME for the large-batch training task.

heading: 4.4 GPT-2 Training
text: In addition to BERT pre-training task, we perform CAME-based training task on another typical large language model, GPT-2. Using the original structure of GPT-2 (Radford et al., 2018b), we specifically adopt GPT-medium (L = 24, H = 1024) with 345M parameters in our experiment. This implementation is based on the code provided by Megatron3. Identically, we take English Wikipedia as the training dataset for this section. Unlike the pre-training of BERT in Section 4.2, we only concentrate on standard training batch size (128) for GPT-2 pre-training.
The empirical results of validation loss are shown in Figure 8. We are able to find that CAME achieves similar convergence and final accuracy compared to Adam, which reveals an impressive improvement over the performance of Adafactor
3https://github.com/NVIDIA/Megatron-LM
0 20 40 60 80 100 120 Steps(K)
4.0
4.5
5.0
5.5
6.0
6.5
7.0
7.5
Va lid
at io
n Lo
ss
Adam Adafactor CAME
Figure 8: Validation loss of GPT-2 language model. CAME demonstrates similar optimization performance to Adam.
with comparable training steps. Moreover, as indicated in Figure 9, the validation perplexity of CAME presents the same convergence performance as Adam but faster convergence speed than Adafactor, which clearly supports the validity of CAME that has fast convergence as in traditional adaptive methods and low memory usage as in existing memory-efficient methods. For instance, the converged validation perplexity of CAME and Adafactor is 50.1 and 56.9 respectively, which yields a considerable improvement of 12.0%.

heading: 4.5 T5 Training
text: Finally, we report empirical results from a different large language model training task: Text-toText Transfer Transformer, T5. Concretely, we follow the architecture of T5 (Raffel et al., 2022) and choose T5-Base (L = 24, H = 1024) with 220M parameters for the experiment. All of our implementations are also based on the code provided by Megatron. Similarly, we consider Wikipedia with 2.5B words as the training dataset in this part. As with the training of GPT-2 in Section 4.4, we only concentrate on standard training batch size (128) for T5.
0 20 40 60 80 100 120 Steps(K)
0
20
40
60
80
100
120
140
160
Va lid
at io
n Pe
rp le
xi ty
Adam Adafactor CAME
Figure 9: Validation perplexity of GPT-2 language model. CAME demonstrates comparable convergence speed with Adam.
The comparison of CAME with Adafactor and Adam is conducted in the same manner as Section 4.4, and corresponding results of validation loss and validation perplexity are illustrated in Figure 10 and Figure 11 seperately. Note that CAME consistently obtains comparable convergence performance for validation loss and validation perplexity on par with Adam, while reducing similar memory usage as Adafactor.

heading: 5 Conclusion
text: In this paper we propose a novel memory-efficient optimizer called CAME, which supports adaptive confidence-based updating guided by the residual between predicted update and generated update. CAME achieves a considerable improvement compared to existing memory-efficient optimizers in the training of large language models, with an ignorable extra memory footprint. Moreover, CAME shows comparable convergence to Adam and LAMB with huge memory reduction. In particular, CAME has proven effective for large-batch training, which serves as an advantageous extension to memory-efficient optimizers. We hope our work will provide insight into memory reduction
of optimizers in future exploration.

heading: 6 Limitations
text: Despite the success of our CAME optimizer in training large language models with memory efficiency, there are still some limitations that need to be addressed in the future.
Our proposed memory-efficient optimizer introduces additional computation costs for the nonnegative matrix factorization of the instability matrix in comparison with Adafactor. We observe, however, that the training time of CAME increases only slightly in our experiments. Beyond that, CAME exhibits minor performance degradation in large-batch training of the BERT-Large model versus LAMB, which allows for further improvement in the future. Meanwhile, it is possible to conduct further experiments on other models in other fields, such as Computer Vision and Reinforcement Learning, thereby exploring the effectiveness of CAME
training under more application scenarios. As a final point, it would be much more helpful to provide an in-depth theoretical analysis of CAME to improve comprehensiveness of the paper.

heading: Acknowledgements
text: Yang You’s research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant and Alibaba grant. We also thank Huawei Noah’s Ark Lab for providing the necessary computing resources and support for datasets.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
section 6.
3 A2. Did you discuss any potential risks of your work? section 6.
3 A3. Do the abstract and introduction summarize the paper’s main claims? section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4.
3 B1. Did you cite the creators of artifacts you used? section 4.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? section 4.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section 4.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? section 4.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? section 4.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 4.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Despite the success of our CAME optimizer in training large language models with memory efficiency, there are still some limitations that need to be addressed in the future. Our proposed memory-efficient optimizer introduces additional computation costs for the nonnegative matrix factorization of the instability matrix in comparison with Adafactor. We observe, however, that the training time of CAME increases only slightly in our experiments. Beyond that, CAME exhibits minor performance degradation in large-batch training of the BERT-Large model versus LAMB, which allows for further improvement in the future. Meanwhile, it is possible to conduct further experiments on other models in other fields, such as Computer Vision and Reinforcement Learning, thereby exploring the effectiveness of CAME
training under more application scenarios. As a final point, it would be much more helpful to provide an in-depth theoretical analysis of CAME to improve comprehensiveness of the paper