Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagation
Authors: Kai Huang
Hanyun Yin
Heng Huang
Wei Gao
Venue: ICLR 2024 poster (2024)
Keywords: Green AI
Large Language Models
Fine-Tuning
Adaptive Backpropagation
Primary area: generative models
TL;DR: This paper presents a new technique of minimizing the FLOPs of LLM fine-tuning with respect to the Green AI requirements, by selecting the best   trainable portions of the model based on their backpropagation costs.

Abstract: Fine-tuning is essential to adapting pre-trained large language models to downstream applications. With the increasing popularity of LLM-enabled applications, fine-tuning has been performed intensively worldwide, incurring a tremendous amount of computing costs that correspond to big carbon footprint and environmental impact. Mitigating such environmental impact directly correlates to reducing the fine-tuning FLOPs. Existing fine-tuning schemes focus on either saving memory or reducing the overhead of computing weight updates, but cannot achieve sufficient FLOPs reduction due to their ignorance of the training cost in backpropagation. To address this limitation, in this paper we present GreenTrainer, a new technique that minimizes the FLOPs of LLM fine-tuning via adaptive backpropagation, which adaptively selects the most appropriate set of LLM tensors for fine-tuning based on their importance and backpropagation cost in training. Experiment results show that GreenTrainer can save up to 64\% training FLOPs compared to full fine-tuning, without any noticeable accuracy loss. Compared to the existing schemes such as Prefix Tuning and LoRA, GreenTrainer can achieve up to 4\% improvement of model accuracy, with on-par FLOPs reduction.

Published as a conference paper at ICLR 2024
TOWARDS GREEN AI IN FINE-TUNING LARGE LAN- GUAGE MODELS VIA ADAPTIVE BACKPROPAGATION
Kai Huang?, Hanyun Yin Heng Huang & Wei Gao
University of Pittsburgh?, University of Maryland, College Park University of Science and Technology of China
k. w..hanggpitt... at.itt.eeee edu, ykisagoodbye@gmag 1. com, heng@umd edu, wei weigao@pite ptttitt... edu
ABSTRACT
Fine-tuning is essential to adapting pre-trained large language models down- stream applications. With the increasing popularity of LLM-enabled applications, fine-tuning has been performed intensively worldwide, incurring tremendous amount of computing costs that correspond to big carbon footprint and environ- mental impact. Mitigating such environmental impact directly correlates to re- ducing the fine-tuning FLOPs. Existing fine-tuning schemes focus on either sav- ing memory reducing the overhead of computing weight updates, but cannot achieve sufficient FLOPs reduction due to their ignorance of the training cost in backpropagation. To address this limitation, in this paper we present Green- Trainer, a new technique that minimizes the FL.OPs of LLM fine-tuning via adap- tive backpropagation, which adaptively selects the most appropriate set of LLM tensors for fine-tuning based on their importance and backpropagation cost training. Experiment results show that GreenTrainer can save up to 64% training FLOPs compared to full fine-tuning, without any noticeable accuracy loss. Com- pared to the existing schemes such as Prefix Tuning and LoRA, GreenTraing can achieve up to 4% improvement of model accuracy, with on-par FLOPs reduction.
INTRODUCTION
Large language models (LLMs) are used foundational tools in generative AI. To be used in down- stream applications, a pre-trained LLM needs to be fine-tuned using the specific application data (Devlin et al., 2018). Intuitively, fine-tuning is less computationally expensive than pre-training due to the smaller amount of training data, but it may result in significantly high energy consumption and carbon footprint when being intensively performed worldwide. Enabled by the democratization of open-sourced LLMs (Candel et al., 2023) and convenient APIs of operating these LLMs (Ott et al., 2019; Wolf et al., 2019), even non-expert individuals can easily fine-tune LLMs for model perfor- mance enhancement or personalization (Scialom et al., 2022: Wang and Gao, 2023). For example, when LLaMA-13B model (Touvron et al., 2023) is fine-tuned by 10k users using A100-80GB GPUs, such fine-tuning consumes 6.9x more GPU hours than pre-training a GPT-3 model (Brown et al., 2020) with 175B parameters. The amount of energy consumed by such fine-tuning is com- parable to that consumed by some underdeveloped countries, and the amount of carbon footprint equivalent to 1000x of that produced by New York-San Francisco flight (aii, 2023).
Mitigating such environmental impact towards Green AI directly correlates reducing the number of floating operations (FLOPs) of fine-tuning, which represents the amount of computational opera- tions and hence energy consumption in training (Schwartz et al., 2020; Huang et al., 2023a). Most existing techniques of optimizing LLM fine-tuning, however, are limited to reducing the memory consumption rather than FLOPs (Malladi al., 2023; Liao et al., 2023). Some other methods re- duce FLOPs by only fine-tuning certain types of model parameters such as bias (Zaken et al., 2021), LayerNorm and output layer weights (Lu et al., 2021), but they impair the model's expressivity and are only applicable to simple non- -eneeeerraaive learning tasks. Instead, researchers suggested keep- ing the original model parameters frozen but injecting additional trainable parameters to the input (Lester et al., 2021; Liu et al., 2022) or internal layers (Li and Liang. 2021; Hu et al., 2023; Huang et al., 2023b). Recent LoRA-
methods (Hu et al., 2021; Zhang et al., 2023) further reduce the overhead of computing weight updates for these injected parameters via low-rank approxima- tion. These methods can minimize the model's 's accuracy loss on generative tasks. However, they still need to compute the activation gradients through the whole model and their FLOPs reduction is hence limited, because the computations of weight updates are only 25%-33% of the total training FLOPs.

---

Published as a conference paper at ICLR 2024
Besides computing weight updates, FLOPs in training are also produced in i) forward propagation and ii) backward propagation of activation gradients. Since complete forward propagation is es- sential to calculate the training loss, we envision that the key to further FLOPs reduction is to take the backpropagation cost of activation gradients, which is >33% of the total training FLOPs, into account and selectively involve only the most appropriate model structures in backpropagation. The major challenge, however, is that selective training will possibly bring model accuracy loss. We minimize the accuracy loss is by adapting such selection in backpropagation to flexible objective of FLOPs reduction, determined by the carbon footprint in energy supply. For example, when such carbon footprint is low due to insertion of renewable energy, using a lower objective of FLOPs re- duction can involve more model structures in training and retain the training accuracy. High carbon footprint, instead, leads to higher objective of FI.OPs reduction for better embracing Green AI.
In this paper, we present GreenTrainer, new technique that realizes adaptive backpropagation for efficient LLM fine-tuning with the minimum accuracy loss. As shown
model params -
backprop
trained
frozen
in Figure 1, given an objective of FLOPs reduction, GreenTrainer adaptively selects the of trainable neu- ral network (NN) tensors in each epoch, based on eval- uation of tensors' importance in training. Such impor- tance evaluation is difficult because NN tensors do not di- rectly associate with input data variables or intermediate features, and most attribution techniques (Sundararajan et al., 2017; Hesse et al., 2021) that evaluate feature im- portance are not applicable. Popular importance metrics, including SNIP (Lee et al., 2018) and Fisher (Liu et al.,
epoch
epoch
epoch 3
Figure 1:
GreenTrainer adaptively se- lects the most appropriate portion of LLM model for fine-tuning
2021), are mainly used in NN pruning to quantify the importance of model weights their current values, but they cannot quantify the importance of weight updates on a tensor to reducing the train- ing loss. Classic metries based on exact accuracy contribution (Lin et al., 2022), weight updates' magnitudes (Li et al., 2016), or random perturbations (Breiman, 2001), on the other hand, are either inaccurate or computationally expensive for LLMs. Instead, our approach adopts a similar rationale with the existing attribution and pruning metrics, and quantifies the contribution of each tensor up- date to the training loss via first-order Taylor expansion over the training loss. In this way, we ensure that the selected tensors can make the maximum contribution to reducing the training loss.
Another challenge is how to precisely profile the training FLOPs. Due interdependency between tensors, their total FLOPs in training is not equal to the summation of their individual FLOPs. Such interdependency is determined by the backpropagation characteristies of NN operators in each ten- sor, but existing FLOPs models cannot link NN operators to tensors based on the computing flow of backpropagation. Some existing work (Kwon et al., 2023) only incorporates the layer-wise wise forward FLOPs into tensor selection, but ignores the computation dependency between layers in backpropa- gation. To tackle this challenge, we rigorously model the cross-tensol dependencies in profiling their backpropagation FLOPs. Based on this model, we develop a dynamic programming (DP) algorithm to find the nearly optimal tensor selection from an exponential number of possibilities (e.g., 2515 for 515 tensors OPT-2.7B 7B model (Zhang et al., 2022)). Therefore, GreenTraine can make sure that the given objective of FLOPs reduction can be met in most cases.
We evaluated GreenTraiinr with three open-sourced LLMs, namely OPT (Zhang al., 2022), BLOOMZ (Muennighoff et al., 2022) and FLAN-TS (Chung et al., 2022), on text generation datasets including SciTILDR (Cachola et al., 2020) and DialogSum (Chen al., 2021). Our results show that GreenTrainer can save up to 64% training FLOPPs compared to full LLM fine-tuning, without any no- ticeable accuracy loss. In some cases, Green can even improve the model accuracy compared to full fine-tuning, by removing model redundancy and overfitting. Compared to existing techniques such Prefix Tuning (Li and Liang, 2021) 1) and LoRA (Hu et al., 2021), GreenTrainee improves the model accuracy by 4% with the same amount of FLOPs reduction, and also provides users with the flexibility to balance between the training accuracy and cost depending on the needs of Green AI.
2 BACKGROUND & MOTIVATION
2.1 TRANSFORMER ARCHITECTURES FOR TEXT GENERATION
Current LLMs are stacked by transformer blocks (Vaswani et al., 2017), each containing Multi- Head Attention (MHA) layer, LayerNorms (Ba et al., 2016), and
Network (FFN).
2

---

Published as a conference paper at ICLR 2024
Given an input sequence X Ruxd with n tokens, the MHA projects tokens into a (Q, K, V) space h times, using h suites of trainable projectors (W w W Jim1,....hh Each projection St Raxd + R= A is defined as Q,, V = x xw performs attention mechanisms to produce O; by weighting V; with the attention scores between Q; and K, The MHA's final output is obtained by concatenating each O,, following linear projection g I nxd - Rnxd with a trainable projector wo
K xw The output (Qi,K, Vi) then
O Softmax (Q:K²' /vahh) Vi:
MHA = Concati(O,, O2,,., W.
To improve the training efficiency, LLMs adopt the teacher-foreing method (Lamb al., 2016) to generate the entire sequence of output tokens in a single forward pass. Specifically, causal masks applied to MHA's attention scores, so that each output token can be predicted from the label tokens at previous positions. With this technique, when being fine-tuned, LLMs can be trained in a standard way like any feed-forward models.
2.2 THE NEED FOR ADAPTIVE BACKPROPAGATION
When being fine-tuned for downstream task, LLMs are usually over- parameterized, because only part of the world knowledge that they learned from pre-training is useful for the cases, only involving some of the model's 's substructures into fine-tuning could have little impact on the model accuracy, but significantly reduces the amount of computations.
target task. In these
Trainable
OPT-2.7B
FLAN-TS-3B
substructure
FLOP (x10"
(%)
(8
Acc. (%)
All params
262.0
23.6
135.7
46.5
Last layers
18166 (319L)
20.8
46.1 (66%)
39.2
Decoder prefix
174.7 (339..)
13.4
55.3 (60%L)
37.6
(We. Wv)
174.7 (33%)L
23.8
90.3 (33%LL
44.7
Table 1: Fine-tuning different substructures of OPT-2.7B and FLAN-T5-3B LLMs on the Dialog- Sum dataset (ROUGE-1 score on the test set used as the accuracy metric)
Existing work has made attempts with fixed selections of some NN components, such the last layers, decoder prefixes (Li and Liang, 2021), and linear projectors (Wq. Wv) (Hu et al., 2021), 1), in fine-tuning. However, due to the interdependencies of NN parameters (Jin et al., 2020), such fixed selections will significantly impair the model accuracy. As shown in Table 1, solely fine-tuning either the last 2 layers decoder prefixes leads up 10% accuracy drop. The reason is that nearby NN substructures with interdependencies on the fixed selections are excluded from fine- tuning, and hence become inconsistent with those selected substructures. Increasing the density of selection, such as including all the linear projectors (Wo, Wy), could mitigate the model accuracy loss, but can save at most 33% FLOPs due to backpropagating activation gradients through trans- former blocks. Naive methods of dynamic selections, such as expanding the trainable portion from the last layer, have the similar limitation.
The deficiency of these existing methods motivates us enforce more flexible and adaptive selection of LLM substructures in backpropagation. In GreenTrainer, we develop tensor importance metric that incorporates parameter dependencies to evaluate how fine-tuning each tensor contributes the trained model's 's accuracy at runtime. Knowledge about such tensor importance, then, allows us to achieve the desired FLOPs reduction while maximizing the model accuracy.
Input data
Layer
Lanyer
y3
L
y4
Prodiction
dy2
dy'
dy
Label
forward
backward
dw3
dw2
dw3
dw4
Figure 2: Backpropagation of a 4-layer dense NN
2.3 FLOPS MODEL OF BACKPROPAGATION
The design of GreenTrainer relies on proper calculation of the selected model substructures' back- propagation FLOPs, which can be decomposed into two parts using the chain rule. For example, as
3

---

Published as a conference paper at ICLR 2024
shown in Figure 2. when training 4-layer dense NN without bias, each layer computes i) dy; as the loss I's gradient w.r.t the activation U/. and ii) dwj as the loss gradient w.r.t weight W,, such that
ØL yy,
aL
dw; aL
ÖL
dy: and the corresponding amounts of FLOPs for computing dy; and dw; are tdy, and Lud,, respectively.
W. dy:+1,,", dyc+1
Vi 0g.+1
Us dy
(2)
aw;
dw;) can be computed from (dy:+1, dw;+1). In particular, even if layer is not selected in fine-tuning, it still needs to compute and pass error gradients (dy;) to the downstream layers. Hence, the amount of computations in backpropagation does not only depend on the selected layers, but also depends on some unselected layers. For example, only Layer 2 is trainable, the total FLOPs for backpropagation will be decided by the cost of computing dw2, dy3 and dy Due to the generality of the chain rule, such rationale of FLOPs calculation is also applicable to other types NN layers.
Based on this rationale, we can construct FLOPs models for LLM substructures. The layer-level model is caase-grained and can lead to inaccurate tensor selection. Some important parameters may be unselected due to other unimportant ones in the same layer. In GreenTrainet we use tensor- level granularity for such selection, which can be well-supported by tensorized NN libraries (e.g., TensorFlow (Abadi, 2016) and Py PyTorch (Paszke et al., 2019)). Weight-level selection, although more fine-grained, is too computationally expensive due the requirement of fine- tne-ggained indexiig.
3 METHOD
To reduce the FLOPs of LLM fine-tuning, intuitive problem formulation is minimize the FLOPs while achieving the desired model accuracy. However, it is hard to determine a proper accu- racy objective in advance, because some accuracy objectives may require very intensive training and the accuracy that we can achieve with our FLOPss budget cannot be pre-estimated before training. Instead, we maximize the training loss reduction while achieving the desired FLOPs reduction:
max (m)) S.t. Tselective S pT
where is binary vector be solved for tensor selection. m parameterizes both the loss reduction (Atore) and per-batch FLOPs of training (Trelective). and Teciective is constrained within a user- specified ratio (p) of the FLOPs of fine-tuning the whole model (Tfuu). For example, P 0.5 means that the FLOPs of fine-tuning should be at most 50% of that in fine-tuning the whole model. In practice, the value of can either be preset or adjusted at runtime in any stage of training.
To identify each tensor's contribution in fine-tuning, we model Atoss as the aggregated impor- tance of selected tensors, and calculate the FLOPs incurred by selected tensors using the FLOPs model of backpropagation in Section 2.3. With this model, Eq. (3) can be rewritten as:
max Aloss(m) s.t. Tfp + m tww + +mm) tay S &pl'ruuts
(4)
where Tjp indicates the per-batch FLOPs of the forward pass, and each pair of variables in (Lay, Law) represents the FLOPS of computing dw) for the corresponding tensor, respectively. Given binary selector m. (m) incorporates all the tensors along the backward pass that contribute to the FLOPS of fine-tuning, by involving in passing the error gradients (dy). For example,
m (0, 0, 1, 0, 1, 0, 0), all the tensors that are in deeper layers than the selected tensors are involved in passing the error gradients, and hence a (m)) = [0, 0, 1,
To ground this formulation and solve m. GreenTrainer consists of three key components: (i) Tensor FLOPs Profiling, which calculates the FLOPs of all NN tensors (i.e., tay and kxw) prior to training; (ii) Tensor Importance Evaluation, which quantifies the contribution of updating each NN tensor to the training quality runtime; (iii) Tensor Selector, which grounds the tensor selection problem using tensors' FLOPs and importances, and provides solutions via dynamic programming at runtime.
3.1 TENSOR FLOPS PROFILING
Standard NN profilers, such as Torch Profiler (Paszke et al., 2019), measure the execution FLOPs of individual NN operators such as matrix multiplication and convolution. However, it cannot be directly linked to NN tensors that participate in these operations. When set of tensors is trained, the training FLOPs of backpropagation are not equal to the summation of individual tensors' FLOPs.
To address this limitation, our approach consists of two steps. First, we convert the layer-based NN structure of LLMs into a lensor-level computing graph, which retains the execution order of all

---

Published as a conference paper at ICLR 2024
Backprop direction
Tensors in MHA layer 1 block
Teneariavel
Graph:
Input Embed. Tensor
Projecter
Bias
Projector K
Bias
Projector
Output Embed. Tensor
Related Backprop Ops!
variable assign,
motmul natmol_ 2
add.
matmmll matmrl
add
matril matrul 6
matnul, B4-3) matrul, (1)
Motch & Agp-
FLOPs
Tensor
(eey
(0,3)
Ta
D2
Figure 3: An sample workflow of tensor FLOPs profiling
tensors' involvements in training. Then, we extract the related backpropagation operators of each tensor, and derive each tensor i's FLOPs in backpropagation (tdy, and tdwe,) by matching and aggre- gating the FLOPs of these NN operators. For example Figure 3, the training of each linear projec- tor (Q, K and V) in an MHA layer should be executed after its corresponding bias tensor's 's training. Training each linear projector, then, will involve two matrix multiplication operators, whose FLOPs in backpropagation will be aggregated. We categorize such rules of matching and aggregation by the type of LLM layers where tensors located, as described below. A specific example about such tensor FLOPs profiling on the OPT-2.7B model is provided in Appendix A.3.
Input & output embedding layers. The input embedding layer contains a trainable embedding tensor that maps each raw token into a dense representation. Given the activation gradient dy from upstream layers, deriving the update of this tensor only involves variable assignment, and we can safely consider (dw; & 0 for any tensor i. If a raw token is mapped to the k-th vector in the embedding tensor during the forward pass, then during backpropagation, dyi+ I from the upstream will be only assigned to k-th row of dwi, such that dw:(s) = dyi-1 1 if s k. otherwise dwi = O. Since the input layer doesn't 't propagate activation gradients, we can also conclude that its lay is O.
Reversely, the output embedding layer projects each token back to the probability space, Intuitively, its (tay. ) can be derived in the same way as we did for the dense layer in Eq. (2). However, in most LLMs, the output embedding layer shares the same trainable tensor with the input embedding layer. This implies that if the output embedding is trainable, then the input embedding will also be involved in training. Hence, all the byy from LLM's output, up to the input embedding layer, should be accumulated to tdy of the output embedding tensor, while its fdw remains unchanged.
Multi-Head Attention (MHA) layer. An MHA layer contains multiple linear projectors as trainable tensors, and their FLOPs in training can be derived in the same way as we did with the dense layer in Eq. (2). Some LLMs (e.g., OPT) also include bias as another type of trainable tensor after such projection. In this case, based on the chain rule, the backpropagation of bias is computed as dy; dyi+1 and dw; = 1 [ddi++, indicating that ldg for bias is O since dy; is identically passed from dy1+1- Idwr of bias can be derived as the FLOPSs of adding up elements in dy:+1 along every feature channel. The attention mechanism in Eq. (1) is backpropagated prior to the projectors. If any of these projectors are involved in training, the attention's 's backpropagation FLOPs must be also calculated, and we accumulate such FLOPs to the corresponding projector tensor ('y''s tdy.
LayerNorm. Given a token, LayerNorm first normalizes its features and uses two trainable tensors and 3 to element-wise multiply with and add to the token, respectively. The operations of multi- plication and addition are similar those in the dense layer, and its FLOPs can be calculated in the similar way. However, the backpropagation FLOPs of normalization operators should be accu- mulated to the previous tensor's tay- If any tensors in the previous layers trained, the FLOPs of propagating the normalization operators should be also included in the FLOPs of the current layer.
Feed-Forward Network (FFN). In the FFN, there is a nonlinear activation function between two dense layers. Following the same method of calculating LayerNorm's 's FLOPs, we accumulate the FLOPS of propagating through this activation function to the bias tensor's tdy in the first dense layer.
3.2 TENSOR IMPORTANCE EVALUATION
A tensor's importance in training can be estimated as the summation of the importances of all its weights. In training, since the model weights are iteratively updated to minimize the training loss, an intuitive approach to evaluating the importance of a weight update in given iteration is to undo this update and check how the training loss increases back as AL = L(w) L(w + Aw), so that a higher value of AL means this update is more important and the weight should be selected. However, computing AL for every weight is expensive. Instead, we estimate the importance of all weights in one shot by smoothing the undo operation described above and computing the loss gradients with respect to the updates that correspond to all the weights. Letting the multiplicative
5

---

Published as a conference paper at ICLR 2024
E [0, 1| denote the undo operation for all the M weights, we can compute the loss gradient @L ur + c Aw) Aw @
@L(u)
(5)
dc
du u==+
where denotes element-wise multiplication. When 0, Eq. (5) becomes importance vector over all weights. Since the loss gradient is parameterized by all weights, the weight importances calculated in this way implicitly incorporate the impact of weight dependencies. A tensor k's im- portance is then calculated as
4. 4=-E, Au: OL/Ow!
(6)
In some cases, when the training process encounters divergence, the values of gradients and caleu- lated tensor importances in Eq. (6) could be very large, eventually leading to overflow when using these importance values for deciding tensor selection in Eq. (4). To address this issue, we could further scale all the tensor importance by the maximum amplitude to improve numerical stability.
Top
selected
PBoLd
Subproblem (N Tip)
t[
Pik.
backprop ELOPs stt
snlected
P0-1,
k-
backgrop depth 5
notselacted
relected
P0k
(a) Subproblem definition
(h) Finding recurrence relations
Figure 4: Solving the tensor selection problem using DP
3.3 TENSOR SELECTION
Since Eq. (4) is nonlinear integer programming problem and hence NP-hard, in GreenTrainer we seek for an approximate solution using dynamic programming (DP). We decompose the whole problem into subproblems constrained by different depths of backpropagation. These subproblems can be sequentially solved from the one with the smallest depth, by using their recurrence relations.
Subproblem definition. As shown in Figure 4(a), we define each subproblem P(k. to maximize the cumulative importance of selected tensors when 1) selection is among the top le tensors' and 2) backpropagation FLOPs is at most DP starts by solving the smallest subproblem Pík = 1,
1] and gradually solves larger subproblems based on the results of smaller subproblems and the recurrence relation of these subproblems, until the target problem P[N, T [un]] is solved.
Recurrence relations of subproblems. The recurrence relation between subproblem Plk, 4) and Pík t, depends whether further select the top tensor & from the solution of Plk 1, t]. as shown in Figure 4(b). Case 1: is not selected, Plk, t] will fall back to Plk 1, t). since the importance of selected tensors will not be further increased. Case 2: If k is selected, then its FLOPs will be included into the solution of P[k t), no matter which other tensors are selected. The FLOPs involved with tensor include 1) the FLOPs to update tensor & and 2) the FLOPs pass activation gradients from the closest selected tensor ke. such as tensor - as shown in Figure 4(b), to tensor k. This implies that Plk, t] falls back to previously solved subproblem P(k- k, t At), where
Al Laws, +
j=k Ldy:
Since Ki unknown in advance, backtrace the previously solved subproblems and explore all the possibilities of ki by reducing the depth of backpropagation from kt and the optimal solution to P(k, 4] is the one with the highest cumulative importance of selected tensors. Based rence relation, we can solve all subproblems by traversing the subproblem space. The time com- plexity of solving each subproblem O(N), and the overall time complexity of DP is O(N?T)wu))
this recur-
4 EXPERIMENTS
In our evaluation. include decoder-only LLMs including OPT (Zhang et al., 2022) and BLOOMZ (Muennighoff al., 2022), and an encoder-decoder I.M,, namely FLAN-TS (Chung et al., 2022),
We consider tensor that is closest to the output as the topmost.

---

Published as a conference paper at ICLR 2024
with LLM sizes ranging from 350M to 6.7B. Our experiments are mainly conducted using the fol- lowing two datasets of abstractive summarization:
SciTLDR (Cachola et al., 2020) is dataset of 5.4K text summaries on 3.2K papers. It contains both author- and expert-derived TLDRs, where the latter is collected by an annotation protocol that produces high-quality summaries with low annotation burden.
DialogSum (Chen et al., 2021) is a dialogue summarization dataset of 13,460 dialogues with manually labeled summaries and topics. It has been demonstrated more challenging than other summarization datasets, such as SAMSum (Gliwa et al., 2019) and CNN/Daily (Nallapati et al., 2016) at similar scale.
We also perform generative QA tasks on WebQuestion (Berant et al., 2013) and PIQA (Bisk et al., 2020) datasets in Appendix A.4. However, we do not consider non-generative tasks such as senti- mental classification, entailment classification and extractive QA, because these tasks are too easy for LLMs and testing them with LLMs will result in exaggerated performance gain over the baseline.
For OPT and BLOOMZ, we follow GPT2-like prompt structures (Radford et al., 2019), "[source seq.] TL;DR:", for summarization tasks to preprocess input data. For FLAN-T5, we adopt the prompt structure "summarize: [source seq.]" used in the original T5 pre-tra pre-training We truncate the source sequences so that the length of every preprocessed input sequence is within 512 tokens. On the test data, we use beam search size of 4, and set the maximum number of generated tokens to 64 for SciTI.DR and 128 for DialogSum. We compare Green' StreenT'ranee Trainer (GT) with the following baselines:
Full
(Full FT) fine-tunes all the LLM parameters and should intuitively achieve the best accuracy of the trained model.
Fine-Tuning Top2 (FT-Top2) only fine-tunes the last two layers, typically the embedding layer and a LayerNorm. The input and output embedding layers are tied for OPT and BLOOMZ, but are not tied for FLAN-TS. This naive baseline only fine-tunes the smallest portion of [.LM parameters and is used to identify whether the dataset is trivial to the LLM. Prefix Tuning (Prefix-T) (Li and Liang, 2021) inserts trainable prefixes into each trans- former block's ' input sequence while freezing the model parameters. For encoder-decoder LLMs, the trainable prefixes are only inserted into the decoder blocks.
LoRA (Hu et al., 2021) is currently the most popular method for efficient LLM fine-tuning. It uses low-rank matrix decomposition to reduce the training cost. We apply LoRA to both query and value projectors, as suggested in (Hu et al., 2021).
In all experiments, we use a batch size of 4 and fine-tune the model for epochs. We use the AdamW optimizer (Loshchilov and Hutter, 2017) at learning rate of 2 x 10-6 with linear schedule and weight decay of 10-2, We use the ROUGE scores 1/22/LL (Lin, 2004) as the accuracy metric, and measure both Peta-FLOPs (PFLOPs) and wall-clock time as the training cost in each run. We measure the end-to-ent cost of training, including the computing costs in forward and backward passes, and the computing costs of tensor importance evaluation and tensor selection using DP.
4.1 TRAINING COST & ACCURACY
We first evaluate the training cost and accuracy of GreenTrainer (GT). As shown in Table 2, for the OPT-2.7B model, GT-0.5 can achieve the required 50% of FL.PPs reduction with most 2% accuracy loss, and GT-0.7 can even achieve 0.2%-3% higher ROUGE scores than Full FT. We hy- pothesize that this is because GT only fine-tunes the most important tensors and hence mitigates the possible overfitting in Full FT. Insufficient trainable parameters can also lead to underfitting, as FT-Top2 has significantly lower ROUGE scores. Similarly, compared to LoRA and Prefix Tuning, GT-0.7 achieves at least 2% higher accuracy with the same amount of training FLOPs.
Similarly, for BLOOMZ-3B, GT-0.5 can save 50% training FLOPs and wall-clock time with < 2% accuracy loss. Compared to Full FT, GT-0.7 achieves the same ROUGE scores on SciTLDR, and 4%-10% higher on DialogSum. With the same amount of training FLOPs, GT-0.7 has 0.4%-1.49% higher ROUGE scores than LoRA. Note that both datasets are non-trivial for the BLOOMZ model, since the naive baseline (FT-Top2) still exhibits high accuracy loss.
For the FLAN-T5-3B model, FT-Top2 achieves similar fine-tuning qualities Full FT with lower FLOPs, indicating that the SciTLDR dataset is trivial for FLAN-TS. In this case, GT-0.34 can achieve the same FLOPs and ROUGE scores by selecting small portion of tensors. On the other hand, FT-Top2 loses accuracy significantly on Sum, but GT-0.4 reduces 54% of training FLOPs and 43% of wall-clock time without noticeable accuracy loss. GT-0.4 also outperforms LoRA by 1% on ROUGE scores and reduces 11% more FLOPs. Compared to Prefix tuning, GT- 0.34 achieves 2%-5% higher ROUGE scores, while reducing the same amount of training FLOPs.
7

---

Published as a conference paper at ICLR 2024
# Model & Method
SciTLDR
DialogSum
PFLOPs
Time
R1/R2/RL
PFLOPs
Time (h)
R1/R2/RL
OPT-27B
Full FT
41.8
0.92
32.9/14,9/27.1
262.0
5.5
23.6/9.5/18.8
FTT
(319
0.61 (349LL)
9.1/4.0/7.6
181.6 (3196.))))))
3.8 (31%
20.8/7.9/17.5
Prefix-T
27.9 (339%)))
0.58 (3796L)
7.6/0.4/6.1
174.7 (339%.)
3.7 (33%-)
13.4/3.3/10.
LoRA
(33%8))))
0.59 (3698L)
28.2/12.1/21.0
174.7 (3318.)
3.6
23.8/9.5/18.8
GT-0.5
20.8 (50%.)
0.46 (509LL)
30.5/13.1/25.2
130.1 (50%-J)
2.7 (51%))
21.478.2/17.6
GT-0.7
29.2 (30%L)
0.68 (269-L)
33.1/15.2/2.6
182.7 (30%
4.0 (27%))
1.021.
BLOOMZ-38
Full FT
47.2
1.0
28.3/12.1/22.5
294.8
26.1/10.6/21.0
FT-
36.5 (23%%))
0.75 (2558L)
23.7/8.8/18.8
227.9 (23%)
4.6 (29%))
22.1/8.5/
Prefix
(338.))
0.68 (349.L)
6.5/2.2/5,
196.5 (339.
4.2 (35%.)
29 29.6/9 4/24,
LoRA
(339
0.69 (3398L)
27
196.5 (33%)4
(34%%))
35.4/14.3/2.6
GT.0.5
(518))))))
0.51 (50RLL)
26.7/10.7/21.2
146.4 (50%.)
3.1 (52%%
24.9/9.5/20.0
GT-007
32.3 (329%)
0.74 (28%kL)
28.0/12.2/22.4
204.7 (31%
4.3 (34%))
36.8/14.7/29.4
FLAN-T5-3B
Full FT
21.7
0.64
17.1/18.5/311.77
135.7
4.0
46.5/20.8/38.5
FT-Top2
(66%)))
0.21 (67%)-)
36.5/18.4//1..5
46.1 (66%)))
1.4 (65%%)
39.2/16.7/32..9
Prefix-T
(63%))
0.23 (64%)L)
16.0/18.2/31.0
55.3 (60%-)
1.7 (57%g)
4//3//1.
LoRA
(33%
0.41 (3658-)
36.6/18.5/31.5
90.5 (33%).)
(38%))
44.7/19.8/37.1
GT-0.34
(65%..))
0.23 (6498L)
36.4/18.4/1.7
53.5 (619%.)
1.4 (6518.))
42.7/18.3/55.
GT-0.4
(549%)))))
0.38 (4156-)
718./18.5/31.
62.5 (54%))
2.3 (43%%)
46.0//007/38.1
GT-0.5
(439.)))
0.44 (318.L)
36.3/17.7/3099
(43%%.))
266 (35%44)
46.2/20.7/38.1
Table 2: Comparison of the training cost & accuracy in LLM fine-tuning. GreenTrainer with an objective p of FLOPs reduction is denoted as GT-p.
4.2 THE IMPACT OF FLOPS REDUCTION OBJECTIVE
To better understand how Green'rranee performs with different objectives of FLOPs reduction, we vary the value of p between 0.36 and 0.8, and compare GreenTrainer with LoRA on the OPT-2.7B model. As shown in Table 3, on the SciTLDR dataset, when the requirement of FLOPs reduction high and corresponds to a value of $0.4, GreenTrainer outperforms LoRA by achieving 2% higher ROUGE scores and saving 25% more FLOPs and wall-clock time. On the other hand, when the value of o increases 0.6, GreenTrainer outperforms the Full FT on ROUGE scores by 0.5% and outperforms LoRA by 5.2%, but saves 40% of training FLOPs and 39% of wall-clock time compared to Full FT. Similar results are also observed the DialogSum dataset. In summary, with different objectives of FLOOPs reduction, GreenTraine can always provide better tradeoffs between the training accuracy and cost, compared to the SOTA baselines.
Method
ScTLLRR
DialogSum
PFLOPs
Time (h)
RI/R2/RI
PFLOPs
Time (h)
R1/R2/RL
Full FT
41.8
0.92
32/9/14.9/27.1
262.0
5.5
23.6995/118.
LoRA
27.9 (33%-}
0.59 (368.L)
28.2/12.1/11.0
174.7 (3396-
3.6 (35%.L
23.8/9.5/18.8
GT-0.36
14.9 (64%))
0.32 (65%L]
4.1/1.7/3.6
92.9 (65%)-1
1.9 (65%.)
15.7/5.0V13.8
0.
16.6 (60%.]
0.36 (619.))
28.6/11.6/33.5
103.4 (6199))
2.2 (60%-)
17.9/6.3/15.
GT-00.5
20.8 (50%-L)
0.46 (50S-L)
30.5/13.1/25.2
130.1 (50%-1)
2.7 (515%L
21.4/8.2/17.6
GT-0.6
25.0 (40%
0,56 (39%
33,4/15.3/27.8
156.6 (40%)
(40%%)
24.099.719..2
GT-0.7
29.2 (30%)L)
0.68 (268-
33. 2/17.
182.7 (30%-1)
(27%%)
26.8/1 1.022 1.6
GT-0.8
33.4 (20%)))
0.77 (16%L)
33.1/15.5/27.6
209.6 (20%)
(20%)
23.9/9.9/19.1
Table 3: Impact of different objectives of FLOPS reduction the OPT-2.7B model
These results also demonstrate that GreenTrainer provides great flexibility in LLM fine-tuning be- tween the training accuracy and cost, by adjusting the value of p. The user can opt of (50.4) to maximize the FLOPS reduction (>60%) with moderate model accuracy loss (3%-4% on the two datasets use). Alternatively, they use high value of p (>0.6) to have the same level of FL.OPs reduction as that of LoRA, but ensure the minimum model accuracy loss or even minor model accuracy improvement. We believe that such flexibility is practically important when fine-tuning LLMs for downstream tasks with different green AI requirements and constraints.
low value
4.3 EFFICACY OF TENSOR IMPORTANCE METRICS
The fine-tuning quality of GreenTrainer builds on proper evaluation of tensor importance. We com- pare our metric (Aw dL to the magnitude-basee based metric (Aw) (Lee et al., 2020) and the gradients- only metric ( (Aji and Heafield, 2017), using the OPT-2.7B model with =0.7. As shown
8

---

Published as a conference paper at ICLR 2024
SciTLDR
DialogSum
Method
PFLOPs
Time (h)
R1/R2/RI
PFLOPss
Time (h)
R1/R2/RL
Full FT
41.8
0.92
32.9/14
262.0
5.5
23.6/95/11888
GT-0.7 (Aw)
29.4 (30%)
0.68 (26% 4)
32.7/15.2/27..2
183.8 (30%L)
(27%.)
24.9/10.2/19.7
GT-0.7 ( 3w
29.4 (30%L)
0.67 (27%
32.8/15.1/27.22
184.0 (30%U)
4.0 (27%-)
25.0/10.2/20.0
GT-0.7 (Aw 0L
29.2 (30%L)
0.68 (26%.)
33.1/15.2/27.6
182.7 (30%L)
4.0 (27%)]
26.8/11.022666
Table 4: Efficacy of Tensor Importance Metries (OPT-2.7B)
in Table 4, with the same objective of FLOPS reduction, using metric (Aw (L for tensor im- portance evaluation achieves the highest model accuracy and outperforms Full FT by 1%-3% on ROUGE scores. This is because maagittude-bsee metrics ignore the dependencies of weight up- dates. Gradient-only metrics only contain the direction information about tensor importance but cannot reflect the intensity of importance. Inaccurate importance measurements will in turn lead to inappropriate selections of trainable tensors.
4.4 IMPACT OF LLM SIZE
A type of LLM may contain several variants with different sizes. To study GreenTrainert perfor- mance with different LLM sizes, performed fine-tuning using the OPT models with sizes ranging from 350M to 6.7B. shown in Table 5, even on small models (OPT-350M), GT-0.5 can save 17%- 21% more training FLOPs than LoRA does, while achieving 2%-4% higher accuracy (on SciTDR) or the same accuracy (on DialogSum). When the model size increases to 2.7B, GT-0.5 outperforms LoRA and GT-0.7 outperforms Full FT on the SciTLDR dataset. On DialogSum, GT-0.7 performs similarly compared to LoRA. For the BB model GT-0.4 can save 27% more training FLOPs than LoRA does on SciTLDR, while achieving the same model accuracy, and similar advantages also be observed when comparing GT-0.5 and GT-0.7 with LoRA. Generally speaking, Green- Trainer's performance advantage widely applies to LLMS with different sizes.
Params & Method
SUTTDDR
DialogSum
R1/R2/R
Time R1/R2/RL
OPT-350M
Fall FT
5.4
0.15
30.9/13.9/2577
33.8
0.92
23.2/9.0/18.5
LORA
(33%-)))
0.10 (33%LL
25 25.9/1
22.5 (33%LL
0.65 (29%-L)
21.5/7 7/1
GT-04
2.1 (619%)
0.06 (60%kL
27.7/12.2/3.44
13.3 (61%%)
0.36 (61%L
17.3/5. 8/14.6
GT-05
2.7 (50%-L)
0.08 (475kL)
29.9/13.2/24.9
16.7 (51%
0.45 (51%L)
21.3/7 8/17.3
GT-0-7
(30%
(20%)
30.6/13.5/25.0
23.6
24.2/9.3/19.3
OPT-L3B
Full FT
20.8 13.9 (33%.L)
0.46
12.1/1433/2664
130.8
25.4/10.//20.2
LoRA
0.31 (339:L)
28. 1/199//222
87.2 (33%.L
1.9 (348.L
24.6/9 9/19.4
GT-044
8.2 (619gL
0.18 (61%))
28.9/11.9/23.8
(61%
1.1 (62%L)
16.9/577/14.6
GT.05
10.3 (50S.L
0.23 (509:L)
30.0/12.7/24.5
64.2 (51%%L
(518))))
2/16.7.4/1.77
GT-077
14.5 (30%L)
0.34 (26%kL
31.2/14.2/25..
90.8 (30%.)
(.1(3)9)
24.99
OPT-2.7B
Full FT
41.8
0.92
32.9/14.9/27.1
262.0
5,5
23.6/9.5/18
LoRA
27.9 (33%
0.59 (36%L)
28. 1/.2//2.1/21.0
174.7 (33%L1
3.6 (35%)L
5/1888
GT-044
16.6 (60%
0.36 (61%L)
28.233
103.4 (61% (11%)
2.2 (60%g)
3/15,4
GP455
20.8 (50%L)
0.46 (50SLL)
30.5/13.1/25.2
130.1 (50%
2.7 (51%L)
21176..//1..6
GT-077
29.2(309.
0.68 (26%.L)
33.1//5.2227.6
182.7 (30%.)
4.0 (279.L
26.8/11.0/216
OPT-6.7B
Full FT
103.9
5.44
92.9/14.9/27.5
649.9
LoRA
(33%g))
28.4/12.3/22.7
433.3 (33%
24
GT-04
41.2 (60%
0.9
28.9/11. 1/233.4
257.9 (60%
5,2
19.1/7.0/16.
GT-55
(519%))
0/2.4//
(49%
2118/8.5/17.3
GT-077
(28%))))))
1.4
1/.533333/77.7
Table 5: Impact of LLM's s model size
5 CONCLUSION
In this paper, we present Treenraiine,, new technique for LLM fine-tuning that allows efficient selection of trainable parameters via adaptive backpropagation, to ensure high training quality while minimizing the computation cost. GreenTrainer saves up to 64% training FLOPs compared to full fine-tuning without noticeable accuracy loss. Compared to the existing technique such as Prefix Tuning and LoRA, Trainer improves the accuracy by up 4% with the same FLOPSs reduction.
For the OPT-6.7B, Full FT GT-0.7 with SumlogSum have the out-of -ol-oooomooyyyy issue on GPUs we use.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers and area chair for their comments and feedback. This work was supported in part by National Science Foundation (NSF) under grant number IIS- 2205360, CCF-2217003 and CCF-2215042.
REFERENCES
2023 Al index report. mtdasttoorro.....ttottttooooro. dtttaaffordeddu/reppppeeeee
hnx
2023.
M. Abadi. Tensorflow: learning functions at scale. Proceedings of the 21st ACM SIGPLAN International Conference Functional Programming, pages 101.
A. F. Aji and K. Heafield. Sparse communication for distributed gradient descent. arXiv preprint 1rXiv:1704.0502,, 2017.
L. Ba, J. Kiros, and Hinton. Layer normalization. arXiv preprint vrXiv:1607.064533 2016.
Berant, Chou, R. Frostig, and Liang. Semantic parsing on freebase from question- answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language process- ing. pages 1533 1544-1544. 2013.
Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
L. Breiman. Random forests. Machine learning, 45:5-32, 2001.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural informa- tion processing systems, 33:1877-1901, 2020.
I. Cachola, K. Lo, A. Cohan, and D. S. Weld. Tidr: Extreme summarization of scientific documents. arXiv preprint arXiv:2004. /501 2020.
A. Candel, J. McKinney, P. Singer, P. Pfeiffer, M. Jeblick, P. Prabhu, J. Gambera, M. Landry, S. Bansal, R. Chesler, et al. h2ogpt: Democratizing large language models. arXiv preprint arXiv:2306.08161 2023.
Y. Chen, Y. Liu, L. Chen, and Y. Zhang. Dialogsum: real-life scenario dialogue summarization dataset. arXiv preprint arXiv:2105.06766 2021.
H. W. Chung, L. Hou, Longpre, Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. De- hghani, S. Brahma, et al. Scaling instruction-finetueed language models. arXiv preprint arXiv:2210. 11416, 2022.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprini arXiv:1810.04800 2018.
B. Gliwa, I. Mochol, M. Biesek, and A. Wawer. Samsum corpus: human- annottted dialogue dataset for abstractive summarization. arXiv preprint arXiv: axXiv:1911.122:5 2019.
R. Hesse, S. Schaub-Meyer, and S. Roth. Fast axiomatic attribution for neural networks. Advances in Neural Information Processing Systems, 34:19513-11524, 2021.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 106.::210609966 2021.
Z. Hu, Lan, L. Wang, W. Xu, E.PP. Lim, R. K.W. Lee, L. Bing, and S. Poria. LIm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304. 0/933, 2023.
10

---

Published as a conference paper at ICLR 2024
K. Huang, B. Yang, and W. Gao. Elastictrainer: Speeding up on-device training with runtime clastic tensor selection. In Proceedings of the s/ Anmual International Conference on Mobile Systems, Applications and Services, pages 36-69, 2023a.
K. Huang, B. Yang, and W. Gao. Modality plug-and-pg -lay:: Elastic modality adaptation in multimodal Ilms for embodied ai. arXiv preprint arXiv:2312.07886 2023b.
G. Jin, X. Yi, L. Zhang, L. Zhang, S. Schewe, and X. Huang. How does weight correlation affect generalisation ability of deep neural networks? Advances in Neural Information Processing Systems, 33:21 1466-1111,,, 2020.
Y. Kwon, R. Li, S. I. Venieris, J. Chauhan, N. D. Lane, and C. Mascolo. Tinytrain: Deep neural network training at the extreme edge. arXiv preprint vrXiv:2307009988 2023.
A. M. Lamb, A. G. ALIAS PARTH GOYAL, Y. Zhang, S. Zhang, A. C. Courville, and Y. Ben- gio. Professor forcing: A new algorithm for training recurrent networks. Advances in neural information processing systems, 29, 2016.
J. Lee, S. Park, S. Mo, S. Ahn, and J. Shin. adyer-ve sparsity for the magnitude-bass -aaiitudee-eeedddddee pruning. arXiv preprint arXiv:2010.07611 2020.
N. Lee, T. Ajanthan, and P. H. Torr. Snip: Single-shot network pruning based on connection sensi- tivity. arXiv preprint arXiv: /810.02300 2018.
B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0862) 2021.
H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. arXiv preprint 1608.vv.08..86 2016.
X. I.. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00196, 2021.
B. Liao, S. Tan, and C. Monz. Make your pre-trained model reversible: From parameter to memory efficient fine-tuning. arXiv preprint arXiv:2306.00477 2023.
C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Lin- guistics. URL ht https:/ / /ww.. acl www.wwb...... org/ anthooooghe /w04-101..
J. Lin, L. Zhu, W.-M. Chen, W.-C Wang, C. Gan, and S. Han. On-device training under 256kb memory. Advances in Neural Information Processing Systems, 35:22941-22954 2022.
L. Liu, S. Zhang, Z. Kuang, A. Zhou, J.-H. Xue, X. Wang, Y. Chen, W. Yang. Q. Liao, and W. Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learming, pages 7021-7032. PMLR, 2021.
X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang, P-tuning: Prompt tuning be com- parable to fine-tuning across scales and tasks. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61-68, 2022.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv:1711.0510.. 2017.
arXiv preprint
K. Lu, A. Grover, P. Abbeel, and Mordatch. Pretrained transformers as universal computation engines. arXiv preprint 1:Xiv:2103.0524: 1, 2021.
S. Malladi, T. Gao, F. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora. Fine-tuning language models with just forward passes. arXiv preprint 2tt.:23055/73333 2023.
N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z.-X. Yong, H. Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint ar 2 1. 1.01766, 2022.
R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang, et al. Abstractive text summarization using sequence- to-sequence mns and beyond. arXiv preprint arXiv:1602.06023, 2016.
11

---

Published as a conference paper at ICLR 2024
M. Out, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng. D. Grangier, and M. Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprins 10Xiv:/904.00000 2019.
A. Paszke, S. Gross, Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, I.. Antiga, al. Pytorch: An imperative style, pigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAl blog, 1(8)99 2019.
R. Schwartz, J. Dodge, N. Smith, and O. Etzioni. Green ai. Communications of the ACM, 63 (12):54-63, 2020.
T. Scialom, T. Chakrabarty, and S. Muresan. Fine-tuned language models are continual learners. In Proceedings the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107-6122, 2022.
M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319-3328. PMLR, 2017.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, Jones, A. N. Gomez, L. Kaiser, and Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
H. Wang and W. Gao. Tackling the unlimited staleness in federated learning with intertwined data and device heterogeneities. arXiv preprint 133.56 2023.
Wolf, L. Debut, V. Sanh, Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Fun- towicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprint
2019.
E. B. Zaken, Ravfogel, and Y. Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106. 10X119900..10199 2021.
Q. Zhang, M. Chen, A. Bukharin, P. He. Y. Cheng. W. Chen, and T. Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint vrXiv:2303.10512 2023.
Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li. X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 2022.
12

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1 REDUCING THE MEMORY USAGE OF TENSOR IMPORTANCE EVALUATION
Our approach to evaluating the importance of NN tensors in Section 3.2 requires caching all the previous model weights and the current gradients, in order to compute Eq. (6). However, doing so significantly increases the GPU memory consumption, especially for modern LLMs with billions of model weights. To reduce such GPU memory usage, we observe that our problem formulation in Eq. (4) will prevent tensors in early layers to be selected for training, due the high costs of propagating their activation gradients in backpropagation. Hence, we could safely exclude these tensors from the trainable portion of L.LM fine-tuning and save significant amount of GPU memory. More specifically, the backpropagation during tensor importance evaluation can be early stopped at certain tensor k such that
Σ bdy. < foft S
Ldy
i.e., the cumulative FLOPs of all the tensors from 1 to k just exceeds our objective of FLOPs re- duction. As shown in Table 6, by applying such early stopping method, could proportionally save GPU memory with respect to the value of p. as a smaller value p leads to smaller ls and the backpropagation can hence be stopped earlier. For example, when p =50, 25% of GPU memory can be saved, and such saving could further increase to 50% when =34%.
Full evaluation
Etrly-stop p 34%
Early-stop p 40%
Early-stop 50%
Early-stop p 60%
Model
OPT-2.7B
10.8
5.5
6.5
8.1
9.7 10.8
FLAN-TS-3B
12.0
6.1
7.2
9.0
Table 6: GPU memory consumption (in GigaBytes) of tensor importance evaluation
A.2 REDUCING THE COMPUTATIONAL COST OF DYNAMIC PROGRAMMING FOR TENSOR SELECTION
In our proposed dynamic programming (DP) approach for tensor selection in Section 3.3, due to the high volume of FLOPs in LLM fine-tuning, the value of Tyull could be very large. To reduce the computational cost of DP, we can reduce the subproblem space by skipping two types of sub- problems: I) invalid ones, whose FLOPs constraint É exceeds the desired constraint (pT /w); 2) redundant ones, whose FLOPs to pass activation gradients to the maximally allowed depth (k) ex- ceeds t. Our preliminary experiment show that, doing so on an OPT model with Pip 50% can reduce the number of subproblems by 5.5x without affecting the optimality of training.
Model
Tg lel
Ti = 1c2
Ty 1c3
T. le4
Ti = 1eō
OPT-2.7B BLOOMZ-3B
0.02/64.1/32.0
0.04/47.6/30.1
0.64/49 8/30.7
7.5/50.0/30.9
76.5/50.0/30.9
0.0001/33.3/9.08
0.007/45.7/55.2
0.21/49 5/27.2
2.3/49 8/27.1
25.3/50.0/27.1
FLAN-T5-3B
0.04/64.9/36.5
0.25/57.1/36.5
3.5/55.3/36.7
41.8/51.8/36.7
449/50.0/36.7
Table 7: The impact of DP resolution Ty on fine-tuning OPT-2.7B, BLOOMZ-3B, and FLAN-T5- 3B I.Mss, on the SciTLDR dataset with p 50%. Each triplet [a/b/c] presents a) the percentage of wall-clock time incurred by DP compared to full fine-tuning, b) the percentage of FLOPs after reduction compared to full fine-tuning, and e) the testing ROUGE-1 score, respectively.
Besides, to further reduce the number of subproblems, we scale tensors' FLOPs (tdwr, tay) by mul- tiplying a factor of Z: tww
Ltw Z], tay Ltag Z].
(9)
where Z Tymi reduces the backropagation FLOPs to a resolution of T. < Trult. The overall time
To
complexity of DP then reduced to O(N2T,) On the other hand, such reduced resolution could increase the ambiguity in DP and affect the training quality. To investigate such tradeoff between the training quality and cost, we conducted preliminary experiments on multiple LLMs. Results
13

---

Published as a conference paper at ICLR 2024
in Table show that, for both OPT-2.7B and BLOOMZ-3B models, setting T, 1e3 reduces the DP overhead to < 1% without affecting the training quality. Similarly, for FLAN-T5-3B, choosing T3 le2 retain good training quality with negligible overhead. On the other hand, when T too small, the solution of DP could inaccurate and hence result in ineffective reduction of the training FLOPs.
plece of the LLM's computing graph represented in python code
statysstates self--g_rroghhideennei self.
batch size token num 512 taken_d dim 2550
key_states hhippppceel--.rrffiiiseeettte -1,
value_states self-_shapecseelf httpre/effvrrossddeeennnns staates, -1, boz)
Tensor-only representation based on tensors' execution order
q_proj
k_ proj
v_ v_proj
Kernel 2560x2560
Bies 2580x1
Kernel 2360x2360
Bias 2560x2
Kerne 2560x2560
Baas 2560x1
(say- fet
(Lay- Fabw
Kernel:
Eyy
Pay
512 x 2560 x 2560
512 2560 2560
Blas:
Fuwe
Pay
(Lay faw) for each tensor
Calculate and match
Kemel:
drj
01.
W." W,
Bias:
dy:
dy:-
B Backpropagation formulas for different types of tansors
dw
1 dy.s
don
W. Op-1
W dries
Figure 5: example tensor FLOOPs profiling in the OPT-2.7B model
A.3 AN EXAMPLE OF TENSOR FLOPs PROFILING IN THE OPT-2.7B MODEL
To better facilitate understanding, we further show an example in Figure about how we profile tensors in the OPT-2.7B models in our experiments. First, we convert the computing graph of the LLM, which implemented in Python code, into a tensor-only representation. The tensors are ordered based on their execution orders in the forward pass, similar to the layer-level graph in Figure 3. We then calculate each tensor's 's FLOPs (fay. tay) based on the backpropagation formulas discussed in Section 3.1. Such calculations are essentially counting the multiplications and being added in their formulas.
Method
Accuracy (%)
PFLOPs
Time (h)
Method
Accuracy (%)
PFLOPs
Time (h)
LoRA
19.6
16.0
LoRA 49.5
174.0
6.27
GT-0,5
28.7
0.50
GT-055
59.2
130.5
4.69
GT-0.6
29.5
14.0
0.61
Table 8: OPT-2.7B on PIQA dataset
Table OPT-2.7B on WebQuestion dataset
A.4 PERFORMANCE ON GENERATIVE QUESTION-ANSWERR TASKS
To better evaluate the performance of Green'rraier other tasks, also conducted experi- ments by using the OPT-2.7B model on WebQuestions and PIQA datasets for generative QA tasks. The WebQuestions dataset contains 6,642 QA pairs using Freebase as the knowledge base. The PIQA dataset focuses on multi-choice QA about physical knowledge with k QA pairs. We adopt the prompt format " (aaeesstio::(////aassssssa (a) /a></=>" for WebQuestions for PIQA,
and "goal: [/)
soll1 where 
 is the EOS token for OPT models. The hyper-parameters for training are the same as the ones described in Section We evaluate the sentence-level accuracy which requires the gen- erated answer to exactly match the ground truth. Note that for PIQA, the generated tokens are still predicted from the entire dictionary of OPT embeddings instead of from the two choices: the first
14

---

Published as a conference paper at ICLR 2024
or the second one, As shown in Table 8 and Table 9, on both datasets, GreenTraing (GT) achieves significantly better accuracy and time efficiency compared to LoRA.
In partieular, the results the PIQA dataset generally lower than those reported in Brown al. (2020). The reason for this accuracy gap is that the way we use the OPT model to generate answers is more challenging than the setup in Brown et al. (2020). According to Section 2.4 in Brown et al. (2020), it formulates the PIQA task as a multi-choice QA task where the answer drawn from a small and predefined candidate set (e.g., ["0", "I""), by comparing the probability scores only over the candidate tokens. In comparison, we strictly cast the problem to open-ended generation, where the candidate set is unknown. In that case, generating correct answers can be more difficult, because the model could generate totally irrelevant answers and increase its chance of making mistakes.
A.5 IMPACT OF FREQUENCY OF TENSOR IMPORTANCE EVALUATION
Our design of GreenTrainer by default, evaluates the importance of tensors and select the set of trainable tensors based on such importance at the beginning of each training epoch. Using the tech- nical approach described in Section 3.1, such tensor importance evaluation is very lightweight, and our experiment results show that the overhead of importance evaluation only 0.2% SciTLDR dataset and 0.01 % on DialogSum dataset, with respect to the entire fine-tuning cost.
On the other hand, in certain cases, the tensor importances, calculated from the model gradient changes, could exhibit non-neglige non-negligible differences within one epoch. In these cases, the flexible design of Green Trainer will allow us to adaptively increase the frequency of tensor importance eval- uation and the corresponding DP-based tensor selection. To demonstrate the impact of such more frequent tensor importance evaluation and DP-based tensor selection, we conducted extra experi- ments using OPT-2.7B model on the WebQuestions dataset and generative QA task, as shown in Table 10.
Frequency of tensor importance evaluation
Accuracy (%)
Time (h)
Every 945 iterations (once per epoch)
28.4
0.50
Every 600 iterations
28.5
0,54
Every 400 iterations
28.2
0.56
Every 200 iterations
27.5
0.64
Table 10: Impact of tensor importance evaluation frequency
The results show that: (1) More frequent tensor importance evaluation brings only very small im- provement on task accuracy. Considering the randomness in different training trials, we believe that such accuracy improvement is negligible, and the accuracy could even drop down by 1 1% when the frequency of evaluation is very high (every 200 iterations). We believe that this is due to accumu- lation of tensor importance evaluation and tensor selection errors, which stem from the first-order approximation in the tensor importance metric and the approximate solution DP. Another possible reason is that the tensor importances are calculated over the training dataset, and too frequent tensor importance evaluation may make the training process overfit to the training dataset. (2) The training cost steadily increases with the frequency of tensor importance evaluation. When the interval of evaluation reduces from 945 iterations to 200 iterations, the training time increases by 28%.
In summary, performing more frequent tensor importance evaluation within each epoch brings little improvement the task accuracy but noticeably increase the training cost. We believe that the tensor importances being evaluated once in each epoch would be sufficiently accurate for appropriate selection of trainable tensors.
A.6 THE NECESSITY OF DYNAMIC TENSOR SELECTION
If the LLM fine-tuning uses a fixed training dataset, it is possible that using a fixed tensor selection decided at the initial phase of training may not result in a significant model accuracy drop, com- pared to runtime tensor selection. However, in practical LLM fine-tuning scenarios, this assumption usually does not hold due to the following two reasons. First, in a lot of L.M fine-tuning scenarios, such as online learning and model personalization, the model is continuously retrained using online data, which is continuously generated at runtime with variant data distributions. Such variant data distributions will surely result in different importances of tensors through the training procedure and
15

---

Published as a conference paper at ICLR 2024
hence require runtime tensor selection. Such online LLM fine-tuning scenarios recently become more and more popular, especially with the possibility of deploying LLMs onto user's ss personal mo- bile devices such as smartphones. Second, even for fixed training dataset, it is also possible that the importances of some tensors may change as the training progresses. In these cases, dynamic tensor selection could improve the trained model accuracy. To verify this, conducted additional experiments using the OPT-2.7B model on the WebQuestions dataset and generative QA task. As shown in Table 11, dynamic tensor selection could make non-negligible contributions to improving task accuracy, with negligible increase of training cost.
Strategy
Accuracy (%)
Time (h)
Fixed lensor selection only in epoch of training Dynamic tensor selection, once each epoch More frequent tensor selection (5 times in each epoch)
27.4
28.4
0.50
27.5
0.64
Table 11: Different strategies of tensor selection
Note that, such improvement of model accuracy would be dependent on the specific dataset and model being used, but these experiment results above demonstrated the necessity of runtime tensor selection. In addition, our experiment results also showed that such tensor importance evaluation and selection indeed incur very little extra computing overhead.
16


Reviews:
[{"review_id": "49tX2LM5ex", "reviewer": "Reviewer_B8AP", "summary": "To address the significant computing costs and environmental impact associated with LLM fine-tuning, this paper proposes \"GreenTrainer,\" a new technique designed to minimize FLOPs through adaptive backpropagation. GreenTrainer selectively fine-tunes the most relevant parts of the LLM based on their importance and the computational cost of their training. Experimental results demonstrate that GreenTrainer can reduce training FLOPs by up to 64% compared to traditional full fine-tuning methods, while maintaining accuracy compared to full-finetuning.", "questions": "It is unclear whether the fine-tuning process includes the DP (data parallelism) overhead or if it is counted separately. It would be helpful to clarify the ratio of DP overhead during fine-tuning.\n\nHow X% of FLOPs saving can lead to a X% reduction in latency in Tab 2? LLMs fine-tuning  is  both  computation- and **memory-intensive**. Even with the same FLOPs, the latency can vary significantly depending on how the computation is allocated. For example, updating last 2 blocks may have similar performance with updating a partial set of all blocks, but the latency of latter one is much higher as it back-propagates to the first layer. To provide a more comprehensive understanding, additional benchmark details are needed to justify these observations.\n\nTo further evaluate the performance of GreenTrainer, more experiments should be conducted to compare it against baselines such as GLUE (which should be relatively inexpensive to execute) and other tasks like hellaswag, webquestions, and piqa. The community is also interested in the performance on Llama. If the 7B model is causing out-of-memory (OOM) issues, reporting the performance with LoAR and GreenTrainer only, or performing on OpenLlama-3B, would be valuable as well.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses an important problem mentioned within it: fine-tuning 10K Llama-13b models can be 6.9 times more expensive than pre-training 175Bs. It is crucial to reduce carbon footprint and energy consumption in order to make LLMs sustainable.\n\nTraditional PEFT methods, such as LoRA and adapter, focus on reducing (learnable) parameters but discuss less about fine-tuning FLOPs and throughput. This work explores ways to actually reduce training costs, and the proposed method allows for flexible trade-offs between computation cost and fine-tuning quality.\n\nThe authors thoroughly compare the performance on SciTLDR and DialogSum datasets, demonstrating that GreenTrainer achieves up to 4% higher performance with faster training speed compared to existing schemes. Furthermore, the authors provide detailed analysis by varying importance metrics (Table 4), FLOPs reductions (Table 3), and LLM model sizes (Table 5).", "weaknesses": "The justification for the effectiveness of adaptive selection is not clear. Language models (LLMs) are typically over-parameterized, and updating a random set of sparse parameters can achieve similar performance. Therefore, it is important to understand why and how adaptive selection properly selects the parameters that are worth fine-tuning. Baselines such as random selection, Fisher, and SNIP should be discussed to provide a comprehensive analysis.\n\nIt should be noted that adaptive selection can introduce additional complexity in the training process. Although the paper presents strategies to reduce this cost (dynamic programming), it still adds a layer of complexity to the training process with a complexity of O(N^2 T). There are concerns that this overhead will grow with the size of the model and soon become non-negligible.\n\nThe benchmarks in the paper only include summarization tasks and do not provide a holistic evaluation. To claim that a method reduces FLOPs without compromising accuracy, it is necessary to evaluate it on more benchmarks, such as GLUE and E2E NLG Challenge. Even within the summarization tasks, GreenTrainer does not show superior performance compared to LoRA under the **same latency budget**, as shown in Table 2 for FLAN-T5, where GT-0.4 and LoRA are shows very close performance and throughput.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HuJ7ftYcnG", "reviewer": "Reviewer_P1Bv", "summary": "The paper proposes GreenTrainer, an approach for improved balancing of task quality and cost tradeoff of model finetuning.  The main idea is that only tensors that are important for the downstream task should be trained during finetuning so as to simultaneously minimize any quality loss and computation (FLOPs) costs. The paper further presents an algorithm for identifying important tensors for a downstream task at runtime to ensure adaptivity. The evaluation results show up to 64% reduction in FLOPs cost with little or no quality loss compared to fine-tuning the full model, and up to 4% quality improvement compared to existing FLOPs reduction techniques.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Model fine-tuning is an important aspect of deep learning, and so the paper is tackling an important and timely problem in trying to reduce the computation costs of finetuning while preserving task quality. \n\nThe idea of identifying important tensors at runtime for the target downstream task is interesting and seems more promising than the existing static approaches. It seems that this would be the right solution for the problem, assuming it could be done efficiently. \n\nThe quality and reduction improvements presented in the evaluation are quite impressive.", "weaknesses": "The main concern is the vagueness on the efficiency or costs of identifying important tensors during finetuning. This makes it difficult for me to judge whether or not this is a practical approach. The tensor selection is simply claimed to be O(N), but I find this to be insufficient to understand the runtime costs. Specifically, I could not glean the following pertinent information from the draft.\n\n1. What is the e2e finetuning time of GreenTrainer compared to baselines.  \n\n2. How frequently are important tensors identified? Is it per iteration, per epoch (as suggested by Figure 1), or some other intervals? What is the frequency for the experimental results? \n\n3. What is the `m` binary vector, or at least the number of important tensors, for the experimental results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8y4l2cSjbs", "reviewer": "Reviewer_CgrN", "summary": "In this paper, the authors propose Green-Trainer for reducing the training cost for adapting pre-trained large language models to downstream applications. The idea is to leverage adaptive backpropagation for adaptively selecting the most appropriate set of LLM\ntensors for fine-tuning based on their importance and backpropagation cost in training. This is achieved by measuring the tensor importance based on the cumulative gradient changes of its weight updates in training. Also, a new FLOPs model is built for finding the optimal tensor from an exponential number of possibilities. Compared with existing fine-tuning methods, the proposed Green-Trainer can reduce 64% training FLOPs while achieving similar accuracy.", "questions": "1. In the experiments, the authors only fine-tune the model for 5 epochs, then it is possible for some fine-tuning methods (eg. FT-Top2) to have the underfitting issue, how about fine-tuning for more epochs?\n\n2. How the running time is measured? There is a lack of hardware specifications.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The problem of reducing the FLOPs when fine-tuning LLM is important. And the authors propose an effective method for achieving this while maintaining the model accuracy. \n\n2. The authors include a clear motivation for conducting adaptive backpropagation to reduce the computation cost of existing fine-tuning methods for LLMs. \n\n3. The authors conduct extensive experiments to demonstrate the effectiveness of the proposed method compared with the baseline methods.", "weaknesses": "1. The  novelty of the proposed method is rather limited. In particular, the method for evaluating the tensor importance has already been proposed for neural network pruning. And the authors did not cite any relevant literature here. \n\n2. The writing of the paper is sometimes confusing. For example, in section 3.1, it is not clear how such profiling is done for different layers in LLM. The authors should give some concrete examples of the FLOPs for these layers to better demonstrate the idea of tensor FLOPs profiling. \n\n3. Since the proposed dynamic programming can only find approximate solutions, there is no discussions on the accuracy of the found solution using the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u6PPwV9Fm8", "reviewer": "Reviewer_U1j2", "summary": "In this paper, the authors propose GreenTrainer, a method to reduce the FLOPs of LLM fine-tuning by selective back-propagation. GreenTrainer consists of three techniques: tensor FLOPS profiling, tensor importance evaluation, and tensor selection (for update). Experiments show that the proposed method can achieve similar accuracy while reducing up to 64% of training FLOPs, which makes training faster and reduces energy usage.", "questions": "Please see the weakness section. Thanks!", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well-written and easy to follow. \n2. The proposed method can reduce the training FLOPs more compared to existing efficient fine-tuning methods like LoRA. \n3. The ablation studies are comprehensive, showing the effectiveness of the tensor selection mechanism and the scalability with larger models.", "weaknesses": "1. The template is highly altered to fit in more content. For example, the author section is removed, which may violate the policy. \n2. The paper misses some references to existing work that uses similar techniques. For example, [a] also uses sparse update for the model and a similar optimization goal as Equation 5 to get the tensor selection scheme, which also considers the backward cost for weights and activations.  Other related works include [b, c]. The authors should discuss and differentiate from existing works. \n3. There are some other efficient training methods focusing more on FLOPs reduction instead of parameter reduction like [d]. The authors may also want to include them as baselines since LoRA and Prefix-tuning are not designed for computation efficiency. \n4. Despite the effective FLOPs reduction, I would expect the method to use more GPU memory compared to methods like LoRA due to the larger optimizer states (since more weights are updated). The authors may also discuss the memory usage compared to the baseline methods, and see whether it can be combined with LoRA to achieve reduced memory and FLOPs at the same time (e.g., only add LoRA to updated tensors). \n5. The tensor importance estimation is based on the gradient information. Is it consistent throughout the training, or will the importance change?\n\n\n[a] Lin et al., On-Device Training Under 256KB Memory\n[b] He et al., Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning\n[c] Kwon et al., TinyTrain: Deep Neural Network Training at the Extreme Edge\n[d] Evci et al., Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)