# Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator
## 4 Method
### Salient Space Choose Strategy  
For fine-tuning, the objective is to integrate new knowledge into the pre-trained weights. Therefore, identifying weights that are sensitive to the fine-tuning dataset is crucial. Following the core principle from Child Tuning [18], our approach considers the singular values and the importance of gradients. This dual consideration helps us design a metric to select the most relevant subspaces for updates.  
Obtaining the gradient importance of each subspace is challenging, as it requires training models to converge, which is time-consuming. Following prior work [18], we approximate \(\mathbf{C}\) using the gradient of \(\mathbf{A}\) after a single step of gradient descent.  
Since the latent space corresponds to a pair from the weight decomposition, using the full-rank decomposition for gradient descent calculations could lead to large calculation and memory demands. To solve this problem, we sparse the \(\mathbf{U},\Sigma\), and \(\mathbf{V}^{T}\) by selecting the top \(2r\) singular values and then compute the gradient-based importance metric.  
While Child Tuning [18] selects subspaces based on element-wise importance, our method focuses on identifying important channels from the SVD decomposition. We use an importance score for the entire channel rather than for each individual element. Inspired by structural pruning [33], we utilize the second-order Taylor expansion to determine the importance of each channel. Given the channel gradient \(g_{\mathbf{A}}=\{g_{a_{i}}\}\), the gradient importance score can be formulated as follows:  
\[s_{a_{i}}=g_{a_{i}}^{T}g_{a_{i}} \tag{11}\]  
As low-rank matrices estimate the gradients of activations, we also consider the magnitude of singular values. Therefore, we use the energy of the singular values (the square of the singular values) to represent their importance. We then normalize these two metrics and combine them using a temperature parameter:  
\[S_{i}=\frac{||\lambda_{i}^{2}||}{max(||\lambda_{i}^{2}||)}+\frac{1}{\alpha} \frac{||s_{a_{i}}||}{max(||s_{a_{i}}||)} \tag{12}\]Here, \(\alpha\) serves as a temperature parameter that helps balance differing normalization values. We present a detailed ablation study to compare the performance of different importance metrics and different \(\alpha\).  
The entire fine-tuning process with LoGE is outlined in Algorithm 1. In practice, the batch size \(b\) is set to 32. The time cost for calculating the importance score is approximately 20 s for the OPT-1.3B model on the Alpaca dataset.  
```
0:\(\theta\), \(\mathcal{D}_{k}\)
1:for layer in model layers do
2:if layer is linear then
3:\(\mathbf{U},\Sigma,\mathbf{V}^{T}\leftarrow\mathbf{W_{0}}\)
4:\(\text{layer}\leftarrow\text{LoGE}(\mathbf{W_{0}},\mathbf{A}^{\prime},\mathbf{ U},\Sigma\mathbf{V}^{T})\)
5:endif
6:endfor
7:Sample \(b\) samples \(d_{k}\) from \(\mathcal{D}_{k}\).
8:Calculate gradient of \(\mathbf{A}^{\prime}\) with \(d_{k}\).
9:Calculate importance index \(S_{i}\) and select \(idx\).
10:for layer in model layers do
11:if layer is linear then
12:\(\text{layer}\leftarrow\text{LoGE}(\mathbf{W_{0}},\mathbf{A},\mathbf{U}_{[idx]}, \Sigma_{[idx]}\mathbf{V}^{T}_{[idx]})\)
13:endif
14:endfor
15:for s in training steps do
16: Update \(\mathbf{A}\) [Training step with LoGE]
17:endfor
18:return\(\theta\)
```  
**Algorithm 1** LoGE for fine-tuning. \(\theta\) represents the model parameters, and \(\mathcal{D}_{k}\) is the dataset for task \(k\).