## 4 Methodology
### Parameter-Efficient Fine-Tuning: Delta-LoRA  
Directly applying full fine-tuning to large models like Hyena-Small (125M) might yield the best results, but is costly and prone to overfitting when labeled data (like specific domain central bank texts) is limited and computational resources are constrained. Therefore, we employ parameter-efficient fine-tuning (PEFT) techniques. Low-rank adaptation (LoRA) [10] is a widely used PEFT method. Its core idea is to freeze most weights \(W\in\mathbb{R}^{c\times d}\) of the pretrained model during fine-tuning and introduce a parallel, trainable low-rank \"bypass\" structure for certain linear layers (e.g., \(W_{Q},W_{K},W_{V},W_{O}\) in MHA or FFN layers). This bypass decomposes the original weight update \(\Delta W\) into the product of two low-rank matrices: \(\Delta W=AB\), where \(A\in\mathbb{R}^{c\times r}\), \(B\in\mathbb{R}^{r\times d}\), and the rank \(r\ll\min(c,d)\). During forward propagation, the output \(h\) is computed as:\[h=Wx+\frac{\alpha}{r}ABx \tag{6}\]  
where \(x\) is the input and \(\alpha\) is a scaling factor. Only the parameters of \(A\) and \(B\) are updated during training, significantly fewer than those in \(W\). After training, \(AB\) can be merged back into \(W\) (\(W_{\text{new}}=W+\frac{\alpha}{r}AB\)), eliminating extra computational overhead during inference. While LoRA significantly reduces fine-tuning costs, its performance often lags behind full fine-tuning because the frozen \(W\) cannot adapt. Delta-LoRA [13] proposes a novel improvement strategy aiming to bridge this gap while maintaining comparable memory and computational efficiency to LoRA. The core idea of Delta-LoRA is to not only update the low-rank matrices A and B, but also leverage the change in the low-rank update to drive the update of the pretrained weights W. The structure and update mechanism of Delta-LoRA are depicted in Figure 6.  
Specifically, at each training step \(t\), first compute gradients \(g_{A},g_{B}\) for \(A\) and \(B\) like in the standard LoRA, and obtain updated \(A^{(t+1)}\) and \(B^{(t+1)}\) via an optimizer (e.g., AdamW). Second, calculate the change (delta) in the low-rank adaptation matrix:  
\[\Delta(AB)^{(t)}=A^{(t+1)}B^{(t+1)}-A^{(t)}B^{(t)} \tag{7}\]  
Third, use this change to update the pretrained weights \(W\):  
\[W^{(t+1)}=W^{(t)}+\lambda\cdot\frac{\alpha}{r}\cdot\Delta(AB)^{(t)} \tag{8}\]  
where \(\lambda\) is a new hyperparameter controlling the extent to which the \"momentum\" or \"change trend\" of the low-rank update is transferred to \(W\). This parameter update typically starts after an initial number of training steps \(K\) (e.g., \(K=500\)). Delta-LoRA allows the pretrained weights \(W\) to participate in the learning process, enabling better adaptation to the downstream task. Theoretically, under certain conditions (no dropout), \(\frac{\partial\mathcal{L}}{\partial W}=\frac{\partial\mathcal{L}}{\partial(AB)}\), making \(\Delta(AB)\) an effective surrogate for the update direction of \(W\), capturing the learning trajectory of the low-rank adaptation. Crucially, updating \(W\) does not require computing or storing the gradients of \(W\) itself or its optimizer states (like first and second moments in AdamW). The update of \(W\) depends solely on the already computed \(A^{(t+1)},B^{(t+1)}\) and their previous values \(A^{(t)},B^{(t)}\). This keeps Delta-LoRA's memory footprint and computational overhead nearly identical to standard LoRA. Standard LoRA implementations often include dropout before \(A\) or \(B\). However, the derivation of Delta-LoRA suggests that dropout breaks the equality \(\frac{\partial\mathcal{L}}{\partial W}=\frac{\partial\mathcal{L}}{\partial(AB)}\). Therefore, the Delta  
Figure 6: Diagram illustrating the Delta-LoRA structure and its update mechanism. Shows the base model weight W, the low-rank matrices A and B, and the delta update influencing W.  
LoRA structure removes dropout from the LoRA bypass to ensure that \(\Delta(AB)\) is a more accurate estimate of \(W\)'s update direction.