Fast Randomized Low-Rank Adaptation of Pre-trained Language Models with PAC Regularization (2024)

Abstract: Low-rank adaptation (LoRA) achieves parameter efficient fine-tuning for large language models (LLMs) by decomposing the model weight update into a pair of low-rank projection matrices. Yet, the memory overhead restricts it to scale up when the model size increases. We propose Randomized LoRA (RLoRA) which adopts Randomized Walsh-Hadamard Transform to achieve significant reduction in the size of trainable parameters compared to LoRA. At the same time, it allows a PAC-Bayes regularizer to be efficiently incorporated to improve generalization. We evaluate the effectiveness of RLoRA on LLMs RoBERTa, GPT-2 and LLaMA-7B using GLUE, E2E and math reasoning benchmarks. With a much lower memory requirement, RLoRA can give similar performance as the SOTA low-rank adaptation methods for these three tasks and significantly better performance under few-shot settings.

heading: 1 Introduction
text: Pre-trained Language Models (PLMs), typically consisting of millions or billions of parameters, have achieved the state-of-the-art performance on NLP tasks (Devlin et al., 2018). While PLMs can be fine-tuned for specific downstream tasks, their increasing scale makes the fine-tuning and model storage formidable.
Parameter-efficient fine-tuning (PEFT) aims to fine-tune a pre-trained model efficiently by adapting only a small set of parameters. Various approaches for PEFT have been explored. i) The adapter-based approach (Houlsby et al., 2019; mahabadi et al., 2021; Pfeiffer et al., 2020; He et al., 2021) introduces trainable adapter modules to achieve fine-tuning with the PLM frozen. This approach however falls short of the inference latency introduced. ii) The masking-based approach updates only some selected subset of the PLM’s parameters (Guo et al., 2020; Zaken et al., 2021;
Fu et al., 2022; Sung et al., 2021). As it requires access to the PLM for updating, its application is restricted. iii) Prefix-tuning (Li and Liang, 2021) and prompt-tuning (Lester et al., 2021) prepend additional tokens to the input embeddings and only train these soft prompts where the contextual information is wrapped to the inputs (or hidden states) without modifying the PLM. This approach however converges much slower than the adapter-based one (Ding et al., 2022; Hu et al., 2021).
Low-rank adaptation (LoRA) (Hu et al., 2021) hypothesizes that the PLM’s weight update for the fine-tuning possesses a low “intrinsic rank” dominating the desirable optimization trajectory (Li et al., 2018; Aghajanyan et al., 2021), and proposes to decompose the update of the weight (e.g., Transformer’s self-attention weight metrics) into two low-rank projection matrices. This reparameteration trick has been shown particularly effective in reducing the number of trainable parameters compared to other PEFT approaches. Yet, the memory overhead is still considerably large when the size of the model increases.
5236
To this end, inspired by the work on random feature projection for kernel machines (Rahimi and Recht, 2007; Liu et al., 2021),we propose randomized low-rank adaptation (RLoRA) which adopts the random matrix approximation to construct the up-projection and down-projection matrices in LoRA. In particular, we adopt the Randomized Walsh-Hadamard Transform (WHT) and a learnable scaling vector to reparameterize the low-rank projections to reduce both the time and space requirements. Using random matrices to reduce the memory requirement has also been explored in some recent work like LoRA-FA (Zhang et al., 2023b), VERA (Kopiczko et al., 2024) and NOLA (Koohpayegani et al., 2023). To contrast, RLoRA adopts the Randomized WHT which is orthogonal and recursively defined, which can further reduce the resources needed for the gradient computation w.r.t. the hidden states.
Also, the proposed RLoRA adopts PAC-Bayes regularization to enhance generalization performance, which is particularly important for adapting PLMs where only a small training set is available. Some related ideas have been explored in Lotfi et al. (2022, 2023), which empirically showed that the PAC-Bayesian theories provide a tighter bound and understanding for the role of model size and generalization ability. Achille et al. (2019); Wang et al. (2021) used the PAC-Bayesian framework to estimate the information in neural networks. The adoption of the Randomized WHT in RLoRA allows the regularization to be efficiently incorporated. To the best of our knowledge, work on incorporating the PAC-Bayes generalization bound for regularizing PEFT is still lacking in the literature.
Our contribution can be summarized as:
1. The number of trainable parameters in LoRA can be substantially reduced by incorporation of Randomized WHT.
2. We show theoretically how the “intrinsic subspace” of the PLM’s weight update is characterized by Randomized WHT.
3. We show how a PAC-Bayes regularization term can be efficiently estimated and incorporated for fine-tuning RLoRA to boost its generalization performance.
4. We conduct comprehensive experiments to demonstrate the effectiveness of RLoRA for fine-tuning PLMs including Roberta-base, Roberta-large, GPT-2 medium and LLaMA7B based on the GLUE, E2E and math reason-
ing benchmarks.
Our code will be made available for sharing via gitHub.

heading: 2 Background
text: In this section, we provide background and related work on low-rank adaptation, random matrix methods, and PAC-Bayes generalization bound.

heading: 2.1 Low-Rank Adaptation (LoRA)
text: LoRA (Hu et al., 2021) starts with the PLM W0 which is frozen and reparameterizes the PLM’s weight update ∆W using two low-rank projection matrices. The forward pass becomes:
h = W0x+∆Wx = W0x+WBWAx (1) where WB ∈ Rdout×r and WA ∈ Rr×din are the trainable parameters, and r ≪ min{din, dout}. The memory overhead for LoRA is still an issue because of: i) high memory consumption to store the activation during the forward pass and to construct the gradient; and ii) the minimum number of trainable parameters lower-bounded by the rank-1 decomposition of the weight matrix which is considerably large as the model size increases.

heading: 2.2 LoRA with Random Matrix Methods
text: Random matrix theory has been applied to achieve PEFT. For instance, LoRA-FA (Zhang et al., 2023b) freezes the down-projection matrix WA which is randomly initialized. VERA (Kopiczko et al., 2024) replaces both WA and WB with random matrices so that the weight update becomes: ∆WV ERA = dWBbWA where only two scaling vectors d, b are learned. NOLA (Koohpayegani et al., 2023) uses multiple random seeds to generate the random basis to form WA and WB , and learn the combination vectors α and β to compute ∆WNOLA = ( ∑k i=1 αiWAi)× ∑l j=1(βjWBj ).
In this paper, random projection methods (Li et al., 2018; Liu et al., 2021) are further explored to reduce the time and storage requirements while maintaining the effectiveness. In particular, we focus on the Randomized Walsh-Hadamard Transform (WHT) as it can efficiently draw a random matrix from a highly structured distribution (Tropp, 2011) (to detailed in the next section).

heading: 2.3 PAC-Bayes Generalization Bound
text: The PAC-Bayes bound is a theoretical framework that provides a probabilistic guarantee on the generalization performance of a learning algorithm.
It is based on the principles of probably approximately correct (PAC) learning and Bayesian inference. Given the true risk R and the empirical risk R̂S , the PAC-Bayes bound can be defined as:
R− R̂S ≤ ES { Eh∼H [ KL ( P (h) ∣∣∣ ∑ h′∈H P (h ′|S)
|H|
)]}
(2) where S represents the training data, H is the hypothesis space, P (h) is the prior distribution over hypotheses, and P (h′|S) is the posterior distribution over hypotheses given the data. The risk difference is upper bounded by the expected KL divergence between the prior and the data-dependent posterior distributions over hypotheses. This implies the PAC-Bayes framework rewards the model has strong prior on it parameters aligned with the data (Lotfi et al., 2022). The proposed RLoRA makes use of a PAC-Bayes bound to guide its finetuning process.

heading: 3 Methodology
text: In this section, we present the details about how the proposed RLoRA. In particular, Section 3.1 presents the Randomized WHT and how it is integrated into RLoRA to gain parameter-efficiency and reduction in computational cost during training. In Section 3.2, we show theoretically how the posterior distribution of the PLM’s weight update based on the reparameterization is associated with a lower-dimensional “intrinsic subspace” characterized by the Randomized WHT. We derive the PAC-Bayes generalization bound in Section 3.3 and explain how the reparameterization by Randomized WHT allows the bound to be efficiently incorporated into RLoRA to regularize the fine-tuning. Fig. 2 shows an overall illustration of RLoRA.

heading: 3.1 LoRA with Randomized WHT Incorporated
text: RLoRA builds upon LoRA. It replaces WA and WB in LoRA with two WHTA and WHTB approximated by Random WHT as well as two trainable scaling vectors α and β, so that the low-rank decomposition of ∆W becomes:
∆W = [diag(β)WHTB][diag(α)WHTA], (3)
where diag(α)ii = αi.

heading: 3.1.1 Efficient generation of projection matrices
text: The projection matrices WHTA and WHTB can be efficiently generated by i) first setting a random
seed and ii) applying the WHT without the need to explicitly store the projection matrix.
Specifically, according to (Choromanski et al., 2017), a d× d Gaussian random matrix R ∈ Rd×d can be approximated by a product of Randomized Walsh-Hadamard matrice H and diagonal matrices S and Bi, given as:
R = 1
σ √ d S
k∏
i=1
HBi. (4)
S is a random scaling and subsample matrix. Bi has elements of independent random signs {−1, 1} along its diagonal. The memory cost of S and Bi is both O(d). The value of k determines how many independent random signals Bi should be multiplied. Increasing the value of k can improve the randomness and the orthogonality of the resulting matrix R. Empirical findings suggest that setting k = 2 or 3 is sufficient for practical applications.
For the matrix H ∈ Rd×d, it can be efficiently generated due to its recursive definition: Hd =[ Hd/2 Hd/2 Hd/2 −Hd/2 ] with H2 = [ 1 1 1 −1 ] . The generation iterates two simple steps: i) dividing the data into two halves, and ii) performing a simple ± operation. Fig. 2(c) shows an example of an 8-dimensional WHT. This fast WHT allows Hx to be computed in O(d log d) time and the transformation basis to be orthogonal.

heading: 3.1.2 Efficient matrix multiplication during backpropagation
text: Assume that the hidden state is denoted as x ∈ Rn×din and the weight as W ∈ Rdin×dout where n is the batch size and din(dout) is the dimension of the input(output) hidden state. The information propagates between layer l and layer l + 1 as:
xl+1 = xlWl
g(W ) = g(xl+1)xl g(xl) = g(xl+1)Wl (5)
where g(W ) is the gradient used to update the trainable parameters and g(xl) is for propagating the gradient to other layers. Given the Randomized WHT adopted, Eq.5 becomes:
xl+1 = αlWHT l(xl)
g(αl) = g(xl+1)WHT l(xl) g(xl) = WHT −1 l (g(xl+1))
(6)
where WHT l(xl) is the multiplication for layer l which can be efficiently computed, WHT−1l () is the reverse projection to layer land αl is the corresponding scaling vector.

heading: 3.2 Intrinsic Subspace and Posterior Distribution of Weight Update
text: To better understand the intrinsic subspace of the PLM’s weight update, we first assume that the posterior distribution of the weight update ∆W follows a matrix-variate Gaussian. Then, we show that the distribution after the reparameterization is associated with a low-dimensional “intrinsic subspace” characterized by the Randomized WHT matrices.
With reference to the analysis in (Rossi et al., 2020), we assume that the posterior distribution of the weight update ∆W = WAWB ∈ Rdin×dout follows a matrix-variate Gaussian distribution, given as ∆W ∼ MN (M ,U ,V ) where M is the mean, and V ∈ Rdout×dout and U ∈ Rdin×din denote the covariance matrices among the rows and columns of ∆W respectively. The vectorized version of ∆W can be expressed equivalently as:
vect(∆W ) ∼ N (vect(M),V ⊗U), (7)
where ⊗ denotes the Kronecker product.

heading: 3.2.1 Intrinsic subspace of ∆W
text: We rewrite ∆W by replacing WA and WB using the Randomized WHT according to Eqs. 4 and 3. For the convenience of the analysis, we assume k = 11 and drop the scalar factor 1
σ √ d . Also, we denote the matrix selecting the first r rows (columns) from WA (WB) as SWA (SWB ). Then, it gives:
∆W = diag(β)SWBHBWBS T WA diag(α)SWAHBWA . (8)
1Note that the result of this analysis can be easily generalized to cases with other values of k. We leave that in the Appendix.
By putting HBWA = [v1,v2, . . . ,vd] where vi = H(i,:)BWA (i,i), vect(∆W ) can be expressed as:
vect(∆W ) = Gα
=   diag(β)SWBHBWBS T WA SWAdiag(v1) diag(β)SWBHBWBS T WA SWAdiag(v2) ...
diag(β)SWBHBWBS T WA SWAdiag(vd)
  α
(9)
where G ∈ Rdindout×r, α ∈ Rr, β ∈ Rdout . The factor STWASWA is essentially randomly selecting r basis vectors from G to construct the projection from α to vect(∆W ).

heading: 3.2.2 Posterior distribution of ∆W
text: Assume that RLoRA’s trainable parameters α,β ∼ N (µα,Σα),N (µβ,Σβ), it gives
vect(∆W ) ∼ N (µαG,GTΣαG). (10)
This implies that the posterior distribution of ∆W is in fact lying on the dα-dimensional subspace of dindout-dimensional space embedded by the orthogonal projector G. The orientation of the space ∆W is controlled by the random directions BWB BWA and the learnable β, which could be regarded as the learnable intrinsic subspace of the network (Aghajanyan et al., 2021).
In addition, we can show that Eq. 10 is equivalent to a matrix-variate Gaussian for ∆W with the parameters M ,U ,V given as:
M = diag(µβ)SWBHBWBS T WA diag(µα)SWAHBWA
U 1 2 = diag(β)SWBHBWBS T WA diag(Σ1/2α )S T WB diag(β) V 1 2 =
1√ tr(U) BWAHS T WA diag(Σ1/2α )
(11)
which can be estimated based on the Randomized WHT. See Appendix B for detailed derivation.

heading: 3.3 PAC-Bayes Regularization
text: We derive an approximation of the PAC-Bayesian generalization bound to guide the learning process for better generation. According to Section 3.2, if α follows the Gaussian distribution, the weight update matrix ∆W will follow the matrix-variate Gaussian. For the clarity reason, we use w to denote the trainable model parameters (i.e., α and β for our case) in the sequel.

heading: 3.3.1 PAC-Bayes generalization bound
text: Suppose the posterior of the model parameters w given the training data S of size N for a specific fine-tuning task is p(w|S) ∼ N (µS ,ΣS) and the oracle prior p(w) = Ep(S)[p(w |S)] ∼ N (µ,Σ) . The expectation of the KL divergence between the posterior and the prior of w over the task-specific data S becomes:
I(w) = Ep(S) [KL (p(w|S), p(w))]
∝ Ep(S) [ log
|ΣS | |Σ| + tr
( Σ
ΣS
)
+(µS − µ)TΣ−1(µS − µ) ]
(12)
where |A| and tr(A) are the determinant and trace of matrix A, respectively. By assuming the covariance of prior and posterior to be proportional, which is common for building PACBayesian bound (Dziugaite and Roy, 2018), the log and trace terms become constant. We have
I(w) ∝ Ep(S) [ (µS − µ)TΣ−1(µS − µ) ]
= Ep(S) [ µTSΣ −1µS ] − µTΣ−1µ. (13)
To achieve a small KL with the data-dependent posterior, the prior should essentially be a good predictor for the posterior, which will be hard without access to the data distribution p(S).

heading: 3.3.2 Estimating prior covariance matrix
text: As the pre-trained network has already converged to a stable solution, we assume that the weight update introduced by fine-tuning for the downstream task are small perturbations to refine this prior. So we first assume ∆W is 0 mean matrix-variate Gaussian. To achieve this, we initialize α,β as α ∼ N (0,Σα) and β = 0. Then, to obtain the bound, we only need to focus on estimating the
prior covariance:
Σ = Ep(S) [ (µS − µ)(µS − µ)T ] (14)
As mentioned, the exact estimation of Σ in requires knowledge of the distribution p(S). Inspired by Wang et al. (2021), we can bootstrap (resample with replacement) from the training data S for NS times to construct the bootstrap datasets {Si}NSi=1.
Σ ≈ 1 NS
NS∑
i=1
[ (µS − µSi)(µS − µSi)T ] (15)
However, to get µSi , we still need to optimize over the bootstrap datasets {Si}NSi=1, which is not practical in neural network training. Instead, we approximate this using the group influence effect which studies the change to the model if we remove a group of data.
Lemma 3.1. Group Influence Function (Basu et al., 2020) Assume that when all samples in a group U are up-weighted by ϵ→ 0, the parameter difference can be approximated as:
µS−U − µS ∆= − 1 1− p 1 |S|H −1 µS
∑ z∈U ∇l(hµS (z))
(16)
where S is the training sample set, ∇l(hµS (z)) and Hµ = ∇2µl(hµS (z)) are the gradient and Hessian of the loss function l defined on the model hµS (), and µS refers to the optimal parameters trained by dataset S. By taking U as a subset resampled from the training data S, we can show that the oracle prior covariance in Eq. 15 can be estimated by
Σ ∝ H−1µ ∑
z∈U ∇l(hµ)
∑ z∈U ∇l(hµ)T (H−1µ )T
∝ H−1µ Fµ,U (H−1µ )T ∝ F−1µ,U (17)
where Fµ,U is the Fisher information matrix estimated using the resampled dataset U . See Appendix B for detailed derivation.

heading: 3.3.3 Estimating the bound for regularization
text: Based on Eq. 17, the generalization bound in Eq. 13 can be estimated as:
I(w) = Ep(S) [KL(p(w|S), p(w))] ∝ Ep(S) [ (µS − µ)TFµ(µS − µ) ]
≃ I(w;S) = ∑
t∈U
[ ∆µT∇µlt(µS)
]2 (18)
where I(w) is further approximated by I(w;S) in the third step as the U is resampled from S. The overall objective function is:
min w L(w;S) + λI(w;S) (19)
where L(w;S) = −∑Ni=1 log(p(yi|xi,w) is the likelihood with yi the output labels in S. To utilize this regularization in practice, we can estimate the information every few steps (e.g. at the end of each epoch) and use this as the regularization term until the next estimation of I(w;S).

heading: 4 Experiment
text: We evaluate the performance of RLoRA using RoBERTa-base/RoBERTa-large (Liu et al., 2019) and GPT-2-medium (Radford et al., 2019). Our experiments encompass a diverse range of tasks, including natural language understanding (NLU) and natural language generation (NLG). Specifically, we evaluate the models on the GLUE benchmark (Wang et al., 2018) for RoBERTabase/RoBERTa-large. For evaluation on GPT-2 medium, to facilitate direct comparison, we adopt the experimental setup outlined in the original work of LoRA (Hu et al., 2021). Additional details regarding the datasets used can be found in Appendix C. All the experiments were conducted using the NVIDIA Tesla V100.

heading: 4.1 Baselines
text: • Full Fine-tuning (FT): The model is initialized
with pre-trained weights and all the parameters are trained.
• Adapter (Adpt) (Houlsby et al., 2019): It inserts adapter layers (two-layer fully connected with bias) between the self-attention and the MLP modules.
• BitFit (Zaken et al., 2021): It freezes the pretrained weight matrix and trains the biases.
• Low-Rank Adaptation (LoRA) (Hu et al., 2021): It represents the weight update using a low-rank decomposition.
• Vector-based Random Adaptation (VeRA) (Kopiczko et al., 2024): It replaces the lowrank matrices in LoRA with random matrices and learn only scaling vectors.

heading: 4.2 Natural Language Understanding
text: We first evaluate our approach on the General Language Understanding Evaluation (GLUE) benchmark which consists of multiple tasks including
paraphrase detection (MRPC), sentiment classification (SST-2), natural language inference (RTE, QNLI) and linguistic acceptability (CoLA). Since the original test sets are not publicly available, we use the original validation set as the test set. For estimating the information term in Eq. 18, we randomly resample 100 training data at the end of each epoch for analysis.
Table 1 shows the fine-tuning results on RoBERTa-base and RoBERTa-large using the GLUE benchmark. For RoBERTa-base model, due to the budget limitation, we did not use the MNLI trick2 to fine-tune the MRPC, RTE and STSB. According to the results on RoBERTa-large, RLoRA can achieve the best results on 6 datasets just like LoRA and VeRA. At the same time, similar to VeRA, RLoRA has the number of trainable parameters subtantially reduced by ∼ 13 times compare to LoRA due to the use of the random WHT for the low-rank projection. Additionally, it is worth noting that while RLoRA can achieve comparable performance as LoRA and VeRA on different datasets, its standard deviation is the smallest among them. This indicates that the PAC-Bayes regularization allows RLoRA to be more robust against the effect due to the random seeds and improve the stability of training results under different random seed settings.

heading: 4.2.1 Effectiveness of PAC-Bayes Regularization
text: To further evaluate the effectiveness of incorporating the PAC-Bayes term for enhancing the generalization performance, we compare the few-shot performance among LoRA, VeRA and RLoRA. We need to re-train LoRA and VeRA to evaluate their performance on the few-shot settings. We also include a version of RLoRA without the PAC-Bayes regularization term for the ablation study to underscore the impact of the regularization. We select five tasks from the GLUE benchmark for this fewshot evaluation and train the models with only 100 and 200 data samples. The performance is evaluated on the original evaluation set. As shown in Table 2, we can observe that only utilizing the Randomized WHT can reduce the number of trainable parameters to 0.02% of the original model (1/35 of LoRA) while achieving comparable performance. Furthermore, incorporating the PAC-Bayes regu-
2Hu et al. (2021) used the checkpoint fine-tuned with MNLI task instead of the pre-trained checkpoint to initialize the model for MRPC, RTE and STSB.
larization can further improve the generalization performance indicating that the model manages to capture the task-related information from the input data. Fig.1 shows an overall performance comparison between RLoRA and other baselines for fine-tuning large models given a small training set. The advantage of RLoRA with PAC regularization can be clearly observed.
To illustrate the importance of using the generalization bound as the information measure for regularizing the fine-tuning process instead of using it only for sub-network selection (like the maskbased approach), we perform experiments to monitor the information change based on Eq. 18 in each layer of Roberta-base during the fine-tuning on RTE and MRPC datasets for 10 epochs. We estimate the layer-wise information at the end of each epoch. Fig. 3 show the temporal dynamics of the information flow during the fine-tuning process. We can observe that at different stages of
training, the information “flows” through the layers. This implies that identifying and fixing some sparse masks on the network could be a too restrictive approach for regularizing the entire training process. In particular, a wrongly estimated mask might steer the whole optimatization process astray.

heading: 4.3 NLG Benchmark
text: For the performance of RLoRA on Natural Language Generation (NLG) models, we make use of GPT-2 medium (Radford et al., 2019). To make a direct comparison, we keep our experiment setup as close as possible to the approach in (Hu et al., 2021). Due to the space limit, we will only present our results on the E2E NLG Challenge.
For fine-tuning LoRA and VeRA, we use the values of the hyperparameters provided in their papers and tune the learning rate from{ 5× 10−1, 1× 10−1, 5× 10−2, ..., 5× 10−5 } .
We further search for the best regularization strength λ from 10−4 ∼ 10−6. Detailed hyperparameter settings can be found in Appendix C.
As shown in Table 3, for generating text using GPT-2 medium fine-tuned with E2E datasets, RLoRA as compared to LoRA, VeRA, and Adpt achieves the best or second-best results across most of the metrics and is competitive with full finetuning. Furthermore, by referring to the last rows in Table 3, we can clearly observe the improvement brought by the PAC-Bayes regularization.

heading: 4.4 Arithmetic Reasoning
text: To evaluate the performance comparison on more recent large model, we finetune the LLaMA7B and the corresponding consumption of computational resources.
By following the experiment settings in (Hu et al., 2023), we evaluate the performance of finetuning LLaMA7B model for math reasoning tasks where a math reasoning dataset with 10K training
data is used and 4 different tasks were being evaluated. Results with * in the following table are taken from the original paper.
The results in Table.4 show that RLoRA can attain performance comparable to other baseline methods (Prefix, Adapter and LoRA) with the fewest number of parameters. The last 4 rows reveal that given similar memory requirements, RLoRA exhibits superior performance than LoRA. Also, comparing with LoRA(r=32) and Adapter, RLoRA can achieve comparable performance with 8× smaller space required for storing the model. The more downstream tasks we need to tackle, the more obvious our storage advantage becomes. For 100 tasks, LoRA would require 20G space, while RLoRA needs only 2G storage for similar performance. In addition, the smallest possible model for LoRA is the one with rank=1 which highly likely will not give good adaptation performance. Yet, its size may still restrict it to be applied under some extreme memory resource constraint situations.
To evaluate the reduction in memory utilization, we compared the resource consumption with minimum requirements for training LoRA. For RLoRA, it is much more computationally efficient due to the adoption of Walsh-Hadamard Transform (WHT). Table.5 shows the minimum Flops and CUDA memory usage needed by LoRA and RLoRA. RLoRA consumes less computational resources than LoRA, and we set the batchsize=1 to get this result.

heading: 4.5 Comparison: RLoRA vs VeRA
text: In comparison with the contemporaneous work VeRA which also generates a random projection matrix as well, RLoRA implements the structured Randomized WHT which can make the weight mul-
tiplication more efficient (O(d log d)). Based on our current implementation (not yet optimized), it can enhance the training from processing 14.7 samples per sec. for VeRA to 15.3 samples per sec. for RLoRA 3. Also, the use of Randomized WHT can effectively scale the PAC-Bayes regularization to large models, leading to improved performance evidenced by our experiment results.

heading: 5 Limitations and Future Work
text: In this paper, we proposed Randomized WalshHadamard low-rank adaptation to reduce the finetuning resource consumption and scale the PACBayesian regularization to larger model. The current estimation of the PAC-Bayes generation bound requires additional gradient computation on the resampled data. Better ways to integrate the estimation of the information into the training process is worth pursuing. In addition, how to select a proper rank for low-rank adaptation is another research issue where methods like AdaLoRA (Zhang et al., 2023c) and IncreLoRA (Zhang et al., 2023a) propose different parameter importance scores for the rank selection. This issue is not yet considered in this paper.

heading: 6 Conclusion
text: We propose the randomized low-rank adaptation RLoRA which integrates LoRA with the
3We train the RoBERTa-base on RTE datasets for 1 epoch to get this result.
Randomized Walsh-Hadamard Transform to gain parameter-efficiency for fine-tuning and PACBayes regularization to further boost generalization capability. Via comprehensive experiments, we demonstrates that RLoRA can achieve comparable performance as LoRA where the number of trainable parameters required is only 0.02% of the original model (and 1/35 of LoRA). Also, under few-shot settings, the PAC-Bayes regularization makes RLoRA outperforms LoRA and other SOTA low-rank adaptation methods. For future work, more effective and efficient ways for estimating the information-based regularization is worth pursuing. In addition, how to selecting the optimal rank is another open research issue.

heading: Acknowledgements
text: This research is partially supported by the Research Matching Grant Scheme RMGS2021_8_06 from the Hong Kong Government.

heading: A Matrix-variate Posterior Distribution Approximated by WHT
text: The matrix W̃ = diag(β)S1HB1ST2 diag(α)S2HB2 = WBHB1S T 2 diag(α)S2HB2 in Eq. 8, where WB = diag(β)S1. The covariance matrices are:
U = 1 tr(V ) E [ (W −M)(W −M)T ]
V = 1 tr(U) E [ (W −M)T (W −M)
] (20)
As the covariance matrix is non-identifiable (Glanz and Carvalho, 2018) which means that for any scale factor
MNm×n(W | M ,U ,V ) = MNm×n(W | M , sU , 1sV ), (21)
we can constrain tr(V ) = 1, and M = S1HB1ST2 diag(µβ)S2HB2
U = E [ HB1S T 2 diag(Σ 1/2 α ϵ)S2HB2B2HS T 2 diag(Σ 1/2 α ϵ)S2B1H ]
= E [ HB1S T 2 diag(Σ 1/2 α ϵ)S2S T 2 diag(Σ 1/2 α ϵ)S2B1H ]
= HB1S T 2 E [diag(Σα)]S2B1H
V = 1 tr(U) E [ B2HS T 2 diag(Σ 1/2 α ϵ)S2B1HHB1S T 2 diag(Σ 1/2 α ϵ)S2HB2 ]
= 1
tr(U) B2HS
T 2 E [diag(Σα)]S2HB2
(22)
Then, the root of U ,V can be found
U1/2 = HB1S T 2 diag(Σ 1/2 α ) V 1/2 = 1√ tr(U) B2HS T 2 diag(Σ 1/2 α )
(23)
Lemma A.1. Suppose W ∼ MNm×n(MT ,V ,U). Let D ∈ Rr×m r ≤ m and C ∈ Rn×p, p ≤ n are both full rank matrices. Then a linear transform of the W , i.e., DWC also follows matrix-variate Gaussian distribution.
DWC ∼ MN r×s(DMC,DUDT ,CTV C). (24)
So, U W̃ = diag(β)S1US T 1 diag(β) and VW̃ = V .

heading: B PAC-Bayesian Estimation
text: When Uψ is bootstrapping from S and Ψ is the selection matrix and ψi = 1 means sample point i will be removed. Ψ is the resampled data index and we use g ∈ Rd×n to denotes the gradient matrix.
µS−U − µS ∆= − 1 1− p 1 |S|H −1 µS
∑ z∈U ∇l(hµS (z))
= 1 1− p 1 |S|H −1 µS
n∑
i=1
ψi∇l(hµS (zi))
so µU − µS ∆= 1 1− p 1 |S|H −1 µS
n∑
i=1
(1− ψi)∇l(hµS (zi))
≃ H−1µS g(1−Ψ).
(25)
Then we have
E [ (µU − µS)(µU − µS)T ]
= E [ H−1µS g(1−Ψ)(H−1µS g(1−Ψ))T ] = H−1µS gE [ (1−Ψ)(1−Ψ)T ] gT (H−1µS ) T
(26)
while Ψ ∼ Poisson(k, 1n , ..., 1n),
E [ (1−Ψ)(1−Ψ)T ] ij
=
{ 1− 2 kn + k(1−n) n + ( k n) 2 i = j
1− 2 kn + k 2 n2 i ̸= j
(27)
In practice as n increases, (E [ (1−Ψ)(1−Ψ)T ] ii ) → 2 and (E [ (1−Ψ)(1−Ψ)T ] ij ) → 1. So
E [ (1−Ψ)(1−Ψ)T ] ≃ (I + 11T ) (28)
and Eq. 26 can be rewritten as:
H−1µS gE [ (1−Ψ)(1−Ψ)T ] gT (H−1µS ) T
= H−1µS g(I + 11 T )gT (H−1µS ) T = H−1µS gIg T (H−1µS ) T ∝ H−1µ Fµ,U (H−1µ )T
(29)

heading: C Experiment Details
text: C.1 GLUE full datasets The detailed hyperparameter settings of our proposed RLoRA can be found in Table 6.
C.2 GLUE datasets for few-shot experiments For our few-shot experiments, we make of GLUE and train the model for 20 epoch with batch size = 32 in all experiments. Detailed hyperparameter settings for our proposed RLoRA can be found in Table 7.
C.3 E2E benchmark The hyperparameter to reproduce the results for the E2E benchmark is shown in Table 8.

heading: Future Work
text: For future work, more effective and efficient ways for estimating the information-based regularization is worth pursuing. In addition, how to selecting the optimal rank is another open research issue