Efficient Learning with Sine-Activated Low-Rank Matrices
Authors: Yiping Ji
Hemanth Saratchandran
Cameron Gordon
Zeyu Zhang
Simon Lucey
Venue: ICLR 2025 Poster (2025)
Keywords: parameter efficient learning
low rank matrices
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We mathematically derive a new approach to parameter efficient learning based on low-rank matrices and sine non-linearities.

Abstract: Low-rank decomposition has emerged as a vital tool for enhancing parameter efficiency in neural network architectures, gaining traction across diverse applications in machine learning. These techniques significantly lower the number of parameters, striking a balance between compactness and performance. However, a common challenge has been the compromise between parameter efficiency and the accuracy of the model, where reduced parameters often lead to diminished accuracy compared to their full-rank counterparts. In this work, we propose a novel theoretical framework that integrates a sinusoidal function within the low-rank decomposition process. This approach not only preserves the benefits of the parameter efficiency characteristic of low-rank methods but also increases the decomposition's rank, thereby enhancing model performance. Our method proves to be a plug in enhancement for existing low-rank models, as evidenced by its successful application in Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF) and 3D shape modelling.

Published as a conference paper at ICLR 2025
EFFICIENT LEARNING WITH SINE-ACTIVATED LOW-RANK MATRICES
Yiping Ji +
Hemanth Saratchandran'
Australian Institute for Machine Learning University of Adelaide DATA61, CSIRO
Australian Institute for Machine Learning University of Adelaide
Cameron Gordon
Zeyu Zhang
Australian Institute for Machine Learning University of Adelaide
Australian National University
Simon Lucey Australian Institute for Machine Learning University of Adelaide
ABSTRACT
Low-rank decomposition has emerged as a vital tool for enhancing parameter ef- ficiency in neural network architectures, gaining traction across diverse applica- tions in machine learning. These techniques significantly lower the number parameters, striking balance between compactness and performance. However, a common challenge has been the compromise between parameter efficiency and the accuracy of the model, where reduced parameters often lead to diminished accuracy compared to their full-rank counterparts. In this work, we propose a novel theoretical framework that integrates sinusoidal function within the low- rank decomposition. This approach not only preserves the benefits of the pa- rameter efficiency of low-rank methods but also increases the decomposition' 's rank, thereby enhancing model performance. Our method proves to be plug-in enhancement for existing low-rank methods, as evidenced by successful ap- plication in Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF) and 3D shape modelling. The code is publicly available at tettts://yiiinge hetpii.ggiitee github. io/ ane__ativated__Pa
INTRODUCTION
the last few years, large-scale machine learning models have shown remarkable capabilities across various domains, achieving groundbreaking results in tasks related to computer vision and natural language processing (Vaswani et al., 2017; Dosovitskiy et al., 2021). However, these models come with a significant drawback: their training necessitates extensive memory footprint. This chal- lenge has spurred the demand for more compact, parameter-efficien architectures. A prominent solution that has emerged is the of low-rank techniques (Hu et al., 2022; Chen et al., 2024; Liu et al. 2024; Kopiezko et al., 2024), which involve substituting the large, dense matrices in large scale models with smaller, low-rank matrices. This substitution shifts the computational complexity from quadratic to linear, making a significant impact on effi- ciency. In the context of sighhcapectty models like Vision Transformers (ViTs) and Large Language Models (LLMs) that utilize millions to billions of parameters, transitioning from dense to low-rank matrices can result in considerable cost savings. Nonetheless, adopting low-rank architectures does introduce trade-off, they typically do not achieve the same level of accuracy as their full-rank counterparts, presenting& a balance between parameter efficiency and model performance.
only simplifies the models but also
* Equal contribution. Correspondence to Yiping Ji Syippingi@adelliiee..d and Hemanth Saratchan-
dran <hemanth. www.:nntw.aaaaaaannnnaaaaaao www.rraannaaeeaaae.cc.cc...

---

Published as a conference paper at ICLR 2025
In this paper, tackle the challenge of balancing parameter efficiency and model performance by introducing a novel technique that enhances the representational capacity low-rank meth- ods. Our approach builds the insight that augmenting low-rank matrices with high- high-frequency sinusoidal functions can increase their rank without adding parameters. We provide a theoretical framework that explains how this modulation increases rank and demonstrate how incorporating this non-linearity into low-rank decompositions enables compact architectures that preserve efficiency while achieving good accuracy across various machine learning tasks.
We direct the reader's attention to figure 1, which showcases our method across various machine learning applications. Our comparisons with standard low-rank methods consistently demonstrate superior performance across these diverse tasks.
Low-Rank
Sine Low-Rank
Low-Rank
Sine Low-Rank
PSNR: 19.81
PSNR 23. 27
IoU:
IoU: 90.8
84.3
(a) NeRF (k=5), 4.7% Full Rank Params
(b) 3D Occupancy (k=1), 2.1% Full Rank Params
Percentage Improvement (%)
Top-1 Accuracy (%)
78
8
Low-Rank
Sine Low-Rank
4
74-
Full-Rand (88M)
30
35
40
45
50
Parameters (M)
WinoGrendd
ARC-e
ARC-c
OBQA
(c) ViT Classification
(d) LoRA (k=32)
Figure 1: Applying drop-in sinc-activation increases the rank of low-rank matrix methods, leading to improved parameter efficiency and performance on a variety tasks including: a) NeRF, b) 3D Occupancy, ViT image classification, and d) Fine-tuning Large Language Models (LoRA).
Our approach's advantages corroborated across range of machine learning applications, includ- ing low rank methods for ViT (He et al., 2022b), LLMs (Hu et al., 2022; Liu al., 2024), NeRF for novel view synthesis (Mildenhall et al., 2020), and 3D shape modeling via Binary Occupancy Fields (Mescheder et al., 2019). Across the board, our approach not only matches the parameter savings offered by low-rank methods but also results in an improvement accuracy, showing its broad ap- plicability and superior performance among diverse machine learning tasks. The main contributions of our paper are:
1. A theoretical framework demonstrating that applying sinusoidal non-linearities to low-rank matrices can effectively increase their rank without introducing additional parameters.
2. Demonstrating that our theoretical framework leads to a drop-in component that can be applied to various low-rank architectures, resulting improved accuracy while maintaining computational and parameter efficiency.
A comprehensive validation of our method across a range of diverse machine learning tasks, including computer vision, 3D shape modeling, and natural language processing.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 LOW-RANK DECOMPOSITION:
Low-rank decomposition stands crucial method across disciplines such as information theory, optimization, and machine learning, providing an approach to reduce memory costs (Strang, 2019; Xinwei et al., 2023). Notably, (Candès et al., 2011) uncovered that matrices can precisely separate low-rank and sparse components through convex programming, linking to matrix completion and recovery. Expanding its application, (Yu et al., 2017) devised low-rank learning framework for convolutional neural networks, enhancing compression while maintaining accuracy. (Sharma et al., 2023) found that performance improvements in large language models could be achieved by elimi- nating higher-order weight matrix components without extra parameters or data. In neural radiance fields, (Tang al., 2022) introduced a rank-residual learning strategy for optimal low-rank approx- imations, facilitating model size adjustments. Additional contributions include (Shi & Guillemot, 2023) with cank-constrained distillation, (Chen et al., 2022) applying vector-matie decomposition, and (Schwarz et al., 2023) using soft-gated low-rank decompositions for compression.
2.2 PARAMETER-EFFICEE LEARNING:
Parameter-efficien learning an important research area in deep learning, merging various tech- niques to enhance model adaptability with minimal resource demands (Menghani, 2023). Tech- niques like parameter-efficient fine-tuning (PEFT) allow pretrained models to adjust to new tasks efficiently, addressing the challenges of fine-tuning large models due to high hardware and stor- age costs. Among these, Visual Prompt Tuning (VPT) stands out for its minimal parameter alter- ation-less than I 1%-in the input space, effectively refining large Transformer models while keep- ing the core architecture unchanged (Jia et al., 2022). Similarly, BitFit offers sparse-finetuning approach, tweaking only the model's bias terms for cost-effective adaptations (Zaken et al., 2022). Moreover, LoRA introduces low-rank adaptation that maintains model quality without additional inference latency or altering input sequence lengths, by embedding trainable rank matrices within the Transformer layers (Hu et al., 2022). Recent studies also combine LoRA with other efficiency strate- gies like quantization, pruning, and random projections for further model compression (Dettmers et al., 2024; Li et al., 2024; Zhang et al., 2024; Kopiczko al., 2024; Liu et al., 2024).
3 METHODOLOGY
We introduce our main technique that we term a sine-activan activated low-rank approximation. The purpose of this technique is to increase the rank of a low-rank matrix without adding any extra parameters.
3.1 PRELIMINARIES
3.1.1 FRRD-FORWARD LAYER
Our technique is defined for feed-forward layers of a neural architecture. In this section, we fix the notation for such layers following Prince (2023). We express feed-forward layer
y = Wx where W E R is dense weight matrix, b b is the bias of the layer, and x is the input from the previous layer. The output y is often activated by a non-linearity a producing oly). The weight matrix W and bias b are trainable parameters of the layer. In contemporary deep learning models, the feed-forward layers' weight matrices, W, are often large and dense yielding a high rank matrix. While the high-rank property of the weight matrix helps in representing complex signals, it significantly adds to the overall parameter count within the network yielding the need for a trade-off between the rank of the weight matrix and overall architecture capacity.
(I)
3.1.2 LOW-RANK DECOMPOSITION
A full-rank weight matrix W can be replaced by low-rank matrices UVT, such that W = UVT, where U E gmak V B and k min(m, n²)
y Wx b (UVT) xb
(2)
3

---

Published as a conference paper at ICLR 2025
This is the most common way reduce the parameter count in feed-forward layer. During the training process, this method performs optimization on and V alternatively, Low-rank multiplica- tion then reduces the learnable parameter count and memory footprint from ((m) to ((k-(((+)))) Although UVT has the same matrix shape as the full-rank matrix W, the rank of UV? con- strained and rank (uV( S k. Thus while we have significantly decreased the number of trainable weights in such a layer, we have paid the price by obtaining a matrix of much smaller rank. In the next section, we address this trade-off by developing a technique that can raise back the rank of low-rank decomposition while keeping its low parameter count.
3.2 THEORETICAL FRAMEWORK
3.2.1 NON-LINEAR LOW-RANK DECOMPOSITION
We introduce a non-linearity transformation into low-rank matrices as follows
= o(w UVT
x x+b
(3)
where <(-) is the non-linearity function, is non-learnable frequency parameter, and is non- learnable parameter to control the gain of the transformation. Our theoretical work will show that ((wx) (wxz) is optimal choice to increase the rank of UVT.
3.2.2 MAIN RESULT
In this section, we provide a theoretical framework that clearly shows how increase the rank of a low-rank decomposition using non-linearity without adding any parameters. We will show that if we choose the non-linearity, in the decomposition defined section 3.2.1, to be a sine function then provided the frequency is chosen high enough, the rank of the matrix 6(w UVT) will be larger than that of UV The proofs of the theorems are given in the appendix A.1.
To begin with, we fix w > and let sin(w A) denote the matrix obtained from a fixed m x n matrix A by applying the function sin(w x) component- sisse A. Assuming A + 0 define A min as:
A
min
Aggl-
(4)
JJ A.i.a
Note that such quantity is well defined precisely because A has finite number of entries and all such entries cannot be zero from the assumption that + 0.
The following theorem relates the rank of sin(w A) to the frequency w and the quantity A
Proposition 1. Fix an m x n matrix A s.f. A + 0. Then
4
Rank(sin(w A)) 2 w
min
if
0 < w S
π
(5)
|
3.40 m.
Proposition 1 shows that if we modulate the matrix sin A) by increasing > 0 then the rank of the matrix sin(w A) can be increased provided
We can apply proposition the context of low-rank decomposition as defined in section 3.1.2. Given a low-rank decomposition UVT with U e pm and V E guxk with k & min(m, n) the following theorem shows how we can increase the rank of the decomposition by applying a sin(w-) function.
3.4 miin.
mam and V E x with l & min(m, n). Assume both U and V are initialized according to uniform distribution U(-1/N, 1/N) where N >> k. Then there exists an we such that the matrix sin(w A) will satisfy the inequality
Theorem 1. Let
Rank(sin(w UVT)) > (an
(6)
provided 2 wo-
4

---

Published as a conference paper at ICLR 2025
Theorem also holds for the case initialize U and V by normal distribution of variance N.
Weight matrices within forward layers are typically initialized using a distribution that is con- tingent upon the layer's s neuron count. When considering low-rank decompositions characterized by (m, n),, the variance of this initialization significantly larger than k; ensuring
matrices Rm and V E R**", where kt < m
distribution is influenced by and n. These dimensions that the condition specified in theorem 1 - that > k - is always met, making this theorem espe- cially relevant for low-rank decompositions in feed-forward layers. For example the most common initialization schemes such as Kaiming (He et al., 2015) and Xavier (Glorot & Bengio, 2010) satisfy the requirements of our theorem.
Theorem offers a viable strategy for maintaining a high-rank characteristic in feed-forwand forward layers while simultaneously minimizing the parameter count. By introducing sinusoidal non-linearitty with sufficiently high frequency w into low-rank decomposition, it's possible to increase the rank of the layer without altering the quantity of trainable parameters.
Sine Low-Rank
Sine Low-Rank
Full-Rank Wfr
Low-Rank Wir UVT
Winn sin(100 UVT) Winntrr sin (2000 UVT)
Figure 2: These figures display weight magnitudes for matrices with dimension 128 x 128. The first figure shows heatmap of full-rank matrix initialized by Kaiming uniform, highlighting lin- independence among rows. The second shows low-rank matrix W UVT R 128 with U, V E IR 128x1 illustrating minimal linear independence. The final pair of figures reveal how ap- plying a sine function element-wise, sin(w UV*), ). with varying w, affects linear independence in low-rank matrices; specifically, 100 and 2000 progressively increase linear independence.
In figure 2 we give a visualization of our method in action. We consider a full-rank matrix, a low- rank matrix, and two sine-actie antevactieted low-rank matrices with different frequencies. By visualizing the weight magnitudes in each matrix via a heatmap, we can clearly see how the sine-activated low-rank matrix increases rank and furthermore how increasing the frequency of the sine function increases the rank in accord with theorem 1.
Building upon equation 3, we explore the application of various non-linear functions low-rank decomposition, with particular focus on the sine function. This choice is inspired by theorem 1, which theoretically demonstrates that applying a sine function effectively increases the matrix rank. In figure 3, we present a comparative analysis of the sine function against other common non- non-linear functions machine learning, such as the sigmoid and ReL.U. The results clearly show that the sine function increases the rank, making it an optimal non-linearity apply to a low-rank decomposition.
Further, theorem suggests that augmenting the frequency the sine function applied low-rank decomposition contributes to further increase in rank. To empirically validate this, conducted experiments applying sine functions of various frequencies to a constant low-rank matrix. The outcomes, depicted in figure 3 (right), corroborate the theorem's 's prediction, showcasing positive correlation between the frequency the sine function and the resultant rank increase.
EXPERIMENTS
This section is dedicated to validating and analyzing the efficacy of our proposed low-rank meth- ods across a spectrum of neural network architectures. To demonstrate the broad applicability and versatility of our approach, examine its performance in three distinct contemporary applications.
5

---

Published as a conference paper at ICLR 2025
1.0
1.0
Full-Rank
Full-Rank
Low-Rank
Sine LR 100
0.8
0.8
Singular Values
Sine LR
Singular Values
Sine LR 2020
Sigmoid LR
Sine LR 40=00
0.6
0.6
ReLU LR
0.4
0.4
0.2
0.2
0.0
0.0
0
100
200
0
100
200
Index
Index
Figure 3: In this figure we depict the singular value spectrum of Kaiming uniform initialized matrix Wr E R256x250 256 and low-rank k 5 approximation matrix Wi, UVT. All singular values are normalized to Left: the spectral advantages of applying a non-linear function o(w - UVT) where is hyper-parameter. Here we see the natural advantages of the sine function such that 6(x) = sin(w-x) Right: manipulating w within the sine function changes these spectral properties.
Specifically, we explore its integration into the fine-tuning of large language models through LoRA (Hu al., 2022), the pretraining of ViT (Dosovitskiy et al., 2021), the reconstruction of scenes using NeRF (Mildenhall et al., 2020). and 3D shape modeling (Mescheder et al., 2019). This collectively underscores our model's adaptability potential to significantly impact various domains within the field of computer vision.
diverse array of low-rank frameworks, highlighting its
4.1 LARGE LANGUAGE MODEL
LoRA is a highly effective strategy for finetuning large pretrained models, as described in (Hu et al., 2022). LoRA targets the adaptation of pretrained weight matrices Wo mmx" by limiting updates to low-rank representation, expressed as Wox AWx Wox UV where U e R= and E x with the rank & < min(m, n}. This method does not introduce additional infer- ence latency or necessitate reducing the input sequence length, thus preserving the model quality. We conduct thorough experiments to evaluate the performance of our novel approach, termed sine LoRA, against the standard LoRA framework, demonstrating the effectiveness of our method.
Dataset. We evaluate the natural language understanding (NLU) task performance on the RoBERTa V3 base model (Reimers & Gurevych, 2019). Specifically, adopt the widely recognized GLUE benchmark (Wang et al., 2018), including CoLA (Warstadt et al., 2018), MRPC (Dolan & Brockett, 2005), QQP, STS-B(Cer et al., 2017), MNLI (Williams al., 2018), QNLI (Rajpurkar al., 2016), and RTE (Dagan et al., 2006; Haim et al., 2006; Giampiccolo et al., 2007: Bentivogli et al., 2009). Furthermore, we evaluate sine LoRA by fine-tuning large scale language models LLaMA 3-8B on commonsense reasoning tasks, which includes BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2019), SIQA (Sap et al., 2019), HellaSwag (HS) (Zellers et al., 2019), WinoGrande (WG) (Sakaguchi al., 2021), ARC-c, ARC-e (Clark et al., 2018) and OBQA (Mibaylov et al., 2018).
Setting. In the Transformer architecture, there are four weight matrices in the self-attention module (W.. q. Wk. W., W.) and two in the MLP module(W up: W down).. To evaluate RoBERTA V3, follow up the LoRA architecture and implement low-rank adaptation only on W, and W,. We study the performance of LoRA and sine LoRA in terms of different rank h = 2, 4. 8. To evaluate LLaMA 3-8B, we implement low low-rank adaptations on five modules (W g. WR. W., W.p. with different rank k 8, 16, 32. For further implementation details see Appendix A.2.1.
up: Waewn)
Results: We replicated the experimental framework of naive LoRA to establish a baseline, and then evaluated our sine LoRA, as detailed in table and table 2. Our results reveal that sine LoRA

---

Published as a conference paper at ICLR 2025
Table 1: Performance and parameter count of the RoBERTa V3 model fine-tuned using the LoRA and sine LoRA methods across varying himxx settings on the GLUE benchmark.
Method
Params
COLA
MRPC
STSB
SST2
RTE
QNLI
MNLI
QQP
Avg.
A
LoRAs-1
36.9K
66.31
90.15
90.15
94.70
78.80
93,06
88.18
87.61
85.63
0.32
Sine LoRAs=1
67.99
90.44
90.85
94.79
78.05
92.76
88.35
87.90
85.95
LoRAs-2
73.7K
68.38
89.42
89.19
95.02
78.27
93,32
89.15
88.57
85
0.45
[
Sine LoRAi-2
68.93
90.79
90.94
94.81
79.10
93.29
88.26
88.70
86.44
LoRA
147.5K
68.56
89.69
88.79
95.23
80.39
93.34
89.78
88.70
86.41
0.73 1
Sine LoRAi=4
68.93
90.86
90.87
95.25
82.00
93.53
89.68
89.18
87.14
LoRA
294.9K
68.62
89.82
89.50
95.25
80.37
93.56
89,86
88.83
86.57
0.42 1
Sine LoRAc-8
68.54
90.22
90.85
95.11
81.82
93.58
89.69
89.38
86,99
Table 2: Performance and parameter count of the LLaMA 3-8B model fine-tuned using the LoRA and sine LoRA methods across varying kimax settings on the commonsense reasoning benchmark.
Method
Params
BoolQ
PIQA
SIQA
118
WG
ARC-e
ARC-c
OBQA
Avg.
LoRAk=I
7.1M
73.58
86.29
79.99
94.92
79.95
63.91
78.7
83
80.04
3.57 [
Sine LoRAE=1
72.69
87.38
79.32
94.39
85.32
75.01
88.64
86.2
83.61
LoRA,
14.2M
72.97
87.43
78.81
72.18
85.80
77.47
88.38
83,20
80.79
2.87 [
Sine LoRAc-s
73.42
86.51
80.3
94.16
85.87
76.36
88.05
84.6
83.66
LoRAA=10
Sine LoRAc-16
28.3M
73.57
73.7
87.65
85.58
79.27
80.76
93.97
94.93
85.71
84.45
75.42
79.1
86,44
89.77
83.2
84.4
84.35
82.9
2.45
[
LoRAk-52
56.6M
70.64
86.13
78.25
91.48
83.19
69.71
85.73
81.4
80.82
2.74 [
Sinc LoRAc=32
72.42
86.51
79.78
93.96
85.16
78.07
87.58
85
83.56
sistently surpasses the performance of the standard LoRA at different rank levels (k), highlighting the effectiveness of the sine function in enhancing the representation capabilities of low-rank matri- ces. Notably, sine LoRA at k 4 not only exceeds LoRA's performance at k = 8 by 0.57 but also halves the parameter count, illustrating significant efficiency and parameter savings. Surprisingly, with the LLaMA 3-8B model, our method with rank 4 already outperforms LoRA with all higher ranks, achieving 83.61 compared to 82.9.
Analysis. Within the LoRA framework, featuring low-rank multiplication component AW UVT, we enhance this low-rank component with a sine function and assess the efficacy of our method. This adaptation amplifies the update significance due the 'intrinsic rank' increase, facil- itated by the sine-activation. Consequently, our approach attains superior performance at reduced rank levels compared to LoRA, effectively decreasing the count of learnable parameters.
Further results. For comparisons to the recent DORA paper (Liu et al., 2024) see Appendix A.2.1.
4.2 PRETRAINING VISION TRANSFORMERS
Vision Transformers have risen to prominence as powerful models in the field of computer vision, demonstrating remarkable performance across a variety of tasks. When pretrained on large-scale datasets such as ImageNet-2IK and JFT-300M, ViTs serve as robust foundational architectures, particularly excelling feature extraction tasks (Deng et al., 2009; Sun et al., 2017). A critical observation regarding the architecture of ViTs is that the two feed-forward layers in each block dedicated to channel mixing contribute nearly 66% of the total model parameter count. In light of this, focused experiments on these specific layers have been conducted to rigorously assess the effectiveness of our proposed method, facilitating a direct comparison with the baseline model.
Experimental setup. We trained the ViT-Small and ViT-Base models from scratch, utilizing the CIFAR- 100 and ImageNet- 1k datasets, respectively, to establish our baseline performance metrics (Deng et al., 2009; Krizhevsky, 2012). The ViT-Small model, characterized by its two MLP layers with input/output dimensions of 384 and hidden dimensions of 1536, was modified by replacing the full-rank weight matrices with low-rank matrices across range of ranks (k). Similarly, the ViT-Base model, which features two MLP layers with input/output dimensions of 768 and hidden dimensions of 3072, underwent a parallel modification, where its full-rank weight matrices were
7

---

Published as a conference paper at ICLR 2025
Top-1 Accuracy (%)
80
Top-1 Accuracy (%)
78
Low-Rank
Low-Rank
Sine Low-Rank
Sine Low-Rank
74
50
Full-Rank (88M)
Full-Rank (22M)
45
30
35
40
45
50
7.5
8.0
8.5
9.0
9.5
10.0
Parameters (M)
Parameters (M)
ViT-Base
(h) ViT-Small
Figure 4: Low Rank VIT classification performance. Use of the sine-activation improves perfor- mance of the low-rank models, and even enables improvement relative the Full Rank model.
substituted with low-rank matrices for range of ranks. For the training of the ViT-Base model, we follow the training methodology described in Masked Autoencoders (MAE) (He et al., 2022a), implementing a batch size of 1024. This structured approach allows us to rigorously evaluate the impact of introducing low-rank matrices to these model architectures. For further implementation details see Appendix A.2.2.
Results. Figure 4 shows the outcomes of training ViT models from scratch on the ImageNet-Ik lk and CIFAR100 datasets, respectively. These findings are compared with those of conventional baseline training of ViT models, which demonstrate that employing aggressive low-rank levels (k) compro- mises accuracy. Remarkably, the ViT-Base model, even when operating at rank of 250 with only 50% of its parameters in comparison the baseline, attains the performance metrics of the baseline on the ImageNet-1k k dataset, albeit at the cost of increased training loss. The use of sine low-rank matrices consistently yields substantial improvements test accuracy across all examined rank lev- els for both datasets. This suggests that the sine function significantly bolsters the representational capacity of low-rank weight matrices, as suggested by the theory in section 3.2.
Analysis. Large models, such as ViT-Base, with an excessively large number of parameters, are prone to overfitting, where they perform well on training data but poorly on unseen data, especially when trained on relatively 'small' datasets like ImageNet- (Xu et al., 2024). Low-rank learning techniques can help designing models that generalize better to new data by encouraging the model to learn more compact and generalizable representations to reduce overfitting. Additionally, while VIT architectures often underperform on smaller datasets, this method introduces a novel approach for efficiently training VIT models using small data collections. For frequency ablations in the rank case, see Appendix A.2.2.
ConvNeXt. order to show that our method works on convolutional only ar- chitectures we implemented sine low-rank ConvNeXt (Liu et al., 2022), leading convolutional architecture for image classification. For implementation details and results see Appendix A.2.3.
Further results:
4.3 NERF
Neural Radiance Fields (NeRFs) represent 3D scene signals by utilizing a set of 2D sparse images (Mildenhall al., 2020). The 3D reconstruction obtained by a forward pass fo y. z, 0 Φ), volving position (z,, z) and viewing direction (0, Φ). We evaluate our methods by training NeRF model on the standard benchmarks L.L.F dataset, which consists
real- hal-world scenes captured by hand-held cameras (Mildenhall et al., 2019). To evaluate our method on NeRF substitute each fully dense layer with low-rank decomposition and use range rank levels (k).
Table 3 provides results the LLFF dataset (Mildenhall et al., 2019; 2020). We report the peak signal-to-non noise ratio (PSNR) with the compression rate representing the percentage of pa- rameters used in comparison to the parameter count of the Full Rank NeRF model. Employing low- rank matrices in NeRF learning reduces parameter count while significantly enhancing compression. However, performance dips with very low-rank levels (k), where models capture minimal informa-
Results.
8

---

Published as a conference paper at ICLR 2025
Table
Quantitative results for NeRF evaluated on the LLFF dataset.
PSNR1
Compression
Femn
Flower
Fortress
Horn
Leaves
Orchids
Room
Trex
Average
Rate
Full-Rank
26.38
27.54
30.93
28.20
21.79
21.33
30.96
27.68
26.85
100%
Low-Ranki-
15.03
14.60
14.74
13.66
12.89
12.50
15.04
13.54
14.00
5.77
1.3%
Sine 1
20.77
20.14
24.13
19.00
15.92
16.25
25.53
16.42
19.77
Low-Ranke Low-Rankt=s
20.64
19.81
24.90
20.40
15.74
16.07
22.74
19.79
20.01
Sine Low-Ranke-a
23.50
23.27
26.78
23.99
18.49
18.90
27.05
22.96
23.11
3.10
4.7%
Low-Ranka Ranke=to
22.83
22.18
25.96
22.76
17.36
18.12
26.12
21.69
22.12
2.27
8.7%
Sine Low- 1o
24.56
24.61
28.01
25.39
19.62
20.02
28.70
24.21
24.39
Low- 2aw-Rank;-30
24.48
24.68
28. 10
25.54
19.36
20.04
38.92
24.24
24.42 25.86
1.45
24.6%
Sine Low-Rande 300
25.71
26.01
29.46
27.16
20.95
21.17
30.18
26.27
Low- Low-Rankg_.d
25.26
26.16
29.50
26.74 74
20.39
20.85
30.00
25.81
25.59
0.77
48.6%
Sine
26.09
26.70
29.75
27.78
21.56
21.37
30.54
27.16
26.36
tion. Our methods, nevertheless, substantially elevate performance. For instance, with k 1. our sine low-rank approach yields an average PSNR of 19.77, outperforming the naive low-rank by 5.77 and achieving a compression rate of merely 1.3%. Even at a 48% compression rate, it surpasses the basic low-rank model by 0.77 PSNR, narrowly trailing the baseline by just 0.49 PSNR, as shown in Figure 5b. Our rate-distortion analysis, applying Akima interpolation for Bjøntegaard Delta calcu- 64.72% and BD-PSNR of 2.72dB, signifying marked improvements
lation, reveals a BD-Rate in compression efficiency (Bjontegaard, 2001; Herglotz al., 2022). Visualization for k results are shown in Figure 5a and more results are shown in Figure 8 in the Appendix.
27.5
Low-Ramt Rank
25.0
22.5
PSNR
20.0
Sine Low-Rank
Sine Low-Rank Rank
17.5
Low- Baw-Rank
15.0
Hank-
20
Compression Rate (%)
(a) Qualitative NeRF results LLFF datasets 1).
(b) Rate- Distortion curve (LLFF average).
Figure 5: (a) Using a non-transformed Low-Rank model leads to complete loss of signal at treme (rank = 1). In contrast, applying a sine- activation function able to reconstruct details 1.3% of the Full-Rank parameters. (b) The Sine Low-Rank NeRF models show significant improvements across the rate-distortion curve relative to the Low-Rank models.
Analysis. NeRF models fit entire 3D scenes, and high training PSNR leads to high testing PSNR (Mildenhall et al., 2020). Employing structured weight matrices could result in a drop in perfor- mance due to the inherent constraints imposed by their structural design. Increasing the matrices' rank enhances their memorization abilities significantly, especially when using a very low k. Start- ing from low frequency, there is rapid and consistent increase in PSNR. Consequently, elevate the rank level k our results gradually align with the baseline NeRFs, which serve as the upper bound. For frequency ablations in the case of rank I and rank see Appendix A.2.4.

---

Published as a conference paper at ICLR 2025
4.4 3D SHAPE MODELING
For this experiment, we evaluate performance on binary occupancy field reconstruction, which in- volves determining whether a given coordinate is occupied (Mescheder al., 2019). Following (Saragadam et al., 2023), we sampled over a 512 x 512 512 grid with each voxel within the vol- ume assigned and voxels outside the volume assigned a We use the Thai Statue, Dragon and Lucy instance from the Stanford Scanning Repository. We evaluate intersection over union (IoU) for the occupancy volumes. We used coordinate- based MLP that includes two hidden layers, each with width of 256 neurons, and employed the Gaussian activation function. The full-rank model achieves an accuracy of 97 (IoU). Figure 6 shows the 3D mesh representation of the Thai Statue, visualized using the low-rank method and the sine low-rank method for k = 5. Applying the sine function to the low-rank matrix resulted in a significant enhancement and more precise shape delineation. Results Dragon and Lucy are given in Table 13 in Appendix A.2.5. A frequency ablation in the rank case is given in Appendix A.2.5.
k=1
k=2
k=5
Rank Sire Low Lawk (Ouss) (IoU: 90.8)
Bank (IoU: 88)
Sine Pank (Ours) (loU: 92)
Low Rank (loU: 96.4)
Sine Low-Rank (Ours) (IoU: 91.4)
(loU: 84.3)
Figure 6: Binary occupancy field reconstruction the Thai Statue. Note that without sine func- tion, the low-rank model is unable reconstruct any finer details for the k = 1 case; however, even at that level the sine low-rank model is able reconstruct fine structural details of the statue, including the trunks of the elephants. The k = 1, k 2 and = 5 model utilizes only 2.1% 2.9% and 5.2%, respectively, of the parameters of the full-rank model.
LIMITATIONS
Our exploration into sine low-rank matrices illuminates their promising capabilities, yet it also has limitation: notably, while these matrices can reach rank levels comparable to their full-rank counter- parts upon the application of sine function, their accuracy falls short. This highlights ongoing challenge in finding the optimal balance between the need for sufficient parameterization to ensure high accuracy and the preferable rank of matrices. Overparameteriizttin is widely recognized in the literature as vital for deep learning models to achieve strong generalization and memorization. Mov- ing forward, developing strategies that not only increase rank but also clearly define the necessary degree of overparameterization will be crucial for creating cost-effective deep learning architectures, presenting intriguing avenue for future research.
CONCLUSION
In this work we have demonstrated that applying a sinusoidal non-linearity improves the accuracy of low-rank approximations by increasing their rank. While simple, this method is highly applicable to parameter-constrainee models such as LoRA, as improves approximation without adding capacity, by overcoming representation limits of the matrix rank. We have fully justified this approach from theoretical first principles. When applied as a drop-in component we showed that this method leads surprisingly large improvements across range of tasks involving low-rank models, including language tasks, image classification, neural radiance fields and 3D shape modelling.
Available https::/ https://graph wtw.apaoi.sstt.ood..oooomm
stanford. edu/date /3pcaan
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Hemanth Saratchandran and Simon Lucey acknowledge support from Commonwealth Bank of Aus- tralia through the CommBank Centre for Foundational Al Research. This funding was essential for the completion of the research described in this publication.
REFERENCES
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. TAC, 7(8):1, 2009.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language, 2019. URL
1911.11641.
Gisle Bjøntegaard. Calculation of average PSNR differences between RD-curves. Technical report, VCEG-M33, Austin, TX, USA, April 2001.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis?
Journal of the ACM (JACM), 58(3):1-33 2011.
Daniel Cer. Mona Diab, Eneko Agirre, Iñiig Lopez-Gazpio, and Lucia Specia. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Ju- rgens (eds.), Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval- 2017), pp. 1-14, Vancouver, Canada, August 2017, Association for Computational Linguisties. URL https: /eeee//alcllaa aclanthologgyy org/ 2001.
doi: (1188653/1/S11-220
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance fields. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pp. 333-350, Berlin, Heidelberg, 2022.
Aochuan Chen, Ziqi Gao, Zijing Liu, Yu Li, and Jia Li. Parameter-efficient fine-tuning via circular convolution. arXiv preprint arXiv:2407. 240..1944, 2024.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905. 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning chal- lenge. arXiv preprint urXiv:1803.05457 2018
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. Machine learning challenges. evaluating predictive uncertainty, visual object clas- sification, and recognising tectual entailment, pp. 177-190. Springer, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 148-255 2009, doi: 10.1 1091109PP.20090......
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems, 36, 2024.
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models, 2023. URL https: //arxvv. org/abs/
aaaaa////11111 22111 1111..169966 11696.
William Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases.. In Proceedings of the International Workshop on Paraphrasing, 2005.

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbor,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICIRR, 2021.
Danilo Giampiccolo. Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACI-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9. Association for Computational Linguisties, 2007.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirfeenth International Conference on Artificial Intelligence and Statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume pp. 785- 794, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015. doi: 10.100//CC222015.12..
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16000-16009, 2022a.
Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter-efficient model adaptation for vision transformers. arXiv preprint arXiv:2203. 16329, 2022b.
Christian Herglotz, Matthias Kränzler, Rubens Mons, and André Kaup. Beyond Bjøntegaard: Limits of video compression performance comparisons. In International Conference on Image Process- ing (ICIP), 2022. doi: 10.1 1119//ccpp4657722222999999999999
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. VeRA: Vector-based random matrix adaptation. In International Conference Learning Representations, 2024.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoftQ: LoRA-fine-tur fine-tuning----- aware quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL httph::s//ppeeeiiii..ccooo net/ fent//forumiien wtw.:miiidii...i.dee 4
Shih- Yang Liu, Chien-Ye Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09355 2024.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s, 2022. URL pestts://arxivv wwtp:/oo///aaaaaae 03545.
Gaurav Menghani. Efficient deep learning: A survey making deep learning models smaller, faster, and better. ACM Computing Surveys, 55(12), mar 2023. ISSN 0360-0300. doi: 10.1 145/3573878898 URL https::////. 115.l...0g011.111/33788899999
12

---

Published as a conference paper at ICLR 2025
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. Proceedings IEEE Conf. Computer Viston and Pattern Recognition (CVPR), 2019.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 1exXiv:809.02000 2018.
Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In 2020.
Simon JD Prince. Understanding deep learning. MIT press, 2023.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pp. 2383-2392. Association for Computational Linguistics, 2016.
Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards a unifying framework for activations in coordinate-mlps, 2022. URL https: arxiv. orgg/bs//////eoooeeo...ooe..eeee /aaa/b///1111 22112111.1111 11111111111 15135.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http::/arxiv..rr abs/ 10084.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adver- sarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Maarten Sap, Hamnah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. SocialIQA: Common- sense reasoning about social interactions. arXiv preprint arXiv:1904.0 09728, 2019.
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G. Baraniuk. WIRE: Wavelet implicit neural representations. In Proceedings the JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18507-18516, June 2023.
Jonathan Richard Schwarz, Jihoon Tack, Yee Whye Teh, Jaeho Lee, and Jinwoo Shin. Modality- agnostic variational compression of implicit neural representations. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 00342-30364. PMLR, 23-29 Jul 2023. URL htttss:/pproeeeiing
mlr wwwl..press. v202222222////22222 2eteeessvv22//ccaarae222 schwarz23aa2 23..mm.......
Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312. 2023.
Jinglei Shi and Christine Guillemot. Light field compression via compact neural representa- tion. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023. doi: 10. 1AI1009SS4449955.2002000 10095668.
Gilbert Strang. Linear Algebra and Learning from Data. Celleslyy-Cambrigge Press, 2019. ISBN 978-0-692-11638--0
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Jiaxiang Tang. Xiaokang Chen, Jingbo Wang, and Gang Zeng. Compressible-compostae NeRF via rank-residual decomposition. arXiv preprint arXiv:2205. 14870, 2022.
13

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804 1804007611 2018.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 1805. /2471, 2018
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT, 2018.
Ou Xinwei, Chen Zhangxin, Zhu Ce, and Liu Yipeng. Low rank optimization for efficient deep learning: Making a balance between compact architecture and fast training. Journal of Systems Engineering and Electronics, PP:1-23, 01 2023. doi: 10.23919//SHE.2223.000
Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, and Zhuang Liu. Initializing models with larger ones. In International Conference on Learning Representations (ICLR), 2024.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7370-7379, 2017.
Elad Ben Zaken. Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficinn fine- tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, Dublin, Ireland, 2022. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can 2019.
machine really finish your sentence? arXiv preprint
Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. LoRAPrune: Pruming meets low-rank parameter-efficient tine-tuning 2024. URL https:// openreview n forum? iid 1d=9kV1ee1g77
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 THEORETICAL FRAMEWORK
In this section we give the proof of proposition and theorem from section 3.2 the paper.
We recall from section 3.2 the following notation: For fixed w > 0, let sin(w- A) denote the matrix obtained from a fixed m X n matrix A by applying the function sin(w x) component-wise
Assuming A # 0 we define 4
1 min
min Ai3
AA,l
(7)
Note that such a quantity is well defined precisely because A has a finite number of entries and all such entries cannot be zero from the assumption that A X 0.
Before we give the proof of proposition from section 3.2 of the main paper, we will prove two lemmas.
Lemma 1. For a fixed m x n matrix A. We have
|ssin 2 (A mini if0 < w <
π
(8)
3 min
where Anin is defined as follows:
A A
min Aijyo
|
(9)
for 1 1sis m and i n.
Proof. Observe by definition of the Frobenius norm that
m ||snnwA)]|e = ΣΣ sin(w
(10)
We then find that
||sin(wA]]? 2 sin(wA )
(11)
The goal is to now find a lower bound (i (min)) ) In order to do this consider the function f(w) = sin(wr) wi 2. where fixed and positive.
Differentiating this function we have
('(w)) = rrco(wxr) 2
x
(12)
To find critical point solve the equation ('(w) find
cos(wwz) 1
(13)
2
We see that equation 13 has the solution war 3. In order to check what type of critical point we need to look at (w(w)
we = 3
("(w) - sin (wx)
(14)
when w 3z implying that the critical point w 3n
maximum point.
Observe that (() = 0 it thus follows that f(w) > 0 on the interval [0, +).
Applying this to the function Am ) we obtain that
A0 A
sin(w A (
if <
(15)
3.40 0
Substituting the lower bound in equation 15 into equation 11 obtain the proposition.
n
15

---

Published as a conference paper at ICLR 2025
The next lemma establishes an upper bound the operator norm of sin(wA). We remind the reader that the operator norm of is denoted by ||A and is defined by
||
sup
||Ax||2
(16)
||=||==1
where æ is a vector and 112 represents the vector 2-norm. can be shown that the operator norm of A is also given by the largest singular value of A see (Strang, 2019).
Lemma 2. Let A be fixed m x n matrix. Then
[|sin(wA)](s & | op
(17)
where /I A| denotes the matrix obtained from A by taking the absolute value and then square root component wise.
Proof. By definition we have
sup ||=||==1
(18)
where || ll2 denotes the 2-norm of a vector.
For any fixed unit vector x we will show how to upper bound the quantity [|sin()A)x113 In order
to do this we will use the fact that for 2 0, we have the bound sin(x) vlzl.
sin( Ban(wee))zg 2
(siinww))rt
(19)
m
2
Σ vi Aiglos
(20)
j=1
= ([([[)
|
x11
(21)
follows that
sup ||||==1
|| sinn
sup ||
11 A
(22)
which implies
|oVIA
[w)n -
(23)
We can now give the proof of proposition from section 3.2 of the main paper. In order to do so we will need the definition of the stable rank of matrix. Assume A is a non-zero matrix. We define the stable rank of by
| SR(A) = ||
(24)
is easy to see from the definition that the stable rank is continuous, unlike the rank, and bounded above by the rank
SR(A) & Rank(A).
(25)
Remark 1. We observe that lemmas 1 and 2 give the main reasons why we chose sine function as the non-linearty to apply on weight matrix A. The periodic nature of sine function that can be controlled by a frequency parameter > 0 is what allows us to obtain a proof of lemmas and 2. When we give a proof of Thm. I we will see that these two lemmas are crucial.
of proposition 3.1 from section 3.2 of main paper. Observe that from equation 25 it suffices to prove the lower bound on SR(A). This immediate from lemma and lemma 2.
16

---

Published as a conference paper at ICLR 2025
Proof of theorem 1 from section 3.2 of main paper. From the assumption of the theorem 1 we have that N >> k. Further, we are assuming that both U and V have entries sampled from u( / (K-1//N,/NN) This means if we let A UV", then there exists C' > 0 such that
4
(26)
min
N
Furthermore, observe that
II
A
IIVIAIIE
&
||A||E
(27)
which implies
4 -min
C
N
w
(28)
1Iv
A
N4
mn
mm
Now observe that from proposition in section 3.2 from the main paper we have that
4
Rank(sin(wA)) 2
(29)
IIviAllop
if 1 3.4 We can rewrite this last condition to say that equation 29 holds if S n.V²
In particular, by using equation 28 we find that there exists wo within the interval 0 S wp v nN²
3
A
))) 2
II 2 2 Rank(A).
(30)
This completes the proof.
Remark 2. Observe that if w is very small, then sin(wz) se we and thus applying a sin simply scales the matrix by w which cannot change rank. It is only when is sufficiently large that that the rank increases. If = 0 then applying sin with frequency ws to the matrix produces the zero matrix which has zero rank and in general will thus have rank less than A.
Remark 3. In general low-rank matrices inherently have fewer degrees of freedom compared to high-rank matrices, which limits their representational capacity. For complex datasets, we hypoth- esize that high-rank weight matrices within neural model provide additional degrees of freedom, enabling the model to capture and learn key features more effectively from the input data. Our empirical results all tasks seem to validate this hypothesis.
A.2 EXPERIMENTS
For the experiments we observed that the gain factor in Equation (3) should be chosen analogously how weights are initialized in (He et al., 2015). In particular we chose g vn, where was the number of rows of the weight matrix. During backpropagation the frequency parameter w scales the gradients which can cause gradient exploding. To mitigate this we found the choice of g yn worked best. We also point out that for the experiments there is no principled way to set w. Therefore, we will obtain by treating it as hyperparameter and tuning it according to best results.
A.2.1 FINE TUNING LARGE LANGUAGE MODELS
Implementation details for Roberta V3 We followed the settings in (Hu et al., 2022) and (Ding et al., 2023). In the Transformer architecture, there are four weight matrices in the self-attention module (W,, q: WK. W., W.) and two in the MLP module( W Woown). To evaluate RoBERTA V3, we follow up the LoRA architecture and implement low-rank adaptation only on W, and W.. We study the performance of LoRA and sine LoRA in terms of different rank l 1, 2, 4, 8. For sine LoRA, we use frequency 200 across all the ranks. We use different learning rate and epoch for different datasets as shown in Table 4.
Implementation details for Llama3-8B We followed the settings in (Liu et al., 2024). In the Transformer architecture, there are four weight matrices in the self-attention module (W q: Wk. W., W.) and two in the MLP module( ( up, W Waowa). To evaluate Llama3-8B, we im- plement low-rank adaptation only on W,, q. Wt. W,, W up: and W down- We study the performance shown in 5.
of LoRA and sine LoRA for different rank k 4, 8, 16, 32 and configurations
17

---

Published as a conference paper at ICLR 2025
Dataset
epoch
CoLA
8c-5
20
SST-2
le-4
10
MRPC
le-4
20
QQP
3e-4
10
STS-B
le-4
20
MNLI
3e-4
10
QNLI
3e-4
10
RTE
1.2e-3
50
Table 4: Learning rate and epoch for each dataset for the Roberta V3 model.
rank
frequency
epoch
800
le-4
3
600
le-4
3
16
200
le-4
3
32
200
le-4
Table 5: Sine LoRA. Frequency, learning rate and epochs for the Llama3-8B model.
DoRA order to compare our method to the current state-of-the-art we apply our sine-LoRA method Weight-Decomposed Low-Rank Adaptation (DoRA) (Liu et al., 2024). DoRA decom- poses pre-trained weights W E |R into two components: magnitude vector E IR 1x1 and lle such that each
direction matrix D E mxx normalized by the column vector-wise norm column remains unit vector.
DoRA decompose low-rank change in the direction matrix into the decomposition AD UVT where U E T and V E R"*k To implement our sine-DoRA, we apply the sine function to this decomposition as per Equation (3). We evaluate the performance of DoRA and sine-DoRA in terms of different rank k 8, 16, 32 in Table 7 and configurations are shown in Table 8. We implement 8 directly as this is not a setting used in (Liu et al., 2024), and compare against reported results for k 16 and k 32. By introducing methods on top of DoRA, our model (Sine- DoRA A = 8) achieve state-of-the-ar results while utilizing only 25% of the parameters required by DoRA(k 32).
Computational cost. Finetuning Llama3-8B takes roughly 6 hours using LoRA, 7 hours using Sine LoRA, 11 hours using DoRA, and hours using Sine DoRA using NVIDIA H100 GPU with 96GB of memory. Training memory cost is shown in Table 6.
A.2.2 VISION TRANSFORMERSS
Implementation details for ViT-Small on CIFAR100: The ViT-Small model, characterized by its two MI. layers with input/output dimensions of 384 and hidden dimensions of 1536, was mod- ified by replacing the full-rank weight matrices with low-rank matrices across range of ranks (k). We use learning rate le-3, batch size 512 and train for 200 epochs. Choices of frequency for different ranks are shown in Table 9.
Table 6: Training memory (GB) cost for LoRA, Sine LoRA, DoRA, Sine Dora finetuning Llama3-8B as reported by Nvidia-SMI.
Method
Rank 8
Rank 16
Rank
LoRA
54.9
55.2
55.3
Sine LoRA
72.0
72.0
72.2
DoRA
75.6
75.9
76.0
Sine DoRA
90.4
90.7
90.8
18

---

Published as a conference paper at ICLR 2025
Table 7: Performance and parameter of the LLaMA 3-8B model fine-tuned using the DoRA and sine DoRA methods across varying kamx settings on the commonsense reasoning benchmark. Results reported the paper.
Method
Params
BoolQ
PIQA
SIQA
HS
WG
ARC-e
ARC-c
OBQA
Avg.
DoRA
73.2
87.7
79.9
94.7
84.5
89.3
78.0
83,2
83.8
14.9M
Sine DoRA
73.9
89.0
81.0
95.3
86.1
90.1
79.0
87.0
85,2
DoRA*E- 1-16
29.1M
74.5
83.8
80.3
95.5
84.7
90.1
79.1
87.2
85.0
|
Sine DoRAg=18
75.1
89.0
81.0
95.3
86.1
90.0
79.3
86.2
85.3
DoRA* 5-22
74.6
89.3
79.9
95.5
85.6
90.5
80.4
85.8
85.2
57.4M
0.1
Sine DoRA;=32
75.8
89.3
80.3
95.9
86.1
90.2
79.4
85.4
85.3
Table 8: Sine DoRA. Frequency, learning rate and epochs for the Llama3-8B model.
Rank
Frequency
Learning Rate
Epoch
8
300
6e-5
16
150
6e-5
32
100
6e-5
Implementation details for ViT-Base on ImageNet-1k:: We followed the settings in (He et al., 2022a). We use batch size of 1024, learning rate of 3e-4 and we train for 300 epochs. Choices of frequency for different ranks are shown in Table
Table 9: Frequencies used for different ranks for ViT-Base and ViT-Small.
rank
5
10
30
60
100
150
250
ViT-small (CIFARI00)
500
500
300
300
300
ViT-Base (ImageNet-1k, 1k)
w
1000
800
800
400
200
200
150
150
Ablation on frequencies: In table 10, we examine the performance of our method on training the ViT-Small model from scratch on the CIFAR100 dataset using different frequencies, when 1.
Table 10: Top-1 Accuracy of ViT-Small (A: = 1) on CIFAR100 with varying frequencies
Frequency w
100
200
300
400
500
600
700
PSNR
55.0
55.8
56.8
58.0
58.1
57.6
57.5
A.2.3 CONVNEXT ON CIFAR100
ConvNeXt is family of convolutional neural networks (CNNs) models introduced in (Liu al., 2022). These models are designed to modernize traditional CNNs architectures by incorporating de- sign elements inspired by Vision Transformers (ViTs) to enhance performance in image recognition.
Implementation details: We employ our methods on ConyNeXt-Tiny model using CIFAR100
datasets and the Timm codebase. Conv consists of 4 stages with block numbers [3, 3, 9, 3] and feature dimensions with [96, 192, 384, 768]. The majority of parameters (50%) in ConvNeXt are used in the last stage, therefore we apply low rank decomposition only to the linear feature layer in this stage. We use batch size of 512, learning rate of 5e-3, and we train for 150 epochs.
Results: In Table 11, present our performance and configurations. We demonstrate that method consistently outperform the naive low rank method, and even outperforms the baseline full- rank method (ConyNext-Ting) with approximately 50% fewer parameters.
19

---

Published as a conference paper at ICLR 2025
Performance and compression rate of ConvNeXX-Timg model trained on CIFARI00 datasets. We use frequency [400, 300, 300] for rank [1, 5, 20] respectively.
Table 11:
Method
# Params
Acc %
Compression Rate %
ConvNeXt-Tiny
27.9M
62.3
100
LR-Con-NcXX--mmyy-
59.5
13.8M
49.5
Sine L.R-ConvNeett-imee 111
61,5
LR-Con-Nexi--innyy--
59.5
13.9M
50.0
Sine LR- Cow.Cooet-Tnyyysssa.....
62.0
LRR-ConwNeXI-Thnny..a2
62.1
14.2M
50.9
Sine LR-Con-NeXt-Tiny--eoocc
62.8
A.2.4 NERF
Implementation details: We followed the settings in (Ramasinghe Lucey, 2022). We use fully connected laycrs-each with 256 neurons, learning rate of 5e-4 and train for 500k iterations. We evaluate the performance of our method by experimenting with different ranks [1, 5, 10, 30, 60), corresponding to frequencies [1400, 800, 600, 400, 300] respectively.
Results:
The full qualitative results on NeRF are given in figure 8.
Ablations: In figure we illustrate the impact of varying frequency on PSNR for cases where k=l (shown on the left) and k=5 (shown on the right).
Computational cost: In Table 12, we present the training memory usage (MB) on NeRF experi- ments across different ranks.
Figure 7: Ablation NeRF results for the LLFF dataset. These two figures show PSNR of NeRF using different frequencies, when (on the left) and (on the right)
23.2
20
23.0
22.8
PSNR
19
PSNR
22.6
22.4
22.2
17
22.0
500
1000
1500
250
500
750
1000
1250
Frequency
Frequency
Table 12:
Training memory usage(MB) on NeRF experiments
Method
rank
rank
rank 10
rank 30
rank 60
Low-Rank
5620
5622
5622
5626
5630
Sine Low-Rank
5630
5630
5632
5634
5638
20

---

Published as a conference paper at ICLR 2025
Figure 8: Qualitative NeRF results for the LLFF dataset (Mildenhall al., 2019; 2020) using rank = and k 5. Using a low-rank model leads to complete loss of signal for l: = 1, however, applying sine is able to reconstruct details even at the extreme low-rank case. At the sine low-rank model is noticeably sharper and clearer than using the low-rank.
k k= 1
k 5
Low-Rank
sine-Low-Rank (Ours)
Low-Rank
sine-Low- Laoe-Low-Rakk (Ours)
Fem
Flower
Fortress
Homs
Leaves
Orchids
Room
Trex
21

---

Published as a conference paper at ICLR 2025
Figure 9: Ablation binary occupancy results for Thai Statue. This figure shows IoU accuracy of 3D shape modeling using different frequencies, when k=1.
90.75
90.50
90.25
ToU
90.00
89.75
89.50
89.25
200
400
600
800
1000
frequency w
A.2.5 3D SHAPE MODELLING
Implementation details: We use fully connected layers-each with 256 neurons, a learning rate of 1e-3 and train for 200 epochs.
Results: Table 13 reports the Intersection over Union (IoU) and Compression Rate of the binary occupancy task using different rank levels (k). Our sine low-rank methods.
Table 13: This table illustrates Intersection over Union for 3D shape modeling (Thai Statue, Lucy and Dragon) across different rank levels (k). It also includes the compression rate, indicating the proportion of parameters utilized relative to the total parameter count of the baseline model, thereby detailing the parameter usage versus model performance at different levels of model complexity. We use frequency [200, 100, 50, 20] for ranks [1, 2, 5, 20], respectively
IoU
Compression Rate
Params
Thai
Lucy
Dragon
Full-Rank
132K
97.2
97.8
98.7
100%
Low-Rankg-1
84.3
79.3
90.4
2.8K
2.1%
Sine How-Rankk=1,w-a
90.8
90.7
94.6
Low-Rank:-2
88.0
89.4
90.9
3.8K
2.9%
Sine Low-Rat Now-Rank
92.0
93.2
96.6
Low-Ranke=5
93.4
94.8
96.9
6.9K
5.2%
Sine Low-Rank,=áwwst
94.3
95.3
97.4
Now-Rank--20
95.4
96.2
98.0
22.8K
16.8%
Sine Low-Rank,=2000m
95.4
96.3
98.1
22


Reviews:
[{"review_id": "j6JpCVn9GC", "reviewer": "Reviewer_C1Nj", "summary": "This paper presents a theoretical framework that integrates a sinusoidal function within the low rank decomposition process to achieve efficient learning. The proposed method was evaluated on several applications such as LLMs parameter efficient fine-tuning, NeRF and 3D shape modeling.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper provides theoretical justification on benefit of the proposed sine-activated low rank matrices without adding additional parameters.\n+ The experimental results show the proposed sine-activated low-rank decomposition works better than the original LoRA.", "weaknesses": "- The paper talks about the benefits of using a sine function for increasing rank, but it could explain more about why this function was chosen over others and why it works. A discussion of its advantages would strengthen this work.\n\n- While the proposed method is applied across various domains, the paper lacks a comprehensive comparison with existing low-rank approximation techniques. There are many existing LoRA variants and improved versions, e.g. [a] [b].\n\n[a] LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning\n[b] MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning\n\n- It would be good to provide space and time complexity analysis of the proposed method. \n\n- There is a lack of quantitative results on a standard dataset for the 3D shape modeling task. It would be good to evaluate the method on other popular tasks such as text to image generation. \n\n- How do we choose the frequency parameter (ω) in practice? Is there an easy way to adjust it? As per the paper, (ω) higher increases the rank. But do we always pick highest (ω) ? How do we effectively use different frequencies within an interval ?\n\n- For each application, the implementation details of the proposed method should be provided.\n\n- Are there any implementation challenges, like increased computation time during training? Does this technique slow down the model\nduring inference, especially in environments with limited resources?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HbQ2t5Dftp", "reviewer": "Reviewer_Gygt", "summary": "This paper proposes the introduction of sinusoidal functions within LORAs to improve model performance across a range of tasks. The claim is to be able to balance parameter efficiency and model performance by increasing the rank with sinusoidal functions without adding parameters. The contributions are a theoretical framework, demonstrations of the performance and evaluation on several tasks such as computer vision, 3D shape modeling and NLP. A large part of the paper is the mathematical analysis of low-rank decomposition and the derivation of a non-linear low-rank decomposition as well as showing mathematically why its rank is larger. The supplementary material holds detailed proofs for the propositions and theorems.", "questions": "Figure 4 in the text is referenced as table 4.\n\nWhy is DoRA cited only once as Lora technique but then there is no reference to it's improved performance?\n\nIn line 182, and 765 is the A_{ij} \\neq 0 maybe a A_{ij \\neq 0} ? It looks as if the indexes should not be 0 and not as if the assumption of A \\neq 0 is formalized. \n\nThe operator norm in equation 5 has one | too much.\n\nWhy is the observation in line 786 important for 11? This should be true in any case.\n\nLine 799 Says Equation equation 13. Probably an issue with cref\n\nThe sentence starting in 855 is somehow messed up.\n\nUsing parameter counts instead of compression rate in Figure 1 for ViT Classification would improve comparison with other state-of-the-art approaches.\n\nWhy does an increased rank improve performance? This point may have been demonstrated by experiments but an intuition could be given. Adding certain noises to the matrix may also increase it's rank without increasing it's parameter count but will probably not lead to improved performance. The link between an increased rank and improved performance could be made clear or hypothesized. Seeing that low-rank adaptation decreases performance may be not sufficient to claim an inverse relationship.\n\nIn line 480 I am not sure it is good to speak of overfitting regarding to NeRFs because this sounds as if it's negative when it is the desired behavior.\n\nFigure 10 seems to indicate the frequency parameter has an optimum. How hard is it to choose it? Are the best results retrieved from a grid search or is there any intuition about it's value beforehand?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is simple and fascinating. If there are no flaws in the proof, which I didn't find, this has a potential large impact in the domain.\n- The mathematical background is well explained, motivated and proven.\n- Figures like Figure 2 help in understanding the impact of the method's parameter.\n- The method is shown on a variety of very different tasks from classification to Nerf.\n- Extreme scenarios are studied like Nerf reconstruction using Lora with rank k=1", "weaknesses": "- The method is only compared on self-implemented baselines and seems to underperform compared to other improvements in the domain. E.g. One of the citations even mentions DoRA but as an example of application of Lora techniques. They do have results of LLaMA3-8B LoRA which are similar to the results achieved in this paper but with DoRA they improve the performance on commonsense reasoning beyond this paper. When presenting an improved LoRA training technique the authors should compare their performance not only to the vanilla LoRA case but to other techniques like DoRA and ideally implement and evaluate an improved DoRA although I do not see with this specific work how that would be applied. Nevertheless, it seems important state-of-the-art here is missing, which is needed to position the paper in the research landscape.\n\nThere are a lot of minor syntax errors or errors in the equations which need to be corrected and do not give a polished impression (See questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PVX1JIOW5P", "reviewer": "Reviewer_HVQH", "summary": "This paper addresses the critical need for compact and parameter-efficient architectures in machine learning. While low-rank techniques are a popular solution for reducing model complexity, the authors introduce high-frequency sinusoidal non-linearity functions to enhance the capabilities of low-rank matrices. The paper provides a theoretical framework demonstrating how this modulation increases the rank of matrices without adding extra parameters. This sine-based non-linearity is incorporated into low-rank decompositions, effectively balancing efficiency and performance improvements across various benchmarks, including Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF), and 3D shape modeling.", "questions": "Please refer the weakness section above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Task Motivation:\nThe paper tackles a highly relevant problem—parameter-efficient learning. While overparameterization is effective for generalization, practical deployment in industry necessitates efficient architectures for cost-effectiveness.\n\nSimple Yet Effective Approach:\nThe introduction of a sine function to augment low-rank decompositions is straightforward but impactful. As shown in Figure 3, the Sine LR method maintains representational power comparable to other activation functions while improving efficiency. The effectiveness of this method is demonstrated across multiple tasks and backbones, including LoRA in LLMs (ROBERTaV3 and LLaMA 3-8B), ViT (ViT-base and ViT-small), NeRF (vanilla NeRF on LLFF), and 3D shape modeling (Thai Statue example).\n\nTechnical Soundness:\nThe theoretical analysis highlights how controlling the frequency coefficient \"w\" in the sine function affects the linear independence of the original matrix \"W\". Figure 2 illustrates this effect, showing that the sine-activated low-rank decomposition facilitates efficient training without increasing the number of model parameters.", "weaknesses": "Lack of Rationale Behind the Sine Function's Effectiveness: While the theoretical and experimental validations in Sections 3.2 and 4 establish the efficacy of the sine activation, the paper doesn't delve into why the sine function specifically enhances parameter-efficient training. I'm left wondering what inherent properties of the sine function drive these results. A clearer explanation of how the sine function contributes to increasing matrix rank or improving performance without adding parameters would significantly strengthen the argument.\n\nHyperparameter \"w\" Tuning:\nThe role of the frequency hyperparameter \"w\" isn't thoroughly addressed. Is there a need to fine-tune \"w\" for each task, or does it generally benefit from being larger? More importantly, how do we determine the optimal \"w\"? A systematic exploration or guidelines for selecting \"w\" across different tasks would be helpful, as the current approach seems to leave this critical aspect underexplored.\n\n\nApplicability to CNN Architectures:\nAlthough the paper demonstrates strong results on various architectures like ViT, LLMs, NeRF, and 3D shape modeling, it leaves a gap concerning CNNs. Given that CNNs are still widely used in practice, it would be valuable to explore whether the sine-activated low-rank method can be effectively applied to CNN architectures. A discussion or preliminary results in this area would broaden the paper’s applicability and relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "At3inVniFx", "reviewer": "Reviewer_EgEA", "summary": "This paper introduces concise and elegant improvement method for Low Rank Adaption (LoRA) and reveals its underlying principle by rigorous formula derivation. In addition to fine-tuning large language model, this method can even also achieving competitive results when directly replacing original full-rank weight, which provides some inspiration for model compression. Although there are some parts that I think are slightly lacking in depth, I would suggest accepting it after the following issues are addressed.", "questions": "Advice & Question:\n1. In formula 3, why the coefficient w and g are non-learnable ? If learnable, will it cause gradient problem? For the same model, has it been considered to provide different w and g for different linear layers? Can the recommended values for these two be provided?\n2. Since other nonlinear functions are almost useless, why not directly give the sine function in Equation 3?\n3. Regarding the right diagram in Figure 3, I don't think we can see a trend of the matrix rank increasing with the increase of frequency w. Because the rank of a matrix is only related to the number of nonzero singular values. So when w is 200 and 400, it seems that the ranks of the two are almost equal.\n4. In the setting of section 4.1, why do not implement low-rank adaptations on all linear modules of LLM?\n5. Does the rank of the decomposed matrix necessarily be less than that of the original matrix regardless of how the frequency is adjusted?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper introduces concise and elegant improvement method for Low Rank Adaption (LoRA) and reveals its underlying principle by rigorous formula derivation. In addition to fine-tuning large language model, this method can even also achieving competitive results when directly replacing original full-rank weight, which provides some inspiration for model compression. Although there are some parts that I think are slightly lacking in depth, I would suggest accepting it after the following issues are addressed.", "weaknesses": "Advice & Question:\n1. In formula 3, why the coefficient w and g are non-learnable ? If learnable, will it cause gradient problem? For the same model, has it been considered to provide different w and g for different linear layers? Can the recommended values for these two be provided?\n2. Since other nonlinear functions are almost useless, why not directly give the sine function in Equation 3?\n3. Regarding the right diagram in Figure 3, I don't think we can see a trend of the matrix rank increasing with the increase of frequency w. Because the rank of a matrix is only related to the number of nonzero singular values. So when w is 200 and 400, it seems that the ranks of the two are almost equal.\n4. In the setting of section 4.1, why do not implement low-rank adaptations on all linear modules of LLM?\n5. Does the rank of the decomposed matrix necessarily be less than that of the original matrix regardless of how the frequency is adjusted?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)