Efficient Learning with Sine-Activated Low-Rank Matrices

## Abstract

Abstract Low-rank decomposition has emerged as a vital tool for enhancing parameter efficiency in neural network architectures, gaining traction across diverse applications in machine learning. These techniques significantly lower the number of parameters, striking a balance between compactness and performance. However, a common challenge has been the compromise between parameter efficiency and the accuracy of the model, where reduced parameters often lead to diminished accuracy compared to their full-rank counterparts. In this work, we propose a novel theoretical framework that integrates a sinusoidal function within the low-rank decomposition process. This approach not only preserves the benefits of the parameter efficiency characteristic of low-rank methods but also increases the decomposition’s rank, thereby enhancing model accuracy. Our method proves to be an adaptable enhancement for existing low-rank models, as evidenced by its successful application in Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF), and 3D shape modeling. This demonstrates the wide-ranging potential and efficiency of our proposed technique.

###### Keywords:

## 1 Introduction

In the last few years, large-scale machine learning models have shown remarkable capabilities across various domains, achieving groundbreaking results in tasks related to vision and natural language processing. However, these models come with a significant drawback: their training necessitates an extensive memory footprint. This challenge has spurred the demand for more compact, parameter-efficient architectures. A prominent solution that has emerged is the use of low-rank techniques, which involve substituting the large, dense matrices in large-scale models with smaller, low-rank matrices. This substitution not only simplifies the models but also shifts the computational complexity from quadratic to linear, making a significant impact on efficiency. In the context of high-capacity models like Vision Transformers (ViTs) and Large Language Models (LLMs) that utilize millions to billions of parameters, transitioning from dense to low-rank matrices can result in considerable cost savings. Nonetheless, adopting low-rank architectures does introduce a trade-off, as they typically do not achieve the same level of accuracy as their full-rank counterparts, presenting a balance between parameter efficiency and model performance.

Addressing this challenge, our work unveils a novel technique that retains the parameter efficiency intrinsic to low-rank methods while achieving superior accuracy. Our method hinges on the realization that augmenting a low-rank matrix with a high-frequency sinusoidal function can elevate its rank without inflating its parameter count. We lay out a theoretical framework elucidating why and how such sinusoidal modulation crucially enhances the matrix’s rank. By utilizing this non-linearity into low-rank decompositions, we design compact architectures that not only maintain their streamlined nature but also deliver improved accuracy across various machine learning tasks.

An example of this insight is depicted in the left image of Fig. 1, revealing the singular value spectra of low-rank versus full-rank matrices in comparison to our sinusoidally enhanced matrix. This enhancement allows our matrix to more closely resemble the spectrum of a full-rank matrix, without an increase in parameters, laying the foundation for our model’s improved accuracy. Further validation of our architecture’s performance is demonstrated in the right image of Fig. 1, which compares three ViT architectures on the ImageNet-1K dataset. Here, our model maintains the same parameter efficiency as the low-rank version yet achieves a performance boost of 3-4% underscoring its superior efficacy.

Figure: Figure 1: In this figure, we illustrate the impact of low-rank approximation and our sine activated low-rank method on the weight matrix spectrum and performance in the ViT-Base model’s Feed-Forward Network. All singular values are normalized to 1. The left part shows the SVD spectrum for matrices initialized with the Kaiming uniform method: full-rank, low-rank, and sine activated low-rank matrices. On the right, we note that while low-rank approximation reduces parameter count, it also lowers performance. Our sine activated low-rank approach can improve the accuracy by approximately 4% without increasing the parameter count.
Refer to caption: extracted/2403.19243v1/vit_rank10.png

Our approach’s inherited advantages are further corroborated across a range of machine learning applications, including variations of ViT, Low-Rank Adaptation (LoRA) methods for LLMs, Neural Radiance Fields (NeRF) for novel view synthesis, and 3D shape modeling via binary occupancy fields. Across the board, our approach not only matches the parameter savings offered by low-rank methods but also secures an improvement in accuracy, attesting to its broad applicability and superior performance. The main contributions of our paper are:

- 1.
A parameter-efficient matrix decomposition that rivals traditional low-rank decompositions in terms of parameter economy while delivering enhanced accuracy.
- 2.
A comprehensive theoretical framework that substantiates our approach, providing a solid underpinning for our methodology.
- 3.
Extensive empirical validation has been conducted across a diverse set of applications, each demonstrating our model’s superior accuracy and effectiveness.

## 2 Related Work

#### 2.0.1 Low-rank decomposition:

stands as a crucial method across disciplines like information theory, optimization, and machine learning, providing a strategic approach to reduce memory costs [, ]. Notably, [] uncovered that matrices can precisely separate low-rank and sparse components through convex programming, linking to matrix completion and recovery. Expanding its application, [] devised a low-rank learning framework for Convolutional Neural Networks, enhancing compression while maintaining accuracy. [] further found that performance improvements in Large Language Models could be achieved by eliminating higher-order weight matrix components without extra parameters or data. In the realm of neural radiance fields, [] introduced a rank-residual learning strategy for optimal low-rank approximations, facilitating model size adjustments. Additional contributions include [] with rank-constrained distillation, [] applying vector-matrix decomposition, and [] using soft-gated low-rank decompositions for compression. More recently, [] implemented a vector-matrix decomposition strategy that allows for test-time compression adjustments.

#### 2.0.2 Parameter efficient learning:

is an important research area in deep learning, merging various techniques to enhance model adaptability with minimal resource demands []. Techniques like parameter-efficient fine-tuning (PEFT) allow pre-trained models to adjust to new tasks efficiently, addressing the challenges of fine-tuning large models due to high hardware and storage costs. Among these, Visual Prompt Tuning (VPT) stands out for its minimal parameter alteration—less than 1%—in the input space, effectively refining large Transformer models while keeping the core architecture unchanged []. Similarly, BitFit offers a sparse-finetuning approach, tweaking only the model’s bias terms for cost-effective adaptations []. Moreover, LoRA introduces a low-rank adaptation that maintains model quality without additional inference latency or altering input sequence lengths, by embedding trainable rank matrices within the Transformer layers []. Recent studies also combine LoRA with other efficiency strategies like quantization, pruning, and random projections for further model compression [, , , ]

## 3 Methodology

In this section, we introduce our technique which we term a sine activated low-rank matrix. The main purpose of this technique is to increase the rank of an initial low-rank matrix without adding parameters.

### 3.1 Notation

#### 3.1.1 Feed-Forward Layer

Our technique is defined for feed-forward layers of a neural architecture. In this section, we fix the notation for such layers. We will express a feed-forward layer as

$$ $\mathbf{y}=\mathbf{W}\mathbf{x}+\mathbf{b}$ (1) $$

where $\mathbf{W}\,{\in}\,\mathbb{R}^{m{\times}n}$ is a dense weight matrix, $\mathbf{b}\,{\in}\,\mathbb{R}^{m{\times}1}$ is the bias of the layer, and $\mathbf{x}$ is the input from the previous layer. The output $\mathbf{y}$ is then often activated by a non-linearity $\sigma$ producing $\sigma(\mathbf{y})$. The weight matrix $\mathbf{W}$ and bias
$\mathbf{b}$ are trainable parameters of the layer.
In contemporary deep learning models, the feed-forward layers’ weight matrices, $\mathbf{W}$, are often large and dense yielding a high rank matrix. While the high-rank property of the weight matrix helps in representing complex signals, it significantly adds to the overall parameter count within the network yielding the need for a trade-off between the rank of the weight matrix and overall architecture capacity.

#### 3.1.2 Low-rank decomposition

A full-rank weight matrix $\mathbf{W}$ can be replaced by low-rank matrices $\mathbf{U}\mathbf{V}^{T}$, such that $\mathbf{W}\,{=}\,\mathbf{U}\mathbf{V}^{T}$, where $\mathbf{U}\in\mathbb{R}^{m{\times}k},\mathbf{V}\,{\in}\,\mathbb{R}^{n{\times}k}$ and $k\,{\ll}\,min(m,n)$.

$$ $\mathbf{y}=\mathbf{W}\mathbf{x}+\mathbf{b}=(\mathbf{U}\mathbf{V}^{T})\mathbf{x }+\mathbf{b}\\$ (2) $$

This is the most common way to reduce the parameter count in the feed-forward layer.
During the training process, this method performs optimization on $\mathbf{U}$ and $\mathbf{V}$ alternatively. Low-rank multiplication then reduces the learnable parameter count and memory footprint rom $O(mn)$ to $O(k\cdot(m{+}n))$. Although $\mathbf{U}\mathbf{V}^{T}$ has the same matrix shape as the full-rank matrix $\mathbf{W}$, the rank of $\mathbf{U}\mathbf{V}^{T}$ is constrained and $rank(\mathbf{U}\mathbf{V}^{T})\,{\leq}\,r$. Thus while we have significantly decreased the number of trainable weights in such a layer, we have paid the price by obtaining a matrix of much smaller rank. In the next section, we address this trade-off by developing a technique that can raise back the rank of a low-rank decomposition while keeping its low parameter count.

### 3.2 Theoretical Framework

In this subsection, we formally describe the simple design of the naive low-rank method and our proposed Sine Activated Low-Rank method. The principles outlined here apply to any dense layers in deep learning models.

#### 3.2.1 Non-linearity low-rank decomposition

We introduce non-linearity transformation into low-rank matrices.

$$ $\mathbf{y}=\frac{\phi(\omega\cdot\mathbf{U}\mathbf{V}^{T})}{g}\mathbf{x}+ \mathbf{b}$ (3) $$

where $\phi(\cdot)$ is the non-linearity function, $\omega$ is a non-learnable frequency parameter, and $g$ is a non-learnable parameter to adjust the gain of the transformation.

#### 3.2.2 Main theorem:

In this section, we provide a theoretical framework that clearly shows how to increase the rank of a low-rank decomposition without adding any parameters. We will
show that if we choose the non-linearity, in the decomposition defined in Sec. 3.2.1, to be a sine function then provided the frequency $\omega$ is chosen high enough, the rank of the matrix
$\phi(\omega\,{\cdot}\,\mathbf{U}\mathbf{V}^{T})$ will be larger than that of $\mathbf{UV}^{T}$. The proofs of the theorems are given in the supp. material.
To begin with, we fix $\omega\,{>}\,0$ and let $\sin(\omega\,{\cdot}\,\mathbf{A})$ denote the matrix obtained from a fixed $m{\times}n$ matrix $\mathbf{A}$ by applying the function $\sin(\omega\,{\cdot}\,\mathbf{x})$ component-wise to $\mathbf{A}$. Assuming $\mathbf{A}\,{\neq}\,0$ we define
$A_{\min}^{0}$ as:

$$ $A_{\min}^{0}=\min_{i,js.t.A_{ij\neq 0}}|A_{ij}|.$ (4) $$

Note that such a quantity is well defined precisely because $\mathbf{A}$ has a finite number of entries and all such entries cannot be zero from the assumption that $\mathbf{A}\,{\neq}\,0$.

The following theorem relates the rank of $\sin(\omega\,{\cdot}\,\mathbf{A})$ to the frequency parameters $\omega$ and the quantity $A_{\min}^{0}$.

###### Proposition 1

Fix an $m{\times}n$ matrix $\mathbf{A}$ s.t. $\mathbf{A}\,{\neq}\,0$. Then

$$ $Rank(\sin(\omega\cdot\mathbf{A}))\geq\omega\bigg{(}\frac{A_{\min}^{0}}{||\sqrt {|\mathbf{A}|}||_{op}}\bigg{)}^{2}\text{ if }0\leq\omega\leq\frac{\pi}{3A^{0}_ {\min}}$ (5) $$

Prop. 1 shows that if we modulate the matrix
$\sin(\omega\,{\cdot}\,\mathbf{A})$ by increasing $\omega\,{>}\,0$ then the rank of the matrix $\sin(\omega\,{\cdot}\,\mathbf{A})$ can be increased provided
$\omega\,{<}\,\frac{\pi}{3A^{0}_{\min}}$. We can apply Prop. 1 to the context of a low-rank decomposition as defined in Sec. 3.1.2. Given a low-rank decomposition $\mathbf{UV}^{T}$ with $\mathbf{U}\,{\in}\,\mathbb{R}^{m{\times}k}$ and
$\mathbf{V}\,{\in}\,\mathbb{R}^{n{\times}k}$ with $k\,{\ll}\,\min\{m,n\}$ the following theorem shows how we can increase the rank of the decomposition by applying a $\sin(\omega\cdot)$ function.

###### Theorem 3.1

Let $\mathbf{U}\,{\in}\,\mathbb{R}^{m{\times}k}$ and
$\mathbf{V}\,{\in}\,\mathbb{R}^{n{\times}k}$ with $k\,{\ll}\,\min\{m,n\}$. Assume both $\mathbf{U}$ and $\mathbf{V}$ are initialized according to a uniform distribution
$\mathcal{U}(-1/N,1/N)$ where $N\,{>}\,k$. Then there exists an $\omega_{0}$ such that the matrix
$\sin(\omega\,{\cdot}\,\mathbf{A})$ will satisfy the inequality

$$ $Rank(\sin(\omega\,{\cdot}\,\mathbf{UV}^{T}))\,{>}\,Rank(\mathbf{UV}^{T})$ (6) $$

provided $\omega\,{\geq}\,\omega_{0}$.

We mention that Thm. 3.1 also holds for the case where we initialize $\mathbf{U}$ and $\mathbf{V}$ by a normal distribution of variance $N$.

Weight matrices within feed-forward layers are typically initialized using a distribution that is contingent upon the layer’s neuron count. When considering low-rank decompositions characterized by matrices $\mathbf{U}\,{\in}\,\mathbb{R}^{m{\times}k}$ and $\mathbf{V}^{T}\,{\in}\,\mathbb{R}^{k{\times}n}$, where $k\,{\ll}\,\min\{m,n\}$, the variance of this initialization distribution is influenced by $m$ and $n$. These dimensions are significantly larger than $k$, ensuring that the condition specified in Thm. 3.1—that $N\,{>}\,k$—is nearly always met, making this theorem especially relevant for low-rank decompositions in feedforward layers. For example the most common initialization schemes such as Kaiming [] and Xavier [] satisfy the requirements of our theorem.

Thm. 3.1 offers a viable strategy for maintaining a high-rank characteristic in feed-forward layers while simultaneously minimizing the parameter count. By introducing a sinusoidal non-linearity with a sufficiently high frequency $\omega$ into a low-rank decomposition, it is possible to elevate the rank without altering the quantity of trainable parameters. This key insight from our theoretical analysis aims to highlight a novel approach to optimizing network structure for enhanced computational efficiency and model performance.

Figure: Figure 2: These figures display weight magnitudes for matrices with dimension $128{\times}128$. The first figure showcases a heatmap of a full-rank matrix initialized by the Kaiming uniform, highlighting linear independence among rows. The second shows a low-rank matrix $\mathbf{W_{\text{lr}}}\,{=}\,\mathbf{UV}^{T}\,{\in}\,\mathbb{R}^{128{\times}128}$, with $\mathbf{U},\mathbf{V}\,{\in}\,\mathbb{R}^{128{\times}1}$ initialized by Kaiming uniform illustrating minimal linear independence. The final pair of figures reveal how applying a sine function element-wise, $\sin(\omega\,{\cdot}\,\mathbf{UV}^{T})$, with varying $\omega$, affects linear independence in low-rank matrices; specifically, $\omega\,{=}\,100$ and $\omega\,{=}\,2000$ progressively increase linear independence and thus visually showing such a strategy increases the rank.

In Fig. 2, we give a visualization of our method in action. We consider a full-rank matrix, a low-rank matrix, and two sine activated low-rank matrices with different frequencies. By visualizing the weight magnitudes in each matrix via a heat map, we can clearly see how the sine activated low-rank matrix increases rank and furthermore how increasing the frequency of the sine function increases the rank in accord with Thm. 3.1.

Building upon Eq. 3, we explore the application of various non-linear functions to a low-rank decomposition, with a particular focus on the sine function. This choice is inspired by Thm. 3.1, which theoretically demonstrates that applying a sine function effectively increases the matrix rank. In Fig. 3 (left), we present a comparative analysis of the sine function against other common non-linear functions in machine learning, such as the sigmoid and ReLU. The results clearly illustrate that the sine function increases the rank, making it an optimal non-linearity to apply to a low-rank decomposition.

Further, Thm. 3.1 suggests that augmenting the frequency of the sine function applied to a low-rank decomposition contributes to a further increase in rank. To empirically validate this, we conducted experiments applying sine functions of various frequencies to a constant low-rank matrix. The outcomes, depicted in Fig. 3 (right), corroborate the theorem’s prediction, showcasing a positive correlation between the frequency of the sine function and the resultant rank increase.

Figure: Figure 3: In this figure we depict the SVD spectrum of a Kaiming uniform initialized matrix $\mathbf{W_{\text{fr}}}\in\mathbb{R}^{256\times 256}$ and a low-rank $k=5$ approximation matrix $\mathbf{W_{\text{lr}}}=\mathbf{U}\mathbf{V}^{T}$, where $\mathbf{U},\mathbf{V}\in\mathbb{R}^{256\times 5}$ are initialized using Kaiming uniform []. All singular values are normalized to 1. On the left we depict the spectral advantages of applying a non-linear function $\phi(\omega\cdot\mathbf{U}\mathbf{V}^{T})$ where $\omega$ is a hyper-parameter. Here we see the natural advantages of the sine function such that $\phi(\mathbf{x})=\sin(\omega\cdot\mathbf{x})$. On the right, we demonstrate empirically how manipulating $\omega$ within the sine function can change these spectral properties.
Refer to caption: extracted/2403.19243v1/diff_fre.png

###### Proposition 1

###### Theorem 3.1

## 4 Experiments

This section is dedicated to validating and analyzing the efficacy of our proposed low-rank methods across a spectrum of neural network architectures. To demonstrate the broad applicability and versatility of our approach, we examine its performance in three distinct contemporary applications. Specifically, we explore its integration into the pretraining of Vision Transformers (ViT)[], the reconstruction of scenes using neural radiance fields (NeRF)[], the fine-tuning of large language models through low-rank adaptation (LoRA) [], and the 3D shape modeling. This collectively underscores our model’s adaptability to a diverse array of low-rank frameworks, highlighting its potential to significantly impact various domains within the field of computer vision.

### 4.1 Pretraining Vision Transformers (ViTs)

Vision Transformers (ViTs) have risen to prominence as powerful models in the field of computer vision, demonstrating remarkable performance across a variety of tasks. When pretrained on large-scale datasets such as ImageNet-21K and JFT-300M, ViTs serve as robust foundational architectures, particularly excelling in feature extraction tasks [, ]. A critical observation regarding the architecture of ViTs is that the two feedforward layers dedicated to channel mixing contribute to nearly 66% of the total model parameter count. In light of this, focused experiments on these specific layers have been conducted to rigorously assess the impact and effectiveness of our proposed method, facilitating a direct comparison with the baseline model.

**Table 1: Top-1 Accuracy and training loss of ViT-Base and sine-ViT-Base training from scratch on the ImageNet-1k dataset with different rank levels $k$. The Compression Rate represents the percentage of parameters used in comparison to the original model’s total number of parameters.**
|  | Top-1 | Change | Training | Change | Compression |
| --- | --- | --- | --- | --- | --- |
|  | Accuracy | Loss | Rate |  |  |
| Baseline | 80.49 | - | 2.41 | - | 100% |
| ViT-Bk=1 | 72.70 | $\mathbf{1.05}$ ↑ | 3.51 | $\mathbf{0.03}$ ↑ | 33.9% |
| sine-ViT-Bk=1 | 73.75 | 3.54 |  |  |  |
| ViT-Bk=5 | 73.57 | $\mathbf{2.41}$ ↑ | 3.51 | $\mathbf{0.28}$ ↓ | 34.5% |
| sine-ViT-Bk=5 | 75.98 | 3.23 |  |  |  |
| ViT-Bk=10 | 73.78 | $\mathbf{3.09}$ ↑ | 3.43 | $\mathbf{0.31}$ ↓ | 35.1% |
| sine-ViT-Br=10 | 76.87 | 3.14 |  |  |  |
| ViT-Bk=30 | 75.54 | $\mathbf{2.74}$ ↑ | 3.13 | $\mathbf{0.04}$ ↓ | 37.3% |
| sine-ViT-Bk=30 | 78.28 | 3.09 |  |  |  |
| ViT-Bk=60 | 78.26 | $\mathbf{0.94}$ ↑ | 3.12 | $\mathbf{0.23}$ ↓ | 40.3% |
| sine-ViT-Bk=60 | 79.20 | 2.89 |  |  |  |
| ViT-Bk=100 | 79.37 | $\mathbf{0.56}$ ↑ | 2.81 | $\mathbf{0.03}$ ↓ | 44.5% |
| sine-ViT-Bk=100 | 79.93 | 2.78 |  |  |  |
| ViT-Bk=150 | 80.31 | $\mathbf{0.94}$ ↑ | 2.70 | $\mathbf{0.05}$ ↓ | 49.8% |
| sine-ViT-Bk=150 | 80.71 | 2.65 |  |  |  |
| ViT-Bk=250 | 80.48 | $\mathbf{0.57}$ ↑ | 2.63 | $\mathbf{0.01}$ ↑ | 60.3% |
| sine-ViT-Bk=250 | 81.05 | 2.64 |  |  |  |

**Table 2: This figure illustrates the Top-1 Accuracy and training loss for the ViT-Small and sine-ViT-Small models, both trained from scratch on the CIFAR-100 dataset across varying rank levels ($k$). It also includes the compression rate, indicating the proportion of parameters utilized relative to the total parameter count of the baseline model, thereby detailing the parameter usage versus model performance at different levels of model complexity.**
|  | Top-1 | Change | Training | Change | Compression |
| --- | --- | --- | --- | --- | --- |
|  | Accuracy | Loss | Rate |  |  |
| Baseline | 65.99 | - | 1.22 | - | 100% |
| ViT-Sk=1 | 54.50 | $\mathbf{3.64}$ ↑ | 2.66 | $\mathbf{0.08}$ ↓ | 33.9% |
| sine-ViT-Sk=1 | $\mathbf{58.14}$ | 2.58 |  |  |  |
| ViT-Sk=5 | 56.04 | $\mathbf{3.48}$ ↑ | 2.63 | $\mathbf{0.14}$ ↓ | 34.8% |
| sine-ViT-Sk=5 | $\mathbf{59.52}$ | 2.49 |  |  |  |
| ViT-Sk=10 | 56.98 | $\mathbf{4.23}$ ↑ | 2.60 | $\mathbf{0.23}$ ↓ | 35.8% |
| sine-ViT-Sk=10 | $\mathbf{61.19}$ | 2.37 |  |  |  |
| ViT-Sk=30 | 58.69 | $\mathbf{8.46}$ ↑ | 2.52 | $\mathbf{0.79}$ ↓ | 40.0% |
| sine-ViT-Sk=30 | $\mathbf{67.15}$ | 1.73 |  |  |  |
| ViT-Sk=60 | 62.07 | $\mathbf{5.64}$ ↑ | 2.33 | $\mathbf{0.63}$ ↓ | 46.6% |
| sine-ViT-Sk=60 | $\mathbf{67.71}$ | 1.70 |  |  |  |

Experimental setup.
We trained the ViT-Small and ViT-Base models from scratch, utilizing the CIFAR-100 and ImageNet-1k datasets, respectively, to establish our baseline performance metrics [, ]. The ViT-Small model, characterized by its two MLP layers with input/output dimensions of 384 and hidden dimensions of 1536, was modified by replacing the full-rank weight matrices with low-rank matrices across a range of ranks ($k$). Similarly, the ViT-Base model, which features two MLP layers with input/output dimensions of 768 and hidden dimensions of 3072, underwent a parallel modification, where its full-rank weight matrices were substituted with low-rank matrices for a range of ranks ($k$). For the training of the ViT-Base model, we adhered to the training methodology described in the Masked Autoencoder (MAE) methodology[], implementing a batch size of 1024. This structured approach allows us to rigorously evaluate the impact of introducing low-rank matrices to these model architectures.

Results.
Tab. 1 and Tab. 2 showcase the outcomes of training Vision Transformer (ViT) models from scratch on the ImageNet-1k and CIFAR100 datasets, respectively. These findings are juxtaposed with those of conventional baseline training of ViT models, which demonstrate that employing aggressive low-rank levels ($k$) notably compromises accuracy. Remarkably, the ViT-Base model, even when operating at a rank of $250$ with only 50% of its parameters in comparison to the baseline, attains the performance metrics of the baseline on the ImageNet-1k dataset, albeit at the cost of increased training loss. Additionally, the incorporation of sine-activated low-rank matrices consistently yields substantial improvements in test accuracy across all examined rank levels for both datasets. This suggests that the sine function significantly bolsters the representational capacity of low-rank weight matrices, as suggested by our theory in Sec. 3.2.

Analysis.
Large models, such as ViT-Base, with an excessively large number of parameters, are prone to overfitting, where they perform well on training data but poorly on unseen data, especially when trained on relatively "small" datasets like ImageNet-1k[][]. Low-rank learning techniques can help in designing models that generalize better to new data by encouraging the model to learn more compact and generalizable representations to reduce overfitting. Additionally, while ViT architectures often underperform on smaller datasets, this method introduces a novel approach for efficiently training ViT models using small data collections.

### 4.2 Large Language Model

Low-Rank Adaptation (LoRA) emerges as a highly effective strategy for finetuning large pre-trained models, as elucidated in []. LoRA specifically targets the adaptation of pre-trained weight matrices $\mathbf{W_{0}}\in\mathbb{R}^{m\times n}$ by limiting updates to a low-rank representation, expressed as $\mathbf{W_{0}}\mathbf{x}+\Delta\mathbf{Wx}=\mathbf{W_{0}x}+\mathbf{UV}^{T}
\mathbf{x}$, where $\mathbf{U}\in\mathbb{R}^{m\times k}$ and $\mathbf{V}\in\mathbb{R}^{n\times k}$ with the rank $k\ll min\{m,n\}$. This method does not introduce additional inference latency or necessitate reducing the input sequence length, thus preserving the model quality.
We conduct thorough experiments to evaluate the performance of our novel approach, termed sine-LoRA, against the standard LoRA framework, demonstrating the enhanced effectiveness of our method.

Dataset.
We evaluate the natural language understanding(NLU) task performance on the RoBERTa V3 base model []. Specifically, we adopt the widely recognized GLUE benchmark [], including CoLA [], MRPC [], QQP, STS-B[], MNLI [], QNLI [], and RTE [, , , ].

Setting.
In the Transformer architecture, there are four weight matrices in the self-attention module ($\mathbf{W_{\text{q}}},\mathbf{W_{\text{k}}},\mathbf{W_{\text{v}}},\mathbf{W_{
\text{o}}}$) and two in the MLP module. We follow up the LoRA architecture and implement low-rank adaptation only on $\mathbf{W_{\text{q}}}$ and $\mathbf{W_{\text{v}}}$. We study the performance of LoRA and sine-LoRA in terms of different rank $k={1,2,4,8}$.

**Table 3: This figure compares the performance and parameter count of LoRA and sine-LoRA models across varying $k_{\text{max}}$ settings on the GLUE benchmark. In every metric, higher scores signify superior performance. Notably, sine-LoRA models consistently outperform their counterparts, demonstrating enhancements in numerous specific metrics and delivering an overall average improvement across all evaluated ranks.**
| Method | #Params | COLA | MRPC | STSB | SST2 | RTE | QNLI | MNLI | QQP | Avg. | Change |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LoRAk=1 | 36.9K | 66.31 | 90.15 | 90.15 | 94.70 | 78.80 | 93.06 | 88.18 | 87.61 | 85.63 | $\mathbf{0.32}$ ↑ |
| sine-LoRAk=1 | 67.99 | 90.44 | 90.85 | 94.79 | 78.05 | 92.76 | 88.35 | 87.90 | 85.95 |  |  |
| LoRAk=2 | 73.7K | 68.38 | 89.42 | 89.19 | 95.02 | 78.27 | 93.32 | 89.15 | 88.57 | 85.99 | $\mathbf{0.45}$ ↑ |
| sine-LoRAk=2 | 68.93 | 90.79 | 90.94 | 94.81 | 79.10 | 93.29 | 88.26 | 88.70 | 86.44 |  |  |
| LoRAk=4 | 147.5K | 68.56 | 89.69 | 88.79 | 95.23 | 80.39 | 93.34 | 89.78 | 88.70 | 86.41 | $\mathbf{0.73}$ ↑ |
| sine-LoRAk=4 | 68.93 | 90.86 | 90.87 | 95.25 | 82.00 | 93.53 | 89.68 | 89.18 | 87.14 |  |  |
| LoRAk=8 | 294.9K | 68.62 | 89.82 | 89.50 | 95.25 | 80.37 | 93.56 | 89.86 | 88.83 | 86.57 | $\mathbf{0.42}$ ↑ |
| sine-LoRAk=8 | 68.54 | 90.22 | 90.85 | 95.11 | 81.82 | 93.58 | 89.69 | 89.38 | 86.99 |  |  |

Results.
We replicated the experimental framework of naive LoRA to establish a baseline, and then evaluated our sine-LoRA, as detailed in Tab. 3, using averages from five random seeds to ensure statistical robustness. Our results reveal that sine-LoRA surpasses the performance of the standard LoRA at different rank levels ($k$), highlighting the effectiveness of the sine function in enhancing the representation capabilities of low-rank matrices. Notably, sine-LoRA at $k=4$ not only exceeds LoRA’s performance at $k=8$ by 0.57 but also halves the parameter count, illustrating significant efficiency and parameter savings.

Analysis.
Within the LoRA framework, featuring a low-rank multiplication component $\Delta\mathbf{W}=\mathbf{UV}$, we enhance this low-rank component with a sine function and assess the efficacy of our method. This adaptation amplifies the update significance due to the "intrinsic rank" increase, facilitated by the sine activation. Consequently, our approach attains superior performance at reduced rank levels $k$, compared to LoRA, effectively decreasing the count of learnable parameters.

### 4.3 NeRF

Figure: Figure 4: Qualitative NeRF results for the LLFF dataset [, ] using rank $k=1$ and $k=5$. Using a Low-Rank model leads to a complete loss of signal for $k=1$, however, applying sine is able to reconstruct details even at the extreme low-rank case. At $k=5$ the sine-Low-Rank model is noticeably sharper and clearer than using the Low-Rank.

**Table 4: Quantitative results for NeRF evaluated on the LLFF dataset [, ]. Our sine-Low-Rank method outperforms the naive low-rank method across all levels $k$. For multiple instances (Fern, Flower, Leaves, Orchids, Room), the $k$=1 sine-Low-Rank model is able to outperform the $k$=5 model. We report the peak signal-to-noise ratio (PSNR) with the compression rate representing the percentage of parameters used in comparison to the parameter count of the Full-Rank NeRF model.**
|  | PSNR$\uparrow$ |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Fern | Flower | Fortress | Horn | Leaves | Orchids | Room | Trex | Average | Change | Compression<br>Rate |
| Full-Rank | $26.38$ | $27.54$ | $30.93$ | $28.20$ | $21.79$ | $21.33$ | $30.96$ | $27.68$ | $26.85$ | - | $100\%$ |
| Low-Rankk=1 | $15.03$ | $14.60$ | $14.74$ | $13.66$ | $12.89$ | $12.50$ | $15.04$ | $13.54$ | $14.00$ | $\mathbf{5.77}$ ↑ | 1.3% |
| sine-Low-Rankk=1 | 20.77 | 20.14 | 24.13 | 19.00 | 15.92 | 16.25 | 25.53 | 16.42 | 19.77 |  |  |
| Low-Rankk=5 | $20.64$ | $19.81$ | $24.90$ | $20.40$ | $15.74$ | $16.07$ | $22.74$ | $19.79$ | $20.01$ | $\mathbf{3.12}$ ↑ | 4.7% |
| sine-Low-Rankk=5 | 23.50 | 23.27 | 26.78 | 23.99 | 18.49 | 18.90 | 27.05 | 22.96 | 23.11 |  |  |
| Low-Rankk=10 | $22.83$ | $22.18$ | $25.96$ | $22.76$ | $17.36$ | $18.12$ | $26.12$ | $21.69$ | $22.12$ | $\mathbf{2.27}$ ↑ | 8.7% |
| sine-Low-Rankk=10 | 24.56 | 24.61 | 28.01 | 25.39 | 19.62 | 20.02 | 28.70 | 24.21 | 24.39 |  |  |
| Low-Rankk=30 | $24.48$ | $24.68$ | $28.10$ | $25.54$ | $19.36$ | $20.04$ | $38.92$ | $24.24$ | $24.42$ | $\mathbf{1.45}$ ↑ | 24.6% |
| sine-Low-Rankk=30 | 25.71 | 26.01 | 29.46 | 27.16 | 20.95 | 21.17 | 30.18 | 26.27 | 25.86 |  |  |
| Low-Rankk=60 | $25.26$ | $26.16$ | $29.50$ | $26.74$ | $20.39$ | $20.85$ | $30.00$ | $25.81$ | $25.59$ | $\mathbf{0.74}$ ↑ | 48.6% |
| sine-Low-Rankk=60 | 26.09 | 26.70 | 29.75 | 27.78 | 21.56 | 21.37 | 30.54 | 27.16 | 26.36 |  |  |

Neural Radiance Field (NeRF) represents 3D scene signals by utilizing a set of 2D sparse images[]. The 3D reconstruction is obtained by a forward pass $f_{\theta}(x,y,z,\theta,\phi)$, involving position ($x,y,z$) and viewing direction $(\theta,\phi)$. We evaluate our methods by training a NeRF model on the standard benchmarks LLFF dataset, which consists of 8 real-world scenes captured by hand-held cameras []. To evaluate our method on NeRF we substitute each fully dense layer with low-rank decomposition and use a range of rank levels ($k$).

Results.
Tab. 4 demonstrates that employing low-rank matrices in NeRF learning reduces parameter count while significantly enhancing compression. However, performance dips with very low rank levels ($k$), where models capture minimal information. Our methods, nevertheless, substantially elevate performance. For instance, with $k=1$, our sine-Low-Rank approach yields an average PSNR of 19.77, outperforming the naive low-rank by 5.77 and achieving a compression rate of merely $1.3\%$. Even at a $48\%$ compression rate, it surpasses the basic low-rank model by 0.8 PSNR, narrowly trailing the baseline by just 0.45 PSNR, as shown in Figs. 4 and 5. Our rate-distortion analysis, applying Akima interpolation for Bjøntegaard Delta calculation, reveals a BD-Rate of $-64.72\%$ and BD-PSNR of 2.72dB, signifying marked improvements in compression efficiency[, ].

Figure: Figure 5: Rate-Distortion curve for NeRF results (average over the LLFF dataset). The sine-Low-Rank NeRF models show significant rate-distortion improvement relative to the naive low-rank NeRF. Evaluating the Bjøntegaard Delta between the models using Akima interpolation shows BD-Rate: -64.72% and BD-PSNR: 2.72dB, indicating substantial improvement in compression quality from using the sine-Low-Rank model.

Analysis.
NeRF models, to a certain degree, tend to overfit entire 3D scenes, and a high training PSNR usually leads to a high testing PSNR. Employing structured weight matrices could result in a drop in performance due to the inherent constraints imposed by their structural design. Increasing the matrices’ rank enhances their memorization abilities significantly, especially when using a very low rank level $k$. Starting from a low frequency, there is a rapid and consistent increase in PSNR. Consequently, as we elevate the rank level $k$, our results gradually align with the baseline NeRFs, which serve as the upper bound.

### 4.4 3D shape modeling

For this experiment, we evaluate the binary occupancy task, which involves determining whether a given space or environment is occupied []. Following [], we sampled over a 512 $\times$ 512 $\times$ 512 grid with each voxel within the volume assigned a 1, and voxels outside the volume assigned a 0. We evaluate intersection over union (IoU) for the occupancy volumes. We used a coordinate-based MLP that includes two hidden layers, each with a width of 256 neurons, and employed the Gaussian activation function. The full-rank model achieves an accuracy of 97 (IoU) and further details are presented in supplementary materials. Fig. 6 shows the 3D mesh representation of the Thai Statue, visualized using the low-rank method and the sine-Low-Rank method for $k=1,2,5$. Applying the sine function to the low-rank matrix resulted in a significant enhancement and more precise shape delineation.

Figure: Figure 6: Binary occupancy field reconstruction on the Thai Statue instance from the Stanford Scanning Repository. Note that without a sine function, the Low-Rank model is unable reconstruct any finer details for the $k=1$ case; however, even at that level the sine-Low-Rank model is able to reconstruct fine structural details of the statue, including the trunks of the elephants. The $k=1$, $k=2$ and $k=5$ model utilizes only 2.1%, 2.9% and 5.2%, respectively, of the parameters of the full-rank model.

## 5 Limitations

Our exploration into sine-activated low-rank matrices illuminates their promising capabilities, yet it also has a limitation: notably, while these matrices can reach rank levels comparable to their full-rank counterparts upon the application of a sine function, their accuracy falls short. This highlights an ongoing challenge in finding the optimal balance between the need for sufficient parameterization to ensure high accuracy and the preferable rank of matrices. Overparameterization is widely recognized in the literature as vital for deep learning models to achieve strong generalization and optimization outcomes. However, our approach, despite increasing the rank, does not provide a clear pathway to achieving an efficient representation that can equal the accuracy of a full-rank model. Moving forward, developing strategies that not only enhance the rank but also clearly define the necessary degree of overparameterization will be crucial for creating cost-effective deep learning architectures, presenting an intriguing avenue for future research.

## 6 Conclusion

In this work, we developed a strategy that elevates the efficacy of general machine learning models, that employ feedforward layers, through parameter-efficient learning without additional parameter costs. Employing low-rank matrices reduces parameter cost, yet overly aggressive reductions, such as to rank=1, can detrimentally impact performance. Our method integrates a sine function within low-rank matrix decompositions to increase the rank of the decomposition leading to yielding an effective method to surmount their constrained representational capacity, thereby boosting model performance while maintaining the parameter cost. We gave a theoretical argument detailing why this method works and went on to validate the theory empirically on a variety of large-scale deep learning models over a broad set of applications. Furthermore, our findings reveal that learning with low-rank matrices contributes to mitigating overfitting, thereby facilitating the effective training of models even on limited datasets.

## 7 Acknowledgement

The authors thank Xueqian Li, Jianqiao Zheng, Shin-Fang Chng, and Daisy Bai for helpful discussion.

## Appendix 0.A Theoretical Framework

In this section we give the proof of Prop. 1 and Thm. 3.1 from Sec. 3.2 of the paper.

We recall from Sec. 3.2 the following notation:
For fixed fixed $\omega\,{>}\,0$, let $\sin(\omega\,{\cdot}\,\mathbf{A})$ denote the matrix obtained from a fixed $m{\times}n$ matrix $\mathbf{A}$ by applying the function $\sin(\omega\,{\cdot}\,\mathbf{x})$ component-wise to $\mathbf{A}$. Assuming $\mathbf{A}\,{\neq}\,0$ we define
$A_{\min}^{0}$ as:

$$ $A_{\min}^{0}=\min_{i,js.t.A_{ij\neq 0}}|A_{ij}|.$ (7) $$

Note that such a quantity is well defined precisely because $\mathbf{A}$ has a finite number of entries and all such entries cannot be zero from the assumption that $\mathbf{A}\,{\neq}\,0$.

Before we give the proof of Prop. 1, we will prove two lemmas.

###### Lemma 1

For a fixed $m{\times}n$ matrix $\mathbf{A}$. We have

$$ $||sin(\omega\mathbf{A})||^{2}_{F}\geq\omega^{2}(A_{\min}^{0})\text{ if }0< \omega<\frac{\pi}{3A_{\min}^{0}}$ (8) $$

where $A_{\min}^{0}$ is defined as follows:

$$ $A_{\min}^{0}=\min_{i,js.t.A_{ij\neq 0}}|A_{ij}|$ (9) $$

for $1\,{\leq}\,i\,{\leq}\,m$ and $1\,{\leq}\,j\,{\leq}\,n$.

###### Proof

Observer by definition of the Frobenius norm that

$$ $||sin(\omega\mathbf{A})||_{F}^{2}=\sum_{i=1}^{m}\sum_{j=1}^{n}sin(\omega \mathbf{A}_{ij})^{2}.$ (10) $$

Now observe that if $\mathbf{A}_{ij}\,{=}\,0$ then the term
$sin(\omega\mathbf{A}_{ij})^{2}\,{=}\,0$ and hence does not contribute to the above sum. Therefore, we find that

$$ $||sin(\omega\mathbf{A})||_{F}^{2}\geq sin(\omega A_{\min}^{0})^{2}.$ (11) $$

The goal is to now find a lower bound on
$sin(\omega A_{\min}^{0})^{2}$. In order to do this consider the function $f(\omega)\,{=}\,sin(\omega x)\,{-}\,\frac{\omega x}{2}$, where
$x\,{\in}\,\mathbb{R}$ is fixed and positive.

Differentiating this function we have

$$ $f^{\prime}(\boldsymbol{\omega})=xcos(\boldsymbol{\omega}x)-\frac{x}{2}.$ (12) $$

To find a critical point we solve the equation
$f^{\prime}(\boldsymbol{\omega})\,{=}\,0$ to find

$$ $cos(\boldsymbol{\omega}x)=\frac{1}{2}.$ (13) $$

Eq. 13 has the solution
$\omega x\,{=}\,\frac{\pi}{3}$. In order to check what type of critical point $\boldsymbol{\omega}x\,{=}\,\frac{\pi}{3}$ we need to look at
$f^{\prime\prime}(\boldsymbol{\omega})$

$$ $f^{\prime\prime}(\boldsymbol{\omega})=-x^{2}sin(\boldsymbol{\omega}x)<0$ (14) $$

when $\omega\,{=}\,\frac{\pi}{3x}$ implying that the critical point
$\omega\,{=}\,\frac{\pi}{3x}$ is a maximum point.

Observe that $f(0)\,{=}\,0$ it thus follows that $f(\boldsymbol{\omega})\,{\geq}\,0$ on the interval $[0,\frac{\pi}{3x}]$.

Applying this to the function $sin(\boldsymbol{\omega}A_{\min}^{0})$ we obtain that

$$ $sin(\boldsymbol{\omega}A_{\min}^{0})\geq\frac{\boldsymbol{\omega}A_{\min}^{0}} {2}\text{ if }0\leq\boldsymbol{\omega}\leq\frac{\pi}{3A_{\min}^{0}}.$ (15) $$

Substituting the lower bound in Eq. 15 into Eq. 11, we obtain the proposition.

The next lemma establishes an upper bound on the operator norm of $sin(\omega A)$. We remind the reader that the operator norm of $A$

###### Lemma 2

Let $A$ be a fixed $m{\times}n$ matrix. Then

$$ $||sin(\omega\mathbf{A})||_{op}^{2}\leq||\sqrt{\omega}\sqrt{|\mathbf{A}|}||_{op }^{2}$ (16) $$

where $\sqrt{|\mathbf{A}|}$ denotes the matrix obtained from $A$ by taking the absolute value and then square root component wise.

###### Proof

By definition we have

$$ $||sin(\omega\mathbf{A})||_{op}^{2}=\sup_{||x||_{2}=1}||sin(\omega\mathbf{A})x| |_{2}^{2}$ (17) $$

where $||\cdot||_{2}$ denotes the $2$-norm of a vector.

For any fixed unit vector $x$ we will show how to upper bound the quantity $||sin(\omega\mathbf{A})x||_{2}^{2}$. In order to do this we will use the fact that for $x\,{\geq}\,0$, we have the bound
$sin(x)\,{\leq}\,\sqrt{|x|}$.

$$ $\displaystyle||sin(\omega\mathbf{A})x||_{2}^{2}$ $\displaystyle=\sum_{i=1}^{m}\bigg{(}\sum_{j=1}^{n}sin(\omega A_{ij})x_{j}\bigg {)}^{2}$ (18) $\displaystyle\leq\sum_{i=1}^{m}\bigg{(}\sum_{j=1}^{n}\sqrt{\omega}\sqrt{| \mathbf{A}_{ij}|}x_{j}\bigg{)}^{2}$ (19) $\displaystyle=||(\sqrt{\omega})\bigg{(}\sqrt{|\mathbf{A}|}\bigg{)}x||^{2}_{2}.$ (20) $$

It follows that

$$ $\sup_{||x||=1}||sin(\omega\mathbf{A})x||_{2}^{2}\leq\sup_{||x||=1}||\sqrt{ \omega}\sqrt{|\mathbf{A}|}x||^{2}_{2}$ (21) $$

which implies

$$ $||sin(\omega\mathbf{A})||_{op}^{2}\leq||\sqrt{\omega}\sqrt{|\mathbf{A}|}||^{2} _{op}.$ (22) $$

We can now give the proof of Prop. 1 of the main paper.
In order to do so we will need the definition of the stable rank of a matrix. Assume $\mathbf{A}$ is a non-zero $m{\times}n$ matrix. We define the stable rank of $\mathbf{A}$ by

$$ $SR(A):=\frac{||\mathbf{A}||_{F}^{2}}{||\mathbf{A}||_{op}^{2}}.$ (23) $$

It is easy to see from the definition that the stable rank is continuous, unlike the rank, and is bounded above by the rank

$$ $SR(\mathbf{A})\leq Rank(\mathbf{A}).$ (24) $$

###### Proof (of Prop. 3.1 from Sec. 3.2 of main paper)

Observe that from Eq. 24 it suffices to prove the lower bound on $SR(\mathbf{A})$. This is immediate from lemma 1 and lemma 2.

We can also give the proof of Thm. 1.

###### Proof (of Thm. 3.1 from Sec. 3.2 of main paper)

From the assumption of the Thm. 3.1, we have that $N\,{\gg}\,k$. Further, we are assuming that both $\mathbf{U}$ and $\mathbf{V}$ have entries sampled from
$\mathcal{U}(-1/N,1/N)$. This means if we let $\mathbf{A}\,{=}\,\mathbf{U}\mathbf{V}^{T}$, then there exists a $C>0$ such that

$$ $A^{0}_{\min}=\frac{C}{N^{2}}.$ (25) $$

Furthermore, observe that

$$ $||\sqrt{|\mathbf{A}|}||_{op}\leq||\sqrt{|\mathbf{A}|}||_{F}\leq||\mathbf{A}||_ {F}$ (26) $$

which implies

$$ $\omega\bigg{(}\frac{A^{0}_{\min}}{||\sqrt{|\mathbf{A}|}||_{op}}\bigg{)}^{2} \geq\omega\bigg{(}\frac{C}{N^{4}}\bigg{)}\bigg{(}\frac{N^{4}}{mn}\bigg{)}= \omega(\frac{C}{mn}).$ (27) $$

Now observe that from Prop. 1 we have that

$$ $Rank(sin(\omega\mathbf{A}))\geq\omega\frac{A^{0}_{\min}}{||\sqrt{|\mathbf{A}|} ||_{op}}$ (28) $$

if $0\,{\leq}\,\omega\,{\leq}\,\frac{\pi}{3A^{0}_{\min}}$. We can rewrite this last condition to say that Eq. 28 holds if
$0\,{\leq}\,\omega\,{\leq}\,\frac{\pi N^{2}}{3}$. In particular, by using Eq. 27 we find that there exists $\omega_{0}$
within the interval $0\,{\leq}\,\omega_{0}\,{\leq}\,\frac{\pi N^{2}}{3}$

$$ $Rank(sin(\omega_{0}\mathbf{A}))\,{\geq}\,\omega_{0}\frac{A^{0}_{\min}}{||\sqrt {|\mathbf{A}|}||_{op}}\,{\geq}\,k\,{\geq}\,Rank(\mathbf{A}).$ (29) $$

This completes the proof.

###### Lemma 1

###### Proof

###### Lemma 2

###### Proof

###### Proof (of Prop. 3.1 from Sec. 3.2 of main paper)

###### Proof (of Thm. 3.1 from Sec. 3.2 of main paper)

## Appendix 0.B Ablation study

In this section, we present additional results from ViT-Small, NeRF, and 3D shape modeling to demonstrate the superiority of our methods.

**Table 5: This table illustrates the Top-1 Accuracy in pretraining the ViT-Small model with rank=1 on the CIFAR100 dataset using different frequencies, $\omega$, of sin($\omega\cdot$)**
|  | PSNR |
| --- | --- |
| Low-Rankk=1 | 54.5 |
| sine-Low-Rankk=1,ω=100 | 55.0 |
| sine-Low-Rankk=1,ω=200 | 55.8 |
| sine-Low-Rankk=1,ω=300 | 56.8 |
| sine-Low-Rankk=1,ω=400 | 58.0 |
| sine-Low-Rankk=1,ω=500 | $\mathbf{58.1}$ |
| sine-Low-Rankk=1,ω=600 | 57.6 |
| sine-Low-Rankk=1,ω=700 | 57.5 |

Figure: Figure 7: Ablation NeRF results for the LLFF dataset. These two figures show PSNR of NeRF using different frequencies when k=1 (on the left) and k=5 (on the right)
Refer to caption: extracted/2403.19243v1/nerfk15.png

### 0.B.1 ViT-Small on CIFAR100

In Tab. 5, we examine the performance of our method on training the ViT-Small model from scratch on the CIFAR100 dataset using different frequencies, when rank=1.

**Table 6: This table illustrates Intersection over Union (IoU) for 3D shape modeling across different rank levels ($k$). It also includes the compression rate, indicating the proportion of parameters utilized relative to the total parameter count of the baseline model, thereby detailing the parameter usage versus model performance at different levels of model complexity.**
|  | Intersection | Compression |
| --- | --- | --- |
|  | over Union | Rate |
| Full-Rank | 97.2 | 100% |
| Low-Rankk=1 | 84.3 | 2.1% |
| Sine-Low-Rankk=1 | 90.8 |  |
| Low-Rankk=2 | 88.0 | 2.9% |
| Sine-Low-Rankk=2 | 92.0 |  |
| Low-Rankk=5 | 93.4 | 5.2% |
| Sine-Low-Rankk=5 | 94.3 |  |
| Low-Rankk=20 | 95.4 | 16.8% |
| Sine-Low-Rankk=20 | 95.4 |  |

Figure: Figure 8: Ablation binary occupancy results for Thai Statue. This figure shows IoU accuracy of 3D shape modeling using different frequencies, when k=1.
Refer to caption: extracted/2403.19243v1/3d_1k.png

### 0.B.2 NeRF

In Fig. 7, we illustrate the impact of varying frequency on PSNR for cases where k=1 (shown on the left) and k=5 (shown on the right).

### 0.B.3 3D shape modelling

Tab. 6 reports the Itersection over Union (IoU) and Compression Rate of the binary occupancy task using different rank levels (k). Fig. 8 illustrates IoU using different frequencies when k=1.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weakness section.
soundness: 3
strengths: + The paper provides theoretical justification on benefit of the proposed sine-activated low rank matrices without adding additional parameters.
+ The experimental results show the proposed sine-activated low-rank decomposition works better than the original LoRA.
summary: This paper presents a theoretical framework that integrates a sinusoidal function within the low rank decomposition process to achieve efficient learning. The proposed method was evaluated on several applications such as LLMs parameter efficient fine-tuning, NeRF and 3D shape modeling.
weaknesses: - The paper talks about the benefits of using a sine function for increasing rank, but it could explain more about why this function was chosen over others and why it works. A discussion of its advantages would strengthen this work.

- While the proposed method is applied across various domains, the paper lacks a comprehensive comparison with existing low-rank approximation techniques. There are many existing LoRA variants and improved versions, e.g. [a] [b].

[a] LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning
[b] MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning

- It would be good to provide space and time complexity analysis of the proposed method. 

- There is a lack of quantitative results on a standard dataset for the 3D shape modeling task. It would be good to evaluate the method on other popular tasks such as text to image generation. 

- How do we choose the frequency parameter (ω) in practice? Is there an easy way to adjust it? As per the paper, (ω) higher increases the rank. But do we always pick highest (ω) ? How do we effectively use different frequencies within an interval ?

- For each application, the implementation details of the proposed method should be provided.

- Are there any implementation challenges, like increased computation time during training? Does this technique slow down the model
during inference, especially in environments with limited resources?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Figure 4 in the text is referenced as table 4.

Why is DoRA cited only once as Lora technique but then there is no reference to it's improved performance?

In line 182, and 765 is the A_{ij} \neq 0 maybe a A_{ij \neq 0} ? It looks as if the indexes should not be 0 and not as if the assumption of A \neq 0 is formalized. 

The operator norm in equation 5 has one | too much.

Why is the observation in line 786 important for 11? This should be true in any case.

Line 799 Says Equation equation 13. Probably an issue with cref

The sentence starting in 855 is somehow messed up.

Using parameter counts instead of compression rate in Figure 1 for ViT Classification would improve comparison with other state-of-the-art approaches.

Why does an increased rank improve performance? This point may have been demonstrated by experiments but an intuition could be given. Adding certain noises to the matrix may also increase it's rank without increasing it's parameter count but will probably not lead to improved performance. The link between an increased rank and improved performance could be made clear or hypothesized. Seeing that low-rank adaptation decreases performance may be not sufficient to claim an inverse relationship.

In line 480 I am not sure it is good to speak of overfitting regarding to NeRFs because this sounds as if it's negative when it is the desired behavior.

Figure 10 seems to indicate the frequency parameter has an optimum. How hard is it to choose it? Are the best results retrieved from a grid search or is there any intuition about it's value beforehand?
soundness: 3
strengths: - The idea is simple and fascinating. If there are no flaws in the proof, which I didn't find, this has a potential large impact in the domain.
- The mathematical background is well explained, motivated and proven.
- Figures like Figure 2 help in understanding the impact of the method's parameter.
- The method is shown on a variety of very different tasks from classification to Nerf.
- Extreme scenarios are studied like Nerf reconstruction using Lora with rank k=1
summary: This paper proposes the introduction of sinusoidal functions within LORAs to improve model performance across a range of tasks. The claim is to be able to balance parameter efficiency and model performance by increasing the rank with sinusoidal functions without adding parameters. The contributions are a theoretical framework, demonstrations of the performance and evaluation on several tasks such as computer vision, 3D shape modeling and NLP. A large part of the paper is the mathematical analysis of low-rank decomposition and the derivation of a non-linear low-rank decomposition as well as showing mathematically why its rank is larger. The supplementary material holds detailed proofs for the propositions and theorems.
weaknesses: - The method is only compared on self-implemented baselines and seems to underperform compared to other improvements in the domain. E.g. One of the citations even mentions DoRA but as an example of application of Lora techniques. They do have results of LLaMA3-8B LoRA which are similar to the results achieved in this paper but with DoRA they improve the performance on commonsense reasoning beyond this paper. When presenting an improved LoRA training technique the authors should compare their performance not only to the vanilla LoRA case but to other techniques like DoRA and ideally implement and evaluate an improved DoRA although I do not see with this specific work how that would be applied. Nevertheless, it seems important state-of-the-art here is missing, which is needed to position the paper in the research landscape.

There are a lot of minor syntax errors or errors in the equations which need to be corrected and do not give a polished impression (See questions)

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Please refer the weakness section above.
soundness: 4
strengths: Task Motivation:
The paper tackles a highly relevant problem—parameter-efficient learning. While overparameterization is effective for generalization, practical deployment in industry necessitates efficient architectures for cost-effectiveness.

Simple Yet Effective Approach:
The introduction of a sine function to augment low-rank decompositions is straightforward but impactful. As shown in Figure 3, the Sine LR method maintains representational power comparable to other activation functions while improving efficiency. The effectiveness of this method is demonstrated across multiple tasks and backbones, including LoRA in LLMs (ROBERTaV3 and LLaMA 3-8B), ViT (ViT-base and ViT-small), NeRF (vanilla NeRF on LLFF), and 3D shape modeling (Thai Statue example).

Technical Soundness:
The theoretical analysis highlights how controlling the frequency coefficient "w" in the sine function affects the linear independence of the original matrix "W". Figure 2 illustrates this effect, showing that the sine-activated low-rank decomposition facilitates efficient training without increasing the number of model parameters.
summary: This paper addresses the critical need for compact and parameter-efficient architectures in machine learning. While low-rank techniques are a popular solution for reducing model complexity, the authors introduce high-frequency sinusoidal non-linearity functions to enhance the capabilities of low-rank matrices. The paper provides a theoretical framework demonstrating how this modulation increases the rank of matrices without adding extra parameters. This sine-based non-linearity is incorporated into low-rank decompositions, effectively balancing efficiency and performance improvements across various benchmarks, including Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF), and 3D shape modeling.
weaknesses: Lack of Rationale Behind the Sine Function's Effectiveness: While the theoretical and experimental validations in Sections 3.2 and 4 establish the efficacy of the sine activation, the paper doesn't delve into why the sine function specifically enhances parameter-efficient training. I'm left wondering what inherent properties of the sine function drive these results. A clearer explanation of how the sine function contributes to increasing matrix rank or improving performance without adding parameters would significantly strengthen the argument.

Hyperparameter "w" Tuning:
The role of the frequency hyperparameter "w" isn't thoroughly addressed. Is there a need to fine-tune "w" for each task, or does it generally benefit from being larger? More importantly, how do we determine the optimal "w"? A systematic exploration or guidelines for selecting "w" across different tasks would be helpful, as the current approach seems to leave this critical aspect underexplored.


Applicability to CNN Architectures:
Although the paper demonstrates strong results on various architectures like ViT, LLMs, NeRF, and 3D shape modeling, it leaves a gap concerning CNNs. Given that CNNs are still widely used in practice, it would be valuable to explore whether the sine-activated low-rank method can be effectively applied to CNN architectures. A discussion or preliminary results in this area would broaden the paper’s applicability and relevance.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Advice & Question:
1. In formula 3, why the coefficient w and g are non-learnable ? If learnable, will it cause gradient problem? For the same model, has it been considered to provide different w and g for different linear layers? Can the recommended values for these two be provided?
2. Since other nonlinear functions are almost useless, why not directly give the sine function in Equation 3?
3. Regarding the right diagram in Figure 3, I don't think we can see a trend of the matrix rank increasing with the increase of frequency w. Because the rank of a matrix is only related to the number of nonzero singular values. So when w is 200 and 400, it seems that the ranks of the two are almost equal.
4. In the setting of section 4.1, why do not implement low-rank adaptations on all linear modules of LLM?
5. Does the rank of the decomposed matrix necessarily be less than that of the original matrix regardless of how the frequency is adjusted?
soundness: 3
strengths: This paper introduces concise and elegant improvement method for Low Rank Adaption (LoRA) and reveals its underlying principle by rigorous formula derivation. In addition to fine-tuning large language model, this method can even also achieving competitive results when directly replacing original full-rank weight, which provides some inspiration for model compression. Although there are some parts that I think are slightly lacking in depth, I would suggest accepting it after the following issues are addressed.
summary: This paper introduces concise and elegant improvement method for Low Rank Adaption (LoRA) and reveals its underlying principle by rigorous formula derivation. In addition to fine-tuning large language model, this method can even also achieving competitive results when directly replacing original full-rank weight, which provides some inspiration for model compression. Although there are some parts that I think are slightly lacking in depth, I would suggest accepting it after the following issues are addressed.
weaknesses: Advice & Question:
1. In formula 3, why the coefficient w and g are non-learnable ? If learnable, will it cause gradient problem? For the same model, has it been considered to provide different w and g for different linear layers? Can the recommended values for these two be provided?
2. Since other nonlinear functions are almost useless, why not directly give the sine function in Equation 3?
3. Regarding the right diagram in Figure 3, I don't think we can see a trend of the matrix rank increasing with the increase of frequency w. Because the rank of a matrix is only related to the number of nonzero singular values. So when w is 200 and 400, it seems that the ranks of the two are almost equal.
4. In the setting of section 4.1, why do not implement low-rank adaptations on all linear modules of LLM?
5. Does the rank of the decomposed matrix necessarily be less than that of the original matrix regardless of how the frequency is adjusted?

Meta-review: Motivated by the need of compact and parameter-efficient architectures in machine learning, this paper proposes a novel theoretical framework that integrates a sinusoidal function within the low-rank decomposition process aimed at lowering the number of model parameters with no or only slight degradation in performance in neural network architectures. The goodness of this approach has been  validated in a number of applications, namely Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF) and 3D shape modelling.

This work originally received almost all positive evaluations (5, 6, 8, 6), turned in unanimous appreciation after rebuttal (6, 8, 8, 6).

Positive aspects regard the approach itself, theoretically well presented and justified, as well as validated by a convincing experimental analysis.

Some remarks have also been raised, mainly oriented at requesting to better specify and explain some parts of the approach and experiments, hyper-parameter tuning and asking clarifications for the motivations of some of the methodological choices.

In conclusion, this paper is accepted for publication to ICLR 2024.

Decision: Accept (Poster)