LRA-QViT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Authors: Beom Jin Kang
NamJoon Kim
Hyun Kim
Venue: ICML 2025 poster (2025)
Keywords: Vision Transformer
Compression
Low-Rank Approximation
Quantization
Primary area: applications->computer_vision

Abstract: Recently, transformer-based models have demonstrated state-of-the-art performance across various computer vision tasks, including image classification, detection, and segmentation. However, their substantial parameter count poses significant challenges for deployment in resource-constrained environments such as edge or mobile devices. Low-rank approximation (LRA) has emerged as a promising model compression technique, effectively reducing the number of parameters in transformer models by decomposing high-dimensional weight matrices into low-rank representations. Nevertheless, matrix decomposition inherently introduces information loss, often leading to a decline in model accuracy. Furthermore, existing studies on LRA largely overlook the quantization process, which is a critical step in deploying practical vision transformer (ViT) models. To address these challenges, we propose a robust LRA framework that preserves weight information after matrix decomposition and incorporates quantization tailored to LRA characteristics. First, we introduce a reparameterizable branch-based low-rank approximation (RB-LRA) method coupled with weight reconstruction to minimize information loss during matrix decomposition. Subsequently, we enhance model accuracy by integrating RB-LRA with knowledge distillation techniques. Lastly, we present an LRA-aware quantization method designed to mitigate the large outliers generated by LRA, thereby improving the robustness of the quantized model. To validate the effectiveness of our approach, we conducted extensive experiments on the ImageNet dataset using various ViT-based models. Notably, the Swin-B model with RB-LRA achieved a 31.8\% reduction in parameters and a 30.4\% reduction in GFLOPs, with only a 0.03\% drop in accuracy. Furthermore, incorporating the proposed LRA-aware quantization method reduced accuracy loss by an additional 0.83\% compared to naive quantization.

LRA-QVIN Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Beom Jin Kang!
Nam Joon Kim
Hyun Kim
Abstract
using various ViT-based models. Notably, the Swin-B model with RB-LRA achieved 31.8% reduction in parameters and 30.4% reduction in GFLOPs, with only a 0.03% drop in accuracy. Fur- thermore, incorporating the proposed LRA-aware quantization method reduced accuracy loss by an additional 0.83% compared naive quantization.
Recently, transformer-base models have demon- strated state-of state-of-the-art performacce across various computer vision tasks, including image classifica- tion, detection, and segmentation. However, their substantial parameter count poses sigmificant chal- lenges for deployment in resource-constrainee environments such as edge mobile devices. Low-rank approximation (LRA) has emerged promising model compression technique, effec- tively reducing the number of parameters in trans- former models by decomposing high-dimensional weight matrices into low-rank representations. Nevertheless, matrix decomposition inherently introduces information loss, often leading a decline in model accuracy. Furthermore, existing studies on LRA largely overlook the quantiza- tion process, which is a critical step in deploy- ing practical vision transformer (ViT) models. To address these challenges, we propose robust LRA framework that preserves weight informa- tion after matrix decomposition and incorporates quantization tailored to LRA characteristics. First, we introduce reparameterizable branch-based low-rank approximation (RB-LRA) method cou- pled with weight reconstruction to minimize in- formation loss during matrix decomposition. Sub- sequently, we enhance model accuracy by inte- grating RB-LRA with knowledge distillation tech- niques. Lastly, we present an LRA-aware quan- tization method designed mitigate the large outliers generated by IRA,, thereby improving the robustness of the quantized model. To validate the effectiveness of our approach, we conducted extensive experiments on the ImageNet dataset
1. Introduction
Transformer-bssee models have demonstrated powerful per- formance in various fields, including computer vision, lan- guage processing, and speech processing (Vaswani et al., 2017; Dosovitskiy al., 2020; Gulati et al., 2020). Specifi- cally, vision transformer (ViT) models have achieved higher accuracy than convolutional neural networks (CNNs) (He et al., 2016; Choi et al., 2019; He et al., 2017) in vision tasks such as image classification (Touvron et al., 2021; Liu et al., 2021), detection (Carion et al., 2020), and segmenta- tion (Strudel et al., 2021). However, ViTs typically involve more parameters and computational complexity than CNNs, making them challenging deploy on mobile and edge de- vices with limited memory and computing resources, unlike high-performance GPU servers (Yin et al., 2022; Lee et al., 2024b). To address these challenges, various compression methods have been proposed, including quantization (Yuan et al., 2022), pruning (Jongho Kim, 2024) and low-rank approximation (LRA) (Lee et al., 2024a; Kumar, 2022).
LRA is an effective method for reducing model size by de- composing weights into low-rank matrices (Hajimolahoseini et al., 2022; Zhang et al., 2023). However, applying LRA often results in significant drop in accuracy due to the loss of weight information. Additionally, as the weights become shallower than those of the original large-scale model, becomes challenging to recover the original accuracy us- ing standard fine-tuning methods. To address these issues, previous studies have introduced fine-tuning methods based knowledge distillation (KD) (Hinton al., 2015). For example, (Guo et al., 2024) proposed method that regular- izes the perturbations caused by LRA while simultaneously applying KD to improve accuracy. (Yu & Wu, 2023) pro- posed a method that decomposes activations and determines the appropriate rank for each layer based on sensitivity to
Department of Electrical and Information Engineering and Research Center for Electrical and Information Technol- ogy, Seoul National University of Science and Technology, Scoul, Republic of Korea. Correspondence to: Hyun Kim <hyunkim scoultech.cc.rr>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Reparameterizatioo
Offinee Addition
Reparam,
Zero
Full-Rank Feature
Layer
FP32
Distill
Weight Quant.
B(w)
Threshold
Threshood
LRA. Branch
Nx
Apply
INT8
Image Input
WADS
Vee
PM
Low-Rank
Calibration
Repuram, Branch
Statie Quant.
FP32
IN18
Singular Value Decomposition
Mitigate
a
Weight
Update
FP32
INT8
Weight
[(w)
Low-Rank Approximation
Weight Reconstruction
Y'
E(a)
Y
(a) RB-LRA & WR
(b) Quantization
Figure The overall framework integrating proposed RB- RB-LRA method with quantization. framework involves two steps: (a) Fine-tuning: Construct the RB RB-LRA layer by initializing the branch weights with discarded matrices from LRA, and apply feature based knowledge distillation (KD) to improve performance. (b) Quantization: Create compressed weights by summing the weights both the LRA and reparameterizable branches, then apply weight quantization and calculate the quantization error. Layers with errors below the threshold are processed with WADS to scale activations and apply quantization.
layer-wise perturbations. However, these approaches are suboptimal in minimizing information loss during weight decomposition, making it difficult to achieve high accuracy only using LRA. Therefore, to further improve the perfor- mance of LRA -RA-based compression methods, more robust LRA-oriented approach is needed.
Based on these motivations, we propose a framework that integrates robust LRA with quantization to efficiently utilize large ViT models on mobile and edge devices. Figure 1 illustrates the proposed integrated LRA and quantization framework. Our contributions are summarized follows:
We introduce a robust reparameterizable branch-based LRA (RB-LRA) and weight reconstruction (WR) meth- ods that complement conventional LRA. The RB- LRA method employs reparameterizable residual branch compensate for L.RA-induced errors. The WR method initializes the RB-LRA branch using the weights removed by LRA, thereby reducing informa- tion loss. Consequently, RB-LRA significantly miti- gates accuracy degradation compared to existing LRA approaches.
Meanwhile, quantization reduces memory requirements and computational overhead by representing 32-bit floating point data as low-bit integers (e.g., INT4, INT8). In resource- constrained environments, such as mobile and edge devices, quantization is essential for memory and power-efficient utilization of ViTs (Choi & Kim, 2022; Kang et al., 2024).
Integrating LRA and quantization methods offers an effec- tive solution for the practical deployment of ViTs on mo- bile and edge devices, which are often constrained by lim- ited computing and memory resources. Specifically, LRA can reduce the number of parameters and computational costs, while quantization enables integer-based operations that enhance inference speed and minimize memory usage. However, the simultaneous use of these two compression techniques can lead to severe accuracy degradation. In fact, we observed substantial outliers in specific channels and tokens after applying LRA. yet existing LRA studies fail take quantization into consideration. Consequently, this observation highlights the need for quantization solution that specifically addresses the effects of LRA.
We observed large activation outliers when applying RB-LRA. To mitigate these outliers, we propose a weight-aware distribution scaling (WADS) method and suggest that per-token quantization is effeetive ap- proach to minimize accuracy drops caused by these outliers.
We applied the proposed RB-LRA and WADS to rep- resentative ViT-based models (DeiT (Touvron et al., 2021). and Swin transformer (Liu et al., 2021)). Con- sequently, the proposed methods not only effectively
2

---

LRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
20.0
posing the output feature map. However, (Guo al., 2024) could not directly address the fundamental information loss that occurs during LRA. Instead, it focused regularizing weight reconstruction perturbations and achieved improved performance by applying KD during fine-tuning. Moreover, (Yu Wu, 2023) decomposed the output feature map rather than the weight matrix, also relying on KD. contrast to these approaches. we propose a novel method that focuses minimizing information loss through the reconstruction of weights removed during the LRA process.
17.5
15.0
Acc. Drop (%)
12.5
10.0
7.5
5.0
2.2. Knowledge Distillation
2.5
Knowledge distillation (KD) is method for distilling knowledge from large model (i.e., teacher model) to smaller model (i.e., student model). KD is typically catego- based KD and feature- based KD. Response-basee based KD involves training the student model to replicate the probability distribution the teacher model's output. For example, (Hinton et al., 2015) uses soft labels generated through label smoothing (Szegedy et al., 2016). This approach enhances generalization performance by training the student model on probability distribution that spans all classes derived from the teacher's 's soft labels. In contrast, feature- based KD trains the student model to mimie the feature maps from specific layers or blocks of the teacher model. For instance, (Zagoruyko Komodakis, 2016) distills attention map features from the teacher model into the student model. This approach aims enhance per- formance by enabling the student model to replicate the feature distributions of the teacher model. Therefore, since LRA can significantly change the distribution of the original model, feature- based KD is an effective method to improve performance.
FQ-VIT
APQ-VIT
RepQ-ViT
AdaLog
ADFG-VIT
IGQ-VIT
Ours
rized into two categories: response- response-based
Figure 2. Comparison of Top-1 accuracy drop with various quan- tization methods and the our proposed compression framework combining LRA and quantization the ImageNet dataset for the DeiT-B model. Detailed numerical results are provided in Ap- pendix A.5.
enhance the trade-off between memory and accuracy but also achieve inference acceleration, making them well-suited for practical deployment on mobile and edge devices. Additionally, the proposed methods out- perform existing quantization methods, as shown in Figure when the DeiT-B model is compressed similar ratio (i.e., INT4 quantization).
2. Related Works
2.3. Quantization
2.1. Low-Rank Approximation
Quantization is a technique that reduces memory require- ments and computational overhead by converting 32-bit floating-point weights and activations into low-bit precision (Gholami et al., 2022; Kim & Kim, 2021). The commonly used uniform symmetric quantization can be expressed as follows:
LRA a method that decomposes a high-dimensional weight matrix into two low-rank matrices. It is primarily used to compress the fully connected (FC) layers, which key components of transformer models. Among various methods, singular value decomposition (SVD)-based LRA (Klema & Laub, 1980) is one of the most frequently adopted approaches. SVD decomposes matrix into a left singular vector matrix, a diagonal singular value matrix, and right singular vector matrix. In this process, the matrix is approxi- mated as a low-rank matrix by removing the smaller singular values while preserving the larger ones. However, removing these singular values leads information loss, which causes drop in accuracy. Accordingly, various methods have been proposed to address accuracy degradation. (Guo et al., 2024) proposed a method that integrates 1 -orrm-based weight perturbation regularization with feature-based KD. (Yu & Wu, 2023) used an eigenvalue decomposition for decom-
2 -
Q(x) - X x.
Q-
where X represents the input activation, and s is the seal- ing factor. a and ,3 denote the maximum and minimum values, respectively. b represents the number of quantiza- tion bits. Quantization the process of mapping data from the range of 32-bit floating-pon values into lower-bit integer domain. However, when large outliers are present, the guantization step can become excessively large, leading to increased quantization error (Ki & Kim, 2022). There- fore, is crucial to minimize accuracy degradation by han- dling these outliers while reducing bit precision. To address
3

---

IRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
outliers effectively, (Xiao al., 2023) used method for smoothing outliers in activations. (Kim et al., 2024) intro- duced quantization-aware distribution scaling (QADS) to handle outliers in MobileNet (Sandler et al., 2018) blocks. (Li et al., 2023) introduced quantization approach lever- aging a reparameterization nique, However, previous quantization studies do not con- sider the characteristics of the model after LRA, although observed large outliers in specific channels and tokens after LRA. This means that quantizing activations with out- liers may severely decrease model performance. To address these issues, we propose novel distribution scaling method and demonstrate that per-token quantization is suitable for LRA.
directly computing the E matrix preserves accuracy, it in- troduces additional parameters during the inference process, thereby negating the memory reduction advantages. There- fore, we design the matrix E as a low-rank residual branch and propose the RB-LRA method derived from this design:
distribution scaling tech-
y y V EX (V' + VU -x)
(5)
where EX (V i' + VU"T + V
In Eq. (5), the structure is formulated as a residual branch (i.e., V, U) relative to the input X, enabling its consolidation into single branch via the application of reparameteriza- tion technique:
y - FX (V++V) =
(6)
3. Proposed Method
The problem of finding the optimal È can be simplified to finding V and U. We determine the optimal V and Ú matrices through fine-tuning.
3.1. Singular Value Decomposition-baset Low-Rank Approximation
First, we describe an SVD-based LRA. Specifically, the SVD for FC layers with weight matrix w E Rmxx is ex-
3.3. Initialization Method of Reparameterizable Branch
pressed as follows:
Since the V. Ü. matrices are not foundation model-oriented elements, the initialization method is crucial. Therefore, we propose WR, which leverages the original weights of the foundation model that are discarded during LRA ap- plication, considering the characteristics of the FC layer operations. In Eq. (3), we remove all submatrices except for those corresponding to the top singular values and their related elements in the U and V matrices. The removed submatrices are expressed as follows:
= USV
(2)
The matrix w is decomposed into U E _mxm ,S Rm and V E R"x= S is diagonal matrix consisting of min(m, n) singular values. Matrices U, V represent the left and right singular vectors, respectively, and are orthonormal matrices. LRA is performed using only the top r diagonal elements of the S matrix. The SVD-based LRA can be expressed as follows:
Udet = Utrl
U' = -Upor) (t {
Sael Sprrii
(3)
Vdet Virsa)
V' = -S MEna))
We obtain the matrices U,S, and V by applying SVD. The value r satisfies r < min(m, n). The process of reconstruct- ing the original FC layer using the removed submatrices and the LRA matrix is expressed as follows:
where U' E m V' E Ruxr are the low-dimensional decompositions of the matrix w, respectively. In this case, the total number of parameters changes from Omm) to O(r(m+ nn). Since r < min(m, n), this results in a signif- icant reduction in the number of parameters. Consequently, we obtain the wt x R v' (U xx).
y' = =- = Concaat T Tx U
(8)
y V =V M
+ (Sac V Myra
3.2. Reparameterizable Branch- based Low-Rank Approximation
We exploit the property that the (SaeVVas)) matrix can be reconstructed via addition, and utilize its value to initial- ize the V matrix. Conversely, since the UTr matrix is con- structed through concatenation, it not fully reconstructed. As a result, the U matrix is initialized to zero and then opti- mized through fine-tuning. When comparing the sizes of the weight elements removed after applying the compression method, the LRA removes 0(m² + n²² - -r(m + +n) weight elements. In contrast, the proposed WR method removes only O(m(n - r)) weight elements. As a result, the pro- posed WR method can significantly reduce information loss
In Eq. (3), the operations of the FC layer involving the approximated V' and U' can be expressed in the following reformulated form:
y = V (U'T)) x + EX
(4)
FX (w V
The matrix E denotes the difference between the founda- tion model's s weight matrix and the LRA matrix. Although
4

---

IRA-QViT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
compared to LRA. Consequently, unlike conventional LRA methods, our method can be fine-tuned achieve optimal performance with minimal loss of weight matrix informa- tion and effectively reduce parameters during inference. In addition, a theoretical analysis of the proposed RB-LRA and WR methods is provided in Appendix A.I.
110
130
12
3.4. Block-Level Knowledge Distillation
Tolon mlees
Tok index
After applying RB-LRA, we perform fine-tuning using KD to achieve high accuracy. When applying KD, selecting the appropriate teacher model and determining which fea- tures from the teacher model to distill are critical consider- ations. RB-LRA compresses a large-scale original model into a shallower model. In this scenario, high accuracy can be achieved by utilizing the large-scale original model as the teacher model. Distilling layer-wise feature knowledge from the teacher model can lead to overfitting during the fine-tuning process (Liu et al., 2023). Therefore, we apply encoder block-level KD, with the loss function designed as follows:
Channel index
Chatmel index
(8)
(b)
Figure 3. Visualization of activation distribution after applying RB-LRA to the Swin-B model. The x-axis represents the channel number. the y-axis represents the token number, and the z-axis represents the absolute value. (a) shows the output activation dis- tribution after applying RB-LRA to the 11th encoder block. In this case, outliers become more severe in certain channels. (b) shows the output activation distribution after applying WADS the 11th encoder block, showing a reduction in outliers compared to (a).
Led (S)( (l)))
(9)
Swin-B model. Based on these observations, we identified two main causes of the decrease in accuracy.
where H represents the output of the I-th encoder in the teacher model. f represents the output of the (-th encoder in the student model, and iı denotes the input to the I-th encoder. The final loss function considering block-level KD loss can be expressed as follows:
First, as shown in Figure 3, we found outliers in certain channels of the RB-LRA layer activation in certain encoder blocks. Based on these observations, we propose an activa- tion scaling method that uses channel scaling vector to mitigate the large outliers in specific channels, as follows:
L = aLee + kLdd
(10)
Y, = Q(X/a)((aW)
(11)
where Lce represents the commonly used cross-entropy loss (Cox, 1958). a and ,3 denote the scale terms of the loss, respectively. For simplicity, we set both a and /3 to 1. Finally, we use a combination of RB-LRA and block-level KD to improve accuracy. In particular, unlike previous methods, the RB-LRA method achieves higher accuracy by applying block-wise KD while minimizing the loss of information caused by applying LRA.
where a and Q(+) represent the channel scaling vector and the quantization function, respectively. X.W, and Y, rep- resent the input activation, weight, and output activation, respectively. The loss function to determine the optimal that minimizes the quantization error is expressed as follows (Kim et al., 2024):
((a) =
- X -x
(12)
3.5. Low-Rank Approximation-Aware Quantization
where Q(-) represents the quantization function. X repre- sents the input activation, and represents the weight. By using this method, the activation quantization error can be minimized. However, when applying this method model with RB-LRA, the weight quantization error becomes severe in certain layers. Therefore, we propose WADS to solve this problem. First, we measure the quantization loss of all RB- LRA layer weights. The quantization loss of the weights can be measured as follows:
We use the commonly used configuration of 8-bit per- channel and per-layer uniform quantization for each of the weights and activations, respectively, as the bascline (Lin et al., 2021). In the baseline model, we observed a drop in accuracy of 2. 19% and 1.30% for Swin-T and Swin-B, respectively, compared RB-LRA with 32-bit floating- point precision. To analyze the causes of this accuracy drop, measured quantization error per layer. Consequently, we found that the RB-LRA layer exhibited significant quantization error. To identify the cause, we analyzed the characteristics of the activation distribution by visualizing the distribution in the RB-LRA layer. Figure 3 shows the activation distribution of a specific RB-LRA layer in the
[(w) = l[Q(W) w|
(13)
where W and Q(-) represent the weight before quantization and quantization function, respectively. We only apply acti- vation scaling for layers with smaller than average values of
5

---

LRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Model
Method
KD Method
Params(M)
GFLOPs
ACC.(%)
Diff.
Baseline
5.7
2.2
72.17
LRA
5.2 (-8.8%)
1.8
68.40
-3.77
DeïT-T
RB-LRA
5.2 (-8.8%)
1.8
70.92
-1.25
RB-LRA KD
Feature
5.2 (-8.8%)
1.8
71.70
-0.47
Baseline
86.6
33.7
81.85
LRA
44.4 (-45.7%)
17.1
78.76
-3.09
DeiT-B
PELA (Guo al., 2024)
Feature
44.1 (-49.1%)
17.0
81.00
-0.85
RB-LRA
44.4 (-45.7 %
17.1
79.93
-1.92
RB-LRA + KD
Feature
44.4 (-45.7%
17.1
81.12
-0.73
Baseline
28.3
8.6
81.37
LRA
21.1 (-25.4%)
6.7
77.30
-4.07
Swin-T
RB-LRA
21.1 (-25,4% )
6.7
80.27
-1.1
RB-LRA + KD
Feature
21.1 (-25.4% %%
6.7
80.49
-0.88
Baseline
88.1
30.3
83.47
LRA
60.1 (-31.8%)
21.1
81.75
-1.72
AAFM+GFM (Yu Wu, 2023)
Feature
60.2 (-31.7%)
82.99
-0.48
Swin-B
PELA (Guo et al., 2024)
Feature
62.2 (-29.4% 44))
21.3
82.50
-0.97
RB-LRA RB-LRA+KD
60.1 (-31.8 9))
21.1
82.88
-0.59
Feature
60.1 (-31.8%)
21.1
83.44
-0.03
Table 1. Comparison of Top-1 Accuracy (%) across various models with RB-LRA applied on ImageNet dataset.
L(w) to reduce the accuracy drop owing weight quanti-
4. Experimental Results
zation errors. Furthermore, we redesign the loss function to find the optimal channel scaling vector. The process of de- termining the optimal channel scaling vector o is expressed as follows:
4.1. Comparison Results on Various Networks
RB-LRA for Image Classification Tasks: We evaluated the performance by applying RB-LRA the DeiT (Tou- vron et al., 2021) and Swin transformer (Liu et al., 2021). Specifically, RB-LRA was applied to all the FC layers within the encoder blocks of each model. All experiments used an initial learning rate of 1e-5, the Adam W optimizer (Loshchilov & Hutter, 2017). and the cosine anncaling scheduler (Loshchilov & Hutter, 2016). Experiments were conducted single A100 GPU within the TiMM (Wight- man, 2019) environment.
a arg min [[a)) w|]}}
(14)
where L(a) refers to Eq. (12). This prevents from be- coming excessively large during optimization, which may cause outliers in the weights. The optimal channel scaling vector is determined using only a subset of the training dataset, enabling optimization without requiring access to the full dataset. As result, by utilizing an activation scal- ing method that considers the weights quantization errors, achieve better performance compared the previous activation scaling method.
To demonstrate the effectiveness of our proposed method combining RB-LRA and block-level KD (i.e., RB- LRA+KD) on image classification tasks, Table compares its performance with state-of-the-art (SOTA) LRA meth- ods, including PELA (Guo et al., 2024) and AAFM + GFM (Yu & Wu, 2023). As the AAFM+GFM method does utilize the full dataset, its performance is compared exclu- sively with that of the Swin-B model, for which full-dataset performance is reported in the referenced paper.
Mcanwhile, we also found that outliers emerged in spe- cific tokens. per-layer quantization applied to activa- tions with these outliers, quantization error may be sub- stantial: however, using finer granularity than per-layer quantization can help reduce quantization errors. Therefore, propose that per-token quantization is appropriate method when considering token outliers and integer domain FC layer operations. Consequently, the proposed WADS method and pok-token activation quantization effectively mitigate the accuracy degradation caused by outliers when applying RB-LRA.
In addition, to demonstrate the mitigation of information loss in the LRA method, we compared the performance of RB-LRA with that of the conventional LRA (i.e., naive SVD- based LRA) without applying the KD method. The experi- mental results show that applying RB-LRA and beoek-level KD enables the DeiT-T and DeiT-B models to reduce param- eters by 8.8% and 45.7% and giga floating-point operations (GFLOPs) by 18.2% and 49.3%, with only 0.47% and 0.73% accuracy drop, respectively. For the Swin-T and Swin-B models, RB-LRA+KD achieved parameter reductions of
6

---

LRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Model
Method
Prec. FP32
Size(MB)
ACC.(%)
Diff.(%)
Baseline(RB-LRA.
20.8
71.70
NaivePTQ
70.90
-0.80
SmoothQuant (Xiao al., 2023)
71.43
-0.27
DeiT-T
Repq-Vit ViT et al., 2023)
INT8
5.2
71.38
-0.32
QADS (Kim et al., 2024)
71.40
-0.30
WADS
71.52
-0.18
Baseline(RB-LLC LRAAA)
FP32
177.6
81.12
NaivePTQ
79.62
-1.50
SmoothQuant (Xiao et al., 2023)
80.26
DeiT-B
Cepq-ViT (Li et al., 2023)
INT8
44.4
80.37
-0.75
QADS (Kim et al., 2024)
79.82
-1.30
WADS
80.56
-0.56
Baseliec(BB-LRA)
FP32
84.4
80.49
NaivePTQ
78.30
-2.19
Smoothhuant (Xiao et al., 2023)
80.00
-0.49
Swin-T
Repq-Vit ViT (Li et al., 2023)
INT8
21.1
80.08
-0.41
QADS (Kim al., 2024)
80.04
-0.45
WADS
80.20
-0.29
Baseline(BB-LRA)
FP32
240.4
83.44
NaivePTQ
82.14
-1.30
Swin-B
SmoothQuant (Xiao et al., 2023) QADS (Kim et al., 2024)
82.76
-0.68
INT8
60.1
82.37
-1.07
WADS
82.97
-0.47
Table 2. Comparison of size (MB) and accuracy between naive quantization, previous quantization methods, and proposed WADS.
25.4% and 31.8%, respectively, along with GFLOPs reduc- tions of 22.1% and 30.4%. These reductions were achieved with minimal accuracy drops of only 0.88% and 0.03%, re- spectively. As a result, the proposed method demonstrated higher accuracy and greater robustness compared to conven- tional LRA approaches. Furthermore, we demonstrate that method performs well tiny networks.
all other methods. This indicates that WASD, which consid- ers the weight quantization error, is the optimal method for combination with RB-LRA. Furthermore, a comprehensive analysis of the variations in quantization error due to the application of WADS is provided in Appendix A.3, while the analysis of performance variations across different quan- tization methods (iee. WADS, per-token) is discussed in A.4.
Quantization: In this subsection, we analyze the perfor- mance of 8-bit naive post-training quantization (NaivePTQ)) and the proposed WADS after applying RB-LRA. We also compare WADS with the SmoothQuant (Xiao et al., 2023), Repq- VIT (I.i al., 2023), and QADS (Kim al., 2024) methods demonstrate WADS effectiveness. The proposed method applies WADS to models trained using both RB- LRA and block-level KD. According to the previous anal- ysis, per-token quantization is applied to activations, and per-channel quantization is applied to weights. We applied static quantization and performed calibration using 32 data samples to determine the optimal scaling factor. Subse- quently, we conducted WADS optimization using the same dataset. This process is performed layer by layer for RB- LRA. finding the optimal channel scaling vector that min- imizes quantization loss through 300 iterations of scaling vector updates. As shown in Table 2, WADS significantly mitigates accuracy drops across all ViT-hased models un- der the same model size. In particular, DeiT-B and Swin-B achieved 0.94% and 0.83% higher accuracy, respectively, compared NaivePTQ. In addition, our proposed method achieved the highest accuracy across all models compared to
4.2. Performance Comparison with SOTA Compression Methods
We evaluate the proposed framework the Swin-B model, comparing its performance against SOTA methods in both quantization and LRA. Since these approaches differ in their original baseline accuracies, we report accuracy degra- dation relative to each method's s respective baseline to en- sure fair comparison. As shown Table our method achieves an accuracy 82.97%, while simultaneously ap- plying LRA and quantization twoo compression techniques that are typically challenging to integrate without significant accuracy loss. Compared to existing LRA-based methods such as PELA and AAFM+GFM, this result demonstrates a highly competitive trade-off between model size and accu- racy. Furthermore, compared to existing quantization meth- ods, our approach minimizes accuracy degradation even with smaller model size of 6.1 MB. These results demon- strate that the proposed framework achieves competitive performance compared to both SOTA LRA methods and other representative compression methods.

---

IRA-QVIT:
Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Method
LRA
Quant
Train
Model Size(MB)
Baseline Acc. (%)
Acc. (%)
PTQ4ViT (Yuan et al., 2022)
84.18 (-1.09)
APQ-VIT (Ding et al., 2022)
84.01 (-1.26)
66.1
85.27
Qrrop (Wei et al., 2022)
84.33 (-0.94)
I&S-ViT (Zhong et al., 2023)
84.94 (-0.33)
PELA (Guo et al., 2024)
248.8
82.50 (-0.97)
AAFM+GFM (Yu Wu, 2023)
240.8
83.47
82.68 (-0.79)
Ours
60.1
82.97 (-0.5)
Table Comparison of size (MB) and Top-1 accuracy on ImageNet dataset between the proposed framework SOTA compression
methods using the Swin-B Model.
Model
Method
Prec.
Size(MB)
Android(ms)
Xavier(ms)
Baseline
FP32
346.4
275.6
150.7
DeîT-B
RB-LRA
FP32
177.6
153.2
73.6
RB-LRA + WADS
INT8
44.4
86.7
59.4
Baseline
FP32
113.2
98.5
61.1
Swin-T
RB-LRA
FP32
84.4
83.6
38.6
RB-LRA + WADS
INT8
21.1
67.3
27.4
Baseline
FP32
352.4
287.4
140.5
Swin-B
RB-LRA
FP32
240.4
226.3
102.2
RB-LRA + WADS
INT8
60.1
155.3
96.2
Table 4. Latency (ms) comparison across methods for executing the DeiT and Swin Transformer models on mobile and edge devices.
4.3. Latency Analysis on Mobile and Edge Devices
tection (Chen al., 2019) environment. For comprehen- sive analysis, we compared the performance of the proposed method with CNN-based ResNet (He et al., 2016) and ViT- based PVT (Wang et al., 2021) backbone networks. As shown Table our method achieves outstanding perfor- mance even on these downstream tasks. In the detection task, achieved average precision (AP) of 42.5 with 15.1% reduction in parameters and a 6. 6.1% reduction in GFLOPs. This represents only a 0.2 AP drop compared with the Swin- T backbone network. Additionally, our method outperforms the Re backbone network by 2.5 AP while using 3.8 M fewer parameters. In the segmentation task, our method achieved an AP of 39.0, which equivalent the accuracy of the PVT-M backbone network. However, it offers bet- ter parameter-accuraat trade-off, reducing both parameters and computational cost (GFLOPs) by 36.5% and 3 31.3%, respectively.
We validate the practical efficiency of our proposed method by measuring latency real mobile and edge devices. Specifically, latency measurements were conducted on an Android smartphone for the mobile device and on the NVIDIA Jetson Xavier platform for the edge device. par- ticular, latency on the Android smartphone was measured using the 3.36 GHz Cortex-X3 main core, while the Ten- sorRT engine employed for latency measurements the NVIDIA Jetson Xavier platform. Table 4 shows that the model incorporating both RB-LRA and WADS achieved in- ference speedups ranging from 1.9x to 3.2x on mobile de- vices and from 1.5x to 2.5x on edge devices. These results validate that the proposed RB-LRA and WADS-based quan- tization method not only optimizes the trade-off between accuracy and memory usage but also delivers significant acceleration benefits, making it practical for deployment in mobile and edge device environments.
further demonstrate the generalizability our approach to diverse applications, we evaluate its effectiveness on the pose estimation task. In particular, we integrate RB-LRA into the ViTPose-B (Xu et al., 2022), followed by fine-tuning and evaluation on the MSCOCO Keypoint dataset. As shown in Table 6, applying RB-LRA achieves 25.7% reduction in model size, with only a 0.9% and 0.6% drop in AP and average recall (AR), respectively. These results highlight the effectiveness of our method in preserving performance while significantly reducing model complexity, even in structurally distinct vision tasks.
4.4. A Ablation Studies
Evaluation on detection and segmentation tasks: To demonstrate the effectiveness of our proposed method, we conducted performance evaluations for detection and seg- mentation tasks using the MSCOCO dataset (Lin et al., 2014). Specifically, we used an ImageNet pre-trained Swin- T model as the backbone network for the Mask-RCNN model. We then applied RB-LRA to the backbone network before training. Experiments were conducted in the MMDe-
8

---

LRA-QViT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Backbone
Params(M)
GFLOPs
Apboz
Apmask
ResNet-50 (He et al., 2016)
44.4
250.2
40.0
36.1
PVT-M (Wang et al., 2021)
63.9
351.2
42.0
39.0
Swin-T (Liu et al.. 2021)
47.8
256.8
42.7
39.3
Swin-T + RB-LRA
40.6
241.2
42.5
39.0
Table 5. Performance comparison of Bask-RCNN models with various backbone networks on the COCO validation dataset.
Model
Method
Params(M)
AP
AR
Model
Method
Params(M)
PPL
WER
ViTPose-B
RB-LRA
Baseline
66.8 (-25.7% %)
89.9
75.0
75.9
80.4
81.0
GPT-2 Medium
RB-LRA
Baseline
249.4 (-29.7% %%
354.8
19.51
18.72
Baseline
116.8
5,4
Conformer-L
RB-LRA
86.2(-26.3%%
5.6
Table 6. Performance of ViTPose with RB-LRA on the COCO
Keypoint Dataset.
Table 8. Evaluation of PPL on the Wikitext- 103 dataset for GPT-2 Medium and WER on the LibriSpeech dataset for Conformer-L., with RB-LRA applied.
Model
Init. Method
Acc.(%)
(%f
Baseline
81.37
Swin-T
Random
79.54
-1.83
WR
80.49
-0.88
Baseline
83.47
the training split of the respective datasets. As shown in Ta- ble 8, despite achieving a compression ratio of 26%~~30% the proposed method incurs less than a 1% increase in PPL and WER. Ultimately, these results demonstrate that our method extends beyond computer vision tasks, exhibiting strong generalizability and scalability across diverse modal- ities.
Swin-B
Random
82.65
-0.82
WR 83.44
-0.03
Table 7. Comparison of Top-1 Accuracy (%) on the ImageNet dataset under varying initialization methods for the branch.
Analysis of performance variations based on initializa- tion methods: We conduct a performance analysis of the WR initialization method. Specifically, after integrating RB- LRA into the Swin-T and Swin-B models, the V branch weights are initialized using either the WR method or a random initialization approach. In this process, random initialization weights W (O 1) utilized. Subse- quently, fine-tuning is performed using the same KD method. As shown in 7, the WR method achieves higher accuracy compared to the random initialization method. This demon- strates that tne-ttunig with the WR method, which reduces information loss, is an effective approach for achieving op- timal accuracy. A more detailed analysis is provided in Appendix A.2.
5. Conclusion
In this paper, we propose a compression solution aimed at enabling memory- and power-efficient deployment of VIT models in resource-constrainet environments, such as mobile and edge devices. The proposed RB-LRA method in- troduces reparameterizable branch specifically optimized to address errors induced by LRA. Furthermore, the WR method initializes the weights of the RB-LRA branch using the weights discarded during the LRA process, effectively minimizing information loss, Additionally, by combining RB-LRA with block-level KD, we effectively balance the trade-off off between compression ratio and accuracy. Subse- quently, we applied optimized quantization that considers the characteristics of LRA. We observed large outliers in certain channels and tokens of the RB-LRA output activa- tions. Based on these observations, we proposed WADS to mitigate these outliers. Through various experiments, we demonstrated that WADS significantly reduces accuracy loss. Additionally, we found that the per-token quantization method effectively handles token outliers making it suitable for quantizing LRA layers. Consequently, our approach, which integrates RB-LRA with our proposed quantization techniques, significantly reduces model size while maintain- ing high accuracy compared to the baseline. As a result, this solution enables the efficient deployment various transformer-basee models mobile and edge environments.
Evaluation of generalization capability across diverse modalities
To assess the cross-domain generalizability of the proposed RB-LRA method beyond computer vision tasks, we evalu- ate its effectiveness on both language and speech processing tasks. For language modeling, we apply RB-LRA the GPT-2 Medium (Radford et al., 2019) and measure per- plexity (PPL.) on the Wikitext- 103 dataset (Merity, 2016). For speech recognition, we integrate RB-LRA into the Conformer-L (Gulati et al., 2020) and evaluate word error rate (WER) on the LibriSpeech test-clean dataset (Panayotov al., 2015). In both cases, fine-tuning is performed using

---

IRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Acknowledgements
international conference on multimedia, pp. 5380-5388,
2022.
This research was partly supported by the Minityyn of Science and ICT), Korea, under the (Inoommtnnt Technology Research Center) support program (IITP-2025- RS-2022-001162255) supervised by the IITP(Intiiute for Information & Communications Technology Planning & Evaluation) and the Technology Innovation Program(or In- dustrial Strategic Technology Development Program) (RS- 02307333, Development and Validation of On-Device Al Semiconductor for Manufacturing Automation Robots through sVLM-Based Situational Awareness) funded By the Ministry of Trade Industry & Energy(MOTTE, Korea).
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.. Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Eckart, C. and Young, G. The approximation of one matrix by another of lower rank. Psychometrika. 1(3):211-218, 1936.
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for effi- cient neural network inference. In Low-Power Computer Vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. Con- former: cnnvolution-uugmene
transformer for speech recognition. arXiv preprint arXiv:2005.0m 100, 2020.
References
Guo, Y., Wang. G., and Kankanhalli, M. Pela: Learning parameter-effictte models with low-rank approximation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 15699-15709, 2024.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229, Springer, 2020.
Chen, K., Wang, J., Pang. J., Cao. Y., Xiong. Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: (BVv::900.07111 2019.
Hajimolahoseini, H., Ahmed, W., Rezagholizadeh, M., Par- tovinia, V., and Liu, Y. Strategies for applying low rank decomposition to transformer-based models. In 36th Conference on Neural Information Processing Systems (NeurIPS2022), 2022.
Choi, D. and Kim, Hardware-friendly logarithmic quan- tization with -reecisireiiion for mobilenetv2. In 2022 IEEE 4th international conference artificial intelli- gence circuits and systems (AICAS), pp. 348-351. IEEE, 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Choi, J., Chun, D., Kim, H., and Lee, H.-J. Gaussian yolov3: An accurate and fast object detector using localization uncertainty for autonomous driving. In Proceedings of the IEEE/CVF International conference on computer vision, pp. 502-511, 2019.
He, K., Gkioxari, G., Dollár, and Girshick, Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv: /503.0253,, 2015.
Cox, D. R. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statisti- cal Methodology, 20(2):215-232, 1958.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint 0rXiv:2106.09685 2021.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized lims. Advances in Neural Information Processing Systems, 36, 2024.
Jiang Y., Sun, N., Xie, X., Yang, F., and Li, T. Adfq- vit:
Ding, Y., Qin, H., Yan, Q., Chai, Z., Liu, J., Wei, X., and Liu, X. Towards accurate post-training quantization for vision transformer. In Proceedings of the 30th ACM
post-training quan- tization for vision transformers. arXiv preprint arXiv:440.002663 2024.
10

---

IRA-QViT:
Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Jongho, L. and Kim, H. Det-vit: High-frequency pruned vision transformer with discrete cosine transform. IEEE Access, 2024.
Liu, Z., Lin, Y., Cao. Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Kang, B. J., Choi, D. H., and Kim, H. Mixed precision quan- tization with serdware-friendde activation functions for hybrid vit models. In 2024 International Conference on Electronics, Information, and Communication (ICEIC), pp. 1-2. IEEE, 2024.
Liu, Z.., Wang, Y., Chu, X., Dong, N., Qi, S., and Ling, H. A simple and generic framework for feature distillation via channel-wise transformation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1 1129-1138, 1138, 2023.
Ki, S. and Kim, H. Activation distribution-basse layer- wise quantization for convolutional neural networks. In 2022 International Conference on Electronics, Informa- fion, and Communication (ICEIC), pp. 1-4. IEEE, 2022.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gra- dient descent with warm restarts. arXiv preprint 2016.
Kim, N. J., Lee, J., and Kim, H. Hyq: Hardware- friendly post-training quantization for con-transformer hybrid net- works. Proceedings of the Thirty-Thite International Joint Conference on Artificial Intelligence, pp. 4291 4299, 2024.
Loshchilov, and Hutter, Decoupled weight decay regu- larization. arXiv preprint arXiv:1711.05101. 2017.
Merity, S. The wikitext long term dependency language modeling dataset. Salesforce Metamind, 9, 2016.
Kim, S. and Kim, H. Zero-centered fixed- point quantization with iterative retraining for deep convolutional neural network-basc based object detectors. IEEE Access, 9:20828- 20839, 2021.
Moon, J., Kim, D., Cheon, J., and Ham, B. Instance- group quantization for vision transformers. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Patern Recognition, pp. 16132-16141, 2024.
Klema, V. and Laub, A. The singular value decomposition: Its computation and some applications. JEEE Transac- tions on automatic control, 25(2):164-176) 1980.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Lib- rispeech: an ast corpus based on public domain audio books. In 2015 IEEE international conference on acous- tics, speech and signal processing (ICASSP), pp. 5206- 5210. IEEE, 2015.
Kumar, A. Vision transformer compression with structured pruning and low rank approximation. arXiv preprint arXiv:2203. 13444, 2022.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Lee, S., Ko. J., and Hong, S. Facto-enn: Memory-efficient cnn training with low-rank tensor factorization and lossy tensor compression. In Asian Conference on Machine Learning, pp. 662-677. PMLR, 2024a.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Lee, S. I., Koo, K., Lee, J. H., Lee, G., Jeong, S., O, S., and Kim, H. Vision transformer models for mobile/edge devices: a survey, Multimedia Systems, 30(2):109, 2024b.
Li, Z., Xiao, J., Yang, L., and Gu, Q. Repq-vit: Scale repa- rameterization for post-training quantization of vision transformers. Proceedings of the IEE//CV Interna- tional Conference on Computer Vision, pp. 17227-17236. 2023.
Strudel, R., Garcia, R., Laptev, I., and Schmid, C. Seg- menter: Transformer for semantic segmentation. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 7262-7272, 2021.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D.,, Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transform- ers & distillation through attention. In International con- ference on machine learning, pp. 10347-10357. PMLR, 2021.
Lin, Y., Zhang, T., Sun, P., Li, Z., and Zhou, S. Fqt Post-training quantization for fully quantized vision trans- former. arXiv preprint arXiv:21 1.1.1.. 13824, 2021.

---

IRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, t.., and Polosukhin, I. Atten- tion is all you need. Advances in neural information processing systems, 30, 2017.
Zhang, M., Liu, F., and Weng, D. Speeding-up and com- pression convolutional neural networks by low-rank de- composition without fine-tuning. Journal of Real-Time Image Processing. 20(4):64, 2023.
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. Pyramid vision transformer: A versatile backbone for dense prediction without convo- lutions. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 568-576 2021.
Zhong, Y., Hu, J., Lin, M., Chen, M., and Ji, R. 1&s- vit: An inclusive & stable method for pushing the limit of post-training vits quantization. arXiv preprint arXiv:231..00126 2023.
Wei, X., Gong, R., Li, Y., Liu, X., and Yu, F. Qdrop: Ran- domly dropping quantization for extremely low-bit post- training quantization. arXiv preprint arXiv:2203.05740, 2022.
Wightman, Pytorch image models. ttttps://githin com/ wtt.ggghtaaanptttrtttte image 2019.
Wu, Z., Chen, J., Zhong, H., Huang, D., and Wang, Y. Ada- log: Post-training quantization for vision transformers with adaptive logarithm quantizer. In European Confer- ence on Computer Vision, pp. 411-427, Springer, 2025.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR. 2023.
Xu, Y. Zhang, J., Zhang, Q., and Tao, D. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in neural information processing systems, 35: 3857 1-8584, 2022.
Yin, H., Vahdat, A., Alvarez, J. M., Mallya, A., Kautz, J., and Molchanov, P. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 108099 2022.
Yu, and Wu, Compressing transformers: features are low-rank, but weights are not! In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 11007-11015, 2023.
Yuan, 7., Xue, C., Chen, Y., Wu, Q., and Sun, G. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization. In European conference on computer vision, pp. 191-207. Springer, 2022.
Zagoruyko, S. and Komodakis, N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv: /0l2.06/20922 2016.
12

---

IRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
A. Addtionna Analysis of the Proposed Method
A.1. Theoretical Analysis of RB-LRA and WR
We conduct a theoretical analysis the proposed RR-LRA and WR methods. Specifically, we begin by reconstructing the
LRA matrix and subsequently evaluate the reconstruction error as follows:
min(m,n)
min (m.
W, = Σ ouuuo
IW - W
Σ
T
Σ
0
(15)
Here, W, _ represents the reconstructed LRA matrix. Eq. (15) follows from the Youart-oung-Mar Theo- rem (Eckart & Young, 1936), and the formulation involving the error matrix E is expressed as follows:
min(m.n) Σ
IW (W, + E)
E
(16)
That is, the optimal error matrix E satisfying the following condition can be obtained in closed form:
min (W, E
(17)
However, the error matrix directly derived from Eq. (17) has a parameter size of O(mm), and incorporating into the model increases the total parameter size to ((2m)), thereby undermining the primary objective of applying LRA. Accordingly, we propose a reparameterizable error matrix É, formulated using low-rank residual-branch structure
preserve the efficiency of LRA as follows:
b= -v +iU + +0O)
(18)
When the proposed RB-LRA and WR are applied, the error matrix E can be formulated as follows:
É=
Σ
with / < min(m, n) - a
(19)
1=
Consequently, the reconstruction error resulting from the application the proposed WR method is formulated as follows:
WwR W, +
RJ
aunno T
(20)
1=1
Consequently, Eq. (16) leads to the following expression:
min(m,n)
IW (W, + E)1 < I - W
Σ
of
(21)
As demonstrated in Eq. (21), the proposed RB-LRA and WR methods achieve more effective reduction in reconstruction error compared to the conventional SVD-based LRA. This implies that the proposed approach minimizes information loss, thereby playing a critical role in accelerating convergence and achieving optimal accuracy during the fine-tuning process.
A.2. Analysis of the Impact of Branch Initialization Method on Activation
We analyze the impact of the initialization method for the V branch on the output activations of the FC layer. Specifically, we apply both the WR initialization and random initialization methods to the V branch, and visualize the output activations of the FC layer to analyze their impact on the activation patterns. Additionally, we visualize the output activations of the FC layer with the SVD-based LRA method applied, in order to compare its effects with those of the WR initialization method. Figure illustrates the FC layer activations based on the initialization method of the V matrix. We employed the DeiT-B
13

---

LRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
Full-Rank Output
LRA Output
2.0
1.0
0.0
WR Init. Output
Random Init. Output
-1.0
-2.
Figure 4. Visualization of the FC layer activations in the DeiT-B model with RB-LRA applied. The x-axis represents the dimension index, while the y-axis corresponds the token index.
model with RB-LRA applied and conducted comparisons without fine-tuning. Furthermore, improve the visibility of the visualizations, the data was normalized within the range of 2- 2]. First, by comparing the activation visualization results of the WR method and the random initialization method, we observe that the WR method generates output similar to that of the full-rank model. In contrast, the use of the random initialization method leads to noise, causing the activation characteristics of the full-rank model to be lost. Ultimately, the presence of noise-like activations in the FC layer suggests that they hinder the attainment of optimal accuracy during the fine-tuning process. Additionally, when compared to the SVD-based LRA method, the WR method is found to better preserve the activation characteristics of the full-rank model. This can be attributed to the effectiveness of the WR method in reducing weight information loss. Consequently, the use of the WR initialization method promotes better retention of the full-rank model characteristics, ultimately contributing the attainment of optimal accuracy.
A.3. Impact of WADS on Activation Quantization Error
To assess the effect of WADS reducing activation quantization error, we perform visualization and analysis of activation quantization errors in both the DeiT and Swin transformer models. Figure 5 shows the activation quantization error for the top 20 layers exhibiting the highest activation quantization errors in the DeiT and Swin Transformer models, as visualized after normalization. As shown in Figure 5, the application of WADS leads to reduction in activation quantization error across all layers. These visualizations and analyses collectively demonstrate that the WADS method effectively mitigates outliers and minimizes quantization error.
DeñT-T T
DeiT-B
Swin-T
Swin
1.0
1.0
1.0
0.8
0.8
0.5
0.8
1666
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0,2
0.2
0.2
0.2
1.5
20
20
Layer Index
Layer Index
Layer Index
Layer Index
Figure 5. Analysis of Activation Quantization Error Changes with WADS Application. The x-axis represents the layer number, while the a-axis represents quantization error. The visualization compares the error for the top-20 layers with the highest quantization error.
14

---

IRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
A.4. Analysis of the performance impact of each proposed quantization method
To analyze the impact of our proposed LRA-aware quantization methods (i.e., WADS and per-token quantization), we performed step-by-step classification performance evaluation the ImageNet dataset. As shown in Table 9, applying per-token quantization to the DeiT-B and Swin-B models improved accuracy by 0.52% and 0.51%, respectively, compared to the NaivePTQ method. These improvements are attributed to the quantization process, which effectively addresses the impact of outliers on specific tokens after applying RB-LRA. Additionally, the WADS resulted in accuracy improvements of 0.42% and 0.32%, respectively. This improvement is due to WADS effectively mitigating outliers in activation -speciff channels while minimizing weight quantization error. These results demonstrate that both the per-token quantization method and the WADS are effective when combined with LRA and RB-LRA, significantly improving classification performance.
Model
Method
Per-Token
WADS
ACC.(%)
NaivePTQ
X
79.62
DeîT-B
LRA-aware
X
80.14
LRA-aware
V
80.56
NaivePTQ
X
X
82.14
Swin-B LRA-aware
X
82.65
LRA-aware
82.97
Table 9. Performance comparison of different quantization methods at each step for the RB-LRA models.
A.5. Comparative Analysis with Existing SOTA Quantization Methods
Accuracy Comparison with Various SOTA PTQ Methods: We evaluate the performance of the proposed compression platform, which combines RB-LRA and WADS, in comparison to existing SOTA quantization methods. Table 10 presents a comparative analysis of the performance of various SOTA quantization methods and the proposed platform, using the FP32 DeiT-B model without LRA and quantization the baseline. Consequently, the proposed method, which combines RB-LRA and WADS, exhibits the best performance in balancing model size and accuracy. For instance, the existing state-of-the-art method, IGQ-VIT, successfully reduced the size of the DeiT-B model to 43.3MB with 4-bit quantization, but incurred a relatively significant accuracy drop of -2.62%. In contrast, our method compressed the model to a comparable size of 44.4MB, while incurring only a minimal accuracy decrease of -1.29%. This demonstrates that the integration of LRA and quantization is highly effective in achieving model compression while maintaining accuracy. Moreover, it underscores that the proposed compression platform provides practical and efficient alternative to existing methods.
Method
Prec.
Size(MB)
ACC.(%)
(%f
BaseLine
FP32
346.4
81,85
FQT (Lin et al., 2021)
64.39
-17.46
APQ-ViT (Ding al., 2022)
67.48
-14.37
RepQ-Vil ViT (Li et al., 2023)
75.61
-6.24
INT4
43.3
AdaLog (Wu et al., 2025)
78.03
-3.82
ADFQ- ViT (Jiang et al., 2024)
78.75
-3.10
IGQ-ViT (Moon et al., 2024)
79.23
-2.62
RB-LRA
FP32 INT8
177.6 44.4
81.12
-0.73
RB-LRA+WADS
80.56
-1.29
Table 10, Comparison of Top-1 Accuracy (%) on the ImageNet datasct across various existing SOTA quantization methods and the proposed method, evaluated on the DeiT-B.
Comparison of Training Cost, Accuracy, and Lateney Trade-ofl's Between SOTA PTQ Methods and the Proposed Framework: We further compare the trade-offs between training cost, accuracy, and latency of the proposed framework against existing SOTA PTQ methods. Table 11 presents the performance of our method, FQ-ViT (Lin et al., 2021), RepQ- ViT (Li et al., 2023), and AdaLog (Wu et al., 2025) on the DeiT-B model. Latency is measured on the NVIDIA Jetson Xavier platform, while GPU time denotes the optimization time evaluated on a single A100 GPU. In terms of accuracy, the proposed method achieves the highest Top-1 accuracy of 80.56% Although it incurs a training time of hours and 56 minutes, this cost justified by the substantial accuracy improvement over existing PTQ methods. Next, terms of
15

---

LRA-QVIT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
latency, existing methods fail achieve meaningful acceleration despite applying 4-bit quantization. This is primarily because support for true 4-bit integer operations remains limited across current edge devices. In most commercial edge hardware, even when 4-bit quantization is applied at the software level, the actual computations are executed using 8-bit integer operations, Consequently, it is difficult to realize the theoretical speedup benefits of 4-bit quantization in practical deployment scenarios. In contrast, the proposed method integrates 8-bit quantization with LRA, effectively reducing both computational complexity and model size. As a result, it achieves the shortest inference latency of 59.4ms. This demonstrates that combining widely supported 8-bit integer operations with structural compression methods such as LRA can yield significant optimization benefits. These results suggest that the proposed method offers excellent practicality and computational efficiency, particularly in realistic edge deployment scenarios.
Method
Prec.
Latency (ms)
Model Size (MB)
GPU Time
Acc.(%)
FQ-ViT (Lin et al., 2021)
110.9
77 s
64.39
RepQ-Vii VIT (Li et al., 2023)
INT4
96.8
43.3
247 s
75.61
AdaLog (Wu et al., 2025)
113.8
2h 47m
78.03
Ours
INT8
59.4
44.4
5h 56m
80.56
Table 11. Comparison of training cost, latency, and accuracy between SOTA PTQ methods and the proposed framework on the DefT-B.
A.6. Differences with Low-Rank Adapter-Bases Parameter Efficient Fine-Tuning
Recent research has focused extensively on adapter-based based methods for parameter-efficient fine-tuning (PEFT) of large Transformer models. Notably, LoRA (Hu et al., 2021) utilizes low-rank adapters update only a limited subset of parameters, thereby significantly reducing GPU memory consumption while preserving the accuracy of full fine-tuning. QLoAA (Dettmers et al., 2024) combines LoRA with the quantization of foundation models, effectively reducing GPU memory consumption during the fine-tuning process. However, while these methods focus reducing GPU memory usage during the fine-tuning process, they do not address the reduction of parameters in the foundation model during inference. In contrast, our approach applies LRA to the foundation model for efficient inference on mobile and edge devices, concurrently utilizing quantization to effectively reduce both the number of parameters and memory usage.
16


Reviews:
[{"review_id": "4C9v5sLQes", "reviewer": "Reviewer_BTg6", "summary": "The authors propose RB-LRA method which introduces a reparameterizable residual branch to compensate for information loss due to LRA. Weight reconstruction (WR) initializes the residual branch with weights discarded during decomposition, mitigating accuracy loss. To further improve accuracy, the method incorporates feature-based KD. Applying LRA creates large activation outliers, leading to increased quantization errors. So, they propose WADS which identifies layers where weight quantization errors are significant and applies per-token quantization and channel scaling to minimize accuracy loss during quantization. With these, they can achieve significant compression and latency improvements on the Swin-B and DeiT-B models with minimum loss in accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their valuable feedback. Refreshing the page (F5) helps generate equations properly!  \n\n# A1) Evaluation of Large-Scale Models\n>- As suggested by the reviewer, we evaluate our framework on ViT-L for large-scale models.\n>- Experimental results show that applying the RB-LRA method improves accuracy by 0.67%.\n>- Furthermore, RB-LRA with WADS achieves 0.18% higher accuracy compared to the baseline while compressing the model by 86.3%, as shown in Table A.\n\n**Table A**\n> |Model|Model Size (MB)|Method|Acc.(%)| \n|-|:-:|-|-|\n|ViT-L|1217.2|Baseline|85.84| \n||669.2|RB-LRA|86.51|\n||167.3|RB-LRA + WADS|86.02| \n# A2) Theoretical Analysis \n>- We provide a brief theoretical analysis of RB-LRA and WR in response to the reviewer's comment. \n>- We first define the error based on the Frobenius norm as follows:\n$$W_r = \\sum_{i=1}^{r} \\sigma_i u_i v_i^{T}, \\quad\n\\lVert W - W_r \\rVert_F^2 =\\left\\lVert \\sum_{i=r+1}^{\\min(m,n)} \\sigma_i u_i v_i^{T} \\right\\rVert_F^2 =\n\\sum_{i=r+1}^{\\min(m,n)} \\sigma_i^2 $$\n>- The error in our RB-LRA method is expressed as follows:\n$$\\lVert W - (W_r + E) \\rVert_F^2 =\\left\\lVert \\sum_{i=r+1}^{\\min(m,n)} \\sigma_i u_i v_i^{T} - E \\right\\rVert_F^2$$ \n>- Thus, the optimization objective can be formulated as: \n$$\\min_{E} \\lVert W - (W_r + E) \\rVert_F^2$$\n>- While $E$ can be directly computed, its shape matches the original weight, increasing the size from $\\mathcal{O}(MN)$ to $\\mathcal{O}(2MN)$ and defeating the purpose of low-rank approximation\n>-  We thus design a reparameterizable error matrix using a residual branch:\n$$\\tilde{E} = \\left( V \\tilde{U}^{T} + \\tilde{V} U^{'T} + \\tilde{V} \\tilde{U}^{T} \\right)$$\n>- The error matrix guides fine-tuning (FT) toward optimal accuracy. \n>- In the setting of Eq. (5), the initial $\\tilde{E}$ matrix with WR applied is expressed as follows:\n$$\\tilde{E} = {V}U^{'T} \\approx \\sum_{i = r+1}^{r+l} \\sigma_i u_i v_i^{T}, \\quad \\text{with } l < \\min(m,n) - r\n$$\n>- Finally, the resulting error incorporating this component is expressed as follows:\n$$W_{\\text{WR}} = W_r + \\tilde{E} \\approx \\sum_{i = 1}^{r+l} \\sigma_i u_i v_i^{T}$$\n>- Consequently, the information loss based on the Frobenius norm for the conventional LRA and the proposed RB-LRA with WR can be compared as follows:\n$$\\lVert W - (W_r + \\tilde{E}) \\rVert_F^2 < \\lVert W - W_r \\rVert_F^2 = \\sum_{i = r+1}^{\\min(m,n)} \\sigma_i^2$$\n>- Ultimately, the proposed initialization method can be seen as better preserving the information in the discarded parameter dimensions.\n>- We further provide a theoretical analysis of how the proposed method achieves optimal performance through FT.\n>- Discarded singular vectors may retain task-relevant information, leading the gradient to exhibit positive projections onto them during FT.\n>- Based on this observation, it is reasonable to assume a positive correlation between the discarded singular vectors and the gradient.\n>- Accordingly, when $\\tilde{V}$ is WR-initialized, it satisfies:\n>$$\n\\mathbb{E}\\left[\\left\\langle \\tilde{V}, \\nabla_W \\mathcal{L} \\right\\rangle_F\\right]\n= \\sum \\sigma \\cdot \\mathbb{E}\\left[\\left\\langle \\nabla_W \\mathcal{L}, v \\right\\rangle\\right] > 0\n$$\n>- Whereas random initialization $\\tilde{V} \\sim \\mathcal{N}(0, 1)$  yields:\n$$\n\\mathbb{E} \\left[\\left\\langle \\tilde{V}, \\nabla_W \\mathcal{L} \\right\\rangle_F\\right] = \\sum \\mathbb{E}\\left[\\tilde{V}\\right]\\left(\\nabla_W \\mathcal{L}\\right) = 0\n$$\n>- The WR method can be seen as being aligned with the gradient, suggesting that it facilitates faster convergence toward the optimal point in the loss landscape.\n>- The effectiveness of our method is demonstrated in Table 5.\n# A3) Comparision with PTQ4ViT\n>- We compare our method with PTQ4ViT as requested.\n>- As shown in Table B, at a similar compression ratio, our method yields a smaller accuracy drop on DeiT-B, indicating a better trade-off.\n\n**Table B**\n>|Model|Method|Prec.|Model Size (MB)|Acc.(%)|\n|-|-|-|:-:|-|\n|DeiT-B|PTQ4ViT|INT4|43.3|64.30|\n||Ours|INT8|44.4|**80.56**|\n# A4) Applicability to Diverse Tasks\n>- Although we attempted to evaluate our method on DiT-XL/2, its architecture demands extensive resources and time, making it infeasible within the rebuttal period.\n> Specifically, one epoch takes 40 hours on a single A100 GPU.\n>- As an alternative, we demonstrated the extensibility of our method by applying it to GPT-2. \n> The results can be found in Table E in our response to reviewer vjxZ\n>- We also evaluate performance on a pose estimation task to demonstrate applicability across diverse tasks\n>- Specifically, we apply RB-LRA to the ViTPose [1] model and evaluate its performance on the COCO dataset.\n>- As shown in Table C, we achieve a compression ratio of 25% with less than 1% AP drop, demonstrating the versatility of our method.\n\n**Table C**\n>|Model|Method|Params|AP|AR|\n|-|-|:-:|-|-|\n|ViTPose|Baseline|89.9|75.9|81.0|\n||RB-LRA|66.8|75.0|80.4|\n# Reference\n>- [1] Vitpose: Simple vision transformer baselines for human pose estimation. NeurlPS'22", "claims_and_evidence": "The paper claims that \n1. RB-LRA effectively reduces parameters with minimal accuracy drop\n2. Knowledge distillation enhances RB-LRA accuracy\n3. WADS effectively reduces Quantization induced accuracy loss\n4. Their method achieves significant inference speedups\nThese claims are supported by experimental evidence\n\nThe paper claims to provide a compression solution that applies to ViT models in general, but only the Swin and DeiT models are tested at tiny base sizes. Further evidence on other models such as the ViT model and larger sized models is yet to be seen.", "methods_and_evaluation_criteria": "They test their method across various baselines on the ImageNet classification task and the Detection and Segmentation tasks on the MSCOCO dataset. Inference speeds are tested on Android and on the NVIDIA Jetson Xavier platforms to measure improvements in the real use cases.", "theoretical_claims": "They claim that the SVD-based initialization of the parameterizable branch reduces information loss compared to LRA. However, no theoretical analysis of this procedure is given.", "experimental_designs_or_analyses": "They cover a wide variety of experiments validating their claims across classification, detection, and segmentation tasks. Their ablation studies in the main paper and appendix shine light on the importance of various design choices they make throughout the paper.", "relation_to_broader_scientific_literature": "This paper builds on prior research in low-rank approximation (LRA), knowledge distillation (KD), and quantization for Vision Transformers (ViTs) by introducing RB-LRA, a parameterizable branch that mitigates information loss from LRA, and WADS, an LRA-aware quantization method that reduces accuracy degradation from outliers. Unlike existing SVD-based LRA methods (Kumar, 2022; Zhang et al., 2023), which suffer from accuracy drops, RB-LRA reconstructs lost weight information through a residual branch. The paper also improves upon prior KD-based LRA methods (Guo et al., 2024) by introducing block-wise KD and optimizing fine-tuning at the transformer block level. Compared to SmoothQuant (Xiao et al., 2023) and RepQ-ViT (Li et al., 2023), WADS integrates per-token quantization and weight-aware scaling, significantly improving robustness for compressed models. Additionally, the paper demonstrates practical improvements in inference speed (1.9×–3.2× on mobile, 1.5×–2.5× on edge devices), making it a comprehensive and efficient compression framework for real-world deployment.", "essential_references_not_discussed": "PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization", "other_strengths_and_weaknesses": "Other strengths:\n1. The ideas in the paper are clearly presented making it easier for the reader to understand the concepts \n2. The paper draws from many methods that boost efficiency in transformers such as LRA, quantization, activation scaling etc.,\n3. They provide performance numbers measured on actual devices making their method ready for immediate deployment.\n4. They cover a wide variety of baseline methods and ablation studies\n\nOther weaknesses\n1. The paper can improve with a theoretical analysis of the design choices they make\n2. Testing their algorithm on larger models such as ViT-H, ViT-L etc.,\n3. Does their method also work on transformers for Generative modeling such as DiT or is it only restricted to discriminative models?", "other_comments_or_suggestions": "See strengths and weaknesses above", "questions_for_authors": "See strengths and weaknesses above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Po0FiBRfpg", "reviewer": "Reviewer_vjxZ", "summary": "This paper proposes RB-LRA, a low-rank approximation scheme integrated with quantization to reduce the number of parameters in vision transformers (ViTs) and mitigate inference delay. To minimize approximation errors introduced by singular value decomposition (SVD), RB-LRA employs block-level knowledge distillation (KD) to fine-tune the approximated matrices. Additionally, to integrate quantization with minimal performance degradation, RB-LRA leverages weight-aware distribution scaling (WADS) to suppress weight outliers. Experimental results on ImageNet with pretrained ViTs demonstrate that RB-LRA effectively reduces the number of parameters while maintaining high accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their valuable feedback and provide the following responses.\n# A1) Additional Computation Analysis\n>- RB-LRA and WADS require one-time fine-tuning (FT) and calibration only during pre-deployment, without inference overhead.\n>- As shown in Table D, we measured the FT time of RB-LRA and compared it with the baseline training time.  \nThe optimization time for WADS is provided in Table I in our response to reviewer MFof.\n>- FT time and baseline training time were measured on a single A100 GPU (batch size 64, 300 epochs for baseline)\n>- RB-LRA converges within 100 epochs, requiring only 15% ~ 21% of baseline training time.\n>- Moreover, all FT and calibration were conducted on public datasets in server-like settings, aligned with common industrial workflows\n>- Overall, our method adds only minor training overhead while achieving both high accuracy and clear gains in post-deployment efficiency, which is central to our contribution.\n>- Moreover, we analyze the trade-off between FT overhead, inference speed, and accuracy improvement.\n>- The results can be found in Table F in our response to reviewer xvP8. We encourage the reviewer to refer to it\n\n**Table D**\n> |Model|Method|GPU Time|GPU Memory (GB)|\n> |-|-|-|-|\n> |DeiT-T|Baseline|20h|5.9|\n> ||RB-LRA|4h 21m|7.2|\n> |Swin-T|Baseline|35h|16.0|\n> || RB-LRA|5h 19m|19.2|\n\n# A2) Clarifying Novelty: Beyond Simple Integration\n> Our contribution lies not in simply combining existing methods, but in the distinct novelty of each component and their cohesive integration into a unified framework.\n> 1. **RB-LRA + WR** \n>- SVD-based LRA suffers from accuracy loss due to discarded components.\n>- We introduce a reparameterizable residual branch that guides FT toward optimal accuracy.\n>- During inference, it merges into a unified form, adding no extra parameters or computations.\n>- This design helps preserve accuracy while reducing memory, suitable for edge deployment.\n> 2. **WADS** \n>- While activation-aware quantization methods exist, we are the first to empirically analyze the emergence of channel- and token-level outliers following LRA.\n>- We propose a weight-aware scaling method guided by quantization loss to effectively suppress these outliers.\n> 3. **Originality in Integrated Design**\n>- Prior works typically address LRA or quantization separately.\n>- We instead propose a unified, deployment-oriented framework that systematically combines novel components such as RB-LRA and WADS.\n# A3) Language Application\n>-  Our method specifically targets the compression of linear layers, and we believe it can be broadly applied across various domains that utilize transformer-based models.\n>- In response to the reviewer’s comment, we applied RB-LRA to GPT-2 Medium on WikiText-103 using pre-trained weights from Huggingface.\n>- To further demonstrate generalizability, we applied RB-LRA to the Conformer [1] model for automatic speech recognition (ASR).\n>- For the ASR task, we used a baseline model trained from scratch on the LibriSpeech 960h dataset using our own training pipeline\n>- As shown in Table E, despite compressing model parameters by 25–30%, our method resulted in only marginal accuracy drops across both NLP and ASR tasks These results demonstrate the strong extensibility of RB-LRA to various transformer-based models and tasks.\n>- We'll include these results in the camera-ready version to demonstrate our method's generality and competitiveness.\n\n**Table E**\n> |Model|Dataset|Method|Params|Perplexity (↓)|\n> |-|-|-|:-:|:-:|\n> |GPT2|Wikitext-103|Baseline|354.8|18.72|\n> |||RB-LRA|249.4|19.51|\n\n> |Model|Dataset|Method|Params|Test-Clean WER (↓)|\n> |-|-|-|-|-|\n> |Conformer|Librispeech|Baseline|116.9|5.4|\n> |||RB-LRA|86.2|5.6|\n# A4) Writing Clarity & Notation Explanation\n>- The matrices $\\tilde{U}$ and $\\tilde{V}$ in Eq. (5) refer to trainable low-rank residual branches introduced to compensate for information loss during the standard SVD-based LRA.\n>- We provide a theoretical explanation of this setup in our response to Reviewer BTg6 and kindly refer the reviewer to that section for further clarification.\n>- If granted the opportunity to submit a camera-ready version, we will define these notations at their first occurrence and provide a brief explanation of their role and motivation in the Proposed Method section to enhance clarity.\n# A5) How to select the rank(r)?\n>- We determined the rank (r) per layer based on the target model size.\n>- Specifically, we retained only the top singular values after SVD to meet the desired compression ratio.\n>- Since singular values are defined as the square roots of eigenvalues of a positive semi-definite matrix, they are always positive and typically sorted by magnitude for truncation.\n>- The exact rank settings are available in our code, and we will include them in the paper for reproducibility in the camera-ready.\n# Reference\n> [1] Conformer: Convolution-augmented Transformer for Speech Recognition. Interspeech'20", "claims_and_evidence": "1. The proposed RB-LRA significantly reduces LRA-induced errors by fine-tuning the approximated matrices using knowledge distillation. The accuracy drops on ImageNet for various ViT models (DeiT-T, DeiT-B, Swin-T, Swin-B) are -0.47%, -0.73%, -0.88%, and -0.03%, respectively.\n\n2. The proposed RB-LRA effectively reduces quantization error through weight-aware distribution scaling (WADS), as validated on ImageNet across several ViT architectures.\n\n3. The proposed RB-LRA significantly reduces inference latency by utilizing quantized approximated parameters. Experiments on Android and Xavier platforms demonstrate the effectiveness of RB-LRA.", "methods_and_evaluation_criteria": "1. The primary evaluation criteria include image classification accuracy on ImageNet and average precision (AP) on the COCO dataset for object detection. Model size is also considered, but it remains the same across all schemes using low-rank approximation.\n\n2. The latency time criteria can validate the effectness of RB-LRA to accelerate inference.", "theoretical_claims": "There is no theoretical contribution in this paper.", "experimental_designs_or_analyses": "1. The experiments are comprehensive across several model architectures, bitwidth and vision tasks (classification, object detection). However, there is no analyses on the additional computation for fine-tuning the approximated matrices and optimizing the scaling factor in Eq. 14. It is more fair to compare both computation overhead and the accuracy improvement.", "relation_to_broader_scientific_literature": "The contribution is relatively incremental, as this paper primarily integrates widely-used techniques such as knowledge distillation and activation-aware quantization. The novelty of the work is quite limited.", "essential_references_not_discussed": "This paper has cited most of the relevant literature.", "other_strengths_and_weaknesses": "1. The novelty is limited, as no new techniques are proposed in this work.\n\n2. The proposed RB-LRA needs extra computation for fine-tuning with data, which restricts the application of this technique.", "other_comments_or_suggestions": "1. Low-rank approximation can also be applied to language models. Evaluating the effectiveness of RB-LRA on LLMs would be valuable, as numerous text-based benchmark tasks are available.\n\n2. Please indicate every notation after using it in the formula. For example, the $\\tilde{V}$ and $\\tilde{U}$  in Eq.5. You should explain what is the meaning of these two notations and why you set up in this way.", "questions_for_authors": "How do you select the rank $r$? Are you selecting all the positive singular values?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cgL3EQfz5c", "reviewer": "Reviewer_xvP8", "summary": "This paper introduces a novel framework that integrates reparameterizable branch-based Low-Rank Approximation (RB-LRA) with Knowledge Distillation (KD) to reduce the number of parameters and inference computational complexity. Additionally, the authors propose an LRA-aware post-training quantization method to enhance the performance of the model after quantization. The experimental results on the image classification task demonstrate that the proposed method achieves promising performance in both full-precision and 8-bit quantized models.\n\n## update after rebuttal\n\nThank you for the authors' rebuttal.\nThe additional experimental results have demonstrated the promising performance of the proposed method.\n\nHowever, regarding points A1) and A2), my intention was to highlight that, as shown in Table 7, the authors' method applies 8-bit quantization to the model after RB-LRA compression, while the comparison methods quantize the ViT to 4 bits. There is no direct comparison or analysis of the computational complexity between 8-bit RB-LRA and 4-bit ViT in this context. If the authors' currently available hardware does not support 4-bit quantization, a theoretical analysis of computational costs could still be conducted to evaluate the potential practical efficiency of this technique. This is particularly relevant given that 4-bit quantization is increasingly feasible for deployment on edge devices in many resource-constrained scenarios.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their valuable feedback. Refreshing the page (F5) helps generate equations properly!  \n\n# A1) Computational Complexity Analysis\n>- We analyze the computational complexity (i.e., FLOPs) and inference speed improvements.\n>- Furthermore, Table 3 demonstrates the actual acceleration achieved on mobile and edge devices.\n>- Our efficiency gains stem from:\n> ### 1. RB-LRA reduces computational complexity \n>- As $\\text{FLOPs} \\propto \\text{parameter size}$, complexity is reduced by a factor of $\\frac{r(m+n)}{mn}$, which is $\\ll 1$.\n>- Our method effectively reduces  FLOPs by decreasing the number of parameters, as detailed in Table 1.\n>### 2. INT8 operation compatibility enabled by quantization\n>- Specifically, when the applied quantization method is implemented using real-quantization (i.e., actual low-bit integer arithmetic rather than simulated quantization), it can be formulated as follows: \n> $$y_{[t, o]} = \\sum_{t=0}^{T} \\sum_{o=0}^{OC} \\frac{1}{S_W[o] \\cdot S_x[t]} \\left( \\sum_{i=0}^{IC} W_q[i, o]x_q[t, i] \\right)$$\n>- $S_W[o]$ and $S_x[t]$ denote the weight scale factor and the activation scale factor, respectively\n>- With compiler support, linear layers that utilize this feature can be converted into integer operations on edge devices, thereby accelerating inference.\n# A2) Comparison of Inference Speed\n>- We measured the inference speed of the 4-bit PTQ methods on a real edge device (NVIDIA Xavier) and the results can be found in Table F.\n>- In Table F, latency refers to the inference latency on edge devices, while GPU time indicates the optimization time of RB-LRA + WADS and each quantization method in a GPU environment.\n>- As a result, the INT4 PTQ methods did not yield meaningful inference speed-up.\n>- This is because edge devices, such as the Jetson Xavier series, provide hardware acceleration up to INT8 precision. \n>- That is, even with INT4 quantization in software, edge devices perform operations similar to INT8 in practice.\n>- To the best of our knowledge, INT4 is unsupported on edge devices and only partially supported on high-end GPUs (e.g., H100).\n>- Ultimately, the proposed framework enables practical INT8 acceleration on commercial edge devices while achieving INT4-level model compression, offering greater deployment benefits.\n# A3) Training Cost, Accuracy Trade-off\n>- Table F shows trade-offs among FT cost, accuracy, and speed.\n>- The proposed method incurs the highest training cost compared to existing methods.  \nHowever, it demonstrates a corresponding improvement in accuracy that justifies the cost.\n>- Additionally, our method achieved the highest accuracy and inference speed on real-world devices.\n>- In edge/mobile scenarios, training is typically performed on GPU servers, while inference is conducted on resource-constrained devices.  \nTherefore, the trade-off between accuracy, latency, and model size becomes particularly critical in such environments. \n>- Notably, our proposed method requires only about 6 hours, which is significantly lower compared to the full baseline training cost.  \nThese results demonstrate that our method performs effectively in real-world deployment scenarios.\n\n**Table F**\n>|Model|Method|Prec.|Latency (ms)|Model Size (MB)|GPU Time|Acc.(%)|\n|-|-|-|:-:|:-:|:-:|:-:|\n| DeiT-B|FQ-ViT|INT4|110.9|43.3| **77 s** |64.39|\n||RepQ-ViT|INT4|96.8|43.3|247 s|75.61|\n||AdaLog|INT4|113.8|43.3|2h 47m|78.03|\n||Ours|INT8|**59.4**|44.4|5h 56m|**80.56**|\n# A4) Compatibility between RB-LRA and SOTA PTQ Methods\n>- First, Table 2 already presents performance comparisons of the proposed RB-LRA model combined with QADS, SmoothQuant, RepQ-ViT, and WADS.\n>- We evaluate INT8 quantization performance by applying methods listed in Table 7, such as AdaLog, to the RB-LRA model.\n>- Since the code for APQ-ViT, ADFQ-ViT, and IGQ-ViT is not publicly available, we apply AdaLog and FQ-ViT to RB-LRA  instead.\n>- As shown in Table G, WADS shows better compatibility with RB-LRA than existing PTQ methods.\n\n**Table G**\n> |Model|Prec.|Method|Acc.(%)|\n|-|-|-|-|\n|DeiT-B|FP32|Baseline(RB-LRA)|81.12| \n||INT8|AdaLog|79.27|\n|||FQ-ViT|77.70|\n|||WADS|**80.56**|\n# A5) Object Detection / Instance Segmentation Task\n>- We conduct experiments on object detection and instance segmentation, with results shown in Table 4.\n>- While a direct comparison with methods like PELA is difficult due to unavailable code and differing baselines (e.g., Swin-B vs. our Swin-T), we instead use accuracy drop as an indirect metric.\n>- As shown in Table H, our method shows comparable degradation, indicating that RB-LRA effectively maintains performance even with smaller models, making it competitive for mobile and edge deployment.\n\n**Table H**\n> |Method|Backbone|Model|Baseline$AP^{box}$|Baseline$AP^{mask}$|$AP^{box}$|$AP^{mask}$|\n|-|-|-|:-:|:-:|:-:|:-:|\n|AAFM|Swin-B|Cascade Mask R-CNN|52.0|45.0|51.9|44.7|\n|PELA|Swin-B|Cascade Mask R-CNN|50.1|-|49.0|-|\n|RB-LRA|Swin-T|Mask R-CNN|42.7|39.3|42.5|39.0|", "claims_and_evidence": "Some claims are not well-supported in the current version:\n\n1. Table 7 demonstrates that the proposed method, when integrated with knowledge distillation (KD) and low-rank adaptation (LRA)-aware INT8 quantization, achieves model sizes comparable to those of INT4 quantized ViTs. However, the paper lacks an analysis of the actual computational complexity and inference speed of these models. Specifically, it would be beneficial to evaluate the real-world inference speeds of the proposed method compared to the baseline methods in Table 7 under identical hardware and model size conditions. This information is crucial for assessing the practical applicability of the proposed framework, as retraining such a model incurs significant computational costs. A well-balanced trade-off analysis between training overhead and inference efficiency would strengthen the paper’s practical contributions.\n2. While the authors highlight the strong performance of their method when combined with INT8 quantization—surpassing some INT4 post-training quantization (PTQ) approaches, as shown in Table 7—there is no direct comparison between their proposed quantization method and these state-of-the-art PTQ methods. For instance, applying AdaLog [2] to the proposed full-precision model and then comparing its performance under INT8 quantization in Table 2 would provide a more comprehensive understanding of the relative strengths and limitations of the proposed approach.\n3. The evaluation of the proposed method is primarily focused on image classification. A broader assessment across other tasks, such as semantic segmentation and object detection, would further substantiate its effectiveness. For example, baseline LRA methods like PELA [1] have demonstrated competitive results in these tasks, and comparing the proposed framework against them in such scenarios would provide valuable insights into its generalizability.\n\nReferences\n[1] Guo et al. PELA: Learning Parameter-Efficient Models with Low-Rank Approximation . In CVPR 2024.\n[2] Wu et al. AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer . In ECCV 2024.", "methods_and_evaluation_criteria": "Some of them are not very satisfactory in current version:\n\n1. While the authors emphasize the robust performance of their method when integrated with INT8 quantization—outperforming certain INT4 post-training quantization (PTQ) techniques, as demonstrated in Table 7—there is a notable absence of direct comparisons between their proposed quantization method and these state-of-the-art PTQ approaches.\n\n2.The evaluation of the proposed method primarily centers on image classification. Expanding the assessment to include additional tasks, such as semantic segmentation and object detection, would further validate its effectiveness. Notably, baseline LRA methods like PELA [1] have achieved competitive performance in these domains. A comparative analysis of the proposed framework against such methods in these contexts would offer valuable insights into its generalizability.\n\n[1] Guo et al. PELA: Learning Parameter-Efficient Models with Low-Rank Approximation . In CVPR 2024.", "theoretical_claims": "The paper does not include any proofs or theoretical claims.", "experimental_designs_or_analyses": "Some of them are not very conviencing. Please see my comments in \"Methods And Evaluation Criteria\"", "relation_to_broader_scientific_literature": "The paper extends prior work on low-rank approximation (LRA), knowledge distillation, and quantization in vision transformers (ViTs) by addressing key limitations in these areas. While existing LRA methods (e.g., PELA) reduce parameters through matrix decomposition, they often suffer from accuracy loss. The proposed reparameterizable branch-based LRA (RB-LRA) mitigates this by introducing weight reconstruction. Additionally, knowledge distillation, widely used in ViT compression (e.g., DeiT), is integrated to further enhance accuracy. Unlike prior LRA methods that overlook quantization, the paper introduces an LRA-aware quantization strategy to handle large outliers caused by decomposition. Empirical results on ImageNet demonstrate superior efficiency-accuracy trade-offs, making the framework highly relevant for real-world deployment.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The proposed method shows promising performance on image classification tasks while significantly reducing inference complexity, which is a valuable contribution to ViT compression.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gdWdzGWcyd", "reviewer": "Reviewer_MFof", "summary": "This paper presents LRA-QViT, a novel framework integrating low-rank approximation (LRA) and quantization to improve the efficiency and robustness of Vision Transformers (ViTs), particularly for deployment in resource-constrained environments such as edge and mobile devices. The authors introduce Reparameterizable Branch-based Low-Rank Approximation (RB-LRA), which mitigates information loss from LRA via weight reconstruction. Additionally, they propose LRA-aware quantization, which addresses outliers induced by LRA using Weight-Aware Distribution Scaling (WADS) and per-token quantization. The method is validated through extensive experiments on ImageNet, demonstrating superior efficiency-accuracy trade-offs compared to existing compression and quantization baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their valuable feedback and provide the following responses.\n# A1) Difference from QAT\n>- Our proposed WADS includes an optimization process distinct from existing PTQ methods, yet it remains fundamentally different from QAT.\n>- As shown in the right part of Figure 1, WADS first measures the layer-wise weight quantization sensitivity of the RB-LRA-applied model (Eq. 13).\n> - Then, layers with sensitivity below a certain threshold are selected, and an optimization is performed to find a scaling vector $\\alpha$ that mitigates activation outliers (Eq. 14).\n>- Notably, the distinction between WADS and conventional QAT lies in the following aspect:  \n\n ---\n>**1. The method uses only 32 calibration images rather than the full training dataset, consistent with the standard static quantization flow adopted in PTQ methods.**  \n>**2. WADS does not require any quantization-aware weight updates over the entire model; it only involves computing MSE-based quantization error and optimizing the scaling vector $\\alpha$.**  \n>**3. As the only optimization target is $\\alpha$ the cost is substantially lower than QAT, which involves full model retraining.**\n---\n>- We measured the WADS application time on an A100 GPU.  \n>- Table I presents the optimization time and additional GPU memory required to optimize the scaling vector when applying WADS.\n>- As a result, the proposed WADS is clearly distinguished from QAT, which requires full training. We kindly refer the reader to Table D in our response to reviewer vjxZ for details on the full-training burden.\n>- Furthermore, the additional GPU memory overhead remains within 250 MB.\n>- As the WADS optimization is performed on high-capacity GPU hardware (e.g., 80GB A100), we believe that the additional 250MB memory usage does not present a significant bottleneck.\n>- Moreover, when deploying to edge or mobile devices, the scaling vector is fixed, eliminating the optimization burden. Therefore, this overhead does not pose a significant limitation in the context of our research objective.\n>- Accordingly, the comparison between WADS and PTQ methods presented in this paper is justified, and WADS is reaffirmed as a practical and effective alternative within PTQ settings.\n\n**Table I**\n>|Model|Method|GPU Time (s)|Additional GPU Memory (MB)|\n|-|:-:|:-:|:-:|\n|DeiT-B|WADS|160.6|192.13|\n|DeiT-T||121.7|27.24|\n|Swin-B||392.6|258.33|\n|Swin-T||131.0|88.73|\n\n# **A2) Comparison with QAT**\n>- Additionally, we analyze the trade-off between model size and accuracy of QAT methods and our proposed method.\n>-  As shown in Table J, our method effectively improves the trade-off between model size and accuracy.\n>- Moreover, from the perspective of training cost–accuracy trade-off, our method requires significantly less computational burden compared to full training, further demonstrating its superiority.\n\n**Table J**\n> |Model| Method | Prec. | Model Size (MB) | Acc.(%) |\n> |-|-|-|:-:|:-:|\n> |DeiT-B|Quantformer [1]|INT4|43.3|79.70|\n> ||I&S-ViT [2]|INT4|43.3|79.97|\n> ||Ours|INT8|44.4|80.56|\n\n# **A3) Comparison with Other Compression Methods**\n\n>- Additionally, we compare the performance of our framework with various compression methods in Table K to demonstrate its superiority.\n>- We compare performance on the Swin-B model using the INT6 quantization method to ensure a fair comparison at a similar model size.  \nIn addition, since baseline performance varies across methods, we compare them in terms of accuracy drop.\n>- On the Swin-B model, our method demonstrates a superior trade-off between model size and accuracy drop compared to all other INT6 PTQ methods.\n>- Although I&S-ViT exhibits a slightly smaller accuracy drop, our method achieves a more balanced and efficient trade-off when considering both model size and accuracy.\n>- Moreover, unlike conventional quantization methods, the RB-LRA shows broad applicability not only to vision tasks but also to various modality tasks, as noted in the responses from reviewers BTg6 and vjxZ. Therefore, we believe it is practically advantageous.\n\n**Table K**\n>|Model|Method|LRA|Quant|Train|Prec.|Model Size(MB)|Baseline Acc.(%)|Acc.(%)|Acc. Drop(%)|\n|-|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Swin-B|PTQ4ViT||$\\checkmark$||INT6|66.1|85.27|84.18| -1.09 |\n||APQ-ViT||$\\checkmark$||INT6||| 84.01 | -1.26 |\n||QDrop [3]||$\\checkmark$|$\\checkmark$|INT6|||84.33|-0.94|\n||I&S-ViT||$\\checkmark$|$\\checkmark$|INT6|||84.94|-0.33|\n||PELA|$\\checkmark$||$\\checkmark$| FP32|248.8|83.47|82.50|-0.97|\n||AAFM|$\\checkmark$||$\\checkmark$|FP32|240.8||82.68|-0.79|\n||Ours|$\\checkmark$|$\\checkmark$|$\\checkmark$|INT8|60.1||82.97|-0.5|\n\n# **Reference**\n>[1] Quantformer: Learning extremely low-precision vision transformers. TPAMI'22  \n[2] I&S-vit: An inclusive & stable method for pushing the limit of post-training vits quantization. arXiv’23  \n[3] QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization. ICLR'22", "claims_and_evidence": "The claims in the paper are well-supported by experimental results:\n- RB-LRA and RB-LRA + KD improve accuracy over naive LRA by introducing a residual branch and fine-tuning strategy, as shown in Table 1.\n- WADS outperforms naive post-training quantization (PTQ), SmoothQuant, and QADS on RB-LRA fine-tuned models.\n- The proposed method enhances practical deployment: The combination of RB-LRA and WADS achieves 1.9×–3.2× inference speedups on mobile devices and 1.5×–2.5× speedups on edge devices, all while maintaining accuracy.", "methods_and_evaluation_criteria": "The methods proposed are well-justified for the problem of efficient ViT fine-tuning and compression. The evaluation is rigorous and includes:\n- ImageNet classification results across DeiT and Swin Transformer models.\n- Comparison with prior LRA methods.\n- Comparison with quantization methods.\n- Ablation studies on object detection and initialization methods.\n- Latency analysis on mobile and edge devices, demonstrating real-world applicability.", "theoretical_claims": "There are no major theoretical claims requiring proof verification.", "experimental_designs_or_analyses": "The experimental design follows a conventional setting and includes:\n- Several reasonable baselines (LoRA for fine-tuning and QADS for post-training quantization).\n- Standard datasets (ImageNet and MS-COCO for downstream tasks).\n\nOne minor concern is whether it is fair to compare WADS with post-training quantization baselines. Based on the description in the paper, WADS appears to resemble quantization-aware training (QAT) rather than pure post-training quantization (PTQ).", "relation_to_broader_scientific_literature": "The paper builds on existing work in LRA and quantization and appropriately cites key references, such as:\n- LRA techniques in ViTs (e.g., SVD-based decomposition, previous PELA methods).\n- Quantization methods (e.g., SmoothQuant, RepQ-ViT, QADS).\n- Knowledge distillation for model compression.\n\nThe integration of LRA and quantization into a unified framework is the core contribution of the paper.", "essential_references_not_discussed": "The paper adequately discusses most essential references.", "other_strengths_and_weaknesses": "I greatly appreciate the authors' contributions to the LoRA + PTQ framework. The presentation and organization of the paper are well-structured. The experiments effectively demonstrate the impact of each component in the framework step by step:\n- Comparing RB-LRA and RB-LRA + KD with LoRA.\n- Comparing WADS with PTQ baselines.\n- Demonstrating the memory and latency benefits of the framework on both Android and edge GPU platforms.\n\nOne minor concern is whether it is fair to compare WADS with post-training quantization baselines. Based on the description in the paper, WADS appears more similar to quantization-aware training (QAT) rather than pure post-training quantization (PTQ).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- It would be good to clarify how much extra time and GPU resources the scaling optimization requires.\n- How does WADS compare to alternative quantization-aware training (QAT) methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers generally agree that the proposed LRA-QViT method is effective for compressing and quantizing ViTs, with promising results on model efficiency and accuracy. However, concerns remain regarding theoretical analysis and comparisons with state-of-the-art methods. Overall, the paper is weakly accepted with a recommendation to address these issues in the final version if accepted.
Decision: Accept (poster)