Title: Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: Low-Rank Optimization

Model Compression

Low-Rank Adaptation

Parameter-Efficient Pre-training

Iteratively Reweighted Least Squares

Log-Determinant
Abstract: Parameter-efficient training, based on low-rank optimization, has become a highly successful tool for fine-tuning large deep-learning models. However, these methods fail at low-rank pre-training tasks where maintaining the low-rank structure and the objective remains a challenging task. We propose the Quadratic Reweighted Rank Regularizer dubbed Q3R, which leads to a novel low-rank inducing training
strategy inspired by the iteratively reweighted least squares (IRLS) framework. Q3R is based on a quadratic regularizer term which majorizes a smoothed log determinant serving as rank surrogate objective. Unlike other low-rank training techniques, Q3R is able to train weight matrices with prescribed, low target ranks of models that achieve comparable predictive performance as dense models, with
small computational overhead, while remaining fully compatible with existing architectures. In experiments, we are able to truncate 60% of the parameters of a ViT-Tiny parameters with marginal loss in CIFAR-10 performance and up to 80% with only 4% accuracy drop. The efficacy of Q3R is confirmed on Transformers across both image and language tasks, including for low-rank fine-tuning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a parameter-efficient training method using a quadratic regularizer that majorizes a smoothed log-determinant as a rank surrogate. The approach enables effective low-rank pretraining and fine-tuning of deep models, demonstrating competitive performance across both vision and language tasks.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed method explicitly induces low-rank structure in deep neural network weights via continuous regularization. It is also robust to the choice of the target rank hyperparameter $r_{target}$, reducing the need for extensive hyperparameter tuning during initialization.\n\nWeaknesses: The experiments on language tasks are limited to fine-tuning only, which may not be sufficient to fully support the authors’ claim.\n Based on the paper’s title and abstract, one would expect competitive performance in pre-training across vision and language domains.\nWhile it may be impractical to train large language models from scratch, a more direct application of the proposed method to pre-trained weights—rather than using a LoRA-style side path—would better support the claim of unified low-rank training.\nThis does not imply that the experiments are poorly designed, but rather that the scope of the evaluation falls short of the expectations set by the paper.\n\nQUESTIONS:\n1, $\\textit{Line 283}$, what is the another tuning hyper-parameter?What is the other tuning hyperparameter mentioned here, in addition to $\\lambda$? Please clarify.\n\n2, The use of $r_{target}$ raises concerns about the sensitivity of the model to the predefined $r_{target}$, the step size, and the weight initialization. \nWhile the sensitivity to $r_{target}$ is briefly addressed in the limitations section, could the authors elaborate on the impact of the other two factors—especially how step size and initialization might affect training stability and final performance? \nFor instance, is it beneficial for the initial weight matrices to have a rank close to $r_{target}$ , or is random initialization sufficient?\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to use a regularizer which reduces the rank of the weight matrices during neural network training. The regularizer is a smooth approximation of the log-determinant of the weight matrix and optimized using the iteratively reweighed least squares (IRLS)\nalgorithm. This leads to an elegantly modified AdamW algorithm, which instead of the regular weight-decay receives a \"modified weight\ndecay\" which reduces the rank of the weight matrix. The method is evaluated in a small-scale setting on CIFAR-10, 100 and NLP tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper was easy to follow, and the mathematical derivations are rigorous building on recent works on IRLS from applied mathematics.\n- The proposed modification of AdamW is elegant, easy to implement and comes with little overhead.\n\nWeaknesses:\n- The experiments seem preliminary and small scale, making it is hard to assess whether the proposed method will work well for larger and realistic settings. The accuracies (?) reported in table 1 are very low for CIFAR-10, a simple LeNet CNN gives over 80%.\n- As the method is motivated as a competitor to LoRA, it would be more convincing to see similar experimental setups as in LoRA papers, for instance, fine-tuning Llama 3 or Qwen rather than RoBERTA models. \n- The use of IRLS for rank regularization is not new and a standard technique from applied mathematics. Given that there is limited novelty in the method, this is a mostly empirical work which requires larger scale experiments to be valuable to the community.\n\nQUESTIONS:\n- What are the numbers in Table 1, is it accuracy or error? The text and figure caption only talk about performance, which makes it hard to assess the results. \n- Is there an inherent overhead in the method which prevents to run it on larger models, such as Llama series, as seen in LoRA papers?\n- An alternative approach to low-rank regularization are proximal algorithms using e.g. proximal operators of the nuclear norm or its nonconvex variants. Is there an inherent advantage of the IRLS approach over proximal algorithms?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Q³R (Quadratic Reweighted Rank Regularizer), a novel regularization framework designed to directly promote low-rank structure in deep neural network weights during training. Unlike traditional parameter-efficient fine-tuning (PEFT) approaches that rely on low-rank updates (e.g., LoRA, LoRITa) or post-hoc truncation, Q³R integrates a smoothed log-determinant surrogate into training via a majorizing quadratic model. The method is computationally tractable through a periodically updated reweighting operator and is embedded into a custom optimizer, AdamQ³R, which decouples the regularization from the adaptive gradient updates. The paper demonstrates that Q³R achieves strong performance on both image classification and NLP tasks, outperforming prior low-rank methods in aggressive compression settings (e.g., ViT-Tiny on CIFAR-10, RoBERTa on GLUE), and unifies low-rank pretraining and fine-tuning in a single framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The use of iteratively reweighted least squares (IRLS) for low-rank deep network training is novel. Prior methods rely on factorized updates or heuristic truncation, whereas Q³R directly optimizes a smooth rank surrogate via majorization.\n\n2. Nice theoretical understanding. The paper provides a rigorous and detailed derivation of the Q³R regularizer and its integration into training via quadratic majorization.\n\n3. Strong empirical experiments: Experiments on ViT-Tiny/ViT-Base (CIFAR-10/100) and RoBERTa (GLUE) convincingly show that Q³R outperforms or matches state-of-the-art baselines (LoRA) especially at high compression ratios.\n\nWeaknesses\n\n1. Limited computational efficiency discussion. While the paper argues that Q³R introduces limited overhead, no explicit wall-clock time comparisons or FLOP breakdowns are provided. Since the method involves periodic SVDs and reweighting operations, this omission makes it hard to assess practicality in large-scale settings.\n\n2. Hyperparameter Sensitivity: Although the authors discuss stability of the regularization weight, the method still requires tuning of it, rank targets, and reweighting periods. The paper does not fully explore how sensitive performance is to these choices.\n\n3, Limited scope of the experiment: The method is tested only on moderate-scale models and datasets (CIFAR). Its applicability to large-scale LLMs (e.g., GPT-class models) remains unknown, such as fine-tuning and pre-training LLM on tasks such as C4.\n\nQUESTIONS:\n1. Can the authors provide wall-clock training time comparisons between Q³R, LoRA, and dense baselines? \n\n2. Can authors provide more discussion and tuning results regarding hyperparameters? How sensitive they are?\n\n3. Can authors evaluate the method in large-scale setting such as LLM fine-tuning or pre-training?\n\nLIMITATIONS:\nYES", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors introduce Q3R, a new method for training deep neural networks that enables effective model compression by promoting low-rank weight structures during training. Unlike methods like LoRA that modify only parts of a pretrained model, Q3R starts with standard full-sized (dense) weight matrices and gradually encourages them to become low-rank through a specially designed regularization term. This regularizer gently pushes the smaller singular values of weight matrices to shrink over time, encouraging a low-rank structure. This allows the model to be effectively compressed after training without significant accuracy loss. For improved convergence of the process, authors also present a custom optimizer called AdamQ3R that helps apply this regularization efficiently. Experiments on ViT and language (RoBERTa) models show that Q3R achieves large reductions in model size, up to 80% fewer parameters, with minimal drops in performance. With this, the presented approach is able to outperform other low-rank methods especially at high compression levels.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written and and easy to understand, It has several strengths:\n1. Proposed approach can be applied during both pre-training and fine-tuning. This makes it more flexible than methods like LoRA that are typically limited to fine-tuning.\n2. The method is based on a solid theoretical foundation using smoothed log-determinant surrogates and quadratic upper bounds.\n3. The approach of shifting a matrix towards low-rank is interesting and designed very intelligently. It avoids overparameterization and post-hoc truncation by encouraging low-rank weights directly during training.\n4. A custom optimizer, AdamQ3R, is designed that helps separate the effects of regularization from standard gradients, improving stability and generalization.\n5. Experimental results across both vision (ViT) and language (RoBERTa) tasks show strong performance, with up to 80% parameter reduction and minimal accuracy loss, outperforming prior methods at high compression levels.\n6. Q3R is compatible with standard architectures and can be integrated without structural changes.\n\nWhile the work is very exciting and relevant for the scientific community, there are several weaknesses as well.\n1. Since the method starts from full-rank systems, the benefit can clearly be considered to be for inference-time and not training time. This also implies that unlike LoRA type approaches, this approach is not suited for budget-constrained systems for training.       \n2. Periodic SVD and reweighting computations introduce extra training-time overhead, especially for larger models.\n3. The approach requires tuning hyperparameters like regularization strength and target rank, which may not transfer well across datasets or tasks.\n4. The experimental evaluation is limited to relatively small and easy benchmarks (CIFAR-10/100 and GLUE), which may not fully reflect the performance and scalability of Q3R on more complex, real-world tasks or larger models.\n5. If I understand correctly, the method promotes low-rank structure but does not strictly enforce a final rank, meaning post-training truncation may still be needed for exact control.\n\nQUESTIONS:\nIt would be great if the authors can address the following questions:\n\n1. Have you considered or experimented with initializing from low-rank approximations or progressively reducing rank during training to improve training-time efficiency, especially for resource-constrained settings? And if you compare this setting with the dense full-scale setting as starting point, do you see performance difference?\n\n2. Could the authors quantify the additional training-time cost introduced by periodic SVD and reweighting operations, and are there strategies (e.g., using approximate or less frequent updates) that might reduce this burden while retaining performance?\n\n3. How robust is Q3R to variations in regularization strength and target rank? Would the authors consider providing heuristics or adaptive mechanisms to guide these choices across tasks and model scales?\n\n4. Since Q3R promotes but does not guarantee a specific rank, could the authors clarify if post-training truncation is required in practice? Would adding an explicit rank constraint during optimization improve rank control without harming performance?\n\n5. Have the authors considered evaluating Q3R on more challenging benchmarks such as ImageNet or large-scale language tasks beyond GLUE to better demonstrate the method’s scalability and generalization potential?\n\nLIMITATIONS:\nYes, however, there is no discussion towards social impact, especially the bias that could be introduced by this compression.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}