Title: QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deep Learning

LLM
Abstract: The combination of linear transformations and nonlinear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex functions. This paper explores the introduction of quadratic transformations to further increase the nonlinearity of the model, with the aim of enhancing the performance of existing architectures. To minimize the additional parameters and computational burden, we propose a lightweight quadratic enhancer that leverages matrix decomposition, weight sharing, and sparsification techniques. This approach introduces only a minimal and negligible increase in parameters and forward computation, while still yielding substantial improvements in model performance. We evaluate the effectiveness of the proposed method across three tasks: text classification, image classification, and fine-tuning large language models (LLMs). In all tasks, our approach demonstrates significant performance gains.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"QuadEnhancer,\" a lightweight module designed to enhance the representational power of deep neural networks by introducing quadratic feature interactions. The core idea is to augment standard linear transformations with a quadratic term. To make this computationally feasible and parameter-efficient, the authors propose a series of design choices: (1) reuse the output of the linear transformation to form the quadratic term, avoiding redundant computations; (2) parameterize the quadratic interactions via a new matrix; and (3) make the matrix extremely sparse, modeling it as a band matrix that can be implemented with a few efficient circular shift operations. In final implementation, this adds only d parameters (where d is the feature dimension) and negligible FLOPs per layer. The authors demonstrate the effectiveness of QuadEnhancer through experiments on image classification (ViT), text classification (GPT-2), and parameter-efficient fine-tuning of large language models (LLaMA and LoRA), showing performance gains across tasks and model scales.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe proposed QuadEnhancer is a simple, general, and seemingly effective method. It acts as a \"drop-in\" replacement for standard linear layers and is applicable to a wide range of architectures (demonstrated on Transformers) and modalities (vision and language). The consistent performance improvements with minimal overhead suggest it could become a standard component in future model design, much like novel activation functions or normalization layers.\n\nThe paper's claims are backed by a convincing set of experiments. The authors evaluate their method on three distinct, important tasks (image classification, autoregressive language modeling, and LLM fine-tuning). The choice of strong, modern baselines (ViT, GPT-2, LLaMA) and standard datasets makes the results credible.\n\nThe paper is well-written and easy to follow.\n\nWeaknesses:\n\nThe choice of K={1} is well-motivated for efficiency, but its impact relative to other choices (e.g., K={-1, 1}) is not explored. An ablation on K would provide a clearer picture of the trade-off between model complexity and performance.\n\nThe decision to exclude the squared term (r=0) is justified by potential FP16 instability. However, a direct empirical comparison (e.g., training with and without this term in both FP16 and FP32) is needed to validate this claim and quantify its effect on both stability and final accuracy. In addition, recently BF16 becomes more popular and is believed to have better stability than FP16.\n\nThe paper is heavily empirical. While the motivation is sound and the results are good, the paper would benefit from a deeper discussion on why these specific local quadratic interactions are effective. For example, a theoretical analysis of how this enhancement affects the function class that the network can approximate would provide more fundamental understanding and elevate the work.\n\nQUESTIONS:\nPlease see weaknesses described above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes QuadEnhancer, a lightweight module designed to enhance non-linearity through quadratic transformation while maintaining low computational complexity. This is achieved via a modular design that can be applied to any layer in a neural network. To reduce the complexity typically associated with quadratic transformations, the authors introduce three key techniques: (1) Low-rank matrices for the quadratic terms, (2) Weight sharing, and (3) Sparsification of the weight matrix. For low-rankness, each weight matrix in the quadratic term is decomposed into the outer product of two vectors, p and q, effectively constraining it to rank 1. To further reduce the number of parameters, the weight matrix W used in the linear term is also reused in the quadratic term's P and Q matrices. Lastly, the matrix Λ, which complements W to form P, is sparsified as a band matrix, maintaining expressiveness while further reducing parameter count. By integrating QuadEnhancer into various models, the authors demonstrate consistent and efficient performance improvements across a range of tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is clearly written and easy to follow.\n- The proposed method is well-formulated with a modular structure, allowing it to be easily integrated into various architectures.\n- The method is evaluated across diverse tasks and backbone models, demonstrating its generality and practical utility.\n\n**Weaknesses**\n- While the paper presents several tricks to reduce complexity, it would be more convincing to include a complexity–accuracy trade-off graph, under settings with smaller models or limited data.\n- The experiment could be strengthened by providing a more explicit comparison with prior work in terms of both performance and computational cost.\n\nQUESTIONS:\nPlease refer to the Weaknesses part.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAdds a single banded matrix of parametric terms to the traditional linear model of parameters for a neural network (that is kd in parameters where d is the number of inputs and k is small (16/32 for LoRA)). With this small increase in parameters it enables quadratic terms to be included on top of standard neural network linear models. Shows a series of experiments on different tasks and datasets showing that this is effective.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1)\tClear presentation of approach in a directly implementable format of vector multiplication.\n2)\tWell-written, well-presented, clear and appears novel (see caveat in W2).\n3)\tApproach is forced towards cross terms, not squared terms for numeric stability.\n4)\tUsing VIT as baseline, on some smaller datasets and re-training LLMs, it shows solid gains given small increases in parameters sizes. This is consistent across a range of problems.\n\nWeaknesses:\n1)\tAlthough it is shown in the introduction that there are other attempts of quadratic based neural network models, including recently, the paper does not offer any experimental comparisons to these. Different architectures may present some challenges here, but this is not discussed. Comparison only include baselines (only VIT and GPT-2 variants), no competitor networks that have quadratic or any additional parametric approaches.\n2)\tThe claim in the introduction – on line 60 that QuadraNet V2 has substantial increases computation and parameters does not seem accurate. The QuadraNet V2 paper uses constant training time and shows small increases in number of parameters vs baselines in its experiments. (Maybe it has compute time increase, but the experiments offset this). Please comment. I may have mis-understood this. It raises a slightly broader question of whether this critique is correct of the other previous literature also.\n3)\tThere is no comparison in terms of fixed parameter numbers, or fixed training times. (Although the tables show large increases proportional to parameter increases).\n4)\tThe FLOPS comparisons are all theoretical (which are very nice numbers in these comparisons), however, the experimental FLOPS are not stated so we don't get to see if the theoretical model can be implemented in this efficient fashion.\n5)\tTraining details only follow a single model from another paper for image classification, and just 20 epochs pre-training, then 10 epochs fine tuning on text classification datasets. This seems a little narrow to understand whether these are sustained results with different numbers of epochs. Are the results robust?\n6)\tIt is slightly surprising that in Tab 8, the 16 rank outperforms 32 in the majority of cases for LLaMA{_,2}-7B. What is the cause of this?\n\nOverall the paper appears good, however, its not clear that the authors have sufficiently established its empirical strength over other approaches to increasing the parameterisation of the standard linear activation model.\n\nQUESTIONS:\n1)\tEmpirical comparison of recent non-linear neural network approaches rather than just a linear baseline.\n2)\tCan QuadraNet V2 be simply dismissed by computational number of parameters? Is this accurate more broadly for the other papers stated?\n3)\tEmpirical FLOPS comparison.\n4)\tDoes the result hold up over broader training conditions? The experimental results have not clearly established this. Conditions shown are too narrow to be confident that the results are generally strong. Especially, does the result generalize for epochs, are they stable?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of improving the expressiveness of neural network architectures by introducing a new feed-forward dense layer design called QuadEnhancer. This module augments traditional dense layers with structured quadratic interactions between features, implemented efficiently via low-rank decompositions and weight sharing. The proposed layer is designed to be a lightweight drop-in replacement for standard MLP layers, providing enhanced modeling capacity with minimal additional cost. The authors present an efficient implementation of QuadEnhancer and analyze some of its theoretical properties. Empirical evaluations include experiments on image classification benchmarks as well as language model fine-tuning, demonstrating improved performance across both domains.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The proposed layer introduces virtually no overhead. The number of additional parameters is negligible, and the computational cost should be minimal in practice. Despite being noted as a limitation, the overhead is likely irrelevant: pointwise nonlinearities after standard dense layers are memory-bound, and with a small $k$, a well-optimized CUDA or Triton kernel fusing the quadratic interaction with the linear and bias terms should match the speed of a standard dense layer.\n2. Experimental results show consistent improvements across multiple architectures and domains, including both image classification and LLM fine-tuning tasks.\n3. The code for all experiments is provided, which supports reproducibility. I have not personally run or verified the code, but the effort to share it is appreciated.\n\n **Weaknesses**\n\n1. No mention of gated linear units: The paper does not mention or compare with widely-used gated MLP variants like [GEGLU](https://arxiv.org/abs/2002.05202) and [SwiGLU](https://arxiv.org/abs/2002.05202), which also introduce quadratic interactions. For example, SwiGLU can be expressed as:\n    \n    - `w1_out = X @ W1`\n    - `w2_out = X @ W2`\n    - `SwiGLU = w1_out * sigmoid(w1_out) * w2_out`  \n        This includes multiplicative terms between transformed inputs and is used in most modern LLMs. Ignoring such strong and relevant baselines is a serious omission.\n2. Hard to assess how meaningful the improvement is: The field is full of small changes to MLP layers, and it's usually possible to show improvement by carefully choosing tasks, hyperparameters, or model sizes. The paper does not provide state-of-the-art results on any benchmark. A method that truly adds value would likely stand out more clearly across many tasks and domains.\n3. No analysis of scaling behavior: Architectural changes often show benefits on small models but lose their impact as model size increases. The experiments in the paper use relatively small-scale models and do not demonstrate behavior on wider range of scales.\n\nQUESTIONS:\n1. Am I correct that the ViT architecture used in your experiments has a two-layer MLP with GELU as the nonlinearity, and does not use SwiGLU or GEGLU? If so, why did you choose this configuration, given that GELU-based MLPs are used in older models, and most modern architectures—including recent LLMs—use SwiGLU instead?\n\n2. Do you have results on models of different sizes? I understand that scaling up may be costly, but even a sweep over smaller models would help clarify how the benefit of your method depends on model size. A plot showing the gain from QuadEnhancer as a function of model size (under a standardized setup) would be very helpful and reduce concerns about cherry-picked results.\n    \n3. Could you clarify how exactly you combined your method with LoRA? If the base architecture remains unchanged (e.g., still uses a two-layer GELU MLP), how does the quadratic enhancement interact with the low-rank LoRA update? Did you modify the `A @ B` structure to insert a quadratic term between them?\n\nLIMITATIONS:\nI agree with authors", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}