QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
Authors: Qian Chen
Linxin Yang
Akang Wang
Xiaodong Luo
Yin Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Deep Learning
LLM
Primary area: deep_learning

Abstract: The combination of linear transformations and nonlinear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex functions. This paper explores the introduction of quadratic transformations to further increase the nonlinearity of the model, with the aim of enhancing the performance of existing architectures. To minimize the additional parameters and computational burden, we propose a lightweight quadratic enhancer that leverages matrix decomposition, weight sharing, and sparsification techniques. This approach introduces only a minimal and negligible increase in parameters and forward computation, while still yielding substantial improvements in model performance. We evaluate the effectiveness of the proposed method across three tasks: text classification, image classification, and fine-tuning large language models (LLMs). In all tasks, our approach demonstrates significant performance gains.

QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
Qian Chen 122 Linxin Yang Akang Wang2.." Xiaodong Luo?,, and Yin Zhang?
School of Science and Engineering. The Chinese University of Hong Kong, Shenzhen, China Shenzhen International Center for Industrial and Applied Mathematics, Shenzhen Research Institute School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
of Big Data, China
Abstract
The combination of linear transformations and non-linear activation functions forms the foundation of most modern deep neural networks, enabling them approximate highly complex functions. This paper explores the introduction of quadratic transformations further increase nonlinearity in neural networks, with the aim of enhancing the performance of existing architectures. To reduce parameter complexity and computational complexity, we propose a lightweight quadratic enhancer that uses low- low-rankness, weight sharing, and sparsification techniques. For a fixed architecture, the proposed approach introduces quadratic interactions between features at every layer, while only adding negligible amounts of additional model parameters and forward computations. We conduct set of proof-of-concep experiments for the proposed method across three tasks: image classification, text classification, and fine-tuning large-lagguage models. In all tasks, the proposed approach demonstrates clear and substantial performance gains.
Introduction
In modern deep learning, the majority of successful architectures are built on the combination of linear transformations followed by non-linear activation functions. This fundamental framework allows neural networks to learn complex mappings from input to output. The linear transformation serves to project the input data into different space, while the non-linear activation function introduces the necessary complexity, enabling the network to model intricate, nigh-dimensional patterns.
The introduction of non- non-linearity has been a key factor in the success of neural networks, enabling them to approximate highly complex functions and capture intricate data patterns. This ability is what allows neural networks to tackle problems ranging from image classification to natural language processing. Over the years, significant advancements have been made the architecture of neural networks, starting with MLPs for simple regression and classification tasks [38 T. followed by CNNs
for image data [22] 21 12 233 RNNs and LSTMs for sequential data [14 42 25 24), and more
recently. Transformer models (5) that dominate a wide range of fields, including natural language computer vision 19 [) and beyond [46] [8].
processing 35
The success of these architectures highlights the importance of incorporating con-linearity to achieve the level of model expressiveness needed for complex real-world tasks. As result, researchers have continually sought ways to introduce more advanced non-linear transformations within neural networks to enhance their capabilities. These explorations have generally followed three main
Conresponding authors: Yin Zhang syinzhang@cuhk.eduu >, Akang Wang <wangakang sribd.en>
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

directions: (i) employing more complex activation functions, (ii) designing non non-linear network modules, and (iii) replacing linear operations with polynomial transformations.
Firstly, recent advancements in neural network activation functions have focused on introducing more complex forms of non-linearity to improve model expressiveness and tackle real-world tasks. For instance, Swish [37. which combines a sigmoid function with a linear term, offers a smoother, more flexible non-linearity, outperforming ReLU in deep networks. Further enhancements include Mish which introduces a combination of tanh and softplus to provide even finer gradients, and GELU probabilistic function used in Transformer models, which incorporates multiple non-linear operations such as tanh and cubic terms to introduce smooth, complex non -ineiniaritty Despite their success, all these activation functions focus on element- wise transformations, capturing non-linearity at the level of individual neurons, yet often fail to exploit the potential for interactions between neurons that could further enhance representational capacity.
31
13
Secondly, there have been attempts to design non-linear network modules that go beyond the simple application of activation functions. For example, the introduction of GRU [3] and LSTM [14] units in recurrent networks adds non-linearity by incorporating gates that regulate the flow of information over time. Additionally, the attention mechanism [5], introduces context-dependent weighting of neurons, allowing the network to focus on more relevant parts of the input. While these modular approaches have shown great success, they are often task- sseciipei and depend on network architecture, making them less universally applicable.
Finally, replacing linear operations with polynomial transformations represents more radical approach to non-linearity. This approach includes polynomial networks 44, which replace standard linear layers with polynomial operations and reduce the number of parameters needed for higher- order terms through tensor decomposition techniques. Recently, [28] extended second-order methods to convolutional neural networks. 7 g explored the advantages of quadratic transformations in terms of both representational power and training efficiency. More recently, [48] 47 combined quadratic methods with neural architecture search, further improving model performance. These methods have demonstrated both theoretically and experimentally that higher-order transformations can enhance a model's representational capacity, thereby boosting its overall performance. However, these approaches have typically been limited due to the substantial increase in parameters and computational cost associated with higher-order terms.
While methods involving more complex activation functions and specialized network modules have demonstrated significant success, polynomial transformations- despite their theoretical poten- tial-have been underexplored in many real-world applications. The main challenge lies in the fact that higher-order terms require considerable number of parameters, which can significantly increase model complexity. For instance, even with aggressive decomposition techniques, the work of 4 still requires (n?) parameters in high-order terms. Nevertheless, as a complementary approach to existing methods. higher-order transformations offer the potential to further enhance expressiveness without conflicting with activation functions or specialized network modules. This motivates the investigation of how polynomial transformations can be integrated with standard non-linearities to improve model performance while maintaining control over computational complexity.
In this paper, we propose a novel enhancement to traditional neural network architectures by in- troducing a quadratic transformation at each linear layer. This quadratic enhancement introduces higher-order interactions between neurons through quadratic terms, while maintaining computational efficiency by reusing the linear activation outputs and sparsifying parameter matrix of the quadratic term. Our method significantly reduces the number of parameters and operations required for standard quadratic transformations, making it light, easily applicable to modern architectures. The key contributions of this paper are as follows:
We introduce novel quadratic transformation technique that enhances non-linearity in neural networks by leveraging quadratic transformations to capture richer interactions between neurons.
We present a sparsified version of the quadratic transformation that reduces the number of parameters, ensuring that the additional computational overhead remains minimal.
We evaluate the effectiveness of our approach through extensive experiments on three tasks, including image classification, text classification, and fine-tuning of large language models (LLMs), showing substantial performance improvements over baseline models.

---

2 Preliminaries
2.1
Notation
Unless otherwise specified, scalars are denoted by normal font (e.g., æ, y). vectors are denoted by bold lowercase letters (i.e., æ, A), and matrices are denoted by bold uppercase letters (c.g.. W. P
and A).
2.2 Standard linear transformation
A typical linear transformation converts input signal R to feature vector R by multiplying a weight matrix W E _dxn as
Wa + b.
(I)
The transformed vector y is then fed to non-linear activation functions. When stacked together in multiple layers, such linear operations, combined with nonlinear activation functions, form the basis for more complex neural network architectures. These layers can be repeated and organized in various ways to tackle wide range of machine learning tasks, from classification to regression, progressively transforming imput data into high-level abstractions.
2.3 Primary objective
The goal of this work is replace the standard linear operation in neural network layer with quadratic function, while keeping activation functions unchanged. Specifically, we aim to replace the linear transformation (1) with quadratic function g : RR" - R. such that the output becomes:
=
g W.A)
where W still represents the parameters of the linear term, and A represents the parameters associated with the quadratic terms. A key challenge is to ensure that the additional matrix A, responsible for quadratic terms, only adds a small number of extra parameters relative the existing ones in W. while the extra computational cost introduced by the quadratic transformation remains minimal compared to the standard linear transformation. We seek enhance the expressiveness of the model by introducing higher-order interactions between features without significantly increasing the model's complexity or computational overhead.
Methodologies
Based on the objective outlined in Section 2.3, this section will provide detailed description of the design of the quadratic function g(æ; W, A).
3.1 Quadratic transformation single layer
Let us begin by considering standard quadratic transformation that introduces additional nonlinearity the linear transformation in Equation
by adding quadratic term. The resulting transformation
is given by:
x: Vi
+ Wr +b,
Vax
where Vi...., Va R=x= are the trainable weights for the quadratic terms, W E gdxn represents the weights for the linear term, and b Rd is the bias vector. However, is, this modification introduces a significant increase in the number of parameters, requiring (dn?) additional parameters, which would substantially increase the complexity of the model.
3.2 Rank-1 matrices for quadratic terms
common technique to reduce the number of parameters in matrices to impose low-rankness. Specifically, set each matrix V,, Vi 1,..., d, in 3 to be rank-1. That is, for two vectors

---

Ps: R",
Vi = p.q.
Vi I....,d. d.
(4)
Consequently, Equation
becomes
= P ② (Qx) Wx + b
(5)
where P Pp,..., ,Pl T Rdxn Q g1..... qa] E Rdxx and @ denotes the Hadamard
(element-wise) product. This approach effectively reduces the number of extra parameters from (dn²) to (2dn), significantly lowering the model's complexity while still allowing it to capture the quadratic interactions between the input features.
3.3
Weight sharing
To further reduce the computational and parameter complexities of the quadratic transformation. we introduce weight sharing. Specifically, we share the weight matrix W between the linear and quadratic terms. By defining:
: AW. Q :
where E Rdxd is a new weight matrix that differentiates the feature space of P and Q, we can rewrite the quadratic transformation as:
(AWx) x) @ (Wæ) Wx + b (A:) ÿ bb
where ý := Wx represents the linear transformation the input. This reuse the weight matrix W provides two key advantages. First, by sharing W across both the linear and quadratic components, we significantly reduce the number of model parameters. Instead of three independent parameters W, P.Q _dxn the model now only needs learn W and A. Second, the linear response = W Ø is computed once and reused in both the linear and quadratic operations, reducing the computational overhead.
3.4 Sparsification of A
While the rank-1 decomposition significantly re- duces the number of parameters in the quadratic term, the weight matrix A still requires (d2) parameters, which could result in substantial increase in model complexity. To address this, we apply a sparsification strategy to by con- verting it into a band matrix. In this structure, non-zero elements are restricted to specific "band". Additionally, we introduce two small triangular regions in the lower-left and upper- right corners, as illustrated in Figure [1] This
Trainoble
Zeros
lines
A
A
A
A. A
A
d
=
k
Figure Sparse structure of A.
allows the band matrix with width be divided into & lines of d-dimensional parameters, where the missing elements of these lines are filled by values from the triangular regions. Hence, the total number of trainable parameters in is reduced to x with being much smaller than (e.g..
k = 1). With this sparse structure, the computation of Aÿ becomes:
Aÿ = Σ A C Roll(gjr).
(8)
rEX:
where K. := -1.0, is the set of shifts with [A] = k. X, represents line of parameters and the function Roll(-) defined as
Roll(y. r) = [ūi+ mod di.-.., Da+trr mod wy
The advantage of using band matrix that it ensures the rank of A remains sufficiently high, even when the number of parameters is significantly reduced (e.g., when à = 1). Specifically, it guarantees that rank(P P) < min (rank(A) (A), rank W)} does not get too small, preserving capacity of the quadratic term. This is crucial for maintaining the expressiveness of the model while reducing both parameter count and computational overhead. In our experiments, we exclude the shift 0, as it produces square terms 2 whereas any non-zero shift produces cross terms. In practice,
representational

---

the square terms are more prone to numerical instabilities (such overflows or exploding gradients) than the cross terms, especially when training with the FP16 precision whose limited dynamic range amplifies numerical instability issues. The rationale behind this design choice is illustrated in Example 3.1 3.1] Consequently, our experiments use K {1), as shown in the right-hand subfigure of Figure [ This choice effectively results in quadratic interactions between nearest-neighbor neurons, excluding the self-interaction.
N(0 1) be independent standard normal random variables. The expectations and variances of the square terms x² and 2z2 as well the cross term z1 are as V Var [+]] = and E[z122] = O, V.V |r[ = The cross retains the expectations and variances of a normal distribution, while the square terms do not. Monte Carlo estimates in Table Tableffurther show that the square terms are far more likely to attain
Example 3.1
Ltd. Let 32 ~
follows: E[aî] = E[ = 1, V ar [=][=]]
large absolute values compared the cross terms.
Table 1: Estimated probability of the pure quadratic and cross terms
probability
t= 4
v= 8
8 16
p ((
4.5 x 10-2
4.7 X 10-3
6.33 x 10
pl 22
v)
7.6 10-3
4.1 10-5
3.37 10-10
3.5 Quadratic enhancer
The complete workflow of the proposed quadratic enhancer is sketched in Figure 2 which is divided into two panels. We recall the conventional linear transformation with bias, upper panel. The lower panel then details the quadratic enhancer itself. First, the linear transformation W æ is computed and fed the enhancer. A set of rolling shifts Roll(-) is applied to ÿ: these shifted copies are linearly combined by a learnable, band-sparse matrix A, producing the refined response Aÿ. after which quadratically augmented feature (Aÿ) @ ÿ is calculated. The final output is then = (Ağ) 3 ÿ + ý + b. Figure 2 illustrates how a single linear transformation can be augmented by the quadratic enhancer. Crucially, the same enhancer block can be attached every linear layer in a neural network, endowing the entire model with richer quadratic interactions while incurring only negligible increase in parameters and computation.
Wx b, in the
W
b
Iaput
Linear Transformation
Oulput
Paramotions
Intermediates
Roll
y/
Matrix multiplication
Quadratic Enhancer
Hadamard product
@
Ag
Figure 2: An overview of the quadratic enhancer.
3.6
Cost analysis
In this section, we evaluate the overhead introduced by the quadratic enhancement in terms of both parameters and inference FLOPs, which ultimately impact the model's efficiency.
Parameters:
The quadratic enhancer introduces a single learnable matrix A E Rdxd but with non-zero entries restricted to bandwidth of width i (where < d, e.g., 1). This results in i x d free parameters in A. In contrast, the standard linear layer employs a weight matrix W E Rd with x n parameters. The relative overhead terms of parameters is thus: ad Olk/n), which negligible in practice given that & < (typically d).

---

Inference FLOPs:
The quadratic enhancer reuses the linear activation ÿ Wx, resulting in only three additional operations. These include: (i) a matrix multiplication Aÿ which requires 2kd FLOPs. (ii) a Hadamard product @ÿ which costs d FI.OPs, and (iii) an addition +ÿ which adds another d FLOPs. Therefore, the total number of extra FL.OPs introduced by the quadratic enhancer is 2 + 1)d FLOPs. For comparison, the original linear transformation Wr requires 2nd FLOPs, and the bias addition adds d FLOPs, leading to a total of 2nd + d FLOPs for the original computation. The relative Okk/n), which, again, becomes negligible for k & n.
overhead is thus: 2(k++I)d
2
4 Experimental results
In this section, we conduct an evaluation of the proposed quadratic enhancer across three tasks. To investigate the performance with minimal overhead, we focus on evaluating the quadratic enhancer with K. {1}. All experiments were conducted using four NVIDIA A100 80GB. The experimental code is publicly available at httpps://gi /ggbtbb........ com/ chitcattarr /uaaauuaaaaaneerrr
4.1 Image classification
Image classification represents one of the most classical and foundational tasks in computer vision. serves as benchmark for evaluating the effectiveness of novel neural architectures and training methodologies. Following common practice. scale dataset, subsequently fine-tuning the pre-trained models on various target datasets. Specifically, we first pre-train on ImageNet-II 1k [20). then fine-tune and evaluate the models across several diverse downstream datasets.
experiments involve initial pre-training a large-
Datasets: Our experiments begin with ImageNet- Ik for the initial pre-training stage. For downstream evaluation, we use six widely recognized benchmarks: Caltech 9 CIFAR-1Ó, CIFAR- [20]. Flowers 32 Food [2). and Pets [33] Caltech is a classical image recognition dataset containing various object categories. CIFAR-IO and CIFAR-100 are commonly used for small-scale visual recognition, with the former covering 10 categories and the latter 100. The Flowers dataset focuses on tine-graine classification of flower species, while the Food dataset includes a variety of food categories. The Pets dataset is centered on distinguishing different pet breeds. Further details about these datasets can be found Table 2]
The Vision Transformer (VIT) serves our baseline model due to its demonstrated effectiveness and wide adoption. By varying key hyperparameters such as hidden size and number of layers, we consider three model sizes: ViT-M, ViT-XT, and ViT-T. Specific hyperparameter configurations are detailed in the Table
Baselines:
Table 2: Image datasets
Dataset
# Classes
Samples
caltech
102
9.0k
cifar10
60k
cifar100
100
60k
flower
102
8.
Training settings: In the pre-training phase, each baseline model and its corresponding quadratic enhancer variant are trained on ImageNet-Ik, Ik. The training parameters, including batch size, learning rate, number of epochs, and total training duration, are consistent with the settings outlined in [29]. After pre-training, each model is fine-tuned individually on all downstream datasets, with performance evaluated on the corresponding validation sets.
pet
37
7.4k
food
101
101k
The experimental results are presented in Table 4 where the column labeled "ImageNet" indicates valida- fion accuracy obtained during pre-training, and subsequent columns represent accuracy scores achieved on each down- stream dataset after fine-tuning. Models incorporating the quadratic enhancer are indicated with "+QE." As shown in Table 4, models equipped with the quadratic enhancer consistently outperform their baseline counterparts across all datasets. Specifically, ViT-M+QE surpasses VIT-M by 1.60% on ImageNet and achieves sub- stantial gains on downstream tasks, notably improving Caltech by 2.55% and CIFAR-100 by 2.34%. Similarly, ViT-XT+QE and ViT-T+QE outperform their respective baselines, demonstrating sig- nificant accuracy boosts across datasets. The quadratic enhancer proves especially beneficial on the challenging Pets dataset, where ViT-XT+QE achieves an impressive 6.94% improvement over
Results:
Table 3: Model parameters of ViTs.
Embed_ding
Layers
FFN_ _im
VIT-M
192
6
768
VTT-XT
128
12
512
VIT-T
192
12
768

---

VIT-XT. Overall, these consistent performance enhancements underline the quadratic enhancer's capability to enrich model effectiveness across various visual classification tasks.
Table 4: Accuracy (%) of VITs with and without the quadratic enhancer.
Model
Params
ImageNet
Caltech
Cifar10
Cifar100
Flowers
Food
Pets
Avg
VIT-M
2.45M
63.70
87.77
96.35
80.25
74.01
83.94
91.03
82.44
ViT-M+QE
2.47M
65,30
90.32
97.09
82.59
75.58
84.63
91.88
83.91
VIT-XT
2.82M
66.04
90.25
96.51
81.24
84.41
85.47
91.03
84.99
ViT-XT+QE
2.83M
67.34
90.77
96.78
82.64
86.37
85.85
97.97
86.82
VIT-T
5.37M
73,96
93.07
97.97
86.13
86.56
88.42
93.87
88.57
VIT-T+QE
5.40M
75.15
94.03
98.03
86.88
87.25
88,59
94.95
89.27
4.2 Text classification
Text classification is cornerstone task in natural language processing, underpinning applications ranging from sentiment analysis to topic categorization. It remains a classical and fundamental benchmark for evaluating advances in language modeling and fine-tuning techniques. Analogous to image classification, modern text classification pipelines typically involve pre-training a general- purpose language model large corpora, followed by fine-tuning specific downstream datasets. In our experiments, we adhere to this convention: we first pre-train on a small-scale corpus, then fine-tune on several text classification benchmarks.
Datasets: For pre-training, we use the WikiText-2 dataset 30), a widely adopted corpus containing over 2 million tokens from English Wikipedia articles. While larger pre-training datasets exist, their computational demands exceed our resource constraints. For downstream text classification, we utilize six standard benchmarks: IMDB (movie review sentiment analysis) 27), Yelp (restaurant review sentiment) [7]. AG-News (topic classification) [4). SST-2 (Stanford Sentiment Treebank) 41, and Emotion (emotion recognition)
39 Detailed dataset statistics are available in Table 5
as our baseline, given its strong performance in language modeling and its widespread applicability across various downstream tasks. We evaluate two models: GPT-2/16 and GPT-2/32, differentiated by the size of their hidden dimension.
Baselines:
We use the GPT-2 architecture
36
Table 5: Text datasets.
Dataset
# Classes
# Samples
IMDB Yelp
2
50k
700k
Training settings: Each model is pre-trained on WikiText-2 for 20 epochs, with batch size 128, learning rate 0.0001, and maximum sequence length of 256 tokens, sufficient to cover most samples. Following pre-training, models are fine-tuned on each classification dataset for 10 epochs, with learning rate 0.00005, batch size 16, and other optimizer settings unchanged.
AG-News
4
120k
SST-2
2
67k
Emotion
6
20k
Table reports the perplexity on WikiText-2 and validation accuracy on each classification dataset. The first column lists model variants, with '+QE' indicating use of the quadratic enhancer. The second column shows the number of trainable parameters; the third column reports WikiText-2 perplexity (lower is better). Remaining columns present classification accuracy. As shown in Table 6 models augmented with the quadratic enhancer achieve consistent improvements. GPT-2/16+QE reduces perplexity from 4.90 to 4.81 and increases average accuracy by 0.91%, 9,,, with notable gains on IMDB and Emotion datasets. GPT-2/32+QE yields a perplexity drop from 4.57 to 4.44, and boosts average accuracy by 0.86%, driven by significant improvements on the Emotion benchmark (from 64.50% to 69.05%). These results demonstrate that even with minimal additional parameters, the quadratic enhancer can enhance language model expressiveness and downstream performance.
Results:
7

---

Table 6: Performance of GPT-2 variants with and without the quadratic enhancer (QE). Wexi is
evaluated in perplexity (lower better), whereas all other datasets are reported in accuracy (%).
Model
Params
WikiText (ppl.)
IMDB
Yelp
AG-News
SST-2
Emotion
Avg Acc.
GPT-2/16
0.71M
4.90
79.68
93.51
90.90
79.70
39.70
76.70
GPT-2/16+QE
0.82M
4.81
81.16
93.64
91.67
78.78
42.80
77.61
GPT-2/32
1,56M
4.57
84.01
93.72
91.97
81.53
64.50
83.15
GPT-2/32+QE
1.61
4.44
83.52
93.80
92.01
81.65
69.05
84.01
4.3 LLMs finetuning
Fine-tuning LLMs through parameter-efficient fine-tuning 11 is critically important due to its capability to efficiently adapt powerful pretrained models to diverse downstream tasks with minimal additional resources. We evaluate the quadratic enhancer's integration into the LoRA algorithm [5 and fine-tune three variants of the open-source LLaMA models
Our tineituning experiments focus on the commonsense reasoning task, a crucial benchmark to assess language models' practical reasoning capabilities. We use several benchmark datasets including BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA. Each dataset assesses different aspects of commonsense understanding and reasoning skills. Detailed descriptions of these datasets are provided in the appendix of [16].
Dataset:
Baselines and training settings: We select three baseline models for our experiments: LLaMA-7B 43. LLaMA2-7B [. and I.LAMA3-BB [TO), reflecting a progression of model capabilities. We employ LoRA, a widely studied parameter-efficient fine-tuning method, known for its efficiency and simplicity. For each baseline, we apply the quadratic enhancer to LoRA adapters with two ranks (r=16 and 323 Our training configurations follow the established settings from prior works [16] 26).
The experimental results are summarized in Table The first three columns specify the model names, methods (with LoRA rank indicated after the ' and '+QE' denoting the quadratic enhancer's use), and the number of parameters trained, respectively. Subsequent columns report accuracy on the commonsense reasoning benchmark datasets. Results for LoRA without QE are taken from prior works 16 26) As shown in Table 7 the quadratic enhancer consistently and significantly improves performance across all model variants and benchmarks, even with half-paramete versions where the rank is 16. Notably, LLaMA2-7B with LoRA/16+QE. achieves an impressive average accuracy improvement of 2.6 2.64% over LoRA/32. Furthermore, LLaMA3- models integrated with QE outperform the baseline LoRA models, particularly on complex reasoning tasks like HellaSwar wag and ARC datasets. This demonstrates the quadratic enhancer's 's strong ability to enhance LLMs' reasoning capabilities with minimal additional computational cost and parameters.
Results:
Table 7: Accuracy (%) of LoRA finetuning on LLaMA variants with and without the quadratic enhancer (QE).
Model
Method
Params
BoolQ
PIQA
SIQA
HellaSwag
WinoG.
ARC-e
ARC-c
OBQA
Avg
LoRA/32
53.5M
68.90
80.70
77.40
78.10
78.80
77.80
61.30
74.80
74.73
LLaMA-7B
LoRA/16+QE
27.6M
69.69
82.64
79.68
$7,11
80.11
79.41
63.99
80.20
77.85
LoRA/32+QE
54.3M
69.14
77.99
74.00
81.29
79.33
64.16
80.80
75.97
LoRA/32
53.5M
69.80
79.90
79.50
83,60
82.60
79.80
77.61
LLaMA2-7B
LoRA/16+QE
27.6M
72.26
82.86
79.78
86.98
83.66
85.35
82.60
$0.25
LoRA/3++QE
54.3M
80.24
90.01
83.03
83.41
80.80
79.77
LoRA/32
54.0M
70.80
85.20
79.90
91.70
84.30
$4,20
71.20
79
80.79
LLaMA3-8B
LoRA/160EE
27.7M
74.40
88.46
80.29
95.45
86.42
91.37
85.0
85.25
LoRA/32+QE
54.7M
74.92
89.44
81.32
95.02
87.29
89.85
79.60
86.20
85.46
4.4
Additional experimental results
In addition to the main results of the three tasks considered, we perform several further experiments to evaluate the performance and scalability of the proposed quadratic enhancer. These experiments provide further insights into its comparative advantages and practical considerations in different settings. The results of these additional experiments are reported below.
8

---

Comparison with quadratic baselines We additionally compare QuadEnhancer with two existing quadratic MLP variants: (i) QuadraNet 48). which uses the quadratic formulation y W @ W5, + War where W.. W., W. E Rdxx are learnable parameters, and (ii) SwiGLU [40, which defines the quadratic interaction as y = (War) ( O Sigmoid(W: Wi @ W2.r), with W. W2 E Rddx All experiments are conducted on the image classification task using the ViT-M model as the backbone, and the number of parameters is matched by adjusting the hidden dimension The results, presented in Table g demonstrate that QuadEnhancer consistently outperforms both quadratic baselines across all evaluated datasets.
Table 8: Accuracy (%) of different quadratic methods.
Method
Param
ImagenetIk
Cifar10
Cifar100
Food
Avg
VIT-M-QuadraNet
2,53M
61.17
95.81
79.08
81,58
79.41
ViT-M+SwiGLU
2.58M
63.25
96.76
80.58
83.91
81.13
ViT-neeQudEnaancer
2.47M
65.30
97.09
82.39
84.63
82.40
Scaling behavior Understanding the scaling behavior of model is essential for evaluating its effectiveness as both data and model sizes increase. While large-scale experiments were not feasible due computational constraints, conduct experiments on data and model sizes ranging from small to moderate scales. These experiments offer insights into the scaling behavior of the proposed quadratic enhancer. All experiments are conducted on the image classification task with varying hidden dimensions d and dataset sizes 8. As shown in Table 9 we observe that the quadratic enhancer yields increasing performance gains as both the model and dataset sizes grow.
Table 9: Accuracy (%) of different scales.
d
24, 8
50k:
= 48, = 100k
= 96,
200
192,
400%
VIT
8.99
19.18
33.
49.59
VIT+QE
9.06
19.95
34.03
50.78
Gain
0.07
0.77
0.89
1.19
Training time comparison To evaluate the trade-off's between computational cost and performance, we report the training times for two tasks: pretraining on ImageNet- 1k (Table (TO) and fine-tuning CIFAR-100 (Table TT). The results show that while the quadratie-enhanced models take slightly longer to converge in the early stages of training, they ultimately achieve superior performance. These findings suggest that while QuadEnhancer introduces some initial overhead, it offers substantial long-term performance gains.
Table 10: Accuracy (%) of pretraining on ImageNet- at different time.
Time(sec.)
Method
1k
10k
20k
30k
40k
50k
VIT
22.28
52.09
58.35
63.51
65.96
65.97
VIT+QE
11.30
51.28
59.07
63.15
67.04
Table 11: Accuracy (%) of finetuning on CIFAR-100 different time.
Time(sec.)
Method
500
1k
2k
3k
4k
5k
6k
VIT
78.79
81.78
83.00
84.21
85.29
85.29
85,29
VI++QE
79.33
81.07
83.43
84.49
85.48
86.15
86.53
Ablation study on We conduct an ablation study to investigate the effect of different choices for the set IC in image classification tasks. Models with various K. sets and hidden dimensions are trained

---

from scratch on the Caltech dataset using ViT-M as the backbone. The accuracy results, shown in Table indicate that increasing the size of K. generally improves performance. The most significant improvement occurs when moving from K. = 0 to K (1), with further increases in K. yielding diminishing returns. This suggests clear trade-off between model complexity and performance.
Table 12: Accuracy (%) when using different K. and hidden size.
Hidden dimension
K
24
96
144
192
288
47.16
54.02
58.20
59.87
60.50
61.13
(1)
48.94
54.79
59.35
60.83
61.39
61.57
(-1,1)
49.38
55.09
59.83
61.28
61.80
61.65
1-2, 2,
49.72
55.16
60.28
61.39
61.50
61.76
Limitation and conclusions
Limitation: A key limitation of the quadratic enhancer is that it introduces additional computational overhead, which, while minimal in terms of parameters and FLOPs, still represents an extra cost compared traditional linear transformations. However, our approach effectively minimizes this impact, ensuring that the performance improvements far outweigh the added complexity.
Conclusions: In this work, construct class of quadratic enhancers enable quadratic interactions among features at a neural network layer. Compared to a fully-connected linear transformation, the quadratic enhancers require only negligible amounts of extra parameters and FLOPs. Our proof- of-concept experiments, conducted across multiple tasks and multiple datasets, have confirmed the potentials of the proposed approach in delivering substantial performance improvements without notably increasing model sizes. Evidences even suggest that, in some cases, our approach may be able significantly reduce model sizes while maintaining even enhancing performance levels. At this point, however, our study is still quite preliminary. More research is definitely needed in this direction to better understand the promises and limitations of the proposed approach.
Acknowledgments
This work was supported by the National Key R&D Program of China under grant and by the Shenzhen Science and Technology Program (Grant No. GXWD20201231 1057222022 Qian Chen, Linxin Yang. and Akang Wang also acknowledge support from the National Natural Science Foundation of China (Grant No. 12301416) and the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515010306 Xiaodong Luo also acknowledges support from the Hetao Shenzhen-Hon, Kong Science and Technology Innovation Cooperation Zone Project (No. WZZSWS-KCCCB-20004
20200901175001000
References
[1] Christopher M Bishop. Neural networks for pattern recognition. Oxford university press, 1995.
[2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 12, 2014, proceedings, part VI 13, pages 446 461. Springer, 2014.
[3] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder- decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078 2014.
[4] Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Pana- gakis, and Stefanos Zafeiriou. Deep polynomial neural networks. IEEE transactions on pattern analysis and machine intelligence, 44(8):4021-4044 2021.
10

---

[5] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language technologies, volume (long and short papers), pages 4171 2019.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
[7] Feng-Lei Fan, Hang-Cheng Dong, Zhongming Wu, Lecheng Ruan, Ticyong Zeng, Yiming Cui, and Jing-Xiao Liao, One neuron saved one neuron earned: On parametric efficiency of quadratic networks. arXiv preprint arXiv:2303.0631 2023.
[8] Feng-Lei Fan, Mengzhou Li, Fei Wang, Rongje Lai, and Ge Wang. On expressivity and trainability of quadratic networks. IEEE Transactions Neural Networks and Learning Systems, 2023.
[9] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[10] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprinl urXiv:2407.21788 2024.
[11] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efffcient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[13] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1rXiv:1606.08415, 2016.
[14] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 18):1733- 1997.
[15] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[16] Zhiqiang Hu, Lei Wang. Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Wei Lee. LIm-adapters: An adapter family for parameter-effiche fine-tuning of large language models. arXiv preprint XXXx:2334400933 2023.
[17] Yelp Inc. Yelp dataset challenge, 2004.
[18] Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: a pre- trained transformer for computational chemistry. Machine Learning: Science and Technology. 30):01502 2022.
[19] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1-41, 2022.
[20] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
11

---

[22] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence Jackel. Backpropagation applied handwritten zip code recognition. Neural compulation, 1(4):541-551, 1989.
[23] Zewen Li, Fan Liu, Wenjie Yang. Shouheng Peng, and Jun Zhou. A survey of convolutional neural networks: analysis. applications, and prospects. IEEE transactions on neural networks and learning systems, 33(12)x6999-7019 2021.
[24] Benjamin Lindemann, Timo Müller, Hannes Vietz, Nasser Jazdi, and Michael Weyrich. A survey on long short-term memory networks for time series prediction. Procedia Cirp, 99:650-65 655, 2021.
[25] Zachary C Lipton, John Berkowitz, and Charles Elkan. A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv: 000/9, 2015.
[26] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
[27] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150,
2011.
[28] Pranav Mantini and Shishr K Shah. Cqnn: Convolutional quadratic neural networks. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 9819-9826. IEEE, 2021.
[29] Sachin Mehta, Farzad Abdolhosseini, and Mohammad Rastegari. Cvnets: High performance library for computer vision. In Proceedings of the 30th ACM International Conference Multimedia, pages 7327-7330, 2022.
[30] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1707:716,, 2016.
[31] Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint arXiv:19085 08681, 2019.
[32] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008
[33] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[34] Narendra Patwardhan, Stefano Marrone, and Carlo Sansone. Transformers in the real world: A survey on nlp applications. Information, 14(4):242, 2023.
[35] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9 2019.
[37] Prajit Ramachandran, Barret Zoph, and Quoe Le. Swish: a self-gated activation function. arXiv preprini 1710.05710.05944 7(1):, 2017.
[38] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propgg errors. nature, (23(6085) 1986.
[39] Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium, October-November 2018, Association for Computational Linguistics.
[40] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.55222,20000
12

---

[41] Richard Socher, Alex Perelygin. Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1631-1642, 2013.
[42] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
[43] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302.13971 2023.
[44] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[46] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 17/0. 10903, 2017.
[47] Chenhui Xu, Xinyao Wang, Fuxun Yu, Jinjun Xiong, and Xiang Chen. Quadranet v2: Efficient and sustainable training of high-order neural networks with quadratic adaptation. arXiv preprini arXiv:2405.03192. 2024.
[48] Chenhui Xu, Fuxun Yu, Zirui Xu, Chenchen Liu, Jinjun Xiong, and Xiang Chen. Quadranet: Improving high-order neural interaction efficiency with hardware-aware quadratic neural net- works. 2024 29th Asia and South Pucific Design Automation Conference (ASP-DAC), pages 19-25. IEEE, 2024.
[49] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We provide a code link in Section 4.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com http://bbb///iiiiee/eeeeeeaaaaaa casssiinnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer to Section 4.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experiments computationally intensive, and we have not yet had enough time to conduct repeated experiments to obtain error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpppp///eerrgs..cccpbba////aaa
Answer: [Yes]
Justification:
The research presented in this paper adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not present any such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer Section 4.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See the code link in Section 4.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No such usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur/i: /ttpaarrpp..cc//ooofeeeeeee////s for what should or should not be described.
20


Reviews:
[{"review_id": "vZ39GoHu8U", "reviewer": "Reviewer_gte1", "summary": "The paper addresses the problem of improving the expressiveness of neural network architectures by introducing a new feed-forward dense layer design called QuadEnhancer. This module augments traditional dense layers with structured quadratic interactions between features, implemented efficiently via low-rank decompositions and weight sharing. The proposed layer is designed to be a lightweight drop-in replacement for standard MLP layers, providing enhanced modeling capacity with minimal additional cost. The authors present an efficient implementation of QuadEnhancer and analyze some of its theoretical properties. Empirical evaluations include experiments on image classification benchmarks as well as language model fine-tuning, demonstrating improved performance across both domains.", "questions": "1. Am I correct that the ViT architecture used in your experiments has a two-layer MLP with GELU as the nonlinearity, and does not use SwiGLU or GEGLU? If so, why did you choose this configuration, given that GELU-based MLPs are used in older models, and most modern architectures—including recent LLMs—use SwiGLU instead?\n\n2. Do you have results on models of different sizes? I understand that scaling up may be costly, but even a sweep over smaller models would help clarify how the benefit of your method depends on model size. A plot showing the gain from QuadEnhancer as a function of model size (under a standardized setup) would be very helpful and reduce concerns about cherry-picked results.\n    \n3. Could you clarify how exactly you combined your method with LoRA? If the base architecture remains unchanged (e.g., still uses a two-layer GELU MLP), how does the quadratic enhancement interact with the low-rank LoRA update? Did you modify the `A @ B` structure to insert a quadratic term between them?", "limitations": "I agree with authors", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The proposed layer introduces virtually no overhead. The number of additional parameters is negligible, and the computational cost should be minimal in practice. Despite being noted as a limitation, the overhead is likely irrelevant: pointwise nonlinearities after standard dense layers are memory-bound, and with a small $k$, a well-optimized CUDA or Triton kernel fusing the quadratic interaction with the linear and bias terms should match the speed of a standard dense layer.\n2. Experimental results show consistent improvements across multiple architectures and domains, including both image classification and LLM fine-tuning tasks.\n3. The code for all experiments is provided, which supports reproducibility. I have not personally run or verified the code, but the effort to share it is appreciated.\n\n **Weaknesses**\n\n1. No mention of gated linear units: The paper does not mention or compare with widely-used gated MLP variants like [GEGLU](https://arxiv.org/abs/2002.05202) and [SwiGLU](https://arxiv.org/abs/2002.05202), which also introduce quadratic interactions. For example, SwiGLU can be expressed as:\n    \n    - `w1_out = X @ W1`\n    - `w2_out = X @ W2`\n    - `SwiGLU = w1_out * sigmoid(w1_out) * w2_out`  \n        This includes multiplicative terms between transformed inputs and is used in most modern LLMs. Ignoring such strong and relevant baselines is a serious omission.\n2. Hard to assess how meaningful the improvement is: The field is full of small changes to MLP layers, and it's usually possible to show improvement by carefully choosing tasks, hyperparameters, or model sizes. The paper does not provide state-of-the-art results on any benchmark. A method that truly adds value would likely stand out more clearly across many tasks and domains.\n3. No analysis of scaling behavior: Architectural changes often show benefits on small models but lose their impact as model size increases. The experiments in the paper use relatively small-scale models and do not demonstrate behavior on wider range of scales.", "was_revised": true, "final_justification": "I remain unconvinced that this work is meaningfully different from many similar papers proposing alternative nonlinearities. Given the popularity of the question “what other nonlinearity can we plug in to improve networks,” I find it inadequate to present a single such nonlinearity, show benchmark results, and base a paper on that. In general, I would score such work as 2, but since the paper is well written and the benchmarks are adequate, I will keep my original score of 3: Borderline reject.", "rebuttal": "> #### W1: No mention of gated linear units\n\nThank you for your insightful comment. We have added clarifications in the revised version, and respond to your concerns in more detail below:\n1. Complementary, not competitive.\n    - Gated MLP variants like GEGLU and SwiGLU are not direct competitors to QuadEnhancer, and thus are not appropriate baselines for comparison. These gated variants are typically applied within FFNs/MLPs, and involve combinations with activation functions. In contrast, **QuadEnhancer is a general mechanism that can be applied to any linear operation across a neural network -- not only FFNs/MLPs, but also convolutions, attention computations, feature projections and other components**.\n    -  Importantly, these approaches are not mutually exclusive—in fact, **our experimental setups already include gated modules** (i.e., GELUs in ViTs and GPT-2s, and SwiGLUs in LLaMAs).\n\n2. Comparison with a more appropriate baseline\n- A more suitable baseline with similar quadratic interaction reformulates linear layers as: $z = (x @ W_1)*(x@W_2) + x@W_3$. We have included new experiments comparing QuadEnhancer (QE) with this baseline (QN) using comparable model sizes. The results, summarized below, show that QE significantly outperforms QN across all datasets in general:\n\n\n|            | params(M) | imagenet1k | caltech | cifar10 | cifar100 | flowers |  food  |   pet  |   avg  |\n|:----------:|:---------:|:----------:|:-------:|:-------:|:--------:|:-------:|:------:|:------:|:------:|\n| ViT-M + QN |   5.47    |   72.15    |  92.14  |  97.79  |  85.36   |  75.39  | 86.83  | 94.05  | 86.24  |\n| ViT-M + QE |   5.40    |   75.15    |  94.03  |  98.03  |  86.88   |  87.25  | 88.59  | 94.95  | 89.27  |\n| ViT-T + QN |   2.53    |   61.17    |  85.36  |  95.81  |  79.08   |  34.50  | 81.58  | 98.31  | 76.54  |\n| ViT-T + QE |   2.47    |   65.30    |  90.32  |  97.09  |  82.59   |  75.58  | 84.63  | 91.88  | 83.91  |\n\nWe hope this clarifies the positioning of QuadEnhancer in relation to gated MLPs and highlights its broader applicability and effectiveness.\n\n\n> #### W2.1: Hard to assess how meaningful the improvement is: The field is full of small changes to MLP layers, and it's usually possible to show improvement by carefully choosing tasks, hyperparameters, or model sizes. \n\nThank you for raising this important point. We fully understand the concern about the potential influence of selective experimental setups. To address this:\n1. Choice of Tasks: We intentionally selected well-established and widely-used benchmarks (though not the latest) in both image classification and language tasks to ensure that our evaluation is grounded and meaningful.\n2. Hyperparameters: To maintain fairness, we directly adopted the hyperparameter configurations from prior work [1,2] for the image classification and LLM fintuning tasks, without additional tuning.\n3. Model Sizes: Due to limited computational resources, our experiments were conducted with relatively small models. Even though, for LLaMA fine-tuning, the model sizes are still quiet standard and consistent with prior studies.\n\n> #### W2.2 The paper does not provide state-of-the-art results on any benchmark.\n\nThank you for your thoughtful feedback. We would like to clarify the intended contribution and our perspective on benchmarking:\n\n1. Focus on foundational improvements: Our primary goal is not to achieve SOTA results on specific benchmarks, but rather to propose a more powerful alternative to standard linear operations. We hope this can serve as a useful building block for the broader community in designing more efficient neural networks.\n2. SOTA involves many external factors: While SOTA results are certainly valuable, they often depend heavily on task-specific tuning, architectural tweaks, and implementation tricks. We view such results as complementary rather than essential to the core contribution of this work.\n3. Future direction: We acknowledge the value of SOTA performance and plan to explore this direction in future work. We appreciate your perspective and will keep it in mind as we continue to develop and evaluate this method.\n\n> #### W3&Q2: No analysis of scaling behavior: Architectural changes often show benefits on small models but lose their impact as model size increases. The experiments in the paper use relatively small-scale models and do not demonstrate behavior on wider range of scales. \n\nThank you for your thoughtful comment. We have added an experiment that examines the effect of model size by varying the hidden dimension of ViT models trained from scratch on the Caltech dataset:\n\n|        |   96  |  144  |  192  |  288  |  300  |\n|--------|:-----:|:-----:|:-----:|:-----:|:-----:|\n| ViT    | 58.20 | 59.87 | 60.50 | 61.13 | 55.35 |\n| ViT+QE | 59.35 | 60.83 | 61.39 | 61.57 | 55.53 |\n| Gain   | 1.15  | 0.96  | 0.89  | 0.44  | 0.18  |\n\nFrom the results, we observe two key trends:\n- (i) As the hidden dimension increases, both ViT and ViT+QE initially improve in accuracy, followed by a performance drop at the largest size due to overfitting.\n- (ii) The gain from QE decreases gradually with model size. This is an expected and reasonable trend, as larger models already possess greater capacity to model complex relationships. \n\n\n> #### Q1: Am I correct that the ViT architecture used in your experiments has a two-layer MLP with GELU as the nonlinearity, and does not use SwiGLU or GEGLU? If so, why did you choose this configuration, given that GELU-based MLPs are used in older models, and most modern architectures—including recent LLMs—use SwiGLU instead?\n\nYes, you are correct. ViTs used in our experiments employ GELUs. We chose this configuration to ensure fair comparison by strictly following the original settings from prior work [1], which used GELU-based MLPs. The same principle applies to our LLaMA fine-tuning experiments, where SwiGLU is part of the original architecture and thus remains unchanged.\nFor further discussion on gated MLP variants, please refer to our response to W1.\n\n> #### Q3: Could you clarify how exactly you combined your method with LoRA? If the base architecture remains unchanged (e.g., still uses a two-layer GELU MLP), how does the quadratic enhancement interact with the low-rank LoRA update? Did you modify the A @ B structure to insert a quadratic term between them?\n\n\nThe linear transformation in LoRA-based fine-tuning is expressed as: $y = (W_0 + AB)x$. We treat the combined matrix as a standard weight matrix $W=W_0+AB$, reducing it to a standard linear form $\\\\tilde{y}=Wx$. This aligns with the formulation in equation (1) of our manuscript, allowing us to apply QuadEnhancer directly on top of the LoRA-adapted weights without altering the LoRA structure itself. The final output after applying QuadEnhancer becomes: \n$\\\\begin{align}\\\\tilde{y}&=(W_0+AB)x\\\\\\\\z&=(\\\\Lambda \\\\tilde{y}) \\\\odot \\\\tilde{y}+\\\\tilde{y}\\\\end{align}$.\nThank you for pointing this out. We have added this clarification in the revised version of the manuscript. \n\n\n---\n\n\n[1] Mehta, Sachin, Farzad Abdolhosseini, and Mohammad Rastegari. \"Cvnets: High performance library for computer vision.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022.\n[2] Hu, Zhiqiang, et al. \"Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.\" arXiv preprint arXiv:2304.01933 (2023)."}, {"review_id": "gy2BkXbd2N", "reviewer": "Reviewer_rPUm", "summary": "Adds a single banded matrix of parametric terms to the traditional linear model of parameters for a neural network (that is kd in parameters where d is the number of inputs and k is small (16/32 for LoRA)). With this small increase in parameters it enables quadratic terms to be included on top of standard neural network linear models. Shows a series of experiments on different tasks and datasets showing that this is effective.", "questions": "1)\tEmpirical comparison of recent non-linear neural network approaches rather than just a linear baseline.\n2)\tCan QuadraNet V2 be simply dismissed by computational number of parameters? Is this accurate more broadly for the other papers stated?\n3)\tEmpirical FLOPS comparison.\n4)\tDoes the result hold up over broader training conditions? The experimental results have not clearly established this. Conditions shown are too narrow to be confident that the results are generally strong. Especially, does the result generalize for epochs, are they stable?", "limitations": "N/A", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1)\tClear presentation of approach in a directly implementable format of vector multiplication.\n2)\tWell-written, well-presented, clear and appears novel (see caveat in W2).\n3)\tApproach is forced towards cross terms, not squared terms for numeric stability.\n4)\tUsing VIT as baseline, on some smaller datasets and re-training LLMs, it shows solid gains given small increases in parameters sizes. This is consistent across a range of problems.\n\nWeaknesses:\n1)\tAlthough it is shown in the introduction that there are other attempts of quadratic based neural network models, including recently, the paper does not offer any experimental comparisons to these. Different architectures may present some challenges here, but this is not discussed. Comparison only include baselines (only VIT and GPT-2 variants), no competitor networks that have quadratic or any additional parametric approaches.\n2)\tThe claim in the introduction – on line 60 that QuadraNet V2 has substantial increases computation and parameters does not seem accurate. The QuadraNet V2 paper uses constant training time and shows small increases in number of parameters vs baselines in its experiments. (Maybe it has compute time increase, but the experiments offset this). Please comment. I may have mis-understood this. It raises a slightly broader question of whether this critique is correct of the other previous literature also.\n3)\tThere is no comparison in terms of fixed parameter numbers, or fixed training times. (Although the tables show large increases proportional to parameter increases).\n4)\tThe FLOPS comparisons are all theoretical (which are very nice numbers in these comparisons), however, the experimental FLOPS are not stated so we don't get to see if the theoretical model can be implemented in this efficient fashion.\n5)\tTraining details only follow a single model from another paper for image classification, and just 20 epochs pre-training, then 10 epochs fine tuning on text classification datasets. This seems a little narrow to understand whether these are sustained results with different numbers of epochs. Are the results robust?\n6)\tIt is slightly surprising that in Tab 8, the 16 rank outperforms 32 in the majority of cases for LLaMA{_,2}-7B. What is the cause of this?\n\nOverall the paper appears good, however, its not clear that the authors have sufficiently established its empirical strength over other approaches to increasing the parameterisation of the standard linear activation model.", "was_revised": true, "final_justification": "Thanks to the authors for a thorough rebuttal. It answers all the points that I raised.\n\nI support this paper.", "rebuttal": "> #### W1&Q1: Although it is shown in the introduction that there are other attempts of quadratic based neural network models, including recently, the paper does not offer any experimental comparisons to these. Different architectures may present some challenges here, but this is not discussed. Comparison only include baselines (only ViT and GPT-2 variants), no competitor networks that have quadratic or any additional parametric approaches. \n\nThank you for your valuale comment. We have added comparisons on image classification with QuadraNets, where the quadratic term is expressed as $y = (x @ W_1)*(x@W_2) + x@W_3$. To ensure fairness, we carefully matched the number of trainable parameters between QuadraNet (QN) and our QuadEnhancer(QE) by adjusting the size of hidden dimensions. The accuracy across several standard benchmarks is summarized below and show that QE significantly outperforms QN in general:\n\n\n|            | params(M) | imagenet1k | caltech | cifar10 | cifar100 | flowers |  food  |   pet  |   avg  |\n|:----------:|:---------:|:----------:|:-------:|:-------:|:--------:|:-------:|:------:|:------:|:------:|\n| ViT-M + QN |   5.47    |   72.15    |  92.14  |  97.79  |  85.36   |  75.39  | 86.83  | 94.05  | 86.24  |\n| ViT-M + QE |   5.40    |   75.15    |  94.03  |  98.03  |  86.88   |  87.25  | 88.59  | 94.95  | 89.27  |\n| ViT-T + QN |   2.53    |   61.17    |  85.36  |  95.81  |  79.08   |  34.50  | 81.58  | 98.31  | 76.54  |\n| ViT-T + QE |   2.47    |   65.30    |  90.32  |  97.09  |  82.59   |  75.58  | 84.63  | 91.88  | 83.91  |\n\nThank you for your comment again, we have added these comparisons to the revised manuscript.\n\n> #### W2&Q2: The claim in the introduction – on line 60 that QuadraNet V2 has substantial increases computation and parameters does not seem accurate. The QuadraNet V2 paper uses constant training time and shows small increases in number of parameters vs baselines in its experiments. (Maybe it has compute time increase, but the experiments offset this). Please comment. I may have mis-understood this. It raises a slightly broader question of whether this critique is correct of the other previous literature also.\n\nThank you for your thoughtful observation, and we apologize for the confusion caused by our earlier wording. Our intent was not to suggest that QuadraNets and other work specifically results in prohibitively large training time or parameter counts than linear NNs, but rather to highlight that the parameter and computational complexity of the quadratic components in prior quadratic models is generally higher than that of QuadEnhancer. For example, models in [3,4] typically involve quadratic terms with $O(n^2)$ parameters and computation (some even up to $O(2n^2)$ [1,2,5]), while QuadEnhancer introduces only $O(n)$ parameters and computational overhead for the quadratic term (with $k=1$). \n\nWe appreciate you raising this point and have added this clarification in the revised the manuscrip.\n\n> #### W3: There is no comparison in terms of fixed parameter numbers, or fixed training times. (Although the tables show large increases proportional to parameter increases).\n\nThank you for your insightful comment. We understand the importance of comparisons with fixed parameter counts or fixed training times. To address this:\n1. Parameter Comparison: The additional number of parameters introduced by the quadratic models is relatively small compared to their linear counterparts. For example, in the text classification task, we observe the parameter counts are 2.47M vs. 2.45M, 2.83M vs. 2.82M, and 5.40M vs. 5.37M. Given the small difference in parameters, we did not perform a comparison based strictly on identical parameter numbers, as the changes are minimal.\n2. Training Time Comparison: To provide further insight into the trade-offs, we have reported training times for two tasks: pretraining on ImageNet-1k and fine-tuning on CIFAR100. The results show that the models with QuadEnhancer (ViT+QE) generally outperform their linear counterparts (ViT) in the later stages of training. Below are the results:\n\nPretrainign on ImageNet-1k\n|time(sec.)|1k|10k|20k|30k|40k|50k|\n|-|-|-|-|-|-|-|\n|ViT|**22.28**|**52.09**|58.35|**63.51**|65.96|65.97|\n|ViT+QE|11.30|51.28|**59.07**|63.15|**66.59**|**67.04**|\n\nFinetuning on Cifar100\n|time(sec.)|500|1k|2k|3k|4k|5k|6k|\n|-|-|-|-|-|-|-|-|\n|ViT|78.79|**81.78**|83.0|84.21|85.29|85.29|85.29|\n|ViT+QE|**79.33**|81.07|**83.43**|**84.49**|**85.48**|**86.15**|**86.53**|\n\n\n> #### W4&Q3: The FLOPS comparisons are all theoretical (which are very nice numbers in these comparisons), however, the experimental FLOPS are not stated so we don't get to see if the theoretical model can be implemented in this efficient fashion.\n\n\nThank you for your comment.  We measured the actual number of FLOPs using NVIDIA's Nsight Compute. We recorded the executed FLOPs for both a standard linear layer and a quadratic-enhanced (QE) layer, each with hidden dimension n=192, averaged over 64 runs. The results are:\n\n| Linear |   QE  |\n|--------|:-----:|\n| 93888  | 94848 |\n\nThis indicates that QE introduces only a ~1.0% increase in actual FLOPs. The theoretical increase is approximately 1/192=0.5%, so the gap is small and reasonable. We note that our current implementation does not yet leverage low-level CUDA or Triton optimization, which could further close this gap.\n\n\n\n> #### W5&Q4:  Training details only follow a single model from another paper for image classification, and just 20 epochs pre-training, then 10 epochs fine tuning on text classification datasets. This seems a little narrow to understand whether these are sustained results with different numbers of epochs. Are the results robust?\n\nThank you for your comment. We clarify below:\n\n1. For image classification, the hyper-parameters followed [7]. The authors of [7] have made efforts to tune the hyper-parameters, thus we didn't change their settings. The training runs for hundreds of epochs (300), and the results are stable in the later stages, indicating robustness.\n\n2. For text classification, we used 20 epochs because the model's perplexity had already converged by then. As shown below, further training beyond 16 epochs yields minimal improvement:\n\n|             | 1        |   4   |   8   |   12  |   16  |   20  |\n|-------------|----------|:-----:|:-----:|:-----:|:-----:|:-----:|\n| GPT-2/16    | 34.47    | 5.58  | 5.16  | 5.00  | 4.90  | 4.90  |\n| GPT-2/16+QE | 6.42     | 5.31  | 5.00  | 4.90  | 4.81  | 4.81  |\n| GPT-2/32    | 6.42     | 5.10  | 4.76  | 4.62  | 4.57  | 4.57  |\n| GPT-2/32+QE | 5.64     | 4.85  | 4.62  | 4.48  | 4.44  | 4.44  |\n\nAnd the finetuning of text classifiction is relatively simpler than the pretraining, thus we used 10 epoches, also yielding a similar coverged results at late training stage. \n\n\n> #### W6: It is slightly surprising that in Tab 8, the 16 rank outperforms 32 in the majority of cases for LLaMA{_,2}-7B. What is the cause of this?\n\nThank you for your question. The performance drop is due to overfitting. Acctually, this phenomenon also occured in the baseline [6] when the rank was increased to 64. QuadEnhancer, by enhancing the model's capacity, causes overfitting to occur earlier, rather than worsening it. \n\n\n----\n[1] Xu, Chenhui, et al. \"Quadranet: Improving high-order neural interaction efficiency with hardware-aware quadratic neural networks.\" 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 2024.\n\n[2] Xu, Chenhui, et al. \"Quadranet v2: Efficient and sustainable training of high-order neural networks with quadratic adaptation.\" arXiv preprint arXiv:2405.03192 (2024).\n\n[3] Chrysos, Grigorios G., et al. \"Deep polynomial neural networks.\" IEEE transactions on pattern analysis and machine intelligence 44.8 (2021): 4021-4034.\n\n[4] Fan, Feng-Lei, et al. \"On expressivity and trainability of quadratic networks.\" IEEE Transactions on Neural Networks and Learning Systems (2023).\n\n[5] Fan, Feng-Lei, et al. \"One neuron saved is one neuron earned: On parametric efficiency of quadratic networks.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2025).\n\n[6] Hu, Zhiqiang, et al. \"Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.\" arXiv preprint arXiv:2304.01933 (2023).\n\n[7] Mehta, Sachin, Farzad Abdolhosseini, and Mohammad Rastegari. \"Cvnets: High performance library for computer vision.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022."}, {"review_id": "RQ1Efa8BHZ", "reviewer": "Reviewer_yMZu", "summary": "The paper proposes QuadEnhancer, a lightweight module designed to enhance non-linearity through quadratic transformation while maintaining low computational complexity. This is achieved via a modular design that can be applied to any layer in a neural network. To reduce the complexity typically associated with quadratic transformations, the authors introduce three key techniques: (1) Low-rank matrices for the quadratic terms, (2) Weight sharing, and (3) Sparsification of the weight matrix. For low-rankness, each weight matrix in the quadratic term is decomposed into the outer product of two vectors, p and q, effectively constraining it to rank 1. To further reduce the number of parameters, the weight matrix W used in the linear term is also reused in the quadratic term's P and Q matrices. Lastly, the matrix Λ, which complements W to form P, is sparsified as a band matrix, maintaining expressiveness while further reducing parameter count. By integrating QuadEnhancer into various models, the authors demonstrate consistent and efficient performance improvements across a range of tasks.", "questions": "Please refer to the Weaknesses part.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- The paper is clearly written and easy to follow.\n- The proposed method is well-formulated with a modular structure, allowing it to be easily integrated into various architectures.\n- The method is evaluated across diverse tasks and backbone models, demonstrating its generality and practical utility.\n\n**Weaknesses**\n- While the paper presents several tricks to reduce complexity, it would be more convincing to include a complexity–accuracy trade-off graph, under settings with smaller models or limited data.\n- The experiment could be strengthened by providing a more explicit comparison with prior work in terms of both performance and computational cost.", "was_revised": true, "final_justification": "The proposed method is clearly described, and the modular design is well structured. However, my concerns were not fully addressed during the rebuttal period. After reading the other reviewers' comments and the authors' response, I noticed a few points that remain unresolved. \n\nThere is no empirical comparison with gated MLP variants like GEGLU or SwiGLU, which are widely used and relevant. While I understand that a fully equivalent comparison may not be possible, including them as baselines or providing ablation results would make the evaluation more robust. \n\nAdditionally, the comparison with prior quadratic models seems somewhat overgeneralized, possibly due to the lack of a related work section discussing how other approaches have addressed efficiency. Including such context would strengthen the paper. \n\nFor these reasons, I adjust my score to 4.", "rebuttal": "Thank you for your kind support and valuable suggestions. Some of the additional experiments are still in progress and could not be completed within the rebuttal period. We will update the results and provide clarification in the discussion stage. We apologize for any inconvenience this may cause."}, {"review_id": "2qEEDp6lBY", "reviewer": "Reviewer_fG69", "summary": "This paper introduces \"QuadEnhancer,\" a lightweight module designed to enhance the representational power of deep neural networks by introducing quadratic feature interactions. The core idea is to augment standard linear transformations with a quadratic term. To make this computationally feasible and parameter-efficient, the authors propose a series of design choices: (1) reuse the output of the linear transformation to form the quadratic term, avoiding redundant computations; (2) parameterize the quadratic interactions via a new matrix; and (3) make the matrix extremely sparse, modeling it as a band matrix that can be implemented with a few efficient circular shift operations. In final implementation, this adds only d parameters (where d is the feature dimension) and negligible FLOPs per layer. The authors demonstrate the effectiveness of QuadEnhancer through experiments on image classification (ViT), text classification (GPT-2), and parameter-efficient fine-tuning of large language models (LLaMA and LoRA), showing performance gains across tasks and model scales.", "questions": "Please see weaknesses described above.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe proposed QuadEnhancer is a simple, general, and seemingly effective method. It acts as a \"drop-in\" replacement for standard linear layers and is applicable to a wide range of architectures (demonstrated on Transformers) and modalities (vision and language). The consistent performance improvements with minimal overhead suggest it could become a standard component in future model design, much like novel activation functions or normalization layers.\n\nThe paper's claims are backed by a convincing set of experiments. The authors evaluate their method on three distinct, important tasks (image classification, autoregressive language modeling, and LLM fine-tuning). The choice of strong, modern baselines (ViT, GPT-2, LLaMA) and standard datasets makes the results credible.\n\nThe paper is well-written and easy to follow.\n\nWeaknesses:\n\nThe choice of K={1} is well-motivated for efficiency, but its impact relative to other choices (e.g., K={-1, 1}) is not explored. An ablation on K would provide a clearer picture of the trade-off between model complexity and performance.\n\nThe decision to exclude the squared term (r=0) is justified by potential FP16 instability. However, a direct empirical comparison (e.g., training with and without this term in both FP16 and FP32) is needed to validate this claim and quantify its effect on both stability and final accuracy. In addition, recently BF16 becomes more popular and is believed to have better stability than FP16.\n\nThe paper is heavily empirical. While the motivation is sound and the results are good, the paper would benefit from a deeper discussion on why these specific local quadratic interactions are effective. For example, a theoretical analysis of how this enhancement affects the function class that the network can approximate would provide more fundamental understanding and elevate the work.", "was_revised": true, "final_justification": "My final justification stays the same as above original review.", "rebuttal": "Thank you for your kind support and valuable suggestions. Some of the additional experiments are still in progress and could not be completed within the rebuttal period. We will update the results and provide clarification in the discussion stage. We apologize for any inconvenience this may cause."}]

Author rebuttal:

Meta-review: On this work, the authors introduce QuadEnhancer, a novel idea that uses quadratic transformations in neural networks to further increase the nonlinearity of the model. To do so, the authors introduce several technical innovations: Low-rank matrices for the quadratic terms, weight sharing, and sparsification of the weight matrix, with the idea being not only improving the performance but keeping the flops of the model not explode.

The paper initially received mixed scores: 2 Borderline Rejects, 1 Borderline Accept and 1 Accept. The reviewers were pretty much unanimous that the paper is well-written and makes sense, and that the results show improvement in several benchmarks. However, there were some issues with the experimental setup (the method is neither SOTA, nor used in particularly large networks) and very importantly some conceptual issues (gated activations can be considered as quadratic transformations). The authors provided a rebuttal showing more experiments and explanations to the reviewers.

After the rebuttal, two of the reviewers kept their scores (Borderline Reject and Borderline Accept), 1 reviewer increased their score from Borderline Reject to Borderline Accept, and one reviewer decreased their score from Accept to Borderline Accept, effectively having a stalemate of the scores. Thus, I carefully read the paper, the rebuttal, the reviewers-authors exchange and the internal discussion. Afterwards, summarizing my thoughts:

- The idea of the paper makes sense, and the authors have done a decent job in keeping the number of parameters and flops from exploding.

- The paper is very well-written, it's hypothesis are supported by a good experimental setup, and the paper provides code which is appreciated.

- Unfortunately, I remain unconvinced when it comes to how the method would work in complex tasks of really large neural networks, especially when combined with gated non-linearities which somehow, albeit orthogonally, tackle the same problem.

The AC and the SAC had a discussion to reach a decision for the paper. They agreed that the paper has merits, with the idea being intuitive and well-explained and the results somehow validating the idea. While they are skeptical if the method would work on larger models, on balance of things, they agree that the paper is more on the accept regime and recommend it to get accepted. Congratulations to the authors and please ensure to integrate all the results from the rebuttal in the paper!
Decision: Accept (poster)