Dendritic Integration Inspired Artificial Neural Networks Capture Data Correlation
Authors: Chongming Liu
Jingyang Ma
Songting Li
Douglas Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: brain-inspired models
dendritic bilinear integration rule
quadratic neuron
deep convolutional neural networks
Primary area: deep_learning_architectures

Abstract: Incorporating biological neuronal properties into Artificial Neural Networks (ANNs) to enhance computational capabilities is under active investigation in the field of deep learning. Inspired by recent findings indicating that dendrites adhere to quadratic integration rule for synaptic inputs, this study explores the computational benefits of quadratic neurons. We theoretically demonstrate that quadratic neurons inherently capture correlation within structured data, a feature that grants them superior generalization abilities over traditional neurons. This is substantiated by few-shot learning experiments. Furthermore, we integrate the quadratic rule into Convolutional Neural Networks (CNNs) using a biologically plausible approach, resulting in innovative architectures—Dendritic integration inspired CNNs (Dit-CNNs). Our Dit-CNNs compete favorably with state-of-the-art models across multiple classification benchmarks, e.g., ImageNet-1K, while retaining the simplicity and efficiency of traditional CNNs. All source code are available at https://github.com/liuchongming1999/Dendritic-integration-inspired-CNN-NeurIPS-2024.

Dendritic Integration Inspired Artificial Neural Networks Capture Data Correlation
Chongming Liu*b, Jingyang Ma*h, Songting Liabe., and Douglas Zhou
"School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, China Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, China "Ministry of Education Key Laboratory of Scientific and Engineering Computing, Shanghai Jiao Tong University, Shanghai, China Shanghaii Frontier Science Center of Modern Analysis, Shanghai Jiao Tong University, Shanghai, China
Abstract
Incorporating biological neuronal properties into Artificial Neural Networks (ANNs) to enhance computational capabilities is under active investigation in the field of deep learning. Inspired by recent findings indicating that dendrites adhere to quadratic integration rule for synaptic inputs, this study explores the computational benefits of quadratic neurons. We theoretically demonstrate that quadratic neurons inherently capture correlation within structured data, fea- ture that grants them superior generalization abilities over traditional neurons. This is substantiated by few-shot learning experiments. Furthermore, inte- grate the quadratic rule into Convolutional Neural Networks (CNNs) using a bi- ologically plausible approach, resulting in innovative integration inspired CNNs (Dit-CNNs). Our Dit-CNNs compete favorably with state-of-the-art models across multiple classification benchmarks, e.g., ImageNet-IK, while retaining the simplicity and efficiency of traditional CNNs. All source code are available at httts::/gii http:///ttttb.ccoo//u////onnnnn/ comm//1uccchonnmmnng99999/ Dendritic- inw.iiciintraatiin inspinedd CNN IPS
-ereuttrres-Dnddittiii
Introduction
While the Artificial Neural Network (ANN) framework has made significant advancements towards solving complex tasks, it still faces problems that are rudimentary to real brains [6]. A notable distinction between the modern ANN framework and the human brain is that the former relies on significant number of training samples, which consumes large amounts of energy, whereas the latter runs on extremely low power (<20 watts) and possesses a strong generalization capability based on few-shot learning. Studies have demonstrated that incorporating dendritic features in ANNs can alleviate these issues and enhance overall performance
However, it difficult to quantify the nonlinear integration of dendrites, which is an essential property that allows individual neurons to perform complex computations 44 As a result, dendritic-ing dendritic-inspired models often employ linear integration with nonlinear activation functions such as ReLU and Sigmoid (7
52
38 30
In light of recent studies revealing that the somatic responses of biological neurons multiple synaptic inputs on dendrites follow a quadratic integration rule [15 28 we explore the computational benefit of the quadratic neuron model 4T This model substitutes the linear integration and nonlinear
Corresponding author, songting @ www...eduuuen.... 7dz @ww@@/jjjju......
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

activation function of traditional point neurons with quadratic integration as follows
f(x) x f(x) - xJ Ar w x+
(I)
The expected value of quadratic term EE>D B [ closely relates to the covariance between variables Y; and žj (z; and 2; are the i-th and the j-th components of vector I, respectively), suggesting an inherent capability for capturing input correlation. This attribute allows biological neurons to exhibit remarkable performance in correlation detection tasks [ 29]. Here, we theo- retically demonstrate that quadratic neurons indeed naturally capture correlation within structured data, which is pivotal in numerous machine learning applications 77 such as language generation [5] and video understanding [23] This intrinsic property endows quadratic neurons with superior generalization capabilities compared to traditional neurons, as evidenced by few-shot learning ex- periments. We further propose a biologically plausible method to integrate them into Convolutional Neural Networks (CNNs). Applying this approach to ResNet [6] and ConvNeXt [34] results in novel CNN
integration inspired ResNet (Dit-ResNet) and Dendritic integration inspired ConvNeXt Xt Pojtt-ConNNett, respectively. Evaluations on CIFAR- 10 and CIFAR- 100 datasets demonstrate that Dit-Res.Nets significantly enhance test accuracy by merely replacing a single layer of point neurons with quadratic neurons. On the ImageNet- IK dataset [TO], Dit-Conv NeXt demonstrate significant enhancements in top-1 accuracy with only a one percent increase in parameters, exhibiting competitive performance compared to state-of-the-an models. Ablation studies further substantiate the effectiveness of quadratic neurons and their capability for capturing data correlation.
This paper is structured as follows: Section 2 reviews previous works related to
models and high-order interactive operations in neural networks. Section 3 presents a theoretical analysis of the computational benefits derived from quadratic neurons, as demonstrated through few-shot learning experiments. Section 4 describes the architecture of Dit-CNNs and evaluates their performance on several computer vision benchmark datasets. Section 5 concludes the paper. All numerical experiments in this paper are conducted using Python and executed on 8 Tesla A100 computing cards with a 7nm GA 100 GPU, featuring 6,912 CUDA cores and 432 tensor cores.
2 Related work
Dendritic-inspired computational models. The significant role of dendrites in the nonlinear
integration within neural systems is well-recognized, enabling neurons to perform intricate tasks
43
49 2]. Recently, numerous studies have explored the implementation of dendritic features in
ANNs from different aspects, yielding encouraging results. In [2 21 37,, the integration of dendritic morphology into ANNs has led to higher test accuracies
simple tasks compared to traditional ANNs. Furthermore, dendritic plasticity rules have been employed to develop learning algorithms in
38 36
effectively replacing the non-biological backpropagation algorithm and achieving enhanced
performance on classification tasks with small data sizes. Additionally, in (30], dendritic nonlinearity considered by adding input to each layer with the Hadamard product rule, which has shown improvements in approximation and classification experiments.
Neural networks with high-order interactions. Recent advancements in the design of deep learning architectures have been predominantly driven by the ability capture high-order statistics among imputs and features. Over recent decades, there has been pivotal shift in the foundational neural networks used for computer vision tasks: moving from comvolution-basse architectures [16] 24 [66 to Transformer-bast
models [T] 33 9. This transition has been driven by the development of the self-attention mechanism, which facilitates quadratic interactions among inputs. Additionally, a novel category of neural networks, referred to as Mamba [4], has demonstrated exceptional performance in tasks that require the processing of extensive high-order statistical information, such as video understanding [32 [9 27). The backbone of Mamba's 's architecture is a state space model that enables high-order interactions among inputs. Moreover, the integration of high-order spatial interactions into CNNs has also been shown to enhance performance significantly [40]. These developments underscore the significance of high-order statistical information in tackling large-scall scale complex tasks and suggest that incorporating high-order interactions as a foundational aspect of model design an effective strategy.
A can always be considered symmetric matrix, attributable to the quadratic form present in f(x).

---

Table 1: The overview of current works with quadratic format.
How quadratic
Biological interpretation
Theory for generalization
Reference
Quadratic format
operation is used
f() (waz)
Pixel-Wise
N
N
48
20
35
60
f()) x² Ar.
Pixel-Wise
N
N
Dit Dit-CNNs
f(x) x² Ax
Channel -ise
Y
Y
Previous works have explored the concept of quadratic integration as a sophisticated alternative to traditional linear summation. Table summarizes various neuron models formulated in quadratic formats, along with their relevant references. The one-rank format quadratic neurons, when applied
across entire networks, is documented in [54
12
4 53], demonstrating enhanced performance
[60] have implemented quadratic neurons on pixel-wiss basis within convolution layers, yielding modest improvements in test accuracy. In contrast, Table iillstraates how our Dit-CNNs approach departs from these models by integrating channel-wise quadratic operations with biological interpretation and providing theoretical analysis of the computational advantages offered by quadratic neurons.
in classification tasks. Studies such as [48] 20, 35
3 Quadratic neurons possess enhanced generalization capabilities
This section provides a theoretical analysis demonstrating how quadratic neurons enhance generaliza- tion capabilities by effectively capturing correlation within structured data. Additional numerical experiments corroborate this assertion. The detailed proofs can be found in Appendix
3.1 Binary classification for normal distributions
Assume that the data points of two classes are equally sampled from two different non-degenerate normal distributions: classı N(1, Σ1), classy N (H2, E2), where H4, RR Σg Rddd (j
1,.). Then the optimal classifier Yopt (x) Rd + {1, 2} can be defined according the sampling probability:
Yopt (I) arg max (x)),
(1,1,
where (I:) is the probability (density function) of sampling point x from distribution N(uj, Ey).
If a single quadratic neuron, as described in Equation (1 (z) xT Ar + w x + b, where A is symmetric matrix), is used to solve the above binary classification task, we can prove the following theorems:
Theorem 1. (Existence) The critical points with respect to the cross-entropy loss L(A, w. b) are given as follows:
Σ1 Σg
A' Eng w² -2(E EnyHa) M2), b² = 1
12 In' u22 + log
i.e.
0L
ØL
ØL
0A b
Ow 1n an, b-=
bb | ---===b-=-
where
L(A, w. b) = Exwelaser (log(1 + c((o))) + Exwelases (1og(1+ te- =-(o)))]]
Moreover, the corresponding classifier generated by this formula the same as the theoretically optimal classifier in Equation (2).
Theorem 2. (Uniqueness) Consider the conditional cross-entropy loss defined on a set S E M(R.).
where M(Rd) denotes the Lebesgue o-algebra on gdd
L(A, w. b () = Eaanotassi (log(1 + ed(x))) + ExwelnsszzzER (log(1 + (
3

---

where
Egwcllssy (g(x)) = -So g(x)p,(z)
then the unique solution satisfying aL(A, w,b (
1A- 0, (L w. (2))
(I, w. (2)
b-= 0,
1A- 100 _8-
0
0A
Ow
Öb
for every Ω E MRR)) is given by:
E1 |Σn]
A" 21
2ig w²= -2(2 p1 "1²2)) p2), 6" -1 p1
u2
T 2ig 122 + log
Quadratic
Two-layer MLP
T
0
Quadratic
Two-layer MLP Uraznin
trainin samples 1010
bareyy sumpley 1920
200 aumerical
213 maseriaa
temmriaa
NN Cas
Task Class, 1. (g)) 0 class N
teaoreicca
theeettbal
Task 2: Class, -N (lo
Diasin soreees 03300
000000 STTOES 4000)
triiking samples 9920
Baaakigg Strepler 6038
theeretica razeiicl
theeretica
naswicaa
maneriial
Figure Visualization of classifier boundaries for two models (a single quadratic neuron with 2-dimensional inputs and a two-layer MLP comprising 2-ReLU(10)-1) across two distinct tasks. For each task, the impact of varying training sample sizes is examined. Model comparisons performed under uniform conditions, utilizing identical random seeds and hyperparameters for fairness. Training is executed using gradient descent with a learning rate of 0.1 over 10,000 epochs. The term "theoretical" denotes the optimal classifier's boundary specified in Equation (2). while "numerical" represents the empirical classification boundary obtained from the model.
Theorem T iidettfiess an analytical critical point for a quadratic neuron with cross-entropy loss, which emerges the optimal classifier. On the other hand, Theorem 2 establishes the uniqueness of this critical point under conditional cross-entropy loss, where Ω denotes the regions occupied by distinct batches of data points. Under these conditions, the quadratic parameters will converge this unique critical point when using the stochastic gradient descent (SGD) algorithm. is demonstrated that, unlike traditional neuron, quadratic neuron inherently possesses the ability converge the optimal classification solution by directly including the covariance matrix. This finding is supported by simulated data in Task 1 (identical covariance) and Task 2 (non-identical covariance) shown in Figure [ which indicates that quadratic neuron consistently achieves the theoretically optimal classification outcome. Moreover, in comparison to a multi-layer perceptron (MLP) with two layers, quadratic neuron surpass traditional neuron in Task 2 which require capturing correlation information from data distributions (require fewer training samples to converge to the optimal solution). This highlights the unique capability of quadratic neurons to identify internal correlations within structured data, offering possible explanation for their superior generalization capability compared traditional neurons, a topic that will be further explored in numerical experiments below.

---

3.2 Few-shot learning experiments on MNIST and Arabic MNIST
From the above classification tasks for normal distributions as depicted in Figure it suggested that quadratic neurons may require fewer training samples in comparison to traditional neurons. We next examine this few-shot learning capability of quadratic neurons on the MNIST [2] and Arabic MNIST datasets [9]] The results, presented in Figure 2 indicate that quadratic neurons outperform MLP with two layers when trained with limited number of samples on both datasets. This evidence shows the enhanced generalization ability of quadratic neurons in applications. Further insights into the mechanism through which quadratic neurons capture data correlation in the MNIST dataset, and related theorem concerning multi-class classification for normal distributions, are detailed in Appendix B
MNIST
Arabic MNIST
98
95
Test accuracy
Test accuracy
90
94
92
85-
90
80
75-
86
Quadratic Neuron
Quadratic Neuron
Two-Layer MLP
70-
Two-Layer MLP
84
0.05
0.01
0.005
0.05 0.005 0.001 Sample size
Sample size
Figure 2: Performance of two models on few-shot learning tasks with MNIST and Arabic MNIST datasets. The first model consists of 10 quadratic neurons with 784-dimensional input, while the second model is a MLP with the configuration of 784-ReLUUs8000 10. Both models are evaluated under identical conditions using the same training protocol, which includes SGD with learning rate of 0.1 and batch size of 100 across 20 epochs. The term 'Sample size' refers to the ratio of the number of training samples to the full training set. Experiments are conducted for each model and sample ratio across ten runs, and the resulting test accuracy is depicted through an error bar plot (error bar represents the upper and lower bound for test accuracy).
4 Integrating quadratic neurons into CNNs with biological plausibility
Retinal ganglion cell
X2
W2
X3
Cone cells
Figure 3: Schematic of the biological interpretation of Dit-CNNs. Dit-CNNs is inspired by neural networks in the visual system. For example, different types of cone cells encode various color (channel) information, and retinal ganglion cells receive inputs from multiple types of cone cells [22), the responses can be modeled as having receptive fields (convolutional kernels) related to different color channels (w) # 11, W2 22, W3 * I3).. When multiple channel inputs are present, traditional CNNs simply linearly sum the corresponding responses. In contrast, dendritic neurons integrate these inputs with an additional quadratic term based on the dendritic quadratic integration rule. This approach leads to the formulation of Dit- Bit-CNNs after simplification.
5

---

The concept of convolution [1]. inspired by the discovery of the receptive field in the cat visual cortex [T8]. laid the foundation for CNNs [26] which have been designed efficiently tackle large-scale vision tasks. The underlying biological interpretation of convolution is that each neuron exhibits varying responses at different image locations based on its receptive field (convolution kernel). Traditionally, CNNs have focused solely on the linear integration of inputs from presynaptic neurons. Specifically, for an input X E RC X Wen, where Cin represents the number of imput layer neurons (channels), and Hn and Win denote the height and width of the input feature, respectively. Given convolution kernels w E IR XCout x (2(+1))) (2/+1) the convolution output, Y Come(X) RCvut x Hout x Wout, represents the response of neuron at location [ī, ki] to inputs from locations [j 1:::: 1, 1: + across all input layer neurons:
Cin
Σ m=1
(,;m,,;;11
X (mm, j + 1
However, acknowledging that neurons with dendrites obey quadratic integration rule, propose novel approach that incorporates quadratic neurons into CNNs as depicted in Figure This adaptation incorporates quadratic integration term among inputs from different neurons (channels). Specifically, with a quadratic integration coefficient Cos XC XCin, the output neuron's ' response in Equation (3) is modified follows:
Cin
Σ m=1
wfm, i, X[m,j
-1: + X" [. AjA 1. []
Quadratic integration
4.1 Evaluations on CIFAR-10 and CIFAR-100
Dataset and models. Our initial experiments utilize the CIFAR dataset, which includes 50,000 training images and 10,000 testing images. The experimental setup follows the ResNet configurations as outlined in [16], comprising models such as ResNet-20, ResNet-32, ResNet-56 and ResNet- 10. To reduce computational demands, we incorporate quadratic neurons specifically into one layer-namehl namely, the second layer of the first block in the second stage of the ResNet architecture [16]. Additionally, to address issues related to gradient dynamics, such as gradient explosion, a Layer Normalization (LN) layer 3 is implemented immediately before the layer equipped with quadratic neurons.
Table 2: Comparative performance of Dit-ResNets and structurally similar models on CIFAR.
Param. (CIFAR10)
Acc. (CIFAR10)
# Param.
Acc. (CIFAR100)
Model
(CIFAR 10FAR100)
ResNet-20(16)
0.27M
91.25%
0.30M
67.26+00.68%
22
0.81M
92.22%
0.84M
67.82+t0.525%
QuadralessNet-225554
0.81M
92.21 %
0.84M
68.02++0.44%
Dit-ResNet-20
0.30M
92.66 %
0.33M
68.66 %
ResNet-32[16]
0.46M
92.49%
0.49M
68.52:00.5%%
QResNet-32[12
1.39M
93.10%
1.42M
69.41:10.489
QuadraResNet-3226 54
1.39M
93.11 %
1,42M
69.54-10.445
Dit-ResNet-32
0.49M
93.17% %
0.52M
69.68 0.32
ResNet-56(16]
0.86M
93.03%
0.89M
70.17++0.679
2.55M
93.66%
2.58M
71.21 +0044004499
QuadraResNet-56(54
2.55M
93.79%
2.58M
70.98 76%
Dit-ResNet-56
0.89M
93.90 %
0.92M
71.40
ResNet-111(16)
1,73M
93.57%
1.76M
70.84:10.769%
1R0
5.17M
93.88%
5.20M
71.58+10.87%,
QuadraResNet-110f5
5.17M
93.77%
5.20M
71.72-10.81%
Dit- Dit-ResNet-110
1.76M
94.33% %
1.79M
72.40-10.85%
Experimental settings. The training process incorporates SGD with weight decay of 0.0001, batch size of 128, and a momentum 0.9. Following the recommendations from [12 quadratic

---

integration parameters are initialized to zero, and distinct learning rate is adopted for these parame- ters. Initially, the learning rate for the quadratic integration matrix set at 1, and 0.1 for all other parameters. These rates is reduced by factor of ten at 80 and 120 epochs, with training concluding 160 epochs. Data augmentation procedures mirror the original protocol: each image is padded by 4 pixels on each side, follows by random cropping of 32 x 32 section from the padded image or its horizontal flip. For testing, single view of the original 32 x 32 image evaluated.
Results. Table 2 presents comparative analysis of CIFAR performance between Net-ResNets and the original ResNet model [16], as well as two adaptations that incorporate quadratic neurons which have shown superior outcomes among existing approaches with quadratic formats 4]] For the CIFAR-I0 dataset, given the extensive benchmarking in prior studies, we conduct runs of our Dit-ResNets, reporting the optimal outcome for comparative analysis. For CIFAR-100, we independently execute all models, documenting average test accuracy and variance (tea Our experiments indicate significant performance enhancements on both CIFAR- 10 and CIFAR-100 datasets after integrating quadratic neurons into a single layer of ResNet, confirming their efficacy. Furthermore, when comparing deeper ResNet model with our Dit-ResNe (e.g.. ResNet-1 10 vs. Dit-ResNet-56), our approach not only boosts performance but also reduces training overhead. This suggests that augmenting the model with quadratic integration parameters more effective than merely increasing network depth- traditionally favored strategy. In comparison to other models employing quadratic neurons 12
12
54]
48
20
35
60
54 our Dit-ResNets consistently achieve the highest test accuracy with the fewest parameters. This underscores the efficacy of our approach in leveraging the computational advantages of quadratic neurons. These neurons, as discussed in Section 3, demonstrate enhanced generalization capabilities.
CIFAR-B5 10
ImageNet-1e -K
Dit-ResNett
DR-ConvWett DeepMAD ConvNNXt
94,0
QReeNett
93.5
ReaNet
Test accuracy
83.5
83.0
7est
82.5
82.0
20
32
$6
110
Network depth
Network size
Figure Visualization of some performance results presented in Table 2 (left) and Table 3 (right).
4.2 Evaluations on ImageNet-1K
Dataset and settings. We extend our investigations to the ImageNet- 1K dataset 10]. which comprises 1.28 million training images and 50,000 validation images across 1,000 categories. Our analysis primarily focuses on top-1 accuracy on the validation set. Training conducted at resolution of 224 x 224, supplemented by a comprehensive suite of data augmentation and regularization strategies. These include RandAugment X mixup [57] cutmix (55), label smoothing (44), layer
scale [7),, random erasing [58), exponential moving average (EMA) (39), and stochastic depth [17] These techniques are inspired by the timm library (50) and methodologies from Touvron al. [45]
Detailed descriptions of these hyperparameters are provided in Appendix
Model description. Mirroring our approach with the CIFAR dataset, we integrate quadratic neu- into single layer of the ConvNeXt model 34 to conserve computational resources. This adaptation results the creation of three distinct models: Dit-ConvNeet--1 Dit-ConwNeXt-S, and Dit-ConWNeXi-B. Specific details about the layer replacement are discussed in the ablation study.
Results. Table presents comparative analysis of Dit- Dit-ConvNNeXts against state-of-the-art models from the existing literature. Compared the original ConvNeXt counterparts, our Dit-ConvNeXts exhibit an average increase of 0.5% in I accuracy, with only marginal average increase of 1% rise
7

---

Table 3: Dit-ConvNeXts versus state-of-the-art (SOTA) models on ImageNet-IK, -K All models listed the table are trained and validated resolution 224 224.
Arch.
Model
# Param.
FLOPs
Top-1 1 acc. (%)
Swin-T
29M
4.5G
81.3
Transformers
DeïT-S
46
22M
4.6G
79.8
VMamba-T 32
22M
5,6G
82.2
State Space Models
VideoMamba-S
26M
4.3G
81.2
ResNer-50 [51
26M
4.1G
80.4
SLaK-T 31
30M
5.0G
82.5
QuadraNet36-T 53
24M
4.1G
82.2
CNNs
DeepMAD-29M
42
29M
4.5G
82.5
ConvNeXt-T
84
29M
4.5G
82.1
Dit-ConvNeXt-I
29M
5.0G
82.6
Transformers
Swin-S
33
50M
8.7G
83.0
VMamba-S 32
44M
11.2G
83.5
State Space Models
VideoMamba-M 27
74M
12.7G
82.8
ResNet-101 51
45M
7.8G
81.5
ResNet-152
51
60M
11.5G
82.0
SLaK-S B1
55M
9.8G
83.8
CNNs
QuadraNet36-S [53
50M
8.9G
83.8
DeepMADD-5MM
42
50M
8.7G
83.9
ConvNeXt-S
34
50M
8.7G
83.1
Dit-ConvNeXt-S
50M
9.2G
83.6
Swin-B 33
88M
15.4G
83.5
Transformers
DeiT-B 46
87M
17.6G
81.8
State Space Models
VMamba-B 32
75M
18.0G
83.7
-LaK-B
31
95M
17.1G
84.0
QuadraNee36-B 53
90M
15.8G
84.1
CNNs
DeepMADD88M
89M
15.4G
84.0
ConvNeXt-B 34
89M
15.4G
83.8
Dit-ConwNeXt-B
90M
16.7G
84.2
in parameter count. Moreover, our models maintain competitive performance against Transformer- based, SSM-based, and CNN-based architectures, highlighting the efficacy and adaptability of Dit-ConvNeXts. Figure further shows that our model consistently improves in accuracy as the model size increases, while other methods tend to saturate, indicating superior scaling property for our model.
4.3 Ablation study
4.3.1 Dit-CNNs capture data correlation
The expectation of the quadratic term for Gaussian variables can be expressed as follows:
E
x Ax = 1 Au + tr(AE).
This equation highlights that the term tr{EE) encapsulates the information derived from data cor- relation. Consequently. Table 4 examinee the impact of this term on the performance of Dit-CNNs tackling complex tasks. The data presented in Table show a notable decline in accuracy for Dit-CNNs when this term is omitted. This underscores the significance of the quadratic neurons within Dit-CNNs, demonstrating their pivotal role effectively capturing data correlation, which essential for task performance.

---

Table 4: The performance of Dit-CNNs and their counterparts, from which the covariance term tr tr(EE) and the quadratic term x² Ax are omitted quadratic neurons (E estimated from training samples).
Performance (Original)
Performance (minus tr( tr(AE)
Performance (minus x" .Ax
Model
Dataset
Dit-ResNet-32
CIFAR-10
93.17%
37.78%
12.11%
Dit-ResNer-32
CIFAR-100
69.68%
22.33%
1.32%
Dit-ConWNeXt-T
ImageNet-1K 82.6%
73.7%
70.6%
4.3.2 Incorporate quadratic neurons with minimal computational overhead
To manage computational costs effectively, only three layers within the ConvNeXt architecture have been identified as suitable candidates for the integration of quadratic neurons, as depicted in Figure 5 We explore the most effective layer for the integration of quadratic neurons and elucidate the rationale behind this choice. Figure Saaree)indicaa indicates that substituting traditional neurons with quadratic neurons in first layer of the block yields the most significant performance improvement. Additionally, negative correlation is observed between the model's ' final accuracy and the accuracy after omitting the quadratic term from Dit-Cone Ne---o post-training. This correlation underscores the critical influence of quadratic neurons the model's efficacy. These findings suggest that Dit-Cle CNtC demonstrating enhanced performance are those where quadratic neurons play a crucial role, highlighting the superior generalization capabilities of quadratic neurons compared to traditional neurons.
Block
Block 3
Input
7x77
IXL,
1xt,
[
Pacc, Ae
[x] 354
1 96
+
d 7wx7 96
|xl, 384
2w I92
81.6
Block
Block
Block
Block
Figure 5: Left: Structure of ConvNeXt highlighting three candidate layers (in red) for integrating quadratic neurons. Right: ImageNet- 1k performance on different Dit-ConvNeXt-T. blue dots indicates top-1 accuracy (left blue vertical axis) while green dots indicates the accuracy post-removal of the quadratic term from Dit-ConveeXt-T after training (right green vertical axis).
4.3.3 Channel/Pixel-wise quadratic neuron utilizations on ImageNet-1K
Table 5: Comparison of quadratic neurons between channel-wise and pixel-wise
Model
Channel/Pixel-win
Top-1 acc. (%)
Channel-wise (Our Method)
82.6
ConvNeXi-T
Pixel-wise
82.2
Channel-wise (Our Method)
83.6
ConvNeXt-S
Pixel-wise
82.9
Channel-wise (Our Method)
84.2
ConvNeXt-B
Pixel-wise
83.7
As previously discussed, our Dit-CNNs employ quadratic neurons a channel-wise manner, which
provides a clear biological interpretation, detailed in Section 4 Meanwhile, employing quadratic
neurons on a pixel-wise basis, akin the quadratic convolution outlined in 20] [35), and [60], leads to scenario where dendritic integration occurs only when neurons receive synaptic inputs from the

---

same neuron. This setup simplifies interactions between different neurons to a linear approximation. method that diverges from biological plausibility within the brain. The superior performance achieved through the channel-wise application of quadratic neurons, as evidenced in Table underscores the efficacy of our model. This result supports the hypothesis that models which more closely mirror brain-like mechanisms tend to exhibit enhanced performance.
5 Conclusion
In this paper, we first provide theoretical demonstration of the computational advantages of quadratic neurons in capturing internal correlation within structured data. These neurons model dendritic integration rules observed in biological neurons. Our empirical evaluations using the MNIST and Arabic MNIST datasets for few-shot learning validate our theoretical assertions. Drawing from the biological interpretation of CNNs, we introduce a biologically plausible method integrate quadratic neurons into CNN architectures, resulting in Dit-CNNs. These Dit-CNNs not only exhibit significant performance enhancements with minimal modifications to their original counterparts but also compete favorably with state-of-the-ar art models. The potential applicability of our approach to other architectures, such as Deep-MAD 42 hints at further performance improvements. The promising results of this research underscore the vast potential of brain-inspired models. Given that the quadratic integration rule of neurons could be confined to specifie brain areas [5] 28), future clectrophysiological experiments could reveal other neuronal integration rules. Our findings could guide the development of brain-inspired deep neural networks by incorporating various integration rules corresponding to different brain areas in distinct layers. Additionally, how to theoretically analyze these new brain-inspired models will be an important issue. It is our aim to extend our analysis concerning high-order statistical information to explore the generalization error of these innovative brain- iaiininsspieed models. We hope that our work will stimulate further investigations into brain-inspired models, ultimately contributing to the quest for artificial general intelligence (AGI).
Limitations and Discussions
Theoretical results for quadratic neurons. Our theoretical framework is established on the assumption that training samples are normally distributed, a simplification that might not fully encapsulate the complexity inherent in real-world datasets. Despite this assumption, our models have demonstrated exceptional performance various tasks including ImageNet-1 K. This underscores the potential of quadratic neurons in capturing correlation within more intricate data distributions. Future efforts will focus on developing more comprehensive theoretical foundation to better understand the mechanisms through which quadratic neurons achieve this capability.
Computational cost of quadratic neurons. While our Dendritic integration inspired Convolutional Neural Networks (Dit-CNNG strategically limit the increase the number of learnable parameters by selectively incorporating quadratic neurons in a singular layer, this modification undeniably raises the computational complexity, as evidenced by an increase in Floating Point Operations (FLOPs). Nevertheless, given the substantial enhancements our Dit-CNNs contribute to model performance, it warrants further investigation into optimizing the efficiency of quadratic neuron deployment. For instance, drawing inspiration from the inherent sparsity observed in the quadratic coefficients of biological neurons neurons--peecffccllll specifically. the pronounced quadratic interactions among synaptic inputs on the same dendritic branch [29]- -it is conceivable to predefine a sparse configuration for the quadratic coefficients within our Dit-CNNs. This approach could potentially reduce computational demands while maintaining performance gains.
Acknowledgments
We thank Ruoyu Sun for his helpful discussions and Zhi-Qin John Xu for his comments. This work was supported by Science and Technology Innovation 2030 Brain Science and Brain-Inspired Intelligence Project with Grant No. 20212D020024; National Natural Science Foundation of China Grant 1227 1361, 12250710674 (S.L.); National Natural Science Foundation of China with Grant No. 12225109, 12071287 (D.Z.) and the Student Innovation Center at Shanghai Jiao Tong University (S.L., D.Z.)
10

---

References
[1] Hagai Agmon-Snir, Catherine E Carr, and John Rinzel. The role of dendrites in auditory coincidence detection. Nature, 393(6682):268-272 1998.
[2] Gal Ariav, Alon Polsky, and Jackie Schiller. Submillisecond precision of the input-output transformation function mediated by fast sodium dendritic spikes in basal dendrites of cal pyramidal neurons. Jourmal of Newroscience, 23(21):7750-77 2003.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton, Layer normalization, arXiv preprint 1607.06607.060 2016.
[4] Jie Bu and Anuj Karpatne. Quadratic residual networks: new class of neural networks for solving forward and inverse problems in physics involving pdes. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pages 675-683. SIAM, 2021.
[5] Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašié. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.002: 2018.
[6] Spyridon Chavlis and Panayiota Poirazi. Drawing inspiration from biological dendrites to empower artificial neural networks. Current Opinion in Neurobiology, 70:1-10, 2021.
[7] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In Proceedings of the IEEE/CVF conference on computer vision and paltern recogni- lion, pages 81 8126-8135 33,, 2021.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
[9] Mostafa Dehghani. Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480 7512. PMLR, 2023.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale bierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins
arXiv:2010.11929, 2020.
[12] Fenglei Fan, Wenxiang Cong, and Ge Wang. A new type of neurons for machine learning. International journal for numerical methods in biomedical engineering, 34(2):e2920, 2018.
[13] Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 193-222 1980.
[14] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. 2023.
[15] Jiang Hao, Xu-dong Wang, Yang Dan, Mu-ming Poo, and Xiao-hui Zhang. An arithmetic rule for spatial summation of excitatory and inhibitory inputs in pyramidal neurons. Proceedings of 2009.
the National Academy of Sciences, (00655):21106-2111
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
11

---

[17] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October I 11-14, 2016, Proceedings, Part IV 14, pages 646-661. Springer, 2016.
[18] David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology. 160(1):106, 1962.
[19] Weiwei Jiang. Mnist-mix: a multi-languuge handwritten digit recognition dataset. IOP SciNotes, 12):025002, 2020.
[20] Yiyang Jiang, Fan Yang, Hengliang Zhu, Dian Zhou, and Xuan Zeng. Nonlinear cnn: improving cnns with quadratic convolutions. Neural Computing and Applications, 32:8507-8516, 2020.
[21] Ilenna Simone Jones and Konrad Paul Kording. Might single neuron solve interesting machine learning problems through successive computations on its dendritic tree? Neural Computation, 33(6):1554 1571, 2021.
[22] Eric R Kandel, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, A James Hudspeth, Sarah Mack, et al. Principles of neural science, volume 4. McGraw-hill New York, 2000.
[23] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 0070000.0.0..... 2017.
[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):4--9,, 2017.
[25] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmistl/ 1998.
[26] Yann LeCun, Lawrence D Jackel, Léon Bottou, Corinna Cortes, John S Denker, Harris Drucker, Isabelle Guyon, Urs A Muller, Eduard Sackinger, Patrice Simard, et al. Learning algorithms for classification: A comparison on handwritten digit recognition. Neural networks: the statistical mechanics perspective, 261627)) 1995.
[27] Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Video- mamba: State space model for efficient video understanding, arXiv preprint a7Xiv:2403.06977. 2024.
[28] Songting Li, Nan Liu, Xiao-hui Zhang, Douglas Zhou, and David Cai. Bilinearity in spa- tiotemporal integration of synaptic inputs. PLoS computational biology, (001))e1004014 2014.
[29] Songting Li, Nan Liu, Xiaohui Zhang, David W McLaughlin, Douglas Zhou, and David Cai. Dendritic computations captured by an effective point neuron model. Proceedings of the National Academy of Sciences, 116330)15544-155553 2019.
[30] Gang Liu and Jing Wang. Dendrite net: A white-box module for classification, regression, and system identification. IEEE Transactions on Cybermetics, 52(12)133774 2021.
[31] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Tommi Kärkkäinen, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprint 200.002007.03620, 2022.
[32] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024.
[33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin wransformr: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[34] Zhuang Liu, Hanzi Mao, Chao-Yuan Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 11976-11986, 2022.
12

---

[35] Pranav Mantini and Shishr K Shah. Cqnn: Convolutional quadratic neural networks. In 2020 25th International Conference on Pattern Recognition (ICPR). pages 9819-9826. IEEE, 2021.
[36] Toviah Moldwin, Menachem Kalmenson, and Idan Segev. The gradient clusteron: A model neuron that learns solve classification tasks via dendritic nonlinearities, structural plasticity, and gradient descent. PLoS computational biology, 17(5)=e00000) 2021.
[37] Varun Ojha and Giuseppe Nicosia. Backpropagation neural tree. Neural Networks, 149:66-83, 2022.
[38] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning hierarchical circuits. Nature neuroscience, 24(7): (017):0000-019, 2021.
[39] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4))838-88) 855, 1992.
[40] Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser Nam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. Advances in Neural Information Processing Systems, 15353-111136 2022.
[41] Sanjeevakumar Redlapalli, Madan M Gupta, and K-Y Song. Development of quadratic neu- ral unit with applications to pattern classification. In Fourth International Symposium Uncertainty Modeling and Analysis, 2003. ISUMA 2003., pages 141-146. IEEE, 2003.
[42] Xuan Shen, Yaohua Wang. Ming Lin, Yilun Huang, Hao Tang, Xiuyu Sun, and Yanzhi Wang. Deepmad: Mathematical architecture design for deep convolutional neural network. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6163-6173, 2023.
[43] Nelson Spruston. Pyramidal neurons: dendritic structure and synaptic integration. Nature Reviews Neuroscience, 93))226--221, 2008.
[44] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 188-2886, 2016.
[45] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. Advances in neural information processing systems, 32, 2019.
[46] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[47] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Go- ing deeper with image transformers In Proceedings of the IEEE/CVF international conference on computer vision, pages 32-42, 2021.
[48] Nikolaos Tsapanos, Anastasios Tefas, Nikolaos Nikolaidis, and Ioannis Pitas. Neurons with paraboloid decision boundaries for improved neural network classification performance. IEEE 2018.
transactions on neural networks and learning systems,
[49] Balázs B Ujfalussy. Judit K Makara, Máté Lengyel, and Tiago Branco. Global and multiplexed dendritic computations under in vivo-like conditions. Neuron, 100(3)579-592, 2018.
[50] Ross Wightman. Pytorch image models.
https://ggichubb.com/rr/ggg///
pytorch-imagee models 2019.
[51] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2/ 10.00476, 2021.
[52] Xundong Wu, Xiangwen Liu, Wei Li, and Qing Wu. Improved expressivity through dendritic neural networks. Advances in neural information processing systems, 31, 2018.
13

---

[53] Chenhui Xu, Fuxun Yu, Zirui Xu, Chenchen Liu, Jinjun Xiong, and Xiang Chen. Quadranet: Improving high-order neural interaction efficiency with hardware-aware quadratic neural net- works. In 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), pages 19-25. IEEE, 2024.
[54] Zirui Xu, Fuxun Yu, Jinjun Xiong, and Xiang Chen. Quadralib: A performant quadratic neural network library for architecture optimization and design exploration. Proceedings of Machine Learning and Systems, 4:503-514, 2022.
[55] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[56] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 818-833. Springer, 2014.
[57] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
[58) Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008 2020.
[59] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024.
[60] Georgios Zoumpourlis, Alexandros Doumanoglou, Nicholas Vretos, and Petros Daras. Non- linear convolution filters for enn-based learning. In Proceedings of the IEEE international conference on computer vision, pages 4761-4766 2017.
14

---

A Proofs of theorems for binary classification task
Assume that the data points of two classes are equally sampled from two distinct non-degenerate normal distributions: class] ~ N (H1, 11), class2 N (22, 22), where Hjj E IR 2; E ax (j 2). The optimal classifier Hope((r) R + 2] is then defined based on the probability of sampling any point z from these distributions. Specifically, the classifier can be expressed as:
Bopt (I:) = arg max Pj (4r),
(
jE(1.2)
where (z(2) denotes the probability (density function) of sampling point x from the distribution (uj Σ,). The decision rule effectively assigns x to the class with the highest probability density, reflecting the most likely class membership based on the normal distribution parameters.
If a single quadratic neuron, as described in Equation ① (f(z) = x" Ar + w I + b, where A is symmetric matrix), employed to solve the binary classification task described above, can prove the following theorems.
A.1 Existence of the optimal solution
Theorem 3. (Existence) The critical points with respect the cross-entropy loss L(A, w. b) given as follows:
Σ1 [22
A ET
2z w² =22 25 p2), 6² =
16 -2 po2 + log
i.e.
OT.
OL
ØL
0A l,',0",b'=0 0, Ow la 1a',=",='=
ab LA -b.= 0,
where
T[A, w.
Fezmelassy (log(1 ed(t)) + Figmalassa (log(1 te- (e))))))]
Moreover, the corresponding classifier generated by this formula is the same the theoretically optimal classifier in Equation (5).
Proof. We express the generalized cross-entropy loss L(A, w. b) as:
L(A, w. b) = 2 / (log(1 [1 + te )p())) log(1 +e -(xx) - -(()))p((z)) du.
The gradients are computed explicitly as follows:
0f 0A 2 (1fe 1 ef(r) pa(x))
OL 1
ef(x)
aA du,
and similarly for AL distribution:
Given the probability density function of the multivariate normal and aL 8b
an
1 11) 1s)) jE (1,2}.
()(x) = (2Th d/2 1 1/2 exp
(21r)
the parameters A, w. are set as:
Ei
A = - w 2((2 p1 Eg p2), b uf''T' p1 2 p2 + log
one can obtain
((x) - -2
1 Ex )xr "T Mu 1 E3 12 + log
[E1]
which ensures that
P2(x)
(
Vr Rd.
Pr (1(x)'
15

---

Consequently,
ef(x)
aL
aL
0L
+ ef(a) (1(z)
1 + ef(a) P2(x) 0, Vz E R
0,
0,
0.
0A
dw
0b
The classifier generated by the quadratic neuron is defined as:
Ymaddl (x) if f(x) 0, f(z) > 0.
Therefore,
Umodel (x) = + (ope(1) = arg max P. (x) = 1. JE(1.2)
Similarly, (x) implies Yopt (1) 2. Hence, Ymodel (2) = Mopt for x Rd
demonstrating the consistency between these two classifiers.
A.2 Uniqueness of the optimal solution
To prove the theorem for uniqueness, we first introduce the following preliminary lemmas:
Lemma 1. Let MRR)) denote the Lebesgue a-algebra on TRd, Consider f(x), a Lebesgue measur- able function on Rdd, satisfying:
1 f() =0: vn E M(R..
It follows that f(x) 0 almost everywhere Rd.
Proof. demonstrate this via contradiction, let us assume, without loss of generality, that: pff f(x) 0}) 0.
Since
0 < n(<x f(() 03) - (z f(x) 1 S Σ u((x [(x) > 1
n=l
then we know there exist nip
z such that u((x f(x) # }) 0. If we set Po fo f(x
no J E M(Rd), then by the property of f(x) we can derive:
p((x f(z)
1 no 0.
0
f(x) du
So
Hence, we have reached a contradiction.
Lemma 2. Assume fIx) = x² Ax + w x + b, a Lebesgue measurable function, equals zero almost
everywhere (f(x) = 0 a.e.), with A being a symmetric matrix. It then follows that A 0, and O.
Proof. Since fz) is continuous, know f(x) for Vo Rd. Then we have:
b= f0) = O.
Setting x ew for some e > O, we obtain:
||ll| + ew" T Aw O,
letting go to zero, we have:
|| = 0 Having established that fx) = of Ar, consider x = e, where e, is the unit vector with a value of 1 at the i-th position and Os elsewhere. Evaluating the function at this vector, have f(e,) ef Ae; = a4s = O. Subsequently, by letting e,t where ej is another unit vector corresponding the j-th position, we obtain the following equation:
au + aij ajs ajj = 0.
Since O₁ = O,jj = and aij = aji (A is a symmetric matrix), we know:
di = Vi,j E (1, d) + A =
16

---

Theorem 4. (Uniqueness) Consider the conditional cross-entrop) loss defined on set Q E M(R)).
where (RR) denotes the Lebesgue a a-aleebra on R.d:
Exwelass, (log(1 + cfi ExNelasss:EE (log(1 + f-1))) ]
L(A, w. b 2)
with
Eaweloss BE (g(x)) = / (1))p)(1))d
then the unique solution satisfies AL w. (2)
1a-,=-,--00, 0, (A w.b (2)
( w. b (2) Öb
,0 b-= 0,
- ====
0A
Ow
for every Ω E (RR.) given by:
2
A" = E - - :²
p2), b² -2(ET'm E21
11' Eg M1
u2 Eg u2 + log
|2
Proof. Following the derivation outlined in the proof of Theorem we obtain:
@L(A, w. 12)
2 2//,
ed(x)
1 1+ (tela))2(aa)
x² du,
0A
(A, w. (0,611))
/ ef(x)
J [Teete) (x)
1 T+)) x du
Ow
aT A, w. b (2)
1 / ( ed(x)
l (1t P(())
ell) P2(x)) P2())) du.
Öb
Hence, if A", wt, and b* satisfy:
ÖL(A, A, w,b [ (2)
1.-.0r.b AL(A, w, (2)
OI(A, w,, 6l (2) 10" .b- 0
la ne b== 0,
0A
Ow
Øb
for every Ω E M(R.). Subsequently, invoking Lemma yields:
ex A a'a+t1"")
(2(x)
pi (x) a.e.
Simplifying this equation, we have
Σ
x² - [2(m Ej 124' (w*)²] x x+ k mn -12 Ex u2 u2+
-b²
0.
which holds almost everywhere. Applying Lemma 2 we obtain:
|Σ [2g
1 = Eد Σズ w² = --2 ExT 8" = /A pn - 15 1 + log
B Quadratic neurons capture correlation on multi-class classification task
For classification task with & classes, assume the training samples for these classes are equally drawn from à normal distributions: class, ~ N(y, Ez). E [k), the optimal classifier Noopt (z). Rdd (1, k) is defined by:
Hopt (x) arg max (z jE(K]
(6)
where (x) represents the probability (density function) of sampling point from the normal distribution NJJ, E,y). Employing quadratic neurons for this multi-class classification task allows for the derivation of theorem analogous to the one discussed in the previous section:
17

---

Theorem 5. (Existence) The critical points with respect to the cross-entropy loss L(0) are given as follows:
A = 23 wj -22; p.j b' =x² =,, 2s Log([()))),, j [k]
where
]
k ((0)= = k Σ Pawelass,
k (.(())_(())
log
1
1=1,
Moreover, the corresponding classier generated by this formula is the same the theoretically optimal classifier as defined in Equation (6).
Proof. The gradients be computed explicitly follows:
0L
k Σ
(b(w))
k
efi(2)
Pi(x)
Σ Ex Ps()) Po(a)) x x² du, j E [M]
x.x"
0A,
2dd (=1,ii)
E ehiw) edi
and similarly for OL and @L that be calculated. From the derivation outlined in the proof of They Jb, Theorem and the choice for critical points as follows
=z w -20 pj. b' n My + log([E,)), j [a].
then we have
ef(())-((()) p,(() (x)
Vr E RR. Vi,j E [k]
which is equivalent to
ch(x)
(fi(a)
E ef(z) Pr(x)
D efile) (z(z), Vo E Rd, Vi E [k].
Thus, we can derive:
OT. = 0, Dw, 0b
ØL = 0, = 0, Aj E ik].
ØT.
0A,-
The classifier generated by quadratic neurons is defined
Ymodel (x) arg max fy (z).
jE(M]
Therefore,
Umodel(2)) = + fz(x) > f((x,, vi E [k]/U}} + (z) =
Hence, (mooei(x) (ope(x) for all in Rd, demonstrating the consistency between these two classifiers.
Theorem 5 demonstrates that in multi-class classification tasks, each neuron captures data correlation by directly relating to the covariance matrix of their respective class. This concept empirically validated through numerical experiments on the MNIST dataset, as depicted in Figure 6 The eigenvectors associated with the largest eigenvalues of the covariance matrices, representing the principal components of each class's 's distribution, reveal the spatial correlation within that class's data. The marked similarity between these eigenvectors and the eigenvectors of the quadratic integration matrices of neurons confirms that quadratic neurons effectively capture correlation from the data distribution. This intrinsic ability of quadratic neurons to capture correlations provides a possible explanation for the superior generalization capability of quadratic neurons.
C Experimental settings of ImageNet training
The training settings for our Dit-ConwNeXt models on ImageNet- 1K are detailed in Table While all Dit- ConvNeXt variants adhere to a unified configuration, certain parameters- nnmelly the stochastic depth rate, learning rate, and weight decay-are individually customized for each model variant. These hyperparameters were optimized based on performance outcomes, utilizing grid search over potential learning rates from the set {0.004, 0.006, 0.008) and weight decay values options (0.04, 0.06, 0.08].
within
18

---

Eigenvector Eigenvector Eigenvector Eigenvector of
2
3
Avg.
of
0.8
cosine similarity
0.6
of
0.4
5
6
0.2
of
0.0
10
Figure 6: Comparison of Eigenvectors between covariance matrices Σj and quadratic weights Aj for j {0, 2, 4, 6, 9}. Left: Visualization of eigenvectors corresponding the largest eigenvalue of Σ alongside the most similar eigenvectors of Aj. Right: Cosine similarity metrics for ten eigenvector pairs depicted on the left.
Table 6: ImageNet-IK training settings.
Training config
Dit-ConvNeXt-T
Dit-ConwNeXt-S
Dit-ConvNext-B
Optimizer
AdamW
AdamW
AdamW
Base learning rate
6e-3
6e-3
4e-3
Weight decay
0.08
0.06
0.06
Optimizer momentum
B1, B2 = 0.9, 0.999
81, B2 = 0.9, 0.999
8,,3 = 0.9, 0.999
Batch size
4096
4096
4096
Training epochs
300
300
300
Learning rate schedule
cosine
cosine
cosine
Warmup epochs
20
20
20
Warmup schedule
linear rand-m9 mantd-99-mtd0.55
linear rand-m9-mstd03
linear
RandAugment [
rand-m 1und-m9-mstd0.5
Mixup [57
0.8
0.8
0.8
Cutmix 55
1.0
1.0
1.0
Random erasing
0.25
0.25
0.25
Label smoothing 44
0.1
0.1
0.1
Stochastic depth
0.1
0.4
0.4
Layer scale [47]
Ic-6
le-6
le-6
Exp. mov. avg. (EMA) 39
0.9998
0.9998
0.9998
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section D
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the Limitations and Discussions part.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
See Section 2 and Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The main text, along with Appendix C contains comprehensive details nec- essary for replicating the study's ' findings, including hyperparameters, data augmentation techniques, etc.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ prapp..cc/pblli/guuddee CoodeSubeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The main text, along with Appendix C contains comprehensive details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Several experiments, including the comparison of top-1 accuracy on ImageNet- IK, do not feature error bars due to adherence to the convention established in prior studies, which omitted such reporting.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See the last paragraph in Section !
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper focuses exclusively on fundamental scientific research and does not address societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Reference section.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "6AMvUbl9oN", "reviewer": "Reviewer_cuyp", "summary": "The paper theoretically demonstrate that quadratic neurons inspired from dendritic computing inherently capture correlation within structured data. The quadratic rules are integrated with convolution networks and established the so-called Dit-CNNs. Experiments on CIFAR and ImageNet datasets demonstrates competitive performance of the proposed model.", "questions": "1. In the ablation study 4.3.1, table 4, why performance dropping level are significantly different for CIFAR and ImageNet?", "limitations": "See weaknesses and questions", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper provides theoretically demonstration of critical points existence of parameters of quadratic neuron. An illustrative binary classification experiment is constructed. The quadratic rules are further verified on MLP network with few-shot MNIST learning task and convolution networks in CIFAR and ImageNet classification tasks. These experiments demonstrate competitive performance of the proposed model to other related methods.", "weaknesses": "1. The paper demonstrates that there exist critical points for the parameters of quadratic neuron, however, there lacks experimental proof that the training process actually drives these parameters towards the critical points. It would be more convincing if the author could provide such evidence, for datasets with different scales.\n2. Compared to previous quadratic dendritic methods, the improvement in accuracy of the proposed model is not so significant, I suggest that the author provide comparisons on computation cost to further demonstrate the advantage of the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n1. Thank you for your suggestion. We have indeed demonstrated the uniqueness of critical points under certain assumptions (Theorem 2 in the main text). Under these conditions, it can also be theoretically proven that if the gradient flow algorithm converges, it will drive the parameters to this unique critical point [1][2]. We will clarify this in the updated version. The numerical results presented in Section 3.1 (Figure 1 with sufficient training samples) further confirm the correspondence between the theoretically identified critical point and the numerically converging one.  \nFor more general cases, such as MNIST, Figure 5 in Appendix B shows that the post-training results are consistent with the theoretically identified critical point established in Theorem 5. For deeper networks and more complex datasets like CIFAR-10 and ImageNet, computing the critical points analytically becomes challenging.\n\n2. Thank you for your valuable feedback. In Table 2 of the main text, we compare our Dit-CNNs with other quadratic methods. Our models show improvements in accuracy while utilizing only about one-third of the number of trainable parameters, highlighting its efficiency.  \nAdditionally, following your suggestion, we visualized the test accuracy in Figure 2.A of the rebuttal materials, which demonstrates that our model consistently improves in accuracy as the model size increases, while other quadratic methods tend to saturate. This indicates superior scaling properties for our approach.\n\n[1] Quarteroni, Alfio, Riccardo Sacco, and Fausto Saleri. Numerical mathematics. Vol. 37. Springer Science & Business Media, 2006.\n\n[2] Ambrosio, Luigi, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2008.\n\n## Questions\nThank you for your insightful question. There are two possible reasons accounting for the difference: architecture and dataset (i.e., ResNet for CIFAR and ConvNeXt for ImageNet). To investigate the reason, we have trained Dit-ResNet on ImageNet-1K. After training, when we omitted the quadratic terms, the test accuracy dropped to 0.1%, which aligns with the Dit-ResNet results on CIFAR-10. So the key factor contributing to the differing performance drop levels in the ablation study is the use of entirely different network architectures for CIFAR and ImageNet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ghB9ZYq4QS", "reviewer": "Reviewer_opWC", "summary": "This paper explores the computational benefits of quadratic neurons, which are inspired by the quadratic integration rules of dendrites. The authors first present the theoretical analysis on binary classification for normal distributions, showing the existence and uniqueness of the solution with a single quadratic neuron. Then, a few-shot learning experiment on MNIST and Arabic MNIST is conducted to show the better performance of quadratic neurons under few-shot training samples. Finally, this paper integrates quadratic neurons into CNNs along the channel dimension, and demonstrates the superior performance of the model on several datasets including CIFAR and ImageNet.", "questions": "Will incorporating quadratic neurons into multiple layers continually increase the performance? How is the suitable layer candidate identified?", "limitations": "The authors discussed limitations in Appendix D.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper considers the quadratic integration rule inspired by biological dendrites and successfully applies it to advanced artificial neural networks such as ConvNeXt.\n\n2. Experiments show promising performance on relatively large-scale datasets, e.g., ImageNet.", "weaknesses": "1. The presentation and organization of the paper are poor and loose. There are also many informal claims without rigorous justification.\n\n(1.1) The link between the theoretical analysis, the so-called “enhanced generalization capabilities”, the few-shot learning advantage, and the integration into CNNs is poor. I do not see that the theoretical analysis of the existence and uniqueness of the solution of a single quadratic neuron under a simplified setting provides effective insight or explanation for the following practice. There is no analysis of deep networks. There is no explanation for the connection between the practice of CNNs and theories.\n\n(1.2) In the discussion of Theorem 1 and 2, it is said using the gradient descent algorithm. However, there is no analysis for training dynamics or learnability in current Theorems.\n\n(1.3) There is no formal definition or rigorous justification for the so-called “superior generalization capability over traditional neuron”.\n\n(1.4) It is claimed that the integration of quadratic neurons into CNNs along the channel dimension is “biologically plausible”. However, I do not see any detailed justification. Indeed, only considering quadratic integration along the channel dimension does not correspond to synaptic connections between neurons.\n\n2. The novelty and significance of this paper are not clear enough. Quadratic neurons have been studied in many previous works, both from theoretical and practical perspectives, e.g., [1] has shown strong results. While this paper interprets it as inspiration from biological neurons, the implementation does not correspond to the biological computation form. So there should be more differentiation with existing quadratic neuron works. Considering Table 1, the biological interpretation in this paper is not strong enough, and there are also theories in previous works [1]. Only changing how the quadratic operation is used makes little contribution.\n\n3. There is no formal definition and analysis showing “enhance data correlation” in the title.\n\n[1] Fan, F. L., Dong, H. C., Wu, Z., Ruan, L., Zeng, T., Cui, Y., & Liao, J. X. (2023). One neuron saved is one neuron earned: On parametric efficiency of quadratic networks. arXiv preprint arXiv:2303.06316.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n1. Thank you for your insightful feedback, and I apologize for any lack of clarity in our presentation.\n\n(1.1) Our main analysis aims to  emphasizes that quadratic neurons inherently capture second-order information from training samples, which enhances their generalization capabilities compared to conventional neurons.  \n\nTo support this, we provide a theoretical proof demonstrating how quadratic neuron capture second-order information to achieve optimal result in binary classification for two normal distributions characterized by first and second-order moments. And our numerical experiments shows that quadratic neurons indeed effectively capture this information while conventional neurons can not.\n\nWe generalize the theorem to multi-class classification tasks (Theorem 5). We also perform numerical experiments on MNIST to verify that quadratic neurons capture second-order information in this case (Appendix B, Figure 5).\n\nIn more complex scenarios, such as CIFAR-10 and ImageNet using deep CNNs, deriving theoretical results becomes challenging. However, previous theoretical analyses provided insights into why quadratic neurons perform better. So we conduct ablation study in Section 4.3.1 to show that quadratic neurons indeed capture second-order information for classification when they performs better in these cases, further supporting our conclusions.\n\n(1.2) Thank you for your observation. Theorem 1 elucidates a possible analytical solution (existence of critical point) for gradient flow algorithm. And Theorem 2 establish the uniqueness of that critical points under certain assumptions. Under these conditions, it can be theoretically proven that if the gradient flow algorithm eventually converges, it will drive the parameters to this unique critical point [1][2]. We will clarify this in the updated version. And the numerical results presented in Section 3.1 (Figure 1 with sufficient training samples) further confirm the correspondence between the theoretically identified critical point and the numerically converging one.\n\n[1] Quarteroni, Alfio, Riccardo Sacco, and Fausto Saleri. Numerical mathematics. Vol. 37. Springer Science & Business Media, 2006.\n\n[2] Ambrosio, Luigi, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2008.\n\n(1.3) The superior generalization capability means the ability of models incorporating quadratic neurons to achieve lower generalization error. For instance, we demonstrate this through the smaller generalization error observed in binary classification tasks where the target function is known, as well as higher test accuracy on the MNIST, CIFAR-10, and ImageNet-1 datasets. We will clarify the meaning of “superior generalization capability over traditional neuron” in the updated version.\n\n(1.4) Thank you for the valuable comments, we have provided a biological interpretation of our Dit-CNN in global rebuttal.\n\n2. Thank you for your feedback. We clarify the novelty and significance as follows: \n- **Biological Interpretation:** Our channel-wise quadratic model offers a comprehensive biological interpretation, as explained in global rebuttal, whereas other quadratic methods lack such evidence.\n- **Efficiency:** Our channel-wise quadratic form demonstrates significantly higher efficiency, utilizing only about one-third of the number of trainable parameters compared to other quadratic methods. This is detailed in Table 2 of the main text, where we also include a comparison with the model referenced in your review ([12] in the main text).\n- **Scaling Property:** We have added a visualization of the test accuracy comparison in Figure 2.A of the rebuttal materials. This shows that our model consistently improves in accuracy as the model size increases, while other quadratic methods tend to saturate, indicating superior scaling property for our model.\n- **Focus on Generalization:** Our analysis does not concentrate on the universal approximation properties of networks, as seen in the theorems of the paper you provided. Instead, we aim to explain the advantages of quadratic neurons from the generalization capability perspective, emphasizing their ability to capture second-order (correlation) information from data distributions. We believe these insights differentiate our work from existing studies on quadratic neurons and contribute to a various understanding of their practical benefits.\n\n3. Thank you for your helpful feedback. In this paper, \"enhanced data correlation\" refers to the model's ability to capture second-order information (correlation) effectively, which is verified through our numerical experiments on MNIST, CIFAR, and ImageNet. To clarify, we will change “enhance data correlation” to “capture data correlation” in the updated version.\n\n## Questions\nThank you for your insightful questions.\n- \"Will incorporating quadratic neurons into multiple layers continually increase the performance?\" \n\nDue to limited time and resources, we have only experimented with incorporating quadratic neurons into multiple layers on CIFAR-10, which resulted in a gradual increase in performance. However, our observations suggested that replacing only one layer is often the best choice for balancing computational cost and performance.\n- \"How is the suitable layer candidate identified?\" \n\nWe discuss this in Section 4.3.2 of the main text. Suitable layer candidates are identified based on the trade-off between computational cost and performance improvement. In our practice, we initially incorporate quadratic neurons into each layer of a fixed architecture, such as ResNet, to evaluate performance in smaller models like ResNet-20. Once a suitable layer for incorporating quadratic neurons is identified, the same replacement is applied to deeper models, such as ResNet-110.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6oi0Iu0zuV", "reviewer": "Reviewer_8aT5", "summary": "This is an interesting paper looking at quadratic neurons and how they impact performance and/or learning rate of ANNs. The quadratic integration is loosely linked to dendritic integration properties of pyramidal neurons in cortex (though it is unclear whether any real resemblence should be granted). This increase in the degree of nonlinearity , when implemented in CNNS (called Dit-CNNs) works to the advantage on benchmarks like imagenet while also retaining the simplicity of conventional CNNs. Of interest and importance are the elegant anaytical solutions exemplified in figure 1 as well as the ablation study shown in table 4. Overall this is an interesting and timely study.", "questions": "could a potentially more nuanced or bio-plausible implementation of dendritic integration further enhance performance? How would that affect the quadratic integration?", "limitations": "it is left somewhat unclear how the dendritic integration along different compartments of a cell and its result in the neuron's output is reflected in quadratic neurons. It would be interesting to look at sub- vs. supra-linear integration effects along the same lines.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is an interesting study looking at the impact of relatively recent findings in neurobiology . Even if how these neurobiological findings are loosely connected to the quadratic model, how the model and the associated networks are characterized is thorough and insightful. The ablation part as well as the work on how to integrate the quadratic neurons with minimal overhead are nice contributions that further substantiate the author's decision to account for higher terms that could potentially be linked to dendritic integration.", "weaknesses": "The authors use the term \"dendritic integration\" loosely without exploring various ways that such integration can occur (sub- vs. supra-linear). It would be interesting to consider such cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\nThank you for your suggestion, it is indeed an interesting idea. Previous work has shown that the dendritic bilinear integration rule can account for both sub-linear and supra-linear cases, depending on the sign of the quadratic coefficient. In our approach, we directly incorporate this rule into artificial neural networks (ANNs) without restricting the integration to only sub-linear or supra-linear cases, specifically, we do not fix the sign of the quadratic coefficients during training, allowing both sub-linear and supra-linear to happen. An additional observation is that, after training, the quadratic coefficients exhibit both positive and negative components, as illustrated in Figure 3.B of the rebuttal materials. This indicates that our models capture both sub-linear and supra-linear effects.\n\n## Questions\nThank you for your insightful question. The dendritic bilinear integration rule is primarily restricted to subthreshold regime before the neuron generate spikes. However, there are other properties of dendritic integration, such as dendritic spikes [1], that warrant further exploration. Due to time constraints, we have not fully investigated these properties. However, examining how to incorporate these features into artificial neural networks (ANNs) and how to combine them with the bilinear integration rule could potentially enhance performance. This is an area that deserves future research.\n\n[1] Gidon, Albert, et al. \"Dendritic action potentials and computation in human layer 2/3 cortical neurons.\" Science 367.6473 (2020): 83-87.\n\n## Limitations\nThank you for your valuable suggestion. The dendritic bilinear integration rule already captures the somatic response when synaptic inputs are integrated from different compartments of a neuron. Therefore, we can model quadratic neurons without explicitly considering sub- and supra-linear integration along these compartments. However, we agree that exploring the integration effects in dendritic compartments could deepen our understanding of neuronal computation, and this area deserves further investigation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SO4SiiJnzK", "reviewer": "Reviewer_mc29", "summary": "The paper introduces a new biologically inspired neural network architecture. Rather than using linear layers followed by a nonlinear activation function, the authors propose a quadratic model instead in the inputs. This form is said to explicitly model the covariance between input features offering better accuracy and generalization.", "questions": "Have the authors look into the structured learned by the quadratic term? \nHow do the CNN filters differ from those previously learned? \nDid the authors compare to other smoother activation functions such as gelu?", "limitations": "The limitations are not outlined in the main text but are in the appendix instead.\nIt would be beneficial to include these in the main text.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors present improved accuracy with fewer parameters than existing state of the art models by simply replacing a few layers. \nThe authors have conducted a thorough series of experiments", "weaknesses": "The relation to biological plausibility is unclear and the link is tenuous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\nThank you for the valuable comments, we have provided a biological interpretation of our Dit-CNN in global rebuttal.\n\n## Questions\n- \"Have the authors look into the structured learned by the quadratic term?\"\n\nThank you for your valuable question. Following your suggestion, we have examined the distribution of the trained quadratic coefficients, as shown in Figure 3.B of the rebuttal materials. We find that most of the coefficients are close to zero, suggesting the potential to further reduce the computational cost of our model. Additionally, the distribution is symmetric and confined to a reasonable range, without extremely large or small values.\n\n- \"How do the CNN filters differ from those previously learned?\"\n\nThank you for your question. Following your suggestion, we visualize the CNN filters of the original ConvNeXt and our proposed Dit-ConvNeXt, as shown in Figure 2.B of the rebuttal materials. This visualization highlights the differences in training results between the two models, indicating that our Dit-CNN is not merely a small modification of the original CNNs.\n\n- \"Did the authors compare to other smoother activation functions such as gelu?\"\n\nThank you for your helpful question. Following your suggestion, we compared our quadratic neuron with the smoother activation function GeLU on binary classification and the MNIST dataset, as shown in Figure 3.A of the rebuttal materials. Our results demonstrate that while GeLU activation performs better than ReLU in the binary classification task, it still does not achieve the optimal results seen with quadratic neurons. Additionally, GeLU activation performs even worse on the MNIST dataset. For more complex datasets, ConvNeXt already utilizes GeLU activation, and the results for this case are presented in Table 3 of the main text. These results are consistent with our theoretical analysis: the computational advantage of quadratic neurons arises from their direct second-order interactions between inputs, enabling them to better capture data correlations, rather than from the smoothness of the quadratic function.\n\n## Limitations\nThank you for your advice. We will move the limitations to the main text in a later revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NIUCDXJ0gD", "reviewer": "Reviewer_gq5j", "summary": "This study explores incorporating channel-wise quadratic neuron, as inspired by dendritic nonlinearity, into artifical neural networks to improve model performance. These models show competitive performance on datasets like CIFAR, ImageNet-1K while maintaining simplicity and efficiency. The theoretical and experimental results highlight the potential of quadratic neurons in enhancing model performance.", "questions": "See above.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a sound and well-supported exploration of quadratic neurons in ANNs. The theoretical analysis gives a good intuitive demonstration on the reason behind the benefit of quadratic neurons. The experimental verification is clear and show clear advantage given by their architecture.\nOverall the paper is well-written and clearly explained.", "weaknesses": "The theortical explaination is limited to highly simplified setting. \nIt is not clear if the emperical analysis on imagenet part is a fair one, give quite sophicated data augumentation is used for their models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n- \"The theortical explaination is limited to highly simplified setting. \"\n\nThank you for your comment. Our theoretical analysis aims to emphasize that quadratic neurons inherently capture second-order information from training samples, which enhances their generalization capabilities compared to conventional neurons. In more complex scenarios, such as CIFAR-10 and ImageNet using deep CNNs, deriving theoretical results becomes challenging. However, theoretical analysis under simplified setting provided insights into why quadratic neurons perform better. Therefore, we conduct ablation study in Section 4.3.1 to show that quadratic neurons indeed capture second-order information for classification when they performs better in these cases, further supporting our conclusions.\n\n- \"It is not clear if the emperical analysis on imagenet part is a fair one, give quite sophicated data augumentation is used for their models.\"\n\nThank you for your valuable feedback. The comparison is fair from two perspectives. Firstly, Our Dit-ConvNeXt employed the same data augmentation techniques as the original ConvNeXt, which indicates that our model does not require significant modifications to achieve notable improvements. Secondly, the state-of-the-art (SOTA) models we compared in Table 3 of the main text also utilize extensive data augmentations and tuning to achieve their results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all the reviewers for their high-quality reviews. To address the reviewers' concerns, we have added more experiments and provided additional details about the model. We hope that our responses effectively address the reviewers' feedback .

 We have provided a comprehensive biological interpretation of our Dit-CNN, as illustrated in Figure 1.C of the rebuttal materials. Our Dit-CNN is inspired by neural networks in the visual system. For example, different types of cone cells encode various color (channel) information, and retinal ganglion cells receive inputs from multiple types of cone cells [1], the responses can be modeled as having receptive fields (convolutional kernels) related to different color channels ($w_1 *x_1, w_2 *x_2, w_3 *x_3 $). When multiple channel inputs are present, traditional CNNs simply linearly sum the corresponding responses. In contrast, neurons integrate these inputs with an additional quadratic term based on the dendritic bilinear integration rule. This approach leads to the formulation of our Dit-CNN after simplification. We believe this integration reflects a more biologically plausible mechanism compared to conventional methods.

[1] Kandel, Eric R., et al., eds. Principles of neural science. Vol. 4. New York: McGraw-hill, 2000.

Meta-review: While the focus in the field has largely centered around building ever more complex and larger architectures of simple individual units, there is a lot of complexity and computation happening at the level of individual neurons in biological circuits. This paper focuses on a relatively understudied issue of where/how to incorporate some of the biophysical apparatus of individual neurons into neural networks. In this sense, I think that this paper contributes to the diversity of efforts in the field and it is useful to stimulate different approaches.

That said, I read the paper and responses and the paper is not very clearly written. The paper could benefit from editing by native speakers or further attention to grammar, prose, and exposition clarity. 

The authors sent a note to the Area Chairs, expressing concern about reviewer opWC. I read the exchange of author responses and reviewer questions. While it is true that the reviewer only succinctly responded to the authors’ response, I do not see any inappropriate action from the reviewer. It is clear to me that opWC read the paper and provided useful critical feedback. Could opWC engage in a longer discussion of each of the rebuttal points? Probably. But overall, I feel that this is a rather reasonable review. 

In all, I share many of the positive points noted by reviewers and I think that this work is different from others and deserves to be accepted, even though the overall discussion is borderline.
Decision: Accept (poster)