Dendritic Integration Inspired Artificial Neural Networks Capture Data Correlation

Authors: chongming  liu
jingyang  ma
songting  li
douglas  zhou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:02:21

Abstract: Incorporating biological neuronal properties into Artificial Neural Networks (ANNs) to enhance computational capabilities is under active investigation in the field of deep learning. Inspired by recent findings indicating that dendrites adhere to quadratic integration rule for synaptic inputs, this study explores the computational benefits of quadratic neurons. We theoretically demonstrate that quadratic neurons inherently capture correlation within structured data, a feature that grants them superior generalization abilities over traditional neurons. This is substantiated by few-shot learning experiments. Furthermore, we integrate the quadratic rule into Convolutional Neural Networks (CNNs) using a biologically plausible approach, resulting in innovative architectures—Dendritic integration inspired CNNs (Dit-CNNs). Our Dit-CNNs compete favorably with state-of-the-art models across multiple classification benchmarks, e.g., ImageNet-1K, while retaining the simplicity and efficiency of traditional CNNs. All source code are available at https://github.com/liuchongming1999/Dendritic-integration-inspired-CNN-NeurIPS-2024.

Main paper: 
 introduction: 
 While the Artificial Neural Network (ANN) framework has made significant advancements towards solving complex tasks, it still faces problems that are rudimentary to real brains [6] . A notable distinction between the modern ANN framework and the human brain is that the former relies on a significant number of training samples, which consumes large amounts of energy, whereas the latter runs on extremely low power (<20 watts) and possesses a strong generalization capability based on few-shot learning. Studies have demonstrated that incorporating dendritic features in ANNs can alleviate these issues and enhance overall performance [52, 38, 30] . However, it is difficult to quantify the nonlinear integration of dendrites, which is an essential property that allows individual neurons to perform complex computations [43, 49] . As a result, dendritic-inspired models often employ linear integration with nonlinear activation functions such as ReLU and Sigmoid [21, 37] .In light of recent studies revealing that the somatic responses of biological neurons to multiple synaptic inputs on dendrites follow a quadratic integration rule [15, 28] , we explore the computational benefit of the quadratic neuron model [41] . This model substitutes the linear integration and nonlinear 2 Related work Dendritic-inspired computational models. The significant role of dendrites in the nonlinear integration within neural systems is well-recognized, enabling neurons to perform intricate tasks [43, 49, 2] . Recently, numerous studies have explored the implementation of dendritic features in ANNs from different aspects, yielding encouraging results. In [52, 21, 37] , the integration of dendritic morphology into ANNs has led to higher test accuracies on simple tasks compared to traditional ANNs. Furthermore, dendritic plasticity rules have been employed to develop learning algorithms in [38, 36] , effectively replacing the non-biological backpropagation algorithm and achieving enhanced performance on classification tasks with small data sizes. Additionally, in [30] , dendritic nonlinearity is considered by adding input to each layer with the Hadamard product rule, which has shown improvements in approximation and classification experiments.Neural networks with high-order interactions. Recent advancements in the design of deep learning architectures have been predominantly driven by the ability to capture high-order statistics among inputs and features. Over recent decades, there has been a pivotal shift in the foundational neural networks used for computer vision tasks: moving from convolution-based architectures [16, 24, 56] to Transformer-based models [11, 33, 9] . This transition has been driven by the development of the self-attention mechanism, which facilitates quadratic interactions among inputs. Additionally, a novel category of neural networks, referred to as Mamba [14] , has demonstrated exceptional performance in tasks that require the processing of extensive high-order statistical information, such as video understanding [32, 59, 27] . The backbone of Mamba's architecture is a state space model that enables high-order interactions among inputs. Moreover, the integration of high-order spatial interactions into CNNs has also been shown to enhance performance significantly [40] . These developments underscore the significance of high-order statistical information in tackling large-scale complex tasks and suggest that incorporating high-order interactions as a foundational aspect of model design is an effective strategy.Table 1 : The overview of current works with quadratic format.Quadratic format How quadratic operation is used Biological interpretation Theory for generalization [54] , [12] , [4] , [53] f (x) = (wax)(w b x)Pixel-Wise N N [48] , [20] , [35] , [60] f (x) = x T Ax Pixel-Wise N N Dit-CNNs f (x) = x T Ax Channel-Wise Y YPrevious works have explored the concept of quadratic integration as a sophisticated alternative to traditional linear summation. Table 1 summarizes various neuron models formulated in quadratic formats, along with their relevant references. The one-rank format of quadratic neurons, when applied across entire networks, is documented in [54, 12, 4, 53] , demonstrating enhanced performance in classification tasks. Studies such as [48, 20, 35, 60] have implemented quadratic neurons on a pixel-wise basis within convolution layers, yielding modest improvements in test accuracy. In contrast, Table 1 illustrates how our Dit-CNNs approach departs from these models by integrating a channel-wise quadratic operations with biological interpretation and providing a theoretical analysis of the computational advantages offered by quadratic neurons.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: See Section 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See the Limitations and Discussions part.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
quadratic neurons possess enhanced generalization capabilities: 
 This section provides a theoretical analysis demonstrating how quadratic neurons enhance generalization capabilities by effectively capturing correlation within structured data. Additional numerical experiments corroborate this assertion. The detailed proofs can be found in Appendix A.Assume that the data points of two classes are equally sampled from two different non-degenerate normal distributions:class 1 ∼ N (µ 1 , Σ 1 ), class 2 ∼ N (µ 2 , Σ 2 ), where µ j ∈ R d , Σ j ∈ R d×d (j = 1, 2).Then the optimal classifier y opt (x) : R d → {1, 2} can be defined according to the sampling probability:EQUATIONwhere p j (x) is the probability (density function) of sampling point x from distribution N (µ j , Σ j ).If a single quadratic neuron, as described in Equation (1) (f (x) = x T Ax + w • x + b, where A is a symmetric matrix), is used to solve the above binary classification task, we can prove the following theorems: Theorem 1. (Existence) The critical points with respect to the cross-entropy loss L(A, w, b) are given as follows:A * = Σ -1 1 -Σ -1 2 , w * = -2(Σ -1 1 µ 1 -Σ -1 2 µ 2 ), b * = µ T 1 Σ -1 1 µ 1 -µ T 2 Σ -1 2 µ 2 + log |Σ 1 | |Σ 2 | . i.e. ∂L ∂A | A * ,w * ,b * = 0, ∂L ∂w | A * ,w * ,b * = 0, ∂L ∂b | A * ,w * ,b * = 0,whereL(A, w, b) = 1 2 E x∼class1 log(1 + e f (x) ) + E x∼class2 log(1 + e -f (x) ) .Moreover, the corresponding classifier generated by this formula is the same as the theoretically optimal classifier in Equation (2). Theorem 2. (Uniqueness) Consider the conditional cross-entropy loss defined on a set Ω ∈ M(R d ), where M(R d ) denotes the Lebesgue σ-algebra on R d :L(A, w, b | Ω) = 1 2 E x∼class1,x∈Ω log(1 + e f (x) ) + E x∼class2,x∈Ω log(1 + e -f (x) ) ,whereE x∼classj ,x∈Ω (g(x)) = Ω g(x)p j (x) dµ,then the unique solution satisfying∂L(A, w, b | Ω) ∂A | A * ,w * ,b * = 0, ∂L(A, w, b | Ω) ∂w | A * ,w * ,b * = 0, ∂L(A, w, b | Ω) ∂b | A * ,w * ,b * = 0 for every Ω ∈ M(R d) is given by: Training is executed using gradient descent with a learning rate of 0.1 over 10,000 epochs. The term "theoretical" denotes the optimal classifier's boundary as specified in Equation (2), while "numerical" represents the empirical classification boundary obtained from the model.A * = Σ -1 1 -Σ -1 2 , w * = -2(Σ -1 1 µ 1 -Σ -1 2 µ 2 ), b * = µ T 1 Σ -1 1 µ 1 -µ T 2 Σ -1 2 µ 2 + log |Σ 1 | |Σ 2 | .Theorem 1 identifies an analytical critical point for a quadratic neuron with cross-entropy loss, which emerges as the optimal classifier. On the other hand, Theorem 2 establishes the uniqueness of this critical point under conditional cross-entropy loss, where Ω denotes the regions occupied by distinct batches of data points. Under these conditions, the quadratic parameters will converge to this unique critical point when using the stochastic gradient descent (SGD) algorithm. It is demonstrated that, unlike traditional neuron, quadratic neuron inherently possesses the ability to converge to the optimal classification solution by directly including the covariance matrix. This finding is supported by simulated data in Task 1 (identical covariance) and Task 2 (non-identical covariance) shown in Figure 1 , which indicates that quadratic neuron consistently achieves the theoretically optimal classification outcome. Moreover, in comparison to a multi-layer perceptron (MLP) with two layers, quadratic neuron surpass traditional neuron in Task 2 which require capturing correlation information from data distributions (require fewer training samples to converge to the optimal solution). This highlights the unique capability of quadratic neurons to identify internal correlations within structured data, offering a possible explanation for their superior generalization capability compared to traditional neurons, a topic that will be further explored in numerical experiments below.From the above classification tasks for normal distributions as depicted in Figure 1 , it is suggested that quadratic neurons may require fewer training samples in comparison to traditional neurons. We next examine this few-shot learning capability of quadratic neurons on the MNIST [25] and Arabic MNIST datasets [19] . The results, presented in Figure 2 , indicate that quadratic neurons outperform MLP with two layers when trained with a limited number of samples on both datasets. This evidence shows the enhanced generalization ability of quadratic neurons in applications. Further insights into the mechanism through which quadratic neurons capture data correlation in the MNIST dataset, and a related theorem concerning multi-class classification for normal distributions, are detailed in Appendix B. Dit-CNNs is inspired by neural networks in the visual system. For example, different types of cone cells encode various color (channel) information, and retinal ganglion cells receive inputs from multiple types of cone cells [22] , the responses can be modeled as having receptive fields (convolutional kernels) related to different color channels (w 1 * x 1 , w 2 * x 2 , w 3 * x 3 ). When multiple channel inputs are present, traditional CNNs simply linearly sum the corresponding responses. In contrast, dendritic neurons integrate these inputs with an additional quadratic term based on the dendritic quadratic integration rule. This approach leads to the formulation of Dit-CNNs after simplification.The concept of convolution [13] , inspired by the discovery of the receptive field in the cat visual cortex [18] , laid the foundation for CNNs [26] which have been designed to efficiently tackle large-scale vision tasks. The underlying biological interpretation of convolution is that each neuron exhibits varying responses at different image locations based on its receptive field (convolution kernel). Traditionally, CNNs have focused solely on the linear integration of inputs from presynaptic neurons. Specifically, for an input X ∈ R Cin×Hin×Win , where C in represents the number of input layer neurons (channels), and H in and W in denote the height and width of the input feature, respectively. Given convolution kernels w ∈ R Cin×Cout×(2l+1)×(2l+1) , the convolution output, Y = Conv(X) ∈ R Cout×Hout×Wout , represents the response of neuron i at location [j, k] to inputs from locations [j -l : j + l, k -l : k + l] across all input layer neurons:EQUATIONHowever, acknowledging that neurons with dendrites obey a quadratic integration rule, we propose a novel approach that incorporates quadratic neurons into CNNs as depicted in Figure 3 . This adaptation incorporates a quadratic integration term among inputs from different neurons (channels). Specifically, with a quadratic integration coefficient A ∈ R Cout×Cin×Cin , the output neuron's response in Equation ( 3) is modified as follows:EQUATIONQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: See the last paragraph in Section 1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
evaluations on cifar-10 and cifar-100: 
 Dataset and models. Our initial experiments utilize the CIFAR dataset, which includes 50,000 training images and 10,000 testing images. The experimental setup follows the ResNet configurations as outlined in [16] , comprising models such as ResNet-20, ResNet-32, ResNet-56 and ResNet-110. To reduce computational demands, we incorporate quadratic neurons specifically into one layer-namely, the second layer of the first block in the second stage of the ResNet architecture [16] . Additionally, to address issues related to gradient dynamics, such as gradient explosion, a Layer Normalization (LN) layer [3] is implemented immediately before the layer equipped with quadratic neurons. Experimental settings. The training process incorporates SGD with a weight decay of 0.0001, a batch size of 128, and a momentum of 0.9. Following the recommendations from [12] , quadratic integration parameters are initialized to zero, and a distinct learning rate is adopted for these parameters. Initially, the learning rate for the quadratic integration matrix is set at 1, and 0.1 for all other parameters. These rates is reduced by a factor of ten at 80 and 120 epochs, with training concluding at 160 epochs. Data augmentation procedures mirror the original protocol: each image is padded by 4 pixels on each side, follows by random cropping of a 32 × 32 section from the padded image or its horizontal flip. For testing, a single view of the original 32 × 32 image is evaluated.Results. Table 2 presents a comparative analysis of CIFAR performance between our Dit-ResNets and the original ResNet model [16] , as well as two adaptations that incorporate quadratic neurons [12, 54] , which have shown superior outcomes among existing approaches with quadratic formats [48, 20, 35, 60, 4] . For the CIFAR-10 dataset, given the extensive benchmarking in prior studies, we conduct ten runs of our Dit-ResNets, reporting the optimal outcome for comparative analysis. For CIFAR-100, we independently execute all models, documenting average test accuracy and variance (mean±std). Our experiments indicate significant performance enhancements on both CIFAR-10 and CIFAR-100 datasets after integrating quadratic neurons into a single layer of ResNet, confirming their efficacy. Furthermore, when comparing a deeper ResNet model with our Dit-ResNet (e.g., ResNet-110 vs. Dit-ResNet-56), our approach not only boosts performance but also reduces training overhead. This suggests that augmenting the model with quadratic integration parameters is more effective than merely increasing network depth-a traditionally favored strategy. In comparison to other models employing quadratic neurons [12, 54] , our Dit-ResNets consistently achieve the highest test accuracy with the fewest parameters. This underscores the efficacy of our approach in leveraging the computational advantages of quadratic neurons. These neurons, as discussed in Section 3, demonstrate enhanced generalization capabilities.Figure 4 : Visualization of some performance results presented in Table 2 (left) and Table 3 (right).Dataset and settings. We extend our investigations to the ImageNet-1K dataset [10] , which comprises 1.28 million training images and 50,000 validation images across 1,000 categories. Our analysis primarily focuses on top-1 accuracy on the validation set. Training is conducted at a resolution of 224 × 224, supplemented by a comprehensive suite of data augmentation and regularization strategies. These include RandAugment [8] , mixup [57] , cutmix [55] , label smoothing [44] , layer scale [47] , random erasing [58] , exponential moving average (EMA) [39] , and stochastic depth [17] . These techniques are inspired by the timm library [50] and methodologies from Touvron et al. [45] . Detailed descriptions of these hyperparameters are provided in Appendix C.Model description. Mirroring our approach with the CIFAR dataset, we integrate quadratic neurons into a single layer of the ConvNeXt model [34] to conserve computational resources. This adaptation results in the creation of three distinct models: Dit-ConvNeXt-T, Dit-ConvNeXt-S, and Dit-ConvNeXt-B. Specific details about the layer replacement are discussed in the ablation study.Results. Table 3 presents a comparative analysis of Dit-ConvNeXts against state-of-the-art models from the existing literature. Compared to the original ConvNeXt counterparts, our Dit-ConvNeXts exhibit an average increase of 0.5% in top-1 accuracy, with only a marginal average increase of 1% rise The expectation of the quadratic term for Gaussian variables can be expressed as follows:E x∼N (µ,Σ) x T Ax = µ T Aµ + tr(AΣ).This equation highlights that the term tr(AΣ) encapsulates the information derived from data correlation. Consequently, Table 4 examines the impact of this term on the performance of Dit-CNNs in tackling complex tasks. The data presented in Table 4 show a notable decline in accuracy for Dit-CNNs when this term is omitted. This underscores the significance of the quadratic neurons within Dit-CNNs, demonstrating their pivotal role in effectively capturing data correlation, which is essential for task performance. To manage computational costs effectively, only three layers within the ConvNeXt architecture have been identified as suitable candidates for the integration of quadratic neurons, as depicted in Figure 5 .We explore the most effective layer for the integration of quadratic neurons and elucidate the rationale behind this choice. Figure 5 indicates that substituting traditional neurons with quadratic neurons in the first layer of the block 3 yields the most significant performance improvement. Additionally, a negative correlation is observed between the model's final accuracy and the accuracy after omitting the quadratic term from Dit-ConvNeXt-T post-training. This correlation underscores the critical influence of quadratic neurons on the model's efficacy. These findings suggest that Dit-CNNs demonstrating enhanced performance are those where quadratic neurons play a crucial role, highlighting the superior generalization capabilities of quadratic neurons compared to traditional neurons. As previously discussed, our Dit-CNNs employ quadratic neurons in a channel-wise manner, which provides a clear biological interpretation, as detailed in Section 4. Meanwhile, employing quadratic neurons on a pixel-wise basis, akin to the quadratic convolution outlined in [20] , [35] , and [60] , leads to a scenario where dendritic integration occurs only when neurons receive synaptic inputs from the same neuron. This setup simplifies interactions between different neurons to a linear approximation, a method that diverges from biological plausibility within the brain. The superior performance achieved through the channel-wise application of quadratic neurons, as evidenced in Table 5 , underscores the efficacy of our model. This result supports the hypothesis that models which more closely mirror brain-like mechanisms tend to exhibit enhanced performance. 
conclusion: 
 In this paper, we first provide a theoretical demonstration of the computational advantages of quadratic neurons in capturing internal correlation within structured data. These neurons model dendritic integration rules observed in biological neurons. Our empirical evaluations using the MNIST and Arabic MNIST datasets for few-shot learning validate our theoretical assertions. Drawing from the biological interpretation of CNNs, we introduce a biologically plausible method to integrate quadratic neurons into CNN architectures, resulting in Dit-CNNs. These Dit-CNNs not only exhibit significant performance enhancements with minimal modifications to their original counterparts but also compete favorably with state-of-the-art models. The potential applicability of our approach to other architectures, such as Deep-MAD [42] , hints at further performance improvements. The promising results of this research underscore the vast potential of brain-inspired models. Given that the quadratic integration rule of neurons could be confined to specific brain areas [15, 28] , future electrophysiological experiments could reveal other neuronal integration rules. Our findings could guide the development of brain-inspired deep neural networks by incorporating various integration rules corresponding to different brain areas in distinct layers. Additionally, how to theoretically analyze these new brain-inspired models will be an important issue. It is our aim to extend our analysis concerning high-order statistical information to explore the generalization error of these innovative brain-inspired models. We hope that our work will stimulate further investigations into brain-inspired models, ultimately contributing to the quest for artificial general intelligence (AGI). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper focuses exclusively on fundamental scientific research and does not address societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: See Reference section. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Theoretical results for quadratic neurons. Our theoretical framework is established on the assumption that training samples are normally distributed, a simplification that might not fully encapsulate the complexity inherent in real-world datasets. Despite this assumption, our models have demonstrated exceptional performance on various tasks including ImageNet-1K. This underscores the potential of quadratic neurons in capturing correlation within more intricate data distributions. Future efforts will focus on developing a more comprehensive theoretical foundation to better understand the mechanisms through which quadratic neurons achieve this capability.Computational cost of quadratic neurons. While our Dendritic integration inspired Convolutional Neural Networks (Dit-CNNs) strategically limit the increase in the number of learnable parameters by selectively incorporating quadratic neurons in a singular layer, this modification undeniably raises the computational complexity, as evidenced by an increase in Floating Point Operations (FLOPs). Nevertheless, given the substantial enhancements our Dit-CNNs contribute to model performance, it warrants further investigation into optimizing the efficiency of quadratic neuron deployment. For instance, drawing inspiration from the inherent sparsity observed in the quadratic coefficients of biological neurons-specifically, the pronounced quadratic interactions among synaptic inputs on the same dendritic branch [29] -it is conceivable to predefine a sparse configuration for the quadratic coefficients within our Dit-CNNs. This approach could potentially reduce computational demands while maintaining performance gains.Consequently,e f (x) 1 + e f (x) p 1 (x) - 1 1 + e f (x) p 2 (x) = 0, ∀x ∈ R d ⇒ ∂L ∂A = 0, ∂L ∂w = 0, ∂L ∂b = 0.The classifier generated by the quadratic neuron is defined as:y model (x) = 1 if f (x) < 0, 2 if f (x) > 0.Therefore,y model (x) = 1 ⇔ y opt (x) = arg max j∈{1,2} p j (x) = 1.Similarly, y model (x) = 2 implies y opt (x) = 2. Hence, y model (x) = y opt (x) for all x in R d , demonstrating the consistency between these two classifiers.To prove the theorem for uniqueness, we first introduce the following preliminary lemmas: Lemma 1. Let M(R d ) denote the Lebesgue σ-algebra on R d . Consider f (x), a Lebesgue measurable function on R d , satisfying:Ω f (x) dµ = 0, ∀Ω ∈ M(R d ).It follows that f (x) = 0 almost everywhere on R d .Proof. To demonstrate this via contradiction, let us assume, without loss of generality, that:µ({x | f (x) > 0}) > 0. Since 0 < µ({x | f (x) > 0}) = µ( +∞ n=1 {x | f (x) > 1 n }) ≤ +∞ n=1 µ({x | f (x) > 1 n }),then we know there existn 0 ∈ Z * such that µ({x | f (x) > 1 n0 }) > 0. If we set Ω 0 = {x | f (x) > 1 n0 } ∈ M(R d ),then by the property of f (x) we can derive:0 = Ω0 f (x) dµ > µ({x | f (x) > 1 n0 }) n 0 > 0.Hence, we have reached a contradiction.Lemma 2. Assume f (x) = x T Ax + w • x + b, a Lebesgue measurable function, equals zero almost everywhere (f (x) = 0 a.e.), with A being a symmetric matrix. It then follows that A = 0, w = 0, and b = 0.Proof. Since f (x) is continuous, we know f (x) = 0 for ∀x ∈ R d . Then we have: b = f (0) = 0. Setting x = εw for some ε > 0, we obtain: ∥w∥ 2 2 + εw T Aw = 0, letting ε go to zero, we have:∥w∥ 2 2 = 0 ⇒ w = 0. Having established that f (x) = x T Ax, consider x = e i, where e i is the unit vector with a value of 1 at the i-th position and 0s elsewhere. Evaluating the function at this vector, we have f (e i ) = e T i Ae i = a ii = 0. Subsequently, by letting x = e i +e j , where e j is another unit vector corresponding to the j-th position, we obtain the following equation:a ii + a ij + a ji + a jj = 0. Since a ii = a jj = 0 and a ij = a ji (A is a symmetric matrix), we know:a ij = 0, ∀i, j ∈ {1, 2, . . . , d} ⇒ A = 0.Theorem 5. (Existence) The critical points with respect to the cross-entropy loss L(θ) are given as follows:A * j = Σ -1 j , w * j = -2Σ -1 j µ j , b * = µ T j Σ -1 j µ j + log(|Σ j |), j ∈ [k], whereL(θ) = 1 k k j=1 E x∼classj   log   1 + k i=1,i̸ =j e fi(x)-fj (x)     .Moreover, the corresponding classier generated by this formula is the same as the theoretically optimal classifier as defined in Equation (6) .Proof. The gradients can be computed explicitly as follows:∂L ∂A j = 1 k R d   k i=1,i̸ =j e fj (x) k i=1 e fi(x) p i (x) - k i=1,i̸ =j e fi(x) k i=1 e fi(x) p j (x)   x • x T dµ, j ∈ [k],and similarly for ∂L ∂wj and ∂L ∂bj that can be calculated. From the derivation outlined in the proof of Theorem 3 and the choice for critical points as followsA * j = Σ -1 j , w * j = -2Σ -1 j µ j , b * = µ T j Σ -1 j µ j + log(|Σ j |), j ∈ [k], then we have e fj (x)-fi(x) = p j (x) p i (x) , ∀x ∈ R d , ∀i, j ∈ [k],which is equivalent toe fj (x) k i=1 e fi(x) p i (x) = e fi(x) k i=1 e fi(x) p j (x), ∀x ∈ R d , ∀i, j ∈ [k].Thus, we can derive:∂L ∂A j = 0, ∂L ∂w j = 0, ∂L ∂b j = 0, ∀j ∈ [k].The classifier generated by quadratic neurons is defined as:y model (x) = arg max j∈[k] f j (x),Therefore,y model (x) = j ⇔ f j (x) > f i (x), ∀i ∈ [k]/{j} ⇔ y opt (x) = j.Hence, y model (x) = y opt (x) for all x in R d , demonstrating the consistency between these two classifiers.Theorem 5 demonstrates that in multi-class classification tasks, each neuron captures data correlation by directly relating to the covariance matrix of their respective class. This concept is empirically validated through numerical experiments on the MNIST dataset, as depicted in Figure 6 . The eigenvectors associated with the largest eigenvalues of the covariance matrices, representing the principal components of each class's distribution, reveal the spatial correlation within that class's data.The marked similarity between these eigenvectors and the eigenvectors of the quadratic integration matrices of neurons confirms that quadratic neurons effectively capture correlation from the data distribution. This intrinsic ability of quadratic neurons to capture correlations provides a possible explanation for the superior generalization capability of quadratic neurons.The training settings for our Dit-ConvNeXt models on ImageNet-1K are detailed in Table 6 . While all Dit-ConvNeXt variants adhere to a unified configuration, certain parameters-namely, the stochastic depth rate, learning rate, and weight decay-are individually customized for each model variant. These hyperparameters were optimized based on performance outcomes, utilizing a grid search over potential learning rates from the set {0.004, 0.006, 0.008} and weight decay values options within {0.04, 0.06, 0.08}. [8] rand-m9-mstd0.5 rand-m9-mstd0.5 rand-m9-mstd0.5 Mixup [57] 0.8 0.8 0.8 Cutmix [55] 1.0 1.0 1.0 Random erasing [58] 0.25 0.25 0.25 Label smoothing [44] 0.1 0.1 0.1 Stochastic depth [17] 0.1 0.4 0.4 Layer scale [47] 1e-6 1e-6 1e-6 Exp. mov. avg. (EMA) [39] 0.9998 0.9998 0.9998NeurIPS Paper ChecklistJustification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).A can always be considered a symmetric matrix, attributable to the quadratic form present in f (x). 


Decision: While the focus in the field has largely centered around building ever more complex and larger architectures of simple individual units, there is a lot of complexity and computation happening at the level of individual neurons in biological circuits. This paper focuses on a relatively understudied issue of where/how to incorporate some of the biophysical apparatus of individual neurons into neural networks. In this sense, I think that this paper contributes to the diversity of efforts in the field and it is useful to stimulate different approaches.

That said, I read the paper and responses and the paper is not very clearly written. The paper could benefit from editing by native speakers or further attention to grammar, prose, and exposition clarity. 

The authors sent a note to the Area Chairs, expressing concern about reviewer opWC. I read the exchange of author responses and reviewer questions. While it is true that the reviewer only succinctly responded to the authors’ response, I do not see any inappropriate action from the reviewer. It is clear to me that opWC read the paper and provided useful critical feedback. Could opWC engage in a longer discussion of each of the rebuttal points? Probably. But overall, I feel that this is a rather reasonable review. 

In all, I share many of the positive points noted by reviewers and I think that this work is different from others and deserves to be accepted, even though the overall discussion is borderline.

Reviews: review: paper_summary: paper_summary: The paper theoretically demonstrate that quadratic neurons inspired from dendritic computing inherently capture correlation within structured data. The quadratic rules are integrated with convolution networks and established the so-called Dit-CNNs. Experiments on CIFAR and ImageNet datasets demonstrates competitive performance of the proposed model.
strength_weakness: strength_weakness: 1. The paper demonstrates that there exist critical points for the parameters of quadratic neuron, however, there lacks experimental proof that the training process actually drives these parameters towards the critical points. It would be more convincing if the author could provide such evidence, for datasets with different scales.
2. Compared to previous quadratic dendritic methods, the improvement in accuracy of the proposed model is not so significant, I suggest that the author provide comparisons on computation cost to further demonstrate the advantage of the proposed model.
questions: questions: 1. In the ablation study 4.3.1, table 4, why performance dropping level are significantly different for CIFAR and ImageNet?
limitations: limitations: See weaknesses and questions
paper_summary: paper_summary: This paper explores the computational benefits of quadratic neurons, which are inspired by the quadratic integration rules of dendrites. The authors first present the theoretical analysis on binary classification for normal distributions, showing the existence and uniqueness of the solution with a single quadratic neuron. Then, a few-shot learning experiment on MNIST and Arabic MNIST is conducted to show the better performance of quadratic neurons under few-shot training samples. Finally, this paper integrates quadratic neurons into CNNs along the channel dimension, and demonstrates the superior performance of the model on several datasets including CIFAR and ImageNet.
strength_weakness: strength_weakness: 1. The presentation and organization of the paper are poor and loose. There are also many informal claims without rigorous justification.

(1.1) The link between the theoretical analysis, the so-called “enhanced generalization capabilities”, the few-shot learning advantage, and the integration into CNNs is poor. I do not see that the theoretical analysis of the existence and uniqueness of the solution of a single quadratic neuron under a simplified setting provides effective insight or explanation for the following practice. There is no analysis of deep networks. There is no explanation for the connection between the practice of CNNs and theories.

(1.2) In the discussion of Theorem 1 and 2, it is said using the gradient descent algorithm. However, there is no analysis for training dynamics or learnability in current Theorems.

(1.3) There is no formal definition or rigorous justification for the so-called “superior generalization capability over traditional neuron”.

(1.4) It is claimed that the integration of quadratic neurons into CNNs along the channel dimension is “biologically plausible”. However, I do not see any detailed justification. Indeed, only considering quadratic integration along the channel dimension does not correspond to synaptic connections between neurons.

2. The novelty and significance of this paper are not clear enough. Quadratic neurons have been studied in many previous works, both from theoretical and practical perspectives, e.g., [1] has shown strong results. While this paper interprets it as inspiration from biological neurons, the implementation does not correspond to the biological computation form. So there should be more differentiation with existing quadratic neuron works. Considering Table 1, the biological interpretation in this paper is not strong enough, and there are also theories in previous works [1]. Only changing how the quadratic operation is used makes little contribution.

3. There is no formal definition and analysis showing “enhance data correlation” in the title.

[1] Fan, F. L., Dong, H. C., Wu, Z., Ruan, L., Zeng, T., Cui, Y., & Liao, J. X. (2023). One neuron saved is one neuron earned: On parametric efficiency of quadratic networks. arXiv preprint arXiv:2303.06316.
questions: questions: Will incorporating quadratic neurons into multiple layers continually increase the performance? How is the suitable layer candidate identified?
limitations: limitations: The authors discussed limitations in Appendix D.
paper_summary: paper_summary: This is an interesting paper looking at quadratic neurons and how they impact performance and/or learning rate of ANNs. The quadratic integration is loosely linked to dendritic integration properties of pyramidal neurons in cortex (though it is unclear whether any real resemblence should be granted). This increase in the degree of nonlinearity , when implemented in CNNS (called Dit-CNNs) works to the advantage on benchmarks like imagenet while also retaining the simplicity of conventional CNNs. Of interest and importance are the elegant anaytical solutions exemplified in figure 1 as well as the ablation study shown in table 4. Overall this is an interesting and timely study.
strength_weakness: strength_weakness: The authors use the term "dendritic integration" loosely without exploring various ways that such integration can occur (sub- vs. supra-linear). It would be interesting to consider such cases.
questions: questions: could a potentially more nuanced or bio-plausible implementation of dendritic integration further enhance performance? How would that affect the quadratic integration?
limitations: limitations: it is left somewhat unclear how the dendritic integration along different compartments of a cell and its result in the neuron's output is reflected in quadratic neurons. It would be interesting to look at sub- vs. supra-linear integration effects along the same lines.
paper_summary: paper_summary: The paper introduces a new biologically inspired neural network architecture. Rather than using linear layers followed by a nonlinear activation function, the authors propose a quadratic model instead in the inputs. This form is said to explicitly model the covariance between input features offering better accuracy and generalization.
strength_weakness: strength_weakness: The relation to biological plausibility is unclear and the link is tenuous.
questions: questions: Have the authors look into the structured learned by the quadratic term? 
How do the CNN filters differ from those previously learned? 
Did the authors compare to other smoother activation functions such as gelu?
limitations: limitations: The limitations are not outlined in the main text but are in the appendix instead.
It would be beneficial to include these in the main text.
paper_summary: paper_summary: This study explores incorporating channel-wise quadratic neuron, as inspired by dendritic nonlinearity, into artifical neural networks to improve model performance. These models show competitive performance on datasets like CIFAR, ImageNet-1K while maintaining simplicity and efficiency. The theoretical and experimental results highlight the potential of quadratic neurons in enhancing model performance.
strength_weakness: strength_weakness: The theortical explaination is limited to highly simplified setting. 
It is not clear if the emperical analysis on imagenet part is a fair one, give quite sophicated data augumentation is used for their models.
questions: questions: See above.
limitations: limitations: Yes.

Comments: comment: Thank you very much for your thoughtful feedback and for taking the time to review our rebuttal. We would like to further clarify your concerns.

Regarding analysis, our theoretical proof demonstrates a clear computational advantage of quadratic neurons: they capture second-order information, which distinguishes them from traditional neurons. Following our observation of a significant improvement in test accuracy in more complex scenarios, we have confirmed that quadratic neurons effectively capture second-order information in these contexts. So the capability of capturing second-order information likely plays an important role in small generalization errors observed in our models.

Regarding biological plausibility, we had considered the quadratic interactions between convolutions and encountered training challenges, such as gradient explosion. Alternatives such as considering both spatial-wise and channel-wise quadratic interactions, may significantly increase the computational cost (e.g.,  a $3\times 3$ convolution with both spatial-wise and channel-wise quadratic interactions leads to a 81 times increase in FLOPs compared to our Dit-CNN). Therefore, we proposed Dit-CNN, which achieves good performance while maintaining practical computational costs. We believe we are the very first to explore quadratic methods based on the dendritic bilinear integration rule. And Our Dit-CNN is inspired by the visual pathways in the neural system, as illustrated in Figure 1.C of the rebuttal materials (a modification with good performance and low computation cost ).

We hope these explanations address your concerns. Once again, thank you for your patience and for raising such constructive points.
We are delighted to see that our clarification and rebuttal were well-received, and we appreciate the increase in the score. Thank you once again for your careful review and valuable recommendations.
I have read the rebuttals. I have updated my score.
I would like to thank the authors for their detailed responses. I acknowledge that the empirical results (efficiency, scaling property, etc.) are good, which I have listed as strengths, while my concerns remain that the theoretical presentation and claims for biological plausibility are not satisfying. Considering the theoretical part, the authors emphasize that they focus on generalization. However, there is a large gap between "capture second-order information (correlation)" and generalization in general settings, and there is no formal analysis for generalization error. Considering biological plausibility, the quadratic term in Dit-CNN is element-wise spatially rather than considering convolution with receptive field, which means some synapses are used for quadratic integration while some are only summed. This discrepancy is still not explained. I understand that these parts may not be actually necessary for brain-inspired algorithms with good performance, but if the authors claim this as an important part of the contribution, I think the current presentation is not complete and satisfying. So I keep my score for the current version of the paper.