Convolution Goes Higher-Order:  A Biologically Inspired Mechanism Empowers Image Classification
Authors: Simone Azeglio
Olivier Marre
Peter Neri
Ulisse Ferrari
Venue: NeurIPS 2025 poster ()
Keywords: Higher-order convolution
Biologically inspired neural networks
Image classification
Convolutional Neural Networks
Biological visual processing
Neural representations
Primary area: deep_learning
TL;DR: We present a new image classification model that extends CNNs with biologically-inspired higher-order convolutions. Outperforms standard CNNs on benchmarks and shows unique representational properties, bridging neuroscience and deep learning.

Abstract: We propose a novel enhancement to Convolutional Neural Networks (CNNs) by incorporating learnable higher-order convolutions inspired by nonlinear biological visual processing. Our model extends the classical convolution operator using a Volterra-like expansion to capture multiplicative interactions observed in biological vision. Through extensive evaluation on standard benchmarks and synthetic datasets, we demonstrate that our architecture consistently outperforms traditional CNN baselines, achieving optimal performance with 3rd/4th order expansions. Systematic perturbation analysis and Representational Similarity Analysis reveal that different orders of convolution process distinct aspects of visual information, aligning with the statistical properties of natural images. This biologically-inspired approach offers both improved performance and deeper insights into visual information processing.

Convolution Goes Higher-Order: A Biologically Inspired Mechanism Empowers Image Classification
Simone Azeglio
Institut de la Vision & Laboratoire des Systèmes Perceptif's INSERM, CNRS, Sorbonne University & École Normale Supérieure PSL 17 Rue Moreau, Paris 75012& 29 Rue d' d'Ulm Paris 75005 s simone. azegl lo@gmail.ccom com
Olivier Marre
Institut de Vision
INSERM, CNRS, Sorbonne University 17 Rue Moreau, Paris 75012
Peter Neri
CHT Erzelli & Laboratoire des Systèmes Perceptifs IIT & École Normale Supérieure PSL Via Enrico Melen 83, Genova 16152 & 29 Rue d' d'Ulm Paris 75005
Ulisse Ferrari
Institut de Vision
INSERM, CNRS, Sorbonne University 17 Rue Moreau, Paris 75012
Abstract
We propose a novel enhancement to Convolutional Neural Networks (CNNs) by incorporating learnable higher-order convolutions inspired by nonlinear biological visual processing. Our model extends the classical convolution operator using a Volterra-like expansion to capture multiplicative interactions observed in biologi- cal vision. Through extensive evaluation on standard benchmarks and synthetic datasets, demonstrate that architecture consistently outperforms traditional CNN baselines, achieving optimal performance with 3h order expansions. Sys- tematic perturbation analysis and Representational Similarity Analysis reveal that different orders of convolution process distinct aspects of visual information, align- ing with the statistical properties of natural images. This caalooicall--issireee approach offers both improved performance and deeper insights into visual infor- mation processing.
Introduction
"The correlation over space translates into correlation over order." Koenderink and Doom
Convolutional Neural Networks (CNNs) have become the cornerstone modern computer vision tasks, demonstrating remarkable performance across wide range of applications [2), (3). [4]. The success of CNNs is largely attributed to their ability to capture local patterns and hierarchical features
in images through their layered structure and weight sharing mechanism [5], [
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, natural images contain complex correlations and higher-order statistics that extend beyond simple linear relationships 77 8 9 These include texture patterns, edge interactions, and intricate geometric structures that are prevalent in real-world visual scenes. Importantly, these correlations exhibit hierarchical structure, with higher-order correlations becoming increasingly sparse [- Traditional CNNs, particularly those with limited depth, often struggle to effectively exploit these higher-order correlations [0], M. [12]. The limitation of standard CNNs in capturing higher-order correlations stems from their reliance on linear convolutions followed by pointwise nonlinearities 13. Although deeper networks can approximate complex functions through the composition of many layers, this approach may be computationally expensive and require large amounts of training data. To address this limitation, we propose a learnable extension of the classical linear convolution
operator incorporate higher-order terms, akin to Volterra expansion [14] [1]]. This approach allows for the direct modeling of multiplicative interactions hetween neighboring pixels, enabling the
network to capture complex local structures more effectively, even in shallow architectures.
We design models by incorporating higher-order convolutions, and compare their performance against standard CNN basclines. We evaluated our models range of datasets, beginning with synthetic images typically used to assess sensitivity to higher-order correlations. Subsequently, we extended our experiments to compare our model against CNN baselines on widely-used datasets including MNIST, MaSSIT CIFARIO, CIFAR 100, Imagenette and Imagenet. Our findings reveal that these higher-order convolutions improve image classification performance, offering promising direction for advancing deep learning models in computer vision tasks.
Interestingly, our results show optimal performance at the 3rd or 4th order, aligning remarkably with the distribution of pixel intensities in natural images as analyzed by Koenderink and van Doorn D and Tkačik et al. [16]. This alignment suggests that our approach effectively captures the fundamental structure of visual information in the natural world. Through systematic perturbation analysis, we isolate the contributions of specific image statistics to model performance, providing insight into how different orders of convolution process distinct aspects of visual information. Furthermore, we employ Representational Similarity Analysis (RSA) 17 to investigate the internal representations learned by our model compared with standard CNNs, revealing distinct geometries that indicate different strategies for encoding and processing visual information.
Our approach addresses the limitations of simple pointwise nonlinearities in capturing local image structure, especially in networks of limited depth or width. In particular, we show that incorporating such computations at the earliest stages of image processing, even within the first layer of a neural network, yields significant benefits. This finding suggests that models benefit from embedding these computations via direct engagement with the superficial structure of images [88
1.1 Biological Inspiration and Related Work
Our work draws inspiration from biological visual systems, where processing extends beyond simple
pointwise nonlinearities and first-order convolutions [99 20 21 22 23 These non non-pointwise
nonlinearities enable more effective extraction and integration of spatial information (24 25 26 27].
beginning in the retina (2 and continuing in the visual cortex where specialized neurons respond to complex features like corners line endings (28).
In computational research, the concept of nonlinear receptive fields has been explored through various approaches. Berkes and Wiskott [29] analyzed inhomogeneous quadratic forms as receptive fields, while Li et al. (30] provided theoretical insights on Volterra-based convolutions in neural networks. Zoumpourlis et al. (31] implemented Volterra-inspire convolutions primarily in the first layer of Wide 28s but failed to demonstrate significant performance improvements and limited their analysis to second-order expansions. In contrast, and sy systematically analyzing expansions up to the 4th order, where we demonstrate that 3dh/4th order expansions yield optimal performance-22 a finding we uniquely connect to the statistical properties of natural images documented by Koenderink and van Doomn [
work extends this foundation by implementing
For video classification, Roheda et al. 32 introduced Volterra Neural Networks using cascaded 2nd filters with low-rank approximations for spatiotemporal tasks, creating separate spatial and temporal streams that are later fused. Our approach differs fundamentally, as we focus on static image classification with direct implementation of higher-order terms (up to 4th order) within the convolution operation itself, preserving parameter independence across orders for more flexible feature learning.

---

While Roheda et al. [32 emphasize computational efficiency through approximations, our work provides deeper insights through systematic perturbation analysis and representational similarity analysis that reveals how different orders process distinct aspects of visual information. Related developments pre-grained image classification include bilinear CNN models 33 and compact
bilinear pooling (34, with applications extending to computational neuroscience
35
Our approach extends and complements these previous higher-order methods by embedding higher- order operations directly within the convolutional operator, creating independent feature maps for each order that are combined before nonlinearities. This design choice enables the network to learn and apply complex nonlinear transformations that better constitute an algorithmic implementation of biological visual processing. Through our structured perturbation analysis, representation geometry analysis, and controlled texture experiments, we provide novel insights that beyond performance improvements, demonstrating why and how higher-order processing benefits visual representation.
1.2 Beyond Pointwise Nonlinearities
Traditional convolutional neural networks typically employ pointwise nonlinearities with weighted sums of inputs as arguments. Expanding these nonlinearities as a polynomial series reveals that different orders of the expansion share the same weights, effectively coupling the terms across orders and leading to inter- and extra-order dependencies:
o
Σ 1=0
W
se ao + (1 + (2222) + 02( + w2
000 + 1liil + w2 + arwi 2x + a2w2 2 +
This tied-weight issue becomes particularly problematic when the network is not deep enough or wide enough to compensate, as the precise bounds for enough depth or width depend on the universal approximation theorem and often challenging to determine exactly [36). Consequently, the standard pointwise nonlinearity with linear summation may fail to capture complex relationships in the data, especially in shallower narrower networks. To address this limitation, we propose a novel approach that can be viewed as an implementation of non-pointwise nonlinearities. Our method, which will be described in detail in Section 2 can be detector model (25) representing receptive field that responds to multiplicative interactions between inputs, similar how some biological neurons process visual information (see Subsection I.I). This approach offers a solution for capturing complex correlations at the level of single mechanism, allowing for more flexible and powerful representations even in compact network architectures.
learnable generalized 2-1)
Higher-Order Convolution
To perform more complex computations while preserving the benefits of locality and weight sharing that are inherent in standard convolutions, we propose the concept of higher-order convolution. This approach extends the traditional convolution operator to include higher-order terms, allowing for more sophisticated feature extraction.
In order to explain our higher-order convolutional layer, we start by considering an input image patch P with elements, reshaped filtering, relating input x and output as follows: y(x) + E w o where wi / == 1,..., n represents the weights of the convolution kernel. We expand this function to include quadratic, cubic, and higher-order terms. The general form of this expansion is:
vector x. Standard convolution can be expressed in terms of linear
v(x) b b+
Σ wis:
Wi
Aik W3 TyryaR +
Extending this to the entire image, express the higher-order convolution operation as:
3

---

1st Order
Image
a
+
Fistten
2nd Order
u
vi
2 x 2 Patch
05
102
Cumulative Explained Variance
+
3rd Order
CNN HoCNN 2nd
0.8
HoCNN
0.54
10
Example Image
18
0.4
10
12
of PCs
Figure I: Extending classical convolution (A) Implementation of higher-order convolution: Input patch flattened vector x. 1 order uses classical convolution with weight-sharing. 2nd and 3rd orders compute outer products respectively one and two times of x with itself and dot product with the corresponding weights. Feature maps are then summed before ReLU activation.
(B) Cumulative explained variance of principal components for standard CNN model vs Higher-ordes CNN models with 2nd and 3rd expansion quantitatively confirms the bee-weighht issue for classical models. Inset: Example of the 32x32 binary texture image used in this analysis, containing all possible 1-, 2-, 3-, and 4-point correlations for 2x2 patches.
Vim = b+
Σ 1.
wo X + ΣΣ
jjhch X Ne+imm+ Xitkmmth
4J
R.A
ijkhpa wi
Xi4i, X14k Xitnntttttttttttt
k.
Here, Yim represents the output at position (1, m) in the feature map, X is the input image, and wi- W2 W3 are the learnable weights for the first, second, and third-order terms respectively. The sums over i, k. h. run over the dimensions of the convolution kernels for each order.
By symmetry considerations, the total number of parameters in this expansion grows as: (x+p)
ntp!
where p represents the order of the term and the size of the kernel. To address potential issues with the relative magnitudes kernels of order greater than the 1s, we apply scaling factor calculated as vRy The scaling factor helps balance the contribution of higher-order terms relative the first-order (linear) terms, promoting stability and preventing higher-order terms from dominating network behavior during training. Figure A illustrates this concept for binary image, showing how the classical convolution is extended to include higher-order terms for single patch. In practice, this operation is applied across entire image, resulting in feature map for each order of expansion, which are then summed together before standard pointwise nonlinearity as ReLU (see for example, Figure B). This layer can functionally replace standard convolutional layer in neural network architectures and is compatible with training through backpropagation.
higher-order terms, we implement a normalization strategy. For

---

2.1 A Quantitative Perspective on the Tied-Weight Issue
To demonstrate how a Higher-order CNN (HoCNN) addresses the tied-weight issue introduced in Sec- tion 1.2 we conducted a systematic analysis of weight independence in the network's representations. We generated a fixed input image (32 x 32 pixels) containing binary textures exhibiting all possible 1-, 2-, and 4-point correlations for 2x2 patches (see inset in Figure BB), corresponding to fixing the z; terms in Equation| 1 We then randomly initialized the same model architecture 10,000 times and analyzed the activations after the convolutional block (Conv/HoConv + BatchNorm ReLU + Max Pooling) for two architectures: CNN with 10 kernels of size 2x2 and a HoCNN with 2 kernels of size 2x2 with higher- order expansions.
To quantify the degree of weight independence, we employed Principal Component Analysis of these activations, measuring the number of principal components (PCs) needed explain 95% of the variance. Our hypothesis was that tied weights would reduce the number of principal components, while greater weight independence would require more components to capture the same variance. The results confirmed our expectations (see Figure [ (87 out of 9610) to explain 95% of the variance, while the HoCNN required 5.3% of PCs (102 out of 1922) with second-order expansion and 8.3 % (159 out of 1922) with third-order expansion. While the CNN has more total components due to its higher channel count (5x), additional analyses with matched channel counts yielded similar results. These findings quantitatively demonstrate that higher-orde convolutions introduce more independent weights, leading to a richer representation space with reduced weight coupling. The increasing number of components needed for variance explanation directly shows how our approach mitigates the tied weight issue inherent in standard CNNs. Additional analyses with various nonlinearities showed consistent results, further supporting these conclusions (see Appendix A..TT.
B): the CNN required only 0.9% of PCs
Experiments & Results
We evaluated our model on a synthetic texture dataset with higher-order correlations and multiple standard benchmarks: MNIST, FashionMNIST, CIFAR- 10, CIFAR-100, and ImageNet. Additional results on Imagenette and fine-grained classification (CUB-200-201)) are provided in Appendix A.6 and A.7
3.1 Synthetic Dataset: Structured Visual Textures
To test our model's sensitivity to higher-order correlations, we generated a synthetic dataset of structured visual textures based on the work of Victor and Conte [37] (see Figure These textures allow precise control over the type and intensity of contained correlations. To generate the textures, we utilized a custom software library (38) implementing the method developed by Victor and Conte [37] This process involves sampling from a distribution of binary textures with specific multipoint correlation probabilities while maximizing entropy. The correlation intensity is parameterized by parity counts of white or black pixels within 1-, 2-, 3-, or 4-pixel tiles (termed gliders, Figure 2). Notably, two-point and three-point gliders create distinct correlations through different spatial configurations: horizontal, vertical, or oblique for two- -oo-ponnt, and L patterns with various orientations for three-point correlations. Our synthetic dataset comprises 2000 training images, 1000 validation images, and 2000 testing images. All reported results are based on the test set.
We framed the problem as image classification task with 10 different classes corresponding to various N-point correlations (N ranging from to 4). To establish a baseline, we first tested a CNN model consisting of 3 blocks (2 convolutional 1st conv layer with 10, 2x2 kernels and 2nd conv layer with 2, 8x2 kernels; both followed by batch normalization, ReLu nonlincarity and max pooling and I fully connected layer). Our results indicate (see Table [ and Figure 2 that this basic architecture fails to correctly discriminate among the different 2- 2-point and 3-point correlations. We then repeated the classification task using three different HoCNNs with expansions up to the 2nd the 3rd and the 4th order kernels (2 kernels of size 2x2) and an additional convolutional (conv layer with 2 kernels of size 8x2 + batch norm. + ReLU + pooling) and fully connected layer replicating the structure of the baseline CNN. These networks produced increasingly good performance [Table and Figure 2 highlighting the need for our proposed higher-order convolution approach. For completeness, we report the number of parameters of the different architectures in Table
5

---

Table 1: CNN and HoCNN performances on Texture classification
Model
Accuracy (%)
Params
CNN (baseline)
59.14
492
HoCNN (up to 2nd order)
82.42
293
HoCNN (up to 3rd order)
89.02
488
HoCNN (up to 4th order)
92.32
1259
1-point (y)
2pponn (9)
Baseline CNN
HoCNN (2* order)
3 40 4 1g
3-point
4ppint
HoCNN (3M arder)
order)
Actual Class
PI p-- Pi
A] P- Al B 6F 9h.
Figure 2: Multipoint correlations and glider classification (A) Textures generated with N-point
gliders (N ranging from to 4), totaling classes when taking into account parity constraints. (B) Confusion matrices for different models, from top left: baseline CNN; top right: Higher-order CNN (HoCNN) with kernels expanded up to the 2nd order; bottom-left: HoCNN with kernels expanded up the 3rd order; bottom-right: HoCNN with kernels expanded up the 4th order. Taken together the four confusion matrices show that higher-orders progressively allow our network to properly capture relevant features for image classification.
3.2 Benchmark Datasets
MNIST, FashionMNIST, CIFAR-10, CIFAR-100
To further validate our approach and ensure its generalizability, we conducted extensive experiments on four well-established benchmark datasets: MNIST, FashionMNIST, CIFAR- 10. and CIFAR-100. We focused on these relatively small datasets to allow for comprehensive testing and analysis. To ensure the robustness of our results and account for the inherent variability in neural network training, we conducted multiple independent runs for each experiment. Specifically, we initialized and trained each model with 50 different random seeds. This approach provides model performance by mitigating the effects of random initializations.
reliable estimate of
We compared our HoCNN with 3rd order kernels (3x3) against standard CNN baseline with equivalent kernel size. Notably, extending to 4th order kernels showed no significant improvement in test accuracy. Complete architectural and training details are provided in Appendix A.9 with parameter comparisons in Table Test accuracy results, averaged over 50 realizations (Figure and Table 2), demonstrate consistent performance advantages for HoCNN across all datasets, with particularly notable improvements on the more complex CIFAR-10 and CIFAR-IO1 tasks. These results suggest fundamentally different representational capabilities in HoCNN (further explored in Section 4). Additional experiments with a deeper and more complex CNN architecture, detailed in Appendix A.2, showed inferior performance compared efficiency of higher-order feature extraction.
simpler HoCNN, highlighting the
Training Details. For all experiments, we used AdamW optimizer 39 with learning rate 0.001, weight decay 5e-4, batch size 64, cross-entropy loss and early stopping. Images were normalized using z-score standardization without data augmentation. Experiments used an NVIDIA RTX 4080

---

Input Images
A
B
Input Images
HoConv order
HoConv n order
HoConv 3 order
Conv
HoConv Block
Conv Block
Batch Norm.
Batch Norm.
ReLU
ReLU
Pooling
Pooling
Predictions
Predictions
Figure (A) Comparison of standard convolution (Conv) and (B) higher-order convolution (HoConv) blocks.
Table Accuracy (with a, standard deviation over multiple initializations) of CNN and HoCNN models across image classification benchmarks. MNIST, FashionMNIST, CIFAR-10, CIFAR-100: 50 runs; ImageNet: 5 runs.
Dataset
Accuracy (%) +
Params
CNN
HoCNN
CNN
НoCNN
MNIST
99.13 + 0.09
99.30 + 0.07
82,706
80,262
PaSHIonMNIST
90.48 0.32
90.95 + 0.35
82,706
80,262
CIFAR-10
69.76 0.64
72.87 + 0.54
82,706
80,262
CIFAR-100
36.92 0.71
40.42 + 0.84
385,006
383,478
Imagenet
68.62 + 0.08
69.47 + 0.09
11,689,512
11,676,252
Ti GPU, with training times of 10-15 minutes for MNISTTFashionMNIIST and 20-30 minutes for CIFAR-10/CIFAL 100 (HoCNN requiring 1.5-2x longer). Complexity analysis is in Appendix A.2
IMAGENET
To investigate the scalability and real-world applicability of our approach, we implemented Higher- order Convolution in R8 4 architecture (HoResNet-18) and evaluated it on ImageNet [40). 5 independent training runs with different random seeds. HoResNet- 18 achieved 69.47% + 0.09% test accuracy compared to ResNet- 18's 68.62% + 0.08%, demonstrating a statistically significant improvement of 0.85% (see Table 2) Results on Imagenette and fine-grained classification (CUB-200-200) 1) are reported in Appendix A.6 and A.7
To ensure statistical robustness, we conducted
HoResNet-18 18 maintains the standard ResNet- 18 structure, but uses higher-order residual blocks in its first stage and standard blocks thereafter. This hybrid design preserves ResNet's architecture while incorporating higher-order convolutions. Parameter counts are shown in Table 2 with architectural details in Appendix A.9.1 These results provide strong evidence for the effiaayy of our higher- order convolution approach, demonstrating successful scaling to deeper architectures and large-scale datasets. To further investigate scalability, conducted extensive experiments with deeper ResNet architectures (ResNet-34 and ResNet-50), confirming that performance improvements persist across model depths (see detailed results in Appendix A.4).
Training Details. For ImageNet, we used standard practices: normalization with mean [0.485, 0.456, 0.406] and standard deviation = [0.229, 0.224, 0.225]: Stochastic Gradient Descent with momentum 0.9 as optimizer. We conducted 5 independent runs with different random seeds ensure statistical robustness. The ImageNet experiments were run on an NVIDIA A100 GPU, requiring approximately 2-3 days per run. The detailed analysis computational requirements and FLOOP measurements for different orders of convolution can be found in Appendix A.2
7

---

4 Image Statistics Sensitivity and Neural Representations
To understand how our Higher-order Convolutional layer (HoConv) processes information differently from standard Convolutional (Conv) layers, we conducted two complementary analyses: an investiga- tion of sensitivity image statisties through perturbations (akin adversarial attacks) and study of representational geometry.
HoCNN's sensitivity to higher-order image statisties has been quantified by conducting systematic perturbation analysis on the CIFAR-10 test dataset using both architectures (50 pretrained models each), while keeping the network trained on the unperturbed images. We generated synthetic textures (1-, 2-, 3-, and 4-point) and interpolated them with test images at different intensity (I) levels (imgperturbed
with varying orders of statistical correlations
(l-)xxmgcIFFAR-IU +Ix perttrrrrraatioo with E [0.05, 0.20)). While HoCNN achieves superior performance on CIFAR-10 10 test set images (see Table 2}. it shows increased vulnerability to higher-order statistical perturbations. The perfor- mance degradation becomes progressively more pronounced as we move from lower to higher-order correlations, with the gap widening at higher perturbation intensities (see Figure A and B for I= 0.12). For instance, with third-order perturbations at I = 0.12, HoCNN's accuracy drops by 78.0% compared to CNN's 73.7% (complete results in Appendix this analysis by showing how HoCNNs are tipically more robust than CNNs for standard corruptions in CIFAR- -ECAR-OCCAAR and CIFAR- 100C [41]. This enhanced sensitivity to statistical perturbations suggests HoCNN's increased capacity to leverage higher- patterns. To better understand how this statisti- cal sensitivity translates into improved classification performance, we examined the model's 's internal representations using Representational Similarity Analysis (RSA) [17 on CIFAR-10 10, analyzing activations from 100 test images (10 per class) averaged across 50 model realizations. The Cony and HoConv blocks exhibit distinet representational geometries (Figure 4 & B), with the latter showing more pronounced class-specific patterns. Analysis of pairwise distance distributions (Figure 4D) reveals that HoConv representations are significantly more dispersed in the high-dimensional space, indicating its ability to capture more diverse and discriminative features. Detailed analysis of individual order components and layer-wise correlations is presented in Appendix
In Appendix [A.5] we complement
(
Together, these analyses demonstrate that HoCNN's superior classification performance stems from its ability to capture and leverage higher-order statistical features in natural images, resulting in richer and more discriminative internal representations.
5 Discussion and Limitations
Our study introduces a novel approach to image classification by extending convolutional neural networks with higher-order learnable convolutions, inspired by complex nonlinear biological visual processing. Our findings indicate that expansion beyond the 4th order is unnecessary, a conclusion supported by both theoretical insights and empirical evidence. Koenderink and van Doorn MI's analysis of natural image statistics reveals that the quadratic, cubic and quartic power dominates approximately 63%, 35%, and 2% of the pixels, respectively. Our experimental results align with this distribution, showing no significant performance gains beyond the 3rd order in benchmark tasks and the 4th in structured visual textures. Through systematic perturbation analysis, we validate this alignment by isolating the contributions of specific image statistics to model performance.
Our approach to structured visual textures connects to mechanistic models in computational neuro- science, particularly those studying the Drosophila visual system's sensitivity to 3-point correlations This connection strengthens the relevance of our model and highlights its potential to
19 20 provide insight into visual processing in both biological and artificial systems. The consistent out- performance of our HoCNN across various datasets demonstrates the effectiveness of incorporating these biologically inspired computations. RSA reveals distinct geometries, providing evidence that our model processes visual information differently from standard CNNs.
5.1 Limitations
Despite the demonstrated benefits, our higher-order convolution approach has several limitations worth acknowledging. The primary challenge is computational complexity, as detailed in Appendix A.2 Our FLOP analysis reveals that 2nd and 3rd order operations require approximately 5x and
18x more computation than standard convolutions, respectively. While we mitigate this by strategic
8

---

4 0:12
In - 0.12
Normalizzed Accuracy wrt CIFAR-10 Test (1 0.12)
0.8
Original
Nomelized Acaurnny
HcONN
0.6
lop 0.12
4. - 0.12
B-
B1
B1
9
Perturbation Statistics
HoConv Block RDM
Conv Block RDM
Log
airplane
automobile
bird
0.15
0.6
dog
frog
0.0
horse
ship
-0.05
0.
airpland actomablee
bnd
aisiaand
acttonatad
Bid
deg
frg
ship
airpland
automabble
frag
hore
ini
Figure 4: Perturbation Analysis and Neural Representations. (A) An exemplar test image (CIFAR- 10) and perturbed examples with different I, 2, 3, 4-point correlations statistics and common intensity 0.12. (B) Normalized (wrt unperturbed CIFAR-10) accuracy for Convolutional (Blue) vs Higher-order (Orange) networks: performances
Higher-order network are systematically worse for more structured perturbations at fixed intensity (I 0.12). (C) Representational Dissimilarity Matrix (RDM) for baseline Convolutional Block (Conv layer, Batch Norm, ReLU, Pooling). (D) RDM for the Higher-order Convolutional block. (E) Log Ratio between the two RDMs, capturing different representational geometries between the two blocks.
placement of higher-order operations in early network layers, this computational overhead represents a significant trade-off for resource-constrainee applications. The increasing parameter count with higher-order expansions also introduces scalability challenges. Although ages kernel symmetry to reduce parameters (e.g., reducing the 3 order parameter count by 4.42x), memory requirements still increase substantially with order. This limits practical applications beyond 4th order expansions, though our experiments indicate minimal benefits beyond this point.
implementation lever-
Additionally, approach shows increased vulnerability to higher-order statistical perturbations (Section 4) suggesting trade-off between enhanced feature extraction capability and certain forms of robustness. We also observed that placement of higher-order layers significantly impacts performance (Appendix A.3),, with deeper placement leading to performance degradation. Finally, our current implementation does not employ low-rank approximations other efficiency techniques commonly used modern architectures, representing an opportunity for future optimization.
5.2 Future Directions
An important consideration is how our higher-order convolution approach relates to Vision Trans- formers [42). which have emerged as powerful alternatives to CNNs. While both capture feature interactions, they operate on fundamentally different principles. Our higher-order convolutions ex- plicitly model multiplicative interactions between neighboring pixels through polynomial expansions, whereas transformers employ bilinear operations with global attention. From mathematical per- spective, attention mechanisms compute pairwise interactions through the query-key product (QKT), which are inherently bilinear, while our approach extends to trilinear and higher-order multiplicative terms within local receptive fields. These differences suggest that higher-order convolutions and vision transformers may be complementary rather than competing approaches, potentially offering different advantages for capturing local complex patterns versus global relationships.

---

A particularly promising direction involves hybrid architectures that leverage both mechanisms: higher-order convolutions could process local complex patterns and higher-order correlations in early layers, while attention mechanisms capture long-range dependencies and global context in deeper layers. This combination could exploit the strengths of both approaches- efficheeeeffiiintttt local feature extraction through higher-order polynomial expansions and flexible global reasoning through self-attention. Investigating such architectures could reveal whether these mechanisms provide complementary or redundant information, and identify optimal integration strategies for different vision tasks.
Beyond architectural hybridization, future research directions include exploring deeper architectures with multiple higher-order convolutional layers and investigating efficiency improvements through low-rank tensor decomposition methods. These techniques, which have proven successful in trans- former architectures 44 could maintain the benefits of higher-order interactions while significantly reducing computational overhead. Additionally, extending our approach to other computer vision tasks beyond image classification, such as object detection and dense prediction tasks, represents an important avenue for validating the generalizability of higher-order convolutions.
In conclusion, our higher-order CNN represents promising direction for more effective and bio- logically inspired computer vision models. By exploiting visual patterns and relationships more efficiently, even in shallow architectures, our approach opens new possibilities for advancing the field. Future research should focus on understanding the interplay between different orders of nonlinearity, their relationship with attention mechanisms, and investigating applications across diverse vision tasks.
Acknowledgments
We would like to thank the reviewers for their valuable feedback and suggestions that helped improve this work. We thank Jan Koenderink and Andrea van Doorn for valuable discussions and for their detailed explanations of their work, which has been a source of inspiration for this research. We are grateful to Felix Wichmann for early discussions during the preliminary stages of this work. We also thank the Sorbonne Center for Artificial Intelligence (SCAl) for providing essential computational resources.
Simone Azeglio acknowledges this work was done within the framework of the PostGenAIGe Paris project with the reference ANR-23-IACL-0007 Additionally, his PhD position was funded by the SCAI, through IDEX Sorbonne Université, project reference ANR-I1-IDEX--4
Olivier Marre would like to thank the ERC Consolidator grant DEEPRETINA (101045253) and the Agence Nationale de la Recherche (ANR) for financial support through Chaire Industrielle A1R-200CE37-000 8-04-Shhooting Star, ANR-22-CE37-002 MyopiaMaster WANR-22-CHIN-0000 PerBaCo, and ANR-RetNettEE. NUTRIACT, 22-.N-R-22--37-0000666
Peter Neri ackmowledges support from IIT (grant IVXX000701), the Agence nationale de la recherche (ANR-10-LABX-0007, -NN-II--DDE-00000-0-000 ANR-22-CE93-0013; and CNRS.
Ulisse Ferrari acknowledges this work was done within the framework of the PostGenAI@ Paris project with the reference ANR-23-IACLL-000) and benefited from financial support by the ANR through grants NatNetNoise (ANR-21 11--2-1-E37-0024) and IHU FOR&SIGHT (ANR-18-IAHU-01) Our lab is part of the DIM C-BRAINS, funded by the Conseil Régional flledd-Frannce
10

---

References
[1] Jan Koenderink and Andrea van Doomn. Local image structure and procrustes metrics. SIAM Journal Imaging Sciences, 11(1):293-323 3244, 2018.
[2] Alex Krizhevsky, Ilya Sutskever, and Geoffre E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[3] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409 1409.1556, 2014.
[4] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[5] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 2015.
[6] Matthew D) Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 818-833. Springer, 2014.
[7] DJ Field. Scale-invariance and self-similar transforran an analysis of natural scenes and mammalian visual systems. Wavelets, fractals, and Fourier transforms, pages 151-193, 1993.
[8] Bruno A Olshausen and David J Field. Natural image statistics and efficient coding. Network: computation neural systems, 7(2):333, 1996.
[9] Odelia Schwartz and Eero P Simoncelli. Natural signal statistics and sensory gain control. Nature neuroscience, 4(8):819-82) 2001.
[10] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2016.
[11] Ivan Ustyuzhaninov, Wieland Brendel, Leon Gatys, and Matthias Bethge. What does it take to generate natural textures? In International conference on learning representations, 2022.
[12] Keyan Ding, Kede Ma. Shiqi Wang, and Ecro P Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence, 44(5):256722581 2020.
[13] Kaicheng Yu and Mathicu Salzmann. Second-order convolutional neural networks. arXiv preprint arXiv: (00tv:1703.000 06817, 2017.
[14] Vito Volterra. Sopra le funzioni che dipendono da altre funzioni. Tipografia della R. Accademia dei Lincei, 1887.
[15] Vito Volterra. Theory functionals and of integral and integro-differential equations. 1930.
[16] Gašper Tkačik, Jason S Prentice, Jonathan D Victor, and Vijay Balasubramanian. Local statistics in natural scenes predict the saliency of synthetic textures. Proceedings of the National Academy of Sciences, 107(42):18 111888199111444 2010.
[17] Nikolaus Kricgeskorte, Maricke Mur, and Peter A Bandettini. Representational similarity anonesis the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008
[18] Jan J Koenderink. The structure of images. Biological cybernetics, 5065):63--37,, 1984.
[19] James Fitzgerald and Damon A Clark. Nonlinear circuits for naturalistic visual motion estimation. Elife, 4:c09123, 2015.

---

[20] Damon A Clark, James E Fitzgerald, Justin M Ales, Daryl M Gohl, Marion A Silies, Anthony M Norcia, and Thomas R Clandinin. Flies and humans share a motion estimation strategy that exploits natural scene statistics. Nature neuroscience, 17(2):29-3333, 2014.
[21] Jerome Y Lettvin, Humberto R Maturana, Warren S McCulloch, and Walter H Pitts. What the frog's s eye tells the frog's s brain. Proceedings of the IRE, (7(14447411940-19955 1959.
[22] Esther Peterhans and R Von der Heydt. Functional organization of area v2 in the alert macaque. European Journal of Neuroscience, 5(5):509-524, 1993.
[23] Jonathan B Levitt, Daniel C Kiper, and J Anthony Movshon. Receptive fields and functional architecture of macaque v2. Journal of neurophysiology, (1tp:/siiooogy.1116)111 7166)2517-2242, 1994.
[24] Gerhard Krieger, Christoph Zetzsche, and Erhardt Barth. Higher-order statistics of natural images and their exploitation by operators selective to intrinsic dimensionality. Proceedings of the IEEE Signal Processing Workshop on Higher-Orde Statistics, pages 147-151. IEEE, 1997.
[25] Christoph Zetzsche and Erhardt Barth. Image surface predicates and the neural encoding of two-dimensional signal variations. In Human vision and electronic imaging: Models, methods, and applications, volume 1249, pages 160-177. SPIE, 1990.
[26] Erhardt Barth, Terry Caelli, and Christoph Zetzsche. Image encoding. labeling, and recon- struction from differential geometry. CVGIP: Graphical models and image processing, 55(6): 428-446, 1993.
[27] Jan J Koenderink and Whitman Richards. Two-dimensional curvature operators. JOSA A, 5(7): 1136-1141, 1988.
[28] David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. Journal of neurophysiology. 1965.
[29] Pietro Berkes and Laurenz Wiskott. On the analysis and interpretation of inhomogeneous quadratic forms as receptive fields. Neural computation, 18(8): 1888-189,, 2006.
[30] Tenghui Li, Guoxu Zhou, Yuning Qiu, and Qibin Zhao. Toward understanding convolutional neural networks from volterra convolution perspective. Journal of Machine Learning Research, 23(31 1):1-50, 2022.
[31] Georgios Zoumpourlis, Alexandros Doumanoglou, Nicholas Vretos, and Petros Daras. Non- linear convolution filters for enn-based learning. In Proceedings of the IEEE international conference computer vision, pages 4761-4769, 2017.
[32] Siddharth Roheda, Hamid Krim, and Bo Jiang. Volterra neural networks (ynns). Journal of Machine Learning Research, 1-2,, 2024.
[33] Tsung- Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear enn models for fine-grained visual recognition. In Proceedings of the IEEE international conference on computer vision, pages 1449-1457, 2015.
[34] Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. Compact bilinear pooling. In Proceedings the 2016.
the IEEE conference on computer vision and pattern recognition, pages 317-326,
[35] Misha B Ahrens, Jennifer F Linden, and Maneesh Sahani. Nonlinearities and contextual influences in auditory cortical responses modeled with multilinear spectrotemporal methods. Journal of Neuroscience, 28(8):1929-194 2008
[36] Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl- Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, (111:501-522) 2020.
[37] Jonathan Victor and Mary M Conte. Local image statistics: maximum-entropy constructions and perceptual salience. JOSA A, 29(7):1313-1345, 2012.
12

---

[38] Eugenio Piasini et al. metex- maximum entopy textures. 2021.
[39]
Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
[40] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee, 2009.
[41] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv:1903 /2261, 2019.
[42] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[43] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09665, 2021.
[44] Jeremy Howard. Imagewang, 2020. URL http:://gttaab.oo///aaaaaa//gaap
[45] Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James DiCarlo. Simulating primary visual cortex the front of cnns improves robustness image perturbations. Advances in Neural Information Processing Systems, 33: 13733-13733 2020.
[46] Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardr. Grounding representation similarity through statistical testing. Advances Neural Information Processing Systems, 34:1556-1555 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's contributions extending convolutional neural networks with higher-order convolutions inspired by biological vision.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes a dedicated "Limitations" subsection (Section 5.1) that comprehensively discusses computational complexity challenges, parameter scaling issues, vulnerability to statistical perturbations, layer placement constraints, and the lack of low-rank approximations in the current implementation.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper doesn't include formal theoretical results requiring proofs. provides mathematical formulations of higher-order convolutions but doesn't 't make claims that require formal theorem-prond structures.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed descriptions of the model architecture (Section 2). training procedures (Section 3.2), and datasets used. Appendices A-F offer comprehensive architectural specifications, hyperparameters, and implementation details sufficient for reproducing the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We included a link to a github repository containing the code used for our experiments in the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/ sealliccguies///// Codesuumissionoolooo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https wttp:/ii/ggiiee//eeeeeeeeeeeeeee http:/uuuddeeeoeeuutiiiiiii www.oodeuumiiisoonnee..o.
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper specifies all relevant training details including optimizer, learning rate, weight decay, batch size, normalization techniques, and data preprocessing steps across all experiments in Section 3.2 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Table 2 reports means and standard deviations (a) over 50 different random
initializations for the benchmark experiments. The methodology section (3.2) explicitly states this approach was used to ensure robustness and statistical reliability of the results.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper includes specific information about compute resources Section 3.2. We specify that experiments on MNIST, FashionMNIST, CIFAR- 10, CIFAR- and Imagenette were conducted using an NVIDIA RTX 4080 Ti GPU, while ImageNet experi- ments used an NVIDIA A100 GPU. We provide concrete training times: 10-15 minutes per model for MNIST/FashionMNIISI 20-30 minutes for CIFAR-1O/CIFAR models taking 1.5-2x longer than CNN counterparts), 4-5 hours for Imagenette models, and 2-3 days for ImageNet models. Additionally, Appendix A.I provides comprehensive analysis of computational complexity, including detailed FLOP measurements for different orders of convolution.
(with HoCNN
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
17

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
The paper presents foundational research neural network architectures for image classification, without immediate application-specffic societal implications, as the results presented here SOTA. focuses on mathematical extension convolution operations rather than application-oriented technology.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper doesn't present models data with high risk for misuse. It focuses on fundamental image classification techniques using standard benchmark datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
18

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites the original sources of all datasets and prior work used, including standard benchmarks (MNIST, CIFAR, ImageNet) and the software library for texture generation (Metex).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Technical Appendices and Supplementary Material
A.1
Code Availability & Reproducibility
The code is available at http:///sshubbboooowsuoooegggeer
A.2
Computational costs versus traditional methods
We analyze the computational requirements of HoCNN versus traditional CNNs. Our HoCNN architecture for MNIST, Fashion MNIST and CIFAR-10 consists of:
First layer: x Volterra kernels (8 channels) with first, second, and third-order interactions
Two standard convolutional blocks (16 and 32 channels)
Two fully connected layers with dropout (p 0.25)
Total parameters: 80,262
In order to provide a comparison with a more complex CNN model, we additionally tested a deeper version of the Baseline CNN model. In terms of architecture the Deep CNN network consist:
Eight convolutional blocks (3 x 3 kernels) with channel progression: - 8 16 16
32 - 32 + 64 + 64
Two fully connected layers with dropout (p 0.25)
Total parameters: 108,130 (25% more than HoCNN)
Despite fewer parameters, HoCNN achieves superior accuracy (avg 72.35% vs 71.20%).
Computational complexity scaling:
Standard convolution: OCCour X Oin x K2 x H x W) Second-order kernels: OC Oin K H W)
Third-order kernels: OCCout X Cin X K x H x W)
Where: Cout = output channels, Cin- = input channels, K = kernel size, H, W feature map height and width.
A.2.1 FLOPs Measurements
To provide a concrete assessment of computational requirements, we conducted detailed benchmark- ing of our higher-order convolution operations compared standard convolutions. For single 3x3 convolutional layer processing an ImageNet-sized input:
Order (standard convolution): 0.2312 M FLOPs (baseline)
Order 1.1651 M FLOPs (5. (5.04x standard convolution)
Order 3: 4.3051 I M FLOPs (18.62x standard convolution)
For complete networks ImageNet-scale inputs:
Standard R 18: 1.82 G FLOPs
HoResNet-18 (1st+2nd order): 2.64 G FLOPs (1.45x increase)
18ResNet-18 (1st+2nd+3rd order): 3.00 G FLOPs (1.65x increase)
These measurements align with theoretical complexity analysis while highlighting how strategic placement of higher-order operations primarily in early layers (see ap-
keeps the overall computational increase manageable. Addi- tionally, our implementation leverages symmetry in the higher-order kernels, significantly reducing the parameter count:
Order 1.80x reduction ((3 x 3)2 81 - 45 unique parameters)
Order 3: 4.42x reduction ((3 3)3 729 165 unique parameters)
21

---

A.2.2 FLOPP-Controlled Experiments
To isolate the impact of our higher-order convolution approach from simply having additional computational capacity, we conducted an experiment comparing our HoCNN with deep CNN having comparable FLOPs on CIFAR-10:
Table 3: Comparison of accuracy with similar computational budgets
Model
FLOPs
Accuracy (%)
Deep CNN
7.09M
71.20
HoCNN
6.81M
72.06
These results confirm that the performance gains from higher-order convolutions cannot be attributed merely to increased computational capacity. Even with slightly fewer FLOPs, our HoCNN achieves 1.21% accuracy improvement over the deep CNN. This substantiates our claim that higher-order convolutions provide a meaningful inductive bias that enables networks to learn more effective representations than those captured by simply scaling standard architectures.
A.3 Placement of Higher-Order Layer & Effects of Larger Kernels in CNN
We conducted extensive experiments to investigate both the optimal placement of higher-order operations within the network hierarchy and the effects of increased kernel sizes in standard CNNs. These experiments provide important insights into the architectural choices made in our main study.
A.3.1 Effect of Higher-Order Layer Placement
We tested the introduction of the higher-order block at different depths the network, specifically at the second and third convolutional layers. Our results reveal consistent pattern across datasets: performance gradually degrades as the higher-order block is placed deeper in the architecture. For CIFAR- 10, accuracy decreases from 71.17 0.51% when the higher-order block is at the second convolutional layer to 70.29 = 0.49% at the third layer. Similarly, for CIFAR- 100, accuracy drops from 37.27 0.80% to 36.09 0.81%.
This performance degradation can be attributed to earlier layers potentially capturing and overfitting to spurious correlations. While standard convolutions theoretically preserve higher-order correlations, the cascade of nonlinearities and pooling operations can introduce and amplify spurious correlations. potentially compromising the network's s ability to learn meaningful representations at deeper layers.
A.3.2 Impact of Kernel Size
We also investigated whether simply increasing kernel sizes in standard CNNs could achieve benefits similar to higher-order operations. Despite larger kernels theoretically being capable of capturing higher-order correlations (as demonstrated by improved performance on our synthetic texture dataset), experiments with 5x5 and 7x7 kernels showed decreased test accuracy compared to our baseline CNN. For CIFAR-10, accuracy decreased to 68.72 72 + 0.63% with 5x5 kernels and 67.51 + 0.61 % with 7x7 kernels. The pattern similar for CIFAR-100, where accuracy dropped 36.04 # 0.76% with 5x5 kernels and 34.01 = 0.72% with 7x7 kernels.
Our choice of 2x2 kernels for the main experiments synthetic textures, which are based on gliders up to 4 pixels arranged in 2x2 patches. The results suggest that while larger kernels might have the capacity capture higher-order interactions, the linear nature of standard convolutions makes challenging to effectively learn these patterns during optimization.
motivated by the generative process of our
A.3.3 Computational Considerations
The placement of higher-order operations in early layers offers computational advantages. In typical convolutional architectures, the number of channels increases with network depth. Introducing higher-order kernels in later layers, where channel count is larger, would substantially increase both parameter count and computational complexity. Our architectural choice of early placement thus provides an effective balance between computational efficiency and performance.
22

---

These findings support our design decision to introduce higher-order operations early in the network, where they can directly process raw image statisties before potential distortion by successive pro- cessing layers. The results demonstrate that explicitly modeling higher-order interactions through dedicated operations provides more effective approach for capturing and leveraging complex statistical features compared to simply increasing kernel sizes in standard CNNs.
A.4 ResNet Scaling Experiments
To thoroughly investigate whether the benefits of our higher-order approach extend to deeper architec- tures, we conducted comprehensive scaling experiments with ResNet-18, ResNet-34, and ResNet-50 architectures. While our main experiments focused on ResNet-18, these additional experiments provide valuable insights into how higher-order convolutions perform as model capacity increases.
The results comparing standard ResNet and HoResNet across different depths are presented in the following table:
Table 4: Performance comparison of ResNet vs. HoResNet across architectures of increasing depth
Dataset
Standard ResNet
Higher-Order ResNet
RI8
R34
R50
HoR18
HoR34
HoR50
MNIST
99.34 + 0.10
99.35 = 0.11
99.38 1 0.10
99.38 E 0.09
99.40 = 0.08
99.41 + 0.08
MNSST
90.85 + 0.32
91.15 = 0.23
91.20
1..00+0.00
91.25
0.29
91.32
+ 0.21
91.34 + 0.25
CIFAR-10
81.26 + 0.57
83.02
= 0.64
84.10
0.59
83.05
0.55
84.06
= 0.49
85.50
0.62
CIFAR-100
60.77 + 0.73
58.05 = 0.69 56.46 0.83
61.85 + 0.81
58.35 0.70
56.55 0.74
The results clearly demonstrate that the benefits of our higher-order approach extend beyond ResNet18. Key observations include:
Consistent improvements across model capacities: Our higher-order networks consis- tently outperform their standard counterparts across all tested architectures and datasets. This suggests that the higher-order interactions captured by approach provide fundamental benefits regardless of network depth.
Scaling behavior: While both standard and higher-order networks show diminishing returns with increasing depth simpler datasets (as expected), HoResNets maintain their advantage. This indicates that the inductive bias provided by higher-order convolutions remains valuable even models scale up.
Performance complex datasets: For CIFAR-10. 10, our HoResNet18 8 achieves accuracy comparable to ResNet34, while HoResNe150 shows the strongest performance overall. This demonstrates the complementary benefits of higher-order operations and increased network depth.
Overfitting considerations: We observe some evidence of overfitting as models get larger, particularly for CIFAR-100, where the limited number of samples per class makes training very deep networks challenging. This explains why we initially focused on more compact architectures our main experiments.
These findings strongly support our claim that higher-order convolutions provide meaningful im- provements that complement traditional scaling strategies like increasing network depth.
A.5 Corruption Robustness Analysis
To evaluate the practical robustness of our higher-order models to real-world image corruptions, we conducted comprehensive experiments using the CIFAR- and CIFAR-I00- benchmarks. These datasets apply 17 common image corruptions at levels of severity to the standard test sets. We report results in terms of normalized accuracy (with respect performance the uncorrupted test set).
Our results reveal a surprising pattern: despite exhibiting increased sensitivity to our controlled higher-order statistical perturbations (Section 4. HoCNN models demonstrate enhanced robustness to common image corruptions:
23

---

Table 5: Normalized accuracy (%) on CIFAR-IO-C with severity level 1
Corruption Type
Baseline CNN
Higher-Order CNN
brightness
99.4 9.2
99.6 5.2
contrast
96.0 + 14.7
96.8 + 6.5
defocus_bile blur
98.8 9.2
99.1 7.0
85.3
86.1 + 8.5
fog
97.8 12.9
98.6 + 6.8
frost
94.2
92.9 + 13.5
gaussian_non noise
77.2
77.5 30.0
glass_blur
58.2
56.6 + 59.0
impulse noise
25.3
89.6 11.6
jpeg_ comp_sssseoinnn
96.3 12.6
97.0 6.0
motion_bluu
88.6 11.2
88.4 15.7
pixelate
97.6 11.1
97.2 5.9
saturate
9 10.4
99.9 + 5.0
Whot_noise
85.8
87.5 + 18.9
snow
90.8 1: 28.1
90.8 - 8.8
spatter
95.0 12.8
95.5 3.1
zoom_blur
81.5 + 30.1
82.5 1 18.0
Table 6: Normalized accuracy (%) on CIFAR- 100-C with severity level
Corruption Type
Baseline CNN
Higher-Order CNN
brightness
99.1 + 9.8
99.2 + 12.3
contrast
93.9 + 12.6
94.9 + 11.5
defocus blur
97.9 11.3
98.2 + 12.4
Blastic_transfom
75.1 13.3
75.6 + 10.0
fog
96.3 10.3
97.0 11.0
frost gaussian noise
84.8
86.0 + 11.8
60.5
64.7 = 16.6
glass blur
21.
3
impulse noise
72.0
7
92.1
motion_ble bbur
80.7
81.5 12.3
pixelate
9
94.0 + 12.1
saturate
98.7
98.6 11.6
shot_noise
74.0 16.8
78.3 15.4
snow
81.0 13.3
82.3 1 12.8
spatter
88.5 + 13.4 13.4
89,4 11.4
zoom blur
73.0 + 21.5
74.2 14.5
For CIFAR-1O-C, our HoCNN outperforms the baseline 13 out of 17 corruptions at severity level 1, with similar patterns observed at higher severity levels.
For CIFAR-100-C, HoCNN shows even more pronounced improvements, outperforming the baseline on 15 out 17 corruptions at severity level
The improvements are particularly notable for corruptions that preserve important image structure (contrast, blur, fog) while maintaining competitive performance on noise-based corruptions.
These findings demonstrate that while the higher-order networks are more sensitive to specific statistical manipulations that target their core functionality, they generally exhibit improved robustness to common image corruptions encountered in real-world scenarios. This suggests that the richer representational capabilities of higher-order convolutions may actually enhance model reliability in practical applications.

---

A.6 Imagenette Results
To investigate the scalability our approach on a moderately-sized dataset, we evaluated HoResNet- 18 on Imagenette [44], 10-class subset sampled from ImageNet. Imagenette provides useful intermediate benchmark between the smaller CIFAR datasets and the full ImageNet, allowing for faster experimentation while maintaining the visual complexity of ImageNet images.
HoResNet- 18 achieved 89.30% test accuracy compared to ResNet- 13%, demonstrating +1.17% improvement (Table 7). Additionally, our HoResNet- 18 architecture achieves superior performance compared to a ResNet- 18 adapted VOneNet [45] moologiaallyinspinn model, while utilizing fewer parameters, as detailed in Appendix
Table 7: Imagenette classification accuracy
Model ResNet-18 18
Accuracy (%)
# Params
88.13 89.30
11,181,647
HoResNet-18
11,168,382
Training Details. For Imagenette, images were normalized using mean [0.5, 0.5, 0.5] and standard deviation [0.5, 0.5, 0.5], in order to align with VOneNet [45] preprocessing. All other training details remain consistent with the benchmarks presented in Section 3.2 The Imagenette experiments were conducted on an NVIDIA RTX 4080 Ti GPU with training times of approximately 4-5 hours per model.
Due to computational constraints, we report single-run results for Imagenette rather than the multiple- run analysis performed for other benehmarks. While this limits our ability assess variance, the improvement is consistent with the trends observed across other datasets, particularly the +0.85% improvement on the full ImageNet with statistical significance confirmed across 5 runs (Section 3.2)
A.7 Fine-Grained Classification: CUB-200-2011
To further evaluate our approach on tasks requiring fine-grained visual discrimination, we tested HoResNet-18 18 on the CUB-200-2011 bird species classification dataset [? J. Fine Fine-grained classification tasks are particularly relevant for assessing higher-order feature extraction, as they require models to capture subtle visual patterns that distinguish between visually similar categories. The CUB-200-201 dataset contains 11,788 images across 200 bird species, with images exhibiting significant intra-class variation in pose, scale, and background.
HoResNet-18 achieved 72.3 1% test accuracy compared ResNet- 18's 68.24%, demonstrating a sub- stantial improvement of +4.07% (Table 8). This larger improvement compared to standard ImageNet classification (+0.85%) suggests that higher-order correlations play an increasingly important role in tasks requiring discrimination of subtle visual patterns. The results align with our hypothesis that natural images contain higher-order statistical structures that are particularly relevant for fne-grained recognition tasks.
Table 8: Fine-grained classification accuracy CUB- CUB-200-2011
Model
Accuracy (%)
# Params
Re
68.24
11,689,512
HoResNet-18
72.31
11,676,252
Training Details. For CUB-200-2011, we followed standard fine-grained classification protocols. Images were resized to 224x224 pixels and normalized using ImageNet statistics (mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225]). The training procedure matched our ImageNet setup: Stochastic Gradient Descent with momentum = 0.9, initial learning rate 0.1 with cosine annealing, weight decay= le-4, and batch size 128. We used standard data augmentation including random crops, horizontal flips, and color jittering. Models were trained for 100 epochs with the best model selected based on validation accuracy. Experiments were run on an NVIDIA 1100 GPU, requiring approximately 12-18 18 hours per
25

---

Due to computational constraints, we report single-run results for CUB-200-2011 rather than the multiple-run analysis performed for ImageNet. However, the substantially larger improvement on this fine-grained task (+4.07% vs. +0.85% on ImageNet) provides strong evidence that higher-order convolutions are particularly beneficial for capturing subtle visual patterns.
A.8 Fine-Grained Classification: Extended Analysis
The CUB-200-2011 dataset presents unique challenges for image classification models due to its fine- grained nature. Bird species often differ in subtle plumage patterns, beak shapes, body proportions, and coloration visual features that require capturing complex spatial relationships beyond simple edges and basic textures. This makes CUB-200-2011 an ideal testbed for evaluating whether higher-order convolutions capture the nuanced statistical structures necessary for fine-grained discrimination.
Our results on LUB-200-201 show a +4.07% improvement (HoResNet-1s 18: 72.31% vs. ResNet- 18: 68.24%), which notably larger than the +0.85% improvement observed on ImageNet. This differential improvement is particularly meaningful: while ImageNet contains diverse object cate- gories with substantial inter-class visual differences, CUB-220-2011 requires discriminating between visually similar bird species. The larger performance gain on CUB-200-201 I suggests that higher- order correlations become increasingly important as the visual discrimination task becomes more fine-grained.
The biological inspiration for our approach aligns well with this finding. Biological visual systems are known to excel at fine-grained discrimination masss-for instance, humans can readily distinguish between similar bird species, fabric textures, or facial identities. Our incorporation of higher-order correlation detection mechanisms may capture some of these capabilities, explaining the particularly strong performance on fine-grained tasks.
We followed standard fine-grained classification protocols for CUB-200- 2011. Images were resized 224x224 pixels and normalized using ImageNet statisties (mean [0.485, 0,456, 0,406], std = [0.229, 0.224, 0.225J). The training procedure matched our ImageNet setup: Stochastic Gradient Descent with momentum 0.9, initial learning rate 0.1 with cosine annealing, weight decay le-4, and batch size 128. We used standard data augmentation including random crops, horizontal flips, and color jittering. Models were trained for 100 epochs with the best model selected based on validation accuracy.
Implementation Details:
Examining the confusion patterns, we observed that HoResNet-18 18 particularly excelled at distinguishing species with subtle plumage pattern differences. The most common errors for both architectures occurred between species with similar body shapes and coloration but different fine-scale patterns patterns-peeiiell the scenarios where higher-order correlations would be expected to provide the most benefit. This error pattern supports our hypothesis that higher-order convolutions enhance the model's ability to capture the complex spatial relationships that define fine-grained visual categories.
Error Analysis:
A.9 Architectures and Training Procedure: MNIST, FashionMNIST, CIFAR-10, CIFAR-100 & Imagenettee
For image classification benchmarks, we employed a baseline CNN and HoCNN architecture, with varying complexity depending on the dataset. For MNIST, FashionMNIST, and CIFAR- 10, we used a simple architecture consisting of three convolutional blocks (Convolutional Layer, Batch Normal- ization, ReLU nonlinearity, and Max pooling) followed by two fully connected layers interleaved with dropout and ReLU nonlinearity (see Figure 3}. For these datasets, we used 9, 18, and 36 kernels respectively for the convolutional layers of the baseline CNN, while the HoCNN used 8, 16, and 32 kernels. This configuration ensured that the baseline CNN had comparable, slightly higher number of parameters (82,706 vs. 80.262). For CIFAR- 100, we increased the kernel counts to 34, 68, and 136 for the baseline CNN, and 32, 64, and 128 for the HoCNN, resulting in 385,006 and 383,478 parameters respectively.
The validation accuracy curves for the CNN and HoCNN on these benchmarks are provided in the Figure 5 further illustrating the consistent performance advantage of the HoCNN across datasets.
26

---

MNIST
FashionMNIST
CIFAR-10
CIFAR-100
72.
40.0
644
Accuracy
58.3
06.0
13.
00.
510
97.
16.00
CWS
Epochs
Epochs
Epocits
Epochs
Figure 5: Validation accuracy curves for CNN and HoCNN on image classification benchmarks. Learning curves comparing the performance of the baseline CNN and the proposed HoCNN on MNIST, FashionMNIST, CIFAR-10, and CIFAR-100 datasets. The HoCNN consistently outperforms the CNN across all benchmarks, demonstrating the effectiveness of incorporating higher-order interactions in the convolutional layers.
Our training strategy utilized the AdamW optimizer with weight decay and learning rate schedul- ing RecduceLROnPPlttaau) We employed Early Stopping with patience for model selection. For MNIST, FashionMNIST, CIFAR-10, and CIFAR- 100, we trained 50 model realizations with different random seeds to ensure reliable performance assessment. For Imagenette, conducted single comprehensive evaluation run.
A.9.1 Architectures and Parameter Counts: Imagenette
Standard ResNet-18 architecture, totaling 11,181,642 parameters:
Initial Block:
Convolutional layer (7 x 7 kernels, 64 channels)
Batch normalization
ReLU activation
Max pooling (3 x 3)
Four main stages, each containing 2 residual blocks:
Stage 64 channels (2 blocks)
Stage 2: 128 channels (2 blocks)
Stage 3: 256 channels (2 blocks)
Stage 4: 512 channels (2 blocks)
Final layers:
Global average pooling
Fully connected layer to output classes
Higher-Order ResNet-18 18 (HoResNNt-188) totaling 11,168,382 parameters:
Initial Block (similar to ResNet-18):
Convolutional layer x kernels, 30 channels)
Batch normalization
ReLU activation
Max pooling (3 x 3)
Four main stages, with a hybrid approach:
Stage 1: 30 channels with higher-order residual blocks (2 blocks)
27

---

Stage 2: 128 channels with standard residual blocks (2 blocks)
Stage 3: 256 channels with standard residual blocks (2 blocks)
Stage 4: 512 channels with standard residual blocks (2 blocks)
Final layers (similar to ResNet-18):
Global average pooling
Fully connected layer to output classes
A.9.2
Training Setup: Imagenette
Data Preprocessing:
Training images: Random resized crop 224 x 224, random horizontal flip, normalization
Test images: Resize to 256 256, center crop to 224 x 224, normalization
Both models use normalization with mean and std of [0.5, 0.5, 0.5] which results in rescaled pixels with mean close to zero (I-0.07, -0.09, -0.15) and std close to 0.5 ([0.55, 0.54, 0.58))
Training Configuration:
Batch size: 64
Loss function: Cross-entropy
Optimizer: AdamW W with learning rate 0.001 and weight decay 5e-4
Learning rate scheduling: Reducel.ROnPlaten. (halves LR after epochs without improve- ment)
Early stopping: Implemented with 12 epochs patience
A.10
Comparison with VOneNet
We compared the performance of our HoResNet-11 18 architecture with ResNet- 18 adapted VOneNet [4], a cetooiccallyinnspieee model designed to better capture the properties of the visual cortex. Our HoResNet-11 18 model achieved superior accuracy the Imagenette dataset (8930% compared the VOneNet model (88.00 (88002), while utilizing fewer parameters (11, 16,,3888 vs. 14,445,322).
This improved performance aligns with the findings reported by the VOneNet authors, who observed lower ImageNet accuracy compared to standard ResNet architectures, despite achieving higher brain- scores (i.e., higher explained variance of neural activity recorded from specific brain areas in the visual cortex). This trade-off between ImageNet performance and brain-score can verified on Brain-Score framework website by sorting models based on their ImageNet top-1 accuracy.
Our results demonstrate that the HoResNet-18 18 architecture, equipped with higher-order convolutions, can outperform biologically-insphe -insiirdd models like VOneNet on standard image classification tasks while maintaining more compact parameter footprint.
A.11 Tied-Weight Issue
A depiction of how Higher-order terms go beyond simple non-pointwise nonlinearities is provided in Figure while PCA analysis with different nonlinearities is provided Figure] ? obtaining similar qualitative results as with ReLU.
A.12 Sensitivity to Image Statistics
More detailed results are presented in Table and T0 bold largest decrease in performance across models).
28

---

A
D132
T1
22
11
22
12
W1
U2
W1
W2
W
Wias +
Σ jm 4m
Wiy + t....
y
y
Figure 6: Beyond pointwise nonlincarities (A) Classical pointwise nonlinearity: the nonlinear function applied after summing the input from the previous layer. A Taylor expansion of this non-linearity reveals the wiig-weight problem addressed in Subsection 1.2 (B) Non-pointwise non- lincarity includes the summation of quadratic or higher-order terms. By modulating the interactions between terms with discovering an efficient trade-off between good approximation and balance different terms.
untie weights in the Taylor expansion: the optimization process is not saddled
Table 9: CNN Model accuracy (%) for each perturbation type and for different intensitics I, in parentheses the decrease in accuracy with respect to the case without perturbation (CIFAR-10 avg test accuracy = 69.76%)
Perturbation
1=0.55
1=0.09
1=0.12
1=0.16
T=0.20
67.0 (-4.0%)
59.9
(-27.6%))
41.4
3-51.22222%
8-
66.5 (-4.7%)
59.7 (-14.5%)
(-27.%%)
40.2 (-42.3%)
31.2 (-55.2%)
57.8 (-17.1%)
41.5 (-40.%))
30.8 (-55.8%%
24.6 (-64.7%%
21.0 (-69.9%)
55.5 (-20.5%)
(-5..2%))
18.0 (-74.2%)
(-81.%)))
11.8 (-8.
8
56.6
32.6 (-53.2%)
19.0
13.9 (-80.1%)
12.2
0_
5-22.22
30.9 (-55.6%)
18.6 (-73.3%)
13.8 (-80.2%)
(-82.9%)
0r
(-22.0%)
31.2 (-55.3%)
18.7 (-73.1%)
13.9 (-80.1%)
11.9 (-82.99)
0
(-22..%))
31.0 (-55.6%%
18.8 (-73.1%)
13.9 (-80.0%)
12.0 (-82.7%)
0
54.2 (-22.3%)
30.7 (-55.9%)
18.5 (-73.5%)
13.7 (-80.3%)
11.9 (-83.0%)
56.1 (-19,6%)
36.0 (-48.4%)
24.0 (-65.7%)
18.3 (-73.7%)
15.8 (-77.3%)
A.13 Neural Representations: Representational Similarity Analysis
To gain deeper insight into how our Higher-order Convolutional layer (HoConv) processes information differently from a standard Convolutional (Conv) layer, we employed Representational Similarity Analysis (RSA) [17 on the CIFAR-10 dataset. We analyzed activations from 100 test images (10 per class) averaged across 50 model realizations to mitigate the effects of random initialization [46]. Our findings reveal distinct representational geometries between the two models. In Figure we consider an alternative distance (hellinger) between the two RDMs, based on the consideration of the fact that values are between and
A.13.1
Order-wise Representational Differences
The 2nd and 3rd order components of the HoConv layer can be analyzed independently and extract different representations, with progressively smaller mean dissimilarity compared to the baseline Conv layer (Figure Ii.. These order-wise differences contribute to the overall representational differences between CNN and HoCNN.
Our analysis of neural representations revealed distinct patterns across different order components (see Figure and and Figure IT). The standard convolutional layer exhibited unimodal distribution of pairwise distances, indicating consistent representation of stimuli. In contrast, the 2nd order
29

---

Cumulative Explained Variance
Cumulative Expleined Variance
LEAKY RELU
GELU
Cumulstiv explained variance
Camulelive explaired variience
0.0
0.8
0.4
0.4
CNN
CNN
HoCNN
HoCNN
HoCNN
HoCNN
0.0
10
12
14
12
14
Purcentage of components
Percentage of components
Cumulative Explained Variance
Cumulative Explained Variance
SIGMOID
MISH
Conilative explained variance
Cumslative explained voriance
0.8
0.6
0.4
0.2
HoCNN 2nd order
HoCNN 2nd order
0.0
10
12
14
10
12
14
Percentage components
Percentage of components
Figure PCA analysis with different nonlincarities: Leaky Relu, Gelu, Sigmoid, Mish
Table 10: HO-CNN Model accuracy (%) for each perturbation type and for different intensities I, in parentheses the decrease in accuracy with respect to the case without perturbation (CIFAR- avg test accuracy 72.87%)
Perturbation
1=0.55
1=0.09
1=0.12
1=0.16
1=0.20
7
70.2 (-3.6%)
(-14.2))))
51.9
(-28.88)
4-43
32.8 (-55.0%
B-
69.3 (-4.9%)
61.7 (-15.3%)
52.1 (-28.5%)
41.3 (-43.3%%
31.7 (-56.5%
3
(-19
40.3 (-44.8%)
(-60
23.2 (-68.2%)
20.2 2-72.33339%
58.5 (-19.7%)
30.6 (-58.0%))
1-.7
12.6 (-82.7%)
11.7 (-84.0%
3/
(-19.3)))%
30.5 (-58.2%)
1-.7
12.5 (-8288%)
11.7 (-83.9%%
0_
56.3 (-22.7%)
29.1 (-60.1%)
(-70.%)
12.1 (-83.4%)
10.9 (-85.0%)
0
(-.1 (-22.8%)
1-.7
12.2 (-83.2%)
11.0 (-84.%%
07
(-22.6)))))
(-59.6%)
(-77.5)
12.3 (-83.1%)
11.0 (-84.9%)
0
56.3 (-22.8%)
29.1 (-60.1%%
16.1 (-77.9%)
12.2 (-83.3%)
11.0 (-85.0%)
58.4 (-19.8%)
35.8 (-50.9%)
22.5 (-69.2%)
17.0 (-76.6%)
14.9 (-79.6%)
component of the HoCNN showed a bimodal distribution, suggesting two distinct scales of represen- tation. This bimodality could indicate the model's 's ability to capture both fine-grained similarities and broader categorical differences between stimuli.
The 3rd order component demonstrated a wider spread of distances, with peak at lower dissimilarity values. This pattern suggests that the 3rd order component might be capturing subtle relationships or features among the stimuli, complementing the representations of the lower-order components.
These findings point more nuanced and hierarchical representation in HoCNNs compared to standard CNNs. The multi-order structure appears to enable the model to simultaneously process information at various scales and levels of abstraction. This could potentially explain the enhanced
30

---

03
38
64
3A
022
Figure 8: (Left) Representational Dissimilarity Matrix (RDM) for baseline Convolutional (Conv) Block (Conv layer. Batch Norm, ReLU, Pooling). (Middle) RDM for the Higher-order Convolutional (HoConv) block. (Right) Hellinger Distance between the two RDMs, capturing different representa- tional geometries between the two blocks
performance of HoCNNs on certain tasks, as they may be better equipped to capture complex, multi-scale patterns in the input data.
However, further investigation is needed to fully understand the underlying mechanisms and im- plications of these representational differences. Future work could explore how these distinet representations contribute to specific aspects of model performance, and whether they align with known principles of biological visual processing.
A.13.2 Layer-wise Representational Differences
The representational difference between CNN and HoCNN becomes more explicit when all orders are summed together and fed to the batch normalization, ReLU, and max pooling layers (see Figure 33. giving rise to the output of our HoConv block. We quantify these differences using Representational Dissimilarity Matrices (RDMs) on a subset of the testing set (Figure[4). We compare blocks instead of single layers because the standard convolution is linear, and equipping it with nonlinearity layer makes the comparison fairer. The Conv and HoConv RDMs show clear structural differences (Figure 4 & B), with the HoConv exhibiting more pronounced class-specific patterns.
A.13.3 Correlation between RDMs across Layers
We further analyzed the average correlation between RDMs across layers the CNN and HoCNN (Figure [O).. The results confirm the divergence in representational structure, with correlations decreasing in higher-order layers. This suggests that the HoConv layer progressively captures more diverse and complex features compared to the standard Conv layer.
A.13.4 RSA on Texture Classification
We extended the analysis presented paragraph of the main paper to the four models trained on synthetic texture data. This analysis aims to highlight the contributions of different kernel orders in texture classification. Following a similar methodology, we randomly selected 10 samples for each texture class from the test set. We then extracted activations at specific layers of the models and used these to construct representational dissimilarity matrices (RDMs), see Figure 12. Figure 13 and
Figure 14
These RDMs provide a visual representation of how the models distinguish between different texture classes at various processing stages. By comparing the RDMs across different layers and between the baseline CNN and HoCNN models, we can gain insights into how higher-order kernels influence representation and discrimination of texture patterns.
The following figures present these RDMs, allowing for comparative analysis how information is processed and transformed through the network layers in both the standard CNN and the HoCNN architectures when dealing with synthetic texture data. This analysis complements our findings from natural image datasets and offers a more comprehensive view of the role of higher-order computations in visual processing tasks.
31

---

Baseline CNN RDM (conv1)
Highor-Orae Order CNN RDM (comv])
10
Absolute RDM Difference
0.40
arplane
butorcbble
0.35
0.8
2eer
6.20
11g
0.4
0.15
Tore
6.10
0.05
Higher-Order CNN
08
0.0
0.25
4.70
0.35
02
0.0
Higher-Order CNN RDM (conv1
3.0
0.40
0.35
6.30
08
4.25
4.
4.10
0.05
30
6.00
Figure 9: Representational dissimilarity matrices across layers. (First row) Representational Dissimilarity Matrices for (left) standard Conv layer (1x layer of the CNN), (middle) 1st order HoConv layer, (right) absolute difference between the two RDMs. (Second row), the RDM for standard Cony not repeated while (in the middle) show the RDM for the activations extracted after the 2nd order HoConv layer; (right) the absolute difference between the standard Conv and the nnd order HoCony RDMs, which presents different representational geometries.
Layer-to-Layer Representational Similarity (Baseline CNN vs Ho-CNN)
0.995
0,919
0,524
Average Corelation
0.744
0.6
0.4
0.0
camdl as comy
cond comrt
come1
com' comrz
com'd comd
tel va
tct vs Rzz
Layer Pairs
Figure 10: Average Correlation between RDMs for matching layers (Baseline CNN vs HoCNN).
32

---

Conv Layer vs Ho-Conv 1st order
B
Conv Layer ve Ho-Conv 2nd order
Ho-Conv
Densily
Conv Layer vs Ho-Conv 3rd order
Conv Block vs Ho-Conv Block
Ho-Conv
Density
0.0
0.3
1.0
Dissimilanity
Dissimilarity
Figure 11: Pairwise distance distribution of RDMs (CIFAR-10). An alternative way represent RDM matrices is to plot the pairwise distance distribution of their entries. (A) We can see that at the 1st order the representations of the standard Conv Layer (here before the nonlinearity) and of the HoConv Layer are practically the same (1st order HoCony Layer is mathematically equivalent to a Conv Layer). (B) Representations for the 2nd order HoConv Layer are different from the standard convolutional layer: by themselves alone they don't bring better representations in terms of dissimilarity if compared with the standard Conv Layer ones. (C) Similar considerations for 3rd order HoConv Layer representations,
(D) When considering the output of the two blocks indeed (same as Fig 5. D), the situation is different: the distribution corresponding to the HoConv representations is considerably shifted, leading "more" dissimilar representations average.
105
04
Figure 12: RDM comparison between baseline CNN and HoCNN 2nd order (summed 1st and 2nd order kernel), on the right absolute difference to highlight representational differences.
33

---

Basaline RDM (baseCNN)
Higher ROM (HOCNN 3id)
Absol.ze Differncee
14
61
Figure 13: RDM comparison between baseline CNN and HoCNN 3rd order (summed 1st 2nd and 3rd order kernel), on the right absolute difference to highlight representational differences.
Bascline ROM (baseChNi)
Higher Order RDM (HOCNN 4th1
Absolute Dfffrence
14
54
01
Figure 14: RDM comparison between baseline CNN and HoCNN 4th order (summed 1st 2nd 3rd and 4th order kernel), on the right absolute difference to highlight representational differences.
34


Reviews:
[{"review_id": "0eXQTcoD4K", "reviewer": "Reviewer_qSKJ", "summary": "This paper introduces Higher-Order Convolutions (HoConv), a learnable extension of classical linear convolution operators that captures higher-order correlations in natural images beyond first-order (linear) relationships. Inspired by the observation that natural scenes exhibit complex statistical dependencies, the authors design a convolutional block that computes and sums responses from filters of order 1 through 4. Each order captures different correlation structures in the image, and the design ensures parameter independence across orders to facilitate more flexible learning. Compared to classical CNNs of similar or even higher parameter counts, HoCNNs consistently outperform on multiple image classification benchmarks. The authors also provide insight into how these higher order convolutions work, by the representational capacity of HoCNNs via synthetic data and PCA-based visualizations.", "questions": "1. Synthetic image visualization: Could you include the binary texture image referenced in Figure 1B and Section 2.1, or at least a representative example? This would help readers better understand what it means to include “all 1–4 point correlations” in 2×2 patches.\n2. PCA experiment design: In your PCA comparison of CNN and HoCNN representations, each point corresponds to a randomly initialized model. Can you clarify the rationale for this setup? Wouldn't it be more informative to fix one model and vary the input instead?\n4. Typo in Figure 4: Please clarify the inconsistency between the image panels and the figure caption in Figure 4. Is panel C the HoConv block or the Conv block?\n5. Extension to SOTA models: You mention potential connections with transformer-based vision models. Can you speculate on how higher-order convolutions could be integrated into hybrid or attention-based architectures?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- Clear motivation and originality: The paper is well-motivated by limitations of traditional CNNs in capturing higher-order pixel correlations. The proposed approach is a principled extension of convolution, grounded in signal statistics and biological plausibility.\n- Novel architecture: While prior work has explored 2nd-order filters, this paper generalizes the idea up to 4th order in a clean, modular way, while ensuring learnability and independence of parameters across orders.\n- Strong experimental design: The authors test the model across a variety of vision benchmarks and provide statistical evidence (via repeated experimental runs) to support the robustness of their improvements.\n- Analysis beyond benchmarks: The paper goes beyond classification accuracy to examine what different orders of convolution capture, using PCA and synthetic datasets.\n- Clarity: The manuscript is overall very well-written and accessible. The motivation, implementation details, and experiments are communicated clearly.\n\n### Weaknesses\n\n- Visualization gap: In Figure 1B and section 2.1, the authors reference a synthetic image containing binary textures that exhibit all 1–4 point correlations within 2×2 patches. This image is not shown, and it’s difficult to develop intuition for it without a visual example.\n- Questionable analysis choice: In the PCA-based analysis comparing CNN vs HoCNN representations, each data point corresponds to a random weight initialization, rather than a fixed network processing different images. This design decision may limit the interpretability of the representation space analysis.\n- Minor clarity/consistency issues: Figure 4 panel labels (C and D) are mismatched between figure and caption. Similarly, textual references to these panels are inconsistent.", "was_revised": true, "final_justification": "This is a technically solid paper and the authors have adequately addressed my remaining questions and concerns in their rebuttal. Therefore, I believe it should be accepted, and I am maintaining my score of 5.", "rebuttal": "Dear Reviewer qSKJ,\n\nThank you for your thorough and constructive review. We appreciate your positive assessment of our work and your recognition of its contributions to understanding higher-order visual processing. We address your specific concerns and questions below.\n\n----\n### W1 & Q1: Visualization of synthetic binary textures\nWe acknowledge this omission. The synthetic image referenced in Figure 1B and Section 2.1 contains randomly distributed 2×2 binary patches designed to exhibit all possible gliders. Specifically what we mean by “ including “all 1–4 point correlations” in 2×2 patches” is that if you look at the distribution of all the possible 2x2 patches, you’ll see representatives of the 1, 2, 3, 4 point (i.e. pixel) gliders. \nAs an intuition you can think about binary (either black or white pixels) white noise: since we’re not imposing any constraint  on neighbouring pixels, in each realization there will be a number of 1, 2, 3 and 4 point (i.e. pixel) correlations. \n While we cannot add figures in the rebuttal, we will include a comprehensive visualization panel in the revised manuscript showing it.\n\n----\n### W2 & Q2: PCA experiment design rationale\nWe appreciate your insight regarding this analysis choice. Our decision to use multiple random initializations and a fixed input image rather than multiple input images and a single random initialization was motivated by the goal of understanding the intrinsic representational capacity of our architecture without training. More in detail: since we are not training our model and the baseline CNN in this experiment, the models haven’t learned how to separate different N-point correlations yet. In this way we only wanted to highlight the fact that the HoCNN has potentially more representational power: more PCs are needed which means higher dimensionality of the representation. \n\nWhile we used this setting for this experiment, we’d like to highlight that for Representational Similarity Analysis in the last section of the paper we are comparing multiple images (with trained networks) from different classes, and representational differences are still present and leading to a gap in performance. \n\n----\n### W3 & Q3: Figure 4 inconsistency\nThank you for catching this error. The correct labeling is:\nPanel C: Higher-order convolution (HoConv) block\nPanel D: Standard convolution (Conv) block\nWe will correct all references in the text and ensure consistency throughout the manuscript.\n\n----\n### Q4: Relationship with Transformers and Self-Attention\nThank you for this question, it touches upon a crucial point regarding the relationship between our proposed higher-order convolutions and the attention mechanisms that empower modern SOTA architectures like Vision Transformers (ViTs). While both approaches are designed to capture complex feature interactions, they do so through mathematically distinct and complementary mechanisms.\n\nTo answer, we can start with the math behind HoCNNs and ViTs:\n\nOur Higher-Order Convolution (HoConv) extends the standard convolution. For a given input patch $\\mathbf{x}$, the output $y(\\mathbf{x})$ up to the 3rd order is:\n$y(\\mathbf{x}) = \\sum_i w_1^i x_i+ \\sum_{i,j} w_2^{ij} x_i x_j + \\sum_{i,j,k} w_3^{ijk} x_i x_j x_k + \\dots$\n\nThe Scaled Dot-Product Attention used in Transformers is formulated as:\n$\\text{Attention}(Q, K, V) = \\text{softmax}\\left(\\frac{QK^T}{\\sqrt{d_k}}\\right)V$\n\nHere, the Query ($Q$), Key ($K$), and Value ($V$) matrices are linear projections of the input tokens $\\mathbf{X}$ (e.g., $Q = \\mathbf{X}W_Q$, $K = \\mathbf{X}W_K$, $V = \\mathbf{X}W_V$).\n\n**Parallels and Differences**\n1. The 2nd-Order Parallelism:\nThe most direct parallel exists between our 2nd-order term and the core of the self-attention mechanism. The term $QK^T$ in attention can be expanded as:\n$QK^T = (\\mathbf{X}W_Q)(\\mathbf{X}W_K)^T = \\mathbf{X}(W_Q W_K^T)\\mathbf{X}^T$\nThis is a quadratic form of the input features $\\mathbf{X}$. It computes multiplicative interactions between all pairs of input tokens. Similarly, our 2nd-order convolution, $\\sum w_2^{ij} x_i x_j$, is also a quadratic form, but computed over a local patch.\nParallelism: Both mechanisms model feature relationships through pairwise multiplicative interactions (a quadratic operation).\n\n2. The 3rd-Order Disparity:\nOne might be tempted to see a 3rd-order interaction in attention through the final multiplication by $V$. The full attention operation is approximately $\\text{softmax}(\\mathbf{X}\\mathbf{X}^T)\\mathbf{X}$. However, this is not a true 3rd-order polynomial. The softmax function is a significant nonlinearity that normalizes the quadratic interactions to form weights. The subsequent multiplication by $V$ is a re-weighting or value-mixing operation, not a simple cubic interaction.\n\n**Difference**: Our 3rd-order convolution is a direct, learnable cubic polynomial ($\\sum w_3^{ijk} x_i x_j x_k$). In contrast, attention's three-term structure computes dynamic weights based on pairwise similarity and applies them to a value representation. It does not explicitly model learnable 3-point multiplicative interactions in the same way.\n\nIn conclusion, HoConv provides a powerful mechanism for learning complex local feature detectors with a strong spatial inductive bias. Self-attention provides a flexible mechanism for learning global, long-range dependencies with dynamic, content-based routing of information.\n\nThese differences suggest promising avenues for future research:\n\n* Hybrid architectures: Integrating HoConv in early layers of transformer networks could enhance local feature extraction before global attention processing.\n\n\n* Order-selective attention: Using attention mechanisms to selectively emphasize different orders of convolution based on image content.\n\t\t\t\n\nOur work on higher-order convolutions therefore offers not only immediate performance improvements but also opens new research directions for integrating complementary approaches to visual representation learning. We have included a brief discussion of these potential extensions in the revised Discussion section, and we thank the Reviewer for prompting us to include these interesting suggestions that may spur further developments in the field."}, {"review_id": "WjMeKZN5hO", "reviewer": "Reviewer_tv7t", "summary": "The article aims to build biologically inspired models for computer vision, based on recently proposed Volterra neural networks. Higher-order CNNs are proposed to achieve this goal, by extending existing CNNs architectures. Through standard benchmarks and synthetic datasets, high-order CNNs are shown to outperform traditional CNNs. Perturbation analysis and representational similarity analysis further reveal that different order of convolution process distinct statistical properties of natural images.", "questions": "-\tClarify how the PCA is done in Section 2.1, i.e. on what vectors?\n-\tIn general, traditionally CNNs can also capture well 2nd order information in natural images, in particular for texture classification since CNNs models can lead to performance improvement (see e.g. Deep Filter Banks for texture recognition and segmentation 2010). I am thus not convinced by the results in Table 1 and Fig 2 which shows that the baseline CN performs poorly. This point should be justified under some constraints of baseline CNNs, such as the depth / width/ filter size limitation. It seems not so clear how the baseline CNN is chosen. This makes the methodology part of the results in this article less convincing. \n-\tOn line 193, where in Appendix A.1 you performed additional experiments with a deep and more complex CNN?\n-\tWhat is the size of L and order in HoResNet-18 model on line 204? How about its computational speed compared to the other models compared in Table 2?\n-\tThe ImageNet result in Table 2 has no +/- analysis. At least 5-10 simulations should be made to estimate this number to show that the improvement of HoResNet-18 is significant.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength: The article makes a step forward to build higher-order CNNs, inspired by Volterra neural networks. Contrary to the latter approach, which is built upon polynomial functions, the proposed model is built upon a cascade of polynomial functions and standard ReLU non-linear CNNs layers. A careful quantitative analysis on the benefit of this approach is done on synthetic texture datasets, as well as image classification benchmarks. \n\nWeakness: Certain points on the weakness of CNNs should be made in a more careful way. This makes the results on the role of higher convolution in CNNs less convincing. Numerical results on ImageNet should also be more conclusive. A better explanation of RSA in Figure 4 would be helpful to understand the results.", "was_revised": true, "final_justification": "I thank the authors for their responses which have addressed my concerns. I am happy to raise my score to accept the article given their original contributions to develop more advanced CNNs for classification problems.\n\nI think the choice of the distance in the RSA analysis in Section 4 should be further discussed (i.e. how to Construct Representational Dissimilarity Matrices, based on e.g. correlation distance, Euclidean distance or the Mahalanobis distance). This may give different conclusions, as suggested in [Representational similarity analysis – connecting the branches of systems neuroscience, 2008].", "rebuttal": "Dear Reviewer tv7t,\n\nThank you for your review and constructive feedback. We address your concerns below.\n\n### Q1: PCA methodology clarification (Section 2.1, L 130-135)\n\nThe PCA analysis is performed on the activations after the convolutional block (Conv/HoConv + BatchNorm + ReLU + Max Pooling). Specifically:\n- We generate a fixed - white noise - synthetic input image containing binary textures with all possible 1-, 2-, 3-, and 4-point gliders.\n- We randomly initialize the network 10,000 times\n- For each initialization, we extract the flattened activation vectors after the first convolutional block\n- PCA is performed on these activation vectors (shape: 10,000 × activation_dimension)\n\nThis methodology isolates the architectural inductive biases by examining how different weight initializations affect the representation space. We apologize if our explanations were not exhaustive. We will add these clarifications to the camera-ready manuscript.\n\n----\n### Q2: CNN baseline performance on texture classification\n\nWe appreciate this important critique. You are correct that standard CNNs can capture second-order statistics effectively in many contexts, especially with **sufficient depth**. Our baseline CNN's poor performance on the synthetic texture dataset (Table 1) is indeed due to specific architectural constraints:\n\n1. **Kernel size limitation**: We used 2×2 kernels motivated by the texture generation process (gliders are defined on 2×2 patches) in the first layer;\n2. **Shallow architecture**: Only 2 convolutional layers to isolate the effect of higher-order operations\n3. **Limited capacity**: 10 kernels in first layer, 2 in second layer\n\nWe have chosen these settings to perform a controlled comparison between CNN and HoConv, not to push their performance at maximum. Indeed, as shown in Appendix A.2, larger kernels (5×5, 7×7) in standard CNNs improve image classification - and so they would do for our HoConv. Yet, this also suggests that while larger receptive fields can help capturing higher-order correlations, our explicit higher-order formulation provides a more generalizable solution even for small kernels. We will explain this clearly in the camera-ready manuscript\n\n----\n### Q3:App. A.1,  Location of deep CNN experiments\n\nIt’s in the second paragraph of A.1. We apologize for the confusion due to an unclear organization of appendix A.1. We will introduce a subsection “Comparison with deeper CNNs”. The deep CNN architecture has:\n- Eight convolutional blocks (3×3 kernels)\n- Channel progression: 8→8→16→16→32→32→64→64\n- Total parameters: 108,130 (25% more than HoCNN)\n- Performance: 71.20% on CIFAR-10 (vs 72.35% for HoCNN)\n\n----\n### Q4: HoResNet-18 architecture details\n\nIn HoResNet-18:\n- **Order**: We use up to 3rd order expansions (1st + 2nd + 3rd order terms)\n- **Location**: Higher-order blocks are used only in Stage 1 (first 2 residual blocks)\n- **Channels**: 30 channels in the higher-order stage (vs 64 in standard ResNet-18)\n\nAs already reported in App. A.1.1, the computational speed values are:\n- Standard ResNet-18: 1.82 GFLOPs\n- HoResNet-18: 3.00 GFLOPs (1.65× increase)\n\n----\n### Q5: Statistical significance for ImageNet\n\nYou raise a valid point about the lack of error analysis for ImageNet results. Due to computational constraints, we performed single runs at the time of the submission. However, we acknowledge this limitation and agree that multiple runs would strengthen our claims. We have been able to collect results for 3 runs at the moment (we are currently running more to get more reliable statistics), results are reported in the table below:\n\n\n| Run  | ResNet-18 (%)   | HoResNet-18 (%)  | Difference (%) |\n|------|-----------------|------------------|----------------|\n| 1    | 68.53           | 69.38            | +0.85          |\n| 2    | 68.72           | 69.61            | +0.89          |\n| 3    | 68.60           | 69.40            | +0.82          |\n| Mean | 68.62 ± 0.10    | 69.46 ± 0.13     | +0.84          |\n\n\n----\n### Additional clarifications on RSA (Figure 5)\n\nRegarding the RSA analysis, we compare representational geometries between CNN and HoCNN:\n- We extract activations from 100 CIFAR-10 test images (10 per class, randomly selected)\n- Compute pairwise distances between activation vectors\n- Construct Representational Dissimilarity Matrices (RDMs)\n- The log-ratio visualization (panel E) highlights where representations differ most\n\nThe key finding is that HoCNN creates more dispersed representations (Larger Log-ratios in Fig. 4E and Figure 11D of Appendix A.9.1), enabling better class separation. We will increase manuscript clarity here.\n\nWe hope these clarifications address your concerns and demonstrate the rigor of our approach."}, {"review_id": "r3kFeSysXZ", "reviewer": "Reviewer_CXmf", "summary": "The authors present an architecture named HoConv that extend standard CNNs with higher-order convolutions, and tested it on image classification.\nA 2D input patch is first linearized into a vector x and then used to perform a set of convolution operations: 1D, 2D, 3D, etc..\nEach time the input is the respective outer products (x, x * x, x * x * x, ...).\nThis helps capture some non-linear interactions between the features in x, not directly possible with one standard convolutional layer.", "questions": "What does the log ratio in Figure 4E show about HoConv? I am not sure how to interpret the visual patterns in it.\n\nDid you consider applying your work on tasks beyond Image Classification? Some tasks might benefit greatly from the ability of HoConv to better capture highy-order image statistics.\n\nDid you consider using a weights sum of the individual convolutions (1st order + w2 * 2nd order + w3 * 3rd order) with learnable weights instead of plain summation? I understand that w2 is just a scaling factor for the W2 matrix, but it would likely improve convergence since it is hard for the individual weights in W2 to learn the same scaling factor all at once.", "limitations": "Discussed in Sec. 5.1 to a good extent.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- AFAIS, the proposed architecture is novel. Similar extensions to CNNs I am aware of are not based on outer products.\n- The authors highlight the presence of tied weights as in inherent shortcoming of standard CNNs that HoConv helps mitigate. \n- The authors designed experiments that demonstrate how HoConv is more sensitive to higher-order image statistics than standard CNNs, based on a deliberately generated texture dataset.\n\nWeaknesses:\n- The comparison with baseline CNNs in terms of accuracy is problematic. It is hard to maintain an exact number of parameters between both models.  Even then, those are not utilized in similar ways, and the FLOPS differences are huge. This latter issue alone gives an unfair advantage to the new method. To appreciate how important that is, consider re-training some popular ImageNet methods for a few more epochs. It is really easy to squeeze  ~1% accuracy improvement on ImageNet purely based on that. \n- Following up on point 1, there are other ways to show the merits of a new architecture, beyond beating baselines on standard benchmarks. I really appreciate your experiment on texture classification which does that very well. I wish there were more targeted examples that highlight those merits. Wouldn't fine-grained semantic segmentation tasks be great candidates? A shallow U-Net struggles a lot with small objects (e.g. traffic lights in CityScapes) and with producing a smooth borders between objects. I felt these are excellent examples where better leveraging natural image statistics would give you an edge.\n- AFAIS, the manuscript does not provide evidence that your method leverages natural image statistics (a key motivation for HoConv), such as the one cited from Olshausen and Field. Synthetic texture images were great to demonstrate higher-order image statistics, but those are far from natural images. The book by Hyvärinen et al provide a variety of further examples (building on the work of Olshausen, Field, and a variety of other authors):\n\nHyvärinen, Aapo, Jarmo Hurri, and Patrick O. Hoyer. Natural image statistics: A probabilistic approach to early computational vision. Vol. 39. Springer Science & Business Media, 2009.\n\nTypos:\n- hellinger -> capitalize\n- on Texture classification -> de-capitalize", "was_revised": true, "final_justification": "The contribution is solid,  and the benefits of the proposed architecture are backed by strong experimental results.", "rebuttal": "Dear Reviewer CXmf,\n\nThank you for your thoughtful review and valuable suggestions. We address your concerns below.\n\n### W1: Fair comparison with baseline CNNs\n\n1. You raise an important point about the FLOP differences: matching parameter numbers might not be fair if FLOPs are significantly different. To account for this, in App. A.1.2, LL464-472 we indeed compared HoCNN with a deep CNN having similar computational budget on CIFAR-10:\n\n   * Deep CNN: 7.09M FLOPs, 71.20% accuracy\n   * HoCNN: 6.81M FLOPs, 72.06% accuracy\n\n   Even with slightly fewer FLOPs, HoCNN achieves 1.21% higher accuracy, demonstrating that improvements aren't merely from increased computation.\n\n2. **Parameter-matched comparisons**: Throughout our experiments (Table 2), we ensured HoCNN has comparable or fewer parameters than baseline CNNs while achieving consistent improvements. In Appendix A.1 (second paragraph) we also report a case where HoConv outperforms a CNN with 25% more parameters.\n\n\n---\n### W2: Applications beyond standard benchmarks\n\nWe greatly appreciate your insightful suggestion about exploring tasks beyond classification. Your comment motivated us to conduct additional experiments, and we're pleased to report promising initial results.\nFollowing your suggestion, we investigated fine-grained visual tasks using the CUB-200-2011 dataset (Wah et al 2011). This dataset, consisting of 200 bird species (classes) is one of the benchmarks of fine grained classification and also presents segmentation masks, it  is particularly challenging due to subtle inter-class differences and limited training data (~6k images), making it an excellent test case for our approach. \n\n**New results:**\n\n1. **Fine-grained classification**: We fine-tuned ImageNet-pretrained ResNet18 and HoResNet18 on CUB-200-2011. Our HoResNet18 showed even larger improvements than on standard benchmarks (72.31 % +- 0.29% vs ResNet18’s 68.24 % +- 0.37%), suggesting higher-order convolutions are particularly beneficial for capturing subtle discriminative patterns. Our result is inline with bilinear pooling CNNs (cite) which are somehow similar to our HoConv, even though the \"higher-order\" mechanism is at the level of the fully connected (last) layer in those\n\n2. **Segmentation**: A key challenge was developing a transposed version of our higher-order convolution for decoder architectures. We successfully implemented this and built U-Net architectures using our pretrained encoders. The HoResNet18-based U-Net consistently outperformed the ResNet18 baseline in mean IoU (87.72 % +- 0.68% mIoU vs ResNet18’s 82.17 % +-  0.75%), demonstrating effective transfer to dense prediction tasks.\n\nThese results validate your intuition that tasks requiring precise local pattern detection benefit from our approach, we will include them in the paper, and we are extremely grateful to this Reviewer for prompting us to explore this novel direction: we will add these results to the appendix. However, even if these preliminary results are very encouraging, we believe that an in depth analysis on other datasets and models would be outside the scope of our paper. Here, instead of testing multiple computer vision tasks, we preferred to focus on image classification, and deepen the understanding of why/how our HoConv layer is effective in capturing higher order correlations in natural images.  \n\n---\n### W3: Connection to natural image statistics\n\nWe acknowledge that our synthetic textures differ from natural images. However,  our perturbation analysis (Section 4 and Fig. 4A and B) exactly show that HoConv leverages higher order correlations to perform classification. There we demonstrate that HoCNN's increased sensitivity to higher-order statistical perturbations in natural images (CIFAR-10) translates to better classification performance, suggesting the model leverages these statistics effectively. \n  \nAdditionally,  **Koenderink (2018) alignment**, our finding that performance peaks at 3rd/4th order directly aligns with natural image statistics where quadratic, cubic, and quartic powers dominate approximately 63%, 35%, and 2% of pixels respectively (Section 5).\n\nFinally, the benchmark models we used for comparison have been developed to perform classification tasks on those datasets, so we felt that we should outperform them within the domain they were developed for to make it a fair comparison. If we select an arbitrary database, showing that our architecture outperforms others may raise suspicion that this result reflects our choice of database, rather than the strength of our model.\n\n\n---\n### Q1: Interpretation of log ratio in Figure 5E\nThe log ratio visualization (Figure 5E) highlights regions where the representational geometries of CNN and HoCNN differ most. Positive values (red) indicate image pairs that HoCNN represents as more dissimilar than CNN, while negative values (blue) show the opposite. The structured pattern reveals that HoCNN creates more distinct class-specific representations, particularly enhancing inter-class dissimilarity while maintaining intra-class similarity. This is complemented by Figure 11 in Appendix where it is possible to appreciate the increased dissimilarity in HoConv’s representations from a distribution perspective. \n\n---\n### Q2: Applications beyond image classification\nYour question prompted valuable exploration. Beyond the CUB-200-2011(Wah et al 2011) results mentioned above, we see potential for higher-order convolutions in:\n\n- **Medical imaging**: Where subtle texture patterns and local correlations are diagnostic\n\n- **Dense Prediction Tasks**: Our new transposed higher-order convolution enables full encoder-decoder architectures for segmentation\n\nThe development of transposed higher-order convolutions significantly expands the applicability of our approach beyond classification. In future work, we intend to explore these directions in more detail. \n\n---\n### Q3: Weighted summation of orders\nThis is an excellent suggestion. We actually implemented and tested this approach during our initial experiments, using learnable coefficients to weight different orders (1st + w2×2nd order + w3×3rd order). Interestingly, we didn't observe consistent improvements over our current implementation with plain summation and normalization factor $s = 1/\\sqrt{n_V}$ (Section 2).\nThe learned weights tended to converge to similar values across runs, suggesting our normalization strategy already provides appropriate scaling. However, we agree that explicit learnable weights could offer more flexibility for certain applications and merit further investigation in domain-specific contexts.\n\n---\n### Typo corrections\nThank you for catching these:\n- We'll capitalize \"Hellinger\" throughout\n- We'll change to \"texture classification\" (lowercase)\n\nWe sincerely thank you for your stimulating comments that led to these additional insights and results. Your suggestions have significantly strengthened our work and we hope that our response helped to clarify your doubts."}, {"review_id": "BJZJj3euiB", "reviewer": "Reviewer_Mz2V", "summary": "The authors propose a novel convolution operator, which aims to solve the tied-weight issue. They intuitively derive this issue via a polynomial expansion of the classical convolution operator and address it by introducing higher orders of convolution into a single convolution layer. According to the paper, higher orders are enabled by the outer product of the two operands of the classical linear-weighted sum, along with an inner product. Experiments show the effectiveness of their method.", "questions": "**Line 90~91** -> ***solved***\n> This tied-weight issue becomes particularly problematic when the network is not deep enough or wide enough to compensate...\n\nOn the one hand, does this mean that this method only shows significant benefit on shallow or narrow CNN models?\nOn the other hand, it seems necessary to provide experiment results to prove that shallow or narrow CNN models can be greatly augmented even beating relatively deep or wide CNN models.\n\n\n**Figure 1** -> ***addressed***\n> Figure 1.\n\nOnly spatial transformation is visualized, so how is the channel transformation implemented? I guess there must be channel transformation, i.e., not a depth-wise-like convolution, or the channel expansion can not be realized from 3, to 30, 128, 256 and 512.\n\nBesides, what are the similarities between the authors' high-order design and the biological structure in the visual system? Providing this would help readers to follow your intuition a lot.\n\n\n**Table 1** -> ***addressed***\n> Table 1.\n\nIt would be better to clearly point out why the number of parameters of 2nd order is even lower than the baseline. As higher orders require much more learnable weights, readers would be doubtful about the numbers unless they dive deep into the appendix.\n\n\n**Table 2** -> ***responded***\n> Table 2.\n\nThe baseline performance, 68.67 of CNN (ResNet18) on ImageNet, is relatively too weak [1] and authors' method even is not as good as the recognized performance of the baseline, i.e., 69.9. It would be better that the authors run their experiments on some well-recognized codebase [1].\n\n\n**Experiments** -> ***the work's effectiveness on segmentation is only validated on a very old toy dataset (agreed by the authors); those of detection is still not responded***\n> Experiments\n\nThe experiments should cover those primary CV tasks, like object detection, instance segmentation or semantic segmentation, or the current reults can not fully support the authors' key claim.\n\n> A.6 Comparison with VOneNet -> ***partly responded, not satisfying***\n\nComparision experiments with other bio-inspired methods should be placed in the main text. If more other bio-inspired methods, CNN methods like CoC [2] and even ViT methods like TransNeXt [3], are provided, it would make this work much more solid.\n\n\n[1] https://github.com/open-mmlab/mmpretrain/tree/main/configs/resnet#image-classification-on-imagenet-1k\n\n[2] Convolution of Convolution: Let Kernels Spatially Collaborate\n\n[3] TransNeXt: Robust Foveal Visual Perception for Vision Transformers", "limitations": "Yes.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n---\n\n1. This work seems promising to me, but the experiments are too limited, just on a tentative level. It would lead to good significance to the community if the authors can supplement more concrete work, like well-design architectures with off-the-shelf pretrained model weights. Good examples are Res2Net, ConvMixer, ConvNeXt, DINO2, HorNet [1], and so on.\n\n2. The authors derive the tied weight issue intuitively using polynomial expanding classifical convolution. To the best of my knowledge, there is no such work ever addressing such a problem.\n\n\nWeaknesses\n---\n\n1. The quality of this paper is quite limited. Key claims are not fully addressed. For example, to fully prove the effectiveness of a novel operator, experiments on primary Computer Vision tasks should be covered, not just classification, also including detection and segmentation. Good examples are mentioned above.\n\n=> ***The work's effectiveness of segmentation is only shown on a very old toy dataset (agreed by the authors); Those of detection has not been validated.***\n\n2. The clarity is this paper is fair. Key designs of this method is clearly stated, but the corresponding biological inspiration is not, making the biological part a mere decoration of the storytelling. A good example could be CoC [2].\n\n=> ***This point was repsonded but not satisfying, especially considering that the authors put the core claim of biological inspiration in their title.***\n\n[1] https://github.com/open-mmlab/mmpretrain/tree/main/configs\n\n[2] Convolution of Convolution: Let Kernels Spatially Collaborate.", "was_revised": true, "final_justification": "Update 3\n---\nThe authors still have not fully addressed my central concern. While I perceive their response as avoiding this core issue, a positive aspect is that their rebuttal has reinforced my assessment.\n\n\n\nUpdate 2\n---\nI would like to re-acknowledge the Significance and Originality or this work. Yet the Quality still needs improvement after rebuttal.\n\nAs for primary cv tasks, the response is not satisfying. The authors use a dataset that is even older and easier than the toy dataset Pascal VOC 2012, let alone the classical benchmark MS COCO 2017. In CUB-200-2011, the dataset used by the authors, there is merely one object box/segment. in each image sample. This is too weak to truely test the effectiveness of a novel conv op in any modern CNN model. Just like in network pruning, some authors claim their algo can prune 99% of a CNN's parameters without undermining the accuracy on CIFAR10 -- Of course this is possible yet meaningless, because most of CV tasks require (pre-)training on ImageNet-scale datasets.\n\nThe results of detection is also not provided.\n\nPlease note that the three primary CV tasks are classification, detection and segmentation. All works that claim they propose an effective novel conv op should and they in fact have tested their method on those primary CV tasks -- Please kindly check the references mentioned above. Please also note that this is the key point supporting my final rating.\n\nAs for biological inspiration, the response is okay yet still provides too limited insightful analogy between the op and the bio to convince the readers that this is not a decoration of storytelling.\n\nAs for parameter efficiency, the response is good.\n\nIn short, I sincerely appreciate the authors' efforts in addressing my concerns and truly expect to see the authors providing the necessary results to make their novel work solid enough.\n\n\n\nUpdate 1\n---\nThank the authors for their efforts in rebuttal. I appreciate the revisions being made. However, after careful consideration, I still believe the initial rating accurately reflects the contribution of the work. I understand the effort involved in the revisions and the time constraints, and I hope my feedback will be helpful for future improvements.\nSpecifically, for example, experiments on those widely-recognized CV tasks should be MS COCO for detection and segmentation, instead of those outdated toy datasets.", "rebuttal": "Dear Reviewer Mz2V,\n\nThank you for recognizing significance and originality of our work. We appreciate your constructive feedback and address your concerns below.\n\n\n### S1 off-the-shelf pretrained model \n\nWe appreciate the reviewer’s suggestion to compare our approach with advanced architectures. In principle, integrating Higher-Order Convolutions into these state-of-the-art networks is an exciting direction. However, each of these models incorporates unique design optimizations – for example, Res2Net introduces multi-scale feature groups within residual blocks, ConvMixer relies on patch-level mixing of features, and ConvNeXt employs large convolutional kernels with specific normalization patterns. Adapting our higher-order convolution mechanism to each such architecture would require careful, case-by-case architectural tuning, effectively resulting in designing and training multiple new hybrid models. Albeit being very relevant future directions, our primary objective was to introduce the HoConv layer as a general concept and demonstrate its effectiveness in a controlled setting, using standard CNN backbones. We focused on baseline architectures to isolate the benefits of higher-order convolutions without the confounding factors introduced by complex architectural tricks. We believe this strategy provides clear evidence of our method’s merit. Expanding the evaluation to every cutting-edge architecture, while valuable, would significantly broaden the scope and detract from the core contribution of our work. We view such extensive comparisons and integrations as future work that extend beyond the space constraints and goals of the current paper.\n\nIn summary, we agree that incorporating HoConv into modern architectures is an important next step, and we thank the reviewer for these suggestions. We will note this in the revised manuscript as a future direction. For now, we hope the results provided (see below for additional tasks) sufficiently validate the effectiveness of our proposed operator.\n\n---\n### W1 Experiments: Extension to primary CV tasks\n\nWe agree with the reviewer that our HoConv layer can provide significant improvement also for other tasks in computer vision.  Following similar suggestions from Reviewer #3, we tested our approach on the CUB-200-2011 dataset (Wah et al 2011). This dataset, consisting of 200 bird species (classes) is one of the benchmarks of fine grained classification and also presents segmentation masks, it  is particularly challenging due to subtle inter-class differences and limited training data (~6k images), making it an excellent test case for our approach. We performed two tasks:\n\n* **Fine-grained classification**: Using, HoResNet18 showed larger improvements than on standard benchmarks (72.31 % +- 0.29% vs ResNet18’s 68.24 % +- 0.37%), validating that higher-order convolutions excel at capturing subtle discriminative patterns.\n\n\n* **Segmentation**: We developed a transposed higher-order convolution layer for decoder architectures, built a U-Net model using our pretrained encoder and used them in the segmentation task. The HoResNet18-based U-Net consistently outperformed the ResNet18 baseline in mean IoU (87.72 % +- 0.68% mIoU vs ResNet18’s 82.17 % +-  0.75%).\n\nThese preliminary results are very encouraging and we will add them in the appendix, but we believe that an in depth analysis would be outside the scope of our paper. Here, we preferred to deepen the understanding of why/how our HoConv layer is effective in capturing higher order correlations in natural images, instead of testing multiple computer vision tasks beyond image classification.\n\n---\n### W2 Biological inspiration clarity\n\nWe acknowledge that biological inspiration needs clearer articulation. A significant amount of literature (ex: Fitzgerald and Clark, eLife, 2015) has shown that neural circuits in early visual systems respond to higher order correlations, beyond pairwise. To implement this response, those circuits use non-linearities and products of more than two local filters applied to their inputs, as for example the multiplication between a low and a high-pass filter convoluted with one image patch with a high-pass convoluted with another patch. A standard convolution layer cannot reproduce those responses (Fitzgerald and Clark,  2015), and to mimic those behaviors CNNs need to stack multiple layers, going much deeper than the visual system itself. Inspired by these limitations of standard CNNs, we developed our higher-order convolution layer, and show that it can capture high order correlations present in the input image.\n\nAfter a first draft going deeper into the comparison between our model and early visual systems, we focused the paper on the ML architecture, removing those biological arguments from the manuscript to improve focus and readability. However the referee is right, as those insights can be helpful to understand the logic beyond our novelties,  we will add them back in a dedicated appendix in the camera-ready version. For this reason, we have extended relevant sections to provide more details regarding biological motivation, as suggested by the Reviewer. Additionally, forthcoming results in a companion paper will showcase how our HoConv layer helps improve prediction of neuronal responses across animal species to visual stimuli\n\n---\n### Q (Line 90-91): Shallow vs deep networks\n\nNo, our HoConv layer is not beneficial only for shallow networks, as shown by our results on difficult image classification datasets (Table 2), where networks are deep.  Additionally, beyond the comparison with CNN matching the number of parameters, in App. A.1 we show that a shallow HoConv network can beat a deeper CNN with 25% more parameters.\nIn summary, our claim isn't that the method only benefits shallow networks, but rather that it provides an efficient alternative to depth/width for capturing the complex correlation patterns present in natural images.\n\n---\n### Q (Figure 1): Channel transformation implementation\n\n* For each order, we have separate learnable weights of shape (out_channels, in_channels, kernel_space_size^order)\n* The input is unfolded into patches, and the higher-order kernel computes higher-order interactions within each patch\n* These are then multiplied by the weight matrix to perform channel transformation\n* All orders (1st, 2nd, 3rd, etc.) are summed to produce the final output\n\nEach order learns its own channel transformation weights (since different orders’ tensors have different shapes), allowing the network to discover which higher-order spatial patterns are most relevant for each output channel.\n\nFor Biological inspiration see above, W2 response.\n\n---\n### Q (Table 1): Parameter count clarification\n\nWe appreciate your concern about the parameter count, which is a crucial ingredient for appreciating HoConv networks. For the results reported in table 1, the CNN has more parameters than the 2nd order HoConv because its first layer is wider (10 channels for the CNN against 2 for the HoConv), while the subsequent layers have the same architecture.  This is explained in LL167-168 and LL172. We have chosen this setting exactly to show that a thin HoConv can outperform a wider CNN with more parameters. On the same line, as shown in App. A.1, HoConv can even outperform a deeper CNN with 25% more parameters.\n\n--- \n### Q Table 2: ImageNet baseline performance\n\nWe agree with the referee, our CNN baseline performance doesn’t match the SOTA by the open-mmlab codebase. The reason is that we trained both models from scratch - following the learning details provided in the manuscript - without the hyperparameters, preprocessing and augmentation tricks that have been designed to push performance at maximum. In our opinion this choice is necessary to allow a controlled comparison between CNN and HoConv networks: It would not have been fair to include those tricks developed for CNNs over the course of years, without doing the same for HoConv. We plan to work on incorporating those tricks in the future and generalize them also for HoConvs, but for the purpose of comparing CNN and HoConv on a level plain field it does not seem critical that they are included.\n\n**Experiments**  See response to W1 above\n\n--- \n### Q Comparison with other bio-inspired architectures\n\nThank you for suggesting our comparison with VOneNet be in the main text. While we agree this is an important comparison, as VOneNet is another bio-inspired convolutional model, we considered it a secondary result that complements our primary findings. Our main focus was to demonstrate the fundamental advantages of higher-order convolutions over standard CNNs, rather than comparing different bio-inspired methods. We will move this comparison as suggested, but retain its role as supporting evidence rather than a central claim.\n\nWe appreciate you bringing “Convolution of Convolution” (CoC) to our attention—we were unaware of this work and will discuss it in the camera-ready version. Their architecture is not an alternative, but rather complementary to our HoConv layer, as HoConv layers could be integrated into CoC. For this reason, benchmarking HoConv against CoC is beyond our contribution’s scope.\n\nRegarding the TransNeXt comparison, we note that it is a Vision Transformer architecture, which is beyond our scope. Our primary goal was to provide a focused study on extending convolution operations, not to compare across fundamentally different paradigms. This focus allows us to isolate the specific benefits of higher-order operations with controlled comparisons.\n\nWe believe our current experimental design, centered on controlled comparisons with standard CNNs, effectively demonstrates the benefits of our approach while keeping the scope manageable given our computational constraints."}]

Author rebuttal:

Meta-review: This paper presents a compelling and novel extension to CNNs through the introduction of learnable, biologically-inspired higher-order convolutions (HoConv), which employ a Volterra-like expansion to capture multiplicative interactions and higher-order statistical dependencies in natural images. This work stands out through its clear biological motivation as well as its careful experimental design and analysis, which demonstrates consistent performance gains over CNN baselines across multiple benchmarks and provides valuable analysis of the distinct representational properties of each convolution order. While reviewers tv7t and CXmf initially raised valid concerns regarding the choice of baselines, the statistical significance of the ImageNet results, the limited task validation beyond classification, and the depth of the biological inspiration, the authors' rebuttal effectively addressed the majority of these points and they increased their score. Although Mz2V maintained that validation on primary vision tasks like detection and segmentation on modern benchmarks (e.g., MS COCO) would be ideal, the AC finds the paper's contributions to be sufficiently significant, technically solid and insightful for acceptance in its current form.
Decision: Accept (poster)