Title: Convolution Goes Higher-Order:  A Biologically Inspired Mechanism Empowers Image Classification
Venue: NeurIPS 2025
Accepted: yes
Keywords: Higher-order convolution

Biologically inspired neural networks

Image classification

Convolutional Neural Networks

Biological visual processing

Neural representations
Abstract: We propose a novel enhancement to Convolutional Neural Networks (CNNs) by incorporating learnable higher-order convolutions inspired by nonlinear biological visual processing. Our model extends the classical convolution operator using a Volterra-like expansion to capture multiplicative interactions observed in biological vision. Through extensive evaluation on standard benchmarks and synthetic datasets, we demonstrate that our architecture consistently outperforms traditional CNN baselines, achieving optimal performance with 3rd/4th order expansions. Systematic perturbation analysis and Representational Similarity Analysis reveal that different orders of convolution process distinct aspects of visual information, aligning with the statistical properties of natural images. This biologically-inspired approach offers both improved performance and deeper insights into visual information processing.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel convolution operator, which aims to solve the tied-weight issue. They intuitively derive this issue via a polynomial expansion of the classical convolution operator and address it by introducing higher orders of convolution into a single convolution layer. According to the paper, higher orders are enabled by the outer product of the two operands of the classical linear-weighted sum, along with an inner product. Experiments show the effectiveness of their method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n---\n\n1. This work seems promising to me, but the experiments are too limited, just on a tentative level. It would lead to good significance to the community if the authors can supplement more concrete work, like well-design architectures with off-the-shelf pretrained model weights. Good examples are Res2Net, ConvMixer, ConvNeXt, DINO2, HorNet [1], and so on.\n\n2. The authors derive the tied weight issue intuitively using polynomial expanding classifical convolution. To the best of my knowledge, there is no such work ever addressing such a problem.\n\n\nWeaknesses\n---\n\n1. The quality of this paper is quite limited. Key claims are not fully addressed. For example, to fully prove the effectiveness of a novel operator, experiments on primary Computer Vision tasks should be covered, not just classification, also including detection and segmentation. Good examples are mentioned above.\n\n=> ***The work's effectiveness of segmentation is only shown on a very old toy dataset (agreed by the authors); Those of detection has not been validated.***\n\n2. The clarity is this paper is fair. Key designs of this method is clearly stated, but the corresponding biological inspiration is not, making the biological part a mere decoration of the storytelling. A good example could be CoC [2].\n\n=> ***This point was repsonded but not satisfying, especially considering that the authors put the core claim of biological inspiration in their title.***\n\n[1] https://github.com/open-mmlab/mmpretrain/tree/main/configs\n\n[2] Convolution of Convolution: Let Kernels Spatially Collaborate.\n\nQUESTIONS:\n**Line 90~91** -> ***solved***\n> This tied-weight issue becomes particularly problematic when the network is not deep enough or wide enough to compensate...\n\nOn the one hand, does this mean that this method only shows significant benefit on shallow or narrow CNN models?\nOn the other hand, it seems necessary to provide experiment results to prove that shallow or narrow CNN models can be greatly augmented even beating relatively deep or wide CNN models.\n\n\n**Figure 1** -> ***addressed***\n> Figure 1.\n\nOnly spatial transformation is visualized, so how is the channel transformation implemented? I guess there must be channel transformation, i.e., not a depth-wise-like convolution, or the channel expansion can not be realized from 3, to 30, 128, 256 and 512.\n\nBesides, what are the similarities between the authors' high-order design and the biological structure in the visual system? Providing this would help readers to follow your intuition a lot.\n\n\n**Table 1** -> ***addressed***\n> Table 1.\n\nIt would be better to clearly point out why the number of parameters of 2nd order is even lower than the baseline. As higher orders require much more learnable weights, readers would be doubtful about the numbers unless they dive deep into the appendix.\n\n\n**Table 2** -> ***responded***\n> Table 2.\n\nThe baseline performance, 68.67 of CNN (ResNet18) on ImageNet, is relatively too weak [1] and authors' method even is not as good as the recognized performance of the baseline, i.e., 69.9. It would be better that the authors run their experiments on some well-recognized codebase [1].\n\n\n**Experiments** -> ***the work's effectiveness on segmentation is only validated on a very old toy dataset (agreed by the authors); those of detection is still not responded***\n> Experiments\n\nThe experiments should cover those primary CV tasks, like object detection, instance segmentation or semantic segmentation, or the current reults can not fully support the authors' key claim.\n\n> A.6 Comparison with VOneNet -> ***partly responded, not satisfying***\n\nComparision experiments with other bio-inspired methods should be placed in the main text. If more other bio-inspired methods, CNN methods like CoC [2] and even ViT methods like TransNeXt [3], are provided, it would make this work much more solid.\n\n\n[1] https://github.com/open-mmlab/mmpretrain/tree/main/configs/resnet#image-classification-on-imagenet-1k\n\n[2] Convolution of Convolution: Let Kernels Spatially Collaborate\n\n[3] TransNeXt: Robust Foveal Visual Perception for Vision Transformers\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an architecture named HoConv that extend standard CNNs with higher-order convolutions, and tested it on image classification.\nA 2D input patch is first linearized into a vector x and then used to perform a set of convolution operations: 1D, 2D, 3D, etc..\nEach time the input is the respective outer products (x, x * x, x * x * x, ...).\nThis helps capture some non-linear interactions between the features in x, not directly possible with one standard convolutional layer.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- AFAIS, the proposed architecture is novel. Similar extensions to CNNs I am aware of are not based on outer products.\n- The authors highlight the presence of tied weights as in inherent shortcoming of standard CNNs that HoConv helps mitigate. \n- The authors designed experiments that demonstrate how HoConv is more sensitive to higher-order image statistics than standard CNNs, based on a deliberately generated texture dataset.\n\nWeaknesses:\n- The comparison with baseline CNNs in terms of accuracy is problematic. It is hard to maintain an exact number of parameters between both models.  Even then, those are not utilized in similar ways, and the FLOPS differences are huge. This latter issue alone gives an unfair advantage to the new method. To appreciate how important that is, consider re-training some popular ImageNet methods for a few more epochs. It is really easy to squeeze  ~1% accuracy improvement on ImageNet purely based on that. \n- Following up on point 1, there are other ways to show the merits of a new architecture, beyond beating baselines on standard benchmarks. I really appreciate your experiment on texture classification which does that very well. I wish there were more targeted examples that highlight those merits. Wouldn't fine-grained semantic segmentation tasks be great candidates? A shallow U-Net struggles a lot with small objects (e.g. traffic lights in CityScapes) and with producing a smooth borders between objects. I felt these are excellent examples where better leveraging natural image statistics would give you an edge.\n- AFAIS, the manuscript does not provide evidence that your method leverages natural image statistics (a key motivation for HoConv), such as the one cited from Olshausen and Field. Synthetic texture images were great to demonstrate higher-order image statistics, but those are far from natural images. The book by Hyvärinen et al provide a variety of further examples (building on the work of Olshausen, Field, and a variety of other authors):\n\nHyvärinen, Aapo, Jarmo Hurri, and Patrick O. Hoyer. Natural image statistics: A probabilistic approach to early computational vision. Vol. 39. Springer Science & Business Media, 2009.\n\nTypos:\n- hellinger -> capitalize\n- on Texture classification -> de-capitalize\n\nQUESTIONS:\nWhat does the log ratio in Figure 4E show about HoConv? I am not sure how to interpret the visual patterns in it.\n\nDid you consider applying your work on tasks beyond Image Classification? Some tasks might benefit greatly from the ability of HoConv to better capture highy-order image statistics.\n\nDid you consider using a weights sum of the individual convolutions (1st order + w2 * 2nd order + w3 * 3rd order) with learnable weights instead of plain summation? I understand that w2 is just a scaling factor for the W2 matrix, but it would likely improve convergence since it is hard for the individual weights in W2 to learn the same scaling factor all at once.\n\nLIMITATIONS:\nDiscussed in Sec. 5.1 to a good extent.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article aims to build biologically inspired models for computer vision, based on recently proposed Volterra neural networks. Higher-order CNNs are proposed to achieve this goal, by extending existing CNNs architectures. Through standard benchmarks and synthetic datasets, high-order CNNs are shown to outperform traditional CNNs. Perturbation analysis and representational similarity analysis further reveal that different order of convolution process distinct statistical properties of natural images.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: The article makes a step forward to build higher-order CNNs, inspired by Volterra neural networks. Contrary to the latter approach, which is built upon polynomial functions, the proposed model is built upon a cascade of polynomial functions and standard ReLU non-linear CNNs layers. A careful quantitative analysis on the benefit of this approach is done on synthetic texture datasets, as well as image classification benchmarks. \n\nWeakness: Certain points on the weakness of CNNs should be made in a more careful way. This makes the results on the role of higher convolution in CNNs less convincing. Numerical results on ImageNet should also be more conclusive. A better explanation of RSA in Figure 4 would be helpful to understand the results.\n\nQUESTIONS:\n-\tClarify how the PCA is done in Section 2.1, i.e. on what vectors?\n-\tIn general, traditionally CNNs can also capture well 2nd order information in natural images, in particular for texture classification since CNNs models can lead to performance improvement (see e.g. Deep Filter Banks for texture recognition and segmentation 2010). I am thus not convinced by the results in Table 1 and Fig 2 which shows that the baseline CN performs poorly. This point should be justified under some constraints of baseline CNNs, such as the depth / width/ filter size limitation. It seems not so clear how the baseline CNN is chosen. This makes the methodology part of the results in this article less convincing. \n-\tOn line 193, where in Appendix A.1 you performed additional experiments with a deep and more complex CNN?\n-\tWhat is the size of L and order in HoResNet-18 model on line 204? How about its computational speed compared to the other models compared in Table 2?\n-\tThe ImageNet result in Table 2 has no +/- analysis. At least 5-10 simulations should be made to estimate this number to show that the improvement of HoResNet-18 is significant.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Higher-Order Convolutions (HoConv), a learnable extension of classical linear convolution operators that captures higher-order correlations in natural images beyond first-order (linear) relationships. Inspired by the observation that natural scenes exhibit complex statistical dependencies, the authors design a convolutional block that computes and sums responses from filters of order 1 through 4. Each order captures different correlation structures in the image, and the design ensures parameter independence across orders to facilitate more flexible learning. Compared to classical CNNs of similar or even higher parameter counts, HoCNNs consistently outperform on multiple image classification benchmarks. The authors also provide insight into how these higher order convolutions work, by the representational capacity of HoCNNs via synthetic data and PCA-based visualizations.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Clear motivation and originality: The paper is well-motivated by limitations of traditional CNNs in capturing higher-order pixel correlations. The proposed approach is a principled extension of convolution, grounded in signal statistics and biological plausibility.\n- Novel architecture: While prior work has explored 2nd-order filters, this paper generalizes the idea up to 4th order in a clean, modular way, while ensuring learnability and independence of parameters across orders.\n- Strong experimental design: The authors test the model across a variety of vision benchmarks and provide statistical evidence (via repeated experimental runs) to support the robustness of their improvements.\n- Analysis beyond benchmarks: The paper goes beyond classification accuracy to examine what different orders of convolution capture, using PCA and synthetic datasets.\n- Clarity: The manuscript is overall very well-written and accessible. The motivation, implementation details, and experiments are communicated clearly.\n\n### Weaknesses\n\n- Visualization gap: In Figure 1B and section 2.1, the authors reference a synthetic image containing binary textures that exhibit all 1–4 point correlations within 2×2 patches. This image is not shown, and it’s difficult to develop intuition for it without a visual example.\n- Questionable analysis choice: In the PCA-based analysis comparing CNN vs HoCNN representations, each data point corresponds to a random weight initialization, rather than a fixed network processing different images. This design decision may limit the interpretability of the representation space analysis.\n- Minor clarity/consistency issues: Figure 4 panel labels (C and D) are mismatched between figure and caption. Similarly, textual references to these panels are inconsistent.\n\nQUESTIONS:\n1. Synthetic image visualization: Could you include the binary texture image referenced in Figure 1B and Section 2.1, or at least a representative example? This would help readers better understand what it means to include “all 1–4 point correlations” in 2×2 patches.\n2. PCA experiment design: In your PCA comparison of CNN and HoCNN representations, each point corresponds to a randomly initialized model. Can you clarify the rationale for this setup? Wouldn't it be more informative to fix one model and vary the input instead?\n4. Typo in Figure 4: Please clarify the inconsistency between the image panels and the figure caption in Figure 4. Is panel C the HoConv block or the Conv block?\n5. Extension to SOTA models: You mention potential connections with transformer-based vision models. Can you speculate on how higher-order convolutions could be integrated into hybrid or attention-based architectures?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-05"}