Brain-inspired $L_p$-Convolution benefits large kernels and aligns better with visual cortex

Authors: Jea Kwon, Sungjun Lim, Kyungwoo Song, C. Justin Lee
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Lp-Convolution, Receptive Field, Multivariate p-generalized normal distribution, Representation Similarity, Visual Cortex, Gaussian Sparsity

Abstract:
Convolutional Neural Networks (CNNs) have profoundly influenced the field of computer vision, drawing significant inspiration from the visual processing mechanisms inherent in the brain. Despite sharing fundamental structural and representational similarities with the biological visual system, differences in local connectivity patterns within CNNs open up an interesting area to explore. In this work, we explore whether integrating biologically observed receptive fields (RFs) can enhance model performance and foster alignment with brain representations. We introduce a novel methodology, termed $L_p$-convolution, which employs the multivariate $L_p$-generalized normal distribution as an adaptable $L_p$-masks, to reconcile disparities between artificial and biological RFs. $L_p$-masks finds the optimal RFs through task-dependent adaptation of conformation such as distortion, scale, and rotation. This allows $L_p$-convolution to excel in tasks that require flexible RF shapes, including not only square-shaped regular RFs but also horizontal and vertical ones. Furthermore, we demonstrate that $L_p$-convolution with biological RFs significantly enhances the performance of large kernel CNNs possibly by introducing structured sparsity inspired by $L_p$-generalized normal distribution in convolution. Lastly, we present that neural representations of CNNs align more closely with the visual cortex when -convolution is close to biological RFs.

Decision: accept (poster)
Reviewer scores: 3, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730208627016

thread_type:
official_review

<thread_object>
creator: reviewer_ozim

<content>
summary:
This paper primarily investigates variations in local connectivity patterns within CNNs, examining whether incorporating biologically inspired connectivity structures can improve model performance and increase alignment with brain representations. Specifically, the authors introduce Lp-convolution, which utilizes the multivariate p-generalized normal distribution (MPND). The proposed adaptable Lp-masks aim to bridge the gap between artificial and biological connectivity patterns, finding optimal configurations through task-based adaptation to enable strong performance in tasks requiring flexible receptive field shapes.

strengths:
* Experimental results indicate that CNN neural representations exhibit a stronger alignment with the visual cortex when the Lp-mask shape approximates a Gaussian distribution. 

* Testing the conformational adaptability of Lp-masks in the Sudoku challenge yielded interesting results, highlighting the flexibility of this approach.

weaknesses:
* Consistency in terminology would improve clarity; alternating between “Lp-convolution” and “Lp-mask” can be confusing. Using a single term throughout would make the concepts easier to follow. 

* The mention of Vision Transformers (ViTs) in the introduction feels tenuous, as they are not included in subsequent experiments, nor are they closely related to the main theme of the paper. 

* In lines 108-110, where it is stated that “CNNs have rectangular, dense, and uniformly distributed connections, as opposed to the circular, sparse, and normally distributed connections in biological neurons,” this description would benefit from supporting references regarding the shapes of receptive fields in biological neurons. It’s also worth questioning whether this statement accurately characterizes CNN weights, as CNNs trained to model retinal ganglion cells, for instance, have demonstrated sparse weight patterns ([1]-[5]). 

* Lines 137-138 mention that “we optimized parameters of p and σ in MPND (Fig.1e, Eq.1)...” However, Eq.1 and the text do not define σ. It’s also recommended that the authors confirm Eq.1’s form by referencing the standard expression of a multivariate Gaussian function. 

* Integrating Lp-masks in CNNs does not appear to significantly improve recognition accuracy across datasets. Comparing this approach to ViTs, it’s unclear if it achieves current state-of-the-art performance. 

* The justification for using large, sparse kernels feels somewhat weak. Aside from achieving marginal improvements in RSM alignment with the visual cortex, it’s unclear how this approach benefits contemporary computer vision tasks. 


References:

[1] Maheswaranathan, Niru, et al. "Deep learning models reveal internal structure and diverse computations in the retina under natural scenes." BioRxiv (2018): 340943. 

[2] Tanaka, Hidenori, et al. "From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction."  Advances in neural information processing systems 32 (2019).

[3] Lindsey, Jack, et al. "A unified theory of early visual representations from retina to cortex through anatomically constrained deep CNNs."  arXiv preprint arXiv:1901.00945 (2019).

[4] Yan, Qi, et al. "Revealing fine structures of the retinal receptive field by deep-learning networks."  IEEE transactions on cybernetics 52.1 (2020): 39-50.

[5] Zheng, Yajing, et al. "Unraveling neural coding of dynamic natural visual scenes via convolutional recurrent neural networks." Patterns 2.10 (2021).

questions:
* In the authors' claim regarding "Lp-convolution with biological constraint," specifically the "Gaussian structured sparsity," what theoretical and empirical evidence supports this biological constraint? 

* Across various experiments, since ppp is a learnable parameter, what typical values does it converge to, and are there any observable trends or variations across different datasets? Could the authors interpret these findings in relation to biological insights?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730212305835

thread_type:
official_review

<thread_object>
creator: reviewer_zvmo

<content>
summary:
This paper introduces Lp-convolution, a novel approach to convolutional neural networks (CNNs) inspired by biological visual processing. The work addresses fundamental differences between artificial and biological visual systems: while traditional CNNs employ rectangular, dense, and uniform connectivity patterns, biological visual systems feature circular, sparse, and normally distributed connections. Additionally, the paper tackles the longstanding challenge that large kernel sizes in CNNs typically don't improve performance despite increased parameters.

The key innovation is the introduction of Lp-convolution, which uses multivariate p-generalized normal distribution (MPND) to bridge these biological-artificial differences. The method implements trainable "Lp-masks" that can adapt their shape through parameters, enabling flexible receptive field shapes that better match biological patterns. Technically, this is achieved by applying channel-wise Lp-masks that overlay onto convolutional kernels, with shape parameters that can be trained for task-dependent adaptation.

The authors demonstrate several significant findings: Lp-convolution improves the performance of CNNs with large kernels, with optimal results achieved when the initial p parameter approaches 2 (matching biological Gaussian distribution). Moreover, neural representations show better alignment with the visual cortex when connectivity patterns are more biologically plausible. 

The practical impact of this work is threefold: it enables effective utilization of larger kernels in CNNs, achieves more biologically plausible artificial neural networks, and maintains compatibility with existing CNN architectures.

strengths:
The paper is overall well-written, with sections that tell a clear, sequential story. The usage of bold characters to highlight important parts is particularly appreciated. This is a very strong contribution across multiple aspects:

* Connectivity patterns as inductive biases are largely unexplored within this community. The biological inspiration effectively guides the search for plausible connectivity patterns, and the approach proposed in this submission is particularly apt. The work presents a complete narrative, from biological mechanism inspiration to the implementation of Lp-convolution for neural activity prediction in V1 and representational similarity analysis.

* The paper's approach to addressing large kernel network training challenges could potentially bridge the performance gap between transformers and CNNs in image classification tasks.

* The mathematical formulation is sound and accessible, with figures (e.g., Figure 1 and 2) that effectively illustrate concepts and build intuition about parameter effects.

* The choice of the Sudoku challenge adds significant value, serving as an excellent demonstration of the model's capabilities in an easily understandable context (especially for $L_p$ mask shapes)

* The Appendix comprehensively addresses potential questions, demonstrating thorough consideration of the work's implications and limitations.

weaknesses:
I don't think significant weaknesses are present in this work. The paper should be accepted as it is.

questions:
**(More of a curiosity)** For future developments of this work, it would be interesting to explore connections with anisotropic diffusion (Perona & Malik, *Scale-space and edge detection using anisotropic diffusion*, 1990). In standard convolution, there exists a well-established mapping between convolution operators and isotropic diffusion processes (as explored in Scale-Space theory, particularly in Koenderink, *The structure of images*, 1987; and Lindeberg, *Scale Space theory in computer vision*, 1994). How might Lp-convolution relate to or extend these theoretical frameworks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699547460

thread_type:
official_review

<thread_object>
creator: reviewer_nn4t

<content>
summary:
This paper introduces $L_p$-Convolution by integrating the multivariate p-generalized normal distribution into a $L_p$ masks for convolution filters. It allows the network to adapt to different receptive field shape and train efficiently with large kernel. The paper show $L_p$-Convolution has a advantage in tasks such as sudoku challenge.

strengths:
The idea of a $L_p$ convolution is original and could have wide application in visual tasks that require flexible receptive field size or in general tasks that require both local and global information from visual input. Showing its ability to transfer to any pretrained network greatly lowers the threshold to apply this for a wide range of tasks. The choices of Sudoku task and the follow up ablation analysis is solid and demonstrates well the strength of this method. Out of all papers that take inspirations from neuroscience and try to utilize it to improve neural nets, this paper stands out in actually providing a fundamentally different implementation of CNNs.

Other that than the points mentioned above, the benchmark testing was thorough and presented clearly.  The visualization of both the $L_p$ mask and convolution is very helpful for understanding the concepts. The writing is very clear.

weaknesses:
The paper does show through table 1-4 that $L_p$-CNNs can train with large kernels and have some advantage in robustness as well as accuracy in benchmark test. However the improvements over baseline models are small. I don't think these numbers convinces me how useful the $L_p$-CNNs could be. Aside from the sudoku task, the paper didn't really show the advantage of efficiently trained large kernal $L_p$-CNNs through a task that actually could really benefit from large kernels. I would suggest including some more tasks that requires processing of context, or even tasks ViT excels at for comparison. 

For the robustness benchmark as well as the Sudoku task, it could be informative to include performance of ViTs as well. 

Lastly, it is pretty well established throughout the paper that $p_{init}=2$ is the most useful for most task, and resembles the most to the biological system. I am not sure it is worth having another section (sec. 6) dedicating to comparing similarity of RSM across different $p$s. If the author were to demonstrate it is also a better model for brain representational alignment that I would recommend doing a more thorough study including more datasets, brain regions and variation of CNNs.

questions:
See weakness for suggestions.

Potential typo:
line 159: a solution to the of large kernel problem in CNN -> a solution of the large ...
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715778004

thread_type:
official_review

<thread_object>
creator: reviewer_7z3f

<content>
summary:
The paper proposes a brain-inspired approach of constraining the weights of convnets with a p-generalized Gaussian envelope. The authors demonstrate some minor improvements in performance on relatively small image datasets such as CIFAR-100 and TinyImageNet. They further claim that the learned representations of more "brain-like" convnets have higher representational similarity to the mouse visual system than their more classical counterparts.

strengths:
+ Novel inductive bias for convnets motivated by biology
 + Overall fairly well-written paper
 + Well-motivated and well-executed experiments

weaknesses:
- The effect sizes are really small, calling into question the practical impact
- Only "toy" datasets are explored
- Experiment on representational similarity not convincing


### Detailed explanation

While I find the paper well motivated and the idea original, I see the paper mostly as a negative result given the small effect sizes observed across most of the tables and the "toy" nature of datasets such as CIFAR-100 and TinyImageNet. The paper now has undergone several revisions that only reinforce this conclusion.

There is a statistically significant improvement due to Lp-Conv for some classical architectures, but not all of them (e.g. ResNet). Generally, the improvements are small (a few percent). Given that architectural modifications alone can now push accuracy on CIFAR-100 >90% (https://arxiv.org/abs/2304.05350v2), the 1–2% improvements in the 60–70% range feel insignificant. Experiments on more modern architectures such as RepLKNet (Table 3) show the same pattern, if anything with decreasing effect size (<1% improvement). Similarly, the transfer learning experiment using ConvNeXt-V2 (Table 4) shows close to no effect. There are no experiments on closer-to-real-world datasets like ImageNet (although that's by now a fairly standard problem that can be done on a consumer GPU), although I should say that I do not expect major effects in that experiment, either. The data simply show that the inductive bias doesn't do much.

The experiment on representational similarity yields equally small effect sizes, again insignificant on many architectures. In addition, the comparison is done to several mouse visual areas, some of which aren't even part of the "ventral" stream for which a convnet trained on image classification would be a reasonable model.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1732537547701

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# Review Summary
We sincerely appreciate the reviewers' time and effort in carefully evaluating our paper. Some comments were incredibly encouraging, acknowledging the value of our efforts so far, while others provided constructive critiques that highlighted expectations aligned with the high standards of ICLR. Addressing these comments has been invaluable in refining our understanding of the impact and utility of our work, as well as its position within the broader field. We briefly summarize the strengths and weaknesses pointed out by reviewers.

1. Strengths
- Novel Idea on Biologically Inspired Inductive Bias **(Reviewer 7Z3f, nN4T, ZVMo)**
- Broad Applicability of Our Method **(Reviewer nN4T, ZVMo, ozim)**
- Comprehensive Experimental Validation and Robustness **(Reviewer 7Z3f, nN4T, ZVMo, ozim)**
- Clear Writing and Effective Visualizations **(Reviewer 7Z3f, ZVMo, nN4T)**
- Demonstration of Model Flexibility with Sudoku Challenge **(Reviewer nN4T, ZVMo, ozim)**
2. Weakness
- Not SoTA or Modest Improvements in Performance (**Reviewer 7Z3f, nN4T, ozim**)
- Limited Datasets and Tasks (**Reviewer 7Z3f, nN4T**)
- Weak Representational Similarity Analysis (**Reviewer 7Z3f, nN4T, ozim**)
- Unclear Relation to ViT Integration (**Reviewer nN4T, ozim**)

# General Response
Our research represents an ambitious attempt to bridge neuroscience and AI, exploring whether **biologically-inspired inductive biases can not only enhance CNNs but also make them more aligned with the brain's mechanisms**. This paper is not the conclusion but the beginning—a foundational step toward conceptualizing and demonstrating this idea's potential.

We recognize the reviewers’ concerns about modest performance improvements, limited engagement with SoTA benchmarks, and the scope of tasks explored. These are valid limitations of our current study, and we appreciate the reviewers’ insights. However, the true value of our work lies in its **vision and foundation**: we have created an **easily pluggable convolutional module** that opens the door for future researchers to build on our findings and explore these possibilities further.

Moreover, this research is more than performance numbers—it provides a unique contribution by showing that biologically-observed mechanisms, when tested in AI models, **reveal meaningful inductive biases critical to visual processing**. This aspect of our work contributes to a deeper understanding of how neuroscience can inform AI, offering a perspective that extends beyond conventional performance-focused computer vision research.

We sincerely hope that reviewers can see this work not as a final statement but as a first, necessary step—**a catalyst for deeper exploration at the intersection of neuroscience and AI**. With this in mind, we have carefully structured our rebuttal to address the reviewers' valuable feedback while staying true to the motivation and purpose behind this study.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces Lp-convolution, a novel approach inspired by the connectivity patterns observed in the brain’s visual cortex. By utilising the multivariate p-generalised normal distribution (MPND), the authors create Lp-masks that allow receptive fields (RFs) in CNNs to adapt in shape, scale, and orientation. This flexibility addresses a key challenge in CNN design known as the large kernel problem, where increasing kernel size often leads to diminished performance. The proposed Lp-convolution overcomes this by enabling large-kernel CNNs to achieve better performance, particularly when the Lp-mask configuration aligns with biologically inspired patterns (e.g. when p = 2, which reflects the Gaussian-like sparsity observed in biological connectivity). Furthermore, the study demonstrates that models employing Lp-convolution achieve stronger alignment with neural representations in the visual cortex, as evidenced by representational similarity analysis (RSA) with mouse visual cortex data.

A key strength and contribution of Lp-convolution lies in its ability to adapt receptive fields in a task-specific manner, enabling CNNs to handle diverse input features more effectively. This adaptability contributes to significant improvements in model robustness, as shown in experiments with CIFAR-100-C, where models with Lp-convolution outperform traditional CNNs. The approach also allows for more efficient transfer learning, allowing existing pre-trained models to incorporate Lp-masks with minimal computational cost and performance drop. Importantly, the method is generalizable, being compatible with a wide range of architectures, from traditional models like AlexNet and ResNet to modern large-kernel architectures like RepLKNet and ConvNeXt.

Nevertheless Lp-convolution introduces certain complexities. The inclusion of trainable parameters (C and p) increases model complexity and requires careful hyperparameter tuning. Model performance is sensitive to the choice of the initial p-value, and selecting the right value for different architectures and tasks can be non-trivial. Moreover, while Lp-masks provide flexibility and task-specific adaptability, the underlying decision-making process within the model is less interpretable than simpler, more transparent CNN designs. The paper also lacks a comprehensive discussion of the computational overhead introduced by Lp-convolution, particularly in large-scale training scenarios.

In summary, this paper provides a neat bridging between biological and artificial intelligence by introducing Lp-convolution, a method that enhances CNN adaptability, robustness, and performance in large-kernel models. While it introduces new complexities, its biologically inspired design offers a fresh perspective on how insights from neuroscience can drive the development of more effective machine learning models, something that was recognised by the majority of the reviewers.