Brain-inspired $L_p$-Convolution benefits large kernels and aligns better with visual cortex
Authors: Jea Kwon
Sungjun Lim
Kyungwoo Song
C. Justin Lee
Venue: ICLR 2025 Poster (2025)
Keywords: Lp-Convolution
Receptive Field
Multivariate p-generalized normal distribution
Representation Similarity
Visual Cortex
Gaussian Sparsity
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Convolutional Neural Networks (CNNs) have profoundly influenced the field of computer vision, drawing significant inspiration from the visual processing mechanisms inherent in the brain. Despite sharing fundamental structural and representational similarities with the biological visual system, differences in local connectivity patterns within CNNs open up an interesting area to explore. In this work, we explore whether integrating biologically observed receptive fields (RFs) can enhance model performance and foster alignment with brain representations. We introduce a novel methodology, termed $L_p$-convolution, which employs the multivariate $L_p$-generalized normal distribution as an adaptable $L_p$-masks, to reconcile disparities between artificial and biological RFs. $L_p$-masks finds the optimal RFs through task-dependent adaptation of conformation such as distortion, scale, and rotation. This allows $L_p$-convolution to excel in tasks that require flexible RF shapes, including not only square-shaped regular RFs but also horizontal and vertical ones. Furthermore, we demonstrate that $L_p$-convolution with biological RFs significantly enhances the performance of large kernel CNNs possibly by introducing structured sparsity inspired by $L_p$-generalized normal distribution in convolution. Lastly, we present that neural representations of CNNs align more closely with the visual cortex when -convolution is close to biological RFs.

Published as a conference paper at ICLR 2025
BRAIN-INSPIRED Ln-CONVOLUTION BENEFITS LARGE KERNELS AND ALIGNS BETTER WITH VISUAL CORTEX
Jea Kwon',, Sungjun Lim2, Kyungwoo Song C. Justin Lee Max Planck Institute, YYonsei University, *Institute for Basic Science jea. kwonempi-sp.ors [1s]9862, s eyonsee .c. cjlBibs re.kr kr
ABSTRACT
Convolutional Neural Networks (CNNs) have profoundly influenced the field of computer vision, drawing significant inspiration from the visual processing mech- anisms inherent in the brain. Despite sharing fundamental structural and rep- resentational similarities with the biological visual system, differences in local connectivity patterns within CNNs open up an interesting area to explore. In this work, explore whether integrating biologically observed connectivity patterns can enhance model performance and foster alignment with brain representations. We introduce novel methodology, termed L-pconnvlution, which employs the multivariate y-generalized normal distribution (MPND). We took advantage of MPND's s conformational flexibility to carefully bridge disparities between artifi- cial and biological connectivity patterns by designing an adaptable Ly-masks. Lp- masks finds the optimal conformation through task-dependent adaptation such as distortion, scale, and rotation. This allows Lop-oonvlution to perform well in tasks that require flexible input field shapes, including not only square-shape but also horizontal and vertical ones. Furthermore, we demonstrate that Lypconnvlution with biological constraint which we call Gaussian structured sparsity significantly enhances the performance of historically successful CNNs with large kernels. Lastly, we present that neural representations of CNNs aligns better with the visual cortex when the conformation of T-p-makks is close Gaussian distribution, biologicially closer condition.
INTRODUCTION
The rise of Vision Transformers (ViTs) has revolutionized the field of computer vision, outperform- ing traditional Convolutional Neural Networks (CNNs) on many tasks and establishing new perfor- mance standards (Dosovitskiy et al. 2020 Liu at.. 2021b Touvron et al. (021). This shift ViT',, however, demands a significant increase in model parameters, larger datasets, and extended training periods (Maurício al. 2023). Under constraints of time and resources, CNNs remain a more prac- tical choice, especially as CNNs often perform better on smaller datasets (Liu et al. 2021a Zhu al. 2023). This advantage is partly due CNNs inherent design that mirrors biological Visual systems like the primary visual cortex (V1) in the brain (Hubel & Wiesel 1962 1965 Fukushima 1980), which serves as strong inductive biases, such hierarchical structures. local feature learning. and parameter sharing, which contribute to their effective generalization capabilities (LeCun et al. 1989 Bartunov al. 2018 Pogodin et al., 2021).
The findings of visual information processing in the brain have greatly influenced the development of CNNs, providing reciprocal platform for understanding neural visual mechanisms (Hassabis) et al., 2017, Zador et al. 2023 Yang & Wang 2020) Lindsay 2021). However, the significant architectural differences between the biological brain and modern computers present engineering challenges that make integrating biological insights into CNNs impractical and often inefficient for problem-solving (Marković et al. 2020). Thus, is important to integrate brain-inspired concepts in manner that is compatible with existing CNN architectures. In this study, we identify connectivity
"This work conducted at the Institute for Basic Science. CCo-coreepoodinng authors.

---

Published as a conference paper at ICLR 2025
patterns as a promising area for innovation. While CNNs typically feature rectangular, dense, and uniformly distributed connections, the brain's 's visual area V1 exhibits circular. sparse, and normally distributed connections (Lerma- -aaaaaaa et al.) 2021: Seeman et al., 2018 Hage et al. 2022 Rossi iet al. 2020) (See Appendix Aa.eedix|A..faiiee for the actual connectivity pattern of V1 neurons). Exploring these distinet connectivity patterns could introduce novel inductive bias that significantly enhance the
performance and efficiency of CNNs, potentially transforming their design and application.
Traditionally successful CNNs have relied on stacking small 3x3 kernels beyond the initial layer (He) et al. 2016 Simonyan & Zisserman 2014), as simply increasing the kernel size does not necessarily enhance performance, which we call large kernel problem, despite its increased trainable parame- ters (Peng et al. (017) (See Table] [ Base vs Large). However, modern CNNs that achieve perfor- mance comparable to ViTs have been adopting significantly larger kernel sizes, ranging from 7x7 to 50x50 (Liu et al.
2022 Ding et al. 2022
Liu et al. 2023
This shift indicates new possibilities and
directions in CNN design.
In this paper, we ask interesting question that could mutually captivate both neuroscience and ma- chine learning communities: Would introducing biological connectivity pattern as novel in- ductive bias into CNN resolve large kernel problem and aligns better with brain's repre- sentations? To answer this question, we introduce convollluu,,,, a novel approach that lever- ages the multivariate p-generalized normal distribution (MPND) to address the disparities between biological and artificial connectivity patterns (Fig. [ Through channel-wise trainable I.p T.p-masks in convolutional layers (Fig. ( we explore their conformational adaptability (Fig. | and (5), re- sulting in enhanced performance in large kernel CNNs and improved alignment with biologi- cal representations (Fig. 6 and Table I). Code, datasets, and pre-trained models are available at https://github. com. cowp///wwooo.jaoooccoooooomoooo wttpc/////aaako//ppooc/o///// ipconv/
2 BRIDGING DISPARITY BETWEEN BIOLOGICAL AND ARTIFICIAL CONNECTIVITY PATTERNS
Standard CNN architectures are typically designed with rectangular, dense, and uniformly dis- tributed connections (LeCun et al., 1998: Krizhevsky al.] 2012] SSmonyynn Zisserman [014]
He et al. 2015), in contrast to the circular, sparse, and normally distributed connections commonly observed in biological neuron (Lerma-Usabiage et al., 202T Seeman et al.) 2018 Hage et al.. 2022). Early studies in biological modeling using CNNs have shown that task specifi adaptations can lead to sparse weight patterns (Maheswaranathan et al. 2018: Tanaka et al.) [019 Lindsey et al. 2019 Yan al., 2020 Zheng et 2021). These insights demonstrate the adaptability of CNNs and highlight the potential for bridging artificial and biological connectivity patterns. To address this, we analyzed the functional connectivity patterns of both biological and artificial systems, by introducing the MPND (Goodman & Kotz 1973). MPND is the key of our paper to bridge the conformational difference between biological and artificial connectivity patterns, or local receptive fields (RFss. Note, While the term 'receptive field' can sometimes include sensory-level inputs and multi-layer interactions in broader context, in this work, we specifically define the RF as a local connectivity pattern between neurons in immediately adjacent layers (See detail in Appendix A.20
Multivariate p-generalized normal distribution Let s represent the d-dimensional random vec- tors indicating specific points within RF. 8p is the receptive center with d-dimensional vector of fixed constants. The relative offset is given by As = s sg- Introducing MPND (Goodman & Kotz, 1973), define the RF using probability density function (PDF) of as following:
exx (-ICAAs)))
(1)
where Cs d d inverse of covariance matrix, ||. | denotes the -op-n raised to the p-th power, / normalization facto and p 2 1. In Figure 1e show some examples of MPNDs with varying values of p and C
1/3 = [(2 +p det(C)) where I' is gamma function, and det(.) denotes the determinant. 1// o 1/a
When we optimize for MPND, C is initialized with
where a determines the scale.
2

---

Published as a conference paper at ICLR 2025
(b)
(e)
Mutivarttt generalized normal distribution
Bolegte Drean
(N
La3
(c)
(d)
Conv1
Aeseoall Hotte
A3
1al
Optimized MPND
Moeas X2
Coryl
Teined Cared
LT Het Cand
Uneaieed Conal
(-patter Menio
Traised Conva -Iaago input
Theined Comv2 -hale Inpul
Untaired Canu3
Unmaiied Comux
Himige hoput
House
-35 23
-25
Figure 1: Local receptive fields from biological and artificial systems can be mathematically reconciled by introducing multivariate p-generalized normal distribution (a-b) Illustration of receptive fields in V1 of mouse brain (a) at layer 2/3 (b). (c-d) Illustration receptive fields AlexNet (c) Conv layer (d). (e) Shapes of MPND with varying parameters of C and (I) Top. visualization of mean receptive fields functional synapses in mouse V1 layer 2/3 (column 1) and AlexNet Conv with varying conditions (columns 2-5). Bottom, optimized MPND receptive fields shown in the first
after MPND optimization; Using Welch's with Holm-B Bonferroni's multiple comparisons correction, all possible combinations between groups were statistically significant (p (p-value-c0.05) except for 'n.s.' (non- (non-significant) denoted in the figure; n=17 for all
conditions. We optimized MPND parameters of and a. where C
1/c
0.5, and Pinit = 2
0 1/a
Constructing biological and artificial RF's from the functional synapses To compare the bi- ological and artificial RFs, we first prepared the 2D offsets of functional synapses relative to the post-synaptic units in both systems: mouse V1 L2/3 and AlexNet Conv1 (Fig. [i-d; See details in Appendix A. [nddd and [..].. To match the scale difference between the two systems, we standardized the mean and unit variance. For the artificial system, we prepared 4 different
relative offsets with cases, using both ImageNet-Ik pre-trained and vundomly-initializes AlexNef with inputs of noises or images (See inputs and corresponding RFs in Appendix A.4). We constructed 2D probability mass functions (PMFs) from the collected offsets, which we call biological or artificial RFs.
MPND effectively models both biological and artificial RFs For the comparison of biological and artificial RFs, we optimized parameters of p and in MPND) (Fig. TT. Eq. over PMFs of biological artificial RFs (Fig. I and g). We show that optimized p* of functional synaptic input patterns of biological neurons were optimized at near 2 (Gaussian-distributec,
See details in Ap- pendix A.3). In contrast, the local RF pattern of pre-trained AlexNet's Convl I was optimized at the range of 3.7 ~ 3.8, and the untrained one was optimized at the range of 11 14 (where input types were less effective). An intriguing observation is the decrease the value of p for the pre-trained AlexNet's first convolutional layer (Convl), bringing it closer the biological RF. Based these findings, we propose that both biological and artificial RFs can be effectively modeled with MPND. Given these findings, we propose to consider that the value of p close 2 is indicative of biological RF, while higher represents RFs to be more artificial.
3 Lp -ONOOOLLUIION INTRODUCING MPND IN THE CONVOLUTION
In the early stages of CNN development, large kernels were not widely adopted, with their use predominantly confined the initial layers (Krizhevsky et al. 2012 Szegedy al.) 2015 2017).
For clarity, we refer trained as pre-trained models and untrained randomly-inttialized models.
3

---

Published as a conference paper at ICLR 2025
(b)
(c)
(d)
(e)
Ly Conv
L, Conv
Cono
(Large)
(Dox [Pac
k/2) (Pmt
256, dest
Leamed channels
Leamne y masks each channels
Graphical illustration of T.p-Com layers and visualization of learned T--masks after traning (a-c) Ly-Conv layers with Oini k/2 and varying Pint (2, 16, 256): Ly-masks overlaid on kernels (red arrows). (d-e) Conventional Conv layers with the kernel size of x & (Base) & (Large). (f-g) Visualized 32 example learned Lp-masks from Concereete AlexNet Conv after training with Tiny Imagenet dataset
Figure 2:
Attempts enlarge kernel size sometimes led decline in classification performance (Peng al., 2017). We confirmed this in Table | Conv (Base) vs Conv (Large), simply enlarging the kernel sizes of previously successful CNNs such as AlexNet, VGG, or ConvNext showed marked drop in performance. Consequently, the more favored strategy was stacking smaller kernels (Ix1, 3x3) (ii-
monyan & Zisserman 2014 He et al. 2016).
Based on our observation in Figure artificial RFs become closer to biological RFs with training (decrease in p of MPND by ImageNet-Ile training in AlexNet Convl), we asked two intriguing questions: 1) Can introducing brain-inspired concept of flexible RF provide solution to the of large kernel problem in CNN? 2) Can CNNs with RFs close to biological ones align better with the representation of the brain? To answer these questions, we introduce -,,-convlution: overlaying channel-wise trainable Lp-masks onto the kernels of CNNs.
Here, we propose the Iop-convolution, which compatible with various convolu- tions. We formulate the T,p-convolution based on the MPND in the convolutional layer by employing channel-wise Lp-masks, which are overlaid on convolutional filters (Fig. (-c)
con-ooovlttioon
First, we define the relative height and width offsets, AS E 22 K.. from the kernel center, (K&/2, K /2), as follows
Kh. 2
Kw 2
AS (Ah, Aw) -h-
for h E K0, - 1], Kw 1], where Kn and Kr, denote the kernel height and kernel width, respectively. Empirically, we utilize the normalized value between 0 and 1. It is noted that S. R denotes all values corresponding to h-th height and w-th width.
Second, we propose the Lm-mask, structured mask matrix, derived from the offset and MPND. Our Lp-mask, M E [0, 1 for all output channel Co, is a soft mask that is proportional to Eq. without normalization factor B as following
Ma exp (-IIC,,,AA AS AS
where C E CCCx222 is the set of 2 covariance matrix for each output channel. In other words, m--- calculates the soft mask Mo.,, for each o-th output channel independently, and each soft mask handles the positional correlation. height and width position, from the offset, AS. Our I.p- mask in Eq, corresponds to RF in Eq. when KÅ = Kw = C. = 1.
Third, propose the Lp-convolution by applying Ly-mask into the convolutional weights W RC.xC. Ka x where C₁ is the number of input channel. For each i-th input channel X; and
4

---

Published as a conference paper at ICLR 2025
convolutional filter weights W, we formulate the corresponding convolution output V
) = > * (W; @ M));
where @ is non-linear function and is the convolution operation.
We note that C and are trainable parameter To ensure the positive definite property of C. and satisfy the norm property with p 2 1, we can employ Cholesky decomposition for C and value clipping for
As shown in Equation our Lp-convolution is generalized version of the traditional convolution. In our settings, it is noted that Lp mask converges to a binary mask as approaches infinity. Empir- ically, Lp mask becomes binary mask for sufficiently large p. as shown Fig. [ If all elements of Ip mask equal one, then Ip-convolution degrades to the traditional convolution. In this situation, both traditional convolution and our Lp convolution have square-shaped RFs for each layer. other words, our conooouluti takes task or data-dependent RFs with varying M by optimizing the C and as shown in Figure 2 Therefore, Lo-coololttion has task-specific RFs with varying distortion, scale, and rotation levels.
In practical terms, we replaced all existing Conv2d layers in the baseline CNN model with Lp-Conv layers by applying function called LpConvert to the baseline CNN model (See pseudo-code in Appendix A.2Tp. To provide further insight into the conformational changes of Ly-masks during model training, present examples of 32 random La-mas from Convl of AlexNet model trained with TinyImageNer (f and in Fig. 2
4 Lp A--CONVOLUTION BENEFITS LARGE KERNEL CNNS
Ln-vovvolttion on traditional models To test whether L,, convoluttni is robustly applicable to traditional CNN architectures, conducted vision classification tasks (See detailed experimental settings in Appendix(A.9) (A9) using the CIFAR-100 and TinylmageNet datasets on models of AlexNet, VGG-16, R8 ResNet-34, and the ConyNext-tiny (Liu et al. 2022). As can be seen in the Table D applying Ly-Masks with approximately double the kernel size generally improves clas- sification performance, whereas simple kernel enlargement fails to achieve similar improvements (highlighted in red). Furthermore. we found that the optimal choice of the hyperparameter Pint varies depending on the model architecture served when Pinit 2 (which is close to biological RFs), with 9 out of 10 cases achieving either the top or second-best results. These results suggest that explicitly applying Gaussian-like structured sparsity to large kernels during the convolution operation may lead to better optimization.
However, the best performance was most often ob-
Biologically-inspreed L.yooovolution improves robustness To assess the robustness of Lp- Models, we utilized CIFAR-100-C, dataset specifically designed for evaluating robustness using corrupted validation data (Hendrycks & Dietterich] 2019). We focused on data corruptions at sever- ity level across all tested architectures (see the attached PDF Figure 1, for all architectures). Our findings reveal that Lp-Convolution of Ly=2 significantly enhances robustness, as demonstrated in Table 2
Ly-convolution modern large kernel CNN Next, we explored the impact of Lp ConCoooluttonn on modern large kernel CNNs like RepLKNNtt (Ding et al.) 2022), which utilizes kernels up to size 31x31. By integrating Lp Lon-oonolttoon into these models, convolution method can enhance the performance of large kernel CNNs. Specifically, we conducted experiments with KepL using and without L, ConCoonvlt For the Lp2-RepL.KNet model, all Cony2d layers were replaced with Lp-Conv (p- = 2) layers without modifying the kernel sizes. As can be seen in Table L, Cy-Convolution achieves a performance improvement with little to no increase in computational cost even in modern large kernel CNN architecture. This indicates that Lp-Convolution can be easily incorporated as a flexible option for researchers aiming to optimize model architectures.
aimed investigate whether our Lp-
*C' and p are updated with the standard backpropagation process. T.p-mask, M, is dynamically generated during forward pracess using C and p.
NNte that although p is parameter that can he adjusted during training, it was not fine-tuned to the point where it overlaps with the initial settings of each Pince condition (See AinendixA..166
5

---

Published as a conference paper at ICLR 2025
Table 1: Top-1 performance on the CIFAR-100 and TinylmageNet datasets in CNNs are reported with 5 trials (meantstd). The symbol f indicates that both and p are frozen parameters during training. x & default kernel size. x I, large kernel size. 1 = 2 x [ + k(=s 2k). For all C layers, C was initialized with 1/ami of diagonals and 0 of off-diagonals, where
k/2, where & represents the default kernel size in each layer of the baseline CNN. Statistical comparison results using Welch's I-test with the base model are marked as follows: 'ns' (p-value 2 0.05). *** (0.01 S p-value < 0.05), **** (0.001 I p-value < 0.01), and sss (p-value < 0.001). The text in bold denotes the best performance, while underlined signifies the second best. Gray indicates baseline performance and red indicates decrease in performance.
CIFAR-100
Layer
Kermel
Pien
AlexNel
VGG-16
ResNet-18
ResNet334
ConyNeet-TT
(Base) Cony
k
66.05 0.33
70.26 0.29
71:22 0.18
12.47 0.23
58.36 4 6:48
(Largo) Conv
1xl
544.53 0.65
"64.822
1122.0
""33.52
0.11
"54.13 1.14
ConC
256
"65.95 + 0.32
".1.03=0.38
0.22
"72.61
0.27
"60.34 2.88
Ly-Co
16
"67.12 0.37
0..28
72,35
0.30
"373.32
"61.30
1.71
1p Conv
fxt
8
66.85 + 0.18
"71.14=0.25
722226
0.228
773.37
0.15
"59.94 4 5.04
TonC
"71.71
0.36
7..00
74.07
"59.34
7.53
L, Cony
2
16..1 0.33
"'72.88
0.30
0..1 73.86
774,95
0.11
*62.61 3,03
ImngemgeNet
Layer
Kerne
Phce
AlexNet
VGG-16
ResNet-18
ResNet-34
ConyNeXt-T
(Base) Cony
x k
52.25 a 0.35
67.75
0.07
66.63 a: 0.51
69.22 a 0.11
70.25 a 0.45
(Large) Conv
1x1
35.52
0.46
666.96
1.50
68.33 0.19
"69.46 = 0.36
668.66 1.50
L, Conv
256
"52,60
0.12
"67.7220.11 0.18
"66.37 0.55
"69.27 = 0.27
"70.45 0.44
L,
16
53,.98 0.50
"699299
667.72
"70.00
*70.62
Ly-Cv
1x1
8
"54.07 0.91
""69.22
0.166
*67.63 + 0:45
69.81 0.23
*70.52 0.36
L, Conv
4
5"5.33 + 0.48
"69.79.00.39
66. 0.50
"69.99 0.44
"77.74 0.37
1 Conv
2
+
"69.96
+ 0.36
770.43 0.24
*33372++0.11
Table 2: SSummarized robustness experiments various Lp CNNs. The table shows the number of wins for each method across di fifferent architectures.
Robustness
Base
Large
Lp=
Lp=4
Li=s
Lip=i
ConNNeXi-T
18
VGG-16
19
0
0
AlexNet
1
4
0
ResNet-18
6
ResNet-34
13
0
0
Counts
21
57
Table 3: Effect of Lp- Convoluuton on large kernel CNN. RepLKNet was trained TinylmageNet for 150
epochs with architecture of [2,2,6,2] blocks and 128,266,115 channels. Presented mean(std).
Model
Kernel sizes Top-1 Avc, (%)
(M)
Flops (G)
RepLKNet [31-2927-133]
66.2 (0.38)
11.68
2.54
Lp2-RepLKNat [31-29-27-13]
67.1 (0.35)
11.79
2.54
Transfer learning with coovoouolt on pretrained models There are countless pretrained CNN models in existence that have achieved state-of- the-art performance using large-scale datasets. Actively reusing these models can offer significant time and economic advantages. Theoretically, an T.p-convolution layer with T.p-masks frozen Cinit and Punt original base model. By employing pretrained weights for the central parameters of the CNN's convolution layer and initializing surrounding weights to zero in the enlarged kernel while using high p value, it is possible to train the Ly-mask without significantly deviating from the model's original performance (Fig.
function identically to the
To investigate this concept, we performed transfer learning experiments using the ImageNet- pretrained ConvNeXt-V2 V2 Tiny model (Woo et al. (2023) across five distinct datasets. Oxford flowers (Nilsback & Zisserman| 2008). Describable Textures (did) (Cimpoi et al. [014], Oxford Pets (Parkhi al. 2012), FGVC Aireraft (Maji et al. (013), and UCF10T (Soomro et al.) 2012). For a fair com- parison, we conducted experiments with identical hyperparameters for each convolution layer. All datasets were trained with 16-shot learning, and the models were evaluated after training for 100 epochs without applying early stopping. As indicated in Table 4 implementing L, Lm-masks with high values of = 16 enhances transfer learning performance beyond that of the pretrained baseline model, whereas simply increasing the kernel size results in significant decline in performance. Our results demonstrate that L, conovlllltio can be effectively integrated with existing successful pretrained models to push performance beyond previous level.

---

Published as a conference paper at ICLR 2025
Figure 3: Schematic illustration of transfer learning setup with Ly LonCoolllt As described in the figure, the pretrained weights were surrounded by zero-initialized parameters, and an Lp-Mask (ranges 0 to 1) is over- laid. This configuration helps start without significantly deviating from the original model's performance.
Table The result ConvNeXt- V2 pretrained ImageNet IK for transfer learning with Up-Convolution using 5 different datasets. We run 16-shot transfer learming and average accuracy over different trials. Avg stands for the average performance across five datasets. Win count indicates the number of dataset that the convolution
layer achieved. Underline: improvement baseline model. Bold, best results among all different models.
flowers
Oxford Pets
FGVC Aireraft
UCF101
Avg
Conv (Base)
95.84
61.12
90.43
55.03
65,98
73.68
Conv (Large)
51.57
8.22
11.33
4.63
20.48
19.24
0
(p=2)
54.25
Lp-Conv (p=4)
95.51
58.20
89.16
43.34
63.94
70.03
0
Lp-Conv (p=8)
95.33
60.53
90.64
54.37
66.16
73.41
Lp-Cony (p=16)
95.92
61.71
90.71
54.92
66.43
73.94
5 CONFORMATIONAL ADAPTABILITY OF LP MASKSS IN SUDOKU CHALLENGE
(8)
Quiz (Input)
Sudoku Lp-ConVee
Solution (Target)
x
4
Column (Bc1)
(b)
[5c5] Bane
Lp ((p2266)
[777] Lp (pm15)
[Tx/] Lp (pm2)
Lange Lange
Average
Square Accunaty
Row/Cal Arouracy
Accuracy
1.0
Eoach
Figure 4: Ly-convolution enhances Sudoku solving efficiency by effectively balancing accuracy between square and row-column puzzles (a) Graphical illustration Sudoku quiz and Sudoku Lp-ConvNet (left), example Sudoku quiz as an input; (middle), basic block repeated L times (L 10, yellow) contains sequential layers of I) reflection padding. 2) Lp-Conv, 3) batch normalization, and 4) activation layers; (right), the example Sudoku solution as target. In Sudoku, 9xx square must be filled in with numbers from 1-9 with repeated numbers in 9x1 rows (blue). 1x9 columns (green), 3x3 squares (orange). (b) Loss and accuracy curves during test session. '(3 3) or '(7 7)' denotes size kernel. L.,"" denotes parameters of Ly-mask frozen. 'Large' denotes a simple enlargement of the kemel,, without a mask.
Our experimental results far indicate that trainable L,-Mask benefits large kernel optimiza- tion. To gain deeper mechanistic insight into this effect, we conducted a specialized experiment, 9 9 Sudoku-solving task (Park) 2018 Oinar 2021 Amos & Kolter 2017 Palm et al. 2018 Wang et al. 2019) which aims to solve multiple goals simultancously (See experimental details in Ap- pendix A.5 As Sudoku challenge necessitates simultaneously achieving three objectives- ensuring

---

Published as a conference paper at ICLR 2025
every row, column, and box contains all numbers from to 9 (Fig. 44) assume that Ly Ly-Masks can evolve their conformation along with training in a task-dependent manner.
Ly Ln-convolution in Sudoku solving: balancing square and row-column imbalances We show that introducing Lp-convolution alleviates the imbalance between square accuracy and row-column accuracy. In Figure 4p, the (3x3) Base model or (7x7) I (Pinit 256) model exhibited an im- balance Square-to-Row/Colume accuracy and showed signs of overfitting after approximately 15 epochs. Next, we tested two trainable Ly-makks with Pinit = 2, 16, which resemble biological (7x7) [Pp(ii = 2) model, we ob- accuracy become more balanced, resulting in remarkable improve- ment in overall Sudoku accuracy (red in Fig. 4p). We speculated that this alleviation of Square-to- Row/Column accuracy imbalance in (7x7) Lp (Pint 2) could be attributed to the task-dependent adaptation of Lp-masks' conformation. To test this possibility, we have designed ablation experi- ments (7x Lp Paant
RF (p = 2) and an artificial RF (p = 16), respectively. served
2) model.
(a) Ly ma-k shapes
(b)
Layer-wise learned Lp mask properties for (7x7) Lp (p=2)
[Case p
y=1
y=2
ye3
.0mg
Cong
Ganoz
.0-g
LyCona
.0-0
Cand
Domd
Gooos
LeCond
Lymo
Lytom
Dony7
Gow7
Candi
4,Cora
5t-g
4,Cavvta
G00dd
160- .00-
45
90"
135
180 1.99 2.00 2.01 2,02 2.03
Scale (o)
Distortion (v)
Rotation (8)
(c)
Ablation vertical orientation selective masks in (7x7) by (p=2)
SET=p SET=8 STt-p 00-0
Recuacyyaa Ratio
1-10
Ablation 6-10
Ablation 1-5
AcciPato cotbox
d-6
ablation range
Figure 5: Task-dependent conformational adaptation L.--makss (a) The shapes Lp-masks when p and varying properties of scale (a)) distortion (7), and rotation (9), which are derived from the singular value decomposition of C; Red box indicates column selective Lp-masks which ablation targets in (c). (b) Layer-wise distribution of learned Ly-maak properties. (c) Selectively ablation of -msss near 90" by gradually increasing the ablation; Ablation in all 10 Lp-conv layers (left), first layers (middle), and last 5 layers (right) respectively.
Ablation of orientation selective masks reveals Ly- -p-masks' conformational adaptability Con- trary to previous large-kernel model that introduces unstructured sparsity directly into filters (Liu al. (023), I.p-convolution with p 2 introduces structured sparsity based on a Gaussian dis- tribution. This approach facilitates covariance analysis of the Gaussian distribution, thereby en- hancing interpretability. Using Singular Value Decomposition (SVD) on C, we extracted three in- terpretable properties of scale (a), rotation (0), and distortion (7) (See conformational analysis in Appendix A.3) Figure 5h shows conformations of -asks inverse calculated from a, B, and ?
Quantitative analysis of (7x7) Lp (pmit 2) model revealed increase in scales when layer deep- ened (Fig. 5b, see visualization in Appendix (.7)). with orientations of horizontal (0.180%) and ver- tical (90%) directions. This indicates the task-dependent adaptation of Tap-mssss, which provide
8

---

Published as a conference paper at ICLR 2025
flexible and structured RFs in visual processing. To confirm these orientation vrientation-sellctiiee selective masks tribute to the balanced Sudoku-solving task, we conducted an ablation test. We classified masks with high distortion (y 3) rrtentation-selectiv masks. Among these, selectively ablated near 90 by gradually increasing the range (close in shape with the red dashed box in Fig. 5h) while tracking changes in column and accuracies. While row and box accuracy exhibited consistent decrease, column accuracy sharply decreased as the / range increased (Fig. [5] and Appendix (A.8). with this trend was notable in the later layers. Together, these results indicate that the conformational adaptability of Lp-man masks enables balanced learning in the Sudoku Sudoku-solving task, thereby contributing overall performance enhancement.
EMPLOYING L, CO-COOVOLNIION FOR BIOLOGICAL SYSTEMS
(a) Naturll Scanss Insgas
regions
DViSp, Wiii JiBat Vilgn, V[aan]]
Stm aomarr Mase Maw
Similrity of
RSM (SSM)
4, Ly-Con Layers
[L....A
(b)
Aneoler
VGG-16
Reatlen-18
Ranhiet-3
Conufeeib-T
Max. SSM (Vs VC)
0.30
4 2
16
8 Piit
16
4
8
2 16
Figure 6: Representational similarity between biological brain and artificial models using natural images (a) Schematical illustration of representational similarity analysis (RSA) from neural activities of mouse VC L2/3 subregions and convolutional layers of EinylmageNet-traineed CNNs; Unit activities in both mouse brain CNNs were obtained from N number of image inputs: N RS matrix was constructed
every subregions or layers by measuring the correlations across unit activitics. The similarity of the RSMs (SSM) between the VI region and CNN Conv layer was measured by Kendall's 's rank correlation coefficient. (b) For the comparison across models, the maximum SSM collected among all pair- wise SSM scores across regions and layers; the two-sample Student's t-test for statistical analysis, & p<0.05 **; p<0.01, ***; p<0.001.
Representational similarity analysis between Lp CNNs and visual cortex Figure al illustrates our approach assessing the alignment of representations between biological and artificial systems. We utilized the standardized dataset from Allen Brain Observatory (de Vries et al. 2020),, which uses 118 images of Natural Scenes and corresponding neural activities recorded from the mouse visual cortex (VC). Our method builds upon established Representational Similarity Analysis (RSA) tech- niques RhallighRazaai & Kriegeskorte 2014. Devereux et al. 2013 Diedrichsen & Kriegeskorte 2017). to compare the representations of CNNs (Bakhtiari et al.
202 Shi et al. 2019) (See details
in Appendix A.IIp.
Based on the observation that Lp--oonvoutionn tends to perform better it approaches biologically observed RFs with Pmmt 2, we posed the question of whether the neural representation artificial models with biological RFs aligns more closely with the representation of VC. To address this, compared the neural representations of Tiny ImageNet-treeee trained CNNs from Table[[by presenting the 1 118 Natural Scene images, with the mouse VC representations. To facilitate model comparison,

---

Published as a conference paper at ICLR 2025
extracted the representative value, maximum SSM, chosen from pair- SSMs across the convolu- tional layers and the VC subregions (See pair-wise SSMs in Appendix 12.2).. In the results, models with Pinit closer to 2 generally exhibited better alignment with the brain (Fig. (p). In summary, we find that T cy--convluttoon tends to achieve better alignment with the brain as it approximates a Gaussian distribution.
Neural activity prediction V1 with L,~CNNs To closely relate to computational neuroscience, neural activity prediction is frequently used alongside representational similarity analysis. Using the recent multimodal dataset with V1 neural activities from freely moving mice (Xu al., (024). we applied L convolltlnn on the CNN of MMVI model and compared the effect of different p values. As can be seen in Table 5] So-Coovolttion with Pini = 2 gives the best prediction performance, which implies that biologically inspired Gaussian sparsity can also benefit CNN architecture for neural activity prediction.
Table 5: V1 activity prediction with CNNN. The MMV1 model is a CNN followed by a GRU, where the
CNN has 3 Conv layers with kernel sizes of [7-7-7] (see details in (Xu et al. 2024). For the GRU, used
sequence length of 4, which corresponds to 192 of history. Here, represents cross-correlation (higher is better), and MSE represents mean squared error (lower is better). Presented as mean(std) from v mice.
MMVI
1p=2
1p=4
Lp=s
Lp=16
Base
Large 0.577 (0.048)
0.595 (0.046)
0.571 (0.076)
0.572 (0.071)
0.584 (0.066)
0.585 (0.052)
MSE
0,0722 (0.012)
0.0828 (0.021)
0.0786 (0.017)
0.0756 (0.018)
0.0746 (0.016)
0.0787 (0.013)
CONCLUSION AND REMARKS
In this study, we introduced novel L, Lp-convvlution, based on the MPND, with the objective of narrowing the gap between artificial and biological RFs, and subsequently crafting neural network modules more aligned with biological structures. Brain-inspired Lypconvvlutionn enables the cultiva- tion of diverse-shaped RFs with Gaussian-based structured sparsity, adaptable to various rotations, distortions, and scales, and tailored for specific tasks. Significantly, Lp-convolution showcases its adaptability and compatibility across an extensive spectrum of CNN models. from the conventional to the contemporary. underscoring its proficiency, especially in contexts involving large kernels. We believe our research serves as noteworthy illustration of the symbiotic relationship between advancements in artificial intelligence and our understanding of neural processes.
The advent of ViTs marked paradigm shift from traditional CNN models, with the Swin Trans- former emphasizing the significance of both attention mechanisms and large receptive fields, thereby renewing interest in large kernel CNNs (Dosovitskiy et al.) 2020 Liu et al. 2021b Vaswani et al.) 202T. Recent innovations such as KepL and SLak have showeased performance comparable to ViTs, highlighting the potential of large kernel CNNs in modern computer vision (Ding et al. 2022 Liu et al. 2023). However, the effectiveness of large kernels in historically successful CNN models remained unexplored until now. In this paper, we succesfuly implemented large-kernel con- volution by overlaying trainable masks to the filters with Gaussian-based structured sparsity, for adjustments in receptive fields tailored to specific tasks. Compared to previous unstructured sparsity approach which may necessitate extensive hyperparameter tuning, our trainable masks streamline the optimization process by automatically adjusting key parameters, thereby facilitating the applica- tion of large kernel training in both traditional and modern CNN architectures.
In biological systems, both anatomical and functional studies have shown that local connectivity patterns and population receptive fields in the visual system display sparse, circular and Gaussian- like distributions in the early visual cortex (Lerma-USsabiga et al. [021 [Seman et al., 2018 HHge et al. 2022). These findings prompted us to investigate whether Gaussian sparsity could serve as beneficial inductive bias in CNNs. Given the fundamental differences in the hardware architectures between CNNs and the brain, it was suspected that introducing Gaussian sparsity might not be an effective choice from an engineering perspective, and its efficacy may not have been effectively tested previously. In our research, addressed this gap by experimenting with Gaussian sparsity in CNNs that feature large kernels, and systematically comparing artificial and biological connectivity patterns through the introduction of the MPND model.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Institute for Basic Science in Korea IBS-RO01-D2.
REFERENCES
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pp. 133--15. PMLR, 2017.
Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self- supervised predictive learning. Advances in Neural Information Processing Systems, 34:25164- 25178, 2021.
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of iioooicallly-motivaeedd deep learning algorithms and archi- tectures, Advances in neural information processing systems, 31, 2018.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Q Ragmen Le. Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703.
Saskia EJ de Vries, Jerome A Lecoq, Michael A Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng, Nicholas Cain. Peter Ledochowitsch, Daniel Millman, et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature neuroscience, 23(1):138-1511 2020.
Barry Devereux, Alex Clarke, Andreas Marouchos, and Lorraine K Tyler. Representational sim- ilarity analysis reveals commonalities and differences in the semantic processing of words and objects. Journal of Neuroscience, 38(48):18906-18116 2013.
Jörn Diedrichsen and Nikolaus Kriegeskorte. Representational models: A common framework for understanding encoding, pattem-component, and reprrssentational-smmiii similarrtt analysis. PLoS com- putational biology, 13(4); 1005508, 2017.
Jörn Diedrichsen, Eva Berlot, Marieke Mur, Heiko H Schüitt, Mahdiyar Shahbazi, and Nikolaus Kriegeskorte. Comparing representational geometries using whitened umbiased-dissance-mmare similarity. arXiv preprint arXiv:2007.02788) 2020.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVÉ conference on computer vision and pallern recognition, pp. 11963-11975, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Reprexentations, 2020.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193-22) 202, 1980.
Irwin R Goodman and Samuel Kotz. Multivariate 0-generalized normal distributions. Journal of Multivariate Analysis, 3(2):204-219, 1973.
Travis Hage, Alice Bosma-Moody, Christopher A Baker, Megan Kratz, Luke Campagnola, Tim Jarsky, Hongkui Zeng, and Gabe J Murphy. Synaptic connectivity to 12/3 of primary visual cortex measured by two-photon optogenetic stimulation. Elife, 11:e7 1003, 2022.
Haldan Keffer Hartline. The response of single optic nerve fibers the vertebrate eye to illumination of the retina. American Journal of Physiology-Legacy Content, 121(2):400-415 1938.

---

Published as a conference paper at ICLR 2025
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inspinn artificial intelligence. Neuron, 95(2):245 200-2255-88,, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Bernhard Hellwig. A quantitative analysis of the local connectivity between pyramidal neurons in layers 2/3 of the rat visual cortex. Biological cybernetics, (2())111-121) 2000.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Purt 14, pp. 646-661. Springer, 2016.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional archi- tecture in the cat's visual cortex. The Journal of physiology. 160(1):106 1962.
David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture in nonstriate visual areas (18 and 19) of the cat. Journal of neurophysiology, 28(2):229-289, 1965.
Seyed-Mahdi Mhaligh-Razav and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain it cortical representation. PLoS computational biology, 10(1)):e1003115 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In Interna- tional conference on machine learning, pp. 1188-1196. PMLR, 2014.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard. Wayne Hub- bard, and Lawrence Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-25 222-22244 1998.
Garikoitz Lerma- Usabiaa Jonathan Winawer, and Brian A Wandell. Population receptive field shapes in early visual cortex are nearly circular. Journal of Neuroscience, 41(11):2420-244 2021.
Grace W Lindsay. Convolutional neural networks model of the visual system: Past, present, and future. Journal of cognitive neuroscience. 33(10):2017-200 2021.
Jack Lindsey, Samuel A Ocko, Surya Ganguli, and Stephane Deny. A unified theory of early visual representations from retina to cortex through anatomically constrained deep cnns. arXiv preprint arXiv: 2019.
12

---

Published as a conference paper at ICLR 2025
Tianlong Liu, Shiwei andrs Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Tommi Kärkkainen, Mykola Pechenizkiy, Decebal C Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. In International Conference Learning Representations, ICLR 2023, 2023.
Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco Nadai. Efficient training of visual transformers with small datasets. Advances in Neural Information Processing Systems, 0--2381-223838 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-1198,, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
Niru Maheswaranathan. Lane T McIntosh, David Kastner, Josh B Melander, Luke Brezovec, Aran Nayebi, Julia Wang, Surya Ganguli, Stephen A Baccus, and Stanford University. Deep learning models reveal internal structure and diverse computations the retina under natural scenes. BioRxiv, pp. 340943, 2018
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- -raiigr visual classification of aircraft. arXiv preprint 1rXiv:1306.515/, 2013.
Danijela Marković, Alice Mizrahi, Damien Querlioz, and Julie Grollier. Physics for neuromorphic computing. Nature Reviews Physics, 2(9):499-510, 2020.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001,
volume 2. pp. 416 423. IEEE, 2001.
José Maurício, Inês Domingues, and Jorge Bernardino. Comparing vision transformers and convo- lutional neural networks for image classification: A literature review. Applied Sciences, 13(9):
5521, 2023.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pp. 722-729. IEBE, 2008
Chingis Oinar. How to solve sudoku with convolutional neural networks (cnn). https://
wrtthub.com/ htt.m//hhhniisoiooorrrr////// www.oooonnaaauuoeooo sol ver. wer.pytorch 2021.
Adriana Olmos and Frederick AA Kingdom. A biologically inspired algorithm for the recovery of shading and reflectance images. Perception, 33(12):1463-1777
2004. 2004.
Bruno Olshausen and K Millman. Learning sparse codes with a mixture-of of-gaussanns prior. Ad- vances in neural information processing systems, 12, 1999.
Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 381(6583):607-669, 1996.
Rasmus Palm, Urich Paquet, and Ole Winther. Recurrent relational networks. Advances in neural information processing systems, 31, 2018.
13

---

Published as a conference paper at ICLR 2025
Kyubyong Park. Can convolutional neural networks crack sudoku puzzles? https://githh))
com/ Kyubyyyyu sudoku, 2018.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012
IEEE conference on computer vision and pattern recognition, pp. 1498-3505. IEEE, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kernel matters- impprveee semantic segmentation by global convolutional network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4353-4361, 2017.
Roman Pogodin, Yash Mchta, Timothy Lillicrap, and Peter E Latham. Towards biologically plau- sible convolutional networks. Advances in Neural Information Processing Systems, 34:13924- 13936, 2021.
Federico Rossi, Kenneth D Harris, and Matteo Carandini. Spatial connectivity matches direction selectivity in visual cortex. Nature, 588(7839))648 644-6622, 2020.
Stephanie C Seeman, Luke Campagnola, Pasha Davoudian, Alex Hoggarth, Travis Hage, Al- ice Bosma-Moody, Christopher A Baker, Jung Hoon Lee, Stefan Mihalas, Corinne Teeter, et al. Sparse recurrent excitatory connectivity in the microcircuit the adult mouse and human cortex. elife, 7:+37349, 2018.
Charles Scott Sherrington. The integrative action of the nervous system. In Scientific and Medical Knowledge Production, 1796-1918, pp. 217-253. Routledge, 2023.
Jianghong Shi, Eric BrownB and Michael Buice. Comparison against task driven artificial neu- ral networks reveals functional properties in mouse visual cortex. Advances in Neural Information Processing Systems, 32, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12 2012.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. Advances in neural information processing systems, 32, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference machine learning, pp. 10347-10357. PMLR, 2021.
J Hans Van Hateren and Arjen van der Schaaf. Independent component filters of natural images compared with simple cells in primary visual cortex. Proceedings of the Royal Society of London. Series B: Biological Sciences.
1998.
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12894- 12904, 2021.
14

---

Published as a conference paper at ICLR 2025
Po- Po-Wei Wang. Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and log- ical reasoning using a differentiable satisfiability solver. In International Conference on Machine Learning. pp. 6545-6554 PMLR, 2019.
Ross Wightman. Pytorch image models. https::/giicuu...om//viigg.ggggt
2019.
Sanghyun Woo, Shoubhík Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16133- 16142, 2023.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Aiwen Xu, Yuchen Hou, Cristopher Niell, and Michael Beyeler. Multimodal deep learning model unveils behavioral dynamics of v1 activity in freely moving mice. Advances in Neural Information Processing Systems, 36, 2024.
Qi Yan, Yajing Zheng. Shanshan Jia, Yichen Zhang, Zhaofei Yu, Feng Chen, Yonghong Tian, Tiejun Huang, and Jian K. Liu. Revealing fine structures of the retinal receptive field by deep-learning networks. IEEE transactions on cybernetics, 52(1))39-50, 2020.
Guangyu Robert Yang and Xiao-Jing Wang. Artificial neural networks for neuroscientists: a primer. Neuron, 10766):008--177, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Anthony Zador, Sean Escola, Blake Richards, Bence Ölveczky, Yoshua Bengio, Kwabena Boahen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, et al. Catalyzing next-generation artificial intelligence through neuroai. Nature communications, 14(1):1597, 2023.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1170. 09442, 2017.
Yajing Zheng, Shanshan Jia, Zhaofei Yu, Jian Liu, and Tiejun Huang. Unraveling neural coding of dynamic natural visual scenes via convolutional recurrent neural networks. Patterns, 2(10),
2021.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmen- tation. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 13001- 13008, 2020.
Haoran Zhu, Boyuan Chen, and Carter Yang. Understanding why vit trains badly on small datasets: an intuitive perspective. arXiv preprint arXiv:2302.0335),, 2023.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX / SUPPLEMENTAL MATERIAL
A.1 BIOLOGICAL RFs
For the biological RF analysis, we have analyzed in vivo intracortical connectivity dataset of Rossi et al. (2020) collected from the mouse primary visual cortex (V1) (Fig. [T). It contains both ex- citatory (CaMK2a) 2a-positive)) and inhibitory (Gad2-positive layer 2/3 neuronal spatial connectivity distribution (Fig. [b) which was determined by recording GCaMP6 signals (calcium activities) of pre-and post-synnn synaptic pairs (Fig. and 7
Given post-synaptic neuron positioned at (Fo, vo). (Fig. I black), and the No number of functional synapse positions nn. Un E (-00, 20) for n = 1,2, 2.., No. then relative offsets are defined as (Az,, Ayn) = (In 20: Vn yo). We summarize functional synapse positions for biological RF
.2....NN,,
as following
Aso = [(An, Aym))
(4)
A.2 ARTIFICIAL RFS
For the artificial RF analysis, we used untrained or pre-trained AlexNe with inputs (224 224)
of 17 images either generated from Gaussian noises or selected among 118 Natural Scenes images datasets (See details Appendix and Fig. [8. When image inputs were shown to AlexNet, we extracted RFs of the functional synapse from the first convolutional layer (Conv 1) (Fig. [ and d).
Given the input X' E RCxxHxW and weights parameters for Artificial RFs WARF E RG K the post-synaptic unit in the Convolution layer, receives weighted-input 2 E RV the results of element-wise multiplication between partial input and filters, where HH- (H- = m.m,m -1 1,nn+++ K W (Fig. D black) for 0 S m I H K₁, and In I W WK-- We calculate the weighted input as the convolution operation without summation across width, height and input channel. As result, weighted input Z has V x C2 K2, x K shape, where V /-HH (H K₁ + 1) x (W K + 1), and Z, E X xk denotes the v-th element of Z. Here, C₁ H, W, K2 and K denote the number of input channels, input height, input width, kernel height and kernel width, respectively. For simplicity, we assume that there is stride one and no sero-padding. For h E [0.... Kn. 1] and w E [0, (0..... K 1]. the relative offsets from the kernel center are defined as follows
KA K 2
As = (Ah, Aw) = (h
W
(5)
Since spatial connectivity pattern in the biological synapse is measured by the functional calcium activities and given as coordinates, we applied a similar approach that of CNN layers. We col- lected No functional weighted-inpute (functional synapses) where N₂ represents the number of cases where each elements of [Z] exceeds a threshold 0. Here, we defined 0 as the standard deviation of | This selection process yielded a different set of functional synapses input-dependent manner. We summarize functional synapse positions for artificial RF as following
As, ((1, k Ah, Aw)) Z Lu, 0}, where As, No-
(6)
FFor the pre-trained model, we used the torchvision's ImageNet- 1k pre-traines trained model
We determined the activity threshold based on a common method used in neuroscience to extract mean- ingful patterns in neural activity, which is similar to calculating the 7-score and typically setting threshold range of to 3 standard deviations identify values that are statistically significant.
16

---

Published as a conference paper at ICLR 2025
A.3 GAUSSIAN DISTRIBUTED FUNCTIONAL SYNAPSES OF POST-SYNAPHIN SYNAPTIC NEURON IN MOUSE VI LAYER 2/3
(c)
Figure 7: Distribution of post-synaptic functional synapses in mouse V1 layer 2/3 (a) Using dataset from Rossi et al. (2020), 3D scatter plot represent relative positions of both excitatory (red) and inhibitory (blue) functional synapses from the soma of the post-synaptic neuron. (b-d) 2d his- togram (left) and Gaussian fitted probability density function (right), showing the laminar organiza- tion functional synapses for all (b), excitatory (c), and inhibitory (d)
A.4 INDIVIDUAL RECEPTIVE FIELDS COLLECTED FROM BOTH BIOLOGICAL AND ARTIFICIAL SYSTEMS
Recisstie feetis lanaiona sprasioos messe Vi Loyenazs indivoloal oyrioatt
syrattitttttooo vouron Seor mocn and aanance nompleeit
AP
Reception functional sycaprue in dieeliet Canall
reeponce retural imaaee inpute Dewo maan and serianoe romnwood)
Receptiow numenooe comppoes Abwrter Candt respome Daam wose inpure bera maan and
1911
1011
1011
1991
Figure 8: Biological and artificial receptive fields with visual stimulus The receptive field dis- cussed in this figure specifically refers the spatial connectivity patterns of synapses. Note that this differs from receptive fields typically associated with low-level visual feature selectivity. (a) Receptive fields of individual neurons in V1 Layer2/3 from the dataset Rossi al. (2020) (b) Re- ceptive fields of untrained or ImageNet- Ik pretrained AlexNet's Convl I layer when Natural Scenes images were shown (c) Receptive fields of untrained ImageNet- pretrained AlexNet's Conv1 layer when Gaussian RGB noise were shown. All receptive fields are zero mean unit variance nor- malized.
17

---

Published as a conference paper at ICLR 2025
A.5 EXPERIMENTAL DETAILS FOR SUDOKU CHALLENGE
We utilized the extensive IM-sudoku dataset Park 2018). a resource also utilized in prior works (Amos & Kolter 2017 Palm al. 2018 Wang al. 2019). Sudoku, widely popular number puzzle, involves organizing digits in a grid such that each row (1x9), column (9x 1), and the Sudoku challenge, where achieving these three objectives simultancously is essential for complete Sudoku solving, there is an advantage that we can test the applicability and effectiveness of evolving RFs in the ly-convvlution. To achieve this, we compared five distinct models of Sudoku CNN: (3x3) Base model, (7x7) Large model, (7 x7) (Piit = 2). (7 (pint 16), and finally (7x7) L (Pinit = 256) (frozen p and model) For numerical stability, we clipped p 2 2 during the training of the Sudoku-solving task. The inputs, targets, and model architecture are outlined in Figure 5p. As illustrated, our model com- prises repeated Conv2dSame blocks, originally introduced in SudokuCNN (Oinar) (022)). Each Conv2dSame block encompasses Reflection padding, followed by conventional Convolutional or Thp Convolutional layer, Batch Normalization, and activation function. The Convolutional layer has 256 channels, and the number of blocks set at I. 10.
box (3x3) contains all numbers from
A.6 CONFORMATIONAL ANALYSIS OF L. MASKS
We defined the properties of scale (a), rotation (0), and distortion (7) through Singular Value De- composition (SVD) on C for each output channel, as shown in the following equation:
C -=UAV
Here, U and V represent 2 x 2 unitary matrices containing the left and right singular vec- tors. A is diagonal matrix containing the singular values (1), Rotation is quantified as 0 arctan sin(V" cos(V (in degrees), providing measure of rotational transformation. Distor- tion is quantified as
offering valuable information about the deformation present in the data. Scale is quantified as V Ao
( indicating the size of mask. In Figure 5a show the example shapes of L---makks by reverse calculating C from the given o, 0, 7-
18

---

Published as a conference paper at ICLR 2025
A.7 LAYER-WISE VISUALIZATION OF Ly-MASKS FOR SUDOKU-Z,CONVNET
Layer-wise visualization all receptive fields Sudoku LpConvNet (p-2)
LpConv1
LpConv2
LpConv3
LpConv4
LpConv5
LpConv6
LpConv7
LpConv8
LpConv9
LpConv10
Figure Layer-wise visualization of -p-masks for Sudoku-L,.conNeet All learned Lp-maske after Sudoku task training of Co-CoonNet(PPut 2). With an increase in layer depth, the sizes of masks get larger.
19

---

Published as a conference paper at ICLR 2025
A.8
ACCURACY CURVES FOR SUDOKU ABLATION EXPERIMENTS ON (P=22)
Ablation in 1~10 layer
Ablation in 1-5 layer
Ablation in 6-10 layer
0.95
acc
acc type
type
0.9
0.94
row
cal
col 0.90
0.8
box
0.92
box
box
0.85
#1
0.90
0.80
type
hyce
0.95
eec_type
Accuracy
0.9
0.94
col
box
box
0.85
#2
0.90
0.95
type
0.94
acc_ _cpp
acc_ ecc_type
0.90
0.8
0.92
box
0.80
#3
0.75
Baseline
89 91
92"
93*
86 - 94
Baseline
91*
88 - 92
93
86 94"
Baseline
89 ~ 91"
92"
87 - 93
86 94"
8/-
87
89
87
88
Theta ablation range
Figure 10: Accuracy curves for Sudoku ablation experiments on Lp(p=2). individual experi-
ments (rows) of vertical mm-kss ablations with different conditions (columns; layer to 10, left; layer to middle; layer to 10, right) (p=2)
20

---

Published as a conference paper at ICLR 2025
A.9 EXPERIMENTAL DETAILS OF VISION CLASSIFICATION TASK
We conducted our training on two datasets: CIFAR-100 Krrizeeskyy et al. 2009) and Tinylma- geNet (Le & Mikolov 2014). 100 comprises 32 x 32 pixel images distributed across 100 classes, while Tiny ImageNet consists of 224 x 224 pixel images spanning 200 categories. Following standard procedures, we reported top-1 accuracy with corresponding mean and standard deviation. Our implementation is based on the PyTorch framework (Paszke et al. 2019). making extensive use of the timm repository (Wightman, 2019). We adopted a training strategy rooted in DeiT Touvron et al. 2021). incorporating techniques such as RandAugment (Cubuk et al.), Mixup (Zhang et al.) 2017). Cutmix (Yun et al., 2019), random erasing (Zhong et al. 2020), and stochastic depth (Huang et al. 2016). The optimization process employed AdamW (Loshchilov & Hutter 2017) with de- fault momentum value of 0.9 and a weight decay set at 5 x 10- We initialized our learning rate at 10 and implemented cosine learning rate schedule. All models underwent training for 300 epochs, utilizzig a batch size of 128. For CIFAR- 100, training was conducted on 2 GTX 1080ti GPUs, while Tesla V100 GPUs were used for TinylmageNet.
A0 THE ALLEN BRAIN OBSERVATORY DATASET
The Allen Brain Observatory dataset de Vries et al. 2020) constitutes comprehensive standard- ized in vivo examination of physiological activity within the mouse visual cortex. encompasses recordings of visually-inducee calcium responses from neurons expressing GCaMP6f. This dataset encompasses cortical activity from nearly 60,000 neurons originating from six distinct visual areas, four layers, and twelve transgenic mouse Cre lines. These recordings were gathered from 243 adult mice in reaction to a diverse set of visual stimuli. In this study, we focused on utilizing the collective neural responses from five visual areas (VISal, VISam, VISI, VISp, VISpm). Layer 2/3 (depth range of 175mm to 275mm), and three mouse lines (Cux2-CreERI, Emx EEnx--IRESCCE Cre) when presenting dataset of natural scenes the mice. This dataset comprised 118 natural images obtained from three different databases (Berkeley Segmentation Dataset (Martin et al. 200T), van Hateren Natural Image Dataset (Van Hateren & van der Schaaf 1998), and McGill Calibrated Colour Image Database (Olmos & Kingdom| 2004)). Further details regarding the experiment can be found in (de Vries et 2020). In our study, we employed both images and neural responses for experiments involving representational similarity analysis to evaluate the correspondence between CNNs and the visual cortex, mirroring earlier investigations (Shi et al. 2019 Bakhtiari al. 2021)
Slc 7-7-IRES2-
A.11 REPRESENTATION SIMILARITY ANALYSIS
While the details of RSA are expertly addressed in Diedrichsen & Kriegeskorte (2017). let us briefly cover our specific approach. We leveraged the codebase provided by Bakhtiari et al.) (2021). In RSA, we generate response matrices (R) for brain regions and neural network layers, with dimen- sions N x M (where N is the number of image inputs and M is the neuron count). Using Pearson correlation, we compute similarities within matrix to construct the N N Representation Simi- larity Matrix (RSM). Additionally, following the methodology Bakhtiari et al. (202I), we applied noise correction by normalizing the RSAs using the noise ceiling values. These values were obtained through comparisons of representations across different mice. For example, the noise ceiling value for VISp is derived by calculating the RSMs of VISp from different animals and taking their median. To assess the similarity between RSMs (SSM), employ Kendall's for robust agreement, which helps mitigate potential bias from measurement noise Diedrichsen et al. (020)).
21

---

Published as a conference paper at ICLR 2025
A.12
PAIR-WISE REPRESENTATION SIMILARITY ANALYSIS BETWEEN ALL CNN LAYERS
AND SUBREGIONS
AlexNet
V15am
VSpm
102
SSM
1.4
0.20
100
1916
2.15
0.0
0.0
2.5
0.0
25
0.0
2.5
2.5
Corv Layer Depth
VGG-16
ViSal
Vibam
VISI
ViSp
V Vispm
2.25
SSM
0.15
0.05
10
Conv Layer Depth
ResNet-18
Vp5
0.24
SSM
2.15
Lo0
Lotk
1.0
Layer Depth
ResNet-34
ViSal
ViSam
VISI
ViSP
V V.5m
0.25
0.20
Lo2
SSM
0.10
0.05
Comv Layer Depth
ConvNeXt-T VIsi
SSM
L516
20
20
20 Conv Layer
20
20
Figure 11: Pair-wise representation similarity analysis between all CNN layers and VC sub- regions. We show the SSM scores for all pairs of Conv layers from CNNs and VC subregions. y-axis, SSM score; x-axis, Conv layer depth. For Max. SSM, choose the highest SSM among all wise SSMs.
22

---

Published as a conference paper at ICLR 2025
A.13
VISUALIZATION OF FUNCTIONAL RECEPTIVE FIELDS OF PRE-TRAINED ALEXNET
Pre-trained AlexNet
vs
(a)
(b)
(c)
Kernels only
Kermels Image inputs
Kernels Noise inputs
W
2 X Wn r
>
Z & Wa 2
(d)
(e)
Functional Connectivity Distribution of Pre-trained AlexNet Conv1 inputs
Functional Connectivits tivicttiityy Distribution of Pre-trained AlexNet Conv1 Noise inputs
Figure 12: Visualization of functional receptive fields of pre-trained AlexNet Convl with image or noise inputs. Visualization first 20 kernels of total 64 without inputs (a; column orders: RGB, R, G. B), with image inputs (b; column orders: RGB, R, G. B, R, G, B) with noise inputs (c; column orders: RGB, R, G. B, R. G, B). (d) Histogram of functional connectivity from (b). (e) Histogram of functional connectivity from (c). W. Weight; X, kernel-sized input; Z, kernel-sized output; 0, activity threshold; O element-wine wise product. See Appendix A.2. for methodological details.
23

---

Published as a conference paper at ICLR 2025
A.14 14
VISUALIZATION OF FUNCTIONAL RECEPTIVE FIELDS OF UNTRAINED ALEXNET
Untrained AlexNet
vs
(a)
(b)
(c)
Kernels only
Kernels Image inputs
Kernels Noise inputs
W
2
- & Warl 2
263
(d)
(e)
Functional connectivity Distribution of untrained AlexNet C Conv1 Image inputs
Functional Connectivity Distribution of untrained AlexNet Convl Noise inputs
Figure 13: Visualization of functional receptive fields of untrained AlexNet Convl with image or noise inputs. Visualization first 20 kernels of total 64 without inputs (a; column orders: RGB, R, G. B), with image inputs (b; column orders: RGB, R, G. B, R, G, B) with noise inputs (c; column orders: RGB, R, G. B, R. G, B). (d) Histogram of functional connectivity from (b). (e) Histogram of functional connectivity from (c). W, Weight; X, kernel-sized input; Z, kernel-sized output; 0, activity threshold; O element-wing wise product. See Appendix A.2. for methodological details.
24

---

Published as a conference paper at ICLR 2025
A.15 VISUALIZATION OF MPND WHEN 2
Figure 14:
Visualization of MPND when p < 2 Given the value of p 1, MPND distribution
becomes diamond shape. When p < L, the distribution becomes cross-like shape.
A.16
POST-TRAINED P-DISTRIBUTION IN Ly LA-MSKSS
A.16.1
OVERALL DISTRIBUTION IN CIFARIO0 100
AlexNet
VGG-16
ResNert-8
ResNett-3
ConvNele-T
atlegol
Figure 15: CIFAR-10--traieed p-distribution of 4, -m-sss
1616.2 OVERALL DISTRIBUTION TINYIMAGENET
AliorNat
VGG-16
ResNert-18
RecNart-4
Figure 16: TinyImageNet--rrinee p-distribution of ma-ks
25

---

Published as a conference paper at ICLR 2025
A.16.3 LAYER-WISE DISTRIBUTION IN CIFAR100
Liyer-wise p-distribution of CIFAR-100-trained Lp -p-masks All values (Median + Stdev) are calculated with of all -askss in each layer, from 5 different trials of CIFAR-100- trained models.
Table 6:
CIFAR-100
Model
Cor
1:99
1,88
0.16
15.25 I FEISS
2,00
3: 0.05
393
0.19
0.52
15:35 1 0.96
AlexNel
1.91 0.07
3.70
0.16
1.6 0.37
15.06 1 0.79
1,00 0.13
3.38
0.52
1.1%
0.36
0.64
2.01
0.00
0.13
0.84
1.9
0.05
1.68
0.11
7445 0.25
15300 0.58
1.9
0.05
1:72
0.12
7.1 0.26
1541 061
L82
0.05
0.13
1,50
0.26
94 0.57
1.71 0.06
3.57
0.17
8.00
0.01
16:00 0.0
ResNet-18
188
0.05
0.05
3661
0.12
7.99
50
0.21
15.47 0:58
18
0.05
1.52
13
1.62
0.05
3.42
0.14
M00
0.00
16.00 000
14
99
0.06
182
0.11
7.72 0.20
2.01
0.30
1,94
0.05
0.10
54
0.41
18
1 74
0.04
1686
0.12
800
000
16
0.0
99 = 0.06
402
0.11
10
021
19
3.02
0.05
0.09
0.25
0.05
3,75
0.10
35 0.51
99
0,05
3.78
0.11
0.25
15
0.58
195 0.0S
3:1
0.12
0.26
15
0.05
2.01 0.05
16 0.52
1.83
0.05
3.52
0.11
7,45
0.25
H3 0.58
Lo9
0.05
0.17
8.00
0.01
1600 0.00
0.05
79
0.10
022
I5
0.52
H4
0.05
3.89
0.12
1,83
0.21
16 0.52
15
L.7
0,05
3885
0.11
7.71 0.21
1606 0.2
ResNet-34
0
18
0.07
559
0.15
7.4
0.21
4 0.56
19
LG5
0.05
3.48
0.14
800
0.01
15.00
000
20
1995
0.05
0.11
7.78
0.21
0.1
01
23
0.06
0.10
0.52
24
3.05
0.05
4,05
0.10
8.12 0.20
16.
0.55
2.03
0.06
4.11
0.12
8.23 0.24
0.1
28
3
0.06
4.06
0.0
0.53
20
0.02
0.06
4.11
0.13
R.32 0.28
36 0.50
1.16
0.00
4.31
0.10
8.59 0.20
17
0.1
33
96
0.05
3.99
0.10
K0S 0.19
13 0.30
34
2.02
0.04
4,05
0.08
8.15 0.13
26 0.26
1.97
0.05
3.95
0.11
8.01 0221
16.L 0.27
2.07
0.0%
2.00 0.06
0.3
1.9
0.05
1888
0.13
7.05 0.31
15.5
0.00
96
0.06
38
0.13
7.69 0.26
5060 0.60
1.95
0.05
VGG
0.00
1.81
96
0.06
1.93
0.11
1.04
0.36
15 04 0.69
2.02
0.06
4404
0.10
8,07
0.24
17 0.67
1.95
0.077
3.98
0.12
7.1
0.18
0.90
2.00
0,05
401
0.00
8,00
0.30
I 05 0.43
X
139
U.TS
137
(1:3)
15
88
0.06
366
0.15
7.09
14.47 0.55
0 05
0
0
99
0.08
3:77
0.14
7.1 0.29
45.41 0.55
K3 0.
1:22
0.16
0.30
15
0
0
CassNeXi-T
11
1.95
L95
0.07
0,07
3.87
89
0.16
0.15
1.83
0.23
15000 0.00
0000
95
0.08
1:78
0.19
7.70
1600 000
0.07
0.07
3.95
0.20
0000 0.00
16
977
391
0.15
7.8 0.25
15:00 0.00
1.97
0.077
395
0.14
190
0.20
1600 000
2.00
0.00
000
21
3
0.00
4,00
0.00
800
0.00
15000 4 000
3.00
+ 0.00
4.00
0.00
8,00 2: 000
1600 2 000
26

---

Published as a conference paper at ICLR 2025
A.16.4 LAYER-WISE DISTRIBUTION IN TINYIMAGENET
Table 7: Layer-wise p-distribution of TinyImmaeNNet-traiie -p-masksAll values (Median +
Stdev) are calculated with of all maskss in each layer, from different trials of TinylmageNet- trained models.c
TinyImageNet
Model
1:91
1.21
1,9
3: 0.06
391
0.28
7.50
0.70
15:1
1:33
AlexNel
199 0.11
146
0.25
1.15 0.66
14.78
1.34
1.71 0.16
297
0.57
6.51
0.58
13.86
0.91
2007 0.08
1.9
0.08
3.75
0.17
7.36 0.30
16.97
1.42
199 -- 0.09
1.75
2- 0.17
0.36
16.02
1864
0.11
0.22
15.33
90
0.08
4.05
0.23
8.33 0.31
17.39
0.33
ResNet-18
1
0.12
348
0.24
2.17 041
7.06
044
15.96
0.11
13
1:75
0.06
3777
0.18
8.44 0.25
17.16
0.27
14
84
0.12
199
0.23
7.07 0.37
16-47
0.08
0.16
15.00
0.20
18
X2
0.06
4.28
0.17
9.25 0
I7
0.16
7
0.10
125
0.16
1.18 0.38
0.06
0.05
0.13
0.67
100 0.09
3.87
0.20
0.53
27:23
30
1.04 0.00
88
0.19
150 0.49
17.14
1.02 0.09
880
0.22
0.40
17.41
2.10 0.10
1.77 0.12
3
0.25
8.33 0.53
55.58
0.98
L85
0.07
0.20
8.7
17.14
0.30
0.12
67
0.22
M.
27
H4
2.05
0.11
888
0.25
1.4 0.55
17
15
0.11
3:75
0.22
1.3%
47
ResNet-34
18
0.12
3
0.26
90 042
96
19
1.75
0.05
3:75
0.18
MAI 0.25
17 13 0.26
L899
0.12
3.52
0.24
0.39
15.04
23
0.12
181
0.27
1.55
24
L8
0.11
388
0.19
1.5 0.34
17.7 107
2.00
0.10
192
0.23
7.1 047
085
28
0.08
3.77
0.16
79
0
20
07
0.10
1
0.21
R335 0.51
18 54
033
99
0,00
390
0.20
1.0 0.38
75
33
0.00
3.51
0.16
6:00 0.30
34
L88
0,09
3.55
0.19
143 0.29
29
1002
L8T
0.12
0.19
041
0.97
0.10
2 0.11
4.09
0.23
1.9 0154
17 38
1.30
1S 0.10
1.12
0.21
7.92 0.49
76
1.95 0.10
VGG
2.84
0.14
398
1.9
0.13
77
0.30
5,50 0.54
34
191
168
7.99 044
0804
2.00 0.15
390
0.29
7.1
0.10
96
86
0.14
7.64 0.36
16.00 + 0.80
401
U.3T
X = T.0
24 2
195
0.10
3-70
0.22
1.42 0.47
14.11
0.92
0.11
71
021
0.15
92
95
0.11
365
0.22
1.11 0.44
14 38
101
95
0.09
0.25
701 0.45
36 0.88
CassNeXi-T
11
945
0.09
0.09
350
3151
0.20
0.20
0,65 0.42
13 034
1 0.89
95
0.10
1.50
0.20
5,2 0400
955
0.11
3.56
5.71
13
16
0.11
158
0.19 1.59
5,50 0.40
14 0.5
0.11
7,05 042
14.44 038
701
0.10
0.77
21
187
0.10
3.5
0.19
121 0.35
14.5
0.77
129
0.09
377
2 0.17
7,5 2 0.55
1500
000.4000 0.76
27

---

Published as a conference paper at ICLR 2025
A.17
LEARNED Ly Lp-MASK PROPERTIES (p=16)
Layer-wise learned Lp mask properties (7x7) Lp (p=16)
Conya
by Convi
asadllim
Convl
Conuz
ConV2
4pCone
Conv2
la-Con3
AyConv
Conva
Como
Canon
GoCond moll
IpCans
kydonss
Comos
Mi Comes
LyConv6
4 Conve
CyCone
4. Convé
Conu7
b Conv?
4-Com?
Conv?
Conv8
Conv8
Conva
LeConwe
fyConve
4onos
Conos
1o-Co030
4p<Convio
Senvoo
Comooo
4
2
45"
90"
135"
180
12
16
20
Scale
(y)
Figure 17: Layer-wise distribution of learned Lp-mask properties (p=16)
A.18 STATISTICS FOR COMPARISON OF OPTIMIZED MPND P VALUES IN ARTIFICIAL AND BIOLOGICAL RFs
Table Holn-Bonferroni corrected multiple comparisons each RFs using Welch's t-test. Lower
diagonal elements denote corrected statistical p-values and upper diagonal elements denote degree
of significance (n.s, not significant; 0.01 <000pp00005 ** <p0011p=0.00; *** p=c0.00:;
Welch's t-test
Mouse V1
Trained
Trained
Untrained
Untrained
(corrected p-values)
Layer 2/3
Image
+ Noise
+ Noise
Image
Mouse V1
##$
***
Layer 2/3
Trained
Image
1.10e-2
n.s
$#8
##*
Trained
1.06e-2
7.6e-1
##*
+ Noise
Untrained
1.23e-15
4.13e-28
1.49e-25
+ Noise
Untrained
8.94e-14
1.59e-25
6.73e-23
1.39e-22
Image
28

---

Published as a conference paper at ICLR 2025
A.19
IMPACT OF Ly- CONVOLUTION (P=2) ON OTHER CNN ARCHITECTURES
Table Top-1 1 performance (meantstd, 5 trials) on the CIFAR- 100 datasets with Lp-Convolution
applied in Von-NeXI-V2-T (Woo et al. 2023), ResNet-50 (He et al. 2016), ResNext-50 (Xie al.,
2017) and DenseNet- 121 (Huang et al. 2017). The symbol indicates Lp-Converted (Pinit - 2) or
not. denotes statistical comparison using Welch's t-test (p 0.001).
CIFAR-100
Conv ConyNeXt--2-T
ResNet-50
ResNeXt-50
DenseNet-121
64.26 # 0,41
73.17 + 0.23
73.55 # 0.57
74.12 # 0.16
65.58 + 0.25
76.66 0.19
77.38 0.36
77.14 0.18
A.20 RECEPTIVE FIELD
The term RF was initially confined to a specific area impacting single neuron in the visual system, as outlined by Sherrington in 1906 and later by Hartline in 1938 (Sherrington [023 Hartline] 1938). was defined distinct region in visual space capable of triggering electrical responses in retinal ganglion cells, thus highlighting immediate and localized neural interactions.
Modern interpretations, however, have substantially expanded the scope of receptive fields. The groundbreaking work of Hubel and Wiesel exemplifies this evolution, revealing how receptive fields process complex visual patterns through multiple layers in the primary visual cortex (VI). This progression from Hartline's narrower viewpoint to a more all-encompassing approach mirrors the intricate, multi-layered nature of sensory processing in the brain. The present-day definition, shaped by Hubel and Wiesel's insights, underscores the dynamic, malii-dimensional nature of neural re- sponses.
In CNN, the concept of receptive field aligns somewhat with Hartline's original idea, focusing LeCon mainly on local input processing (Fukushima) 1980 LeCun al. 1989). In CNNs, RF typically de- notes the localized interactions among successive layers, reflecting Hartline's 's emphasis localized sensory inputs. In our analysis, we align with more restricted definition of the receptive field, akin to the original concepts from Sherrington and Hartline. This approach corresponds to the local connectivity between adjacent layers in CNNs, aiming to provide a clearer and more focused understanding in both biological and computational contexts.
29

---

Published as a conference paper at ICLR 2025
A.21 PYTORCH-STYLE PSEUDOCODE FOR Lp- CONVOLUTION
Algorithm PyTochh-style pseudocode for L Convoluttn
import torch
import torch.nn as nn
import arch..n.functional as F
from torchvision.models import alexnet
class L=Conn2d(mn.Con220):
def (self, p_init, sigma_init, in_channels, out_channels, Kernel_size, stride, padding, kwargs):
# Create parameters p
params_p torch.ones( out_channels p_init
params_ C torch. out_channels,
params [[:,000] = 1/sigma_ing init
params_C[;,11.) = 1/sigma_inis
self.p wan.Parameter params-p)
self.C Parameettt params C)
def forward(self, input):
# Create channel-wise Ip_masks from parameters p and C
1p_masks www..channel.vccc...pmaaase msell. self.C)
# Overlay Ip pa_sssss on weight
masked weight weight ip masks
return F.con22d(inpu masked weight, bias, kernel size, stride, padding, **wwaggs)
def LpConvertmmodel p-init):
Convert all mn. ConCon22 layers into LpConv2d
for i in range(num.layenn layers):
layer = model.layers[[]
if isinstance(layer, nn. Convd)::
model LpConv2d p_init,
sigma.init, in chamnels=layer.n channels,
out channels=layer.out channels, kermeelsizeclayerkkenel...i stride=layer.stridee, padding=llaer.padddnng **Jayen.extra.argg)) return model
Example -pConvert on Alexnet for TinylmageNet
mase_moder (aaaeeet(nuumclassss=2222 1p2_model LpConver((base.mon _moee, p-init=2)
30

---

Published as a conference paper at ICLR 2025
A.22
COMPARISON OF VIT AND LP2-CNNS
Model
Top-1 (%)
FLOPs (G)
Params (M)
ViT-32x32
49.88
4.37
87.6
ViT-16x16
54.20
16.87
86.0
AlexNet
52.25
0.71
57.82
Lp2-AlexNet
54.13
3.41
68.6
Lp2- VGG-16
69.96
83.74
200.5
Lp2-Reet 122-ResNet--18
68.45
9.86
61.5
Re2--34
70.43
19.93
116.6
Lp2-ConvNeXi-T 70.72
3.42
33.8
Table 10: Lp2-AlexNet achieves comparable performance to ViT-16x16 on TinylmageNet with
significantly lower parameter counts and computational cost, demonstrating its efficiency.
A.23 THE JUSTIHICATION FOR USING LARGE, SPARSE KERNELS
The use of large kernels enables the model to cover the input space more effectively with fewer layers compared smaller kernels (Ding et al., 2022 Luo et al. 2016). However, simply increasing the kernel size does not guarantee performance improvements, as shown in Table I (Base vs. Large). This is presumably due to larger kernels inadvertently incorporating irrelevant global information, which can hinder performance compared to smaller kernels that rely on locality inductive biases to extract local features hierarchically. This is where sparsity plays key role. We introduce sparsity constraints to optimize the usage of large kernels, ensuring they focus on only relevant global information while mitigating the disadvantages of naively expanding kernel sizes, supported by Sudoku experiments.
A.24 EVIDENCE FOR GAUSSIAN SPARSITY AS BIOLOGICAL CONSTRAINTS
A.24.1 1.THEORETICAL EVIDENCE
Sparse Coding Theory Sparse Coding Theory posits that neural systems optimize sensory rep- resentations by minimizing redundancy. Learning a sparse code for natural images leads to the emergence of simple-cell receptive field properties (Olshausen & Field 1996). This process can be linked with Gaussian priors, where synaptic weights follow Gaussian distribution with most connections being weak and a few strong, promoting efficient information encoding (Olshausen & Millman 1999).
Effective Receptive Field (ERF) Theory In convolutional neural networks, the actual influence of input pixels on an output neuron decreases in Gaussian manner from the center of the theoretical receptive field (Luo al.) 2016). This means that while the theoretical receptive field defines the maximum possible area of influence, the ERF is effectively smaller and Gaussian-shaped, with central pixels contributing most significantly to the neuron's output.
A.24.2 EMPIRICAL EVIDENCE
Supporting References These two references demonstrate both anatomical and functional dis- tribution synapses predominantly following a Gaussian-like distribution in the visual cortex (Hell- wig. 2000 Rossi et al. 2020
Analysis Result We have demonstrated Gaussian distribution with the in vivo functional synapse data in) alive mouse V1 in Appendix A.3
31


Reviews:
[{"review_id": "fUAX7zhtT7", "reviewer": "Reviewer_7Z3f", "summary": "The paper proposes a brain-inspired approach of constraining the weights of convnets with a p-generalized Gaussian envelope. The authors demonstrate some minor improvements in performance on relatively small image datasets such as CIFAR-100 and TinyImageNet. They further claim that the learned representations of more \"brain-like\" convnets have higher representational similarity to the mouse visual system than their more classical counterparts.", "questions": "None", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "+ Novel inductive bias for convnets motivated by biology\n + Overall fairly well-written paper\n + Well-motivated and well-executed experiments", "weaknesses": "- The effect sizes are really small, calling into question the practical impact\n- Only \"toy\" datasets are explored\n- Experiment on representational similarity not convincing\n\n\n### Detailed explanation\n\nWhile I find the paper well motivated and the idea original, I see the paper mostly as a negative result given the small effect sizes observed across most of the tables and the \"toy\" nature of datasets such as CIFAR-100 and TinyImageNet. The paper now has undergone several revisions that only reinforce this conclusion.\n\nThere is a statistically significant improvement due to Lp-Conv for some classical architectures, but not all of them (e.g. ResNet). Generally, the improvements are small (a few percent). Given that architectural modifications alone can now push accuracy on CIFAR-100 >90% (https://arxiv.org/abs/2304.05350v2), the 1–2% improvements in the 60–70% range feel insignificant. Experiments on more modern architectures such as RepLKNet (Table 3) show the same pattern, if anything with decreasing effect size (<1% improvement). Similarly, the transfer learning experiment using ConvNeXt-V2 (Table 4) shows close to no effect. There are no experiments on closer-to-real-world datasets like ImageNet (although that's by now a fairly standard problem that can be done on a consumer GPU), although I should say that I do not expect major effects in that experiment, either. The data simply show that the inductive bias doesn't do much.\n\nThe experiment on representational similarity yields equally small effect sizes, again insignificant on many architectures. In addition, the comparison is done to several mouse visual areas, some of which aren't even part of the \"ventral\" stream for which a convnet trained on image classification would be a reasonable model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W46MlQXG3b", "reviewer": "Reviewer_nN4T", "summary": "This paper introduces $L_p$-Convolution by integrating the multivariate p-generalized normal distribution into a $L_p$ masks for convolution filters. It allows the network to adapt to different receptive field shape and train efficiently with large kernel. The paper show $L_p$-Convolution has a advantage in tasks such as sudoku challenge.", "questions": "See weakness for suggestions.\n\nPotential typo:\nline 159: a solution to the of large kernel problem in CNN -> a solution of the large ...", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The idea of a $L_p$ convolution is original and could have wide application in visual tasks that require flexible receptive field size or in general tasks that require both local and global information from visual input. Showing its ability to transfer to any pretrained network greatly lowers the threshold to apply this for a wide range of tasks. The choices of Sudoku task and the follow up ablation analysis is solid and demonstrates well the strength of this method. Out of all papers that take inspirations from neuroscience and try to utilize it to improve neural nets, this paper stands out in actually providing a fundamentally different implementation of CNNs.\n\nOther that than the points mentioned above, the benchmark testing was thorough and presented clearly.  The visualization of both the $L_p$ mask and convolution is very helpful for understanding the concepts. The writing is very clear.", "weaknesses": "The paper does show through table 1-4 that $L_p$-CNNs can train with large kernels and have some advantage in robustness as well as accuracy in benchmark test. However the improvements over baseline models are small. I don't think these numbers convinces me how useful the $L_p$-CNNs could be. Aside from the sudoku task, the paper didn't really show the advantage of efficiently trained large kernal $L_p$-CNNs through a task that actually could really benefit from large kernels. I would suggest including some more tasks that requires processing of context, or even tasks ViT excels at for comparison. \n\nFor the robustness benchmark as well as the Sudoku task, it could be informative to include performance of ViTs as well. \n\nLastly, it is pretty well established throughout the paper that $p_{init}=2$ is the most useful for most task, and resembles the most to the biological system. I am not sure it is worth having another section (sec. 6) dedicating to comparing similarity of RSM across different $p$s. If the author were to demonstrate it is also a better model for brain representational alignment that I would recommend doing a more thorough study including more datasets, brain regions and variation of CNNs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qtrgx9PiR8", "reviewer": "Reviewer_ZVMo", "summary": "This paper introduces Lp-convolution, a novel approach to convolutional neural networks (CNNs) inspired by biological visual processing. The work addresses fundamental differences between artificial and biological visual systems: while traditional CNNs employ rectangular, dense, and uniform connectivity patterns, biological visual systems feature circular, sparse, and normally distributed connections. Additionally, the paper tackles the longstanding challenge that large kernel sizes in CNNs typically don't improve performance despite increased parameters.\n\nThe key innovation is the introduction of Lp-convolution, which uses multivariate p-generalized normal distribution (MPND) to bridge these biological-artificial differences. The method implements trainable \"Lp-masks\" that can adapt their shape through parameters, enabling flexible receptive field shapes that better match biological patterns. Technically, this is achieved by applying channel-wise Lp-masks that overlay onto convolutional kernels, with shape parameters that can be trained for task-dependent adaptation.\n\nThe authors demonstrate several significant findings: Lp-convolution improves the performance of CNNs with large kernels, with optimal results achieved when the initial p parameter approaches 2 (matching biological Gaussian distribution). Moreover, neural representations show better alignment with the visual cortex when connectivity patterns are more biologically plausible. \n\nThe practical impact of this work is threefold: it enables effective utilization of larger kernels in CNNs, achieves more biologically plausible artificial neural networks, and maintains compatibility with existing CNN architectures.", "questions": "**(More of a curiosity)** For future developments of this work, it would be interesting to explore connections with anisotropic diffusion (Perona & Malik, *Scale-space and edge detection using anisotropic diffusion*, 1990). In standard convolution, there exists a well-established mapping between convolution operators and isotropic diffusion processes (as explored in Scale-Space theory, particularly in Koenderink, *The structure of images*, 1987; and Lindeberg, *Scale Space theory in computer vision*, 1994). How might Lp-convolution relate to or extend these theoretical frameworks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is overall well-written, with sections that tell a clear, sequential story. The usage of bold characters to highlight important parts is particularly appreciated. This is a very strong contribution across multiple aspects:\n\n* Connectivity patterns as inductive biases are largely unexplored within this community. The biological inspiration effectively guides the search for plausible connectivity patterns, and the approach proposed in this submission is particularly apt. The work presents a complete narrative, from biological mechanism inspiration to the implementation of Lp-convolution for neural activity prediction in V1 and representational similarity analysis.\n\n* The paper's approach to addressing large kernel network training challenges could potentially bridge the performance gap between transformers and CNNs in image classification tasks.\n\n* The mathematical formulation is sound and accessible, with figures (e.g., Figure 1 and 2) that effectively illustrate concepts and build intuition about parameter effects.\n\n* The choice of the Sudoku challenge adds significant value, serving as an excellent demonstration of the model's capabilities in an easily understandable context (especially for $L_p$ mask shapes)\n\n* The Appendix comprehensively addresses potential questions, demonstrating thorough consideration of the work's implications and limitations.", "weaknesses": "I don't think significant weaknesses are present in this work. The paper should be accepted as it is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TEK5zZd9zH", "reviewer": "Reviewer_ozim", "summary": "This paper primarily investigates variations in local connectivity patterns within CNNs, examining whether incorporating biologically inspired connectivity structures can improve model performance and increase alignment with brain representations. Specifically, the authors introduce Lp-convolution, which utilizes the multivariate p-generalized normal distribution (MPND). The proposed adaptable Lp-masks aim to bridge the gap between artificial and biological connectivity patterns, finding optimal configurations through task-based adaptation to enable strong performance in tasks requiring flexible receptive field shapes.", "questions": "* In the authors' claim regarding \"Lp-convolution with biological constraint,\" specifically the \"Gaussian structured sparsity,\" what theoretical and empirical evidence supports this biological constraint? \n\n* Across various experiments, since ppp is a learnable parameter, what typical values does it converge to, and are there any observable trends or variations across different datasets? Could the authors interpret these findings in relation to biological insights?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Experimental results indicate that CNN neural representations exhibit a stronger alignment with the visual cortex when the Lp-mask shape approximates a Gaussian distribution. \n\n* Testing the conformational adaptability of Lp-masks in the Sudoku challenge yielded interesting results, highlighting the flexibility of this approach.", "weaknesses": "* Consistency in terminology would improve clarity; alternating between “Lp-convolution” and “Lp-mask” can be confusing. Using a single term throughout would make the concepts easier to follow. \n\n* The mention of Vision Transformers (ViTs) in the introduction feels tenuous, as they are not included in subsequent experiments, nor are they closely related to the main theme of the paper. \n\n* In lines 108-110, where it is stated that “CNNs have rectangular, dense, and uniformly distributed connections, as opposed to the circular, sparse, and normally distributed connections in biological neurons,” this description would benefit from supporting references regarding the shapes of receptive fields in biological neurons. It’s also worth questioning whether this statement accurately characterizes CNN weights, as CNNs trained to model retinal ganglion cells, for instance, have demonstrated sparse weight patterns ([1]-[5]). \n\n* Lines 137-138 mention that “we optimized parameters of p and σ in MPND (Fig.1e, Eq.1)...” However, Eq.1 and the text do not define σ. It’s also recommended that the authors confirm Eq.1’s form by referencing the standard expression of a multivariate Gaussian function. \n\n* Integrating Lp-masks in CNNs does not appear to significantly improve recognition accuracy across datasets. Comparing this approach to ViTs, it’s unclear if it achieves current state-of-the-art performance. \n\n* The justification for using large, sparse kernels feels somewhat weak. Aside from achieving marginal improvements in RSM alignment with the visual cortex, it’s unclear how this approach benefits contemporary computer vision tasks. \n\n\nReferences:\n\n[1] Maheswaranathan, Niru, et al. \"Deep learning models reveal internal structure and diverse computations in the retina under natural scenes.\" BioRxiv (2018): 340943. \n\n[2] Tanaka, Hidenori, et al. \"From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction.\"  Advances in neural information processing systems 32 (2019).\n\n[3] Lindsey, Jack, et al. \"A unified theory of early visual representations from retina to cortex through anatomically constrained deep CNNs.\"  arXiv preprint arXiv:1901.00945 (2019).\n\n[4] Yan, Qi, et al. \"Revealing fine structures of the retinal receptive field by deep-learning networks.\"  IEEE transactions on cybernetics 52.1 (2020): 39-50.\n\n[5] Zheng, Yajing, et al. \"Unraveling neural coding of dynamic natural visual scenes via convolutional recurrent neural networks.\" Patterns 2.10 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)