Brain-inspired $L_p$-Convolution benefits large kernels and aligns better with visual cortex

Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-01 02:49:17

Abstract: Convolutional Neural Networks (CNNs) have profoundly influenced the field of computer vision, drawing significant inspiration from the visual processing mechanisms inherent in the brain. Despite sharing fundamental structural and representational similarities with the biological visual system, differences in local connectivity patterns within CNNs open up an interesting area to explore. In this work, we explore whether integrating biologically observed receptive fields (RFs) can enhance model performance and foster alignment with brain representations. We introduce a novel methodology, termed $L_p$-convolution, which employs the multivariate $p$-generalized normal distribution as an adaptable $L_p$-masks, to reconcile disparities between artificial and biological RFs. $L_p$-masks finds the optimal RFs through task-dependent adaptation of conformation such as distortion, scale, and rotation. This allows $L_p$-convolution to excel in tasks that require flexible RF shapes, including not only square-shaped regular RFs but also horizontal and vertical ones. Furthermore, we demonstrate that $L_p$-convolution with biological RFs significantly enhances the performance of large kernel CNNs possibly by introducing structured sparsity inspired by $p$-generalized normal distribution in convolution. Lastly, we present that neural representations of CNNs align more closely with the visual cortex when $L_p$-convolution is close to biological RFs. This research shines a light on the potential of brain-inspired models that merge insights from neuroscience and machine learning, with the hope of bridging the gap between artificial and biological visual systems.

Main paper: 
 introduction: 
 Convolutional neural networks (CNNs) were initially inspired by the early discoveries from the brain's visual system, particularly the groundbreaking work of Hubel and Wiesel on receptive fields (RFs) in the primary visual cortex (V1) (Hubel & Wiesel, 1962; 1965) . This laid the foundation for the Neocognitron, a precursor to CNNs, which focused on the idea of local connections (Fukushima, 1980) . Building upon the backpropagation (Werbos, 1974; Rumelhart et al., 1986) , and the concept of weight sharing, LeCun and his colleagues applied it to CNNs, leading to the development of the first practical CNN, known as LeNet (LeCun et al., 1989) . Fueled by advancements in backpropagation, large datasets, and GPU training, the landmark success of AlexNet in 2012, which significantly outperformed existing models in the ImageNet challenge, marked a turning point for CNNs (Krizhevsky et al., 2012) .In recent times, CNNs have become a prominent model for understanding biological visual processing due to both structural and representational similarity between them (Lindsay, 2021) . The CNN's architecture closely mirrors the visual system: 1) RGB channels emulate retinal computations, 2) stacked layers correspond to various visual areas (such as V1, V2, V4, IT), 3) deeper layers excel at detecting complex features, and 4) local receptive fields. In terms of representational similarity, Khaligh-Razavi & Kriegeskorte (2014) demonstrated that among 37 different computational models, CNN trained with supervision exhibited the best performance and alignment with the brain. Recently, CNNs like VGG and ResNet have become instrumental models for visual processing models in neuroscience, owing to their strong representational alignment with the brain (Khaligh-Razavi & Kriegeskorte, 2014; Simonyan & Zisserman, 2014; Shi et al., 2019; He et al., 2016; Bakhtiari et al., 2021) .• Introduction of MPND to model the disparity between biological and artificial RFs.• Proposal and implementation of L p -convolution leveraging adaptable L p -masks.• Demonstration of task-dependent conformational adaptability of L p -masks with Sudoku challenge. • Significant performance gains in large kernel CNNs in vision classification tasks.• Improved alignment of neural representations with brain-inspired L p -convolution.Code, datasets, and pre-trained models are available at https://anonymous.4open. science/r/lpconv-E39D. 
bridging biological and artificial receptive fields: 
 The distinct characteristics of dense, uniform local RFs in CNNs, as opposed to the sparse, morphologically structured connectivity patterns in biological neurons, pose a challenge for directly comparing their local RFs. To address this, we analyzed the functional connectivity patterns of both biological and artificial systems, by introducing the MPND (Goodman & Kotz, 1973) . MPND is the key of our paper to bridge the conformational difference between biological and artificial RFs. While the concept of the receptive field generally encompasses sensory-level inputs and adjacent layers, in this paper, we specifically refer to RF as the local connectivity patterns between neurons in adjacent layers.Multivariate p-generalized normal distribution Let s represent the d-dimensional random vectors indicating specific points within RF. s 0 is the receptive center with d-dimensional vector of fixed constants. The relative offset is given by ∆s = s -s 0 . Introducing MPND, we define the RF using as probability density function (PDF) of s as following:EQUATIONwhere C is d × d covariance matrix, ∥ • ∥ p p denotes the L p -norm raised to the p-th power, β is normalization factor1 , and p ≥ 1. In Figure 1e , we show some examples of MPNDs with varying values of p and C.Constructing biological and artificial RFs from the functional synapses To compare the biological and artificial RFs, we first prepared the 2D offsets of functional synapses relative to the post-synaptic units in both systems: mouse V1 L2/3 and AlexNet Conv1 (Fig. 1a-d ; See details in Appendix A.1 and A. 2). To match the scale difference between the two systems, we standardized the relative offsets with zero mean and unit variance. For the artificial system, we prepared 4 different cases, using both ImageNet-1k pre-trained and randomly-initialized AlexNet2 with inputs of noises or images (See inputs and corresponding RFs in Appendix A.4 ). We constructed 2D probability mass functions (PMFs) from the collected offsets, which we call biological or artificial RFs. n=17 for all conditions. We optimized MPND parameters of p and σ, where C = 1/σ 0 0 1/σ , σ init = 0.5, and p init = 2 MPND effectively models both biological and artificial RFs For the comparison of biological and artificial RFs, we optimized parameters of p and σ in MPND (Fig. 1e , Eq. 1) over PMFs of biological or artificial RFs (Fig. 1f and g ). We show that optimized p * of functional synaptic input patterns of biological neurons were optimized at near 2 (Gaussian-distributed; See details in Appendix A.3). In contrast, the local RF pattern of pre-trained AlexNet's Conv1 was optimized at the range of 3.7 ∼ 3.8, and the untrained one was optimized at the range of 11 ∼ 14 (where input types were less effective). An intriguing observation is the decrease in the value of p for the pre-trained AlexNet's Conv1, bringing it closer to the biological RF. Based on these findings, we propose that both biological and artificial RFs can be effectively modeled with MPND, particularly within the range of p = [2, 16] . Given these findings, we propose to consider that the value of p close to 2 is indicative of biological RF, while a higher p represents RFs to be more artificial. L p -convolution Here, we propose the L p -convolution, which is compatible with various convolutions. We formulate the L p -convolution based on the MPND in the convolutional layer by employing channel-wise L p -masks, which are overlaid on convolutional filters (Fig. 2a-c ). 
l: 
 First, we define the relative height and width offsets, ∆S ∈ R 2×K h ×Kw from the kernel center, (K h /2, K w /2), as followsEQUATIONwhere K h and K w denote the kernel height and kernel width, respectively. Empirically, we utilize the normalized value between 0 and 1. It is noted that S •,h,w ∈ R 2 denotes all values corresponding to h-th height and w-th width.Second, we propose the L p -mask, structured mask matrix, derived from the offset and MPND. Our L p -mask, M ∈ [0, 1] Co×K h ×Kw for all output channel C o , is a soft mask that is proportional to Eq. 1 without a normalization factor β as followingEQUATIONwhere C ∈ R Co×2×2 is the set of 2 × 2 covariance matrix for each output channel. In other words, L p -mask calculates the soft mask M o,•,• for each o-th output channel independently, and each soft mask handles the positional correlation, height and width position, from the offset, ∆S. Our L pmask in Eq. 3 corresponds to RF in Eq. 1, whenK h = K W = C o = 1.Third, we propose the L p -convolution by applying L p -mask into the convolutional weights W ∈ R Ci×Co×K h ×Kw , where C i is the number of input channel. For each i-th input channel input X i and convolutional filter weights W i , we formulate the corresponding convolution output Y i asEQUATIONwhere ϕ is non-linear function and * is the convolution operation.We note that C and p are trainable parameters3 . To ensure the positive definite property of C o and satisfy the L p -norm property with p ≥ 1, we can employ Cholesky decomposition for C and value clipping for p.As shown in Equation 4, our L p -convolution is a generalized version of the traditional convolution.In our settings, it is noted that L p mask converges to a binary mask as p approaches infinity. Empirically, L p mask becomes a binary mask for sufficiently large p, as shown in Fig. 1 . If all elements of L p mask equal one, then L p -convolution degrades to the traditional convolution. In this situation, both traditional convolution and our L p convolution have square-shaped RFs for each layer. In other words, our L p -convolution takes task or data-dependent RFs with varying M by optimizing the C and p as shown in Figure 2 . Therefore, L p -convolution has task-specific RFs with varying distortion, scale, and rotation levels.In 2 ). 
conformational adaptability of l p -masks in sudoku challenge: 
 To assess the adaptability and effectiveness of evolving RFs in L p -convolution, we conducted wellestablished 9 × 9 Sudoku-solving task (Park, 2018; Oinar, 2021; Amos & Kolter, 2017; Palm et al., 2018; Wang et al., 2019 ) (See experimental details in Appendix A.5). We selected the Sudoku task for its challenging nature, as it necessitates simultaneously achieving three objectives-ensuring every row, column, and box contains all numbers from 1 to 9-(Fig. 3a ).L p -convolution in Sudoku solving: balancing square and row-column imbalances As shown in Figure 3b , the (3×3) Base model, or (7×7) L † p (p init = 256) model exhibited a significant imbalance in Square-to-Row/Column accuracy and showed signs of overfitting after approximately 15 epochs. The (7×7) Large model demonstrated improvement in the balance of Squareto-Row/Column accuracy and overall Sudoku accuracy, possibly due to the enlarged RFs. Along with these control experiments, we tested two trainable L p -masks with p init = 2, 16, each of them closely resembles a biological RF (p = 2) and an artificial RF in (p = 16), respectively. In (7×7) L p (p init = 2) model, we observed highly balanced Square-to-Row/Column accuracy, resulting in remarkable improvement in overall Sudoku accuracy compared to (7×7) L p (p init = 16) or (7×7) Large model (red and green in Fig. 3b ). We speculated that this alleviation of Square-to-Row/Column accuracy imbalance in (7×7) L p (p init = 2) could be attributed to the task-dependent L p -masks' conformational adaptability. To test this possibility, we have designed ablation experiments on (7×7) L p (p init = 2) model.Ablation of orientation selective masks reveals L p -masks' conformational adaptability Contrary to a previous large-kernel model that introduces unstructured sparsity directly into filters (Liu et al., 2023) , L p -convolution with p = 2 introduces structured sparsity based on a Gaussian distribution. This approach facilitates covariance analysis of the Gaussian distribution, thereby enhancing interpretability. Using Singular Value Decomposition (SVD) on C, we extracted three interpretable properties of scale (α), rotation (θ), and distortion (γ) (See conformational analysis in Appendix A.5) . Figure 4a shows conformations of L p -masks inverse calculated from α, θ, and γ.Quantitative analysis of (7×7) L p (p init = 2) model revealed an increase in scales when layer deepened (Fig. 4b , see visualization in Appendix A.7), with orientations of horizontal (0,180 • ) and vertical (90 • ) directions. This indicates the task-dependent adaptation of L p -masks, which provide flexible and structured RFs in visual processing. To confirm these orientation-selective masks contribute to the balanced Sudoku-solving task, we conducted an ablation test. We classified masks with highly distorted (γ > 3) as orientation-selective masks. Among these, we selectively ablated near 90 • by gradually increasing the range (close in shape with red dashed box in Fig. 4a ) while tracking changes in column and row accuracies. While row and box accuracy exhibited a consistent decrease, column accuracy sharply decreased as the θ range increased (Fig. 4c and Appendix A.8), with this trend was notable in the later layers. Together, these results indicate that the conformational adaptability of L p -masks enables balanced learning in the Sudoku-solving task, thereby contributing to overall performance enhancement. 
l p -convolution benefits large kernel cnns in vision classification task: 
 Next, we asked whether our designed L p -convolution would be beneficial for enhancing the performance of historically successful CNNs. To test this, we conducted vision classification tasks (See detailed experimental settings in Appendix A.9) using the CIFAR-100 and TinyImageNet datasets on models of AlexNet, VGG-16, ResNet-18, ResNet-34, and the ConvNext-tiny (Liu et al., 2022) (Table 1 ). When simply increasing the kernel size, we observed a significant decrease in performance for all models except ResNet. This is consistent with a previous report of a decrease in vision classification performance with larger kernels (Peng et al., 2017) .As depicted in Table 1 , when parameters are frozen with a notably large p (p = 256 and σ = k 2 ), L p -Conv exhibits performance comparable to the baseline model (refer to subfigures c and d in Fig. 2 for graphical illustration). This suggests that initializing the parameters of L p -Conv with a similar experimental configuration can contribute to the stable learning of L p -masks, avoiding significant performance degradation compared to the baseline model. Notably, in contrast to the large kernel models, all instances of L p -Conv, regardless of the value of p init , exhibit stable performance gains, as indicated in Table 1 . Intriguingly, the condition p init = 2 (which is close to biological RFs) recorded the highest performances, ranking first in seven out of ten cases and second in two. This observation suggests that employing L p -convolution to increase the kernel size can enhance the performance of historically successful CNN models as well as the recent models, particularly when p init = 2, which aligns closely with biologically observed RFs. This underscores the potential benefits for CNNs from incorporating brain-inspired structured sparsity and flexible RF adaptability. 
representational similarity analysis between l p -cnns and visual cortex: 
 Figure 5a illustrates our approach to assessing the alignment of representations between biological and artificial systems. We utilized the standardized dataset from Allen Brain Observatory (de Vries et al., 2020), which uses 118 images of Natural Scenes and corresponding neural activities recorded from the mouse visual cortex (VC). Our method builds upon established Representational Similarity Analysis (RSA) techniques (Khaligh-Razavi & Kriegeskorte, 2014; Nguyen et al., 2020; Mehrer et al., 2020; Devereux et al., 2013; Diedrichsen & Kriegeskorte, 2017) , to compare the representations of CNNs (Bakhtiari et al., 2021; Shi et al., 2019) (See details in Appendix A.11) . Based on the observation that L p -convolution tends to perform better as it approaches biologically observed RFs with p init = 2, we posed the question of whether the neural representation of artificial models with biological RFs aligns more closely with the representation of VC. To address this, we compared the neural representations of TinyImageNet-trained CNNs from Table 1 by presenting the 118 Natural Scene images, with the mouse VC representations. To facilitate model comparison, we extracted the representative value, maximum SSM, chosen from pair-wise SSMs across the convolutional layers and the VC subregions (See pair-wise SSMs in Appendix A.12). In the results, models with p init closer to 2 generally exhibited better alignment with the brain (Fig. 5b ). In summary, we find that L p -convolution tends to achieve better alignment with the brain as it approximates a Gaussian distribution. 
related works: 
 Resurgence of large kernels in convolution In the early stages of CNN development, large kernels were not widely adopted, with their use predominantly confined to the initial layers (Krizhevsky et al., 2012; Szegedy et al., 2015; 2017) . Attempts to enlarge kernel size sometimes led to a decline in classification performance (Peng et al., 2017) . Consequently, the more favored strategy was stacking smaller kernels (1x1, 3x3) (Simonyan & Zisserman, 2014; He et al., 2016) .The advent of Vision Transformers (ViTs) marked a paradigm shift from traditional CNN models, with the Swin Transformer emphasizing the significance of both attention mechanisms and large receptive fields, thereby renewing interest in large kernel CNNs (Dosovitskiy et al., 2020; Liu et al., 2021; Touvron et al., 2021; Vaswani et al., 2021; Liu et al., 2021) . Recent innovations such as RepLKNet and SLaK have showcased performance comparable to ViTs, highlighting the potential of large kernel CNNs in modern computer vision (Ding et al., 2022; Liu et al., 2023) . However, the effectiveness of large kernels in historically successful CNN models remains unexplored.In this paper, we implemented large-kernel convolution by applying trainable masks to existing filters and introducing Gaussian-based structured sparsity for adjustments in receptive fields tailored to specific tasks. This methodology contrasts with recent approaches that incorporate unstructured rowrank sparsity directly into filters, which may necessitate extensive hyperparameter tuning. Our train- able masks streamline the optimization process by automatically adjusting key parameters, thereby facilitating the application of large kernel training in both traditional and modern CNN architectures.Gaussian distribution as receptive field of visual information processing In biological systems, computational neuroimaging studies have uncovered that population receptive fields (pRFs) in the human visual system exhibit Gaussian distributions, shedding light on population-level insights (Wandell & Winawer, 2015) . Recent research underscores the predominance of circular pRF shapes in the early visual cortex (Lerma-Usabiaga et al., 2021).In the realm of artificial systems, the concept of effective receptive fields (ERFs) was introduced by Luo et al. (2016) . Their findings indicate that ERFs in CNNs display Gaussian-like properties and are significantly smaller than their theoretical counterparts, thereby providing critical insights into how information is integrated across layers (Luo et al., 2016) .The prevailing understanding is that the Gaussian-like nature of RFs in both biological and artificial systems arises from the cumulative effect of convolutions through stacked layers. However, in our study, we show that the inherently Gaussian-like RFs -not a result of the cumulative stacking of layers -, could be beneficial in visual processing and more accurate in the explanation of the brain's neural representation. 
conclusion: 
 The emergence of CNNs stands as one of the most significant events in the history of artificial intelligence (Krizhevsky et al., 2012) . CNN-based models have led to the development of numerous commercial applications, generating substantial industrial value (Alzubaidi et al., 2021) . Notably, CNNs have not only played a pivotal role in engineering but have also become a major model in the study of visual processing in the brain (Lindsay, 2021) . However, with the recent advent of Vision Transformers, the throne of dominance as a vision model appears to be shifting away from CNNs (Dosovitskiy et al., 2020 ). Yet, the emergence of large-kernel CNNs has prompted a reevaluation of the potential of CNNs Ding et al. (2022) ; Liu et al. (2023) .In this study, we introduced a novel L p -convolution, based on the MPND, with the objective of narrowing the gap between artificial and biological RFs, and subsequently crafting neural network modules more aligned with biological structures. Brain-inspired L p -convolution enables the cultivation of diverse-shaped RFs with Gaussian-based structured sparsity, adaptable to various rotations, distortions, and scales, and tailored for specific tasks. Significantly, L p -convolution showcases its adaptability and compatibility across an extensive spectrum of CNN models, from the conventional to the contemporary, underscoring its proficiency, especially in contexts involving large kernels.We believe our research serves as a noteworthy illustration of the symbiotic relationship between advancements in artificial intelligence and our understanding of neural processes. 
 
 Appendix: A.1 BIOLOGICAL RFS For the biological RF analysis, we have analyzed in vivo intracortical connectivity dataset of Rossi et al. (2020) collected from the mouse primary visual cortex (V1) (Fig. 1a ). It contains both excitatory (CaMK2a-positive) and inhibitory (Gad2-positive) layer 2/3 neuronal spatial connectivity distribution (Fig. 1b ) which was determined by recording GCaMP6 signals (calcium activities) of pre-and post-synaptic pairs (Fig. 7 and 6 ).Given a post-synaptic neuron positioned at (x 0 , y 0 ), (Fig. 1b black), and the N b number of functional synapse positions x n , y n ∈ (-∞, ∞) for n = 1, 2, ..., N b , then relative offsets are defined as (∆x n , ∆y n ) = (x n -x 0 , y n -y 0 ). We summarize functional synapse positions for biological RF as followingEQUATIONA.2 ARTIFICIAL RFS For the artificial RF analysis, we used untrained or pre-trained AlexNet4 with inputs (224 × 224) of 17 images either generated from Gaussian noises or selected among 118 Natural Scenes images datasets (See details in Appendix A.10 and Fig. 7 ). When image inputs were shown to AlexNet, we extracted RFs of the functional synapse from the first convolutional layer (Conv1) (Fig. 1, c and d ).Given the input X ∈ R Ci×H×W and weights parameters for Artificial RFs W ARF ∈ R Ci×K h ×Kw , the post-synaptic unit in the Convolution layer, receives weighted-input Z ∈ R V ×Ci×K h ×Kw , the results of element-wise multiplication between partial input and filters, whereZ v=m * (H-K h +1)+n = X •,m:m+K h -1,n:n+Kw-1 ⊙ W (Fig. 1d black) for 0 ≤ m ≤ H -K h and ≤ n ≤ W -K w .We calculate the weighted input as the convolution operation without summation across width, height and input channel. As a result, weighted input Z has V × C i × K h × K w shape, where V = (H -K h + 1) × (W -K w + 1), and Z v ∈ R Ci×K h ×Kw denotes the v-th element of Z. Here, C i , H, W , K h and K w denote the number of input channels, input height, input width, kernel height and kernel width, respectively. For simplicity, we assume that there is stride one and no zero-padding. For h∈ [0, • • • , K h -1] and w ∈ [0, • • • , K w -1],the relative offsets from the kernel center are defined as followsEQUATIONSince spatial connectivity pattern in the biological synapse is measured by the functional calcium activities and given as coordinates, we applied a similar approach to that of CNN layers. We collected N a functional weighted-inputs (functional synapses) where N a represents the number of cases where each elements of |Z| exceeds a threshold θ. Here, we defined θ as the standard deviation of |Z|5 . This selection process yielded a different set of functional synapses input-dependent manner. We summarize functional synapse positions for artificial RF as following ∆s a = {(v, k, ∆h, ∆w)|Z v,k,h,w > θ},We utilized the extensive 1M-sudoku dataset (Park, 2018) , a resource also utilized in prior works (Amos & Kolter, 2017; Palm et al., 2018; Wang et al., 2019) . Sudoku, a widely popular number puzzle, involves organizing digits in a grid such that each row (1×9), column (9×1), and box (3×3) contains all numbers from 1 to 9. In the Sudoku challenge, where achieving these three objectives simultaneously is essential for complete Sudoku solving, there is an advantage that we can test the applicability and effectiveness of evolving RFs in the L p -convolution. To achieve this, we compared five distinct models of Sudoku CNN: (3×3) Base model, (7×7) Large model, (7×7) L p (p init = 2), (7×7) L p (p init = 16), and finally (7×7) L † p (p init = 256) (frozen p and C model) 2. For numerical stability, we clipped p ≥ 2 during the training of the Sudoku-solving task. The inputs, targets, and model architecture are outlined in Figure 4a . As illustrated, our model comprises repeated Conv2dSame blocks, originally introduced in SudokuCNN (Oinar, 2021) . Each Conv2dSame block encompasses Reflection padding, followed by a conventional Convolutional or L p -Convolutional layer, Batch Normalization, and an activation function. The Convolutional layer has 256 channels, and the number of blocks is set at L = 10.We defined the properties of scale (α), rotation (θ), and distortion (γ) through Singular Value Decomposition (SVD) on C, covariance matrix for each output channel, as shown in the following equation:EQUATIONHere, U and V represent 2 × 2 unitary matrices containing the left and right singular vectors. Λ is a diagonal matrix containing the singular values (λ 0 , λ 1 ). Rotation is quantified as We conducted our training on two datasets: CIFAR-100 (Krizhevsky et al., 2009) and TinyIma-geNet (Le & Mikolov, 2014) . CIFAR-100 comprises 32 × 32 pixel images distributed across 100 classes, while TinyImageNet consists of 224×224 pixel images spanning 200 categories. Following standard procedures, we reported top-1 accuracy with corresponding mean and standard deviation.EQUATIONOur implementation is based on the PyTorch framework (Paszke et al., 2019) , making extensive use of the timm repository (Wightman, 2019) . We adopted a training strategy rooted in DeiT (Touvron et al., 2021) , incorporating techniques such as RandAugment (Cubuk et al.), Mixup (Zhang et al., 2017) , Cutmix (Yun et al., 2019) , random erasing (Zhong et al., 2020) , and stochastic depth (Huang et al., 2016) . The optimization process employed AdamW (Loshchilov & Hutter, 2017) with a default momentum value of 0.9 and a weight decay set at 5 × 10 -2 . We initialized our learning rate at 1 × 10 -3 and implemented a cosine learning rate schedule. All models underwent training for 300 epochs, utilizing a batch size of 128. For CIFAR-100, training was conducted on 2 GTX 1080ti GPUs, while 2 Tesla V100 GPUs were used for TinyImageNet.The Allen Brain Observatory dataset (de Vries et al., 2020) constitutes a comprehensive standardized in vivo examination of physiological activity within the mouse visual cortex. It encompasses recordings of visually-induced calcium responses from neurons expressing GCaMP6f. This dataset encompasses cortical activity from nearly 60,000 neurons originating from six distinct visual areas, four layers, and twelve transgenic mouse Cre lines. These recordings were gathered from 243 adult mice in reaction to a diverse set of visual stimuli. In this study, we focused on utilizing the collective neural responses from five visual areas (VISal, VISam, VISl, VISp, VISpm), Layer 2/3 (depth range of 175mm to 275mm), and three mouse lines (Cux2-CreERT, Emx1-IRES-Cre, Slc17a7-IRES2-Cre) when presenting a dataset of natural scenes to the mice. This dataset comprised 118 natural images obtained from three different databases (Berkeley Segmentation Dataset (Martin et al., 2001) , van Hateren Natural Image Dataset (Van Hateren & van der Schaaf, 1998), and McGill Calibrated Colour Image Database (Olmos & Kingdom, 2004) ). Further details regarding the experiment can be found in (de Vries et al., 2020) . In our study, we employed both images and neural responses for experiments involving representational similarity analysis to evaluate the correspondence between CNNs and the visual cortex, mirroring earlier investigations (Shi et al., 2019; Bakhtiari et al., 2021) .While the details of RSA are expertly addressed in Diedrichsen & Kriegeskorte (2017) , let us briefly cover our specific approach. We leveraged the codebase provided by Bakhtiari et al. (2021) . In RSA, we generate response matrices (R) for brain regions and neural network layers, with dimensions N × M (where N is the number of image inputs and M is the neuron count). Using Pearson correlation, we compute similarities within matrix R to construct the N × N Representation Similarity Matrix (RSM). Additionally, following the methodology of Bakhtiari et al. (2021) , we applied noise correction by normalizing the RSAs using the noise ceiling values. These values were obtained through comparisons of representations across different mice. For example, the noise ceiling value for VISp is derived by calculating the RSMs of VISp from different animals and taking their median.To assess the similarity between RSMs (SSM), we employ Kendall's τ for robust agreement, which helps mitigate potential bias from measurement noise Diedrichsen et al. (2020) . β = [(2Γ(1 + p -1 )) d • det(C)] -1where Γ is gamma function, and det(•) denotes the determinant.For clarity, we refer to trained as pre-trained models and to untrained as randomly-initialized models.C and p are updated with the standard backpropagation process. Lp-mask, M, is dynamically generated during forward process using C and p.For the pre-trained model, we used the torchvision's ImageNet-1k pre-trained modelWe determined the activity threshold based on a common method used in neuroscience to extract meaningful patterns in neural activity, which is similar to calculating the Z-score and typically setting a threshold at a range of 2 to 3 standard deviations to identify values that are statistically significant. 



Reviews: review: title: title: Exciting and New Convolution Idea , but orthogonal experiments
main_review: main_review: I think this paper should be presented at the ReAlign workshop given the amount of work the authors have put into it. However my main concern is at a higher-level I don't think that $L_p$ convolutions have really driven the point home.

Experiment 1) Fitting to Biological data. I can't really see this in Figure 1, and it would be better if the assessment here was quantitative and not qualitative. I am only really convinced by the Mouse V1 L2/2 data with a proper $L_p$/MPND kernel fit, but I can not say the same about the other ones.

Experiment 2) Sudoku Challenge. This is interesting but authors are benchmarking against themselves, and I think the proper control here should be no L_p vs L_p included across networks with matching number of parameters -- though it seems that this is shown in the 7x7 plot vs 7x7 with varying values of p with p=2 being the best and close to the baseline model. Overall, I was confused to see why this benchmark in particular was picked to showcase the use of $L_p$ convolutions.

Experiment 3) ablation, I am not sure what authors were trying to show here.

Experiment 4) RSA experiments, seems like $p=2$ is a winner, but the t-test is done against the same model varying values of $p$ so while this results is encouraging I'm not sure if this is the right control, or perhaps another one like pre-fixed receptive fields like the ones of Dapello, Marques et al. NeurIPS 2020.

In the Computer Vision benchmark, I have the same set of comments. I overall think that perhaps another set of experiments should have been picked to show-case why $L_p$ is useful in the first place. My main take-home from the paper from what I have read so far is that $L_p$ are interesting, however I would have liked to see a stronger message like the following (that will of course require another set of experiments):

* Systems trained with $L_p$ convolutions are more adversarially robust than systems that do not have them.
* Systems trained with $L_p$ convolutions can render/synthesize visual stimuli with little to no aliasing and/or better Frechet Inception Distance (FID) than systems that do not have them.
* Systems trained with $L_p$ convolutions require less training data than systems trained without them.
* Systems trained with $L_p$ convolutions achieve higher performance in task [XYZ] vs systems that do not have them.
* RSA analysis of systems trained with and without $L_p$ convolutions show that systems with $L_p$ convolutions have higher RSA score than systems without it.

All-in-all perhaps only focusing on one or two of these experiments would be ideal, rather then 4 or 5.
strength_weakness: strength_weakness: The amount of work that authors have given in this paper is exceptional. I still think however that the experiment shown do not drive the point home.

I would still like this paper to be presented at the workshop. I think authors could get a lot of great feedback from attendees.
title: title: Promising work suggesting that brain-inspired changes to CNN receptive fields improves model performance and brain-alignment.
main_review: main_review: Large kernels with Lp-masks show improved performance on vision tasks and higher brain alignment, particularly when p=2 (which is most closely aligned with biological receptive fields). While I believe these are promising results, I think the paper could be improved by clarifying the method, motivating the choice of metrics (you don’t see Sudoku, TinyImageNet classification, and Mouse visual cortex productivity sandwiched together every day), and expanding on the neural alignment component (additional analyses of these brain data, perhaps application to other brain data).

I might have misunderstood the methods, but I found it a bit difficult to follow. Is it the case that the method is to introduce large kernels with weight masks? If so, that seems straightforward, and could be clarified in the introduction. When the masks are initialized and frozen, are they sampled from a distribution or is there a fixed exhaustive set?

Why the Sudoku task?

Classification results limited to CIFAR-100 and TinyImageNet, do they hold for larger scale models (e.g., Imagenet trained)?

Why the mouse visual cortex? Why not submit these models to BrainScore?

What is the noise ceiling on the neural data? How well do untrained models, or standard CNNs trained on TinyImageNet vs. full ImageNet do?

Does the result hold if you use different alignment procedures (e.g., allowing feature re-weighting)?

I might have misunderstood the setup, but it was unclear to me why the inductive bias of the Lp mask was needed, since the model could in theory emulate this by adjusting the weights, or am I missing something critical about the implementation? It would be helpful if the authors could provide some intuition for why the masks are needed/beneficial.
strength_weakness: strength_weakness: Creative bio-inspired computer vision improves brain-model alignment, promising if brain-alignment advantage holds up when compared against untrained or standard models, and other (human, non-human primate) brain data.
title: title: Highly relevant topic, expansive scope; clarity could have been improved
main_review: main_review: This paper asks whether integrating biologically-inspired receptive field (RF) structure into CNNs can improve their task performance and brain alignment. The authors introduce a novel method called Lp-convolution, which uses the multivariate p-generalized normal distribution to instantiate learnable Lp-masks that specify the model's RFs. If I understand correctly, this enables the convolution process to accommodate different distortions, scales, and rotations of the connectivity between one layer and the next. Particular settings of the distributional parameters are considered more "brain-like" than others (via comparison to mouse visual cortex).

The authors present a series of results in favor of the utility of Lp-Convolution, showing: (i) faster learning and better performance on Sudoku (presumably because effective models of that task benefit from elongated vertical and horizontal receptive fields); (ii) improved accuracy on CIFAR-100 and Tiny ImageNet (presumably because the structured RFs provide some form of beneficial regularization); and (iii) that TinyImageNet-trained models with a Gaussian-like RF structure (more closely mimicking that of mouse visual cortex) tend to have higher brain alignment than those with less brain-like RF structure. 

These results suggest that incorporating biologically inspired RFs could be a rich, promising direction for future exploration in both computer vision and neuroscience.

- Strengths
	- This is a fairly impressive scope for a workshop paper. The authors attempt to introduce a new architectural motif, provide fairly extensive visualizations to help the reader understand the method, and attempt to separately validate the method both in the context of computer vision tasks and in analysis of brain alignment.
	- This work addresses an extremely relevant gap between human and machine visual systems (RF properties), that has received relatively little attention so far. Re-Align definitely seems like an ideal venue for this work. 
	- Lp-Conv seems to be a fairly intuitive idea and (if i'm understanding correctly) may be of interest both to folks that primarily work on both neuro and on AI.
- Weaknesses
	- The paper is fairly dense. The logic could be significantly streamlined so that the average reader would have an easier time digesting the (many) methods and results. I feel that too much space in the Introduction (and Discussion) was dedicated to describing the history of CNNs, rather than previewing (and reviewing) the full scope of the paper, the logic of the analyses, and the main findings. 
	- Along these lines, my understanding of the work (and its implications) was limited by the sprawling nature of the paper, the heavy-handed use of symbols and notation, and the existence of 21 appendices. I appreciate the authors' attempt to be diligent and thorough, but these factors made it hard to zero in on the specific ideas and results that were most relevant.  
	- As a result, I feel that I understand the methods and results at a surface level, but that several more read-throughs would be required to really feel confident in my evaluation of the work...
	- Among my specific questions and confusions are: 
		- The use of weight sharing/tied weights in CNNs seems critically relevant to this work, but the connection was not made in a clear way.
		- The Sudoku analyss came a bit out of nowhere, and could have been more strongly set up / motivated earlier in the paper rather than waiting for Section 4.
		- The RSA analyses appear to be missing a pretty important comparison to baseline models trained without Lp-convs... this would really help neuroAI readers understand the utility of this method.
		- The benefits and limitations of models with large kernels could have been spelled out more clearly.


Despite these limitations, because this work is so clearly relevant for the Re-Align community, and because the results are provocative, I am fairly convinced that this work should be accepted for a poster presentation. I am sure the authors and community would mutually benefit from being able to discuss the work.
strength_weakness: strength_weakness: The approach seems well motivated, certainly novel, and potentially of interest in both neuroscience and AI. I think this is a really important question to be working on.