Enriching ConvNets with pre-cortical processing enhances alignment with human brain responses

Authors: niklas m üller
h steven scholte
iris i a groen
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-08 18:08:35

Abstract: Convolutional Neural Networks (ConvNets) are the current state-of-the-art for
modelling human visual processing whilst also performing tasks on a human per-
formance level. Convolutional features can be seen as analogous to visual recep-
tive fields and thus render them biologically plausible. However, spatially-uniform
sampling and reuse of features across the entire visual field do not accurately rep-
resent structural properties of the human visual system. Here, we present empir-
ical findings of incorporating functional and structural properties of the human
retina into ConvNets on their alignment with human brain activity. We show
that predictions of human EEG data using ConvNets features improve by using
foveated stimuli and that differential spatial sampling in ConvNets explains sev-
eral qualities of EEG recordings. We also find that color and contrast filtering of
inputs in turn do not yield improved predictions. Overall, our results suggest that
incorporating biologically plausible spatial sampling is important for increasing
representational alignment between ConvNets and humans.

Main paper: 
 introduction: 
 Deep neural networks are currently state of the art predictive models of human brain activity and behaviour (Güc ¸lü & van Gerven, 2015; Khaligh-Razavi & Kriegeskorte, 2014) . The kernels of a trained convolutional neural network (ConvNet) resemble parts of the computational and representational principles of both early visual cortex, i.e. simple and complex cells in V1, and higher-level visual cortex, in terms of object representations (Lindsay, 2021) . However, the performances in predicting human neuroimaging data using features of ConvNets are still below the theoretical maximum of explainable variance (Cichy et al., 2016; Gifford et al., 2022; Huang et al., 2017; Wen et al., 2018) , indicating potential to improve their alignment with humans.In this paper, we argue that certain, built-in features of ConvNets are limiting their accuracy and plausibility as models to study the neural mechanisms underlying early human visual processing. One limitation comes from the spatially uniform input sampling. This architectural feature led to a tremendous success of ConvNets in solving computer vision tasks (Krizhevsky et al., 2012; He et al., 2016; Milletari et al., 2016; Mascarenhas & Agarwal, 2021; Kriegeskorte, 2015; VanRullen, 2017) as reusing convolutional features across the entire visual field reduced training parameters. While this decreased computational resources, it also introduced a gap between the physiology of ConvNets and that of the human retina.The human retina is structurally divided into areas with a high density of retinal ganglion cells (RGCs) at the center of gaze called fovea, and areas with a lower density of RGCs, referred to as periphery (Oyster et al., 1981; Wässle et al., 1989; 1990; Kreiman, 2021) . This distinction is relevant in the context of object recognition. To perceive small object details, humans move their center of gaze, i.e., the fovea, in order to sample visual information about the object with high resolution. Peripheral vision is important in situations when a lot of new visual information needs to be processed, such as turning around to a new scene. While not having high acuity, the periphery is optimized to process the essence of vast amounts of information, i.e., the gist of a scene, at a high temporal resolution (Biederman et al., 1974; Potter, 1975; 1976; Thorpe et al., 1996; Greene & Oliva, 2008; Sharan et al., 2009) .While differential sampling between the fovea and the periphery is theoretically implementable in ConvNets, the current strategy of ConvNets of uniformly sampling the input by reusing features solves tasks like object recognition a lot more efficiently and can lead to good predictions of higher visual cortex (Yamins et al., 2014) . However, enhanced task efficiency does not necessarily reflect enhanced brain alignment (Konkle et al., 2022; Kubilius et al., 2019) , emphasizing that the brain solves many more tasks under strong energy and anatomy constraints that ConvNets are not subjected to (Dupont et al., 1993; Orban et al., 1996; DiCarlo et al., 2012) . The brain therefore needs to carefully select the right amount of information in an efficient manner, e.g., through differential spatial sampling. We hypothesize that augmenting ConvNets with differential spatial sampling will increase their representational similarity with humans.The retina is converting photoreceptor signals to a color-opponent signal (Koenderink et al., 1972) that is different from the RGB (red-green-blue) coded input of ConvNets. Adding a color-opponent transformation to the ConvNet input should therefore enhance their biological plausibility and increase neural prediction performance. Furthermore, it is well-known that structures involved in visual processing between the retina (beyond bipolar cells) and the primary visual cortex, such as RGCs (Easter Jr, 1968) or the lateral geniculate nucleus (LGN) (Bonin et al., 2003) , process information in a non-linear fashion (Hennig et al., 2002; Macleod et al., 1992; Benardete & Kaplan, 1997) . Thus, we add a pre-filter that applies a non-linear contrast-based edgemap-algorithm.Concretely, we add six pre-filters to a ConvNet that have different levels of biological plausibility. We train these augmented ConvNets on object classification on high resolution images containing a single object. Object images are cropped from high resolution street scene images from a new dataset using manually annotated bounding boxes. We collect human EEG data during visual presentation of these high resolution scene images and evaluate how well the learned ConvNet features are able to predict differences in event-related-potentials (ERPs) across images. We use cross-validated linearized encoding models to map the convolutional features onto the measured EEG data and use the prediction performances as an indicator for the most successful pre-filter. We present empirical findings for ConvNets augmented with a variety of pre-filter and hypothesize that the degree of biological plausibility will predict the neural prediction performance. Furthermore, leveraging the high temporal resolution of EEG recordings, we are able to investigate differences in temporal processing of visual stimuli. The approach of applying a biologically plausible pre-filter to the ConvNet aims at changing the geometry of the learned representational space and thereby at increasing the representational similarity between humans and ConvNets (Sucholutsky et al., 2023) . 
related work: 
 Augmented ConvNets The approach of augmenting ConvNets with static filters that are not subject to the training data has already led to positive results in increasing representational alignment between humans and ConvNets. ConvNets trained for object recognition augmented with a V1-like layer explain more variance in the primate V1 while simultaneously being more robust to adversarial attacks and other image perturbations (Dapello et al., 2020) . This showcases that static, i.e., non-learned filters can enhance the neural prediction performance of ConvNets significantly.The fact that foveal input has higher information density than peripheral input is reflected in the projections of these retinal regions to the (primary) visual cortex. This effect of cortical magnification, where foveal regions are represented in disproportionately more cortical volume than peripheral regions has been known for decades (Cowey & Rolls, 1974; Levi et al., 1985) . Wang et al. (2021) found that this effect simulated by spatial adaptive gaussian blur can be used to enhance biological plausibility of the training of ConvNets while maintaining task performance. Related work by Deza & Konkle (2020) shows that ConvNets with spatial adaptive foveation display increased robustness against image perturbations while maintaining task performance even on compressed information. Together, these findings should be taken as evidence that the highly optimized human retina performs principled computations leading to a robust visual percept that can be adapted for use in computational models. Moreover, da Costa et al. (2023) propose a method of foveation that transforms regular images into irregularly-spaced images according to the distribution of retinal ganglion cells in the primate retina guided by empirical findings. This method of Ganglion Cell Sampling (GCS) leads to a hierarchy of receptive field properties measured using conventional RF mapping techniques (Dumoulin & Wandell, 2008) that is similar to that of the human visual hierarchy measured using fMRI.Peripheral Vision and Scene Perception The above findings motivate the use of foveation as a method of increasing representational alignment. However, prior studies on human visual perception indicate that peripheral visual processing also plays an important role, especially for scene perception (Rosenholtz et al., 2012 ), gist processing (Larson & Loschky, 2009) , face perception (Park et al., 2023) , and even for providing context (Wu et al., 2018; Aldegheri et al., 2023) in object detection, segmentation, or transformation. Wu et al. (2018) show that peripheral information can help improve object recognition performance and robustness to noise in ConvNets. Furthermore, Groen et al. (2013) showed that image-computable scene statistics based on simple pre-cortical filtered inputs (including COC and edgemaps, but also other filters (Scholte et al., 2009; Groen et al., 2012) ) are predictive of human EEG data elicited by visual stimulation and allow to disambiguate different models in terms of their predictive power. 
data collection: 
 Subjects Human EEG data was collected from 31 participants. The institution's Ethical Committee approved the experiment and all participants gave written informed consent before participation and were rewarded with study credits. Four subjects were excluded from analysis because their recordings were incomplete.Stimuli / Image dataset A new dataset of 6130 ultra-high resolution outdoor scenes was used for both human data collection and ConvNet training. Images contain diverse street scenes, have an original resolution of 5468x3672 pixels, and are stored in an uncompressed format. Each scene was manually annotated with object bounding boxes for a set of 16 common outdoor object classes (see Appendix Table 1 ). Additionally, a set of 27 indoor scene images were taken to serve as target images in the human experiment.Experimental Procedure Subjects were presented with outdoor scene images described above downsampled to a resolution of 2155x1440 pixels using a rapid serial visual presentation (RSVP) paradigm (Intraub, 1981; Keysers et al., 2001; Grootswagers et al., 2019) . In total, the dataset consisted of 4800 images of which every subject was presented with a subset of 702 images. 1/3 of all images were repeated 10 times throughout the entire experiment while the other 2/3 were repeated 5 times. Resulting EEG data will be averaged across repetitions per unique images to increase the Signal-To-Noise-Ratio (SNR). As commonly done (Gifford et al., 2022) , averaging across 5 repetitions yields a low-SNR train dataset to fit the encoding model while averaging across 10 repetitions yields a high-SNR test dataset. Stimuli were shown on a 24-inch monitor with a resolution of 2560x1440 pixels and a refresh rate of 120 Hz. Subjects were seated 63.5 cm from monitor and were using a head rest such that stimulus spanned 50x29.5 degrees of visual angle.Each subject performed 240 trials. In every trial a series of 20 images of either only outdoor scenes (non-target trial) or 19 outdoor scenes and exactly one indoor scene (target trial) was presented.Participants were asked to perform a target detection task by pressing either 'Y' for a target trial or 'N' for a non-target trial, at the end of the trial. In each trial, subjects were first presented with 3500 ms gray screen and a red fixation cross in the center of the screen. Next, 20 stimulus images were presented for 100 ms and a 300 ms gray screen each. The fixation cross was presented continuously. At the end of the trial, subjects had 3500 ms to respond with a button press and were instructed to blink before the start of the next trial. The session was divided into 8 blocks each followed by an enforced break of at least 30 seconds with self-paced continuation. Stimuli were presented and behavioural data collected using PsychoPy (Peirce et al., 2019) .EEG Acquisition and Preprocessing EEG data was collected using a Biosemi 64-channel Ac-tiveTwo EEG system (Biosemi Instrumentation) with an extended 10-20 layout, modified with two additional occipital electrodes (I1 and I2, while removing electrodes F5 and F6). Eye movements were monitored with electro-oculograms (EOGs) and using an EyeLink100 desktop-mounted system. EEG recording was followed by offline referencing to external electrodes placed on the earlobes. Resulting EEG data was pre-processed using the MNE software package (Gramfort et al., 2013) as follows: high-pass filter at 0.1 Hz (12 dB/octave) and a low-pass filter at 30.0 Hz (24 dB/octave) followed by two notch filters at 50 and 60 Hz; automatic removal of deflections > 300mV ; epoch segmentation in -100 to 400 ms from stimulus onset; occular correction using the EOG electrodes (Gratton et al., 1983) ; baseline correction between -100 and 0 ms; and conversion to Current Source Density responses (Perrin et al., 1989) . Resulting ERP data was averaged across repetitions per unique image, thus resulting in an sERP specific to each subject, electrode and image.Model Training We trained five alexnet instances (Krizhevsky et al., 2012) for 30 epochs on object classification using object crops as outlined above. Objects were resized to 400x400 pixels, and the model was trained to assign one of 16 classes. The dataset consists of 82.236 objects which are split into non-overlapping sets for training (80%), validation (10%), and testing (10%).Each model instance is augmented with one pre-filter (see 3.2.1). Importantly, all pre-filters are static, i.e., no parameters depend on the training context. We report the top-1 accuracy of each model on the object crop test set. All models were trained and tested using PyTorch 1.11.0 on a NVIDIA A100 using NVIDIA CUDA 11.3 with cuDNN v8.2.0 support.Full-Scene In the default image condition, models are trained on RGB object crops. During feature extraction full-scene images are used, i.e., images are resized to a resolution of 2155x1440 pixels without cropping. Images are encoded using RGB-encoding.Center-Crops In the center-crop condition, models are trained on the default RGB object crops. During feature extraction the center-crop filter crops out the central part of the 2155x1440 full-scene images. Here, we included crops of sizes 400x400, 224x224, 112x112, and 64x64 pixels.Ganglion Cell Sampling We use the Ganglion Cell Sampling (GCS) technique from da Costa et al. ( 2023) that transforms an image, such that the central part of the image is magnified and magnification decreases non-linearly as a function of increasing eccentricity according to an empirically derived distribution (Watson, 2014) of retinal ganglion cells in the primate retina. We use the default settings reported in table 1 in da Costa et al. ( 2023) and apply the transform to object crops during training and to full-scenes during feature extraction.Color Opponent Channels As an alternative color coding to RGB, we use color opponent channels (COC). An RGB-coded image is converted into: intensity , Blue-Yellow-opponent , and the Red-Green-opponent (Koenderink et al., 1972) . The COC-transform is applied both during training on object crops and during feature extraction on the full-scene images.Edgemaps We use an edgemap-algorithm using multiscale contrast magnitude maps from Groen et al. ( 2013). These edgemaps are originally computed on COC-transformed images but can, however, also be computed directly on the RGB-images. Thus, we treat the COC-transform and the Edgemap-transform as two orthogonal computation, that can be combined or not. The RGB or COC input is first rectified and then normalized. Further, we compute contrast magnitude maps by convolving the image with a static bank of multiscale filters of different sizes (Zhu & Mumford, 1997; Croner & Kaplan, 1995; Ghebreab et al., 2009; Bonin et al., 2005) . Subsequently, minimum reliable scale selection is applied (Elder & Zucker, 1998) . The edgemap-algorithm is applied both during training on the object crops and during feature extraction on the full-scene images.Encoding Model We built linearized encoding models to map convolutional features onto the ERP amplitude for each subject, electrode and time point. We used the same pre-filters as applied during training to extract features for the scene stimuli also used during the human experiment. We used the pre-trained features from all max pooling layers (features.2 referred to as 'layer1', features. For every model instance, we flattened the extracted features per stimulus in the low-SNR set across all layers and performed a principle component analysis (PCA) across these features. The projections per stimulus of the first 100 PCs then formed the design matrix. The ERP amplitudes for every subject, electrode, and time point was then regressed onto the design matrix. On the extracted features of the stimuli of the high-SNR test set, we applied the same PC projections and then used the fitted encoding model to predict the ERP amplitude for every subject, electrode, and time point. We then quantified each model's predictive performance by calculating the cross-validated Pearson correlation coefficient between predicted and real ERP amplitude.For every ConvNet instance we built four more specialized encoding models. For each specialized encoding model we flattened the extracted features from one of the four layers per stimulus in the low-SNR set. The encoding model is then identically built and cross-validated as described in 3.2.2.We calculated the noise ceiling, i.e., theoretical maximum of explainable variance of the ERP amplitude across stimuli, per subject, electrode and time point. We split the ERP signal across ten repetitions for the high-SNR set into two non-overlapping splits of 5 repetitions each. For each subject, electrode and time point, we estimated the lower bound of the noise ceiling as the correlation between the first split of 5 repetitions and the second split of 5 repetitions and the upper bound as the correlation between the first split of 5 repetitions and all 10 repetitions. 
spatial sampling: 
 Foveal vs. Peripheral Sampling Figure 2 shows the average prediction performance across subjects for two posterior visual electrodes using the general encoding model for the RGB model using different spatial sampling. We perform t-tests on pairs of conditions and show the t-values across the whole scalp for selected time points for significant fdr-corrected p-values with alpha = 0.05. First, we tested for a difference between the predictions of an RGB-trained alexnet on full-scenes vs. using 400x400 pixel center-crops. We include topoplots for three time points: 95 ms, 134 ms, and 192 ms. We find that the full-scene model yields significantly better predictions at 95 ms than the center-crop model thus predicting ERP data better, earlier in time.Crucially, at the time point of peak performance (134 ms) the center-crop model (r = 0.56) outperforms the full-scene model (r = 0.46, with t = 8.63, p < 0.0001 for α < 0.05 ) This advantage persists even when cropping to 224x224 pixels (shown in line plots; see appendix Fig. 5 for crops at 112x112 and 64x64 pixels). Any differences between full-scene and center-crop have disappeared at 192 ms and later. Predictions of the same model using randomly centered 400x400 crops (purple line) are significantly worse than all other models post-stimulus (max r = 0.14). However, valid predictions can still be found around 134 ms as random crops can overlap with central parts. This comparison shows that drastic differential sampling between foveal and peripheral input to a ConvNet can yield qualitatively different predictions. Peripheral sampling leads to accurate early predictions while foveal sampling yields later but even more accurate predictions.Retinal Spatial Sampling Above, we showed that including visual features from either the foveal/central part of the image or the periphery helps explain distinct properties of ERP signals elicited by high resolution scene images. Next, we asked whether foveal and peripheral features can be combined more optimally by taking into account the sampling determined by the density distribution of RGCs in the human retina that dictates a non-uniform spatial sampling.We tested for differences between the center-crop model and the same RGB-trained model tested on GCS-transformed images (Fig. 2 second topoplot row). The GCS model outperformed the centercrop model both early (95 ms) and late (192 ms) in time, but not in between (134 ms). Further, the GCS model outperforms the full-scene model at 134 ms, showing a higher maximum performance (r = 0.57, Fig. 2 third topoplot row). Thus, the GCS model simultaneously explains ERP data at early time points as well as yielding maximum prediction results at all later time points.Retinal Sampling at Train or Test Time So far, we showed that applying the GCS-transform during feature extraction improves prediction performance. This result is thus an effect of a transformation of the input space but not the representational space. Hence, the resulting improvements stems from a input-dependent weighting of visual features and not a training-dependent change of convolutional features. We thus asked, whether applying the GCS-transform already during training would improve predictions even more. We find that indeed, the prediction performance increases for the GCS-trained model for both early (95 ms) and intermediate (134 ms) time points, but not later (see Fig. 2 last topoplot row). FDR-corrected t-tests reveal a significant improvement especially at the intermediate time point for posterior electrodes.We show the results for the specialized encoding models for layer1 and layer2 in Fig. 3 for models using different color coding. The specialized encoding model for layer1 shows that the RGB model (max r = 0.45) outperforms the COC model (max r = 0.31) at every time point and most strongly in at posterior electrodes (t = 10.22 and p < 0.0001 for Iz at 134 ms). Moreover, the COC-Edges model significantly outperforms the RGB model at 134 ms at electrodes Iz (r = 0.50) and Figure 4 : (left) Top-1 accuracy on held-out test set for ConvNets trained using different pre-filter (indicated in colors). (right) Prediction performances for specialized encoding models averaged across all subjects, all 19 posterior electrodes and all time points between 50 ms and 300 ms after stimulus onset. Colors indicate the pre-filter that the ConvNet was trained and tested with (except "GCS": trained on RGB, tested on GCS). Prediction performance decreases for later ConvNet layers and GCS. Advantages of RGB and COC are layer-dependent while Edgemap-trained models yield robust prediction for every layer.Oz (r = 0.52). In turn, the specialized encoding model for layer2 shows that the RGB model is outperformed by all other models. Not only achieve the COC, the RGB-Edges and the COC-Edges models a higher maximum performance but also yield better predictions at early time points.Using the general encoding model (see appendix Fig. 6 ), we find the same pattern of results as for layer1. Further, we find no differences between RGB model and RGB-Edges or COC-Edges model at all time points. However, both the RGB-Edges model and the COC-Edges model outperform the COC model at around 130 ms (max r = 0.45 for RGB-Edges model and r = 0.47 for COC-Edges model for Iz at t = 0.1344 s). Overall, we see a clear disadvantage of COC model features for predicting EEG response. The RGB-Edges model performs equally well as the RGB model, showing that the Edgemap-algorithm does not yield better model features compared to RGB. However, the COC-Edges model shows that the Edgemap-algorithm can yield features that recover the prediction performance when compared to the COC model.To conclude, the RGB model is in most cases the best performing model, together with the edgemap models. For very specific and localized cases we see robust and significant improvements of especially the COC-Edges model using the layer-wise encoding model.Classification performance of ConvNets decreases with increasing biological plausibility of added pre-filters (see Fig. 4 left) . Taking the RGB model as a baseline for (blue dashed line) a drop in test accuracy can be observed for both edges models but mainly for the GCS-trained model. Moreover, applying GCS to an RGB-trained model results in a drastic drop of test accuracy resulting from the distribution shift of the input image.Furthermore, the specialized encoding models show that later layers generally lead to a decreased ERP amplitude prediction performance than earlier layers (see Fig. 4 right ). These layers are commonly associated with a stronger tuning for abstract object classes more than low-level visual features. We therefore see a mismatch between suitability of model features for object classification and for predicting EEG responses. 
discussion: 
 In this paper, we evaluated methods to increase the representational alignment of brain activity (human EEG) and computational models (ConvNets). ConvNets trained on object recognition are often compared to the human ventral visual stream which is thought to be highly involved in representing objects. However, the comparison assumes that visual processing starts with the cortex, while in biological organisms, substantial processing occurs in peripheral sensory organs and pathways, in this case the eye and the thalamus. Here, we attempted to strengthen the plausibility of using ConvNets as models of human visual processing by augmenting ConvNets with pre-filters that are known to be computed in the retina and the LGN. We find that differential spatial sampling as an augmentation to ConvNets improves their predictive power while alternative color coding and contrast filters show no or only minimal improvements for EEG predictions.The human fovea is known to have an effective resolution that is orders of magnitudes larger than that of the periphery. ConvNets sample their input spatially uniform and thus do not approximate the differential sampling of the human retina. The most drastic way of modelling this differential sampling that favours information from the center is to remove peripheral visual information. By using only the a small central part of the input to the ConvNet we show that a large part of the variance in the ERP signal is elicited by foveal stimulation, as randomly taken crops have much lower predictive power. This effect is strongest for very posterior electrodes and highlights the amount of processing that is happening in the primary visual cortex on foveal visual information (cortical magnification). However, the extent of cortical magnification could be overestimated based on these results as a large part of the visual cortex is located in the cerebral fissure and might therefore not contribute to the EEG signal.Crucially, the high temporal resolution of EEG recordings enables us to distinguish between early and late time windows of ERP responses. We find that ConvNets processing full-scenes can predict earlier parts of the ERPs that are associated with the fast gist processing that is largely dependent on peripheral information. These early predictions are substantially worse for the center-crop model.Furthermore, the GCS model can account for both, early responses associated with gist processing and for peak prediction performance later in time. Thus, the amount and weighting of information resulting from the GCS-transform seems to be optimal for predicting ERPs elicited by natural scenes.The spatial invariance of ConvNets might pose a crucial limitation to how similar ConvNets can be to the human visual cortex and how much variance of EEG data they can explain. Here, we see that altering the spatial distribution of the input to a ConvNets improves predictions. We find additional positive effect of including the GCS-transform during training implying a change of the underlying representational geometry across all layers that is beneficial for predicting ERP responses.Object Classification Performance All models shown in this paper were trained on object classification. Thus, features are not optimized to predict ERP amplitudes and increased prediction performance using the linearized encoding model is a byproduct of the used filters. We see a trend of decreasing object classification accuracy for models using pre-filters with increasing biological plausibility. However, any model that predicts the brain better might also be of risk of showing a decrease in task performance for any single task but might exhibit a broader generalizability. GCS-trained models show the strongest deficit for object classification. Object crops for training are taken from the full scene by using the smallest rectangle that covers the whole object, often resulting in tightly cropped images. Applying the GCS-transform essentially reduces the information density in the periphery thereby removing important object-related information. Thus, a drop in performance, alongside with increased reliance on local features should be expected. A more optimal way of applying the GCS-transform would be to use it directly on the full-scene (or a larger part of the scene) and thereby emulating the main function of foveation, i.e., to sample an object of interest with a high spatial resolution.Color Coding and Contrast Filter Augmenting a ConvNet with color or contrast pre-filter has so far not led to major improvements in predicting human EEG recordings. Especially, the linear COC filter is only beneficial after several layers of non-linear processing in the ConvNet. The non-linear contrast filters indeed strengthen this notion as predictions are improved for both early and later layers. However, the default RGB model is the overall best performing model, especially given the computational overhead of the pre-filters. Nonetheless, historically ConvNets have been optimized to process RGB coded input for object classification. It is therefore not surprising that changing the input coding does not yield improvements when compared to an already optimized default. In turn, contrast filtering might yield both task performance improvements as well as neural prediction improvements when trained on task thats e.g., require robustness to noise.We found a discrepancy of optimal ConvNet features for object classification and for predicting human EEG data. Additional tests for robustness against adversarial attacks or image perturbation could inform about the benefits of model features that can be used to predict human EEG data. As human vision is highly robust, we would hypothesize that ConvNets augmented with biologically plausible pre-filter exhibit higher degrees of robustness.To conclude, differential spatial sampling has a strong positive effect on the predictive power of ConvNets for predicting human EEG. While GCS-training improved predictions even further, we also see degraded object recognition performance. Future work should explore applying GCS transforms to model features instead of images. Thereby, the distribution shift of the ConvNet input can be mitigated while keeping the benefit of the GCS transform. Overall, we see differential spatial sampling as an input-transformation as an important asset for neural encoding models.  
 
 Appendix:  



Reviews: review: title: title: Interesting, rigorous and clear
main_review: main_review: The paper sets out to test the addition of biologically plausible constraint to convolutional neural networks with the aim of investigating the correspondence between performance of ConvNets and human EEG data. For this purpose, the authors use a series of pre-filters added to ConvNets, focusing on non-uniform spatial sampling, color and constrast, inspired by retinal processing. They use linearised encoding models to map convolutional features onto EEG data (ERPs) and use the prediction performance as an indicator for the pre-filter success. The author systematically investigate the prediction performance of the different pre-filters, and show the non-uniform spatial sampling leads to improved prediction of EEG responses at certain time points.
The paper is very clear, demonstrate rigorous methodology, novel contribution to the field and of high interest to the community.
While the papers shows an interesting and important addition to existing literature, several concerns include:
1. While the benchmark for successful performance was predicting differences of EEG responses across, the ERPs themselves are not shown and it is not clear whether there were indeed substantial differences between them.
2. The aim was to better predict EEG responses. Nevertheless, the tasks of the ConvNets (object classification) and humans (target detection) were very different from one another. In fact, the outdoor scenes were non-targets in the EEG experiment, therefore it is not clear what the level of their processing was, even in early areas of the visual cortex. Indeed, while the pre-filters improved prediction of EEG responses, they did not necessarily improve object classification by the ConvNets.
3. As the author noted, a more suitable non-uniform sampling pre-filter could have included varying sampling within the same image (centre/periphery). It would be nice to see the effect of such pre-filter.
4. Because the pre-filters did not result in better object classification, it would be useful to discuss potential solutions for that for future research, and ways in which both EEG predictions and model performance can be improved.
strength_weakness: strength_weakness: The research presented in the paper in of high quality, the findings are useful and interesting to the community, and it sets a good direction for future research.
title: title: This study investigates the incorporation of functional and structural properties of the human retina into ConvNets to enhance their alignment with human brain activity. Their work used foveated stimuli and implementing differential spatial sampling in ConvNets, showing improvements in predicting human EEG data. However, no evidence found that incorporating color and contrast filtering of inputs yields enhanced predictions.
main_review: main_review: 1. Clarity: The paper presents its ideas clearly and understandably. However, there are some sections where additional clarification may be beneficial, particularly regarding the implementation of human-inspired pre-filters, to ensure complete comprehension by all readers (especially for the ML community).

2. Correctness: This paper shows strong evidence that foveated stimuli and differential spatial sampling can improve the alignment between ConvNets and human brain activity. Interestingly, the paper also observes some pre-filters such as color and contrast filtering that exist in human perception but do not lead to improvement in the alignment. Further causal analysis is needed to understand what kinds of features contribute to the improvement. 

3. Novety: This paper presents a novel and promising way to improve the alignment between the neural networks and the brain response.
strength_weakness: strength_weakness: This paper demonstrates the enhancement of ConvNets' credibility as models for human visual processing by integrating pre-filters computed in the human retina and the LGN. Its findings resonate well with the workshop's overarching theme, and the insights into convergence and divergence presented herein point toward a timely and important topic for achieving better alignment between humans and machines.
title: title: Retinal sampling grids improve predictivity of Human EEG responses
main_review: main_review: This paper explores the impact of several biologically motivated preprocessing strategies on the ability of ANNs to predict EEG responses of humans in response to natural scenes. 

Strengths:
- Several of the applied interventions yield improved predictability. 
- The strength of the measurement modality (EEG) is leveraged by examining the timecourse of predictivity. I found the full-view vs. center-cropped result (where full view better predicted early onset responses to be particularly interesting. 
- The EEG dataset collected seems like a valuable contribution on its own. 

Weaknesses:
- Most of the effect sizes seem to be quite small. I.e. in figure 2 there are a range of sites where there are statisticlally significant differences between settings (as visualilzed in the top plots), but the absolute differences in predictivity seem marginal when examining the left two panels. 
- I am skeptical about the setting where GSC is applied during feature extraction to networks trained without this preprocessing step. This would seem to me to put the inputs "out of distribution" from the network's perspective and I am not sure I would expect the extracted representations to be very meaningful. The authors could check this by evaluating classification performance for the vanilla trained model when given GSC processed inputs.
- I am less familiar with the idea that edgemaps are comparable to a precortical processing step. I think the motivation for its inclusion in this study could be made more clear. 

Questions:
- In figure 4 GCS is missing from the right panel, or am I missing something?
strength_weakness: strength_weakness: The topic is very well aligned with the workshop, a new dataset was collected, and the experiments seem rigorous and thorough.
title: title: Submission53 presents interesting initial findings that could be supplemented with more integration into relevant work and more experiments
main_review: main_review: The authors try to implement more biologically plausible CNNs to match neural human EEG data by drawing inspiration from the retina. They compare their methodology to previous work and show increase in performance to several previous methods. While the authors already do a pretty good job of centering their work in current literature, there are some missed opportunities. In addition, a few more experiments and interpretative analysis would bring the paper to the next level of interest. Lastly, some changes to the formatting of the paper would aid greatly in clarity.

1) The authors cite the presence of the fovea within the retina as an inspiration for more biologically plausible networks. The fovea of the retina is critical, but other retinal properties also lend to the initial processing of the retina feeding to the LGN, such as the diverse roles of amacrine cells and bipolar cells in complex computations (beyond just edge detection). 

2. It is not entirely clear to me exactly which methodologies the authors are applying are novel, and if none, are they simply comparing to on a novel dataset.

3. While training a neural network to predict neural activity is interesting, good performance on this does not necessarily indicate representational alignment. Instead, is there a way to characterize alignment of the network with the visual task-trained (object recognition) networks?

4. Combinations of filters (RGC and color, etc) may be interesting to add into the mix.

5. It would be interesting to probe further into how RGC and edge-detection are truly impacting the activity and representations of the networks through perturbation experiments or otherwise.

6. Please revise the captions of the figures for clarity and consider using sub-panels (A, B, C). Please order the references alphabetically by last name. It's hard to see the differences in Figure 2, I am not convinced the effects are meaningful.

7. Please consider checking out the work of the following labs for deep models of visual systems: Stephen Baccus (Stanford), Dan Yamins (Stanford), Nancy Kanwisher (MIT)
strength_weakness: strength_weakness: The findings are interesting and should definitely be presented to the wider community.
title: title: The paper is well-structured and relevant to the theme of Re-Align Workshop of ICLR 2024.
main_review: main_review: The paper is evaluating the role of six biologically inspired pre-cortical filters on the alignment of the intermediate representations of a deep neural network - AlexNet - with acquired human EEG data. The six filters inspired by the functional and structural properties of retina include: 1) full-scene RGB images, 2) centre-cropped RGB images, 3) Ganglion spatial sampling of RGB images as a differential spatial sampling method, 4) converting RGB images to colour-opponent channels, 5) converting RGB images to edge-maps, 6) converting transformed colour-opponent channels to edge-maps. The EEG data is obtained for 31 participants exposed to high-resolution natural scene images composed of 16 common outdoor object classes. For computational modelling, AlexNet has been trained over the same set of stimuli and object classes. Each model is augmented by one of those six pre-filters which are static and don't have any training parameters. Furthermore, the pre-trained features from all max pooling layers and the penultimate fully-connected layer of AlexNet, after some processing, have been used to regress out the ERP amplitudes of EEG data for every subject, electrode, and time point. The final results show that Ganglion spatial sampling of RGB images as a differential spatial sampling and biologically plausible filter, can improve EEG prediction performance, and thus the model-brain alignment. Other filters do not have that much effects. Overall, I think the paper has been clearly written, correct, novel, and interesting to the community. I suggest only two minor improvements. First, it would be great if the authors motivate more - probably considering the biology of the retina - the reasons for choosing those six filters. Second, it would be great if the authors elaborate more on why they choose this amount of stimuli, object classes, and training epochs considering the number of training parameters in AlexNet. Basically, my question is that how they can make sure that they cannot improve the test accuracy of the AlexNet by increasing training data and epoch without overfitting. There is also one small typo in line 25 of page 9 of the paper. I think "ny" should be written as "any".
strength_weakness: strength_weakness: As I mentioned, I think the paper is well-written, clear, correct, novel, and interesting to the community.

Comments: comment: We thank the reviewer for the comments and for pointers and references for further improvements!

Regarding comment 3, we want to emphasise that our method indeed trains ConvNets to perform object recognition and not to directly predict neural data. We then build a linear encoding model, that performs a linear regression on the human EEG data using the pre-trained features from the ConvNet resulting in behavioural alignment (ConvNet can classify objects) and representational alignment (ConvNet features can be linearly mapped onto human EEG data).
This is indeed the suggested way of characterising alignment using the visual task-trained network.