ReAlnet: Achieving More Human Brain-Like Vision via Human Neural Representational Alignment

Authors: zitong  lu
julie d golomb
yile  wang
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-02 23:24:26

Abstract: Despite the remarkable strides made in artificial intelligence, current object recognition models still lag behind in emulating the mechanism of visual information processing in human brains. Recent studies have highlighted the potential of using neural data to mimic brain processing; however, these often reply on invasive neural recordings from non-human subjects, leaving a critical gap in our understanding of human visual perception and the development of more human brain-like vision models. Addressing this gap, we present, for the first time, ‘Re(presentational)Al(ignment)net’, a vision model aligned with human brain activity based on non-invasive EEG recordings, demonstrating a significantly higher similarity to human brain representations. Our innovative image-to-brain multi-layer encoding alignment framework not only optimizes multiple layers of the model, marking a substantial leap in neural alignment, but also enables the model to efficiently learn and mimic human brain’s visual representational patterns across object categories and different neural data modalities. Furthermore, we discover that alignment with human brain representations improves the model’s adversarial robustness. Our findings suggest that ReAlnet sets a new precedent in the field, bridging the gap between artificial and human vision, and paving the way for more brain-like artificial intelligence systems.

Main paper: 
 introduction: 
 While current vision models in artificial intelligence (AI) are advanced, they still fall short of capturing the full complexity and adaptability inherent in the human brain's information processing. Deep convolutional neural networks (DCNNs) have reached a performance level in object recognition that rivals human capabilities Lecun et al. (2015) , and many studies have identified similarities in the hierarchical structure between DCNNs and the ventral visual stream Cichy et al. (2016) ; Güc ¸lü & van Gerven (2015) ; Kietzmann et al. (2019) ; Lu & Golomb (2023a) ; Yamins et al. (2014) . However, the alignment between DCNNs and human neural representations remains deeply inadequate, whether compared with human electroencephalography (EEG) or functional magnetic resonance imaging (fMRI) data. Enhancing the resemblance between visual models and the human brain has become a critical concern for both computer scientists and neuroscientists. From a computer vision perspective, brain-inspired models often exhibit higher robustness and generalization, crucial for realizing true brain-like intelligence; meanwhile, from a cognitive neuroscience perspective, models that more closely mirror brain representations can significantly aid in our exploration of the brain's visual processing mechanisms.Given these challenges and limitations, the pivotal question arises is how we can leverage our understanding of the human brain to enhance current AI vision models. Conventional approaches have limitations in emulating the complexity of the human brain's visual information processing, even with increased model depth and layers Rajalingham et al. (2018) . This limitation has prompted 2023)). However, limited studies have focused on directly using neural responses to complex visual information as feedback to improve the model's similarity to human brains. Our research focuses on a third approach -utilizing human brain neural activity data to realize brain-like models. This approach represents a more direct alignment strategy between models and the human brain, unconstrained by variations in model structure or pre-training methods, potentially marking a crucial step towards achieving greater resemblance to the human brain. Thus, our central research question emerges: Can we use human brain activity to align ANNs on object recognition and achieve more human brain-like vision models? Related Work. Several previous studies have already started to try to apply neural data to machine learning especially deep learning models. The earliest attempt was to apply human fMRI signals to amend the classification boundary of SVMs and CNNs to achieve better category classification performance Fong et al. (2018) . Some more recent studies started to let the models learn neural representations. One common way is to add a similarity loss to increase the representational similarity between models and neural activity (neural recordings from mouse V1, monkey V1 or IT) during the training Dapello et al. (2023); Federer et al. (2020) ; Li et al. (2019) ; Pirlot et al. (2022) . Another strategy from Safarani et al. (2021) is to add an additional task based on an encoding module to predict monkey V1 neural activity. Both similarity-based method and multi-task framework can achieve more brain-like representations and improve model robustness. However, these neural align-ment studies have two key challenges: (a) Dependence on animal instead of human neural activity. This limits the direct applicability and relevance of findings to human visual processing, and it is harder to enable models to effectively learn the human brain's representational patterns based on the low data quality. (b) Single brain region or single model layer alignment. On the one hand, previous studies could only align a single early or late layer in CNN and/or align the model with a certain brain region, V1 or IT. On the other hand, it remains unclear which specific brain region should align with which particular layer of the model, leading to potential misalignment and inaccuracies.Additionally, a recent study focused on video emotion recognition first applied a representational similarity-based method to align CNN with human fMRI activity Fu et al. (2023) . However, it is noteworthy that they focused on simpler emotion recognition tasks, may fall short in the more complex and diverse domain of object recognition which has larger space and multitude of object categories. Therefore, our work addressed this by employing an additional encoding module that goes beyond mere similarity. This module predicts human neural activity and is trained to autonomously extract complex visual features, offering a more effective approach for aligning the model with human neural representations in object recognition.Contributions. To bridge the gap between AI vision and human vision, we propose a more human brain-like vision model, ReAlnet, effectively aligned with human brain representations based on a novel and effective encoding-based multi-layer alignment framework. We summarize our contributions and novel findings as follows:• To the best of our knowledge, we are the first to directly align object recognition models using non-invasive neural data recorded from human brains, which opens new possibilities for enhancing brain-like representations in models based on human brain activity.• We propose a novel image-to-brain encoding-based representational alignment framework that optimizes multiple layers of the network simultaneously, which effectively improves the model's similarity to human brain representations across different modalities (both human EEG and fMRI).• Our representational alignment framework allows us to obtain a personalized vision model by aligning with individual's neural data.• Aligning with human neural representations can improve the model adversarial robustness. 
re(-presentational)al(-ignment)net: 
 Here we describe the human neural data (EEG data for the alignment, and both EEG and fMRI data for testing the similarity between models and human brains) we used in this study, the alignment pipeline (including the structure, the loss functions, and training and test methods) for aligning CORnet representations with human neural representations to obtain ReAlnet, and the evaluation methods for measuring representational similarity between models and human brains and adversarial robustness. Basic architecture of ReAlnet: We have chosen the state-of-the-art CORnet-S model Kubilius et al. (2018; 2019) as the foundational architecture for ReAlnet, incorporating recurrent connections akin to those in the biological visual system and proven to more closely emulate the brain's visual processing.EEG generation module: In addition to the original recurrent CNN structure, we have added an EEG generation module designed to construct an image-to-brain encoding model for generating realistic human EEG signals. Each visual layer is connected to a nonlinear N × 128 layer-encoder (Enc-V1, Enc-V2, Enc-V4, and Enc-IT correspond to Layer V1, V2, V4, and IT) that processes through a fully connected network with a ReLU activation. These four layer-encoders are then directly concatenated to form an N × 512 Multi-Layer Visual Encoder, which is subsequently connected to an N × 340 EEG encoder through a linear layer to generate the predicted EEG signals.Here N is the batch size.Therefore, we aim for the model to not only perform the object classification task but also to generate human EEG signals which can be highly similar to the real EEG signals when a person views the certain image through the EEG generation module with a series of encoders. During this process of generating brain activity, ReAlnet's visual layers are poised to effectively extract features more aligned with neural representations.Alignment Loss: Accordingly, the training loss L A of our alignment framework consists of two primary losses, a classification loss and a generation loss with a parameter β that determines the relative weighting:L A = L C + β • L G (1)L C represents the standard categorical cross entropy loss for model predictions on ImageNet labels:EQUATIONHere, y i represents the i-th image, and p i represents the probability that model predicts the i-th image belongs to class i out of 1000 categories. However, the correct ImageNet category labels for images in THINGS dataset are not available. Therefore, we adopt the same strategy as in Dapello et al. (2023) , using the labels obtained from the ImageNet pre-trained CORnet without neural alignment as the true labels to stabilize the classification performance of ReAlnet.L G is the generation loss, which includes a mean squared error (MSE) loss L M SE and a contrastive loss L Cont between the generated and real EEG signals. This contrastive loss is calculated based on the dissimilarity (one minus Spearman correlation coefficient) between generated and real signals, aiming to bring the generated signals from the same image (positive pairs) closer to the corresponding real human EEG signals and make the generated signals from different images (negative pairs) more distinct. L G is calculated as followed:EQUATIONEQUATIONHere, S i and Ŝi represent the generated and real EEG signals corresponding to the i-th image.Training procedures: Unlike CORnet which trained on the same ImageNet dataset, ReAlnet additionally trained on individual EEG data consists of 10 personalized ReAlnets, 1 per EEG subjects. Each network were trained to minimize the alignment loss including both classification and generation losses with a static training rate of 0.00002 for 30 epochs using the Adam optimizer. We used a batch size of 16, meaning the contrastive loss computed dissimilarities of 256 pairs for each gradient step. Also, we trained various ReAlnets using four different β weights (β = 1, 10, 100, 1000) separately. In total, we trained 40 ReAlnets (4 β weights × 10 subjects).Representational similarity analysis (RSA): RSA is used for representational comparisons between models and human brains Kriegeskorte et al. (2008) based on first computing representational dissimilarity matrices (RDMs) for models and human neural signals, and then calculating Spearman correlation coefficients between RDMs from two systems.To evaluate the similarity between models and human EEG, the shape of each RDM is 200 × 200, corresponding to 200 images in THINGS EEG2 test set. For EEG RDMs, we applied decoding accuracy between two image conditions as the dissimilarity index to construct EEG RDM for each timepoint and each subject. For model RDMs, we input 200 images into each model and obtained latent features from each visual layer. Then, we constructed each layer's RDM by calculating the dissimilarity using one minus Pearson correlation coefficient between flattened vectors of latent features corresponding to any two images. To compare the representations, we calculated the Spearman correlation coefficient as the similarity index between layer-by-layer model RDMs and timepointby-timepoint neural EEG RDMs. To evaluate the similarity between models and human fMRI, the shape of each RDM is 50 × 50, corresponding to 50 images in Shen fMRI dataset test set. For fMRI RDMs, we calculated one minus Pearson correlation coefficient between voxel-wise activation patterns corresponding to any two images as the dissimilarity index in the RDM for each ROI and each subject. For model RDMs, similar to the EEG comparisons above, we obtained the 50 × 50 RDM for each layer from each model. Then, we calculated the Spearman correlation coefficient as the similarity index between layer-by-layer model RDMs and neural fMRI RDMs for different ROIs, assigning the final similarity for a certain brain region as the highest similarity result across model layers due to the lack of a clear correspondence between different model layers and brain regions. All RSA analyses were implemented based on NeuroRA toolbox Lu & Ku (2020) .Adversarial attacks: For performing white box adversarial attacks, we used Fast Gradient Sign Attack (FGSA). We evaluated top-5 classification accuracies on ImageNet with epsilon ranged from 0 to 0.06 for each model.  
improved similarity in realnets to human eeg: 
 Here, for each of the 10 human subjects, we calculated (1) the similarity between their EEG data and the single CORnet, and (2) the similarity between their EEG data and the subject-matched ReAlnet. ReAlnets show significantly higher similarity to human EEG neural dynamics for all four visual layers (Layer V1: 70-130ms and 160-200ms; Layer V2: 60-200ms; Layer V4: 60-200ms; Layer IT: 70-160ms) than the original CORnet without human neural alignment (Figure2A). Further statistical analysis of each layer's similarity improvement (ReAlnet -CORnet) and improvement ratio ((ReAlnet -CORnet) / CORnet) also indicate that at the similarity peak timepoint, there is a maximum of an 8% similarity improvement and an 80% improvement ratio, with the average improvement for the 50-200ms time-window being over 5% and the average improvement ratio over 40% (Figure2B). Additional comparisons also show that ReAlnet is more human brain-like than not only CORnet, but also ResNet and CLIP (Figure1B).These results suggest three findings: (1) Our multi-layer alignment framework indeed improves all layers' similarity to human EEG representations.(2) Every ReAlnet with individual neural alignment exhibits improved similarity to human EEG compared to the basic CORnet.(3) ReAlnets demonstrate the generalization of improvement in human brain-like similarity cross object categories, as the image categories used for testing were entirely absent during the alignment training.Although ReAlnet demonstrates higher similarity to human EEG, a question arises: Does ReAlnet learn representations specific to EEG, or more general neural representations of the human brain? To ensure that our alignment framework enables the model to learn representations beyond the single modality of EEG, we utilized additional human fMRI data to evaluate the model's cross-modality representational similarity to human fMRI.Excitingly, we indeed observed an increase in this cross-modal brain-like similarity. The results indicate that even though ReAlnets were aligned based on human EEG data, they still resemble the human brain more closely on fMRI data compared to CORnet (Figure2B). Also, there is a significant correlation of ReAlnets' similarity improvement compared to CORnet between EEG and fMRI (r=.9204, p < .001) (Figure2C).These findings further highlight three points: (1) Across multiple ROIs, ReAlnets exhibits higher human fMRI similarity than CORnet. ( 2 (2) This variability increases with the depth of the layers (from Layer V1 to Layer IT, Figure3A and Figure3B). This may also suggest a trend of increasing individual variability from primary to higher visual cortical areas in human brains.Using white-box FGSA, we also discovered that ReAlnets, aligned with human neural representations, have increased adversarial robustness against adversarial attacks (Figure3C). The left panel of Figure3C shows a slight increase in adversarial robustness in ReAlnets compared to CORnet at around Epsilon = 0.02. However, the original classification performance (Epsilon = 0) of ReAlnets is lower than that of CORnet, due to the absence of correct labels for images in THINGS EEG2 dataset. To make a fairer comparison, we aligned the classification accuracy at Epsilon = 0 as the baseline to observe the relative decline in accuracy for both ReAlnets and CORnet as Epsilon value increases. The corrected results (Figure3C right) demonstrate more pronounced adversarial stability in ReAlnets.The results presented above are based on a generation loss weight β set to 100. We further explored the impact of this β value on the performance of ReAlnet. Theoretically, a higher β should lead to stronger learning of human neural representations. However, is a larger β always better? Our findings suggest otherwise. We observed that with an increase in β, ReAlnets show greater similarity to human EEG and fMRI (Figure4A and Figure4B) and more pronounced individual variability within models (Figure4C). However, an increase in β also reduces the improvement of adversarial stability (the improvement at β = 100 was less significant than at β = 1 or 10) (Figure4D). Moreover, at excessively high values (β = 1000), ReAlnet's adversarial robustness was even lower than the original CORnet without neural alignment (Figure4D). Therefore, this analysis suggests that: (1) It justifies our use of β = 100 as a weight that balances the trade-offs and maximizes advantages of ReAlnet. (2) β is a parameter that could be manipulated differently in future research depending on research goals.For the control experiments, we tested two aspects: (1) How does contrastive learning influence model-to-brain alignment? (2) If we disrupt the pairing of each image with the EEG signal from the same subject but elicited by viewing a different image, can the model still learn the neural representation patterns of the human brain? Accordingly, we trained two additional sets of ReAlnets based on human EEG data from ten subjects in THINGS EEG2 dataset, termed as W/o ContLoss models (without the constrastive loss component) and Unpaired models (where the pairing between images and EEG signals was disrupted).The results of the control experiments reveal: (1) W/o ContLoss models still exhibit an improvement in human brain similarity compared to CORnet. However, while the similarity to human EEG did not decrease compared to ReAlnet, the similarity to cross-modality human fMRI significantly decreased. This suggests that the contrastive loss component in our alignment framework aids Re-Alnet in extracting more cross-modality brain visual representation features. (2) Unpaired models failed to enhance brain similarity, which show no significant improvement in brain similarity compared to CORnet, indicating that the training process requires the model to effectively learn the specific neural visual features corresponding to each image. Only in this way can the model become more human brain-like and then exhibit higher similarity to the human brain across different object images, categories, and human neuroimaging data modalities. Regarding ReAlnet itself, it warrants further exploration to ascertain what specific information has learned from the alignment with human brains. The fact that different generation loss weights do not significantly impact the behavioral performance but do enhance its similarity to human brains suggests that nodes in the model, which originally did not encode category-specific information, may have been optimized Federer et al. (2020) . More analyses of the neural network's internal representations may be needed to delve into this. Also, from a reverse-engineering perspective, attempting to understand the brain-like optimization process of the model could further aid in unraveling the mechanisms by which our brains process visual information Ayzenberg et al. (2023) ; Cic ( 2019 Certainly, it is important to highlight that ReAlnet transcends being merely a specific vision model; it represents a pioneering framework potentially applicable for aligning any AI model with brain activity. On the one hand, this alignment framework can be extended to other neural modalities, such as fMRI and MEG (dimensionality reduction might be necessary for extensive neural data features), paving the way for the development of variants like ReAlnet-fMRI and ReAlnet-MEG. On the other hand, the ambition is to adapt this framework to a wider range of models and tasks in the future, including language and auditory processing and self-supervised or unsupervised models, leading to innovations such as ReAlnet-Language, ReAlnet-Auditory, and self-supervised or unsupervised versions of ReAlnet. 
conclusion: 
 Our study transcends traditional boundaries by employing a groundbreaking alignment framework that pioneers the use of human neural data to achieving a more human brain-like vision model, Re-Alnet. Demonstrating significant advances in bio-inspired AI, ReAlnet not only aligns closely with human EEG and fMRI but also exhibit hierarchical individual variabilities and increased adversarial robustness, mirroring human visual processing. We hope that our alignment framework stands as a testament to the potential synergy between computational neuroscience and machine learning and enables the enhancement of any AI model to be more human brain-like, opening up exciting possibilities for future research in brain-like AI systems. 
 
 Appendix: We tested the classification accuracy of ReAlnets on ImageNet at different β values (Figure 8 ). Importantly, to ascertain that the observed decrease in accuracy was not due to the additional generation task compromising classification performance, but rather the absence of correct ImageNet labels for images in THINGS EEG2 dataset, we trained a ReAlnet with β = 0. This ReAlnet excluded the EEG signal generation module but underwent fine-tuning with images from THINGS EEG2 dataset. The results indicated that the ReAlnet with β = 0 also experienced a similar level of decline.  


Decision: We invite this work for presentation as a poster at the workshop. However, we strongly recommend that the authors temper and properly scope the title of the paper, as the evidence presented is insufficient to justify broad and advanced claims of "achieving human brain-like vision" with the methods presented here (for which analogous methods have been proposed before and many ongoing developments will be presented at this workshop), and the notion of "human neural representational alignment" employed only covers a single method for representational alignment (for which there are many alternative methods; see Sucholutsky et al. (2023)).

Reviews: review: title: title: A promising paper that fits the workshop well, but embedding in literature and claims of robustness can be improved
main_review: main_review: - Clarity: the paper’s aims and claims are very clear. Some details of the methods require elaboration.
- Correctness: the results largely appear correct, some of them require additional controls and raise new questions.
- Novelty: the paper claims to be the first to directly align object recognition models using non-invasive human brain data. I think this is correct, although the paper could benefit from more careful embedding in related literature on end-to-end training on neural data and EEG encoding/decoding (see comments below).
- Interest to the community: This paper is highly likely to be of interest to the workshop.
strength_weakness: strength_weakness: - Demonstrating increased robustness of on object recognition through representational alignment with non-invasive human data is very interesting.
- The generalisation to a second non-invasive modality (fMRI) is impressive.
- The simplicity of the approach (1 non-linear+ 1 linear mapping of 4 layer network to downsampled and vectorized EEG) is attractive.
- The paper already includes several important controls (training on unpaired data, training without contrastive loss, training with beta set to 0).
- The paper raises a lot of interesting questions for further analyses, e.g. what aspects of the CorNet representations change when adding the EEG-based loss, why some subject’s aligned ReAINets generalize better to fMRI than others, why such networks show increased adversarial robustness, etc.‘
title: title: Incremental, but highly relevant work
main_review: main_review: Clarity 4/5
Correctness 5/5
Novelty 4/5
Interest 4/5

The authors propose -- ReAlnet -- a framework for aligning convolutional neural network representations with human EEG and fMRI signals to achieve more human brain-like vision models. 

Their experiments show representational similarity to both EEG and fMRI after alignment and increased adversarial robustness.
strength_weakness: strength_weakness: Manuscript was well written and quite thorough
title: title: Official Review of Submission29 by Reviewer TKXo
main_review: main_review: **Summary:** \
In the study, the authors propose a new vision model called ReAlnet consisting of a CORNet-S backbone and adding an EGG generation model. By using an alignment loss that not only optimizes for classification prediction but also EEG signal prediction they show that while reaching a slightly less ImageNet prediction accuracy compared to vanilla CORNet-S an improved similarity to human EEG signal even generalizing to fMRI signal without being trained for it. This ReAlnet exhibits additional robustness against white-box (FGSA) adversarial attacks. 

---

**Clarity:** good \
**Correctness:** good \
**Novelty:** good \
**Interest to the community:** good 

---

**Strong Points:** 
- **Very well-written** manuscript which facilitates the understanding of the work and results.  
- **Good clarity:** The aim of the paper is clear. The ReAlnet offers a great contribution to further understanding which visual features make models more aligned with the human visual system. The alignment loss seems to be an innovative and interesting way to make DCCNs more aligned with human brain measurements.     
- **Great figures**, visualizing the various results clearly and comprehensively.
- The research done here seems timely, and relevant for various fields - including neuroscience to better understand the processing of the visual system for object recognition; Human-AI alignment improving the alignment between comp. models and brain signals, and computer vision by creating more robust vision models. 

--- 

**Questions:** 
Question 1: It is a cool finding that the EEG signal fine-tuned model also generalizes to alignment with fMRI measurements. Would you expect similar results when adding an fMRI activity prediction unit and testing on EEG?
Question 2: Why does introducing a ß (1, 10) result in a drop in Top1/Top5 ImageNet accuracy, and why is it higher for ß =100 and then drops again?
strength_weakness: strength_weakness: N/A