Revealing spatial-frequency channels in an ensemble encoding model of human fMRI

Authors: furkan  ozcelik
ajay  subramanian
najib j majaj
denis g pelli
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-23 22:49:56

Abstract: Humans recognize objects by means of a narrow octave-wide spatial-frequency filter called a ``channel''. Behaviorally, this channel can be revealed using critical band masking, a classic psychophysical method that measures sensitivity of recognition accuracy to noise added at each spatial frequency. We want to learn how this channel emerges from the activity of the brain areas involved in object recognition. Critical band masking relies on measured human accuracy in categorizing filtered-noise-perturbed images. To get an analogous accuracy from the brain response, we use N-way Representational Classification Accuracy (N-RCA). For each noise condition used to perturb images in critical band masking, our score measures how often the brain response to the noisy image is more correlated with the response to the original image when compared with responses to $N-1$ other images. This captures how well the brain area's activity informs categorization of a noisy image. We apply the critical band masking paradigm to these accuracies to reveal spatial frequency channels. We then characterize the spatial frequency channel in each visual ROI of an ensemble fMRI encoding model. Our long term goal is to measure channels directly from human fMRI data. Here, we find that the channel bandwidth equals the 1-octave human channel bandwidth and is conserved across model visual ROIs: V1 to V4 and category-selective areas.

Main paper: 
 introduction: 
 Critical band masking, a classic psychophysical method, measures sensitivity of human recognition to noise added in various spatial-frequency bands [1] . Using this method, previous studies found that humans use a narrow, octave-wide band of spatial-frequencies, called a channel, to recognize letters, faces and natural objects [2, 3, 4] . But object recognition is a complex visual process thought to involve several brain areas -early visual areas (V1-V4) and category-selective areas (IT, FFA, PPA, VWFA, etc.). How do the spatial-frequency preferences of these areas contribute to the octave-wide behavioral channel? Behaviorally, critical band masking on each trial asks the human to perform N-way categorization of an image perturbed with noise of some strength (SDs) restricted to one of several spatial-frequency (SF) bands. Categorization accuracy for each noise condition is then used to compute a noise sensitivity threshold for each spatial-frequency. Taken together, these thresholds trace out an octave-wide, inverted-U-shaped curve, the spatial frequency channel (shown in Figure 1 ). Subramanian et al. [4] used this method to measure behavioral channels in humans and machines performing 16-way ImageNet object recognition. Our goal is to apply their stimuli and method to study spatial-frequency channels in various brain areas. To do so, we need a way to measure a brain area's "categorization accuracy".V�T-B-16 Bandw�dth: 3.05 octaves Human Bandw�dth: 0.80 octaves To deal with this challenge, we propose a proxy score, called N-way Representational Classification Accuracy (N-RCA) that can be computed using brain responses. Our score (depicted in Figure inspired from pairwise similarity comparison analysis [5] , measures how often the representation of the noisy image correlates better with the representation of the original image than those of N -1 other noisy images. Thus, N-RCA is analogous to behavioral classification accuracy, and hence can be used to compute sensitivity thresholds of a brain area's response across spatial frequency conditions, which can then be used to derive a channel.In our analyses, we set N to be 16. This allows us to match chance accuracy 100 N % with chance accuracy in [4] 's behavioral experiments. This new N-RCA score allows us to apply critical band masking to brain responses. This reveals channels in an ensemble of the encoding models of human fMRI data from the Algonauts 2023 challenge [6] .To ensure a representative selection, we picked 8 models that spanned diverse training datasets and architectures. We followed Yang et al.'s [8] for training the encoding models. Yang et al. used factorized (in terms of layer, space and scale) and topologically smooth features for encoding. These constraints help the models achieve smoother representations and higher confidence in prediction results. We trained 6 encoding models for the backbone architectures including CLIP [9] , DiNOv2 [10] , SAM [11] , MAE [12] , MoCov3 [13] and ViT-B-16 [14] (ImageNet) which exhibited high encoding performance in Yang et al. [8] . We also added two variations of DeiT3 [15] architectures trained on ImageNet, DeiT3-B-16 and DeiT3-S-16, due to their low-bandwidth results in behavioral critical band masking analysis [16] . We used data from Subject 1 of the Algonauts challenge which consists of 9841 (image, fMRI trial) pairs. The data was split into 9000 training samples, 200 validation samples (used for ensemble model selection described below), and 641 test samples. All models were trained for 20 epochs. Like [4] , we tested our models using grayscale ImageNet validation images reduced to 20% contrast perturbed with Gaussian noise at 4 strengths (σ ∈ {.02, .04, .08, .16}) and filtered within 7 octave-wide spatial-frequency bands centered at f ∈ {1.75, 3.5, 7, 14, 28, 56, 112} cycles/image. For each model, we collected the encoded representations of layers corresponding to visual areas: V1-V4, and ROIs for Body(EBA, FBA), Face (FFA, OFA), Place (OPA, PPA), and Word (OWFA, VWFA). To consider the best-case encoding model, i.e., an encoding model of human fMRI data that performs better than any single-network encoding model listed above (a closer proxyEnsemble Encoder X � or�g Y � or�g X � no�sy(σ,f ) Y � no�sy(σ,f ) X j no�sy(σ,f ) Y j no�sy(σ,f ) Ensemble Encoder 𝑑 j for 𝑑 � < 𝑑 j 𝑓(𝑥) = for 𝑑 � ≥ 𝑑 j 𝑑 � Figure 2: Diagram of N-RCA method.For each noise condition, (σ, f ), we select ith stimulus image (X i orig ) and corresponding brain responses (Y i orig ) obtained from Natural Scenes Dataset [7] as the ground-truth image and response (we repeat this for every image in the test set), we apply noise to the ground-truth stimulus to obtain corresponding noisy ground-truth image (X i noisy(σ,f ) ) then this image is passed through the Ensemble Encoder to get predicted fMRI response (Y i noisy(σ,f ) ). Then, we select N -1 other from the test set (excluding ith image) to construct our subset with N images for the comparison (we repeat this for every possible combination of N images in the test set). We apply noise to the every image we have chosen to get noisy comparison images (X j noisy(σ,f ) , where j ̸ = i) then we pass them through the Ensemble Encoder to get predicted fMRI responses (Y j noisy(σ,f ) ). We compute the correlation distance (d) between groundtruth fMRI response (Y i orig ) and predicted fMRI response of the noisy groundtruth image (Y i noisy(σ,f ) ) to obtain d i , and also between groundtruth fMRI response (Y i orig ) and predicted fMRI response of the noisy comparison images (Y j noisy(σ,f ) ) to obtain d j . Finally, we compare d i to all d j distances in the subset of N samples, if d i is smaller than all d j distances then we mark the result as 1, else we mark it as 0. We repeat this computation for all of the possible subsets and get the average score by getting the mean of all subsets which results as our R-NCA score.to a human fMRI data), we use an ensemble prediction strategy to compute N-RCA. For each voxel in each noise-frequency condition, we pick the encoding model with the lowest L1 distance over the validation set to predict responses for the test set.We employed the N-RCA method to generate accuracy heatmaps for each visual area. We set N = 16 with the intention of comparing our results with behavioural channels presented in Subramanian et al. [4] . However, even when a different value of N was used, the results did not vary significantly as long as the threshold was also modified accordingly. 50% accuracy thresholds were then computed for each spatial frequency (after normalizing the scores by the score of original RGB image encoding, which is a ceiling for each ROI) and a Gaussian function was fit to reveal channels.We observed that for different noise conditions different models stand out with their correlation performances. While CLIP model is best performing model for original color images with 0.257 R 2 Score (not presented in CBM experiments), DeiT3-S-16 and DiNOv2 are selected more (in terms of voxels) in the ensemble encoder model for different noise conditions. We also observed that in every noise condition, ensemble encoder performs better than any individual model. For example, in the case of testing with original color images ensemble encoder model has achieved 0.264 R2 Score. Although it is just a slight difference in the case of color images (which are not included in CBM experiments), it ensured that encoding model to be always better than the best individual performing model for each noise condition. [4] for human behavior, and broader channel measured using N-RCA in one of the deep neural networks, ViT-B-16. Both human and network channels have inverted-U shapes. We characterize human and network channels by 3 properties: bandwidth (log full-width at half max), center frequency (frequency of max. noise sensitivity), and peak noise sensitivity. In this paper, we only focus on bandwidth results since it explains human-machine differences as mentioned in Subramanian et al. [4] Ensemble Encoder Bandw�dth: 0.75 octaves  
conservation of bandwidth: 
 Subramanian et al. [4] showed that the channel of 76 vision networks, trained to maximize objectrecognition performance, is 2-4 times wider than the 1-octave-wide human channel. The larger bandwidth of networks suggests that they use different features for recognition. How do the encoding model for human fMRI data compare? Fig. 3 plots bandwidths of this ensemble encoding model over several visual ROIs including early visual areas (V1-V4) as well as category-selective areas (bodies, faces, places, words). We see that the channel bandwidth is conserved across the model for all visual ROIs and equals 1-octave, like the human channel. When we examine the ROI-specific bandwidths, in different hypotheses, we could have expected an increasing or decreasing trend, or randomly changing values while we passing from early visual areas to category-specific layers but instead what we observe is a constant channel throughout the ROIs. 
conclusion: 
 We introduced N-RCA score to critical-band masking to study spatial-frequency preferences across brain areas. We test this method on an ensemble of 8 diverse encoding models of human fMRI data and observe that bandwidth is conserved across all visual ROIs and equal to 1 octave.As future work, we plan to apply this score to compute channels directly from human fMRI data and compare those with layer-wise channels derived from state of the art deep neural networks. 
 
 Appendix:  


Decision: In light of the reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Nice but preliminary
main_review: main_review: The paper introduces a metric called N-way Representational Classification Accuracy (N-RCA), which aims to computationally model "channels"—a psychophysical observation that humans use narrow, octave-wide bands of spatial frequencies, known as channels, to recognize letters, faces, and natural objects. The results reveal an inverted U-shaped channel, with the center frequency closely matching human data, but the bandwidth and peak noise sensitivity diverging from human behavior. Despite this difference, the findings are exciting. However, there are several fundamental limitations in the study:

Lack of Methodological Detail: The paper provides insufficient detail about the methodology. The variable "N" in N-RCA likely has a significant impact on the results, yet there is no discussion on its variation or how it affects outcomes. Moreover, the use of eight different encoding models is mentioned, but there is little explanation of how these models were chosen and whether using a different set of models could lead to different results. It is also unclear if all eight models are necessary for the analysis.

fMRI Data and Conceptualization of Noise Response: The study uses fMRI data from general brain responses, but it is uncertain whether fMRI responses would change in the way conceptualized here in reaction to different levels of noise. This raises questions about the applicability of the findings to actual human brain responses.

Stimulus Processing Choices: The method tests grayscale ImageNet validation images reduced to 20% contrast, perturbed with Gaussian noise at four different strengths (σ ∈ {0.02, 0.04, 0.08, 0.16}), and filtered within seven octave-wide spatial-frequency bands centered at frequencies ranging from 1.75 to 112 cycles per image. These values appear to be based on best guesses, without a clear rationale provided for these choices. The lack of justification weakens the robustness of the study’s findings.

Focus on Channel Bandwidth: The paper’s most significant contribution seems to be showing that the model can preserve channel bandwidth to one octave. However, it remains unclear whether optimizing for this behavior has unintended side effects on other model properties. The authors acknowledge the need for further steps, including directly computing channels from human fMRI data and comparing them with layer-wise channels derived from state-of-the-art deep neural networks. Additionally, it is crucial to provide a rationale for model choices and to conduct robustness analyses for these decisions.
title: title: The topic and results are quite interesting, but the paper would benefit from additional technical information and more detailed results.
main_review: main_review: The author proposed using N-RCA with critical-band masking to study spatial frequency preferences across different brain areas. They explored several deep neural networks (8 models) with noise-contaminated natural images at various frequencies. They found that all the deep networks exhibited consistent bandwidth across all visual ROIs. The topic and results are quite interesting.

I have several questions:

1. The author mentioned using N-RCA to measure brain responses; however, there isn’t enough detailed information or mathematical explanation to clarify it. While the author does provide some background on N-RCA, the amount of information presented can make it difficult for readers to follow the main topics. I suggest that the authors include a bit more concise information about N-RCA to enhance understanding.

2. I noticed that the author fed noise images into deep neural networks and then measured the predicted fMRI responses, subsequently calculating the correlation between the real and predicted fMRI responses. However, two aspects are not very clear. First, how was the predicted fMRI response measured? Was linear regression or another method used? Second, I would appreciate it if the author could report the correlation values. This information could be included in the appendix if there isn’t enough space in the main text.

3. Following up on the previous question, I wonder about the relationship between physical distance (or Euclidean distance) and perceptual distance (the distance represented in the brain, e.g., neural responses to clean and noise-contaminated images.). By adding noise to clean images, you alter the physical distance (clean vs noise images), which is then fed into deep neural networks. Assuming that these networks perform functions similar to those of the brain, how does physical distance relate to perceptual distance in both the brain and the deep networks?