BrainACTIV: Identifying visuo-semantic properties driving cortical selectivity using diffusion-based image manipulation
Authors: Diego Garcia Cerdas
Christina Sartzetaki
Magnus Petersen
Gemma Roig
Pascal Mettes
Iris Groen
Venue: ICLR 2025 Poster (2025)
Keywords: brain
selectivity
visual cortex
fMRI
manipulation
variation
diffusion
neuroscience
Primary area: applications to neuroscience & cognitive science
TL;DR: BrainACTIV manipulates a reference image to drive or suppress activity in a target brain region using pretrained diffusion models, providing insights into neural selectivity for experimental purposes.

Abstract: The human brain efficiently represents visual inputs through specialized neural populations that selectively respond to specific categories. Advancements in generative modeling have enabled data-driven discovery of neural selectivity using brain-optimized image synthesis. However, current methods independently generate one sample at a time, without enforcing structural constraints on the generations; thus, these individual images have no explicit point of comparison, making it hard to discern which image features drive neural response selectivity. To address this issue, we introduce Brain Activation Control Through Image Variation (BrainACTIV), a method for manipulating a reference image to enhance or decrease activity in a target cortical region using pretrained diffusion models. Starting from a reference image allows for fine-grained and reliable offline identification of optimal visuo-semantic properties, as well as producing controlled stimuli for novel neuroimaging studies. We show that our manipulations effectively modulate predicted fMRI responses and agree with hypothesized preferred categories in established regions of interest, while remaining structurally close to the reference image.  Moreover, we demonstrate how our method accentuates differences between brain regions that are selective to the same category, and how it could be used to explore neural representation of brain regions with unknown selectivities. Hence, BrainACTIV holds the potential to formulate robust hypotheses about brain representation and to facilitate the production of naturalistic stimuli for neuroscientific experiments.

Published as a conference paper at ICLR 2025
BRAINACTIV: IDENTIFYING VISUO-SEMANTI PROPERTIES DRIVING CORTICAL SELECTIVITY
USING DIFFUSION-BALE BASED IMAGE MANIPULATION
Diego García C Christina Sartzetaki', Magnus Petersen", Gemma Roig", Pascal Mettes and Iris Groen University of Amsterdam, The Netherlands, Sioethe-Universsine Frankfurt, Germany
Reference
Minimiize activation
Maximize
Sodyy Arot
Vinaal Ward Sures Aenis
Kieech
Figure 1: BrainACTIV manipulates a reference image to maximize or minimize the activity of target region in the human visual cortex. By analyzing the resulting image variations, quantify visuo-semantic representations that underlie selective responses in the human brain. In the examples above, manipulations enhance hypothesized preferred categories in specific brain regions known to exhibit selectivity for faces, bodies, words, and places, respectively.
ABSTRACT
The human brain efficiently represents visual inputs through specialized neural populations that selectively respond to specific categories. Advancements in gen- erative modeling have enabled data-driven discovery of neural selectivity using brain-optimized image synthesis. However, current methods independently gen- erate one sample
time, without enforeing structural constraints on the genera- tions; thus, these individual images have no explicit point of comparison, making it hard to discern which image features drive neural response selectivity. To ad- dress this issue, we introduce Brain Activation Control Through Image Variation (BrainACTIV), a method for manipulating a reference image to enhance or de- crease activity in target cortical region using pretrained diffusion models. Start- ing from a reference image allows for fine-grand mme-grained and reliable offline identifica- tion of optimal Aisuo-semantic properties, as well as producing controlled stimuli for novel neuroimaging studies. We show that our manipulations effectively mod- ulate predicted fMRI responses and agree with hypothesized preferred categories in established regions of interest, while remaining structurally close to the refer- ence image. Moreover, we demonstrate how our method accentuates differences between brain regions that are selective to the same category, and how it could be used explore neural representation of brain regions with unknown selec- tivities. Hence, BrainACTIV holds the potential to formulate robust hypotheses about brain representation and to facilitate the production of naturalistic stimuli for neuroscientific experiments.
"Corresponding author; di ego aero-ggrrdda@@gaaa c.m

---

Published as a conference paper at ICLR 2025
INTRODUCTION
The discovery of brain regions that selectively respond to specific image categories raises intriguing questions about their underlying neural representations (Grill-Spector and Weiner 2014). While traditional approaches to measuring neural selectivity relied on a few hand-selected image cate- gories, recent studies guide generative models with brain encoder gradients to activate category- selective regions of interest (ROIs) in human visual cortex (Ratan Murty et al. 2021 Ozcelik and VanRullen 2023 Luo al., 2023). These studies pioneered data-driven exploration of neural lectivity by optimizing random noise vectors to synthesize maximun-activatioo images, allowing the formulation of new hypotheses about the representations in each ROI. However, none of them explicitly enforce structural constraints on the generations; hence, the images are independently sampled without an explicit reference point. This process naturally leads varied set of images of which some characteristics are preferred by ROIs and others are randomly produced by the gen- erative model. Disentangling these factors is essential for understanding the neural representations underlying caleeory--ellccivvty and for determining the relative contribution of visual versus seman- tic features to neural representation, a key debate across scene (Groen et al. (017). object (Bracci et al. 2017). face (Vinken et al. 2023) and word (Janini et al. 2022) perception.
We introduce Brain Activation Control Through Image Variation (BrainACTIV). method for ma- nipulating reference image to increase or decrease predicted brain activity in target cortical region, sce Figure I BrainACTIV uses IP-Adptter (Ye et al. (023)) to prompt a pretrained diffusion model with brain-optimal embeddings obtained through spherical interpolation in CLIP space. Ini-
tial diffusion latents are computed with SDEdit (Meng et al., (2022) to retain the low-level structure of the reference image. The manipulation of reference image (Goetschalekk al., 2019 Papale let al. 2024) ensures a reliable comparison point for the synthesized stimuli, isolating the effect of brain optimality on the latter. Besides straightforward visual interpretation, our method facilitates quantifying differences in visuo-semantic and mid-level image features using computer vision tech- niques, highlighting those preferred by a brain region of interest. Moreover, the use of real image as reference enables the integration of BrainACTIV into novel hypothesis-driven studies.
We validate BrainACTIV by targeting fMRI responses in well-established category- selective ROIs, confirming that their predicted activation is successfully modulated by our image variations and that the visuo-semantic properties highlighted by them agree with previous neuroscientific work. Additionally, we demonstrate how our method can accentuate differences between similar regions of interest, providing insights into the specific role each region in visual processing. Finally, describe how researchers select between semantic variation and low-level structural control when using BrainACTIV for experimental stimulus design. Our contributions are:
The use of image manipulation to maximize or minimize responses in higher visual cortex: this guarantees that the changes made to the original image come from the objective of creasing or decreasing the brain activation, rather than stochasticity in the image generation process, with the original image serving for activity baseline comparison.
The use of automated methods to quantify semantic category presence and mid-level im- age features to characterize each ROI in finer detail, circumventing the need for human behavioral assessments.
The identification of differences in stimulus representation between similar brain regions beyond category selectivity, by accentuating these differences in reference image.
The introduction of BrainACTIV as a controllable method for neuroscientific stimulus gen- eration, describing how researchers can modify the degree of low-level visual changes when generating image variations.
Our code is available aggibbbbbcom/dgggggeedddsstttttn
2 RELATED WORK
Category Selectivity in the Higher Visual Cortex. Different regions in high-level areas of the human visual cortex exhibit selectivity for specific semantic categories like faces, bodies, places, and words (Kanwisher et al. [997] McCarthy et al. [997] Downing et al.; 2001 (eeleen and Down-
ing 2005 Epstein and Kanwisher 1998 McCandliss et al. 2003). Reliable characterization of
2

---

Published as a conference paper at ICLR 2025
each region requires measuring neural responses to large sets of images and finding those that elicit maximal activity (Ratan Murty et al., 202I). However, experimental constraints and the high di- mensionality of image space make it impossible to test all potential stimuli. Neuroscientists have traditionally narrowed this search by focusing on hand-selected stimuli, but this risks overlooking relevant features that could not be conceived priori. Deep neural networks (DNNs) trained on large-scale datasets of brain recordings have been adopted as "brain encoders" to make rapid and highly accurate predictions of neural responses to large volumes of images (Khosla et al. 2021). Moreover, deep generative models such as diffusion models Ho et al. (2020); Song et al. (2020) can synthesize novel stimuli by sampling from rich image priors constrained to the domain of natural images. Our work combines brain encoders and diffusion models to highlight semantic properties that drive functional selectivity in the visual cortex, enabling the formulation of new hypotheses more robustly and objectively than current data-driven approaches.
Image Variation with Diffusion Models and CLIP. Diffusion models treat the data generation process iterative noise removal, progressively refining random noise 2т ~ (O, 1) into struc- tured data Ty--, T2+1, Ft, 2t- ..., Ty through a trained denoising network. This process can be guided to synthesize samples from conditional distribution, done by text-to-image (T2I) models (Nichol et al.] [022] Saharia et al.] (022). Stable Diffusion (Rombach et al.] (022) enables efficient T21 synthesis by representing data in lower-dimensional latent space. Image prompting allows for generating variations of a reference image I, preserving its style and content. IP-Adapter Ye ctt al. (2023) introduces additional cross-attention layers in the denoising network of pretrained T2I models, incorporating information extracted by a CLIP image encoder (Radford ct al. 2021). To
preserve low-level structural fidelity the reference image, SDEdit (Meng et al. 2022) initializes the denoising process at an intermediate step by injecting noise to I up to timestep tg = T with with
Y E [0, 1] and using Tio as starting point. Our work employs IP-Adapter and SDEdit on Stable Diffusion to generate image variations conditioned on brain-derived CLIP embeddings.
Optimal Visual Stimulus Generation. Previous studies have successfully used gradients from DNN-based brain encoders to produce stimuli that maximally activate parts of the macaque and mouse visual cortex (Bashivan et al., 2019; Walker et al. 2019: Ponce et al. 2019). Later approaches steered random noise vectors within generative models using encoder gradients to synthesize optimal stimuli for category-selective visual regions in macaques (Pierzchlewicz et al. 2024) and the human brain: NeuroGen (Gu et al.] (2022) and Ratan Murty al. (2021) used GANs (Goodfellow et al., 2014), while BrainDiVE (Luo et al. 2023) improved stimulus quality and semantic specificity by using diffusion models and a CLiP-based brain encoder. Diffusion-basse
generation has proven ef- fective in "brain decoding" settings, where a visual stimulus is reconstructed based on elicited brain activation patterns (Chen et al. 2023 Scotti et al., 2023 Zeng et al.) 2023 Ozcelik and VunRullen, 2023). In contrast, BrainDiVE and BrainACTIV synthesize novel stimuli that maximize predicted activity in specific brain regions. Because the noise vectors in BrainDiVE are randomly sampled for each synthesized image, this process leads to a varied stimulus set that shares some characteristics (i.e., those preferred by the region) and differs in others (i.e., those randomly produced by the gener- ative model). This introduces the need for human behavioral studies to interpret large image sets to disentangle these features. Instead, our method of brain-targeted image variation ensures a point of comparison for each synthesized stimulus, directly disentangling the effect of brain activity optimal- ity and allowing the quantification of semantic and mid-level image features relevant to the targeted cortical region using computer vision techniques. Concurrent work by Prince et al. (022)) explores the accentuation of pixel-based features in an image through gradient ascent to modulate brain ac- tivations; further, work by Papale et al. (2024) explores image perturbation through a GAN-based brain decoder (Dado et al. (024)) to study tuning properties of monkey IT neurons. In contrast, we leverage diffusion models and spherical interpolation in CLIP's latent space to study broader regions in the human visual cortex.
3 METHODS
Given a real reference image I, we aim to produce variations highlighting semantic selectivity prop- erties of a target cortical region. First, we explain how condition diffusion models on a brain- derived signal to synthesize variations that increase or decrease predicted activations (Figure 2). Then, we describe how to quantify differences in semantic and mid-level image features to identify properties preferred by each region.
3

---

Published as a conference paper at ICLR 2025
1
Deriving modulation embeddings brain encoder
Interpolating in CLIP space
ROI activations
10
Reference Image
10
Rain
Training
CUP mg
Linear Regression
W
Spherical Interpelation
CLIP
min
Smix
3 Generating variations with diffusion model
Region of interest [ROI]
2T. *I..
Mezimize
Objective
IP Adapter
Model
Minimite
prompt"
Category selectivity:
Bodies
Words
Figure 2: Brain-targeted image variation pipeline. (1) Modulation emheddings are derived from CLIP-based brain encoder. (2) For reference image, we produce intermediate embeddings using spherical interpolation in CLIP space. (3) IP-Adapter conditions a pretrained diffusion model
on the intermediate embeddings generate images that maximize minimize activity in category- selective ROIs; SDEdit helps retain low-level structural similarity to the reference image.
3.1 BRAIN-TARGETED IMAGE VARIATION
CLIP's semantically rich image embeddings have displayed high representational similarity to the higher visual cortex (Conwell al. 2023 Wang et al. (023), making them suitable choice for representing and manipulating semantic content in the original image Z. Specifically, we move the image embedding zI CIIPing(I) towards optimal endpoints that derive from paired images and fMRI recordings. We refer to these endpoints as modulation embeddings.
First, similarly to BrainDiVE (Luo al. 2023 we fit a brain encoder Hxx33 R that transforms images J into brain activations where the latter are single values representing the average of voxel-wise beta values belonging to the region of interest (ROIf The brain encoder consists of two parts. The first is a frozen CLIP image encoder that outputs D D-dimensional vectors. The second is regularized linear regression model on normalized CLIP embeddings:
CLIP ing
||CLIP ing (T)I
-
Due to the linear relationship between normalized embeddings and activations, RDD can be thought of vector in CLIP space that points the direction of maximal activity for ROI. Likewise, the negated weights -w point in the direction that minimizes it. Therefore, we define two ROI-specific modulation embeddings, Zmax and Zmin: through the unit-norm weight vector:
W
w
|
Zmin
(2)
Luo et al. (024) explain how to close the modality gap between CLIP embeddings of natural images
and mmax First. for each image M, in set of K natural images M M2,
.Mx
While we use ROI-wise averaged brain responses, this method could be straightforwardly adapted to smaller cortical regions or even single voxels.
4

---

Published as a conference paper at ICLR 2025
softmax score with temperature is computed through
exp(Scoo ( ei)/T)
score;
Σx= exp (Zmax. ex)/T)
where = CLIP img (M;) and ( is the cosine similarity function. Then, mmax is projected to the space of CLIP embeddings for natural images through decoupled weighted sum of the image embeddings:
K
K
proj 2
FIAX
scores ||ex|||
SCOTC
||ex|]
Ck
(4)
A similar procedure be followed for miin- In the following, we assume both mmax and 2min projected unless otherwise stated.
Next, we use modulation embedding mmax and the reference image embedding zI to produce inter- mediate embeddings II..
max using spherical linear interpolation:
sin (1 -0)
sin(a 0) =max
I
Sin(
ZI
(0n(
(5)
where 0 cos coss-
(
the angle between the vectors, a E [0, 1] indicates the extent of Zmax Larger values of a are thus expected to increase activations in the
| [F
rotation, and >/mxx | target ROI. An analogous operation with Zmin yields intermediate embeddings =
Finally, we perform guided image synthesis with Stable Diffusion (Rombach et al. 2022) to generate the image variations Ip.. To incorporate the brain-optimized semantic information from I into In, use IP-Adapter Ye et al. 2023) to prompt the diffusion model with 27 or =I., (skipping the adapter's prepended image encoder). To retain the low-level structure of in L., we obtain the initial diffusion latents It, through SDEdit with to T. where T is the total number of denoising steps and y E [o, 1]. The hyperparameters a and 7 specify the degree of semantic variation and structural control in the manipulations (subsection 4.5
3.2 QUANTIFYING INFORMATION ACTIVITY- MAXIMIZING AND MINIMIZING IMAGES
We identify the effect of brain optimization in the image variations I. by quantifying differences in category presence and mid-level image features with respect the reference I. We focus on 16 categories based on previous research on cortical representation and behavioral judgments (Huth al. 2012, King et al. 2019 Hebart al., 2020); faces, hands.feet. people. animals, plants. furniture, tools, clothing, electronics, vehicles, landscapes, buildings, rooms, and text. For each cat- egory, we build a representation embedding with CLIP. A challenge in doing so that single-word word descriptions are typically insufficient to capture all possible category instances. Therefore, we build the embeddings using an overcomplete set of concrete nouns from WordNet (Miller) 1995) classi- fied by a large language model (details in appendix subsection A. Hence, we measure category presence through cosine similarity between an image embedding and the category's 's embedding.
To illustrate how BrainACTIV
reveal not only high-level categorical, but also low-level struc- tural changes in oraimiopt images, we compute a number mid-level features: entropy, the minimum number of bits needed encode the gray level distribution in local neighborhood, as a loose quantification of texture/clutter, which is known to affect many aspects of human vision Rosenholtz et al.] (2007); and inspired by prior work showing that metrics of 3D scene structure represented in scene-selective ROIs (Lescroart and Gallant 2019: Dwivedi et al. 2021 1, Sarch et al., 2023), we also computed metric depth, estimated with the ZoeDepth network (Bhat
(2023). and Gaussian curvature and surface normals, computed with the XTC network (examples for reference NSD images can be found in Appendix subsection A.I.).
RESULTS
4.1 SETUP
We use the Natural Scenes Dataset (NSD) Allen et al., 2022), a large dataset of whole-brain high-resolution fMRI responses from eight human subjects. Each subject viewed 10,000 nat-
5

---

Published as a conference paper at ICLR 2025
(A)
(B)
Minimite
Reference smaga
Maximize
Tewords
Towards
Encoder predictions
Imspdtaiin
diferance
0.73
FFA
encoderr
BBA
WHFA
FFA
0.25
0.00
EBA
0.50
VIITA
NWRT encodee
0.10
0.00
6P
PPA
0.90
0.00
RSC
1.0
08
0.6
044
02
0. 02 a4 de co 10
Interpolatio
Interpolation diaa
DR0AST ensader
CLP cepeee
Figure 3: Example variations and modulation results. (A) Example image variations show the effect of activation maximization and minimization in each ROI; line plots show predicted differ- ences activation with respect to the reference image. (B) ROI activations predicted by DINO- ViT encoder (top), fwRF encoder (middle), and CLIP encoder (bottom) function of interpolation with each modulation embedding, averaged across test images and subjects.
ural images from the MS COCO dataset (Lin et al., 2014) repeated three times across multiple scanming sessions. The fMRI beta values are z-scored within their original session and averaged repetitions. Following standard practice in fMRI encoding (van Gerven 2017, Naselaris jet al. 2011), we split the data into a shared test set consisting of the 1,000 images seen by all subjects and subject-specific training set with the remaining images. The training sets are used to analytically derive weights w during modulation embedding derivation, as well as for embed- ding projection (using т 0.01). We use projection set consisting of 400,000 images from the ainaaaicnneBeeeee research-safe dataset (Schuhmann et al. 2022). We use the functional localizer masks included in NSD (thresholding at > 5) to define cortical regions of interest.
(Rombach et al. 2022 for guided image synthesis and a pretrained IP-Adapter (ip-adapter sd15) (Ye et al, 2023 for image embedding conditioning. For consistency with these models, we use Open- CLIP's ViT-H// H/14 CLIP architecture with DLION2B-S322-BVV 9K pretrained weights (Radford al. 2021 Ilharco et al. 2021 Schuhmann et al. (022). We use separate brain encoder to predict activations in our experiments. Its architecture consists of DINOv2 (Oquab et al. 2023) as a feature extractor, followed by an ensemble of single-layer vision transformers (ViTs) (Dosovitskiy et al.) 2021) and multilayer perceptrons, inspired by Adeli et al. (202)). To ensure the robustness of our method, we employ additional feature-weighted receptive field encoder YSe- and Naselaris 2018 Allen et al.] 2022) (available through the Neural Encoding Dataset (Gifford and Cichy][2024) in our validation procedure. We refer to these as DINO-VIT encoder and fwRF encoder, respec- tively. Importantly, neither encoder is CLIP-based hence, they do not share the same latent space as the encoders used derive the modulation embeddings. Details on architecture and prediction performance of all encoders can be found in the appendix subsection A.2
We employ pretrained Stable Diffusion model sstableddiffuiion
4.2 MODULATING ACTIVITY IN BRAIN REGIONS OF INTEREST
We validate BrainACTIV by targeting six previously identified regions of interest in the higher vi- sual cortex: fusiform face area (FFA), extrastriate body area (EBA), visual word form area (VWFA), occipital place area (OPA), parahippocampal place area (PPA), and retrosplenial cortex (RSC) (pre-

---

Published as a conference paper at ICLR 2025
1.2 (4)
LPIPS (4)
ROI
Random
Maximize
Minimize
Random
Maximize
Minimize
FFA
31.9
30.11:54
0.27
0.27
EBA
30.7 5..8
30.8+5.2
0.24
=.28
VWFA
29.9 1:888
30.514.7
0.2611..0
0,28 10.10
19.2
0.55=0.09
OPA
:1.3
29.94:5.5
0.29:0.11
1.24
PPA
327
29.4
0.26
RSC
31.1 14.9
32.1
0.27
0.29 100.10
Table 1: Structural control metrics. Image variations remain structurally similar to the reference image even at a for maximization and minimization objectives; hence, they serve as a reliable comparison point to quantify preferred features. Additional baselines in Appendix (subsection
(opendix(subsection A.
cise location can be found in Allen et (022)), First, we identify six mutuaally exclusive subsets of images in NSD that share broadly similar semantic contexts: wild animals, birds, vehicles, vehicles.people in sports, food, and furniture. We define each subset by filtering the pixel-wise category annotations (ooseediissubscction
made available with COCO (appendix /subsection A.3). These subsets are employed to enforce the diversity of image selection in our experiment.
For each subject and each ROI, we select the 20 test images from each subset with measured re- sponses closest baseline activation (i.e., the average ROI activation across all test images). Be- cause initial experiments showed that modulation embeddings are highly similar across subjects (see Appendix subsection A.9). we opt to use the average all subject-specific zmax and Zmin (before projection), excluding the subject which predictions are made. Hence, activity in each subject through a signal (averaged žmux or zmin) derived exclusively from the rest of the subjects' data. We manipulate each of the 120 selected test images with interpolation values E (0.1,0.2, .... 0.9, }. producing 20 variations in total for each image. For SDEdit, we use a log- arithmic warm-up schedule for 7 up a value of 7 0.6 for a 1. Note that a 0 corresponds to the unaltered test image-the the diffusion model is not used. Next, we predict activations for each of them using the appropriate subject- and ROI-specific DINO-ViT encoder and fwRF encoder. To compute the predicted difference in activation for each variation, we subtract the prediction of the reference image.
modulating brain
(A) displays example variations for each ROI, along with the predicted differences in ac- tivation. Note that the effect of on the magnitude of these differences varies for each image, as an effect of its features and the ROI's sensitivity to these. To study a region's iss selectivity, we look for features that consistently appear or disappear over a wide range of contexts. Thus, our analy- focus on the general effect of BrainACTIV over the whole selection of test images. Additional examples can be found in the appendix subsection A.4
Figure 3)
First, we verify that the reference images serve as a reliable comparison point by measuring how structurally similar they are to the variations. Following Meng et al. (2022). we compute image L2 distance and LPIPS (Zhang al. 2018) between reference and variations, averaged over images and subjects. As a baseline, we compare 1,000 random pairs in the test set. Table shows that structural similarity is preserved maximization and minimization (a 1) for all ROIs
Next, we look at the effect of our variations on the DINO-ViT and fwRF encoder outputs to verify that BrainACTIV successfully increases and decreases predicted ROI responses, Figure (B) shows these predictions as a function of a for all ROIs (averaged over images and subjects). We observe a stable increase and decrease across ROIs for both encoders, confirming that our method modulates predicted activations. The plots show an expected lag in activity increase/decrease up to a = 0.4 due our y schedule since we intended the initial variations be close to the reference image. Furthermore, we observe similar effect with the CLIP encoder used to manipulate the images.
4.3 QUANTIFYING VISUO-SEMANTES CHANGES IN IMAGE VARIATIONS
In this section, we verify that the category selectivity suggested by BrainACTIV for each ROI agrees with established neuroscientific findings. To this end,
the manipulations from subsection 4.2
to identify the categories whose presence is increased when activations are maximized (Figure 4 top row). The plots display differences in category presence (relative to the reference) averaged over

---

Published as a conference paper at ICLR 2025
FFA
EBA
VWFA
OPA
PPA
RSC
622
Maximize
Objective
23
23 C4 da-b
Minimize
CA InSerpuation Loc)
Itopelation
Intrgndatonn Clac)
Figure 4: Top categories for each region. Each plot displays the difference in category presence (see subsection 3.2) with respect to the reference image as a function of a. Top-5 categories per ROI for maximization (top) and minimization objective (bottom) are ranked by the highest measured difference. Results agree with hypothesized preferred categories.
all images and subjects. FFA increases the presence of faces, agreeing with Kanwisher et al. 1997) and McCarthy al. (1997); EBA increases body parts (Downing et al. 2001); VWFA increases
text McCandliss al. 2003) and tools/food, potentially suggesting preference for text on small ob- jects OPA, PPA, and RSC increase manmade structures/scenes (Kamps et al. 2016 Epstein and
Kanwisher 1998 Mitchell al., 2018). Category presence also increased during activation mini- mization Figure 4 bottom row): FFA, EBA, and VWFA respond minimally for scenes/structures (particularly for FFA, landscapes), OPA and PPA to people/plants, and RSC to food. These mini- mizations are also broadly consistent with existing literature: the opposite preference of face- versus place-selective regions is commonly observed in IMRI (e.g. Silson al. 2022 Margalit al., 2020), and the minimal preference for plants in place-regions could reflect a preference for built/man made structure (Cukur al., 2016; Groen al.) 202I). However, others are novel: e.g. minimal prefer- ence for food in RSC has, to our know mnogledge, not been reported before.
Entropy
Surface
KM
Depth
Surface normal
Gaussian
Surface
curvature
normal
Figure 5: Mid-level features. Correlation between predicted ROI activation differences and mid- level feature differences each pixel location an image.
Because ACTiV allows the generation of image variations over a wide range of activation val- ues, we can use it study the correlation between predicted activation differences and mid-level feature differences for different locations in an image (Figure 5). Results suggest an important role of surface orientation in differentiating scene- and object-selective regions, with FFA, EBA, and VWFA preferring surfaces pointing outwards and OPA, PPA, and RSC preferring surfaces pointing inward, as reported before controlled stimulus sets (Cheng et al.) 202T). Moreover, the enhanced correlations with surface normals in scene-selective regions are consistent with their reported sen- sitivity to 3D configurations (Lescroart and Gallant] 2019). Depth correlations further emphasize differences between scene-selective regions, with RSC showing higher correlation with deeper depth values. This could potentially reflect a role for RSC coding perceived egocentric distances (Persichetti and Dilks, 2016). Together with section 4.2 these results demonstrate the validity of BrainACTIV as a data-driven method that reproduces known properties of visual cortex and can help formulate ine-grained new hypotheses about image properties driving brain activations.
'I is important to note that the projection of modulation embeddings to the space of CLIP image embed- dings necessarily biases the representativity of particular features towards objects in the projection set that most frequently hold these features (e.g., small size - food).
8

---

Published as a conference paper at ICLR 2025
(A)
(8)
Reference image
nes t
ment
proept box
RO1
1
RO1 2
minibar
OPA
built- in bed
OFA
FFA
dr awe sleeper
Peeppt on casines
blai
sushi bar cafe
OWFA
WIFA
wet bar room i lounge
PPA
reception
desk
poptroos Gataraaee ca La Lounge
cockta
cash
bar
OPA
PPA
poolroom room
walk- k-kthhouu h
RSC
FFA
walk up we lounge
KOWFA OPA
VNITA
OOWFA PPA
VWFSfe OOVIA
OPA
Ofte
APA
axtin
sun par se stairwell
-9S -8.0 -5.5 -5.0 Entropy difference
--4-2--1 6-1.2-08 Depths differance
-10.5 Curvetures diffrrecce
-255 -220
concourse
Figure 6: Differences between similar ROIs. (A) Top nouns emphasize differences between scene- selective ROIs. (B) (Top) Example variations accentuating differences for three ROI pairs: OFA- FFA, OWFA-VWFA, OPA-PPA. (Bottom) Differences in mid-level feature values (with respect to reference image) of each ROI, averaged and z-scored over test images.
4.4 IDENTIFYING DIFFERENCES BETWEEN SIMILAR ROIS
BrainACTIV can also be used to identify what distinguishes one region from another beyond cate- gory selectivity, important step toward understanding broader functional organization principles in the visual cortex. We perform a top-nouns analysis (Figure (A)) to identify the WordNet nouns whose presence increases the most already highlights differences between the three scene-selective regions: OPA prefers local scene el- ements, while PPA and RSC prefer more global views (Kamps al.) 2016 Henderson al., 2008); additionally, RSC prefers corridor-like scenes, potentially related to its role in navigation (Mitchell al. 2018). These results further demonstrate how BrainACTIV improves upon BrainDiVE and NeuroGen while retaining their fine tine-grained distinction capabilities.
maximization results for |subsection 4.2] This analysis
However, BrainACTIV can also be adapted to directly generate new hypotheses about ROIs with similar seletiry-selectivitty through accentuation of differences between ROIs by manipulating reference image. To demonstrate this, we here target three pairs of ROIs that are selective to the same category: face-selective OFA and FFA, word-selective OWFA and VWFA, and place-selective OPA and PPA. For each pair, we create accentuation embeddings by subtracting the modulation embeddings of each ROI from another. We randomly sample 50 images from the test set and manipulate each of them with interpolation values a E {0.1, 0.2, ..., 0.9, 1) toward the accentuation embeddings. For SDEdit, Figure 6) (B) displays example variations for each ROI pair and measured differences in entropy. |ppendixxsubseetion
exponential warm-up schedule up to y 0.6 for a 1.
depth, and curvature. Additional examples can be found in Appendix (subsection A.5] The features accentuated on each side represent preferred visual properties that distinguish the regions.
AraiV suggests higher preference for text in OFA and a higher preference for faces in FFA, despite both being face-selective. For OWFA, we identify a higher preference for cluttered coarse- grained elements, evidenced by higher entropy values; VWFA shows a preference for text on small items. Finally, OPA and PPA differ in sensitivity to depth as analyzed in subbsection 4.2) These new hypotheses can be validated by using these images as experimental stimuli in new fMRI studies.
4.5 PRODUCING NOVEL EXPERIMENTAL STIMULT
BrainACTIV generates synthetic stimuli that differ from real reference image along hypoth- esized tuning dei-vee in data-driven manner-for a particular ROI. These paired images be employed stimuli for novel neuroscientific experiments (Figure 7 (A)). To facilitate its use for researchers and illustrate the available design choices, we briefly show the effect our two axyerparrantees--ineppppatai
a and SDEdit 7-on the resulting images (Figure (B). Both hyperparameters decrease semantic similarity and structural fidelity to the reference image (as evi-

---

Published as a conference paper at ICLR 2025
(A)
(8)
(C)
Example: maximization of PPA
(Hypathesized] Tuning axis of RC
SDSley
CJIP coaine wwn.aarrryyyy... 211
stimalas
strealas
Contrast activations
Interpolation
Figure 7: Effect of a and y for novel stimuli. (A) Schematic of the use of BrainACTIV on neuro- scientific experiments. (B) Different values of and 7 present choice between structural fidelity and semantic variation. (C) Predicted activations are more strongly modulated when semantic con- tent is closer mmax or Zmini lower and yy result in higher semantic similarity and structural fidelity to the reference image.
denced by lower cosine similarities and higher LPIPS metries, respectively) (Figure (C)). At the same time, we observe that lower semantic similarity and lower structural fidelity result in higher changes in predicted activations. This to expected from the design of our modulation embed- dings. However, two distinct alternatives exist: choosing lower a and higher y results in variations that mostly retain the semantic content of the reference image while the spatial arrangement differs (depending how well CLIP can capture it). Conversely, higher and lower favor the low-level structure of the reference image while more strongly varying the semantic content.
5 DISCUSSION
We introduced BrainACTIV,, a method for modulating predicted brain responses through image ma- nipulation. To our knowledge, are the first work to generative models-paatticllate particularly, diffusion models-to manipulate reference images with the goal of maximizing or minimizing activations in the human visual cortex. Our results show the potential of approach for fine-grained and liable identification of visuo-semantic properties preferred by specialized neural populations. This information can be used formulate new hypotheses about visual representations in the brain.
We propose that our generative framework can be employed by neuroscientists to design precisely controlled and innovative experimental paradigms to disambiguate the role of low-, mid- and high- level features, whose inherent correlations in natural images complicates the ability to isolate their effect on brain responses (Malcolm al., 2016 Lescroart et al.] 2015). We here primarily demon- strate our approach on brain regions with known ditegory-selectivve
but also explore earlier visual regions and anterior IT (see Appendix (subsection A.6 and (subsection A.7) to highlight how Brain- ACTIV could help interpret 'no-mans land' regions of cortex (Bao et al. 20200) whose functional specialization is less well understood. Future work can explore BrainACTIV manipulation frame- work in alternative stimulus modalities, such natural language (Luo al. 2024 Tuckute al.) 2024). Finally, given BrainACTIVs reduced computational need relative to prior work, we believe holds potential for exploration of selectivity in closed-loop paradigms where activations are con- tinuously updated along an optimization trajectory (e.g. Ponce et al. 2019).
Our method has some limitations. First, because we employ pretrained models for image synthesis and representation, our results are subjected to biases in their training data. These biases might over- represent certain categories through correlations with specific image features, producing misleading results. We encourage future work to use fine-tuned models and domain-specific representation spaces to explore finer- finer-grained selectivity within smaller specialized cortical regions. Second, our work relies on brain encoders to validate the effective modulation of brain activity. While we have taken measures to ensure the robustness of our results, future work should validate BrainACTIV's predicted activations against novel brain recordings. In summary, BrainACTIV unlocks the possibil- ity to test existing and generate novel hypotheses about neural representations in visual cortex using brain-guided image diffusion with structural control.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
The presentation of this paper at the conference was financially supported by the Amsterdam ELLIS Unit (European Laboratory for Learning and Intelligent Systems).
REFERENCES
Adeli, H., Minni, S., and Kriegeskorte, N. (2023). Predicting brain activity using transformers. bioRxiv preprint.
Allen, E. J. et al. (2022). A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25: 16--126.
Bao, P., She, L., McGill, M., and Tsao, D. Y. (2020). A map of object space in primate inferotem- poral cortex. Naurr, 583(7814):103--10)
Bashivan, P., Kar, K., and DiCarlo, J. (2019). Neural population control via deep image synthesis. Science, 364.
Bhat, S. F. et al. (2023). Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint.
Bracci, S., Ritchie, B., and de Beeck, H. O. (2017). On the partnership between neural represen-
tations of object categories and visual features in the ventral visual pathway. Neuropsychologia, 105:153-164.
Chen, Z. et al. (2023). Seeing beyond the brain: Masked modeling conditioned diffusion model for human vision decoding. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Cheng, A., Walther, D. B., Park, S., and Dilks, D. D. (2021). Concavity as a diagnostic feature of visual scenes. Neurolmage, 232: 17920.
Cichocki, A. et al. (2009). Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE transactions on fundamentals of electronics, communications and computer sciences, 92:708-721.
Conwell, C. et al. (2023). What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? bioRxiv preprint.
Çukur, T., Huth, A. G., Nishimoto, S., and Gallant, J. L. (2016). Functional subdomains within scene-selective cortex: parahippocampal place area, retrosplenial complex, and occipital place area. Journal of Neuroscience, 16247)310
Dado, T. et al. (2024). Brain2GAN: deature-disentangled neural encoding and decoding of visual perception in the primate brain. PLoS computational biology, 20.
Dosovitskiy, A. al. (2021). An image worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR).
Downing, P. E., Jiang. Y., Shuman, M., and Kanwisher, N. (2001). A cortical area selective for visual processing of the human body. Science, 293:2470-2473.
Dwivedi, K., Bonner, M. F., Cichy, R. M., and Roig, G. (2021). Unveiling functions the visual cortex using task-specific deep neural networks. PLoS compulational biology,
Engel, S., Zhang, X., and Wandell, (1997). Colour tuning human visual cortex measured with functional magnetic resonance imaging. Nature, 388:68-71
Epstein, R. and Kanwisher, N. (1998). A cortical representation of the local visual environment. Nature, 392:598-601.
Gifford, A. T. and Cichy, R. M. (2024). The neural encoding dataset. In preparation.
11

---

Published as a conference paper at ICLR 2025
Goetschalckx, L., Andonian, A., Oliva, A., and Isola, P. (2019). Ganalyze: Toward visual definitions of cognitive image properties. In Proceedings of the ieee/cvf international conference on computer vision, pages 5744-5753.
Goodfellow, 1. al. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, 27:139-144.
Grill-Spector, K. and Weiner, K. S. (2014). The functional architecture of the ventral temporal cortex and its role in categorization. Nature Reviews Neuroscience, 15(8):536 548.
Groen, L. I., Silson, E. H., and Baker, C. I. (2017). Contributions of low-and high-level properties to neural processing of visual scenes in the human brain. Philoxophical Transactions of the Royal Society B: Biological Sciences, (71((17))2006011))
Groen, I. I., Silson, E. H., Pitcher, D., and Baker, C. I. (2021). Theta-burst tms of lateral occipital cortex reduces bold responses across category-selectiv areas in ventral temporal cortex. Neu- rolmage, 230: 117990.
Gu, 7. et al. (2022). NeuroGen: Activation optimized image synthesis for discovery neuroscience. Neurolmage, 247:118812
Hebart, M. H., Zheng, C. Y., Pereira, F., and Baker, C. I. (2020). Revealing the multidimensional mental representations of natural objects underlying human similarity judgements. Nature Human Behaviour, 4:1 173-11131 185.
Henderson, J. M., Larson, C. L., and Zhu, D. C. (2008). Full scenes produce more activation than close-up scenes and diagnniic objects parahippocampal and retrosplenial cortex: an fmri study. Brain and Cognition, 66:40-49.
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-68511
Huth, A. G., Nishimoto, S., Vu, A. T., and Gallant, J. L. (2012). A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76: 1210-22-2
Ilharco, G. et al. (2021). OpenCLIP. Zenodo (software package).
Janini, D., Hamblin, C., Deza, A., and Konkle, T. (2022). General object-based features account for letter perception. PLoS computational biology, 18115220
Kamps, S. et al. (2016). The occipital place area represents the local elements of scenes. Neu- rolmage, 132:417-424.
Kanwisher, N., McDermott, J., and Chun, M. M. (1997). The fusiform face area: A module inhuman extrastriate cortex specialized for face perception. Journal of Neuroscience,
Khosla, M. et al. (2021). Cortical response to naturalistic stimuli is largely predictable with deep neural networks. Science Advances, 7.
Khosla, M. et al. (2022). A highly selective response to food human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 32:4159-4171.
King, M. L. et al. (2019). Similarity judgments and cortical visual responses reflect different prop- erties of object and scene categories in naturalistic images. Neurolmage, 197:368-38 -82..
Lescroart, M. D. and Gallant, J. L. (2019). Human scene-selective areas represent 3d configurations of surfaces. Neuron, 101(1):18-1992.
Lescroart, M. D., Stansbury, D. E., and Gallant, J. L. (2015). Fourier power, subjective distance, and object categories all provide plausible models of bold responses in scene-selective visual areas. Frontiers computational neuroscience, 9:135.
12

---

Published as a conference paper at ICLR 2025
Lewis, M. et al. (2020). BART: Denoising srquence-t-sequeeee pre-training for natural language generation, translation, and comprehension. Proceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 7871-7880.
Lin, T.-Y. al. (2014). Microsoft COCO: Common objects in context. European Conference on Computer Vision, pages 740-755.
Luo, A. F., Henderson, M. M., Tarr, M. J., and Wehbe, L. (2024). BrainSCUBA: Fine-graie -g-iina captions
natural captions of visual cortex selectivity. International Conference on Learning Rep- resentations (ICLR).
Luo, A. F., Henderson, M. M., Wehbe, L., and Tarr, M. J. (2023). Brain diffusion for visual ex- ploration: Cortical discovery using large scale generative models. 37th Conference on Neural Information Processing Systems (NeurIPS).
Malcolm, G. L.. Groen, and Baker, C. I. (2016). Making sense of real-world scenes. Trends in cognitive sciences, 20(1 1):843-866.
Margalit, E., Jamison, K. W., Weiner, K. S., Vizioli, L.,, Zhang, R.-Y., Kay, K. N., and Grill-Spector, K. (2020). Uttaa-high-resooluton fmri of human ventral temporal cortex reveals differential repre- sentation of categories and domains. Journal of Neuroscience, 40(15):3008-300 3024.
Mazer, J. A. et al. (2002). Spatial frequency and orientation tuning dynamics in area v1. Proceedings of the National Academy of Sciences, 19:1644-16500
McCandliss, B. D., Cohen, L., and Dehaene, S. (2003). The visual word form area: Expertise for reading in the fusiform gyrus. Trends in Cognitive Sciences, 7:293-299.
McCarthy, G., Puce, A., Gore, C., and Allison, T. (1997). Face-specific processing the human fusiform gyrus, Journal of Cognitive Neuroscience, 9:605-610.
Meng, C. et al. (2022). SDEdit: Guided image synthesis and editing with stochastic differential equations. International Conference on Learning Representations (ICLR).
Menon, R. S., Ogawa, S., Strupp. J. P., and Uğurbil, K. (1997). Ocular dominance in human vl demonstrated by functional magnetic resonance imaging. Journal of Neurophysiology, 77:2780- 2787.
Miller, G. A. (1995). Wordnet: A lexical database for english. Communications of the ACM, 38:39- 41.
Mitchell, A. S. et al. (2018). Retrosplenial cortex and its role in spatial cognition. Brain and Neuroscience Advances, 2:1-13.
Naselaris, T., Kay, K. N., Nishimoto, S., and Gallant, J. L. (2011), Encoding and decoding in fmri. Neurolmage, 56:400-410.
Nichol, A. et al. (2022). Glide: Towards photorealistic image generation and editing with text-guided diffusion models. International Conference on Machine Learning.
Oquab, M. et al. (2023). DINOv2: Learning robust visual features without supervision. arXiv preprint, 204.00333...71193.
Ozcelik, F. and VanRullen, R. (2023). Natural scene reconstruction from fMRI signals using gener- ative latent diffusion. Nature Scientific Reports, 13.
Papale, P., De Luca, D., and Roelfsema, P. R. (2024). Deep generative networks reveal the tuning of neurons in and predict their influence on visual perception. bioRxiv, pages 2024-10.
Peelen, M. V. and Downing, P. E. (2005). Selectivity for the human body in the fusiform gyrus. Journal of Newrophysiology, 93:603-608
Persichetti, A. S. and Dilks, D. D. (2016). Perceived egocentric distance sensitivity and invariance across scene-selective cortex. Cortex, 77:155-163.
13

---

Published as a conference paper at ICLR 2025
Pierzchlewicz, P., Willeke, K., Nix. A., Elumalai, P., Restivo, K., Shinn, T., Nealley, C., Rodriguez, G., Patel, S., Franke, K., et al. (2024). Energy guided diffusion for generating neurally exciting images. Advances in Neural Information Processing Systems, 36.
Ponce, C. R. al. (2019). Evolving images for visual neurons using deep generative network reveals coding principles and neuronal preferences. Cell, 177:999-1009.
Prince, et al. (2024). Dissecting visual population codes with brain-guided feature accentuation. Cognitive Computational Neuroscience Conference.
Radford, A. et al. (2021). Learning transferable visual models from natural language supervision. International Conference Machine Learning-
Ratan Murty, N. A. al. (2021). Computational models of category-selective brain regions enable high-throughput tests of selectivity. Nature Communications, 12:55409.
Rombach, R. et al. (2022). High-resolution image synthesis with latent diffusion models. JEEE/CVF
Conference on Computer Vision and Puttern Recognition (CVPR), pages 10674-10685.
Rosenholtz, R., Li, Y., and Nakano, L. (2007). Measuring visual clutter. Journal of vision, 7(2):7~ 17.
Saharia, C. et al. (2022). Photorealistic text-to-image diffusion models with deep language under- standing. 36th Conference on Neural Information Processing Systems.
Sarch, G. H., Tarr, M. J., Fragkiadaki, K., and Wehbe, L. (2023). Brain Dissection: IMRI-trained Networks Reveal Spatial Selectivity in the Processing of Natural Images. 37th Conference on Neural Information Processing Systems (NeurIPS).
Schuhmann, C. al. (2022). LAION-5b: An open large-scale dataset for training next generation image-text models. Thitty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Scotti, S. et al. (2023). Reconstructing the Mind's Eye: IMRI-to-Image with Contrastive Learning and Diffusion Priors. 37th Conference on Neural Information Processing Systems (NeurIPS).
Silson, E. H., Groen, I. I. and Baker, C. I. (2022). Direct comparison of contralateral bias and face/scene selectivity in human occipitotemporal cortex. Bruin Structure and Function, 42054)1440--1421
Song, Y. et al. (2020). basee-based generative modeling through stochastic differential equations. International Conference on Learning Representations (ICLR).
St-Y G. and Naselaris, T. (2018). The feature- receptive field: an interpretable encoding model for complex feature spaces. Neurolmage, 180:188-202.
Tootell, B. at. (1998). Functional analysis of primary visual cortex (v1) in humans. Proceedings of the National Academy of Sciences, 95:81 95:811-98 1-817.
Tuckute, G. et al. (2024). Driving and suppressing the human language network using large language models. Nature Human Behaviour, 8:544-561.
van Gerven, M. A. (2017). A primer on encoding models in sensory neuroscience. Journal of Mathematical Psychology, 76:172-183.
Vinken, K., Prince, S., Konkle, T., and Livingstone, M. (2023). The neural code for "face cells" is not face-specific. Science Advances, 133))
Walker, E. Y. et al. (2019). Inception loops discover what excites neurons most using deep predictive models. Nature Neuroscience, 22:2060-2065.
Wang, A. Y. et al. (2023). Better models of human high-level visual cortex emerge from natural language supervision with large and diverse dataset. Nature Machine Intelligence, 5:1415- 1426.
14

---

Published as a conference paper at ICLR 2025
Williams, A., Nangia, N., and Bowman, S. (2018). A broad-cod coverage challenge corpus for sentence understanding through inference. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long Papers), pages 1112-1122.
Ye, H. et al. (2023). IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint, arXiv:2308.06722
Yin, W., Hay, J., and Roth, D. (2019). Benchmarking zero-shot text classification: Datasets, eval- uation and entailment approach. Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCNLP), pages 3914-3923.
Zeng, B. al. (2023). Controllable mind visual diffusion model. arXiv preprint.
Zhang, R. et al. (2018). The unreasonable effectiveness of deep features as a perceptual metric. Computer Vision and Pattern Recognition Conference (CVPR).
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 CATEGORY REPRESENTATIONS IN CLIP
We build sppecit CLIP embeddings quantify the presence of each category (faces, hands, feet, people, animals, plants, food, furniture, tools, clothing, electronics, vehicles, land- scapes, buildings, rooms, and text) an image through cosine similarity with the image embedding. First, we gather large list of Noouss 17, 086 concrete nouns from WordNet (Miller 1995 by collecting hyponyms the following synsets:
amphibian.n .3
fish. .1
publ icaiiiaaiinn on on.i.onnn..
articie.n.02
food.n. 02
reptile. en.01
bird.n.01
instrumentalityy.ncoo
room.
body.of.water.n. n. 01
land.n.C4
sign.n.02 02
building.n.01
person.n.0l
vehicle.n.00
commodity.n.C1
placental.n.0
way.n.
correspon.dence.nog 01
plant.n.02
written.record.nn
external atternal.body.aatt.com facility.n.01
plaything.n.01
geological..oommtion..
Then, we perform zero-shot classification of each noun into one of the categories using baroookkbartttaaraeee trained on the MultiNLI dataset (Williams et al. 2018 Yin et al., 2019). Instead of using the cate- gory names as labels, we build custom labels:
a version of the language model BART (Lewis et al. (0200)
faces: "related to faces, eyes, nose, mouth"
hands: "related hands, arms, fingers"
feet: "related to feet, legs, toes"
people: "related to people, humans, persons"
animals: "related to animals, creatures, fauna"
plants: "related to plants, greenery, flora"
food: "related to food, meals, eating"
furniture: "related to furniture, household items"
tools: "related to tools, equipment, instruments"
clothing: "related clothing, textiles, garments"
electronics: "related electromies, gadgets, devices"
vehioles: "related vehicles, transportation, travel"
landscapes: "related to natural areas, landscapes, outdoors"
buildings: "related to urban areas, buildings. structures" "related indoors, rooms, interiors"
text: "related written text, signs"
The resulting class probabilities gathered in matrix Ypob [0, No x16 where each sums
up to 1. We weigh the probabilities by the salience of each category with respect to the rest for each noun to obtain Ysal:
Y Yproblass
Y 1s (Ypobles
Σn (Yyroole,s
Next, we compute embeddings for each of the nouns using CLIP's text encoder. make these more robust, average the embeddings obtained through 18 prompt templates (e.g.. "a photo 0 or "a good photo of the ()-") used originally by CLIP for image classification (Radford et al. 2021 We normalize these embeddings and gather them matrix Znouns E R. N.Num XTU24
Finally, we use a regularized linear regression model on Znouns to predict Y.. and analytically derive the weights Wnouns Dx16 Each column in the weight matrix then functions as our representation for each category. We notice that the text category is difficult to represent through this method; therefore, instead compute its embedding by encoding the phrase "text on an object" using each of the 18 prompt templates and averaging them. Figure 8) displays each category's 's representative examples from the Natural Scenes Dataset (NSD) Allen et al. 22022). as well as salient WordNet nouns.
16

---

Published as a conference paper at ICLR 2025
Faces
Hands
Feet
People
tist g love thumb
ESTCORE bOCKEY
huma bumanist
man
TORERT FROM
human head
person
Animals
Plants
Food
Furniture
conver tible Scauis AIEUSR 15DADE
sinne BLaTe
scorch Soorru Tincon
cradle
Tools
Clothing
Electroiccs
Vehicles
detteake saed shart outton coa botton Lironing og
aa Wheel
Car AN syrote
witte Cha
Text
Natural outdoors
Man-made outdoors
Man-made indoors
glen 1py trail dune park bench
Skyscrape BURN
addresss poster
Sweeper CC blockbuster
DOUTRAL bear A0W
Tanor hall
Figure 8: Representative examples (high cosine similarity) from NSD for each category, together with top WordNet nouns as classified by facebook/bart -argte-m-i--- mnli.
17

---

Published as a conference paper at ICLR 2025
A.2 BRAIN ENCODER PERFORMANCE AND DETAILS
Encocor
Subject
NA DATA 776
WP
DINO Encoder
MRE Encoder
CLIP Encocer
Da terorcce
Subject
DINO-VIT Encaber
[nRT Encudee
CLIP EnLUCH
Subject
DINOVIT Encocer
Subject
Eepatred
(WN
P6. Ne WIT O3A
MA WPF
DIN Encoder
MRR Encoder
Encocer
Expli oo teforce
Subject
DINO-VIT Encoder
MRRE Encoder
CLIP Encocer
Subject
10.
DING-VIT Encocer
(wRF
Encocer
Subject
D85 1WA
PM
RA UK WMA
DINO-VIT Encocer
MRR Encoder
Encocer
Subject
Expahhed
Figure 9: Encoding performance (explained variance) of DINO-VIT fwRF, and CLIP encoders for each subject and each region of interest. Black lines indicate the estimated noise ceiling from Allen al. 2022) (maximum over ROI voxels).
DINO-VIT Encoder. Adeli et al. (2023) explored the use of a pretrained 12-layer DINOv2 model Oquab al. 2023) feature extractor for single-layer transformer that learns ROI-specific
18

---

Published as a conference paper at ICLR 2025
queries, which are later linearly mapped voxel activations. They train 22 models per subject, differing on the choice layer used to extract features from DINOv2 and targeted ROIs. Finally, they employ ensemble approach to produce their voxel-wise predictions. We simplify this process by fixing single architecture and training it once for each category-selective ROI (9 models total extract the output of each of the 12 layers in DINOv2 and pass each of them through a separate single-layer vision transformer (VIT) (Dosovitskiy et al. 2021). The output CLS token of each VIT is used by multilayer perception (MLP) predict voxel-wise activations for the ROI. The outputs of the 12 MLPs are aggregated through a learnable linear layer to produce our final predictions. All models are trained for 15 epochs with early stopping, using learning rate of le-4 and batch size of 64 samples.
per subject). Specifically,
RwRF Encoder. The Neural Encoding Dataset (NED) Giffoor and Cichy, 2024) provides pre- trained brain encoders for the NSD dataset (Allen al., 2022). These encoders are feature-weighte receptive field encoding models (St-Yves and Naselaris 2018), neural networks trained end-to-end to predict neural responses.
19

---

Published as a conference paper at ICLR 2025
A.3 NSD IMAGE SUBSETS
We identify six mutually exclusive subsets of images in NSD Figure 10p to enforce diversity in our validation experiment. We filter the pixel-wise category annotations from COCO (Lin et al. 2014) as specified in Figure each row also indicates the size of each subset for training and test sets.
Wild animals
Birds
Vehicles
People sports
Furniture
Figure 10: Example images per subset.
Subset
Size (training set)
Size (test set)
COCO Categories
include: horas sheep cow elephant, zolr graffe
Wild anmmals
773x30
150
person, bicycle, motor cyclk, amplaan, trai, tuick, hanans, spple sundwrcch arange
canoe dog, pizza.
include: hird
Beds
129474
24
exclude:
sandwich,
broocoll cancot hot dog, donut, cake, home, sheap, elephent
grafie,
includer bicycle, motorcycle, axplane, tram, truck, bost
Venicles
123
eoclude: par banana, apple, senderch
broocali carde, hat
horse.
sheep. cow rphaart bear zebra giralfe, bind cat
include: person frahee, antodtnnnd, sports ball, kite, bosebal besebea glane skatebnard surfboard,
People sports
R30+51
101
exolude: horse, shoop. cow, elephant bear zebra, giaafie, aid dog bicycle, motor cyttrrycl,, amplane, bus, tmin trcck bost bananss apple, sandwicch orange, brncool, canot hot dog pirza, dinutt, cake
include: banonn, apple candwich, orange broocolt, canoot hot dog, pizza, donut cake
Fond
509+33
59
enckuder person, horee allepp cow elephenn beet zebra graffee bird eat, dog, bloycle oa, motoroycle, atrlaan, bus, trai, truck, boat
include: cheir, couch, potted
Famiture
883+40
108
exclude: perzon, horze, sheep, cow clephant, best zebra, graffe, bind est dog, braycle, car motorcycle. airptars, bux train, track, bowk, banee apole, sandwichh one go, broccol, cartot dog pizza, donut, cake, dining table,
Figure 11: Overview of the size and COCO categories used to define each subset. Upper rows (in the rightmost column) indicate categories present all images. Bottom rows indicate categories that were explicitly excluded.
20

---

Published as a conference paper at ICLR 2025
A.4 ADDITIONAL IMAGE VARIATIONS
FUSIFORM FACE AREA (FFA)
ST
$2
N
$3
Figure 12: Example image variations (two per subject). Middle column: reference images. Left: minimization of FFA. Right: maximization of FFA.
21

---

Published as a conference paper at ICLR 2025
EXTRASTRIATE BODY AREA (EBA)
SB
Figure 13: Example image variations (two subject). Middle column: reference images. Left: minimization of EBA, Right: maximization of EBA.
22

---

Published as a conference paper at ICLR 2025
VISUAL WORD FORM AREA (VWFA)
S5
Rosnt
Rotes
S6
S8
Figure 14: Example image variations (two per subject). Middle column: reference images. Left: minimization of VWFA. Right: maximization of VWFA.
23

---

Published as a conference paper at ICLR 2025
OCCiPiTaL PLACE AREA (OPA)
S5
36
S8
Figure 15: Example image variations (two per subject). Middle column: reference images. Left: minimization of OPA. Right: maximization of OPA.
24

---

Published as a conference paper at ICLR 2025
PARAHIPPOCAMPAL PLACE AREA (PPA)
58
Figure 16: Example image variations (two per subject). Middle column: reference images. Left: minimization of PPA. Right: maximization of PPA.
25

---

Published as a conference paper at ICLR 2025
RETROSPLENIAL CORTEX (RSC)
S6
S8
Figure 17: Example image variations (two per subject). Middle column: reference images. Left: minimization of RSC. Right: maximization RSC.
26

---

Published as a conference paper at ICLR 2025
A.5
ADDITIONAL EXAMPLES: DIFFERENCES BETWEEN REGIONS
65
Figure 18: Example image variations accentuating one region (left) from another (right) in refer- ence image (middle).
27

---

06 d2 04 as as 10 Published as a conference paper at ICLR 2025
A.6 EARLY- AND MID-LEVEL ROIS
We showcase the use of BrainACTIV on early- to mid-level regions of interest. In particular, we target V1, V2, V3, and V4 for Subject These regions are known to be selective for low-level image properties such as orientation (Tootell et al.) 1998), ocular dominance (Menon et al. 1997p. color (Engel et al. 1997). and spatial frequency (Mazer et al. 2002). We follow similar procedure as outlined in subsection 4.2 Figure 2I shows example variations for each ROI, Wigure 19 shows successful modulation of predicted activations in these ROIs. Because BrainACTIV employs CLIP's image space to represent and modify the reference image's s content, the manipulations mainly display semantic changes. However, these changes likely reflect semantic associations or co-occurrences with the low- and mid-level properties preferred by the ROIs, learned by CLIP during pre-training. For example, the appearance light bulbs and cluttered elements VI, or colorful objects in V4. Hence, conclusions and hypotheses formulated from these results must be cautious regarding semantic selectivities.
Figure disisplays measured low- and mid-leved image features for both optimal endpoints (averaged over all reference images). importantly, we identify changes in color saturation and entropy (texture) that are not present for high-level ROIs (see subsection A.8
08
Predicted activation (18000-311 encoder))
V2c
06
Vic
predicted artivation (Talt encoder)
07-
02-
00
-0.1
-044
-02-
-0.3-
-0.6
18
08
24
62
50
60
0A
03
08
1. 10
088
0.6
0.1 022
Interpolation
Interpolation
Interpolation Seeee
Interpolation Kima
Krm
Figure 19; ROI activations predicted by DINO-VIT encoder (left) and fwRF encoder (right) function of interpolation a with each modulation embedding, averaged across test images.
V1
V3
brightnesa tress
soturat
wamth
11. 0.755
0.75
1.5 6.75
0.75 15
0.75
0.75
355 4.75 (standard deviations)
(standard
(standard deviations)
Legend:
Maximal activation
Minimal activation
Figure 20: Quantification of average low-level and mid-level image features for variations at maxi- mal (black) and minimal (red) activation.
28

---

Published as a conference paper at ICLR 2025
Reference
Maximize
Minimize
Referrnnc
Mazimize
V2
Mininze
Refference
Mazimize
W3
Minimae
Reference
Maximize
V4
Minimize
Figure 21: Example reference images (top) with their corresponding maximization (middle) and
minimization (bottom) a = results. DINO-ViT predictions are displayed next to each image.
29

---

Published as a conference paper at ICLR 2025
A.7 EXPLORATION ANTERIOR IT CORTEX
We showcase the use of BrainACTIV to explore brain representations in regions for which selectiv- ity is less well-understood than that of the areas targeted in subsection 4.2 Partieularly, we target the anterior IT cortex. To do so, we first identify four components in the data by performing non- negative matrix factorization (NMF) (Cichocki al., 2009p on the NSD data matrix of anterior IT (number of images x number of voxels) for subject 5, similarly to Kholaa et al. (2022) 's exploration of the ventral visual cortex. This operation yields two lower-dimensional matrices W and H whose product approximates the original data matrix. W represents the relative contribution of each iden- tified component to each the voxels in anterior IT. H represents the response of each component to all visual stimuli.
Figure 22 displays each component in W overlayed on flat cortical surface map. Higher values indicate greater contribution of each component to the response profile of voxel. Importantly, the four components define subdivisions anterior IT that distinctly represent visual stimuli. Once we have identified the four components, for each component and computing a modulation vector, as outlined subseectoo 3.1 Then, we perform image manipulations in subsection 4.2) Example results displayed in Figure 24
define sub-ROIs by taking the 100 most relevant voxels
Component
Component 2
Component 3
Component
Figure 22: Relative contribution of each NMF component to the voxels in the anterior IT cortex.
Component 1
Component 2
1ish loaf kaiser roll pasta cinnamon roll
villa
Playhous
playhouse
shate downcast
wans
a
base
sausage rollrollmops
rolled
bowling alley
building
toad-in -e--in-thee
strudel the-hole
snoked eel
Sn
rocket
O
channey
ruin
baked potato
Frncch pastry cookie potatobear cl aw roll cross bun
smoked mackerel
abattoir
outbui
Iding
house
magazine
re
refri eefrigeras
ingerttrrrr
g
pariison
ransturter
bu
jacket potato
bloater onion
pillbox guar guardhous dhous esignal
Component 3
Component 4
reveler
raver
bunny show
or organa
organ-gria
-ean-grinder
r showman
rino
g
irl
bill
prompt
pal
anguinn
bOX
minstrel
barker
super
supernumerare
bar
surrit
grruple
rapper
pr
prompt
box
stop
b-girl
banana
greenrood
tobacconist
show
end
is
bill
barkettler
cinema
man
settler
poster
cash
cheer
dancer stage door geek
leader earty giri
sign painter snake
fakar
diessper
taxi
snake
charmer
Figure 23: Top-nouns analysis of the image manipulations for each component illustrates
what each
of the corresponding sub-ROIs is suggested be most responsive
30

---

Published as a conference paper at ICLR 2025
To characterize the selectivity suggested by these results, perform top-nouns analysis in Fig ure 23 Inspection these nouns together with manipulated images suggests the existence of sub- divisions within anterior IT that responsive to food, places, people, and text/objects. Interest- ingly, the visual characteristics of component 2 and 3 are different from what we observed for FFA, PPA, OPA, and RSC. For example, we to gender-specific separation for people. These results can be used to formulate new hypotheses about neural representations in anterior IT, which must be tested through neuroimaging studies.
Componen
Component 2
Component 3
Component
Figure 24: Example manipulations for each the sub-ROIe computed for anterior IT. The middle column shows reference images; minimization results are the left; and maximization results on the right.
31

---

Published as a conference paper at ICLR 2025
A.8
DIFFERENCE IN LOW-LEVEL IMAGE FEATURES
FFA
EBA
VIFA
0PA
PPA
RSC
00urtti
H
155
07s
Defference
Diffenece
Deffrrnce
Differeace
Defference
Offferecce
deviations)
Istandard deviations)
Istandard
Legend:
Maximal activation
Minimal
Figure 25: Quantification of low-level properties (brightness, saturation, color warmth) shows that these remain unaffected by variations at maximal (black) and minimal (red) activation, unlike mid- level features.
32

---

Published as a conference paper at ICLR 2025
A.9 SIMILARITY BETWEEN MODULATION EMBEDDINGS
BEFORE AVERAGING OVER SUBJECTS
09
08
00
03
OPA
PPA
RSC
105
116
1.00
000
030
080
641
0.5
065
0.0
Supject
Figure 26: Cosine similarity between subject-specife modulation embeddings zmax for each ROI before averaging over subjects.
AFTER AVERAGING SUBJECTS
FFA
EBA
VWFA
0399
0/95
0.66
0,906
DAES
0.994
0:975
4.jeet
Suzjert
OPA
PPA
RSC
0350
2593
0.96
2507
0,004
3494
Subject
Figure 27: Cosine similarity between subject-specific modulation embeddings Zmax for each ROI after averaging over subjects.
33

---

Published as a conference paper at ICLR 2025
A.10 SIMILARITY ACROSS RANDOM SEEDS
FUSIFORM FACE AREA
Reference image
16436
51197
Figure 28: Example manipulations maximizing FFA (a 1) show great similarity across random seeds the diffusion model, emphasizing that BrainACTIV isolates the effect brain optimality.
PARAHIPPOCAMPAL PLACE AREA
Reference image
Figure 29: Example manipulations maximizing PPA (a 1) show great similarity across random seeds of the diffusion model, emphasizing that BrainACTIV isolates the effect of brain optimality.
34

---

Published as a conference paper at ICLR 2025
A.11
ILLUSTRATIVE NSD EXAMPLES PER MID-LEVEL FEATURE
Brightness
Warmth
Saturation
Maximal
Minimal
Entropy
Depth
Curvature
Maximal
Minimal
Surface normal
Surface normaly y
Surface normal z
Maximal
Minimal
Figure 30: Examples from the NSD dataset displaying maximal and minimal pixel-averaged values for the different mid-level features we employ in this study.
35

---

Published as a conference paper at ICLR 2025
A.12 ADDITIONAL STRUCTURAL CONTROL BASELINES
To provide additional context on the structural control metrics for BrainACTIV (Table 1). we com-
pute these metries stimuli synthesized without brain-conditioned targeting, namely 0 (equiv- alent to simply passing the reference image through IP-Adapter, as would be commonly done for generating image variations).
We compute L2 distance and LPIPS between the reference image and the synthesized images using different values of SDEdit's 7 (examples in Figure 3T. Intuitively, the difference between these synthesized images is that = shows how CLIP "interprets" the semantic content in the refer- ence, while y has no effect from CLIP and y in-between are an interpolation between these two endpoints. Hence, all of these represent different interpretations of synthesis without brain conditioning.
Ref
0.1
0.2
0.3
0.5
0.8
4567
Figure 31: Examples for synthesis without brain conditioning using 0 and differing
Metrics in Figure 32 show that images closer to y 0 are structurally very similar the reference
image, while this similarity decreases approaches 1. In these plots, we highlight y 0.6, the maximal value used by BrainACTIV to compute results in Table
(4)
LPIPS ( (
60
0.35
0.30
40
0.25
0,20
0.15
20
0.10
10
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.2
SDEdit
SDEdit y
Figure 32: Lg and LPIPS metrics for differing values of when a = 0. Averaged over test samples.
36


Reviews:
[{"review_id": "sOFgFkytgs", "reviewer": "Reviewer_mur3", "summary": "In this paper, the authors present a method for generating maximizing or minimizing images for a brain ROI using Stable Diffusion. The method uses the NSD dataset, and builds on previous work using this dataset and diffusion models to generate activating images. Specifically, starting from a source image, the method changes this image to maximize or minimize the activation in an ROI (though the estimation of a CLIP vector characterizing the ROI from an encoding model). This makes it possible to detect the change in category representations as different regions are maximized. The method can also be used to contrast two ROIs, even ones that process the same category.", "questions": "- Regarding Weakness 1: How many images are used to compute the selectivity results (e.g. figure 4), why are those chosen? What is the predicted activity associated with the maximizing and minimizing images? What are the statistics related to these activities? Could it be used to make predictions about how such images would activate or deactivate brain regions?\n- Regarding Weakness 2: how do we understand the minimizing categories? Is it just a factor of having to step away from the preferred categories or is there truly a reduction in activity from baseline? Is this supported in the literature?\n- Regarding Weakness 3: What hypotheses can be predicted from the results comparing two regions?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The method nicely builds up on previous work (which it acknowledges well). \n- It allows a more concrete evaluation of images across ROIs due to the presence of the common starting image. \n- The formulation allows for both maximization and minimization.\n- It is possible to contrast multiple ROIs along the same brain pathway. \n- The results makes sense in terms of the discovered selectivities.", "weaknesses": "1- More quantification of the observed effects is needed, or at least more details about them. \n2- More should be discussed in terms of the meaning of the maximizing/minimizing categories and the results. It is not clear what are novel  results about the brain and what is a replication.\n3- The most novel results appear to be in figure 7. However these are not discussed in terms of significance or hypotheses.\n4- Luo et al 2024 is used in the methods but doesn't figure in the related work section. How does the current work compare with the method used there?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lSeET4VWBq", "reviewer": "Reviewer_82Gz", "summary": "At its core, the methodology employs a CLIP-based brain encoder that transforms visual inputs into corresponding brain activation measured with fMRI, with the goal to generate interpretable images that excite or inhibit certain brain areas. The proposed method, BrainACTIVE, allows for incorporating semantic information from modulation embeddings as well as lower-level image information directly into the generation process by seeding a diffusion process specific target images. The authors validate their proposed model by training separate encoders to predict the brain's activity to the synthesized images.", "questions": "- The usefulness of the top-nouns analysis is unclear to me, and the authors don't discuss the outcome of these results in the discussion section. In general, applying BrainACTIV seems very promising when applying it to areas in the visual hierarchy that are less well understood, going beyond largely confirmatory analyses. It would be great if the authors could include further discussion points how the top-noun analysis could best be utilized in the discovery of new feature or category selectivities.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper overall is very well written, clear, with a straightforward exposition and extensive experiements. Sufficient detail is provided to understand all key components of the data, model architecture, and training.", "weaknesses": "While the paper is very well presented, I'm taking issue with the related work section, the selection of experiments, and the overall novelty. Considered as a whole, the paper can be improved significantly by addressing these concerns.\n\n\n**Major Concerns**\n- *Related works*: there is a wealth of articles surrounding generative models for optimizing stimuli for neuronal data, especially using fMRI, see related literature (1, 2, 3, 4). ]The focus of the related work sections seems to lie on category selectivity in visual cortex and and diffusion models using CLIP. More emphasis on the wealth of related work could be placed to compare the proposed method to other approaches\n\n- *Comparing predictive performance*: Having separate encoding models is a clear strength of this paper. However, the predictive performance of the CLIP encoder model was not convincingly shown. Showing that also the CLIP-based encoding model has a comparable performance on the held-out test set (as shown in Fig. 9) as the DINO and fwRF would provide valuable insight.  A similarly interesting analysis would be including the CLIP-encoding model into Fig. 3 to understand the change in neuronal activity from the generating model. Lastly, it is important to demonstrate how much of the variations in the images is due to the random seed in the generator models - by training different models and measuring the similarity of the optimal images, and by measuring the variability of activations in the evaluator models (DINO and fwRF)\n\n- *Analyses of mid-level images features*: This analysis shows much promise but doesn't yield much insight. To establish the validity of these measures, it would have been important to show that the metrics are consistent for either handpicked or parametric stimuli. Similarly, these metrics could be very useful to study early- to mid-level ROIs.\n\n- *Overall novelty*: Taken together, while this approach is promising, it doesn't deliver a new insight into either category selectivity of mid-level image feature processing of visual cortex. The analyses are mostly confirmatory with respect to the current state of understanding of visual processing. The authors could take one of the so called \"no-mans-land\" ROIs and discover novel categories that drive these regions, such as anterior IT.\n\n\nLiterature:\n- (1) https://medarc-ai.github.io/mindeye\n\n- (2) https://mind-vis.github.io\n\n- (3)  https://arxiv.org/abs/2305.10135\n\n- (4)  https://proceedings.neurips.cc/paper_files/paper/2023/hash/67226725b09ca9363637f63f85ed4bba-Abstract-Conference.html", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WycFfN1typ", "reviewer": "Reviewer_Dc87", "summary": "The paper introduces a new model BrainACTIV to generate synthetic images optimizing specific brain responses from specific areas. Their model incorporates recent works on diffusion models conditioning to generate images that vary structurally with brain signal for better interpretations of brain representations and downstream analysis.", "questions": "line 323: what's the reasoning behind choosing images with responses closest to baseline activation?\nTable 1: it might help to show with image what you means by \"preserve structural fidelity\" since \"preserved\" can be a relative concept. It might even be helpful to compute the numbers for images synthesized from other papers for comparison.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is novel in tackling a common interpretability problems faced among works of synthesizing image based on brain signal. These synthetic images usually requires a lot of guesswork to confirm they are representing information from the brain. The IP-Adaptor is a clever way to induce structural biases into diffusional models so that the generated image vary reliably on the axis that encode the most salient representation that a brain areas cares about. This model has great potential to be extended to different brain data modality and it seems like it only requires a CLIP based brain encoding model for adaptation.\n\nThe second part of the paper analyzing mid-level features provides good examples of potentially making inferences from these synthetically generated images, further making use of the interpretability of the method.", "weaknesses": "The paper should perhaps incorporate predictive performance the Dino-ViT and fwRF encoder of these brain areas in Figure 3b so reader could have a better sense of how big of a difference the manipulation of images could make.\n\nMy main critique of this paper is perhaps more high level. Now that we have a model that could generate very clean optimized images for specific brain areas, can one use it for making new discoveries/inferences about how the brain represent visual information? Most of the work that synthesize images with brain responses has been proving that model can generate what we already know about the brain (e.g. Face image with FFA data) but rarely do they inform what we don’t already know. I am aware this paper might want to establish itself as a proof that this method could work in generating testable hypothesis but I am not yet convinced that it is substantially different than all the other papers in this subfield.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W0cPgYK67m", "reviewer": "Reviewer_eXwK", "summary": "The paper proposes a method for manipulating a reference image so as to selectively enhance (or decrease) the corresponding predicted brain activity in a given target region. The image manipulation is performed with pretrained diffusion models, and the brain activity prediction relies on training a brain decoder to perform regression between a large image dataset (NSD) and the corresponding brain activation patterns.", "questions": "* “we opt to average the embeddings over all subjects, excluding the subject on which predictions are made. Hence, we modulate brain activity in each subject through a signal derived exclusively from the rest of the subjects’ data”: This statement is vague enough to be interpreted in different ways (as there are multiple components in the pipeline that could be calculated over one or multiple subjects). I would suggest clarifying things with the corresponding variable names from equations (1-5).\n* On line 224, the methods description switches from computation of variables related to the target region of interest (e.g. $z_{max}$), to the introduction of the specific reference image and its manipulations $z_I$. It would help to state this explicitly.\n* Table 1 provides structural metrics for manipulated images together with a lower baseline computed from random images. I would suggest adding an upper baseline calculated from diffusion-based image variations without brain signal conditioning.\n* In Figure 6, it could be helpful to include the reference image (e.g. in the top-left corner).\n* The literature review on “optimal visual stimulus generation” misses (at least) one reference from the BrainDiffuser paper (Ozcelik et al, 2023).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The proposed image manipulation technique is non-trivial, and goes much beyond deploying an off-the-shelf diffusion system. It leverages state-of-the-art diffusion systems and adapters.\n* The manipulated images are convincing, and compatible with known selectivity in the target brain regions. \n* The technique allows to draw further subtle distinctions between brain regions that have been traditionally associated with similar category selectivity, and could be used to inform future neuroscience experiments.\n* The method allows to control the trade-off between semantic variation and structural similarity with two hyperparameters.", "weaknesses": "The paper shares weaknesses with most brain decoding studies, in the sense that the results describe predictions of a model trained from a specific dataset and a specific (combination of) deep learning model(s).  These predictions will ultimately require validation from actual experiments. However, this is well acknowledged in a paragraph on “limitations”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o93Ajv6Rsv", "reviewer": "Reviewer_5W1K", "summary": "The authors introduce BrainACTIV, a method for generating images modulated by the responses of a specific brain region, enabling interpolation between maximum and minimum activation levels. Additionally, they present two hyperparameters that regulate the semantic and structural variations of the generated images. The authors claim that this approach has the potential to provide insights into neuroscientific experiments.", "questions": "1. A comparison with related work [1-2] would be valuable if feasible.\n2. The response to brain activation seems to be influenced by the fMRI encoder (specifically the CLIP encoder used in this paper), so its performance should be addressed in the experimental section.\n3. How do the four mid-level features—entropy, metric depth, Gaussian curvature, and surface normals—contribute to advancements in neuroscience discoveries?\n4. How do the two hyperparameters regulate the trade-off between semantics and structure? Increasing either hyperparameter seems to lead to deviations from the reference image, but it’s unclear how this establishes a trade-off between the two aspects. What is the rationale behind this approach?\n\n[1] Gu Z, Jamison K W, Khosla M, et al. Neurogen: activation optimized image synthesis for discovery neuroscience[J]. NeuroImage, 2022, 247: 118812.\n\n[2] Luo A, Henderson M, Wehbe L, et al. Brain diffusion for visual exploration: Cortical discovery using large scale generative models[J]. Advances in Neural Information Processing Systems, 2024, 36.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors first achieved controllable image generation by manipulating a reference image to enhance or suppress specific target brain regions.\n2. Based on the example images, particularly the interpolated manipulated reference images in Figure 1, the proposed method appears effective, especially concerning the functional regions of interest defined by neuroscience.\n3. The article is highly readable, thanks to its clear writing and presentation.", "weaknesses": "1. The motivation needs to be better articulated. While the article emphasizes the controllable modification of the reference image, it lacks a clear rationale for this approach. What advantages does this controllable generation offer for guiding neuroscience discoveries compared to existing methods?\n2. The technical novelty is limited, as the article primarily implements IP-Adapter and SDEdit to achieve controllable modifications of reference images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)