SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table S.5: Human evaluation of the difference between face-selective ROIs. Evaluators compare groups of images corresponding to OFA and FFA; comparisons are done within GT and generated images respectively. Questions are posed as: "Which group of images has more X?"; options are FFA/OFA/Same. Results are in %. Note that the "Same" responses are not shown; responses across all three options sum to 100.
Which cluster is more...	vegetables/fruits	healthy
S1	S2	S5	S7	S1	S2	S5	S7
Food-1 NSD	17(4.3)	21(4.8)	27(5.1)	36(3.5)	28(5.8)	22(3.7)	29(6.2)	40(4.0)
Food-2 NSD	65(7.2)	56(6.4)	56(5.7)	49(3.9)	50(7.1)	47(4.9)	54(6.0)	45(4.3)
Food-1 BrainDiVE	11(7.0)	10(6.0)	8(6.6)	11(6.5)	15(6.2)	16(6.0)	20(7.2)	17(7.1)
Food-2 BrainDiVE	80(7.3)	75(8.0)	67(9.8)	64(7.4)	68(7.7)	68(7.3)	46(9.3)	51(7.8)
Which cluster is more...	colorful	far away
S1	S2	S5	S7	S1	S2	S5	S7
Food-1 NSD	19(5.5)	18(6.1)	13(2.8)	27(3.5)	32(6.6)	24(4.7)	23(6.5)	28(4.2)
Food-2 NSD	42(6.4)	52(5.6)	53(6.5)	42(6.4)	34(7.0)	39(8.1)	36(7.9)	42(7.3)
Food-1 BrainDiVE	6(3.8)	9(5.7)	11(5.7)	16(4.9)	24(6.8)	18(6.4)	27(8.9)	18(6.0)
Food-2 BrainDiVE	79(7.9)	82(6.9)	65(7.6)	61(8.9)	39(10.1)	51(9.0)	39(8.8)	40(8.8)
Table S.6: Human evaluation of the difference between food clusters. Evaluators compare groups of images corresponding to food cluster 1 (Food-1) and food cluster 2 (Food-2), with questions posed as "Which group of images has/is more X?". Comparisons are done within NSD and generated images respectively. Note that the "Same" responses are not shown; responses across all three options sum to 100. Results are in %.
Which cluster is more...	angular/geometric	indoor
S1	S2	S5	S7	S1	S2	S5	S7
OPA-1 NSD	45(7.2)	58(9.4)	49(7.7)	51(9.0)	71(5.6)	88(4.9)	80(5.1)	79(5.6)
OPA-2 NSD	13(4.0)	12(2.4)	14(2.9)	16(4.5)	7(3.2)	8(3.7)	11(3.0)	14(4.3)
OPA-1 BrainDiVE	76(7.8)	87(8.6)	88(6.6)	76(7.8)	89(5.6)	90(5.7)	90(4.7)	85(5.3)
OPA-2 BrainDiVE	12(4.9)	3(2.0)	4(1.5)	10(4.2)	7(3.2)	7(3.2)	5(2.1)	8(2.4)
Which cluster is more...	natural	far away
S1	S2	S5	S7	S1	S2	S5	S7
OPA-1 NSD	14(3.8)	3(2.0)	9(4.1)	10(2.8)	10(2.4)	1(0.9)	6(2.9)	8(2.4)
OPA-2 NSD	73(3.4)	89(7.4)	71(6.4)	81(6.1)	69(4.6)	93(3.8)	81(6.5)	85(5.5)
OPA-1 BrainDiVE	6(3.2)	6(1.5)	9(3.6)	6(2,9)	1(0.9)	3(2.8)	3(2.8)	8(5.6)
OPA-2 BrainDiVE	91(5.7)	91(3.6)	83(6.9)	90(5.5)	97(2.8)	92(6.6)	91(5.6)	88(7.4)
Table S.7: Human evaluation of the difference between OPA clusters. Evaluators compare groups of images corresponding to OPA cluster 1 (OPA-1) and OPA cluster 2 (OPA-2), with questions posed as "Which group of images is more X?". Comparisons are done within NSD and generated images respectively. Note that the "Same" responses are not shown; responses across all three options sum to 100. Results are in %.
12 Brain encoder  $R^2$
Figure S.19: Visualization of  $R^2$  on test set images. We evaluate  $R^2$  on the  $\sim 1000$  images shared by all subjects. Note that voxels in early visual or outside of higher visual are not modeled.
In Figure 12 we show the  $R^2$  of the brain encoder as evaluated on the test images. Our brain encoder consists of a CLIP backbone and a linear adaptation layer. We do not model voxels in the early visual cortex, nor do we model voxels outside of higher visual. Our model can generally achieve high  $R^2$  in regions in known regions of visual semantic selectivity.
13 OFA and FFA visualizations
In this section, we visualize the top-10 NSD and BrainDiVE images for OFA and FFA. NSD images are selected using the fMRI betas averaged within each ROI. BrainDiVE images are ranked using our predicted ROI activities from 500 images.
Figure S.20: Results for face-selective ROIs in S1.
Figure S.21: Results for face-selective ROIs in S2.
Figure S.22: Results for face-selective ROIs in S5.
Figure S.23: Results for face-selective ROIs in S7.
14 OPA and food visualizations
S1 food
S2 food
S5 food
S7 food
S1 OPA
S2 OPA
S5 OPA
S7 OPA
Figure S.24: Clustering within the food ROI and within OPA. Clustering of encoder model weights for each region is shown for four subjects on an inflated cortical surface.
Consistent with Jain et al. [5], we observe that the food voxels themselves are anatomically variable across subjects, while the two food clusters form alternating patches within the food patches. OPA generally yields anatomically consistent clusters in the four subjects we investigated, with all four subjects showing an anterior-posterior split for OPA.
Figure S.25: Comparing results across the OPA clusters for S1 and S2.
Figure S.26: Comparing results across the OPA clusters for S5 and S7.
Figure S.27: Comparing results across the food clusters for S1 and S2.
Figure S.28: Comparing results across the food clusters for S5 and S7.
15 Training, inference, and compute details
Encoder training. Our encoder backbone uses ViT-B/16 with CLIP pretrained weights laion2b_s34b_b88k provided by OpenCLIP [65, 66]. The ViT [74] weights for the brain encoder are frozen. We train a linear layer consisting of weight and bias to map from the 512 dimensional vector to higher visual voxels  $B$ . The CLIP image branch outputs are normalized to the unit sphere.
$$
M _ {\theta} (\mathcal {I}) = W \times \frac {\operatorname {C L I P} _ {\operatorname {i m g}} (\mathcal {I})}{\| \operatorname {C L I P} _ {\operatorname {i m g}} (\mathcal {I}) \| _ {2}} + b
$$
Training is done using the Adam optimizer [75] with learning rate  $1r_{\mathrm{init}} = 3e - 4$  and  $1r_{\mathrm{end}} = 1.5e - 4$ , with learning rate adjusting exponentially each epoch. We train for 100 epochs. Decoupled weight decay [76] of magnitude decay  $= 2e - 2$  is applied. Each subject is trained independently using the  $\sim 9000$  images unique to each subject's stimulus set, with  $R^2$  evaluated on the  $\sim 1000$  images shared by all subjects.
During training of the encoder weights, the image is resized to  $224 \times 224$  to match the input size of ViT-B/16. We augment the images by first randomly scaling the pixels by a value between [0.95, 1.05], then normalize the image using OpenCLIP ViT image mean and variance. Prior to input to the network, we further randomly offset the image spatially by up to 4 pixels along the height and width dimensions. The empty pixels are filled in using edge value padding. A small amount of gaussian noise  $\mathcal{N}(0,0.05^2)$  is added to each pixel prior to input to the encoder backbone.
Objective. For all experiments, the objective used is the maximization of a selected set of voxels. Here we will further draw a link between the optimization objective we use and the traditional CLIP text prompt guidance objective [14, 59]. Recall that  $M_{\theta}$  is our brain activation encoder that maps from the image to per-voxel activations. It accepts as input an image, passes it through a ViT backbone, normalizes that vector to the unit sphere, then applies a linear mapping to go to per-voxel activations.  $S \in N$  are the set of voxels we are currently trying to maximize (where  $N$  is the set of all voxels in the brain),  $\gamma$  is a step size parameter, and  $D_{\Omega}$  is the decoder from the latent diffusion model that outputs an RGB image (we ignore the euler approximation for clarity). Also recall that we use a diffusion model that performs  $\epsilon$ -prediction.
In the general case, we perturb the denoising process by trying to maximize a set of voxels  $S$ :
For the purpose of this section, we will focus on a single voxel first, then discuss the multi-voxel objective.
In our case, the single voxel perturbation is (assuming  $W$  is a vector, and that  $\langle \cdot ,\cdot \rangle$  is the inner product):
$$
\begin{array}{l} \epsilon_ {t h e t a} ^ {\prime} = \epsilon_ {t h e t a} - \sqrt {1 - \alpha_ {t}} \nabla_ {x _ {t}} \left(\gamma M _ {\theta} \left(D _ {\Omega} \left(x _ {t} ^ {\prime}\right)\right)\right) \\ = \epsilon_ {t h e t a} - \sqrt {1 - \alpha_ {t}} \nabla_ {x _ {t}} (\gamma M _ {\theta} (\mathcal {I} _ {\mathrm {g e n}})) \\ = \epsilon_ {t h e t a} - \gamma \sqrt {1 - \alpha_ {t}} \nabla_ {x _ {t}} (\langle W, \frac {\mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}})}{\| \mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}}) \| _ {2}} \rangle + b) \\ \end{array}
$$
We can ignore  $b$ , as it does not affect optimal  $\mathrm{CLIP}_{\mathrm{img}}(\mathcal{I}_{\mathrm{gen}})$
$$
\equiv \epsilon_ {t h e t a} - \gamma \sqrt {1 - \alpha_ {t}} \nabla_ {x _ {t}} (\langle W, \frac {\mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}})}{\| \mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}}) \| _ {2}} \rangle)
$$
Now let us consider the typical CLIP guidance objective for diffusion models, where  $\mathcal{P}_{\mathrm{text}}$  is the guidance prompt, and  $\mathrm{CLIP}_{\mathrm{text}}$  is the text encoder component of CLIP:
$$
\epsilon_ {t h e t a} ^ {\prime} = \epsilon_ {t h e t a} - \gamma \sqrt {1 - \alpha_ {t}} \nabla_ {x _ {t}} (\langle \frac {\mathrm {C L I P} _ {\mathrm {t e x t}} (\mathcal {P} _ {\mathrm {t e x t}})}{\| \mathrm {C L I P} _ {\mathrm {t e x t}} (\mathcal {P} _ {\mathrm {t e x t}}) \| _ {2}}, \frac {\mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}})}{\| \mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}}) \| _ {2}} \rangle)
$$
As such, the  $W$  that we find by linearly fitting CLIP image embeddings to brain activation plays the role of a text prompt. In reality,  $\| W\| _2\neq 1$  (but norm is a constant for each voxel), and there is likely
no computationally efficient way to "invert"  $W$  directly into a human interpretable text prompt. By performing brain guidance, we are essentially using the diffusion model to synthesize an image  $\mathcal{I}_{\mathrm{gen}}$  where in addition to satisfying the natural image constraint, the image also attempts to satisfy:
$$
\frac {\operatorname {C L I P} _ {\operatorname {i m g}} \left(\mathcal {I} _ {\operatorname {g e n}}\right)}{\| \operatorname {C L I P} _ {\operatorname {i m g}} \left(\mathcal {I} _ {\operatorname {g e n}}\right) \| _ {2}} = \frac {W}{\| W \| _ {2}}
$$
Or put another way, it generates images where the CLIP latent is aligned with the direction of  $W$ . Let us now consider the multi-voxel perturbation, where  $W_{i}, b_{i}$  is the per-voxel weight vector and bias:
We move  $\frac{\gamma}{|S|}$  outside of the gradient operation
$$
\begin{array}{l} = \epsilon_ {t h e t a} - \frac {\gamma}{| S |} \sqrt {1 - \alpha_ {t}} \nabla_ {x _ {t}} (\sum_ {i \in S} M _ {\theta} (D _ {\Omega} (x _ {t} ^ {\prime})) _ {i}) \\ = \epsilon_ {t h e t a} - \frac {\gamma}{| S |} \sqrt {1 - \alpha_ {t}} \nabla_ {x _ {t}} (\sum_ {i \in S} \left[ \langle W _ {i}, \frac {\mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}})}{\| \mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}}) \| _ {2}} \rangle + b _ {i} \right]) \\ \end{array}
$$
We again ignore  $b_{i}$  as it does not affect gradient
$$
\equiv \epsilon_ {t h e t a} - \frac {\gamma}{| S |} \sqrt {1 - \alpha_ {t}} \nabla_ {x _ {t}} (\sum_ {i \in S} \langle W _ {i}, \frac {\mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}})}{\| \mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}}) \| _ {2}} \rangle)
$$
We can move  $\sum$  outside due to the distributive nature of gradients
$$
= \epsilon_ {t h e t a} - \frac {\gamma}{| S |} \sqrt {1 - \alpha_ {t}} \sum_ {i \in S} \left[ \nabla_ {x _ {t}} (\langle W _ {i}, \frac {\mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}})}{\| \mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}}) \| _ {2}} \rangle) \right]
$$
Thus from a gradient perspective, the total gradient is the average of gradients from all voxels. Recall that the inner product is a bilinear function, and that the CLIP image latent is on the unit sphere. Then we are generating an image that
$$
\frac {\mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}})}{\| \mathrm {C L I P} _ {\mathrm {i m g}} (\mathcal {I} _ {\mathrm {g e n}}) \| _ {2}} = \frac {\sum_ {i \in S} W _ {i}}{\| \sum_ {i \in S} W _ {i} \| _ {2}}
$$
Where the optimal image has a CLIP latent that is aligned with the direction of  $\sum_{i\in S}W_i$
Compute. We perform our experiments on a cluster of Nvidia V100 GPUs in either 16GB or 32GB VRAM configuration, and all experiments consumed approximately 1,500 compute hours. Each image takes between 20 and 30 seconds to synthesize. All experiments were performed using PyTorch, with cortex visualizations done using PyCortex [77].
CLIP prompts. Here we list the text prompts that are used to classify the images for Table 1. in the main paper.
face_class = ["A face facing the camera", "A photo of a face", "A photo of a human face", "A photo of faces", "A photo of a person's face", "A person looking at the camera", "People looking at the camera", "A portrait of a person", "A portrait photo"]
body_class = ["A photo of a torso", "A photo of torsos", "A photo of limbs", "A photo of bodies", "A photo of a person", "A photo of people"]
scene_class = ["A photo of a bedroom", "A photo of an office", "A photo of a hallway", "A photo of a doorway", "A photo of interior design", "A photo
of a building", "A photo of a house", "A photo of nature", "A photo of landscape", "A landscape photo", "A photo of trees", "A photo of grass"]
food_class = ["A photo of food"]
text_class = ["A photo of words", "A photo of glyphs", "A photo of a glyph", "A photo of text", "A photo of numbers", "A photo of a letter", "A photo of letters", "A photo of writing", "A photo of text on an object"]
We classify an image as belonging to a category if the image's CLIP latent has highest cosine similarity with the CLIP latent of a prompt belonging to a given category. The same prompts are used to classify the NSD and generated images.