Title: Retinotopy Inspired Brain Encoding Model and the All-for-One Training Recipe
Venue: NeurIPS 2023
Accepted: no
Keywords: brain encoding model

computer vision

neuroscience
Abstract: Brain encoding models aim to predict brain voxel-wise responses to stimuli images, replicating brain signals captured by neuroimaging techniques.   There is a large volume of publicly available data, but training a comprehensive brain encoding model is challenging.  The main difficulties stem from a) diversity within individual brain, with functional heterogeneous brain regions; b) diversity of brains from different subjects, due to genetic and developmental differences; c) diversity of imaging modalities and processing pipelines.
We use this diversity to our advantage by introducing the All-for-One training recipe, which divides the challenging one-big-model problem into multiple small models, with the small models aggregating the knowledge while preserving the distinction between the different functional regions (ROI). 
Agnostic of the training recipe, we use biological domain knowledge of the brain, specifically retinotopy, to introduce inductive bias to learn a 3D brain-to-image mapping that ensures a) each neuron knows which image regions and semantic levels to gather information, and b) no neurons are left behind in the model (starvation).  
We pre-trained a brain encoding model using over one million datapoints from five public datasets spanning three imaging modalities. To the best of our knowledge, this is the most comprehensive brain encoding model to the date. We demonstrate the effectiveness of the pre-trained model as a drop-in replacement for commonly used vision backbone models. Furthermore, we demonstrate the application of the pre-trained model to brain decoding. Code and the model checkpoint will be made available. 

Abstract

Brain encoding models aim to predict brain voxel-wise responses to stimuli images, replicating brain signals captured by neuroimaging techniques. There is a large volume of publicly available data, but training a comprehensive brain encoding model is challenging. The main difficulties stem from a) diversity within individual brain, with functional heterogeneous brain regions; b) diversity of brains from different subjects, due to genetic and developmental differences; c) diversity of imaging modalities and processing pipelines. We use this diversity to our advantage by introducing the All-for-One training recipe, which divides the challenging onebig-model problem into multiple small models, with the small models aggregating the knowledge while preserving the distinction between the different functional regions. Agnostic of the training recipe, we use biological knowledge of the brain, specifically retinotopy, to introduce inductive bias to learn a 3D brain-to-image mapping that ensures a) each neuron knows which image regions and semantic levels to gather information, and b) no neurons are left behind in the model.

We pre-trained a brain encoding model using over one million data points from five public datasets spanning three imaging modalities. To the best of our knowledge, this is the most comprehensive brain encoding model to the date. We demonstrate the effectiveness of the pre-trained model as a drop-in replacement for commonly used vision backbone models. Furthermore, we demonstrate the application of the model to brain decoding. Code and the model checkpoint will be made available.

0.36

0.39

0.42

0.45

0.48

In-Distribution Dataset Pearson's R

0.17

0.18

0.19

0.20

0.21

0.22

Pearson's R

AFO

CLIP-RN50x4

DiNOv2-ViTB

IM-RN50

IM-ViTB

-0.2

0.0

0.2

0.4

0.6

0.8

0.18

0.20

0.22

2 Holdout Datasets

Pearson's R

0.36

0.39

0.42

0.45

0.48

In-Distribution Dataset Pearson's R

-0.3

0.0

0.3

0.6

0.9

21

DiNOv2

layer2

layer5

layer8

layer11

ConvBlocks

B1

B2

B3

B4

x

y

z

u1

u2

x

y

z

w2

softmax

TopyNeck

w1

w3

w4

w1

w2

w3

w4

x

x

x

x

+

RetinaMapper

LayerSelector

one

voxel

RetinaMap

fMRI EEG MEG

RetinaGrid

tanh

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18

veROI

NSD_01

NSD_02

NSD_04

NSD_05

NSD_06

NSD_07

NSD_08

ALG

HCP

EEG

MEG

0

9000

Voxel Counts

NSD_01

veROI

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18

1 Introduction

There is a growing body of research in neuroscience that utilizes brain encoding models. The model 22

predicts voxel-wise brain response to visual stimuli, and it can be depicted as a multi-task regression 23

problem where each voxel is a task. The brain encoding model serves as a computational counterpart 24

to biological brains Wen et al. (2018). The common practice for building brain encoding models 25

is to use pre-trained models from image classification Deng et al. (2009), text-to-image alignment 26

Radford et al. (2021), or self-supervised tasks Oquab et al. (2023). These pre-trained models may 27

excel at their benchmarked task; however, Schrimpf et al. (2018) show that the image-classification 28

benchmark score does not align with prediction performance in brain encoding. 29

Building a model from all data sources poses a significant challenge due to heterogeneity in data: 30

diversity in functional sub-modules within each brain, b) genetic and developmental differences 31

across subjects, c) inconsistent imaging techniques and pre-processing pipelines. The current best 32

practice is to build Region-of-Interest (ROI) 1 models over subjects from the same dataset Cichy et al. 33

Willeke et al. (2022) Allen et al. (2022), where ROIs are predefined by well-studied anatomical 34

and functional properties of the brain voxels. However, the ROI-model approach lacks the potential 35

benefits for ROIs to aggregate knowledge and collaborate. This issue can be mitigated to some extent 36

by adjusting the granularity of ROIs. This work proposes a multi-stage All-for-One (AFO) training 37 recipe that explicitly lets ROIs aggregate knowledge while keeping the main training objective less 38 challenging than training for one all-ROI model. Borrowing the idea of 'Dark knowledge' distillation 39 Hinton et al. (2015), we use denoising to ensure the aggregated knowledge is clean. 40

Biological domain knowledge of the brain, specifically retinotopy, can be explored to design a 41 better model Lurz et al. (2021). The retina cells are physically wired through the optic nerve to the 42 lateral geniculate nucleus, which connects to the visual cortex. Thus, visual cortex cells preserve the 43 topological structure of images projected to the retina. This study explicitly defines a RetinaMapper 44 function that replicates retinotopic mapping. An obvious solution is learning a forward mapping that 45 transforms 2D RetinaGrid into a neuron in a 3D brain location. However, such forward mapping 46 can not guarantee to be surjective: every 3D neuron location is the mapped from at least one 2D 47 RetinaGrid . Our solution is to model the RetinaMapper from the inverse perspective, mapping 3D 48 neuron to 2D RetinaGrid . RetinaMapper is learned without ground-truth supervision, but still exhibits 49 retinotopic behavior, as shown in our results. 50

51

52

53

54

55

56

A well-reported phenomenon is that neuron voxels are mapped to shallow to deep layers of a feedforward neuron network Takagi and Nishimoto (2022). This motivates the common practice of selecting the best layers for each voxel. But per-voxel hyper-parameter tuning is highly noisy and prone to overfitting; previous studies overcome this by choosing the same layers for each ROI. In this study, we propose a LayerSelector module that enforces spatial proximity, thus allowing a flexible and robust selection of layers.

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

2 Related work

The field of computational neuroscience has been actively exploring the task of brain encoding, highlighting from Kay et al. (2008) Naselaris et al. (2011), surveyed by Wen et al. (2018). There are several initiatives and benchmarks: The brain-score Schrimpf et al. (2018) initiative compares frozen image backbone models using a PCA and linear regression pipeline. The PCA approach allows for a fair comparison of vision models with different latent dimensions. Additionally, Conwell et al. (2022) utilized a similar frozen PCA pipeline to benchmark various vision models on the NSD dataset. The Algonauts challenge Cichy et al. (2021) benchmarks end-to-end trained model without the constraint of frozen model and PCA dimension reduction. The Sensorium benchmark Willeke et al. (2022) worked on invasive mouse V1 imaging data. The Things initiative Hebart et al. (2023) provides fine-grid image captions which can be used for hypotheses testing. These datasets and benchmarks cover a wide range of imaging modalities, and preprocessing and denoising pipelines Kay et al. (2013) Prince et al. (2022). The All-for-One training recipe aims to leverage all of these diverse data sources to pre-train a comprehensive brain encoding model.

The neuroscience community has extensively applied brain encoding models to unravel the biological mechanisms underlying brain function. St-Yves et al. (2022) employed transfer learning techniques with brain encoding models to investigate the hierarchical organization of the brain. Franke et al. (2022) applied the model to study color coding in mouse neurons. The NeuroGen framework Gu et al. (2022) combined brain encoding models with image generation models, they utilize gradient-based methods to manipulate stimulus images. Bashivan et al. (2019) generated maximally excited images for populations of neurons and presented these images to subjects to validate the conclusions. On the other hand, there are fruitful studies of brain decoding 2 without a brain encoding model Takagi and Nishimoto (2022) Gu et al. (2023) Lu et al. (2023) Gu et al. (2023). Their framework is to take a pre-trained text-conditioned image generation model Ho et al. (2020) Rombach et al. (2022), then train a mapping function that aligns brain patterns to the text-condition embeddings space. However, we argue that decoding without a pre-trained encoding model is less efficient: Firstly, this pipeline is tightly linked to the pre-trained image generation model. Also, this pipeline face challenges in effectively utilizing heterogeneous data from various imaging modalities. We argue that decoding with a frozen encoding model is more efficient as this approach is agnostic to the specific image generation model.

Previous studies also explored incorporating retinotopy into the brain encoding model. Allen et al. 87 (2022) fits Gabor filters of various sizes and locations for each voxel. Lurz et al. (2021) also employed 88

the RetinaMapper , but their work focuses on training with the same imaging modality and one single 89 ROI. In contrast, our approach tries to model the whole visual brain with diverse data sources. 90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

3 Method

The voxel-wise encoding model (Fig 2) comprises three main components: Firstly, the backbone processes the input image and extracts latent image features from its intermediate layers. Next, the neck component compresses the feature vector for each voxel. Finally, the head applies a linear regression model to fit a prediction for each voxel. Let M l ∈ R D × H k × W k be the feature map output from the frozen backbone, where l is the layer index, k is the down-scale factor, we refer the H k × W k grid as RetinaGrid . The brain encoding model can be formulated as learning a mapping function F (Eq 1), where N depends on the imaging modality 3 . N MRI := ( X × Y × Z ) × 1 , N EEG := C × T , N MEG := ( X × Y × Z ) × T

3.1 TopyNeck

RetinaMapper The biological retinotopy process is mapping f : R H k × W k → R X × Y × Z . RitinaMapper aims to replicate this mapping. However, f can not guarantee to be surjective: every 3D neuron location is the mapped from at least one 2D RetinaGrid . Instead of the forward mapping f , we learn a reverse injective mapping f ′ : R X × Y × Z →R H k × W k and use tanh activation function to guarantee the output 2D coordinates lies within the RetinaGrid . The RetinaMapper is formulated as

where p ∈ R N× 3 is the voxel's spatial coordinate, PE is sinusoidal positional encoding function, u ∈ R N× 2 is coordinates in the RetinaGrid . During training, a small non-trainable variance σ is introduced u ′ ∼ N ( u, σ ) . At inference time σ is set to 0. At each u ′ , linear interpolation is performed to obtain a 1-D feature vector m l ∈ R N× D for each layer l . Furthermore, Another 1-D feature vector q l = MLP ( GlobalAvgPool ( M l ) , GlobalMaxPool ( M l )) is added to m l . Parameters of RetinaMapper is shared for all layers. Figure 2 and 4 show examples of such mapping. The color dots in RetinaGrid indicate which 3D neuron layers it is from. The blank area indicates image regions that are unused for prediction.

LayerSelector Early visual to downstream regions have growing receptive field sizes and neurons' latent representation of the stimuli image grows abstract. This motivates matching voxels to layers in feed-forward neuron networks. But selecting the best or top layers for each voxel is suspected to be overfitting. LayerSelector enforce spatial proximity formulated as

where η ∈ R N× L . The 1-D feature vectors sampled from various layers at RetinaGrid is reduced as m ∗ i = ∑ L η l i m l i . Regularization loss l ent = ∑ L η l log η l is applied to prevent converging to a local minimum that only selects one single layer.

3.2 All-for-One training recipe

Dividing neuron voxels into ROIs loses ROIs' potential to aggregate knowledge and collaborate. Mixing can also negatively affect individual voxel performance, making learning more challenging. The AFO recipe aims to gather the benefits from both dividing and mixing. Multiple stages models are trained (Figure 3): In stage one, each ROI model is trained separately. In stage two, each ROI model is trained to distill the dark knowledge Hinton et al. (2015) from all other ROIs, but the ground truth loss is only applied on the target ROI, other ROIs are helpers, and their parameters were discarded after training. Model checkpointing and early stopping are conditioned only on the target ROI. In stage three, the final model is trained with all ROIs as outputs, with dark knowledge and ground truth loss. The final product is one comprehensive all-ROI model.

131

132

133

134

135

136

137

138

139

140

141 142 143 144 145 146 147 148 149

GT: Grount Truth

Stage1

Model1

ROI1

Modeln

Stage2

Model1

GT

loss

GT

loss

Helper

ROI

loss

DK

loss

GT

loss

Modeln

discarded after training

Stage3

Model

ROI1

DK

loss

GT

loss

ROIn

DK

loss

GT

loss

ROIn

DK

ROI1

3.3 Voxel-wise encoding ROI

We need a unified ROI parcellation that is defined for all subjects from various imaging modalities. To generate such a unified ROI, we utilize the final linear regression weight, which is extracted from an average of 10 all-ROI models. We start by performing Euclidean distance k-means clustering on the weights to reduce the dimension of voxel counts. Subsequently, Ward's method applies hierarchical clustering to find the cluster centroids. This hierarchical clustering results in a dendrogram. We cut the dendrogram at a hand-picked threshold to identify the veROIs . By adjusting this threshold, we can control the granularity of the veROIs .

4 Experiments

4.1 Datasets

We utilize 7 publicly available datasets for our experiments (Table 1). Details are provided in Allen et al. (2022) Van Essen et al. (2012) Cichy et al. (2021) Hebart et al. (2023) Gifford et al. (2022) Chang et al. (2019). We use only voxels from the visual brain. Each dataset was divided into training, validation, and test sets with a ratio around 90 : 6 : 4 . For the Things datasets, we use repeatedly represented images as the test set. All the experiment results are reported from the test set unless specified. The HCP video was split into chunks of 20 seconds to ensure no data leak, and a time delay of 4 seconds between video frames and fMRI frames was applied Khosla et al. (2021), blank resting-state segments are not discarded. For video stimulus, we extracted frames at a rate of one frame per second. We only use one frame for the ALG dataset.

Notably, except for the NSD dataset, all subjects from other datasets viewed the same set of images. 150 As a compromise for computation intensity, we concatenated the voxels from ALG EEG MEG 151 subjects into each single large brain, voxel's spatial coordinates are placed in an evenly spaced grid. 152 For the HCP dataset, a group average was performed due to the large number of subjects and the 153 lower SNR in each individual subject. All datasets have spatial coordinates for voxels except the 154 EEG dataset, EEG voxel's spatial coordinates are generated from dummy sequential numbers. 155

156

157

158

159

160

161

162

163

164

165

166

167

168

4.2 TopyNeck probing

RetinaMapper In Figure 4, for NSD subjects, early visual voxels were mapped to span most of the RetinaGrid , while downstream-region voxels remained concentrated in the center. The ablation study presented in Table 2 further demonstrates the outstanding importance of the RetinaMapper for early visual voxels in NSD subjects. This alignment with retinotopy design motivation. However, for other low SNR datasets, no clear retinotopic mapping was observed, suggesting that the RetinaMapper may not be necessary in such cases, and a constant mapping to the center could be sufficient.

LayerSelector In Figure 5, for subject NSD_01, a smooth transition from shallow to deep layers was observed. This alignment with the design motivation. Ablation study in Table 2 also indicates significant improvement for NSD subjects compared to un-weighted averaging layers or selecting a single layer. However, for low SNR datasets, the trend was to select only the last layer (Figure 4), suggesting that the LayerSelector module may not be necessary in such cases.

x

y

z

x'

y'

RetinaMapper

RetinaMap

RetinaGrid

Virtual

MapBack

NSD_01

NSD_02

NSD_04

NSD_05

NSD_06

NSD_07

NSD_08

ALG

HCP

EEG

MEG

layer2

layer5

layer8

layer11

SPLR

IPSIPS2

IPS5

IPS3

SPL1

PS1

IPST

IPS2

IPS3h

PS2IPS4PS

V3AIPSO

IPSO

V3A

V3B

V3B

LO2

V3

V

V1

LO1

hMTz

ISW

V4

V4

V2V3

OFA

LON

V2

-VO1OFA

VO2

FFA

V02L

FFA

PPA

PPA

layer2

layer5

layer8

layer11

PSI

IPS2

IPST

V3A

P

MS1

LOR

vk

VJ

LO1

GCA

Log

LO1

V38

AST

LO1

V3B

AST

V

LO1

V3B

MST

OFA

YO2

D

OFA

2

OFA

V02l

VO1

PA

OFA

VO2

1OM

2

V4

V02

OFA

FFA

PPA

PPA

PPA

PPA

PPA

PA

PPA

PPA

9.2

0.4

0.4

0.6

9.7

9.4

0.6

9.2

0.4

0.6

4.3 All-for-One recipe results

In Table 3, a significant performance gap between the S1 and S2 models indicates the effectiveness 169 of aggregating knowledge among ROIs. We also study a randROI that has the exact same number of 170 ROIs and number of voxels inside each ROI. S1 and S2 gap is not observed in the randROI approach, 171 as randROI already covers all types of voxels in every ROI. Furthermore, the model trained with 172 ground truth (NoDK) as helpers shows little to no improvement over the S1 model. This suggests 173 that the quality of the helper ROI is critical for the AFO recipe, as involving noisy helpers makes the 174 training process unnecessarily challenging. In this context, dark knowledge plays a crucial role as 175 denoising. However, solely dark knowledge distillation doesn't have a great impact as can be inferred 176 from the small gap between randROI S1 and S2 models. 177

4.4 veROI results 178

Figure 6 shows veROI on cortex across all NSD subjects, early visual areas is centered around 179 veROI_5 (blue) and downstream areas centered around veROI_9 (green), voxels that drop out from 180 the field of view in early visual areas are centered around veROI_16 (red). The score for each veROI 181 for subject NSD_01 can be found in Figure 8, where veROI_12 onward is mainly for the low SNR 182 voxels. From the heatmap in Figure 2 we can also observe that veROI_12 onward is mainly HCP, 183 EEG, and MEG subjects. 184

185

186

187

188

189

4.5 Brain decoding

Methods In this study, brain decoding refers to the task of ranking and retrieving candidate images from a candidate set, retrieved images are to match a given brain response pattern. The decoding pipeline involves forwarding each candidate image through the brain encoding model and measuring Pearson's correlation coefficient between the model's prediction and the ground truth.

Results The experiments are conducted on 500 validation images as candidate images. As a 190 qualitative analysis, Figure 7 and Figure 9 demonstrate that when conditioning on the early visual 191 area or veROI_5, texture and orientation are more preserved in the decoded images. Conversely, 192 when conditioning on downstream ROIs, semantic concepts are more preserved. Additionally, Figure 193 8 shows that image retrieval achieves high accuracy when conditioned on early ROIs. Quantitative 194 exploration of the functional roles of ROIs is beyond the scope of this study. Future work may involve 195

198

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18

NSD_01

NSD_02

NSD_04

NSD_05

NSD_06

NSD_07

NSD_08

investigating semantic concepts with image generation models. Furthermore, the gradient of the 196 encoding model can be utilized to facilitate image generation and manipulation. 197

GT

Top1

Top2

Top3

Top4

Top5

(a) Match all voxels

early

GT

Top1

Top2

Top3

Top4

Top5

mid

late

(b) Match one ROI

libanl

all

early mid

late

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

0.0

0.2

0.4

0.6

0.8

1.0

Image Retrieval

ROI

top1

top5

r

0.0

0.2

0.4

0.6

0.8

1.0

Performance Score

Pearson's R

4.6 Implementation details

We use smooth L1 loss with β = 0 . 01 , regulirazation loss l ent is scaled down by λ = 0 . 00003 . 199 AdaBelief optimizer Zhuang et al. (2020) is employed with lr = 0 . 003 , batchsize = 128 , 200 weight_decay = 0 . 0001 , ( β 1 , β 2 ) = (0 . 9 , 0 . 999) . Notably, we mix subjects in one mini-batch, 201 and the effective batch size for each subject is less than the total. Due to memory constrain, we 202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

GT

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

randomly sample up to 8000 voxels for each training datapoint, there is 436,715 voxels totaling all subjects. Early stopping is configured with patience = 20 epochs, we define one epoch as 10% of the total training data. Greedy Model Soup Wortsman et al. (2022) is applied at the top 10 validation checkpoints. Backbone is kept frozen except LayerNorm running statistics is updated. Input resolution is 224 × 224 and the feature from backbone layers are all of the size 768 × 16 × 16 . The attached trainable convolution block is three zero-padded 5x5 convolutions with skip connection and LayerNorm , C = 768 . The last convolution layer reduces the dimension to D = 256 . We trained all models on single NVIDIA RTX 2080 Ti 12GB GPUs at a reduced clock speed of 1140Mhz, single-subject all-ROI models consume half to 1 GPU hour, all-subject single-ROI models consume 3 to 5 GPU hours, all-subject all-ROI models consume 10 GPU hours. The complete AFO recipe total around 300 GPU hours. Models are trained Pytorch Lightning Falcon (2019) mixed precision FP16. To boost training speed, MLPs in RetinaMapper and LayerSelector are pre-optimized by a single-subject all-ROI model, they are loaded and kept frozen in the AFO recipe, this gives 2 times faster convergence speed.

5 Conclusion and Limitations

We proposed the AFO recipe alongside veROI to address the issue of heterogeneity in publicly available datasets. To the best of our knowledge, our pre-trained model constructed with over 1 million data points is the most comprehensive brain encoding model to date. The model shows superior performance when transferred to small hold-out datasets. As demonstrated by our brain decoding experiments, the pre-trained model could facilitate further neuroscience research.

We also designed TopyNeck inspired by retinotopy, which showed retinotopic behavior despite having no ground truth supervision for the retinotopic mapping function. However, the retinotopic behavior diminishes when the target dataset SNR is low, e.g. EEG, MEG. This suggests a simple alternative approach is sufficient in such a case.

References

Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F., Charest, I., Hutchinson, J. B., Naselaris, T., and Kay, K. (2022). A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience , 25(1):116-126. Number: 1 Publisher: Nature Publishing Group.

Bashivan, P., Kar, K., and DiCarlo, J. J. (2019). Neural population control via deep image synthesis. Science , 364(6439):eaav9436. Publisher: American Association for the Advancement of Science.

Chang, N., Pyles, J. A., Marcus, A., Gupta, A., Tarr, M. J., and Aminoff, E. M. (2019). BOLD5000, a public fMRI dataset while viewing 5000 visual images. Scientific Data , 6(1):49. Number: 1 Publisher: Nature Publishing Group.

Cichy, R. M., Dwivedi, K., Lahner, B., Lascelles, A., Iamshchinina, P., Graumann, M., Andonian, A., Murty, N. A. R., Kay, K., Roig, G., and Oliva, A. (2021). The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion. arXiv:2104.13714 [cs, q-bio].

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

Conwell, C., Prince, J. S., Alvarez, G. A., and Konkle, T. (2022). Large-Scale Benchmarking of Diverse Artificial Vision Models in Prediction of 7T Human Neuroimaging Data. Pages: 2022.03.28.485868 Section: New Results.

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). ImageNet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255. ISSN: 1063-6919.

Falcon, W. A. (2019). Pytorch lightning. GitHub , 3.

Franke, K., Willeke, K. F., Ponder, K., Galdamez, M., Zhou, N., Muhammad, T., Patel, S., Froudarakis, E., Reimer, J., Sinz, F. H., and Tolias, A. S. (2022). State-dependent pupil dilation rapidly shifts visual feature selectivity. Nature , 610(7930):128-134. Number: 7930 Publisher: Nature Publishing Group.

Gifford, A. T., Dwivedi, K., Roig, G., and Cichy, R. M. (2022). A large and rich EEG dataset for modeling human visual object recognition. NeuroImage , 264:119754.

Gu, Z., Jamison, K., Kuceyeski, A., and Sabuncu, M. (2023). Decoding natural image stimuli from fMRI data with a surface-based convolutional network. arXiv:2212.02409 [cs, q-bio].

Gu, Z., Jamison, K. W., Khosla, M., Allen, E. J., Wu, Y., St-Yves, G., Naselaris, T., Kay, K., Sabuncu, M. R., and Kuceyeski, A. (2022). NeuroGen: Activation optimized image synthesis for discovery neuroscience. NeuroImage , 247:118812.

Hebart, M. N., Contier, O., Teichmann, L., Rockter, A. H., Zheng, C. Y., Kidder, A., Corriveau, A., Vaziri-Pashkam, M., and Baker, C. I. (2023). THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. eLife , 12:e82580. Publisher: eLife Sciences Publications, Ltd.

Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531 [cs, stat].

Ho, J., Jain, A., and Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. arXiv:2006.11239 [cs, stat].

Kay, K. N., Naselaris, T., Prenger, R. J., and Gallant, J. L. (2008). Identifying natural images from human brain activity. Nature , 452(7185):352-355.

Kay, K. N., Rokem, A., Winawer, J., Dougherty, R. F., and Wandell, B. A. (2013). GLMdenoise: a fast, automated technique for denoising task-based fMRI data. Frontiers in Neuroscience , 7.

Khosla, M., Ngo, G. H., Jamison, K., Kuceyeski, A., and Sabuncu, M. R. (2021). Cortical response to naturalistic stimuli is largely predictable with deep neural networks. Science Advances , 7(22):eabe7547.

Lu, Y., Du, C., Wang, D., and He, H. (2023). MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion. arXiv:2303.14139 [cs].

Lurz, K.-K., Bashiri, M., Willeke, K., Jagadish, A., Wang, E., Walker, E. Y., Cadena, S. A., Muhammad, T., Cobos, E., Tolias, A. S., Ecker, A. S., and Sinz, F. H. (2021). Generalization in data-driven models of primary visual cortex.

Naselaris, T., Kay, K. N., Nishimoto, S., and Gallant, J. L. (2011). Encoding and decoding in fMRI. NeuroImage , 56(2):400-410.

Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. (2023). DINOv2: Learning Robust Visual Features without Supervision. arXiv:2304.07193 [cs].

Prince, J. S., Charest, I., Kurzawski, J. W., Pyles, J. A., Tarr, M. J., and Kay, K. N. (2022). Improving the accuracy of single-trial fMRI response estimates using GLMsingle. eLife , 11:e77599.

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020 [cs].

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752 [cs].

Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, J., Schmidt, K., Yamins, D. L. K., and DiCarlo, J. J. (2018). Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like? Pages: 407007 Section: New Results.

St-Yves, G., Allen, E. J., Wu, Y., Kay, K., and Naselaris, T. (2022). Brain-optimized neural networks learn non-hierarchical models of representation in human visual cortex. Pages: 2022.01.21.477293 Section: New Results.

Takagi, Y. and Nishimoto, S. (2022). High-resolution image reconstruction with latent diffusion models from human brain activity. Pages: 2022.11.18.517004 Section: New Results.

Van Essen, D. C., Ugurbil, K., Auerbach, E., Barch, D., Behrens, T. E. J., Bucholz, R., Chang, A., Chen, L., Corbetta, M., Curtiss, S. W., Della Penna, S., Feinberg, D., Glasser, M. F., Harel, N., Heath, A. C., Larson-Prior, L., Marcus, D., Michalareas, G., Moeller, S., Oostenveld, R., Petersen, S. E., Prior, F., Schlaggar, B. L., Smith, S. M., Snyder, A. Z., Xu, J., Yacoub, E., and WUMinn HCP Consortium (2012). The Human Connectome Project: a data acquisition perspective. NeuroImage , 62(4):2222-2231.

Wen, H., Shi, J., Zhang, Y., Lu, K.-H., Cao, J., and Liu, Z. (2018). Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision. Cerebral Cortex , 28(12):4136-4160.

Willeke, K. F., Fahey, P. G., Bashiri, M., Pede, L., Burg, M. F., Blessing, C., Cadena, S. A., Ding, Z., Lurz, K.-K., Ponder, K., Muhammad, T., Patel, S. S., Ecker, A. S., Tolias, A. S., and Sinz, F. H. (2022). The Sensorium competition on predicting large-scale mouse primary visual cortex activity. arXiv:2206.08666 [cs, q-bio].

Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. (2022). Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv:2203.05482 [cs].

Zhuang, J., Tang, T., Ding, Y., Tatikonda, S., Dvornek, N., Papademetris, X., and Duncan, J. S. (2020). AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients. arXiv:2010.07468 [cs, stat].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors make two claimed contributions:\n1. They propose an architecture for an encoding model. This architecture consists of three key components.\n* A frozen DINO trained ViT-B backbone + learnable convolutional layers on top of ViT feature maps\n* A differentiable spatial sampling layer (implemented via pytorch grid_sample, similar to [1]), where the 2D coordinates are predicted from 3D voxel coordinates\n* A differentiable softmax based layer selector\n2. They propose an \"All-for-One\" training recipe. Which incorporates the following:\n* \"dark knowledge distillation\" (typically referred to as knowledge distillation or network distillation in most other machine learning works), where they use ROI specific networks to train larger networks\n* They propose a new parcellation across brains which they call \"veROIs\", which is extracted via k-means clustering of voxel weights.\n\nTo validate this method, the authors visualize the learned spatial sampling grids and how the preferred layer varies across voxels. The authors further perform an ablation study of the encoder, and perform image retrieval using their network.\n\nThe authors provide an illustration of how the sampling grid evolves during training in the supplementary.\n\n[1] Jaderberg, Max, Karen Simonyan, and Andrew Zisserman. \"Spatial transformer networks.\" Advances in neural information processing systems 28 (2015).\n\nSTRENGTHS:\nThe paper is novel in the combination of techniques, which uses online-learned sampling grids with soft backbone layer assignment to predict the brain activations, this design is reasonably biologically motivated.\n\nThe authors proposal of clustering based ROI assignment is also novel in the context of decoding (image retrieval).\n\nThe paper provides definitions and dimensions for most variables where appropriate, and the figures are illustrative.\n\nMy view is that the proposed decoder architecture is moderately novel in the context of encoding models. There is insufficient information (lack of detail) to judge the all-for-one training.\n\nWEAKNESSES:\nThe paper is interesting, however my key concern lies in the lack of details for the all-for-one training scheme, and the lack of evaluation in comparing against simple linear-regression + backbones (ResNet, CLIP ViT [1])\n* On the clarity of the paper\n  * The high level clarity of the paper regarding the all-for-one training is poor and could use significant improvement. \n  * It is not clear how exactly the veROIs are used. Is the encoding model ultimately at the voxel-level? If so, are the veROIs used for the network distillation and image retrieval steps? Do you share the backbone and just use independent linear weights for the voxels?\n  * It is not clear which subjects you use for Stage 1/2/3. Do you train with one subject's ROI? Or do you train on all subjects' single ROI (using stage 1 as an example). If so, do you take veROI-1 for example, then train a model on all subjects from all modalities for veROI-1?\n  * It is not clear how you get the all-ROI model to derive the veROIs. Is this model trained on all subjects and all modalities? Do you train just the last linear weights? Or you use the DINO features? Do you train the conv weights?\n  * It is not clear how you train across subjects and datasets where the number of voxels are not the same. This is a central claim in your paper.\n\n* On motivation\n   * Currently the justification for repeated distillation is weak. Your experiments show that distillation does help from a performance standpoint, however your motivation differs from the typical use of distillation (which is to accelerate inference using a student model). Could you better motivate this? \n\n* On prior work\n  * For the topic of brain encoding models that utilize differentiable spatial sampling, I recommend citing [2,3] which are similarly brain motivated, as well as [4] which is one of the more significant papers that uses differentiable sampling.\n\n* On the soundness of the baseline\n  * For the \"FrozenRM\" encoder, you mention that every voxel is mapped to the center. Since you use DINO based on ViT, there is an extra classifier token (usually the first token). Typically when using ViT based architectures for spatially invariant tasks, this is the embedding you use. I recommend modifying the FrozenRM baseline, or adding an additional baseline when you use this token\n  * The same criticism applies to the GlobalPool token, I recommend adding/replacing a baseline where the global pooled representation is replaced with the classifier token\n\n* On the lack of baselines\n  * Currently the authors perform ablation studies, but do not perform qualitative or quantitive comparisons against other works [1]. The paper would be strengthened by adding comparisons to linear-regression based single subject voxel-wise encoding models based on different architectures (global pooled ResNet features , VGG, Gabor features, GIST features) trained on ImageNet, or different objectives (CLIP/OpenCLIP/EVA-CLIP, DINO, any of the SSL/Masking work) adapted to a single subject. I don't expect their multi-subject network to necessarily perform better (nor would it be a negative if they perform worse), but some evaluations are still necessary. \n\n* Minor\n  * Figure 4 was quite confusing to me. In the retinagrid case, the colors indicate spatial extent. However in the retinamap, the colors indicate layer selection. I would ask that you provide additional clarity here.\n  * The notation in Table 3 is quite confusing, you do not specify what veROIsX is. It is implied that it corresponds to what stage of network distillation you use. Please clarify this. \n  * Line 199 \"regulirazation\", minor misspelling. \n\nOn balance, the paper is interesting from an architectural standpoint. I would be happy to take another look at the paper if the authors can clarify their training scheme and add additional baselines. \n\n[1]  Conwell, Colin, et al. \"What can 5.17 billion regression fits tell us about artificial models of the human visual system?.\" SVRHM 2021 Workshop@ NeurIPS. 2021.\n\n[2] Mahner, Florian, et al. \"Learning Cortical Magnification with Brain-Optimized Convolutional Neural Networks.\" Conference on Cognitive Computational Neuroscience. 2022.\n\n[3] Jun, Na Young, Greg Field, and John Pearson. \"Efficient coding, channel capacity, and the emergence of retinal mosaics.\" Advances in neural information processing systems 35 (2022): 32311-32324.\n\n[4] Jaderberg, Max, Karen Simonyan, and Andrew Zisserman. \"Spatial transformer networks.\" Advances in neural information processing systems 28 (2015).\n\nQUESTIONS:\nSee weaknesses section for questions\n\nLIMITATIONS:\nThe authors adequately address the limitations of their model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops an all-for-one training model to address the challenge of one big-model problem by converting it into multiple small models, in which the small models aggregate the knowledge while preserving the distinction between the different functional regions. With the proposed method, biological knowledge of the brain, particularly retinotopy, is used to introduce inductive bias into a 3D brain-to-image mapping that ensures a) each neuron knows which regions and semantic levels to gather information, and b) no neurons are left out. Overall, it is an interesting paper, however, there are several concerns about the machine learning novelty, validating the empirical studies, and the clear presentation of the proposed method.\n\nSTRENGTHS:\nPlease refer to the question section\n\nWEAKNESSES:\nPlease refer to the question section\n\nQUESTIONS:\nThe followings are the major concerns and minor comments:\n\n\n1) It is still unclear to me as to what the novelty of this paper is in terms of machine learning. There may be some contributions to computational neuroscience in this paper; however, which part of the machine learning approach is new in this paper? \n\n\n2) My second concern is the design of the empirical studies. Even though the study contains several beautiful figures (and a movie in supplementary material), more numerical analyses would be helpful to convince the reader that the proposed method is effective. There is a lack of regular analysis and conventional machine learning metrics (such as accuracy, dice, etc.) which make it difficult to understand the results. Specifically, Pearson’s correlation is not an accuracy metric to evaluate the proposed model. \n\n\n3) The proposed method should be benchmarked in comparison with related state-of-the-art techniques.\n\n\n4) In this paper, the notations are confusing. In regular papers, scalers are denoted by small letters, vectors are defined with small letters (highlighted in bold), and matrices are denoted by capital letters using bold. In this paper, there are a lot of conflicts. It is so hard to trace what is a set, a matrix, or even a distribution. \n\n\n5) The proposed method can be summarized in the form of an algorithm or pseudocode.\n\n\n6) There are some minor linguistic and typo problems in this paper.\n\nLIMITATIONS:\nPlease refer to the question section", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript addresses the challenge of generating brain encoding models (specifically for visual stimuli) - which seeks to predict brain responses at the voxel level to visual stimuli. A challenge facing brain encoding is heterogeneity in data modality, individual variability, and functional differences across brain region. Existing models often address the problem of heterogeneity by fitting separate models for different brain regions. However, the authors seek to to fit a single model that encompasses the entire visual brain by leveraging a dark knowledge distillation method in which each ROI distills the dark knowledge present in the other ROIs.  The authors evaluate their method on a variety of functional imaging datasets spanning fMRI, MEG, and EEG.\n\nSTRENGTHS:\nOriginality: The authors developed a novel training pipeline utilizing dark knowledge distillation in order to allow ROIs to collaborate during training. The authors also develop a new method for incorporating retinotopy, a biologically realistic feature, into their encoding model.\n\nQuality: The model seems to work across different data modalities (e.g. EEG, MEG, and fMRI), making it broadly applicable.\n\nClarity: The manuscript provides several ablation studies that investigate the limitations of the approach and which features of the architecture are important for improvement.\n\nSignificance: The study introduces a new approach for creating visual brain encoding models from functional imaging data. Their approach could be employed by other groups studying non-visual stimuli as well.\n\nWEAKNESSES:\nUnless I am mistaken, the authors do not evaluate their model against the state of the art in  computational speed, making it hard to evaluate whether their model represents a significant improvement, as the improvement in correlation appears to be modest. Furthermore, the message of the paper is a little unclear - what is the main breakthrough the authors are trying to present: the all-for-one training, retinotopy, or ability to work with multiple data modalities. Evaluation against state of the art should occur for each of these topics. Finally, the model is trained on data across fMRI, MEG, and EEG modalities, but the held out data consists only of fMRI data, making evaluation of model generalizability difficult.\n\nQUESTIONS:\nSee weakness section for list of suggestions/questions.\n\nLIMITATIONS:\nLimitations are addressed except for points raised earlier about comparison to state of the art.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the brain encoding task, which predicts brain voxel-level responses to image stimuli. The authors aim to train a comprehensive brain encoding model using the vast amount of public data from diverse imaging modalities and numerous participants. The proposed method, the All-for-One training recipe, divides the one-big-model to multiple small models and aggregates the knowledge together in inference time. An intriguing technique the authors propose is to use retinotopy to introduce inductive bias to learn the mapping.\n\nSTRENGTHS:\n1. The figure quality (especially aesthetics) is not often seen in NeurIPS submissions. Take a look at Figure 2, 4, 5 and 6. Those figures are on par with Nature family submissions. \n2. Decent ablation studies.\n\nWEAKNESSES:\n1. The design choice is not the most straightforward to process. I have some trouble understanding the logic behind the three-staged design of the proposed All-for-One recipe — specifically, why is it necessary to have these 3 stages?\n2. RetinaGrid and RetinaMap seems a bit far-fetched. By far, my impression is that the authors are simply trying to draw an analogy from image formation process in the retina and provide a fancy visualization. The authors are welcome to defend with explanations.\n3. Lack of other alternative baselines.\n\nQUESTIONS:\n1. It is a little bit weird to see an MEG and an EEG dataset in 5 other fMRI datasets. Did you conduct experiments to see if the inclusion of MEG and EEG is significantly harming the performance?\n2. What is the rationale for enforcing a strong prior on one-to-one mapping between the brain voxel-level responses and the pixels in the image feature maps?\n3. Do you think it will be beneficial to include some brief summaries/teasers in the Abstract and/or Introduction regarding the brain decoding investigations?\n\nLIMITATIONS:\nNothing that I am aware of.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the task of brain encoding model, which aims to predict brain voxel-wise responses to stimulus images. First, the paper proposes the All-for-One (AFO) training recipe, which enhances interactions among multiple ROI models to handle the large diversity within the data. Second, the paper introduces the RetinaMapper to learn a 3D brain-to-image mapping and the LayerSelector to selectively merge features from multiple layers. Finally, the paper trains the model on a very large scale of data and demonstrates the effectiveness of the model through extensive qualitative and quantitative analysis.\n\nSTRENGTHS:\n1. The overall motivation is clear, and the techniques used are straightforward and reasonable. For example, enforcing cross-model interaction/distillation is a reasonable way to enhance each model, and selectively merging information from multiple layers with different receptive field sizes is sensible.\n\n2. The experiments are extensive, and adequate qualitative and quantitative analysis is provided. For example, Table 2 demonstrates the effectiveness of TopyNeck, and both Figure 4 and Figure 5 show the effectiveness of the LayerSelector.\n\n3. The work pre-trains the model in large-scale data, which is impressive.\n\nWEAKNESSES:\n1. need to provide more details and insights into specific techniques\n2. missing more explanations/references\n3. some writing issues\n\nQUESTIONS:\n1. Although the overall motivation is clear, many details and insights of specific techniques are missing and require more explanations/references. For example:\n\n1） It is unclear why the authors introduce non-trainable variance to 'u' and add global features to 'm^l'.\n\n2） The authors do not explain why selecting the best or top layers for each voxel is suspected to be overfitting and how their method addresses this problem. \n\n3） The regularization loss presented in L119 only increases/decreases entropy, and it is unclear how it can prevent convergence to a local minimum that only selects one single layer. The authors should refer to the IM loss mentioned in [1] for the correct term.\n\n4） The details about the distillation in L267 are missing, and the authors should present the concrete form of the distillation loss.\n5） Some hyper-parameters are missing due to the lack of details mentioned above, such as the weights of the regularization loss and distillation loss.\n\n[1] Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation, in ICML 2020.\n\n2. Some of the presentations need improvement:\n\n1) The abbreviations and notations used are numerous, and it would be helpful to simplify them. For example, RetinaMapper, RetinaGrid, AFO, and TopyNeck. The notations used in the experimental section, such as in Table 2 and Table 3, are also confusing and should be simplified.\n\n2) Figure 2 is unclear, and the authors should clearly label which parts correspond to the backbone, TopyNeck, and head.\n\nLIMITATIONS:\nThe authors have addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}