SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison of experimental setups For the GA optimization, we use the established method for generating MEIs that has been tested in vivo Walker et al. [17]. However, we perform a comparison study to show that the parameters chosen are selected to maximize the performance of the GA method. We rerun the MEI optimizations using the AdamW optimizer and find a significant decrease in performance in comparison to the SGD optimizer  $(r = 0.69)$ . We also run the MEIs for 100 steps instead of 1000 and also find a performance decrease  $(r = 0.95)$  (Fig. S2).
Generating MEIs in color The diffusion model generates color images, so in principle, it can generate color MEIs. We attach some examples (Fig. S4). Since the encoding models are trained
Supplementary Figure S1: a) Comparison of the mean shift distance between the center of mass for the attention masks against the test correlation of the neurons. b) Example attention maps of neurons responding to different neurons. The blue dot shows the center of mass. The examples show that the neuron shifts its center of mass.
on grayscale images because the animals only saw grayscale images the colors in these may not be meaningful. However, if one were to use color stimuli it would be possible to generate MEIs that are colored and potentially meaningful.
MENIs vs ImageNet search We compare the generated MEIs  $(\lambda = 1)$  to a standard approach for finding natural images for individual neurons. To that end, we perform a search across  $100k$  images from the ImageNet dataset [68] to find the top-1 most activating image for a particular unit. We then compare the predicted activations of the top-1 ImageNet image and the generated MEIs  $(\lambda = 1)$  in the cross-architectures paradigm (Fig. S8). We find that the generated MENIs drive comparable activation to the top-1 ImageNet images. Like in the MEI generation paradigm, EGG can thus significantly speed up the search for activating natural images, as it does not need to search through millions of images.
Supplementary Figure S2: Comparison of different experimental setups of MEI optimization using the GA method. a) Use of SGD vs AdamW optimizer. The SGD optimizer outperforms the AdamW optimizer on within architecture evaluation. b) Increasing the number of steps slightly decreases the within architecture performance.
Supplementary Figure S3: Variability dependent on the seed used for generating MEIs in the Gaussian model and the Attention model. Each column represents a different seed and each row a different neuron. Results shown for the Gaussian and Attention models.
Supplementary Figure S4: Examples of MEIs in their original color version (before converting to grayscale). Top row is the direct output RGB images from the diffusion model, the bottom shows the grayscale version. Each column corresponds to a different neuron.
Supplementary Figure S6: Examples of images generated using EGG diffusion in the Monkey V4 with different energy scales  $\lambda \in \{1,2,5,10\}$ . Generated for the Gaussian model. Units not matched with the images shown for the Attention model.
Supplementary Figure S7: Examples of images generated using EGG diffusion in the Monkey V4 with different energy scales  $\lambda \in \{1,2,5,10\}$ . Generated for the Attention model. Units not matched with the images shown for the Gaussian model.
Supplementary Figure S8: Comparison of the MEIs  $\lambda = 1$  activations to the top-1 most activating ImageNet images per neuron in the cross-architecture domain. Line fit obtained via Huber regression with  $\varepsilon = 1.1$ . In the left panel, three points at (11, 65), (9, 70), and (16, 120) are not shown for visualization purposes.
Supplementary Figure S9: Examples of reconstructions using GD across various training lengths. Increasing the training does not bring the generated image closer visually to the GT nor EGG.
Supplementary Figure S10: Reconstruction examples from the Gaussian model. Generated using EGG diffusion and gradient descent.
Supplementary Figure S11: Reconstructions from the Attention model. Top row in each panel is the ground truth image, middle is our EGG generated reconstruction and last row is the GD optimized reconstruction. Bar plot shows the performance of both EGG and GD in the within and cross architecture paradigms in terms of  $L_{2}$  distance.
eess
Gaussian Model Reconstructions from Real Neurons
Supplementary Figure S12: Re constructions from real neurons using the Gaussian model. Top row in each panel is the ground truth image, middle is our EGG generated reconstruction and last row is the GD optimized reconstruction. Bar plot shows the performance of both EGG and GD in the within and cross architecture paradigms in terms of  $L_{2}$  distance.
Reconstructions Performance
A
B
Supplementary Figure S13: Setup for the human perceptual evaluation. The voluntary participants were presented with 50 images with the GT image always in the middle and the GD or EGG reconstructions were placed randomly on each side of the GT image. The participants were provided with the question "Which of the images looks more like the image in the center?" They were provided with context text: "In our study, we are reconstructing images from the brain activity. We have two methods to do so and we want to find out which one looks better to the human eye. Your participation is entirely voluntary, and you have the right to withdraw at any time without providing a reason. Please note that all responses will be kept confidential, and the data collected will be used solely for research purposes. Your identity will remain anonymous, and your personal information will not be disclosed to anyone."