Brain decoding: toward real-time reconstruction of visual perception
Authors: Yohann Benchetrit
Hubert Banville
Jean-Remi King
Venue: ICLR 2024 poster (2024)
Keywords: brain decoding
neuroimaging
image generation
visual perception
Primary area: applications to neuroscience & cognitive science
TL;DR: Decoding perceived images from brain activity (MEG) data facilitates the study of visual perception dynamics.

Abstract: In the past five years, the use of generative and foundational AI systems has greatly improved the decoding of brain activity. Visual perception, in particular, can now be decoded from functional Magnetic Resonance Imaging (fMRI) with remarkable fidelity. This neuroimaging technique, however, suffers from a limited temporal resolution ($\approx$0.5\,Hz) and thus fundamentally constrains its real-time usage. Here, we propose an alternative approach based on magnetoencephalography (MEG), a neuroimaging device capable of measuring brain activity with high temporal resolution ($\approx$5,000 Hz). For this, we develop an MEG decoding model trained with both contrastive and regression objectives and consisting of three modules: i) pretrained embeddings obtained from the image, ii) an MEG module trained end-to-end and iii) a pretrained image generator. Our results are threefold: Firstly, our MEG decoder shows a 7X improvement of image-retrieval over classic linear decoders. Second, late brain responses to images are best decoded with DINOv2, a recent foundational image model. Third, image retrievals and generations both suggest that high-level visual features can be decoded from MEG signals, although the same approach applied to 7T fMRI also recovers better low-level features. Overall, these results, while preliminary, provide an important step towards the decoding - in real-time - of the visual processes continuously unfolding within the human brain.

Published as a conference paper at ICLR 2024
BRAIN DECODING: TOWARD REAL-TIME RECONSTRUCTION OF VISUAL PERCEPTION
Yohann Benchetrit' Hubert Banville
Jeanll- Jean-Rémi King*.²
FAIR, Meta, Laboratoire des Systèmes Perceptifs, École Normale Supérieure, PSL University (ybenchet rit, hubert jb, jeanremi) @eeee...aommmmmmm.oo.e. com
ABSTRACT
In the past five years, the use of generative and foundational Al systems has greatly improved the decoding of brain activity. Visual perception, particular. can now be decoded from functional Magnetic Resonance Imaging (fMRI) with remarkable fidelity. This neuroimaging technique, however, suffers from lim- ited temporal resolution (=0.5 Hz) and thus fundamentally constrains its real-time usage. Here, we propose an alternative approach based on magnetoencephalog- raphy (MEG), neuroimaging device capable of measuring brain activity with high temporal resolution (=55,00 Hz). For this, we develop an MEG decoding model trained with both contrastive and regression objectives and consisting of three modules: i) pretrained embeddings obtained from the image, ii) an MEG module trained end-to-end and iii) pretrained image generator. Our results are threefold: Firstly, our MEG decoder shows 7X improvement of image-retrieval over classic linear decoders. Second, late brain responses images are best de- coded with DINOv2, a recent foundational image model. Third, image retrievals and generations both suggest that high-level visual features can be decoded from MEG signals, although the same approach applied to 7T fMRI also recovers better low-level features. Overall, these results, while preliminary, provide an important step towards the decoding in real-time of the visual processes continuously unfolding within the human brain.
INTRODUCTION
Automating the discovery of brain representations. Understanding how the human brain rep- resents the world is arguably one of the most profound scientific challenges. This quest, which originally consisted of searching, one by one, for the specific features that trigger each neuron, (e.g-
Hubel & Wiesel| [1922; Keeeee & Nadel (197)); Kanwisher et al. (1977)) )), is now being automated by Machine Learning (ML. in two main ways. First, as a signal processing tool, MI algorithms are trained extract informative patterns of brain activity in data-driven manner. For example, Kami-
lani & Tong 2005) trained support vector machine to classify the orientations of visual gratings from functional Magnetic Resonance Imaging (IMRI). Since then, deep learning has been increas-
ingly used to discover such brain activity patterns (Roy et al. 2019 Thomas et al. 2022 Jayaram Barachant 2018 Défossez et al., 2022, Scotti al. 2023). Second, ML algorithms are used as functional models of the brain. For example, Yamins et al. (2014) have shown that the embedding of natural images pretrained deep nets linearly account for the neuronal responses to these images in the cortex. Since, pretrained deep learning models have been shown to account for a wide variety of stimuli including text, speech, navigation, and motor movement (Banino et al. 2018 Schrimpl
et al. 2020 Hausmann et al.] 2021: Mehrer et al. 202 Caucheteux al. 2023).
Generating images from brain activity. This observed representational alignment between brain activity and deep learning models creates new opportunity: decoding of visual stimuli need not be restricted to a limited set of classes, but now leverage pretrained representations to condi- tion subsequent generative AI models. While the resulting image may be partly "hallucinated", interpreting images can be much simpler than interpreting latent features. Following long series
"Equal contribution.

---

Published as a conference paper at ICLR 2024
of generative approaches (Nishimoto al. 2011 Kamitani & Tong 2005 VanRullen & Reddy 2019 Seeliger et al. 2018), diftusion techniques have, in this regard. sigmificanlly improved the generation of images from functional Magnetic Resonance Imaging (fMRI). The resulting pipeline typically consists of three main modules: (1) a set pretrained embeddings obtained from the im- age onto which (2) fMRI activity can be linearly mapped and (3) ultimately used to condition
pretrained mmage-generation model (Ozcelik & VanRullen 2023; Mai & Zhang 2023 Zeng et al.
2023 Ferrante et al. 2022 These recent IMRI studies primarily differ the type of pretrained
Saage-generatio model that they use.
The challenge of real-time decoding. This generative decoding approach has been mainly applied to fMRI. However, the temporal resolution of fMRI is limited by the time scale of blood flow and typically leads to one snapshot of brain activity every two seconds a time scale that challenges its clinical usage, e-g. for patients who require brain-computer-inteffaee (Willett al., 2023 Moses
et al. 2021 Metzger et al. 2023 Défossez et al. 2022). On the contrary, magnetoeneephallggaaal (MEG) can measure brain activity much higher temporal resolution (=x5000 Hz) by recording the fluctuation of magnetic fields elicited by the post-synaptic potentials of pyramidal neurons. This higher temporal resolution comes cost, however: the spatial resolution of MEG limited to z300 sensors, whereas fMRT measures x z100,000 voxels. In sum, fMRI intrinsically limits our ability to (1) track the dynamics of neuronal activity, (2) decode dynamic stimuli (speech, videos, etc.) and (3) apply these tools to real-time use cases. Conversely, it is unknown whether temporally- resolved neuroimaging systems like MEG are sufficiently precise to generate natural images real- time.
Our approach. Combining previous work on speech retrieval from MEG DDéfossez et al. 2022) and on image generation from IMRI (Takagi Nishimoto 2023 Ozcelik & VanRullen 2023), here develop three- module pipeline trained to align MEG activity onto pretrained visual embed- dings and generate images from a stream of MEG signals (Fig.
i
Images
model
Retrieval
Generation
Regression
MEG
Brain model
Identify
Generate
Image
Image
Align
Align
Align
Gen
Brain
Brain
Brain
Figure 1: (A) Approach. Locks indicate pretrained models. (B) Processing schemes. Unlike image generation, retrieval happens latent space, but requires the true image in the retrieval set.
Our approach provides three main contributions: MEG decoder (1) yields 7X increase in
performance as compared to linear baselines (Fig. 2 (2) helps reveal when high-level semantic
features are processed in the brain (Fig. and (3) allows the continuous generation of images from temporally-resolve brain signals (Fig. Overall, this approach thus paves the way to better understand the unfolding of the brain responses visual inputs.
METHODS
2.1 PROBLEM STATEMENT
We aim to decode images from multivariate time series of brain activity recorded with MEG as healthy participants watched a sequence of natural images. Let X, E RCT the MEG time window collected image 7, presented to the participant, where C' is the number of MEG
2

---

Published as a conference paper at ICLR 2024
channels, T is the number of time points in the MEG window and [1. with N the total number of images. Let 2; E R be the latent representation of L, with F the number of features, obtained by embedding the image using pretrained image model (Section 2.4.. As described in more detail below, our decoding approach relies on training a brain module fo RCxT + R to maximally retrieve predict I through 2;, given X,
2.2 TRAINING OBJECTIVES
We different training objectives for the different parts of our proposed pipeline. First, in the case of retrieval, we aim to pick the right image 1, (i.e., the one corresponding to X;) out of a bank of candidate images. To do so, train f using the CLIP loss (Radford al. 202T (i.e., the InfoNCE loss (Oord et al., 2018) applied both main-to-image and image-to-brait directions) on batches of size B with exactly one positive example,
(CLIP(O) 1 Σ i=1
exp(s(Zi, Z:)TT) + log exp(s(Z;; 2.)/T) log Σ exp(p (5(2), (2)2,21)/7) 2/)/T)
(I)
Σ k= exp (z( zi)/T)
where is the cosine similarity, z; and 2; fo (X;) are the latent representation and the correspond- ing MEG-based prediction, respectively, and is a learned temperature parameter.
Next, go beyond retrieval and instead generate images, train fo directly predict the latent representations 2 such that we can use them to condition generative image models. This is done using a standard mean squared error (MSE) loss over the (unnormalized) z; and 2;:
(MSE(
Σ
21
NF
Finally, we combine the CLIP and MSE losses using convex combination with tuned weight to train models that benefit from both training objectives:
Commbined = >LCLIP + MMSE
(3)
2.3 BRAIN MODULE
We adapt the dilated residual ConvNet architecture Défossez et al. (2022), denoted as fo, to learn the projection from an MEG window X; E RCx to latent image representation E R.F. The original model's output Ybacktone E I XT maintains the temporal dimension of the network through its residual blocks. However, here we regress a single latent per input instead of a sequence of T latents like in Défossez al.) (2022). Consequently, we add a temporal aggregation layer to reduce the temporal dimension of Y baackoone obtain ÿagg FR.... We experiment with three types of aggregations: global average pooling, a learned affine projection, and an attention layer. Finally, we add MLP heads, i.e., one for each term in CCombined. to project from F" to the F dimensions of the target latent. Additional details on the architecture can be found in Appendix A.I
We run a hyperparameter search to identify an appropriate configuration of preprocessing, brain module architecture, optimizer and CLIP loss hyperparameters for the retrieval task (Appendix A.2 The final architecture configuration for retrieval is described in Table ST and contains e.g. 6.4M trainable parameters for F 768. The final architecture uses two convolutional blocks and an affine projection to perform temporal aggregation (further examined in Appendix A.
For image generation experiments, the output of the MSE head is further postprocessed as in Ozcelik VanRullen (023), , i.c., we z-score normalize each feature across predictions, and then apply the inverse z-score transform fitted the training set (defined by the mean and standard deviation of each feature dimension on the target embeddings). We select À LCombined by sweeping over (0.0, 0.25, 0.5, 0.75) and pick the model whose top-5 accuracy is the highest the "large test set" (which is disjoint from the "small test set" used for generation experiments; see Section 2.8). When training models to generate CLIP and AutoKI. latents, simplify the task of the CLIP head
3

---

Published as a conference paper at ICLR 2024
by reducing the dimensionality of its target: use the CLS token for CLIP- Vision (FMSE 768), the "mean" " token for CLiP-Text (FMSE = 768), and the channel-average for AutoKL latents (FMSE 4096), respectively.
Of note, when comparing performance on different window configurations e.g. to study the dynam- ics of visual processing in the brain, train different model per window configuration. Despite receiving a different window of MEG as input, these models use the same latent representations of the corresponding images.
2.4 IMAGE MODULES
We study the functional alignment between brain activity and variety of (output) embeddings obtained from deep neural networks trained in three different representation learning paradigms, spanning a wide range of dimensionalities:: supervised learning (VGG-19), image-text alignment (CLIP), and variational autoencoders. When using vision transformers, we further include two ad- ditional embeddings of smaller dimensionality: the average of all output embeddings across tokens (mean), and the output embedding the class-token (CLS). For comparison, we also evaluate our approach human-engineered features obtained without deep learning. The list of embeddings is provided in Appendix A.3 For clarity, we focus our experiments on representative subset.
2.5 GENERATION MODULE
To fairly compare our work to the results obtained with fMRI results, follow the approach of Ozcelik & VunRullen 2023) and use a model trained to generate images from pretrained embed- dings. Specifically, we use latent diffusion model conditioned on three embeddings: CLIP-Vision (257 tokens x 768), CLIP-Text (77 tokens x 768), and variational autoencoder latent (AutoKI,; (4 x 64 64). In particular, we use the CLip-Text embeddings obtained from the THINGS object- category of stimulus image. Following Ozcelik & VanRullen (2023), we apply diffusion with 50 DDIM steps, a guidance of 7.5, a strength of 0.75 with respect to the image-to-image pipeline, and mixing of 0.4.
2.6 TRAINING AND COMPUTATIONAL CONSIDERATIONS
Cross-participant models are trained on a set of x63,000 examples using the Adam optimizer (Kingma & Ba 2014) with default parameters (3, =0.9, 8g=0.999), a learning rate of 3 10-4 and a batch size of 128. We use early stopping on a validation set of 155,800 examples randomly sampled from the original training set, with a patience of 10, and evaluate the performance of the model on held-out test set (see below). Models are trained a single Volta GPU with 32 GB of memory. We train each model three times using three different random seeds for the weight initialization of the brain module.
2.7 EVALUATION
Retrieval metrics. We first evaluate decoding performance using retrieval metrics. For a known test set, interested in the probability of identifying the correct image given the model predic- tions. Retrieval metrics have the advantage of sharing the same scale regardless of the dimensionality of the MEG (like encoding metrics) or the dimensionality of the image embedding (like regression metrics). We evaluate retrieval using either the relative median rank (which does not depend on the size of the retrieval set), defined the rank of a prediction divided by the size the retrieval set, or the top-5 accuracy (which is more common in the literature). In both cases, we use cosine similarity evaluate the strength of similarity between feature representations (Radford et al.) 2021).
Generation metrics. Decoding performance is often measured qualitatively well quantita- tively using a variety of metries reflecting the reconstruction fidelity both in terms of perception and semantics. For fair comparison with fMRI generations, we provide the same metrics as Ozcelik & VanRullen (023). computed between seen and generated images: PixCorr (the pixel-wise correla- tion between the true and generated images), SSIM (Structural Similarity Index Metric), and SwAV (the correlation with respect to Nes-Reeee output). On the other hand, AlexNet(2/5), Incep- tion, and CL.P the respective 2-way comparison scores of layers 2/5 of AlexNet, the pooled last

---

Published as a conference paper at ICLR 2024
layer of Inception and the output layer of CLIP. For the NSD dataset, these metries reported for participant only (see Appendix A.4)
To avoid non-representative cherry-picking, we sort all generations on the test set according to the sum of (minus) SwAV and SSIM. We then split the data into 15 blocks and pick images from the best, middle and worst blocks with respect to the summed metric (Figures S2 and S5
Real-time and average metries. It is common in fMRI to decode brain activity from prepro- cessed values estimated with a General Linear Model. These "beta values" are estimates of brain responses to individual images, computed across multiple repetitions of such images. To provide fair assessment of possible MEG decoding performance, we thus leverage repeated image presen- tations available in the datasets (see below) by averaging predictions before evaluating metries and generating images.
2.8 DATASET
We test our approach on the THINGS- dataset (Hebart et al. 2023). Four participants (2 female, male; mean age of 23.25 years), underwent 12 MEG sessions during which they were pre- sented with set of 22,448 unique images selected from the THINGS database (Hebart et al.. 2019). covering 1,854 categories. Of those, only subset 200 images (each one of a different category) was shown multiple times the participants. The images were displayed for 500 ms each, with variable fixation period of 10001200 ms between presentations. The THINGS dataset additionally contains 3,659 images that were not shown to the participants and that we use augment the size of our retrieval set and emphasize the robustness of our method.
MEG preprocessing. We use a minimal MEG patt-preppoo pipeline as in Défossez et al. Raw data from the 272 MEG radial gradiometer channels is downsampled from 1,200 Hz to Hz. The continuous MEG data is then epoched from -500 ms to 1,000 ms relative to stimulus onset and baseline-corrected by subtracting the mean signal value observed between the start of an epoch and the stimulus onset for each channel. Finally, we apply a channel-wist robust scaler Pedrrgosa al. 2011 and clip values outside of [-20, 20) minimize the impact of large outliers.
2022
Splits. The original split of Hebart et al. (2023) consists of 22,248 uniquely presented images, and 200 test images repeated 12 times each for each participant (i.e., 2,400 trials per participant). The use of this data split presents challenge, however, as the test set contains only one image per category, and these categories are also seen in the training set. This means evaluating retrieval performance on this test set does not measure the capacity of the model to (I) extrapolate to new unseen categories of images and (2) recover a particular image within a set of multiple images of the same category, but rather only to "categorize" it. Consequently, we propose two modifications of the original split. First, we remove from the training set any image whose category appears in the original test set. This "adapted training set" removes any categorical leakage across the train/test split and makes it possible to assess the capacity of the model to decode images of unseen image categories (i.e., a "zero-shot" setting). Second, we propose a new "large test set" that is built using the images removed from the training set. This new test set effectively allows evaluating retrieval performance of images within images of the same category We report results on both the original ("small") and the "large" test sets enable comparisons with the original settings of Hebart et al. (2023.. Finally, we also compare our results to the performance obtained by similar pipeline but trained on IMRI data using the NSD dataset (Allen et al. 2022
(see Appendix A.4.
3 RESULTS
ML as an effective model of the brain. Which representations of natural images are likely to maximize decoding performance? To answer this question, we compare the retrieval performance obtained by linear Ridge regression models trained to predict one of 16 different latent visual rep- resentations given the flattened MEG response X; each image 1; (see Appendix A.5 and black
We leave out images of the original test set from this new large test set, keeping them would create discrepancy between the number of MEG repetitions for training images and test images.
5

---

Published as a conference paper at ICLR 2024
transparent bars in Fig. 2 While all image embeddings lead to above-chance retrieval, supervised and text/image alignment models (e.g. VGG, CLIP) yield the highest retrieval scores.
ML as an effective tool learn brain responses. We then compare these linear baselines deep ConvNet architecture (Délossez et al. 2022) trained the same dataset to retrieve the match- ing image given an MEG window Using deep model leads to 7X improvement over the linear baselines (Fig. 2 Multiple types of image embeddings lead to good retrieval performance, with VGG-19 (supervised learning), CLIP- Vision (text/image alignment) and DINOv2 (self-supervised learning) yielding top-5 accuracies of 70.33=2.80%% 68. 08.66.42.8499 8.4%
2.86% respectively (where the standard error of the mean is computed across the averaged image-wise metrics). Similar conclusions, although with lower performance, decoding cannot rely solely on the image category but also requires discriminating between multiple images of the same category. Representative retrieval examples are shown in Appendix A.7
be drawn from our "large" test set setting, where
Large
CLIP-Visoon (mean)-
CLIP-Vision (CLS)
CLiP-Text (mean)
Latent kind
Color histogram
Image/text allgnment
real)
Feature engineering
HOG
Variational autoencoder
Local binary patterns (LBP)
Self supervipe leamiing
FFT 20 Ilog-PSD and angle)
Supervised learning
Linear baseline
AutoKL
VOVAE
DINOv2
VGG-19
Top-5 acc (%)
Top-5 acc (96)
Figure 2: Image retrieval performance obtained from a trained deep ConvNet. Linear decoder base- line performance (see Table (22 is shown with a black transparent bar for each latent. The original "small" test set (Hebart al. 2023) comprises 200 distinct images, each belonging to different category. In contrast, our proposed "large" test set comprises 12 images from each of those 200 categories, yielding a total of 2,400 images. Chance-level is 2.5% top-5 accuracy for the small test set and 0.21 % for the large test set. The best latent representations yield accuracies around 70% and 13% for the small and large test sets, respectively.
Temporally--reoovee image retrieval. The above results are obtained from the full time window (-500 to 1,000 ms relative to stimulus onset). To further investigate the feasibility of decoding visual representations as they unfold in the brain, we repeat this analysis on 100-ms sliding windows with stride of 25 ms (Fig. B). For clarity, we focus a subset of representative image embeddings. As expected, all models yield chance-level performance before image presentation. For all embeddings, first clear peak can be observed for windows ending around 200-275 ms after image onset. A second peak follows for windows ending around 1500200 ms after image offset. Supplementary analysis (Fig. (S7) further suggests these two peak intervals contain complementary information for the retrieval task. Finally, performance quickly goes back to chance-level. Interestingly. the recent self-supervised model DINOv2 yields particularly high retrieval performance after image offset.
Representative time-resolved retrieval examples are shown Appendix A.7 Overall, the retrieved images tend come from the correct category, such as "speaker" or "brocoli", mostly during the first few sub-windows (f < s). However, these retrieved images do not appear to share obvious low-level features the images seen by the participants.
While further analyses of these results remain necessary, it seems that (1) our decoding leverages the brain responses related both the onset and the offset of the image and (2) category-level information dominates these visual representations as early as 250 ms.
We use À in Coombined solely concerned with the retrieval part of the pipeline here.

---

Published as a conference paper at ICLR 2024
Small test set
Large test set
50
6-
acc (%)
40
Latent
4-
CLIP-Vision (CLS)
30
Color histogram
AutoKL
20
DINOv2 (CLS)
10
VGG-19
0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Emax (5)
tmax (s)
Figure 3: Retrieval performance of models trained on 100-ms sliding windows with a stride of 25 ms for different image representations. The shaded gray area indicates the 500-ms interval during which images were presented to the participants and the horizontal dashed line indicates chance- level performance. Accuracy peaks a few hundreds of milliseconds after both the image onset and offset for all embeddings.
Table 1: Quantitative evaluation of reconstruction quality from MEG data on THINGS-MEG (com- pared fMRI data on NSD (Allen al. 2022) using a cross-valiie valldaeadd Ridge regression). We report PixCorr, SSIM, AlexNet(2), (S)) Inception, SwAV and CLIP and their SEM when meaning- ful. particular, this shows that fMRI betas as provided in NSD are significantly easier to decode than MEG signals from SHINGS-MEG
Low-level
High-love
Dataset
PixCorr 1
SSIM 1
AlexNer(2)1 1
AlexNet(5)
Inception
CLIP
SWAV
0.917
MEGG
(averaged across all trials within subject)
0.076 = 0.005
0,336
0.007
0.736
0.826
0.671
0.767
0.844 = 0.004
THINGS-MER
(averaged acmes
subjects)
THINGS-MEG
0.088 0.011
0.327 0.014
0.695
0.753
0.593
0,700)
0.630
0.007
(no average)
Generating images from MEG. While framing decoding as a retrieval task yields promising results, it requires the true image to be in the retrieval set well well-posed problem which presents limited use-cases in practice. To address this issue, we trained three distinct brain modules to predict the three embeddings that generations from (A) "growing" windows, i.e., where increasingly larger MEG windows (from [O, 100] [O, 1,500] ms after onset with 50 increments) are used condition image generation and (B) full-length windows (i.e., -500 to 1,000 ms). Additional full-window representative generation examples are shown in Appendix A.8 As confirmed by the evaluation metrics of Table | (see Table S4 for participant the true mage. However, most generations appear preserve a relatively small amount low-level features, such as the position and color of each object. Lastly, we provide a sliding window analysis of these metrics in Appendix A. These results suggest that early responses to both image onset and offset primarily associated with low-level metrics, while high-level features appear more related to brain activity in the 200-500 ms interval.
(see Section 2.5) to generate images. Fig. 4 shows example
metrics), many generated images preserve the high-level category of
The application of very similar pipeline & VanRullen 2023) using a simple Ridge regression - shows image reconstructions that share both high-level and level features with the true image (Fig. S2. Together, these results suggest that it is not the reconstruction pipeline which fails to reconstruct low-level features, but rather the MEG signals which comparatively harder to decode.
analogous fMRI dataset Allen al. 2022 Ozcelik

---

Published as a conference paper at ICLR 2024
A
Seen
100 ms
150 ms
200 ms
250 ms
450 ms
800 ms
1500 ms
B
Generaced
Figure 4: Handpicked examples of successful generations. (A) Generations obtained on growing windows starting at image onset mmms) and ending at the specified time. (B) Full wildow genera- tions (-500 to 1,000 ms).
4 DISCUSSION
Related work. The present study shares several elements with previous MEG and electroen- cephalography (EEG) studies designed not to maximize decoding performance but to understand the cascade of visual processes the brain. In particular, previous studies have trained linear mod- els to either (1) classify a small set of images from brain activity (Grootswagers et al. 2019 King Wyart 202T), (2) predict brain activity from the latent representations of the images (Cichy al.)
2017) (3) quantify the similarity between these two modalities with representational similarity
analysis (RSA) (Cichy et al., 2017 Bankson et al. 2018, Grootswagers al. 2019 Gifford al.
2022). While these studies also make use of image embeddings, their lincar decoders are limited to classifying a small set of object classes, or to distinguishing pairs images.
In addition, several deep neural networks have been introduced to maximize the classification of speech (Défossez et al. 2022), mental load (Jiao et al. 2018) and images PPllazoo et al., 2020 McCartney et al. 2022 Bagchi & Bathula 2022) from EEG recordings. In particular, Palazzo al.
(2020) introduced a deep convolutional neural network to classify natural images from EEG signals.
However, the experimental protocol consisted of presenting all of the images of the same class within
8

---

Published as a conference paper at ICLR 2024
single continuous block, which risks allowing the decoder to rely autocorrelated noise, rather than informative brain activity patterns (Li et al. 2020). In any case, these EEG studies focus on the categorization of relatively small number of images classes.
In sum, there is, to our knowledge, no MEG decoding study that learns end-to-end to reliably gen- erate an open set of images.
Impact. Our methodological contribution has both fundamental and practical impacts. First, the decoding of perceptual representations could clarify the unfolding of visual processing in the brain. While there is considerable work on this issue, neural representations are challenging to interpret because they represent latent, abstract, feature spaces. Generative decoding, the contrary, can provide concrete and, thus, interpretable predictions. Put simply, generating images at each time step could help neuroscientists understand whether specific potentially unanticipated textures or object parts are represented. For example, [Cenn et al.] (2023) showed that generative decoding applied to IMRI can be used decode the subjective perception of visual illusions. Such techniques can thus help to clarify the neural bases of subjective perception and to dissociate them from those responsible for "copying" sensory inputs. Our work shows that this endeavor could now be applied to clarify when these subjective representations arise. Second, generative brain decoding has con- crete applications. For example, has been used conjunction with encoding, to identify stimuli that maximize brain activity (Bashivan et al. 2019). Furthermore, non-invasive brain-computer in- terfaces (BCI) have been long-aw long-awaited by patients with communication challenges related to brain lesions. BCI, however, requires real-time decoding, and thus limits the use of neuroimaging modal- ities with low temporal resolution such as fMRL. This application direction, however, will likely require extending our work to EEG, which provides similar temporal resolution to MEG, but is typically much more common in clinical settings.
Limitations. Our analyses highlight three main limitations to the decoding of images from MEG signals. First, generating images from MEG appears worse at preserving low-level features than a similar pipeline on 7T fMRI (Fig. S2). This result resonates with the fact that the spatial resolution of MEG (+= cm) is much lower than 7T fMRI's (* mm). Moreover, and consistent with previous find- ings (Cichy al. 2014 Hebart et al. 2023), the low-level features can be predominantly extracted from the brief time windows immediately surrounding the onset and offset of brain responses. As a result, these transient low-level features might have a lesser impact on image generation compared to the more persistent high-level features. Second, the present approach directly depends on the pre- training of several models, and only learns end-to-end embeddings. Our results show that this approach leads to better performance than classical computer vision features such as color histograms, Fast Fourier transform and histogram of oriented gradients (HOG). This is consistent with a recent MEG study by Défossez et al. (2022) which showed, in the context of speech decoding, that pretrained embeddings outperformed fully end-to-end approach. Nevertheless, it remains to be tested whether (1) fine-tuning the image and generation modules and (2) combining the different types of visual features could improve decoding performance.
align the MEG signals to these pretrained
Ethical implications. While the decoding of brain activity promises to help a variety of brain- lesioned patients (Metzger et al. 2023 Moses et al. 2021 Détossez et al. 2022 Liu et al. 2023 Willett et al. 2023). the rapid advances of this technology raise several ethical considerations, and most notably, the necessity to preserve mental privacy. Several empirical findings are relevant to this issue. Firstly, the decoding performance obtained with non-invasive recordings is only high for per- ceptual tasks. By contrast, decoding accuracy considerably diminishes when individuals are tasked to imagine representations (Horikawa & Kamitani 2017 Tang al. 2023). Second, decoding per- formance seems to be severely compromised when participants are engaged in disruptive tasks, such as counting backward (Tang al. 2023). In other words, the subjects" consent is not only a legal but also and primarily a technical requirement for brain decoding. To delve into these issues effectively, we endorse the open and peer-reviewed research standards.
Conclusion. Overall, these results provide an important step towards the decoding of the visual processes continuously unfolding in the human brain.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was funded in part by FrontCog grant ANR-17-EURE-0011 JRK for his work at PSL.
REFERENCES
Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob Prince, Logan Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 2661):116-126, 2022.
Subhranil Bagchi and Deepti R Bathula. BOGG-Con/Transformee for single-trial EEG-based visual stimulus classification. Pattern Recognition, 129:108751 2022.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: arXiv: 1409.0433, 2014.
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap. Piotr Mirowski, Alexander Pritzel, Martin Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(77065):29-4433 2018.
B.B. Bankson, M.N. Hebart, I.A.. Groen, and C.I. Baker. The temporal evolution of conceptual object representations revealed through models of behavior, semantics and deep neural networks.
Neurolmage, 178:172-182, 2018. ISSN 1053-8119 19. doi: http:://doi.rr//10.000666 nerroomma.e 2018.05.037. URL https:/ //tp://wwww..mm sci encedi rect com http.c//cccceeeeeeattcclleeeee/e 91053811918300
Pouya Bashivan, Kohitij Kar, and James DiCarlo. Neural population control via deep image synthesis. Science, 364(6433):caav4430 2019.
G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
Thomas Carlson, David A Tovar, Arjen Alink, and Nikolaus Kriegeskorte. Representational dynam- ics of object vision: the first 1000 ms. Journal of vision, 13(10))1-1,2011 2013.
Thomas A Carlson, Hinze Hogendoorn, Ryota Kanai, Juraj Mesik, and Jeremy Turret. High tempo- ral resolution decoding of object position and category. Journal of vision, 11(10):9-9, 2011.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Evidence of predictive coding hierarchy in the human brain listening to speech. Nature human behaviour, 7(3):30-441, 2023.
Fan Cheng, Tomoyasu Horikawa, Kei Majima, Misato Tanaka, Mohamed Abdelhack, Shuntaro Aoki, Jin Hirano, and Yukiyasu Kamitani. Reconstructing visual illusory experiences from human brain activity. bioRxiv, pp. 2023-06, 2023.
Radoslaw Martin Cichy, Dimitrios Pantazis, and Aude Oliva. Resolving human object recognition in space and time. Nature neuroscience, 17(3):455-422, 2014.
Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, and Aude Oliva. Dynamics of scene representations in the human brain revealed by magnetoenccepaaloraphhy and deep neural net- works. Neurolmage, 153:346-358, 2017.
Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. De- coding speech from non-invasive brain recordings. arXiv preprint arXiv:2208. /2266, 2022.
Matteo Ferrante, Tommaso Boccato, and Nicola Toschi. Semantic brain decoding: from fMRI to conceptually similar image reconstruction of visual stimuli. arXiv preprint arXiv:2212.0672.
2022.
Alessandro T Gifford, Kshitij Dwivedi, Gemma Roig, and Radoslaw M Cichy. A large and rich EEG dataset for modeling human visual object recognition. Neurolmage, 264:11975: 2022.
Tijl Grootswagers, Amanda K Robinson, and Thomas A Carlson. The representational dynamics of visual objects in rapid serial visual processing streams. Neurolmage, 188:668-679, 2019.
10

---

Published as a conference paper at ICLR 2024
Sébastien B Hausmann, Alessandro Marin Vargas, Alexander Mathis, and Mackenzie W Mathis. Measuring and modeling the motor system with machine learning. Current opinion in neurobiol- ogy, 70:11-23, 2021.
Martin N Hebart, Adam H Dickter, Alexis Kidder, Wan Kwok, Anna Corriveau. Caitlin Van Wick- lin, and Chris Baker. THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PloS one, 14(10):0023379. 2019.
Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Yaziri-Pashkann and Chris I Baker. THINGS-data, multi- modal collection of scale datasets for investigating object representations in human brain and behavior. eLife, 12:c82580, feb 2023. ISSN 2050-084X. doi: 10.7554/cLife.82555 82588. URL
https: /oi. org 10. 7554/ eLi fe. 82580
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1): 2017.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional archi- tecture the cat's visual cortex. The Journal of physiology, 160(1):106, 1962.
Vinay Jayaram and Alexandre Barachant. MOABB: trustworthy algorithm benchmarking for beis. Journal of neural engineering, 15(6):066011, 2018.
Zhicheng Jiao, Xinbo Gao, Ying Wang. Jie Li, and Haojun Xu. Deep convolutional neural networks for mental load classification based EEG data. Pattern Recognition, 76:582-595, 2018.
Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature neuroscience, 8()):679-685, 2005.
Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: module in human extrastriate cortex specialized for face perception. Journal of neuroscience, 17(11):4302- 4311, 1997.
Jean-Rémi King and Valentin Wyart. The human brain encodes a chronicle of visual events each instant of time through the multiplexing of traveling waves. Journal of Neuroscience, 41(4)): 7224-7233, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.6980 2014.
Ren Li, Jared S Johansen, Hamad Ahmed, Thomas V Ilyevsky, Ronnie B Wilbur, Hari Bharad- waj. and Jeffrey Mark Siskind. The perils and pitfalls of block design for EEG classification experiments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1361)3316-333, 2020.
Yan Liu, Zehao Zhao, Minpeng Xu, Haiqing Yu, Yanming Zhu, Jie Zhang, Linghao Bu, Xiaoluo Zhang, Junfeng Lu, Yuanning Li, et al. Decoding and synthesizing tonal language speech from brain activity. Science Advances, 9(23))eadh0478, 2023.
Weijian Mai and Zhijun Zhang. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint wrXiv:230.007226 2023.
Ben McCartney, Barry Devereux, and Jesus Martinez-del Rincon. zero-shot deep metric learn- ing approach to brain-computer interfaces for image retrieval. Knowledge-Rased Systems, 246: 108556, 2022.
Johannes Mehrer, Courtney J Spoerer, Emer C Jones, Nikolaus Kriegeskorte, and Tim C Kietzmann. An ecologically motivated image dataset for deep learning yields better models of human vision. Proceedings of the National Academy of Sciences, 118(8):c201 1417118, 2021.
Sean L Metzger, Kaylo T Littlejohn, Alexander B Silva, David A Moses, Margaret P Seaton, Ran Wang, Maximilian E Dougherty, Jessie Liu, Peter Wu, Michael A Berger, et al. A high- performance neuroprosthesis for speech decoding and avatar control. Nature, pp. 1-10, 2023.

---

Published as a conference paper at ICLR 2024
David A Moses, Sean L Metzger, Jessie R Liu, Gopala K Anumanchipalli, Joseph G Makin, Pengfei Sun, Josh Chartier, Maximilian E Dougherty. Patricia M Liu, Gary M Abrams, et al. Neuroprosthesis for decoding speech in a paralyzed person with anarthria. New England Journal of Medicine, 385(3):217-227, 2021.
Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Re- constructing visual experiences from brain activity evoked by natural movies. Current biology, 21 21(19):641-1144, 2011.
John O'Keefe and Lynn Nadel. The hippocampus cognitive map. Behavioral and Brain Sci- ences, 2(4))487-494, 1979.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748. 2018
Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using genera- tive latent diffusion. Scientific Reports, 13(1):15666, 2023.
Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Danicla Giordano, Joseph Schmidt, and Mubarak Shah. Decoding brain representations by multimodal learning of neural activity and visual features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43 (1)333)33- 3849, 2020.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Yannick Roy, Hubert Banville, Isabela Albuquerque, Alexandre Gramfort, Tiago H Falk, and Joce- lyn Faubert. Deep learning-based electroenceehallggrapyy analysis: a systematic review. Journal of neural engineering, 16(5):051001, 2019.
Martin Schrimpf, Idan Blank, Greta Tuckute, Carina Kauf, Eghbal Hosseini, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. Artificial neural networks accurately predict lan- guage processing in the brain. BioRxiv, pp. 2020-06, 2020.
Paul S Scotti, Atmadeep Banerjee, Jimmic Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, et al. Reconstructing the mind's s eye: IMRI-to-image with contrastive learning and diffusion priors. arXiv preprint arXiv:2305. 18274, 2023.
Katja Seeliger, Umut Güçlüü Luca Ambrogioni, Yagmur Güęlütürk,, and Marcel AJ van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolm- age, 181:775-785, 2018.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv, 2023. doi: 111p:////222.....1111......11..1 URL https: 11, 18.517004
// //www. biorxiv..vvvv www.bbiwwww..o.//ooeeeeeeeeee
Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. Semantic reconstruction of con- tinuous language from non-invasive brain recordings. Nature Neurosience, pp. 1-9, 2023.
Armin Thomas, Christopher Ré, and Russell Poldrack. Self-supervised learning of brain dynamics from broad neuroimaging data. Advances in Neural Information Processing Systems, 32:51 21 21269, 2022.
Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. scikit-image: image processing in python. PeerJ, 2:e453, 2014.
12

---

Published as a conference paper at ICLR 2024
Rufin VanRullen and Leila Reddy. Reconstructing faces from fMRI patterns using deep generative neural networks. Communications biology. 2(1):193, 2019.
Francis R Willett, Erin M Kunz, Chaofei Fan, Donald T Avansino, Guy H Wilson, Eun Young Choi, Foram Kamdar, Matthew F Glasser, Leigh R Hochberg, Shaul Druckmann, al. high- performance speech neuroprosthesis. Nature, pp. 1-6, 2023.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optiminn hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):661--6222 2014.
Bohan Zeng, Shanglin Li, Xuhui Liu, Sicheng Gao, Xiaolong Jiang, Xu Tang. Yao Hu, Jianzhuang Liu, and Baochang Zhang. Controllable mind visual diffusion model. arXiv preprint arXiv:2305. 10135, 2023.
13


Reviews:
[{"review_id": "aZRciEFWrN", "reviewer": "Reviewer_4PGh", "summary": "This contribution concerns the interesting topic of decoding/retrieval and reconstructing of visual input from MEG data (THINGS-MEG data set). The approach is based on representations of images and MEG data using multiple architectures and multiple levels of generalization.\n\nThere is a rich literature on decoding and reconstructing visual and audio stimulus from brain recordings, so novelty is somewhat limited.\n\nDecoding is evaluated as retrieval in closed and open set conditions (the latter using zero-shot setting).\nRetrieval is based on linking by learning to align MEG and image representations \n\nThe reconstruction of visual input is based on generative models, using frameworks that have been developed elsewhere (Ozcelik and Van Rullen). \n\nCompared to the very rich literature on methods based on MEG and other modalities, this study has an increased focus on temporal resolution of the retrieval process and furthermore, they use diffusion models for conditional generation.", "questions": "Based on MEG we have high time resolution and SNR. In the temporally resolved analysis, it is interesting that VGG outperforms the more advanced for the direct image (after onset) while the more complex image representations dominate retrieval based on the after image (after image offset). Missing a discussion of this interesting finding.\n\nThe generative performance while evaluated in a number of metrics with good consistency among the metrics. Yet, we are missing uncertainty estimates to weigh the evidence in this case\n\nVisually the generated imagery is intriguing. However, we miss a discussion of the notable lack of fine grained semantic relatedness (generation seems primarily to pick up on texture, object scale(?) and high-level semantics eg. man-made vs natural)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Compared to the very rich literature on methods based on MEG and other modalities, this study has increased focus on temporal resolution of the retrieval and furthermore using sota diffusion models for conditional generation.\nIt is concluded that retrieval interesting peaks following image onset and image offset (the latter based on the after-image presumably). Retrieval performance is good for several image representations (VGG and DINOv2)\nThe generative performance is evaluated in a number of metrics, there is good consistency among the metrics.\nVisually the generation makes sense.\nUseful to see examples  stratified over good, bad and ugly cases.", "weaknesses": "There is a rich literature on decoding and reconstructing visual and audio stimulus from brain recordings, so novelty is somewhat limited.\n\nBased on MEG we have high time resolution and SNR. In the temporally resolved analysis, it is interesting that VGG outperforms the more advanced representations for the direct image (after image onset) while the more complex image representations dominate retrieval based on the after-image (following image offset). We miss a discussion of this interesting finding.\n\nThe generative performance is evaluated in a number of metrics with good consistency among the metrics. Yet, we are missing uncertainty estimates to weigh the evidence in this case\n\nVisually the generated imagery is intriguing. However, we miss a discussion of the notable lack of fine grained semantic relatedness (generation seems primarily to pick up on texture, object scale(?) and high-level semantics eg. man-made vs natural)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MkclGYsUfK", "reviewer": "Reviewer_hpxq", "summary": "In this paper the authors propose a method to decode brain activity. The main idea is to train an MEG decoder which maps MEG signals to a feature space which is then used to reconstruct images using a pretrained image generator. \n\nThe authors show that MEG decoder which is a DNN leads to 7 times improvement over linear decoders which is a common approach in neuroscience studies. Image generation results suggests that it is possible to reconstruct semantically accurate from MEG activity while low-level details are difficult to reconstruct.", "questions": "Please refer to weaknesses section for points to address in rebuttal. \n\nOverall this paper has some new contributions but authors make some claims which do not have sufficient support in the results. Therefore, my recommendation would be to either tone down the claims or present good evidence to back them up", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. 7x improvement in decoding accuracy over linear decoders. This is an important result which will encourage neuroscience researchers to use DNNs for decoding MEG/fMRI signals.\n2. Clear presentation of methods (Figure 1, Section2).", "weaknesses": "1. The reconstruction results are not impressive. Even the best examples shown in Figure 5 often do not have the reconstructions of image of same or related category.  Therefore, the title is misleading as the main contribution of this paper in my opinion is DNN based MEG decoder and retrieval results and is not correctly reflected in the title.\n2. The decoder is trained using a combination of two loss functions : MSE loss and CLIP loss (equation 3, line 91). There seems to be no ablation study investigating what is the impact of each loss function in retrieval performance. There is one figure in supplementary material Fig S2 E but I am not sure whether it indicates two terms of CLIP loss or two terms of overall loss (CLIP + MSE).\n3. In Line 110 authors mention that they select  lambda by sweeping over {0.0, 0.25, 0.5, 0.75, 1.0} and pick the model whose top-5 accuracy is the highest on the large test. Is the hyperparameter search for lambda done on test data?\n4. The claim in the abstract \"MEG signals primarily contain high-level visual features\" does not have sufficient evidence based on the reconstruction results only. It has been shown in literature (even in Things dataset paper Figure 8) that fMRI responses of early visual cortex (which can decode low-level features) are correlated with MEG responses (Cichy et al. 2014, Hebart et al. 2023) in early time windows. Therefore, a stronger evidence is required to back this claim. A possible explanation why the reconstructions can not recover low-level details might be that temporal aggregration layers leads to suppresion of low-level features which are present in a smaller time-window around 100ms. Another possible explanation is that we are predicting a high-level feature  (DINOv2/CLIP etc.) from MEG which may not need information from low-level features and thus the image generated also lack these details. \n5. The main result of the paper is 7x improvement over linear decoders. It is not clear where exactly this result is in the paper. A reader needs to compare results in supplementary and Figure 2 in the main text. Simply adding shaded bar in Figure 2 for linear decoder next to each bars can improve clarity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "idTWMdd4zt", "reviewer": "Reviewer_hWdQ", "summary": "In this paper, the authors developed a model based on contrastive and regression objectives to decode MEG, resulting in 7X improvement in image retrieval over a classic linear decoder. The promising results in image retrieval and generation are significant in that the presented approach allows the monitoring of the unfolding of visual processing in the brain based on MEG signals, which have much higher temporal resolution than fMRI. The work yields two potentially interesting observations: (1) late responses are best decoded with DINOv2, and (2) MEG signals contain high-level features, whereas 7T fMRI allows the recovery of low-level features, though it would be worthwhile to articulate or speculate what these findings mean for understanding the cascade of visual processes in the brain.", "questions": "What do the two observations tell us about the unfolding of visual perceptual processes?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The work is significant in that there is no MEG decoding study that learns end-to-end to reliably generate an open set of images. Thus, it can potentially be considered a ground-breaking in this area of research, even though the techniques used are not necessarily novel from an ML perspective.", "weaknesses": "The decoding work is supposed to provide new insights to the cascade of visual processing and the unfolding of visual perception in the brain.  The authors need to articulate better what insights the current observations (mentioned in the Summary) actually provide us.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pn5gD8wJT6", "reviewer": "Reviewer_F12m", "summary": "The authors perform MEG conditioned visual decoding.\n\nCompared to other works that leverage fMRI, MEG is a different information source that presents unique challenges.\n\nThe authors use an align then generate strategy, where they learn a function that takes as input the MEG signal, and train it to align with a CLIP latent using a weighted sum of infoNCE and MSE loss. For image generation, they use Versatile Diffusion and regress the needed conditioning variables from MEG. \n\nThey observe that it is possible to recover high level semantics in the reconstructed images.", "questions": "Please see the weakness's section for questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "MEG decoding of full images is an under-explored area compared to fMRI based decoding, and it is a harder task, given the low channel count relative to the tens of thousands of voxels in fMRI. To my knowledge, this is the first time that image decoding has been demonstrated using MEG.\n\nThe paper is methodologically sound, outlining different training objectives for different parts of the proposed pipeline. The paper provides systematic benchmarks, showing that their MEG decoder leads to reasonable image retrieval and image generation.\n\nI applaud the authors for showing \"representative\" retrieval and best/mean/worst decoding results, which helps gauge the effectiveness of the method.\n\nIt is also interesting that they found MEG capable of recovering high level semantics. Although it is not fully clear if this is a limitation of MEG or their method (should probably discuss more).", "weaknesses": "As with other deep decoding papers, it is not super clear what the ultimate scientific insight is. This is not a criticism specific to this paper, but more generally aimed at current decoding works which leverage powerful image priors and deep non-linear decoding/embedding functions. \n\nIn this aspect, this paper is better than most, as their Figure 3 provides some insight on the temporal dynamics of decoding. I think it would benefit the paper to add some discussion (not necessarily experiments) on extending this to EEG based decoding, or other potential practical applications or scientific insights. \n\n**General clarifications:**\n\nThe clarity of many of their methods could be improved. The author repeatedly references Defossez et al. in reference to their methods. But in the main text it is not super clear. Concretely I would like the authors to clarify the following:\n1. What the the MEG conv \"encoder\" convolving over?\n\n2. How do you combine the MEG channels?\n\n3. What temporal aggregation layer did you use? You mention global pooling, affine, and attention. Which layer did you end up using? Because you discuss this and then never talk about which method you ended up using.\n\n4. How do the different aggregation layers work? I ask this question in the context of Figure 3. Because you discuss using a 1500ms window, then shift to a 250ms window. Do you train a new model? Do you re-use the 1500ms model but change the aggregation? If you do train a new model, are you taking multiple 250ms windows and supervising with the same image target? For the sliding window, what is the step size?\n\n5. For Figure 2, in the supervised models (VGG, ResNet, etc.) are you using the last layer (1000 imagenet classes layer), or the post-pooled layer. \n\n6. For retrieval, are you always using cosine/dot-product similarity?\n\n**Minor format error:**\n1. The authors have ICLR 2023 in the header, when it should be ICLR 2024. And they have line numbers, which do not seem to be present in the default ICLR template.\n\n**Minor clarifications:**\n1. Can you clarify if $N$ (line 75) denotes the number of images? It doesn't seem like you define $N$ prior/after using it.\n2. To provide more context, can you mention in line 84 that you are using the infoNCE loss, rather than just mentioning the CLIP loss.\n3. In section 2.2, can you clarify if you are normalizing $\\hat{z}$ to norm = 1 for eq. 1, and not assuming a fixed norm for eq. 2? Otherwise it seems like the two losses would have trivially the same optima, but I guess you are trying to have one loss align the direction, and have a second loss align the direction + norm.\n\n**Additional citations:**\n\nThe author discusses one approach towards decoding, but I would appreciate if the author could also discuss the brain gradient conditioned image generation work listed below, the most recent of which also leverage GANs/Diffusion models:\n\nInception loops discover what excites neurons most using deep predictive models (**Nature 2019**); Neural population control via deep image synthesis (**Science 2019**); Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences (**Cell 2019**); Computational models of category-selective brain regions enable high-throughput tests of selectivity (**Nature Communications 2021**); NeuroGen: Activation optimized image synthesis for discovery neuroscience (**Neuroimage 2022**); Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models (**NeurIPS 2023**); Energy Guided Diffusion for Generating Neurally Exciting Images (**NeurIPS 2023**)\n\nOverall I think the paper is sound, interesting, and provides good insight on neural decoding from an often overlooked modality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)