dataesr/scientific-paragraphs-categorization huggingface dataset:
, 2019; Miyazaki et al. , 2016). All participants provided written informed consent to take part in the study and were given a compensation of 45€ for their participation. The study has been approved by the local ethics committee (i. e. Comité d'Éthique de Recherche) of the University of Strasbourg. Experimental Protocol. The experiment used the Unity software  (Unity technologies, v. 2019. 3. 9f1)  to create the virtual environment. The HTC Vive Eye Pro (HTC Corp. ) controllers and headset were used to immerse the participants in VR. Participants wore both the VR and EEG headsets while sitting on a chair. Several benefits motivated the use of VR to perform the task. Firstly, it facilitates the engagement of the participant to the task at hand. Secondly, VR allows a better trade-off between fully-controlled experimental settings and conditions close to everyday life (e.
The same four robots were displayed across tasks but only the two central ones were used for the simultaneity judgment task. The participant switched tasks every block to reduce boredom. At the end of each experimental block, a break was proposed to the participant to remove the VR headset. Given the research questions investigated in this article, only data collected from the SJ task are presented. Simultaneity judgment task and stimuli. At the beginning of each trial, a red light representing a fixation cue appeared in between the two central robots displayed in the center of the visual field (Fig. 1 ). This fixation cue disappeared following 1300 ms of gaze fixation without blinking or saccades. After a random inter-stimulus interval (ranging from 800 to 1000 ms), visual targets appeared.
At first, the onsets of blinks were identified with the HTC SRanipal SDK, detecting blinks via individual eye openness. Because blinks were rare events given the non-blinking requirements to trigger the fixation cue offset, trials containing at least one blink occurring around the onset of the targets (i. e. -600 to 200ms) were discarded (4. 8% of total trials). Then, raw data of each trial were interpolated with a spline method to increase the temporal precision, followed by the calculations of the derivations of the speed of the vertical and horizontal eye movements. Saccades were detected using an established algorithm  (Engbert, 2006)  based on the gaze's velocities. An elliptic threshold criterion for saccades detection was determined in 2D velocity space based on the horizontal and the vertical velocities of the eye movement.
Saccade rates were smoothed using a moving average window of 100 ms, as in  Betta & Turatto (2006). Preliminary analyses compared the results with a moving window of 50 ms (as in  Dankner et al. , 2017; Denison, Yuval-Greenberg, & Carrasco, 2019; Tal-Perry & Yuval-Greenberg, 2020)  and did not affect the data interpretation, other than reducing the signal/noise ratio. EEG acquisition and analysis. EEG activity was continuously collected using a Biosemi ActiveTwo 10-20 system with 64 active channels at 1024 Hz sampling rates and the ActiView software. The electrode offset was kept below 20 mV. The offset values were the voltage difference between each electrode and the CMS-DRL reference channels. EEG analyses were performed with MNE-Python v. 0. 22. 0  (Gramfort et al. , 2014 (Gramfort et al.
, 2017)  of power analysis as it increases the signal/noise ratio without eliminating a possible group difference of EEG power before the first target onset. Also, the normalization approach renders the data normally distributed, which allows the use of standard parametric tests. Control analyses of the pre-and post-target onsets alpha power using the baseline-free approach did not change the effects reported below. Preliminary observation of both ITC and power values recorded after the first target onset revealed two clusters of interest, an alpha-band (8-14 Hz) cluster recorded over parieto-occipital areas and a theta-band (4-8 Hz) cluster recorded over fronto-medial areas. Thus, for statistical analysis, we selected the alpha-band ITC and power recorded at the parieto-occipital electrode POz.
Statistical Analyses. Time series analyses compared two intervals of interest: 1) the anticipatory period representing the pretarget interval starting from -1000 ms and finishing -100 ms before the first target onset (as in  Denison et al. 2019) , and 2) the processing period representing the interval starting 100 ms and finishing 300 ms after the first target onset. Repeated-measures analyses of variances (rANOVAs) analysis were performed using RStudio (v. 0. 99. 489) and the rstatix (v. 0. 6. 0) package with a Greenhouse-Geisser correction when within-subject factors (Block, Response, Time-Window, and SOA) violated the sphericity assumption. Planned comparisons analysis was performed with Tukey's HSD tests corrected for multiple comparisons with the false-discovery rate method. Shapiro's test was used to evaluate the normal distribution of the data.
Discussion Is the ability to discriminate in time visual events enhanced in video game players? In this cross-sectional study, we evaluated the temporal discrimination abilities in vision at the millisecond level in VGPs and NVGPs using a simultaneity judgment task performed in virtual reality, while measuring EEG and oculomotor activities. During the task, participants had to judge whether two targets located on the left and right visual fields appeared simultaneously or not. The asynchronies between targets were either 0 ms, 11 ms, 33 ms or 66 ms and were equiprobable. Behavioral data indicated an overall enhanced ability in VGPs to detect all visual asynchronies. No difference in reaction times nor response bias between groups was revealed, thus ruling out the possibility of different decision-making strategies.
In the second stage, we use the image-to-image framework of a latent diffusion model (Versatile Diffusion) conditioned on predicted multimodal (text and visual) features, to generate final reconstructed images. On the publicly available Natural Scenes Dataset benchmark, our method outperforms previous models both qualitatively and quantitatively. When applied to synthetic fMRI patterns generated from individual ROI (region-of-interest) masks, our trained model creates compelling "ROI-optimal" scenes consistent with neuroscientific knowledge. Thus, the proposed methodology can have an impact on both applied (e. g. brain-computer interface) and fundamental neuroscience.
These deep generative models included Variational Autoencoders (VAE), Generative Adversarial Networks (GAN), and recently Latent Diffusion Models (LDM). Most of these studies used existing deep generative models, pretrained on large-scale data, and then learned a mapping (with simple regression or more advanced neural network architectures) to reconstruct the corresponding latent variables from the brain signals. This general method was used to reconstruct images with different levels of complexity such as faces  8, 9  , single-object-centered images  10  , and more complex scenes  11, 12. Most of the earlier works on natural scene reconstruction studied either the Generic Object Decoding  13  or the Deep Image Reconstruction 10 datasets curated by the Kamitani Lab. These datasets consist of 1200 training and 50 testing images from the ImageNet  14  dataset and they differ in the number of fMRI repetitions for training and testing images.
Chen et al. 20  utilized a sparse masked brain modeling on large-scale fMRI data and then trained a double-conditioned diffusion model for visual decoding. Recently, Allen et al. curated another dataset for visual encoding and decoding studies called Natural Scenes Dataset  11. For this dataset, 8 subjects viewed thousands of images from the COCO  21  dataset. COCO images contain multiple objects and they are more complex in nature compared to ImageNet images. Because of the number, diversity, and complexity of images included, the NSD dataset-although very recent-is becoming the de facto benchmark for fMRI-based natural scene reconstruction. Thus, it is the dataset that we chose for the present work. There are already three studies that reconstructed images from this dataset, and we can use them as baselines against which to compare our model's performance.
The first one is by Lin et al. 12  , who utilized the Lafite framework that adapts the StyleGAN2 model for text-to-image generation. Takagi et al. 22  devised a method based on Stable Diffusion, using captions for the semantic information and latent variables from images for the low-level information. Gu et al. 23  improved upon Ozcelik et al. 's  19  IC-GAN framework, by establishing a surface-based convolutional network to process fMRI data instead of using vectorized data in the regression models; they also trained an encoder network to predict pose information, instead of performing latent optimization. The above studies have fostered advances in reconstructing images with high fidelity, especially in the case of objectcentered images (i. e. , ImageNet images from the Kamitani dataset).
Yet, reconstructing scenes with multiple objects and complex semantic descriptions (i. e. , COCO images from the NSD dataset) remains a challenge. Given the remarkable recent success of latent diffusion models  24  in generative AI applications such as text-to-image generation  [24] [25] [26] [27] [28]  , we reasoned that brain decoding studies could also take advantage of such models. Thus, we propose here a visual reconstruction framework called "Brain-Diffuser", relying on the powerful generation capabilities of Versatile Diffusion 28 , a model conditioned on both vision and language representations acquired from the pretrained CLIP 29 model. Our framework consists of two stages. The first stage, illustrated in Figure  1 , generates a low-level reconstruction of images (akin to an "initial guess") using a Very Deep Variational Autoencoder (VDVAE)  30.
Dataset We used the publicly available Natural Scenes Dataset (NSD), a large-scale 7T fMRI dataset  11. The NSD was collected from 8 subjects viewing images from the COCO 21 dataset. Each image was viewed for 3 seconds, while subjects were engaged in a continuous recognition task (reporting whether they had seen the image at any previous point in the experiment). For our study, we used the 4 subjects (sub1, sub2, sub5, sub7) who completed all trials. The training set that we used thus contained 8859 images and 24980 fMRI trials (up to 3 repetitions for each image), and 982 images and 2770 fMRI trials for the test set. We averaged fMRI trials for the images that had multiple repetitions. We also used the corresponding captions from the COCO dataset. Test images are common for all subjects, while training images are different.
We used the provided single-trial beta weights, obtained using generalized linear models with fitted hemodynamic response functions and additional GLMDenoise and ridge regression procedures ('betas_fithrf_GLMdenoise_RR'). We masked preprocessed fMRI signals using the provided NSDGeneral ROI (Region-of-Interest) mask in 1. 8 mm resolution. The ROI consists of [15724, 14278, 13039, 12682] voxels for the 4 subjects respectively, and includes many visual areas from the early visual cortex to higher visual areas. For further details on this dataset and the corresponding fMRI preprocessing steps, we refer the reader to the initial paper describing the Natural Scenes Dataset 11. Test fMRI data (X test ) are passed through the trained Regressor to obtain predicted latent variables (ẑ test ). These predicted latent variables are fed to the pretrained VDVAE Decoder to get the low-level reconstruction ( Ŷlow ) of the test images (Y test ), which will serve as a sort of "initial guess" for the second stage.
Low-Level Reconstruction of Images using VDVAE (first stage) A Variational Auto-Encoder (VAE)  31  is a generative model trained to capture an input distribution (such as an image dataset) via a low-dimensional latent space, constrained to follow a particular prior distribution (e. g. Gaussian). When the input dataset takes on a more complex distribution, training a Variational Autoencoder (VAE) can be challenging. Indeed, prior work has found that datasets consisting of natural scene images require many latent variables with complex distributions for which a simple VAE would not suffice; this is why the Very Deep Variational Autoencoder (VDVAE) was introduced  30. The VDVAE is a hierarchical VAE model, with several layers of conditionally dependent latent variables, each layer adding different details from coarse to fine when transitioning from top to bottom.
This hierarchical structure helps the VDVAE learn sufficiently expressive latent variables to represent complex distributions like natural scene images. q φ (z | x) = q φ (z 0 | x) q φ (z 1 | z 0 , x). q φ (z N | z <N , x) (1) p θ (z) = p θ (z 0 ) p θ (z 1 | z 0 ). p θ (z N | z <N ) (2) For our study, we used the model provided in  30  , trained on a 64 × 64 resolution ImageNet dataset, and consisting of 75 layers; we only utilized the latent variables from the first 31 layers for the sake of size in regression, since we observed that adding further layers did not make much difference in the reconstruction results (at test time, the latent variables from the remaining layers are sampled according to the prior distribution given in equation (  2 )). In the training stage, we fed images to the encoder part of the VDVAE to extract latent variables for each training image (as described in Figure  1 ).
Final Reconstruction of Images using Versatile Diffusion (second stage) Although the VDVAE was helpful to reconstruct the layout of the image, it is not sufficient for the high-level features, nor does it produce fully naturalistic pictures. For that, we use the Versatile Diffusion 28 model in the second stage of our reconstruction framework. Versatile Diffusion is a recently proposed latent diffusion model (LDM)  24. LDMs have become highly popular after their success in high-resolution text-to-image generation. In order to train an LDM, first an autoencoder (with encoder E(. ) and decoder D(. )) is trained on a large-scale image dataset to learn a compressed representation of images x 0 , i. e. a latent space z 0 = E(x 0 ). Then, the forward diffusion process is applied to these latent variables z 0 by adding Gaussian noise in successive timesteps (described in equation (  3 ), where t represents the timestep, ᾱt indicates a coefficient derived from the standard deviation of the Gaussian noise, and ε represents the Gaussian noise).
z t = √ ᾱt z 0 + 1 -ᾱt ε (3) 4/17 L LDM = E t,z 0 ,ε,y ∥ε -ε θ (z t ,t, τ θ (y))∥ 2 (4) The Versatile Diffusion model (see Figure  2 ) is a latent diffusion model with different pathways which allow us to condition the generation process on both text and image features to guide the reverse diffusion process. It is possible to provide CLIP-Vision, CLIP-Text, or both features as conditions in the reverse diffusion process. It is also possible to initialize the reverse diffusion with latent variables obtained from a particular image, rather than from a purely random distribution-this is the image-to-image pipeline that we will use to take advantage of our "initial guess" image from stage 1. The Versatile Diffusion model that we utilized in our framework was trained on the Laion2B-en  32  dataset with 512 × 512 resolution images and corresponding captions.
CLIP (Contrastive Language-Image Pre-training)  29  is a multimodal model designed to assist in different tasks that involve natural language processing and computer vision. It is trained in a contrastive learning approach, where features gathered from images vs. text captions are projected onto separate latent spaces of identical dimensions: CLIP-V refers to the latent space for images and CLIP-T for captions. Similarity scores (e. g. cosine similarity) of the latent space projections for matching images and captions are optimized throughout training. CLIP is widely used as a feature extractor, due to its high representational capabilities. The CLIP network used in Versatile Diffusion is based on the transformer architecture (ViT-L/14) and pretrained on a large-scale contrastive task. In stage 2, we thus train two regression models, the first one between fMRI patterns and CLIP-Vision features (with 257 × 768-dim extracted from the corresponding images where the first vector with 768-dim represents the category-related embedding and the remaining 256 embeddings represent the patches acquired from the images) and the second one between fMRI patterns and CLIP-Text features (77 × 768-dim extracted from the COCO captions associated with the corresponding images where the 77 embeddings correspond to the number of tokens given to the model as inputs).
At testing time, we use the image-to-image pipeline of the latent diffusion model. First, we encode the image reconstructed with the VDVAE model (stage 1) with the AutoKL Encoder (after upsampling the image from 64 × 64 to 512 × 512) and add noise to the latent vector for 37 steps of forward diffusion (corresponding to 75% of the 50 steps of full diffusion, which is a commonly used value in the image-to-image pipeline of LDMs. ). In this image-to-image pipeline, it is necessary to first add some amount of noise to the latent values using forward diffusion, since LDMs generate images via denoising using reverse diffusion (without noise on the image, the reverse diffusion step would end up with no change). Then, we feed this noisy latent as initialization to the diffusion model and denoise it for 37 steps while conditioning with the predicted CLIP-Vision and CLIP-Text features (stage 2).
Comparison with state of the art How do these findings compare to the state of the art? We contrast the qualitative results of our model with three other existing models in Figures  5  and 6. Lin et al. 12  was the first study that used the NSD dataset for reconstruction. They are similar to our model in terms of utilizing both image and text features as conditions, but they used a StyleGAN2 model instead of an LDM. Takagi et al. 22  is the only other study (in addition to ours) to use a latent diffusion model for reconstructing images from the NSD dataset. Finally, Gu et al. 23  used an Instance-Conditioned GAN model trained on ImageNet. In Figure  5 , we compared our results with previous studies for the same set of images as in Figure  3  In Figure  6 , we compare our method against the same three baselines, but using image reconstructions that were reported by the original authors in their papers, which might be more representative of each method's performance.
Finally, when we compare our results to Gu et al. , we can see that, although both appear good at reconstructing images with similar semantics, structural aspects are less well represented in their reconstructed images (e. g. unrealistic warped shapes for the train, bus, and building). In contrast, the shape and texture details of our model are more realistic. Since their model has a BigGAN backbone, with few parameters to encode the entire layout of the image (including the object's class, its pose, size, and location), and since it is trained on a single-object-centric dataset (ImageNet), the model seems to be limited in reconstructing complex scenes with multiple objects. On the other hand, since LDMs include a spatially organized map of features, it is more convenient for them to represent multiple objects; as an example, we see one train in the third image although there are two trains in the groundtruth image, and in the reconstructed image from our model.
Which brain regions are used? In order to understand the relationship between brain regions and the various components of our model (VDVAE, CLIP-Vision, CLIP-Text), we performed a region-of-interest (ROI) analysis of the regression weights. We used 4 visual ROIs derived from population receptive field (pRF) experiments, and 4 ROIs derived from functional localization (fLoc) experiments. All experiments were provided along with the NSD dataset by the original authors. These ROIs are as follows (region names following the terminology adopted in Allen et al. 11  ): V1 is the concatenation of V1 ventral (V1v) and V1 dorsal (V1d), and 10/17 similarly for V2 and V3; V4 is the human V4 (hV4); the Face-ROI consists of the union of OFA, FFA-1, FFA-2, mTL-faces, and aTL-faces; Word-ROI consists of OWFA, VWFA-1, VWFA-2, mfs-words, and mTL-words; Place-ROI consists of OPA, PPA, and RSC; Body-ROI consists of EBA, FBA-1, FBA-2, and mTL-bodies.
They also appear more naturalistic compared to reconstructions from earlier studies (Figure  6 ). When evaluated quantitatively, we saw that Brain-Diffuser outperforms previous models in both high-level and low-level metrics. After advancing the state-of-the-art in image generation applications  [24] [25] [26] [27] [28]  , it appears that latent diffusion models can also be used to improve the state-of-the-art in fMRI-based image reconstruction. Although latent diffusion models are very recent  24  , we noted at least two competing studies that used LDMs for fMRI-based image reconstruction. Chen et al. 20  proposed MinD-Vis, a method based on an LDM conditioned on image category labels (rather than text captions) to reconstruct images from the Kamitani dataset. As mentioned above, this is a less challenging, single-object-centered dataset; thus, their results cannot be directly compared with ours, obtained using the richer and more complex NSD dataset.
Takagi et al. 22  , on the other hand, used NSD and were thus included in our quantitative comparisons. There are multiple possible reasons why our model performed better than theirs, on both low-level and high-level metrics. Beyond the use of distinct pretrained LDMs (Stable Diffusion  24  for Takagi et al. vs. Versatile Diffusion 28 in our study), our framework contains several improvements such as the use of VDVAE reconstructions for low-level details (Figure  1 ) and the dual conditioning on CLIP-Vision and CLIP-Text features (Figure  2 ), which together resulted in better qualitative and quantitative results. There are several ways in which this work may be pursued in the future. First of all, it will be important to test and validate our method on other image-fMRI datasets. As deep generative models will likely continue to improve at a breakneck pace, it is probable that there will soon come models better suited for complex scene reconstruction from fMRI signals.
Of course, among a pool of many generative models, it may not be a trivial task to select the most appropriate ones and to experiment on them, and adapt them for brain decoding and image reconstruction. If future generative models reach a ceiling in their ability to linearly explain brain activity, we may need to look for better alternatives than just doing ridge regression between fMRI patterns and latent variables. These alternatives (non-linear regressions, deep hierarchical networks), however, may require larger training datasets to learn the correspondence between fMRI patterns and visual features than ridge regression. It is possible that our reconstructions would benefit from including larger brain regions (or even the whole-brain) in our analysis.