dataesr/scientific-paragraphs-categorization huggingface dataset:
'Speech module' evaluation To evaluate our approach, we compare these decoding performances to those obtained with models that target different representations of speech (Table  2 ). While a model trained to predict the Mel spectrogram with a regression objective ('Base model' in Table  2 ) is systematically higher than chance, the use of a contrastive loss ('+ Contrastive') leads to decoding gains that range from 2% (for the Brennan and Hale dataset  31  ) to 42. 7% (for the Gwilliams dataset  33  ). This gain is further supplemented by targeting the latent representations of the Mel spectrogram ('+ Deep Mel'). The latent representations of speech sounds, however, appear to be best identified with a pretrained speech module, that is, by using wav2vec 2. 0, a model pretrained with self-supervised learning on speech sounds only, rather than by jointly learning speech and MEG and EEG representations (Table  2 ).
'Brain module' evaluation To evaluate the elements of the brain module, we performed a series of ablation experiments, and trained the corresponding models on the same data (Extended Data Fig. 2 ). Overall, these ablations show that several elements impact performance: performance systematically decreases when removing skip connections, the spatial attention module, and the initial or final convolutional layers of the brain module. These results also show the importance of clamping the MEG and EEG signals. Finally, additional experiments show that the present end-to-end architecture is robust to MEG and EEG artefacts, and requires little preprocessing of the MEG and EEG signals (Supplementary Sections A. 3 and A. 4).
Impact of the number of participants To test whether our model effectively leverages the inter-individual variability, we trained it on a variable number of participants and computed its accuracy on the first 10% of participants. As shown in Fig. 2c , decoding performance steadily increases as the model is trained with more participants on the two MEG datasets. This result shows that our model effectively learns neural representations that are common across participants, while also accommodating participant-specific representations through the participant layer described in Methods.
Decoded representations best correlate with phrase embeddings What type of representation does our model use to decode speech from brain signals? This interpretability question is notoriously difficult to address  22, 34. Figure  3  illustrates this issue: it shows the probability of each word given the MEG data of five representative participants listening to the phrase 'Thank you for coming, Ed'. Extended Data Fig. 3  shows additional predictions for five representative segments of the Gwilliams dataset  33. In both cases, it can be difficult to judge whether the decoder's error tends to be related to the phonology or to the semantics of the actual sentence. To address this issue, we analyse the single-word and single-segment predictions of our model with a linear model. Specifically, we train a linear regression to predict the softmax probability of the true word estimated by the decoder, given different set of features, ranging from low-level representations (for example, phonemes) to high-level representations (for example, phrase embedding; see Methods for details).
Problem formalization We aim to decode speech from a time series of high-dimensional brain signals recorded with non-invasive MEG or EEG while healthy volunteers passively listened to spoken sentences in their native language. How spoken words are represented in the brain is largely unknown  [37] [38] [39]. Thus, it is common to train decoders in a supervised manner to predict a latent representation of speech known to be relevant to the brain  16, 34, [40] [41] [42]. For example, the Mel spectrogram is often targeted for neural decoding because it represents sounds similar to the cochlea  43. We study four datasets, two using MEG signals and two using EEG signals. We name each dataset by its author and year. For each dataset, we report the number of channels, the number of participants and the total duration in hours.
Model Contrastive loss. We reasoned that regression may be an ineffective loss because it departs from our objective-that is, it requires maximally distinguishing different speech segments apart. Indeed, a regression objective stems from the principle that all of the dimensions of the Mel spectrogram are (1) equally important and (  2 ) are scaled appropriately: the L2 objective inclines the model to predict low and high frequencies equally well, even if (1) some frequencies (for example, very low) may be irrelevant to speech and (2) some frequencies may vary in orders of magnitudes lowers than others. To relax this constraint, we opted for a contrastive objective and thus replaced the regression loss with the 'CLIP' loss (originally for Contrastive Language-Image Pre-Training) by ref. 44, which was originally designed to match latent representations in two modalities, text and images.
Unlike the regression objective, this contrastive loss leads the model to find a combination of features that maximally discriminates samples in the batch. Consequently, the model is naturally inclined to focus on the informative dimensions of the Mel spectrograms and to scale them appropriately. We implement the CLIP loss as follows. Let X be a brain recording segment and Y ∈ ℝ F×T the latent representation of its corresponding sound (also known as 'positive sample'). We sample N - 1 negative samples Ȳ j∈{1,…,N-1} over our dataset and we add the positive sample as Ȳ N = Y. We want our model to predict the probabilities ∀j ∈ {1, … , N}, p j = ℙ [ Ȳj = Y]. We thus train a model f clip mapping the brain activity X to a latent representation Z = f clip (X) ∈ ℝ F×T. The estimated probability can then be approximated by the dot product of Z and the candidate speech latent representations Y j , followed by a softmax: p j = e ⟨Z, Ȳ j ⟩ ∑ N j ′ =1 e ⟨Z, Ȳ j′ ⟩ , (2) with 〈⋅, ⋅〉 the inner product over both dimensions of Z and Ŷ.
We then train f clip with a cross-entropy between p j and p j. Note that for a large enough dataset, we can neglect the probability of sampling twice the same segment, so that we have p j = j=N , and the cross-entropy simplifies to L CLIP ( p, p ) = -log( p N ) = -⟨Z, Y⟩ + log ( N ∑ j ′ =1 e ⟨Z, Ȳ ′ j ⟩ ). (3) Following ref. 44, we use the other elements of the batch as negative samples at train time. At test time, the negative samples correspond to all of the segments of the test set but the positive one. Brain module. For the brain module, we introduce a deep neural network f clip , input with raw MEG and EEG times series X and a one-hot encoding of the corresponding participant s, and outputs the latent brain representation Z, with the same sample rate as X. This architecture consists of (1) a spatial attention layer over the MEG and EEG sensors followed (2) by a participant-specific 1 × 1 convolution designed to leverage inter-individual variability, which input to (3) a stack of convolutional blocks.
( 4 ) The output is given by a softmax attention based on the evaluation of a j at each input position (x i , y i ): ∀j ∈ {1, … , D 1 }, SA(X) (j) = 1 ∑ C i=1 e a j (x i ,y i ) ( C ∑ i=1 e a j (x i ,y i ) X (i) ) (5) with SA the spatial attention. In practice, as a j is periodic, we scale down (x, y) to keep a margin of 0. 1 on each side. We then apply a spatial dropout by sampling a location (x drop , y drop ) and removing from the softmax each sensor that is within a distance of d drop = 0. 2 of the sampled location. The initial motivation for spatial attention was to allow for a cross-dataset model to be defined in a way that would generalize across a diverse number location and set of sensors. Interestingly, we observed this layer to introduce an inductive bias that is beneficial to the prediction accuracy (Extended Data Fig.
2 ). See Extended Data Fig. 4  for a visualization of the learnt attention maps over each dataset. We then add a 1 × 1 convolution (that is, with a kernel size of 1) without activation and with the same number D 1 of output channels. The R values quantify the extent to which phonemes, word frequency, part-of-speech, word embedding and phrase embedding predict the probability of the predicted word to be correct. Error bars are the s. e. m. across participants (Table  1 ).
Speech module. The Mel spectrogram is a low-level representation of speech inspired from the cochlea and is thus unlikely to match the rich variety of cortical representations  38. Consequently, we replaced the Mel spectrograms with latent representations of speech. For this, we propose either to learn these representations end-to-end ('Deep Mel' model) or to rely on those learnt by an independent self-supervised speech model (wav2vec 2. 0; ref. 29). End-to-end speech representations with Deep Mel. The 'Deep Mel' module uses the same deep convolutional architecture to the brain module devoid of the participant block, and thus simultaneously learns to extract speech and MEG and EEG representations such that they are maximally aligned. By definition, and unlike wav2vec 2. 0, Deep Mel sees only the audio used in the MEG and EEG datasets.
Datasets We test our approach on four public datasets, two based on MEG recordings and two based on EEG recordings. All datasets and their corresponding studies were approved by the relevant ethics committee and are publicly available for fundamental research purposes. Informed consent was obtained from each human research participant. We provide an overview of the main characteristics of the datasets in Table  1 , including the number of training and test segments and vocabulary size over both splits. For all datasets, healthy adult volunteers passively listened to speech sounds (accompanied by some memory or comprehension questions to ensure participants were attentive), while their brain activity was recorded with MEG or EEG. In Schoffelen et al. 32  , Dutch-speaking participants listened to decontextualized Dutch sentences and word lists (Dutch sentences for which the words are randomly shuffled).
The study was approved by the local ethics committee (the local Committee on Research Involving Human Subjects in the Arnhem-Nijmegen region). The data are publicly and freely available after registration on the Donders Repository. In Gwilliams et al. 33  , English-speaking participants listened to four fictional stories from the Masc corpus  57  in two identical sessions of 1 hour  30. The study was approved by the institutional review board ethics committee of New York University Abu Dhabi. In Broderick et al. 58   20. Consequently, we first resampled all brain recordings down to 120 Hz with Torchaudio 59 and then split the data into training, validation and testing splits with a size roughly proportional to 70%, 20% and 10%, respectively. We defined a 'sample' as a 3 s window of brain recording with its associated speech representation.
As decoding speech production can be challenged by the presence of muscular activity, we here evaluate this model on four public datasets where healthy participants listened to natural speech. Our analyses suggest that advanced MEG and EEG preprocessing does not provide a major advantage in the current decoding task and that a simple baseline correction followed by a robust scaler and clamping suffices. In addition, not only does our participant-specific layer improves decoding performance but also this performance increases with the number of participants present in the training set. These findings, combined with both the rise of publicly available datasets and the potential to learn informative features from unannotated data  73, 74  , suggest that this brain module may be a stepping stone for building a foundational model of brain recordings.
Our analyses confirm that this approach outperforms (1) a supervised decoding of the Mel spectrogram as well as (2) 'Deep Mel', that is, latent representations of the Mel spectrogram optimized for decoding solely from the present MEG and EEG datasets. Finally, the inspection of the decoding predictions suggests that our model primarily captures the lexical and contextual features captured by modern word embeddings and language models. So far, however, what these high-level features represent and how these representations are structured and manipulated remain to be determined.
Furthermore, the performances of our model are based on vocabularies that do not fully overlap with those used in the training set (Table  1 ). For example, for the Gwilliams dataset, the decoding performance reaches 40% in spite of the fact that nearly 36% of the words were never presented during training. Overall, such zero-shot decoding shows the versatility of this approach and opens the possibility to decode from even larger vocabulary. Second, although our model's performance remains modest, it may not be too distant from the performance obtained with invasive recordings of brain activity. Indeed, decoding the perception of isolated words from a vocabulary of n = 50 words leads to a top-1 accuracy of 22. 7% on average, but up to 42. 9% in the best participants (Supplementary Section A.
Data availability The data from Schoffelen et al. 32  were provided (in part) by the Donders Institute for Brain, Cognition and Behaviour with a 'RU-DI-HD-1. 0' licence. The data for Gwilliams et al. 33  are available under the CC0 1. 0 Universal licence. The data for Broderick et al. 58  are available under the same licence. Finally, the data from Brennan and Hale 31 are available under the CC BY 4. 0 licence. All audio files were provided by the authors of each dataset.
Elucidating the developmental process of an organism will require the complete cartography of cellular lineages in the spatial, temporal, and molecular domains. We present Zebrahub, a comprehensive dynamic atlas of zebrafish embryonic development that combines single-cell sequencing time course data with light-sheet microscopy-based lineage reconstructions. Zebrahub is a foundational resource to study developmental processes at both transcriptional and spatiotemporal levels. It is publicly accessible as a web-based resource, providing an openaccess collection of datasets and tools. Using this resource we shed new light on the pluripotency of Neuro-Mesodermal Progenitors (NMPs). We find that NMPs are pluripotent only during early axis elongation before becoming exclusively mesodermal progenitors.
Single-cell RNA sequencing (scRNAseq) technologies have provided complementary means of tracking cell lineages based on their transcriptomes over developmental time  [11] [12] [13] [14] [15] [16] [17] [18] [19] [20]. Here, we combine light-sheet microscopy with scRNAseq of individual developing zebrafish embryos to construct a comprehensive multimodal atlas of cell lineages and molecular states at unique spatiotemporal resolution. We provide the data along with the analytical tools as an integrated and publicly available resource: Zebrahub. Using Zebrahub we investigated the Neuro-Mesodermal Progenitors (NMPs) -a conserved population of pluripotent axial elongation progenitors that generate skeletal muscles and posterior neural cells  [21] [22] [23] [24] [25] [26]. These cells play a crucial role in vertebrate axis elongation; however, their developmental dynamics remain unclear.
Using a combination of multimodal techniques, we have elucidated the differentiation dynamics of NMPs throughout the whole process of vertebrate axis elongation. 14 hpf embryo transcriptome colored by cell type (3862 cells total). Third, Zebrahub's web portal allows the exploration of scRNAseq data online. (b) UMAP representation of the entire dataset of 40 single-embryo transcriptomes colored by developmental stage (120k total cells). (c) Same but colored by annotated cell types. (d) Maximum intensity projections of a 12hour light-sheet live-imaging of a zebrafish embryo expressing histone-mCherry (cyan) and mezzo-GFP (magenta). Arrows indicate the mezzo-positive cells. (e) Similar imaging but on an embryo expressing the photoconvertible protein Dendra2. Photoconverted cells are in red (blue arrows).
RESULTS Time-resolved single-embryo deep scRNAseq dataset of zebrafish development. Zebrahub's sequencing resource consists of a dataset containing 120,444 cells from 40 individual zebrafish embryos and larvae spanning 10 developmental stages from endof-gastrulation embryos to 10-day larvae (10-, 12-, 14-, 16-, 19-, 24hours post-fertilization, as well as 2-, 3-, 5-and 10-days postfertilization). To capture single-cell RNA sequencing variability across individuals, the cell dissociation protocol was optimized to avoid embryo pooling thus achieving non-pooled single-embryo single-cell RNA sequencing (Fig. 1a , Supp. Fig. 1  & Methods). Four fish were sequenced for each of the ten developmental stages thus obtaining 40 individually sequenced embryos. Using strict quality control criteria (Supp.
Cell type clustering and annotation. To visualize the transcriptomes for these 120,444 cells we generated uniform manifold projections (UMAP  27  ) for each developmental stage (Fig. 1a  -center and Supp. Fig. 3 ) and the whole dataset (Fig. 1b ). We observed branching patterns that recapitulate the progression of embryonic development, with cells taken from early embryos located at the beginning of branches and cells obtained from older juveniles positioned towards the outer areas (Fig. 1b  and Supp. Video 1). A total of 529 cell clusters (Leiden clustering) were annotated by examining enriched genes and cross-referencing with the published literature as well as the zebrafish database ZFIN  28  (Supp. Table  1 ). The different cell types identified were then grouped into higher-level tissue types (Fig.
1g , Supp. Video 4 and methods). To help users navigate and use these very large datasets we developed an in silico fate mapping plugin for the multidimensional image viewer napari (Fig. 1h  and Supp Fig. 15 ). This software allows users to conduct virtual fate mapping experiments by marking territories and following the trajectories of cells and their descendants. All imaging datasets and tools are accessible online at zebrahub. org and together with the scRNAseq datasets constitute a foundational multimodal resource to study vertebrate development.
Inter-individual transcriptomic variance between sibling embryos. The single-embryo resolution and high sequencing depth of our scRNAseq datasets provide a unique opportunity to study the variability of gene expression during development by comparing the single-cell transcriptomes of siblings (Fig. 2a ). To quantify the differences in gene expression patterns between siblings, we developed a framework comparing gene expression distributions across cells, utilizing the Kullback-Leibler divergence  34  (Fig. 2b ). A Kullback-Leibler divergence of 0 indicates that the two distributions are identical, while values greater than zero indicate increasing divergences between the distributions (Fig. 2b ). For example, at 12 hpf, multiple genes showed seemingly identical expression patterns between siblings (e.
g. , neural tube to neurons in Fig. 3c -top ). In contrast, matured or fully differentiated cell states did not transition into any other state (e. g. , endothelial cells in Fig. 3c -bottom ). The cell state transitions were recapitulated in a coarse-grain graph (Fig. 3d ), supporting the notion that NMPs transition to both mesodermal and neural states. Next, we examined their molecular and spatiotemporal dynamics. Sox2/tbxta gradient predicts the future transcriptomic state of NMPs. We zoomed in on the NMPs and the adjacent clusters and re-projected the data on a new UMAP (fig. 4a , Supp. Fig. 10c ). In this projection, the NMPs stay positioned between the presomitic mesoderm (PSM) and neural tube (Fig. 4a  and Supp. Fig. 11 ). The RNA velocity analysis on the new projection showed the same mesodermal and neural flows originating from the NMP population (Fig.
Pseudotime lineage reconstruction supports NMPs' fate restriction. To further understand the developmental progression of NMPs along the neural-mesodermal axis, we reconstructed pseudotime developmental trajectories  48. These trajectories recapitulate the expression of genes that play crucial roles in neural and mesodermal differentiation  49  (Fig. 4j ). We also identified the key cell types in these trajectories (Fig. 4k ) and visualized their distribution in developmental time (Fig. 4l ). These findings were consistent with our previous observations, indicating that the majority of NMPs moving in the neural direction are early-stage cells, while the cells moving in the mesodermal direction are from all stages (Fig. 4k , l -red bar). Overall, our time-resolved scRNAseq dataset indicates that the fate of NMPs changes over time, with early NMPs, predicted to have the potential for both mesodermal and neural fates, whereas after 16 hpf they are restricted to the mesodermal lineage.
DISCUSSION We introduced Zebrahub, a collection of single-embryo single-cell temporally resolved transcriptome and image datasets for the comprehensive analysis of zebrafish development. All datasets, designs, and software are accessible for download, sequencing and imaging datasets can be explored interactively, and image-based lineages can be visualized using our napari  52  plugins. This resource includes a wide range of multidisciplinary innovations such as a protocol for high-efficiency single-embryo cell dissociation, a framework for studying the transcriptomic variability of individual embryos, an open-source design for adaptive multi-view light-sheet imaging and photo-manipulation, as well as computational tools for in silico fate mapping from light-sheet imaging data. Furthermore, in contrast to previous scRNAseq studies  [11] [12] [13] [14] [15] [16] 19  , our resource encompasses a broader developmental period, attains non-pooled resolution at the single-embryo level, and our gene coverage per embryo is sufficient to measure transcriptome variability between individuals.
This delay would prevent these cells from prematurely differentiating and thus integrating a mismatched tissue. In contrast, when the axial progenitor territory is more stable (no repeller), and cell flow is coordinated, the NMPs remain pluripotent competent, but give only rise to mesoderm. This suggests the importance of tissue kinematics in modulating effective NMP differentiation fates. Interestingly, our data, and in particular our RNA velocity study, also suggests that the NMPs might be at the origin of other axial tissues, namely notochord and floor plate. Altogether, our results challenge the classical view of lineage commitment after gastrulation and support the idea of late and broad differentiation plasticity. Zebrahub's collection of accessible datasets and tools is a foundational resource to the community.
By combining imaging and sequencing datasets we joined the strengths of both modalities. Beyond NMPs, we expect that this resource will help the community to shine a light on other key developmental processes. Zebrahub is poised to expand with more developmental stages, novel multiomic datasets, and ultimately a consensus digital lineage reconstruction of multiple embryos. Integrating existing datasets, including from other species, will enable the creation of a comprehensive atlas of vertebrate embryogenesis, initiating a new era for developmental and evolutionary biology.