dataesr/scientific-paragraphs-categorization huggingface dataset:
PDB continues to adapt in the new century by adopting new data formats to accommodate the much larger structures being solved. The development of metrics for assessing structure validity both globally and locally at the nucleotide level is proving key to understanding the significance of structural variation and mapping out the dynamic behavior of nucleic acid structures in response to interactions with small molecules, proteins, and other nucleic acids. Cryo-EM is rapidly displacing X-ray crystallography for very large complexes and molecular machines that cycle through multiple functional states and are hard to crystallize in unique states. New metrics for cryo-EM are in progress and should soon be available  (164). In this context, referees and journal editors have also a major role to play, to prevent incorrect structures from reaching the literature and worse, the archival databases.
Referees are encouraged to request (and in many cases are actually provided with) complete and detailed statistics tables, validation reports and quality indicators, coordinates, and electron-density maps. Journal editors, as well as authors, should comply with such requests despite fierce competition for publication. The PDB has played an important role by providing sophisticated metrics and automation that provides rapid feedback to contributors of data when they submit structures. Databases constitute an absolute necessity for advancing science, to facilitate the work of scientists in the same and related disciplines. They are repositories of observational data organized with carefully curated dictionaries and controlled vocabularies upon which future science can build and develop. In the immediate future, there are at least two main challenges to which we would like to draw attention.
Owing to the sizes and complexities of modern databases, these two challenges are interconnected and interdependent. First, the sustainability and maintenance of databases for the use of a now internationalized community of scientists raise the issue of how the associated costs shall be equitably apportioned; these are recurrent costs to support trained and competent personnel, together with the maintenance and constant upgrading of the extensive infrastructure required to keep up with the growth in the size and complexity of the data. Short-term grants with a dedicated focus are not adapted to maintain large international resources  (164). New business models are being explored and promoted  (165). The Global Biodata Coalition is an initiative started by the International Human Frontier Science Program Organization that according to their website (https://globalbiodata.
org) aims "to stabilize and ensure sustainable financial support for the global biodata infrastructure and in particular to identify for prioritized long-term support a set of Global Core Data Resources that are crucial for sustaining the broader biodata infrastructure. " And, secondly, databases fundamentally comprise wellorganized archives of chosen sets of objects. In structural biology, the objects are mainly molecular sequences and structures. However, as these two types of data attach to the same natural entities, they need to be integrated to grasp biological function and evolution more holistically. New developments in cryo-electron tomography  (166) , which allows visualization of macromolecules in situ, will open up visualization of macromolecules in their cellular environment. Maybe one day we will also include digital descriptions of phenotypes, including bones, skeletons, or wings in order to relate their forms and colors to the underlying genes and genetic networks.
In the light of evolution It would take a whole book with several dozens of figures to convey the amazing knowledge accumulated on RNA structures and architectures since the beginning of RNA structural biology and the advent of the PDB. Only glimpses of RNA structural biology are presented here, and we would like to apologize for not mentioning the contributions of many structural biologists, especially those using NMR spectroscopy, a field that contributed several seminal structures, for example, on RNPs and telomerase  (168) (169) (170) (171) (172) , with unique critical data on RNA dynamics  (173) (174) (175) (176) (177). Almost each RNA structure still brings some surprising and unexpected features, and these unveil the constraints of the physicochemical interactions and the congruent accommodations due to the historical contingencies of biological evolution.
As appropriately reminded by John Helliwell  (185) , in the end, these breakthroughs would not have been feasible without the PDB. concepts, tools database he contributed to develop. Neocles earned his PhD from Yale with Peter Moore as mentor. He became professor of Chemistry at Bowling Green State University in 1987. With Peter Moore, he worked using NMR on the 5S rRNA and showed the key role of Mg ions and the presence of non-Watson-Crick pairs. Later using sequence comparisons, he could identify loop E modules in RNA. Together we published a nomenclature for classifying logically and nonambiguously non-Watson-Crick. Neocles Leontis and his group at Bowling Green developed many websites and tools for analyzing and compiling RNA structures.
Bioinformatics analyses The resulting sequences were analyzed using R  (Team, 2019)  workflow combining dada2 v. 1. 16 (Callahan et al. , 2016)  and FROGS 3. 1. 0 (Escudié et al. , 2018)  software. Reads were filtered, merged and assigned to ASVs with dada2 and the ASVs were assigned to species using the FROGS affiliation tool. Adapters were first removed using cutadapt v. 1. 12 with Python 2. 7. 13. Reads were then filtered using the dada2 filterAndTrim function, with a truncation length of 250 bp for ITS1 forward and reverse reads and 275 and 200 bp for 16S forward and reverse reads, respectively. This truncation reduced the error rate while still allowing the merging of most reads. The error model was then calculated using the learnErrors function. Reads were dereplicated using derepFastq and the dada2 core sample inference algorithm was executed.
Forward and reverse reads were then merged with a minimum overlap of 20 bp. The resulting sequences were saved in a sequence table using makeSequenceTable. Chimera were removed using the removeBimeraDenovo function. The amplicon sequence variants (ASV) in the sequence table were then assigned to species using FROGS affiliation v3. 2. 2 with silva 138  (Quast et al. , 2013)  for 16S and Unite 8. 0  (Nilsson et al. , 2019)  for ITS1. Unite was completed with ITS1 reference sequences from yeast species usually found in sourdough. Multi-affiliations were dealt with by assigning the lowest common taxonomy level to multi-affiliated ASVs. Samples were rarefied to the minimum number of reads for each barcode, or 1000 reads using the rarefy_even_depth function of the R (v. 4. 1. 0) phyloseq package (v.
Statistical analysis A Wilcoxon-Mann-Witney test was performed to compare the diversity index between the flour and sourdough samples. The correlation between flour and sourdough diversity was computed using a Spearman rank-order correlation test. Both tests were computed using the R package stats v 3. 6. 2, with the wilcox. test and cor. test functions, respectively. A Mantel test was performed to test the link between geographical distances for sourdoughs and Bray-Curtis distance matrices, using the R ape package (v. 5. 5) mantel. test function.
However, the number of bakers sharing yeast ASVs was quite low : four, six, and eight bakers shared at least one K. bulderi, K. humilis and S. cerevisiae ASV respectively, so we were not able to perform a robust statistical analysis. A population genomic analysis of K. bulderi, K. humilis and S. cerevisiae from sourdoughs would shed more light on the relative impact of gene flow and selection on the evolution of these sourdough yeasts. The genomes of K. bulderi and K. humilis were released recently (BioProject accession number PRJEB44438 in the NCBI BioProject database, https ://www. ncbi. nlm. nih. gov/bioproject/) and this will enable the conduct of these studies. In conclusion, this evaluation of the bacterial and fungal composition of flour and sourdough showed that their microbiotas overlapped little.
Acknowledgments We would like to thank  Sylvain Santoni  and  Audrey Weber  for the Illumina sequencing and their valuable advices. This work was partly supported by a grant from the  Fondation de France  (Gluten : mythe ou réalité ?). Authors thank  Dominique DESCLAUX ,  Kristel MOINET , and all the bakers and farmer-bakers that have shared their sourdough, flour and knowledges.
The use of international laboratory terminologies inside hospital information systems is required to conduct data reuse analyses through interhospital databases. While most terminology matching techniques performing semantic interoperability are language-based, another strategy is to use distribution matching that performs terms matching based on the statistical similarity. In this work, our objective is to design and assess a structured framework to perform distribution matching on concepts described by continuous variables. We propose a framework that combines distribution matching and machine learning techniques. Using a training sample consisting of correct and incorrect correspondences between different terminologies, a match probability score is built. For each term, best candidates are returned and sorted in decreasing order using the probability given by the model.
Introduction The use of international laboratory terminologies (e. g. LOINC) inside hospital information systems is required to conduct data reuse analyses through inter-hospital databases. Well-known strategies from the ontology matching field of computer science have already been proposed, like string-based or language-based models  [1] , to standardize local terminologies toward an international reference. Distribution matching is an instance-based matching technique  [2]  that performs terms matching based on the statistical similarity of their respective sets of instances. This technique is an extension of two-sample hypothesis testing to compare distributions. Recent developments on distribution matching include a comprehensive evaluation of schema matching techniques  [3] , where a Wasserstein distance-based distribution matching algorithm  [4]  competes with state-of-the-art schema-based matching techniques.
Datasets and Scenarios Our framework is assessed using data from the freely accessible database MIMIC-III and from the Lille University Hospital's laboratory terminology, the reuse of which for medical research purposes has been authorized by the CNIL in 2019 (reference number 2202081). All models were fine-tuned using 5-fold cross-validation on a training dataset and evaluated on a testing dataset. The training and testing datasets both contain the complete terminologies (i. e. the whole terms). Splitting is performed at random on the series of measurements according to a 70% (train) / 30% (test) ratio. We propose two experimental scenarios to assess the model's behaviour : Case 1: We worked on a subset of 54 terms from the MIMIC-III database and 101 terms from the Lille University Hospital's laboratory terminology.
Discussion In this work, we applied distribution analysis to match laboratory terminologies between hospitals. The objective was to explore the use of distribution-based similarity measures for terminology matching, implement and benchmark this technique against uncurated laboratory data from the MIMIC-III database and the Lille University Hospital. The selected model was able to give the correct correspondence among the 5 best candidates for 95% of the 101 terms considered. As illustrated by the overall PrecisionRecall AUC and features' importance, distribution-based similarity measures such as the KS statistic and the Hellinger distance strongly improve the performance of the decision model compared to the absolute difference of the means or standard deviations. A second use case (positive control) consisted in matching the MIMIC-III dataset against itself.
Methodological Issues As opposed to conventional language-based matching techniques, distribution matching does not rely on the quality and richness of terminologies. Indeed, it showed to be resilient to data anomaly when tested on uncurated datasets. However, in its current state, our framework remains sensitive to mismatching unit systems between data sources. In practice, the Hellinger distance relies on kernel density estimates which are sensitive to ill-behaving data sample and requires cpu intensive numerical integration. In spite of this limitations, our model still remains accurate thanks to the combination of other distribution-based features using ensemble learning. At last, our framework yet supports only univariate distribution of continuous variables.
Then we present in detail seven different instances of DVAE that were recently proposed in the literature, with an effort to homogenize the notations and presentation lines, as well as to relate these models with existing classical temporal models. We reimplemented those seven DVAE models and we present the results of an experimental benchmark conducted on the speech analysis-resynthesis task (the PyTorch code is made publicly available). The paper is concluded with an extensive discussion on important issues concerning the DVAE class of models and future research guidelines. 13. 1.
It is thus replaced with a parametric approximate posterior distribution (i. e. , an inference model) that is implemented with a DNN. Since the observed data likelihood function is also not tractable, the estimation of the model parameters is done by chaining the inference model (aka the encoder in the VAE framework) and the generative model (the decoder ) and maximizing a lower bound of the log-likelihood function called the variational lower bound (VLB) over a training dataset. 5 From now on, we refer to this general variational inference and training methodology as the VAE methodology. In summary, the VAE methodology enables deep unsupervised representation learning while providing efficient inference and parameter estimation in a Bayesian framework. As a result, the seminal papers  (Kingma and Welling, 2014; Rezende et al.
Dynamical VAEs As a Deep BN, the original VAE of  (Kingma and Welling, 2014)  does not include temporal modeling. This means that every data vector from a dataset is processed independently of the other data vectors (and the corresponding latent vector is also processed independently of the other latent vectors). In the years following the publication of  (Kingma and Welling, 2014; Rezende et al. , 2014) , the VAE methodology was extended and successfully applied to several more complex Deep BNs. In particular it has been applied to Deep BNs with a temporal model, i. e. , DDBNs, dedicated to the modeling of sequential data exhibiting temporal correlation. This was reported in a series of papers including  (Bayer and Osendorfer, 2014; Fabius and van Amersfoort, 2014; Krishnan et al. , 2015; Chung et al.
In contrast, and remarkably, the training phase is quite similar between models since it is consistently based on chaining the encoder and decoder and maximizing the VLB over a training dataset, that is applying the VAE methodology, possibly with a few adaptations and refinements. In the end, it is difficult to say if we have to see those models as variational DDBNs, that is DDBNs immersed in the VAE framework, or as Dynamical VAEs (DVAEs), that is VAEs including a temporal model for modeling sequential data. They are probably both! In the following of the paper, as well as in its title, we chose to use the second term, that is Dynamical VAEs, since we assume that the term "VAE" is currently more popular than the term "DBN," and "Dynamical VAEs" gives a more speaking first evocation of those models, compared to "variational DDBNs.
In short, an extended benchmark of DVAE models is not yet available. On the other hand, conducting an extended benchmark is a huge amount of work, since there are many possible configurations for the models, and many tasks for evaluating them. In particular, as we will see in this review, it is not yet clear how to evaluate the degree of "disentanglement" of the extracted latent space. For all those reasons (and also because this review paper is already quite long!), we limit the benchmark to a simple task in the present review. We plan to compare more extensively the models in future studies. • Finally, the code re-implementing the seven DVAE models and used on the benchmark task is made available to the community. The open-source code and the best trained models can be downloaded at the following repository: https://github.
com/XiaoyuBIE1994/DVAE-speech. We have also taken care, in the code, to follow the unified presentation and notation used in the paper, making it, hopefully, a useful and pedagogical resource. In summary, we believe that comparison of models across papers is a difficult task in essence, whatever the efforts spent by the authors of the original papers, because of the use of different notations, presentation lines, missing information, etc. We hope that the present review paper and accompanying code will help the readers in more easily accessing the technical substance of DVAEs, and see their cross-connections and their connections with pre-existing classical models. Before we give the outline for the rest of the paper, it is useful to mention what we do not present in this review paper: • All the DVAE models we review in a detailed manner consider continuous latent random variables.
VAE encoder Training the generative model (2. 1) consists in learning the parameters θ from a training dataset X = {x n ∈ R F } Ntr n=1 , where the training vectors x n are assumed i. i. d. This is usually done by finding the values of parameters θ that maximize the training data marginal log-likelihood log p θ (X). Because the relation between z and {µ θx ,f (z), σ 2  θx ,f (z)} is modeled by a DNN, p θ (X) is intractable. 1 Therefore, the posterior distribution p θ (z|x) is intractable too, and thus using the classical Expectation-Maximization (EM) algorithm is not possible. Variational inference is an elegant methodology that overcomes these problems and enables us to efficiently train the VAE. It is grounded in two principles: i) Because the posterior distribution p θ (z|x) is intractable, an approximate posterior distribution of z is introduced, denoted q φ (z|x) ≈ p θ (z|x), that has a similar general form to the generative conditional distribution p θ (x|z) and that plays the role of the encoder, i.
Data grouping involves a certain amount of supervision during training. Although there is no temporal aspect in  (Bouchacourt et al. , 2018) , we will see later that the use of different latent variables to encode different levels of information in the data can be applied to sequential data characterized by features with different time resolutions. Improved VAE decoders: As for using more sophisticated decoders, the authors of  (Chen et al. , 2017)  considered an autoregressive conditional density of the form p θx (x|z) = i p θx (x i |z, x nb  [i]  ) with application to 2D-image modeling, where x i is the i-th pixel of the image and x nb  [i]  are the neighboring pixels. The autoregressive part is typically implemented with an RNN (van den  Oord et al. , 2016b). Ideally, the local statistics of an image (e.
3) (3. 4)). And the same for combinations of multi-layer RNNs and LSTMs, i. e. , multi-layer LSTM networks. In summary, in all the following, we assume that equations (3. 3) (3. 4) are a "generic" or "high-level" representation of an RNN of arbitrary complexity. Notation remark: Although it is a bit abusive, to clarify the presentation and the links between the different models, we use the same generic notation d x for the generating function in (2. 5) and (3. 4), and we will do that all throughout the paper (and the same for d h and for d z to come). So far, the above RNNs are totally deterministic: Given u 1:T and h 0 , x 1:T is totally determined. Such networks are trained by optimizing a deterministic criterion, e. g. , the mean squared error (MSE) between target output sequences from a training dataset and corresponding actual output sequences from the network.
Recurrent Variational Autoencoders (RVAE) A recurrent VAE (RVAE) was introduced in  (Leglaive et al. , 2020). This model was used to represent the clean speech signal in a speech enhancement application. It was combined with a Gaussian noise model with Non-negative Matrix Factorization of the variance, within a Bayesian framework. The parameters of the RVAE were estimated offline on a larget dataset of clean speech signals, using the VAE methodology (maximization of the VLB). A variational Expectation-Maximization (VEM) algorithm was then used for the estimation of the remaining parameters from a noisy speech signal, and probabilistic Wiener filters were then derived for speech enhancement. Here we present only the RVAE model.
Note that the model can be optimized at the segment level instead of at the sequence level, that is each data segment can be used as a batch dataset. According to the authors of  (Hsu et al. , 2017b) , this can solve scalability issues when training sequences become too long. DRAW: A somehow dual model of  (Fabius and van Amersfoort, 2014)  has been proposed in  (Gregor et al. , 2015)  and called DRAW for Deep Recurrent Attentive Writer: DRAW considers a sequence of latent vectors z 1:T to encode a single static but highly structured data x (a digit image or a low-resolution image from the CIFAR database). The generative model is of the general form p θx (x|z 1:T ). It involves the iterative construction of a sequence x1:T that can be seen as the sequence of images resulting from the "natural" drawing of x over time.
Experiments In the following, we present an experimental benchmark of the seven DVAE models that we detailed in the previous chapters. We showcase this benchmark with the task of speech analysis-resynthesis, that is encoding and reconstructing speech signals, that here will play the role of the data x 1:T , encoded into, and resynthesized from, a latent vector sequence z 1:T. We first specify in Section 13. 1 the implementation of each model. We then describe the experimental protocol, dataset, model training and evaluation metrics in Section 13. 2. Finally, we provide and discuss the obtained results in Section 13. 3. We recall that the opensource code and the best trained models can be downloaded at the following repository: https://github. com/XiaoyuBIE1994/DVAE-speech.
VRNN We recall that, unlike STORN, VRNN employs a shared RNN for inference and generation, with internal state vector h t. Furthermore, VRNN explicitly introduces feature extractors for x t and z t , and uses the extracted features to feed the different encoder and decoder modules. Feature extraction: ϕ x (x t ) is an MLP(x t , 256, Tanh) and ϕ z (z t ) is an MLP(z t , 32, Tanh, 64, Tanh). Generation: d h (ϕ x (x t-1 ), ϕ z (z t-1 ), h t-.
Dataset and preprocessing For the present speech analysis-resynthesis task, we used the Wall Street Journal (WSJ0) dataset  (Garofolo et al. , 1993) , which consists of speech read from Wall Street Journal news. We used the speaker-independent, medium (5k words) vocabulary subset of the corpus. More precisely, the si tr s subset (around 25 hours) was used for training, the si dt 05 subset (around 2 hours) was used for validation, and the si et 05 subset (around 1. 5 hours) was used for testing. The raw speech waveform was sampled at 16 kHz. Analysis-resynthesis was processed with the DVAEs in the time-frequency domain. To this aim, the speech signals were preprocessed with the short-time Fourier transform (STFT) using a 32-ms sine window (512 samples) with 50%-overlap to obtain sequences of 257-dimensional discrete Fourier spectra (for positive frequencies).