dataesr/scientific-paragraphs-categorization huggingface dataset:
Evaluation of CNN-HS on FEMTO Dataset Datasets of bearings in condition 1 (1800 rpm and 4000 N, C1) and condition 2 (1650 rpm and 4200 N, C2) were used in our experiment. Each condition has seven run-to-failure datasets (we denote them as B1 to B7). Two datasets from each operating condition were used as the training dataset, and the trained model was tested on five other datasets in the same condition. The results of the prediction in C1 are compared with previous methods, RMS and AE, in Figures  9  and 10. Again, RMS and AE require a pre-specified threshold value for HS division, and the threshold value is represented by the red horizontal line. FPT, determined by a threshold value in RMS and AE, and by a trigger mechanism in CNN-HS, is represented as a vertical red dotted line. The FPTs measured from all the experiments in both C1 and C2 are recorded in Table  3.
Discussion By experimental evaluation, the feasibility of our model for HS division is demonstrated. Our model is able to capture faulty features ahead of previous methods, as demonstrated in the feature maps, which generate a significant amount of lead time on the maintenance time horizon. To verify this capability, we conducted another test for the feasibility of our method for HI prediction. Thanks to the capability of the initial wearing feature extraction, our proposed method can make earlier FPT predictions than the signal-based and unsupervised data-driven method that requires specified thresholds. In order to verify the performance of the proposed method, more experimental evaluations on the reliability of the dataset are needed. There are other benchmark datasets, but we could not work with them due to the following reasons.
NSP is a scalable and signal-independent data-wrangling method, but it requires multiple channel signals. Other open benchmark datasets containing multiple channels of vibration data were not found. As of now, there are no datasets available with labeled ground truth FPT. As such, this work has been developed in the face of this existing difficulty and thus remains among the first works of this kind. For efficient HS division using supervised learning, there is a need for an extensive amount of degradation datasets under variable operational conditions. Wearing patterns are varying in laboratory test benches as well as in real-world applications. Some wearings degrade gradually, while others degrade rapidly. A general supervised HS division method can be developed if a large amount of data covering such varying wear is made available.
Impact of Combination of Training Data In our proposed method, training data from the same operating condition is needed, and the selection of the training dataset of our model plays a critical part in the performance. During our experimental phase, we tested different combinations of training data to observe how the variation in bearing degradation in the training set affected the model in both C1 and C2. The impact of the different combinations of training data in terms of the average FPT to the total duration ratio is shown in Table  4. These results show that the predicted FPT varies by up to 34. 45%. The bearing datasets under accelerated degradation show two phenomena: bearing that shows gradual degradation and bearing that shows rapid degradation. From our intuition, a suspicion arises as to whether the model is predicting healthy conditions to be unhealthy prematurely since, in the training phase, relatively healthy conditions might have been labeled as unhealthy since rapid degradation was shown near the end.
Impact of Labeling Ratio of the Wearing Process In the evaluation, NSP from the first 5% of the total degradation process was labeled as healthy, and the last 5% as unhealthy. This labeling makes the ratio of used training data to unused training data in the training dataset (denoted as labeling ratio) equal to 1:9. The labeling ratio of 1:9 is determined by experimental comparisons: from 1:9 to 5:5. In each condition of the FEMTO dataset, two bearing datasets are selected as training data, and the rest of the datasets are tested. First, FPT is predicted using the trained model in each test dataset. Based on the predicted FPT, the test dataset is divided into a healthy condition period (HCP) and an unhealthy condition period (UCP). The ground truth of HS division is set to 'healthy' for HCP data, and to 'unhealthy' for UCP data.
Effectiveness of the CNN-HS Model and NSP in Representing Degradation Patterns To verify the effectiveness of the CNN-HS model and NSP in representing degradation patterns, we conducted an experiment on the FEMTO dataset. The experiment we proposed was to predict RMS values using NSP images and CNN-HS. RMS, which is a signal-based method, uses the original time series vibration data. The RMS prediction model using NSP and CNN-HS is shown in Figure  13. In the first step, the time series vibration data is transformed into NSP images and the NSP image is used as an input image of CNN-HS. In the training procedure, the RMS value of the sliding frame corresponding to the NSP image is set as an input label of CNN-HS. Unlike the proposed method, NSP images from the total degradation process are used in the training procedure.
In the testing phase, the trained CNN-HS is used to compute out RMS values of NSP images acquired from the test data. The RMS values for each bearing under two operating conditions in the FEMTO dataset are shown in 14. Because each bearing has a different degradation pattern and defect level, the pattern of RMS values and the maximum RMS value on each bearing and condition are different. To reduce the scale difference in RMS values, we saturate the maximum RMS value to a threshold value. In this experiment, we evaluated the CNN-HS with a threshold value of two because the minimum value among the maximum RMS values on bearings is two. In order to verify the accuracy of the RMS prediction using NSP and CNN-HS, the root mean square error (RMSE) is used to calculate the accuracy of the prediction values.
Conclusions In this paper, we propose a novel method of supervised health stage prediction using CNN for bearing wear. In signal-based and unsupervised data-driven methods, threshold values have to be manually specified, a method which is prone to either high false alarm rates or low recall. The combination of NSP, which allows us to integrate a frequency analysis that helps to extract faulty wear features, and a CNN-based binary regression model, rids the necessity of designating a threshold, one of the most crucial problems in early fault detection. Our CNN-HS model effectively distinguishes healthy and unhealthy states in an unambiguous manner by changing the anomaly detection problem into a binary regression problem, utilizing a simple trigger mechanism. In order to minimize the labeling process in supervised learning, a small portion of the dataset is utilized for training the model.
, 2017)  or RegulomeDB  (Boyle et al. , 2012) , most of them mainly dedicated to coding SNVs. Even the PreTIS software  (Reuter et al. , 2016)  dedicated to 5'UTR variants only predicted a "moderate" impact of the mutation. We then realize that there was no easy-to-use tool that can automatically read VCF files and detect/annotate SNVs that potentially create premature ATG codon and upstream open reading frame (uORF). At least 2 situations can result from the creation of a uAUG by a SNV (uaugSNV) located in a 5'UTR region. In the first case, the created uAUG could be in frame with the main coding sequence (CDS) with no associated premature stop codon, likely producing a longer protein. In the second case, the 5'UTR variant could create a uORFUpstream open reading frames are sequences formed by a uAUG, associated with a stop codon preceding the end of the main CDS.
Information about these elements are of key importance to predict as accurate as possible the impact of a loorf at the protein level. In absence of an automated method, this type of predictions is very timeconsuming and not adapted to the huge number of variants identified by new generation sequencing nor to the large variety of uORF. To fill this gap, we develop a R program, MORFEE (for Mutated upstream ORF dEtEction), that detects, in a given VCF file, any 5'UTR SNV that could potentially create premature ATG codon and provides some annotation about their potential impact on the natural protein. The application of MORFEE to two genetic datasets illustrates that loorf are a neglected class of non-coding variations that could contribute to human diseases.
Workflow The algorithm of MORFEE is written in R language and can run on all operating systems which have an R interpreter (including Linux, macOS and Windows). MORFEE starts with a minimal VCF file (i. e. with at least the *chr*, *position*, *reference allele* and *alternate allele* fields) as an input , but can also work with already ANNOVAR-annotated  (Wang et al. , 2010)  VCF files. MORFEE has some R packages dependencies which are available on CRAN or Bioconductor repositories and uses the GENCODE database. In a first step, MORFEE reads the input VCF file and use ANNOVAR (that has then to be beforehand installed) through the wrapper function *vcf. annovar. annotation* to annotate all variants. This step is skipped if the input file has already been annotated. Then, 5'UTR variants associated to any transcript are selected and GENCODE database is used to extract necessary transcripts information to generate the complementary DNA sequence at selected variant.
Results We first applied MORFEE to the 1000Genomes reference dataset (phase 3 v. 20130502) and identified 5,161 uaugSNVs among which 4,639 could be loorfs (Supplementary Table  S1 ). Among these candidate loorfs, 107 are mapping to loci associated with clinical disorders as reported in ClinVar (version 20190305) or to loci identified in Genome Wide Association studies (GWAS). These observations warrant further deep investigation to determine whether such loorfs could contribute to explain GWAS signal. For instance, the FRMD5 rs492571 is in strong linkage disequilibrium with several SNPs reported to associate with triglycerides and HDLcholesterol levels  (Teslovich et al. , 2010) , all these SNPs being considered as intronic. Interestingly, 7 protein coding transcripts, harbouring different CDS and 5'UTR, are associated with the FRMD5 gene.
The rs492571 SNP is located into introns in 6 over these isoforms, but in the 5'UTR of the transcript NM_001286491. 2, at c. -487 position. Based on MORFEE predictions, the A>G substitution at this position (rs492571), creates a new start codon in the 5'UTR of the NM_001286491. 2 isoform, thus generating a fully upstream uORF carrying 35 codons (including the stop codon; i. e. 34 amino acid). These data suggest that the rs492571 could be a good culprit candidate for the observed GWAS signal whose functionality deserve to be further investigated. MORFEE was also applied to the vcf file derived from the whole genome sequencing analysis of 200 unrelated patients with idiopathic venous thrombosis (VT) from the MARTHA study  (Oudot-Mellakh et al. , 2012)  (Supplementary Materials). Full results listing all identified uaugSNV are given in Supplementary Table  S2.
MORFEE identified 1,096 uaugSNVs containing 702 uncommon variants (minor allele frequency < 1% or never been reported). Among them, 635 were loorfs , including two variants mapping to candidate VT genes: PROC rs116169054 and PLAT rs763007802. MORFEE predictions indicate that the PROC SNP rs116169054 could generate an overlapping 35 codons uORF by creating a uAUG in the 5'UTR of the transcript NM_000312. 4 while the PLAT rs763007802 is predicted to create a uAUG in the 5'UTR of the transcript NM_000930. 5, resulting in the production of a 3 codons uORF. Preliminary experimental work (Supplementary Materials) suggest that the presence of the PROC variant, but not that of the PLAT variant, is associated with decreased levels of the corresponding protein (Supplementary Fig 1. ). The effect of the PROC rs116169054 loorf on decreased protein C levels could be compatible with this variant contributing to VT risk, protein C deficiency being a major risk factor for VT.
Funding This work was supported by the  GENMED Laboratory of Excellence on Medical Genomics  [ ANR-10-LABX-0013  to OS]. D. A was partially financially supported by a  ANR-18-PERM-0004  grant. FLV was partially supported by a  ANR-17-ECVD-0005-01  grant. DA. T was supported by the  EPIDEMIOM-VTE Senior Chair  from the  Initiative of Excellence of the University of Bordeaux.
We introduce dual-decoder Transformer, a new model architecture that jointly performs automatic speech recognition (ASR) and multilingual speech translation (ST). Our models are based on the original Transformer architecture  (Vaswani et al. , 2017)  but consist of two decoders, each responsible for one task (ASR or ST). Our major contribution lies in how these decoders interact with each other: one decoder can attend to different information sources from the other via a dual-attention mechanism. We propose two variants of these architectures corresponding to two different levels of dependencies between the decoders, called the parallel and cross dual-decoder Transformers, respectively. Extensive experiments on the MuST-C dataset show that our models outperform the previously-reported highest translation performance in the multilingual settings, and outperform as well bilingual one-to-one results.
Furthermore, our parallel models demonstrate no trade-off between ASR and ST compared to the vanilla multi-task architecture. Our code and pre-trained models are available at https://github. com/formiel/speech-translation. 9 For a hypothesis of length L, a length penalty p means a score of pL will be added to the (log probability) score of that hypothesis (Section 4. 2). Therefore, longer hypotheses are favored, or equivalently, shorter hypotheses are "penalized". 10 We also tried sacreBLEU (Post, 2018) and found that the results are identical. 11 This is for a fair comparison with the results of  Inaguma et al. (2020) , presented in Section 5. 4.
Our model is inspired by  Liu et al. (2020) , but the interaction between ASR and ST decoders is much tighter. 1 Finally, experiments show that our model outperforms theirs on the MuST-C benchmark  (Di Gangi et al. , 2019). Our contributions are summarized as follows: (1) a new model architecture for joint ASR and multilingual ST; (2) an integrated beam search decoding strategy which jointly transcribes and translates, and that is extended to a wait-k strategy where the ASR hypothesis is ahead of the ST hypothesis by k tokens and vice-versa; and (3) competitive performance on the MuST-C dataset in both bilingual and multilingual settings and improvements on previous joint ASR/ST work.
Related Work Multilingual ST Multilingual translation  (Johnson et al. , 2016)  consists in translating between different language pairs with a single model, thereby improving maintainability and the quality of low resource languages. Gangi et al. (2019)  adapt this method to one-to-many multilingual speech translation by adding a language embedding to each source feature vector. They also observe that using the source language (English) as one of the target languages improves performance. Inaguma et al. (2019)  simplify the previous approach by prepending a target language token to the decoder and apply it to one-to-many and many-to-many speech translation. They do not investigate many-to-one due to the lack of a large corpus for this. To fill this void,  Wang et al. (2020a)  release the CoVoST dataset for ST from 11 languages into English and demonstrate the effectiveness of many-to-one ST.
Dataset To build a one-to-many model that can jointly transcribe and translate, we use MuST-C (Di Gangi et al. , 2019), which is currently the largest publicly available one-to-many speech translation dataset. 7 MuST-C covers language pairs from English to eight different target languages including Dutch, French, German, Italian, Portuguese, Romanian, Russian, and Spanish. Each language direction includes a triplet of source input speech, source transcription, and target translation, with size ranging from 385 hours (Portuguese) to 504 hours (Spanish). We refer to the original paper for more details.
Training and decoding details Our implementation is based on the ESPnet-ST toolkit  (Inaguma et al. , 2020). 8 In the following, we provide details for reproducing the results. The pipeline is identical for all experiments. Models All experiments use the same encoder architecture with 12 layers. The decoder has 6 layers, except for the independent-decoder model where we also include a 8-layer version (independent++) to compare the effects of dual-attention against simply increasing the number of model parameters. Text pre-processing Transcriptions and translations were normalized and tokenized using the Moses tokenizer  (Koehn et al. , 2007). The transcription was lower-cased and the punctuation was stripped. A joint BPE  (Sennrich et al. , 2016)  with 8000 merge operations was learned on the concatenation of the English transcription and all target languages.
All models are trained for 550K steps, corresponding to 25 epochs. Following  Inaguma et al. (2020) , we use the average of five checkpoints with the best validation accuracies on the dev sets for evaluation. We compare the results with the previous work  (Gangi et al. , 2019)  in the multilingual setting. In addition, to demonstrate the competitive performance of our models, we also include the best existing translation performance on MuST-C  (Inaguma et al. , 2020) , although these results were obtained with bilingual systems and from a sophisticated training recipe. Indeed, to obtain the results for each language pair (e. g. en-de),  Inaguma et al. (2020)   weights of (respectively) the encoder and decoder for ST training. This means that to obtain the results for the 8 language pairs, 24 independent trainings had to be performed in total (3 for each language pair).
The results in Table  2  show that our models achieved very competitive performance compared to bilingual one-to-one models  (Inaguma et al. , 2020) , despite being trained for only half the number of epochs. In particular, the par++ model achieved the best results, consistently surpassing the others on all languages (except on Russian where it is outperformed by the bilingual model). Our results also surpassed those of Gangi et al. (  2019 ) by a large margin. We observe the largest improvements on Portuguese (+1. 94 at line 5, +1. 50 at line 6, and +1. 47 at line 7, compared to the bilingual result at line 1), which has the least data among the 8 language pairs in MuST-C. This phenomenon is also common in multilingual neural machine translation where multilingual joint training has been shown to improve performance on low-resource languages  (Johnson et al.
Conclusion We introduced a novel dual-decoder Transformer architecture for synchronous speech recognition and multilingual speech translation. Through a dual-attention mechanism, the decoders in this model are at the same time able to specialize in their tasks while being helpful to each other. The proposed model also generalizes previously proposed approaches using two independent (or weakly tied) decoders or chaining ASR and ST. It is also flexible enough to experiment with settings where ASR is ahead of ST which makes it promising for (one-to-many) simultaneous speech translation. Experiments on the MuST-C dataset showed that our model achieved very competitive performance compared to state-of-the-art.
Acknowledgements This work was supported by a  Facebook AI SRA  grant, and was granted access to the  HPC resources of IDRIS  under the allocations  2020-AD011011695  and  2020-AP011011765  made by  GENCI. It was also done as part of the  Multidisciplinary Institute in Artificial Intelligence MIAI@Grenoble-Alpes  ( ANR-19-P3IA-0003 ). We thank the anonymous reviewers for their insightful feedback.
Smaller datasets include Europarl-ST (Iranzo-Sánchez et al. , 2020)  andMaSS (Boito et al. , 2020). Recently, a very large many-to-many dataset called CoVoST-2(Wang et al. , 2020b)  has been released, while its predecessor CoVoST(Wang et al. ,  2020a)  only covers the many-to-one scenario.
There is a long history of ecological research on the role of traits in predicting shifts in community composition along environmental gradients  (Grime, 1977; Poff, 1997; Southwood, 1977; Townsend & Hildrew, 1994; Winemiller, Fitzgerald, Bower, & Pianka, 2015). Several initiatives have collated trait information for different groups of organisms, and multiple traits are routinely measured on individual organisms or referenced from databases after taxonomic identification  (BirdLife International, 2019; FishBase Consortium, 2018; Kattge et al. , 2011; Schmidt-Kloiber & Hering, 2015; Vieira et al. , 2006). Applications of trait-based ecology have driven important new insights into biodiversity responses to global environmental change  (Brown et al. , 2018; Díaz et al. , 2016; Newbold et al.
To take an example from the freshwater realm, previous global  (Brown et al. , 2018) , continental  (Blanck & Lamouroux, 2006; Statzner, Bis, Dolédec, & Usseglio-Polatera, 2001; Statzner, Dolédec, & Hugueny, 2004 ) and river basin scale  (Dolédec et al. , 1999; Heino, Schmera, & Erős, 2013)  meta-analyses have shown that while trait-environment relationships are generally consistent across those scales, systematic spatial effects on trait structure are often clearly evident. For instance, working in alpine streams,  Brown et al. (2018)  reported a significant contraction of functional richness and a shift in trait structure towards taxa with smaller body sizes and shorter life cycles, as well as shifts in diets, at higher latitudes. In contrast,  Statzner et al. (2001)  reported low variability in trait structure and trait response to disturbance among a collection of datasets from various stream types in Europe.
| Integrating open data sources to test assumptions of trait-based ecology There are now >1 billion species' occurrence records in the Global Biodiversity Information Facility (GBIF), and >1. 7 million sequences publicly available in the Barcode of Life Database. The Open Tree of Life now has >2. 6 million tips in its synthetic phylogenetic tree. High-resolution climate (e. g. WorldClim) and elevation (e. g. SRTM) data are openly available and readily integrated into large-scale statistical models  (Fick & Hijmans, 2017). This creates new opportunities for trait-based research at large scales that have yet to be fully realized  (Culina, Crowther, Ramakers, Gienapp, & Visser, 2018; Violle, Reich, Pacala, Enquist, & Kattge, 2014). We stipulated three basic criteria for trait independence: (a) there should be minimal statistical correlations between traits; (b) traits should be minimally constrained by phylogeny; and (c) trait structure, and its response to environmental change, should be minimally variable with respect to spatial coordinates at the scale considered.
Observed pairwise correlations (r obs ) are then ranked among the null distributions to derive two-tailed p values, Bonferroni adjusted for multiple pairwise comparisons. For applications involving exclusively binary or continuously measured traits, this step may be simplified to a standard correlation test. In cases where binary or continuously measured traits are mixed with fuzzy coded traits then our framework may be applied directly. To enable phylogenetic constraints to be quantified, a phylogenetic tree is constructed with tips corresponding to species within the taxa included in the trait database at mixed taxonomic levels (Figure  1b ). A trait distance matrix is produced from the observed traits and the constraint quantified as the cumulative correlation between traits and phylogenetic distance, compared to a null hypothesis of no phylogenetic autocorrelation.