dataesr/scientific-paragraphs-categorization huggingface dataset:
AWS batch implementation In addition to the regular implementation of lncEvo, a separate execution script designed to be run in the cloud is provided (Fig. 4 ). The pipeline can be launched either on a local computer or as an AWS EC2 instance; the latter is suggested for heavy workloads or long runs. AWS Batch dynamically provides the optimal quantity and type of computer resources based on the volume and specific resource requirements of the submitted batch jobs. Nextflow automatically manages the computing environment, allocates resources, and initiates and terminates nodes for each task according to the executed directed acyclic graph (DAG), job queues and computing environments. AWS batch settings are located within an awsbatch. config file. Nextflow retrieves the credentials from the user ~/.
Output The main output data LncEvo generates output data for each of the three consecutive steps: transcriptome assembly, the lncRNA search and the lncRNA conservation analysis (Fig. 5 ). The transcriptome is available in GTF format for downstream processing and in a FASTA file, which contains the sequences of the transcripts. The transcript expression values can be found in a tab-separated file in the expression directory. The predicted lncRNAs are reported in GTF and FASTA formats as well, but there is also a TSV file with extended annotation data, such as the relationships of lncRNAs to the reference annotations, the results of the protein coding potential evaluation or the expression values provided in TPM units. The output of the conservation study is presented in two files: one file contains the top-ranked orthologs selected based on the exonic identity, and the other file contains all of the possible conserved counterparts; the lncRNAs in both files are presented with the calculated conservation metrics.
If a cross-species chain file is created, it is also stored for possible future runs using the same or other samples. Importantly, for lncRNA identification and conservation studies, interactive reports are provided in the form of HTML files that present the statistics in columns and the associations in rows. Fig. 5  The structure of the output files, with an analysis of sequences from Canis lupus familiaris compared with human lncRNAs shown as an example. The QC folder contains multiple quality reports, with multiQC. html containing data from a number of software utilities. The set of predicted lncRNAs is kept in the PREDICTION folder, whereas the assembled transcriptome is available in the species_name_final. gtf and species_name_final. fasta files. The file hsapiens. clfamiliaris. over.
Quality reports and log files Nextflow generates an HTML execution report with a number of metrics regarding workflow execution. The file is organized into three sections: Summary, Resources and Tasks. The summary section reports the execution status, launch commands, overall execution time and supplementary workflow metadata. The Resources section contains the plots of the distributions of resource usage for each workflow process, which are generated using the plotly. js JavaScript library. Finally, the Tasks section lists all executed tasks with their status and the actual command that was applied. There are also detailed reports for each of the lncEvo analytical steps. For transcriptome assembly, the FastQC HTML report provides the results of the quality control check of the raw input sequences; Fastp provides the quality control reports for the RNA-Seq data in a single HTML document.
Performance and test run To test lncEvo, five species were selected: human (Homo sapiens), mouse (Mus musculus), chimp (Pan troglodytes), horse (Equus caballus) and dog (Canis familiaris). In the conservation study, humans were used as a reference. The data used for testing were fetched from ENA using a Python script [https ://githu b. com/wwood /ena-fast-downl oad]. Data from the following Sequence Read Archive runs were analyzed: SRR4421334, SRR4421792, and SRR4421350 for humans; SRR7771840, SRR7771842, SRR7771843, and SRR7771846 for mice; SRR1602576, SRR1758919, and SRR1758927 for chimps; SRR9133801, SRR10140550, and SRR10205788 for horses; SRR10915304, SRR10915305, and SRR10915307 for dogs. Extensive details regarding analysis of human and mouse lncRNAs are provided in Additional file 1.
Overall, it is recommended to use the fast options. Additional file 1 compares results obtained with medium and medium_fast parameters, with human lncRNAs used as a reference and a mouse as a target species. To reduce the computational time at the slncky stage, the target lncRNA dataset is divided into batches of 2500 transcripts each, which can be executed in parallel with the AWS Batch executor (one can use up to 10 forks at once). One of the main advantages of lncEvo compared with the methodology used by us previously  [4]  is that lncEvo enables the parallelization of tasks. For transcriptome assembly using AWS Batch, the pipeline can handle as many samples at once as the cluster capacity allows, whereas for local execution, this is true only for tasks that do not utilize all available CPUs; however, it still represents a meaningful improvement for the pipeline.
Results of the test runs Using the tested datasets, we predicted from ~ 5000 lncRNA transcripts in dogs up to ~ 25,000 in mice (Fig. 6 ). The differences across species may be explained by the quality of the available annotations and the number and sources of samples used in analysis (i. e. , samples from diverse types of tissues, organs and cell lines). For humans, most of the lncRNAs fall into one of the three Cuffcompare class codes: " = " (equal to known ENSEMBL transcript), "i" (intronic variants of known genes) and "j" (novel isoforms of known genes) (Fig. 6 ). In the other species, most of the transcripts represent novel intergenic transcripts (class code "u"); this reflects the fact that most species have poor lncRNA annotations, which justifies our strategy of identifying the lncRNAs ourselves rather than relying on publicly available datasets.
Once a transcript is detected in a syntenic region, the crossspecies alignment is scored relative to a set of random intergenic regions from one of the genomes-the score value (hence a decision whether a lncRNA is conserved) is going to be different, depending on which species is used as a query. Finally, we checked the performance of lncEvo on a manually selected set of eleven lncRNAs, whose evolutionary conservation has been previously studied. For that purpose, we looked into human lncRNAs and their conserved counterparts in mice (Additional file 1). The analysis demonstrates lncEvo is able to efficiently detect lncRNAs conserved between humans and mice, such as MALAT1 or GAS5, often yielding multiple splicing isoforms of the lncRNA gene. As expected, it finds no orthologs for a primate-specific lncRNA (LINC00473).
annotations; "=": complete match with the intron chain; "j": potential novel isoform; "i": transcript falls entirely within a reference intron; "o": generic exonic overlap with a reference transcript; "u": unknown intergenic transcript; "x": exonic overlap with a reference transcript on the opposite strand. The distributions reflect the quality of the ENSEMBL annotations, but the volume of the sequencing data and the types of samples used also have an impact on the distributions (some tissues possess a much more diverse repertoire of lncRNAs than other tissues).
Downstream analyses LncEvo produces a diversity of tabular data that can be fed into analytical platforms, thus enabling to query, filter the data, and plot graphs, among others. For users who deal with a number of species or transcriptomes and want to track a specific gene, transcript or a group of transcripts, such as coexpressed transcripts or ultraconserved lncRNAs, we have implemented Athena with AWS, thus providing a relatively straightforward way to manipulate the data using SQL syntax, without the requirement to build the actual relational database. Athena is also fully integrated with AWS QuickSight for data visualizations. On the other hand, for those wishing to obtain insight into the overall data statistics, including advanced plotting, SweetViz reports are a feasible option.
b The extent of the overlap with conserved human lncRNAs in the four analyzed species. c The color of the dots represents the species in which a given human transcript is conserved. The exonic identity is represented by the X axis, while locus identity is shown on the Y axis computational infrastructure. Additionally, we focused on extensive quality control and provided options for downstream analysis, such as AWS Athena. For brief and convenient comparisons of the obtained results, SweetViz reports are generated.
Availability and requirements Project name: lncEvo: automated identification and conservation study of long noncoding RNAs Project home page: https ://gitla b. com/spiri t678/lncrn a_conse rvati on_nf Operating system(s): platform independent Programming language: Nextflow DSL Other requirements: >= Java 8, >= Nextflow 2. 20, >= Docker 19. 03 License: MIT Any restrictions to use by non-academics: none Availability of data and materials Source code and the most recent version of lncEvo are freely available under MIT license in the GitLab repository: https ://gitla b. com/spiri t678/lncrn a_conse rvati on_nf. An archived version, referenced in the manuscript, is available at https ://doi. org/10. 5281/zenod o. 42284 73 as well as at https ://gitla b. com/spiri t678/lncrn a_conse rvati on_nf/-/relea ses/1.
0. A Docker image is available at Docker Hub: https ://hub. docke r. com/repos itory /docke r/spiri t678/lncrn a_conse rvati on. Example result data from our test runs (predicted lncRNAs and their conserved counterparts in another species) are available on AWS s3: https ://diffs uff. s3. amazo naws. com/for_paper /test_run_data. tar. gz. Additionally, interactive summary reports for the test runs can be obtained from https ://diffs uff. s3. amazo naws. com/for_paper /test_run_repor ts. tar. gz.
Introduction The rapid growth of seismology in Europe -from the ~30 openly available off-line stations in 1987 when the Observatories and Research Facilities for European Seismology (ORFEUS) was initiated, to the current situation with thousands of stations, many available in near real-time -resulted in ORFEUS evolving from a single centralized archive to a federated set of data centres. The concept of a centralized ORFEUS data centre thus changed through a number of European infrastructure projects (e. g. NERIES, NERA, SERA, EPOS-PP, EPOS-IP) towards an efficient federation of data centres (nodes) governed within ORFEUS, which is now known as the European Integrated Data Archive (EIDA) (see  Data and Resources). The process, envisaged in 2011 was completed with the formal establishment of EIDA in 2013.
EIDA is the federated data and service infrastructure within ORFEUS aiming at secure archiving of seismic waveform data and metadata gathered by European seismic networks and research infrastructures, and provide the geoscience research communities seamless access to data. The EIDA archive is comprised of more than 12,000 stations with focus on the Euro-Mediterranean region (Figure  1 ), affiliated with both permanent and temporary networks. These networks are operated by the data suppliers, and include data from broad-band, short-period, strong motion, infrasound, and Ocean Bottom Seismometers (OBS) sensors. Table  1  provides an overview of the available stations by status, sensor type and accessibility. Data are disseminated via twelve EIDA nodes distributed across Europe in a seamless way through a suite of standardized web services.
With the establishment of EIDA, ORFEUS significantly increased the availability of data from 12 TB in 2011 to 170 TB in 2013 and to nearly 600 TB in 2020 (Figure  2 ). The transition from a single data centre towards a distributed system produced not only and scalability of the system as well as to rely on standard interfaces rather than specific software. As a replacement, EIDA has adopted the standard web services defined by the seismological community (see  Data and Resources). EIDA data access today is based on a portfolio of web services which continue to be extended through collaborative work among the data centres coordinated within ORFEUS and in synergy with the European Plate Observing System (EPOS) developments (see Data and Resources). In this paper, we present the EIDA service infrastructure from standard services of the International Federation of Digital Seismograph Networks (FDSN) to external clients using these EIDA services through ad-hoc developed services to implement the distributed and interoperable system.
The EIDA service portfolio EIDA provides access to seismic waveforms and associated quality information through the various nodes, as shown in Figure  3. Access to more than 12,000 stations from National Research Infrastructures across Europe is provided via a suite of standard FDSN web services and additional web services developed by the ETC. In the following subsections each EIDA service will be introduced briefly, see Data and Resources for documentation and usage about all services.
Fdsnws-dataselect The standard FDSN "dataselect" service is implemented by each EIDA node to retrieve waveform data in miniSEED format, the standard for seismic waveform data within the FDSN. This service handles all EIDA data requests. Its interface is almost identical to the fdsn-station web service, but without the capability to select by geographical location. In addition to that, this web service includes a method (queryauth) to request restricted data by authorized users. The authentication is based on the basic digest authentication implemented in all usual web servers. Users should be aware that it is common for EIDA implementations of this service to apply limits to the amount of data which can be requested. Thus, large requests should be split into sets of smaller requests. For those cases, there are some smart clients which can do this automatically (e.
WFCatalog This quality control service  (Trani et al. , 2017)  has been developed to meet user requests to be able to evaluate data quality before download and hence query for relevant data only. The web service provides detailed information on the contents of the waveform data in an archive. In particular the following features and quality parameters are provided: gaps, statistical values of background noise, availability, overlaps, quality flags. It may be used for quickly exploring metrics calculated on the waveforms before downloading the data, or by clients to fulfill user specific requirements. The API follows the style used by the FDSN web services with some specific additional features. The metrics are computed on fixed daily intervals (day boundaries), in case of gaps metrics are computed for each continuous data segment within the given day.
Routing service The Routing Service  (Quinteros, 2017)  was designed to assist users and clients to locate data within a federated, decentralized collection of data centres such as EIDA. This service can be queried in order to locate the data. Some smart clients (fdsnws_fetch, ObsPy routing client) are also provided to offer the user an integrated view of the entire EIDA, hiding the complexity of its internal structure. This service was developed starting from the former concept of routing table used previously in EIDA when federation was first implemented using ArcLink. It allows scalability both in terms of nodes and services. The current EIDA routing service is not only used by the users to locate specific data but is also used to synchronize the routes within the federation, replacing the old synchronization mechanism that involved merging and replicating the EIDA inventory at all nodes.
Today, only the routes are synchronized and merged such that all nodes know about all routes. The routing service also allows definition of primary (authoritative) routes alongside secondary routes. If more than one node attempts to declare conflicting primary routes for data, this will be detected and moderated. The service harvests the routes (datasets) declared by all nodes to expose them in the official EIDA Routing Service running at ODC, where client tools expect to find it. The approach used by the Routing Service to cope with the declaration of services, datasets and priorities in case of multiple copies, has been adopted by FDSN in its last FDSN Plenary Meeting (Montreal, 2019). There, a standard metadata schema was approved to declare data centres and their available datasets. Thus, all information available in the EIDA Routing Service can be harvested by FDSN to provide a detailed declaration of the services and data holdings available at the nodes (see Data and Resources).
Authentication service Although only a minor part of EIDA data are embargoed, a new authentication and authorization infrastructure (AAI) for the federated system was needed in order to deal with these restrictions, to get better statistics regarding data usage, and also to possibly expand services in the future for authenticated users. In order to have a system that is easy to maintain and that is scalable with services and data in a federated context, authentication and authorization have been decoupled. The service redirects the user to its home institution (if it is affiliated to eduGAIN) to sign in and, in the case of a successful sign in, it provides a digitally signed token in which all available attributes are present. While this works very well for authentication, it may be insufficient to decide if access to restricted data should be granted.
Federator The EIDA Federator can automatically route and retrieve requests federated between EIDA nodes. It provides a single service entry point for the entire EIDA holdings for the fdsnws-station and -dataselect services, and the EIDA WFCatalog service. This prevents users from having to query the routing service before making data requests to the individual EIDA nodes. The federator currently does not support authentication and hence cannot be used to download restricted datasets. Specifications are compliant to the federated web services at the end points: (fdsnws-station, fdsnws-dataselect, WFCatalog). The service has been developed and is operated by ETHZ.
ObsPy An additional smart client has been developed in collaboration with the ObsPy community  (Beyreuther et al. , 2010) , namely the routing client which is capable of using the routing service and the EIDA authentication service. Similar to fdsnws_fetch the routing client is requesting data directly from the end points after liaising with the routing and authentication central services (Figure  3 ). This smart client is widely used to access EIDA data, as a large number of EIDA users use ObsPy / Python for download and data processing.
EPOS-ICS The EPOS-ICS portal is successfully integrating the EIDA services, and the EIDA services are indeed part of the EPOS Seismological Thematic Core Service (EPOS-S TCS). Therefore, data from EIDA is made available to the geoscience community along with other types of data through an interoperable platform for data discovery, access, and processing. This includes also the concept of virtual networks which is used for example by the AlpArray community, and the EPOS Near Fault Observatories (NFO) to collect related stations together.
Interactive access The EIDA portal also provides browser-based interactive access using WebDC3  (Bianchi et al. , 2015). WebDC3 is a web interface to SeisComP standard seismological services allowing users to conveniently discover seismic stations distributed from EIDA data centres, explore events in a number of seismic catalogues, build and submit requests for data and metadata, and finally download the results in different formats. Requests can be built using either absolute time windows or by station-event combinations suitable for different data processing pipelines. The web interface supports tokens generated by the EIDA authentication service.
Data management and dissemination EIDA data policies are coordinated among the various EIDA nodes within ORFEUS. When becoming an EIDA node, a new data centre must commit to an open data policy, is expected to demonstrate a minimum service availability (95%), ensuring in house at least one redundant copy of the data for hot backup and optional cold backup off site (in place at some nodes). Ingestion of data and metadata at each node is performed according to its own procedures with all nodes requested to carefully check consistency of metadata with data suppliers. Some of the EIDA services are synchronized daily through the nodes (e. g. routing service and logging) and all are monitored via their own or federated tools. At all data centres a dedicated technical person is available to provide response to arising issues, in general within one working day.
Through the use of standard services, metadata and data formats, the long term commitment of all EIDA nodes to curate data with a good level of data FAIRness  (Wilkinson et al. , 2016 ) is also achieved. EIDA ensures that seismic networks within its archives have a DOI (Digital Object Identifier), following the FDSN guidelines  (Evans et al. , 2015)  and most EIDA nodes can mint a DOI for the networks they host. EIDA promotes rich metadata, including standard licences where possible (e. g. Creative Commons CC BY 4. 0 or similar). DOIs are also integrated in the seismic station metadata (FDSN StationXML) to formally establish the link with the DOI metadata that can be automatically harvested for a specific seismic network. EIDA requires the users to provide proper reference to the data suppliers.
This can be done by citing the seismic networks with their associated DOIs, or in rare cases when these are unavailable, by using network name and/or FDSN network code. Taking advantage of the development in the SERA project (see Data and Resources) the EIDA group has been working on metadata challenges and proposed solutions to also integrate other types of data than those historically present in the EIDA nodes (e. g. OBS, Infrastructure monitoring, Infrasound). Following these developed guidelines data and metadata from the French and the German OBS pools, collected within the AlpArray project (AlpArray Seismic Network, 2015) have been archived for the first time in a consistent way at more than one node with common pre-processing and metadata creation procedures. In the process of improving workflows, also for network operators and data suppliers, EIDA developers participate in the development of the new FDSN documentation for StationXML, partially sponsored also by ORFEUS.
Discussion and Conclusions With the adopted governance and technical setup, the EIDA system has demonstrated that federated approaches, although more complex and difficult to maintain, are a valid solution to serve users of seismological data when resources are distributed and for political and financial reasons not available in a single institution. With the growing demand for large volumes of data, distributed archives have become a very attractive solution to minimize failures related to single access points. The fundamentals of the present EIDA system is a modular scalable infrastructure based on standard interfaces. Development of a logging system fully compliant with the European General Data Protection Regulation (GDPR) is in progress alongside with a new interactive portal to access data making use also of the quality metrics to let users assemble tailored data sets.