dataesr/scientific-paragraphs-categorization huggingface dataset:
How to use ISA-Tab? Imagine a situation in which a collection of seeds of a number of crop varieties is given to researchers at different sites to compare the influence of the local environment on yield. They perform separate trials on, assumingly, the same set of objects, in similar-but not exactly the same-experimental designs. All general information about Biosource, Environment, Treatments and Experimental Design is to be given in separate Study files for each site. Data can later be aggregated across locations according to the obligatory attribute "Geographic location". Imagine another situation, where an experiment is performed in one location, and many different researchers take samples from it, taking note of the identifier of the plant they analyse. In such a case, there is just one Study file, and a number of Assays for the individual researchers to record detailed description of handling of the samples and measurements.
Dataset I Data contained in 'dataset_basic_GMI_Atwell' (Additional file 1) comes from the investigation described by Atwell et al. [59] , and concerns Arabidopsis accessions. The data was downloaded in the ISA-Tab format from GWA-Portal at GMI. It has been formatted according to the basic phenotyping configuration. The Study file "s_Study1. txt" lists all the Biosources, i. e. Arabidopsis accessions, which are annotated by their identifiers in the GMI's accession list. There are multiple replications of each accession; each one is assigned a unique Sample Name. The Sample Names are repeated in the Assay file "a_study1. txt" which links them to the rows of the Derived Data File "d_data. txt" through Assay Name column. The columns of the Derived Data File correspond to the 107 phenotypic variables stored on the GWAS platform and defined in the Trait Definition File named "tdf.
txt". This example illustrates a situation in which the structure of the ISA-Tab archive does not reflect any actual experiment; the data, exported from an intermediary database, are in fact detached from most of their original metadata. Therefore, the information that is to be conveyed is very simple. One may say that in this situation the ISA-Tab structure, even in its basic configuration, is too complicated. However, obeying the rules even for simple datasets enhances greatly their interoperability.
Dataset II Data contained in 'dataset_field_IPGPAS_Polapgen' (Additional file 2) were obtained in a project aimed at studying reaction to drought in populations of barley recombinant inbred lines (RIL)  [60]. The GeH population, obtained from a cross between Georgie and Harmal, consisting of 100 lines, was observed in a two-year field experiment in 2012 and 2013. The RILs and their parental forms constitute 102 biosources defined at the Study level in two files "s_study1. txt" and "s_study2. txt", corresponding to the two years. The most important environmental data concerning soil type, field size, sowing density, and day temperature are provided as values of Parameters of the appropriate Protocols. Some information required by MIAPPE was not available, therefore a few columns in both Study files are empty.
The phenotyping done on samples taken from field experiments is described in two Phenotyping Assay files, "a_study1_phenotyping_ field2012. txt" and "a_study1_phenotyping_field2013. txt". In the experiment eight phenotypic traits were measured; these are named and annotated in the Trait Definition File "tdf_polapgen_field. txt". Additionally, two environmental variables were recorded: "water vapor pressure deficit" and "total precipitation"; they are also described in the Trait Definition File. The observations of phenotypic traits and of environmental variables are contained in data files "d_polapgen_field2012. txt" and "d_polapgen_ field2013. txt", corresponding to the two assays. The GeH RIL dataset represents a very common case of a multi-environment study made with the same set of plant accessions.
Dataset III The experiment described and data contained in  [53]  have been acquired in the frame of a series of validation experiments in IPK's high throughput plant phenotyping system for small plants. It assessed the effect of plant rotation during imaging (Factor "rotating"/"stationary") as well as of soil covers (Factor "covered"/"uncovered") on growth and development of 484 Arabidopsis plants. The dataset contains raw and processed images, extracted phenotypic features relevant for quantification of biomass (growth) and manual validation data. Detailed information about the experimental procedures and results can be found in  [50]. The study has been described according to a MIAPPE-compliant ISA-Tab phenotyping configuration (Greenhouse Study) and was a part of data descriptor article  [54].
The raw image files can be found in the "1135FA_images" folder. The subfolders are ordered and categorized into "camera_sensor" (vis/fluo/nir), "cam-era_view" (top/side) and "das" (day after sowing). The corresponding ISA-Tab files (Investigation, Study, and Assay files) for the semantical description are located in the "metadata" folder. This dataset demonstrates the application of the ISA-Tab configuration (and MIAPPE) for a high throughput phenotyping experiment comprising time series measurements with different camera sensors. On the basis of this example the integration and representation of further related data (novel sensors, and importantly, environmental data) will be done at IPK.
Conclusions The results of research funded from public resources are expected to be publicly available, not only as a proof that the research was done, but also as a source of knowledge, or even input for further analyses. Open access to data is usually provided through open repositories (e. g. Dryad or Zenodo). They implement different policies of data formatting and description. Some accept objects (including datasets) of any type, assigning them simply an ID; others require adding a set of general attributes describing an object; some more ask for a specific data format. Repositories and databases of particular institutions and projects provide their own way for storage and access to data, most suitable for their needs, with an increasing policy toward Open Access. Future usability of datasets dispersed across all those repositories relies upon numerous factors: possibility to extract a specific dataset together with its metadata, comprehensible dataset formatting, completeness of its description, and clear meaning of individual elements of this description.
Compatibility with other experimental results, also those of different types, is also not to be neglected in the context of data interoperability. Our work has been aimed at moving phenotyping data towards these objectives: • The MIAPPE document, defining recommendations for phenotypic dataset description elements, helps to provide the right metadata in the dataset. • The ISA-Tab format allows experimental metadata formatting, and thus inclusion of all important information within the dataset, making it exchangeable and independent of a data repository's metadata policy. Flexibility of the format allows to export databases' internal structures as ISA-Tab, while the definite rules for element arrangement make the experimental process traceable. • The ISA-Tab Phenotyping Configuration provides mapping of MIAPPE requirements to ISA-Tab structures for the basic phenotyping situations, and thus facilitates dataset construction.
Thanks to holding information on ontologies for particular attributes, it supports data annotation. A list of recommended ontologies for annotation of particular elements of experiment description assists in choosing formal terminology to clarify the wording, and thus avoiding ambiguity of the description. Ontological annotation is accommodated in ISA-Tab datasets. The Minimum Information About a Plant Phenotyping Experiment document has been constructed as a result of consultations with a number of research groups within the transPLANT project and beyond, especially EPPN and DPPN. Although it is focused on classical phenotyping experiments, some attention in MIAPPE is also given to less frequently performed, but nonetheless important, experiments in aquatic and biotic conditions. Yet, a real application of MIAPPE in such situations would require more discussion with relevant practitioners.
The same remark applies to observational studies. Based on experimental data from high throughput plant phenotyping experiments at IPK using the Lem-naTec platform, a first version of a high throughput phenotyping configuration has been prepared. This work builds the basis for a comprehensive plant phenomics experiment documentation and data publication pipeline. Indeed this kind of experiments comprising automated multisensor-imaging-based procedures can produce terabytes of data for each experiment. Handling such Big Data needs dedicated technologies and the level of resolution of related experimental metadata to be represented and published using ISA-Tab archives is still a matter of discussion. The selection of an adequate level of detail (geographical location of every single pot vs. location of the greenhouse), data volume (whether to remove low quality images or not) and processing stage (raw images vs.
compressed/processed images) for data publication is linked to the technical capability of publication servers as well as institutional or journal policies. Nevertheless, the continuous documentation of the data lifecycle is a basic requirement for a consistent and seamless creation of ISA-Tab archives. We hope that the discussion with interested parties dealing with this type of experiments will allow a general or platform-specific High-Throughput Phenotyping Assay to be developed. The textual nature of the ISA-Tab format makes it directly readable for everyone, without the need for any special software and support from computer scientists or bioinformaticians. Similarly, the construction of a dataset is possible manually, in a text or spreadsheet editor, by filling in a prepared template.
A more advanced option is the preparation of an own implementation of data export/import as ISA-Tab based on the format specification to combine ISA-Tab with existing databases and tools. ISA-Tab is also supported by a free software suite, ISA-Tools, developed by ISA group  [61]  and members of the community. There are a number of tools and APIs for dataset construction, validation, analysis, management, and export to other formats. Certain functionalities of this official tools distribution are not yet provided, but the implementation of new user-friendly environments for dataset management is in progress  [62, 63]. Further development of tools supporting formatting of data according to the given rules is an important step to promote adoption of the metadata standards. Since the textual nature of ISA-Tab makes it not particularly convenient for automatic processing, the possibility to export ISA-Tab dataset structure to other formats is a useful feature.
The existing tools provide, among others, JSON and RDF representations, as well as OWL for compatibility with the Linked Data. ISA-API  [64]  is going to further simplify programmatic approach to data formatting and management. The ISA-Tab format has been accepted by the Nature Publishing Group for dataset publication, which additionally popularizes the format and encourages new users. More work is needed to achieve a widespread acceptance of the policy of data publication in the form of open resources. The FAIR Data Principles  [58]  that define the properties of a good dataset are a convenient remainder of the targets that are to be aimed at. Acceptance of the rules described in this paper will help to achieve these targets by the plant phenotyping community.
Funding The work was supported by the  European Union  under the  FP7 thematic priority Infrastructures , Project No. 283496  transPLANT and Project No. 28443 EPPN-European Plant Phenotyping Network , by the  German Federal Ministry of Education and Research (BMBF)  Grant No. 031A053A/B/C DPPN-German Plant Phenotyping Network , and by the  Infrastructure Biologie Santé.
The ATLAS EventIndex is the catalogue of the event-related metadata for the information collected from the ATLAS detector. The basic unit of this information is the event record, containing the event identification parameters, pointers to the files containing this event as well as trigger decision information. The main use case for the EventIndex is event picking, as well as data consistency checks for large production campaigns. The EventIndex employs the Hadoop platform for data storage and handling, as well as a messaging system for the collection of information. The information for the EventIndex is collected both at Tier-0, when the data are first produced, and from the Grid, when various types of derived data are produced. The EventIndex uses various types of auxiliary information from other ATLAS sources for data collection and processing: trigger tables from the condition metadata database (COMA), dataset information from the data catalogue AMI and the Rucio data management system and information on production jobs from the ATLAS production system.
Introduction The ATLAS experiment  [1]  runs a complex system that employs detectors with millions of readout channels, and gathers billions of events in data taking periods that may extend to several years. It produces several petabytes of data per year, with different physics conditions and data formats. Different types of metadata (data about data) make possible the effective usage of this tremendous amount of information. Dataset metadata deal with data storage organization. Run-wise metadata deal with slowly changing parameters and links to the data-taking conditions. It is useful to have a catalogue containing information about the basic properties of event data and references to their storage locations, and also providing a means of rapid search through the event metadata. The EventIndex  [2]  is a system designed to be a complete catalogue of ATLAS events, real and simulated data at all processing stages.
There are a number of use cases for the EventIndex:  Event picking: a request for reference to the specific event in given format and processing type.  Event counting: counting the number of times that a specific trigger decisions were passed. The information is collected by the EventIndex using trigger tables from the Condition Metadata Store (COMA)  [3]. Results are provided to users also by COMA.  Duplicate finding: detecting duplicated data within the scope of a dataset or physics collection. The information is supplied to the users through the ATLAS Metadata Interface (AMI)  [4]. Different data types  [5]  are processed by the EventIndex and may be treated differently. Top priority for the EventIndex are Analysis Oriented Data (AOD), both for real and simulated (MC) data. The indexing of AODs provides also indexing of RAW data through the provenance information that is picked up.
MC datasets containing generated events (EVNT) are indexed for duplicate event detection. Routine indexing of derived data (DAODs) is currently not performed, because of a short useful lifetime and large amount of data (10x AODs). Still the capability to index DAOD's is preserved; a very small subsample is indexed for that purpose. We avoid repeated processing of the trigger data for the same events, so we have trigger processing enabled only for real data AODs from Tier-0 (including spill-over) and for MC AODs. The trigger processing is disabled for real data AOD from reprocessing and for DAODs.
EventIndex structure and dataflow The EventIndex infrastructure consists of a few autonomous components, providing the collection and processing of EventIndex data. The modular approach allowed sharing the developments between different institutions. The main components and dataflow are presented in Figure  1. The information is collected by event processing tasks ("producers") which can run at Tier-0 (initial reconstruction at CERN) or on Grid sites (downstream processing). Event metadata collected by the producers is sent via the ActiveMQ [6] message broker to the Hadoop [7] store at CERN. At CERN messages are read by Consumer program and stored as Hadoop Mapfile (a file-based map from keys to values) objects. Data are validated for completeness and duplicated events, and decoding of the trigger information performed using trigger tables from COMA.
Users can access EventIndex information through the query services: command line interface and web services with graphical and text/plain output suitable for the ATLAS PanDA  [8]  (Production and Distributed Analysis) system  [9]. On the Grid the EventIndex uses ProdSys2  [10] , a second generation ATLAS production system, to index datasets as soon as they appear in the AMI catalogue and are marked as valid and completed. Information on data produced/modified daily on the Grid is gathered by automatic scripts by querying the AMI metadata catalog for VALID datasets from specific datasets from mc_15 and data_15 projects, and soon mc_16 and data_16 too. Then a catalogue in the ATLAS Distributed Data Management system  [11]  (Rucio) is queried for the identifiers of the datasets to feed the EventIndex processing jobs.
Collected datasets are attached to the special "technical containers" by cron script, to be picked up by the EventIndex production tasks. There is one technical container for each type of data to be indexed and trigger information processing setting. A new set of technical containers is being created when a new version of the EventIndex producer appears. For each dataset that in the technical container, ProdSys2 creates a Grid task that extracts event data and sends information to the "consumers" running on servers at CERN  [12]. This operation is done in the following steps (see Figure  2  contains all the files (GUIDs) and all the events, and detecting multiple processing of the same file and job failures. The consumers send back statistical information, containing the counts of messages, files and events to the broker to report the status.
The list of trigger chain names, obtained after decoding are then stored in updated event records. The EventIndex uses trigger tables replicated from COMA to HBase for the trigger decision decoding. The trigger tables use SMK as key and ChainName:ChainCounter pairs as columns. Additionally there is a Run:SMK table for verification. As of the beginning of 2016 the trigger tables contain information on 616 SMKs and 10526 runs. Trigger table replication is done by a cron job running daily. Automatic replication is in progress since March 2015. The EventIndex is a source of data on trigger event counts for COMA. The EventIndex program does the counting on a server within the CERN network providing information through a RESTful service that can be accessed from outside CERN. We use AMI information for identifying new and changed datasets that have to be processed by the EventIndex.
We query AMI through a python interface for a list of complete dataset containers of certain types with last modification time on the previous day and flagged as "valid". As large datasets in ATLAS can be produced by a number of tasks, containers are made of one or few task ID (TID) datasets that have to be specified as an input for the EventIndex production job. Information on such datasets is obtained ATLAS data management system (Rucio). The current procedure based on dataset modification times has a number of drawbacks. It is difficult to detect datasets that became corrupted or obsolete. Dataset modification times may change for reasons not related to their contents, requiring unnecessary re-importing of data. There is no feedback to AMI that may be useful to tag corrupted data based on the result of the EventIndex verification.
It was proposed to introduce a special table in the AMI for information exchange between the two systems. Information on the new, changed and invalid datasets will be placed by AMI to a special field, AMI_DS_STATE, with a timestamp. The EventIndex will put dataset import status into another field, EI_DS_STATE, providing to AMI information on data consistency problems detected by the EventIndex. This machinery is currently in the process of implementation by the AMI Team. The development of the EventIndex started in 2012 and at the beginning of 2015 all its components were operative. The import of data from the previous years of LHC operation started in February 2015 and now is complete. Run 2 data import started when it became available in summer 2015 and now it flows continuously from the Tier-0 and Grid and is available to users almost in real time.
):  The producer process extracts EventIndex information from each event in a file and fills an intermediate file on the worker node. Event data are then read from the intermediate file, trigger masks are compressed, duplicate information is dropped and records are encoded into JSON to build the text message.  EventIndex uses the Streaming Text Oriented Messaging Protocol [13] (STOMP) to transport information between producer and consumers. The message size is set small (1-10 kB).  Messages are received by the ActiveMQ message broker and queued for the consumers.  Messages are finally read by EventIndex consumers, decoded and organized into Hadoop MapFile objects.  Consumers also run a validation step, verifying that the EventIndex for a collection (dataset).
This information is further processed at the VariantParser step to calculate the fraction of deviant reads in each group. This information is used to identify loci showing significant differences between groups. The CNVs detected using VariantParser are annotated and scored using the DoC information from paired-end sequencing data. As a test case we used two data sets. 1) Mate pair (MP) data generated from two wild and two domestic rabbits (average insert size of 4. 5 kb and average coverage of 3x) for PEM signatures (Figure  1 ). 2) Rabbit paired-end (PE) sequencing data from pooled samples of wild and domestic rabbit populations with an average coverage of 10x per population  (Carneiro et al. , 2014)  was used for DoC information. For this particular test case, DevRO uses information of two groups (wild and domestic rabbits) to find SVs with significant frequency differences between the two groups.
Methods The input of DevRO is a set of aligned MP or PE reads in SAM/BAM format  (Li et al. , 2009). All input BAM files from multiple populations are analyzed jointly, to avoid pairwise comparisons in the end. The pipeline DevRO consists of three modules 1) VariantCaller 2) VariantParser and 3) VariantAnnotate that are used to detect inversions, deletions and duplications when comparing two or more populations as well as deletions in a reference genome seq. Variant Caller. We used BAM files  (Li et al. , 2009)  from MP data as an input to DevRO VariantCaller. In this step, we scan the genome in windows of 1kb to search for PEM signatures using discordant reads (Figure  1 ). The discordant or deviant reads have (I) abnormal mapping distance in comparison to average mapping distance, the threshold for declaring a PEM as deviant is that it deviates more than three standard deviations from the mean which is a signature for deletion or insertion, (II) abnormal relative mapping orientation (in case of duplication or inversion) as shown in Figure  1.