dataesr/scientific-paragraphs-categorization huggingface dataset:
A unique aspect of the AGP is the open community process of assembling the Research Network and analyzing these data, which are released immediately on data generation. Analysis details are shared through a public forum (GitHub, https://github. com/knightlab-analyses/american-gut-analyses). Scientific contributions to the project were made through a geographically diverse Research Network represented here as the American Gut Consortium, established prior to project launch and which has grown over time. This model allows a "living analysis" approach, embracing new researchers and analytical tools on an ongoing basis (e. g. , Qiita [http://qiita. microbio. me] and GNPS  [34] ). Examples of users of the AGP as a research platform include educators at several universities, UC San Diego Athletics, and the American Gastroenterological Association (AGA).
1A ). This research was performed in accordance with the University of Colorado Boulder's Institutional Review Board protocol number 12-0582 and the University of California San Diego's Human Research Protection Program protocol number 141853. Self-reported metadata were collected through a web portal (http://www. microbio. me/americangut). Samples were collected using BBL culture swabs (Becton, Dickinson and Company, Sparks, MD) and returned by mail. Samples collected in Australia and the United Kingdom were shipped using domestic post within each country to an aggregation site and stored at Ϫ80°C at the aggregation site until shipment to the United States. Shipment to the United States was done on dry ice using a certified shipping service. All samples were handled and processed in kind with other American Gut samples.
For more information, please find an extensive benchmark of shipping conditions in reference 17. All samples were processed using the EMP protocols. Briefly, the V4 region of the 16S rRNA gene was amplified with barcoded primers and sequenced as previously described  (51). Sequencing prior to August 2014 was done using the 515f/806r primer pair with the barcode on the reverse primer (52); subsequent rounds were sequenced with the updated 515f/806rB primer pair with the barcode on the forward read  (52). Sequencing batches 1 to 19 and 23 to 49 were sequenced using an Illumina MiSeq; sequencing for batches 20 and 21 was performed with an Illumina HiSeq Rapid Run, and round 22 was sequenced with an Illumina HiSeq High-Output. 16S data processing. The 16S sequence data were processed using a sequence variant method, Deblur v1.
Rarefaction was used to mitigate uneven sequencing depth in accordance with the benchmarking in reference 56. Diversity calculations were computed using scikit-bio 0. 5. 1 with the exception of UniFrac  (13) , which was computed using an unpublished algorithmic variant, Striped UniFrac (https://github. com/biocore/unifrac), which scales to larger data sets and produces results identical to previously published UniFrac algorithms. In brief, unweighted UniFrac computes a dissimilarity between two samples by summing up the amount of branch length that is unique to each sample and divides this by the sum of the branch length that is common to the two samples. Metadata curation. To address the self-reported nature of the AGP data and the ongoing nature of the project, basic filtering was performed on the age, height, weight, and body mass index (BMI).
BMI categorization was considered only for adults over the age of 20 years, since the description of BMI in children is based on their age and sex. Education level was considered for adults over the age of 25 years. This threshold was used to match the available data from the U. S. Census Bureau  (57). The percentage of the American Gut participants was calculated as the fraction of individuals who reported results for that variable. U. S. population data are from the 2010 census (58); U. S. Census Bureau reports (57); Centers for Disease Control reports on obesity  (59) , diabetes  (60) , inflammatory bowel disease (IBD)  (61) , and smoking  (62) ; and a report from the Williams Institute (63) (Table  S2 ). Alpha-and beta-diversity analyses within American Gut. OTU tables generated in the primary processing step were rarefied to 1,250 sequences per sample.
Meta-analysis movie. American Gut samples from all body sites were combined with data from an infant time series (ITS)  (49) , a fecal transplant study  (44) , and recent work characterizing the microbiome of patients in the intensive care unit  (24). The combination of the data sets in the figshare supplementary movie S2 in reference 50 required that all sequences were trimmed to an even length of 125 nucleotides. All projects except for the infant time series were sequenced using an Illumina instrument. In order to combine the data, we expressed the Illumina and non-Illumina data through a common reference database. Specifically, the Deblur sOTUs from the Illumina data were mapped against the Greengenes (19) database (13_8 release) using 99% similarity; the associations between the input sOTUs, and their cluster memberships, were used to construct an OTU table based on the original sOTU-per-sample sequence counts (i.
e. , summing the counts for all sOTUs in a common OTU). The infant time series data were picked using a closed reference OTU picking approach against the same reference at the same similarity. The infant time series data set followed a closed reference OTU picking approach using 99% similarity. The resulting two tables (from Illumina-generated data and the ITS data set) were merged and analyzed using the Greengenes 99% tree. The table was rarefied to 1,250 sequences per sample. Principal-coordinate projections were calculated based on unweighted UniFrac distance  (13). The principal-coordinate analysis was visualized and animated in EMPeror 1. 0. 0-beta8-dev (66, 76). The movie was captured in QuickTime (Apple, Cupertino, CA) and edited with Premiere Pro (Adobe, San Jose, CA). Integration with the Earth Microbiome Project.
A precomputed 100-nt Deblur BIOM table representing the data in reference 1 was obtained from ftp://ftp. microbio. me/emp/release1/otu_tables/ deblur/. One-hundred-nucleotide (100-nt) Deblur tables were also obtained from Qiita for Hadza fecal samples (Qiita study identifier [ID] 11358  [22] ), ICU microbiome samples (Qiita study ID 2136  [24] ), and a longitudinal series which includes samples immediately prior to and following a large bowel resection (Qiita study ID 10283, EBI accession no. ERP105968, unpublished); all samples were processed using the EMP Illumina 16S V4 protocol. The EMP data set used a minimum sOTU count of 25; the same threshold was applied to the other data sets included prior to merge. Blooms as identified by the method in reference 15 were removed from all samples.
This collection of BIOM tables was then merged, yielding an OTU table representing 40,600 samples. sOTUs were restricted to those already present in the EMP 100-nt fragment insertion tree, which represents 329,712 sOTUs. The table was then rarefied to 1,000 sequences per sample, and unweighted UniFrac was computed using 768 processors with the aforementioned Striped algorithm. Visualizations and animations were performed using EMPeror v1. 0. 0b12. dev0. Extreme diet study state assessment. The sequence data from reference 25 were processed by Deblur to assess 16S sOTUs in common with the AGP processing above. In order to assess a state difference with PERMANOVA, we needed to control for sample independence within the longitudinal sampling. To do so, we randomly selected one sample from each individual per diet, computed PERMANOVA, and repeated the process 100 times.
For all of the MS1 features detected across all samples, only ~1 to 5% are  (35)  present in an individual sample. Molecular networking. Raw mass spectrometry data files were converted to the. mzXML format using Bruker Data Analysis software and uploaded to the GNPS (https://gnps. ucsd. edu/) MassIVE mass spectrometry database (https://massive. ucsd. edu/). Molecular networking  (35)  was performed to identify spectra shared between different sample types and to identify known molecules in the data set. All annotations are at level 2/3 according to the proposed minimum standards in metabolomics  (36). The data were filtered by removing all MS/MS peaks within Ϯ17 Da of the precursor m/z. MS/MS spectra were window filtered by choosing only the top 6 peaks in the Ϯ50-Da window throughout the spectrum.
Chemical annotations were carried out by automatic matching of fragmentation spectra to multiple databases using Global Natural Product Social Molecular Networking (GNPS)  (34)  and then examining the data at the MS/MS level by molecular networking  (35). The goal is to retrieve spectra with identical and similar fragmentation patterns and combine them into consensus nodes and clusters, respectively. The consensus node spectra are then compared against public MS/MS libraries to provide molecular annotations  (80). Further annotations could be suggested by examining the molecular network (35) (enabling the propagation of annotations). The links to the job for the molecular networks used for this study can be found at http://gnps. ucsd. edu/ProteoSAFe/status. jsp?taskϭ9bd16822c8d448f59a03e6cc8f017f43 (antibiotic subset) and http://gnps.
coli bloom in the AGP samples and 2 contained the 16S from other blooms recognized by the work in reference 15. Shotgun sequence processing. Raw Fastq files were processed using Atropos v1. 1. 5 (102)  to remove adapters and low-quality regions. Putative human genome contaminations were identified and removed by using Bowtie 2 v2. 3. 0 (103) with the "--very-sensitive" option against the human reference genome GRCh37/hg19. Sequences were assigned taxonomy using Kraken v1. 0. 0 (95) against the "standard" database built according to the Kraken manual, which contains all complete bacterial, archeal, and viral genomes available from NCBI RefSeq as of 3 August 2017. Results were processed using Bracken v1. 0. 0 (104) to estimate the relative abundance of species-level taxa. Metagenome sequencing data were assembled using SPAdes v3.
Of these, 240 belong to the antibiotic ("Abx") cohort, and 203 belong to the "plant" cohort. We ran FragGeneScan (v1. 30 "run_FragGeneScan. pl -completeϭ0 -trainϭillumina_10 -genomeϭ Ͻinput. faϾ -outϭϽout. fileϾ") (108) on quality-controlled reads to infer partial protein sequences and then ran HMMER (v3. 1b2 "hmmsearch --cut_tc --cpu 4 --tblout Ͻout. fileϾ Ͻresfam. dbϾ Ͻinput. fileϾ") (109) against the ResFam database, version 1. 2. 2  (110, 111) , to identify putative antibiotic resistance genes. The ResFam database contains entries for 170 antimicrobial resistance (AMR) gene families. Of these, 122 belong to at least 1 of 16 antibiotic classes, all 170 belong to at least 1 of 21 mechanism classes, and 42 belong to 5 beta-lactamase classes in the Ambler classification. We normalized the counts against the total number of reads per sample.
1. Data availability. All sequence data and deidentified participant responses can be found in EBI under project PRJEB11419 and Qiita study ID 10317. Per-sample EBI accession numbers and download links partitioned by the sequencing run are described in Table  S1. The sOTU table following bloom removal containing the 9,511 samples used in the manuscript can be accessed from figshare in both read count  (113)  and relative abundance  (114) , as well as the corresponding phylogeny  (115) , sample metadata  (116) , collated alpha-diversities  (117) , and unweighted  (118)  and weighted (119) UniFrac distance matrices. For the mass spectrometry, data are accessed through https://gnps. ucsd. edu: MSV000080187 (antibiotic study cohort), MSV000080186 (plant consumption frequency study cohort), and MSV000081777 (cell cultures of blooms).
For the novel annotated structures of N-acyl amides, the reference MS/MS spectra at GNPS are CCMSLIB00004679232, CCMSLIB00004679233, CCMSLIB00004679234, CCMSLIB00004679235, CCMSLIB00004679236, CCMSLIB00004679237, CCMSLIB00004679238, CCMSLIB00004679239, and CCMSLIB00004679240. All of the raw data are publicly available at the UCSD Center for Computational Mass Spectrometry (36) (data set ID MassIVE MSV000080179). The additional sequence data generated from the American Gut samples were deposited in EBI under the American Gut accession (ERP012803), and the storage sample data were deposited under their accession (ERP015155). The deidentified participant and sequence data for the American Gut are available from the European Bioinformatics Institute under accession number ERP012803. Per-sample accession numbers are included in the supplemental material (Table  S1 ).
Sensors gained a significant role in the Internet of Things (IoT) applications in various industry sectors. The information retrieved from the sensors are generally stored in the database for post-processing and analysis. This sensor database could grow rapidly when the data is frequently collected by several sensors altogether. It is thus often required to scale databases as the volume of data increases dramatically. Cloud computing and new database technologies has become key technologies to solve these problems. Traditionally relational SQL databases are widely used and have proved reliable over time. However, the scalability of SQL databases at large scale has always been an issue. With the ever-growing data volumes, various new database technologies have appeared which proposes performance and scalability gains under severe conditions.
INTRODUCTION The growth of the IoT domain has increased importance for processing a large amount of sensor data. Sensor-based databases are a particular type of database with a constant predictable data flow  (Kepka et al. , 2017)  in a fixed interval of time. The data flow is generally fixed with the sensors which are then attached to the units. These units are physical entities, for instance: a tractor in the field, field node with several connected sensors or a smart-phone  ( Řezník et al. , 2016) ,  (Kepka M. , 2012). Sensor networks are widely used in several surveillance applications. These applications help us get the useful data for post-processing and analysis.
SensLog Case Study SensLog is web-based sensor data management system suitable to process data both from static and mobile sensors. The SensLog application consists of server-side part and database part. Raw measurements from sensors are taken to gateways  (Kepka et al. , 2017). These gateways send the flow of measurements to the database with the SensLog API. Sens-Log database schema was designed and developed during recent years with features respecting standard ISO 19156  (Probst, 2008). With the help of Sens-Log team, we have observed SensLog sensor-based data model and storage  (Kepka et al. , 2013) , which, as in traditional data-intensive applications, relies on the relational database. The current implementation of SensLog data model in PostgreSQL database system  (Pos, 2018)  seems to be still effective.
Objectives Over the past years, Softeam research team has conducted several studies on the methods for modeldriven design of data-intensive applications  (Mar, 2018; da Silva et al. , 2014; Aurélio Almeida da Silva and Sadovykh, 2014). Part of this work is the migration problem to help the architects and software engineers to move easier from one data storage technology to another with automated tools. In the context of the DataBio project  (Dat, 2018)  which focuses on data-driven bioeconomy, we develop the modeldriven NoSQL migration tools in  Modelio (Mod, 2018). Our major hypothesis is that the migration process should start with analysis and denormalization of relational structures  (Mar, 2018). The next steps are the transformation to NoSQL (e. g. Mon-goDB) meta-model and deployment.
BACKGROUND In the world of software engineering, we have seen continuous changes in languages, platforms, and architectures. However throughout relational database has been the default choice for serious data storage in the industry context. With the growing data volumes, the need to integrate data from multiple sources and continuous strive for scale, there came new challenges in data storage. Thus, new database technologies such as MongoDB (Mon, 2018b),  Cassandra (Cas, 2018 ), Neo4j(Neo, 2018)  and Azure table storage (Azu, 2018) came into existence. NoSQL databases provided the advantage of building systems  (Pokorny, 2013)  that were more performing, scaled much better and were easier to program for. Various proof of concepts and surveys have shown benefits to apply NoSQL databases  (Pokorny, 2013).
It has been proven particularly advantageous for the databases which do not rely much on the relations and/or could be denormalized with some measures  (Pokorny, 2013; Dhanachandran, 2012). One of the promising cases in this respect could be the sensor industry. Working with SensLog team who specializes in their tool for processing sensor queries, we furt- hermore explore the idea of processing sensor queries with the warehousing approach that relies on a document-oriented NoSQL database  (Balazinska et al. , 2007). In this approach, we principally have a data lake and an analytics database. The data lake is populated by the sensor gateway, which stores the sensor data in a predefined way for all the insertions. Then, this data could be transformed into a form that is easier for analytics application.
MIGRATION The current SensLog application is implemented with a PostgreSQL database. The SensLog application encounters certain performance limitations and our goal was to understand the bottlenecks and propose a more efficient architecture utilizing NoSQL concepts. Therefore, to start developing proof of concept, we had to explore the existing relational schema of the current SensLog application. We chose to translate the SQL schema into a data model using  Modelio SQL Designer (SQL, 2018). Our working hypothesis is denormalizing the data model in order to map it to NoSQL  (Mar, 2018). Therefore, after observing complex relations among the tables in the SensLog data model, we investigated certain methods for denormalizing those tables towards getting to the MongoDB(Mon, 2018b) database. The main idea is to achieve a state where tables can be transformed into related documents and mapped to MongoDB.
The denormalization methods are discussed in  (Mar, 2018). We applied methods starting with changing the connections to IDs to substitute relations among the MongoDB collections. In the second iteration, we substituted joints by embedding certain tables  (Kanade et al. , 2014). This method should be applied strategically depending on the amount of data to be duplicated. The good indication is the complexity of joint relations that can be observed with Modelio SQL Designer. Once deciding to migrate we used a specific Mongify tool (Mon, 2018a) where we could script these denormalization rules and translate the relational schema to a MongoDB database structure. That way, we could instrument the methods for ID replacement and embedding tables  (Jadhav, 2017). The figure  2  gives a snapshot of tables achieved after migration as seen in the Robo3T.
EVALUATING RESULTS Once both systems, the legacy one and migrated one, were ready, we could proceed with the performance testing and compare both database systems. To achieve this we needed to set up a testbed consisting of two identical cloud machines hosting Post-greSQL and MongoDB. On both machines, we set up the SensLog data models and deployed the historical data. With these two machines we organized the test plans that included the following scenarios: • Throughput on insertion query. • Response Time on insertion query. • Throughput and Response Time with varying number of users varying from 10 to 30 as discussed with the SensLog team. • Throughput and Response Time with specific queries. • Throughput and Response Time with specific schema set. Considering the limits of infrastructure, we concentrated our experiments with micro cloud machines.
Performance Testing The graphs below illustrate the results achieved with JMeter(Jme, 2018) performance testing tool. Those graphs compare the behavior of both PostgreSQLbased and MongoDB-base SensLog applications. In this first set of experiments, we have concentrated on insertion testing  (DataStax, 2018). We stressed both the machines by a test case involving a large number of users (i. e. sensors) to insert data each second in order to determine the maximum number of users to be served with a stable Response Time. We chose the most critical part of the data model in order to demonstrate the impact of the insertions on the system. In our context, the SensLog application heavily relies on the Observation table where all measurements are stored. Hence, we concentrated insertion queries on this table.
SensLog team targets that application is capable to handle 30 sensor data insertions per second in a typical scenario. Thus, in the first part of our experiments, we measured the Response Time in this typical scenario on our testbed. We run test suite on both machines simultaneously and observed the difference. The results are shown in the snapshots below. We observe in the figure  3  that it is going towards exponential curve whereas in figure  4  we get linear performance stability in the NoSQL MongoDB insertions. In terms of throughput during insertion, we observed MongoDB performing three times more than the PostgreSQL machine. In the second iteration, we decided to investigate in more details the MongoDB behavior by fine-tuning different parameters such as the number of users and the ramp-up time.
In the figure  5  we observed a consistent Response Time when MongoDB was stressed with 15 users for ramp-up time of 1 minute. There was a big jump after several insertions which is most likely because of the memory shortage in the small Amazon machine we have utilized in our proof of concept. From this lap, we can also see the behavior of Mon-goDB machine when it is near to getting full. That is a very appreciable behavior of a little jump in a still consistent way comparing to completely breaking up as in the PostgreSQL case. In the next experiment, we did the same for 30 users and observed similar behavior. Thus, with our experiments, we proved our hypothesis on linear consistency when using MongoDB. The data insertions are done with the help of scripts in the JMeter tool  (Jme, 2018)  by generating relevant fake data as below.
CONCLUSION AND FUTURE WORK The major point with the sensor database is the constant flow of data from the sensor networks. This flow if decoupled from the relational-based database, using the technique of data warehousing  (Balazinska et al. , 2007)  can benefit the organization of the insertion in the database with respect to the Response Time. In SensLog database, we also observe the existing Post-greSQL heavily utilizing SELECT queries. MongoDB lacks support in SELECT queries and hence we weigh on preserving a part of the original architecture which will still have a centralized database operating with PostgreSQL. From our tests, we observed SensLog Response Time graph in MongoDB to be growing linear compared to the equivalent PostgreSQL relational database. Therefore, we consider that a data warehouse on MongoDB will be a good solution for SensLog database, while still keeping the centralized database for the reporting system.
If there is any architecture decision to be taken for SensLog application, we would point to deploying MongoDB servers in the front receipting part to get direct data from the Sensors. We can later strategically utilize this database as a data pool for postprocessing. Thus, our experimentations on the basis of proved concept with MongoDB sensor data model, we expect a performance optimization of threefold at the optimistic level in this architecture system. This study is an important step forward for justifying our efforts for further research in the direction of automated tools for model-driven SQL to NoSQL migration with Modelio.