dataesr/scientific-paragraphs-categorization huggingface dataset:
Next, we continue to investigate the cost of our HX-MATCH algorithm by pre-computing the cross-matching with a relatively large radius lr. We thus build an intermediary DataFrame M which contains the IDs of the matched records from R and S according to lr along with their distance and store it on HDFS. The dataset schema is represented by the following attributes (sourceID 1 , sourceID 2 , dist) where sourceID 1 and sourceID 2 correspond to source identifier in R and S respectively and dist is the spherical distance between the actual objects identified by sourceID 1 and sourceID 2. Thus, the subsequent cross-matching queries can be replaced by simple selection in M and equi-join queries, which is much more efficient than the initial computation of a distance matching. Precisely, all needed is to filter M on dist ≤ where is the requested radius in the cross-matching query, and to retrieve other attributes from R and/or S to make a join with R and/or S on respective sourceIDs.
This can be expressed as follows: R R. sourceId1=M. sourceId1 σ dist< (M ) M. sourceId2=S. sourceId2 S To study the impact of this pre-computing technique on our algorithm, we matched the two biggest catalogs GAIA DR1 and IGSL with a radius of 7 arc-seconds. We first used HX-MATCH to build the intermediary dataset M. This tooks 3 hours, including the materialization of M on HDFS. Although this seems high compared to a one shot cross-matching with a smaller radius, it will be amortized when it comes to repeated queries with various distance criteria, as long as this distance is lower than the one used to build M. Figure  16  shows the overall speedups of this optimization. The comparison between the two options shows that using the intermediary dataset, the cross-matching computation achieves 4x speedups.
kNN Query Figure  17  shows the performance of the kNN query on GAIA datasets in ASTROIDE, Spark SQL and SIMBA. We select a random point from the input dataset, fix k to 10 and measure the execution time of the query. In Figure  17 , we  study the effect of increasing the data size from 50 millions to 1. 2 billions. ASTROIDE outperforms Spark SQL since Spark SQL requires scanning the whole dataset to execute a kNN query. ASTROIDE achieves better performance than SIMBA, because it scans less partitions. In general, one or two partitions are sufficient to cover the kNN result. In addition, we study the effect of increasing k, we varied k from 10 to 100 and fixed the data size to about 50 millions of objects. Figure  18  shows that the performance of ASTROIDE, SIMBA and Spark SQL are not affected by k.
Conclusion In this paper, we presented ASTROIDE, a solution that combines the scalability and cost-effectiveness of distributed data processing engines with Spark and the expressiveness of astronomical data servers with ADQL. ASTROIDE achieves this objective through efficient partitioning using HEALPix, customized strategies in the Catalyst optimizer and an expressive query interface. ASTROIDE extends Spark to support main astronomical queries. Experiments on real datasets from the on-going spatial mission GAIA demonstrate that ASTROIDE is obviously much faster than Spark SQL. It also outperforms SIMBA which is not specialized in astronomical data thanks to the use of an adapted HEALPix partitioning algorithm. We plan improve our query optimizer by adding optimization rules and strategies to execute queries.
Declination of s d = d 1 -d 2 = Difference in declination between r and s a = Difference in right ascension between r and s Cone Search is one of the most frequent queries in the astronomical domain. It returns a set of stars whose positions lie within a circular region of the sky. Given a dataset R and a circle c defined by a sky position p and a radius sr around that position, a cone search query returns all pairs of points r ∈ R within c. Formally 2 : Cone-Search(R, c) = {r | r ∈ R, SD(r, c. p) ≤ c. sr}.
Acknowledgments The present paper has been partially supported by  CNES  ( Centre National d'Etudes Spatiales ) and by the  MASTER  project that has received funding from the  European Union 's  Horizon 2020 research and innovation programme  under the  Marie-Slodowska Curie  grant agreement N. 777695. We would like to thank  Veronique Valette  from the  CNES  and  Frederic Arenou at Paris Observatory  for their cooperation. We are also thankful to  François-Xavier Pineau  and colleagues from the  CDS (Centre de Données astronomiques de Strasbourg)  for their help regarding the ADQL parser. This work has made use of data from the  European Space Agency (ESA)  mission GAIA (https://www. cosmos. esa. int/gaia), processed by the GAIA Data Processing and Analysis Consortium (DPAC, https://www.
Declination of r d 2 = 2 Declination of s d = d 1 -d 2 = Difference in declination between r and s a = Difference in right ascension between r and s Cone Search is one of the most frequent queries in the astronomical domain. It returns a set of stars whose positions lie within a circular region of the sky. Given a dataset R and a circle c defined by a sky position p and a radius sr around that position, a cone search query returns all pairs of points r ∈ R within c. Formally 2 : Cone-Search(R, c) = {r | r ∈ R, SD(r, c. p) ≤ c. sr}.
For instance, the following SQL query finds all rows with positions within 0. 0833 degrees of ra=266 and dec=-29 in a table called gaia: SELECT * FROM gaia WHERE (2 * ASIN(SQRT(SIN((dec-(-29)/2) * SIN((dec-(-29))/2) + COS(-29) * COS(-29) * SIN((ra-266)/2) * SIN((ra -266)/2))) <= 0. 0833) 5. 1 Cone Search QueryA cone search query takes a dataset R, a point p and a radius sr as inputs and returns the records in R that overlap the cone defined by p and sr. Spark SQL has to scan the entire input table to execute a cone search query Without any accelerating data structure (e. g. index).
Subcommittee. A library with a 600-bp insert-size was constructed by applying the 10X Genomic Chromium system. The Chromium Controller instrument (10X Genomics Inc. ) was used for barcoding long stretches of DNA in Genome Gel Beads. Subsequently, these DNA fragments were reduced to smaller sizes for short read sequencing on an Illumina Hiseq platform (Illumina), generating 2 × 150 paired-end sequences. Sequencing was performed by Novagene. After removing low-quality reads, 285. 67 Gb clean sequences were retained for further genome analyses. Quality control was undertaken using the trimmomatic program with default parameters (java -jar trimmomatic-0. 36. jar PE ILLUMINACLIP:TruSeq3-PE. fa:2:30:10 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15 MINLEN:50). supernova software (version 1. 2. 2) was applied to construct the scaffolds using the barcode and reads data generated from 10X Genomic libraries.
| Genome-wide genetic diversity We used bowtie  (Langmead & Salzberg, 2012)  version 2. 2. 5 to map the retained sequence reads of both species to their reference genomes. We used samtools  (Harris, 2007; Li, 2011)  version 1. 3. 3 to filter out reads with a mapping quality below 20 and applied the command "grep -v 'XS:i'" to filter out reads which mapped to multiple locations, retaining 676,985,798 C. pygargus and 522,854,805 C. capreolus reads. We subsequently called single nucleotide polymorphisms (SNPs) using samtools, bcftools-1. 9  (Narasimhan et al. , 2016)  and vcftools-0. 1. 13 (Danecek et al. , 2011) , and used linux command line tools to count on a per-contig basis the number of heterozygous sites, the total number of sites with genotype information and the spacing between adjacent heterozygous sites.
We also used the default settings of -t15 and -r5. For C. capreolus we set the minimum read depth (d) to 7 and the maximum read depth (D) to 42; for C. pygargus we set d to 13 and D to 78. We set the generation time parameter to 4-6 years  (Nilsen et al. , 2009). We assumed a mammalian mutation rate per site per year  (Kumar & Subramanian, 2002)  of 0. 22 × 10 -8 and 1. 1 × 10 -8 per generation assuming a generation time of 5 years and a linear relationship. Although we cannot know the correct mutation rate precisely, this estimate is consistent with a broader survey of rates  (Lynch, 2010) , and we illustrate the effect of varying the mutation rate (Figure  S4 ). Bootstrapping was undertaken with 100 replicates. For comparison we generated a PSMC analysis of red deer (Cervus elaphus) using GenBank data for the genome published by  Bana et al.
02. 00. We used a sliding window approach to screen the C. capreolus and C. pygargus genomes for regions with above average spacing between heterozygous sites (i. e. , genomic regions with depleted genetic variation), using the rollapply function of the R-package "zoo" version 1. 8. 6  (Zeileis & Grothendieck, 2005). We set both stepsize and windowsize to 100 datapoints (i. e. , 100 adjacent ROHs). A window was marked as an outlier window based on two criteria: (a) the window should contain at least one ROH in the top 0. 05% (= 5%/windowsize) of all ROHs; and (b) the window should contain at least n ROHs in the top 5% of all ROHs located on the respective contig, with n averaging 15, the exact number being dependent on the number of windows (nwin) on the contig, as described by the R command: n = qpois((1-0.
2). To further consider divergence time, we used an alternative method whereby the third codon positions of exomes were compared between the two species. From a two-way alignment of C. pygargus versus C. capreolus exomes we selected third codon positions, and subsequently removed all sites which had missing data in one or both species. We counted the total length of the retained alignment, as well the number of sites which differed between the species. All steps were performed using linux command line tools (i. e. , grep, awk, tr and sed).
| Capreolus pygargus genome annotation First, we used ltr _ finder  (Xu & Wang, 2007 ; RRID:SCR_015247) and repeatmodeller  (Smit & Hubley, 2018; version 1. 0. 4 ; repeatmodeler, RRID:SCR_015027) to find repeats. repeatmasker  (Tarailo-Graovac & Chen, 2009 ; version 4. 0. 5, with parameters -nolow -no_is -norna -parallel 1) was then used to search for known and novel transposable elements (TEs) by mapping against the de novo repeat library and Repbase TE library (version 16. 02). Subsequently, tandem repeats were annotated using tandem repeat finder  (version 4. 07b; 53;  with parameters 2 7 7 80 10 50 2000 -d -h). In addition, we used repeatproteinmask software  (Tarailo-Graovac & Chen, 2009 ; with parameters -no LowSimple -p value. 0001) to identify TE-relevant proteins. In total, the repeat sequences cover about 1.
genwise (version wise 2. 2. 0) was then used to annotate structures of the genes. To aid in the subsequent annotation of the genome, we used transcript sequences (kindly provided by Dr Wang from  Wang et al. , 2019)  to predict gene structure and update the evidencemodeler (evm) consensus predictions, adding untranslated region annotations and models for alternatively spliced isoforms by using pasa software  (Haas, 2003; version 2. 1. 0; RRID:SCR_014656). Thereafter, we used evm software  (Haas et al. , 2003; version 1. 1. 1 ; evidencemodeler, RRID:SCR_014659) to merge the de novo and homology gene sets and form a comprehensive, nonredundant gene set, resulting in 21,777 protein-coding genes. To infer gene functions, we searched the GO, KEGG, TrEMBL and SwissProt databases for best matches to the protein sequences from the previous step by using blastp (version 2.
| dN/dS tests We blasted the exons of the 21,777 C. pygargus genes (excluding untranslated regions) to the C. capreolus genome as well as to Bos taurus  (Elsik, 2009; GCA_002263795. 2) and other published cervid genomes, namely: Rangifer tarandus  (Li et al. , 2017; PRJNA391754)   davidianus  (Zhang et al. , 2018; GCA_002443075. 1) , Cervus elaphus  (Bana et al. , 2018; GCA_002197005. 1) , Cervus albirostris  (Chen et al. , 2019; GCA_006408465. 1) , Hydropotes inermus  (Chen et al. , 2019;  GCA_006459105. 1), Odocoileus hemonius (mule deer), Muntiacus muntjak  (Chen et al. , 2019; GCA_006409035. 1) , Muntiacus crinifrons  (Chen et al. , 2019; GCA_006408485. 1) , Hyelaphus porcinus  (Chen et al. , 2019)  and Muntiacus reevesi  (Chen et al. , 2019; GCA_006408525. 1). We used bedtools  (Quinlan & Hall, 2010)  version 2.
19. 1 getfastafrombed to extract the blast hits from the reference genomes, and subsequently muscle  (Edgar, 2004)  version 3. 8. 31 for multiple alignments. For each gene we selected the hits with the highest bitscores, giving preference to exons residing on the same contig or chromosome. The credibility of the gene alignments was verified by visual inspection of a subsample of the genes, as well as by using the concatenated alignments to generate a maximum likelihood species tree with 100 bootstrap replicates using the software raxml  (Stamatakis, 2014)  with Bos taurus as the outgroup  (Elsik et al. , 2009;  GCA_002263795. 2). Genes were partitioned for first and second codon positions versus third codon positions. (The actual raxml command: raxml -fa -m GTRGAMMA -p 12345 -o B_tau -x 12345 -# 100 -s input.
Evidence for selection at a "candidate" locus (locus with a relevant putative function based on earlier studies; see Results), pard3 (see  Boroviak et al. , 2015 ; Table  S2 ), was further assessed based on amino acid properties using the program treesaap  (Woolley, Johnson, Smith, Crandall, & McClellan, 2003). We included all test properties available in the program in the initial run and tested screening window sizes of 20 and 100 codons. Alignments for illustration were generated in geneious, which was also used to generate a neighbour joining phylogeny based on 100 bootstrap replicates and the Tamura-Nei model for locus ZDHHC23 (g09868).
| Gene ontology analysis We searched for gene ontology (GO) terms using fatigo (Al-Shahrour, Diaz-Uriarte, & Dopazo, 2004) on the Babelomics 5 platform (http:// babel omics. bioin fo. cipf. es/). GO term functions were identified with reference to the human database (chosen due to the relatively complete level of annotation and functional analysis, and the lack of a suitable reference from a closely related species). A Fisher exact test for 2 × 2 contingency tables (testing for over-representation in list 1) was used to check for significance. Significance was corrected for multiple testing using the Benjamini and Hochberg's false discovery rate (FDR)-controlling procedure. GO biological processes, molecular function, cellular component, GOSlim GOA, Interpro and Genome-scale metabolic network databases were searched, filtering terms by five to 500 annotated IDs in each database.
| ddRAD sample selection We generated SNP data sets for C. capreolus populations from four localities: Ayrshire (Scotland), East Anglia (England), Wurttemberg (Germany) and Aurignac (France), sampled over geographical areas of comparable size. The East Anglian roe deer are the descendants of an anthropogenic reintroduction of 12 individuals stocked from the Wurttemberg population in 1884, found to have just one mtDNA control region haplotype (744 bp) among 40 sequenced individuals  (Baker & Hoelzel, 2013). The Ayrshire population represents the endemic UK population that became isolated from the roe deer population in mainland Europe around 6000-7000 years ago with the flooding of Doggerland (a land bridge that connected Britain to the continent when the Scandinavian and British ice sheets reached their maximum extent).
05. The actual size selection was executed with a Sage Science PippinPrep machine. The Phusion High-Fidelity kit was used for a 13-cycle PCR (denaturation step: 62°C for 20 s; annealing step: 72°C for 45 s; extension step: 72°C for 5 min). The Aurignac data set was generated independently for a separate study  (Gervais et al. , 2019)  and incorporated opportunistically. It was also produced using the ddRAD strategy  (Peterson et al. , 2012) , and with the same frequent cutting enzyme (MspI) but with a different 6-bp cutting enzyme (EcoR1) with size-selected libraries of 270-330 bp. All libraries were paired-end sequenced for 125 bp on an Illumina HiSeq_2500 (version 4 chemistry) machine.
| SNP calling and filtering The two sequencing lanes for the East Anglia, Ayrshire and Wurttemberg data sets combined producing 602. 6 million reads, which were trimmed to 110 bp and demultiplexed using the software stacks-1. 35 (Catchen, Amores, Hohenlohe, Cresko, & Postlethwait, 2011). Over 6. 1 million reads had to be discarded due to either low quality, an ambiguous radtag, or a missing read mate, resulting in an average number of 7. 3 million read pairs per sample (SD: 5. 8 million, min. : 0. 7 million, max. : 32. 4 million). Paired reads were aligned against both the newly generated C. pygargus genome as well as the Cervus elaphus genome  (Bana et al. , 2018)  using the software bowtie version 2. 2. 5. We chose the red deer genome as a second reference genome because at the time red deer was the species closest to Capreolus spp.
with a genome assembly up to chromosome level. Overall per-sample alignment rates ranged around 98% and 65% for alignment against C. pygargus and Cervus elaphus respectively, with over 65% and under 50% of the reads aligning concordantly to one location within the genome. samtools version 1. 3. 3 was used to filter out reads which aligned to more than one location in the genome, which aligned disconcordantly, and reads with a mapping quality below 20. The Aurignac data set was treated in the same way except that the reads were trimmed to 117 bp rather than 110 bp. Trimming settings differ because quality control of the Aurignac data set was executed at CEFS. SNPs were called using the stacks refmap pipeline with default settings. Loci for which at least 30% of all individuals had a read depth below 8 were removed.
pgdspider 2. 0. 9. 0  (Lischer & Excoffier, 2012)  and plink version 1. 90  (Purcell et al. , 2007)  were used to convert the output from genepop format to a genlight object, implemented in the R package adegenet-2. 1. 1  (Jombart, 2008). All samples with more than 25% missing data were removed, retaining 107 samples (i. e. , Ayrshire: 25, Aurignac: 29, East Anglia: 23, Wurttemberg: 30). SNPs with more than 10% missing data were filtered out, as well as SNPs with one copy of the minor allele and SNPs which belonged to the 1% class of SNPs with the highest read depths. We also filtered out a small number of SNPs that mapped to the same location of the reference genome even though they belonged to different stacks loci, and which therefore probably represent unexpected behaviour of the refmap pipeline.
Analyses for heat map dissimilarity, site frequency spectra plots, and admixture analyses were executed in R, using either in-house-built functions or functions implemented in the adegenet-2. 1. 1, ape-5. 3 (Jombart & Ahmed, 2011; Paradis, Claude, & Strimmer, 2004) , stampp-1. 5. 1  (Pembleton, Cogan, & Forster, 2013 ) and lea-2. 8. 0  (Frichot & Francois, 2015)  packages. Pairwise population F ST measures were calculated using an in-house built script following  Weir & Cockerham (1984). For DAPC (run in adegenet) we set the number of PCs to 1/3 the number of individuals, the number of clusters equal to the number of populations, and the number of discriminant functions set to 3. For population structure estimation (run in lea) we set K (putative number of populations) to 2-6, alpha to 10 and tolerance to 0.
| Stairway plots The demographic histories of our populations were inferred using the stairway _ plot _ v2 software  (Liu & Fu, 2015). We chose this method due to its frequent use and demonstrated utility for RADseq data and for considering time frames relevant to our study. Similar to our PSMC analyses, we set the mutation rate to 1. 1 × 10 -8 and the generation time to 5 years. The population-specific folded site frequency spectrum (SFS) vectors were generated with a custom-built script which binned SNPs in classes based on their number of copies of the minor allele, and which subsequently calculated the size of each bin (i. e. , number of SNPs within each bin).
Risk assessment for mixtures of chemicals requires to investigate the magnitude of their potential adverse effects on living organisms. This is usually done by assessing how experimental toxicological mixture data depart from the model of Loewe additivity. Several recent scientific studies propose to perform this task using an ad hoc method known as model deviation ratio (MDR) method. Moreover, the first official European regulatory document for the study of combined exposures explicitly recommends the use of the MDR method (EFSA Scientific Committee et al. Guidance on harmonised methodologies for human health, animal health and ecological risk assessment of combined exposure to multiple chemicals. EFSA Journal, 2019). We show here that the MDR method is not rooted in statistical principles and can lead to erroneous claims.
1 ) assuming that Loewe additivity holds. The doses investigated were 0, 0. 4, 0. 8, 1. 5, 3, 6, 12 and 20. The ratio on the ray was 0. 7 (proportion of first compound in mixture). At each dose or combination of doses, a number n of replicates was simulated. We carried out simulations for values of n equal to 10, 36, 100 and 500. The R code of this simulation experiment is made available as supplementary material for details and full reproducibility. The main aspects of our simulations are illustrated in Fig. 2  and in Table  1  and can be summarized as follows: • The statistical distribution of the MDR statistic depends strongly on the number n of replicates, with small n data sets displaying larger variability of the MDR statistics, cf Fig. 2  where the variance and the maximum of the simulated sample decreases with n.
The question is then to evaluate the significance of this value. As  x y n z ( , , , ) i i i i i p 1, , for k in 1, …,K do Simulate response values z (k) = (z 1 , …, z p ) (k) under design = … x y n ( , , ) end for i i i i p 1, Compute empirical quantiles q α/2 and q 1-α/2 of … (MDR ˆ, , MDR ˆ) K (1) ( ) Reject hypothesis of Loewe additivity if q q MDR ˆ( , ) /2 1 /2 In the above, α is a pre-defined level of type I error, K is the number pseudo-datasets simulated. It should be large enough to estimate correctly the quantiles q α/2 and q 1-α/2. In our experience, any number of the order of 1000 provided accurate results. A p-value can also be obtained as the proportion of pseudo-datasets such that > MDR ˆMDR k ( ). The method above explicitly takes into account the sample size and the uncertainty in the estimation of EC50's.
The decision provided by the predictive model is based on a simple, yet powerful score computed by adding, subtracting, or dividing cumulative abundance of microbiome measurements. Results: Tested on >100 datasets, we demonstrate that predomics models are simple and highly interpretable. Even with such simplicity, they are at least as accurate as state-of-the-art methods. The family of best models, discovered during the learning process, offers the ability to distil biological information and to decipher the predictability signatures of the studied condition. In a proof-of-concept experiment, we successfully predicted body corpulence and metabolic improvement after bariatric surgery using pre-surgery microbiome data. Conclusions: Predomics is a new algorithm that helps in providing reliable and trustworthy diagnostic decisions in the microbiome field.