dataesr/scientific-paragraphs-categorization huggingface dataset:
| MODEL DESCRIPTION For the model described here, we consider that K i eDNA water samples, obtained from either spatial or temporal replicates, are collected at each one of a total of I sites for which we want to infer animal density. The eDNA data are denoted w ik for each site i = {1, … I} and each sampling replicate k = {1, … K i }. Data w ik must be a quantitative metric of eDNA concentration (e. g. , ng/l or DNA copies per ml). Ideally, the K i samples for a site i are extracted from spatially replicated water samples pooled together and homogenized. Water replicates would typically be collected at selected locations inside the sampling unit (site; e. g. , pond, wetland, section of river), at a single time, thus ensuring population closure. For diffusion-limited lentic waters where eDNA concentrations are likely to reflect space use of the target species (e.
Parameter r quantifies across-replicates variability and overdispersion in values w ik. The likelihood formulation for this model is as follows: where, p i = r μ i +r , which also referred to as the probability of trial success of the negative binomial process. For the Negative Binomial model, data inputs must be integers. Ideally, eDNA concentration is quantified as a number of DNA copies, which does not require any transformation, as in our carp example. Otherwise, eDNA values must first be transformed as integers, like we did in our salamander example.
| Proof of concept data As a proof of concept, we applied the method on an experimental dataset on common carp, where fish density was known without error. This experiment consisted in a total of I = 11 non-empty mesocosms and K i = 3 eDNA sampling replicates per mesocosm (site), taken at 1-week intervals. Here, temporal replicates were used instead of the "snapshot" spatial replicate design recommended above, but because it was a controlled experiment, variability in environmental conditions was not an issue. For full details on this case study, see  Doi et al. (2015). Model performance was quantified by the root mean squared error and 95% C. I. realized coverage of repeated animal density estimates ( Di ) provided by the model in a cross-validation study. We used an exhaustive leave-p-out cross-validation approach where p represents the number of sites for which animal density was assumed unknown (i.
e. , eDNA-only sites) and thus used for validation. The number J = I-p of dual data sites represents the subset of data used for model calibration. We assessed the model for four different scenarios of model calibration: I-p = {2, 3, 4, 5} dual data sites (I = 11; hence p = {9, 8, 7, 6}). Cross-validations were exhaustive: For each scenario, the learning or testing process was done for all possible ways C p I of subdividing the original dataset. For instance, for I = 11, with p = 8, a total of C p I = 165 validation repetitions were performed. For each individual repetition, we proceeded as follows: (1) J sites, for which we kept both eDNA and density data, were used as dual data sites, and p sites, for which only eDNA data were kept, were used as eDNA-only sites for the analysis; (2) model outputs provided density estimates Di,i≠j for a total of p = I-J eDNA-only sites; (3) the p estimates Di≠j obtained as model's outputs were compared to the "known" density values that were left out of the analysis.
| Field dataset To evaluate the potential applicability of the method in natural settings, we applied the model to data obtained from a stream salamander survey performed in summer 2011 in the South Fork Salmon River Sub-basin, Idaho, USA  (Pilliod et al. , 2013). During this field study, eDNA was collected and quantified at the downstream edge of a survey reach just prior to salamander sampling in five streams (see  Pilliod et al. , 2013 for details). Idaho giant salamanders were surveyed using single-pass electrofishing in a 500-m stream reach (defined as a site) of each stream and salamander density was then estimated as the number individuals captured divided by the area searched per stream. The 500-m reach was randomly selected and predefined as the spatial sampling unit or site. While the rate of eDNA loss per distance of stream is unknown, it has been estimated to occur on the scale of 100-200 m  (Wilcox et al.
5°C; other sites: x = 9. 7°C), and depth (sites used: x = 25. 2 cm; other sites: x = 21. 4 cm). Only the stream wet width differed significantly (sites used: x= 491. 8 cm; other sites: x= 332. 9 cm). Nonpaedomorphic adult salamanders, which are terrestrial, were never captured during electrofishing surveys, and thus were assumed to contribute little to no eDNA to the water. The whole dataset consisted of I = 5 sites (i. e. , a 500-m upstream stretch in five different streams) and the number of per-site water sampling replicates (K i ) varied from 3 to 12. For two sites K i = 3, for two others K i = 9 and for one site K i = 12. The data for the two sites with nine samples actually consisted of three replicated samples from each of three slightly different methods of water collection (GrabFilter, GrabHold, Instream; see  Pilliod et al.
, 2013). We were confident in using all nine samples together because (1) the difference between methods only concerned water collection and storage (not eDNA processing and analysis), and (2) previous investigation had shown no evidence to suggest differences in the amount of eDNA captured by each method for these samples  (Pilliod et al. , 2013 (Pilliod et al. , , pp 1126)). To assess models' performance on this dataset, we used the same cross-validation approach as described above, but with the caveat that salamander densities were not known perfectly. Because densities were estimated from a field sampling technique (electrofishing) that cannot provide perfect detection of individuals, we acknowledge that there is inherent uncertainty in the density data that feed the model, which likely affects the accuracy of our estimator.
| Results Before running analyses with our new modeling approach, we looked at correlations between values of eDNA concentration and animal density, to get a sense of the "quality" of information contained in the dataset. Correlation was high for both the proof of concept (r =. 84, Figure  1a ) and the field datasets (r =. 79, Figure  1b ). We also quantified the degree of overdispersion in eDNA, using average values of per-site variance-to-mean ratio (VMR), with VMR > 1 indicating overdispersion and VMR < 1 indicating underdispersion. This information was especially useful to compare performances of the different models considered. We found a high degree of overdispersion in the carp eDNA data, with the per-site VMR averaging 182. 2. Here, no data transformation was applied because the original eDNA data were quantified as a number of DNA copies per ml, which are already integer values.
| Field dataset Similarly to the proof of concept dataset, we found the Negative Binomial model to perform better than the Normal model (Table  1 , Figure  3 ). Here, only 2 and 3 dual data sites were used to inform the eDNA/salamander density relationship, but it represented 40% and 60% of all the sites (I = 5), respectively. The realized coverage was 1. 00 for both scenarios assessed.
| DISCUSSION We provided a model framework to estimate animal density using eDNA data in combination with known animal density values obtained for a subset of sampled sites. Cross-validation analyses, performed using a dataset for which fish densities were known with certainty, re-   (Furlan et al. , 2016). The main limit of our method, currently, lies in the fact that it (a) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2 dual data sites _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 3 dual _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2 dual data sites _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 3 dual data sites estimates of animal density. We therefore emphasize that it will be important to pursue model developments of this approach in order to (1) add a detection-based process for density estimation in the modeling frame and (2) explicitly incorporate uncertainty of Di values when modeling the eDNA-density relationship (i.
does not explicitly account for uncertainty in animal density values Dj. Uncertainty was not an issue with the carp data because the number of animals in each mesocosm was known. These data were ideal to assess model performance as proof of concept scenario, but they may not reflect field levels of eDNA in the environment. Using survey estimates of animal density (from the traditional surveys) as data points, without consideration of the associated uncertainty, can create biases and erroneously inflate our confidence in the eDNA-basedF I G U R E 2Results of the cross-validation study from the common carp dataset (mesocosm experiment). Each point represents the density estimate obtained from (a) the Normal model and (b) the Negative Binomial model, for cases where fish density was known for 2 (black), 3 (red), 4 (green), or 5 (blue) sites (i.
e. , dual data sites). The horizontal black dashes represent the known values of animal density Results of the cross-validation study from the Idaho giant salamander dataset (from electrofishing sampling of wild populations). Each point represents the density estimate obtained from the (a) Normal and (b) Negative Binomial models, for cases where salamander density was "known" for 2 (black) or 3 (red) sites (i. e. , dual data sites). The associated error bars, representing the 95% C. I for each individual estimate, are also showed here. The horizontal black dashes represent the relative values of animal density obtained from field data. For the Normal model, the higher limit of the 95% C. I. is not visible for a few scenarios because it out of scale (upper limits between67. 3 and 6,778). This is of course a clear indication of the poor precision of this model.
ACKNOWLEDGMENTS Funding for this research was provided by the  U. S. Geological Survey-Amphibian Research and Monitoring Initiative (ARMI) , as well as the  Environment Research and Technology Development Fund  ( 4-1602 ) of the  Ministry of the Environment, Japan  and the  Core Research for Evolutional Science and Technology (CREST)  program from the  Japan Science and Technology Agency. This is ARMI contribution number  608. Nathan Chelgren  and several anonymous reviewers provided helpful comments on this manuscript. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the  U. S. Government.
Can data sharing in rodent phenotyping help with replicability? Laboratory mice and rats are the main mammalian models currently used for high throughput genomic and behavior genetic research, and are employed primarily to explore and test gene function. This is considered by some to be "the great challenge facing biologists today"  (Collins et al. , 2007). Rodent models are used extensively as part of preclinical development and testing of treatments for disease in humans, in genomic research  (Collins et al. , 2007; Beckers et al. , 2009) , and also in research of the central nervous system (CNS) and behavior (e. g. ,  Crawley, 1985; Gerlai et al. (1995) ;  Logue et al. , 1997; Dulawa et al. , 1997; Gerlai, 2002a; Gerlai et al. , 2002; Musatov et al. , 2006). For obvious reasons, the reproducibility (within the same study) and replicability (across studies) of phenotyping has crucial implications for their translational relevance.
Bioinformatics is a well-established discipline in the life sciences, traditionally concerned primarily with DNA, RNA and protein sequence data, which are stored in public databases as a primary research tool. The idea that phenotypic data are also worthy of storing, analyzing and reanalyzing  (Gerlai, 2002a)  is not so widely established yet, but the value of phenotype data integration has been recognized methodologically and in practice  (Chesler et al. , 2003) , and phenotype data standards emerged early  (Grubb et al. , 2004). Collaboration, community efforts, data sharing, and public phenotyping databases have an important role in today's field of rodent phenotyping. Among many other utilities, they also offer unique opportunities for researching and controlling reproducibility and replicability.
These public databases and data sharing projects are instructive in informing replicability studies at different levels: from reanalyzing other researchers' data to contributing their own data, and even constructing and maintaining public databases and community projects. Reanalysis of shared phenotyping data enhances their utility and scientific value, potentially substituting for additional animal studies, thus reducing animal use without compromising actual reproducibility and replicability. This section reflects experience with projects collecting phenotyping data across several laboratories, in some cases over long time periods, frequently through collaboration with researchers from other institutes and disciplines, and frequently contributing phenotyping data to public databases and services, and/or to meta-analysis and reanalysis by other researchers  (Crabbe et al.
, 1999; Chesler et al. , 2002a,b; Collaborative Cross Consortium, 2004; Wolfer et al. , 2004; Kafkafi et al. , 2005; Wahlsten et al. , 2006; Mouse Phenotype Database Integration Consortium, 2007; Mandillo et al. , 2008; Morgan et al. , 2009; Beckers et al. , 2009; Baker et al. , 2011; Richter et al. , 2011; Collaborative Cross Consortium, 2012; Bogue et al. , 2014; Grubb et al. , 2014; Heller et al. , 2014; Karp et al. , 2014; Koscielny et al. , 2014; Maggi et al. , 2014; de Angelis et al. , 2015; Bogue et al. , 2016; Karp et al. , 2017; Kafkafi et al. , 2017). The projects described in the rest of this section will be used to address multiple issues of replicability and reproducibility in the following sections. The Mouse Phenome Database (MPD), a data resource that emerged from a research effort at The Jackson Laboratory, stores primarily individual (per subject, as opposed to just group means and standard deviations) phenotype values, along with in-depth phenotyping protocol information, as contributed by researchers from all over the world  (Maddatu et al.
, 2012; Grubb et al. , 2014; Bogue et al. , 2015 Bogue et al. , , 2016)). It allows for trait correlation and examination of trait stability across strains, data sharing, dissemination and integration, facilitating the discovery of convergent evidence. At the time of writing the MPD contains several hundred measures of widely studied behaviors collected in multiple laboratories in inbred strains and now also includes per subject data from genetic mapping studies in the QTL Archive. Several among the meeting participants contributed their results to the MPD, and data from the MPD were used for several studies presented in the meeting. The GeneWeaver. org database  (Baker et al. , 2011)  employs curated user-submitted and published gene sets from GWAS, QTL mapping, genome-wide gene expression analysis, text mining, gene co-expression, expert lists, curated annotations, and many other data sources drawn from major public data resources.
It included at the time of the meeting ~80,000 gene sets from 9 species including humans rats, zebrafish, drosophila and mice. GeneWeaver applies several algorithms to analyze the convergent evidence for relations among these sets of genes and behaviors or other biological constructs derived from many independent experimental studies. e. g. , for those implicated in alcohol preference and withdrawal  (Bubier et al. , 2014; see Section 9). GeneNetwork is a database that enables searching for ~4000 phenotypes from multiple studies in the BXD, HXB, and in other recombinant inbred rodent families, as well as in other model organisms and even humans  (Mulligan et al. , 2017). GeneNetwork employed a somewhat different strategy than MPD in that it did not rely solely on researchers submitting their data.
Instead the database operators extracted the data from the scientific literature and integrated them into a uniform format  (Chesler et al. , 2003). This strategy required a considerable effort, but also expanded the range of studies and possible forms of analysis. In many cases, however, per subject phenotype data were not available. GeneNetwork uses both routine and advanced statistical methods to extract, explore, and test relations among phenotypes and underlying genetic variation. It enables complex queries in real time, including very fast QTL mapping. Similar to MPD, GeneNetwork can also be used to correlate any phenotype with all other phenotypes in the database across strain means, within or between studies, enabling the exploration of the replicability of phenotypes, even before relating them to the genotype.
Any new phenotype can be correlated with any previously documented phenotypes across multiple strains. The increasing number of possible combinations grows exponentially with the rate of the added data. In the future, these two data resources, the per strain phenotype data storage with thorough protocol documentation in MPD, the Rat Genome Database, and genetic analysis suite in GeneNetwork. org will be more closely integrated  (Mulligan et al. , 2017). The public database of the International Mouse Phenotyping Consortium (IMPC) is intended to be "the first truly comprehensive functional catalogue of a mammalian genome"  (Morgan et al. , 2009; Koscielny et al. , 2014). The IMPC is a community effort to knock out ~20,000 genes and generate ~20,000 mutant mouse lines over the next 10 years, phenotype them using comprehensive and standardized high-throughput assays, and make them freely available to researchers over the world as animal models  (De Angelis et al.
, 2015). Minor protocol differences exist in some cases, the composition of the battery varies across centers, and of course, a litany of laboratory-related factors (housing, husbandry, experimenter, room dimensions, ambient noise, caging styles, etc. ) differ across centers. Data from the IMPC database are currently being used for several studies of replicability  (de Angelis et al. , 2015; Karp et al. , 2015; Kafkafi et al. , 2017). A large data set used to analyze replicability across laboratories  (Kafkafi et al. , 2017)  was first presented in the meeting, consisting of data from multiple databases and multi lab studies contributed by several researchers, including  Wolfer et al. (2004) ,  Richter et al. (2011) ,  Wahlsten and Crabbe (2003, downloaded  from the MPD) and knockout data downloaded from the IMPC database  (Morgan et al.
, 2009; Koscielny et al. , 2014). This dataset records per subject results of individual animals, amounting to one of the most extensive reanalysis of multi-lab studies, enabling estimation of the typical replicability in the field (see Section 4), as well as demonstrating the random lab model (Section 6) and GxL-adjustment (Section 8) advocated for estimating replicability. GxL-adjustment explicitly relies on systematic data sharing as a proposed strategy for addressing replicability across laboratories in rodent phenotyping.
Replicability issues in mouse phenotyping -how serious are they, really? This seemingly simple empirical question is not simple to answer, for several reasons: there is no consensus over the correct ways to analyze and estimate replicability (Open Science Collaboration, 2015;  Gilbert et al. , 2016 , see also  Section 6) , and only a few attempts have been made at systematic analysis across several studies and/or laboratories with the objective of estimating replicability in a quantitative way (see also  Sections 3 and 8). Here, using careful reanalysis and meta-analysis of data from the multi-lab studies and public phenotyping databases detailed in the previous section, we give a general assessment. Most of the participants in the meeting seemed to agree that there are real and serious problem of reproducibility and replicability in mouse phenotyping, but also that some specific phenotyping results are highly replicable, especially when the genotype effect size is large.
Several speakers in the meeting presented such plots comparing laboratories and studies, and both the MPD and the GeneNetwork software (see Section 3) generate them by request, and even run a fast search in their database for phenotypes that correlate with any given phenotype across strains  (Mulligan et al. , 2017). Such plots frequently indicate considerable correlation between strain means across studies, indicating some replicability, although there is no clear criterion for how much correlation indicates sufficient replicability. The heterogenization experiment of  Richter et al. (2011,  see Section 8 for more detail) was orchestrated across six laboratories, more than in any other multi-lab experiment in the field of rodent phenotyping. It concluded that these laboratories, while still much fewer than all potential phenotyping laboratories over the world, already contribute a large component of variation, apparently considerably larger than the variation introduced by systematic heterogenization of two factors (test age and cage enrichment).
This study therefore concluded that "differences between labs are considerable and unavoidable". There are many potential confounders in studying genetically modified mice that are difficult to control  (Schellinck et al. , 2010)  and they are likely to differ across laboratories and studies. Studies utilizing phenotyping data from several knockout lines and associated controls across research centers of the IMPC were presented in the meeting. These studies found that test day at each phenotyping institute was a considerable source of variation and encompassed multiple variance sources (e. g. human operator, litter, cage, reagents etc. , see also  Karp et al. , 2014 , de Angelis et al. , 2015). Spreading testing across time functions as a form of heterogenization. It is not clear yet to what extent a multi-batch workflow  (Karp et al.
, 2014)  captures the interaction of genotype with the laboratory, which is a different effect. In a recent large dataset comprised of multiple previous studies, each including several genotypes measured across several laboratories  (Kafkafi et al. , 2017 , see Section 3), cases were demonstrated that may be termed "opposite significant", i. e. , there is a crossover interaction with genotype and laboratory such that one genotype produces significantly higher mean of an outcome measure in one laboratory while significantly lower in another laboratory (see Fig. 1  right for a conceptual illustration, assuming environments E1 and E3 represent two different laboratories). In other words, these laboratories would have reported opposite discoveries. Opposite significant cases are not rare: examples were found in most of the multi-lab datasets in the study, although as expected they are more common in datasets that include a larger number of laboratories.
However, in most multi-lab datasets (specifically all 8 but one) the majority of genotype effects were replicable when using the random lab model criterion for a replicable genotype effect (see  Section 6). In these same datasets, the proportion of "non-replicable positives", i. e. , genotype differences that were found significant within a single laboratory (using the typical t-test at the level of α = 0. 05) but did not replicate across all laboratories (using the random lab model) ranged between 19% and 41%  (Kafkafi et al. , 2017). This result can be regarded as an estimation of the proportion of non-replicable "discoveries" in single-lab studies in the field. It could be argued that the true value is higher, since the general standardization level in the field is probably lower than the standardization level in the multi-lab studies used to derive the above proportion (but see  Section 7).