dataesr/scientific-paragraphs-categorization huggingface dataset:
https://doi. org/10. 1371/journal. pcbi. 1008957. g003 considered individually, and the individual PPV are averaged) of only about 0. 6, while almost all large MSA lead to initially accurate contact predictions. The figures also contain a contact prediction for randomized data from Null model II. In the upper panels we observe a very weak contact signal; it results from the fact that conserved sites tend to lead to larger phylogeny-induced spurious couplings, but they also tend to be concentrated in proteins, e. g. in active sites or the protein core, and consequently to have a higher contact fraction. The histograms in Figs  3  and 4  are derived from individual samples of the Null models. One might expect that they change from sample to sample. While this is the case for individual couplings, the histograms remain remarkably unchanged when comparing samples, cf.
e. we can conclude that the first predicted contacts are influenced only to a very limited degree by phylogenetic couplings. However, it is also striking that, across the studies protein families, the phylogeny-caused couplings in Null models II and III almost reach the DCA-score threshold found before for accurate contact prediction. This suggests that the suppression of phylogenetic biases in the data (or their better consideration in model inference), may shift this threshold down and therefore allow for predicting much more contacts. The potential gain would be limited due to the finite depths of the MSA, whose effects are assessed by Null model I. We can therefore quantitatively assess the potential in removing phylogenetic effects by the following hypothetical DCA output: on all contacts in our dataset DS2 and DS3 we use the standard plmDCA scores derived from natural data, and on all non-contacts we remove phylogenetic effects by using couplings derived by running plmDCA on samples of Null model I.
The resulting hypothetical PPV curves are given in Fig  5  as purple lines: they are substantially higher than the real PPV obtained from the original data. In the case of the large MSA in DS2, we find a broad plateau of almost perfect PPV close to one, starting to drop only after about 50 top residue pairs, but staying above 90% even after 100 pairs, as compared to about 70% for the real DCA predictions. Even in the small-to-medium-depth MSA of DS3 the potential effect of removing phylogenetic effects is considerable, even if the finite-sample effect is much more pronounced. While in the real data only less than 60% of the considered protein start with a true-positive prediction, the hypothetical phylogeny-removed prediction starts with a PPV above 70%. Since coevolution-based scores are also input to most of the recent deep-learning-based contact predictors, we could imagine that corrections for phylogenetic effects would also improve the accuracy of these methods.
Data Collection We collected our data between March and August 2016 through a combination of interviews and document analysis. Some additional (secondary) data were collected in 2018. Data were collected in four steps, summarized in Table  1. First, we analyzed a variety of documents, including strategic plans, previous studies on the Convener's coffee sector work, its annual plans and reports, and communication materials (e. g. , websites). We also engaged in informal conversations with Convener employees and conducted an in-depth interview with the (then) Program Director to gain in-depth insight into the Convener's work and strategy since its coffee program's inception in 2012. Our second step consisted of a questionnaire, in which the six CCs were asked to provide (1) general information about their platform's geographical scope, date of inception, etc.
These 15 additional stakeholder interviews provided in-depth insights into the differences between the six country contexts in terms of the implementation of the Convener's strategy, as well as sensitizing us to variety in the local context, appreciation, and experience with CSPs in general. We deliberately sought to include a wide variety of perspectives on the Convener's work to complement the Convener's own perspective with that of the 'convened' organizations: In each country, we aimed to include (besides the CC) at least one private sector, and one nonprofit interviewee. All interviews were recorded and transcribed verbatim, resulting in over 400 pages of interview transcript. The final step of data collection took place in 2018 when we collected additional secondary data from the Convener's website and read (consultancy) reports to update us on the developments in its coffee program between 2016 and 2018.
Furthermore, it showed very clearly the variety in implementation (and challenges) across each of the six countries, whilst the strategy of the Convener for each of these countries, and the results achieved, were very similar. We proceeded along this line of discovery when we continued our data analysis, comprising a thorough round of coding using the qualitative data analysis software package MaxQDA. Because of our interest in the role of the Convener in the CSPs, we started by coding the interviews with the Convener's Program Director and the six CCs, as these interviews provided the most information on the Convener's strategy, as well as the relationship between the Convener and the CCs in the developmental phase of the (international) coffee program and the national platforms. The interviews with other stakeholders involved in the platforms, which we coded in the next phase, were instrumental in contextualizing the results found in this first coding round and complemented the Convener and CCs' perspectives on how the mission-driven convener had strategically organized the CSPs.
right panel). We controlled the system and collected data using a custom LabVIEW TM script (LabVIEW 2014 version 14. 0. 1 (32 bit)) through a 16-bit National Instruments A/D Card output channel (National Instruments, Austin, TX), driving an MTS TM Magnetic Resonance Imaging gradient amplifier capable of delivering up to 200 A rms at ± 345 V (MTS Automation, Horsham, PA, USA). We delivered Direct Current (DC) and AC stimulations using a transcranial current stimulation device (StarStim, Neuroelectrics, Spain) driven by the NIC software (Neuroelectrics Instrument Controller, version 1. 4. 1 Rev. 2014-12-01) via Bluetooth. We used a force plate (OR6-7-1000, AMTI, USA) to collect participant's body sway at 1 kHz according to 6 degrees of freedom: forces and moments data each in the 3 dimensions.
D. DATA ANALYSIS The COP time series were filtered with a low pass bidirectional 4 th order Butterworth zero-phase digital filter with a cutoff frequency of 5 Hz. The use of a residual analysis with a customized Matlab program (MatLab version 9. 3 -The MathWorks Inc. , USA) determined the cutoff frequency. We computed sway characteristics using a customized Matlab program. Classically, sway variables are analyzed on anteroposterior (AP) and mediolateral (ML) axes separately, but here we favored planar analyzes over one-dimensional analyses for mainly two reasons listed thereafter. First, balance is best controlled by coordinating the body in space in both dimensions simultaneously  [32]. Second, biomechanical factors bias AP-ML analyzes  [33] ,  [34] , which may be particularly impacted in a protocol sensitizing the vestibular function and involving a heavy customized headset coil exposure system.
Finally, monaural electrical stimulations were used (different from the binaural bipolar montage in  [21] , which induce oblique deviations in the AP and ML plane  [35]. Therefore, we favored planar analyzes over one-dimensional analyses. Electrical stimulations of the vestibular system impact both the quality (sway spatial orientation) and the quantity of movement (sway size)  [29]. Both were therefore considered. To investigate the acute stimulation effects, we measured the sway differences between the 5 seconds period before stimulation onset (PRE-STIM) and the 5 second stimulation period (STIM) for all our analyzes. Spatial orientation was estimated by first conducting a Principal Component Analyses (PCA) on COP datasets to compute PRE-STIM and STIM 95% confidence interval ellipses  [36]  for each trial (Fig.
2  right panels). Then the barycenter was found at the intersection of the major and minor axes of each ellipse. To facilitate the analysis, the mean of the PRE-STIM COP dataset was subtracted from both the PRE-STIM and STIM datasets, centering all the PRE-STIM barycenters on zero. To estimate the spatial direction of sway we found the angle theta (θ) between 0 degrees and STIM barycenters (Fig. 2.
E. STATISTICAL ANALYSIS We performed all linear statistical analyses using R version 3. 3. 2 [40]  and all circular statistics using the CircStat toolbox in Matlab  [41]. A level of significance of α = 0. 05 was adopted throughout data analysis. Differences in all three CTRL conditions were analyzed with a one-way repeated measure analysis of variance (ANOVA). To investigate the effect of wearing the stimulation device (ON vs OFF) as well as the effect of our positive control (DC vs CTRL), we implemented paired t-tests to analyze ρ and speed. Two-way ANOVAs (2 stimulation modalities (AC / MF) × 6 conditions (CTRL plus five frequencies)) for repeated measures were used to test the effect of frequency of the time-varying exposure types on ρ and speed. Rao's spacing test for circular uniformity was used to determine whether θ was distributed uniformly.
Therefore computeraided diagnosis (CAD) systems have become a second opinion reader by implementing appropriate image processing algorithms to improve radiologist's diagnoses  [5]. Radiologists determine the cancer malignancies according to certain features such as shape, margin, orientation and textural echo pattern of cancerous objects  [3]. All of these features can be extracted properly only after the investigated objects are correctly localized. And with the growing abundance of US datasets from continuous screening results, manual localization will be a tiring, tedious and prone to human error. That is why automated object detection techniques are a key factor in the success of CAD systems  [6]. In addition, the enrichment of this capability will make the latest CAD technology survive and compete in the era of industrial revolution 4.
The calculation results of five above metrics are presented in table 2. It can be observed that MoRbAC object detection coincides mostly with its ground truth. Worth noting that decisive steps in the CAD procedure are segmentation and feature extraction. Failure in these stages exacerbates clinical recommendations produced by CAD. By localizing each of suspected cancerous objects, these two CAD procedures will be easier to do with accurate result even on an abundant dataset from repeated or continuous US screening.
Motivation: One of the most widespread methods used in taxonomy studies to distinguish between strains or taxa is the calculation of average nucleotide identity. It requires a computationally expensive alignment step and is therefore not suitable for large-scale comparisons. Short oligonucleotide-based methods do offer a faster alternative but at the expense of accuracy. Here, we aim to address this shortcoming by providing a software that implements a novel method based on short-oligonucleotide frequencies to compute inter-genomic distances. Results: Our tetranucleotide and hexanucleotide implementations, which were optimized based on a taxonomically well-defined set of over 200 newly sequenced bacterial genomes, are as accurate as the short oligonucleotide-based method TETRA and average nucleotide identity, for identifying bacterial species and strains, respectively.
Moreover, the lightweight nature of this method makes it applicable for large-scale analyses. Availability and implementation: The method introduced here was implemented, together with other existing methods, in a dependency-free software written in C, GenDisCal, available as source code from https://github. com/LM-UGent/GenDisCal. The software supports multithreading and has been tested on Windows and Linux (CentOS). In addition, a Java-based graphical user interface that acts as a wrapper for the software is also available.
Datasets The threshold optimization dataset consisted of 287 bacterial genomes, covering the major phylogenetic lineages  (Forterre, 2015). This dataset was constructed by the Belgian Coordinated Collection of Microorganisms. Selected bacterial strains were cultured and genomic DNA was extracted using either a modification of the procedure of  Pitcher et al. (1989 ), Gevers et al. (2001)  and  Wilson (2001)  or using a Maxwell V R 16 Tissue DNA Purification Kit, after a prior enzymatic lysis step in case of gram-positive strains (Supplementary Table  S1 ). DNA integrity and purity were evaluated on a 1. 0% (w/v) agarose gel and by spectrophotometric measurements at 234, 260 and 280 nm, respectively. DNA concentration was determined with the QuantiFluor V R ONE dsDNA System (Promega Corporation, Madison, WI, USA).
Library preparation and whole-genome sequencing were performed by the Oxford Genomic Center (University of Oxford, United Kingdom). Paired-end sequence reads (PE150) were generated using the Illumina HiSeq 4000 platform (Illumina Inc. , San Diego, CA, USA). Quality check and trimming of the raw sequence reads was performed using the programs FastQC (http://www. bioinformatics. babraham. ac. uk/projects/fastqc/) and Trimmomatic  (Bolger et al. , 2014). The genome assembly of the raw reads was performed using Shovill (https://github. com/tseemann/ shovill), which relies on SPAdes  (Bankevich et al. , 2012)  and pilon  (Walker et al. , 2014) , and evaluated with QUAST  (Gurevich et al. , 2013). Annotation was performed using RAST  (Aziz et al. , 2008). Quality check of the assemblies was performed by determining the presence and absence of 107 single copy core genes using bcgTree  (Ankenbrand and Keller, 2016) , guanine-cytosine content distribution and 16S DNA-based identification.
To test the performance of the similarity threshold, a dataset consisting of 285 species with 20 bacterial genomes per species, and a dataset consisting of 16 228 species with one genome per species were analyzed (Table  1 ). These datasets are referred to as the 'species-balanced' and 'distinct-species' datasets, respectively. Both datasets were subsets from the National Center for Biotechnology Information (NCBI)'s reference sequence database (RefSeq). Taxonomic relations between these genomes were obtained from the NCBI taxonomy database (accessed July 12, 2018). To test the performance of our approach for differentiating strains within bacterial species, i. e. 'types'  (Li et al. , 2009) , a typing dataset (subset of the threshold optimization dataset) was built, containing 63 genomes of six species for which traditional molecular typing data were available.
Burkholderia stabilis and Burkholderia cepacia strains were subjected to multi-locus sequence typing (MLST) based on atpD, gltB, gyrB, lepA, phaC, recA and trpB  (Baldwin et al. , 2005; Spilker et al. , 2009). Pandoraea apista and Pandoraea pnomenusa were typed with recA. Lactobacillus reuteri and Lactobacillus rhamnosus were typed using amplified fragment length polymorphism  (Vancanneyt et al. , 2006). Individual accession numbers for the sequencing data are provided in Supplementary Table  S1. The full lists of accession numbers for RefSeq subsets are provided in Supplementary Datasets S1 and S2.
Comparison with existing methods for purposes of identification To compare our results with existing methods, we computed TETRA (own implementation), and MASH (v2. 0) dissimilarities. For TETRA, the existing implementation in JSpeciesWS is not designed to handle large numbers of genomes efficiently, and as such had to be rewritten. Testing on a dataset containing 23 genomes obtained from RefSeq showed that our implementation did not differ substantially from the JSpeciesWS implementation, with differences only occurring in highly fragmented genomes (Supplementary Table  S2 ). The relative accuracy of the methods was illustrated using the standard receiver operating characteristic (ROC) curves associated with the results and by computing Matthew's correlation coefficient (MCC) as a function of the choice of taxonomic cutoff.
Comparison with existing methods for purposes of typing Bacterial typing is performed on organism-by-organism and case-bycase basis, and as such, there does not currently exist a single typing scheme that can be applied in a more general context. We evaluated the applicability of our method as a predictor of typing-based comparisons on a dataset containing 63 genomes, belonging to 6 species (typing dataset). Unlike the dataset used for evaluating the identification capability of our method, this dataset was not designed to draw general conclusions, but still provides a good indication about the performance of the different methods as the selected species enable comparing multiple levels of taxonomic similarity. Our results were compared with TETRA, MASH and OrthoANI. Moreover, since this dataset only contained 63 genomes, it was computationally feasible to obtain ANI values for paired comparison using OrthoANI.
Optimization of the similarity threshold Optimization of the similarity threshold t was performed using the hill climb approach. The parameter to be maximized was defined as the distance between the centroids of distributions pertaining to two different classes of comparisons, e. g. 'same species' and 'same genus', divided by the sum of the standard deviation of both distributions. d A; B ð Þ¼ l B À l A r A þ r B ( 12 ) where A and B are distributions, l A and l B are their means, and r A and r B are their standard deviations. We preferred this approach to a more typical classifier evaluation, such as the area under the ROC curve, because the distributions associated with comparing genomes belonging to the same species and genomes belonging to the same genus have almost no overlap in the datasets used to estimate the threshold.
Implementation We implemented the software, named GenDisCal, as a standalone command-line application, and provided a Java-based graphical user interface wrapper with a simplified use that provides appropriate presets and also includes most of the options available through the command-line. The computational requirements of this software rely almost exclusively on the choice of signature length (Fig. 1 ). While lengths above eight would remain feasible if a smaller number of genomes were used, these provide neither a computational advantage nor an accuracy advantage over other methods (data not shown) and thus were not considered here. In addition, we performed the same analysis using MASH. For FastANI and OrthoANI, computational costs for the RefSeq dataset were prohibitively large, and smaller datasets were used instead (Table  2 ).
For TETRA, the publically available implementation in JSpeciesWS is not designed to handle more than 15 genomes at once and hence is not considered here. MASH has roughly the same speed as PaSiT6. In order to demonstrate GenDisCal's performance, it was tested against other tools on a Linux server with an Intel Xeon CPU E5-2690 v4, running at 2. 6 GHz, with 56 cores and 252 GB of RAM (16 cores were used in all tests).
Parameter optimization In a first step, the similarity threshold was optimized using a curated dataset containing 287 genomes with an unambiguous taxonomic assignment. For different oligonucleotide lengths as well as similarity threshold values, we computed the distance between the distributions associated with 'same species' comparisons and 'same genus' comparisons (Fig. 2 ). The optimal values for each oligonucleotide length were refined using the hill climb method (Table  3 ). We found that for tetranucleotides, a similarity threshold of 0. 0144 produced the most distinct distributions of any oligonucleotide length. However, we decided to round this value up to 0. 02, since performance is more affected by underestimation than overestimation (Fig. 2 ). With the threshold set to 0. 02, the distance between means of the distributions being 3.
Identification of bacterial species We defined a species-level cutoff value based on the output of the threshold optimization dataset and a similarity threshold of 0. 02. In this dataset, all genomes with distances below 0. 36 belong to the same species (Figs  3a  and 4b ), with the exception of comparisons between Lactobacillus plantarum and Lactobacillus paraplantarum genomes which also fall into this range. The 0. 36 species-level cutoff value is also valid when applied to the species-balanced dataset (Fig. 3b ). However, the distribution from genomes belonging to the same genus ('same genus' distribution) overlaps with the distribution associated with comparisons of genomes belonging to the same species ('same species' distribution). The number of inter-genera distances below 0. 6 is smaller for the distinct-species dataset (Fig.
3c ) than for the species-balanced dataset, indicating that the overlap of the same-genus and same-species distributions in the latter is due to the inclusion of 'outlier' genera. In order to assess whether or not our results improve the stateof-the-art, we compared the relative accuracy of PaSiT to TETRA and MASH using the species-balanced dataset by assessing the ROC curves and MCC associated with the results (Fig. 4 ). The performance of PaSiT4 is comparable with the other methods when it comes to distinguishing species.
Bacterial typing In a next step, we used a dataset with genomes of known bacterial types (typing dataset) to evaluate whether PaSiT could be used for bacterial typing. To illustrate the bacterial typing capabilities of the different methods, heatmaps of complete distance matrices are shown (Fig. 5 ) as well as histograms (Fig. 6 ). In terms of 'same species' comparisons, values for the genomes belonging to B. stabilis and Burkholderia cenocepacia are clearly above the 'same species' cutoffs using PaSiT4 and OrthoANI-based dissimilarity, but fall into the ambiguous zone when using TETRA (see Supplementary Tables S3-S8 for exact values). On the other hand, values for the L. reuteri genomes are below the 'same species' All comparisons had to be performed twice due to input format limitations, so time could potentially be halved.
Conclusion ANI was proposed as a new standard for bacterial identification based on genome coherence by  Richter and Rossello-Mora (2009)  and remains to date one of the most effective ways to compare entire genomes to each other. However, until recently the computational cost of such methods was prohibitive for large datasets such as the ones presented here. While this problem has largely been solved by   S7  and S8 , respectively. (Color version of this figure is available at Bioinformatics online. ) the introduction of ANI approximations using MinHash techniques, it can be further reduced by using tetranucleotide frequencies as a basis for comparison. Moreover, these MinHash-based methods utilize only part of the entire genome to perform comparisons, as opposed to frequency methods. The PaSiT method is on par with TETRA in terms of accuracy when using tetranucleotides (PaSiT4).
A stand-alone version of TETRA, which is able to handle large-scale comparisons, is currently not available. Therefore, a rigorous comparison is not possible. Nevertheless, GenDisCal contains an implementation of TETRA  (Teeling et al. , 2004a) , which was used to compare its accuracy to the proposed method. Interestingly, while both methods have a similar accuracy, they are not strongly correlated, with an R 2 of 0. 7 on the 'same species' distribution from the species-balanced dataset (Supplementary Fig. S3 ). This indicates that the proposed method exploits the available data differently. Indeed, by reducing the influence of large differences between components of signatures on the overall distance, cases where only a small part of the signature differs produce a small distance, even if the parts that differ do so significantly.