dataesr/scientific-paragraphs-categorization huggingface dataset:
7 × 10 5 , for c = 0. 1 and L = 400, up to around n = 1. 1 × 10 9 fragments, when c reaches its maximal value. This is way beyond the configurations considered for instance in  Patil et al. (2012) , where SVM models were learned from approximately 10 4 fragments drawn from 1, 768 genomes. To assess the performance of these models we consider two sets of 134, 319 fragments, of respective length 200 and 400, drawn from the 52 complete genomes that are not in the reference database used to train the models. Performance is measured by first computing, for each species, the proportion of fragments that are correctly classified, and considering its median value across species. In a multiclass setting, this indicator is indeed less biased towards over-represented classes than the global rate of correct classification.
Increasing the number of fragments confirms this hypothesis : performance systematically increases or remains steady with k for c ≥ 1, and for k ≥ 8, the performance is significantly higher than that obtained at c = 0. 1, for both length of fragments. Increasing the coverage from c = 1 to c = 10 has a positive impact in both cases, although marginally for fragments of length 400. Further increasing the number of fragments does not bring any improvement. Altogether, the optimal configuration on this mini dataset involves k-mers of size 12 and drawing fragments at a coverage c ≥ 10 for the two lengths of fragments considered. Further increasing the size of the k-mers did not bring improvements, and actually proved to be challenging. Indeed, as mentioned above, VW proceeds by hashing the input features into a vector offering at most 2 32 entries.
This hashing operation can induce collisions between features, which can be detrimental to the model if the number of features becomes too high with respect to the size of the hash table. This issue is even more stressed in a multiclass setting, where the number of hash table entries available per model is divided by the number of classes considered. On this dataset, 51 models have to be stored in the hash table, which reduces the number of entries available per model to 2 32 /51 ∼ 2 32-6 = 4 13. We have empirically observed that performance could not increase for k greater than 12 and actually decreased for k-mers greater than 15. We now compare these results to two well-established approaches: a comparative approach based on the BWA-MEM sequence aligner  (Li, 2013)  and a compositional approach based on the generative NB classifier  (Rosen et al.
, 2011). The NB experiments rely on the FCP implementation  (Parks et al. , 2011)  and are carried out in the same setting as VW: we compute profiles of k-mers abundance for the 356 genomes of the reference database, and use them to affect test fragments to their most likely genome. BWA-MEM is configured to solely return hits with maximal score (option -T 0). Unmapped fragments are counted as misclassifications, and a single hit is randomly picked in case of multiple hits, in order to obtain a species-level prediction. This latter random hit selection process is repeated 20 times and the performance indicator reported below corresponds to its median value obtained across repetitions. Results are shown in Figure  2. We first note that k-mer based approaches, either generative or discriminative, never outperform the alignment-based approach.
Evaluation on the small and large reference databases We now proceed to a more realistic evaluation involving a larger number of candidate microbial species and a larger number of reference genomes, using the small and large reference databases. We learn classification models according to the configuration suggested by the evaluation on the mini database: we consider k-mers of size 12 and a number of fragments allowing to cover each base of the reference genomes 10 times in average. We limit our analysis to fragments of length 200, which leads to learn models from around n = 1. 38 × 10 8 and n = 2. 56 × 10 8 fragments for the small and large reference databases, respectively. Note that due to the larger number of species involved, around 2 32-8 = 4 12 and 2 32-10 = 4 11 entries of the VW hash table are available per model for each of these reference databases.
Based on the results with the mini database, this should be compatible with k-mers of size 12. We evaluate the performance of the models on fragments extracted from the 193 genomes of the validation database and draw a number of reads necessary to cover each base of each genome once in average, which represents around 3. 5 × 10 6 sequences. Results obtained by VW are again compared to that obtained by the two baseline approaches involved in the previous proof of concept, namely BWA-MEM and NB, and are shown in Table  2. We first note that for the small reference database, the performance of VW and BWA-MEM are very similar (median species-level accuracy of 92. 4% and 93%, respectively). The NB classifier, on the other hand, has a significantly lower performance, with 8% less in median accuracy than the alignment-based approach.
5 × 10 6 sequences as well. Homopolymeric error models. To evaluate the impact of homopolymeric errors, we consider the three error models implemented in Grinder : Balzer  (Balzer et al. , 2010) , Richter  (Richter et al. , 2008)  and Margulies  (Margulies et al. , 2005). Results are shown in Figure  3. We first note that this kind of errors has a very limited impact on BWA-MEM: only the Margulies model turns out to be detrimental, with a drop of less than 1% for both the small and large reference databases. The Balzer and Richter models have a limited impact on the compositional approach as well: a drop of less than 1% is observed as well in most cases (except with the NB classifier, where a drop of almost 2% is observed using the large reference database and the Richter model). The Margulies model, on the other hand, has a much more severe impact on the performance of k-mer based approaches.
While a relatively limited performance drop of around 1. 5% is observed with VW using the small reference database, the NB shows a drop of more than 5%. Considering the large reference database, both approaches show a drop of almost 8%, which therefore leads to a gap of more than 10% and up to 20%, for VW and NB respectively, compared to the performance of the alignment-based approach. This discrepancy is therefore significantly higher than the one observed from fragments, where VW and NB have performance lower than that of BWA-MEM by around 4% and 12%, respectively, on the large reference database. Analyzing the error profile of the reads obtained by Grinder reveals that both the Balzer and Richter models lead to a median mutation rate of 0. 5% (meaning that half of the 200 bp simulated reads show more than one modified base), while this rate raises to 3% with the Margulies model.
While this can readily explain why this latter model had a stronger impact, it suggests that what may be seen as a relatively moderate modification of the sequences (6 bases out of 200) can have a severe impact on compositional approaches. Mutation error model. To study the impact of general mutation errors, we consider the 4th degree polynomial proposed by  Korbel et al. (2009)  and implemented in Grinder. Using the default values proposed by Grinder, we empirically observe a median mutation rate of 10. 7%. This value is much more important than what is expected by current NGS technologies, and is probably due to the fact that this model was calibrated  Top: small reference database. Bottom: large reference database. These figures give the median species-level accuracy obtained with VW (purple), NB (green) and BWA (grey).
Each approach has been evaluated on three datasets simulated according to three different error models: Balzer  (Balzer et al. , 2010) , Richter  (Richter et al. , 2008) , and Margulies  (Margulies et al. , 2005). from shorter reads. Indeed, the median mutation rate decreases to around 1. 5% when we reduce the length of the reads to 30, in agreement with the results of the original publication  (Korbel et al. , 2009). To investigate in details the impact of mutations within reads of length 200, we modify the parameters of the error model in order to gradually increase the median mutation rate from 1% to 10%, by 1%. This therefore leads to simulating 11 datasets, since we consider in addition the default Grinder configuration. Results are shown in Figure  4. We first note that this type of errors has a very limited impact on alignment-based approach: even at the higher rate of mutation considered (median mutation rate of 10.
7%), the performance drops by around 1% with respect to the performance obtained with fragments, for both the small and large reference databases. On the other hand, the performance obtained with compositional approaches steadily decreases when the mutation rate increases. Using the small reference database, the impact is more severe for NB than for VW : a drop of up to 10% is observed in the former case (from 85. 1% for fragments down to 75. 2% for a mutation rate of 10. 7%) and almost 6% in the latter (from 92. 4% down to 86. 7%). The drop is even more severe using the larger database in both cases. Interestingly while it remains relatively constant around 10% for mutation rate greater than 4% with NB (hence twice the gap observed between the small and large datasets using fragments), it keeps increasing with VW and reaches 24% at the highest mutation rate considered.
As a result, VW is outperformed by NB using the large reference database for mutation rates greater than 8%. Although these extreme configurations are not realistic regarding the current state of the NGS technologies, we emphasize, in agreement with the previous experiment on homopolymeric errors, that significant drops are observed with compositional approaches for moderate mutation rates, especially for large number of candidate species. For instance, with a mutation rate of 2%, the performances of VW and NB drop respectively by 4 and 5% with the large reference database, while this has no impact on the alignment approach. In this more realistic setting, the alignment-based approach shows markedly higher performances: it provides a median species-level accuracy of 91. 7%, while VW and the NB classifier reach 83.
9% 74. 8%, respectively. q q q q q q q q q q q 2 4 6 8 10 50 60 70 80 90 100 mutation rate (%) median species-level accuracy q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q VW NB BWA small DB large DB Figure  4 : Robustness to sequencing errors: mutation-based models. These figures give the median species-level accuracy obtained with VW (purple), NB (green) and BWA (grey), obtained with the small (dashed lines) and large (solid lines) reference databases. Each approach has been evaluated on 11 datasets simulated according to different mutation rates (from 1 to 10. 7%) with the error model proposed in  Korbel et al. (2009).
Classification speed Last but not least, we now turn to the comparison of the comparative and compositional approaches in terms of prediction time. This aspect is indeed of critical importance for the analysis of the large volumes of sequence data provided by next-generation sequencing technologies, and constitutes the main motivation of resorting to k-mer based approaches. To perform this evaluation we measure the time taken by BWA-MEM and the k-mer based approaches to process the 30 test datasets involved in the previous experiments (1 fragments dataset, 3 reads datasets with homopolymeric errors and 11 reads datasets with mutation errors, for the two reference databases considered). This allows us to investigate the impact of the number of species involved in the reference database, as well as the amount of sequencing noise in the reads.
For 193 and 774 species and k-mers of size 12, this amounted to 12 and 48 gigabytes, respectively. Computation times are measured on a single CPU (Intel XEON -2. 8 Ghz) equipped with 250 GB of memory, and summarized in Figure  5. The time needed to classify each read or fragment dataset by the k-mer approach shows little variation, for a given reference database. The median value obtained across test datasets reaches 5. 4 and 9. 1 minutes, using the small and large reference database, respectively, hence about a two-fold difference. This therefore amounts to classifying around 6. 5 × 10 5 and 3. 9 × 10 5 reads per minute, respectively. BWA-MEM shows a different behavior. We observe that the time varies more across reads and fragment datasets, and tends to increase with the amount of sequencing noise.
Discussion In this work, we investigate the potential of modern, large-scale machine learning approaches for taxonomic binning of metagenomics data. We extensively evaluate their performance when the scale of the problem increases regarding (i) the length of the k-mers considered to represent a sequence, (ii) the number of fragments used to learn the model, and (iii) the number of candidate species involved in the reference database. We also investigate in details their robustness to sequencing errors using simulated reads. We consider two baselines for this evaluation: a comparative approach based on the BWA-MEM sequence aligner and a compositional approach based on the generative NB classifier. We demonstrate in particular that increasing the number of fragments used to train the model has a significant impact on the accuracy of the model, and allows to estimate models based on longer k-mers.
Finally, we confirm that compositional approach achieve faster prediction times. This is indeed systematically the case in the various configurations listed above, with predictions obtained 3 to 15 times faster by compositional approaches, and, interestingly, depends on the number of candidate species and the level of sequencing noise. We emphasize, however, that fast predictions can only be obtained provided that the classification models are loaded in memory, hence for a memory footprint that scales linearly with the number of candidate species and exponentially with the size of the k-mers, which can become important for large reference databases and long k-mers. At least three simple extensions could be envisioned to make compositional approaches more competitive in accuracy with the alignment-based approach, faster, and to limit their memory footprint.
Figure3: Robustness to sequencing errors: homopolymer-based models. Top: small reference database. Bottom: large reference database. These figures give the median species-level accuracy obtained with VW (purple), NB (green) and BWA (grey). Each approach has been evaluated on three datasets simulated according to three different error models: Balzer (Balzer et al. , 2010) , Richter (Richter et al. , 2008) , and Margulies (Margulies et al. , 2005).
There are several technologies for monitoring sedation status based on electroencephalogram (EEG) analysis, but these were developed primarily for monitoring the depth of anaesthesia and their validity for use in ICU patients is uncertain  [16]. Although these algorithms correlate with clinical sedation scores, their discriminant ability is limited, in part because of interference from facial frontal electromyelogram (fEMG) signals during arousals and in lighter sedation states  [17, 18]. There have been few randomised trials of these technologies, and the available data do not support clinical effectiveness  [13, 19]. We recently described a novel technology for continuous monitoring of patient status in ICU patients, the Responsiveness Index (RI), based on fEMG activity  [20] [21] [22]. The algorithm uses fEMG data acquired via adhesive surface forehead electrodes, and utilises the previous 60 minutes of fEMG activity to assess the frequency and intensity of arousals during ongoing treatments.
Control group (group 1) All patients were attached to the Responsiveness monitor, and data were acquired using a laptop computer. For these patients a monitoring device was used, which did not display any RI data throughout the study. RI data could only be accessed by code by the research team (to check quality etc. ), which was not available to clinical staff. The monitor presented an automated prompt every hour during the study to assess the RASS score and enter it into the study case record form (CRF). The prompt flashed on the monitor until a score was entered, and the exact time of entry logged. The local sedation protocol encouraged staff to target a RASS score of >-3 and to undertake a sedation hold on a daily basis unless clinically contraindicated.
Intervention group (group 2) As for group 1, all patients in the intervention group were attached to the Responsiveness monitor, and data collected using a laptop computer. For these patients RI data were presented as a continuous trend, with a numeric value (0-100), and a colour code (red, amber, green) according to our previous published work, which suggested a value with a red colour had sensitivity to detect patients who may be excessively sedated  [20]. An example of a monitor during use is shown in Fig. 1. Prompts accompanied the colour of RI presented on the screen, to encourage sedation reduction for patients with a red or amber RI value (RI <40). Nurses were asked to alter sedation using clinical judgement to transition patients out of the red RI range and adjust sedation to achieve values in the amber/green range.
Data collection Baseline data were age, gender, admission diagnosis, time from hospital to ICU admission, acute physiology and chronic health evaluation (APACHE) II score, Charlson For intervention patients, the screen presented a running trend over several hours, with the most recent value presented on an arbitrary scale of 0-100, with a colour code of RED (<20), AMBER (20 to 40); and GREEN (>40). In addition to the colour trend, the following prompts were automatically presented according to RI value: RED high risk of over-sedation. Ensure adequate analgesia if responsive to stimuli, e. g. , suctioning/physiotherapy. Reduce sedation dose if no eye-opening to physical stimuli. AMBER: moderate risk of over-sedation. Ensure adequate analgesia. Reduce sedation if no eye-opening to physical stimuli.
GREEN: low risk of over-sedation. Ensure adequate analgesia. Continue current sedation unless patient agitated comorbidity index, sequential organ failure assessment (SOFA) score (excluding neurological score) for the first 24 hours (range 0-20), RASS score at the start of monitoring, and intravenous sedation and analgesic drug use in ICU prior to RI monitoring. Hourly data collection during the intervention period was RASS score, and intravenous sedative and analgesic dose. The use of formal sedation holds was recorded on a daily basis. The following predefined sedation-related adverse events were recorded: unplanned extubation, unplanned removal of vascular catheter, unplanned removal of nasogastric/enteral tube, episode of myocardial ischaemia, myocardial infarction and agitation requiring pharmacologic treatment.
Analysis The intervention period was defined as the start of monitoring to 48 hours post ICU admission. Baseline RI was defined as the first valid RI value recorded. For RI analyses patients in whom monitoring was removed because the patient was extubated were assumed to have an RI value >20 during the post-extubation period. Periods during which NMBAs were used, plus the 30 minutes after discontinuation, were censored from the analysis period. No deaths occurred during the 48-hour intervention period, so competing risk of death was not relevant. RASS data were pre-processed to impute missing data using several rules: first, if RASS scoring was not performed during a particular hour, the subsequent hourly RASS value was utilized in the analysis; if the subsequent hour had no RASS value, the previous hourly value was used; if neither value was available then that hour was considered to have no RASS data point.
For comparisons of total intravenous sedative and analgesic dose during the intervention periods propofol and alfentanil equivalents were calculated for patients who received midazolam and/or morphine as an alternative or addition to propofol/alfentanil. We used a conversion factor of 1 mg midazolam = 10 mg propofol, and 1 mg morphine = 100 μg alfentanil. Prior to the trial we had no data to inform the distribution of RI values at enrolment. As the intervention aimed to promote transition from an RI value of <20 (red colour code) to higher RI values (amber/green colour code) we undertook a post hoc subgroup analysis for patients with baseline RI <20. We justified this analysis because for this group the protocol directed nursing staff to decrease sedation at enrolment, whereas for patients with RI values >20 the protocol did not direct nurses to alter sedation based on the intervention.
Results Ninety patients were randomised. No consent was obtained post-randomisation in 12 patients (data could not be utilised according to local ethics approval) (4 in the intervention group and 8 in the control group), a protocol deviation occurred in 4 patients (the randomisation procedure was incorrectly followed by clinical nursing staff in 2 patients, and monitoring was not started by clinical staff during sedation in 2 patients despite randomisation to the intervention group). We included 74 patients in the analysis (36 in the intervention group and 38 in the control group). Patient characteristics and baseline data are shown in Table  1. The patient groups were similar at baseline with respect to all measured variables. Monitoring was started within 8 hours of intubation for >75 % of participants.
Acknowledgements MK, PL, KU, PP, and MOKS are employees of  GE Healthcare. The study was funded through a grant from  GE Healthcare  to  Edinburgh University  and  NHS Lothian  (where the study was undertaken). KE received salary support from  GE Healthcare  as part of the grant to  Edinburgh University  and  NHS Lothian. We thank the nursing staff of the  Intensive Care Unit, Edinburgh Royal Infirmary Scotland  for their enthusiastic support of the trial and for participating in the nursing interviews.
g. , ribosomal ITS2, nuclear EF-1α, mitochondrial cytochrome b, mitochondrial cytochrome oxidase subunit 1) which can be compared with sequences in an online database to identify the sand fly species  (Latrofa et al. , 2011; Erisoz Kasap et al. , 2013; Gutiérrez et al. , 2014). However, this is time consuming and requires access to specialist equipment. More rapid molecular methods for the identification and differentiation of sand fly species include (polymerase chain reaction) PCR of the ribosomal ITS2 region  (Khalid et al. , 2010; Latrofa et al. , 2012)  and PCR-RFLP (restriction fragment length polymorphism) of the mitochondrial DNA fragment cytb -nd1  (Latrofa et al. , 2012)  or 18S ribosomal RNA gene  (Tiwary et al. , 2012). Both methods allow the separation of species based on differences in fragment sizes, and allow the identification of a large number of samples in a short time with limited resources.
We successfully purified MRGPRD + , C-LTMRs and a heterogeneous population of sensory neurons and subjected their RNA contents RNA-deep sequencing (RNA-seq). The subsequent RNA-seq experiment led to the generation of unique sets of data representative of pure transcriptome profiles of each subset. As a result of this pioneering approach, we established the combinatorial expression of the sets of genes that could dictate the functional specializations of MRGPRD + neurons and C-LTMRs. Herein we provide details regarding the experimental design, the quality controls and statistical analysis of the data deposited at Gene Expression Omnibus under the accession number GSE64091.