dataesr/scientific-paragraphs-categorization huggingface dataset:
A breakdown of the results on the Prospective dataset, showing each drug separately, is given in Figure  5. Mykrobe and ARIBA, as expected, are very similar except for the high false-positive rate for isoniazid for ARIBA, resulting in a PPV of 82% for ARIBA compared to 94% for Mykrobe. Manual inspection revealed that the extra incorrect calls were caused by assembly errors in the katG gene, resulting in false-positive calls because the gene appeared to be incomplete. The memory usage and run time averaged across all runs of each tool is given in Table  6 , and in Extended data files we have: summary boxplots in run_time_boxplots. pdf  40 and memory_boxplots. pdf  41 , and the complete data in run_time_memory. tsv  42.
Impact of Mykrobe on initial choice of WHO-recommended regimen The World Health Organisation (WHO) now recommends systematic access to drug susceptibility testing and specific TB treatment regimens based on individual drug resistance profiles. We therefore ask, if we were to base the choice of a patient's initial multi-drug therapy purely on the genotyping results from Mykrobe, how accurate would this choice of treatment regimen be? We encoded the most recently published WHO TB treatment recommendations (https://github. com/iqbal-lab-org/tb-amr-benchmarking/blob/master/ python/evalrescallers/who_treatment. py), and used this to infer both the recommended drug regimen implied by the known phenotype of each sample, and that implied by Mykrobe's resistance prediction (on the combined global and prospective datasets).
We are able to compare and contrast these approaches, as we are using the same data, and conclude that the novel-mutation-aware approach is preferable to that used by Mykrobe (and, as far as we can tell, all other benchmarked tools). Essentially, novel (off-catalogue) mutations are an effective flag for very major errors (missed resistance). Our feeling is that the cost (delays for susceptible samples containing novel mutations) is outweighed by the benefit (avoiding some false susceptible calls). Modifying Mykrobe to follow that policy is therefore something we would like to do in the future. Of the other tools we tested, MTBSeq and TB-profiler would most easily be able to adopt this approach, as they both already do full SNP discovery for every sample. In general Mykrobe outperforms all the other benchmarked tools, having the best sensitivity and best or second-best specificity on each of the three datasets.
Mykrobe supports error-prone ONT reads, and we show that ONT and Illumina data produce identical results, although we would wish to have tested on more than five samples. Motivated by the fact that low frequency resistant populations will sweep to fixation under drug pressure, and by the results in the Mykrobe predictor paper 1 , Mykrobe predicts resistance if it detects known resistance alleles at low frequencies (in practise above 10%, for illumina data only). For isoniazid, rifampicin and the second line drugs, this results in detection of more resistant isolates, at limited cost in false positives (see Table  3 ). However for ethambutol and pyrazinamide the proportion of these extra calls that are false is unacceptably high (38% and 15%, respectively). Possible reasons for different performance for these drugs are given above; future versions of Mykrobe are likely to exclude minor alleles for these drugs.
We introduce the idea of measuring how sequencing-based diagnostic for TB drug resistance impact potential regimen choice, and include a powerful way to visualise it. The idea is explicitly not to suggest this could directly advise physicians, but instead to provide a different metric for clinically important errors. Our analysis has limitations. First, we have only partial phenotyping data for second line TB drugs, and in fact none for the new and repurposed TB drugs. Second, WHO guidelines are ever evolving and may differ from clinical expert opinions when it comes to the choice of personalized therapeutic regimens. Nevertheless, we can clearly see that the richer resistance profile from WGS data leads to more differentiated choices of regimen. We look forward to impending large and fully phenotyped datasets, for example from the CRyPTIC consortium, where tens of thousands of global samples will have quantitative phenotype data for 14 drugs.
Command-line version. Installation of the command line version is via cloning the git repository and running some install commands (documented here: https://github. com/Mykrobe-tools/mykrobe), or by using bioconda: conda install -c bioconda mykrobe Drug resistance predictions can be obtained from a sample using the single command of the form: mykrobe predict --format json --seq reads. fastq --output out. json sample_name species where "sample_name" can be replaced with the name of the sample being run (any text will do). The "species" parameter can be "tb" or "staph" to use the built-in panels, or "custom" for a usergenerated panel (an example is shown later). The output file "out. json" contains variant and drug resistance calls, and details of the run, such as the version number of Mykrobe and input parameters.
The first is a change from G to GC at position 12 in the DNA sequence of the pncA gene. The second variant is an amino acid change from D to G at position 94 in the embB gene. The final variant is eight nucleotides upstream of the fabG1 gene, where the T in the reference genome is changed to A, C, or G (i. e. X is a wildcard). Note that all positions are relative to the gene, not the complete genome sequence. The following command produces a FASTA file of probe sequences mykrobe variants make-probes reference. fasta -g reference. gb -t panel. tsv > probes. fasta where reference. fasta and reference. gb are FASTA and GenBank format files of the reference sequence and annotation respectively. Note that background variants can also be included in the probe set, which requires the variants in one or more VCF files, and MongoDB to be installed.
The following BASH commands will add variants to a new database. db=$PWD/mongo-db/ mkdir $db mongod --quiet --dbpath $db & sleep 10 for f in 'cat vcfs. txt' do mykrobe variants add --db_name mtb $f reference. fasta done In the above commands, the file vcfs. txt contains a list of the VCF file names (one file name per line), and the variants from each VCF file are added in turn to the database. Then the same command as above can be run to make the probe sequences, but with the extra option --db_name mtb. In addition to the probes FASTA file, a JSON file that maps each variant to a list of drugs is required to run the "predict" subcommand of Mykrobe. For example, the JSON file corresponding to the above example three variants is as follows. {"pncA_G12GC": ["Pyrazinamide"], "gyrA_D94G": ["Ciprofloxacin", "Ofloxacin", "Moxifloxacin"], "fabG1_T-8X": ["Isoniazid"]} The probes FASTA file probes.
Containerised releases. A Docker container can be obtained using the command docker pull quay. io/biocontainers/mykrobe:0. 7. 0--py37h2666aa9_0 Variant panel development. The new Mykrobe variant catalogue was developed using the Training and Global datasets as follows (see also Figure  2 ). We started with the catalogue of variants from 20 for the first line drugs, which we call NEJM-2018. This was used together with the remaining drugs from the existing Mykrobe predictor release panel (Walker-2015), and the fluoroquinolones separated into ciprofloxacin, moxifloxacin and ofloxacin. This combined set of variant sites was called candidate panel 1 (CP1). Note that this includes all frameshifts of length 1 and 2 in pncA and katG, implying resistance to pyrazinamide and isoniazid respectively. Mykrobe was run on the Training dataset using CP1 and the results were compared against the known phenotypes for those samples.
All variants that had positive predictive value less than 5% were removed from CP1, to make candidate panel 2 (CP2). Next, the process was repeated, removing variants from CP2 to make candidate panel 3 (CP3), based on the results of running Mykrobe with CP2 on the Global data set. All removed variants are provided in the Source data file removed_variants. tsv  32. Finally, three removed variants were reinstated into the panel because they were found to have positive predictive value greater than 5% on later datasets (these were identified using the results from the Walker-2015 panel). The final panel is included as the default set of variants in release 0. 7. 0 of Mykrobe. Comparison with other tools. Since our combined Training, Global, and prospective datasets consist of more than ten thousand samples, we only benchmarked Mykrobe against tools which could be run on the commandline (not web or user-interface applications).
Second, unlike Mykrobe, ARIBA was not set to report resistance-conferring synonymous changes in gene sequences. We note that ARIBA code version 2. 13. 2 was run for this study, but using the built-in panel from version 2. 13. 3, which corresponds to the final Mykrobe release panel. These versions only differ in that 2. 13. 2 has a variant panel corresponding to CP3. Mykrobe and ARIBA explicitly have the fluoroquinolones ciprofloxacin, moxifloxacin, and ofloxacin in their variant catalogues. However, KvarQ, MTBseq and TB-Profiler simply call fluoroquinolone resistance without distinguishing between the three drugs. Therefore when reporting results, a fluoroquinolone call was taken to imply resistance to all three of ciprofloxacin, moxifloxacin, and ofloxacin. The benchmarking pipeline was implemented using Nextflow  48  , Singularity  49   Porechop (version 0.
2. 3) was then used to trim adapters from the reads and to do additional basecalling. When demultiplexing is done by Porechop on the Albacore demultiplexed folders, Porechop will assign reads to a barcode where both Albacore and it agree. All reads where there is disagreement between the two are placed in the "unclassified" category. The resulting FASTQ files for each barcode were aligned to the M. tuberculosis H37Rv reference (NC_000962. 3 51 ) using minimap2  52  with the map-ont option. The resulting SAM file was sorted, indexed, and converted to FASTQ using SAMtools  53. When converting the SAM file to FASTQ, all unmapped reads (SAM flag 0x4) were filtered out. The final Mykrobe panel from release 0. 7. 0 was run on the samples, separately on each of the Illumina and ONT reads. For ONT reads, the option --ont was used to trigger the use of the dynamic genotype confidence cutoff, as described above.
1. Reviewer Expertise: WGS of mycobacteria, phylogenetics, molecular epidemioogy, clinical bioinformatics I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Major comments: PAGE 5: The geographic breakdown of training, prospective and global sample subsets seems disproportionate. Presumably, this is due to the training and prospective datasets from specific sites having more reliable or complete data and functional susceptibility information? The multi-phased development and validation of the resistance prediction panel is otherwise well-described (text and Figure  2 ) and adequate, but the geographic sampling bias should at the very least be explained. PPV and NPV performance metrics were all impressive, with low overall error rate; critical discrepancies were adequately addressed or explained.
Without delving into the primary data, the dataset tables on Page 10 using the final Mykrobe panel would seem to suggest significant underlying differences in PPV/NPV/VME/ME between datasets. This could be an issue if the characteristics of the three test populations used to develop and refine the final Mykrobe panel are not relatively homogenous. Minor comments: PAGE 9: Interestingly (Figure  4 ), ciprofloxacin resistance false positivity seems to have increased slightly with Mykrobe-CP3, without corresponding shifts in the prediction of other fluoroquinolones. This is noted on Page 9, which implicates D94A in gyrA, and discusses adjustments that were made to CP3 to accomodate. While I'm not by any means an expert in the specific mechanisms of fluoroquinolone resistance in TB, presumably Mfx and Ofx should also show some degree of 2.
4. fluoroquinolone resistance in TB, presumably Mfx and Ofx should also show some degree of impact? Seems odd. PAGE 11-12: Assessing the impact of minor alleles and unknown mutations was a welcome addition to this report. Thank you. PAGE 13-14: Comparative benchmarking with other software, using default databases and settings. Mykrobe performance appears to be most on-par with ARIBA (Page 13), albeit with significantly higher peak RAM utilization (Page 15). Summary tables from Page 13 and graphical breakdowns in Figure  5  may be slightly redundant and some of this could likely be condensed or moved to a supplemental. The flow diagram on Page 16 is particularly useful in teasing apart the genotypic prediction and proposed drug regimen data. These data clearly underscore the difficulty in consistently assigning regimens from TB genotype data.
Additionally, the sensitivity and specificity stated in the abstract are not found elsewhere in the paperthey can be calculated with data given, but would be beneficial to have in results. The methods section should be organized to mirror the order of the results section as this would enable readers to quickly reference methods when reading the results section. In the methods variant panel development section, it is unclear what data sets CP2 and CP3 are run on to generate the next panels. This is clear in the figure  2 , but text was unclear.
catalog update, ability to use other mutation catalogs, improved identification of non-tuberculosis Mycobacterial species, and ability to use tool with Oxford Nanopore data. The software is freely available at mykrobe. com and has source code and documentation on Github. Additionally the data presented in the paper can be found on figshare. The development of the mutation panels and data interpretation is demonstrated on three datasets: training, prospective, and global set. The benchmarking against four other prediction tools is useful to show improvements and performance. Although only five Oxford Nanopore samples were used, they did show agreement in drug resistant variant calls. The case study of using predictions to guide personalized regimens is an extremely useful comparison and Mykrobe illustrates valuable clinical application of this type of tool.
Is the rationale for developing the new software tool clearly explained? YesIs the description of the software tool technically sound? YesAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? YesIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? YesAre the conclusions about the tool and its performance adequately supported by the findings presented in the article? Yes No competing interests were disclosed. Competing Interests:Reviewer Expertise: Bacterial phylogenetics, Infectious disease molecular epidemiology, and NGS technology I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
These final variants are given in Source data panel. final. tsv 35  panel. final. json 36  which are the same as those used by default in the new Mykrobe release version 0. 7. 0. The accuracy statistics of candidate panels 1, 2, and 3 on the Training, Global and Prospective datasets are given in Table 1(a), (b), and (c) respectively. The results of the final release panel on each of the datasets are in Table.
and Python scripts, and is freely available under the MIT licence (code available here: https://github. com/iqbal-lab-org/tb-amr-benchmarking ; commit version used was 3e768c025ee54f154116dc6b2f7100a844081609). Running time and peak RAM usage of the tools was gathered using the output of the UNIX command time -v, using the fields "Elapsed (wall clock) time" and "Maximum resident set size". The benchmarking pipeline outputs all data in a single JSON file (included as Extended data file pipeline. json. gz 50 ), which was then used to generate the results of this study, including all tables and figures, with the separate code repository (code here: https://github. com/iqbal-lab-org/tb-amr-benchmarking-paper/ ; the version/commit used to generate this paper was 59e1b442dd04ec8e16656334ab9ea7d5cefaa871).
37urce data Figshare: sample_data. tsv. https://doi. org/10. 6084/m9. figshare. 7556789. v129. This project contains European Nucleotide Archive accession numbers for Illumina sequence data and associated metatdata. Figshare: ont. tsv. https://doi. org/10. 6084/m9. figshare. 7605443. v137. This file contains the run time and peak memory usage of each tool on each sample. This file contains the number of true-and false-positive calls for each tool and each reported variant. This file contains the otuput of the benchmarking pipeline. Figshare: who_regimens. tsv. https://doi. org/10. 6084/m9. figshare. 7605476. v2 54. This file contains drug regimens defined by the WHO.
v1 ZI was funded by a  Wellcome Trust/Royal Society Sir Henry Dale Fellowship , grant number  102541. MH was supported by Grant information:  Wellcome Trust/Newton Fund-MRC Collaborative  Award [200205]; and  Bill & Melinda Gates Foundation Trust  [ OPP1133541 ]. TMW is a  Wellcome Trust   Clinical Career Development Fellow , grant number  214560. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
1 2 1 Is the rationale for developing the new software tool clearly explained? YesIs the description of the software tool technically sound? YesAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? YesIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? YesAre the conclusions about the tool and its performance adequately supported by the findings presented in the article? Yes No competing interests were disclosed. Competing Interests:Reviewer Expertise: Bacterial phylogenetics, Infectious disease molecular epidemiology, and NGS technology I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
a) Candidate panel 1, Training dataset Drug TP FP TN FN VME (95% CI) ME (95% CI) PPV (95% CI) NPV (95% CI) Ethambutol 209 121 2807 22 9. 52(6. 4-14. 0%) 4. 13(3. 5-4. 9%) 63. 33(58. 0-68. 3%) 99. 22(98. 8-99. 5%) Isoniazid 564 32 2538 38 6. 31(4. 6-8. 5%) 1. 25(0. 9-1. 8%) 94. 63(92. 5-96. 2%) 98. 52(98. 0-98. 9%) Rifampicin 373 40 2714 11 2. 86(1. 6-5. 1%) 1. 45(1. 1-2. 0%) 90. 31(87. 1-92. 8%) 99. 6(99. 3-99. 8%) Amikacin 56 3 702 8 12. 5(6. 5-22. 8%) 0. 43(0. 1-1. 2%) 94. 92(86. 1-98. 3%) 98. 87(97. 8-99. 4%) Capreomycin 51 36 669 10 16. 39(9. 2-27. 6%) 5. 11(3. 7-7. 0%) 58. 62(48. 1-68. 4%) 98. 53(97. 3-99. 2%) Ciprofloxacin 19 4 243 0 0. 0(0. 0-16. 8%) 1. 62(0. 6-4. 1%) 82. 61(62. 9-93. 0%) 100. 0(98. 4-100. 0%) Kanamycin 11 1 528 6 35. 29(17. 3-58. 7%) 0. 19(0. 0-1. 1%) 91. 67(64. 6-98.
Candidate panel 2, Global dataset Drug TP FP TN FN VME (95% CI) ME (95% CI) PPV (95% CI) NPV (95% CI) Ethambutol 1338 434 3499 230 14. 67(13. 0-16. 5%) 11. 03(10. 1-12. 1%) 75. 51(73. 5-77. 5%) 93. 83(93. 0-94. 6%) Isoniazid 2781 54 2704 184 6. 21(5. 4-7. 1%) 1. 96(1. 5-2. 5%) 98. 1(97. 5-98. 5%) 93. 63(92. 7-94. 5%) Pyrazinamide 750 177 2408 264 26. 04(23. 4-28. 8%) 6. 85(5. 9-7. 9%) 80. 91(78. 3-83. 3%) 90. 12(88. 9-91. 2%) Rifampicin 2634 60 2990 136 4. 91(4. 2-5. 8%) 1. 97(1. 5-2. 5%) 97. 77(97. 1-98. 3%) 95. 65(94. 9-96. 3%) Amikacin 100 5 1495 21 17. 36(11. 6-25. 1%) 0. 33(0. 1-0. 8%) 95. 24(89. 3-97. 9%) 98. 61(97. 9-99. 1%) Capreomycin 97 26 1600 42 30. 22(23. 2-38. 3%) 1. 6(1. 1-2. 3%) 78. 86(70. 8-85. 1%) 97. 44(96. 6-98. 1%) Ciprofloxacin 1 4 33 4 80. 0(37. 6-96. 4%) 10. 81(4. 3-24.
(c) Candidate panel 3, prospective dataset Drug TP FP TN FN VME (95% CI) ME (95% CI) PPV (95% CI) NPV (95% CI) Ethambutol 71 48 4171 1 1. 39(0. 2-7. 5%) 1. 14(0. 9-1. 5%) 59. 66(50. 7-68. 0%) 99. 98(99. 9-100. 0%) Isoniazid 312 19 3931 17 5. 17(3. 3-8. 1%) 0. 48(0. 3-0. 7%) 94. 26(91. 2-96. 3%) 99. 57(99. 3-99. 7%) Pyrazinamide 102 23 4047 23 18. 4(12. 6-26. 1%) 0. 57(0. 4-0. 8%) 81. 6(73. 9-87. 4%) 99. 43(99. 2-99. 6%) Rifampicin 133 35 4140 0 0. 0(0. 0-2. 8%) 0. 84(0. 6-1. 2%) 79. 17(72. 4-84. 6%) 100. 0(99. 9-100. 0%) Amikacin 3 0 59 1 25. 0(4. 6-69. 9%) 0. 0(0. 0-6. 1%) 100. 0(43. 9-100. 0%) 98. 33(91. 2-99. 7%) Capreomycin 3 0 59 0 0. 0(0. 0-56. 1%) 0. 0(0. 0-6. 1%) 100. 0(43. 9-100. 0%) 100. 0(93. 9-100. 0%) Ciprofloxacin 30 3 199 19 38. 78(26. 4-52. 7%) 1. 49(0. 5-4. 3%) 90. 91(76. 4-96.
29(17. 3-58. 7%) 0. 0(0. 0-0. 7%) 100. 0(74. 1-100. 0%) 98. 88(97. 6-99. 5%) Moxifloxacin 17 4 550 4 19. 05(7. 7-40. 0%) 0. 72(0. 3-1. 8%) 80. 95(60. 0-92. 3%) 99. 28(98. 2-99. 7%) Ofloxacin 19 3 764 6 24. 0(11. 5-43. 4%) 0. 39(0. 1-1. 1%) 86. 36(66. 7-95. 2%) 99. 22(98. 3-99. 6%) Streptomycin 340 16 1357 43 11. 23(8. 4-14. 8%) 1. 17(0. 7-1. 9%) 95. 51(92. 8-97. 2%) 96. 93(95. 9-97. 7%) (b) Global dataset Drug TP FP TN FN VME (95% CI) ME (95% CI) PPV (95% CI) NPV (95% CI) Rifampicin 2634 60 2990 136 4. 91(4. 2-5. 8%) 1. 97(1. 5-2. 5%) 97. 77(97. 1-98. 3%) 95. 65(94. 9-96. 3%) Amikacin 100 4 1496 21 17. 36(11. 6-25. 1%) 0. 27(0. 1-0. 7%) 96. 15(90. 5-98. 5%) 98. 62(97. 9-99. 1%) Capreomycin 97 25 1601 42 30. 22(23. 2-38. 3%) 1. 54(1. 0-2. 3%) 79. 51(71. 5-85. 7%) 97. 44(96. 6-98. 1%) Ciprofloxacin 1 4 33 4 80.
0(37. 6-96. 4%) 10. 81(4. 3-24. 7%) 20. 0(3. 6-62. 4%) 89. 19(75. 3-95. 7%) Kanamycin 88 3 1410 47 34. 81(27. 3-43. 2%) 0. 21(0. 1-0. 6%) 96. 7(90. 8-98. 9%) 96. 77(95. 7-97. 6%) Moxifloxacin 14 1 85 3 17. 65(6. 2-41. 0%) 1. 16(0. 2-6. 3%) 93. 33(70. 2-98. 8%) 96. 59(90. 5-98. 8%) Ofloxacin 329 19 1348 59 15. 21(12. 0-19. 1%) 1. 39(0. 9-2. 2%) 94. 54(91. 6-96. 5%) 95. 81(94. 6-96. 7%) Streptomycin 462 27 2281 217 31. 96(28. 6-35. 6%) 1. 17(0. 8-1. 7%) 94. 48(92. 1-96. 2%) 91. 31(90. 1-92. 4%) (c) Prospective dataset Drug TP FP TN FN VME (95% CI) ME (95% CI) PPV (95% CI) NPV (95% CI) Ethambutol 71 49 4170 1 1. 39(0. 2-7. 5%) 1. 16(0. 9-1. 5%) 59. 17(50. 2-67. 5%) 99. 98(99. 9-100. 0%) Isoniazid 312 19 3931 17 5. 17(3. 3-8. 1%) 0. 48(0. 3-0. 7%) 94. 26(91. 2-96. 3%) 99. 57(99. 3-99. 7%) Pyrazinamide 102 22 4048 23 18.
4(12. 6-26. 1%) 0. 54(0. 4-0. 8%) 82. 26(74. 6-88. 0%) 99. 44(99. 2-99. 6%) Rifampicin 133 32 4143 0 0. 0(0. 0-2. 8%) 0. 77(0. 5-1. 1%) 80. 61(73. 9-85. 9%) 100. 0(99. 9-100. 0%) Amikacin 3 0 59 1 25. 0(4. 6-69. 9%) 0. 0(0. 0-6. 1%) 100. 0(43. 9-100. 0%) 98. 33(91. 2-99. 7%) Capreomycin 3 0 59 0 0. 0(0. 0-56. 1%) 0. 0(0. 0-6. 1%) 100. 0(43. 9-100. 0%) 100. 0(93. 9-100. 0%) Ciprofloxacin 34 3 199 15 30. 61(19. 5-44. 5%) 1. 49(0. 5-4. 3%) 91. 89(78. 7-97. 2%) 92. 99(88. 8-95. 7%) Kanamycin 4 0 54 0 0. 0(0. 0-49. 0%) 0. 0(0. 0-6. 6%) 100. 0(51. 0-100. 0%) 100. 0(93. 4-100. 0%) Moxifloxacin 19 3 41 1 5. 0(0. 9-23. 6%) 6. 82(2. 3-18. 2%) 86. 36(66. 7-95. 2%) 97. 62(87. 7-99. 6%) Ofloxacin 21 3 42 1 4. (a) Training, (b) Global, and (c) Prospective datasets. TP, true-positive; FP, false-positive;TN, true-negative; FN, false-negative; VME, very major error rate (falsenegative rate); ME, major error rate (false-positive rate); PPV, positive predictive value; NPV, negative predictive value.