dataesr/scientific-paragraphs-categorization huggingface dataset:
Data synthesis and statistical analysis analysis will be conducted using Stata software version 16. 1 (Stata Corp. , College Station, Texas). The Kolmogorov-Smirnov test will be used to check the normality of the extracted data. Descriptive statistics will be used to describe the characteristics of all eligible studies. Normally distributed data will be presented as mean (standard deviation), and non-parametric data will be expressed as median (interquartile range). Categorical data will be presented as proportions (%). If the studies retrieved report quantitative data that can be combined, the extracted data will be aggregated into a meta-analysis. Heterogeneity will be assessed using the Q and I 2 statistics. A p-value <0. 10 or an I2 >50% suggests that statistical heterogeneity may exist (16).
The CRyPTIC project designed a variant of the MYCOTB plate, called UKMYC5, that contains 14 antibiotics, including bedaquiline (BDQ), delamanid (DLM), clofazimine (CFZ) and linezolid (LZD) but not pyrazinamide (PZA) (figure  1a ). Based on a multi-laboratory study that examined the inter- Each country is coloured depending on the number of originating samples using a logarithmic scale. PAS: para-aminosalicylic acid. and intra-laboratory reproducibility of the UKMYC5 plate, and determined the optimum reading methods and incubation period  [17] , CRyPTIC subsequently modified the design by removing para-aminosalicylic acid (PAS) and extending/changing the concentration of certain drugs, leading to the 13-drug UKMYC6 plate (figure  1b ). In this article we propose epidemiological cut-off values (ECOFF/ECVs) for the UKMYC series of plates to enable subsequent research on this dataset.
The ECOFF/ECV is the highest MIC observed within a phenotypically wild-type ( pWT) population, usually defined as the MIC which encompasses 99% of that population  [20] , and allows interpretation of an MIC value as "susceptible" or "resistant", which is crucial to the decision on whether to prescribe a drug. The standard approach requires uncensored MICs and assumes that the pWT population can be readily identified, either because the population has been minimally exposed to the drug or because the MIC distribution is strongly bimodal. These conditions are not universally met in our dataset and we shall therefore identify a genotypically wild-type (gWT) population from which we can either measure the ECOFF/ECV directly or via a Gaussian fitted using interval regression, a statistical technique that can fit to censored data.
Since the magnitude of MIC measurement error is anticipated to be much greater than the error in the genetic sequencing, we constructed an MIC quality assurance (QA) process to minimise the measurement error of the MICs (supplementary figure  S3 ). This measured each MIC using up to three independent methods and only MICs where two of these methods concur are allowed into the final dataset. Overall, 77% of all MIC measurements passed the MIC QA process.
MIC histograms are different for different drugs As expected, the MIC histograms differ between drugs (figure  2 , and supplementary figures S5 and S6); the MICs for some compounds form bimodal distributions (isoniazid (INH), kanamycin (KAN), amikacin (AMI), rifampicin (RIF) and rifabutin (RFB)) and therefore conform to the classical binary paradigm whereby an isolate is either "resistant" or "susceptible". CRyPTIC aimed for half the isolates collected to be multidrug resistant (MDR), and the MIC histograms for INH and RIF are consistent with this. Given this bias towards MDR in the dataset, one would expect appreciable resistance to ethambutol (EMB), ethionamide (ETH), both fluoroquinolones and both aminoglycosides. Both drugs belonging to the latter class indeed have a subset of isolates with very high MICs.
The MIC histograms for the remaining compounds (EMB, ETH, moxifloxacin (MXF) and levofloxacin (LEV)) are not bimodal, hence it is unclear whether they can be adequately described by two log-normal distributions. Since the remaining drugs on the plates (BDQ, DLM, CFZ and LZD) have not yet been widely used, and for some countries were not even available to treat TB, one expects little resistance in the dataset and hence it is likely these MIC histograms are "phenotypically wild-type" ( pWT). All the MIC histograms are truncated/censored at either one or both ends and some are severely truncated with the mode MIC occurring in the lowest dilution (AMI, RFB and DLM). Our large dataset allows us to use reproducible, algorithmic approaches for estimating the 99th percentile of the wild-type population.
In addition, since ECOFFinder requires a single consistent MIC distribution and our dataset is composed of two plate designs, two ECOFF/ECVs are returned for each drug. For many drugs these are very similar, but for EMB and DLM the estimates are almost a doubling dilution different. To overcome the problem that our MIC histograms are truncated we applied interval regression, an established statistical method for fitting normal distributions to truncated data  [26, 27]  which, unlike conventional maximum-likelihood algorithms, takes into account that observations are properly represented by intervals. The entire dataset containing measurements from both plate designs can then be considered simultaneously, resulting in a single pair of log-normal distributions that describe the MIC histograms on both plate designs (supplementary figure  S7 ).
The histograms can be reproduced online  [21]. photograph taken of the UKMYC plate after 14 days incubation so we could run the QA process. As a result, isolates from only nine CRyPTIC laboratories made up this dataset (supplementary table  S8 ), and the number of confirmed MICs varied between 2594 and 4078 by drug, with a mean of 3263 (supplementary table  S9 ). Visually, the resulting MIC histograms are simpler and more likely to be adequately described by a single log-normal distribution (supplementary figure  S8 ). Directly measuring the ECOFF/ECV from the gWT population Directly determining the MIC of the 99th percentile from the gWT population is an attractive option since it requires no further assumptions. This is not usually possible since typically either one cannot discern the wild-type population and/or there are an insufficient number of isolates.
The large size of our dataset and the inclusion of genetic information enables us to directly measure the ECOFF/ECV (figure  3  and supplementary figure  S9 ). Our dataset is enriched for resistance, hence the proportion of resistant samples misclassified as susceptible due to sample mislabelling is likely to be of the order of a few percentage points, even after we have removed some putative mislabelled samples (Methods). This makes directly identifying the 99th percentile challenging, hence we shall also consider the 97. 5th and 95th percentiles. All three percentiles for the MIC histograms of the gWT population are at most two doubling dilutions apart, except for LEV (UKMYC5) and INH (UKMYC6). The latter has an appreciable number of isolates that, despite being classified as gWT, have elevated MICs.
These are likely due to some remaining samples that were mislabelled and illustrates the difficulty in using the 99th percentile to define an ECOFF/ECV due to its sensitivity to errors in the dataset, especially when the prevalence of resistance is high, as in the case for INH in our dataset. We shall take forward the values for the 99th percentiles (table 2) but will bear in mind that a high amount of variation may indicate strain mislabelling.
5th and 99th percentiles are all shown. See figure  1  for drug abbreviations. The analysis and figure can be reproduced online  [21]. The 99th percentile determined by direct measurement for INH for the UKMYC6 dataset was discounted due to the large range of MICs spanned by the percentiles; this is most likely due to resistant samples misidentified as susceptible due to laboratory mislabelling. INH has the highest prevalence of resistance in the dataset and hence would be affected most. Our starting point is therefore the corresponding value for the UKMYC5 dataset (0. 1 mg•L -1 ). The ECOFFinder results were ignored for INH since the gWT MIC histogram is truncated on both plate designs and hence the fits were poor. Visually, the gWT MIC histograms (supplementary figure  S8 ) do not appear to follow a log-normal distribution and consequently    FIGURE 4  Interval regression is able to fit a log-normal distribution to the minimum inhibitory concentration (MIC) histograms of the genotypically wild-type (gWT) isolates for all 13 drugs on the UKMYC6 plate.
Data from both plate designs were considered simultaneously, hence the resulting distributions are those the algorithm considers to best describe both the UKMYC5 (supplementary figure  S10 ) and UKMYC6 datasets. See figure  1  for drug abbreviations. See the TSV file in the supplementary material for the numerical data. The data can be reproduced online  [21]. interval regression overestimates the 99th percentile (figure  4  and supplementary figure  S10 ). The ECOFF/ ECV of 0. 1 mg•L -1 for INH is therefore less well supported than the values for the remaining drugs. Direct measurement of the 99th percentiles for RIF were 0. 5 and 0. 25 mg•L -1 for the UKMYC6 and UKMYC5 datasets, respectively. Again, ECOFFinder was not used due to concerns about truncation. Visually, the gWT MIC histogram appears more normal in character and the 99th percentile derived from the interval regression fit is 0.
52 mg•L -1. Our proposed consensus ECOFF/ECV for RIF is hence 0. 5 mg•L -1. Direct measurement produced a consistent value of 4 mg•L -1 for EMB, which is supported by interval regression and ECOFFinder for the UKMYC5 dataset (the concentration range on the UKMYC6 plate was more truncated). Both fluoroquinolones behaved similarly: direct measurement gave a value of 1 mg•L -1 for the 99th percentile for both compounds for the UKMYC6 dataset, but 2 and 4 mg•L -1 for the UKMYC5 dataset (MXF and LEV, respectively). These values for the latter dataset were very sensitive to the exact percentile used in the definition (supplementary figure  S9 ), again suggesting that these gWT populations may contain a small number of mislabelled resistant samples. Both drugs have the same concentration range on both plate designs, are only moderately truncated and hence ECOFFinder would be expected to give reasonable results.
These, along with the result of the interval regression (figure  5 ), result in an ECOFF/ECV of 1 mg•L -1 for both fluoroquinolones. Direct measurement indicates the 99th percentile for KAN is 8 and 4 mg•L -1 for the UKMYC6 and UKMYC5 datasets, respectively, while it produces the consistent value of 1 mg•L -1 for AMI. The MIC histogram of the latter is too truncated for ECOFFinder to function correctly and visually the log-normal fitted by interval regression appears to have overestimated the 99th percentile as 1. 6 mg•L -1 ; hence, we propose an ECOFF/ECV for AMI of 1 mg•L -1. The KAN MIC histograms are less truncated and interval regression better describes the gWT population; these data support an ECOFF/ECV of 4 mg•L -1 for KAN. For ETH, direct measurement produces a consistent value of 4 mg•L -1 , which is supported by interval regression and ECOFFinder.
The MIC histogram of RFB is extremely truncated and hence only direct measurement is likely to be effective; it estimates 0. 12 mg•L -1 to be the 99th percentile for both datasets, which is therefore our ECOFF/ECV. Direct measurement yields consistent values of 0. 25 and 1 mg•L -1 for BDQ and LZD, respectively, with each supported by both interval regression and ECOFFinder. Our ECOFF/ECV for DLM is 0. 12 mg•L -1 since this is the direct measurement which is the same for both datasets and it is supported by interval regression. Lastly, direct measurement for CFZ suggests the 99th percentiles are 0. 25 and 0. 5 mg•L -1 for the UKMYC6 and UKMYC5 datasets, respectively. The latter has more variation and hence we propose its ECOFF/ECV is 0. 25 mg•L -1.
Discussion We have proposed ECOFF/ECVs for research use for 13 different anti-TB compounds for the UKMYC series of broth microdilution plates using an aggregated dataset of 20 637 TB samples collected worldwide by 14 CRyPTIC laboratories based in 11 countries on five continents. The UKMYC6 plate design (figure  1b ) not only contains the first-line drugs RIF, INH and EMB, but also all of the Group A medicines, one of the two Group B medicines (CFZ) and five of the seven Group C medicines recommended by the WHO for treating cases of MDR-TB  [19]. As such these plates offer near comprehensive phenotypic AST as well as a standardised, scalable phenotype that, once analysed with linked genomic data, will inform clinical decisions where routine diagnostics switch to genome sequencing. We caution that while the ECOFF/ECVs proposed herein have been derived using the largest collection of M.
tuberculosis samples to date, the methods do not conform with those laid out by EUCAST. That said, our analyses illustrate that the EUCAST definition of an ECOFF/ECV as the 99th percentile of the wild-type population  [20]  is difficult to apply in practice since firstly it is not always possible to define which isolates are "phenotypically wild-type" (pWT) without engaging in a circular argument. We were able to avoid this here by defining a "genotypically wild-type" (gWT) population. The second problem is that using the 99th percentile to define the ECOFF/ECV places a very stringent upper limit on the total error rate, which becomes harder to meet as the prevalence of resistance in any dataset increases. Despite our efforts, we see evidence that our gWT populations for some compounds contain >1% resistant isolates for some drugs which, for example, hampers the use of direct measurement.
One can deconstruct the error in determining an ECOFF/ECV using a microtitre plate into sample selection biases, data entry and labelling errors, inoculation and incubation errors, measurement errors, errors in defining the wild-type population, uncertainties arising from censored data, and errors in fitting a curve to the resulting MIC histogram. In addition to the obvious benefits in collecting such a large and diverse dataset (table  1 , figure  1  and supplementary table  S2 ), we have been careful to minimise measurement errors (supplementary figure  S3 ) and have also used a principled method to attempt to remove some putative mislabelled samples (Methods). By defining a gWT population and either applying interval regression to fit normal distributions (figure  4 ) or directly measuring the 99th percentile (figure  5 ), we have also minimised the final two sources of error.
Despite these steps, further sources of error no doubt remain. Another key weakness of this study is the lack of PZA, which due to its preference for acidic conditions, is currently unable to be successfully incorporated onto broth microdilution plates, although there is hope that this could be rectified in future  [48]. The debate about how to define and calculate ECOFF/ECVs will continue and new approaches will be suggested  [49] [50] [51] [52]. However it evolves, larger and more geographically diverse TB datasets, such as presented here, will bring more confidence and rigour to the AST of clinical TB samples. We hope also, that as clinical microbiology transitions into a data-driven science, our proposed method of directly measuring the required percentile of the gWT population will gain traction due to its simplicity and reproducibility as the genetics of M.
In future work the CRyPTIC project will apply the ECOFF/ECVs proposed here not only to further optimise a genetic catalogue for the first-line anti-TB compounds  [6]  but also to extend coverage to second-line, repurposed and new compounds, with the aim of covering as many of the drugs recommended by the WHO for treating MDR-and XDR-TB  [19]. Clearly the numerical data being collected by the CRyPTIC Consortium also lends itself to the development of a genetic catalogue for anti-TB compounds that can make quantitative predictions; such a catalogue would naturally take account of additivity, epistasis and non-linear effects. Finally, the tools and data-driven approaches developed here could be applied to other pathogens, especially other mycobacteria.
Genetic sequencing and interpretation Sequencing arrangements differed slightly between each CRyPTIC participating laboratory. All sequencing was performed using Illumina machines and hence the input to our genetic sequencing pipelines was a matched pair of FASTQ files containing the short reads. Data integrity was ensured throughout by tracking the MD5SUM hashs of the FASTQ files. Human and HIV reads were removed from the raw sequence data as follows. Reads were mapped to the reference genome H37Rv, the human genome version GRC38, the HIV reference NC_001802. Samples were excluded from the dataset if they had either >100 000 unfiltered SAMtools variant calls (a weak filter applied to detect samples contaminated with the wrong species) or an average read coverage of ⩽15 when mapped to reads covering the H37Rv reference.
The samples that pass these criteria and have paired phenotype data are named the GPI (geno-pheno intersection). Variant calls were removed if they overlapped a set of masked positions as previously defined  [55]. This mask consists of 324 971 positions from the H37Rv reference with self-blast matches (https://github. com/iqbal-lab-org/cryptic_tb_callable_ mask/commit/43ec21319209b23f648f32e4868bdf07cf09f2a0). Version 3 of the H37Rv strain (NC_000962. 3) was used as the TB reference genome throughout. The resulting VCF files were then transferred to the CRyPTIC data warehouse where they were interpreted.
Data warehousing With the exception of the compressed FASTQ files, all data (VCFs, images, MIC metadata, genetic variants and catalogue predictions) were aggregated and stored in a hierarchical file system using the Python datreant 1. 0. 2 module  [59] , which allowed for data discovery, tagging and filtering. Updates were performed by inhouse Python scripts. Plate metadata was downloaded from CliRes2 using the zeep Python SOAP client.
QA of MIC readings Central to our QA process is the photograph taken of the plate after 14 days of incubation using the Vizion instrument by the laboratory scientist. Images were deduplicated by checking the MD5SUM was unique. The remaining images were first read by bespoke software, AMyGDA  [60, 61] , which detects the locations of the wells and, by measuring the growth in each well, estimates an MIC for all drugs. For 54. 7% of all measurements the MICs measured by the laboratory scientist and AMyGDA were identical (supplementary figures S4 and S5a), and therefore passed the QA process. Images of the 45. 3% of cases where these two methods disagreed were uploaded to a Citizen Science project, hosted by the Zooniverse platform, called BashTheBug  [62]. Each image was classified by at least 11 different volunteers and the median reading was taken to be the consensus.
Overall, therefore, we have a high degree of confidence in 77. 4% of the MIC measurements since two or more independent methods concur on the value. Finally, in 22. 6% of cases all three methods gave a different answer (supplementary figure  S5d ); these are excluded from further analysis. All these proportions are averaged over all drugs; there is significant variation between drugs (supplementary tables S7 and S8). Putative mislabelled samples Some 44 samples were assumed to be mislabelled samples as defined by being gWT but having both an INH MIC ⩾1. 6 mg•L -1 and an RIF MIC ⩾4 mg•L -1. This corresponds to 0. 8% of the dataset, which is likely an underestimate. All 44 samples were removed.
Data analysis and graphs All data analysis, with the exception of the interval regressions and ECOFFinder, was performed using Python 3. 8 in conjunction with Pandas 1. 2. 1  [63]  and NumPy 1. 19. 5  [64]. Graphs were plotted using Matplotlib 3. 3. 4 [65]  and GeoPandas 0. 8. 2 (https://geopandas. org).
Reproducibility The raw data ( photographs of 96-well plates, genetic variant call files) along with a series of data tables related by a schema can be downloaded from the European Bioinformatics Institute at http://ftp. ebi. ac. uk/ pub/databases/cryptic/. In addition, one can reproduce nearly all the tables and figures, along with the supplementary data, in a browser window (i. e. no installation required) using Python code we have made publicly available  [21].
Acknowledgements: We are grateful to  Claudio Köser  ( University of Cambridge, Cambridge, UK ) for providing the ECOFFinder programme and for helpful comments, the EUCAST ESGMYC subcommittee chaired by  Emmanuelle Cambau , especially  John Turnidge , for helpful discussions, and all the BashTheBug volunteers for the time and energy they have contributed. We thank  Faisal Masood Khanzada  and  Alamdar Hussain Rizvi  ( NTRL, Islamabad, Pakistan ),  Angela Starks  and  James Posey (Centers for Disease Control and Prevention, Atlanta, GA, USA) , and  Juan Carlos Toro  and  Solomon Ghebremichael  ( Public Health Agency of Sweden, Solna, Sweden). CRyPTIC Consortium members and affiliations:  Philip W. Fowler  ( University of Oxford, Oxford, UK ),  Ivan Barilar  ( Research Center Borstel, Borstel, Germany ),  Simone Battaglia  ( IRCCS San Raffaele Scientific Institute, Milan, Italy ),  Emanuele Borroni  ( IRCCS San Raffaele Scientific Institute, Milan, Italy ),  Angela Pires Brandao  ( Oswaldo Cruz Foundation ,  Rio de Janeiro, Brazil and Institute Adolfo Lutz, São Paulo, Brazil) ,  Alice Brankin  ( University of Oxford, Oxford, UK ),  Andrea Maurizio Cabibbe  ( IRCCS San Raffaele Scientific Institute, Milan, Italy ),  Joshua Carter  ( Stanford University School of Medicine, Stanford, CA, USA ),  Daniela Maria Cirillo  ( IRCCS San Raffaele Scientific Institute, Milan, Italy ),  Pauline Claxton  ( Scottish Mycobacteria Reference Laboratory, Edinburgh, UK ),  David A.
R. Siqueira Oliveira  is supported by  FAPESP  ( 17/16082-7 ). L. Ferrazoli received financial support from  FAPESP  ( 2012/ 51756-5 ). B. Zhu is supported by the  National Natural Science Foundation of China  ( 81991534 ) and the  Beijing Municipal Science and Technology Commission  ( Z201100005520041 ). N. T. Thuong is supported by the  Wellcome Trust   International Intermediate Fellowship  ( 206724/Z/17/Z ). G. Thwaites is funded by the  Wellcome Trust. R. Warren  is supported by the  South African Medical Research Council. J. Carter  is supported by the  Rhodes Trust  and  Stanford   Medical Scientist Training Program  ( T32 GM007365 ). A. Lalvani is supported by the  NIHR Health Protection Research Unit  in Respiratory Infections at  Imperial College London. S. Grandjean Lapierre is supported by the  Fonds de Recherche en Santé du Québec.
C. Nimmo  is funded by the  Wellcome Trust  ( 203583/Z/16/Z ). A. Van Rie is supported by  Research Foundation Flanders (FWO)  under grant number  G0F8316N  ( FWO Odysseus ). G. Meintjes was supported by the  Wellcome Trust  ( 098316 ,  214321/Z/18/Z  and  203135/Z/16/Z ), and the  South African Research Chairs Initiative of the Dept of Science and Technology and National Research Foundation (NRF) of South Africa  ( 64787 ). The funders had no role in the study design, data collection, data analysis, data interpretation or writing of this report. The opinions, findings and conclusions expressed in this manuscript reflect those of the authors alone. L. Grandjean was supported by the  Wellcome Trust  ( 201470/Z/16/Z ), the  National Institute of Allergy and Infectious Diseases of the National Institutes of Health  ( 1R01AI146338 ), the  GOSH Charity  ( VC0921 ) and the  GOSH/ICH BRC  (www.