dataesr/scientific-paragraphs-categorization huggingface dataset:
Implementation in Uranie and application to the use-case The general organisation of Uranie's ANN is detailed in three steps in the following part and displayed in Figure  8. The first layer, where the vector of entries is stored, is called the input layer. The last one, on the other hand, is called the output layer while in between lies the single hidden layer, composed of n H hidden neurons. The first step is the definition of the problem: what are the input variables under study, how many neurons will be created in the hidden layer, what is the chosen activation function. The second step is the training of the ANN. Using the full database L , two mechanisms are run simultaneously: • the learning itself. By varying all the synaptic weights contained in the parameter Ξ, the aim is to produce the output set ŷ = f Ξ (x), that would be as close as possible to the output stored in L then keep the best configuration (denoted as Ξ * ).
The difference between the real output and the estimated one is measured through a loss function which could be, in the case of regression, a quadratic loss function such as L(y, ŷ) = 1 2 ||y -ŷ|| 2 From there, one can define the risk function R(Ξ) used to transform the optimal parameters search into a minimisation problem. The empirical risk function can indeed be written as R(Ξ) = 1 n S n S ∑ i=1 L(y i , f Ξ (x i )) • the regularisation. Since the ANN is trained only on the L ensemble, the surrogate model could be trained too specifically for this sub-part of the input space which might not be representative of the overall input space. To avoid this, the learning database is split into two sub-parts: one for the training (see previous bullet), and one to prevent the over-fitting to happen.
For every newly tested parameter set Ξ, the generalised error (computed as the average error over the set of points not used in the training procedure) is determined. While it is expected that the risk function is becoming smaller when the number of optimisation steps is getting higher, the generalised error is also becoming smaller at first, but then it should stabilise and even get worse. This flattening or worsening is used to stop the optimisation. This procedure is stochastic: the splitting of the L ensemble is done using a random generator, so does the initialisation of the synaptic weights for all the formal neurons. It is important then to export the constructed neural network as running twice the same methods will not give the same performances. Figure  9  represents the distribution of the thermal gauge values (as defined in Equation (  8 )) estimated by the surrogate model ( θ ) as a function of the ones computed by the complete model (θ ) in a test database containing 2000 points, not used for the training.
Introduction The modelisation relies on the assumption that the deterministic output y(x) can be written as a realisation of a gaussian process Y (x) that can be decomposed as Y (x) = m(x) + Z(x) where m(x) is the deterministic part, called hereafter deterministic trend, that describes the expectation of the process and Z(x) is the stochastic part that allows the interpolation. This method can also take into account the uncertainty coming from the measurements. In this case, the previously-written Y (x) is referred to as Y Real (x) and the gaussian process is then decomposed into Y Obs (x) = m(x) + Z(x) + ε(x), where ε(x) is the uncertainty introduced by the measurement. To construct the model from the training database L , a parametric correlation function can be chosen along with a deterministic trend (to bring more information on the behaviour of the output expectation).
Implementation in Uranie and application to the use-case The kriging approximation in Uranie is provided through the gpLib library  [21]. Based on the gaussian process properties of the kriging  [22] , this library can estimate the hyper-parameters of the chosen correlation function in several possible ways, then build the prediction model. More details about these steps are provided hereafter and in the gpLib tutorial  [23]. The first step is to construct the model from a training database L , by choosing a parametric correlation function, amongst the list below, for which l is the vector of correlation lengths and ν is the vector of regularity parameters: • Gauss: defined with one parameter per dimension, as c(δ x) = exp -∑ n X k=1 δ x k l k 2. • Isogauss: defined with one parameter only, as c(δ x) = exp -|δ x| 2 l 2.
Depending on various conditions, convergence can be difficult to achieve. Once done, the kriging surrogate model can be applied to the testing database to get predicted output values and their corresponding uncertainties. It is, however, possible, even before using a testing database, to check the specified covariance function at hand, using the Leave-One-Out technique (Loo). This method consists in the prediction of a value for y i using the rest of the known values in the training site, i. e. y 1 ,. , y i-1 , y i+1 ,. , y n S for i = 1,. , n S. From there, it is possible to use the Leave-One-Out prediction vector (y Loo i ) i=1,. , n S and the expectation ȳ to calculate two criteria: the Mean Square Error (MSE) and the quality criteria Q 2 Loo defined as MSE Loo = 1 n S n S ∑ i=1 (y i -y Loo i ) 2 and Q 2 Loo = 1 - n S ∑ i=1 (y i -y Loo i ) 2 (y i -ȳ) 2.
The first criterion should be close to 0 while, if the covariance function is correctly specified, the second one should be close to 1. Another possible test to check whether the model seems reasonnable consists in using the predictive variance vector (σ 2 y Loo i ) i=1,. , n S to look at the distribution of the ratio (y iy Loo i ) 2 /σ 2 y Loo i for every point in the training site. A good modelling should result in a standard normal distribution. The kriging technique has been applied twice to illustrate its principle and the results are gathered in Figure  10. In the first case, it is used on a monodimensional thermal gauge evolution as a function of the dimensionless time, see Figure  10a. On this figure, the black points represent the training database while the blue and red ones are respectively the real output values and their estimated counterpart from the kriging model using the testing database.
In the case where a maximum should be determined, all the techniques remained, but the objective is changed (inverted) to get back to a minimum search. In order to do so, Uranie offers many solutions thanks to its external dependencies: Minuit It is ROOT's package to perform singleobjective optimisation problem, without constraint. It provides two algorithms • Simplex: it does not use the first derivatives, it is insensitive to local optima, but without guarantee of convergence. • Migrad: a fairly sophisticated gradient descent one that is able to escape from some local optima. NLopt It is a library for nonlinear optimisation providing algorithms for single-objective optimisation problem, with or without constraint. The list of algorithms implemented in Uranie can be found in  [2]  along with a small description of their principle, taken from NLopt  [9].
Introduction The EGO technique is the perfect illustration of method combination, introduced in Section 1. 3, where all the building blocks are already implemented in Uranie. It might be useful in the case where the code is high time/cpu/memory consuming (so one wants to limit as much as possible the computation) given that there might be several local minimum in which one doesn't want to fall (preventing from using local optimisation such as gradient ones for instance). Instead of having to choose a starting point (as done in Section 5. 1. 2), the idea is to provide a training database which is supposed to be representative of the problem, so whose size cannot be too small with respect to the dimension of the ongoing analysis. Once done, a kriging model is constructed. Let f min = min(y 1 ,.
, y n t ) be the current best function value (n t being the size of the kriging training database). It is true, that before computing the output of the code for a given input vector x, we are uncertain about the value y(x). On the other hand, y(x) is not completely unknown as we can assimilate it to ŷ(x) its realisation through the kriging surrogate model, which is provided along with its standard deviation s(x). With this hypothesis, it is possible to compute the probability of the real value to be below the actual minimum f min. Different distances below the line y = f min , are associated with different density values. If we weight all these possible improvements by the associated density value, we get what we call the "expected improvement" (EI). The improvement for a given point x is I = max( f minŷ(x), 0) which is a random variable as ŷ(x) models our uncertainty about the code's value at x.
The other parameters are the coordinate of the maximum, whose measurement can be turned into an optimisation by looking for the minimum of the opposite function -h(t). This is exactly what has been done and this analysis can be summarised in Figure  25  where, in the upper part of every pad, the blue line is the real "unknown" function used, the black points are the training database, the red line is the approximation given by the kriging model along with its uncertainty and the green point is the latest point included in the training database from the previous iteration. The bottom pad of every plot shows the inverted expected improvement (-EI) which is minimised using the evolutionary algo-rithm to determine the next location to be included in the training database. Going though Figure  25 , one can find back the different steps described in Section 5.
1. 2. The idea is to compare the results given in Figure  21  to the one presented in Figure  26. A training database of 20 locations has been generated and the algorithm has been run over 32 more computations in order to get the same number of code assessments (1560, as already computed in Section 5. 1. 3). Figure  26  shows the training database (black points) and the 32 new locations computed using the EGO algorithm (blue points) along with the best minimum found (red box). The newly computed locations are split into three categories: the first ten ones in purple, the following ten ones in green and the last twelve ones in blue. Thanks to this splitting, it is possible to check that the optimisation exploration remains global as there can be green and even blue new locations computed far away from the global minimum.
Acknowledgements We thank  Miguel Brun-Usan ,  Lisandro Milocco ,  Renske Vroomans  and  Jukka Jernvall  for useful comments, and  Sophia Hagolani-Albov  for correcting grammar and spelling in the text. The  CSC (Finnish IT Center for Science)  provided computer time. We thank the  Konrad Lorenz Institute  for its support.
is released under MIT license at Mykrobe https://github. com/mykrobe-tools/mykrobe. We incorporate mutation catalogues from the CRyPTIC consortium et al. (  2018 ) and from Walker et al. (  2015 ), and make improvements based on performance on an initial set of 3206 and an independent set of 5845 Illumina M. tuberculosis sequences. To give estimates of error rates, we use a prospectively collected dataset of 4362. Using culture based M. tuberculosis isolates DST as the reference, we estimate to be 100%, 95%, 82%, 99% Mykrobe sensitive and 99%, 100%, 99%, 99% specific for rifampicin, isoniazid, pyrazinamide and ethambutol resistance prediction respectively. We benchmark against four other tools on 10207 (=5845+4362) samples, and also show that gives concordant results with nanopore data. Mykrobe We measure the ability of -based DST to guide personalized Mykrobe therapeutic regimen design in the context of complex drug susceptibility profiles, showing 94% concordance of implied regimen with that driven by phenotypic DST, higher than all other benchmarked tools.
Introduction The software tool Mykrobe predictor  1  , released in 2015, identified isolates to the species level and predicted the drug susceptibility testing (DST) profile of Staphylococcus aureus and Mycobacterium tuberculosis directly from genomic sequencing data. Mykrobe predictor was developed to address four needs. First, it provided robust genotyping of single nucleotide polymorphisms (SNPs), insertions and deletions (indel) and gene alleles, independently of the location of mutations (where "location" is used both in the sense of coordinate, and in the sense of chromosomal versus plasmid). It also did this with error rates that did not depend on the phylogenetic position of the isolate of interest, a characteristic of all reference-genome methods  2. We did this by building a de Bruijn "genome graph" representation of known variation in the species, incorporating both resistant and susceptible alleles, and also nearby variation that might affect k-mer matching (equivalent to neighbouring mutations affecting a PCR probe (see Figure  1  of 1)).
There were a number of improvements we wished to make over the 2015 Mykrobe predictor codebase. First, we wanted a cleaner codebase. Second we wanted users to be able to specify their own resistance catalogue. The latest Mykrobe predictor catalogue, updated since publication, was based on Walker et al. 3. Walker et al. derived this catalogue by analysing the same 3500 samples which were used in the initial Mykrobe predictor performance analysis  1  -thus no independent test or validation dataset were available to properly estimate error rates. Finally, we wanted to further develop the support for nanopore data which had been improved by the use of a slightly different statistical model  4  , but only tested on N = 5 replicates of a single strain of M. bovis substrain BCG. A number of other tools for detection of resistance-associated alleles have been released: ABRicate (Seemann), ARIBA  5  , ARGs-OAP  6  , ARG-ANNOT  7  , CASTB 8 , KvarQ  9  , MTBseq 10 , PhyResSe  11  , RAST  12  , ResFinder  13  , RGI  14  , SRST2  15  , SSTAR  16  , and TB-Profiler  17.
21 and some new pncA mutations from 22. It also characterised all frame-shifts in pncA and katG as causing resistance to pyrazinamide or isoniazid respectively. Using phenotyping as gold standard, they measured performance characteristics of WGS-based DST, including sensitivity and specificity, on the samples for which a prediction can be made with high confidence. They propose a workflow where a sample is sequenced, and if an "unknown significance" mutation is seen, then no resistance/susceptibility prediction is made, but the sample is instead referred for phenotyping. This provides a means whereby healthcare systems can leverage the advantages of WGS for rapid and comprehensive DST, be informed when a given sample is pushing against the limits of our knowledge, and revert to slower but well-understood and trusted phenotyping in order to provide clinicians with reliable results, and prospectively enrich available databases.
In this study we evaluate our new lightweight offline species-identification and resistance prediction tool, named simply Mykrobe 23. The species-identification methodology is unchanged since 1 , but after publication Mykrobe predictor was heavily evaluated by Public Health England, and in order to pass their acceptance criteria, we improved our species-informative probes. We report the results of applying these prospectively at PHE below, but this study primarily concerns improvements in resistance prediction. We first combine the resistance catalogue from The CRyPTIC Consortium et al. 20  with the pre-existing second-line drug resistance catalogue from Walker et al. 3. We treat this catalogue as our first candidate catalogue and improve it in an iterative process, evaluating it on successive datasets.
Drug resistance prediction Data. We re-use the sequencing and phenotype data from 1,3 and 20; the latter included only first-line phenotypes, so in this study we augment that data with second-line phenotype data. We also remove in advance those samples which were identified as likely sample-swaps in 20. We split the samples into three disjoint datasets: • Training set: N = 3206 isolates, from 1 • Prospective set: N = 4362 isolates from 20; specifically these were the only isolates that were sampled prospectively (from Italy, Germany, the Netherlands and the UK) to enable realistic error-estimates. • Global set: N = 5845, the remaining isolates from 20. See Figure  1  and Source data sample_data. tsv  29 for a per-country breakdown of the datasets.
Initial development of resistance panel. We started with a mutation catalogue (candidate panel 1, CP1) built by combining those from 20 for first-line drugs and 3 for second-line drugs. An iterative process was used to remove variants from the panel after running Mykrobe on each dataset (see Methods and Figure  2  for details). First, CP1 was evaluated on the Training set. This resulted in the removal of six variants from the panel which had positive predictive value less than 5%, producing candidate panel 2, CP2 (see Methods for details). CP1 is given in Source data panel. CP1. tsv 30 , panel. CP1. json 31 , and the six variants removed from CP1 to generate CP2 are in Source data removed_variants. tsv  32. We then measured the performance of Mykrobe with this panel on the large Global dataset.
Resistance prediction performance on Global dataset. We show in Figure  3  a comparison of three sets of predictions on the Global dataset: those from 20, those from Mykrobe using the panel ("Walker-2015") used by our previous software Mykrobe predictor, and those from Mykrobe using CP2 (complete results are in Extended data file accuracy_stats. tsv  33 ). The greatest difference in performance based on Walker-2015 and CP2 is in pyrazinamide, where sensitivity increases from 25% in Walker-2015 to 74% in CP2, at the cost of specificity falling from 98% to 94%. Numerous rare variants were the source of reduced pyrazinamide specificity in CP2, most of which also cause true-positives (Extended data file variant_counts. tsv 34 ). However, 21 of these variants resulted in 29 false-positives and no true-positives and were therefore removed from CP2 to make CP3.
We did however notice that of these 21, 16 were in the resistance catalogue from 22, which provides direct experimental evidence for their impact, and so we may reinstate them in future given further data. The difference between Walker-2015 (86% sensitivity, 89% specificity) and CP2 (84% sensitivity, 89% specificity) for ethambutol is explained by two variants in the embB gene. The first site, M306, is included in all panels. However, in Walker-2015 it is present as any non-synonymous change, whereas in NEJM-2018 (and hence CP1 and CP2) only the more specific M306I and M306V are included. The variant M306L (in all samples    respectively. These results can be seen in Table  1b. We discuss below how the approach taken in 20 improved on this VME despite using essentially the same catalogue. Resistance prediction performance on Prospective dataset Different resistance mutations occur at varying frequencies across the world.
Therefore, in order for an error-rate estimate to be generalisable into clinical practise, the underlying data needs to be sampled in a way that is representative of prevalence somewhere. We used the Prospective dataset to get realistic estimates of error rates in the (low burden) countries from which these data were sampled. We show in Figure  4  a comparison of the predictions for that dataset from 20, Mykrobe using the old Mykrobe predictor panel (Mykrobe-2015)  3  , and Mykrobe with CP3. Sensitivity, specificity, and error rates are shown in Table  1c. Very Major Error rate (missed resistance) for the first-line drugs was 0. 0%, 5. 2%, 1. 4%, 18. 4% for rifampicin, isoniazid, ethambutol, pyrazinamide respectively, with corresponding negative predictive values of 100. 0%, 99. 6%, 100. 0%, 99.
01) and second, coverage on minor allele due to a minor population at frequency 0. 2. In practise, with coverage around 30-50x, this results in a minor population being detected when has frequency above around 8-10%. Having made minor changes to the catalogue from CP1 to CP2, we measured performance on the Global dataset, to finalise catalog and parameters (described below) before evaluating on the (final) Prospective dataset. The one remaining parameter that could be changed was the frequency of minor allele in the second model above, set to 0. 2. We measured the impact of resistance calls driven by minor alleles using CP2 on the Global set -results are shown in Table  3. These results are broadly in concordance with those reported in 1 on the Training set. For second line drugs, the use of minor variants increases sensitivity without impacting false positive rate.
In contrast to the above, for second-line drugs the pattern is different, with lower error rates for minordriven calls. The number of samples with second-line phenotypes is too low to draw strong conclusions. We would like to see these results replicated with comprehensive phenotyping and deep sequencing. In theory the Mykrobe statistical test described above could have been modified to use a second model with higher minor-allele frequency than the current value of 0. 2 -essentially demanding a higher frequency to reduce false positives. Indeed, it might be possible to fit different thresholds per drug, improving results on our dataset. However, we had concerns about generalisability. A recent publication  39  showed how sequencing the same isolate with Illumina HiSeq, MiSeq and NextSeq could give different pictures of minor allele variation, including very different minor allele frequencies.
Evaluating policy of ignoring unknown mutations in resistance genes A key component of the approach taken by the CRyPTIC consortium in 20 was to refuse to make predictions if an unknown mutation was detected in a resistance-associated gene, and divert such samples to phenotyping. By contrast, Mykrobe genotypes known SNPs and indels, but does not detect novel mutations, and makes predictions of phenotypes based on these data only. In particular, if no resistance mutation from its catalogue is detected, Mykrobe makes a Susceptible prediction. Our prior expectation was that this would come at the cost of a higher false susceptible rate for Mykrobe. This was indeed the case -see the red components of the bars on the right hand side of Figure  3. However, in addition, on both the Global and Prospective datasets, we find that Mykrobe calls more true positives and negatives (i.
Finally, we recall again that the global set is a heterogeneous combination of datasets sampled in different ways. The prospective set is prospectively sampled, though from low-burden countries in Europe. On that dataset we find the Mykrobe approach pays far less penalty, missing 0, 9, 2 and 17 samples resistant to rifampicin, isoniazid, ethambutol and pyrazinamide, respectively, while detecting 215, 184, 486, 76 susceptible samples earlier (without phenotyping).
Benchmarking against other tools In order to separate the effect of mutation catalogue and genotyping method, we gave ARIBA the same catalogue (namely the final release panel described above) as Mykrobe (with small differences, see Methods for details). For the other tools, we simply used their default catalogues in their latest versions at the time of benchmarking. The sensitivity, specificity, and error rates for Mykrobe, ARIBA, KvarQ, MTBseq, and TB-Profiler when applied to the Prospective dataset, summarised across all drugs, is given in Table  5  (full data are in the Extended data file accuracy_stats. tsv  33 ). On each of the three datasets, Mykrobe is the most sensitive, has the lowest very major error rate (false susceptible error rate), and the highest negative predictive value. Mykrobe has the best specificity, major error rate (false resistant rate) and positive predictive value on the Training dataset, and the best or second-best results for those statistics on the Global and Prospective datasets.