dataesr/scientific-paragraphs-categorization huggingface dataset:
The baseline system was based on the Indri index with stemming and without stop words (language model). Part of speech tagging was performed by TreeTagger. The summarization algorithm was TermWatch  [21]. In 2012, the text corpus was presented by an updated Wikipedia dump from November 2011. The query set was dramatically changed. It consisted of approximately 1000 real tweets written in English collected from informative accounts such as @CNN, @TennisTweets, @PeopleMag, @science etc. However, the task remained the same: to provide a summary up to 500 words in the TREC format. In 2013 there were 598 tweets in English to be contextualized from the Wikipedia dump of November 2012. For all test collections, 50 tweets 4  were selected to evaluate the informativeness of the summaries  [21]. For each of those topics, all submitted passages were merged into a pool.
INEX Snippet Retrieval Task Search engines return to a user an immense volume of results that it is impossible to read. Therefore, to define whether a web page is relevant to a query without clicking on a link, a search engine provides a user with snippets. Snippets are small text passages appearing under every search result. For the Snippet Retrieval Track 2012, the data collection consisted of the dump of the Wikipedia of October 2008 annotated with YAGO  [23]  and 35 topics. Participants should provide 20 snippets per topic limited to 180 characters  [24]. In 2013, the Snippet Retrieval track was using the same document collection as the Tweet Contextualisation track, based on a dump of the English Wikipedia from November 2012. The description, and narrative (intent) provided the idea of the user information need.
g. jackknifing (use of the subsets of the reference summaries, each of which is missing one reference) or bootstrapping (random replacement of points in the data set). This is important, since comparing a reference to itself leads to the maximal score, but ignoring it results into different number of references. In case of re-sampling, the final score is the mean of all computed values  [36]. In  [8] , Louis and Nenkova suggested an automatic approach for summary evaluation without a gold standard. Instead of a set of reference summaries, a full text is used. The authors suggested to estimate summary scores by Kullback-Leibler divergence, Jensen Shannon divergence, and cosine similarity measure. Although these metrics have some correlation with traditional gold-standard based ROUGE scores, ROUGE-1 demonstrated better results.
Scientific Articles. For our experiments we used the data from the ISTEX (Excellence Initiative of Scientific and Technical Information) platform  6   The second data set we used was a cleaned recent English Wikipedia XML dump created by the INEX organizers for Tweet Contextualization Track  [41]. All notes, history and bibliographic references were removed. Thus, a page was composed of a title (title), an abstract (a) and sections (s). A section had a header (h). Abstract and sections contained paragraphs (p) and entities (t) referring to other pages. We selected the 100,000 longest articles. However, only 43,611 articles had both abstracts and not-empty sections and were kept for experiments. This data set is referred as Wikipedia.
INEX has a high accuracy on ISTEX data but very poor accuracy on other collections. In all cases, the difference with the random baseline is significant. The table testifies that GRAD significantly outperforms the baseline assigning score randomly as well as the state-of-the-art vocabulary overlap based metrics (INEX, ROUGE). Since the used methods for summary generation are the extractive ones, the metrics based on the overlap between terms should have a low performance. ROUGE and COS are in average twice as worse as the random baseline on both test collections. Although, the INEX measure showed quite competitive results on the ISTEX data set, on the Wikipedia and environmental science articles its results are much lower than the results obtained by RAND baseline. Thus, we can conclude that full texts can not serve as a gold standard especially in case of extractive summaries since in this case the measures are mainly reduced to the ratio of the size of a summary over the one of a full text.
Ability to Find the Native Summary among Summaries of Different Texts The second experiment aimed at evaluating the capacity of metrics to identify the appropriate abstract among human written summaries and follows the second part of the framework we suggested. We performed pair-wise comparison between each of 31 documents and abstracts coming from different texts (31 × 31 = 961 comparisons, 496 different text pairs). These documents on environmental sciences were selected from the ISTEX database. We used the same documents as in  [9]. In Figures  1 to 4  we present the results of the subtraction of the score assigned to original abstracts and and the score of abstracts coming from the other texts when considering these 961 cross-text comparisons and various metrics. In these figures, if an abstract coming from another text is preferred by a metric over the native abstract, the point is below the zero line.
Discussion and Conclusion 670 In this paper, we discuss a wide range of evaluation frameworks for automatic summarization, including data sets and metrics used, as well as main competitions in the field, for the last 18 years. Manual assessment is time and money consuming and, therefore, not applicable to real time scenarios or large collections. The expert nature of metrics, such as responsiveness, retention rate, metrics used in INEX Snippet Retrieval track, makes it impossible for re-use, since manual assessment requires expert intervention for 675 each summary under evaluation. Moreover, on average, assessment agreement is about 70% due to the fact that judges may have different opinions about summary quality and evaluation metrics  [29]. Inter-rater agreement is traditionally measured by Cohen's kappa  [73]  or Krippendorff's alpha in the case of an arbitrary number of coders  [74].
The use of the full text rather than a set of reference summaries for summary evaluation provides poor results  [8]  since traditional overlap-based metrics are mainly reduced to the ratio of the size of a summary to the size of a full text since they are designed to be compared with summaries created by humans. To provide evidence, we proposed a completely automatic framework for the evaluation of metrics to assess automatically produced summaries which does not require any human annotation. We conducted experiments on Wikipedia data set and a collection of scientific articles from the ISTEX database. The obtained results show that metrics based on vocabulary overlap are not suitable for measuring the quality of a summary with regard to a full text. However, the GRAD measure significantly outperformed overlapbased baselines on both test collections in distinguishing human written abstracts from generated summaries of poor quality.
that contains collections of scientific literature in all disciplines covering journal archives, digital books, databases, texts corpora etc. from the following publishers: Elsevier, Wiley, Springer, Oxford University Press, British Medical Journal, IOP Publishing, Nature, Royal Society of Chemistry, De Gruyter, Ecco Press, Emerald, Brill, Early English Books Online. We collected articles that contain authors abstracts, editorial or/and web summaries. We selected 4,234 full texts in TXT format with corresponding summaries provided as XML descriptions. We call this data set ISTEX. We also selected 4,535 articles with abstracts about environmental sciences from ISTEX (referred as Environmental). ISTEX documents are labeled by Web of Science categories. So, we retrieved documents labeled Environmental Studies or Environmental Science.
Wood proposes to call the Hordwell alligator A. Hantoniensis'  (Wood, 1844: p. 351). A second account of Hordle Cliff was published by  Wood (1846: p. 6 ), in which he reported, under the genus Alligator, that: 'A considerable portion of the skeleton of a crocodile, resembling the New World type, for which I propose the specific name Hantoniensis'. Wood (1846: pls 1, 6, 7 ) also figured some cranial remains, a femur and a cervical vertebra, under the name 'Alligator Hantoniensis'. Because the 1844 publication lacks an accession number or figures to identify the material,  Wood (1846)  is the accepted authority for the name Alligator hantoniensis. Both the Marchioness of Hastings and Owen presented new crocodylian remains from Hordle Cliff at the Seventeenth Meeting of the British Association for the Advancement of Science held at Oxford in June 1847, which were published as reports the following year  (Hastings, 1848; Owen, 1848).
, 2011 , were scored based on Blanco et al. (2015) , and we also updated scores for Allodaposuchus precedens based on  Martin et al. (2016). Eight additional characters (C180-187) from  Brochu & Storrs (2012)  were also added to the character list. One new character state was added to each of characters 146 and 147 following  Delfino et al. (2008) , to better characterize the unique morphology of the posterior margin of the otic aperture in allodaposuchids. Our revised dataset comprises 103 taxa scored for 187 characters. The data matrix (nexus and tnt files), and character list are provided in Supplementary Information 1-3 respectively.
Because homoplastic characters might or might not bear a phylogenetic signal, or are characters that evolve faster than nonhomoplastic ones, downweighting them in respect to highly phylogenetically significant ones appears to be a logical solution  (Farris, 1969; Goloboff, 1993; Chippindale & Wiens, 1994; Haszprunar, 1998). Indeed, the inclusion of downweighted homoplastic characters has been shown to increase phylogenetic accuracy both in simulations and with real morphological and molecular datasets (e. g. Chippindale & Wiens, 1994; Källersjö et al. , 1998; Goloboff et al. , 2008 Goloboff et al. , , 2018;; Goloboff, 2014). Given that implied weighting, as implemented in the software TNT, provides the only repeatable and user-friendly methodology to date, we applied this strategy herein. However, traditional implied weighting can be negatively influenced by missing data, especially in cases where homoplastic characters are only scored in a few operational taxonomic units (OTUs).
The lack of information in these characters can lead to a misleadingly low homoplastic rate when convergent OTUs are grouped together  (Goloboff, 2014; Tschopp et al. , 2018). In order to counteract this negative influence of missing data,  Goloboff (2014)  implemented extended implied weighting, which downweights homoplastic characters not only in relation to their homoplastic rate, but also based on the amount of missing data. As has been shown by  Goloboff et al. (2018) , this methodology seems to perform best, especially when using morphological datasets. Both implied and extended implied weighting use a concavity constant 'k' to define the strength of the downweighting: the lower the k-value, the stronger the downweighting function. Although there is no biological reasoning behind the choice of a specific k-value  (Turner & Zandee, 1995) , it is probably reasonable to exclude extreme values, and to test and compare the impact of various k-values on tree topology  (Goloboff, 1995; Tschopp & Upchurch, 2019).
ACKNOWLEDEGEMENTS We would like to thank  Paul Barrett ,  Sandra Chapman ,  Susannah Maidment  and  Lorna Steel  for providing access to the Diplocynodon hantoniensis materials held in the NHMUK, and  Matt Riley  for providing access to material in the CAMSM. We also acknowledge those who have facilitated collections' visits to study additional Diplocynodon remains ( Ronan Allain ,  Krister Smith  and  Torsten Wappler ). We also thank  Michael Stein  for sharing photographs of the humerus of Kambara from the QM. We gratefully acknowledge the  Willi Hennig Society , which has sponsored the development and free distribution of TNT. JPR is funded by a  NERC  PhD studentship. PDMs research was supported by a  Royal Society University Research Feollowship  ( UF160216 ). ET holds a  Theodore Roosovelt Memorial Fund  and  Division of Paleontology Postdoctoral Fellowship from the Richard Gilder Graduate School  at the  American Museum of Natural History.
It could instead be more relevant to reduce a specific uncertainty source, or add new locations to be included in a learning database for building a surrogate model. The Uranie platform has been developed in order to gather the methodological developments coming both from the academic and the industrial world and provide them to the broadest audience possible. This is done, keeping in mind few important aspects such as: • Open-source: the platform can be used by anyone, and every motivated person can investigate the code and propose improvements or corrections. • Accessibility: the platform is developed on Linux but a windows-porting is performed. Even though it is written in C++, it can also be used through Python. • Modularity: the platform is organised in modules so that one should only load requested modules and that analysis can be organised as a compilation of fundamental bricks.
The Uranie platform Uranie (the version under discussion here being 3. 11. 0) is an open-source software dedicated to perform studies on uncertainty, sensitivity analysis, surrogate model generation and calibration, optimisation issues. Developed by the nuclear energy division of the CEA 1 and written in C++, it is based on the ROOT  [1]  platform (discussed in Section 1. 2). The platform consists in a set of so-called technical libraries, usually referred as modules (represented as the green boxes in Figure  1 ), each performing a specific task. Some of them are considered low-level, in the sense that they are the foundation bricks upon which rely the rest of the modules, which can be considered more methodologically-oriented (dedicated to a specific kind of analysis). In the rest of this section, the main modules, used throughout this paper, will be briefly described (in terms of role) starting with the DataServer one, which is the spine of the Uranie project, as shown in Figure  1.
DataServer module The DataServer library is the core of the Uranie platform. It contains all the necessary information about the 1 Alternative Energies and Atomic Energy Commission, Saclay, France. The nuclear energy division is usually referred as Den. variables of a problem (such as the names, units, probability laws, data files, and so on. ), the data itself (if information have been brought or generated) and it allows to perform very basic statistical operations (computing averages, standard deviations, quantiles. ).
Uranie installation and external dependencies Even though the Uranie platform is developed on Linux operating systems, a Windows-version has also been made. In order to check and guarantee the best portability possible, the platform is tested daily on seven different Linux distributions and Windows 7. Getting the source of the Uranie platform can be done at the Sourceforge web page: https://sourceforge. net/projects/uranie/. Once the sources have been retrieved, it is highlyadvised to follow the instruction listed in the README file to perform the installation. On top of the code itself, this installation brings Uranie documentation, among which: • a methodological manual (both html and pdf format,  [2] ). It gives a shallow introduction to the main methods and algorithms, from a mathematical point of view, and provides references for the interested reader, to get a deeper insight on these problematics.
Compulsory dependencies • ROOT: Discussed thoroughly below, the version used here is v5. 34/36. • Cmake: Free and open-source software for managing the build process of compiled software, the version used here is v3. 7. 1  [5]. • CPPUnit: Unit testing framework for C++ programming, the version used here is v1. 13. 1  [6]. The ROOT system is an open-source object oriented framework for large scale data analysis. It started as a private project in 1995 at CERN  3  and grew to be the officially supported LHC analysis toolkit. ROOT is written in C++, and contains, among others, an efficient hierarchical object-oriented database, a C++ interpreter, advanced statistical analysis (multi-dimensional histogramming, fitting, minimisation, cluster finding algorithms) and visualisation tools. The user interacts with ROOT via a graphical user interface, the command line or batch scripts.
The command and scripting language is C++ (using the interpreter) and large scripts can be compiled and dynamically linked in. The object-oriented database design has been optimised for parallel access (reading as well as writing) by multiple processes. The ROOT system is developed in C++ (but can be called with other languages such as Python or Ruby though) and is well maintained and documented. Uranie is built as a layer on ROOT and, as a result, it benefits from numerous features of ROOT, among which: • the C++ interpreter (CINT); • the Python interface: it provides an automatic transcription of Uranie-classes into Python • an access to SQL databases; • many advanced data visualisation features; • and much more.
Optional dependencies • OPT++: Libraries that include non linear optimisation algorithms written in C++, the version used here is v2. 4  [7]. As this package is not maintained anymore, a patched (and recommended) version is included in the Uranie archive. • FFTW: Library that computes the discrete Fourier transform (DFT) in one or more dimensions, of arbitrary input size, the version used here is v3. 3. 4  [8]. • NLopt: Library for nonlinear optimisation, the version used here is v2. 2. 4  [9]. • PCL: (Portable Coroutine Library) Implements the low level functionality for coroutines, the version used here is v2. 2. 4. • MPI: (Message Passing Interface) Standardised and portable message-passing system needed to run parallel computing, the version used here is v1. 6. 5  [10]. • CUDA: (Compute Unified Device Architecture) Parallel computing platform and programming model invented by NVIDIA to harness the power of the graphics processing unit (GPU), the version used here is v8.
Paper layout This paper will describe several typical analysis that can be run using the Uranie platform. It is not meant to be fully exhaustive concerning the methodology behind the introduced techniques, but also concerning the methods and options implemented in Uranie. In many cases, though, a sub-part called to go further will introduce briefly the important, yet not discussed, options and solutions that can be offered to the user. Given what has been seen in Figure  4 , the use-case used throughout the rest of this paper will be the PTFE case. The first introduced concept will be the generation of surrogate model (see Section 2). Three different techniques will be applied on an pre-produced design-ofexperiments, that will be called the training database, describing the lowest level of complexity of the usecase (only considering the dimensionless variables x ds and t ds ).
Surrogate model generation In this part, different surrogate models will be introduced to reproduce the behaviour of a given code or function. The aim of this step is to obtain a simplified model able to mimic, within a reasonable acceptance margin, the output of both a training and a test database, along with an important improvement in terms of time and memory consumption. The full analytic model, detailed in Equation (  9 ), plays the role of the complex model that should be approximated. To do so, a training database will be produced, composed of n S locations (n S being set to 40 here), written as L = {(x j , y j ), j = 1. n S } (15) where x j = (x j ds ,t j ds ) and y j = θ (x j ds ,t j ds , B i = 4). The Biot number is set to 4 as only the PTFE case will be considered (see Table  1 ).
Three different techniques will be applied: the polynomial chaos expansion, the artificial neural network and the kriging approximation. Each and every method will have a brief introduction before being applied to our use-case. The interested readers are invited to go through the references for a more meticulous description. An example of making practical use of the the kriging surrogate is provided in Section 5. The starting point will always be the loading of the training database in an TDataServer object which is the spine of Uranie. // Create the TDataServer object TDataServer * tds = new TDataServer ( " tdsObs " , " The training DB " ) ; // Load the data in the dataserver tds -> fileDataRead ( " t r a i n i n g _ d a t a b a s e. dat " ) ;.
Implementation in Uranie and application to the use-case In Uranie, the implementation of the polynomial chaos expansion method is done through the NISP library (  [16] ), NISP standing for Non-Intrusive Spectral Projection. Originally written to deal with normal laws, for which the natural orthogonal basis is Hermite polynomials, this decomposition can be applied to few other distributions, using other polynomial orthogonal basis, such as Legendre (for uniform and log-uniform laws), Laguerre (for exponential law), Jacobi (for beta law). The PC coefficients are estimated through a regression method, simply based on a least-squares approximation: given the training database L , the vector of output y(n S ) is computed with the code. The regression are estimated, given that one calls the correspondence matrix H(n S , p) and the coefficient-vector β , by a minimisation of ||y -Hβ || 2 , where y = y 1 y 2.
y n S , (17) H =      Ψ 1 (x 1 ). Ψ p (x 1 ) Ψ 1 (x 2 ). Ψ p (x 2 ). Ψ 1 (x n S ). Ψ p (x n S )      , (18) β =      β 1 β 2. β p      (19) This leads to write the general form of the solution as β = (H T H) -1 H T y which means that the estimation of the points using the surrogate model are given through ŷ = Hβ = H(H T H) -1 H T y = Py, where P = H(H T H) -1 H T. Here, the P matrix links directly the output variable and its estimation through the surrogate model: this formula is useful as it can be used to compute the Leave-One-Out uncertainty. Figure  5  represents the distribution of the thermal gauge values (as defined in Equation (  8 )) estimated by the surrogate model ( θ ) as a function of the ones computed by the complete model (θ ) in a validation database containing 2000 locations, not used for the training.
To go further There are several points not discussed in this section but which can be of interest for users: • Based on regression method explained in Section 2. 1. 2, Uranie also provides a method to estimate the best degree possible, relying on the Leave-One-Out estimation, limiting the range of tested degree, given the learning database size (n S ). • PC coefficients can be estimated using an integration methods (instead of the regression) which relies on specific design-of-experiments (usually sparse-grids) that are oftenly smaller, in terms of computations, than the regularly-tensorised approaches  [16]. • When the PC development is done on the natural polynomial basis of the stochastic laws (listed in Section 2. 1. 2), the PC coefficients can be combined and transformed into Sobol coefficients (discussed in Section 4) providing both a surrogate model and a sensitivity analysis.