dataesr/scientific-paragraphs-categorization huggingface dataset:
Generated Tests As far as test generation is concerned, we unfold the standard metadata to write interface tests for C and Fortran. As shown in Figure  4 , this process relies on simple code generation to create individual test files (1) creating the right arguments for the call and then (2) calling for the MPI function. This test then checks if the symbol is exported by the MPI interface and if the arguments are correctly provided. However, this automatic test generation does not validate the correct implementation of the call itself as it would have required more extensive testing incompatible with scaffolding. We consider this second approach as future work and we consider leveraging Large Language Models (LLMs) to do so procedurally. In summary, what is tested is the compilation of the given symbol with the provided compiler, configurable as per the PCVS profile and target MPI runtime.
Integration with Spack As depicted in Figure  6 , to pull the last release of a given runtime, we relied on the Spack package manager. This package manager, specifically tailored for HPC was designed to build packages from their sources. Naturally, all the main HPC runtime and compilers are provided through this interface. The Spack repositories are periodically probed for a new version and if there is one, the test suite is run and submitted to the bank. This result bank saves in Git format all the results including those from other versions. Conjointly, the reporting portal monitor the result bank and render results when new data are available. This approach has the advantage that it is very generic. We just need to provide a list of MPI runtimes as input and by calling Spack we can install and switch between multiple versions.
MPI Test-Suite We have run the PCVS exhaustive interface test suite on multiple implementations of MPI, Open MPI  [6]  and MPICH  [7]  which are production level implementations and MPC  [15] , a more research oriented implementation. To do so, as previously discussed we leveraged the Spack package manager to compile and deploy the runtimes. We used the default configuration without specific tweaks. We then simply run the test suite after loading the respective MPI implementations. We have thus targeted the various implementations over C, Fortran. The test suite runs for all functions on the interface and test standards from 1. 0 to 4. 0 included. As 4. 0 is very recent it is expected that several failures are linked to the absence of implementation and not to runtime errors. Tests are grouped using labels including standard levels, type of tests (e.
Deploying the Test-Suite The best way to get PCVS is through the PyPI repository: More information may be found at https://pcvs. readthedocs. io/. Using these simple instructions, you should be able to reproduce our results on your local MPI implementation, leading to the execution of the test suite as shown in figure  10. It is the exact same process we have followed to expose such results on https://mpicheck. pcvs. io/.
CONCLUSION In this paper, we presented our implementation of an exhaustive MPI test suite generated from the standard metadata. To run these tests, we utilized the Parallel Computing Validation System (PCVS), a parallel test-suite runner designed for HPC. We highlighted the increase in the number of MPI functions over time and the need for automated testing to track symbols in the various bindings. Our classification of the MPI standard functions, helped gather tests on a standard-revision basis. We described the architecture of PCVS, its capabilities, and implementation details. We then outlined the implementation of tests in different output languages (C and Fortran, with support for any revision). Additionally, we presented and discussed the results of the test suite, followed by an evaluation of its scalability, reducing the execution time to less than one minute for the whole interface test suite.
FUTURE WORK This paper presented the first iteration of our validation system. We started with symbols as they are one of the simplest thing to test. We will soon complement the tests with the constants and datatypes defined by the standards. In a second time, as these tests are only interface tests, allowing to check if the symbol is present, and not if it is operational, we plan to complement them with functional tests. As it can be cumbersome to write tests validating all the functions of the MPI standard, we plan to leverage a Large Language Model (LLM) to generate the corresponding code, as such models are surprisingly good at manipulating code. In addition as MPI is working on defining an ABI, we plan to implement the corresponding tests as part of our interface test suite, the same could be done for other languages bindings such as RUST and Python.
# A f t e r t h i s command you s h o u l d have # ' pcvs ' i n your p a t h pip3 install pcvsYou may clone mpicheck and (optionally) generate the PCVSenabled test-suite with the generator script: # R e t r i e v e t h e t e s t -s u i t e git clone https : / / github. com / ceahpc / pcvsbenchmarks cd pcvsbenchmarks # O p t i o n a l ( pre -g e n e r a t e d ) # w i l l c r e a t e. / MPI / mpicheck / t e s t s / cd. / MPI / mpicheck / python3 mpicheck. py You can then run the test suite and then start the reporting server on http://localhost:5000 with: # C r e a t e a d e f a u l t p r o f i l e w i t h MPI t e m p l a t e # I t i m p l i e s t o f i n d MPI w r a p p e r s i n t o PATH pcvs profile create -t mpi user. mpi # Run t h e d i r e c t o r y u s i n g t h i s p r o f i l e # R e s u l t s a r e s t o r e d l o c a l l y i n a h i d d e n d i r.
These observations can be used to assess the optical quality through a measurement of the PSF. During Flight#3, we observed Jupiter at its maximum elevation of ≃ 17 • during about 30 min at the start of the flight. We obtained eight individual maps using eight positions of the HWP, sampling the available analysis range uniformly. The maps were obtained at two different scanning angles to enable low frequency noise removal. The data were corrected for the responses calculated on the residual atmospheric signal and the ICS calibration signals, and corrected for the effect of the detector time constants through deconvolution, as described in  [1]. The signal was then processed using the Scanamorphos map-making software described in  [38]  and as used in its polarization version in  [37]  to produce maps of the Stokes parameters I, Q and U and the corresponding variances and co-variances.
The above processing is computed independently for each detector array and we use the PSF of each array as measured in maps of Jupiter computed for that array only. The data processing steps proceed clockwise from top-left. The I, Q and U PSF as measured on Jupiter are rotated to follow the time-dependent sky rotation. They are then convolved to the high resolution Herschel satellite sky map of the considered object (here the Orion A region) at 250 µm sky to produce instantaneous maps of the Q and U leakage. The instantaneous maps are re-observed to produce a correction time-line which is subtracted from the original data. Our final leakage-corrected, science-ready maps are constructed by applying our map-making algorithm to this corrected timeline. To first order, the processing therefore takes into account the focal-plane variations of the leakage shown in Fig.
Acknowledgements PILOT is an international project that involves several European institutes. The institutes that have contributed to hardware developments for PILOT are  IRAP  and  CNES in Toulouse (France) ,  IAS in Orsay (France) ,  CEA in Saclay (France) ,  Sapienza University in Rome (Italy) ,  Cardiff University (UK)  and the  Scientific Support Office at ESTEC (NL). This work was supported by the  CNES. It is based on the PILOT data obtained during three flight campaigns operated by  CNES , under the agreement between  CNES  and  CNRS / INSU. It benefited from the active participation of  CNES ,  IRAP ,  IAS ,  CEA  and  ESTEC  to the flight campaigns. This work was supported by the  Programme National "Physique et Chimie du Milieu Interstellaire" (PCMI)  of  CNRS / INSU  with  INC/INP  co-funded by  CEA  and  CNES.
The automatic extraction of materials and related properties from the scientific literature is gaining attention in data-driven materials science (Materials Informatics). In this paper, we discuss Grobid-superconductors, our solution for automatically extracting superconductor material names and respective properties from text. Built as a Grobid module, it combines machine learning and heuristic approaches in a multi-step architecture that supports input data as raw text or PDF documents. Using Grobid-superconductors, we built SuperCon 2 , a database of 40324 materials and properties records from 37700 papers. The material (or sample) information is represented by name, chemical formula, and material class, and is characterized by shape, doping, substitution variables for components, and substrate as adjoined information.
Introduction In recent years, with the creation of computational databases, such as the Materials Project (MP)  [1]  and the Open Quantum Materials Database (OQMD)  [2] , and then experimental data repositories such as NIMS MDR (http://mdr. nims. go. jp)  [3] , focus has been steadily shifting towards a data-driven design of materials, which is often called Materials Informatics (MI). Such an approach is expected to accelerate the exploration of functional materials because it is not limited to the intuition or experience of very little genius researchers. In this new paradigm, the efficient use of data to guide experiments and material property prediction through the use of machine learning methods takes center stage. For example, data-driven methods have been used to search/design magneto-caloric materials  [4] [5] [6] , photo-catalysts for hydrogen splitting  [7] , thermoelectrics  [8] , and superconductors  [9].
In such a data-driven search, one of the most important keys lies in the availability of the data, which should at least should consist of compositions of materials and their physical properties. In the specific case of superconductivity, most of the data-driven works  [9] [10] [11]  rely on a single database: SuperCon (http://supercon. nims. go. jp). SuperCon is a structured database of superconductor materials and properties;it was developed at the National Institute for Materials Science (NIMS) in Japan. At the time of writing this paper, SuperCon contained about 33000 inorganic and 600 organic materials and is the "de-facto" standard in data-driven research for superconductors materials (about 4400 articles contain the mention "SuperCon database" in Google Scholar). However, the SuperCon harvesting process is currently fully manual "from scratch": humans have to read the human-readable printed matter such as PDF documents and enter the information into the system.
The efficiency is directly proportional to the number of available human curators. Considering the cost of database construction, it is necessary to consider an assisted or alternative system that improves throughput while ensuring data quality equivalent to that of manual extraction. As a solution, we are developing a hybrid data extraction method from scientific literature that combines automation using text data mining and manual curation. The automated system extracts and formats potential data and proposes them to the curator as "pre-cooked" structured data by  (1)  highlighting the relevant entities on the original document and (2) pre-filling the extracted information in a tabular format. In building the automated part of this hybrid system (training, evaluation) we used SuperMat, an annotated linked dataset from scholarly documents in superconductor research, which we recently constructed  [12].
In this work, we present Grobid-superconductors: a system that automatically extracts structured information related to superconductor materials and properties from scientific literature. The tool is a specialised module of Grobid (Generation of Bibliographic Data)  [13] , a machine learning library designed to parse and structure scientific documents. Grobid provides an open-source platform for building specialised modules including astronomical entities recognition  [14] , dictionaries  [15] , software mentions  [16] , and physical measurements extraction  [17]. Grobid provides several built-in features including access to PDF document layout information, citation resolution, bibliographic information consolidation through biblio-glutton  [18]  (a fast open-source reference matching service for CrossRef data), and a diverse set of machine learning (ML) architectures from a fast linear Conditional Random Field (CRF) to the latest state-of-the-art deep learning implementations.
Using Grobid-superconductors and other sub-tools, we established a pipeline to process a large number of documents and obtain an automated database of superconductor materials and properties. We processed 37770 papers from ArXiv (https: //arxiv. org) and obtained a database of 40324 records. This new database, named SuperCon 2 , can become an automated staging area for SuperCon, when bridged by a curation interface. During the project, there was also an opportunity to focus on properties that that have recently become of greater interest and that are underrepresented in SuperCon. For example, the "pressure" applied to obtain superconductivity (about 20 records in SuperCon), has gained attention because it can radically change the physical structure of a material. In addition, the "method" used to measure the superconducting transition temperature T c (about 600 records in SuperCon) can be used to semantically recognise multiple T c 's obtained from the same material or sample (e.
Grobid-superconductors We developed Grobid-superconductors as a Grobid module following principles (multistep, sentence-based, full-text-based) discussed in a previous preliminary study  [19]. Grobid has several advantages: 1) it can be integrated with pdfalto (https://github. com/kermitt2/pdfalto), a specialised tool for converting PDF to XML, which mitigates extraction issues such as the resolution of embedded fonts, invalid character encoding, and the reconstruction of the correct reading order, 2) it allows access to PDF document layout information for both machine learning and document decoration (e. g. , coordinates in the PDF document); and, 3) it provides access to a set of high-quality, pre-trained machine learning models for structuring documents. Grobidsuperconductors is structured as a three-steps process illustrated in Figure  1  and described in the Sections 2.
1, 2. 2, and 2. 3. Abstract versus full-text At the time of writing this paper, we are aware of related works that utilise text from abstracts as training data for machine learning. The main reason for using abstracts is that they are usually freely available as text  [20] , and contain condensed information  [21, 22]. Accurately parsing the full-text presents more challenges, however, but they are mitigated by Grobid and, the full-text contain a broader range of information, including the sample preparation process, negative results (e. g. , absence of superconductivity for certain samples), and background information (e. g. , reports on other materials from referenced works). Thus, grobid-superconductors is built to support full-text documents. Paragraphs versus sentences Another question related to natural language processing (NLP) is whether to use sentence-based or paragraph-based text.
As indicated in Table  1 , the F1-score increased by 17. 94 percentage points when the sentence-based text was used. For the Entity Linking (EL) task, we want to maximise precision. In our previous work  [24]  we noticed that limiting linking entities within the same sentence (versus paragraph) would obtain higher precision (68. 7% versus 57%) at the expense of lower recall (6. 5% versus 10. 7%), and F1-score (11. 87% versus 18. 01%). Therefore, in both our tasks we found evidence that a sentence-based dataset is more beneficial than paragraph-based dataset.
Overview As illustrated in Figure  3  the "superconductor parser" extracts the main superconductor-related information by aggregating the resulting entities from two ML models. The Superconductors ML model was developed based on the SuperMat schema  [12] , and the Quantity ML model was developed in a separated Grobid module for measurement extraction  [17]  and the output is limited to only temperatures and pressures. Overlapping entities are merged, exacted duplicates are removed, and the largest entities (in terms of string length) are preserved. The resulting entities are summarised in Table  2. Entities of type <material>, which may contain mixed heterogeneous information, are passed in the cascade to the "Material parser" which aggregates ML and other tools. First, the entity is passed through a Material ML model to segment and identify its content (Table  3 ).
Machine Learning study In this section we discuss the novel ML models we have trained for extracting specialised entities: the Superconductor ML model and the Material ML model (Figure  3 ). SuperMat  [12] , our training dataset, contains 164 papers as of the time of writing and is composed of annotated full-text and layout features from PDF documents. For both ML models we trained and evaluated the following four architecture/implementations: linear CRF (CRF), bidirectional LSTM with CRF  [26]  (BidLSTM CRF), bidirectional LSTM with CRF with Features  [26]  (the same as (BidLSTM CRF) with an additional input channel for features; BidL-STM CRF FEATURES), and SciBERT  [27]  using a CRF as the activation layer (Scibert). The ML models are interfaced by Grobid, which uses the Wapiti  [28]  implementation for linear CRF, and DeLFT (Deep Learning For Text)  [29]  for deep learning models.
Superconductor ML model Holdout set The holdout set evaluation consists in using a fixed part of a dataset for validation. The selection must be performed to reproduce the same distribution of entities of the original dataset. We assembled the holdout set by manually selecting 32 documents (24%) from SuperMat, making sure they had a similar ratio of examples, entities and unique entities with the remaining 76% (132 documents) which was used as training set (Figure  4a ). Maintaining the same rate for entity type distribution between the two sets was more challenging: on average, we obtained about 15-18% of labels of each type in the holdout set (Figure  4b ), except for the <material> label (23%). We defined the "out-of-domain" ratio as the number of unique entities from the holdout set that were not in the training set.
1, 0. 25, 0. 5 and 1. 0 did not provide stable evidence suggesting scoring improvements when testing against the holdout set. Evaluation The best results were obtained by Scibert with an F1 of 77. 03% and a recall of around 80. 69% (Table  4 ). The features did not provide any improvements with RNN models: BidLSTM CRF and BidLSTM CRF FEATURES resulted in the same F1 score. This result comes as a surprise because features such as superscript/subscript were expected to be determinants for recognising material sequences. The <pressure> label had the lowest performance scores in all architectures. We believe that 274 training examples are not a sufficient large number considering that pressure expressions can be dependent on the context because they can refer to different types of pressures (e.
69%) of entities labelled as <tc>. Scibert shows good generalisation capacity for unseen examples or examples appearing in different contexts. For example, in Figure  6a , only Scibert correctly extracts "above 100K", while CRF misses it completely and BidLSTM CRF misses "above". In the training data, "above 100K" is not present, but "below 100K" and " 100K" are present, and several other entities contain the token "above" and Scibert can understand that the token "above" is relevant to the temperature. In a second example (Figure  6b ), only Scibert can correctly extract "W-C nanowire" which is not present in the SuperMat training data. Unfortunately, we cannot check whether "above 100K" or "W-C nanowire" are also present in the dataset used in the pre-train of SciBERT by their authors  [27]  because the data are not available.
Material ML model To train the Material ML model we created a special dataset with an additional layer of labels (Table  3 ), which included the material information represented by entities annotated as <material> in the SuperMat documents. Holdout set In this model we created an independent holdout set because the manual annotation work is performed on smaller chunks of text and requires less effort than annotating sentences as when we developed SuperMat. We used material data extracted from a dataset of 500 documents (500-papers) from three publishers: American Institute of Physics (AIP), American Physical Society (APS) and Institute of Physics (IOP)  [24]. The resulting holdout set has a average coverage greater than 25% (Figure  7 ) and an average "out-of-domain" ratio of 83. 93% (Figure  8 ).
End to end evaluation End-to-end evaluation (E2EE) measures the capacity of the system from the PDF documents until the final linked results. We limited the scope of the E2EE to the triplet 'material-T c -pressure' which, at the moment, is the backbone upon which the database is built. We performed the E2EE on the "500-papers" dataset where we manually examined the resulting database as follows: 1) we marked invalid records and 2) we identified the cause of failure from a predefined set of five error types (Figure  9 ): • Quantity extraction: quantity entities (pressure, temperature) are not correctly extracted. We measured this error separately to identify the failure that could be shared with the Quantity ML model. • T c classification: the temperature is wrongly classified as superconducting T c.
Conclusion In this work, we present our solution for automatically building a database of materials and properties from scientific literature. Our contribution is composed of: 1) Grobid-superconductors, a specialised open-source system that processes PDF documents combining ML and rule-based methods to extract and link relevant information in superconductors research; 2) a pipeline allowing large-scale document processing; and 3) a visualisation interface for rapid data exploration, which includes PDF document information enrichment. We made SuperCon 2 , a database with 40324 records of superconductors materials and properties, including the applied pressure and the T c measurement method. SuperCon 2 is available in text format at https://github. com/lfoppiano/supercon. In the future, we plan to improve our tools by 1) extracting more properties, such as crystal structure type, space groups type, and lattice structure; 2) training supervised models for the "Linking step"; and 3) extending the interface to support data correction toward efficient curation.
Document Creation Using Tweets as the basis of a retrieval task is not as straightforward as using newswire, where the articles produced are naturally seen as documents. An individual Tweet can lack context, making relevance assessment difficult. Tweets are both responded to and copied in the form of Retweets, so a rich tree structure exists that can be used to combine Tweets. We aggregated replies to Tweets to form linear conversation threads. Each such thread served as a single document. We began with Twitter's random 1% feed, known at the Twitter "spritzer" stream available from the Internet Archive. 3  We separated the data by language using Twitter's language ID, selecting the Chinese  4  and Persian subsets of the collection. We augmented our Tweet collection in two ways. First, to ensure we could generate coherent topics, and to support alignment with HC4, we allowed our assessors to include in the collection Tweets they identified through live search of Twitter.