dataesr/scientific-paragraphs-categorization huggingface dataset:
Funding: The work on the ozone data was supported by the  Russian Science Foundation  (project  RNF-19-12-00171 ). Yu. L. B. acknowledges the support from the  Academic D. Mendeleev program of Tomsk State University. Collaboration with  A. Barbe ,  S. Mikhailenko  and  E. Starikova  in spectra analyses for S&MPO is acknowledged. VB and CR wish to thank the  OSU THETA of Franche-Comté Bourgogne  and  DatOSU  project for their support. The  ExoMol  project has been supported by  ERC  Advanced Investigator Project  267219 , and  STFC  Projects No. ST/M001334/1  and  ST/R000476/1. The HITRAN database is supported through the  NASA Aura  and PDART grants  NNX17AI78G  and  NNX16AG51G , respectively. The implementation of the Query Store was partially funded by the European Project  RDA EU3  (funded under  H2020-EINFRA-2014-2 , project  ID :  653194 ).
A portion of the research described in this paper was performed at the  Jet Propulsion Laboratory, California Institute of Technology , under contract with  NASA ,  US  government sponsorship acknowledged. The work on the Spectr-W 3 update was supported by the  Russian Foundation for Basic Research  (project Nr. 14-07-00863 ) and the  Russian Academy of Sciences Presidium Fundamental Research Program Nr. 13. Developments of the interface with the VAMDC portal to the NIFS database is partly supported by the  NIFS program of Promoting Research by Networking among Institutions  (grant No. 01411702 ). EK acknowledges  Raja Ramanna Fellowship  from  Dept. of Atomic Energy, Govt. of India. PEARL-KAERI database has been partly supported by the  National Research Foundation of Korea  ( NRF-2019R1A2C1009529 ).
Support for VAMDC has been provided by the  UKRI/STFC  under grant  ST/M007731/1. The Paris team received fundings from  Paris Observatory  through its Paris Data Center (PADC), from  CNRS  through the overheads of the  VAMDC  project, from  LERMA  through the overheads of the SUP@VAMDC projects, from  ASOV  ( Action Spécifique VO France). TJM is grateful to the  STFC  for support through grants  ST/M007766/1  and  ST/P000312/1. The  SSHADE  project has received funding from the  European Union 's  Horizon 2020 research and innovation programme  under grant agreements No  654208  ( Europlanet 2020 RI), No 871149 (Europlanet 2024 RI ) and from the  Observatoire des Sciences de l'Univers de Grenoble and INSU. Support  for the VAMDC consortium has been provided through the VAMDC and the  SUP@VAMDC  projects funded under the " Combination of Collaborative Projects and Coordination and Support Actions" Funding Scheme of The Seventh Framework Program.
Franche- 5. 5. 55 Comté region in France [181]. Sustainability issues are also related to human training and capabilities. In that respect, VAMDC has trained engineers and post-docs who are still involved in the fields of software developments, database management, development of standards for the benefit of the public sector in the VAMDC collaboration and in other domains, as well as in the private sector. Impact on Open Science Initiatives and International Data Alliances VAMDC delegates are active and have a high reputation in international data alliances, where they bring the VAMDC requirements and expertise about open-data sharing: • VAMDC has been active in the Research Data Alliance since 2014 (Section 3. 2. 2), when VAMDC became an early pilot for the data-citation recommendation. Starting from 2016.
Many applications need concerted access to different databases, just what is provided through VAMDC. Here, we present a few of these applications that demonstrate the use and benefit of VAMDC. 4. 1. CASSIS Software CASSIS 59 (Centre d'Analyse Scientifique de Spectres Instrumentaux et Synthétiques) is a standalone software written in Java, freely delivered to the community for help in visualising, analysing and modelling observations from ground or space-based observatories. It has been developed at IRAP since 2005 and is part of the OVGSO data centre that aims at promoting VO technology. It displays any spectra (ASCII, FITS or GILDAS/CLASS format or the result from a query to any SSAP 60 VO or 57https://github. com/xnx/pyvalem 58 https://www. djangoproject. com/ 59 http://cassis. irap. omp.
After a brief reminder on general concepts used in data cataloguing activities, this chapter provides information concerning the architecture and design recommendations for the implementation of catalogue systems for the ENVRIplus community. The main objective of this catalogue is to offer a unified discovery service allowing cross-disciplinary search and access to data collections coming from Research Infrastructures (RIs). This catalogue focuses on metadata with a coarse level of granularity. It was decided to offer metadata representing different types of dataset series. Only metadata for so-called flagship products (as defined by each community) are covered by the scope of this catalogue. The data collections remain within each RI. For RIs, the aim is to improve the visibility of their results beyond their traditional user communities.
Data catalogues also avoid duplication of data. Data catalogues exist to collect, create and maintain metadata. These records are indexed in a database and end-users should access the information through a userfriendly interface. This interface should offer common data search functionalities allowing users to narrow down their search according to different criteria: keywords (controlled vocabularies), geographic location, temporal and spatial resolution, and data sources. Data catalogues have become an important pillar in the data management lifecycle. Indeed, almost every step of the data lifecycle is described in the metadata fields or accessible through the data catalogue online interface. Curated data are described by effective and structured metadata (cf. Riley's list above) providing information about data collection (e.
Hence a standard is a standard insofar as there is an institutional or organizational standardization unit developing and maintaining a standard -whereas all parties and persons involved agree this institution to be trustworthy and reliable. Some relevant standards are mentioned below. ISO 19115 [5] is an internationally adopted schema for describing geospatial data. As indicated in their website "it provides information about the identification, the extent, the quality, the spatial and temporal aspects, the content, the spatial reference, the portrayal, distribution, and other properties of digital geographic data and services. " DataCite 2   [6] is an international consortium founded in 2009 with an emphasis to make explicitly research data citable, giving them a 'value' during the scientific process: "a persistent approach to access, identification, sharing and re-use of datasets" [6]  3.
Data Catalogues Tools There are many tools used by scientific communities to create data catalogues. Two example tools used by the environmental and Earth science research communities are GeoNetwork and CKAN. GeoNetwork  13  is an open-source software allowing the creation of customised catalogue applications. This tool is mainly used for describing and publishing geographic datasets and is related to ISO 19115/19139. CKAN  14  is an open-source Data Management System widely used in the world of open data. It uses essentially some Dublin Core metadata elements  15  but allows for an infinite extension of additional attributes thus making interoperation difficult. EUDAT B2FIND uses CKAN for its frontend. Independently of the software used, protocols exist for sharing metadata between data catalogues, in particular OGC-CSW  16  , OAI-PMH  17  , SPARQL  18  and others.
Upload and Indexing B2FIND´s search portal and GUI is based on the open-source portal software CKAN, which comes with Apache Lucene SOLR Servlet allowing indexing of the mapped JSON records and performant faceted search functionalities. CKAN was created by the Open Knowledge Foundation (OKFN) and is a widely used data management system. CKAN has a very limited internal metadata schema 39 which has been enhanced for B2FIND while creating additional metadata elements as CKAN field "extra". B2FIND offers a full text search, results may be narrowed down using currently 11 facets (including spatial/temporal search and facets <Discipline>, <ResourceType>, <Publisher>, <Contributor>, <Language>, <Community>, <Tags> and <Creator>). "Community" here is the data provider where B2FIND harvests from.
Flagship Implementation The implementation of ENVRIplus Flagship catalogue in B2FIND faced two main challenges: 1) how to integrate metadata records that are representing Research Infrastructures rather than Datasets, and 2) how to represent these RIs as part of ENVRIplus The Data Documentation Initiative (DDI) is an international standard for describing the data produced by surveys and other observational methods in the social, behavioural, economic, and health sciences. https://ddialliance. org/. The Component MetaData Infrastructure (CMDI) provides a framework to describe and reuse metadata blueprints. Description building blocks ("components", which include field definitions) can be grouped into a ready-made description format (a "profile"). https://www. clarin. eu/con tent/component-metadata.
A. RI Dataproducts As described above B2FIND is first and foremost a search portal for research data that should be findable across scientific disciplines. It is not primarily meant to be a search portal for other information as e. g. funding bodies, site information or research infrastructure descriptions. Concerning RIs that are part of ENVRIplus, most of them have their own search interface and some of them have already made their repositories harvestable. Thus, the flagship implementation started with harvesting already existing RI endpoints (DEIMS 41 , NILU 42 , EPOS, SeaDataNet, Euro-Argo, AnaEE, ICOS Carbon Portal 43 ) and integrating them as "Communities" into a B2FIND testing machine  44  , which means representing their data as e. g. "DEIMS". One challenge on B2FIND side was to develop the software stack 45 in order to be able to harvest from CSW endpoints.
B) B2FIND/Flagship architecture Initially, the Flagship catalogue should have been visible in a way that would display both ENVRIplus as the main project and each RI as a part of it. CKAN allows to create "Groups" and "Subgroups"; however, B2FIND is constructed as CKAN "Group" and its "Communities" as CKAN "Subgroups" which means that a further distinction between ENVRIplus and RIs could not be implemented. In order to enable a search for RIs the decision was to create a 'Community' = ENVRIplus and use the metadata element https://deims. org/. https://www. nilu. no/en/. The data centre of ICOS, https://icos-cp. eu/. http://eudat7-ingest. dkrz. de/dataset. B2FIND uses CKAN only for GUI and search interface while the backend is developed B2FIND code, it´s Open Source on GitHub: https://github.
com/EUDAT-B2FIND. <Contributor> as a distinctive feature (Fig. 1 ). As the flagship implementation enforced B2FIND to enhance its metadata schema (to enable a faceted search via <Contributor>) it was implemented on a test machine at DKRZ 46. The demonstrator may be seen here: http://eudat7-ingest. dkrz. de/dataset?groups=envriplus. As described above B2FIND links to a certain resource by using persistent identifiers (if offered within the metadata) in order to increase the reliability of a digital resource (Fig. 2 ). Therefore, an internal 'ranking' is used: if a DOI is provided it will be displayed, both as a link to the Landing page and additionally as a small icon on the single record entry page. If no DOI but another PID (e. g. a Handle) is offered this one will be shown, both as a link and as an icon.
How Does It Work? Although the model can be implemented in many ways (including object-oriented, logic programming and triplestores), most often it is implemented as a relational database but with a particular approach thus ensuring referential and functional integrity. CERIF has the concept of base entities representing real-world objects of interest and characterised by attributes. Examples are project, organization, research product (such as dataset, software), equipment and so on. The base entities are linked with relationship entities which describe the relationships between the base entities with a role (such as owner, manager, author) and date-time start and end so giving the temporal span of the relationship. In this way versioning and provenance are 'built-in'. CERIF also has a semantic layer (ontologies).
Some Existing Use Cases EPOS uses CERIF for its catalogue because of the richness for discovery, contextualisation and action and because of the built-in versioning and provenance, important for both curation and contextualisation. The architecture of the software associated with the catalogue (ICS: Integrated Core Services) is based on microservices (Fig. 5 ). The implementation uses PostgreSQL as the RDBMS and has been demonstrated on numerous occasions (Fig. 7 ). A mechanism for harvesting metadata from the various domain groups of EPOS (TCS: Thematic Core Services) and converting from their individual metadata schemes to CERIF has been implemented including an intermediate stage using EPOS-DCAT-AP -a particular application profile of the DCAT standard [11]. (Figure  6 ). CERIF thus provides EPOS users with a homogeneous view over heterogeneous assets allowing cross-disciplinary research as well as within-domain research.
The current work is to provide the user interface software to be used at that location. In parallel work proceeds on (a) converting CERIF to the metadata format based on DataCite and integrated with CKAN used at EUDAT for inclusion in the EUDAT B2FIND catalogue. Unfortunately, conversion from the B2FIND catalogue (based on CKAN) to CERIF is not possible because the records cannot be made available by the hosting organisation, largely due to resource limitations. CERIF is natively FAIR since it supports all four aspects of the FAIR principles. Because of its referential and functional integrity, formal syntax and rich declared semantics CERIF is more machine-actionable than most metadata standards which usually require human intervention to interpret the metadata e. g. for the composition of workflows.
VRE4EIC and ENVRI To prototype the use of CERIF as a joint catalogue service combining datasets from multiple RIs for use by a single VRE, a collaboration was established between ENVRIplus and the VRE4EICproject. VRE4EIC concerned itself with the development of a standard reference architecture for virtual research environments, as well as the prototyping of exemplar building blocks as prescribed by that reference architecture. In particular, the project consortium developed VRE4EIC Metadata Service to demonstrate how data from multiple RIs might be harvested using a variety of protocols and techniques and then provided via a common portal. X3 ML mappings [12] from standards such as ISO 19139 [10] and DCAT to CERIF [13] were used to automatically ingest metadata published by different RIs to produce a single resource catalogue.
6 55. Metadata harvested from external sources were converted to CERIF RDF 54 https://virtuoso. openlinksw. com/. 55 https://www. eurocris. org/cerif/main-features-cerif. using the X3 ML mapping framework 56 ; the mapping process itself was as illustrated in Fig. 9 : 1. Sample metadata, along with their corresponding metadata schemes, were retrieved for analysis. In addition to metadata from ENVRI and EPOS also records from CRIS (Current Research Information Systems which describe projects, persons, outputs, and funding) were harvested. 2. Mappings were defined that dictate the transformation of selected RDF and XML based schemas into CERIF RDF. 3. Metadata is retrieved from different data sources in their native formats, e. g. as ISO 19139 or CKAN 57 metadata (specifically as used in B2FIND within EUDAT in the context of ENVRI).
4. These mappings could then be used to transform the source metadata into CERIF format. 5. The transformed metadata was then ingested into the CERIF metadata catalogue. Once ingested, these metadata became available to users of the portal, who could query and browse the metadata catalogue upon authentication via a front-end authentication/authorisation service. X3ML mappings were constructed using the 3M Mapping Memory Manager 58. Among other functions, 3M supported the specification of generators to produce unique identifiers for new concepts constructed during translation of terms. Mappings into CERIF RDF were produced for Dublin Core, CKAN, DCAT-AP, and ISO 19139 metadata, as well as RI architecture descriptions in OIL-E. The VRE4EIC Metadata Service demonstrated many desirable characteristics for a catalogue service, those being: a flexible model in CERIF for integrating heterogeneous metadata; a tool-assisted metadata mapping pipeline to easily create or refine metadata mappings or refine existing mappings; and a mature technology base for unified VRE catalogues.
It was judged however that more development was needed in the discovery of new resources and the acquisition of updates through some automated polling/harvesting system against a catalogue of amenable sources. In this respect, RI-side services for the advertisement of new resources or updates to which a VRE can subscribe to trigger automated ingestion of new or modified metadata would be particularly useful. A notable feature of CERIF is how it separates its semantic layer from its primary entity-relationship model. Most CERIF relations are semantically agnostic, lacking any particular interpretation beyond identifying a link. Almost every entity and relation can be assigned through a classification that indicates a particular semantic interpretation (e. g. that the relationship between a Person and a Product is that of a creator or author or developer), allowing a CERIF database to be enriched with concepts from an external semantic model (or several linked models).
The vocabulary provided by OIL-E (Chapter 6) has been identified as a means to further classify objects in CERIF in terms of their role in an RI, e. g. classifying individuals and facilities by the roles they play in research activities, datasets in terms of the research data lifecycle, or computational services by the functions they enable. This provides additional operational context for faceted search (e. g. identifying which processes generated a given data product) but providing additional context into the scientific context for data products (e. g. categorising the experimental method applied or the branch of science to which it belongs) is also necessary. Environmental science RIs such as AnaEE and LTER-Europe are actively developing better vocabularies for describing ecosystem and biodiversity research data, building upon existing SKOS vocabularies.
Future Directions and Challenges for Cataloguing To demonstrate cataloguing capabilities a two-pronged approach was adopted. Some records describing 'data products' were created from several RIs and ingested by B2FIND. This exposed the effort of metadata mapping but also the capability of a catalogue with metadata from different domains with unified syntax (but not necessarily unified semantics). This catalogue certainly demonstrated the potential for a homogeneous view over heterogeneous assets described by their metadata converted to a common format. However, the relatively limited schema used in EUDAT B2FIND means that some richness from the original ENVRI RI metadata records was lost. Separately the EPOS metadata catalogue of services was used as an exemplar of the use of CERIF for integrated cataloguing, curation and provenance and via the associated VRE4EIC project the harvesting, mapping and conversion to CERIF of heterogeneous assets from multiple sources was demonstrated.
Furthermore, CERIF provided a richer metadata syntax and semantics although -of course -if the source ENVRI RI catalogue had only limited metadata the full richness could not be achieved. There was some investigation in VRE4EIC of enhancing metadata by inferential methods since the formal syntax, referential and functional integrity and declared semantics of CERIF lend themselves to logic processing. The objective of these two parallel exercises was to allow RIs to see what can be achieved -and what effort is necessary -in the integration of heterogeneous metadata describing assets to permit homogeneous cross-domain (re-)use of assets. Further enhancements and improvements of the mapping (from various metadata formats used by the RIs to a canonical format) are necessary before the ENVRIplus records could be published and be searchable in the production B2FIND portal.
Within EPOS 17 different metadata formats had to be mapped and converted to be ingested into the CERIF catalogue and made available for (re-)use and in VRE4EIC further heterogeneous assets were added. The effort of correct matching and mapping between metadata standards should not be underestimated but -once achieved -can provide homogeneous access over heterogeneous asset descriptions and hence support a portal functionality allowing the end-user to gain interoperability. As indicated by K. Jeffery (see Chapter 7: the choice of the metadata elements in the catalogue (including their syntax and semantics) is crucial for the processes not only of curation but also of provenance and catalogue management and utilisation for dataset discovery and download. The RIs have different metadata formats and each has its own roadmap or evolution path improving metadata as required by their community.
Unfortunately, there are many metadata standards, some general (and usually too abstract for scientific use) and some detailed and domain-specific (but not easily mapped against other formats). The need for rich metadata is becoming generally accepted. As mentioned by authors from the EOSC Pilot project [16] "Minimum and common metadata is useful for data discovery and data access. Rich metadata formats can be complex to adopt, but have the advantage of making data more "usable" by both humans and machines". It is planned to continue -in the ENVRI community -with the EUDAT B2FIND catalogue (maintained by EUDAT) and also to continue the work with CERIF (maintained by EPOS), anticipating the need for richer metadata than the B2FIND schema for at least some of the ENVRI RIs. CERIF already can handle the functionality associated with services -and other RI assets -as required in the EOSC (European Open Science Cloud).
In particular, EUDAT/B2FIND is concentrated on datasets whereas the EPOS CERIF catalogue -while also handling datasets, workflows, software, equipment and other assets -initially concentrated on services to ensure alignment with the emerging EOSC. A mapping between CERIF and the draft metadata standard for EOSC services has been done. The overall strategy is to make cataloguing technology available to the ENVRI RIs for them to choose how they wish to proceed, considering also other International obligations for interoperability which may determine particular metadata standards. This means that it is likely for the foreseeable future that ENVRI will need to support a range of metadata standards -among the RIs, internationally and also to align with general efforts such as schema. org from Google and associated dataset search -but that to interoperate them a canonical rich metadata schema will be required.