dataesr/scientific-paragraphs-categorization huggingface dataset:
As the underlying data model for the unified catalogue is RDF-based, queries constructed using the Portal are submitted to the underlying database using SPARQL 1. 1. It is possible for users to directly construct SPARQL queries via the Portal, or to edit as SPARQL queries constructed via the graphical Web interface; it is a principle of the Portal's design however that most users should never need to. X3ML mappings are described using the 3M Mapping Memory Manager 24. 3M is a Web application, that can be run in an servlet container environment such as Apache Tomcat  25  , which allows for mappings to be viewed, shared and edited as part of a community via any standard Web browser. Mappings are described by mapping rules relating subject-property-object triples from the source scheme to equivalent structures in the target scheme, subject to various syntactic conditions, as illustrated in Figure  6.
As a catalogue front-end, the Portal can be compared to community aggregators such as EUDAT's B2FIND service, which uses CKAN for content management. Our portal uses the CERIF standard to structure its data, which provides a flexibility not found in more rigid schemes such as used for B2FIND, but this flexibility comes at the cost of additional complexity in the underlying data model. This is offset however by the use of a robust metadata mapping pipeline and by a simple-to-use user interface for constructing queries. We can also compare with RIspecific data portals such as the ICOS Carbon Portal 28  for greenhouse gas data. The ICOS portal uses Semantic Web technologies just as our portal, but only serves data for a specific RI, and uses a more specific set of facets for locating specific datasets.
Further Development The VRE4EIC Metadata Portal was demonstrated to the ENVRI community cluster of environmental science RIs in Europe as well as directly to the European Plate Observing System (EPOS)  29  , with sample data harvested from a cross-section of RIs across the ecosystem and solid earth science domains, further augmented by synthetic data for a total data corpus of approximately 53 million RDF triples as of October 2018. Feedback was broadly positive, but indicated a number of specific improvements that would make this kind of metadata portal more useful to the RI community. We now examine these key improvement areas, and discuss any ongoing developments which address them.
Better handling of under-defined metadata CERIF was originally designed based on a relational database model and so consequently defines a number of strictly disjoint entity classes without any kind of default hierarchy as would typically be found (for instance) in an ontology. This grants CERIF a certain flexibility (since it allows arbitrary relations to be defined between any two entities), but can cause difficulties where metadata elements are under-defined. As an example, CERIF strictly distinguishes between people, organisation units and facilities, but does not formally define a more generic 'agent' concept. In a number of cases however, we found that source metadata records would define certain agents (such as 'publisher', 'creator' or 'owner') that could be people, organisations or institutes without any definitive way to distinguish between them.
Without the means to map to a precise entity class, these concepts would be lost upon translation into the strict CERIF standard, thus raising the question of how to manage 'graceful degradation' where important but semantically-ambiguous entities are to be found in harvested metadata. One possibility is to make use of CERIF's support for probabilistic relations between entities, or to make use of additional information sources to disambiguate entities. The use of additional external information sources, such as registries of entities (people, organisations, institutions, etc. ) used in metadata records from a given source (e. g. the RIDE database  [40]  for EPOS) is being investigated to help with disambiguation of under-defined metadata. Text-based analysis could also help (given that most ambiguous fields typically use free text), but essentially trades away precision for greater recall, which may not be acceptable for catalogues made available to scientists as 'production-ready'.
Greater exploitation of common terminology A notable feature of CERIF is how it separates its semantic layer from its primary entity-relationship model. Most CERIF relations between two entities are semantically agnostic, lacking any particular interpretation beyond identifying a link. Almost every entity and relation can be assigned though a classification drawn from a classification scheme that indicates a particular semantic interpretation (e. g. that the relationship between a Person and a Product is that of a 'creator'), allowing a CERIF database to be enriched with concepts from an external semantic model (or several linked models). Though different vocabularies (e. g. ISO 19115 codelists) were investigated in the VRE4EIC project as possible classification schemes for CERIF entities and relations in the context of environmental science, a harmonised set of schemes would be needed for any particular instance of the portal to fully exploit CERIF's semantic layer.
The vocabulary provided by OIL-E has been identified within VRE4EIC as a means to further classify entities and relations between entities in CERIF in terms of their role in an RI, e. g. classifying individuals and facilities by 'persistent data' 'QA-assessed data', 'annotated data' Service 'computational object' 'catalogue service', 'data broker' Table  1 : Examples of OIL-E classifications of CERIF entities: the OIL-E concept that acts as the classification scheme is identified along with examples of sub-concepts that act as classification instances. For readability, the concepts' RDFS labels rather than URIs are used. the roles they play in research activities, datasets in terms of the research data lifecycle, or computational services by the functions they enable. This provides additional operational context for faceted search (e.
g. identifying which processes generated a given data product), but providing additional context into the scientific context for data products (e. g. categorising the experimental method applied or the branch of science to which it belongs) is also necessary. An overview of OIL-E concepts that can be used to classify various CERIF entities and relations was published by the VRE4EIC project  [41] ; Table  1  shows some examples of such classifications. Classifying CERIF entity classes such as person, facility, result product or service using OIL-E concepts such as environmental scientist, data provider, persistent dataset and virtual laboratory is simple enough, but OIL-E can also be used to classify various classes of RI activity involving interactions between instances of CERIF entity in a way that is particularly suitable for describing time-bounded events involving those entities.
Integration of semantic search facilities The identification of synonymous, subsuming and intersecting terms provides the basis for better semantic search, whereby a greater range of data products with similar characteristics can be retrieved on query without necessarily sharing precisely the same controlled vocabulary for their metadata. Currently, the VRE4EIC portal principally supports faceted search based explicitly on entity classes with keyword filters, but making use of linked vocabularies would allow for more 'free text' searches alongside structured search, and would also simplify the task of integrating resource metadata from multiple catalogues, as it would reduce the need to map all metadata values into a single master vocabulary (with the likely resulting loss of nuance), while still retaining the benefits of cross-RI search and discovery.
Discussion Any sustainable VRE cataloguing solution will need to address certain challenges, including how to integrate new RI resources, handle updates to standards, scale with ever-greater data volumes, and ensure proper attribution of credit for data made available to researchers. All of these challenges require both technical and governance solutions broadly supported by research communities, requiring continued collaboration between various interest groups. In this section we make observations on topics that relate to VRE development in general, and indicate where our own contributions intersect with them.
Integrating new resources More data, more data resources and more research infrastructure all place additional pressure on 'seamless' integrative environments. Standardisation in protocols, schemes and vocabularies remain the best mechanisms for dealing with greater heterogeneity in distributed data volumes, but there is always need for data mapping, especially across disciplinary boundaries. As joint catalogues are mainly concerned with metadata rather than the actual datasets themselves, aggregated catalogue data do not tend to fit the profile of 'big data' in terms of raw volume, but the act of synthesis and integration itself is still challenging. The use of frameworks such as X3ML and flexible target schemes such as CERIF or GeoDCAT-AP  [43] , can make this integration viable, while the use of a standard metadata mapping framework with tool support (e.
Maintaining the catalogue It will be necessary to periodically refresh the content of any joint catalogue as datasets are cleaned, extended and updated at their source. Datasets hosted by different RIs have varying update regimes, meaning that a single policy (e. g. update every 24 hours) is not practical. In practice, updates can be pulled by the VRE (via periodic polling of RI resources) or pushed by the RI (by broadcasting updates to subscribers). The latter approach is desirable, but requires RI communities to support some kind of subscription mechanism for VREs. The Euro-Argo RI 35 has been developing a data subscription service for researchers  [46] ; a VRE subscription service running on similar principles may be feasible. Whether a push or pull model is used to acquire metadata updates, a joint catalogue should maintain a history of changes to metadata, as an aid both to search and to general reproducibility.
In the geospatial domain prominently occupied by current environmental science RIs, most standards have been developed independently of the Semantic Web, with recommendations such as INSPIRE 36  being all but disjoint from it, though technologies such as GeoSPARQL 37  do attempt to address this by bridging the capability gap. This current separation poses a barrier for integration of geospatial catalogues published via CSW or OAI-PMH into the Semantic Web, and adaptors are still needed to query such data sources and present responses in RDF format (e. g. [54] ); generic standards such as R2RML  [55]  make it easier now however to construct RDF-based views on relational data in databases, which helps create more seamless interoperability between the Semantic Web and other data frameworks.
Data objects and workflows Most scientific investigations follow a workflow, and there have been a number of workflow management systems developed with different characteristics and target applications  [56] , several of which have been applied to science  [57]  and to VRE applications  [58]. The use of ontologies for verification and validation of workflows has already been explored (e. g. [59] ), and the ability to construct and validate such workflow specifications using metadata from service catalogues demonstrates that the cataloguing problem is not wholly centred on datasets, but to any resource that researchers may want to discover and access. Integration between workflow systems and provenance recording is essential to the reproducibility of research results. To this end, VREs must be able contribute as well as use provenance data in their integrated workflow systems (e.
Related to this, there remains a broader question in data science regarding the accessibility of 'dark data'  [62] datasets produced by individual researchers and small research teams not represented by any of the large RIs and therefore not discoverable via their catalogues, but perhaps only by smaller institutional repositories. In this paper, we considered only the coupling of VREs with the curated assets of formally amalgamated RIs with mature data management systems and policies, but it may also be worth giving more consideration as to how data produced in the long tail of science can be made visible via a VRE; for example, making use of the catalogues of open repositories such as Zenodo 38 , or increasing the visibility of small institutional repositories.
Governance The construction, deployment and maintenance of joint catalogues for use by VREs is as much a governance problem as a technical one. The use of standard protocols and terminology that make the production of joint catalogues much easier requires consensus across the research community, and this requires effective forums in which stakeholders can hold dialogues and agree best practices-hence community initiatives such as ENVRI for the environmental and earth science RI community, and RDA for data science in general, which then can make recommendations. These recommendations influence the work of VRE developers. There is strong correlation between the metadata element set recommended by the RDA Metadata interest group and the primary entities defined by CERIF; likewise, the e-VRE reference architecture was conceived to directly address the need for standard VRE architectures espoused by the RDA VRE interest group.
The use of standard terminology for classifying entities is also heavily influenced by communities such as ENVRI. This influ-38 https://zenodo. org/ ence is natural, and demonstrates the importance of such bodies to this kind of work. In general, governance of evolving standards for data science belong to those initiatives that have the support of the scientific community. For VRE developers, this means that close collaboration is essential, as is the agility to adapt to the continued evolution of the RI resource landscape. This adaptability is key, because while the fundamental requirements of data science as embodied (for example) by the FAIR principles are becoming increasingly well-understood, the specific technologies and methods used to address these principles will continue to change; hence there also need to be equivalent principles for generic, flexible VRE design, at the core of which is how we gather resource metadata into searchable, expressive catalogues as the basis for nearly all key VRE services.
Conclusion In this paper we linked the development of VREs (alternatively referred to as science gateways or virtual laboratories) to the outgrowth of dedicated RIs providing curated data services to research communities, and argued the need for new VREs that can be freely coupled with different RI resources based on the evolving requirements of researchers and of data science. In particular, we argued that to provide researchers seamless discovery and access of RI resources, it is necessary to build VREs that can interface with as wide a range of resources as possible, brokered by catalogue services either provided by the RIs themselves or created at the VRE side by harvesting metadata from RI resource catalogues. In order to realise such a network of linked infrastructure and catalogue services however, we asserted that some degree of metadata mapping is essential to facilitate cross-RI search and discovery, mostly due to the fundamental diversity of metadata schemes, vocabularies and protocols used to access resource catalogue data published by different RIs, but also due to idiosyncracies in how such schemes, vocabularies and protocols are used in practice.
We examined how metadata might be aggregated into a single logical catalogue, comparing the approach of actually harvesting metadata to make a single physical catalogue versus simply brokering requests redirected towards multiple separate catalogues. We also looked at how such aggregation is currently being performed within RIs and RI clusters. Focusing on scenario of the creation of a joint catalogue drawing metadata from heterogeneous sources, we outlined a methodology for building such a catalogue based on the e-VRE reference architecture. We also examined the steps required for building a robust mapping pipeline for handling heterogeneous metadata, which is essential for building such catalogues. We provided an example in the VRE4EIC Metadata Portal of how our methodology is applied in practice; in this instance, the VRE4EIC project took the approach of building a single centralised catalogue using CERIF, a European research information standard, as a framework for aggregating resource metadata from different metadata catalogues provided by members of the ENVRI cluster of environmental and earth science RIs.
We described the use of the X3ML framework to produce effective mappings from XML-based metadata records to RDF data suitable for building a unified knowledge graph. We used 3M, an X3ML editor and transformation platform, to translate ISO 19139 XML, CKAN, Dublin Core, DCAT-AP and OIL-E data into CERIF RDF for ingestion into a CERIF RDF knowledge graph hosted within a Virtuoso data store. Based on the feedback given by RIs involved in VRE4EIC and in the ENVRI community, we identified key areas where more work was necessary, and described the ongoing development that is being done or is planned to address these areas for future iterations of the Portal and other e-VRE services. Finally, we discussed more broadly some of the issues that may bear impact on VRE and VRE catalogue development in general, such as the refreshing of metadata records, the coupling of VREs with other types of service provided by RIs, and the need to closely follow the activities and recommendations of community initiatives for establishing standards for data science in specific domains and in general.
Introduction Currently, there are several sets of in situ bio-optical data, worldwide, suitable for validation of ocean-colour satellite data. Whereas some are managed by the data producers, others are in international repositories with contributions from multiple scientists. Many have rigid quality controls and are built specifically for ocean-colour validation. The use of only any one of these datasets would limit the number of data in validation exercises. It is, therefore, vital to acquire and merge all these datasets into a single unified dataset to maximize the number of matchups available for validation, their distribution in time and space, and, consequently, to reduce uncertainties in the validation exercise. However, merging several datasets together can be a complicated task. First it is necessary to acquire and harmonize all datasets into a single standard format.
Second, during the merging, duplicates between datasets have to be identified and removed. Third, the metadata should be propagated throughout the process and made available in the final merged product. Ideally, the compiled dataset would be made available as a simple text table, to facilitate ease of access and manipulation. In this work such unification of multiple datasets is presented. This was done for the validation of the ocean-colour products from the ESA Ocean Colour Climate Change Initiative (OC-CCI), but with the intent to serve the broader user community as well. A merged dataset is not without drawbacks: it is likely to be large and so not always easy to manipulate; because the merging is done on pre-existing, processed databases, it is not possible to have full control of the whole processing chain; the dataset would be a compilation of observations collected by several investigators using different instruments, sampling methods and protocols, which might eventually have been modified by the processing routines used by the repositories or archives.
To minimize these potential drawbacks, we have, for the most part, incorporated only datasets that have emerged from the long-term efforts of the ocean-colour and biological oceanographical communities to provide scientists with high-quality in situ data, and we implemented additional quality checks on the data to enhance confidence in the quality of the merged product. Nevertheless, it is still recognized that different and unpredictable uncertainties may affect data from the diverse sources as a result of the application of a variety of field/laboratory instruments, methods and data reduction schemes. In Sect. 2 the methodologies used to harmonize and integrate all data, as well as a description of individual datasets acquired, are provided. In Sect. 3 the geographic distribution and other characteristics of the final merged dataset are shown.
This is the second version of the compilation of global bio-optical in situ data described by  Valente et al. (2016). A track-change file of the manuscript of the first version can be found in the Supplement. The new version has more data and a higher temporal and spatial coverage. The increases in the number of observations are mainly for chla, rrs and aph. In comparison with  Valente et al. (2016) , the observations of chla and aph have doubled in number and provide a better spatial coverage, especially in the Southern and Arctic Ocean. The rrs values also increased in number, but not as much in spatial coverage, because most of the new observations came from fixed locations. The present second version is a compilation of data from sources used in the first version (MOBY, BOUS-SOLE, AERONET-OC, SeaBASS, NOMAD, MERMAID, AMT, ICES, HOT and GeP&CO) plus data from additional sources (AWI, ARCSSPP, BARENTSSEA, BATS, BIOCHEM, BODC, CALCOFI, CCELTER, CIMT, COAST-COLOUR, ESTOC, IMOS, MAREDAT, PALMER, SEA-DATANET, TPSS and TARA).
The main differences from the first version are (1) some of the data sources used in the first version were updated (MOBY, AERONET, SeaBASS and HOT), (2) new data sources were added, (3) a new variable was compiled (total suspended matter), (4) the format of the database was modified and (5) two new flags were added. Concerning the change in format, in  Valente et al. (2016)  the compilation was provided as one unique two-dimensional table. Now, given its increased size (136 250 rows and 1286 columns compared with 80 524 rows and 267 columns previously), the table has been broken into three smaller tables that relate to each other via one unique key identifying each row. One additional table is also provided to help with data manipulation. Despite this change, the compilation should still be viewed conceptually as one unique table, and as such, it is still described in that way.
One requirement for chla_fluor measurements was that they were made using in vitro methods (i. e. based on extractions of chlorophyll a). Although this severely decreased the number of observations, since in situ fluorometry (e. g. fluorometers mounted on CTDs) is widely available in oceanographic databases, it was decided to exclude such data because of potential problems with the calibration of in situ fluorometer data. The variable chla_hplc was calculated by summing all reported chlorophyll a derivatives, including divinyl chlorophyll a, epimers, allomers and chlorophyllide a. The two chlorophyll variables are retained separately in the database to facilitate their use. HPLC measurements could be considered of higher quality, but fluorometric measurements are more numerous. Thus one option for users is to use chla_fluor only when there are no chla_hplc measurements available.