dataesr/scientific-paragraphs-categorization huggingface dataset:
For instance, the popular NASA Coordinated Data Analysis Web (CDA-Web) interface https://cdaweb. gsfc. nasa. gov/ allows selection by mission name and general instrument types, with datasets and variables described by text strings and time ranges, as provided by the ISTP metadata internal to the datasets; but the system works only with data stored in specific data formats, CDF and netCDF (see section 2. 2. 4. 1). Some archives also provide direct access to data files in a directory hierarchy, where selection metadata are essentially the names of the directories and files, which requires custom parsing to find, for instance, the begin time of the data in a file (see section 2. 2. 4. 4). This parsing can be described with standardized template descriptions, such as, https://github. com/hapi-server/uri-templates/wiki/Specification, which is more useful when datasets and filenames follow common recommendations https://spdf.
gsfc. nasa. gov/guidelines/filenaming_recommendations. html. Like the NASA CDAWeb, the ESA heliophysics archives offer the possibility to select multiple datasets by time range to either download or visualize them. Advanced search tools are also available such as the Cluster data mining tool, allowing science based searches over an entire archive (https://caa. esac. esa. int/data-mining/data-mining/ ). However, search capabilities could be improved to make use of the full potential of the detailed metadata included in the files. For instance, detailed metadata could be used to access and retrieve data related to direct current (DC) electric field (already stored in the Cluster Science Archive (CSA) metadata). Thanks to these metadata, users could retrieve not only measurements from classic DC doubleprobe sensors, but also high-level DC electric field data products derived from particle experiments, which are already available (e.
Data formats. A data format specifies how information or data content is digitized and organized (and stored) in a data file. It is easy to imagine even for a simple, time-series dataset written into an ASCII (American Standard Code for Information Interchange) table or a spreadsheet, correct use of the data is possible only if proper metadata descriptions of the data file (type and format) are available to the user before it can be independently used. Community users are often confronted with (1) not knowing the specific file and data formats in advance of acquiring the data, (2) having to deal with resources in different formats that they might not be familiar with, and (3) not knowing what tools are available and suitable for dealing with the data once they have acquired it. The lack of knowledge about file and data formats and related tools invariably hinders the effective use of the data resources by the wider Heliophysics and Space Weather community.
Due to their special capability of handling multidimensional data records and embedding of structured metadata, the so-called ''self-describing and selfdocumenting" data formats, such as the Common Data Format (CDF, https://cdf. gsfc. nasa. gov/), the Flexible Image Transport System (FITS, https://fits. gsfc. nasa. gov/), and the Network Common Data Form (netCDF, https:// www. unidata. ucar. edu/software/netcdf/), have been the mainstay of data formats used for storing heliophysics resources. Since the availability of metadata is guaranteed for these file formats, they are often preferred to the more traditional formats (e. g. , ASCII) that require separate metadata documentations, which could be lost easily without having the embedded metadata. In addition, the use of standard data formats with standard metadata specification agreed upon at the international level [such as the ISTP Guidelines (https://github.
com/IHDE-Alliance/ ISTP_metadata/) for CDF and netCDF files or the World Coordinate System (WCS) for FITS (https://fits. gsfc. nasa. gov/fits_wcs. html)] would ensure interoperability of the resources stored in those file formats. The drawback of these binary standard formats, however, is that the embedded metadata is not fully accessible without opening the data files, making the data less convenient to be searched and found (i. e. , less supportive of the ''F" in FAIR). An intermediate solution was found in the ASCII-based Cluster Exchange Format (CEF) with associated structured metadata and a detailed metadata dictionary that was developed by ESA for the Cluster and the Double Star missions. It enables storing of multi-dimensional time series datasets and images. Structured metadata is embedded at the top of any file and read by various data analysis software.
Providing clear, unambiguous meaning of the contents of the data is of utmost importance for supporting point (4) above and enabling independent use of the data. Given the diversity of measurement platforms, instruments, measurement types, and measured quantities in heliophysics, there is a great deal of complexity in heliophysics data. Similar measurement quantities appearing in different datasets may have different coordinate systems, units or measurement cadence. Without proper definitions and explanations, it would be hard for datasets to be utilized by users from different disciplines. This is akin to different people trying to communicate in different languages. The most effective way to enable independent use of the data by diverse users is the adoption of a metadata standard that can serve as a lingua franca between different disciplines, data services and systems.
We describe in the next subsection some general guidelines on how data contents at the measurement parameter level should be described in order to enable independent usability of the data. 2. 2. 4. 3. Metadata for parameter descriptions. While global attributes of a digital resource are needed for identifying and locating the resource, they do not contain specific information about the contents of the resource. And, while the data file format specifies how resource contents are organized, parameter descriptions are needed for explaining the contents of the resource. Complete and unambiguous parameter descriptions are the key to understandability, thus making the resource independently usable at the end of path (4) in Fig. 1  and ensuring the ''I" and ''R" in the FAIR principles. Metadata provides detailed descriptions of data formats, datasets, parameters, time conventions, and dataset and file naming conventions enabling effective data analysis and browsing with generic easy-to-use software and web services.
Restricting metadata descriptions to standard, and thus uniform, representations would limit the number of equivalent possibilities which software must deal with and would thus foster interoperability. Conventions help standardize ways to name things, represent relationships, locate data in space and time, and abstract the general data models to represent the data semantics. This enables the development of reusable applications with powerful data extraction, gridding, analysis, visualization, and processing capabilities. These standards embody the data provider's experience and capture the meaning in the data, making the data semantics accessible to humans as well as software tools. Higher-level abstractions such as coordinate systems and standard names for physical quantities further facilitate comparing different datasets and distinguishing between variables.
The ISTP Metadata Guidelines mentioned above (section 2. 2. 4. 1) have been in use by the traditional space physics community over the last few decades mostly for time series data. It provides an internally and logically complete set of metadata to enable a dataset to be correctly and independently usable, especially by generic (non-dataset specific) automated processing, analysis, and display software. The guidelines define self-documenting (internal) metadata for data stored in CDF and netCDF files and include general file naming conventions. Data are time-ordered and time-identified, with standardized time formats. The ISTP Metadata Guidelines define a set of required and optional global (dataset-level) and variable (parameter-level) attributes. Variable attributes can point to other variables by name and carry arguments, and thus may convey information about relationships among variables.
The ISTP guidelines may thus provide a good candidate for paving the way toward standardizing parameter descriptions for time series data. 2. 2. 4. 4. Data access protocols. Data access protocols are used to access, retrieve, and deliver a specific element of digital content to the user. Before the execution of an access protocol, however, the specific element of digital content defined by, e. g. , a dataset name, measurement parameters and time range, must first be identified and located, requiring the use of metadata (see paths (2) and (3), or (3a), and (4) in Fig. 1 ). This means that metadata is needed to first define the resource object for an access protocol to access, satisfying the ''A" in the FAIR principles. If the same access protocol is used generally by different data sources (data providers or repositories in Fig.
1. Two protocols are used at the NASA SPDF to access time series data: HAPI (https://hapi-server. org/servers/,  Weigel et al. 2021a ) and the Coordinated Data Analysis System (CDAS) Web services. They and a few others currently in use for accessing heliophysics resources are described briefly below and summarized in Table  3. The HAPI protocol is an API with endpoints that allow access to time series data values at the parameter level within one or more data collections. Since HAPI is focused on data access, the required HAPI-specific metadata is not intended for complex search and discovery. However, the metadata schema can provide information as to where more descriptive details for any dataset could be found. The HAPI API is based on REpresentational State Transfer (REST) principles (https://en.
There is also a Python library (https://pypi. org/project/cdasws/) which provides a simple python interface to the CDAWeb data and services. Solar data collections at the SDAC are primarily delivered through the Virtual Solar Observatory (VSO). VSO provides access to distributed solar data collections stored around the world, all described with the VSO data model. Two popular tools in the solar community have been developed to access the VSO API: the IDL Solar SoftWare (SSW) package (SolarSoft) and the SunPy  (Mumford, et al. , 2021)   3 ). This data access protocol enables access to both data and metadata for time series and remote sensing data. The Cluster Science Archive (CSA; see Table  2 ) and Solar Orbiter ARchive (SOAR; see Table  2 ) in particular contain more than two thousand datasets with detailed metadata, some even described in SPASE (see section 3.
, 2018). Even though the SPASE information model and the SPASE Group got their start in the international community at the grassroots level, their development, maintenance, and operations have primarily been sponsored by NASA. The SPASE Group, however, is still an international heliophysics community group and remains a grassroots effort to ensure community involvement and as such is open to all interested parties (https://spase-group. org/connect. html). In addition, NASA has established a SPASE metadata working team (SMWT, https://hdrl. gsfc. nasa. gov/smwt_home/smwt_index. html) to assist the community in implementing the SPASE model and to produce and maintain SPASE metadata. Consequently, NASA currently maintains the largest, openly-accessible SPASE metadata registry on Github (https://github.
We note here that all heliophysics digital resources (dataset, model, modeled data, etc. ) can be cited similarly by using DOI URLs and SPASE description landing pages. Due to its specific applicability to heliophysics data, SPASE has been adopted by the COSPAR Panel on Space Weather in 2018 as a recommended metadata standard for describing space physics and space weather resources (COSPAR Panel on Space Weather, 2021). The subsections below give an overview of the SPASE metadata schema and a description of how heliophysics digital resources can be described and referenced, enabling their archival, searchability, accessibility, understandability, and independent reusability.
Identifying datasets and tracking their origins Minimizing barriers to the flow of data is the key to data sharing and enabling effective use of the resources by the international community. In an environment where resources are freely exchanged, however, it is easy to lose track of the origin or the evolutionary track of a resource. It is thus important to identify and track the origins of and changes to resources while sharing the resources in accordance with the FAIR principles  (Wilkinson et al. , 2016). We describe here how the origin of a resource is built into the SPASE metadata model and used for tracking. We then describe in section 3. 5 below how digital resources can be cited and referenced by using Digital Object Identifiers (DOIs). It is important to note that the origin of an object, its source or history of ownership, is referred to as the provenance of the object in the English language [see for examples, the Merriam-Webster (https://www.
While a Naming Authority is typically the funding agency that sponsored the creation of the resource, many projects, observatories, or even instruments, particularly in ground-based facilities, are sponsored or managed by multiple agencies. It is sometimes difficult to determine which agency holds the ultimate oversight responsibility. In that case, the SPASE Group has recommended a set of rules for establishing a NamingAuthority for both space-based and ground-based resources (see https:// spase-group. org/services/naming-authority. html, where a number of currently defined Naming Authorities are also listed). The rest of the ResourceID URI in the example for an observational dataset above then consists of item identifications in the Data domain (ResourceTypes), the Origination domain (Project, Observatory, and InstrumentType), and the specification of the most distinctive or representative characteristic of the resource (e.
g. , time cadence) to make up a unique path for identifying the described resource (see also section 4. 1. 3. 1). ResourceIDs for other types of resources (see Fig. 2 ) can be constructed similarly to the above example. Since all digital resources are uniquely identified by their corresponding ResourceIDs, the SPASE URI structure thus provides a logical way to organize all the SPASE resources on the SPASE registry (see https://hpde. io for the SPASE registry landing pages). We should note here that there is not yet a globally recognized general SPASE metadata registry. While the SPASE Group maintains the largest SPASE metadata registry (https://github. com/hpde/), individual data services such as the Automated Multi-Dataset Analysis tools (AMDA) of the CDPP, ESAC science data centre (ESDC), and EuroPlanet, have also set up their own metadata registries based on their adopted metadata models: SPASE, TAP, and EPN-TAP, respectively.
Digital resource description As pointed out in section 2. 2. 4. 2 (metadata), a user must be able to do four things in order to use the resource effectively and independently after retrieving a digital resource. As noted also in section 2. 2. 4. 3 (parameter descriptions), independent usability of a dataset depends on the clarity, correctness, completeness, and self-sufficiency of the metadata describing the dataset. Fig. 3  shows an example of the hierarchy of SPASE fields and labels for identifying, locating, and understanding each of the parameters of a numerical data resource. The NumericalData resource schema has two important metadata containers that need careful attention when generating data product descriptions. The first is the ResourceHeader, which includes a ResourceName text box and a Description text box among other things (see Fig.
3 ). The ResourceName should list a unique dataset ''title" along with the data processing level and time resolution. The Description text field allows one to provide a much more complete account of the dataset content. One can utilize other parts of the ResourceHeader schema to shorten the overall Description and reduce the work required to write SPASE descriptions. For instance, one can provide the DOI and PublicationInfo for an already published paper or other resource that describes the data (see section 3. 5). InformationURL metadata can also be used to link to web sites that describe the data product, though it is important to remember that URLs are not guaranteed to have the persistence of DOIs. It is also important to provide a complete list of the people involved in all aspects of the data product including generation, access, and archival by using the Contact portion of the header schema.
A more complete view of the SPASE metadata layout, including the simulation extensions, can be found in Fig. 1  of  Roberts et al. , (2018). To understand the meaning of each tag or term in Fig. 3  and the associated information pertaining to a dataset being described, the readers are encouraged to consult the SPASE metadata model (https://spasegroup. org/data/model/index. html) (SPASE Group, 2021) and the SPASE dictionary (https://spase-group. org/data/ model/search/index. html). Since not everyone is well versed in XML and the SPASE schema, the SMWT has developed a web-based SPASE editor (https://xmleditor. spase-group. org/) for anyone to create and edit SPASE XML documents without needing to know XML and SPASE. Under the auspices of the NASA HDRL, there is ongoing work by the SMWT to assist the community in creating and maintaining SPASE descriptions, and registering them at the SPASE registry mentioned earlier in section 3.
One of the projects called ADAPT, which stands for Active Data Archive Product Tracking  (Bargatze, 2018; Bargatze et al. , 2022) , leverages metadata embedded in data resources that are stored in self-documented data formats, such as CDF and netCDF, to automate the generation of SPASE data product descriptions. The ADAPT tool kit currently comprises a core set of IDL programs designed to create, populate, and write data product descriptions for each resource type defined by the SPASE data model. The ADAPT IDL routines are also supplemented by a set of support software built to harvest metadata from self-documented data products archived at and made available from data repositories and services (sections 2. 2. 2 and 2. 2. 4). The support software also scrapes textual information from other sources, including previously generated SPASE resource descriptions already registered and in use in the global Heliophysics data environment (Fig.
1 ). The Heliophysics data environment, ADAPT and its support software thus form an ecosystem in which the process of SPASE metadata generation and updating can be effectively automated by regularly harvesting metadata and rerunning ADAPT over any new and updated metadata. In the present context, ADAPT uses metadata present in ISTP compliant data files (section 2. 2. 4. 3) to generate SPASE descriptors via mapping of CDF (operational) and netCDF (developmental) global and variable attribute content into the appropriate text field elements appearing in SPASE (see Appendix B). The ISTP-to-SPASE metadata mapping is relatively robust, but it cannot always be trusted to generate valid SPASE data product descriptions. Also, sometimes the ISTP attribute text needs standardization, correction for errors, or translation of an ISTP term into its SPASE equivalent cognate.
Model and simulation data description Fig. 4  shows a more detailed ontology of the SPASE extensions for describing simulation models and their data (https://spase-group. org/data/simulation/). The readers are reminded that arrows in the schematic point in the direction of association, as in Fig. 2. Generally speaking, simulation models and model results are described separately although model run descriptions should also include the model identifiers and their associated setup for executing the model runs. These extensions were added to the SPASE model in parallel with the observational heliophysics data products beginning in version 2. 2. 4 in 2015. The SPASE model simulation extensions were originally developed by members of the Integrated Medium for Planetary Exploration (IMPEx) project, which was funded by the European Union under the Seventh Framework Programme with the goal of bridging ''the gap between observational databases and scientific modeling tools'' (see https://impex-fp7.
Resource citation and referencing A Digital Object Identifier (DOI) provides a unique, stable reference to a piece of digital content including articles, observational datasets, model output, and software. The DOI overarching authority is the International DOI Foundation at https://doi. org/ Many members of this group, notably CrossRef (https://www. crossref. org/) mainly for scholarly articles and DataCite (https://datacite. org/) for datasets, give users the ability to ''mint" (create) DOIs. There is considerable leeway in what qualifies as a data ''digital object," and it can vary from a table of numbers to a petabyte of files or images, such as the nearly 3 petabytes of data collected by the Solar Dynamics Observatory during its five-year nominal mission. The DOI is a handle to a ''landing page" that provides not only a persistence reference to a resource but also the full metadata description of the resource, such as access URLs and methods, descriptions of content including variables, caveats for data use, and potentially other things (as shown in Fig.
This form of DOI referencing for digital resources is more versatile than the traditional use of DOIs for referencing publications. There are various degrees of granularity that can be invoked in minting DOIs. The general practice for NASA Heliophysics is that any NASA dataset that has a SPASE description and is publicly accessible is assigned a DOI. For example, the 1-min resolution magnetometer data from the ISEE-2 spacecraft in various coordinate systems has been collected as a set of files that are registered as a SPASE dataset. It has been given a DOI (https://doi. org/ 10. 21978/p8t923). Clicking on the DOI (note that the full URL for the DOI is the expected form of the DOI) brings up the landing page that includes an example of how to cite the dataset along with access methods (under AccessInformation; see Figs.
3  and 8 ) and a great deal of other metadata. All of the information on the landing page is essentially the SPASE description. Value-added data products, derived from the measurements of multiple spacecraft, like the very popular OMNI datasets have also been assigned DOIs (e. g. , https://doi. org/10. 48322/mj0k-fq60 for the OMNI version 2, 1-min resolution). Although over 10,000 datasets, mostly under the NASA NamingAuthority, have already been described using SPASE, NASA alone still has some way to go to mint DOIs for all the SPASE-registered data products. A given SPASE description of a resource, with PublicationInfo provided under the ResourceHeader [see Fig. 2  in  Roberts et al. (2018)  or Fig. 3 ], can nevertheless contain all the information required to mint a DOI. Using simple software tools, all the pertinent information can be extracted to create a landing page that will become the permanent DOI reference page.