dataesr/scientific-paragraphs-categorization huggingface dataset:
This is not surprising, given the technical and programmatic difficulties involved in exchanging and interpreting data produced at different institutions. In fact, not too long ago, several of us had to physically visit instrument home institutions to obtain the native data in their most useful, archival form: microfiche, print or 9 inch track tapes. Things have changed significantly since then. First, data repositories are now digitally accessible quickly and openly, through data repositories (e. g. , NASA's Space Physics Data Facility, SPDF, or NOAA's National Centers for Environmental Information, NCEI) or distributed databases (e. g. , Virtual Observatories, VxOs). Metadata (data descriptors) facilitating queries and on-line documentation are widely available for users to interpret these data.
) to kinetic scales (electron or ion gyroradius or inertial lengths). The emergent H/GSO (optimized through the utilization of THEMIS's fuel that allows optimal phasing of the THEMIS orbits with MMS orbits and ground-based observatories) has been endorsed as a concept by the Heliophysics Decadal Survey  [National Research Council 2013]  as part of its DRIVE initiative. This trend of increasing, distributed databases from multiple missions is expected to continue as Constellation Class missions  [Angelopoulos and Panetta 1998; Siscoe 1998 ] come closer to reality and ground-based assets increase in number, data volume and complexity. Increasing the science return gained from common data storage guidelines and an opendata environment is highly desirable to capitalize on the programmatic changes at NASA's Heliophysics division regarding data access.
At a time of an exponential increase in the number of space physics missions and ground-based assets, which need to work tightly together towards an efficient H/GSO, the duplicative effort of querying, acquiring, ingesting into a common analysis platform and visualizing time series data, including images, becomes disproportionately large compared to the time available for analysis and interpretation. Tools for efficient utilization and scientific exploration of such datasets are therefore highly desirable to increase the scientific return in this golden era of space research. The need for standardized ingestion and analysis tools to work across disparate space physics datasets has existed for decades but no comprehensive and robust solution had emerged until recently. This was not due to lack of trying.
g. , on WIND  (Lin et al. 1995)  and Cluster  (Rème et al. 1997). With the advent of the THEMIS mission, which structured its data analysis environment to benefit from these packages starting from its first proposal in 1998  (Delory et al. 1998 ) and implemented during its science analysis planning phase starting in 2003  (Angelopoulos 2008) , a general-purpose analysis system took shape independent of, parallel to, and using the same design principles as SSW. The THEMIS Data Analysis System (TDAS) relied heavily on legacy software but introduced (through a critical mass of software engineers) a regimen of quality assurance, suite-testing, version control, and backwards compatibility testing (subject to CDF and IDL version constraints), strong ties with SPDF, and community support. TDAS was designed to incorporate groundbased observatory images through CDFs and be expandable to ancillary datasets that were useful to enhance THEMIS's system-science objectives (e.
g. , more than 100 ground-based observatories, geomagnetic indices, NOAA GOES, WIND, ACE, etc. ). By the launch of the THEMIS mission in 2007 FITS files had also been incorporated in TDAS for better interface with SSW. Soon thereafter other projects (NASA's Van Allen Probes; the Japan Aerospace Exploration Agency's Energization and Radiation in Geospace, ERG  [Miyoshi et al. 2018a [Miyoshi et al. , 2018b]] ; the Japanese inter-university upper atmosphere global observation network project IUGONET  (Hayashi et al. 2013 ; http://www. iugonet. org); SuperDARN; NASA's Mars Atmosphere and Volatile Evolution mission, Maven  [Jakosky et al. 2015] ) started utilizing it to conduct or prepare for their own multi-instrument, multi-mission data analysis. Now, a decade after the first TDAS operational implementation, community endorsement of its design principles has led to the design of a software development superstructure, SPEDAS, a generalized software development platform, with distributed software developing teams and a core team of developers coordinating mission or instrument "plug-ins" based on a set of guidelines.
This flexibility tailors the installation to the user, and therefore ensures the widest possible utilization of SPEDAS by the community, maximizing the science return from Heliophysics' missions. In late 2017 NASA HQ provided SPEDAS modest funds to ensure continued community support of legacy missions (as well as of ancillary non-NASA missions critical for the emerging H/GSO), while the Ionospheric Connection Explorer  (ICON, Immel et al. 2018)  and Parker Solar Probe (PSP,  Fox et al. 2016)  have also started to develop their own plugins in anticipation of launch. Recent collaborations with the SPDF have led to successful ingestion of all ISTP-compatible SPDF data into the same software system as proof of concept of an emergent powerful new analysis platform. Thus, at this stage SPEDAS represents a grass-roots software development platform that requires only modest resources to maintain thanks to the in-kind contributions from dozens of programmers across inter-agency and international missions and ground-based observatories.
SPEDAS was developed on the premise of platform independence. Therefore, as software technologies evolve, options for SPEDAS's future evolution to keep up with the trends have to be thoughtfully considered. In that regard, it should be emphasized that the specific software implementation (presently using IDL) was based on considerations of code heritage, software development and maintenance cost, robustness, expert support staff availability, and other technical considerations. The SPEDAS structure, however, is transferable to other machine-independent languages or architectures. SPEDAS is thus expected to adjust as external (industry) conditions change and the relative weight of those considerations evolve with time. In this paper we introduce SPEDAS to the user and software developer communities.
In Sect. 2 we overview SPEDAS' design philosophy and goals, and in Sect. 3 we discuss its historical evolution that led to many of its design choices. Readers interested in learning quickly what SPEDAS is, how it can be used for analysis or how it can serve their project's needs (e. g. , with a plug-in) can bypass these two sections. Section 4, intended for the user community, outlines the SPEDAS top-level capabilities in various modes of use. It is a self-sufficient introduction to SPEDAS with references to more extended documentation, tutorials, YouTube videos and on-line recorded webinars. Section 5, intended for developers, explains the SPEDAS architecture, expandability, flexibility, data access control, "plug-in" concept implementation details, the division line between project ("plug-in") team and SPEDAS (core) team support and the personnel and software management structure and practices.
g. , the emergence of Python). SPEDAS was intended to support an efficient implementation of the H/GSO, in accordance with the intent of the aforementioned policy document and the recommendation of the National Academy's Heliophysics decadal survey to optimize Heliophysics/Geospace resources across missions from multiple agencies, and NSF's Geospace Environment Modeling, GEM, program  (Wiltberger 2015). Towards that goal, SPEDAS was designed to help support multi-mission, multi-instrument analysis by facilitating searching, downloading, and ingesting scientific datasets from space physics missions via their own mission data portals, mirror sites, the Space Physics Data Facility (SPDF), Virtual Observatories (VxOs) and/or NASA data centers, and by providing a number of visualization and analysis tools to take full advantage of the archived and currently streaming data.
(D) Modularity of code. Routines need to be separable into data introduction (query and data ingestion) routines, generic data analysis routines, and generic plotting and data export routines, in order to allow flexibility in contributions from projects, developers and the user community. This ensures that a distributed team of software developers, all experts in their own areas, are able to contribute efficiently to the SPEDAS project, rendering it most effective for the science community. (E) Community Support. It is critical that good documentation, user training, new plugin or analysis code development guidelines, a "help-line" for users, code maintenance and quality assurance are present. This allows SPEDAS to remain stable, robust and flexible. The current system implementation adheres to the above design principles and reflects the best choices that were made at a given instant in time.
Those were typically instrument-team efforts, treating other instrument or other mission data as ancillary products. Ancillary products were ingested mainly through key-parameter datasets whose delivery was contractually required to SPDF during the ISTP era. One particular effort related to solar optical instruments that has blossomed into a generic, multi-instrument, multi-mission piece of software with qualityassurance provisions, came from the solar physics community, SolarSoftWare, as was discussed earlier. Multi-instrument, multi-spacecraft space physics software for in-situ measurements naturally grew out of various team efforts utilizing higher-level language routines. UCLA's DFS (a modular analysis package operating on flatfile binary files) was rewritten in IDL in 1994, and was introduced and used internally at JHU/APL by the lead author of this paper as a standalone analysis package for ISTP data analysis.
Legacy code in the low-level language C was still critical for introducing FAST data into the user's analysis environment. Thus, the early version of the FAST IDL-based package relied on binary data introduced into IDL using the C-based SDT as the interface, leaving only the ancillary key parameter data from ISTP missions and indices to be introduced by standardized "*intro*" routines. By the late '90s it was apparent that although the initial FAST data analysis model was proposed and designed to primarily handle its data input and visualization through the C-based SDT, analysis was predominantly done at UCB and at affiliated institutions using the FAST IDL package, due to its platform-independence and low-maintenance requirements. However, as computers were becoming faster, different flavors of this IDL analysis package were transcribed for different instruments on SSL missions and proliferated quickly  (WIND, Cluster, other).
Conflicts in routine naming prohibited (or severely impeded) joint event analysis in the absence of a concerted effort to instill quality-assurance, the software remained limited to the confines of each individual mission, each cannibalizing code from the other. The first implementation of platform-independent (in-situ) space physics software that adhered to most of the design elements of SPEDAS was developed as the THEMIS Data Analysis System (TDAS) in the early 2000's. Using ISTP-compatible CDFs and routines that introduced Level-1 (uncalibrated, raw data) as well as L2 (calibrated data in physical units) it created an end-to-end solution for pre-launch instrument testing, post-launch on-the-fly data inspection and validation, and archival science data analysis. Additionally, given the THEMIS needs for all-sky imager (ASI) data access and analysis, CDFs for ASI images were also produced by the team (Sect.
At that time THEMIS was entering its third extended mission phase. ERG, IUGONET, NASA's Balloon Array for Radiation-belt Relativistic Electron Losses (BARREL) mission  [Millan et al. 2011]  and MMS had already opted to use the same software system to create their own, full-fledged, multi-instrument analysis packages. Other projects concurrent to THEMIS and its lunar extension, ARTEMIS, were also introduced into the TDAS software distribution (e. g. ,  Kaguya [Kato et al. 2010] ) benefiting the emergent H/GSO. At that time it was realized that the potential of TDAS to ingest archived data from most legacy missions (going back to Apollo), currently on-going space missions and ground-based observatory datasets which were parts of the H/GSO, as well as missions which were then under development (ICON, PSP) was very powerful.
In response to the growing requests for support of multiple missions the TDAS team initiated a code re-organization to create SPEDAS Version 1. 0, in its present incarnation, in August 2014. SPEDAS, a software superstructure, incorporated THEMIS's TDAS as a standard "plug-in" no different in priority or hierarchy than other missions. SPEDAS thus became an entity independent of THEMIS and its home institution. SPEDAS started to reside in http://spedas. org, and was operated as a grass-roots software development environment to serve the community of Heliophysics users with in-kind support from THEMIS and allied projects. While the initial responsibility of launching, organizing, maintaining, and supporting the greater community rested upon the initiating team, the expectation was that this responsibility could naturally evolve to other, more heavily endowed missions in the future.
However, by 2016 the task of supporting other missions was already exceeding the professed role of the THEMIS mission for H/GSO coordination, especially in an extended phase. Luckily, in response to a NASA request for proposals for a Heliophysics "one-stop-shop" analysis service, allowing efficient use of multi-instrument, multi-spacecraft datasets, the SPEDAS team was selected to carry out these services for 5 years, starting in September 2017. At that point, SPEDAS was endorsed by NASA and the team started to pro-actively incorporate additional missions (Cluster, Geotail,. ) and support extended services (link to Autoplot, development of Python solutions, link to Heliophysics API, etc. ). The current paper describes the status of SPEDAS well into its first formal year of existence under NASA support, and four years after the initial concept was first launched.
Inevitable operating system upgrades are taken care of by IDL and its version upgrades, while the SPEDAS software is maintained by the core SPEDAS team, to ensure that no new bugs are introduced from new IDL versions, while maintaining (and documenting) backwards compatibility to the extent possible. SPEDAS offers standardized access and import routines from distributed data repositories, written (at least for the highest resolution and highest quality mission data) by the individual mission teams (the experts). Multiple data processing levels and versions of the same dataset are allowed, if available at the remote site, with SPEDAS behavior controlled by standard keywords. Science processing and graphics are also standardized, containing a powerful set of legacy routines, good for all space physics datasets.
2. Data from more satellites for this event will be shown later. introduce such multi-instrument, multi-mission and ground-based observatory data, while demonstrating the large scale connections across space and ground for this event. For the purposes of Fig. 2 , shown in file "Figure02. pro" in the Supplementary Materials is a text file which serves as an IDL script, or IDL program (in this case the former); the commands can be inserted (cut-and-paste one-by-one or in multiple lines) in the IDL command window for execution. Programs like it can be run as a script in its entirety with a single command, in this case: "Figure2" (or equivalently in the IDL window press buttons "Compile", then "Run"). The latter choice should be used judiciously and is only warranted if the script has been completely debugged prior to execution.
pro. The most efficient way of viewing, analyzing and processing the data is by using them directly in their tplot variable form. Higher-level commands in SPEDAS are used towards this goal. For example, the command: "tplot, 'mms4_epd_eis_extof_proton_flux_omni_spin' " (here single quotes for the tplot variable are used on both sides but in the user's commandline double quotes are also acceptable) will result in a single-panel plot without the user having to specify any limits or colors or other plot attributes (assuming the quantity's dlimits were set properly upon loading by the load routine, if that was well-written by the instrument team). This is in fact a major advantage of the tplot system. It places the onus of the proper introduction and plotting of the variables on the instrument software team.
SPEDAS GUI Mode of Use The second main SPEDAS mode of use is the GUI. This does not necessitate a paid IDL license, but only a "Virtual Machine" IDL installation, which is free. This installation is bundled with the SPEDAS free distribution and the user need not download it from the IDL site separately (the SPEDAS team has paid the additional fees to be able to distribute it directly). The intent of the Virtual Machine is to allow third party software vendors to distribute IDL-based higher level applications (e. g. , medical tomography) that can be run on client machines without the functionality of IDL in command-line. However, thanks to SPEDAS's routine "calc", which can operate on tplot variables, arrays and variables, after parsing, interpretation and internal redirection to IDL operations, the Virtual Machine accepts data analysis through "calc".
The SPEDAS protocol is thus to introduce data even into the GUI mode first as tplot variables from the standard tplot variable load routines (behind the scenes). Once the data becomes accessible in memory the information it contains (data arrays, limits etc. ) lends itself to GUI data object representation too, in parallel to tplot representation. Therefore, GUI and command-line modes are entirely compatible with each other, and can be used in parallel (an action on the data in one mode has an immediate effect on the data views through the other mode). Thus, the command-line user can take advantage of the additional GUI functionality, such as searching and introducing any ISTPcompatible SPDF dataset; easily navigating through and selecting many mission products without any familiarity with data location and their read routines; menu-driven data analysis, high-quality plotting and visualization capabilities, to name some.
This builds mutual trust in the analysis and permits elevated scientific discourse even in an era of numerous, voluminous datasets available at their highest cadence at distributed sites. Fig. 5  An example of SPEDAS web-services mode of use: the ERGWAT tool. This web-based form creates, in the background, SPEDAS commands that load, analyze and plot data, to the user's specification. These commands are executed at the server institution in the SPEDAS command-line mode, and the resultant plot (in portable network graphics, PNG, form) is served on the user's screen. The user need not know SPEDAS, or have an IDL license (it is free), and the connection up-/down-link speed requirements are modest, set by transmission of (small size) commands and PNGs, not data. The web-services mode is also useful for education and capacity building in space science, especially at remote schools or developing nations  [Miyoshi et al.
All steps are taken through the web browser, so that knowledge of IDL or SPEDAS is not required. Specifically, ERGWAT users need not be familiar with IDL to understand the state of their program's execution or infer, from someone else's command-line history, the steps they would need to take on the ERGWAT menu to recreate that plot. After obtaining the desired plot on the screen, the associated postscript file can be downloaded from ERGWAT via the web-browser for use in publications. The web-services mode can operate free-of-license for the user as all operations are executed at a number of licenses at the server institution. In addition to its potential for promoting research by engaging researchers to readily conduct data analysis and obtain plots with little upfront time investment, the web-services mode has high potential for education and capacity building in space physics.
Tutorials exist in the SPEDAS wiki (http://spedas. org) "Introduction" section for many levels of users. The un-initiated will find the YouTube Channel video link "Starting with SPEDAS" most useful; it contains installation instructions and how to create one's first plot. The "Introductory Examples" and "SPEDAS Screenshots" offer command-line and GUI examples of how to download and plot some data. Beyond that, tutorials that have been presented previously at meetings (e. g. , in conjunction with the American Geophysical Union's general assembly or at the annual GEM meetings) are shown under "SPEDAS Presentations" and represent the next step of sophistication in tutorials. They expose the user to the full complement of software capabilities. Tutorials for individual missions also exist (for THEMIS, MMS, BARREL and others) in subdirectory "older presentations".
Beyond the aforementioned generic SPEDAS and plug-in tutorials, there exist twodozen detailed examples (crib-sheets) in the software distribution under subdirectory "general/examples" showing how to introduce, analyze and plot SPEDAS data in command-line mode. Many mission-specific crib-sheets also exist at the top-level of each mission under a subdirectory "examples". For instance, "projects/mms/examples/basic" contains ∼ 3 dozen examples showing basic functionality on how to introduce MMS mission instrument data and operate on it and plotting it, whereas "projects/themis/examples/advanced" includes ∼ 3 dozen examples on how to perform advanced operations, typically on multiple THEMIS instruments or spacecraft. The user can navigate through the various subdirectories for each plug-in, and use a combination of pertinent crib-sheets and.
html documentation to become quickly familiar with the code, the data and their use. Community support on SPEDAS is streamlined for tracking and resolution through email help-requests to SPEDAS_Science_Support@ssl. berkeley. edu. Those are re-directed by the software team lead to the cognizant software engineer or developer. Direct communications between user and software engineer are then encouraged. Under the GUI main menu, a Help button streamlines GUI help requests through the same channels. The user may also contact directly the developer who has been identified at the header information of the routine in question as the major (or latest) contributor to it (with cc to the above SPEDAS_Science_Support email). Such open communications are conducive to efficient problem solving, accurate information exchange and SPEDAS package improvement for the benefit of the entire community.
SPEDAS for Developers SPEDAS was designed as a generic software development environment for ingestion of data from distributed datasets, data analysis and plotting by general purpose space physics tools, facilitating collaborative multi-instrument, multi-mission studies. A few basic guidelines can help facilitate project participation in this environment (space mission or ground-based observatory), as explained in this section. There exist on-line examples and documentation for developers of command-line and GUI code. Additionally, several existing plug-ins (with a variety of instrument types, number of satellites and data volumes) form a good starting point for any new project to modify and build their own. The SPEDAS core team, under NASA/HQ support, facilitates such development by providing advice and experience to project software developers, and by checking for plug-in inter-compatibility through suite testing, code conflict identification and resolution, and maintenance of a stable distribution.