dataesr/scientific-paragraphs-categorization huggingface dataset:
A major revamp of the code in another language is therefore not as daunting. Here we describe what, in the IDL implementation, bestows SPEDAS its properties, but it should be recognized that any other language (e. g. , Python) with similar device independence, CDF library support and robustness as IDL could potentially replace the SPEDAS internal engine. This possibility is discussed in Sect. 6. The discussion below explains how the current implementation (IDL-based) addresses the SPEDAS design principles listed in Sect. 2. (A) Platform Independence The majority of SPEDAS code is IDL based, and the majority of the data ingested in SPEDAS are in ISTP-compatible CDF files. As IDL has internally implemented higher level routines for accessing, interrogating, reading and writing CDF files, using SPDF-provided software for accessing such files, the platform independence of SPEDAS is thus guaranteed by the platform independence of IDL and its CDF libraries.
SPEDAS's reliance on IDL to develop is main routines, as well as to enable new (userprovided) routines, does not preclude non-IDL code from being integrated in its arsenal using IDL provisions for externally linking C, Fortran and Python code. This happens by compiling such external code in the user's machine (or equivalent), then placing the dynamically loadable libraries or modules (. dll or. dlm objects) in appropriate directories within the user's IDL files so they are accessible from the IDL higher-level code. The user can do this for their own, home-grown code. The current SPEDAS version also distributes. dlls and. dlms for legacy routines, such as the Tsyganenko mapping routines [see: http://geo. phys. spbu. ru/ ~tsyganenko/modeling. html and Tsyganenko 2013] and associated field line tracing.
To enable use by the widest possible sector of the community, the SPEDAS team has pre-compiled such codes for the most common operating systems, and is distributing the libraries/modules along with the SPEDAS software (or they can be downloaded from the SPEDAS repository). Similar flexibility exists in data storage. Although highly recommended, CDF files are not a hard requirement for SPEDAS plug-ins. For example, THEMIS electrostatic analyzer (ESA) files  [McFadden et al. 2009] , the workhorse of ESA data analysis, are non-CDF, L0 (Level-0) files which are distributed as binary (even across disparate platforms) and accessed as such by IDL's platform independent code (big-endian versus little-endian interpretation is implemented on-the-fly according to the user's machine, along with calibration with the latest calibration parameters).
The L2 ESA files, on the other hand, are derived products (calibrated and in physical units), in CDF format, for release to SPDF in a manner satisfying NASA requirements for data dissemination. For other missions too (e. g. , MMS), the majority of their data are released as CDFs at their highest resolution, as L2 products (calibrated and in physical units). They have been vetted by the teams as ready-for-release, and satisfy NASA requirements and the needs of both the teams and the community for the highest quality and resolution. However, other non-CDF, platform-independent data formats are also supported by SPEDAS. These include netCDFs (e. g. , GOES and ICON datasets) and FITS files (e. g. , for the Solar Magnetic Activity Research Telescope, SMART, from Hida Observatory of Kyoto University, part of the IUGONET plug-in).
Such standardized data format choices not only conform to NASA's open data policy, and provide robust associated software and human resources (e. g. , SPDF support, community peer support), but also facilitate the SPEDAS environment's platform independence. (B) Project Independence SPEDAS is a list of generic routines that avoid a "core" program but rely on a set of agreements to represent, load, manipulate and plot the data (as tplot or GUI variables representing physical quantities which link time, data, and associated information, as explained in Sect. 4. 2). Figure  6  shows this hierarchy at the present time. Organization by directories, arranged by function (general, external, projects) and subdirectories therein, cleanly separate the general SPEDAS functions (under directory "general") from external (to SPEDAS) but still general purpose software of significant critical mass (under "external"), and project plug-ins (under "projects").
The command-line mode and its associated crib-sheets offer fast and comprehensive access to the data, whereas the GUI allows quick and intuitive access to the code by the uninitiated. To avoid duplication and errors the SPEDAS software implements the GUI as an added layer on top of command-line code, relying on the same underlying routines. As explained in Sect. 4. 2, tplot-variable content is accessible by GUI variable and vice-versa with a simple mapping upon user request. This allows the user to invoke operations on the data from either mode, even though operations are actually done by the same routines regardless of the mode from which they were invoked. This is possible due to the use of IDL pointers in memory, which allows IDL structures (such as tplot variables and GUI variables) to point to the data without actually accessing memory.
An intermediate option can be exercised by any project to aid analysis by the community. This entails a bundled license purchase of 50 or 100 seats, which can then be distributed to the project's science team at a low incremental cost per user. On centralized projects the costs can be borne by the project lead institution. If that is an educational institution and students are involved in research throughout the team then the IDL license cost per user is approximately 10 times lower than a standalone license cost. For example, THEMIS owns and distributes ∼ 100 licenses to its team members and reassigns them every few years depending on need. (D) Modular Code Structure Effective data analysis relies on a modular set of codes. Introduction, data analysis, and visualization/plotting routines can thus be used to conduct effectively powerful analysis and document it or communicate it concisely.
SPEDAS follows these rubrics with crib sheets or GUI documents (Sect. 4. 2). Multiple threads woven within each crib-sheet or GUI document (e. g. , one per product, or dataset) can create intermediate products along the way, which can be viewed and scrutinized, then combined into higherlevel products for final analysis and plotting. Text-based crib-sheets or GUI documents retain records of past work, can be shared with colleagues and can be electronically published in journals referring to plots or tables of results to succinctly and accurately represent the processing done to the raw data. Aside from flexibility for the user, this code modularity is reflected also in the directory structure of SPEDAS (Fig. 6 ) which is conducive towards balancing the development effort as a set of different human resources: (i) a core development team (general), (ii) individual project (plug-in) software development teams (projects), (iii) other organizations contributing data and software packages, e.
Large efforts of general utility, such as Tsyganenko models and CDAWlib functions, are bundled separately under "external", allowing interfaces, updates and QA to be managed most efficiently with clear lines of communication and responsibility. Help from experienced SPEDAS software developers ensures quality, successful software creation. Each piece of code is internally structured to ensure documentation, comments, names and developer contact information, and, in many cases, test suites for regression testing and QA. Software modularity is also important to ensure that user codes do not become overly complex, that the software development roles which mirror this modularity are well understood, and lines of management and responsibility are clear. Code debugging also benefits from this modularity both on the user and on the software developers' side.
g. , for integration and test data, early mission data or uncalibrated quantities) and a clear and simple directory structure. This is implemented as file retrievals using https requests, which enables password-protection. Such protection can be implemented on a portion of the remote data, even when the majority of the dataset is open-access. User credentials are saved at the start of a session and apply for the duration of the IDL session. The two main guidelines for facilitating such a distributed data system are that: (i) the remote data are organized in directories by quantity, then date (. project/instrument/mode/ yyyy/mm/dd as appropriate), where version numbers are allowed within the same directory, signified by file extensions (e. g. , filename. v2. 3); and (ii) the data are internally time-ordered and have time overlaps removed (to avoid inconsistencies during plotting and data searching and for multiple files to be uniquely and accurately patched into longer time-ordered intervals per user request).
g. at 10 min or 2 hour or 1 day). Such tips to projects by the core team can significantly simplify the load routines, reducing the effort by the project plug-in code developers so that the typically limited personnel resources will be better utilized for enhancement of science. (E) Support Structure SPEDAS is more than just code: it is a community of developers serving a large community of users (several wear both hats). Software evolves and companies may disappear, but the needs for analysis, visualization, and efficient science return from multi-instrument, multi-spacecraft missions are continuing and, in fact, growing. Because the community and its needs are long-term, and because human interactions (agreements) take typically longer than writing well-defined code, the underlying software itself is secondary compared to the buy-in and satisfaction of the community's short and long-term needs.
Therefore, a robust and flexible support structure is required to maintain the current code, prevent breaks or internal incompatibilities and protect it from future unexpected software industry evolution. This is accomplished by a core team of developers, funded by NASA/HQ, in part directly and in part through NASA and international project contributions (SPDF, THEMIS, MMS, ERG, etc. ). It also receives in-kind contributions (software, advice, test support) from users and other (interagency, small business) stake-holders (SPDF, ERG, IUGONET, NOAA, Cottage Systems,. ). The core team releases SPEDAS code documentation for users, in the form of. html daily updates, writes and updates development guidelines, maintains current code (bug fixes) and enhances/expands it as requested by the community, and vets new code submitted by the community.
NASA/HQ currently holds a modest SPEDAS support contract (1. 6FTE) and the remainder of the core SPEDAS personnel is contributed by individual projects (contractual authority delegates power to the core team for decision-making by consensus). The core team is comprised of NASA/HQ, the SPEDAS contract PI, the SPEDAS software lead and key developers funded or contributed from major projects and the main stakeholders. This team of a dozen individuals acts as a board of directors that prioritizes (based on consensus by the experts) the on-going tasks and community requests, addresses the immediate SPEDAS goals for community support and code inter-operability, and periodically assesses new features and long-term evolution as funding and time permit. The core team meets weekly to discuss the software development status and plan future work.
This includes top-level description, prioritization and assignment of new bug reports and enhancement requests; status updates on tasks in progress and of any issues or assistance needed; declaration of closures; coordination with other missions and stakeholders; and planning for upcoming software releases, conferences, tutorials and webinars. Participation of other stakeholders (e. g. science team members or software developers working on other missions) is also welcomed and encouraged, on an as-needed basis. These weekly SPEDAS management meetings mostly provide updates, assess priorities and outline possible development strategies. Prolonged, detailed technical discussions are reserved for splinters, often initiated at the weekly management meeting, in order to keep the weeklies focused and efficient.
The SPEDAS team would then create simple test scripts (ideally a minimal standalone script that doesn't require any SPEDAS code) in order to reproduce the problem and pass the conditions and symptoms along to Harris Geospatial via its technical support contact. These communications would often result in suggested workarounds (SPEDAS software changes) for the affected IDL release, with the issues often being fixed in the next IDL release. Recently, however, the SPEDAS team has taken a proactive approach and has upgraded its IDL technical support contract in order to preview impending IDL releases to perform beta-testing. Now the team executes suite testing and proactively looks for potential issues, with the goal of having them corrected before the impending IDL release is rolled-out to the public.
Before being officially announced, SPEDAS release software builds are put through rigorous (and time-consuming) QA process. This entails using test-suites to thoroughly exercise new and upgraded features to ensure that they operate as designed for a wide range of use cases and workflows. In the command-line mode, test-suite results are compared to known successful completion results, to ensure that changes and bug fixes did not have unexpected interactions with other parts of the system. In the GUI mode, test suites consist of a scripted series of operations for a tester to perform manually, with the expected results documented. Tests for both modes are conducted using multiple IDL versions, to ensure compatibility with older and current IDL releases. They are repeated for different operating systems, to test for platform-specific issues.
Each round of tests is followed by a round of bug fixing and another cycle of testing with attention on potentially affected routines, until the code converges to a state that passes all tests. The software release is then packaged and uploaded to the SPEDAS web site. The SPEDAS wiki and documentation are updated to conform to the new release. The SPEDAS executables incorporating IDL Virtual Machine are then built for Linux, Windows, and Mac and are also updated on the web. Finally, the availability of the release is announced on the SPEDAS mailing list, GEM, SPA, and CEDAR newsletters, and the SPEDAS blog page, including links to release notes, documentation, and crib sheets illustrating new capabilities. The QA process prior to major SPEDAS version upgrades could take up to two months, even with automated test scripts for most of the command-line mode.
However, they greatly facilitate use of existing routines by conforming to current practices. Each one can be overcome at the expense of incremental work by the plug-in development team. The most basic desirement is (i). It too can be violated by having the IDL code be an external call to Fortran, C or other code, that has been pre-compiled for a number of platforms and distributed as a. dll or. dlm. However, use of predominantly IDL code allows the user to take full advantage of the native environment of SPEDAS including the associated maintenance through future upgrades or operating system version changes. The most important routine for a plug-in's command-line mode is the load routine. If conforming to the standard SPEDAS practices it should specify time range, satellite and/or instrument, data type and any other identifiers specific to the program, such as operational mode, level of processing and possibly also coordinate system.
These generic tools enable SPEDAS standard load routines to identify the remote files to fulfill the load request, spawn an HTTPS (preferred), FTP, or HTTP request to download these files from a remote repository to a local data cache, and load the variables from the downloaded files into IDL as tplot variables. Given the similarity of data across space physics missions, existing load routines of a similar data type can easily be found and repurposed by the project plug-in developers with only minor changes. Such existing code can serve either as a close example for modification, or as a starting point for code development from scratch. In either case, this simplifies the development process and reduces the need for outside developers to have in-depth knowledge of the SPEDAS framework's internal implementation details.
Similarly, specialized command-line analysis and plotting routines for a new plug-in can be developed by building on existing SPEDAS tools that perform same or related functions. GUI features for a new plug-in are optional but quite useful, as they enable loading, analysis, summary plot creation, and efficient science interactions across the community without a paid license. They also allow novice IDL users a quick access to the plug-in without much of a learning curve on the use of the plug-in's command-line syntax. To facilitate such GUI features, which can be viewed as Application Programming Interfaces (APIs), the SPEDAS team has documented the GUI API development process at the SPEDAS wiki (under Documentation → Developers' Guide → GUI). The standard SPEDAS software distribution provides the necessary infrastructure for such development.
SPEDAS Expansion Plans The Heliophysics user community is best served when as many missions and data sets as possible are supported by a common software toolkit such as SPEDAS, providing a rich variety of analysis and visualization tools to the widest sector of the community. Additionally, the SPEDAS core development team strives to build interfaces with other known data ingestion or plotting packages to increase interoperability and offer alternate ways of looking at the data. The two that are presently amenable to this are Autoplot and CDAWeb. A promising new direction is also PySPEDAS, which employs a complete rewrite of the SPEDAS functions using Python. These are explained below. First, with regards to new mission engagement, the SPEDAS team reaches out to missions as early as possible when the data project management plan is being written and the data products and analysis systems are being defined.
It identifies tools or techniques that would be helpful in implementing SPEDAS support for their data analysis needs. At the very least the SPEDAS team reviews and understands the mission's plan and makes recommendations, such that small steps can be taken which do not impede (but rather facilitate) data ingestion into SPEDAS. When implemented early such recommendations are easy to do (e. g. ISTP-compliant metadata, appropriate file sizes and directory layouts, well-defined data access policies etc. ). Conversely, the SPEDAS team also strives to take any necessary steps within the SPEDAS distribution that address the needs of the new mission. For example, password-protection of a portion of the dataset was implemented for the needs of ERG and then employed by MMS for its early orbit data distribution-it is now a robust SPEDAS feature.
Ideally, a token new mission is willing to go the route of an integrated analysis package for its instruments. This was the case for MMS, which benefited from an early development of its load routines and construction of crib-sheets, resulting in high science return by both the MMS team and the Heliophysics community. Towards that goal, MMS invested approximately one full-time equivalent person per year to facilitate interfacing between its instrument teams and SPEDAS, which was necessary and sufficient for a multi-spacecraft, multi-instrument mission of this size. Other projects (e. g. ERG) elected to invest into their own interfaces at the instrument software support level, and provide periodically well vetted plug-ins. For those projects, the SPEDAS core team provided support and advice through frequent communications, code reviews and beta testing.
As of this writing, there are plug-ins under development to support the upcoming ICON and Parker Solar Probe missions, however the exact implementation plans are still in flux. There are also plans to develop plug-ins for additional legacy data sets: GEOTAIL, Venus Express (VEX) and Mars Express (MEX), LANL-GEO, and newly reprocessed data sets from GOES and CLUSTER. Next, with regards to other software packages, the SPEDAS team is also proactively engaged in interfacing with them to enhance the Heliophysics user experience. Autoplot is a Java-based tool which was originally developed to query and fetch data from diverse sources identified at Heliophysics' Virtual Observatories (VxOs) and is funded by NASA. These distributed datasets are indexed and served by metadata which adhere to the Space Physics Archive Search and Extract (SPASE) guidelines, established by the Heliophysics Data and Model Consortium (HDMC).
More than a dozen Virtual Observatories pertaining to space physics data exist now, and their metadata serve as a powerful way to locate datasets serving particular quantities (e. g. , magnetic fields in space) across multiple missions. Autoplot is a flexible tool for accessing and viewing these datasets. Many teams have gotten accustomed to its use for visualization of their favorite datasets. While Autoplot provides a visualization but not a data analysis environment, a combination of Autoplot and SPEDAS could be a powerful addition to the Heliophysics discipline, as it could supplement it with data analysis capability. An Autoplot-SPEDAS bridge could allow disparate Autoplot-accessed data to be analyzed together using SPEDAS and the joint analysis products to be visualized by either SPEDAS or Autoplot.
Information on how to do these operations is in the SPEDAS wiki under Documentation → Autoplot Interface. Additionally, CDAWeb, a world wide web-based plotting tool which provides access to the vast array of space missions and other products at SPDF, can also benefit from and aid SPEDAS in their common quest to better serve the Heliophysics community. Specifically, CDAWeb serves as a data query, ingestion and visualization of the SPDF data repository. This repository contains data from most US NASA missions going back to the Apollo program-the most recent ones in ISTP-compatible CDF format. The CDAWeb userinterface is hypertext markup language (HTML)-based and relies on common gateway interface (CGI) scripts to spawn IDL runs at local machines and display dynamically the data on the web. Its toolkit, a library of IDL routines to query, access, and correctly plot data stored in mission CDFs, is CDAWlib (NASA-sponsored and freely accessible).
Further data analysis can proceed in SPEDAS GUI, or after mapping the data into tplot variables using the "Manage Tplot and GUI Variables" tab under I/O Data in the main menu Most notably, the SPEDAS team has implemented an interface to the CDAWeb data access via the SPEDAS GUI, called: "Load Data using CDAWeb". This brings up a session which allows the user to select mission, instrument and data product (Fig. 8 , left panel) then seamlessly map data into GUI variables and plot them with a few mouse clicks (Fig. 8 , right panel). This allows a SPEDAS user to conduct science from a large array of historical missions but also from those current missions whose data may not yet be directly available as SPEDAS plug-ins. Finally, the SPEDAS core development team is exploring the possibility of implementing some subset of SPEDAS in Python to experiment with its usefulness.