dataesr/scientific-paragraphs-categorization huggingface dataset:
The CDPP approach to multi-source data analysis (applying the above principles) is illustrated in Fig. 1. It resulted in a flexible and attractive online tool, AMDA, which allows the user to combine and plot data from heterogeneous sources. As of today, AMDA serves more than 35 space missions and ground observatories (past, active or in preparation), various planetary and solar indices, and supporting models. While displaying data is the main feature of the service, it also comes with other scientifically valuable features; among them is the "data mining" capability, enabling to search data with combinations of predefined mathematical functions, and initially called "conditional search". This function produces "time tables" or lists of events (that comply with the conditional criteria applied on datasets and time intervals) which are a common and very much used object in space physics for which data mostly take the form of time series (data records indexed in time order).
First, and most importantly as it was the initial goal, AMDA helps produce science results. This paper aims at showing that this goal is achieved by facilitating the life of scientists, from undergraduate students to senior researchers, which is very probably illustrated by 1/the increasing usage of the tool and the partly correlated number of publications (Section 6. 1), and 2/the variety of use cases (Section 6. 4) testifying of eased mass treatments, data mining, classification and vizualisation. Second, and it was not planned so, the achievement of AMDA helps the CDPP as a whole being recognized for its expertise in data distribution and exploitation, such that in return AMDA was, and is, involved in various data-centered projects (e. g. Helio, Europlanet, IMPEx, some of which are shortly described below).
The purpose of this paper is to describe the basics of AMDA, how it is built (Architecture, Section 2), what it is concerned with (Data, Section 3), what it performs (Functionalities, Section 4) and how it lives and communicates in the ever-growing ecosystem of tools and databases (Interoperability, Section 5). Finally an important part (How useful ? Section 6) is devoted to applications which show the variety of ways AMDA has been used by the scientific community. The description of the tool makes necessary the mentions of a number of concepts related to space science, IT technologies, and organizations which heavily rely on acronyms. Therefore a list of the most cited acronyms is provided in appendices to help the reader. This paper offers a significant extension of the description of AMDA which is already available in the literature but based on an out-dated version of the tool  (Jacquey et al.
Architecture The size, abstraction and complexity of AMDA's code base has greatly evolved over the years. The architecture presented here corresponds to the latest version of the tool, although the general concepts were established long ago. Ergonomics are a key characteristics of AMDA's ambition. First, as an online tool, no installation is required. Data are readily accessed and manipulated by users through a combination of objects and graphical elements. These objects are viewed and handled through the Workspace Explorer, the central part of the Graphical User Interface (GUI) which is presented as laptop desktop. These workflows are made possible via a modular architecture organized in "layers" glued together by web-services and standardized protocols. These concepts are described in the following sections.
Parameter A parameter is the central object in AMDA and represents a time dependent quantity. It may be a scalar, a vector or higher order array (2D, 3D), for instance a proton density, a magnetic field vector or a particle flux depending on energy and incidence angles. A parameter has a time resolution and may have a unit. Parameters have a common name (e. g. , | b| or 'density') and an ID ('c1_btot' for the magnetic amplitude of Cluster 1 at 4s resolution). The user can build user-defined parameters from basic parameters available in the Workspace Explorer. For a user-defined parameter called 'name', the ID will simply be 'ws_name'. This ID is visible in the plot layout (see Section 4. 1) and is used to derive new parameters in the editing window (see Section 4. 2). For a given instrument on a particular observatory (in a broad sense: spacecraft or ground-based facility) the classical hierarchy is such that a parameter is the leaf of the data tree "mission > instrument > dataset > parameter".
TimeTable A TimeTable is a collection of couples (start time and end time) representing a list of events. It can be used to represent any list of intervals of user interest, from physical events like Interplanetary Coronal Mass Ejections or plasma boundary crossings to instrumental mode of operations, etc. A TimeTable has a name, a number of intervals, and optionally (but very recommended) a description composed of a free text. A TimeTable in AMDA may come from different sources conditional search (data mining), as discussed above, see Section 4. 3 upload, in various possible formats, see Section 4. 5 mouse selection for start and stop times directly on the plot window, see Section 4. 1 TimeTables may be manipulated (merge, intersection) in a special window described below in Section 4. 4.
Catalogue A catalogue is an extension of the TimeTable object. Both types of objects will eventually be merged into one single concept but, at present, for historical and technical reasons, there is still a distinction in AMDA. Catalogues have two first columns similar to those of TimeTables (i. e. a start time and a stop time) followed by an unlimited number of additional columns. This additional information of various types (float, integer, string, …) corresponds, for instance, to the mean value of a given parameter on the corresponding interval (or the min, max, median, or any other statistical quantity), a flag (instrument mode, plasma region), a comment, a URL, a DOI, etc. Catalogues in AMDA may come from the statistics module (see Section 4. 6) or from upload (see Section 4. 5). Discussion on a common catalogue format in space physics took place under the auspices of the SPASE consortium (in which the CDPP participates) and led to the HPEvents format 8 which is now adopted in AMDA.
Navigating in the GUI: the Workspace Explorer The Workspace Explorer is the main window used to navigate in AMDA resources and to monitor actions. It is unique to each user and is divided into three tabs as shown on Fig. 2 : resources, operations, and jobs, all of which gather AMDA and user data together with the results of data treatments. Resources is the main location for data either coming from the AMDA database, or remote databases, or more generally all data uploaded by the user (My DataBase). It is also where new objects defined by the user are stored: derived parameters, TimeTables and catalogues. Objects shared between all AMDA users (TimeTables and catalogues) are also accessible there in "Shared TimeTables"/"Shared catalogues" directories. Each user is allocated a dedicated storage capacity of 200 Mb with no time limit; each session is logged and saved for future use.
Some users regularly use and update their workspace since the beginning of AMDA 15 years ago. The second tab is "operations" where requests for plotting and data mining are saved (download and statistics requests are currently being implemented). Clicking on a saved request automatically fills the plot/data mining interface with the corresponding settings. Finally the "jobs" tab enables users to monitor actions which take too long to be run interactively and have therefore been transferred to the batch mode. Active jobs are in "Jobs in Progress" and, when terminated (either correctly or with error), go into "Finished Jobs/Results". Information on a given job is given in the tooltip. The middle panel of Fig. 2  shows different types of AMDA resources. The icon before each mission name refers to the type of environment the resource is about: planet, the solar wind, comet.
Web-services and communication As it can be seen on Fig. 3  there are two ways of accessing AMDA data, the first one being through the GUI as it is explained in previous Sections. The second access is via webservices and is mostly used for machine to machine communication, for instance when a user wants to automate some data retrieval and treatment outside of AMDA. These webservices are developed with the two protocols REST and SOAP. For the simpler REST access, via a URL in a browser, dedicated documentation is available 9  ; alternatively the SOAP implementation is also described 10. The list of available webservices is the following: auth: returns a token to use as an API security parameter for get-Dataset, getOrbits and getParameter; this token has a 10min validity limit getDataset: provides a full dataset (one or a series of parameters) for a given period of time getObsDataTree: provides the hierarchy of public access data getOrbits: provides the trajectory of a spacecraft for a given period of time getParameter: provides a single parameter for a given period of time getParameterList: provides the list of AMDA parameters (including those from the "Derived parameters" directory if information on the user is provided) getStatus: gets the current status of the request according to a process ID getTimeTable: provides the contents of a TimeTable or Catalogue Fig.
2. The Workspace Explorer with the three tabs "resources" (left), "operations" (top right) and "jobs" (bottom right). The middle image is a view of AMDA database resources (by alphabetical order, from A to J only on this view) and shows directories for spacecraft missions, radar, indices, and ephemeris. getTimeTableList: provides the private list of TimeTables owned by a user. When called without userID, this web-service returns the list of shared TimeTables. isAlive: checks whether AMDA services are available or not These webservices were initially developed during the IMPEx project (see Section 3. 2) and have been in use since then. Recently the SciQLop project (SCIentific Qt application for Learning from Observations of Plasmas https://github. com/SciQLop/SciQLop) has become one of the intensive users (see Section 6.
Data from large archives As explained above, the objectives of AMDA are to present a wide variety of plasma data in an homogeneous way such that mathematical treatments and science analysis can be easily performed with standardized methods. This is the reason why AMDA restricts itself to a limited number of parameters types: time series of arrays of one or several dimensions. Large public archives generally contain more types: distribution functions, altitude profiles, spectra, maps and images but cannot offer the ability to plot and analyze all of them. The science committee of AMDA therefore makes a choice of which public data are duplicated in AMDA from the various large, agency-led databases that mainly comprise of the Planetary Data System (NASA/PDS 11 ), the Coordinated Data Analysis Web (NASA/CDAWeb 12 ), the Cluster and Double Star Science Archive (ESA/CSA 13 ), and the Planetary Science Archive (ESA/PSA 14 ).
A data conversion is also applied as the internal AMDA format is netCDF (CDF to netCDF and PD3/4 to netCDF converters are used). Smaller, mission-focused databases are also accessed for data duplication (THE-MIS at Berkeley/SSL, SWARM at ESA/EO). Original data are retrieved, converted and copied but specific uses may require some processing, for instance producing a new, averaged data sets or omni directional flux from multiple anode datasets (information on these treatments are shown to the user). Finally, an important aspect of the databases is to maintain and regularly update orbits, trajectories and attitudes of spacecraft and celestial bodies. This is done thanks to the SPICE kernels from NAIF and ESAC (meta kernels are used for recent ESA missions 15 ).
2) and some use cases are detailed in G enot et al. (2018a,b,c) including one on Mercury plasma environment which also involves AMDA data mining functionality (see Section 4. 3). The numerical values of the time series are not stored within the AMDA database but accessed remotely via web-services. They can be accessed from the Workspace Explorer in the directory "Parameters/Remote Database: Simulations". The articles cited above give full information on the IMPEx standards and protocols developed and used during the project and they will not be repeated here. An important point is that the list of data providers can be easily extended and it was done so at several occasions after the project itself ended. It should be noted that a prototype access to simulations at CCMC 18 was operational in AMDA before this facility started reworking a data model inherited from SPASE (see Section 5.
1); this shift should now facilitate a larger scale deployment of this access. More recently the French ANR-funded project "TEMPETE" extended the IMPEx concepts to other codes and for time-dependent simulations  (Romanelli et al. , 2019). The new simulations will be equivalently visible in AMDA by re-employing IMPEx concepts. At CDPP, along with regular updates of data from external databases, models are also run routinely and the results are accessible via AMDA. This is the case of the 1D MHD solar wind propagation model  (Tao et al. , 2005)  which provides plasma and magnetic parameters at all planets. The 1D nature of the model permits rapid execution and light volumes of data such that datasets are produced from  Tao et al. (2005) 1990 onward for all planets. Solar wind conditions at spacecraft in cruise phase (Juno, BepiColombo) and around comet (Rosetta) are also presented.
User data The users can upload their own files (see Section 4. 5 for specific formats) and hence creates their own database (this is even facilitated via a system of file masks and wildcards). The parameters which are thus defined can be plotted and combined like any other (internal) AMDA parameters. They are visible in "Parameters/My DataBase". 11 https://pds. nasa. gov/. 12 https://cdaweb. sci. gsfc. nasa. gov/. 13 https://www. cosmos. esa. int/web/csa/. 14 https://archives. esac. esa. int/psa/. 15 https://www. cosmos. esa. int/web/spice/. 16 http://impex-fp7. oeaw. ac. at/is no longer online, see also http://impex. latmos. ipsl. fr/. 17 This has been illustrated in an "ESA story" available at https://sci. esa. int/we b/solar-system/-/60416-exploring-planetary-plasma-environments-from-yourlaptop.
Private instances In specific cases, an instrument team arranges for its data to be visible in AMDA only to a restricted list of individuals. This can happen during the proprietary phase of the data or during the generation of higher level (L3) data products or during the review of some specific datasets (following cross-calibration for instance). AMDA administration can grant access to a restricted list of users based on the logins. This enables a team with limited funding to access all AMDA functionalities without developing its own dedicated, costly system. Such an example, on a large scale, was the case of the Rosetta Plasma Consortium (see Section 6. 2 for details).
Constructing new parameters One of the originalities within AMDA is its calculator, or parameter editor, which enables parameters to be combined from heterogeneous instruments and produce a new, user-defined parameter. The temporal resolution of this new parameter is also decided by the user. Once saved, the parameter can then be used in plots, conditional search (data mining, see next section) or be inserted in the construction of another derived parameter. Technically the derived parameter is not calculated and inserted in the database; it is rather internally defined as an XML file and computed on-the-fly each time it is used for one of the purposes listed above. Examples of common derived parameters are plasma β, Mach number, Alfv en velocity, solar wind -magnetosphere coupling functions … which require the combination of magnetic field and plasma moments.
this protocol has been integrated in AMDA such that data are now distributed via HAPI thanks to the use of the official node-js HAPI server and the implementation of a binding to the AMDA REST web services. 22  This will enhance the visibility of AMDA and its datasets via a facilitated access; for instance plotting AMDA datasets in Autoplot  (Faden et al. , 2010) , another popular plotting tool, is now straightforward. From the earliest version of AMDA, the integration of the Simple Access Messaging Protocol (SAMP) developed by the astronomy community (IVOA) has been ensured. This is continued today and AMDA can now broadcast data (parameters), TimeTables and catalogues to companions tools connected via SAMP. More generally applications of this protocol within the CDPP tools have been discussed in G enot et al.
obspm. fr, or from interfaces embedded in tools). The mapping between the relevant EPNcore and SPASE dictionary elements has been done by the AMDA team for the internal database. The same registry that is used in AMDA is also used for VESPA. The EPN-TAP protocol allows access to granules, which are typically files. Each file is described in a row of the relational database, and the SPASE XML registry is used to populate the rows, after a translation from SPASE to EPNCore. Some metadata not defined in EPNCore are added as optional for EPN-TAP. For example "observed region" (SPASE) is translated to "spase region" (EPNCore). Data from AMDA may be searched from VESPA, using the common registry. Users searching for data of interest through, for example, the VESPA portal may choose several search criteria, most of them compatible with SPASE (instrument, timespan …).
Communication with other tools The interoperability of CDPP tools at large is also discussed in the respective reference papers for 3DView  (G enot et al. , 2018a,b,c; G enot et al. , 2018a,b,c)  and the Propagation Tool  (Rouillard et al. , 2017). 3DView enables the display of 3D planetary and heliospheric scenes with spacecraft trajectories along which observed plasma data can be plotted as illustrated on Fig. 7. An access to AMDA data via its web-services has therefore been implemented into 3DView. For a given scene including spacecraft, the AMDA database is filtered such that the user is presented with the most relevant data. The Propagation Tool allows users to connect solar perturbations seen at the Sun and their in-situ observations at planets and probes by computing time shifts with several models (radial or SEP propagation, co-rotation).
A built-in link to AMDA, based also on the web services, is presented to the Propagation Tool user to display in-situ data in a dedicated window as illustrated on the use case shown on Fig. 8. AMDA and Propagation Tool additional functionalities are complementary for following ICMEs in the heliosphere: the spacecraft hit by a given CME are predicted with the Propagation Tool, then the data from these spacecraft can directly be plotted in AMDA to analyze the in-situ plasma properties (as illustrated, for instance, in  Witasse et al. (2017)  and  Grison et al. (2018) ). Finally, in order to simplify the way users access CDPP data, the long-term CDPP archive 24  at CNES (called SIPAD, to be replaced by REGARDS mid 2021) has implemented the web services to give full exposure and access to AMDA datasets in a dedicated branch "Missions@AMDA"; this is illustrated on Fig.
, 1994)  data in a few steps: The first step was to analyze and to treat Christon's catalog  (Christon et al. , 1998)  which divides the 2 years "distant tail" GEO-TAIL campaign with respect to its transit inside the various magnetospheric regions (solar wind, dawn & dusk magnetosheath, magnetotail, boundary layer, plasma sheet, lobes). From this catalog, 14 time-tables have been generated, each one associated with a particular region and a level of geomagnetic activity. 28 hese time-tables were then used to map the likelihood of encounter of the various magnetospheric regions around L2. The time-tables were used to extract the data from AMDA in respect to the observed region. This allowed to compute the occurrence frequency of GEOTAIL (or any other probe) in each particular region. The time-tables were also used to extract (from the AMDA database) subsets of detailed (including species, energy and angular information) GEOTAIL particle data with respect to the observed regions and geomagnetic activity.
AMDA in the machine learning era: use of web-services The large amount of data held in the AMDA database associated with the availability of (REST) web-services which enables easy access to parameters makes AMDA a very suitable repository for machine learning applications. The 'learning' phase of these techniques indeed requests the ingestion of numerous 'events' characterized by one or multiple parameters. From simple command lines one or multiple parameters can be retrieved on one or multiple intervals (TimeTables). The example below shows how to build the command in a Python code. One strength of the AMDA architecture is that it offers an identical access method for parameters from the database as well as user defined parameters. This means 'paramID' in the command above can be the magnetic field magnitude measured by Cluster1 or whatever complex parameters that the user has defined in their workspace (for instance an integration over energy channels of a particle spectrogram).
This provides a powerful means to embed pre-processing in a pipeline where this pre-processing is done by the AMDA server. This has the advantages of reducing the data transfer over the network while the coding is lighter (in the example above only the scalar integral will be transferred instead of the full spectrogram). To date, the AMDA database and web-services have been used for several machine learning applications including plasma boundaries detection around Mars/Venus (from Mars & Venus Express, MAVEN data, see  (Garnier, submitted) ) and electron diffusion region associated with magnetic reconnection at the magnetopause (MMS data, see  (Lenouvel et al. , 2021) ). More applications have also been investigated in the frame of a contract between CNES and AKKA (IT company). The ultimate goal is to provide AMDA users with catalogues of events which are automatically updated upon new data ingestion.
This kind of development is currently in a prototyping phase. For instance on Fig. 12  an illustration of the global pipeline for bow shock detection at Mars is illustrated. On the right upper part of the Figure, time series of plasma parameters are plotted with a colour chart describing the detected regions: solar wind, bow shock or planetary environment. The general principle will be the same for each envisioned applications: a model is trained (beforehand by the AMDA team and collaborators) to recognize regions or events and produce a dedicated catalogue. The catalogue is then interpreted (like on Fig. 12  with the colour chart) to give the user an enhanced view of the raw data. Proposing value-added products like these predictions is one of the main objectives of applying machine learning techniques on the AMDA large database.
Model run at AMDA: the HelioPropa service Following a collaboration with C. Tao (now at NICT, Japan), a version of her 1D MHD code  (Tao et al. , 2005)  for solar wind propagation has been implemented in AMDA. The architecture is the following: solar wind observations at L1 (magnetic field and plasma moments) available in AMDA are taken as inputs to the model which is run automatically every day to produce estimates of solar wind conditions at distant locations, planets and spacecraft in their cruising phases (for such large scale estimates, conditions at the planet and at orbiting spacecraft are identical). Backward propagation is applied toward Mercury and Venus, and also toward the inner heliosphere spacecraft, Solar Orbiter and Parker Solar Probe. Cruising spacecraft include Juno, Rosetta and BepiColombo.