AIDA is not actually using DOI for produced data. However, the introduction of
persistent and unique identifiers using service like DataCite
(https://datacite.org/dois.html) will be evaluated during the project for
simulations and enhanced data.

##  Making data openly accessible

Different types of storage are available at Cineca: disk, tape, structured db.
Depending on the typology, dimension and the use (internal, external) of data
different solutions can be used. For example, enhanced datasets can be hosted
in a MySql instance. Moreover, a cloud environment based on Openstack is also
available to create virtual machine to set up custom databases installation
(MySql, Postgres, Neo4j, etc.)

In general, Cineca storage is "user oriented": Each user has its own space in
the $HOME filesystem (with back up) and some space in the $CINECA_SCRATCH
filesystem (without back up and for a short time) where to store data for all
the projects he is involved in. Moreover, on demand, a $TAPE filesystem is
available for saving data on magnetic media. However, two "project oriented"
filesystem are available: WORK (for data analysis) and DRES (for data
storage).

For the AIDA project, a DRES directory has been created on request of Aida
project. It is onlystorage resource, based on GSS technology. It is
characterized by:

  * an Owner (a user who owns that resource and is allowed to manage it), 

  

  * some possible Collaborators (users who can access the resource but not manage it) 

  

  * a validity time, an extension and a storage type 

  

  * some possible computational Projects (all collaborators of the project can access the resource) 

  

Currently, three main types of DRES are available for AIDAdb initial setup
(Table 6)

_Table 6: Storage resources for AIDAdb._  
  
<table>  
<tr>  
<th>

_**Name** _

</th>  
<th>

_**Description** _

</th>  
<th>

_**Available space in TB** _

</th> </tr>  
<tr>  
<td>

FS

</td>  
<td>

a storage area consisting in a normal unix FileSystem object

</td>  
<td>

10.0

</td> </tr>  
<tr>  
<td>

ARCHIVE

</td>  
<td>

a storage area for long-time archiving that is actually maintained mainly on
magnetic tape via LTFS technology

</td>  
<td>

200.0

</td> </tr>  
<tr>  
<td>

REPO

</td>  
<td>

a more sophisticated data repository for long-time archiving, where data are
described by metadata and different security levels are available. The Data
Repository (REPO) is based on iRODS technology

[1]

</td>  
<td>

0.5

</td> </tr> </table>

Results of the numerical simulations will be made available on disk or tape
storage. In particular a description of the numerical code, initial
conditions, boundary conditions as well as the electromagnetic fields data
obtained directly from the selected simulations will be stored. These data
will be immediately accessible to all components of the Consortium and will be
made accessible to people external to the project by the end of the project at
maximum, if possible even before the end. Spacecraft data summarized in the
ICD are public and come from ESA or NASA DB and are accessible to any user
following the ESA or NASA procedures. AIDA will furthermore produce new high-
level data products from the used spacecraft data that will include catalogues
of features and events detected by ML and AI algorithms. These data sets will
also be stored on the Cineca DB and will be made available to the public.

Cineca Data Repository [2] is a service to store and maintain scientific data
sets, built in a way that allows a user to safely back-up data and at the same
time manage them through a variety of clients, such as web browser, graphical
desktop and command line interfaces. The service is implemented over iRODS [1]
(integrated Rule-Oriented Data System), which relies on plain filesystem as
backend to store data, represented as collections of objects, and on databases
for metadata. The service's architecture has been designed to scale to
millions of files and petabytes of data, joining robustness and versatility,
and to offer to the scientific communities a complete set of features to
manage the data life-cycle:

  * Upload/Download: the system supports high performance transfer protocols like GridFTP, or iRODS multi-threads transfer mechanism, and large interoperable ones like HTTP. 

  

  * Metadata management: each object can be associated to specific metadata represented as triplets (name,value,unit), or simply tagged and commented. This operation can be performed at any time, not just before the first upload. 

  

  * Preservation: the long-term accessibility is granted by means of a seamless archiving process, which is able to move the collections of data from the on-line storage space to a tape based off-line space and back, according to general or per-project policies. 

  

  * Stage-in/stage-out: the service is enabled to move data sets, requested as input for computations, towards the HPC machines' local storage space, commonly named “scratch”, and backwards as soon as the results are available. 

  

  * Sharing: the capability to share single data objects or whole collections is implemented via a unix-like ownership model, which allows us to make them accessible to single users or groups. Moreover a ticket based approach is used to provide temporary access tokens with limited rights. 

  

  * Searching: the data are indexed and the searches can be based on the objects location or on the associated metadata. 

  

During the project we will evaluate also the use of EUDAT services B2SAFE and
B2FIND (Cineca is a EUDAT partner). EUDAT Collaborative Data Infrastructure is
a Service-oriented, Community driven, Sustainable and Integrated initiative
developed thanks to FP7 and H2020 European programmes. The aim of EUDAT is to
create a single European e-infrastructure of interoperable data services. It
is based on a network of nodes (computing or data centers across 15 European
nations) that provide a range of services for upload and retrieval,
identification and description, management, replication and data integrity,
plus supplementary services needed to run the infrastructure. EUDAT adopts a
data model and a set of technical standards and policies shared with
scientific communities

[3].

B2SAFE [4] allows one to replicate datasets across different data centers in a
safe and efficient way while maintaining all information required to easily
find and query information about the replica locations. To this end,
information on locations along with other important information is stored in
PID (Persistent identifier) records. It is based on iRODS and an instance is
available in Cineca.

B2FIND [5] is a metadata service providing a discovery portal, which allows
users to find data collections within an international and inter-disciplinary
scope. It is based on a metadata catalogue of research data collections stored
in EUDAT data centers and other repositories. Harmonization of the metadata
descriptions collected from heterogeneous sources enables not only the
presentation in a consistent form but also the faceted search across
scientific domain boundaries.

For the storage of software product AIDA is using GitLab
(https://gitlab.com/aidaspace/aidapy). GitLab is a web-based hosting service
for version control, mostly used for computer code. It offers the
functionality of distributed version control and source code management,
providing several collaboration features like task management, feature
requests and bug tracking for every project.

Projects on GitLab can be accessed using the standard Git command-line and a
web interface. GitLab also allows registered and unregistered users to browse
public repositories.

After registration on Cineca UserDB, users can access data and metadata,
depending on their role in the project (participant, external user...) using:

  * Secure Shell (SSH) 

  

  * Desktop Based Tool 

  

  * Grid Services using GSI-SSH 

  

  * Cineca Data repository tool based on iRODS 

  

The access through AidaPy for external users will be allowed without
registration on Cineca Userdb but with app registration. An anonymous access
is also available through iRODS.

Data will remain available at Cineca for 2 years after the end of the project
and solutions for the preservation after this period will be examined by AIDA
project.

##  Making data interoperable

To facilitate interoperability we will enumerate the numerical simulations
using different capital letters referring to the physical environment as e.g.,
solar wind, magnetosphere, magnetotail and so on followed by the main physical
process of interest as , e.g. magnetic reconnection, Kelvin-Helmholtz
instability and so on. We will also use the L or E letter to distinguish the
numerical approach, Lagrangian or Eulerian, respectively. The same terminology
will be used for space data. A scientific standard vocabulary will be used for
all data types in the data set.

Data quality assurance and quality control will be maintained throughout the
project, utilizing strategies for [6]:

  * Preventing errors from entering a dataset. Typical types of errors include for example (a) incorrect or inaccurate data entered in the repository, or (b) data or metadata not recorded due to human error or anomalies in the field. 

  

  * Ensuring quality of data for entered data in order to identify potential data contamination. Typical examples include (a) checking for null values in the data (like NaN etc.) so as to avoid any missing, impossible or anomalous values within the data and (b) making sure that data line up in proper columns. 

  

  * Monitoring and maintaining the quality of data during the project in order to prevent data contamination. 

  

Upon uploading data on the AIDApy repository, the user will be forced to run a
local script/procedure that will collect some non-privacy information and
initially check if the data that are going to be uploaded are relevant and
valid. The user will need to include info like:

  * Digital identity, in order to know who is entering data to the AIDA repository 

  

  * Disclaimer that the data do not contain any harmful item (e.g. virus free) 

  

  * Data information like the type, size, number of scenes etc. 

  

Regarding data quality assessment (QA), three aspects are generally found in
remotely sensed imagery and can be included in the AIDApy tool for performing
data QA:

  * Radiometric Quality, which is affected by sensor characteristics and detector responses such as striping, drop lines, noise and band missing 

  

  * Atmospheric Quality, which is dependent on the circumstances at the imaging time such as cloud cover and haze 

  

  * Geometric Quality, which is either dependent on sensor characteristics and also satellite situation such as attitude, position, velocity and perturbations. 

  

AIDApy will study the usability and inclusion of some visual and statistical
analysis tools for performing QA on data entered in the repository [6],
summarized in Table 7:

_Table 7: Visual and statistical tools for QA._  
  
<table>  
<tr>  
<th>

_**Quality Defect** _

</th>  
<th>

_**Visual diagnosis** _

</th>  
<th>

_**Statistical Diagnosis** _

</th> </tr>  
<tr>  
<td>

Striping

</td>  
<td>

Different overall brightness of adjacent lines

</td>  
<td>

Significantly different variance and mean of adjacent lines

</td> </tr>  
<tr>  
<td>

Drop Line

</td>  
<td>

Null scan line

</td>  
<td>

Zero variance of a line

</td> </tr>  
<tr>  
<td>

Noise

</td>  
<td>

Dark and bright points at the background

</td>  
<td>

Radiometric anomalies

</td> </tr>  
<tr>  
<td>

Band Missing

</td>  
<td>

Lack of data in a band

</td>  
<td>

Zero variance of a band

</td> </tr> </table>

#  Increase data re-use

During the project AIDA consortium will choose an OpenSource license for data
access through the AIDApy software.

For numerical simulation data the access will be provided by Cineca based on
single user request. The high level data that will be produced using ML and AI
algorithms will be also made available to the public on the basis of a single
user request.

Data generated by AIDA will be also reused in other scientific national or
international projects or existing databases. An example might be ingesting
into ESA science archives a number of AIDA high-level products obtained from
both ESA data and numerical simulations. The focus would be on ESA
Heliophysics Science Archives of completed or long-duration missions such as
the Ulysses

Final Archive [7], the Cluster and Double Star Science Archive [8] and the
SOHO Science Archive [9]. During the AIDA project, discussions will be carried
out with ESA and an agreement will be pursued between the project’s PI and ESA
to manage the interfacing between AIDA’s and ESA’s databases. This will happen
through a relevant ICD (Interface Control Documents) consisting of the rules
for pushing-pulling data, together with lists of metadata to be ingested in
ESA databases. Following the ICD specifications, Cineca will create an archive
including the project’s results in order to make as easy as possible the data
ingestion in the ESA archives. Cineca will create the procedures to load the
project’s data into the ESA archives.

#  Allocation of resources

There are no immediate costs anticipated to make the data produced FAIR. Data
will be processed using the tools developed in the AIDA framework and stored
in Cineca systems. Parts of the AIDAdb relevant to specific European missions
will be migrated also to the ESA database. The databases created by AIDA
(AIDAdb) will be public and interfaced with community-based databases such as
the databases maintained by ESA.

For documentation, the EUDAT service B2DROP is used. B2DROP [10], based on
ownCloud, is a user-friendly and trustworthy storage environment, which allows
users to synchronize their active data across different desktops and to easily
share this data with peers. The service is directed towards all European
scientists whose institute does not host such storage.

#  Data security

For the duration of the project, data will be stored on the Cineca storage
protected against unauthorized access by means of standard Cineca
authentication. Appropriate access levels will be granted by definition of
roles in the project.

#  Ethical aspects

There are no ethical issues in the generation and analysis of simulations and
solar data. There are no human subjects or samples involved. Personal data of
project participants will be treated following GDPR through the Cineca Userdb.

#  Long term plans

AIDA will continually revise the present plans to take opportunity to the
evolving scenario on scientific databases and in particular databases for
space sciences. As the project will mature, we plan to use every opportunity
to seek continuity to the activities of both AidaPy and AIDAdb. We envision
AidaPy to become a community open software package for python that will
continue to expand and be refined from the contributions of the community of
users. AIDAdb, instead, requires resources of hardware and that requires
financial support. We will seek opportunities in future EU and ESA plans for
the continuation of a database of simulation results that will retain long
term the data of published results, of key simulations and will expand with
new input from the community.