# Presentation: 230490

**C. Huc/CNES, D. Boucon/CNES-SILOGIC,****D.M. Sawyer/NASA/GSFC, J.G. ****Garrett/NASA-Raytheon**

**Producer-Archive Interface Methodology****Abstract Standard**
**PAIMAS**

**Notes:**
We are going to talk about archiving and the relations, interfaces, and interactions that exist between a data producer and a service in charge of archiving these data.
To avoid any confusion, we will start by stating what we mean by archive:
Archive:  An organization that intends to preserve information for access and use by a Designated Community.
This definition is taken from a CCSDS and ISO standard, the OAIS Reference Model, which we will be coming back to later.

## Contents

-  **Why a new standard?**
**Space data archiving**
**Needs for standardization**
**What standardization will bring**
**Abstract standard status**
** ****Proposed methodology****	**
**Context - description**
**The 4 phases**
** ****Specialization**
** ****Aid Tool for the use of the Methodology**
** ****Towards an implementation standard**
** ****Conclusions**

## Why a new standard?Space data archiving: archive types

**Notes:**
In the space sector, archives can be set up at different levels depending on the organizational structure implemented.
- within a mission control system:
The constitution of technological databases for satellite monitoring – such database must be able to be preserved
 the archive of mission data (most often scientific), such as the archives of the European XMM-Newton or ISO missions (Infrared Space Observatory) in Spain
or older scientific missions such as IUE (International Ultraviolet Explorer)

- multi-mission archives of scientific data centres such as the NSSDC, the PDS or the CDPP

## Data Producers: 	
the operating systems of each mission
the mission processing and archiving centres
the scientific teams in the laboratories
Archive
specialized independent organizations (CDS)
specialized services within the agencies (NSSDC)
services common to several organizations (CDPP)

**the operating systems of each mission**
**the mission processing and archiving centres**
**the scientific teams in the laboratories**
**Archive**
**specialized independent organizations (CDS)**
**specialized services within the agencies (NSSDC)**
**services common to several organizations (CDPP)**

***Why a new standard?******Space data archiving: key roles and players***

**Notes:**
Producers: a complex context, most often in the framework of international cooperation

##  The relations between archives and data Producers are rarely simple and easy:
nonconformity of received data
unclear and imprecise definition of the data to be delivered,
failure to meet delivery schedule,
late detection of errors in archived data,
non-management of modifications
 ==> Can be detrimental to archived information quality and the cost of the operation.
Ever increasing diversity of the producers
Data complexity
Each project develops its own methodology on the basis of a process that is roughly the same from one project to another
 ==>  Work duplicated, no generality, excessively high costs, etc.

**nonconformity of received data**
**unclear and imprecise definition of the data to be delivered,**
**failure to meet delivery schedule,**
**late detection of errors in archived data,**
**non-management of modifications**
** ****==>**** Can be detrimental to archived information quality and the cost of the operation.**
**Ever increasing diversity of the producers**
**Data complexity**
**Each project develops its own methodology on the basis of a process that is roughly the same from one project to another**
** ****==>  ****Work duplicated, no generality, excessively high costs, etc.**

***Why a new standard?******Needs for standardization: problems***

**Notes:**
Established facts:

Examples of numerous difficulties encountered :

* Nonconformity of the data received by the archives with respect to what was expected
 ...

## Why a new standard?Needs for standardization: multiple standards

- Generic abstract
- standard

- Community
- abstract standard

- PAIMAS
- Implementation
- standard

**Approved by the CCSDS**

**Development in progress**

***Why a new standard?******Needs for standardization: multiple standards***

**Notes:**
PAIMAS is a methodological standard defining the general concepts and the process (major work steps and the associated actions)
==> PAIMAS also defines the methodology for producing a similar standard specific to a particular community (such-and-such a scientific community, such-and-such a space mission, etc.) so that the particularities of that community can be taken into account. The development of such standards falls under the responsibility of each community concerned.
PAIMAS also constitutes the methodological framework within which a so-called implementation standard will be produced by the CCSDS: the implementation standard addresses the practical aspects of implementing and verifying the process in terms of languages and models.

##  Ease some typical  problems (as noted earlier)
 Make it easier to follow schedules and commitments
 Help with human resources turnover
 Anticipate new problems
 Address the legal aspects early
 Increase data quality of archive holdings
 Have weight and credibility for adoption
And finally, save time (and money)

** ****Make it easier to follow schedules and commitments**
** ****Help with human resources turnover**
** ****Anticipate new problems**
** ****Address the legal aspects early**
** ****Increase data quality of archive holdings**
** ****Have weight and credibility for adoption**
**And finally, save time (and money)**

***Why a new standard?******What standardization will bring***

##  Today: CCSDS standard (blue book) and standardization with ISO in progress

Various communities have helped produce this document : 
space agencies (CNES, NASA, ESA, ...)
archivists (NARA, ...) and 
librarians (BNF -French National Library, RLG-Research Library Group).

**Various communities have helped produce this document : **
**space agencies (CNES, NASA, ESA, ...)**
**archivists (NARA, ...) and **
**librarians (BNF -French National Library, RLG-Research Library Group).**

***Why a new standard?******Abstract standard status***

## Contents

-  **Why a new standard?**
**Space data archiving**
**Needs for standardization**
**What standardization will bring**
**Abstract standard status**
** ****Proposed methodology****	**
**Context - description**
**The 4 phases**
** ****Specialization**
** ****Aid Tool for the use of the Methodology**
** ****Towards an implementation standard**
** ****Conclusion**

***Methodology******Context: OAIS environment***

**Notes:**
Starting point: the OAIS reference model

External data flows between the Producer, the consumer and the OAIS.
The producer establishes a submission agreement with the archive, which identifies the deliverable SIP (Submission Information Packages), the content of which has been negotiated between the producer and the archive.

The data consumer establishes an order for the desired information with the OAIS. The order identifies one or more archived objects - the Archival Information Packages (AIP) - that are of interest. These AIPs are retrieved from the archive, transformed if necessary, and delivered to the consumer as "Dissemination Information Packages".

## MethodologyContext: PAIMAS focus

- AIP : Archival Information Package, SIP : Submission Information Package, DIP : Dissemination Information package

***Methodology******Context: PAIMAS focus***

**Notes:**
Functional entities and their respective interfaces (only the main data flows are illustrated)

 1. Ingestion: ensures the services relating to the acceptance of the Submission Information Packages (SIP) received from the data producers, and to the preparation for the storage and management of the data within the archive.
 2. Storage: ensures the services relative to the storage, maintenance and retrieval of the Archival Information Packages (AIP), for example, receiving the SIPs and subjecting them to a quality assurance check.
 3. Data management: manages enriching, updating and access to the description information and the administrative data used to manage the archive, e.g. performing the database updates.
 4. Administration: manages the overall functioning of the archive. 
5. Preservation Planning:  provides the services and functions for monitoring the environment of the OAIS and providing recommendations to ensure that the information stored in the OAIS remains accessible to the Users over the long term, 
 6. Access: helps the data users determine whether an item of information exists or not, and to find its description, location and availability; enables the users to ask for and receive data products. 

## MethodologyContext: PAIMAS focus

- AIP : Archival Information Package, SIP : Submission Information Package, DIP : Dissemination Information package

**Ingest**

***Methodology******Context: PAIMAS focus***

##  
 The archive project is broken down into 4 main phases:
Preliminary Phase,
Formal Definition Phase,
Transfer Phase,
Validation Phase.

 Each phase has extensive action tables.

 Specification for a community.

-  **The archive project is broken down into 4 main phases:**
**Preliminary Phase,**
**Formal Definition Phase,**
**Transfer Phase,**
**Validation Phase.**
** ****Each phase has extensive action tables.**
** **
** ****Specification for a community.**

***Methodology******Description***

**Notes:**

Breakdown into main phases and sub-phases to be taken into account in chronological order.

Each sub-phase consists of a set of actions grouped in action tables.

Coverage of all aspects of the problem: technical, cost, security, legal, etc.

## MethodologyThe phases: relationships

**Data ready to archive**

***Methodology******The phases: relationships***

***Preliminary ***
***Agreement***

***Dictionary***
***Formal model***
***Submission Agreement***

***Transferred ***
***object files***

***Validation ***
***agreement****** ***

**Phase objective**

**Preliminary **
**Phase**

**Formal Definition **
**Phase**

***Anomalies***

**Validate the transferred **
**objects**

- Identification and preliminary
-  agreement
-    **Define the information **
**   ****to be archived**
**resources estimation**

- Negotiate the Submission
-  **Develop agreement (data to **
** ****be delivered, complementary **
** ****elements, schedule)**

- Actual transfer of the data
**   ****Actual transfer of the **
**   ****objects**

**Notes:**
This schematic diagram shows the entire process described by the method, from the first contact between the producer and the archive through to the integration of data in the archive with:

* the objective of each phase,
* the outputs from each phase.

The process begins at a very early stage, from the first phase of contact between the producer and the archive. The phases follow on from one another in chronological order (there can be loopbacks between the transfer and validation phases). Each phase is itself further broken down to end up with series of actions, some of which can be performed independently of one another (we will see an example later on): the methodology comprises some thirty action tables (29 to be precise) taking into account of all the possible factors in the negotiation. 

Each phase is described in greater detail further on in the presentation.

## Methodology Preliminary phase: context

**Preliminary Agreement**

***Methodology****** Preliminary phase: context***

**Archive**

**Producer**

**Notes:**
Very important phase that determines the basis on which the producer and archive will set up the project.
This phase must be carried out as early as possible, before the creation of the data themselves if possible. 

This phase results in a preliminary agreement that determines whether to proceed to the next phase or to stop the project if it is unfeasible.

Here is the content of the phase in greater detail ...

## MethodologyPreliminary phase: sub-phases

***Methodology******Preliminary phase: sub-phases***

**Notes:**
This phase is divided into 3 sub-phases:

* 1st contact: identification of contacts, exchange of information,
* preliminary definition, feasibility and evaluation: this part is subdivided into several sections to be addressed, such as:
 - information: preliminary definition of the data to be archived and the complementary information necessary for their archiving,
 - preliminary evaluation of the volumes of data to archive,
 - analysis of the legal and contractual aspects (intellectual property, conditions of access to data, etc.),
 - permanent impact on the archive (renewal of media, precautions to take to avoid losing data, etc.),
 - study of the associated costs,
 - ....

* establishing of a preliminary agreement: in the light of the preceding studies, this agreement determines continuation to the next phase. It contains, among other things, a conclusion on feasibility from the technical, resources and task-sharing aspects, and a schedule.

## MethodologyFormal Definition Phase: context

**Preliminary Agreement**

**Dictionary**
**Data Model**
**Submission Agreement**

***Methodology******Formal Definition Phase: context***

**Notes:**
Formal definition phase: 

The entry point for this phase is the preliminary agreement established in the preceding phase.
This phase serves to formally define the entire process that has already been defined in a preliminary manner. It concludes with an agreement between the producer and the archive.
This phase results in the creation of:
* a Data Dictionary,
* a data model,
* an Agreement.

This phase aims at providing a full, precise and formal description of all the deliverable data and the conditions of delivery.
The formal definition of the deliverable data (Data Dictionary, Data model, validation) shall be based on existing standards and languages (DEDSL and XML for the dictionary, for example) or future ones (e.g. XFDU for the packaging).
The objective of the methodology in this part is to remain general without proposing an implementation process, this being the purpose of the implementation standard that we will discuss later on.

## MethodologyFormal Definition Phase: sub-phases and action table

***Methodology******Formal Definition Phase: sub-phases and action table***

## MethodologyTransfer Phase

**Transfer Phase**

**Actual transfer of the objects:**
** ****carry out the transfer test**
** ****manage the transfer**

**Data**
**Model of object files to deliver**
**Schedule**

**Transferred object files**

***Methodology******Transfer Phase***

**Notes:**
This is the phase of actual data transfer from the producer to the archive, based on the formal method defined in the preceding phase (communication procedures, information packaging, etc.).

The packaging can be based on the XFDUs to define the content of the delivered package, and the means of reconstituting the expected objects from the delivered objects. 
For example, a large size SPOT image can be transferred as several smaller files which will then have to be concatenated for archiving on arrival. 

The aim of this phase is to ensure that transfer takes place as planned (for both producer and archive), and in accordance with the planned schedule. 

## MethodologyValidation Phase

**Validation Phase**

**Validate the transferred objects:**
** ****carry out the validation test**
** ****manage the validation**

**Transferred object files**

**Data ready to archive**

**Anomalies**

***Methodology******Validation Phase***

**Producer**

**Validation **
**acknowledgement**

**Notes:**
This is an important phase for the quality and conformity of the archived data. 
It permits package validation such as was defined in the formal phase.
There can be a level of automatic (and if necessary systematic) validation, carried out as soon as the packages are received (e.g. verification of file sizes, verification of file structures according to their standardized syntactic description – EAST language).
There can also be an in-depth validation level. This can take place after reception of a coherent information package, on reception of a full set of data, because some checks can require the presence of files delivered at different times (e.g. verification of completeness of the delivery with respect to the planned delivery – number of files, time periods).

Looping back with the transfer phase (and the producer) can take place if anomalies are detected.
This results in the construction of a package ready for definitive archiving.

PLEASE NOTE: the methodology provides for the management of changes of data and of the archiving plan over time.

## Contents

-  **Why a new standard?**
**Space data archiving**
**Needs for standardization**
**What standardization will bring**
**Abstract standard status**
** ****Proposed methodology****	**
**Context - description**
**The 4 phases**
** ****Specialization**
** ****Aid Tool for the use of the Methodology**
** ****Towards an implementation standard**
** ****Conclusion**

## Specialization

** ****Steps involved to define a community standard**
**terminology,**
**data dictionary and information model,**
**standards,**
**common tools.**
**Analyze each action of the generic standard (add actions if necessary)**

***Specialization***

**Notes:**
Steps in defining a community standard: 
*replace the generic terminology by the terminology of the domain,
*define a data dictionary and an information model that will be used by the archiving projects,
*identify other standards useful to the project,
*identify common tools

Analyze each action of the generic standard to determine whether:
it is applicable to the context,
it is not applicable,
it must be modified.

## Specialization: Example 

**F-4****: Define the formats, coding rules, and standards **
**        ****to be applied for the objects to be delivered**

**Generic Action (Methodology****)**

***Specialization: Example****** ***

**Notes:**
The CDPP will create the following specific action from action F-3 of the methodology. This action will be valid for any project participating in the CDPP.

The supervisors of a CDPP archive project then defined "concrete actions" derived from the CDPP action, which are characterized by attributes such as the person responsible for the action and the action completion date.

## Contents

-  **Why a new standard?**
**Space data archiving**
**Needs for standardization**
**What standardization will bring**
**Abstract standard status**
** ****Proposed methodology****	**
**Context - description**
**The 4 phases**
** ****Specialization**
** ****Aid Tool for the use of the Methodology**
** ****Towards an implementation standard**
** ****Conclusion**

## Aid Tool

**A methodology application and archive project aid tool would be a real advantage for both the Producer and the Archive. **
**First mockups produced**

***Aid Tool***

**Notes:**
Objectives of the aid tool: 

1. Analyze the action tables: addition, deletion or modification of actions,
2. Monitoring of the actions and phases of the Archive Project,
3. Give easy access to the standard (hypertext navigation, etc.).

To go further on the subject: pending validation and issuance of the standard. 

## Aid Tool

***Aid Tool***

**Notes:**
Mockup: Tracks archive project status, and also supports access to materials needed for each sub-phase.

## Contents

-  **Why a new standard?**
**Space data archiving**
**Needs for standardization**
**What standardization will bring**
**Abstract standard status**
** ****Proposed methodology****	**
**Context - description**
**The 4 phases**
** ****Specialization**
** ****Aid Tool for the use of the Methodology**
** ****Towards an implementation standard**
** ****Conclusion**

## Towards an implementation standardContext

- PAIMAS
- Generic abstract
- standard

**Implementation **
**standard**

**Methodology application aid**

**Languages**
**Tools**** **

**  **

**  **

**Archive project**

***Towards an implementation standard******Context***

**Notes:**
From the diagram shown at the beginning of the presentation: 

The implementation standard model comes into play at several levels. 

* it allows the defining of a general implementation process (applicable in highly varied contexts) that is flexible (can take into account the particularities of each context)  for the definition and transfer of the data.

*  to PAIMAS:  it proposes a general and very simple transfer data model and the implementation of a basic dictionary.
*  to the community standard: extension of the basic dictionary to a community dictionary.
*  to the archive project: construct a diagram of the project data, taking into account the constraints of data granularity and sequencing between the data and over time. 

The standard will be based upon languages and tools.

## Towards an implementation standardApproach

**generic data model**
**transfer data model (generic objects)**
**transfer data dictionary (generic descriptors)**
**community dictionary**
**diagram of data to transfer for the **
**            ****archive project**
**       **
**transfer sequencing and chronology  **
**transfer session**

**Implementation **
**standard**

**Languages**
**Tools**** **

**XML**

**XML Schema**

**UML**
**XMLScheme**

**DEDSL**

**DTD**
**XMLScheme**

**XFDU**

***Towards an implementation standard******Approach***

## Towards an implementation standardRelationships

***Towards an implementation standard******Relationships***

**CLUS-CIS-CODIF-Jm**

**CLUS-CIS-CODIF-EAST**

**CODIF**

**CIS**

**HIA**

**CLUS-CIS-HIA-EAST**

**CLUS-CIS-HIA-Jn**

**CLUSTER**

**Notes:**
Example in the framework of the CLUSTER project: an experiment, 2 data sets and an EAST descriptor for each data set.

The model enables the set of deliverable elements to be defined through a tree structure: data and descriptors of these data.

## Towards an implementation standardSchema

**Delivery plan**
**all the elements to be delivered**
**easy to understand from the Producer side**
**expression of constraints**

***Towards an implementation standard******Schema***

**Notes:**
The final step is a delivery plan that is simple to view and understand from the XML diagram that models this plan.

The plan contains all the deliverable elements, along with the delivery constraints applicable to these elements: granularity and sequencing in time.

## Conclusion

***Conclusion***

** ****PAIMAS identifies: **
**the phases in the process of transferring information,**
**the objective of the phases,**
**the actions that must be carried out,**
**the expected results.**
** **
**PAIMAS is a basis: **
**for the identification of standards and implementation guides,**
** ****for identification and development of a set of software tools.**

## End of presentation