Data Provenance: Use Cases for the ESO archive, andinteractions with the Virtual Observatory
J. Santander-Vela1,2, A. Delgado1, N. Delmotte1, M. Vuong1
1 European Southern Observatory

2 Instituto de Astrofísica de Andalucía (CSIC)

J. Santander-Vela

In the Virtual Observatory era, where we intend to expose scientists, or software agents on their behalf, to a stream of observations from all existing facilities, the ability to access and tofurther interpret the origin, relationships, and processing steps on archived astronomical assets (their Provenance) is a requirement for proper observation selection, and quality
assessment. In this poster we present the different use cases Data Provenance is needed for, the challenges inherent to the ESO archive and their link with ongoing work in the IVOA.

Data Provenance and Astrophysics
Data Provenance in e-Science is the tracking of the origin of any piece of information that has been recorded, with knowledge of all processing steps undergone.
In the astronomical realm, we track data Provenance in order to answer questions like what was done to the photons captured by this archived entity, and who
were responsible for it? And we need to be able to answer that question because for typical astronomical workﬂows (such as that shown in Figure 1) acrosslarge datasets Provenance must be obtained, and used throughout the workﬂow, in order to get meaningful results.

List of targets

Find existing data
per target and band

Filter out data regarding
quality parameters

Manipulate retrieved
datasets

DataprovenanceinformationneededFor the typical detection and processing steps in astronomy, a certain parameter (i.e., energy ﬂux) is measured and registered in ﬁles by a telescope’s
detectors, which are related to photon emission from an astronomical source (See Figure 2). Concrete questions for data Provenance are which were theﬁles used to create a particular data product (Associations), and which are the data products created from it? (Inverse Association). Additionally, weneed to know are we using the same source data (photons) for any two given data products? (Photon Accountancy), as the use of the same photons in
diﬀerent data products which are later combined can produce unreliable science. Plus, we need to be able to track parameters related to the models foremission, absorption and detection used by the reduction soware.
uses
knowledgeofraw files

data products

F1

Figure 1 Astronomical, multiwavelength analysis work-ﬂow
using e-Science tools such as theVO , s h ow i ng w h e re d at aProvenance information is arequirement for e-science.

Galactic &Extragalactic
extinction

AstronomicalSource

P1

F2

Atmosphere(Seeing &Opacity)

P2

DataReduction
Software

FnPm

Telescope, Filters, Detectors…

photon emission

Figure 2
Elements of astronomical data Provenance: datareduction pipelines, or standalone soware are used to convert
some property measured with the telescope detectors into up to nﬁles (Fn), and later into m physical parameters and data products(Pm). For that, they use knowledge of the telescope and detectors
settings (for details on the light-path, for instance, see [Delgado2009] in this poster session), plus modelling of the telescope,source, and the diﬀerent absorbing media photons have traversed.

data generation

Requirements for Data Provenance Systems: High-Level Use Cases

Data Quality

From a general classification of data Provenance systems’ properties, such as that shown in Figure 3, we can see that for some systems Provenanceis considered simply informational. Ideally, however, Provenance can be used to attribute dataset originators, replicate derivations of parameters,and ultimately assess dataset quality (both being retrieved, and being archived). If the Provenance collection starts with observatory systems,
audit trails can be created to see which people and systems have been responsible for each particular processing step from the origin.

UsageAudit Trail

SettledReplication

In study

Attribution

Data Provenance can be data or process oriented (but data oriented Provenance is called, in the IVOA context, Characterisation), orprocess oriented, so the metadata for Provenance outside of Characterisation must be process oriented.

InformationalProcess

Orientation

DataCoarse

Subject

Granularity

Figure 4, on the other hand, shows a classiﬁcation of the diﬀerent use cases that have been compiled as desireable for the ESO archive.
Any implementation of astronomical data Provenance within ESO must take into account those use cases.

Fine
Annotation

Data ProvenanceTaxonomy

Representation

Data Discovery Services

Storage

SchemeInversion

Information

Syntactic

Contents

Semantic

KindIn file
LinkedObservation preparation

Exposure Time
Calculations

Dataset Documentation

Technical Queries

Organisational Queries

Ambient Queries

Concerns

Publications

Dissemination

Scalability
Overhead

Visual GraphsQueries

Similar project provenancequeryA posteriori DataDiscovery-Like queries

Reference DB of opticalelements

On-the-fly creation of IST
query forms

Quality control queries

Observation Logs

Observing programmescheduling

ADP Logs

Figure 4 Classiﬁcation of use cases compiled for the ESO archive. e main group is Data Discovery using Provenance information (mostlythrough inverse association queries), while similar queries are desireable for dataset documentation (using direct association queries).

Service APIs

Figure 3: Taxonomy (based on [Simmhan 2005]) of the diﬀerent
aspects to be taken into account for any kind of e-science
Provenance system.
Extended Provenance

Glossary
Characterisation Observation metadata
within a multi-axis space (usuallyspace, time, frequency and observedproperty).
Curation Project-related metadata, suchas what kind of project does anobservation belong to, but also
encompassing data collections whichhave been but together a posteriori.
e-Science Enhanced science performed
through the use of networked,distributed resources. See VO.
IVOA International Virtual Observatory
Alliance, organisation in charge ofcreating the standards for thedescription and interoperation ofastronomical observations in archives.
http://www.ivoa.net

VO Virtual Observatory, a federation ofdata archives sharing the samedescription, including Curation,Characterisation and Provenancemetadata, plus common data accessprotocols.

References

e remaining elements to be set up for the establishment of a proper data Provenance, as shown
on Figure 3, are Granularity, Storage, Representation, and Dissemination.
‣ Granularity must be chosen to serve all use cases above, and also imposes restrictions on Storage (to beable to cope with the chosen Granularity; Storage is an implementation detail not relevant to consumers.)‣ Representation is the way Provenance is expressed when shared with requesting entities (asopposed to Storage). Hierarchical trees, series of keyword/value pairs, or even plain-text ﬁlesfor informational use cases. However, the rich nature of the objects described by Provenance(for instance, see [Delgado 2009] for a detailed description of the light-path), Representation
could be XML-based, for easier conversion to other IVOA formats.
‣ Dissemination, finally, is the way Provenance information is to be available for shared datasets.
Decisions in this respect include creating Provenance services to satisfy the use cases in Figure 4.
Provenance of astrophysical datasets is being discussed within the IVOA Working Group for Data
Modelling (DM WG). A Representation, in the form of a complete IVOA Observation data model,
has already been proposed in [Santander 2009] (see Figures 5 and 6), and is currently under study.
There is also a complete FITS keyword/metadata database already in place at ESO [Vuong 2008],
which will be one of the basis for establishing Storage and Granularity of the future ESO Provenanceservices. Currently, that facility stores more than five thousand million keyword-value pairs, formore than ten million raw file entries, and already includes metadata versioning support (arequirement for Provenance audit trails and historical dataset documentation).

Conclusion
By identifying the elements needed for a successful data Provenance management system, and theselection of already in place systems, the ESO archive will be able to provide even more advanced
scientific-oriented data products and services, while at the same time benefiting from existing IVOA
standards for the Virtual Observatory, and helping in their development, leading by example.

[Delgado 2009] Delgado, Delmotte, & Vuong, “Instrumental
Provenance of ESO Archival Data”, Proceedings of theADASS XIX, ASP Series (in preparation)[Santander 2009] Santander-Vela, “Integration of radio
astronomical archives and tools in the Virtual Observatory
framework,” Ph.D. Thesis, University of Granada, May 2009

Data Model for Observation

Observation

Curation

ObsData

PolicyCharacterization

Coverage

Resolution

ProvenancePrecision

TargetPackagingSensitivity

Accuracy

Data Model for
Dataset Characterisation

Figure 5 Proposal [Santander 2009] for an IVOA Observation
Data Model. e green highlight shows the concept of extended
Provenance, which adds spatial (Target) and project-oriented(Curation) Provenance to the Instrumental, Ambient andProcessing Provenance, shown in Figure 6 .
Observation

Provenance
Software…

ProvenanceObservation metadata
related to the origin of every
realisation of an observation (i.e., filesor data products), from instrumental
configuration, to ambient conditions,and processing steps.

Implementing Data Provenance at ESO: Challenges and Jump-Starts

ProcessingCalibration

Organisational
Project
Metadata

Observation

ProvenanceAmbient

Instrumental

Figure 6 Detailed hierarchical Representation of the elements ofdata Provenance, with emphasis in non-organisational Provenance.

[Simmhan 2005] Simmhan, Plale, & Gannon, “A Surveyof Data Provenance in e-Science,” SIGMOD Record 34, 3,Sept. 2005

[Vuong 2008] Vuong et al., “Applications of the ESO
metadata database”, Proceedings of the SPIE, vol. 7016,
pp. 70161M-70161M-8, July 2008