Building an Interoperable NASA Astronomy Archive

a collaborative proposal of NASAs
Astrophysics Data Centers Executive Committee

1. Introduction
We propose a framework for the coordination and integration of NASA's astronomy data 
resources.  In this introduction we discuss the motivation for this effort and briefly sketch 
the outline of the project, the NASA Interoperability Layer Elements (or NILE).  The 
next section describes the specific efforts we propose.  The last section discusses the 
organization of the project, its management, and the cost and schedule for NILE 
implementation.

1.1 NASA Strategic Goals in Astronomy
NASA's role in astronomy is founded on the ability of space missions to explore 
wavelength regimes, resolutions and other domains unreachable from the ground.  From 
its very first astronomy missions, NASA has promoted the panchromatic view of the 
heavens, relating the observations made in X-ray to optical, infrared to radio, and 
combining the high spectral response of one mission or instrument with the high spatial 
resolution of another.   The recognition that understanding astrophysical processes arises 
not from a single observation, but from careful use of data with complementary 
characteristics, suffuses the roadmaps that have recently been developed in NASA's 
Origins and Structure and Evolution of the Universe  (SEU) themes. 

One of the current investigations in the Origins roadmap is the chemical evolution of the 
universe.  For hot plasmas astronomers measure abundances using X-rays. For cooler 
plasmas and stars, optical and UV spectra reveal the constituents. While for still cooler 
gases and dust, radio and infrared emission is key.   The SEU's Beyond Einstein roadmap 
sets understanding black holes as one of its major goals.  The standard model of black 
holes in AGN has needed inputs from all regimes from radio through gamma-ray and 
resolutions spanning the multi-degree fields of two-lobed radio sources, to the current 
limits of radio VLBI and ultimately we hope to the exquisitely fine resolution enabled by 
X-ray interferometry.

This need to combine and contrast information from diverse sources is central to many of 
the science goals of the roadmaps: the evolution of galaxies, and the birth of stars, the 
structure of the microwave background, ....  When LISA sees a star falling into a black 
hole, we will surely wish to compare LISA's results with any gamma-ray burst or X-ray 
flare that results.  When NGST makes a census of the stellar nursery, our understanding 
of the dynamics of the cluster will be bolstered by X-ray measurements and quantitative 
comparisons with detailed theoretical calculations.

Both of the roadmaps recognize the importance of building archive systems that enable 
the community to combine data from multiple sources effectively.  The SEU roadmap 
explicitly calls out the need for a NASA role in the NVO effort when discussing the 
research and analysis activities.   The Origins roadmap, written slightly earlier, describes 
the need for NASA archive system to continue at the forefront of scientific research and 
to enable easy use of data from a given satellite in any science context that may come 
up.    In this white paper we propose a structure for the NASA and its archive services to 
respond to the imperatives of NASA's plans and to provide the data archive resources 
that will be needed to reach NASA's science goals.

1.2 Interoperability among NASA Data Centers
NASA data centers have long recognized the need to work together and with ground or 
foreign collaborators to build systems that meet our users' needs.  The Astronomy Data 
Centers Collaborative Committee (ADCCC) was formed as an informal group to ensure 
that each center was cognizant of the efforts of the others and to minimize duplication of 
effort.  The ADCCC was formally recognized by NASA and reorganized into the 
Astronomy Data Centers Executive Committee (ADEC) in the beginning of 2001.   The 
interoperability working group of the ADEC categorized dozens of specific instances in 
which NASA centers were interoperating.   However these efforts were largely ad hoc.  
None provides full coverage to distributed resources and given their bilateral nature they 
tend to make fragile interfaces, easily broken by changes at either end.

Without a more formal structure for building interoperable systems, it has become clear 
that it is difficult to avoid wasted effort and often impossible to reach useful goals. 
Recently a number of data centers have begun delivering information connecting 
bibliographic references to data sets at the centers.  Each center developed its own 
software for this task, duplicating the effort and requiring a separate interface between the 
Astronomy Data System and each center.  When we try to deal with a more complex 
problem, e.g, what patches of the sky have been observed by all of several different 
instruments, we go beyond the kind of services that the centers can build without 
dedicated explicit resources.   

The remainder of this white paper details how we propose to set up this framework for 
cooperation.   We propose a collaborative effort of all of NASA's existing astronomy 
data centers to build the NASA Interoperability Layer Elements (NILE).  This effort, 
under the auspices of the ADEC, will provide an integrated NASA astrophysics archives 
for the scientific community and can be the foundation for major new initiatives in 
building archive services.  The NILE will consist of thin layers on top of the archives' 
existing capabilities, providing new services to users and avoiding substantial redesign of 
working archives already optimized for specialists in the appropriate disciplines.

1.2.1 Integrating Archives: The User View
Users will see benefits from NILE at three levels.  NILE will enable our services to 
access remote data as if it were local.  So rather than users having to sequentially query 
each of our archives, they can start at the one that is most convenient.  If they have begun 
by looking for information on a particular object, perhaps they get their data through 
NED.  If they are analyzing a set of X-ray observations, perhaps the HEASARC or 
CXC's interfaces are their portals.  Each of centers interfaces will be able to link to data 
and resources at all of the other centers.  Almost all of the existing interoperability links 
function at this level.  NILE will enable us to support these links in a uniform fashion and 
greatly increase the number and uniformity of resources included.

A second level of integration not only links users to diverse resources, it allows the user 
to combine this information meaningfully.  For catalog resources this means that users 
can correlate tables stored at different centers.  For archives, there are data models that 
encapsulate how archive datasets can be used for analysis.  Data models address 
questions like how the data is stored and also how different files are related: how the 
background and instrumental response are characterized, where to find the spatial or 
energy resolution, and so forth.   The individual archives will draw on their own expertise 
in conjunction with other Virtual Observatory data model working groups to develop 
these high level descriptions of their data products.  

Initially the data models will be descriptive, they will describe the science information 
the datasets represent and indicate how they may be used.  As the models are developed 
and we gain confidence in them, we may make the data models more proactive, the data 
model becomes an API that allows the user to interact with archive data to perform 
specified scientific tasks.  At this stage the ADEC team will draw out commonalities 
between the different archives data models and establish a common NASA data 
dictionary in the context of emerging international standards.

The third level of integration is building entirely new functionality on top of the lower 
level constructs.  In the past decade NED and the ADS have provided fundamentally new 
ways for astronomers to do archival research.  We should continue to look for such 
revolutionary approaches.  One interesting approach is to build a complete catalog of 
every object detected by any NASA mission.    Each object would be tied to all its 
observations using the kinds of links described above.  Such a project might tackle some 
very difficult questions: how can one handle extended or composite objects, and how do 
we classify objects.  These seem challenging today - much like the issue of handling the 
diverse astronomical nomenclature did prior to the advent of the NED and SIMBAD 
name resolvers.

1.2.2 Integrating Data Centers: The System View
Parallel to these three user views are the three distinct elements that comprise NILE.  The 
lowest level NILE element is a set of relatively simple interfaces that each center 
implements to enable remote access to its resources.  These interfaces will initially 
include access to the catalog queries and data retrieval as well as a few more specialized 
services, e.g., a specialized query asking for all observations that might overlap a region, 
or a query to retrieve the data model associated with a dataset.

The next element is a set of integrating tools.  In some cases one archive will know 
exactly what remote service it wishes to call, e.g., Chandra archive services might 
explicitly include a search for HST observations of the same region.  However, other 
services will not want hard links to remote services: they will query a NASA registry of 
services to find, e.g., all the catalogs of infrared sources.  So as IRSA includes new 
catalogs, these become available to other sites automatically.  Similarly, the need for 
cross-correlations of catalog information from multiple sites is a capability that all of our 
centers can use.  Rather than requiring each center to build an explicit cross-match 
service, we anticipate building a basic service that each center can customize to its 
particular needs.  

The last element of NILE is in building new capabilities into our interfaces.   For existing 
systems this involves retrofitting our user tools to look for and retrieve remote resources 
using the lower level elements.  Clearly this part of the NILE effort will be specific to 
each data center.  However, we shall also be designing the NILE interfaces as a template 
which describes how users, catalogs and archive services interact.  For new archives and 
missions the NILE interfaces should provide a guide as to how to build local services as 
well as capabilities to link to remote ones.  New interfaces such as the comprehensive 
NASA object database can also be built.  These may involve collaborations or resources 
outside of NILE, but the NILE interfaces are the foundation on which they are built.

1.3 NILE and the Virtual Observatory
There are many efforts underway both within the United States and around the world to 
build a Virtual Observatory where astronomy data and resources are available 
wherever and to whoever needs them.  Interest in the VO has been galvanized by the 
selection of the VO as one of the highest priorities identified in the latest decadal review 
of the state of astronomy.   NILE is envisaged as a major element of NASA's response to 
the opportunities afforded by the growing interest in the VO, but it reflects NASA's 
priorities and strategic goals.  

A number of the lower level interfaces and protocols that we anticipate using within 
NILE are being prototyped within the broader VO contexts.   These will be adapted for 
use within NILE.  Many NASA institutions are already participating within the VO and 
have been influential in the development of these interfaces.   The NILE framework 
should increase the influence of NASA centers within the VO efforts and ensure that 
NASAs goals and objectives are fully addressed. 

Current VO efforts outside of NASA have focused on the development of infrastructure 
that will support the VO.  NILE is aimed at using and extending this infrastructure to 
build a seamless NASA archive.  The NASA data centers include many of the most 
mature and successful data centers throughout the astronomical community.  They 
comprise a natural core element to the burgeoning Virtual Observatory.

The coherence of NASA's mission goals may also be very helpful to the broader VO.  
E.g., it is the intent of the Virtual Observatory to enable confrontation of theoretical data 
with observation data.   NASA's support for theory programs tied to mission objectives 
may provide a context where the VO can begin to address this ambitious goal. NILE 
interfaces need not be limited to archival data.  With its clear strategic goals, mature 
archives, and existing resources, NASA centers are natural leaders for the Virtual 
Observatory.


2. The NILE System

2.1 Interfaces

2.1.1 Catalog Interface
A catalog is something users can query to get a table of results.  Most of our current 
interfaces fall into this category.  The table of results may be a list of potential papers in 
the ADS, a list of nearby targets in NED, or observations from MAST or the CXC.  The 
NSF NVO program has already developed some simple prototype interfaces for catalog 
queries including the Cone Search interface and the Simple Image Access protocol.  
Much of the work of the ADEC's Interoperability Working Group has been in developing 
prototypes as well: downloads of bibliography link tables and a resource discovery 
service.   The new VOTable format has been widely used to format catalog responses.

These early interfaces will not be adequate.  Queries will need to support a common 
interface for queries by at least position, region, bibliographic reference, target name, 
observation identifier, author/observer, object type, time and resource.  The interface 
protocol will also need to support more complex query concepts, joins and correlations 
among tables.   Catalog services at NASA archives are built on top of existing relational 
and object data base engines.  In some cases users can pass SQL directly through to these 
databases, in others scientists have been shielded from learning SQL.  At this level in 
NILE, the users are not astronomers, but other archive centers.  Providing more complete, 
if complex, query access to the underlying database is appropriate and should be 
straightforward.   Other VO efforts are looking at enhancing the capabilities of the 
existing Catalog query interfaces.  NILE will supplement these efforts and implement 
them in the NASA context.

2.1.2 The Archive Access Interface and Data Models
The archive access interface defines how a site provides remote access to the 
observational data in its archive.  This interface allows users to download the data.  A 
special case is the interface to cut-out services such as those at MAST (DSS), IRSA 
(2MASS and IRAS) or the HEASARC (SkyView).   

We want users to be able to do more than just get bits and bytes.  The physical archive 
access protocols will be closely integrated with data models that describe the semantics 
of the underlying data.  Data models enable us to describe datasets that may be stored in 
different formats, but which are nonetheless comparable.  E.g., an image may be stored 
using a FITS image format in one archive and as a list of photons in another.   Several 
data models may be applicable to the same data (the events list could also be used as a 
light curve).  By providing some standardized information on the meaning of the contents 
of the archive, we greatly enhance the ability of scientists who are not specialists in a 
given instrument to use its data. 

Initially the use of data models in NILE will be descriptive.  The data model indicates the 
kind of data that is available in the archive, which files are which types, and describes 
how that data can be transformed into 'standard' models.   With increasing maturity, 
NILE will use the data models prescriptively.  Data will be returned to the user in 
standard formats.  In the longer term data models may also be useful in the catalog 
interfaces.  There we already share a common model of tables with fixed columns and 
scalar values.  However as we move into object-oriented databases where the results of a 
query can be more complex data models may again glue together data from different 
systems.

2.1.3 Region search capability.
TBD [This is the capability to find all observations that overlap a user specified region 
(or point) using the actual coverage of the observations]

2.1.4 Preview Protocols
One of the issues that all Virtual Observatory system must address is to shield users from 
irrelevant resources.  While there may be thousands of catalogs and many dozens of 
missions with archival data, it is unlikely that a user will want to query all of these and 
most users will not thank us for inundating them with irrelevant queries. One of the most 
common reasons missions will be inappropriate is that they simply do not have data at the 
position, time or energy that the user wants.   To help remote services avoid inappropriate 
links, a set of preview protocols will be developed.  These mimic the archive and catalog 
protocols in their invocation, but are intended to be relatively lightweight.  Rather than 
returning actual results, they return estimates of how many results would be returned if 
the 'real' service were queried with the same parameters.

2.2 Integrating tools.
The NILE protocols provide interfaces through which an archive exposes its holdings.  In 
a number of cases systems at several different sites will need to perform the same kinds 
of merging of information from multiple NASA sites.  Rather than duplicate the effort at 
all of those sites, we propose to develop a few key integrating capabilities.  Individual 
sites will specialize these capabilities for their specific requirements.

As NILE develops we anticipate that new opportunities for common software will 
emerge.  The tools listed here represent basic core components.

2.2.1 Cross Correlation
Correlating the results of queries from multiple sites may be the most ubiquitously useful 
capability that NILE provides to both sites and users.  NILE will develop a powerful 
cross-match capability for joining results obtained using a catalog interface query. IRSA 
has led the NSF NVO team in building specialized cross-match capabilities for large 
tables, and the free tools can quickly support SQL queries of smaller XML files (e.g., 
with hundreds or thousands of rows).    A challenge remains in combining the ability to 
do sophisticated queries with the possibility of dealing with very large databases.   The 
correlator will itself implement the NILE catalog interface over the database comprised 
of all of NILE catalog services.

The correlator will support joins, unions, correlative queries and anticorrelations.  A 
scientist may be interested in the brightest infrared sources of some class that have not 
been observed by HST.  Multiple tables may be joined.  Users will be able to include 
their own tables for use in joins.  Spatial joins may use the 2-D indexing developed 
schemes developed at the SDSS, HTM indices , or the HEALPIX  indices used for 
analysis of microwave background data.  A curator of several key microwave background 
data sets, LAMDBA and NASA may wish to lead the effort to provide tools to merge 
data between datasets indexed using these two schemes.

2.2.2 Resource Registries and Service Models
One of the major vulnerabilities of the current links between NASA archives is the 
fragility of the URLs themselves.  One a link has been established between sites, it places 
a restriction on the development of that site, and there is no standard way for a site to 
indicate that its interface has changed.  Similarly as our sites grow and develop we need a 
way to publish the existence of new resources and to provide a systematic way of 
describing the resources we have.  Resource registries are being extensively used in the 
commercial world where the UDDI protocol is dominant.  The GLU service developed 
by the CDS provides an alternative implementation for resource registries in the 
astronomical context.

The resource registry will use the NILE catalog interface to support queries that enable 
service to discover the existence and characteristics of services and to learn how these 
services are to be used.   The registry works based upon a set of data models describing 
basic data services: catalog/archive services like those for MAST, HEASARC, CXC and 
IRSA, cutout services like the DSS, SkyView and IRSA 2MASS server, the bibliographic 
search service supported by SIMBAD, and several others.  As we explore new ways to 
present data to users we will doubtless extend and refine our service models.

Appropriate metadata, the regimes included in the service, the physical parameters 
available, the epoch of the data, will be included in a resource catalog that users or 
services can query.   The returned descriptions include a prcis of the specific protocols 
describing how the service can be used in terms of the basic service models.

2.3 Using NILE
The goal of NILE is not to build interfaces, it is to use them. .  Users retrieving HST 
images using MAST will be able to get comparable images from IRSA and the 
HEASARC without leaving Starview.   The LAMBDA archive will have transparent 
access to diffuse images at IRSA or in SkyView.  Users may select objects within NED on 
the basis of the archival information available for the object as well as the object 
parameters themselves.   The ADS can become a major index not just for publications but 
for organizing data: the growing links between publications and datasets will provide a 
network of references that will support a whole new style of Google-like searches.  

2.3.1 Enhancing existing services
Specific goals for each center are described in the following paragraphs.

2.3.1.1 ADS
2,3.1.2 CXC
2.3.1.3 HEASARC
The HEASARC provides a number of interfaces to the query high-energy data and other 
resources.  Using NILE, the HEASARC Browse interface will provide correlative data 
from all wavelengths as products for observations by high-energy imaging instruments. 
The NILE catalog access and cross-correlation tools will be available, so that users can 
perform immediate comparisons of observations and targets detected by any NASA 
mission.  The SkyView cut-out service will use the NILE archive protocols and registry to 
locate and query any remote cut-out services.  Conversely SkyView will be available as a 
NILE resource to mosaic data from any survey data set. 
2.3.1.4 IRSA
2.3.1.5 LAMBDA
2.3.1.6 MAST
2.3.1.7 NED

[Do we need Michelson center here]

2.3.2 New Initiatives 
While the initial use of NILE will be to enhance our existing systems, we also envisage 
major new capabilities that can be built on top of this system.  Building on the experience 
of NED in integrating NILE into its existing interface, we see the potential for a 
comprehensive object catalog that may provide a whole new way to integrate archival 
data.  Rather than using the observation based model of traditional archives, or the 
positional criteria of cut-out and mosaicking services, the comprehensive object archive 
organizes data by linking all that is known about a given object: physical parameters, 
references, archival data, and user inputs.  This object database will tackle several very 
complex issues: how one object may be part of another, extended emission and object 
classification.  This full effort may go beyond the NILE initiative.  It involves not only 
NASA but all of astronomy, and may have a scope beyond what NILE can support.   
However within our project we can see NILE developing the data model describing the 
astronomical object and implementing that data model on our existing catalogs.  With 
just these elements and the NILE archive interface, we can make substantial progress 
towards a comprehensive object catalog.

Another area of great interest is in data mining of NASA catalogs and archives.  Once we 
have developed standardized interface protocols and data models on top of these, we may 
find that archival analysis projects are possible on a massive scale that was previously 
infeasible.  We may need to begin to provide user analysis resources, and a protocol for 
invoking them, near the archives since the element of data systems growing the slowest is 
network capacity.  Here the NVO efforts experience with grid computing may be directly 
applicable to NILE.

3. Program Management and Budget
3.1 Program Management
3.1.1 Overall program management
[how do we run this overall.  Can the ADEC do this or do we need a program manager]
3.1.2 Development matrix
[A matrix showing who leads which activity.]

3.1.3 Schedule
3.2 Budget