Building an Interoperable NASA Astronomy Archive

*a collaborative proposal of NASA's*

Astrophysics Data Centers Executive Committee

# 1. Introduction

We propose a framework for the coordination and integration of NASA's
astronomy data resources. In this introduction we discuss the motivation
for this effort and briefly sketch the outline of the project, the NASA
Interoperability Layer Elements (or NILE). The next section describes
the specific efforts we propose. The last section discusses the
organization of the project, its management, and the cost and schedule
for NILE implementation.

## 1.1 NASA Strategic Goals in Astronomy

NASA's role in astronomy is founded on the ability of space missions to
explore wavelength regimes, resolutions and other domains unreachable
from the ground. From its very first astronomy missions, NASA has
promoted the panchromatic view of the heavens, relating the observations
made in X-ray to optical, infrared to radio, and combining the high
spectral response of one mission or instrument with the high spatial
resolution of another. The recognition that understanding astrophysical
processes arises not from a single observation, but from careful use of
data with complementary characteristics, suffuses the roadmaps that have
recently been developed in NASA's Origins and Structure and Evolution of
the Universe (SEU) themes.

One of the current investigations in the Origins roadmap is the chemical
evolution of the universe. For hot plasmas astronomers measure
abundances using X-rays. For cooler plasmas and stars, optical and UV
spectra reveal the constituents. While for still cooler gases and dust,
radio and infrared emission is key. The SEU's Beyond Einstein roadmap
sets understanding black holes as one of its major goals. The standard
model of black holes in AGN has needed inputs from all regimes from
radio through gamma-ray and resolutions spanning the multi-degree fields
of two-lobed radio sources, to the current limits of radio VLBI and
ultimately we hope to the exquisitely fine resolution enabled by X-ray
interferometry.

This need to combine and contrast information from diverse sources is
central to many of the science goals of the roadmaps: the evolution of
galaxies, and the birth of stars, the structure of the microwave
background, .... When LISA sees a star falling into a black hole, we
will surely wish to compare LISA's results with any gamma-ray burst or
X-ray flare that results. When NGST makes a census of the stellar
nursery, our understanding of the dynamics of the cluster will be
bolstered by X-ray measurements and quantitative comparisons with
detailed theoretical calculations.

Both of the roadmaps recognize the importance of building archive
systems that enable the community to combine data from multiple sources
effectively. The SEU roadmap explicitly calls out the need for a NASA
role in the NVO effort when discussing the research and analysis
activities.[^1] The Origins roadmap, written slightly earlier, describes
the need for NASA archive system to continue at the forefront of
scientific research and to enable easy use of data from a given
satellite in any science context that may come up.[^2] In this white
paper we propose a structure for the NASA and its archive services to
respond to the imperatives of NASA's plans and to provide the data
archive resources that will be needed to reach NASA's science goals.

## 1.2 Interoperability among NASA Data Centers

NASA data centers have long recognized the need to work together and
with ground or foreign collaborators to build systems that meet our
users' needs. The Astronomy Data Centers Collaborative Committee (ADCCC)
was formed as an informal group to ensure that each center was cognizant
of the efforts of the others and to minimize duplication of effort. The
ADCCC was formally recognized by NASA and reorganized into the Astronomy
Data Centers Executive Committee (ADEC) in the beginning of 2001. The
interoperability working group of the ADEC categorized dozens of
specific instances in which NASA centers were interoperating.[^3]
However these efforts were largely ad hoc. None provides full coverage
to distributed resources and given their bilateral nature they tend to
make fragile interfaces, easily broken by changes at either end.

Without a more formal structure for building interoperable systems, it
has become clear that it is difficult to avoid wasted effort and often
impossible to reach useful goals. Recently a number of data centers have
begun delivering information connecting bibliographic references to data
sets at the centers. Each center developed its own software for this
task, duplicating the effort and requiring a separate interface between
the Astronomy Data System and each center. When we try to deal with a
more complex problem, e.g, what patches of the sky have been observed by
all of several different instruments, we go beyond the kind of services
that the centers can build without dedicated explicit resources.

The remainder of this white paper details how we propose to set up this
framework for cooperation. We propose a collaborative effort of all of
NASA's existing astronomy data centers to build the NASA
Interoperability Layer Elements (NILE). This effort, under the auspices
of the ADEC, will provide an integrated NASA astrophysics archives for
the scientific community and can be the foundation for major new
initiatives in building archive services. The NILE will consist of thin
layers on top of the archives' existing capabilities, providing new
services to users and avoiding substantial redesign of working archives
already optimized for specialists in the appropriate disciplines.

### 1.2.1 Integrating Archives: The User View

Users will see benefits from NILE at three levels. NILE will enable our
services to access remote data as if it were local. So rather than users
having to sequentially query each of our archives, they can start at the
one that is most convenient. If they have begun by looking for
information on a particular object, perhaps they get their data through
NED. If they are analyzing a set of X-ray observations, perhaps the
HEASARC or CXC's interfaces are their portals. Each of centers
interfaces will be able to link to data and resources at all of the
other centers. Almost all of the existing interoperability links
function at this level. NILE will enable us to support these links in a
uniform fashion and greatly increase the number and uniformity of
resources included.

A second level of integration not only links users to diverse resources,
it allows the user to combine this information meaningfully. For catalog
resources this means that users can correlate tables stored at different
centers. For archives, there are data models that encapsulate how
archive datasets can be used for analysis. Data models address questions
like how the data is stored and also how different files are related:
how the background and instrumental response are characterized, where to
find the spatial or energy resolution, and so forth. The individual
archives will draw on their own expertise in conjunction with other
Virtual Observatory data model working groups to develop these high
level descriptions of their data products.

Initially the data models will be descriptive, they will describe the
science information the datasets represent and indicate how they may be
used. As the models are developed and we gain confidence in them, we may
make the data models more proactive, the data model becomes an API that
allows the user to interact with archive data to perform specified
scientific tasks. At this stage the ADEC team will draw out
commonalities between the different archives' data models and establish
a common NASA data dictionary in the context of emerging international
standards.

The third level of integration is building entirely new functionality on
top of the lower level constructs. In the past decade NED and the ADS
have provided fundamentally new ways for astronomers to do archival
research. We should continue to look for such revolutionary approaches.
One interesting approach is to build a complete catalog of every object
detected by any NASA mission. Each object would be tied to all its
observations using the kinds of links described above. Such a project
might tackle some very difficult questions: how can one handle extended
or composite objects, and how do we classify objects. These seem
challenging today -- much like the issue of handling the diverse
astronomical nomenclature did prior to the advent of the NED and SIMBAD
name resolvers.

### 1.2.2 Integrating Data Centers: The System View

Parallel to these three user views are the three distinct elements that
comprise NILE. The lowest level NILE element is a set of relatively
simple interfaces that each center implements to enable remote access to
its resources. These interfaces will initially include access to the
catalog queries and data retrieval as well as a few more specialized
services, e.g., a specialized query asking for all observations that
might overlap a region, or a query to retrieve the data model associated
with a dataset.

The next element is a set of integrating tools. In some cases one
archive will know exactly what remote service it wishes to call, e.g.,
Chandra archive services might explicitly include a search for HST
observations of the same region. However, other services will not want
hard links to remote services: they will query a NASA registry of
services to find, e.g., all the catalogs of infrared sources. So as IRSA
includes new catalogs, these become available to other sites
automatically. Similarly, the need for cross-correlations of catalog
information from multiple sites is a capability that all of our centers
can use. Rather than requiring each center to build an explicit
cross-match service, we anticipate building a basic service that each
center can customize to its particular needs.

The last element of NILE is in building new capabilities into our
interfaces. For existing systems this involves retrofitting our user
tools to look for and retrieve remote resources using the lower level
elements. Clearly this part of the NILE effort will be specific to each
data center. However, we shall also be designing the NILE interfaces as
a template which describes how users, catalogs and archive services
interact. For new archives and missions the NILE interfaces should
provide a guide as to how to build local services as well as
capabilities to link to remote ones. New interfaces such as the
comprehensive NASA object database can also be built. These may involve
collaborations or resources outside of NILE, but the NILE interfaces are
the foundation on which they are built.

## 1.3 NILE and the Virtual Observatory

There are many efforts underway both within the United States and around
the world to build a "Virtual Observatory" where astronomy data and
resources are available wherever and to whoever needs them. Interest in
the VO has been galvanized by the selection of the VO as one of the
highest priorities identified in the latest decadal review of the state
of astronomy.[^4] NILE is envisaged as a major element of NASA's
response to the opportunities afforded by the growing interest in the
VO, but it reflects NASA's priorities and strategic goals.

A number of the lower level interfaces and protocols that we anticipate
using within NILE are being prototyped within the broader VO contexts.
These will be adapted for use within NILE. Many NASA institutions are
already participating within the VO and have been influential in the
development of these interfaces. The NILE framework should increase the
influence of NASA centers within the VO efforts and ensure that NASA's
goals and objectives are fully addressed.

Current VO efforts outside of NASA have focused on the development of
infrastructure that will support the VO. NILE is aimed at using and
extending this infrastructure to build a seamless NASA archive. The NASA
data centers include many of the most mature and successful data centers
throughout the astronomical community. They comprise a natural core
element to the burgeoning Virtual Observatory.

The coherence of NASA's mission goals may also be very helpful to the
broader VO. E.g., it is the intent of the Virtual Observatory to enable
confrontation of theoretical data with observation data. NASA's support
for theory programs tied to mission objectives may provide a context
where the VO can begin to address this ambitious goal. NILE interfaces
need not be limited to archival data. With its clear strategic goals,
mature archives, and existing resources, NASA centers are natural
leaders for the Virtual Observatory.

## 2. The NILE System

## 2.1 Interfaces

### 2.1.1 Catalog Interface

A catalog is something users can query to get a table of results. Most
of our current interfaces fall into this category. The table of results
may be a list of potential papers in the ADS, a list of nearby targets
in NED, or observations from MAST or the CXC. The NSF NVO program has
already developed some simple prototype interfaces for catalog queries
including the Cone Search interface and the Simple Image Access
protocol. Much of the work of the ADEC's Interoperability Working Group
has been in developing prototypes as well: downloads of bibliography
link tables and a resource discovery service. The new VOTable format has
been widely used to format catalog responses.

These early interfaces will not be adequate. Queries will need to
support a common interface for queries by at least position, region,
bibliographic reference, target name, observation identifier,
author/observer, object type, time and resource. The interface protocol
will also need to support more complex query concepts, joins and
correlations among tables. Catalog services at NASA archives are built
on top of existing relational and object data base engines. In some
cases users can pass SQL directly through to these databases, in others
scientists have been shielded from learning SQL. At this level in NILE,
the users are not astronomers, but other archive centers. Providing more
complete, if complex, query access to the underlying database is
appropriate and should be straightforward. Other VO efforts are looking
at enhancing the capabilities of the existing Catalog query interfaces.
NILE will supplement these efforts and implement them in the NASA
context.

### 2.1.2 The Archive Access Interface and Data Models

The archive access interface defines how a site provides remote access
to the observational data in its archive. This interface allows users to
download the data. A special case is the interface to cut-out services
such as those at MAST (DSS), IRSA (2MASS and IRAS) or the HEASARC
(*SkyView*).

We want users to be able to do more than just get bits and bytes. The
physical archive access protocols will be closely integrated with data
models that describe the semantics of the underlying data. Data models
enable us to describe datasets that may be stored in different formats,
but which are nonetheless comparable. E.g., an image may be stored using
a FITS image format in one archive and as a list of photons in another.
Several data models may be applicable to the same data (the events list
could also be used as a light curve). By providing some standardized
information on the meaning of the contents of the archive, we greatly
enhance the ability of scientists who are not specialists in a given
instrument to use its data.

Initially the use of data models in NILE will be descriptive. The data
model indicates the kind of data that is available in the archive, which
files are which types, and describes how that data can be transformed
into 'standard' models. With increasing maturity, NILE will use the data
models prescriptively. Data will be returned to the user in standard
formats. In the longer term data models may also be useful in the
catalog interfaces. There we already share a common model of tables with
fixed columns and scalar values. However as we move into object-oriented
databases where the results of a query can be more complex data models
may again glue together data from different systems.

**2.1.3 Region search capability**.

TBD \[This is the capability to find all observations that overlap a
user specified region (or point) using the actual coverage of the
observations\]

### 2.1.4 Preview Protocols

One of the issues that all Virtual Observatory system must address is to
shield users from irrelevant resources. While there may be thousands of
catalogs and many dozens of missions with archival data, it is unlikely
that a user will want to query all of these and most users will not
thank us for inundating them with irrelevant queries. One of the most
common reasons missions will be inappropriate is that they simply do not
have data at the position, time or energy that the user wants. To help
remote services avoid inappropriate links, a set of preview protocols
will be developed. These mimic the archive and catalog protocols in
their invocation, but are intended to be relatively lightweight. Rather
than returning actual results, they return estimates of how many results
would be returned if the 'real' service were queried with the same
parameters.

## 2.2 Integrating tools.

The NILE protocols provide interfaces through which an archive exposes
its holdings. In a number of cases systems at several different sites
will need to perform the same kinds of merging of information from
multiple NASA sites. Rather than duplicate the effort at all of those
sites, we propose to develop a few key integrating capabilities.
Individual sites will specialize these capabilities for their specific
requirements.

As NILE develops we anticipate that new opportunities for common
software will emerge. The tools listed here represent basic core
components.

### 2.2.1 Cross Correlation

Correlating the results of queries from multiple sites may be the most
ubiquitously useful capability that NILE provides to both sites and
users. NILE will develop a powerful cross-match capability for joining
results obtained using a catalog interface query. IRSA has led the NSF
NVO team in building specialized cross-match capabilities for large
tables, and the free tools can quickly support SQL queries of smaller
XML files (e.g., with hundreds or thousands of rows). A challenge
remains in combining the ability to do sophisticated queries with the
possibility of dealing with very large databases. The correlator will
itself implement the NILE catalog interface over the database comprised
of all of NILE catalog services.

The correlator will support joins, unions, correlative queries and
anticorrelations. A scientist may be interested in the brightest
infrared sources of some class that have *not* been observed by HST.
Multiple tables may be joined. Users will be able to include their own
tables for use in joins. Spatial joins may use the 2-D indexing
developed schemes developed at the SDSS, HTM indices[^5], or the
HEALPIX[^6] indices used for analysis of microwave background data. A
curator of several key microwave background data sets, LAMDBA and NASA
may wish to lead the effort to provide tools to merge data between
datasets indexed using these two schemes.

### 2.2.2 Resource Registries and Service Models

One of the major vulnerabilities of the current links between NASA
archives is the fragility of the URLs themselves. One a link has been
established between sites, it places a restriction on the development of
that site, and there is no standard way for a site to indicate that its
interface has changed. Similarly as our sites grow and develop we need a
way to publish the existence of new resources and to provide a
systematic way of describing the resources we have. Resource registries
are being extensively used in the commercial world where the UDDI
protocol is dominant. The GLU service developed by the CDS provides an
alternative implementation for resource registries in the astronomical
context.

The resource registry will use the NILE catalog interface to support
queries that enable service to discover the existence and
characteristics of services and to learn how these services are to be
used. The registry works based upon a set of data models describing
basic data services: catalog/archive services like those for MAST,
HEASARC, CXC and IRSA, cutout services like the DSS, *SkyView* and IRSA
2MASS server, the bibliographic search service supported by SIMBAD, and
several others. As we explore new ways to present data to users we will
doubtless extend and refine our service models.

Appropriate metadata, the regimes included in the service, the physical
parameters available, the epoch of the data, will be included in a
resource catalog that users or services can query. The returned
descriptions include a précis of the specific protocols describing how
the service can be used in terms of the basic service models.

## 2.3 Using NILE

The goal of NILE is not to build interfaces, it is to use them. . Users
retrieving HST images using MAST will be able to get comparable images
from IRSA and the HEASARC without leaving Starview. The LAMBDA archive
will have transparent access to diffuse images at IRSA or in *SkyView.*
Users may select objects within NED on the basis of the archival
information available for the object as well as the object parameters
themselves. The ADS can become a major index not just for publications
but for organizing data: the growing links between publications and
datasets will provide a network of references that will support a whole
new style of Google-like searches.

### 2.3.1 Enhancing existing services

Specific goals for each center are described in the following
paragraphs.

### *2.3.1.1 ADS*

### *2,3.1.2 CXC*

### *2.3.1.3 HEASARC*

The HEASARC provides a number of interfaces to the query high-energy
data and other resources. Using NILE, the HEASARC Browse interface will
provide correlative data from all wavelengths as products for
observations by high-energy imaging instruments. The NILE catalog access
and cross-correlation tools will be available, so that users can perform
immediate comparisons of observations and targets detected by any NASA
mission. The *SkyView* cut-out service will use the NILE archive
protocols and registry to locate and query any remote cut-out services.
Conversely *SkyView* will be available as a NILE resource to mosaic data
from any survey data set.

### *2.3.1.4 IRSA*

### *2.3.1.5 LAMBDA*

### 2.3.1.6 MAST

### 2.3.1.7 NED

\[Do we need Michelson center here\]

### 2.3.2 New Initiatives 

While the initial use of NILE will be to enhance our existing systems,
we also envisage major new capabilities that can be built on top of this
system. Building on the experience of NED in integrating NILE into its
existing interface, we see the potential for a comprehensive object
catalog that may provide a whole new way to integrate archival data.
Rather than using the observation based model of traditional archives,
or the positional criteria of cut-out and mosaicking services, the
comprehensive object archive organizes data by linking all that is known
about a given object: physical parameters, references, archival data,
and user inputs. This object database will tackle several very complex
issues: how one object may be part of another, extended emission and
object classification. This full effort may go beyond the NILE
initiative. It involves not only NASA but all of astronomy, and may have
a scope beyond what NILE can support. However within our project we can
see NILE developing the data model describing the 'astronomical object'
and implementing that data model on our existing catalogs. With just
these elements and the NILE archive interface, we can make substantial
progress towards a comprehensive object catalog.

Another area of great interest is in data mining of NASA catalogs and
archives. Once we have developed standardized interface protocols and
data models on top of these, we may find that archival analysis projects
are possible on a massive scale that was previously infeasible. We may
need to begin to provide user analysis resources, and a protocol for
invoking them, near the archives since the element of data systems
growing the slowest is network capacity. Here the NVO efforts experience
with grid computing may be directly applicable to NILE.

# 3. Program Management and Budget

## 3.1 Program Management

3**.1.1 Overall program management**

\[how do we run this overall. Can the ADEC do this or do we need a
program manager\]

### 3.1.2 Development matrix

\[A matrix showing who leads which activity.\]

### 3.1.3 Schedule

## 3.2 Budget

[^1]: Beyond Einstein, p 90.

[^2]: Origins Roadmap, p 45?, p 66

[^3]: Garching ITWG paper.

[^4]: Decadal review reference.

[^5]: HTM (SDSS reference)

[^6]: HEALPIX reference