# Presentation: 211750

## NASA’s Evolution of ESEData and Information Systems

**NASA’s Evolution of ESE****Data and Information Systems**

**Martha Maiden,**

- Program Executive for Data Systems
- Program Planning and Development DivisionNASA HQ

**Land Processes DAAC Science Advisory Panel Meeting**
**September 10, 2003**

## NASA’s current satellite capabilities and future missions
The current elements of the ESE data and information system
The challenges to our current data and information systems
Steps toward an evolution of the ESE data and information systems
NASA’s policies and plans for the long term archive

- The current elements of the ESE data and information system
- The challenges to our current data and information systems
- Steps toward an evolution of the ESE data and information systems
- NASA’s policies and plans for the long term archive

- Presentation Overview

## ESE Current Measurement Capabilities

**Aqua**

**Terra**

**TRMM**

**SORCE**

**SeaWiFS**

**GRACE**

**ICESat**

**Jason**

**NOAA/POES**

***The Earth Observing System -- systematic measurement of interactions among land, oceans, atmosphere, ice & life***

***Exploratory missions to probe key Earth system processes globally for the first time***

**NOAA/GOES**

*Operational precursor / Technology demos*

**EO-1: ALI & Hyperion**

**ACRIMSAT**

**TOMS**

**ERBS**

**TOPEX/**
**Poseidon**

***Operational weather services missions with NOAA***

**Landsat**

## ESE Next Generation Missions

**Synthetic Aperture Radar**

**Chemistry /**** Climate Mission**

**Ocean Surface Topography Mission**

**NOAA / NPOESS**

*Next generation systematic measurement missions to extend / enhance the record of science-quality global change data.*

*Research missions to probe key Earth system processes globally for the first time*

*Operational weather services missions with NOAA/DOD*

**NOAA/GOES-R**

**NPOESS Preparatory Project**

**Aerosol Mission**

**Landsat Data**
**Continuity Mission**

**Solar Irradiance Measurement **

**Global Precipitation Measurement**

**Total Column Ozone **

**Orbiting Carbon Observatory**

*Future research measurements: Soil moisture, Advanced gravity, Ocean carbon, Cold climate processes, Vegetation recovery*

**In Formulation**** **
**Candidate Future Missions**

**Aura**

**Cloudsat**

**Calipso**

**EO-3: GIFTS**

**Aquarius**

## Cross-Cutting Science Observations to Support ESE Science Roadmaps

## Strategic Planning to Implementation through Science Roadmaps

**Six Science Focus Area Roadmaps**

- ESE
- Strategy

**Observation System Implementation**

**Measurement and **
**Technology**
**Requirements**

## The Earth Science Enterprise

**Flight Operations,**

**Data Capture,**

**Initial Processing,**

_**Backup Archive**_

**Data**

**Transport**

_**to DAACs**_

**Science Data**

**Processing,**

**Info Mgmt,  Data**

_**Archive, & Distribution**_

**Distribution,**

**Access,**

**Interoperability,**

_**Reuse**_

**Spacecraft**

**NASA**

**Integrated**

**Services**

**Network**
**(NISN)**
**Mission**
**Services**

**WWW**

**Value-Added**

**Providers**

**Interagency**

**Data**

**Centers**

**Int’l Partners**

**& Data**

**Centers**

_**Data Acquisition**_

**Ground**
**Stations**

**Tracking**

**& Data**

**Relay Satellite**

**(TDRS)**

**Research**

**Users**

**Education**

**Users**

**Distributed**
**Active**
**Archive**
**Centers**

**Science**
**Teams**

**Data **
**Processing**
**&**
**Mission**
**Control**

**Polar Ground Stations**

- REASoNs

## NASA’s Ground Network

-  50 ground station antennas; 7 geographic antenna locations
-  4 different owner/operator models
-  14 contracts for commercial services

**GN Antenna Map (Number of Antennas)**

**Fairbanks, AK**

**Svalbard, Norway**

**Wallops,VA**

**Merritt Island, FL**

**Santiago, Chile**

**McMurdo**

**Hartebeesthoek,****South Africa**

**26**

**Owner/Operator Model**

**Primary Support Category**

**6**

**1**

**1**

**1**

**1**

**2**

**1**

**1**

**White****Sands, ****NM**

**Ultra Long Duration Balloons**

**Shuttle Launch**

**Notes:**
Typical of the heterogeneity in GN is the Fairbanks, Alaska area.  
AGS – Lockheed, 		11.3M Rx 8-9 Ghz
PF1 – Datalynx sub to Lockheed 	11M
Universal Space Network 	13M Rx 7.8-8.5 GHz
ASF-University of Alaska 	10M 8.025-8.4, 11.3M 8-9 GHz
Gilmore Creek – NOAA. 	3x13M 7.25-8.5 GHZ

## Information Systems:  EOS Mission Control

**Earth Science Mission Operations**
- Manage command and control spacecraft power, dynamics,
- Monitor instrument operations, build composite observation schedule
- Manage spacecraft recorder dumps and schedule  retrieval  with ground stations and space network
- House Landsat operations for USGS, and instrument operations for SeaWinds/ADEOS II

**Notes:**
The Earth Science Mission Operations (ESMO) control center is a part of the Earth Science Data Information System (ESDIS)

## ESE Data Center Locations

- A total of 50 widely distributed data centers (some of which are at the same location).
- ESE has recently updated our peer reviewed data and information producing centers through the Research, Education and Applications, Solutions Network Cooperative Agreement Notice (REASoN CAN) for development of next-generation architectures.

**Distributed Active Archive Centers (DAACs)**
**REASoN Projects**

**9**

**2**

**6**

**2**

## EOSDIS Architectural Data Flow

*The end-to-end mission data handling responsibility must handle the information from photons in to science and operational products out.*

- Activity

- Schedule,

- Instrument

- Status

**DAACs**

**Users**

- Science Data Products

**SIPS**

- <  Search and Order

- Results  >

- L1+

***EMSn / ESSn / NISN networks not shown***

**GS**

- Science

- (52Mbps)

**EDOS**

**LZPF**

**EOC**

**EDOS GSIF**

- X

- -

- Band

**FDS**

- FDS

- Products

- R

- B

- D

*******

***--***

***Science Team ***

***Responsibility***

**OMI**

**VFD***

**AURA **

**ISTs**

- EDG

- History Data, Carry

- -

- out Files

- (Instrument  & GBAD )

- Scheduling Requests,

- Instr

- . Loads, Analysis Requests

**--**

***Modified***

***For Aura***

**SCFs**

**Ground Stations +**

**GSIFs**

- @ Ala. & Nor.

- TDRSS Tracking, clock,

- contingency Cmds & Tlm

- DPREP O/A

- RT HK, Schedules, Planning Aids, Analysis Data

- Definitive Orbit

- L0 PDSs, EDSs

**HIRDLS NRTP ***

**OMI NRTS ***

**--**

***Aura Specific***

**Selected Users**

- SW, OPF

- From OMI PI SCF

- L0+, Aux., Anc.

- Emergency GS @ WGS

- OMI Direct Broadcast

**KNMI Server ***

- }

- PB HK (524 Kbps)

- RBD

- Schedules

- Mission Data

- Science Data

- Support Data

- Products

- by order

- DAPs

- Cmds, Tlm

- via S

- -

- Band

- Science Data

- via  X

- -

- Band

- Memory

- Dumps>

- <Loads

- < Commands

- Real

- -

- time HK >

**FDF**

- SN Tracking

- Data

- HK Tlm

- GS Tracking

- LOF Reports

- SN Tracking

- Pass

- Results

- UPD >

- Schedules

- Products

- SN

- Schedules

**PVVF**

- Acquisition

- Data

**DSMC**

- GS Tracking

- DAPs

- GES

- GES

- &

- ASDC

- WOTIS

- < GCMR

- GS

- Schedules

**E**

**R**

**P**

**S**

- ERPS

**E**

**R**

**P**

**S**

- ERPS

- < Cmds

- Tlm >

- EMOS

**WSC**

- E

- R

- P

- S

- Activity

- Schedule,

- Instrument

- Status

**DAACs**

**Users**

- Science Data Products

**SIPS**

- <  Search and Order

- Results  >

- L1+

***EMSn / ESSn / NISN networks not shown***

**GS**

- Science

- (52Mbps)

**EDOS**

**LZPF**

**EOC**

**EDOS GSIF**

- X

- -

- Band

**FDS**

- FDS

- Products

- R

- B

- D

*******

***--***

***Science Team ***

***Responsibility***

**OMI**

**VFD***

**AURA **

**ISTs**

- EDG

- History Data, Carry

- -

- out Files

- (Instrument  & GBAD )

- Scheduling Requests,

- Instr

- . Loads, Analysis Requests

**--**

***Modified***

***For Aura***

**--**

***Modified***

***For Aura***

**SCFs**

**Ground Stations +**

**GSIFs**

- @ Ala. & Nor.

- TDRSS Tracking, clock,

- contingency Cmds & Tlm

- DPREP O/A

- RT HK, Schedules, Planning Aids, Analysis Data

- Definitive Orbit

- L0 PDSs, EDSs

**HIRDLS NRTP ***

**OMI NRTS ***

**--**

***Aura Specific***

**--**

***Aura Specific***

**Selected Users**

- SW, OPF

- From OMI PI SCF

- L0+, Aux., Anc.

- Emergency GS @ WGS

- OMI Direct Broadcast

**KNMI Server ***

- }

- PB HK (524 Kbps)

- RBD

- Schedules

- Mission Data

- Science Data

- Support Data

- Mission Data

- Science Data

- Support Data

- Products

- by order

- DAPs

- Cmds, Tlm

- via S

- -

- Band

- Science Data

- via  X

- -

- Band

- Memory

- Dumps>

- <Loads

- < Commands

- Real

- -

- time HK >

**FDF**

- SN Tracking

- Data

- HK Tlm

- GS Tracking

- LOF Reports

- SN Tracking

- Pass

- Results

- UPD >

- Schedules

- Products

- SN

- Schedules

**PVVF**

- Acquisition

- Data

**DSMC**

- GS Tracking

- DAPs

- GES

- GES

- &

- ASDC

- WOTIS

- < GCMR

- GS

- Schedules

**E**

**R**

**P**

**S**

- ERPS

**E**

**R**

**P**

**S**

- ERPS

- < Cmds

- Tlm >

- EMOS

**WSC**

- E

- R

- P

- S

*For AURA*

***Mission Systems***

***Science Systems***

## Information Systems:  A Growing Demand for Earth Science Data and Information

**Ingesting, processing, and archiving an unprecedented volume of climate and Earth science data (currently ~2TB/day).**
**We are benchmarking capabilities and processes for handling the capacities for future operational  needs (e.g., NPOESS).**

**NASA provides access to Earth system science data, information, and services to millions of unique users.**
**Over the next decade, ESE will ensure the timely delivery of Earth Science information at an affordable cost by evolving to a more open, distributed set of data systems and service providers.**

## Challenges  to NASA’s ESE Data Systems

- Enable flexibility within data systems to adapt to new data stream(s) or to changes in current processing streams.
- Assure products flowing from processing stream meet the needs of the focus area science teams - scientists working in conjunction with data systems experts, and in consultation with their communities.
- Identify and create interfaces that facilitate the flow of data to modeling efforts (e.g. carbon assimilation)  - “one size does not fit all”.  Enable seamless ‘hooks’ into data mining and high performance computing environments.
- Evolve from past instrument-focused processing systems to *measurement* oriented data systems within an interoperable framework that will help guide the flow of information and services, improve performance and access, and measurement focused and can be distributed geographically and logically.
- Meet the challenges of transitioning EOSDIS elements to a new DIS paradigm.  What parts do we evolve?  How to evolve these pieces without breakage or disruption?

## The Evolution in ESE Data Management  

***More NASA control***
***in data management***

***Less NASA control***
***in data management***

- The Evolution in ESE Data Management

_**Pre-EOSDIS**_**  ****(before 1994)**
- Data held by science researchers or data centers; data difficult to locate
- Varying storage organization
- Long-term data preservation issues

_**EOSDIS Core System**_**  (1999 to present)**
- Data held by DAACs
- Uniform data management environments
- Distributed data system
- Hard to be “all things to all people”

_**EOSDIS Version 0**_**  (1994 to present)**
- Data held by DAACs, affiliated with researchers
- Heterogeneous data management
- Locate data thru V0 IMS, a federated data system
- Commitment to archive quality

_**ESIP Federation**_**  (1998 to present)**
- Experimental not operational
- Natural clustering
- More interest in individual research than federation evolution

_**SEEDS/REASoN**_

**The intent of the SEEDS study and the REASoN CAN is to focus ESE data system evolution to:**
- increase NASA’s flexibility to adapt the network of data systems & service providers;
- enable access for NASA’s Applications program and its educational programs;
- improve cost effectiveness throughout the data system development and operational life cycle;
- leverage the capabilities, expertise, and lessons learned from existing data systems; and
- assure long-term data stewardship and continuity of services.

## Next Generation Data Processing Architecture

**The Strategic Evolution of ESE Data Systems (SEEDS) study developed a framework for the evolution of data systems and service providers in the 2004 – 2015 timeframe - a move from centralized systems to distributed, heterogeneous systems.**

- SEEDS Study Objectives:
- Ensure timely delivery of ES information at an affordable cost
- Maximize availability and utility of ESE products
- Define a unifying framework for a more distributed network
- Assure continued effectiveness of an increasingly distributed and  heterogeneous network  of systems and services
- Engage the community on
-     data management objectives and solutions

## The Strategic Evolution of ESE Data Systems

***# of Participants & Users ***

**Disparate researchers working independently - difficult to exchange data and information.**

**Emerging interoperability based on a set of minimal standards of interoperability - communication increases.**

**Communities form around common themes or interests - specialized methods for advanced information sharing are developed.**

**Heterogeneous communities are connected via a robust set of standards and interfaces.  Innovation proceeds rapidly as ideas from one community are shared among all.**

***Difficulty to Interoperate ***

**DAACS - tightly coupled data system based on logically centralized, geographically distributed centers.**

**SIPs, ESIPs, NewDISS and RESACs leverage heterogeneous activities and expand processing, archive and distribution capabilities through rapid prototyping.**

**The SEEDS study has concluded with recommendations and a framework for developing the next generation of distributed and heterogenous data systems.**

**PIs collect and store data sets.**

**Computer networking is difficult due to the variety of standards specific to highly technical IT research communities.  Cost of computing infrastructure is high and only available to narrow range of users.  **

**The merging of many networks into one common naming space - the World Wide Web.  HTTP, URL and other protocols are widely adopted. **

***Computing***
***$$***

**Rapid development in internet-based services and functions.  Groups develop custom tools for expanding access to on-line resources.  Ease of use increases the number of users.**

**The REASoN projects add to ESE next generation of data systems, while the evolution of EOSDIS elements continues.**

## Summary of Near Term Data System Challenges

**Missions to Measurements** will require data systems to support data coming from multiple missions and instruments.  The emerging data systems should be capable of rapidly adapting to rolling waves of capabilities derived from new instruments and measurements.
- Foster **discipline/measurement focused science communities** to develop consensus views on important data and data system issues - broad community adoption and buy-in of algorithms, selection of products, sharing of expertise, etc.
**SEEDS** activities in support of data system evolution will produce new opportunities to share and exchange solutions across communities.  Information sharing from among theme communities will assure that data system solutions from one theme area can inform and possibly be adopted by others, including applications.
- Make initial steps in the development of **sensor web technologies**.

## Evolve Earth Science driven data management from missions to measurements to support

- Response to challenges: An Earth Science Knowledge Environment

*Evolve Earth Science driven data management from missions to measurements to support*
* ‘**Data to Outcomes to Impacts’*
* **by infusing science community expertise and knowledge into transparent and seamless data and information systems.*

## A Data System Evolution Prototype:

- NPOESS

***Community Agreed ***
***Standards and Protocols***

_**MAIN PROCESSING SYSTEM**_
*Data from multiple satellite and instrument types*
*In-situ, ancillary, and other data*

- Public
- Archive

- Data

**Flexible**
**Test**
*Community*
*System*
*Processing*
*“**Services”*

**TEAM**
**Collaborative**
*FTP*
*Capabilities:*
*Product Analysis*
*& Review*

***Geographically & Logically Distributed***

***Reconfigurable & Scalable***

- CZCS

- Terra
- MODIS

- Aqua
- MODIS

- NPP
- VIRS

- PI Selected
- through open
- peer review

- MOBY

- Products

- The Ocean Color Measurement Processing System

- SIMBIOS

- SEAWIFS

- Rolling Waves of New Capabilities

-   Processing embedded within the science focus areas
-   Distributed capabilities  capturing “rolling waves”
-   Community, participation, consensus and community services
-   Move processing capabilities closer to the science teams
-   Engage expertise through peer review selection

## Science Focus Area

**Science Focus Area**

**Reconfigure/**
**Pointing**
**Requests**

**Data **
**Products**

**Data**

**Sensor Pool**

**Sensor Control Systems**

**• ****Other Collaboratories **
**    ****Products**
**• ****Validation/Calibration **
**• ****Computational **
**    ****Resources (Grids, ...)**
**• ****Tailored Products**

**Distributed**
**Long-Term **
**Data**
**Archiving **

## Managing the End-to-End Information Flow of the Future

**Predictions**

**Calibration, Transformation To Characterized Geophysical Parameters**

**Interaction Between**
**Modeling/Forecasting**
**and Observation**
**Systems**

**Interactive**
**Dissemination**

**Multi-platform, multi-parameter, high spatial and temporal resolution, remote & in-situ sensing**

**Advanced Sensors**

**Data  Processing & Analysis**

**Information**
**Synthesis**

**Access to Knowledge**

## The Future of Knowledge and Data Sharing

**Peer**

**Peer**

**Peer**

**Peer**

**DAAC**

**DAAC**

**DAAC**

**DAAC**

**US_Gvt**

**US_Gvt**

**US_Gvt**

**US-Gvt**

**REASoN**

**REASoN**

**REASoN**

**REASoN**

**Peer**

**Peer**

**Foreign_Partner**

**Foreign_Partner**

- Which data, products and
- models are available to study
- the Chesapeake Bay pollution?

***Integration of:***
***  *****• ****DAAC Minimum Levels of Services**
-         **e.g.,  Long-Term Data and Products**
**   ****• ****Peer-to-Peer or Peer-to-****Institutions**
**           ****Prototypes: GCMD; ECHO; Mercury;**
**	****    ****Nepster/Direct Readout; UNITE /ESML**

**Commercial_Entity**

**Commercial_Entity**

**Commercial_Entity**

## Earth System Science Today & Tomorrow: Linking Measurements-Missions-Models To Improve Scientific Impact

- Earth System Science Today & Tomorrow: Linking Measurements-Missions-Models To Improve Scientific Impact

## Towards an ESE Knowledge Environments Vision

**Format Translation**

**High-Accuracy **
**Multi-Sensor ,**
**Multi-Platform Analysis**

**Intelligent Agents**

**Human-Computer **
**Interfaces**

**Collaborative Environments**

**Autonomous Metadata **
**Generation**

**Context Sensitive Adaptive **
**Processing**

**2010**

**2015**

**Time**

**Increasing Capability**

**Real- or Near-Real Time, **
**Transparent, Seamless and Automatic Data Fusion **
**and Knowledge Sharing**

**EOSDIS**

**Now**

**Petabytes of Data**

**  ****Prototypes of Distributed Info Systems**

**Megabytes of Knowledge**

**Distributed and Query-Driven Knowledge Systems **

**Distributed Immersive **
**Visualization**

**Cyberinfrastructure**

**SEEDS**

## Our Vision -- An Integrated Global Observing System

**Optical**
**Crosslink**

**Passive**
**Optical**

**Ka Crosslink**

**In-situ User**
**PC Based GS**

**Comm Gateway**

**Metadata**
**Warehouse**

**Commercial**
**Communication **
**Network**

**Active **
**Optical**

**Data Mining/**
**Date Fusion Sites**

**Ka**

**Optical Crosslink**

**Ka**

**Ka**

**Multiple vantage points, multiple partners**

## Long-term Archive:  NASA’s Policies and Plans

## LTA Policy

- NASA’s agency mandate is research
- ESE mission data products are managed by NASA’s Distributed Active Archive Centers for the life of the missions
- NASA’s partner agencies (NOAA and USGS) have a mandate for operations
- These agencies will provide LTA services for ESE mission data products
- NASA-NOAA MOU on EOS - 1989
- 	NOAA will use its best efforts to...Assume responsibility at a time to be agreed upon for active long-term archiving and appropriate science support activities for atmospheric and oceans data for the EOS program
- NASA-USGS MOU on EOS - 1993
- USGS will fund long-term archive functions. USGS will fund archive and distribution functions, including operations and maintenance costs for EOS and related data more than 3 years old.

## Call to Action from the Science Community

- It is essential that the LTA program preserve key long term data and information including the definitive version of the EOS Level 1 data and derived products...and any other data sets/products needed to interpret them...and any other data sets needed for calibration, and validation... (USGCRP, LTA Workshop Report, 1998).
- A long-term archive should be established and operated in the simplest way possible to meet user needs and program goals.  A long-term archive is not only for today’s generation of users but also for the next generation of scientists and citizens whose needs have yet to be expressed but must be provided for (NRC CES, Issues in the Integration of Research and Operational Satellite Systems for Climate Research, 2000).
- NOAA should begin now to develop and implement the capabilities to preserve in perpetuity the basic satellite measurements, i.e., radiances and brightness temperatures (NRC CES, Ensuring the Climate Record, 2000).

## USGS Perspective

- USGS is responsible for LTA of ESE land data
- USGS’s NSLRSDA
- National Satellite Land Remote Sensing Data Archive at EDC
- Established by legislation in 1992
- Provides archive for MSS, TM, AVHRR (LAC to 1989), Corona, and ETM+
- $5M annual budget
- Archive Advisory Committee with representation from academia, government, industry, NGOs, and international partners

## USGS LTA Status

- Global Land Cover Characteristics	Completed
- Global Land 1-KM AVHRR			FY04
- N. American Landscape Char.		FY04
- Remaining V0 Data Sets*			FY04
- EOS Data Sets*					TBD
- Significant challenges exist...
- in defining the levels of service applied to LTA of NASA data
- in defining the time-table to make the transition to LTA
- and in securing the resources needed to do the job

****A form of archive-in-place is under consideration as a mechanism for USGS LTA of NASA data. An annex to the NASA/USGS MOU for the long-term archive of NASA's land science data that specifically addresses the transfer of “EOSDIS Version 0” data sets has been generated and is signature cycle.***

## Backup Slides

## National Earth Science Data Holdings

- Graph updated May 31, 2003

## Assumptions

- Chart in page 2 shows data generated by NASA, NOAA and USGS in the respective bars, without regard to where they are archived. (e.g., Landsat-7 and ASTER data are included under NASA even though they are archived at USGS’s EROS Data Center)
- Estimates for NPP and GPM volumes are included in NASA bars
- NASA data volumes are shown based on design lifetimes (typically 5 years) for the various satellite missions; generally satellites last significantly longer, and as data are accumulated beyond design lifetimes, the volumes rise faster than shown in later years
- Allocations of data volumes to agencies will change as long-term archives are implemented with NOAA and USGS

## NOAA Strategy and Plans

- NOAA is responsible for ESE climate data
- Discussion on LTA of NASA Earth Science Data in NOAA on-going since about 1993
- NOAA’s CLASS (Comprehensive Large-Array data Stewardship System)
- NOAA initiative for archive, access and distribution to NESDIS data products
- Will provide data services for NPP and NPOESS
- “Statement of Intent” signed by G. Asrar and G. Withee for inclusion of NASA Earth Science LTA within CLASS (see back-ups)
- A joint NASA-NOAA team developed a draft Implementation Plan as called for in the Statement of Intent
- A joint NASA/NOAA team is working to define the interface and levels of service requirements that will enable a test of transfer of MODIS low level (Level 0/1)data from NASA’s Goddard DAAC to CLASS to occur in FY03

## “Statement of Intent” signed July 3, 2001