# Presentation: 319626

## ECHO: Foundational Middleware for a Science Cyberinfrastructure

Robin Pfister (robin.g.pfister@nasa.gov)
Keith Wichmann (wichmann@gst.com),
Beth Weinstein (beth.weinstein@nasa.gov)

MODIS Science Team Meeting
March 22, 2005

**Robin Pfister (robin.g.pfister@nasa.gov)**
**Keith Wichmann (wichmann@gst.com),**
**Beth Weinstein (beth.weinstein@nasa.gov)**
**MODIS Science Team Meeting**
**March 22, 2005**

**Notes:**
ECHO is a middleware that has currently been applied to the Earth science domain to help form a science cyberinfrastructure.

## ECHO is...

**Is an Enabling Framework that allows interoperability among diverse and distributed, data, service and client systems**
**Is a metadata clearinghouse and order broker.  In the near-future will also be a granule-level service broker**
**Is an Open System: Enabling a Collaborative Community**

**Notes:**
When we originally started defining the concept, our objective was to come up with a solution to address data access issues that existed at the time. 
Issues included:
A capability gap between what users wanted and the system provided at that time
 And, Inflexibility in data access options.
The result was an Enabling Framework that allows interoperability among diverse, distributed systems allowing for sharing of data, services and clients by the community.
So, The term ECHO is more than just an acronym. It represents the functionality of the system in that it echo’s systems to other systems. It currently does this through data and service registries and an order broker.
In Early ‘06 it will have a granule-level service broker that will broker the exchange of information between data and service systems to achieve virtual seamlessness in order to streamline data access for science users.
As an open system ECHO enables a collaborative community.

## Background

**Users of Earth Science Data were confined to a single user interface built as a one-size-fits-all solution**
**Search and retrieval data access paradigm, searches were executed at the archives. Many factors contributed to slow or non-performance**
**Scientists spent a significant amount of time performing data access activities**
**NASA was paying **a single contractor** to develop **all** needed functionality into a single system**

**Notes:**
Prior to ECHO...
As you may recall, users were confined to a single user interface that was fixed to the interoperability middleware so that no alternate user interfaces could be supported.
It provided single point access to over one and a half thousand very unique datasets via a one-size-fits all solution for cross-DAAC access.

The search and retrieval paradigm was the only option based on the architecture. No other paradigms were supported.
In this paradigm, searches are executed at the provider site and users were dependent on the slowest responding archive.  
Due to this and other factors, overall performance was slow. 
Also, this paradigm requires the users to know something about the content of the archives to form an intelligent query. 
It was common for users to get a results set of either zero, or thousands of granules – too many to filter through.

## Background

**Individuals and communities can share data, services and tools.**
**Scientists save time by using tools tailored for their data access needs, cutting out unnecessary steps in the data access process**
**Tools can be developed by anyone, ***saving NASA funds*** and ***increasing quality* **through more competition**

**ECHO**

**Data & Service **
**Providers - Any**

- Orders

- Collection
- & Inventory
- Catalog +
- Sample
- Images

- Service &
- Order
- Broker

- Provider APIs

- Client APIs

- WWW Browsers

- Data
- Holdings

**EDG & Other Clients:**
**Machine - Machine &**
**Human - Machine**

- Search,
- Browse &
- Order
- Requests

**Notes:**
In moving to ECHO we decided to architecturally separate the notion of Clients from the interoperability middleware.  We took the opportunity to take a fresh look at the middleware capabilities and develop those that were more in line with what the user community wanted.

So, With ECHO....

Anyone can share data, services and tools extending available earth science resources from the science community members to other science community members.

This enables a truly collaborative system with open exchange.

Scientists save time by using tools tailored for their needs cutting out unnecessary steps in the data access process

Tools can be developed by anyone. This leverages resources and efforts of other groups.

## ECHO Context

**ECHO acts as middleware between resource (data or service) partners and client partners.**
**Clearinghouse of spatial, temporal, numeric and textual Earth Science metadata**
**Order Broker**
**User and provider account service**
**Services clearinghouse and broker (future)**
**Capabilities are in line with (and expected to be driven by) GEOSS.**
**Data Partners provide information about their data holdings, and client partners develop software (either machine-machine or human-machine) to access this information.**
**End users who want to search ECHO's metadata must use one of the ECHO clients.**

**Notes:**
The idea is to provide a mechanism for the science community to share their resources and to control the view and access to those as they wish -- Putting the power of the system into the hands of the community.

Our job is to focus on the underlying fabric of non-science services to allow for that exchange to happen. And I refer to this fabric as Cyber-infrastructure.

This approach is in line with (and in the future will be driven by) the momentum of the Global Earth Observation System of Systems (or GEOSS).

## ECHO Context

**Client**
**Extensibility**

**Data**
**Extensibility**

**Service Extensibility**

**Graphical User Interfaces (applets, active pages, etc.)**
**Machine-to-Machine Interfaces**

**New Data Partners**
**- ****Can establish policies for their own data**
**New Collections/Data Types**
**Access Mechanisms**

**Types of services:**
- Data Services: e.g.
-   collection, subsetting
-  Search Services: e.g.
-   gazetteer, thesaurus (future)
-  Administrative Services:
-   e.g. accounting (future)

**Service interactions:**
-  Order Options
-  Advertised
-  Brokered (future- ECHO 8.0)
- - Orchestration/Chaining (future)

**Service Views:**
- - Service View
- - Data View

**Notes:**
ECHO presents an open messaging framework based on XML and SOAP

Data Partners  are organizations that gather, generate, process and/or distribute Earth Science Data.  The DAACs are primary examples but we envision that someone with field data on their desktop would also be a provider if they choose to share their data.

Client Partners are organizations that develop software that uses ECHO’s APIs to interact with Earth Science metadata on behalf of end users.  These clients can be general purpose or very specialized; they can be human-machine or machine to machine; and they can follow a variety of data access paradigms - not just search and retrieval.

Service Partners are organizations that develop software or services that operate on data, (and in the future, that provide client component functions)  that are advertised in ECHO to provide value-added capabilities that can be offered through the clients. In the future they will be brokered to achieve a virtual seamlessness.  

Application Developers - are a combination of all the above. They develop a targeted solution which leverages ECHO as middleware to access community resources, to contribute their output products and their services for access back to themselves (and to others if desired).

So, ECHO is NOT a GUI or Client, but an enabling framework.  

If everything works properly ECHO is completely invisible to the end user.

## Why use ECHO?

**The multi-organizational content of ECHO provides a broadened set of resources to a growing number of Earth science applications and interdisciplinary research efforts. **
**The more resources provided, the richer the set of available resources.**
**ECHO streamlines access to digital data and materials and brokers orders and other services from Clients to Data Partners.**
**More data access paradigms and approaches are supported.**
**Human-Machine and Machine to Machine**
**“****Navigation and discovery” and “Search and retrieval”**
**Subscriptions**
**Virtual seamless access from the end user’s perspective.**
**Because scientists will have access to clients tailored to their needs, valuable time can be saved and refocused on research.**

**Notes:**
So why would anyone want to use ECHO?

The rich set of resources that can be offered for a variety of purposes is certainly a key value.

Streamlined and tailored access to data and services - for example if you only want to query on 5 pieces of metadata out of the 300 available, you can build a client that shows you that simplified view. 

Virtual seamlessness

All this serves to save time in data access so scientists can refocus that time on their primary job of research.

....(more on next page)

## Why use ECHO? (cont.)

**ECHO puts the control of the resources in the hands of the providers of those resources. **
**ECHO has tools for providers to mange and control access to their contributed resources.**
**Establish the best spatial search approach (details in back-up slides) for their data (orbital search, geodetic, or Cartesian)**
**Improved availability over previous architecture (details in back-up)**
**ECHO provides tracking services for both the Data and  Client Partner. **
**In the future, new technologies can be infused using the services APIs.**
**Back-up slides also illustrate:**
** ****Recent performance improvement. Performance improvement is an ongoing effort (more details on availability and performance improvements in back-up slides)**

**Notes:**
ECHO puts the control of the resources in the hands of the providers of those resources. 

Providers can manage and control access to their metadata and data

Providers can specify the spatial search mechanism that best suits their data. This will maximize accuracy in the results sets.

ECHO provides improved availability

Also provides tracking services.

And when service brokering and chaining are available, ECHO will also be a mechanism for technology infusion.

There are sev eral slides in the Backup slide section of my presentaiton that describe more detail on some of these points
And of course I’ll provide this slide set to post on your website for later use.

## Current State of ECHO

**Current Operational Version: 5.5.3**
**Version 6.0 is being externally tested**
**Version 7.0 is ready for external test**
**Version 8.0 development is getting started**
**Information on future releases ECHO Web site.**

**Notes:**
ECHO is iteratively developed and currently provides core capabilities but the opportunity for additional capability remains.

You can affect the capability of ECHO through participation in the ECHO Technical Committee that meets by telecon every week.

## Client API Improvements

**Client Workshop (March 10-11) kicked off 2005 as the "year of the client". **
**Prioritize API areas for development of the Client Libraries and the Reference client.  **
**Reference Client will provide a view with 'best practices' code that can be copied and pasted into other applications.  **
**The workshop also encouraged component sharing among the existing client providers. **
**Additionally, We've repackaged the complete set of toolkit source code, java documentation, and libraries.   **
**This repackaging fills in all the remaining "holes" in the toolkit library**
**Included is a sample demonstration client- this demo client was always there - it is updated to reflect the 6.x changes.**

**Notes:**
Over the past year the operations team was focused on getting data represented into ECHO. Now that we have over 40 million granules represented we have dubbed This year as the Year of the Client.

Our client workshop a couple of weeks ago kicked off this effort that will 
serve to improve client API usability and 
encourage re-use of  UI components among client developers.  

For example, how many shopping carts do you really need.  If someone develops one and it works well, then why not re-use it and save some money.

## ECHO Holdings Report

**Operational Holdings Summary as of 3/1/2005**

|  | Collections | Granules | Browse |
| --- | --- | --- | --- |
| Public | 1233
 | 35,178,394
 | 5,308,581 |
| Restricted | 176 | 6,483,460 | 2,047,530
 |
| Total | 1,409 | 41,661,854 | 7,356,111
 |

**Notes:**
This is the current data holdings in ECHO.

The public holdings are those that are visible to the public.

The restricted holdings are those that are restricted for viewing and ordering.

The data provider has control over who can see their metadata, and who can order  their data. We provide tools that allow data providers to set these kinds of access controls.

## ECHO Data Partners

**ECHO Data Partners**

**Operational**
**ORNL DAAC (Oak Ridge National Laboratory Distributed Active Archive Center) **
**SEDAC (Socioeconomic Data and Applications Center)**
**LP DAAC (Land Processes EOS Core System (ECS) DAAC) **
**GES DAAC (GSFC Earth Sciences ECS DAAC)**
**ASF DAAC (Alaska SAR Facility DAAC)**
**In Test**
**LaRC DAAC (Langley Research Center ECS DAAC)**
**NSIDC DAAC (National Snow and Ice Data Center ECS DAAC)**
**In Development**
**PO.DAAC (Physical Oceanography DAAC)**
**SSC SDP (NASA Stennis Space Center Science Data Purchase)**
**GES DAAC V0 (GSFC Earth Sciences Version 0 DAAC)**

**Notes:**
We have 5 operational data partners - their data are on the operational system and viewable by the public

We have 2 in Test - their data are on the operational system and not viewable by the public, but will be soon

We have 3 in Development – These partners are still working to resolve metadata issues, working towards historical load on the operational system, most of these partners are still on the test system (but some are on the operational system)

## ECHO Client Partners

**Operational**
- Mercury EOS** – ORNL client **
- Power User Interface** – script for bulk ordering**
- SIMECC **(Simple MODIS ECHO Client) – MODIS data search and order**
**In Test**
- WIST** –  EOS Data Gateway using ECHO **
- ASF ECHO Client - **Alaska SAR Facility client**
- WISRD** (Web Interface for Searching, Subsetting, Stitching, Resampling, Regriding, and Reformating Data)  – NSIDC Client to search for swath, scene, and gridded data**
- MODIS Website** – Client via existing MODIS website and ORNL’s shopping cart**
- Data Validation User Interface** – Client for MODIS Land Data Validation Team **
**Under development**** **
- Annoterra Version2 ** – Link between EOS newsfeeds, GCMD, and ECHO granules**
- SNOWI-E** – NSIDC client**
- NEO** - NASA Earth Observations client **
- Invasive Species Data Service
- OGC/NSDI Client Adaptor - will allow any OGC/NSDI compliant client to plug in. Earth-Sun Gateway (ESG) is an example )
**There are plans for other clients (MODIS rapid response, Adaptive Sensor Fleet)**

**Notes:**
We have 3 operational client partners

5 in Test

5 more in development

And a few that are still in planning stages.

## Client Partner Support

**The ECHO Operations Team (ECHO Ops) is the point of contact for direct interaction between ECHO, its Partners, and end users**
**ECHO Ops is responsible for the operation and maintenance of the ECHO operational and partner test systems**
**ECHO Ops general support for Partners includes:**
**Assisting Partners in understanding the ECHO API and DTDs**
**Ingest management and accounting - weekly summaries and metrics on all ingest jobs**
**Problem tracking and resolution**
**Advertising Data Partner holdings and availability of new datasets**
**Promoting and engaging new Client Partners**
**Reference client in progress (best practices code reuse)**
**Improve website content to provide “front office” for client developer support**
**Improve overview materials**
**Create sense of thriving project and community**
**Reorganize developer reference materials into developer center**
**Develop multilevel documentation to ease use of APIs**

**Notes:**
The ECHO Operations Team is available to help partners get started using ECHO and can help make connections between partners for sharing of ideas and code. 

They’re responsible for the operation and maintenance of ECHO.

They provide general support to help partners in using APIs, problem tracking and resolution, promoting interaction among the partners for re-use and many other things.

They’re building a reference client that will serve as a ‘best practices’ example where the code can be lifted and re-used by other client developers.

They maintain a website that serves as their ‘front office’ for support

They also develop documentation that organizes views of the APIs so they’re simpler to understand.

## Contact Information

**ECHO Ops **
_**[echo@killians](mailto:echo@killians.gsfc.nasa.gov)**__**[.](mailto:echo@killians.gsfc.nasa.gov)**__**[gsfc](mailto:echo@killians.gsfc.nasa.gov)**__**[.](mailto:echo@killians.gsfc.nasa.gov)**__**[nasa](mailto:echo@killians.gsfc.nasa.gov)**__**[.](mailto:echo@killians.gsfc.nasa.gov)**__**[gov](mailto:echo@killians.gsfc.nasa.gov)**_
**301-867-2071 **
**ECHO website**
_**[http:](http://eos.nasa.gov/echo)**__**[//eos](http://eos.nasa.gov/echo)**__**[.](http://eos.nasa.gov/echo)**__**[nasa](http://eos.nasa.gov/echo)**__**[.](http://eos.nasa.gov/echo)**__**[gov/echo](http://eos.nasa.gov/echo)**_
**Holdings Summary, Upcoming Functionality, APIs and DTDs**
**Real-time status**
**ECHO mailing lists**
**echo-all, echo-status, echo-client, echo-data**
**ECHO Technical Committee (ETC) Meetings**
**Tuesday @ 3:00pm ET**
**Contact ECHO Ops for dial-in information**

**Notes:**
Here is contact information and some mailing lists.

echo-all: General mail list for use by the extended ECHO community. 
echo-status: List for members who want to receive notification of system downtime and failure alerts. 
echo-client: Forum for client partners and our Dev and Ops teams to discuss client related issues. 
echo-data: Forum for data partners and  our Dev and Ops teams to discuss data related issues.
.... (next slide DVUI intro) 

## DVUI Demonstration

**Data Validation User Interface Proof-of-concept**
**Illustrates ECHO can support very different search and access paradigms from EDG**
**Resulting from this effort were improvements to ECHO that are being released in upcoming iterations**
**This is an example of something that couldn’t be done in the old heritage architecture**
**Potential re-users of this concept (or code) need to make sure that the data they need for their application are made visible by the providers of those datasets to the potential users of the client (this was a major reason DVUI wasn’t able to go operational before)**

**Notes:**
Now Keith will demonstrate the proof of concept that Beth Weinstein (who is now head of the operations group) developed for her project as part of Goddard’s Professional Intern Program.  
The DVUI illustrates a different data access paradigm – navigation and discovery. 
This particular implementation is based on a dynamic queries approach, where the user navigates metadata dynamically through slider bars and visual displays.  
This shows the flexibility enabled by ECHO

MOREOVER For this application, users needed to dynamically view 3 years worth of data from 37 datasets over 31 different data validation sites. And this view needed to be refreshed daily. Because of the various limitations in the pre-ECHO system, It was estimated that a single instance (not considering refresh) of the 3 x 37 x 31 results set would take 45 person days to generate.  There was no way to meet the user requirement with the old architecture.  ECHO not only enables this query, it enables daily updates so this community of users can access up-to-date data on a daily basis.

DVUI shows the value of having a metadata clearinghouse like ECHO to serve client views.

(....demo.....)

(post demo....)    Our experience in building this proof-of-concept also shows the importance of the client community and end user community to work with and encourage the data providers to populate the appropriate datasets into ECHO so that these kinds of systems can actually go operational as intended.

## Backup

## DVUI Proof-of-Concept Client

## Architecture Overview: System Drivers 

**Design for Change - with ever-changing technology, science needs and policies we needed to build a highly-evolvable system**
**Present an open API for organizations to connect their own data or service or to build user interfaces for specialized views of these resources**
**Make it easy for providers of Earth Science data and services to participate in the system**
**Provide searches that respond quickly**
**Broker orders for both data and services**
**Minimize operational costs**
**Build upon advances in industry and use e-commerce systems as a model**
**Build a system that can be scaled up to handle large numbers of requests**

## Access Control Lists

**ECHO supports allowing the Data Partner to restrict visibility of their Collection or Granule metadata, as well as the ability to order**
**Since ECHO contains 10s of millions of records, granules to be access controlled are determined based on a set of rules**
**Fixed time rules (i.e. hide data acquired between September 11th, 2001 and September 30th, 2001)**
**Relative time rules (i.e. prevent orders for data for 30 days after it is produced)**
**Restriction flag setting**
**Specific collection or granule**
**Access controls are evaluated at the last possible moment so that changes are taken into account**
**Providers block access to everyone, and then grant access to groups**
**ECHO provides a separate function for managing what users are in a group**
**PUMP (Provider User Management Program) is a GUI tool for managing access controls and other Data Partner related functions**

## About Spatial Data

**ECHO leverages Oracle’s spatial capabilities to perform spatial search and to represent Partner metadata**
**Both collections and granules can be spatially represented**
**If a collection or granule represents the entire Earth, there is no need to spatially index it **
**ECHO is asking Providers to set a spatial keyword to “Global” in order to indicate this condition**
**Version 5.5 will have a special Global flag for the Partner to set**
**Oracle 9i version 2**

## Cartesian, Geodetic and Orbit Models

**Cartesian Model**
**The Earth is represented by a rectangle (-180, -90, 180, 90)**
**Spatial objects cannot cross the International Date Line**
**Spatial objects cannot cross the poles**
**Polygons can be as large as the whole Earth**
**Geodetic Model**
**The Earth is represented by a spheroid**
**Spatial objects may cross the International Date Line**
**Spatial objects may cross the poles**
**No single polygon may have an area equal to or larger than half of the Earth’s area**
**Orbit Model**

## Cartesian View of Sample Polygon

## Geodetic View of Sample Polygon

## Services Provided For Searching

**Spatial Search**
**Cartesian**
**Geodetic**
**Orbit-based**
**Temporal Range Searches**
**Keyword Searches**
**Numeric Searches**
**Product Specific Attribute Searches**
**Extensibility to support varying, specialized inventory search capabilities. E.g  spatial models**

## Improved Availability: Old Architecture 

- 200TB

- 8TB

- 100TB

- 1PB

- 1GB

- Net Result: 200 terabytes of data not searched.
- Search time dependent on lowest common denominator.

- Partner down for maintenance, network not available, etc.

**Data**
** ****Partner**

**Data**
** ****Partner**

**Data**
** ****Partner**

**Data**
** ****Partner**

**Data**
** ****Partner**

**EDG**

## Improved Availability: ECHO Architecture

- Partner down for maintenance, network not available, etc.

- 200TB
- (35MB New)

- 8TB
- (2MB New)

- 100TB
- (0 New)

- 1PB
- (670MB New)

- 1GB
- (100MB New)

**Net Result**: 35 megabytes of data not searched.
- Search time dependent on ECHO only.

**ECHO**

**Client**

**EDG**

**Client**

**Data**
**Partner**

**Data**
**Partner**

**Data**
**Partner**

**Data**
**Partner**

**Data**
**Partner**

## Online Access or Order

**Some Data Partners make the actual data available online.  In this case, the OnlineAccessURL is included in the query result, and users can access the data without going through the order process provided in Order Entry Service. **
**If the data is not online, the Order Entry Service allows users to create, modify, quote, and submit orders.**
**The user can also check the status of the order if it is updated by the Data Partner.**
**One order consists of one or more provider orders.  Each provider order represents the catalog items that a user wants from a particular Data Partner.**
**Each provider order consists of one or more order line items, with each line item describing the catalog item that is being requested including any requested options for that item.**
**Catalog Items in ECHO are either granules or collections**
**Only items that are orderable are Catalog Items**

## Subscription Service

**The subscription service automatically keeps users or clients informed of updated metadata**
**Don’t have to issue a query every night**
**Only updated metadata is sent**
**The user can subscribe to the following options**
**Collection metadata updates of one or all collections **
**Granule metadata updates of a collection**
**Both collection metadata and granule metadata updates of a collection**
**The metadata can be delivered by either FTP or e-mail**
**Must be a registered user to use this service**
**Email address of user is used to report errors**
**The subscription service honors the Access Control Lists by only delivering the data that the registered user has access to**

## Query Performance

**Query Performance**

- 			_**5.5.2**_**	**_**5.5.3**_
**GranuleUR**
**# of Runs****	****0****447****	****0****13**
**Min:****	****0****120****	****152**
**Max: ****	****0****439****	****205 **
**Mean:****	****0****185****	****178 **
**DataSetID, Browse Only**
**# of Runs****	****0****350****	****0****13**
**Min:****	****00****45 ****	****0****59**
**Max: ****	****0****378****	****108**
**Mean:****	****00****83****	****0****77**
**DataSetID, Temporal**
**# of Runs****	****0****440****	****0****10**
**Min:****	****00****40****	****0****71**
**Max: ****	****0****447****	****176**
**Mean:****	****00****95****	****105**
**DataSetID, Temporal, Spatial**
**# of Runs****	****0****173****	****0****11**
**Min:****	****0****311****	****0****15**
**Max: ****	****3575****	****0****98**
**Mean:****	****1810****	****0****40**

**Notes:**
5.5.3 released into operations March 3 
Time metrics are reported in seconds.
DataCenterID (e.g. LPDAAC_ECS) is specified in all of the benchmark queries listed above. In addition, queries that reference DataSetID are performed on a LARGE dataset (more than 200K granules in dataset) from a Data Center with large catalog (more than 10 Million total granules).
Version 5.5.2 metrics collected over 30 days (2/1/05 – 3/1/05), with approx. 100 data points collected for each benchmark query.
Version 5.5.3 metrics collected during testing on a system configured to closely resemble the operational system; approx. 30 data points were collected for each benchmark query over a 5 day period.
5.5.3 introduces a drastic improvement in spatial query performance: It only applies when the Collection is specified in the query
Similar approach to what was done for spatial will be applied to other tables: Additional attributes, Granules

## System Availability

**12/1/2004 - 3/1/2005**
**Operational System Uptime**
**Overall average:****	****      **98.5**%**
**December 2004:****	****      96.4%**
**January 2005:****	****      99.6% **
**February 2005:****	****      99.4%**
**Partner Test System Uptime**
**Overall average:****	****      **99.8**%**
**December 2004:****	****      99.8%**
**January 2005:****	****      99.8% **
**February 2005:****	****      99.7%**

**Notes:**
In the last 6 months, we have recorded Operational system availability above our stated goal (>98%) in 5 of 6 months.
However, periodically there are GSFC facilities issues, ECHO hardware, and system maintenance issues that can create problems in providing high availability.
Redundant fail-over ECHO (end of year)

## WIST

**Warehouse Inventory Search Tool – general ECHO client based on EDG GUI**
**WIST 0.2**
**ECHO 6.0 compatible**
**June 1, 2005 **
**WIST 0.3**
**ECHO 7.0 compatible**
**Sept 1, 2005**
**Draft of EDG to WIST transition plan**
**Distributed to the DAAC Managers this week**
**Telecons **