# Presentation: 697234

## MODIS Distributed Active Archive Centers (DAACs)Level 1 Processing/ReprocessingLevel 1+ Archive and Distribution

- Presentation to MODIS Science Team
- December 19, 2001
- Steve Kempler (GES DAAC)
- 301-614-5765
- steven.kempler@gsfc.nasa.gov
- Greg Scharfen (NSIDC DAAC)
- John Dwyer (EDC DAAC)

## Level 1 Processing/Reprocessing: Highlights Summary

**Level 1 Processing/Reprocessing: Highlights Summary**

- > 280 consecutive full days of Level 1 Collection 1 data processed
- > 315 full days of Level 1 Collection 3 data processed
- - Full day means > 98% of data per day processed
- - Exception: Lost Level 0 data (i.e., data the DAAC did not see)
- L1 Production has been upgraded to process at 3X, seeing bursts of over 4X
- ECS Release 6A04 set us back in processing during the summer, but we have recovered well and have taken action to reduce future new release issues
- Integration of GES DAAC developed, Simple Scalable Script-Based Science Processor (S4P) into ECS demonstrates more efficient reprocessing, and now, processing
- Weekly meetings with ESDIS have lead to identifying and resolving system instabilities over time
- Weekly meeting with MODIS Technical Team address production issues that ensure data processing that maximizes data output
- Daily meeting with MODAPS to discuss processing status and plans

*Several data production metrics tools have been developed by the DAAC*
*   ...**allowing system behavior to become better understood*
*	**     ...**in turn, allowing production strategies to be reliably implemented that maximize output*
* **	**          ...**based on priorities*

## Actual and Currently Planned Increase in Processing Capacities

|  | 12/1999 | 12/2000 | 2/2001* | 1/2002* | 12/2003
** |
| --- | --- | --- | --- | --- | --- |
| Terra processing | 1.2X | 1.2X | 1.2X | 1.2X | 1.2X |
| Terra reprocessing |  | 1X | 2X | 2X | 2X |
| Aqua processing |  |  |  | 1X | 1X |
| Aqua reprocessing |  |  |  |  | 1X |
| Total Processing Capacity | 1.2X | 2.2X | 3.2X | 4.2X | 5.2X |

- *   Additional 1X installed earlier in the year than originally planned
- ** Additional 1X may be installed earlier in the year than planned, depending when AQUA is launched
- At the GES DAAC, a rate of 1X equals processing a day of MODIS Level 1 data, in a day.

## GES DAAC total performance for production, archival and ingest

- 326

- 330

**X-Rate**

**(based on 1 day's amount of processing)**

- S4PM Reprocessing

- S4PM Forward

- PDPS

- S4PM Reprocessing on (SPG01)

- S4PM

- Forward

- Installation

- Bldg 32

- power outage

- 6A.04 Installation

- Focus on

- improving

- system

- performance with

- S4PM

- Focus on

- improving system

- performance with

- PDPS

- Reprocessing

- disk upgrade

- DOYs 254-255 & 261 shut down

- production to increase push of

- reprocessing data to MODAPS

- Installation

- of numerous

- system

- patches

- Multiple system problems

- Installation of

- numerous hardware

- upgrades

- Statistics up to November 27 (331)

- Multiple

- system

- problem

- S4PM

- (SPG01)

- Installation

## GDAAC Level 1 Production of MODIS Data Daily Average Volume by Month of Production 

**GDAAC Level 1 Production of MODIS Data Daily Average Volume by Month of Production **
**February 2000 to November 2001**

## Slide 6

## Availability of MODIS Level 2  Oceans Sea Surface Temperature as of November 29, 2001 3:00PM

**Availability of MODIS Level 2  Oceans Sea Surface Temperature as of November 29, 2001 3:00PM**

## Availability of MODIS Level 2 Atmospheres Cloud Products 

**Availability of MODIS Level 2 Atmospheres Cloud Products **
**as of November 29, 2001 3:00PM**

## >569.4

**GSFC**

**EDC**

**NSIDC**

**TOTAL**

**TB/month**
**sustained**

**Total (TB), **
**so far**

*** Last three months averaged **

**43.1**

**15.2**

**  ****0.9**

** *******

**59.2**

**445.9**

**116.1**

**7.4**

**>569.4**

## Slide 10

**Notes:**
From Gary Alcott

## Slide 11

## Daily Average Gigabytes of MODIS Data Distributed each Month by the NSIDC DAAC.  

- Daily Average Gigabytes of MODIS Data Distributed each Month by the NSIDC DAAC.

## Slide 13

## Distribution Issues:  What we are doing

- Enhancing Version 0 Web-Based Hierarchical Ordering Mechanism (WHOM) as an alternative data -ordering engine. (**PLEASE USE THESE: YOUR FEEDBACK HELPS ENSURE THAT THESE TOOLS ARE THE CORRECT  ONES**)
- Added subsetting (parameter, spatial, temporal)
-  Added on-demand subsetting
-  Output of above tools are in HDF and binary formats
- Developed subsampling PGE, also providing HDF and binary fromat outputs
- Developing in-house reprojection tools (EDC) and mosaicing enhancements (NSIDC)
- Pointing to tools developed by the MODIS Atmospheric Team including Granule Locator and GranuleVisualizer.
- Adding pointers to the GES DAAC wepage to navigate users to tools, status, and help
- Provided browse products
- Developed a data mining environment by which  science algorithms that reduce data volume before transmission can be integrated.
- Implementing 'data pools’, very large data staging archives, that will greatly enhance the amount of data that can be distributed in a given time period.
- Linking to research initiatives (e.g., NASA Cold Land Processes at NSIDC) and field campaigns
- Collaborating with universities by providing them data so that they can act as redistribution sites.

## Distribution Issues:  Digging In

- The GES DAAC has initiated a complete data ordering to data receipt analysis:
- 1.  Engage MODIS data users (initially, MODIS Team members and GES DAAC User Working Group members) to categorize data access/usage patterns and problems
- 2.  Analyze and identify, from data ordering to data receipt, the operational and functional mechanics and bottlenecks of the system, based on above data access/usage categories
- 3.  Prioritize (will need user input) and implement changes needed to ensure that the system allows for data access/usage at an acceptable level:
- 	- The GES DAAC will coordinate and resolve operational bottlenecks
- 	- ESDIS Project will coordinate and resolve functional bottlenecks
*This activity requires full commitment from the GES DAAC, users, and ESDIS Project to efficiently identify and remedy distribution issues*

## Use Access-Related Issues(Contributions from all 3 DAACs)

- Need high level graphical view of archive holdings (temporal and/or spatial)
- Need to formally support alternative search and order tools at all DAACs
- Need temporal, spatial and parameter subsetting capability as part of EDG
- Need more HDF-EOS tutorials and tailored display and manipulation tools and offer other formats or conversion to other formats (e.g., Geotiff, flat binary)
- Need to address data management of near-real time (bent-pipe) data
- Need to implement routine data reconciliation procedures
- Need to minimize impact of DAAC system downtime
- Need to automatically import projection information into COTS packages.
- Need to support subscription services
- Need to describe specific differences between Collection 1 and Collection 3
- Need to provide global datasets conveniently
- Need to translate QC flags (no tools)
- Need to provide global coverage of the 250m data.
- Need to match filenames as delivered with granule-id that was ordered.
- Users like the Global Browse websites, but not all that is shown has always been archived yet (EDC)

- G. Scharfen, 19 Dec 01

## Ways to Access MODIS Data   
EOS Data Gateway (EDG)
Access directly or via DAAC Home Pages through Links button
Data is sent by tape or pulled by requestor
DAAC-specific interfaces
GSFC: Terra-WHOM
Anonymous FTP
Access via DAAC Home Page through On-line FTP button
Subscription 
Data automatically pushed to requestor

- EOS Data Gateway (EDG)
- Access directly or via DAAC Home Pages through Links button
- Data is sent by tape or pulled by requestor
- DAAC-specific interfaces
- GSFC: Terra-WHOM
- Anonymous FTP
- Access via DAAC Home Page through On-line FTP button
- Subscription
- Data automatically pushed to requestor

- MODIS Data Access

## DAAC User Services

- GSFC:
- MODIS Data Support Team.  Lead: Greg Leptoukh 301-614-5253
- User Services Group:
- 301-614- 5473 (ECS) or 301-614-5224 (V0)
- daac_usg@gsfcsrvr4.gsfcmo.ecs.nasa.gov (ECS) daacuso@daac.gsfc.nasa.gov
- http://daac.gsfc.nasa.gov
- NSIDC
- MODIS Snow and Ice Product Team.  Lead: Greg Scharfen 303-492-6197
- ECS Science Outreach Coordinator: Siri Jodha Singh Khalsa 303-492-1445
- User Services:
- HELP DESK - 303-492-6199
- nsidc@kryos.colorado.edu
- http://nsidc.org/NASA/MODIS/
- EDC DAAC
- User Services MODIS Data Specialist:  Carolyn Gacke  605 -594-6822
- ECS MODIS Science Data Specialist:  Calli Jenkerson  605 -594-2638
- User Service:
- Brenda Jones:  605 -594-6503; Toll Free  866 -573-3222 (866 LPE DAAC)
- edc@eos.nasa.gov
- http://edcdaac.usgs.gov

## DAAC user support for MODIS products(slide provided by NSIDC, adapted for all DAACs)

*Team approach*: the MODIS Product Teams includes representatives from several "resource groups" including operations, writers, user services, web technicians, systems engineering, programming and science groups
- MODIS *web pages* include:
- documentation (metadata, catalog pages, guide documents)
- access to browse
- access to the EDG, WHOM
- user tutorials: using the EDG, converting HDF-EOS data (more coming)
- data manipulation and display tools (more coming)
*user services* responds to questions within 24hours
*distribution* by ftp (coming very soon:CD, DVD, DLT, 8mm)
*outreach: *e.g. Conferences, papers, brochure and user surveys

- G. Scharfen, 19Dec 01

## BACKUP

## DAACs

- L0
- Processing
- (ESDIS/
- EDOS)

**GES DAAC**

- Ingest, Archive
- & MODIS L1
- Processing &
- Reprocessing
- (ESDIS/ECS)

- Ingest
- DAO Data
- (GES DAAC)

**DAO**

**NOAA **
**& FDD**

**V0, V1**

**MODAPS**

- L2+
- Processing &
- Reprocessing
- (SDST)

**MODIS Calibration**
**Science Team (MCST) and supporting scientists for Level 1 data**

**L2+**
**Validation **
**at the Science Computing Facilities (SCFs)**

**GES DAAC**

**NSIDC DAAC**

**EDC DAAC**

- Archive &
- Distribute L1+
- (ESDIS/ECS)
-  and DAO,
- ancillary data
- (GES DAAC)

- Archive &
- Distribute
- Land Data
- (ESDIS/ECS)

- Archive &
- Distribute
- Snow & Ice
- (ESDIS/ECS)

**Science**
**Appli-**
**cations**
**Educ-**
**ation**

**DAACs**

- DAO
- Data

- L0
- Data

- Ancillary
- Data

- L2+
- Data

- L1
- Data

- QA
- Metadata
- Update

- QA
- Metadata
- Update

- L1 Data

- DAO,
- Anciillary Data

- L2+
- Data

- DATA!

**LaRC**
- For CERES Processing

- L2
- Data

- MODIS Data Flow
- (Development Team in parentheses)

## Slide 22

## Slide 23