# Presentation: 696760

## MODAPS Reprocessing Collections 4 and 5

- Ed Masuoka
- July 13, 2004

## MODIS Processing Systems

**EDOS**
- L0
- Processing
- (ESDIS/
- EDOS)

**DAO**

**GES DAAC**

- Level 1
- Processing:
- Calibrated
- Radiance
- & Geographic
- Location
- (ESDIS/ECS)

**MODIS Team**
- Q/A ,Validation and Science

**User**
**Community**

- DAO
- Data

- L0

- MODIS
- Products

- Update Product
- Quality Metadata

-   Level 1 Current day

- Ancillary
- Data

- Level 2 &
- Level 3
- Science
- Products

- DATA PRODUCTS

- Ground
- Stations
- White
- Sands
- Fairbanks
- Svaalbard

**MODAPS**

- Level 2,3
- Processing:
- Higher
- Level
- Science
- Production
- (MODIS Team)

- 34,300 files

- <1GB

- Ancillary  Data

- Reprocessed.

- 68,600 files

- Current

- 24
- files

- Near Real
- Time Processing
- System for
- MODIS and
- AIRS (NOAA)

- MODIS
- Rapid
- Response

- Ocean Color
- Data Processing
- System

- Ocean Color
- community

- Applications
- Community,
- Public
- and Media

- Weather,
- Fish/Wildlife...

- L1AOS

**Disk Farm**

- L0

- 9,000

**Notes:**
MODAPS is part of the larger MODIS processing stream of EDOS and DAACs
Product flow and responsibilities (EDOS L0, GES DAAC L1, MODAPS L2+ and DAACs ingest and distribution to community
6x => 3,120 GB per day produced,  2,640GB shipped to DAACs, 510GB to Team
Processing is not near real-time (EDOS 1 day behind, GES DAAC 1-2 days behind,  MODAPS 2-3 days behind real-time)
To meet near real-time production needs for customers (US Forest Service, USDA, EPA and DOD) the MODIS Rapid Response System produces and distributes custom products for fires, floods, drought, battle field planning 

Backup Information
Calculations of current volume:
160+18+330+12= 522GB with reduction in Oceans 80+18+330+12=440GB
1,170 GB per 1x  [220GB ingest, 510GB to Science Team, 440GB to DAACs] EDC 

## MODIS products per day @ 1X*

| Type  | Produced by | Product Generation Executives | Volume GB/day | 
Files/day |
| --- | --- | --- | --- | --- |
| Level 0 | EDOS | N/A | 70 | 
12 |
| Level 1+ | GES DAAC
MODAPS | 6  DAAC
2  MODAPS | 235 | 
4,500 |
| Oceans SST, Atmospheres | MODAPS | 16 | 18 Atmos 
51 SST | 
2,400 Atmos
2,950 Oceans |
| Land | MODAPS | 27 | 312 | 
9,500 |
| 
Snow/Sea Ice | MODAPS | 9 | 14 | 
2,300 |
| Totals  | All (MODAPS only) | 60 (54) | 700 (495) | 

24,600 (22,008) |

- * Sustained land + snow&ice: Collection 4:   5.6X         Collection 5:  9X (7X reprocessing, 2X forward)

## Aqua Collection 4 (Atmospheres)

- Atmospheres Collection 4 Aqua Reprocessing
- Collection 4 reprocessing for the interval July 4, 2002 through February 1, 2004 (578 days) began on January 11, 2004
- Atmosphere data were reprocessed in parallel with land at approximately 4 data-days/production day (4x)
- 2004 data were processed early to ensure a continuous 2004 Collection 4 data set
- On March 9 after 37% of the data had been processed the Atmosphere team identified problems in the L2 cloud top algorithm and the L3 weekly and monthly aerosol algorithms and software correcting the problem was developed and delivered to MODAPS
- To date all 2003 and 2004 data have been reprocessed
- Starting mid-July 2002 data will be reprocessed using L0 data acquired from the GES DAAC and will be completed mid-August

## Aqua Collection 4 (Land)

- Land Collection 4 Aqua Reprocessing
- Collection 4 reprocessing for the interval July 4, 2002 through February 1, 2004 (578 days) began on January 22, 2004 and was completed on July 5, 2004
- EDC DAAC ingested products from reprocessing at 3.6x*
- Combined land products (Terra/Aqua) will be completed in December, 2004 using Terra products ordered from EDC DAAC
- *2.8x  was the baseline ingest rate for reprocessing at EDC

## Aqua Collection 4 (SST)

- Sea Surface Temperature: Aqua Reprocessing
- Aqua Oceans SST forward processing was converted to Collection 040 on data-day 89, 2004
- Collection 041 reprocessing for the interval March 26, 2002 through March 29, 2004 (633 days) began on April 4, 2004
- Oceans SST products were generated at approximately 30x on a separate machine from land and atmospheres reprocessing
- On April 27 after 70% of the oceans SST data had been reprocessed two problems were identified by OCDPS: a problem with the global attribute metadata associated with the HDF re-formatter for the ancillary Reynolds and a problem with the spatial transformation of the Reynolds data in the SST code
- Corrected code was obtained from Miami in early-June and will be placed in forward processing starting with data day 7/19/04
- SST 042 reprocessing on-hold pending Ocean discipline decision

## Collection 5 Reprocessing

- Collection 5 reprocessing for Terra and Aqua will produce 8.2 data-years of land and sea surface temperature and 7.6 years of atmospheres products
- Product generation in MODAPS will begin from L0 to speed up overall throughput to cut at least 7 months off Collection 5 schedule and provide more flexibility in discipline processing
- 1 year (2003) of Terra, Aqua and combined products from Collection 5 will be produced within the first 2-4 months of discipline processing to enable early development of software improvements for Collection 6

## MODAPS Architecture

- SGI Origin systems produce Level 3 weekly through quarterly products + L1
- CPU farm - 2 processor Linux systems for processing Level 1 through Level 3 daily products
- Disk farm - 2 processor Linux systems running handle compression of Level 0 products
- Database server – 4 processor Linux system running Sybase tracks all files and processing and distribution jobs
- Web server on external network provides access to files for science team
- All systems shown here are on internal networks behind firewalls

**Switched Internal Networks 100baseT-GIGE**

**Gigabit Ethernet **
**Switch**

- Database
-  server

- Disk farm

- CPU farm

- 28-64 cpu SGI Origin

**Notes:**
 Processing strategy is to keep inputs for processing and products being shipped to the DAACs on the 60TB of online storage.
When all the L1B granules don’t arrive for a day that day is held open until they do, 8-day, 16-day and 32-day products are also held open.  3-4 weeks of data may be in this state when there are problems at upstream systems (EMOS, EDOS or GES DAAC). 
Databases are at the center of MODAPS production.  These keeps track of all files in the production system and nearline storage
Now 20,016 files per  1x, 6X=>    40K forward 80K reprocessing = 120K files
4-8p Dell servers running Linux and Sybase SQLserver
Databases are dumped once per day to backup server where checks run

## Collection 5 (Reprocessing Rates)

- With the new processing approach the following key questions exist:
**Can MODAPS generate L1A, geolocation, L1B and cloud mask products that match those produced by the GES DAAC ?** Extensive tests have been conducted with Terra and Aqua and matching results have been obtained by MODAPS.  During the Collection 5 tests will be conducted once per month to ensure that matching values are obtained. **Can MODAPS process atmospheres from L0 input with a single machine at the required rate of 11x ?** Tests have demonstrated that MODAPS can perform this processing at 15x
**Can MODAPS process land and SST at 3.5x on each of the two production machines?** Results so far indicate that each system can sustain production rates in excess of 4x
**Can the GES DAAC send L0 data at 11x (lower rates are required after the GES DAAC starts Collection 5 reprocessing in October?** Tests indicate DAAC can send L0 at rates exceeding 15x
**Can the EDC DAAC ingest land products at 7x ?** Land will reduce its archived product volume from 330GB to 220GB/dataday. DAAC has indicated that it can ingest this data volume at 7x. Tests are scheduled for late July to verify this

## Collection 5 (Software Deliveries)

- Software delivery dates have been arrived at with the disciplines
- The schedule is driven solely by the software delivery dates below
- Key delivery dates for science codes are:
- L1A and geolocation Terra and Aqua: 9/10/04 and 12/3/04
- L1B Terra and Aqua: 9/24/04 and 1/9/05
- Initial delivery of Atmospheres Terra and Aqua: 6/15 and 12/9/04
- Initial deliveries of Land Terra and Aqua: 10/1/04 and 1/18/04
- Initial deliveries of Oceans SST: 2/1/05 and 5/1/05
- Based these dates and allowing for I&T/science tests/patches:
- Atmosphere reprocessing will start 12/04 and finish 7/05
- Land will start 6/05 and finish 12/06

## Collection 5 (Science Tests)

- Atmospheres science test plan
- Two 16-day tests will be performed for Aqua and Terra followed by a patch test
- The second of these tests will include the Collection 5 Level 1 software and will use Collection 4 land codes in order that both disciplines can validate the Level 1 and cloud mask software
- Atmospheres reprocessing will begin in late-November, 2004 and will be paused after 30 data-days have been produced to allow for a final discipline review of the products
- Land science test plan
- Terra
- Two 32-day science tests will be performed
- Two tests with representative tiles over a 12 month period will be performed each followed by a patch test
- Aqua
- One 32-day test will be performed
- One test with representative tiles be performed followed by a patch test

## Concluding thoughts

- Changes to MODAPS such as processing from Level 0 will increase our reprocessing rate and offer greater flexibility in reprocessing individual disciplines
- Software deliveries are on track for Collection 5
- Timely evaluation of test results and delivery of patches are vital to keeping Collection 5 on schedule but we’ve built more testing into the schedule
- With a smaller staff supporting software integration, testing and production, we can not easily make up schedule slips which will delay Collection 6 reprocessing