# Presentation: 945924

## CERES FM-5

**NPP Science Processing**
**Delta Design Review**
**GSFC, MD**
**August 26, 2008**
- Erika Geier
- Jim Closs
- Denise Cooper
- Sunny Sun-Mack
- Lee Bodden
- Michael Little
**NASA Langley Research Center**

## Agenda

**CERES Introduction**
**Operations Concept**
**Design Assumptions**
**Science Data Processing**
**Requirements**
**Instrument Subsystem Design**
**Clouds Subsystem Design**
**Testing Strategy**
**ASDC Implementation**
**Schedule**
**Risk Analysis**
**Issues/Concerns**

## CERES Mission Ground Systems

**NASA Langley Research Center (LaRC) provides critical cloud and Earth radiation budget climate data records to support global climate change research**** **
**Past ERBE and current CERES instruments on TRMM (1), Aqua (2), and Terra (2), have provided 24 years of sustained radiation budget measurements**
**LaRC has collected over 30 Instrument years of CERES data from TRMM, Terra, and Aqua and has processed over 90% of that data to date**

## CERES Current Capabilities

**Extensive LaRC CERES experience accumulated through the support of the five CERES instruments currently in orbit**
**Existing scientific expertise for the development, calibration, management and application of CERES data products**
**Data Center Operations already in place providing ingest, archival, production, management and distribution of CERES data products**
**Existing Science Computing Facility (SCF) for scientific analysis, investigations and development of CERES data products**
**Highly-experienced staff already in place supporting:**
- Ongoing algorithm investigation, development and enhancements
- Implementation of algorithms into CERES Data Management System
- Enhancements and development to current and new CERES data production systems
- Production and distribution of CERES data products in a configured operations
**The CERES DM task was successfully appraised at CMMI Capability Level 3 /Equivalent Maturity Level 2**

## CERES Climate Analysis Research 

- Science

**CERES Climate Analysis Research **
**System (CARS) Organization**

- Data Management Team
- (DMT)

- Atmospheric Sciences Data Center
- (ASDC)

-  Implements algorithms
-  Maintains software
-  Verifies data
-  Assists in validation
-  Provides CM and
-      documentation support

-  Ingests data
-  Places operational
-     software in production
-  Produces data sets
-  Distributes data sets
-  Archives data
-  Provides User Services

-  Derives & refines
-      algorithms
-  Validates algorithms
-  Validates CERES data
-      sets
-  Writes Quality Summary

***Algorithm Development******	******Algorithm Implementation******	******            Data Production ***

**Notes:**
What you need to remember from this slide is really the flow of information.

Science (ST and support) passes science algorithms and other requirements to the DMT.
The DMT prepares the production software from these requirements and passes this software on to the
ASDC where the CERES data are processed using this software.

## Required CERES CARS Functionality

**Provide Land PEATE sub-sampling software for aggregated VIIRS SDRs (Science and DMT)**
**Acknowledge data from SDS Land PEATE (ASDC)**
**Ingest data from SDS Land PEATE & other sources (ASDC)**
**Produce, archive, distribute data products (ASDC)**
**Report status of collection (ASDC)**
**Support user access to collection (ASDC)**

## CERES Operations Concept

**CERES instrument raw data transmitted to the C3S and IDPS**
**IDPS creates CERES Raw Data Records (RDRs) and provides RDRs to the SDS SD3E and to ADS/CLASS**
**SD3E provides CERES RDRs and VIIRS xDRs to the Land PEATE**
**Land PEATE passes CERES RDRs along to ASDC**
**Land PEATE runs VIIRS Sub-sampler; provides output to ASDC**
**ASDC ingests and archives inputs coming from Land PEATE**
**ASDC generates and archives higher-level CERES products**
**ASDC distributes products to science researchers and public**

## CERES Operational Data Flow

- SD3E

- L-PEATE

- ADS

- IDPS

- SDS

- All RDRs

- All xDRs

- CERES RDRs,
- VIIRS Sub-samples & aerosols

- C3S

- Commands, Loads
- and requests

- RDRs
- SDRs
- EDRs
- IPs

- CERES
- Instrument
- Ops Team

- Svalbard

- Cmd
- &
- Tlm

- Mission
- Data

- Existing data, agreements

- Data
- Providers

- Mission Notices
- and Data

- RDRs to PST

- CERES
- CARS
-  Data
-  and
- Science
- Operations

- Mission Notices
- and Data requests

- Data
- Distribution

- Data
- Users

- Reuse existing systems and interfaces
- System enhancements for NPP CERES

- NPP Systems

- NOAA Systems

- CERES Systems

## CERES Data Processing Design Assumptions 

**CERES CARS data processing development and operations leverages the existing personnel, procedures, production systems, and infrastructure already in place supporting CERES instruments on EOS Terra and EOS Aqua Missions**
**CERES CARS will use existing interfaces at the LaRC ASDC**
**CERES CARS software development is extensively reusing existing Terra/Aqua processing codes**
**Primary required NPP related enhancements are:**
- Use CERES raw data in new Raw Data Record (RDR) format**:**
- Develop a preprocessor to distribute/sort RDR contents into Aqua-like files that can be used as inputs to existing CERES processing stream
- Use VIIRS data products:
- Use VIIRS sub-sampled calibrated radiance and aerosol products in place of instantaneous MODIS data.

## NOAA IDPS Assumptions

**Acquire CCSDS Packets**
**HDF Wrap CCSDS Packets**
**Generate & Stage RDRs for all APIDs for SDS & CLASS**
**Maintain CDFCB**
**Provide samples of each CERES RDR**
**Provide 28 hours of CERES RDRs for testing CERES codes at LaRC**

## SDS/Land PEATE Assumptions

**Subscribe to SD3E for all CERES RDRs (Science, Diagnostic, and Housekeeping/Telemetry)**
**Acquire CERES RDRs and forward to ASDC via existing Land PEATE interface**
**Support CERES reordering of missing data**
**Produce aggregated VIIRS SDRs (MODIS HDF-EOS format) in 5 minute granules (VIAE, VMAE, VDAE). Sub-sample and ship 5 minute granules (VIMD) to ASDC**
**Produce VIIRS aerosol in 5 minute granules (VAOT) and ship to ASDC**
**Post-launch: When VIIRS calibration and/or algorithm changes, reprocess VIIRS from covers-open forward and ship to ASDC to support CDRs**

## Software Development Process

- New Requirement
- Received, Logged,
- and
- Analyzed

- Requirement Accepted and Assigned

- Subsystem Updates and Unit Testing

- Updates Integrated Into Baseline

- Subsystem Science Testing

- Pre-CM Testing

- Delivery to CM

- CM Testing

- Release to SIT

- Operational Testing at ASDC

- ValRx Testing

**Requirements**_**Management**_

**Subsystem Product Integration and **_**Verification**_

**CM Validation and **_**Product Integration**_

_**ASDC Validation**_** **

**Software Development Process**

**Software Development Process is in the following document currently under CERES Management review:**
- Clouds and the Earth’s Radiant Energy System (CERES)National Polar-orbiting Operational Environmental Satellite
- System (NPOESS) Preparatory Project (NPP)Data Management System (DMS)Software Management Plan Version 0

## Science Data Processing Approach for CERES FM-5

**Science Data Processing ****Approach for CERES FM-5**

_**Processing is divided into 2 Streams: **_
**Instrument and ERBE-like**
- Autonomous stream, dependent only on CERES data
- In production within 48 hours of power-on
- Aids in verifying Instrument health
- Critical for establishing CERES calibration/validation
- Produces ERBE-like data sets
- Data sets exist for CERES instruments on TRMM, Terra, and Aqua
- Similar data sets exist for ERBE instruments on ERBS, NOAA-9 and NOAA-10
**Fused data sets beginning with Cloud/Convolution/Inversion processing and the SSF**
- Use inputs from imager and other data sources
- Higher resolution imager data is Point Spread Function (PSF) weighted
- Produces higher quality climate data
- Data sets exist for CERES instruments on TRMM, Terra, and Aqua
- Will not be in production immediately after covers open
- Requirement is to ingest VIIRS radiance, geolocation, and aerosol data when it becomes available from Land PEATE

## Simplified CERES Processing Flow

**Simplified CERES Processing Flow**

- SS4
- Compute Clouds,
- Convolve with
- CERES, compare
- TOA/surface
- fluxes

- CERES
- RDRs

- SSI
- Instrument

- SS2
- Instantaneous
- ERBE-like

- SS3
- ERBE-like
- TISA

- ES-4

- ES-9

- BDS

- ES-8

- IES

**Autonomous Subsystems affected **
**by changing NPP inputs**

**Autonomous CERES Subsystems**

- VIIRS
- Radiances
- Geolocation
- Aerosols

- SS9
- TISA
- Gridding

- SS10
- TISA
- Averaging

- SRBAVG

- SSF

- SFC

- SS5
- Compute fluxes
- at levels

- SS6
- TISA
- Gridding

- SS7 & 8
- TISA
- Averaging

- AVG

- SYN

- CRS

- FSW

- ZAVG

**Subsystems affected by NPP input**** **

## Current CERES Climate Data Record Production Architecture

**Current CERES Climate Data Record Production Architecture**

## CERES Input Data

**CERES Input Data**

| Type of Data | Parameter
Description | Freq | Source | Comments |
| --- | --- | --- | --- | --- |
| CERES RDRs | Instrument level 0 data, attitude, and ephemeris | ~131/day
 | Land PEATE | Expecting 3 types of RDRs: Science, Telemetry, and Diagnostic |
| VIIRS L1B Calibrated Radiance Data | Imager radiances & Geolocation 
Aerosols |  288/day
~144/day | Land PEATE | Have identified radiance subsets and will provide associated code |
| Aerosol data  | Aerosol (Coln) Optical thickness, type/size | 1/day | MODAPS | For Terra/Aqua using MODIS MOD08 and MATCH. Plan to do same for NPP  |
| Ozone data | Ozone Profile | 1/day SMOBA
1/day OMI | NCEP SMOBA 
OMI  |  |
| Meteorological data | 3-D Met Data
2-D atmospheric data
2-D constants | 4/day
24/day
1 | GMAO |  |
| Precipitable Water | 2-D constants | 2/day | Global Hydrology Resource Center (GHRC) |  |
| Geostationary data | MCIDAS data from 5 geostationary satellites per month | 120/day | University of Wisconsin Space Science and Engineering Center (SSEC) | Only every 3rd hour is used for production |
| SURFMAP(Snow/Ice) | Snow/Ice Map | 4/day | NCEP/NESDIS |  |
| SURFMAP(Snow/Ice) | Snow/Ice Map | 1/day | NSIDC |  |

- Existing Data Source

## CERES Publicly Available Output Products

**CERES Publicly Available Output Products**

| 
Data Product | 
Data Product Name | 
File Freq |
| --- | --- | --- |
| BDS | Bidirectional Scan | 1/day |
| ES-8 | ERBE-Like Inst TOA Filtered Radiances | 1/day |
| ES-9 | ERBE-Like Monthly Regional Averages | 1/mo |
| ES-4 | ERBE-like Monthly Geographical Averages | 1/mo |
| SSF | Single Scanner Footprint TOA/Surface Fluxes & Clouds | 1/hr |
| SFC | Monthly Gridded Radiative Fluxes & Clouds | 36/mo |
| SRBAVG | Monthly TOA/Surface Averages | 5/mo |
| CRS | Clouds and Radiative Swath | 1/hr |
| FSW | Monthly Gridded Radiative Fluxes & Clouds | 60/mo |
| SYN | Synoptic Radiative Fluxes and Clouds | 1/day |
| AVG | Monthly Regional Radiative Fluxes and Clouds | 1/mo |
| ZAVG | Monthly Zonal and Global Radiative Fluxes & Clouds | 1/mo |

## Data Production Scenarios

**Data Production Scenarios**

**CERES strives to receive all possible FM-5 level 0 data and VIIRS data**
**Instrument**
- Ingest manager at ASDC to monitor missing RDR files and request those files from Land PEATE
- Land PEATE has agreed to  provide CERES 99% of RDR files produced by IDPS within 1 month
- Data is actually processed three times
- Baseline1-QC run ~ 6 hours after end of the day.  Used for quick instrument assessment.  Files not needed after Edition1-CV processes.
- Edition1-CV run ~5-20 days after the end of the month.  Assume all available data has been ingested. Calibration/Validation data set used to compute gains and spectral response functions.
- Edition2 run 6-12 months later using Edition1-CV as input.  Applies best available instrument gains and spectral response functions.
**Cloud/Convolution/Inversion processing of the SSF**
- Ingest manager at ASDC monitors missing MODIS files and requests missing files from MODAPS.  VIIRS files to be handled in similar fashion.
- Once in routine production, Land PEATE has agreed to provide CERES 99% of available VIIRS files within 2 months of data date.
- Requires CERES and VIIRS radiance/geolocation data to avoid data gaps in SSF files
- If either one missing, a data gap results
- VIIRS aerosol data also needed.  If unavailable, aerosol parameters set to CERES default fill values and no data gap results.
- Current Terra and Aqua SSFs typically run 6-12 months after data date.  NPP processing will be very similar to Terra and Aqua.

## CERES CARS Requirements Management 

**CERES CARS Requirements Management **

**CERES Level 3 and Level 4 requirements are being baselined in a CM-controlled document**
**CERES Level 3 requirements have been reviewed by the SDS and will be provided to the SDS for inclusion in the SDS Requirements Specification**
**Requirements change requires approval by Project Management and the Configuration Control Board**
**Requirements are mapped into subsystem(s) for implementation**
**Requirements are mapped into tests for verification and validation**

## CERES Data Sources

- Land PEATE

-  CERES CARS

- CERES RDRs

- VIIRS Geo/Rad

- VIIRS Aerosols

- VIIRS Subsample code

- CERES RDRs

- Sub-sampling

- VIIRS Geo/Rad

- VIIRS Aerosols

- Instrument
- processing

- ERBE-like
- processing

- Clouds
- processing

- Archive,
- Distribution

- User Community

**CERES Data Sources**

- SD3E

- 1- The Land PEATE receives CERES RDRs and VIIRS xDRs from the SD3E. 2- CERES RDRs are passed directly to the CERES CARS, where they are processed by the Instrument subsystem to
-     produce inputs to the ERBE-like and Cloud subsystems.
- 3- The Land PEATE produces MODIS-like VIIRS radiance/geolocation and aerosol files.
- 4- The Land PEATE sub-samples the MODIS-like VIIRS radiance/geolocation data using software provided by CERES.
- 5- The Land PEATE sends the sub-sampled VIIRS files and the MODIS-like VIIRS aerosol files to the CERES CARS.
- 6- The CERES CARS runs the downstream subsystems, beginning with Clouds, as data becomes available.
- 7- Climate-quality data products are archived and made available to the user community along with Data Quality Summaries and user support.

## CERES SDS Requirements

**CERES SDS Requirements**

**NPP CERES FM-5 Science Data Segment Requirements:**
**The ASDC Ingest subsystem shall have the capability of ingesting, verify by Checksum, and archiving CERES RDRs, sub-sampled VIIRS radiance/geolocation files, and VIIRS aerosol data received from the Land PEATE.**** **
**The CERES CARS Preprocessor of the Instrument Subsystem shall have the capability of producing level-0, attitude, and ephemeris files as expected by the processing software.**** **
**The CERES DMT shall have the capability of generating and delivering to the ASDC the following Science Subsystem code:****- Instrument (at launch)****- ERBE-like (at launch)****- Clouds and subsequent subsystem software (12+ months after****  launch)**** **
**The CERES DMT shall provide software to the Land PEATE to sub-sample VIIRS radiance/geolocation files, and work with the Land PEATE to ensure that the sub-sampling code executes properly.**

## CERES SDS Requirements

**CERES SDS Requirements**

**NPP CERES FM-5 Science Data Segment Requirements (cont.):**
**The CERES DMT shall have the capability of modifying and redelivering production code for all 12 CERES Subsystems to produce CERES NPP data products.**** **
**The ASDC shall have the capability of generating, archiving, and disseminating climate-quality data products.**** **
**The CERES DMT and Science Team shall have the capability of validating and writing quality summaries for all CERES data products.**** **
**The ASDC shall have the capability of providing quality summaries and user support for data customers.**

## Requirements Verification

**Baseline LaRC FM-5 L4 requirements by placing CERES CARS Requirements Specification Document on NX (CERES CARS, 09/2008)**** **
**Baseline SDS FM-5 L3 requirements by adding CERES to SDS Requirements Specification Document (SDS, 09/2008)**
**Map FM-5 L3 requirements to SDS Testing Scorecard and schedule (Vic Buczkowski, 10/2008)**
**Produce Requirements Traceability Matrix to map LaRC L4 requirements to CERES CARS Subsystems and test cases (CERES CARS, 10/2008)**
**Conduct FM-5 SDS L3 testing as documented in SDS Testing  Scorecard (SDS/LaRC Test Team, (02/2009)**
**Conduct FM-5 LaRC L4 testing as documented in LaRC Requirements Traceability Matrix (LaRC Test Team, (mid-2009)**

## Instrument Subsystem Design Approach

**Instrument Design Approach**
**and**
**CERES DMT RDR Tests**
**Denise Cooper**
**Contributing Team Members:**
**Ashley Alford**
**Dale Walikainen**
**Mark Timcoe**

## Instrument Design Approach

**Extensive reuse of existing software for instrument subsystem**
- Proven system used for current CERES processing
- Majority of software written in Ada
**Newly developed code to be in C++**
- RDR Preprocessor
- Creates 24-hr Level-0 and Toolkit compliant Ephemeris and Attitude data files
**Ada code will be updated to allow processing of FM-5 data **
**PGE to process incoming NPP Level-0 data will be the first priority, other PGEs will be converted following successful testing of this PGE.**

## Instrument Design Approach

**PGE providing data to ERBE-like already in C, only changes will be to allow processing of FM-5 data.**
**Initial version of all code will use the SDP Toolkit due to schedule constraints.  Updates to remove dependency on the SDP Toolkit expected after launch.**
**CERESlib updates to create a C-version of existing F90 Metadata creation routines.**
- Facilitate integration of existing Metadata with Ada, C and C++ software.

## NPP Instrument Subsystem

- L0
-  Data

- Geolocate & Calibrate

- BDSs

- IESs

- Pre-Es8
- Generator

- Pre-ES8

- To Convolution

- To ERBE-Like

- Attitude
- Data

- Preprocessor

- ~133 Science &
- Diagnostic RDRs
- with Diary data

- 2 files needed to process
- (time span noon yesterday -
- noon today)

- 14 files needed to process
- (time span hr 22 day before -
- hr 00 next day)

- Up to 3 files per day
- (Science, Calibration &
- Diagnostic)

- Up to 24 files per day
- (HDF4 Vdata containing
- Geolocation values
- Filtered radiances)

- Up to 6 files per day
- BDS, BDSS, BDSD,
- BDSM, BDSP & BDSG
- (HDF4 SDS & Vdatas containing
- Geolocation values
- Raw data counts
- Filtered radiances
- Instrument H&S info)

- 1 file per day
- (Binary containing
- Geolocated,
- Filtered radiances)

- Ephemeris
- Data

- Attitude
- Data

- Attitude
- Data

- Attitude
- Data

- Attitude
- Data

## ERBE-like Spectral Corrections

- Modtran
- Data

- Unfiltering
- Coefficients

- Unfiltering

**ERBE-like Spectral Corrections**

- No code changes needed to support NPP

## ERBE-like Inversion

- Daily
- Inversion

- Pres8

- EID6

- ES4

- ES9

- ES8

- Monthly
- Averaging

**ERBE-like Inversion**

- From Instrument Subsystem

- No code changes needed to support NPP

- 1 file per day
- (Binary containing
- Geolocated,
- Filtered radiances)

- 1 file per day
- (HDF-EOS4 containing
- Geolocated
- Filtered & Unfiltered radiances &
- fluxes)

- 1 file per day
- (Binary database
- Info)

- 1 file per month
- (HDF4 containing
- Regional, Zonal  & Global
- Gridded fluxes)

- 1 file per month
- (HDF4 containing
- Global Average fluxes)

## CERES DMT RDR Tests

**Objectives**
- Ensure the RDR preprocessor is able to read the CERES Science and Diagnostic RDR HDF5 files and create the properly formatted Level-0, Ephemeris and Attitude data files that are compatible with the updated Ada and the converted C++ code.
**Description**
- Assume ingest of RDRs by ASDC;  Alternative is to access RDRs stored in specified location if not available from the ASDC
- Test with selected RDRs on the development platform to provide output that will be used by the Level-0 data processor to show that output Level-0, Ephemeris and Attitude data is as expected by this PGE.
- Test on the development platform with minimum of 28-hrs of RDR data through the Level-0 PGE, on to the Pre-ES8 generator to create the data used by ERBE-like.

## CERES DMT RDR Tests

**Description (Cont’d)**
- Pre-delivery testing on production platform to ensure all systems work on the production platform and generate scientifically equivalent data.
- Ensures all software works as expected on the production platform before delivery
- Provides expected output used by CM to verify their test results
- CM testing using Instrument Subsystem Test Plan to verify that all necessary updates have been delivered and provide output equivalent to the provided expected output.
- ASDC SIT operational testing:  e.g. Dec. 31, Jan. 1, Feb. 28/29, etc.
**Participants**
- Ashley Alford, Denise Cooper, Mark Timcoe, Land PEATE, LaRC ASDC SIT

## CERES DMT RDR Tests

- RDRs
- Science & Diagnostic

- ~133 per data day
- (Level-0 packets &
- Diary data)

- RDR
- Pre-processor

- Science
- Level-0 File

- Calibration
- Level-0 File

- Diagnostic
- Level-0 File

- Total 10391 packets
- Per data day

- Ephemeris
- Data Files

- Attitude
- Data Files

- Data files
- (Noon previous day -
- Noon next day)

- 12 Data files
- (2 hr data per file)

## Instrument Schedule

**RDR Format Documentation Provided (Jun. 2008)**
**Unpacking of RDR Wrapper Implemented (Jul. 2008)**
**Conversion of RDR Contents to CERES Level 0 Format Implemented (Sep. 2008)**
**Selection and Conversion of Attitude/Ephemeris Data Implemented (Sep. 2008)**
**Informal Testing with Actual RDR ****(dependent on IDPS providing early RDR)**** (Oct. 2008)**
**Instrument Subsystem Enhancements Integrated and Tested (Dec. 2008)**
**Instrument Subsystem Verification Testing Complete ****	****(Jan. 2009)**
**Delivery to ASDC through DMT CM for Formal Validation Tests (by Mar. 13, 2009)**

## Clouds Subsystem Design Approach

**CERES Clouds Subsystem Design Approach**
**and**
**CERES DMT VIIRS Tests**
**Sunny Sun-Mack**
**Contributing Team Members:**
**Yan Chen**
**Walt Miller**

## CERES SSF Processing(Subsystem 4)

**CERES SSF Processing****(Subsystem 4)**

**Subsystem 4 - ****Determine Cloud**** ****Properties, TOA and Surface Fluxes, is made up of 3 different sets of code**
**Clouds - VIIRS inputs will necessitate changes to support NPP**
- Processes the imager data
- Resulting pixel-level output written to temporary file
**Convolution - minor changes needed to support NPP**
- Merges the pixel-level data with the CERES footprint data
- Resulting footprint-level output written to a temporary file
**Inversion - no change needed to support NPP**
- Reads the footprint level output file containing CERES radiance data and cloud properties
- Computes unfiltered radiances, TOA fluxes, and surface fluxes
- Generates the SSF data product which is archived and publicly released

## NPP Impact on CERES Clouds and Convolution Design Approach

**VIIRS will replace MODIS as imager source**
- Ability to generate certain cloud properties changes
- Some radiances, CO2 slicing, may be missing
**CERES IES (input from Instrument Subsystem) interface will remain the same**
**New Point Spread Function (PSF) ancillary file will be generated to account for VIIRS imager resolution**
**SSF interface change possible as CERES migrates to Edition3 **

## Clouds Subsystem Design Approach

**Newly developed VIIRS sub-sampling code at LaRC**
**VIIRS Sub-sampling Preprocessor**** **
- Written in C language
- Input:
-                 NPP_VIAE    ( Radiances from Imaging “I” Channels )
-                 NPP_VMAE  ( Radiances and geolocations from
-                                         Moderate-Resolution “M” Channels)
-                 NPP_VDNE   ( Radiances and geolocations from Imaging
-                                          Day-Night Band (channel)  “DNB” )
- Output:
-                  NPP_VIMD_SS ( Radiances and geolocations from all above
-                                               three Imaging, Moderate and DNB,  but
-                                               sub-sampled and sub-channeled with
-                                               certain channels being averaged )

## Clouds Subsystem Design Approach

**VIIRS sub-sampling code will run at the Land PEATE**** **
**The package to be delivered to Land PEATE by **
**    ****CERES CM at LaRC**
- VIIRS sub-sampling code
- Process Control Generator to produce process control file (PCF)
- Input data and Expected sub-sampled output
- Test Plan (PDF)
**VIIRS sub-setting code will be run at Land PEATE **
- The output from VIIRS sub-sampling code, NPP_VIMD_SS,  will be
-     produced at Goddard Land PEATE
- Land PEATE will then deliver NPP_VIMD_SS to LaRC ASDC
**LaRC ASDC will ingest and archive NPP_VIMD_SS**
**NPP_VIMD_SS will be the Cloud Imagery Data (CID) as input to**
**     ****CERES Clouds Subsystem. **

## VIIRS Sub-setting Data Formats

**Sub-setting NPP_VIAE **
**           ****Resolution: 375 meter**
**           ****Radiances from VIIRS Imaging Channels **

**Sub-setting Channels**

## VIIRS Sub-setting Data Formats

**Sub-setting NPP_VIAE ( Cont’d)**
**                                                 **

**M Pixels (NPP_VIMD_SS)**

**I Pixels (NPP_VIAE)**

-  **Averaging 4 Imaging Pixels to 1 Moderate Resolution Pixel**

## VIIRS Sub-setting Data Formats

**Sub-setting NPP_VIAE ( Cont’d)**

**Sub-setting Samples**

**NPP_VIAE**** **

**NPP_VIMD_SS**** **

**Scan-lines x Pixels = 4608 x 6400**

**Scan-lines x Pixels = 2304 x 3200**

**Every other pair of scan-lines**

**Every other pair of pixels**

## VIIRS Sub-setting Data Formats

| SDS Name (NPP_VIAE) | Description | Dimensions | SDS Name (NPP_VIMD_SS) | Dimensions |
| --- | --- | --- | --- | --- |
| Radiance_Img_I1 | Radiances for I1 in moderate resolution | (6144,6400) | Radiance_Img_I1_Avg | (1536,1600) |
| Radiance_Img_I3 | Radiances for I3 in moderate resolution | (6144,6400) | Radiance_Img_I3_Avg | (1536,1600) |
| Radiance_Img_I4 | Radiances for I4 in moderate resolution | (6144,6400) | Radiance_Img_I4_Avg | (1536,1600) |
| Radiance_Img_I5 | Radiances for I5 in moderate resolution | (6144,6400) | Radiance_Img_I5_Avg | (1536,1600) |
| Reflectance_Img_I1 | Reflectance for I1 in imagery resolution | (6144,6400) | Reflectance_Img_I1_Sub
Reflectance_Img_I1_Avg | (3072,3200) (1536,1600) |
| Reflectance_Img_I3 | Reflectance for I3 in imagery resolution | (6144,6400) | Reflectance_Img_I3_Sub
Reflectance_Img_I3_Avg | (3072,3200) (1536,1600) |
| BrightTemp_Img_I4 | Brightness temperature for I4 at imagery resolution | (6144,6400) | BrightTemp_Img_I4_Sub BrightTemp_Img_I4_Avg | (3072,3200) (1536,1600) |
| BrightTemp_Img_I5 | Brightness temperature for I5 at imagery resolution | (6144,6400) | BrightTemp_Img_I5_Sub
BrightTemp_Img_I5_Avg | (3072,3200) (1536,1600) |
| QF_VIIRS_I1_SDR_1 | Quality control flag for I1 | (6144,6400) | QF_VIIRS_I1_SDR_1_Sub | (3072,3200) |
| QF_VIIRS_I1_SDR_3 | Scan quality control flag for I1 | (192,4) | QF_VIIRS_I1_SDR_3 | (192,4) |
| QF_VIIRS_I3_SDR_1 | Quality control flag for I3 | (6144,6400) | QF_VIIRS_I3_SDR_1_Sub | (3072,3200) |
| QF_VIIRS_I3_SDR_3 | Scan quality control flag for I3 | (192,4) | QF_VIIRS_I3_SDR_3 | (192,4) |
| QF_VIIRS_I4_SDR_1 | Quality control flag for I4 | (6144,6400) | QF_VIIRS_I4_SDR_1_Sub | (3072,3200) |
| QF_VIIRS_I4_SDR_3 | Scan quality control flag for I4 | (192,4) | QF_VIIRS_I4_SDR_3 | (192,4) |
| QF_VIIRS_I5_SDR_1 | Quality control flag for I5 | 6144,6400) | QF_VIIRS_I5_SDR_1_Sub | (3072,3200) |
| QF_VIIRS_I5_SDR_3 | Scan quality control flag for I5 | (192,4) | QF_VIIRS_I5_SDR_3 | (192,4) |

**1. Sub-setting NPP_VIAE ( Cont’d): ****SDSs Kept**

## VIIRS Sub-setting Data Formats

**                                                 **

| SDS Name        (NPP_VIAE) | Description | Dimensions | SDS Name (NPP_VIMD_SS) | Dimensions |
| --- | --- | --- | --- | --- |
| Radiance_Img_I2 | Radiances for I2 in moderate resolution | (6144,6400) |  |  |
| Reflectance_Img_I2 | Reflectance for I2 in imagery resolution | (6144,6400) |  |  |
| QF_VIIRS_I2_SDR_1 | Quality control flag for I2 | (6144,6400) |   |  |
| QF_VIIRS_I2_SDR_3 | Scan Quality control flag for I2 | (192,4) |   |  |

**Sub-setting NPP_VIAE ( Cont’d):  ****The SDSs not being subset **

## VIIRS Sub-setting Data Formats

| NPP_VMAE |  |
| --- | --- |
| M1 = 0.412 m | M3 = 0.488 m |
| M2 = 0.445 m | M4 = 0.555 m |
| M5 = 0.672 m | M7 = 0.865 m |
| M6 = 0.746 m | M8 = 1.24 m |
| M10 = 1.61 m | M9 = 1.378 m |
| M12 = 3.7 m | M11 = 2.25 m |
| M13 = 4.05 m | M14 = 8.55 m |
|  | M15 = 10.763 m |
|  | M16 = 12.013 m |

**2. Sub-setting NPP_VMAE **
**        ****Resolution: 750 meter**
**         ****Radiances and Geolocation from VIIRS Moderate Channels**

**Sub-setting Channels**

| NPP_VIMD_SS |
| --- |
| M3 = 0.488 m |
| M4 = 0.555 m |
| M7 = 0.865 m |
| M8 = 1.24 m |
| M9 = 1.378 m |
| M11 = 2.25 m |
| M14 = 8.55 m |
| M15 = 10 763 m |
| M16 = 12.013 m |

## VIIRS Sub-setting Data Formats

**Sub-setting NPP_VMAE ( Cont’d)**

**Sub-setting Samples**

**NPP_VMAE**** **

**NPP_VIMD_SS**** **

**Scan-lines x Pixels = 3072 x 3200**

**Scan-lines x Pixels = 1536 x 1600**

**Every other scan-line**

**Every other pixel**

## VIIRS Sub-setting Data Formats

| Latitude |
| --- |
| Longitude |
| Solar Zenith Angle |
| Solar Azimuth Angle |
| Sensor Zenith Angle |
| Sensor Azimuth Angle |
| Satellite Range |
| Terrain Height |

**All Geolocation Info in NPP_VMAE is included in the Subset NPP_VIMD_SS**

| Scan Start Time |
| --- |
| Scan Mid Time |

**NPP_VIMD_SS:  **
**SDS (64 bits float)**** **

**NPP_VMAE:  **
**Global Attributes**

**2. Sub-setting NPP_VMAE (Cont’d) **

**NPP_VMAE and NPP_VIMD_SS**

## VIIRS Sub-setting Data Formats

**2. Sub-setting NPP_VMAE (Cont’d): ****SDSs Kept**

| SDS Name | Description | Dimensions | SDS Name | Dimensions |
| --- | --- | --- | --- | --- |
| Radiance_Mod_M11 | Radiances for M11 in moderate resolution | (3072,3200) | Radiance_Mod_M11_Sub | (1536,1600) |
| Radiance_Mod_M14 | Radiances for M14 in moderate resolution | (3072,3200) | Radiance_Mod_M14_Sub | (1536,1600) |
| Radiance_Mod_M15 | Radiances for M15 in moderate resolution | (3072,3200) | Radiance_Mod_M15_Sub | (1536,1600) |
| Radiance_Mod_M16 | Radiances for M16 in moderate resolution | (3072,3200) | Radiance_Mod_M16_Sub | (1536,1600) |
| Radiance_Mod_M3 | Radiances for M3 in moderate resolution | (3072,3200) | Radiance_Mod_M3_Sub | (1536,1600) |
| Radiance_Mod_M4 | Radiances for M4 in moderate resolution | (3072,3200) | Radiance_Mod_M4_Sub | (1536,1600) |
| Radiance_Mod_M7 | Radiances for M7 in moderate resolution | (3072,3200) | Radiance_Mod_M7_Sub | (1536,1600) |
| Reflectance_Mod_M11 | Reflectance for M11 in moderate resolution | (3072,3200) | Reflectance_Mod_M11_Sub | (1536,1600) |
| Reflectance_Mod_M3 | Reflectance for M3 in moderate resolution | (3072,3200) | Reflectance_Mod_M3_Sub | (1536,1600) |
| Reflectance_Mod_M4 | Reflectance for M4 in moderate resolution | (3072,3200) | Reflectance_Mod_M4_Sub | (1536,1600) |
| Reflectance_Mod_M7 | Reflectance for M7 in moderate resolution | (3072,3200) | Reflectance_Mod_M7_Sub | (1536,1600) |
| Reflectance_Mod_M8 | Reflectance for M8 in moderate resolution | (3072,3200) | Reflectance_Mod_M8_Sub | (1536,1600) |
| Reflectance_Mod_M9 | Reflectance for M9 in moderate resolution | (3072,3200) | Reflectance_Mod_M9_Sub | (1536,1600) |
| BrightTemp_Mod_M14 | Brightness temp. for M14 at mod resolution | (3072,3200) | BrightTemp_Mod_M14_Sub | (1536,1600) |
| BrightTemp_Mod_M15 | Brightness temp for M15 at mod resolution | (3072,3200) | BrightTemp_Mod_M15_Sub | (1536,1600) |
| BrightTemp_Mod_M16 | Brightness temp for M16 at mod resolution | (3072,3200) | BrightTemp_Mod_M16_Sub | (1536,1600) |

## VIIRS Sub-setting Data Formats

**                                                 **

| SDS Name | Description | Dimensions | SDS Name | Dimensions |
| --- | --- | --- | --- | --- |
| QF_VIIRS_GEO_MOD_2 | Quality control flag for geo location | (3072,3200) | QF_VIIRS_GEO_MOD_2_Sub | (1536,1600) |
| QF_VIIRS_M11_SDR_1 | Quality control flag for M11 | (3072,3200) | QF_VIIRS_M11_SDR_1_Sub | (1536,1600) |
| QF_VIIRS_M11_SDR_3 | Scan quality control flag for M11 | (192,4) | QF_VIIRS_M11_SDR_3 | (192,4) |
| QF_VIIRS_M14_SDR_1 | Quality control flag for M14 | (3072,3200) | QF_VIIRS_M14_SDR_1_Sub | (1536,1600) |
| QF_VIIRS_M14_SDR_3 | Scan quality control flag for M14 | (192,4) | QF_VIIRS_M14_SDR_3 | (192,4) |
| QF_VIIRS_M15_SDR_1 | Quality control flag for M15 | (3072,3200) | QF_VIIRS_M15_SDR_1_Sub | (1536,1600) |
| QF_VIIRS_M15_SDR_3 | Scan quality control flag for M15 | (192,4) | QF_VIIRS_M15_SDR_3 | (192,4) |
| QF_VIIRS_M16_SDR_1 | Quality control flag for M16 | (3072,3200) | QF_VIIRS_M16_SDR_1_Sub | (1536,1600) |
| QF_VIIRS_M16_SDR_3 | Scan quality control flag for M16 | (192,4) | QF_VIIRS_M16_SDR_3 | (192,4) |
| QF_VIIRS_M3_SDR_1 | Quality control flag for M3 | (3072,3200) | QF_VIIRS_M3_SDR_1_Sub | (1536,1600) |
| QF_VIIRS_M3_SDR_3 | Scan quality control flag for M3 | (192,4) | QF_VIIRS_M3_SDR_3 | (192,4) |
| QF_VIIRS_M4_SDR_1 | Quality control flag for M4 | (3072,3200) | QF_VIIRS_M4_SDR_1_Sub | (1536,1600) |
| QF_VIIRS_M4_SDR_3 | Scan quality control flag for M4 | (192,4) | QF_VIIRS_M4_SDR_3 | (192,4) |
| QF_VIIRS_M7_SDR_1 | Quality control flag for M7 | (3072,3200) | QF_VIIRS_M7_SDR_1_Sub | (1536,1600) |
| QF_VIIRS_M7_SDR_3 | Scan quality control flag for M7 | (192,4) | QF_VIIRS_M7_SDR_3 | (192,4) |
| QF_VIIRS_M8_SDR_1 | Quality control flag for M8 | (3072,3200) | QF_VIIRS_M8_SDR_1_Sub | (1536,1600) |
| QF_VIIRS_M8_SDR_3 | Scan quality control flag for M8 | (192,4) | QF_VIIRS_M8_SDR_3 | (192,4) |
| QF_VIIRS_M9_SDR_1 | Quality control flag for M9 | (3072,3200) | QF_VIIRS_M9_SDR_1_Sub | (1536,1600) |
| QF_VIIRS_M9_SDR_3 | Scan quality control flag for M9 | (192,4) | QF_VIIRS_M9_SDR_3 | (192,4) |

**2. Sub-setting NPP_VMAE (Cont’d): ****SDSs Kept**

## VIIRS Sub-setting Data Formats

**2.  Sub-setting NPP_VMAE ( Cont’d):  ****The SDSs not being subset**
**                                                 **

| SDS Name | Description | Dimensions | SDS Name | Dimensions |
| --- | --- | --- | --- | --- |
| Radiance_Mod_M1 | Radiances for M1 in moderate resolution | (3072,3200) |  |  |
| Radiance_Mod_M10 | Radiances for M10 in moderate resolution | (3072,3200) |  |  |
| Radiance_Mod_M12 | Radiances for M12 in moderate resolution | (3072,3200) |  |  |
| Radiance_Mod_M13 | Radiances for M13 in moderate resolution | (3072,3200) |  |  |
| Radiance_Mod_M2 | Radiances for M2 in moderate resolution | (3072,3200) |  |  |
| Radiance_Mod_M5 | Radiances for M5 in moderate resolution | (3072,3200) |  |  |
| Radiance_Mod_M6 | Radiances for M6 in moderate resolution | (3072,3200) |  |  |
| Radiance_Mod_M8 | Radiances for M11 in moderate resolution | (3072,3200) |  |  |
| Radiance_Mod_M9 | Radiances for M11 in moderate resolution | (3072,3200) |  |  |
| Reflectance_Mod_M1 | Reflectance for M1 in moderate resolution | (3072,3200) |  |  |
| Reflectance_Mod_M10 | Reflectance for M10 in moderate resolution | (3072,3200) |  |  |
| Reflectance_Mod_M2 | Reflectance for M2 in moderate resolution | (3072,3200) |  |  |
| Reflectance_Mod_M5 | Reflectance for M5 in moderate resolution | (3072,3200) |  |  |
| Reflectance_Mod_M6 | Reflectance for M6 in moderate resolution | (3072,3200) |  |  |
| BrightTemp_Mod_M12 | Brightness temperature for M12 at moderate resolution | (3072,3200) |  |  |
| BrightTemp_Mod_M13 | Brightness temperature for M12 at moderate resolution | (3072,3200) |  |  |

## VIIRS Sub-setting Data Formats

**2.  Sub-setting NPP_VMAE ( Cont’d):  ****The SDSs not being subset**
**                                                 **

| SDS Name | Description | Dimensions | SDS Name | Dimensions |
| --- | --- | --- | --- | --- |
| QF_VIIRS_M10_SDR_1 | Quality control flag for M10 | (3072,3200) |  |  |
| QF_VIIRS_M10_SDR_3 | Scan quality control flag for M10 | (192,4) |  |  |
| QF_VIIRS_M12_SDR_1 | Quality control flag for M12 | (3072,3200) |  |  |
| QF_VIIRS_M12_SDR_3 | Scan quality control flag for M12 | (192,4) |  |  |
| QF_VIIRS_M13_SDR_1 | Quality control flag for M13 | (3072,3200) |  |  |
| QF_VIIRS_M13_SDR_3 | Scan quality control flag for M13 | (192,4) |  |  |
| QF_VIIRS_M1_SDR_1 | Quality control flag for M1 | (3072,3200) |  |  |
| QF_VIIRS_M1_SDR_3 | Scan quality control flag for M1 | (192,4) |  |  |
| QF_VIIRS_M2_SDR_1 | Quality control flag for M2 | (3072,3200) |  |  |
| QF_VIIRS_M2_SDR_3 | Scan quality control flag for M2 | (192,4) |  |  |
| QF_VIIRS_M5_SDR_1 | Quality control flag for M5 | (3072,3200) |  |  |
| QF_VIIRS_M5_SDR_3 | Scan quality control flag for M5 | (192,4) |  |  |
| QF_VIIRS_M6_SDR_1 | Quality control flag for M6 | (3072,3200) |  |  |
| QF_VIIRS_M6_SDR_3 | Scan quality control flag for M6 | (192,4) |  |  |

## VIIRS Sub-setting Data Formats

| DNB = 0.7 m |
| --- |

| Latitude |
| --- |
| Longitude |
| Solar Zenith Angle |
| Solar Azimuth Angle |
| Sensor Zenith Angle |
| Sensor Azimuth Angle |
| Satellite Range |
| Terrain Height |

**Subset every other scan-line and every other pixel **

**3.  Sub-setting NPP_VDNE**

**Resolution: 750 meter**
**Radiances and Geolocation from VIIRS Day-Night Channel**

**Only One channel-->No Sub-setting Channel**

**Keep All Geolocation SDSs**

**Sub-setting Sample**

## VIIRS Sub-setting Data Formats

**File Size Summary**
**         ****INPUT:**
**             ****NPP_VIAE  ---> 460 MB / granule**
**             ****NPP_VMAE ---> 560 MB / granule**
**             ****NPP_VDNE ----> 330 MB / granule**
**         ****OUTPUT**
**              ****NPP_VIMD ---> 400 MB / granule**
**               **

-  **~ 70% Data Reduction**

## NPP CERES Cloud Subsystem

**NPP CERES Cloud Subsystem**

- GMAO
- MOA

- Clear
- Reflectance
- History

-  **Clouds**
**Processor**

**TRMM VIRS CID**

**MODIS Terra CID**

**MODIS Aqua CID**

**NPP VIIRS CID**

**Cloud Imager Data (CID)**

**EIPD **
**(Cookie Dough)**

- EQCG

- EQCB

- ECVS

- ECV

**Output**

**NPP VIIRS Subset: NPP_VIMD_SS**

**To CERES Convolution**

## Convolution Subsystem Description

- Cloud Pixel

- Convolution

- Inversion

- X 24 hours

- Instrument

- IES

- FQCI

- SSFAI

- SSFI

- EIPD
- Cookie dough

- FQC

- PSF

**Convolve imager radiances, cloud properties, and aerosol into CERES footprint**

## CERES DMT VIIRS Tests

**      ****Objectives**
**             ****Ensure the VIIRS Sub-sampler processor is able to:**
-             **(1)  Read**
-                       NPP_VIAE ( Imager resolution radiances),
-                       NPP_VMAE ( Moderate resolution radiances and geolocations)
-                       NPP_VDNE (Day / Night band radiances and geolocations)
-             **(2) Subset**
-                       sub-channel
-                       sub-sample
-                       averaging of imagery resolution data
-             **(3)  Produce VIIRS sub-sampled output**** **
-                       NPP_VIMD_SS

## CERES DMT VIIRS Tests

**Description**
- If less than a couple of hours of NPP VIIRS data is available, then Land PEATE pushes VIIRS NPP_VIAE,  NPP_VMAE, and NPP_VDNE to a CERES/VIIRS designated computer
- For larger volumes of data, the Land PEATE sends all test NPP VIIRS data types to the ASDC for ingest, and then subsequently staged for access by the VIIRS Sub-sampler on a CERES/VIIRS computer
- Execute VIIRS Sub-sampler on local CERES/VIIRS computer to create and verify sub-sampled output using input VIIRS data from Land PEATE
- Load VIIRS Sub-sampler and selected VIIRS test data in the file space and computer designated for CERES processing at the Land PEATE, and then conduct testing at the Land PEATE and verify sub-sampled output
- CERES DMT delivers VIIRS Sub-sampler (tar files) and test plan to the CERES CM for delivery to the Land PEATE
- Land PEATE conducts VIIRS Sub-sampler testing creating sub-sample
- Land PEATE verifies sub-sample and also send sub-sampled output to either the ASDC or to a designated CERES/VIIRS computer for additional verification

## CERES DMT VIIRS Tests

**NPP VIIRS**
**Sub-setting**
**Processor**

**NPP_VIAE**

**NPP_VDNE**

**NPP_VMAE**

**NPP_VIMD_SS**

**  ****Required Input**

**  ****Expected Output**

**Participants**
- Sunny Sun-Mack, Yan Chen, Tammy Ayers, Walt Miller, CERES Science Team members: Patrick Minnis, Norman Loeb, Tom Charlock, Dave Kratz and Dave Doelling,  Land PEATE, and LaRC ASDC

## CERES CARS Delivery Processto the Land PEATE

**Following unit, integration and verification testing by the CERES DMT, the VIIRS Sub-sampler must be delivered to the Land PEATE for final testing**
**CERES DMT creates tar files containing source code, process control generator, selected NPP VIIRS input data and expected sub-sampled output**
**CERES DMT updates Test Plan for testing at the Land PEATE**
**CERES DMT delivers the above tar files along with the Test Plan to CERES CM**
**CERES CM delivers the tar files and Test Plan to the Land PEATE’s designated machine.  A “Delivery Notification” email is sent to the Land PEATE and all concerned**

## CERES CARS Delivery Processto the Land PEATE (cont.)

**Land PEATE conducts VIIRS Sub-sampler testing**
**When the Land PEATE testing is successfully completed, the Land PEATE:**
- Sends an email to all concerned regarding the status
- Promotes the delivery to their production environment
- Makes the delivery (as tar files) available in a designated area for CERES CM to retrieve
- Sends a notification email to CERES CM that this action has taken place.
**Upon receiving this notification, CERES CM: **
- Retrieves the tarred delivery
- Places it in the CERES CM repository
**Subsequent changes to CM controlled software are documented and tracked using a CERES Software Configuration Change Request (SCCR) and redelivered through CERES CM**

## NPP VIIRS Sub-sampler Schedule

**Data Format of VIIRS Subset for CERES Defined (June 1, 2008) Completed on Time.**
**Data Product Catalog Pages Documenting VIIRS Subset for CERES Defined (August 1, 2008)  Completed on Time**
**Sub-sampler Code Complete and Running Locally at LaRC on Linux Machine (October 1, 2008)**
**Goddard Land PEATE Accepts the Sub-sampler Code Delivery and Produces Sub-sampled Output from Input Data (December 1, 2008)**
**Verify a Few Days of Sub-sampled Data Produced at GSFC (January 15, 2009) **
**ASDC Ingests Sub-sampled VIIRS Data; Verified by CERES DMT (February 26, 2009)**

## CERES DMT Testing Approach

**The CERES DM task was successfully appraised at CMMI Capability Level 3 /Equivalent Maturity Level 2 **
**SSAI is appraised at CMMI Capability Level 3/Equivalent Maturity Level 2**
**CERES DMT uses a structured CMMI-based development process that includes unit and integration testing**
**Functional (e.g., verification) tests are executed to verify requirements for all enhancements**
**Regression tests are executed to verify existing functionality**
**Delivery packages for CERES and VIIRS Sub-sampler to include:**
- Test Plan
- Expected output
- All inputs
- Code and scripts
**Every CERES code delivery (to ASDC or Land PEATE) is under configuration management (CM) control**
**Documented in CERES NPP DMS Software Management Plan Version 0 **

## CERES CARS Testing Strategy

**Conversion of RDRs to Level 0 data will be tested by the DMT using defined formats and available test RDRs prior to delivery of code to the ASDC**
- Testing conducted in development environment
**Transmission of RDRs by the Land PEATE to the ASDC will be tested when:**
- CERES RDRs are available to the Land PEATE from the SD3E
- Interface between ASDC and the Land PEATE has been established
**Final testing of conversion of RDRs to Level 0 data will involve:**
- Testing in the CERES operations test environment
- Land PEATE sending all three RDR data types to the ASDC
- ASDC ingesting data and acknowledging receipt to the Land PEATE
- Instrument subsystem reading RDRs staged from ASDC archive and correctly converting to Level 0 format

## CERES CARS Testing Strategy

**VIIRS Sub-sampler will be tested by the DMT using test data provided by the Land PEATE prior to code delivery to the Land PEATE**
- Testing conducted in development environment
- Testing also conducted on designated Land PEATE computer
**Transmission of sub-samples by the Land PEATE will be tested when:**
- VIIRS Sub-sampler has been tested and delivered to the Land PEATE
- VIIRS data is available to the Land PEATE from the SD3E or other source
- Interface between ASDC and the Land PEATE has been established
**Final testing of Sub-sampler will involve:**
- Execution of Sub-sampler by the Land PEATE to produce sub-sampled data product
- Transmission of the sub-sampled data to the ASDC for ingest and archive
- CERES DMT stages data from ASDC archive and verifies sub-sample

## DMT NPP Schedule

**DMT NPP Schedule**

**10/1/08: VIIRS sub-sampling software complete and tested at LaRC**
**12/1/08: VIIRS sub-sampling software running at SDS Land PEATE, expected sub-sampled output has been verified by CERES DMT**
**1/15/09: Large set of VIIRS sub-sampled output produced at SDS Land PEATE and verified at LaRC by CERES DMT**
**3/13/09: Instrument subsystem code needed to support NCT3 complete and tested by DMT**
**5/15/09: Instrument subsystem code delivered to ASDC, tested, and promoted to operations**
**08/13/09: Instrument subsystem code needed to support NCT4 complete and tested by DMT**
**Interim delivery to correct NCT3 problems and prepare for NCT4**
**10/15/09: Instrument subsystem code delivered to ASDC, tested, and promoted to operations**
**11/19/09: Instrument subsystem code needed to support launch complete and tested by DMT**
** ****1/15/10: Instrument subsystem code delivered to ASDC, tested, and promoted to operations**

## CERES CARS Build Schedule

**CERES Data Management System **
**Projected Build promotion dates to ASDC Operations:**
**Build 1:  NCT3 Functionality****	****-****	****May 2009**
**Build 2:  NCT4 Functionality****	****-****	****October 2009**
**Build 3:  Launch-Ready System****	****-****	****January 2010 **

## ASDC Implementation Approach

**ASDC Implementation Approach**
**Michael Little**

## ASDC Supports CERES PI in Ingest, Archive, Distribution, Production

**Ingesting and Archiving inputs for all Data Products**
- Use by Production in Producing Data Products
- Use by Science Team in analysis and algorithm refinement
- Long-term stewardship of data including migration across media changes
- Moving from Tape archive with Disk Cache to Disk archive with Tape Backup
**Distribute CERES Data Products**
- External Customers order data products via ASDC Order Tool, ESDIS ECHO
- ASDC Order Tool provides interim support
- ESDIS ECHO provides access to all EOS data products
- Subscription Services
- Plan to supply CLASS with subscription service as an offsite backup
- Internal users via file sharing
- Moving from Request/deliver model to online Fiberchannel/NFS
**Produce Data Products for CERES PI**
- Accept and test Production codes from DMT
- Accept Production Requests from DMT
- Run codes consistently to produce expected data products
- Now by procedure; expect automation within one year

## ASDC Evolution Approach

**Provide CERES Production Services at ASDC**
- Meet constraints of CERES PGEs
- Avoid increasing workload on DM/Science workforce
- Support FORTRAN, Ada, and IDL languages
- Meet increasing memory and disk access requirements
- Use lessons learned to improve CERES production
- Improve I/O bandwidth limitations to allow multiple instances of some subsystems
- Reduce unplanned outages by increasing storage reliability
- Reduce dependence on active tape archive to improve labor intensive staging processes
- Increase production capacity for FM-5 and reprocessing Terra/Aqua
- Replace current hardware at end-of-life
**Provide Access to CERES Data Products for Internal Users**
- Production
- LaRC Science Community
- CERES Data Management Team
**Improve Order Throughput for External Customers**

## ASDC & SCF Integrated ArchitectureFiberchannel Connectivity

- 256 GB

**ASDC & SCF Integrated Architecture****Fiberchannel Connectivity**

- Fiberchannel Switch

- 64 x86 SMP
- 256 GB

- 56 P6
- 256GB

- 56 P6
- 256GB

- 56 P6
- 256GB

- 56 P6
- 256GB

- 220TB
- DS4800

- 220TB
- DS4800

- 220TB
- DS4800

- 220TB
- DS4800

- 220TB
- DS4800

- 64 x86 SMP
- 128 GB

- 16 x86 HS12
- 64 GB

- Common
- SANhead
- (NFS)
- (AFP)
- (Samba)

- ASDC JBOSS
- Server
- & TS-1
- & Ingest

- ASDC SDP Production

- 64 x86 SMP
- 256 GB

- 56 P6
- 256GB

- SCF Cluster
- (Access through
- Sun Grid Engine)

- 220TB
- DS4800

- 220TB
- DS4800

- SCF Disk Array

- Instrument

- Instrument

- ERBE-like

- SRB/Power

- SRB/Power

- SARB

- 14 4P6@8GB

- SCF Interactive Processors

- ASDC Disk Archive
- (R/W Production)
- (R/O SCF)

- Consodine

- TISA

- SOFA

- Clouds

- Clouds

- TISA

- 220TB
- DS4800

- WWW

- WWW

- 64 x86 SMP
- 128 GB

- Cloud Modeling
- Cluster (Xu)

- 64 x86 SMP
- 128 GB

- ECS DUE
- Processing

- Db
- Server

- Db
- Server

- Tesla

- Tesla

- 3755

- 3755

- GPU Experiment (YongHu)

- Sim
- Server

- FC
- Monitor

- LB1

- LB1

## ASDC & SCF TCP/IP Architecture

- 220TB
- DS4800

**ASDC & SCF TCP/IP Architecture**

- LaRCNet Switch

- 220TB
- DS4800

- 220TB
- DS4800

- 220TB
- DS4800

- 220TB
- DS4800

- 4x16 x86 SMP
- 128 GB

- SH1
- x3650

- Common SANhead
- (NFS, AFP, Samba)

- ASDC JBOSS Server & TS-1

- ASDC SDP Production (SGE)

- SCF Cluster (SGE)

- 220TB
- DS4800

- 220TB
- DS4800

- SCF Disk Array (R/W)

- Instrument

- Instrument

- ERBE-like

- SRB/Power

- SRB/Power

- SARB

- 14 4P6@8GB

- SCF Interactive Processors

- ASDC Disk Archive (R/W Production)(R/O SCF)

- SARB

- TISA

- SOFA

- Clouds

- Clouds

- TISA

- 220TB
- DS4800

- WWW

- WWW

- 64 x86 SMP
- 128 GB

- Cloud
- Modeling
- Cluster

- 64 x86 SMP
- 128 GB

- ECS DUE Processing

- ANGe
- Db
- Servers

- ANGe
- Db
- Servers

- Tesla

- Tesla

- 3755

- 3755

- GPU Experiment

- Sim
- Server

- Load
- 2

- Magneto

- 56 P6 256GB

- 56 P6 256GB

- 56 P6 256GB

- 56 P6 256GB

- SH2
- x3650

- 192.168.16.x

- FC
- Monitor

- ASDC SGE Head

- SCF SGE Head

- Load
- 1

- LaRCNet 100Mbps Ethernet
- Private Net (DS Mgt)
- Private Net (GPFS)
- Cluster & Mgt Net (SGE)
- Green Indicates SubSys Function

**Legend**

- WWW Proxy
- Servers

- Cluster Mon
- (HW TBD)

- 64 x86 SMP
- 256 GB

- 64 x86 SMP
- 256 GB

- 64 x86 SMP
- 256 GB

- 56 P6
- 256GB

- Zamboanga

- Ingest Drop Box

- Center Firewall

**NISN WAN**

- IBM Director
- x3650

## ASDC Ingest Processes for NPP

- To be
- Resolved

- Metadata
- For Orderable
- Data Products

**ASDC Ingest Processes for NPP**

**ANGe**
**Drop-Box**

**ASDC**
**Archive**

**ANGe**
**Core db**

**IMS**
**db**

**ASDC Ingest Processor**

- Metadata

- Location
- & GUID

- Subscription
- Files

- PDR

- Files

**ASDC**
**Delivery**
**Server**

- Files

- PDR

- Files

- PDR

- Files

**ECHO**
**Metadata**
**Adapter**

**NOAA**
**CLASS**

## NPP Impacts to the ASDC

| Impact | Resolution |
| --- | --- |
| Increased storage capacity and performance needed for new CERES data set | Additional disks purchased with fiberchannel connectivity to production processors.
Higher density tapes expand capacity of tape archives. |
| Increased processing capability needed for new CERES data set | Additional processors purchased and added to Linux cluster increasing processing capacity.
Faster data access from new disk archive vs. tape archive. |
| Additional CERES data production requirements effect on staffing | Improvements in staging/ingest/archive reliability will offset staff workload for production.
Staff experienced in processing CERES data products. |
| Operating procedures required for new data set and experience of staff  | Job production using on existing Subsystem DMT Ops Manuals requiring minor updates for NPP.
Staff experienced in processing CERES data products. |
| Ability of facility to accommodate additional hardware with respect to space, HVAC and power | ASDC has space/HVAC for expansion.
Power upgrades 09/08/08 |
| Increased data load impact on network throughput | New architecture will improve internal data production network throughput and access by SCF users
May need additional bandwidth to Land PEATE for VIIRS |
| User access to current and future CERES data products | New architecture will speed ordering CERES data products
New data products will be available through current ordering tools when released to public |

## Ingest/Archive Sizing Estimates

**ASDC can accommodate expected CERES data volumes from the Land PEATE**

| CERES RDRs: Science, Diagnostic, and Telemetry
                                    File sizes are 1370.8 KB, 685.5 KB and 27.5 KB; excluding HDF overhead, about 700 KB/file
	Estimate:  (1370.8 + 700) KB/file * 131 files/day + 685.5 KB/file * 1 file/day+ 27.5 KB/file * 12 files/day
                                            = 272.2903 MB/day |
| --- |
| Sub-sampled VIIRS radiance and geolocation data:
	Output File Size: Approximately 400 MB per 5-min file
	Estimate:     400 MB/file * 288 files/day = 115,200.0 MB/day |
| VIIRS Aerosol data:
	File Size:  ~ 632 MB per 5-min file, as of June 1, 2008
                                             57 MB per 5-min file if changing to 6 x 6 aggregation for both Land and Ocean             	                     32 MB per 5-min file if changing to 8 x 8 aggregation for both Land and Ocean.
	Estimate:	   ~12 files per hour or ~288 files/day * 632 MB/file = 182,016 MB/day
		   ~12 files per hour or ~288 files/day *   57 MB/file =   16,416 MB/day
	Note:  The Land PEATE expects to switch to 6x6, or possibly 8x8, aggregation prior to launch |
| Total Volume - Large Aerosol Files   =  297,488.29 GB/day  108,583.226 GB/yr   or   108.58 TB/yr
Total Volume - Small Aerosol Files	 =  131.888.29 GB/day    48,139.226 GB/yr   or    48.14 TB/yr |

## Security Compliance

**ASDC Approval to Operate Based on August, 2007 Review**
- System Security Plan: SC-010-M-LRC-1000
- Approving Official: Steve Jurczyk, Deputy Center Director
- System Owner: John Kusterer, ASDC Manager
- System Security Manager: Michael Little, ASDC Systems Engineer
- Re-certification in progress based on CIO mandated schedule re-alignment
- Re-certification based on ANGe/IBM installation to occur next spring
- Include consolidation of SCF and ASDC Systems into single System
**IT Security Issues**
- Reconciliation of OCIO Mandated ODIN conversion with ASDC ITS needs
- ODIN to take over support of workstations, desktops, etc
- ASDC to retain control/responsibility for servers
- Implementation of unannounced ITS measures by OCIO sometimes interferes with ingest and delivery over network
- Agency CIO mandated transfer of all Center Firewalls to NISN is not fully understood

## CERES Production Code Test & Evaluation

**Proven Processes Based on 10 Years of Experience in CERES Code Delivery,****Acceptance, Testing, Evaluation and Production **
**CERES DMT Develops Code for Production of Data Products**
- PGE(s) for each Sub-system (SS) tested and internally verified
- DMT and ASDC develop test cases
- Delivery to ASDC includes sample data, Operators Manual, code
- Delivery Evaluation and Code Compilation by DMT Configuration Management comparing to sample data products and using test cases
- Delivery to ASDC for SS Integration and Test to evaluate producibility and completeness and comparison to sample data products
- ASDC and DMT work together to identify and resolve any problems through Software Trouble Tickets
- Promotion to Production status and ASDC CM when approved by Operations Readiness Review (ORR)
**Data Product Production Validation Testing**
- Once code is in production, sample data products (ValRx) are produced with appropriate key months
- ValRx data products are evaluated by Science Team and approved prior to release into an orderable status
- Unsuitable results may result in re-delivery
- Release also requires a Quality Summary by the SS Science Lead

## ASDC Validation Testing Approach

**ASDC Validation Testing Approach**

**Systems delivered to the ASDC undergo Validation testing in a controlled environment**

**DMT Subsystems  under CERES CM delivered to the ASDC for Validation Testing**

**Delivered Subsystems inspected for completeness by ASDC CM, unique testing scenarios** **identified**

**Subsystems validated in operations testing environment for all new requirements and enhancements**

**Delivered subsystems regression tested for existing functionality**

**Operations Readiness Review (ORR) following successful Validation Testing and/or mitigation of existing issues**

**Problem reports generated and tracked for all identified problems**

**Problem report resolutions incorporated into configured software for redelivery (if needed) to the ASDC for validation**

**Data**
**Production**
**and**
**Operations**

**CERES CM**

**ASDC CM**

**ASDC**
**Validation**
**Testing**

**ASDC CCB**
**And Project**
**Management**

**ASDC**
**Regression**
**Testing**

**ASDC**
**Operations**

***CERES FM-5 Testing to Operations Life-Cycle ***

***Problems ***

***Resolutions ***

## NPP-Level Testing and Status

**NPP-Level Testing and Status**

**Testing of the RDRs requires the SD3E to access data from the IDPS, the RDRs are then accessed by the Land PEATE and sent to ASDC**
- ASDC will ingest the RDR data and acknowledge receipt
- ASDC will test data production when available after 05/09
**Testing of VIIRS data requires Land PEATE to run sub-sampler and send sub-sampled radiance/geolocation and aerosol data to the ASDC**
- Land PEATE must be able to generate or access valid VIIRS data
- ASDC will ingest data and acknowledge receipt
**Ingest capability will be ready to support Early EEO**
**Ingest and data production capability will be ready to support NCT3**
**Interim delivery to address NCT3 problems to support NCT4 success **
**All NPP-related functionality available at L-90 days for Launch **

**Notes:**
I’m assuming these are part of the ASDC slides and this is where the transition to Mike Little occurs

## CERES ASDC RDR Tests

**CERES ASDC RDR Tests**

**Objective**
- Verify that the Land PEATE can access CERES RDRs from the SD3E and successfully transfer the data to the ASDC for ingest and archive
**Description**
- SD3E reads CERES RDR data from IDPS or through alternative source
- Land PEATE accesses RDRs from the SD3E and transfers data to the ASDC
- ASDC ingests RDRs and acknowledges receipt back to Land PEATE
- ASDC archives data for staging and processing by Instrument Subsystem
- ASDC will test data production when systems and RDRs available after 05/09
- Additional reorder and data management functions will also be tested
**Input**
- CERES RDRs generated by IDPS or from Proxy data provided to Land PEATE
**Output**
- CERES RDRs ingested and archived at ASDC
- CERES Level 0 data for instrument subsystem processing available after 05/09
**Participants**
- ASDC, Land PEATE, SD3E, CERES DMT
- IDPS when CERES RDR capability is available

**Notes:**
I’m assuming these are part of the ASDC slides and this is where the transition to Mike Little occurs

## CERES ASDC VIIRS Tests

**CERES ASDC VIIRS Tests**

**Objective**
- Verify execution of the VIIRS Sub-sampler by the Land PEATE and the transfer of the sub-sampled radiance/geolocation and aerosol data to the ASDC
**Description**
- SD3E reads VIIRS data from IDPS or through alternative source
- Land PEATE must be able to generate or access valid VIIRS data
- Land PEATE executes VIIRS Sub-sampler and sends sub-sample to ASDC
- ASDC ingests sub-sampled data and acknowledges receipt to the Land PEATE
- ASDC archives data for staging and processing by Clouds subsystem
- Sub-sampled and aerosol data examined by science and DMT staff
- ASDC will test data production when systems are available after 05/09
- Aerosol data ingest and archive tested when available from the Land PEATE
**Input**
- VIIRS data generated by IDPS or from Proxy data provided by Land PEATE
**Output**
- VIIRS sub-samples generated by the Land PEATE
- VIIRS aerosol data product generated by the Land PEATE
**Participants**
- ASDC, Land PEATE, SD3E, CERES DMT, IDPS

**Notes:**
I’m assuming these are part of the ASDC slides and this is where the transition to Mike Little occurs

## Operational Support

**System Upgrades will Enhance Current Capabilities**
- CERES production system already runs at the ASDC
- ANGe storage management upgrade should be operational this week
- Hardware upgrades to be phased in with existing CERES Ingest, Archive, Distribution and Production capabilities in Dec09, reducing risk to NPP needs
**Procedures and Documentation are available and tested**
- Operations staff with CERES processing experience already in place with defined, proven procedures
- ASDC staff already work closely with Land PEATE staff through interactions supporting MODIS data transfers provided from the same data center (MODAPS)
**System availability will remain at 24x7**
- Current production model has minimized need for 24x7 staffing
- System hardware and architecture evolution will increase system reliability and reduce workforce requirements
**Improved system performance and additional CERES data products will lead to increased customer satisfaction**

## Documentation and Agreements

**Most documentation and agreements already exist as part of CERES on TRMM, Terra, and Aqua**
- http://science.larc.nasa.gov/ceres/docs.html
- Documents may be updated if needed
- New documents may be written explicitly for NPP, to meet special requirements
**Overall roles and responsibilities are documented in CERES Science and Data Products Working Agreement**
- Working agreement between CERES IT and NPP Project Office
- May need to be revisited when NPP transfers to NOAA leadership
- Working Agreement in hands of NPP SEWG
**Operations Agreement (OA) being developed between ASDC and Land PEATE**
- Based on existing MODAPS OA for MODIS data sent to ASDC
- On-going relationship between MODAPS and ASDC for MODIS
**No OA planned for ADS/CLASS interface**
- Expect to use standard customer subscription agreement

## ASDC Supports NPP Schedule

**08/31/08****	****Initial Operational Capability (IOC): CERES in ANGe**
**09/15/08****	****Complete Installation/Configuration of IBM hardware**
**Sep-Nov08****	****Test/Characterize IBM Installation**
**11/13/08****	****Initial Delivery of CERES RDR to ASDC Ingest**
**Nov08****	****Complete LaTIS conversion into ANGe**
**Nov/Dec08****	****Design/Develop/Test NPP Inst, VIIRS into ANGe**
**11/26-12/23/08****	****Complete CERES RDR and VIIRS ICD**
**12/01/08****	****IOC IBM ASDC+SCF Archive and Processing System**
**Jan/Feb09****	****Debug Aerosols/VIIRS Sub-sample into ANGe with Land PEATE**
**05/08/09****	****SDS Test2:  Instrument RDRs into ANGe from Land PEATE**
**05/15/09****	****Build 1 ORR:  NCT3 Functionality**
**06/09-15/09****	****NCT3**
**07/01/09-02/02/10****	****Design/Develop/Test Other NPP into ANGe**
**10/15/09****	****Build 2 ORR:  NCT4 Functionality/NCT3 Problem Corrections**
**12/02-09/09****	****SDS Test3/NCT4**
**01/15/10****	****Build 3 ORR:  Launch-Ready System**
**02/02/10****	****Instrument/VIIRS IOC**
**03/02/10****	****MRR**
**06/02/10****	****NPP Launch**

## Overall Schedule

**CERES Schedule is baselined**
**CERES Schedule incorporated into NPP Schedule**
**Schedule available as separate handout**

## Resource Assessment

**Required resources have already been assessed and hired**
**Current experienced CERES staff leveraged for CERES FM-5**
**CERES FM-5 staff augmented with key hires to address technical challenges**
- Major SSAI job fair resulted in hundreds of resumes
**Backfill positions created in current CERES support filled with new talent**
**Resource estimate based on approximate lines of code to be generated**
**Using industry productivity standards, CERES CARS has the resources in place needed to do the job**
**Additional resources needed for NPP testing, processes, and working groups not included in industry standards**
**Additional issues and requirements expected**
**Ongoing CERES projects supported by SSAI can provide short-term support if critical needs arise**

## Risk Analysis

| Raw Data 
Record 
(RDR) 
Formats Not
Fully
Defined | Given that CERES Raw Data Record (RDR) formats are not fully defined or understood in time, there is a possibility the code needed to support testing may not be complete. | 2 | 1 | Technical
Schedule | Erika 
Geier | M | 05/29/08  

The CERES team
is developing a
giver/receiver list
and this item is on the list.
07/23/08  

IPO/Raytheon and
Janet Smith all
Know about this risk. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Unavailability 
of CERES 
Raw Data 
Record 
(RDR) Test
Data | Given that CERES Raw Data Record (RDR) test data from IDPS is not available prior to testing, there is a possibility the code needed to support testing may not be complete. | 3 | 1 | Technical
Schedule | Erika
Geier | W | 07/23/08  

We have requested
test data; although,
it doesn't sound
like we'll get
meaningful test data. |

**Title****	****	****Description****	****	****L     C      Impact    Owner  Action        Comments**

- Legend:  L – Likelihood        C – Consequence           M – Mitigate            W - Watch

## CERES Issues/Concerns

**Need single sample of the Science, Diagnostic, Housekeeping/Telemetry RDRs to verify that we understand format no later than 10/15/08**
- CERES data content may be proxy, sample, or fill data
- Prefer attitude/ephemeris not be fill data
**Desire 3 days (~ 28 consecutive hours) of CERES RDRs to run through preprocessor and Instrument Subsystem to aid in code development no later than 11/14/08**