# Presentation: 844369

## Algorithms Testing at NGST

**Merit Shoucri & Bruce Hauss**
**NGST System Engineering**
**Models & Simulations Team**
**NPP Science Team Kick-off Meeting**
**Annapolis, MD**
**4-6 November 2003**

*Abbreviated Version*

## Content

**NGST - System Engineering ’s Work with the Science Algorithms**
**Testing the Algorithms**
**Integrated Weather Products TestBed (IWPTB) **
**Monitoring Science during Sci-to-Ops Conversion**
**Areas of Potential Collaboration with the NPP Science Team**

## NGST Is Required to Verify Algorithms Performance

**NGST System Engineering is responsible for verifying (before  launch) that the delivered science algorithms **_**meet, or are on the path to meeting**_**, the SSPR NPOESS/NPP System Specification (Appendix D) **
**There exists a structured process of accepting, testing, verifying performance and delivering algorithms for operational conversion (at IDPS)**
- Science algorithms (SDR and EDR) are delivered from subcontractors
- Algorithm Theoretical Basis doc. (ATBD), code and software documentation are evaluated for correctness and self-consistency
- Algorithms are tested against a performance verification matrix, results are documented
- Any algorithm errors (theoretical basis, coding or documentation), tuning or improvements needed to meet requirements are attended to, and complete updates are implemented
- An initial Operational Algorithm Description Document (OAD) is generated and a package containing the above and sample test datasets is delivered to IDPS for operational code conversion
- All algorithms are under configuration management and an Algorithms CCB

## Algorithm Testing Approach -Wicket 1 and Wicket 2 Definitions

**The concept of Wicket 1 and Wicket 2 testing is an internal NGST-SE definition for its testing procedures in order to assess maturity of algorithms delivered by sensor subcontractors, set criteria for acceptance and performance verification**
**Wicket 1 test process:**
- SE tests code with subcontractor data.  Do products meet the Subcontor’s own requirements? – If yes, Wicket 1 is passed
- If Wicket 1 is not passed, SE sends algorithms back to subcontractor
- Wicket 1 Endpoint:  SE/PL officially approve algorithm delivery from subcontractor
**Wicket 2 test process:**
- SE expands test data if needed and possible
- SE determines if products meet NGST System Spec requirements using NGST test data
- If yes, ACCB directs delivery of algorithms to IDPS (in effect, Wicket 2 is passed)
- If no, potential algorithm rework commences until algorithm performance criteria are met
- Wicket 2 Endpoint: SE officially states that requirements have been met
**Code Stable Delivery:**
- If algorithm is on path to meeting System Spec., SE can drop algorithm to IDPS as a “code stable” delivery while other algorithm-related tasks are attended to

## NGST has Already Tested and Delivered to IDPS Seven Algorithms

| Algorithm | Wicket 1 Data Sets Provided by Vendor | Wicket 2 Data Sets Used by NGST SE |
| --- | --- | --- |
| VCM | 1 MODIS Granule (1x1354x2016 pixels)
s10_DAY_MID_LAT_SPRING
No truth data provided
Stratifications included:
All daytime | 4 MODIS Granules (4x1354x2016 pixels)
MOD_2001_196_2240
MOD_2001_213_1210
MOD_2001_213_1220
MOD_2002_032_1750
Truth data by manual analysis
Stratifications included:
All daytime
3 Global Data Set Granules (3x684x168)
TD1_v27A (1330 Orbit)
TD1_v27B (1730 Orbit)
TD1_v27C (2130 Orbit)
Truth data by simulation
Stratifications included:
All except coastal daytime |

| Algorithm | Wicket 1 Data Sets Provided by Vendor | Wicket 2 Data Sets Used by NGST SE |
| --- | --- | --- |
| SST | 1 MODIS Granule (1x1354x2016 pixels)
Florida_Feb_19_2001
No truth data provided
Stratifications included:
All daytime | 3 Global Data Set Granules (3x684x168 pixels)
TD1_v27A (1330 Orbit)
TD1_v27B (1730 Orbit)
TD1_v27C (2130 Orbit)
Truth data by simulation
Stratifications included:
All
Each data set was analyzed once using no sensor noise, then using 3 different sensor noise models representing nominal conditions, and then finally using the three sensor noise models under off nominal conditions.  In total 21 test cases covering over 7 million pixels around the globe. |

| Algorithm | Wicket 1 Data Sets Provided by Vendor | Wicket 2 Data Sets Used by NGST SE |
| --- | --- | --- |
| NCC | 1 MODIS Granule (1x1354x2030 pixels)
VIIRS_EV_DNB_Imagery
Quality determined by inspection
Stratifications included:
Terminator | 12 MODIS Granules (12x1354x2030 pixels)
VIIRS_Imagery_EDR_1420_v3
VIIRS_Imagery_EDR_2001213_1210D_v5
VIIRS_Imagery_EDR_2001213_1210N_v8
VIIRS_Imagery_EDR_2001213_1220D_v6
VIIRS_Imagery_EDR_2001213_1220N_v9
VIIRS_Imagery_EDR_2002032_17500D_v7
VIIRS_Imagery_EDR_2002032_1750N_v10
VIIRS_Imagery_EDR_2235_v4
VIIRS_Imagery_EDR_250_v0
VIIRS_Imagery_EDR_515_v1
VIIRS_Imagery_EDR_610_v2
VIIRS_Imagery_EDR_fixed
Quality determined by inspection
Stratifications included:
Terminator, day, and night |

| Algorithm | Wicket 1 Data Sets Provided by Vendor | Wicket 2 Data Sets Used by NGST SE |
| --- | --- | --- |
| ACO | 1 MODIS Granule (1x1354x2016 pixels)
Florida_Feb_19_2001
No truth data provided
Stratifications included:
Carribean and Mid-latitude Atlantic | 1 MODIS Granules (1x1354x2016 pixels)
MOD_2001_213_1220
Truth data from MODIS
Stratifications included:
Tropical Atlantic
1 Global Data Set Granule (1x684x168)
TD1_v27A (1330 Orbit)
Truth data by simulation
Stratifications included:
All |

| Algorithm | Wicket 1 Data Sets Provided by Vendor | Wicket 2 Data Sets Used by NGST SE |
| --- | --- | --- |
| OC | 1 MODIS Granule (1x1354x2016 pixels)
Florida_Feb_19_2001
No truth data provided
Stratifications included:
Carribean and Mid-latitude Atlantic | 2 MODIS Granules (2x1354x2016 pixels)
MOD_2000_129_1554
MOD_2001_213_1220
Truth data from MODIS
Stratifications included:
Mid-latitude Atlantic 
Tropical Atlantic |

| Algorithm | Wicket 1 Data Sets Provided by Vendor | Wicket 2 Data Sets Used by NGST SE |
| --- | --- | --- |
| VI | 1 MODIS Granule (1x1354x2016 pixels)
LandUnitTest
No truth data provided
Stratifications included:
Western United States and Mexico | 3 Global Data Set Granules (684x168 pixels)
TD1_v27A (1330 Orbit)
TD1_v27B (1730 Orbit)
TD1_v27C (2130 Orbit)
Truth data by simulation
Stratifications included:
All
Each data set was analyzed once with no sensor noise and once with representative sensor noise.  In total 6 test cases comprising 689,472 pixels.
Analysis of effects of band-to-band registration also carried out and documented in the test results |

**“****Assembly line” process for**
**Science algorithm testing**

## Example of Algorithm Testing  Using Simulated Test Data Sets

| Algorithm | Wicket 1 Data Sets Provided by Algorithm Subcontractor | Drop Criteria/Wicket 2 Data Sets Used by NGST SE |
| --- | --- | --- |
| SST | 1 MODIS Granule (1x1354x2016 pixels)
Florida_Feb_19_2001
No truth data provided
Stratifications included:
All daytime | 3 Global Data Set Granules (3x684x168 pixels)
TD1_v27A (1330 Orbit)
TD1_v27B (1730 Orbit)
TD1_v27C (2130 Orbit)
Truth data by simulation
Stratifications included:
All
Each data set was analyzed once using no sensor noise, then using 3 different sensor noise models representing nominal conditions, and then finally using the three sensor noise models under off nominal conditions.  In total 21 test cases covering over 6 million pixels around the globe. |

## NGST Assessment of SST EDR Performance

_**“**__**Favorable” Conditions**_
**Within 40****° of nadir**
**271 K – 285 K**
**AOT < 0.2, **_**and**_
**Confidently clear in 3x3; no cirrus**

_**“**__**Unfavorable” Conditions**_
**> 40****° from nadir**
**271 K – 313 K**
**AOT > 0.2, **_**or**_
**Not confidently clear in 3x3; or cirrus**

**Will stratify precision spec for favorable/unfavorable conditions**
**Internal algorithm breakpoints across SST range are likely source of error**
**Expanding and improving testing**
**Possible upgrades identified; results from testing will determine appropriate approach**

- Requirement - Uncertainty

- Requirement* - Precision

- Requirements not met

- 0

- 0.1

- 0.2

- 0.3

- 0.4

- 0.5

- 0.6

- 270

- 275

- 280

- 285

**Temp (K)**

**Error (K)**

- Uncertainty

- Precision

- Precision requirement not met*

- 0

- 0.1

- 0.2

- 0.3

- 0.4

- 0.5

- 0.6

- 270

- 280

- 290

- 300

**Temp (K)**

**Error (K)**

- Uncertainty

- Precision

- Requirement - Uncertainty

- Requirement* - Precision

## Example of Algorithm Testing  Using Proxy or Real Test Data Sets

| Algorithm | Wicket 1 Data Sets Provided by Algorithm Subcontractor | Drop Criteria/Wicket 2 Data Sets Used by NGST SE |
| --- | --- | --- |
| NCC | 1 MODIS Granule (1x1354x2030 pixels)
VIIRS_EV_DNB_Imagery
Quality determined by inspection
Stratifications included:
Terminator | 12 MODIS Granules (12x1354x2030 pixels)
VIIRS_Imagery_EDR_1420_v3
VIIRS_Imagery_EDR_2001213_1210D_v5
VIIRS_Imagery_EDR_2001213_1210N_v8
VIIRS_Imagery_EDR_2001213_1220D_v6
VIIRS_Imagery_EDR_2001213_1220N_v9
VIIRS_Imagery_EDR_2002032_17500D_v7
VIIRS_Imagery_EDR_2002032_1750N_v10
VIIRS_Imagery_EDR_2235_v4
VIIRS_Imagery_EDR_250_v0
VIIRS_Imagery_EDR_515_v1
VIIRS_Imagery_EDR_610_v2
VIIRS_Imagery_EDR_fixed
Quality determined by inspection
Stratifications included:
Terminator, day, and night |

## VIIRS Imagery Input - MODIS Scene (Terminator Condition)

- 500

- 1000

- 1500

- 2000

- 200

- 400

- 600

- 800

- 1000

- 1200

- 1400

- 1600

- 1800

- 2000

- W/cm2/sr

**Map Projection: Space Oblique Mercator**

## EDR Performance Analysis - AVTP

**Version 2.3 of the CrIMSS EDR algorithm uses ATMS channels and demonstrates that we meet the performance specifications**
**We are utilizing CrIS proxy data generated from real AIRS measurements (Joel Susskind) to test the algorithms in preparation for processing real-world data**

- EDR Code

- Simulated Scenes
- NCEP Profiles
- IR and MW Emissivities
- Cloud Model
- Etc.

- Ground Truth

- Spacecraft Effects
- Jitter
- Pointing Accuracy
- Etc.

- Sensor Performance
- Noise
- Line-shape
- Co-registration
- Etc.

- Computed Profiles

- Forward Model

- SDR Generator

- Proxy (AIRS) Scenes
- Mapped to simulate CrIS

## CrIMSS EDR Performance - Clear Conditions - NGST Assessment

| System-Level TPMs | IORD | System Spec | CBE Performance |
| --- | --- | --- | --- |
| AVTP. Uncert. (Clear, Surface-300mb) | 1.6K | 0.9K | 0.65K |
| AVTP Uncert. (Clear, 300-30mb) | 1.5K | 0.98K | 0.54K |
| AVTP Uncert. (Clear, 30-1mb) | 1.5K | 1.45K | 0.62K |
| AVTP Uncert. (Clear, 1-0.3mb) | 3.5K | 3.5K | 1.03K |
| AVMP Uncert. (Clear, Surf. –600mb) | 20% or 0.2g/kg | 14.1% | 7.1% |
| AVMP Uncert. (Clear, 600-300mb) | 35% or 0.1g/kg | 13.8% | 9.3% |
| AVMP Uncert. (Clear, 300-100mb) | 35% or 0.1g/kg | 11.7% | 20.9% |

**Worst Case Altitude Bin**
**Version 2.3 CrIMSS Algorithms**
**NGST Global dataset, training data independent from test data **

## CrIMSS EDR Performance - Uncertainty - NGST Assessment

## NPP Science Algorithm Drop Schedule

## Simulated and Proxy Data Used to Test and Verify Algorithm Performance

**Simulated data used:**
- To assess performance over a broad range of environmental conditions, typical and extreme
- To isolate and assess the impact/ effects of different sources of error
- To assess impacts of sensor test & characterization results
**Proxy data used:**
- To assess performance of EDR retrieval algorithms under real-world phenomenology and vagaries found in actual sensor/spacecraft data
- To provide sensor data for scenes that cannot be easily simulated
- To produce large volumes of data for fully exercising the operational data processing system

**Simulated data is based on ****an end-to-end simulation of the system, employing****:**
- A compilation of global/regional environmental scene datasets
- Validated radiative transfer models
- Rigorous models of the sensors and spacecraft platform
**Proxy data uses ****SDRs from a heritage sensor to emulate NPP sensor SDRs and requires****:**
- Calibrated radiance data (SDRs) from a heritage sensor with similar characteristics to the NPP sensor
- A validated model to map values of the radiance from the heritage sensor to those of the NPP sensor
- A validated source of “truth” EDR data or best available information

## IWPTB to Assess EDR Performance Based on Simulated Sensor Data

## EDR Retrieval Accuracy Assessment Approach Based on Proxy Sensor Data

**Map to Development Sensor**

**Emulated Radiance**

**EDRs**

**Data Match-up**

**Surrogate Sensor SDR**

**Metrics, Plots, Graphics**

**to IDPS**

**Operational Retrieval Code**

**Sensor Variables**

**Science RetrievalCode**

**Elements from Sensor Contractors**

**Heritage Sensor SDR, EDR & Telemetry**

**In-situ data & NWP GDAS**

**Ancillary Data**

**“****Truth” EDR Value**

## Existing MODIS Proxy Datasets Cover a Range  of Geographic Locations and Times

- MODIS L1B Data (SDR) and Associated Products (EDR “truth”) collected for seasonal diversity:

## Providing Ground-Truth for MODIS Proxy Data

**Whenever possible, insitu data is used to provide “truth” values for comparison with retrieved VIIRS EDRs**
**An alternative approach has been to use MODIS L2 swath products as “truth” values for comparison to retrieved VIIRS EDRs which are based on the corresponding L1B product as input**
- Approach cannot be used for VIIRS EDRs for which there is no corresponding MODIS product (e.g., MODIS does not have a sea-ice age product)
- Should be restricted to only those MODIS products that have been validated to Stage 2 or 3, although sanity checks of VIIRS performance using Stage 1 validated products may be allowable
- Will require obtaining a better understanding of the validation status of each of the MODIS L2 products and the associated retrieval uncertainties
- Plan is to review available product validation reports and to contact individual MODIS product validation teams to obtain additional information
**To be useful for verification of VIIRS algorithms, the corresponding MODIS products must have uncertainties comparable to the NPP/NPOESS requirement**

## Status of Radiative Transfer Models Used in Simulations

| NPOESS Sensor | Sensor Contractor RTM | IWPTB RTM | STATUS of IWPTB RTMs |
| --- | --- | --- | --- |
| VIIRS | MODTRAN3.7
UCLA LBL-RTM
6S | MODTRAN4V1R2
UCLA LBL-RTM
6S | MODTRAN is a community standard model with known fidelity and limitations
Ocean BRDF model has been implemented in MODTRAN
Augmenting our capabilities with UCLA RTM & 6S
Investigating limitations of different models to correctly capture phenomenology |
| CrIS/
ATMS | AER-OSS
(AMSU is ATMS surrogate) | AER-OSS /
NGST- ARMSS | IWTPB testing performance as part of EDR retrieval performance and IDPS tests
Examining validation using AER LBLRTM |
| CMIS | AER CMIS-OSS | NGST-ARMSS | Components compared with published results for other MW models
End-to-end RT compared with field experiments on previous NGST programs
Sanity check with simulated SSM/I and AMSU data
Ongoing validation effort using SSM/I and TMI data |

## NGST-SE Keeps Watch on Science Content of Operational Algorithms 

**SE Science Algorithm team has the following additional responsibilities:**
- Monitors science content of delivered algorithms to IDPS during Sci-to-Ops conversion
- Verifies developed operational algorithm performance during pre-launch System Testing
- Produces and regenerates LUTs during pre-launch phase
- Supports Cal/Val effort and other post-launch science needs

***Sci-to-Ops Conversion Process***

**S1**
**Rehost**

**S5**

**S2**
**IPO**

**S3**
**Error**
**Handling**

**S4**
**Optimize**

**EOB **
**Test**

**EDR Char. Test**

**TR - full**

**OAD  Redline**

**Peer Review**

**TR - sub**

**OAD  Redline**

**Peer Review**

**TR - full**

**TR - sub**

**OAD  SDRL**
**(+30 days)**

**TR - full**

**OAD  SDRL**
**(+30 days)**

**Graceful**
**Degradation**

## Supplemental Algorithm Work Captured as Post-drop Follow-up Tasks

**Sample supplemental tasks required to complete “Code Stable” deliveries**

## Areas of Potential Cooperation with the NPP Science Team

**NGST has two major challenges in order to meet its algorithm delivery schedule and not impact NPP launch schedule**
- Acquire validated test datasets to train regression coefficients and verify algorithm performance
- Complete supplemental post-delivery tasks necessary to “fix” identified problems with the algorithm: LUT, training regression coefficients, missing code unit, missing ATBD information, ...
**NPP Science team can help in one of many ways:**
- Provide answer to questions: Expert Advice
- Provide verified/validated datasets
- Provide direct support to NGST
**US Govt funds both NASA and the SSPR Contractor (NGST/Raytheon team)**
- This should ease transfer of data from NPP Science Team to NGST and vice-versa
- Need to establish protocol for such exchanges