# Presentation: 546158

## Critical Design ReviewThe Blended TPW Products December 16, 2008

- Presented by:
-  Limin Zhao1, Stan Kidder3 and Peter Keehn4
- Other Contributors: Andy Jones3, John Forsythe3, Jianbin Yang4, Jerry Guo4, Ivan 			Tcherednitchenko4, Aiwu Li 4, John Paquette1 and  Ralph Ferraro2
- 1 NOAA/NESDIS/OSDPD
- 2 NOAA/NESDIS/STAR
- 3 CIRA
- 4 Perot Systems Government Service

## Review Agenda

|   |   |   |
| --- | --- | --- |
| Introduction | 1:00-1:15 | Zhao |
| PDR Report and Actions  | 1:15-1:25 | Zhao/Kidder |
| Project Requirements   | 1:25-1:35 | Zhao/Paquette/Kidder |
| Algorithm Theoretical Basis | 1:35-2:20 | Kidder/Forsythe/Jones |
| System Design | 2:20-3:10 | Keehn/Zhao/Jones/Yang/Guo/ Tcherednitchenko |
| Break | 3:10-3:15 |   |
| Quality Assurance | 3:15-3:25 | Zhao/Kidder |
| Operations Concept | 3:25-3:45 | Zhao/Kidder/ Paquette |
| Risks and Actions | 3:45-3:50 | Zhao/Kidder |
| Summary | 3:50-4:00 | Zhao |
| Discussion | 4:00-4:10 | All |

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Outline

- Introduction
- PDR Report and Actions
- Project Requirements
- Algorithm Theoretical Basis Review
- System Design
- Quality Assurance
- Operations Concept
- Risks and Actions
- Summary

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Introduction

- Presented by
- Limin Zhao
- NOAA/NESDIS/OSDPD

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Contents

- Project Background
- Project Objectives
- Project Stakeholders
- Project Plan and Timeline
- Review Objectives

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Driving Factors
TPW products currently available from various satellites/sensors with different biases due to sensor and algorithm difference, which makes the products hard for use by forecasters
Large artifacts exist when simply overlay observations from different sensors with different time and sample sizes

What Forecaster Need is an Unified Product 
Mitigating the individual biases of various data resource to produce a more meteorologically significant product
Providing a seamless global coverage without gaps to allow the analysis of moisture transition between land and ocean

Existing Capability
Experimental Blended TPW products that merges AMSU, SSM/I, GOES and GPS TPWs into unified products
Used by satellite analysts and weather forecasters for years in helping them on analysis and forecasting of heavy rain and flooding events.

- TPW products currently available from various satellites/sensors with different biases due to sensor and algorithm difference, which makes the products hard for use by forecasters
- Large artifacts exist when simply overlay observations from different sensors with different time and sample sizes
- What Forecaster Need is an Unified Product
- Mitigating the individual biases of various data resource to produce a more meteorologically significant product
- Providing a seamless global coverage without gaps to allow the analysis of moisture transition between land and ocean
- Existing Capability
- Experimental Blended TPW products that merges AMSU, SSM/I, GOES and GPS TPWs into unified products
- Used by satellite analysts and weather forecasters for years in helping them on analysis and forecasting of heavy rain and flooding events.

- Project Background

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Transition the user approved blended TPW products from research into operation
Integrate the blended TPW product system into the ESPC operation environment
Make the products operational available to users
Develop tools for product quality monitoring

Provide the operational POES/GOES/GPS blended TPW regional products within three hours of observation (or 30 minutes of data receipt from SOCC) to NWS:
The blended TPW product over four AWIPS regions (Alaska, Hawaii, Puerto Rico and Super National) in AWIPS format
The TPW percentage of normal over four AWIPS regions in AWIPS format

Provide the operational POES/GOES/GPS blended TPW global product to NESDIS/SAB
The blended TPW product in McIDAS Area format
The TPW percentage of normal in McIDAS Area format

Provide the operational POES/GOES/GPS blended TPW global image product to Science on Sphere.
The blended TPW product over global in PNG/GIF image format

- Integrate the blended TPW product system into the ESPC operation environment
- Make the products operational available to users
- Develop tools for product quality monitoring
- Provide the operational POES/GOES/GPS blended TPW regional products within three hours of observation (or 30 minutes of data receipt from SOCC) to NWS:
- The blended TPW product over four AWIPS regions (Alaska, Hawaii, Puerto Rico and Super National) in AWIPS format
- The TPW percentage of normal over four AWIPS regions in AWIPS format
- Provide the operational POES/GOES/GPS blended TPW global product to NESDIS/SAB
- The blended TPW product in McIDAS Area format
- The TPW percentage of normal in McIDAS Area format
- Provide the operational POES/GOES/GPS blended TPW global image product to Science on Sphere.
- The blended TPW product over global in PNG/GIF image format

- Project Objectives

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Project Stakeholders - Integrated Product Team (IPT)

- IPT Lead:  Limin Zhao and John Paquette (OSDPD)
- IPT Backup Lead: Ralph Ferraro (STAR) and Stan Kidder (CIRA)
- NESDIS team:
- STAR:  John Forsythe (CIRA), Andrew Jones (CIRA) and Ingrid Guch
- OSDPD: Sheldon Kusselson
- OSD:  Tom Schott
- Data Center:  NCDC (Phil Jones)
- Others: Peter Keekn and Ivan Tcherednitchenko (PSGS)
- User team
- Lead: Kevin Schrab/Andy Edman, NWS
- Others: NESDIS/SAB, NOAA/HMT program
- Oversight Panel (OP) lead and product priority:  Bob Kuligowski and Limin Zhao; Priority # 1
- Other OPs involved: ICAPOP

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Project Stakeholders - Customers and Users

- NWS HPC – Hydrometeorological Prediction Center
- NWS WR – Western Region
- NWS SPC – Storm Prediction Center
- NWS NHC – National Hurricane Center
- NWS AWIPS – Weather Forecast Offices
- NWS NAWIPS – National AWIPS
- NWS TPC – Tropical Prediction Center
- NESDIS/SAB – Satellite Analysis Branch
- GDS SOS – Science on Sphere
- FTP users

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Project Stakeholders –Operation and Maintenance

**OSDPD**
- Run the blended TPW product processing system in the ESPC
- Tailoring the products to fit into AWIPS requirements
- Distribute the products to users

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Project Plan - Task and Schedules

**Tasks:**
- Defined in “POES-GOES-GPS blended hydrometeorological products.ppt”
**Key Milestones:**
- Project Milestones
- Preliminary Design Review – July, 2008
- Critical Design Review – Nov, 2008
- Operational Readiness Review – Feb., 2009
- SPSRB Briefing – Feb., 2009
- Operations Commence – Feb., 2009
- User Milestones
- Mar 08: Static test
- July 08: Stream Test
- Jan 09: Operational data available

- Feb 14, 2008

- July 17, 2008

- Oct 16, 2008, routinely

- Dec 16, 2008

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Project Timeline (1/2)

**Mar 08:**** Development Phase Begins**
- Mar 08: IPT Branch Lead informed to begin product development
- Mar 08: Initial Archive Requirements identified
- Mar 08: Quality Monitoring Concept Defined
- Mar 08: Static test data file provided to NWS/AWIPS (OB9.x)
- May 08: Long-term Maintenance Concept Defined
- (July) Jun 08:  Preliminary Design Review
- Jun 08:  Development processing system defined
- (Nov ) Sep 08: Critical Design Review
- Oct 08: Code is prepared for implementation
- Oct 08: Final Archive requirements identified
- Oct 08: Operational and backup processing defined

- July 31, 2008

- Feb 14, 2008

- Dec 16, 2008

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## July 08:  Pre-operational Phase Begins
July 08: Pre-operational product output evaluated & tested
July 08: Final/Streaming product delivered to NWS/AWIPS
Nov 08: Operational and backup processing capabilities in place
Nov 08: Final IT Security Concept Defined
Jan 09: Code transitions to operations; all documentation is complete
Jan 09: Operational and backup capabilities reach ops status
Jan 09: Brief SPSRB capability is ready to operational

Feb 09:  Operational Phase Begins
Feb 09: SPSRB declares product operational
SPSRB Secretaries/manager update the SPSRB product metrics web pages
OSD updates Satellite Products database 

- July 08: Pre-operational product output evaluated & tested
- July 08: Final/Streaming product delivered to NWS/AWIPS
- Nov 08: Operational and backup processing capabilities in place
- Nov 08: Final IT Security Concept Defined
- Jan 09: Code transitions to operations; all documentation is complete
- Jan 09: Operational and backup capabilities reach ops status
- Jan 09: Brief SPSRB capability is ready to operational
**Feb 09:****  Operational Phase Begins**
- Feb 09: SPSRB declares product operational
- SPSRB Secretaries/manager update the SPSRB product metrics web pages
- OSD updates Satellite Products database

- _[Back to Index](slide2.xml)_

- Project Timeline (2/2)

- July 17, 2008

- Jan 31, 2009

- Feb 18, 2009

- Oct 16, 2008

**DPEAS  ported on North in Oct, 2008**

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Review Objectives

- Review the
- PDR Report
- Project Requirements
- Algorithm Theoretical Basis
- System Design and Description
- Quality Assurance Procedures
- Implementation and Operations Concept

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Outline

- Introduction
- PDR Report and Actions
- Project Requirements
- Algorithm Theoretical Basis Review
- System Design
- Quality Assurance
- Operations Concept
- Risks and Actions
- Summary

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## PDR Report and Actions

- Presented by
- Limin Zhao
- NOAA/NESDIS/OSDPD

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Contents

- PDR Report Overview
- Risk Assessment and Mitigation – status changes since PDR
- Responses to Actions from PDR
- CDR Entry Criteria
- CDR Exit Criteria
- PDR Summary

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Preliminary Design Review Report Overview

- This section serves as the Preliminary Design Review Report (PDRR)
- The purposes of the PDRR are:
- To provide documented evidence that the PDR was conducted and closed
- To address actions identified in PDR
- To provide the status change of project risks and associated risk mitigation actions since PDR
- The PDRR includes an updated assessment of one project risk identified at PDR, which has been closed.
- The PDRR includes the responses to 6 non-risk-associated actions requested by the PDR reviewers, which have been all closed.
- The new identified risks are addressed at Section 8 in this CDR.

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Risk Assessment and Mitigation – Status changes since PDR

**Risk:** Nearly one year delay on the porting of DPEAS from PC to the ESPC IBM platform due to the issues on the SATEPS migration.
**Risk Assessment:** High  Low
**Impact:**. The products will most likely not be generated on time from the desired platform to support the NWS operational milestone
**Mitigation: **Products will be delivered from the PC-based system when waiting for the migration be done  The porting to IBM AIX is done, and the project is most likely will be on schedule to deliver the operational products from the required platform.
**Status: **Open  Closed. A different risk is addressed in the CDR.

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Responses to Actions from PDR (1/2)

| Subject/Reviewer | IT Capacity Requirement/ Tom Schott |
| --- | --- |
| Action | Recommend adding a capacity planning slide @CDR to show IT resource needs for this application. Ricky said he has a guide you should consider |
| Responses | We have the application run on AIX now, and the ESPC IT team has conducted the IT assessment for the application. Detail requirements are presented on CDR. |
| Status | Closed |

| Subject/Reviewer | Archive/ Tom Schott |
| --- | --- |
| Action | At CDR, please provide an update on archive. How much are you done locally and where do we stand with CLASS archive? |
| Responses | No archive request from users to archive AWIPS files.  The NWS will archive the AWIPS data files at NCDC.  OSDPD is working with NCDC to see if the global data files can be archived. More details on archive plan is addressed at CDR. |
| Status | Closed |

| Subject/Reviewer | Risk Management/ Tom Schott |
| --- | --- |
| Action/Risk | A risk for your future enhancements for this project is “availability of MIRS improved over land TPW operationally”. You may want to provide an update on this at CDR. |
| Responses | The operational MIRS TPW over land from NOAA-18, MetOp-A are available since March, 2008 at the ESPC, and is available from SSMIS since September, 2008. So this risk is considered closed.  |
| Status | Closed |

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Responses to Actions from PDR (2/2)

| Subject/Reviewer | Schedule Details/Tom Schott |
| --- | --- |
| Action | You should include a schedule for standing up the IBM Linux capability in your overall schedule. |
| Responses | The IBM AIX is eventually chosen for the operational implementation. We update the schedule to include the information in CDR. |
| Status | Closed |

| Subject/Reviewer | Web Monitoring/ Ricky Irving |
| --- | --- |
| Action | Will CIRA provide the website (code) for us to install in ESPC? Are there any other monitoring tools other than visual respect of images? |
| Responses | CIRA has delivered the current Web tool for OSDPD to implement in the ESPC. OSDPD will develop tools for system health and product timeliness monitoring. The CDR provides more details on the monitoring procedures.  |
| Status | Closed |

| Subject/Reviewer | Science Review/ Tom Schott |
| --- | --- |
| Action | Should have a few slides that explain the science approach to develop the blended TPW |
| Responses | We have include a section for Algorithm Theoretical Basis Review in CDR. |
| Status | Closed |

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## CDR Entry Criteria

- PDR Risks and Actions Report
- Review of CDR for the blended TPW Products
- Requirements
- Algorithm Theoretical Basis
- Software Architecture
- Quality Assurance
- Operations Concept

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## CDR Exit Criteria

- Critical Design Review Report
- The CDR Report (CDRR) will be compiled after the CDR
- The report will contain:
- CDR presentation
- Actions
- Comments

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## PDR Report – Summary

- PDR Report – Summary

**This PDR report has closed the Blended TPW products PDR: **
- 1 Risk has been updated and closed with closure criteria.
- 6 non-risk-associated actions have been taken and these requests are now closed.
- 2 CDR Entry criteria have been established
- 1 CDR Exit criterion has been established

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Outline

- Introduction
- PDR Report and Actions
- Project Requirements
- Algorithm Theoretical Basis Review
- System Design
- Quality Assurance
- Operations Concept
- Risks and Actions
- Summary

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Project Requirements

- Presented by
- Limin Zhao, Stan Kidder and John Paquette

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Contents

- Project Requirements
- Products Requirements
- AWIPS Operational Requirements
- IT Capacity Requirements
- Dissemination Requirements

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Project Requirements

- SPSRB
- #9902-19 POES/DMSP Blended Products: #0708-0023, POES-GOES Blended Hydrometeorological Products
- Summary: Generate operational blended TPW and RR products from multi-satellites, including POES, MetOp, DMSP, GOES and GPS. ..., etc.
- Others
- NWS Operations and Services Improvement Process (OSIP) 06-084
- NWS OB9.x
- User requests from various WFOs
- NOAA’s Hydrometeorological Testbed (HMT)
- NOAA’s Scientific Data Stewardship (SDS) program
- NOAA Mission Goal supported:
- Ecosystems, Climate, Weather and Water.
- Mission priority:
- Mission Critical - Cannot meet operational mission objectives without this requirement.

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

**Document Title: **POES-GOES Blended Hydrometeorological Products
**Product specifications / needs: **
- Environmental Parameters: Total Precipitable Water (TPW), percentage of TPW normal
- Satellite Sources: POES/AMSU, MetOp/AMSU, DMSP/SSMI, GPS, GOES Sounder
- Geographical Coverage: global
- Horizontal resolution: 16 km
- Measurement Accuracy: greater of 3mm or 10%
**Data Latency: **0.5 ~ 3.0 hrs.
**Refresh:  **per 6 hours (3 hours if possible).
**Timeliness **(e.g., how soon after observation is product needed?): 30 mins

- Products Requirement

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## AWIPS Operational Requirements

**Product specifications: **
- Environmental Parameters: Total Precipitable Water (TPW), percentage of TPW normal
- Satellite Sources: POES/AMSU, MetOp/AMSU, DMSP/SSMI, GPS, GOES Sounder
- Geographical Coverage:  four AWIPS regions over Alaska, Hawaii, Puerto Rico and Super National
- Horizontal resolution: 16 km
- Measurement Accuracy: greater of 3mm or 10%
**Data Latency****: **0.5 ~ 3.0 hrs.
**Refresh****:  **per 6 hours ( 3 hours if possible).
**Timeliness**** : **30 mins
**OB9.0 Schedule**
- Static Test data: March 2008
- Stream Test Data: July 2008
- Operational Data: January 2009

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## IT Capacity Requirements

- OS Requirements (IBM AIX and Dell Linux)
- Runs on Windows at CIRA
- Ported from Windows 2003 Server platform to the IBM AIX platforms at ESPC
- Dell Linux for the QC/VAL processing
- Memory/CPU Requirements
- Memory (Max: 2048 MB, Typical is ~1536 MB)
- CPU Requirements (1 CPU)
- Storage Requirements
- Total per run cycle (~ 7 GB)
- Source Code (~ 40 MB)
- Executable Code (~ 20 MB)
- Timing Requirements
- Run Frequency (every 1 hour)
- Run Duration (variable duration, based on available inputs, typical is ~20 minutes)
- Key Additional IT Requirements
- Network connectivity to Servers (DDS, SATEPSDIST4/ADDE, SATEPSDIST1, GINI, etc.)
- Standard ESPC security procedures (secure shell, sftp)

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Dissemination Requirements

**NWS AWIPS Users**
- Data Format: AWIPS
- Data Type: 4 AWIPS Sectors
- Sever: GINI
- Access: FTP push through ACFN
**NESDIS/SAB and NWS/NAWIPS Users**
- Data Format: McIDAS
- Data Type: Global
- Sever: ADDE
- Access: McIDAS commands
**Science on Sphere User**
- Data Format: PNG/GIF
- Data Type: Global
- Sever: FTP
- Access: FTP Pull
**Others (including CLASS/NCDC)**
- Data Format: HDF-EOS
- Data Type: Global
- Sever: DDS
- Access: FTP Pull/Push

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Outline

- Introduction
- PDR Report and Actions
- Project Requirements
- Algorithm Theoretical Basis Review
- System Design
- Quality Assurance
- Operations Concept
- Risks and Actions
- Summary

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Algorithm Theoretical Basisfor theBlended TPW Products

- Presented by
- Stan Kidder, John Forsythe, and Andy Jones
- CIRA

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Contents

- The Need for Blended Products
- The Algorithms
- Blended TPW Algorithms
- Over-Ocean Algorithm
- Over-Land Algorithm
- Percent of Normal TPW Algorithm
- Implementation Issues
- Product Validation and QC
- For More Information

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## The Need for Blended Products

- “Forecasters today are faced with many sources of data. What they need are meteorologically significant data fields blended from all available data sources, not many maps of observations from individual sources”
- —Kidder, S. Q., and A. S. Jones, 2007: A blended satellite total precipitable water product for operational forecasting. *J. Atmos. Oceanic Technol.*, **24**, 74-81.

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## The Need for Blended Products

- If one simply composites data from different sensors on the same map, artifacts can result
- A “smart” blending algorithm is needed to reduce artifacts and thus make a product that is useful to forecasters.

- SSM/I swaths

**TPW (mm)**

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## The Need for Blended Products

- After the blending algorithm is applied, the artifacts are largely eliminated, which leaves the forecaster free to concentrate on the meteorology of the situation.

- SSM/I swaths

**TPW (mm)**

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## The Need for Blended Products

- The forecaster also needs to quickly judge the severity of the situation
- Percent of Normal TPW compares TPW with the NVAP weekly normal TPW. Yellow areas are 200% or more of normal, and are threat areas.

**TPW (mm)**

**Percent of Normal TPW**

**TPW**

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## The Algorithms

- Three algorithms:
- Over-Water Blending
- Over-Land Blending
- Percent of Normal

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Over-Water Blending Algorithm

- Blends AMSU and SSM/I TPW
- Assumes that over a five-day period, each instrument views the same TPW distribution over the global ocean.
- Uses Histogram Matching to make SSM/I TPW look like NOAA 18 AMSU
- Removes scan-angle effects

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Over-Water Blending Algorithm

- A cubic polynomial is fit to the reference CPDF for each scan angle of each instrument
- Note that SSM/I  (blue) tends to be wetter than AMSU (red), especially in the middle range of TPW

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Over-Water Blending Algorithm

- The cubic polynomial adjustment  (black dashed line) largely eliminates the difference between SSM/I and AMSU
- This technique uses four moments of the distribution for the correction, not just a “bias adjustment”

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Over-Water Blending Algorithm

- Each orbit of each satellite is corrected with the histogram matching method and mapped onto the output map.

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Over-Water Blending Algorithm

- The orbits for the 12-hour period prior to runtime are composited by averaging the pixels, or ...

- Average Composite

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Over-Water Blending Algorithm

- The orbits for the 12-hour period prior to runtime are composited by averaging the pixels, or by overlaying newer pixels on top of older pixels

- Overlaid Composite

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Over-Land Blending Algorithm

- Neither the MSPPS AMSU system nor the FNMOC SSM/I system retrieves TPW over land
- MIRS does retrieve TPW over land, and we will be working on assimilating the MIRS TPW data

- No data over land!

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Over-Land Blending Algorithm

- Over land, we “fill the holes” (another type of blending) in the Overlaid TPW product with Barnes-analyzed GPS TPW (first priority), or ...

- GPS TPW data over land

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Over-Land Blending Algorithm

- Over land, we “fill the holes” (another type of blending) in the Overlaid TPW product with Barnes-analyzed GPS TPW (first priority), or with GOES PW data (second priority)

- GOES PW data over land

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Over-Land Blending Algorithm

- After blending the GPS TPW and/or the GOES PW data, we have the final Blended TPW Product, which is a global product with holes over portions of North America filled in.

- The Blended TPW Product

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Percent of Normal Algorithm

- The Percent of Normal TPW Product is produced by dividing the TPW value at each pixel by the NVAP* Weekly Normal TPW
- Brown areas are below normal, blue areas are above normal, and yellow areas are above 200% of normal

- The Percent of Normal TPW Product

- *Randel, D. L., T. H. Vonder Haar, M. A. Ringerud, G. L. Stephens, T. J. Greenwald and C. L. Combs, 1996: A new global water vapor dataset. *Bull. Amer. Meteor. Soc*., **77**, 1233–1246.

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Implementation Issues

- The biggest implementation issue is in timing the runs such that the desired input data are available. This is particularly true of the GPS TPW data, which are available only between 20 and 50 minutes past the hour.

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Product Validation and QC

- These blending algorithms do not fundamentally change the accuracy of the products which are blended.
- Over Water
- Since we currently use the NOAA 18 AMSU TPW as the standard, the accuracy of the blended product is approximately the same as for the MSPPS AMSU TPW, or approximately or 10% RMS (Ralph Ferraro, personal communication; Ferraro et al., 2005: NOAA Operational Hydrological Products Derived From the Advanced Microwave Sounding Unit. *IEEE Trans. Geosci. Remote Sens.,* 43, 1036–1049.]

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Product Validation and QC

- Over Land
- GPS data: ± 0.75–1.5 mm (Seth Gutman, personal communication; Smith et al., 2007: Short-Range Forecast Impact from Assimilation of GPS-IPW Observations into the Rapid Update Cycle. *Mon. Wea. Rev*., **135**, 2914–2930.)
- GOES PW: ± 2 mm (Schmit et al., 2002: Validation and use of GOES Sounder moisture information. *Wea. Forecasting*, **17**, 139–154.)
- No separate validation effort is planned
- QC is accomplished by
- Visually inspecting the images, and
- Responding to user comments

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## For More Information

- See the *Blended TPW Products Algorithm Theoretical Basis Document (ATBD)*

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Outline

- Introduction
- PDR Report and Actions
- Project Requirements
- Algorithm Theoretical Basis Review
- System Design
- Quality Assurance
- Operations Concept
- Risks and Actions
- Summary

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## System Design

- Presented by
- Peter Keehn, Limin Zhao, Andy Jones, Jianbin Yang and Jerry Guo

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Contents

- System Overview
- DPEAS Framework
- Blended TPW Processing
- McIDAS/AWIPS Processings
- QC/VAL Processing

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Product Characteristics:
Blended TPW, percentage of TPW normal
Refresh: every 6 hr, nominally at 00, 06, 12, and 18 UTC, but can be extended to every hour.
Map Projection:  Mercator.
Coverage: global ocean and CONUS, excluding poles (71°N to 71°S)
Resolution: 16 km at equator
Input data:
AMSU and SSM/I TPW are available in the ESPC through MSPPS
NOAA & METOP: ~1.5MB/orbit  * 14 orbits * 5 satellites * 5 days = 525MB (includes intermediate files)
DMSP: ~3.5MB/orbit * 14 orbits * 1 satellites * 5 days = 245MB (includes intermediate files)
GPS TPW data acquired through NOAAPort
~45KB/file * 4 per hour * 24 hours = 4.32MB
GOES PW are available in the ESPC through another operational product generation system (SFOV)
~13MB/file * 2 satellites * 24 hours = 624MB
TPW weekly mean provided by NASA Water Vapor Project (NVAP -  Randel et al. 1996).  Normal period is 1988 – 1999.
Output data:
HDF-EOS – Distributed from DDS to CLASS (Optional) (~ 2*20 MB per run)
McIDAS – Distributed from SATEPSDIST4 through ADDE sever(~2*3.5 MB per run)
AWIPS – Distributed from GINI through AWIPS Network Control Facility( ~ 4*2.0 MB per run)

- Blended TPW, percentage of TPW normal
- Refresh: every 6 hr, nominally at 00, 06, 12, and 18 UTC, but can be extended to every hour.
- Map Projection:  Mercator.
- Coverage: global ocean and CONUS, excluding poles (71°N to 71°S)
- Resolution: 16 km at equator
- Input data:
- AMSU and SSM/I TPW are available in the ESPC through MSPPS
- NOAA & METOP: ~1.5MB/orbit  * 14 orbits * 5 satellites * 5 days = 525MB (includes intermediate files)
- DMSP: ~3.5MB/orbit * 14 orbits * 1 satellites * 5 days = 245MB (includes intermediate files)
- GPS TPW data acquired through NOAAPort
- ~45KB/file * 4 per hour * 24 hours = 4.32MB
- GOES PW are available in the ESPC through another operational product generation system (SFOV)
- ~13MB/file * 2 satellites * 24 hours = 624MB
- TPW weekly mean provided by NASA Water Vapor Project (NVAP -  Randel et al. 1996).  Normal period is 1988 – 1999.
- Output data:
- HDF-EOS – Distributed from DDS to CLASS (Optional) (~ 2*20 MB per run)
- McIDAS – Distributed from SATEPSDIST4 through ADDE sever(~2*3.5 MB per run)
- AWIPS – Distributed from GINI through AWIPS Network Control Facility( ~ 4*2.0 MB per run)

- System Overview

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## System Overview 

- Hardware:
- IBM AIX machines (North, Emerald, Diamond)
- Data servers (DDS, SATEPSDIST4/ADDE, SATEPSDIST1, GINI, etc)
- Software:
- DPEAS/TPW software
- Fortran90, C/C++
- Shell scripts, Perl, Java, JavaScript
- Processing Control and Scheduler (OPUS)
- Toolkits, HDF, HDF-EOS
- Security:
- Follow standard ESPC security procedures
- Use of secure shell and sftp
- Secure access to the systems

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## System Overview

- Use standard OPS directory structure
- /bin
- Contains compiled executables
- /data
- Contains input and output data files.
- /logs
- Contains all log files
- /monitoring
- Contains files related to the monitoring program
- /scripts
- Contains all script files
- /setup
- Contains configuration and setup files
- /src
- Contains all source code
- /work
- Contains temporary working files

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## System Overview

- North/Emerald/Diamond
**DPEAS**

- ESPC Diamond
**AMSU & SSMI TPW**

- ESPC satepsdist1
**GPS & GOES TPW**

**McIDAS **
**Processing**

**AWIPS **
**Processing**

**SOS Image **
**Processing**

- Satepsdist4
**ADDE server**

- Satepsdist4

- Satepsanon (gp16)
**SOS TPW image**

**NAWIPS**

- Satepsdist1
**Product Server**

**GINI**

**AWIPS**

**Web Farm**
**Monitoring**

**DDS**

- Operational Blended TPW System Diagram

**CLASS**

**SAB**

**Users**

**Cyclone**
**QC/VAL**

**Web Farm**
**Monitoring**

- sftp
- push

- sftp
- push

- sftp push

- sftp push

- sftp push

- sftp push

- sftp push

- sftp pull
- (spider)

- direct access

- ANCF/SBN

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## System Overview 

- System Overview

- sftp satepsdist1
- GPS TPW

- AMSU-A
- SSMI TPW

- sftp satepsdist1
- GOES TPW

- DPEAS Framework

- McIDAS
- AWIPS
- Processing

- Blended TPW Products Processing

- Simplified Operational Blended TPW System Diagram

- Product
- Dissemination

- QC/VAL Processing

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## System Overview

- sftp satepsdist1
- GPS TPW

- AMSU-A
- SSMI TPW

- sftp satepsdist1
- GOES TPW

- GPS files are transferred via automated script from SATEPSDIST1 and converted to the proper format.

- GOES 11 and 12 files are transferred via automated script from SATEPSDIST1 and uncompressed.

- AMSU-A and SSMI files are made available through MSPPS on Diamond in HDF-EOS format

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## DPEAS Framework

- DPEAS - Data Processing and Error Analysis System
- Created to overcome the inherent difficulties of working with multiple data formats.
- The software design is centered on the HDF-EOS format.
- It has proven to be highly successful at tackling the cross-sensor data use issues (Jones and Vonder Haar, 2002).
- Runs the blended TPW processing code.

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System - Overview

- DPEAS runs the blended TPW processing code.
- Blended TPW consists of two parts
- Ocean TPW
- AMSU-A (NOAA, METOP)
- SSM/I (DMSP)
- Land TPW
- GPS
- GOES

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System – Over Ocean

- Part 1: Ocean TPW
- Locate latest 5 days worth of AMSU and SSM/I data
- Convert input data format to DPEAS HDF-EOS format
- Create TPW correction
- Apply TPW correction and remap
- Composite corrected data and write output
- GIF
- McIDAS
- HDF-EOS

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System – over Land

- Part 2: Land TPW
- Locate most recent GPS and GOES (11 and 12) data
- Read the TPW composite file generated from the previous processing
- Merge GPS values
- Merge GOES East and West values
- Compute TPW percent
- Write output
- McIDAS
- HDF-EOS

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System -Flow Charts

- Start TPW

- Convert Data

- NOAA/AMSU-A
- DMSP/SSMI
- METOP/AMSU-A

- Create TPW
- correction

- Apply correction
- and remap

- Composite
- corrected data

- End TPW

- HDF-EOS

- ASCII

- HDF-EOS

- HDF-EOS
- McIDAS
- GIF

- Start TPW2

- Get Data

- Merge GPS

- Merge GOES

- Compute
- % TPW

- GPS
- GOES-E
- GOES-W

- Write Output

- HDF-EOS

- HDF-EOS

- End TPW2

- McIDAS

- McIDAS

- HDF-EOS

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System

| Interface Item | Interface Type | Source | Description |
| --- | --- | --- | --- |
| AMSU-A | Input | MSPPS | TPW from NOAA-15,16,17,18 METOP-2 |
| SSMI | Input | MSPPS | TPW from DMSP 13 |
| GPS | Input | NOAAPort | TPW over CONUS |
| GOES | Input | SFOV | TPW over CONUS from GOES 11 and 12 |
| SAB_GLOBAL_Mercator_16km.txt | Input | DPEAS | Map projection parameters (static) |
| GPSMETStation.txt | Input | DPEAS | GPS station locations (static) |
| nvap_weeklyaverage.std | Input | NASA | NVAP weekly normal TPW (static) |

- Input Data

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System

| Interface Item | Interface Type | Source | Description |
| --- | --- | --- | --- |
| Reformatted AMSU and SSMI | Input/Output | DPEAS | HDF-EOS format preferred by DPEAS |
| TPW correction | Input/Output | DPEAS | ASCII file containing correction coefficients |
| Remap | Input/Output | DPEAS | HDF-EOS file containing corrected TPW |
| Merged GPS and GOES | Input/Output | DPEAS | HDF-EOS merged TPW from GPS and GOES |
| Merged GPS and GOES | Output | DPEAS | McIDAS merged TPW from GPS and GOES |
| Composite TPW | Input/Output | DPEAS | HDF-EOS composite ocean TPW |
| Composite TPW | Output | DPEAS | GIF image of composite ocean TPW |
| Composite TPW | Output | DPEAS | McIDAS area file of composite ocean TPW |

- Intermediate Data

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System

| Interface Item | Interface Type | Source | Description |
| --- | --- | --- | --- |
| Composite TPW | Output | DPEAS | HDF-EOS composite TPW |
| Composite TPW | Output | DPEAS | McIDAS area file of composite TPW |
| % of normal TPW | Output | DPEAS | HDF-EOS % of normal TPW |
| % of normal TPW | Output | DPEAS | McIDAS % of normal TPW |

- Output Data

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System – Run Schedule

- get_GPS_data.sh
- Every 30 minutes at :25 and :55 past the hour
- get_GOES_data.sh
- Every hour at :01 past the hour
- run_TPW.sh
- Every 6 hours (00, 06, 12, 18) at :15 past the hour
- clean_up.sh
- Once a day at 02:05

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System -Monitoring Procedures (1/2)

- OPUS
- DPEAS Process Monitor
- Java tool to display status of each processing script

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System - Monitoring Procedures (2/2)

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Blended TPW System - QC/QA Procedures 

- Each process script will...
- contain error checking with operator notification of problems
- produce a log file
- Blended TPW scripts
- get_GPS_data.sh
- Verify data was transferred and exists on local host
- get_GOES_data.sh
- Verify data was transferred and exists on local host
- run_TPW.sh
- Verify 5 days of AMSU-A and SSMI data exist
- Verify GPS and GOES data exist
- Verify TPW part 1 completed successfully
- Verify TPW part 2 completed successfully

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## System Environment
ESPC IBM Linux Partition
CPUs: 3 dual-CPUs processors with 4.2 GHz
Memory: 1.99GB
Input data:
Blended TPW and TPW anomaly push from the blended TPW Processing on Diamond
Output data:
Blended TPW and TPW anomaly with calibration in McIDAS and GIF/PNG image formats 
Distribution
McIDAS files distributed from through ADDE sever to SAB and NAWIPS users
GIF/PNG images distributed through ESPC anonymous FTP to SOS users

- ESPC IBM Linux Partition
- CPUs: 3 dual-CPUs processors with 4.2 GHz
- Memory: 1.99GB
- Input data:
- Blended TPW and TPW anomaly push from the blended TPW Processing on Diamond
- Output data:
- Blended TPW and TPW anomaly with calibration in McIDAS and GIF/PNG image formats
- Distribution
- McIDAS files distributed from through ADDE sever to SAB and NAWIPS users
- GIF/PNG images distributed through ESPC anonymous FTP to SOS users

- McIDAS Processing -  Overview

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## McIDAS Processing – Flow Chart

-    **NAWIPS**

- Add Calibration

***SAB***

- Blended TPW from DPEAS

- New Image

- No

- Stop

- Yes

***SOS image ***
***Generation***

**AWIPS**
** ****Processing**

- ESPC ADDE Server

- ESPC FTP Server

*Invoke Spider *
*Entry*

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## McIDAS Processing – Directory Structure

- Directory Structure of the System
- bin
-       /home/mcidas/bin - C executables
- data
-    	 /home/mcidas/data – Mcidas data files, Spider
- 	 /data2/BLTPW – Output Mcidas AREA files
- logs
-  	/data2/BLTPW/logs – processing status and error messages
- lib
-  	 /home/mcidas/bin – Mcidas functions & subroutines
-       /home/oper/mcidas/bin – Spider functions & subroutines
- src
-       /home/mcidas/bltpw – Source code and scripts for Mcidas processing

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## McIDAS Processing – Run Schedule

- getBLTPW.sh
- Run every 15 minutes at :00, :15, :30, :45 past the hour
- logclean.sh
- Run daily at 05:23Z

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## System Environment
ESPC IBM Linux Partition
CPUs: 3 dual-CPUs processors with 4.2 GHz
Memory: 1.99GB
Input data:
Blended TPW and TPW anomaly products with calibration pull from the McIDAS Processing 
Output data:
Blended TPW and TPW anomaly in AWIPS format over 4 AWIPS regions
Distribution
AWIPS files are first push to the ESPC Products Server 
Pull by GINI
Distributed through ANCF to AWIPS users

- ESPC IBM Linux Partition
- CPUs: 3 dual-CPUs processors with 4.2 GHz
- Memory: 1.99GB
- Input data:
- Blended TPW and TPW anomaly products with calibration pull from the McIDAS Processing
- Output data:
- Blended TPW and TPW anomaly in AWIPS format over 4 AWIPS regions
- Distribution
- AWIPS files are first push to the ESPC Products Server
- Pull by GINI
- Distributed through ANCF to AWIPS users

- AWIPS Processing - Overview

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## AWIPS Processing – Flow Chart

- Generate Remapped AREA files

- AREA files from McIDAS Processing

- Image Updated?

- Create AWIPS Header

- Re-write image data

- Yes

- No

- Stop

- Product Server
- SATEPSDIST1

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## AWIPS Processing – Directory Structure

- Directory Structure of the AWIPS Processing
- src
-        /home/mcidas/poesawips/src - contains C source code and make files
- bin
-         /home/mcidas/poesawips/blend/bin - C executables
-         /home/mcidas/bin – Mcidas commands
- scripts
-        /home/mcidas/poesawips/blend/scripts – shell and Perl scripts
- data
-    	/home/mcidas/data – Mcidas local files
- 	/data/gini/blend – Output AWIPS files
- logs
-  	/home/mcidas/poesawips/blend/logs -- processing status and error messages
- working files
-        /home/mcidas/poesawips/blend/etc – configuration files

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## AWIPS Processing – Run Schedule

- Blend_awips_TPW.sh
- Run every hour at :48 past the hour
- Awips_blend_sftp.pl
- Run every hour at :58 past the hour
- Cleanup.pl
- Run daily at 23:05Z

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## System Environment
Dell Linux
CPUs: 8 dual-CPUs with 3.66 GHz
Memory: ~ 2.0 GB
Input data:
Blended TPW and TPW anomaly push from the blended TPW Processing on Diamond
Blended TPW and TPW anomaly push from the McIDAS/AWIPS Processing on SATEPSDIST4
Output data:
Blended TPW and TPW anomaly images
Distribution
ESPC Intranet
ESPC Internet

- Dell Linux
- CPUs: 8 dual-CPUs with 3.66 GHz
- Memory: ~ 2.0 GB
- Input data:
- Blended TPW and TPW anomaly push from the blended TPW Processing on Diamond
- Blended TPW and TPW anomaly push from the McIDAS/AWIPS Processing on SATEPSDIST4
- Output data:
- Blended TPW and TPW anomaly images
- Distribution
- ESPC Intranet
- ESPC Internet

- QC/VAL Processing - Overview

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## QC/VAL Processing – Flow Chart

- Start QC/VAL

- Get TPW data

- Latest TPW &
-  TPW anomaly

- Create Images

- Make Web Pages

- Send Files to
- Intranet
- & Internet

- End QC/VAL

- TPW in
- HDF-EOS

- PNG/GIF files

- JavaScript

- JavaScript
- PNGs

- Internet
- Server

- Intranet
- Server

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## QC/VAL Processing – Directory Structure

- Use standard OPS directory structure
- /bin
- Contains compiled executables
- /data
- Contains input and output data files
- /logs
- Contains all log files
- /scripts
- Contains all script files
- /setup
- Contains configuration and setup files
- /src
- Contains all source code

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## QC/VAL Processing – Run Schedule

- make_bTPW_image.sh
- Run every hour at :55 past the hour
- cleanup.sh
- Run daily at 00:10Z

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Outline

- Introduction
- PDR Report and Actions
- Project Requirements
- Algorithm Theoretical Basis Review
- System Design
- Quality Assurance
- Operations Concept
- Risks and Actions
- Summary

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Quality Assurance

- Presented by
- Limin Zhao/Stan Kidder
- OSDPD/CIRA

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Contents

- System-Level Quality Control
- Process Quality Assurance
- Product QC Monitoring

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## System-Level Quality Control 

- System-Level Quality Control

- Data Status Check
- Scripts check for existence of all input files before processing.
- Scripts check for existence of all intermediate & output files during/after processing
- Following the pre-set production rule, the processing will either continue or stop with error message be send to the maintenance personnel and/or operator to take proper actions.
- Processing Status Check
- Exit code is returned from each major processing based on the pre-set QC procedures
- The driver scripts check the exit code and trigger the proper error message be send to either maintenance personnel or operators to take proper action
- Job Status Check
- Status of product generation and dissemination are checked, and ensured no hung jobs,  and products are generated and distributed to users in the required time window.
- Built-in QC Procedures:
- The data from each individual product is considered missing if its value is out-of-bounds.
- The data is considered missing if no reliable data from each individual product but no additional QC flag is currently output.

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Process Quality Control

- Configuration Management:
- Operational changes and promotions follow the ESPC standard tracking and verification procedures, and all changes are overseen by the ESPC CCB
- All codes, static files and documents will be base-lined and change controlled under the repository with the Revision Control Software (RCS).
- Promotion follows the EPSC SCM procedure, and goes through
- Unit Testing
- Integration/System Stress Testing
- Operational Implementation
- Debugging and Diagnostics
- Script logs, main processing (algorithm) logs, post-processing logs for detailed investigation and diagnosis of the causes for job failures.
- Recovery of failures or data losses:
- Processing failures are recoverable as long as inputs available, including processing failures, delayed inputs, disk crashes (lost outputs).

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Product Quality Control

- Near-real time web-accessible image are used as the main QC tools
- To visually inspect noticeable bad data or data gaps, and ensure the product looks reasonable from a forecaster’s point of view – SAB analyst plays a major role at this aspect.
- To compare against the individual products used in the blending to ensure the merge product is not degraded.
- Science monitoring at CIRA
- _[http://](http://amsu.cira.colostate.edu/TPW)__[amsu.cira.colostate.edu](http://amsu.cira.colostate.edu/TPW)__[/TPW](http://amsu.cira.colostate.edu/TPW)_
- _[http://](http://amsu.cira.colostate.edu/gpstpw/)__[amsu.cira.colostate.edu/gpstpw](http://amsu.cira.colostate.edu/gpstpw/)__[/](http://amsu.cira.colostate.edu/gpstpw/)_
- Codes for generating the Web pages have been shared with OSDPD for operational monitoring, and the planned web link is
- _[http://www.osdpd.noaa.gov/ml/air/bTPW](http://www.osdpd.noaa.gov/ml/air/bTPW)_

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Slide 95

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Outline

- Introduction
- PDR Report and Actions
- Project Requirements
- Algorithm Theoretical Basis Review
- System Design
- Quality Assurance
- Operations Concept
- Risks and Actions
- Summary

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Operation Concepts

- Presented by
- Limin Zhao
- OSDPD

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Contents

- Operational TPW System Architecture
- Products Generation
- Products Monitoring
- Products Maintenance
- Products Dissemination
- Products Archive

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Operational TPW System Architecture

- North/Emerald/Diamond
**DPEAS**
**(Blended TPW)**

- ESPC Diamond
**AMSU & SSMI TPW**

- ESPC satepsdist1
**GPS & GOES TPW**

**McIDAS **
**Processing**

**AWIPS **
**Processing**

**SOS Image **
**Processing**

- Satepsdist4
**ADDE server**

- Satepsdist4

- Satepsanone
**SOS TPW image**

**NAWIPS**

- Satepsdist1
**Product Server**

**GINI**

**AWIPS**

- Web Farm
**Monitoring**

**DDS**

**CLASS**

**SAB**

**Users**

- Cyclone
**QC/VAL**

- Web Farm
**Monitoring**

- sftp
- push

- sftp
- push

- sftp push

- sftp push

- sftp push

- sftp push

- sftp push

- sftp pull
- (spider)

- direct access

- ANCF/SBN

- The Operational Blended TPW System Diagram

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Products Generation 

- The Operational jobs will be controlled and monitored by the chosen operational job scheduler, such as, OPUS.
- The blended TPW products generation system run on IBM platforms
- With 24/7 support for robust and reliable products

**Status**

**Error Message**

-  AMSU & SSM/I,
- GPS, GOES

- Jobs triggered
- & monitored
-  through OPUS

- TPW
- Products

- Send error message
- to product manager

**Error**

- Drive
- Scripts

- TPW
-   PGS*

- * PGS– Product Generation System

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Products Monitoring

**Multi-level monitoring tools will be set-up and used  at the ESPC operation, including **
- Operators’ monitoring of the job flow and processing status through OPUS GUI interface under 24/7 to ensure the system run normally
- Automated email notification for product generation and distribution anomalies to the PAL and maintenance personnel
- Web-based monitoring of the products generation, timeliness and distribution status (_[https://sandia-intranet.nesdis.noaa.gov/DPEAS/inter/bTPW_monitor](https://sandia-intranet.nesdis.noaa.gov/DPEAS/inter/bTPW_monitor)_)
- Web-based imagery QC monitoring (_[http://www.osdpd.noaa.gov/ml/air/bTPW](http://www.osdpd.noaa.gov/ml/air/bTPW)_)

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Example of Web-based Products Monitoring

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Products Maintenance (1/2)

- Operation Supports
- The production machines are maintained by the ESPC IT team under 24/7
- ESPC help desk monitoring for any job failure, data outage and anomaly
- The primary and backup personnel are identified to cover maintenance needs, and ensure problems be fixed on time
- OM and SMM will be developed to support O&M needs
- The ESPC CM procedures  are enforced for the operational changes and promotions, and any codes and documents changes will be overseen by the ESPC Change Control Board
- Multi-level monitoring capability will be deployed as shown in the previous slide
- Re-processing scripts will be developed to recover any lost of data files without interfering with the co-exist operational processing

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Products Maintenance (2/2)

- Maintenance Supports
- Processing failures due to machines problems, such as, disk storage, CPU overloaded, etc.
- Processing terminates abnormally or is hung for whatever reason, such as, delayed or problematic inputs, bugs, ...etc.
- Products dissemination problems
- Products degradation due to channel/sensor/satellite problems, ...etc.
- Adding and/or deleting the satellite data as its availability changes
- Algorithm updates/enhancements implementation
- The PAL will work with CIRA scientists for any science maintenance needs when identified
- Emergency Recovery Capability
- The blended TPW will be set-up on the CIP (Critical Infrastructure Protection) facility at Wallops, VA.
- Any promotions and changes in the ESPC operation will be made in the CIP at the same time
- The blended TPW products can be continued in the event of an unexpected and sustained loss of service at the NSOF ESPC

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Products Dissemination (1/2)

- The ESPC SATEPSDIST servers will be the staging location for distribution of the blended TPW products in McIDAS and AWIPS formats
- ADDE server
- Push to GINI
- The ESPC Data Distribution Server (DDS)
- CLASS pull for archive
- Push/Pull to/from real-time users for HDF-EOS format
- Current Distribution List
- AWIPS
- NAWIPS
- SAB Precip team
- SOS

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Products Dissemination (2/2)

- The output product will be distributed immediately after being generated through GINI (AWIPS), ADDE (SAB, NAWIPS, etc), DDS (potential CLASS), FTP (SOS images) servers.
- Images of products will be sent to the web for QC monitoring.

- TPW
- Products

**Web/FTP**
**(images)**

**ADDE**

**GINI**

**SAB**

**NAWIPS**

**AWIPS**

**NCDC**

**DDS**

**CLASS**

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Products Archive

- Short Term Archive
- 5 days time retention requirement on the ESPC operational machines
- Storage of about ~ 10 GB is needed with the current configuration, i.e., all inputs and outputs are storage for 5 days.
- Automatic Shell/Perl scripts used to clean up the old files.
- Long Term Archive
- For AWIPS regional data files, the NWS has a special arrangement with NCDC to archive the AWIPS products. No archive requirement for NESDIS.
- OSDPD has taken initiative to work with NCDC to see if the global data files should be archived. Discussion is ongoing.
- Interface is in place to transfer data from ESPC to CLASS/NCDC if need.
- When finalized, a Data Submission Agreement (DSA) will be submitted to NCDC.

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Documentation

- Algorithm Theoretical Basis Document – prepared by CIRA
- System Description Document – prepared by CIRA
- User Manual - prepared by CIRA – prepared by CIRA
- Interface Control Document  - prepared by CIRA
- System Maintenance Manual – will be prepared by OSDPD
- Operations Manual – will be prepared by OSDPD
- AWIPS Operation Requirements – prepared jointly by OSDPD and NWS
- Data Submission Agreement – will be prepared jointly by OSDPD and NCDC *(Phil Jones ) *
- Metadata File - will be prepared jointly by OSDPD and NGDC *(Anna Milan ) *

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Outline

- Introduction
- PDR Report and Actions
- Project Requirements
- Algorithm Theoretical Basis Review
- System Design
- Quality Assurance
- Operations Concept
- Risks and Actions
- Summary

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Risks and Actions

- Presented by
- Limin Zhao/Stan Kidder
- OSDPD/STAR

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Risk Assessment and Mitigations (1)

**Risk 1:** The scheduled project milestone for the operation commence is one month behind of the user required operational milestone.
**Risk assessment:** Low; The AWIPS B9.0 implementation at NWS most likely will be postponed at their end.
**Impact:** Minimal based on NWS users’ feedback.
**Mitigation:** The TPW products have being provided to AWIPS routinely since Oct 2008, and no problem has been reported so far. The Initial Operational Capability (IOC) with 24/7 support can be provided to support users’ milestone requirement in January, 2009 while the IPT is warping-up all other SPSRB requirements for operational, such as, Documentations.
**Status:** Open

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Risk Assessment and Mitigations (2)

**Risk 2:** Different platforms and compilers are used at CIRA and OSDPD.
**Risk assessment:** Low.
**Impact:**** **Potentially on** **R2O transition cost; minimal.
**Mitigation:**** **OSDPD has set-up the baseline system on AIX and identified all system-related changes when porting from PC to AIX. The results have been validated against that from the PC-based system. For any future changes in algorithm, CIRA will coordinate with OSDPD and adopt the incremental delivery method. The delivery memo will clearly state what changes have been made and how to implement them.
**Status:** Open

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Outline

- Introduction
- PDR Report and Actions
- Project Requirements
- Algorithm Theoretical Basis Review
- System Design
- Quality Assurance
- Operations Concept
- Risks and Actions
- Summary

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Summary

- Presented by
- Limin Zhao/Stan Kidder
- OSDPD/CIRA

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Review Objectives Have Been Addressed

- PDR Report have been reviewed
- Project Requirements have been reviewed
- Algorithm Theoretical Basis has been reviewed
- Software system architecture and interfaces have been reviewed
- Plans for quality assurance have been reviewed
- Operation concepts have been reviewed
- Risks and Actions have been reviewed

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Next Steps

- Prepare CDR Report
- Perform the operational readiness review, including code review
- Brief SPSRB the capability is ready to operational

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008

## Open Discussion

**The review is now open for free discussion**

- Critical Design Review: The Blended TPW Products

- Dec 16, 2008