# Presentation: 691215

## INTERGRATING  IN SITU NETWORKS & STANDARDIZING DATA QUALITY ASSURANCE

***INTERGTATED SURFACE DATA PROCESSING SYSTEM***
**Stephen Del Greco**
**National Climatic Data Center**

- NOAA Integrated Surface Observing System (ISOS) Work Group Meeting, Asheville NC, July 20-22, 2004

## MISSION         

- The National Climatic Data Center (NCDC) is mandated by the National Oceanic & Atmospheric Administration (NOAA) to provide the following services for the nation’s meteorological and climatological data:
-       *  Acquisition
-       *  Quality Assessment & Improvement
-       *  Archive
-       *  Dissemination
-       *  Describe the Climate

## HISTORY

- Current In-Situ Networks are processed with Quality Assurance/Control, (QA/QC’d) based on network, not element
- For example; The NOAA Automated Surface Observing System (ASOS), Network Volunteer Cooperative Observers Network (COOP) & Climate Reference Network (CRN) are processed using systems unique to the individual network
- This is also the case for the Global Climate Observing System Network (WMO) as well as other various Meso networks owned and operated by local and state governments or the military
- The global Integrated Surface Hourly dataset, integrated from several sources has a unique QA/QC system.
- Often current final QA/QC is performed when all data for the month are received

## OBJECTIVE

- Today’s climate related issues dictate the need for high quality homogenous data sets, often in near real time
- Towards that end, NCDC is designing new QA/QC processing systems to replace existing systems
- The goal is to move towards fully automated QA/QC validation for hourly/daily/monthly weather data, keeping in mind that interactive QC will take place when situations warrant
- The new system processes data on a daily or hourly basis instead of the current end of month time scale
- Any QA/QC standardization needs collaboration among NOAA line offices, State and Regional Climate Centers (HPRCC, linear regression, FSL mesonets)

## NCDC Climate Services Partnerships

**NCDC Climate Services Partnerships**

**Partnership Cycle**

## OBJECTIVE

- While similar,  QA/QC rules and algorithms for like parameters from different observing networks are not standard
- Current technology provides a means for integrating like data into one standard format and processing these data through standardized QA/QC algorithms & procedures
- NCDC is designing a new QA/QC system - Integrated Surface Data Processing System (ISDPS)
- ISDPS is an end to end system for processing in-situ data where QA/QC is network independent and based on reporting frequency (hourly, daily, etc..)
- ISDPS integrates assessment techniques and links algorithms into one unified system (ISD input/output)

##        IN SITU NETWORKS	

**INGEST **
**       ****IN SITU NETWORKS**

**DATA PROCESSING**
**STOVEPIPE SYSTEMS**

**PRODUCTS **
**NETWORK DEPENDENT**

**CRN**

- ISH

## INGEST 

**INGEST **
**IN SITU NETWORKS**

**DATA PROCESSING**
**INTEGRATED FORMAT**

**INTEGRATED PRODUCTS**

**CRN**

- ISH

##     BENEFITS OF INTEGRATION 

- Reduction of subjectivity and inconsistencies among data sets that span multiple observing networks and platforms
- Standardize QA/QC based on reporting time (QC methodology for hourly temp data independent of network)
- Standardized products are more easily developed
- Collective experience and expertise leads to a better product
- Software is modular for ease of modification
- Conformance of data to documentation (reference manuals, FMH, etc.)

## NCDC CONSISTENCY CHECKS

- Mathematical and meteorological
- Maxima > minima with other values between maximum and minimum
- Physically plausible ranges
- Physically plausible combinations of data
- Extremes (Wakeby probability model, empirical curve fitting, statewide extreme)
- Source-specific rules
- Original vs. original, replacement vs. original, replacement vs. replacement
- Spatial checks (TempVal, PrecipVal)

- NOAA Baseline data sets ASOS, COOP, CRN, ISD

## Model:	

**Model:****	**
**Rapid Update Cycle (RUC)**
**Satellite:****	**
**GOES - GMSRA**
**GMSRA with Night Screen**
**Auto Estimator**
**Hydro Estimator**
**Blended GOES / Microwave **

**PRECIPVAL DATA SOURCES:**
**In-Situ:****	**
**ASOS/AWOS**
**CRN**
**Radar:****	**
**Stage IV****	****	**
**Digital Precipitation Array (DPA)**

## ASOS Hourly QC Algorithms (Examples)

| Element Name: MTR_RAW.Ctemp, MTR_RAW.Ctemp_10  |
| --- |
| Description: Temperature  |
| Field Size:    4,5 digits respectively  |
| Source(s): ASOS high resolution ingest/reformat software and SRRS  |
| 1. Test: Check for blank (missing value) in both elements.  This entry indicates the absence of temperature data due to the inoperability of the temperature sensor |
| Action:	If both fields contain only blanks, allow processing to proceed on to the next field  |
| Flag(s): None |

-  ***Refer to the SCIADS Documentation Manual for QA review for all weather parameters***

**355 Checks**

## ASOS Hourly QC Algorithms (Examples)

| 2. Test: Check ‘Ctemp’ for a possible negative sign (-) in the first position and numeric characters in the remaining positions.  |
| --- |
| Action:	If a valid entry is not found, flag the field as an error and proceed to the next field.  |
| Flag(s): HR026 - Temp invalid |
| 3. Test: Check ‘Ctemp’ to see if the value is between -50 and 50 inclusively  |
| Action:	If the value is outside the expected range, flag the field as an error and proceed to the next test.  |
| Flag(s):HR027 - Temp outside range |

## Cooperative Observers Network

**Error detection daily COOP temps ****include the following rules:**
**Limits Check (rule group 1) – **An extremes table constructed by state and month for maximum and minimum temperatures is compared against.  Limits check failures occur when observation values exceed the table values for the current, previous and subsequent months. Failures may result in deletion from the file
**Internal Consistency Check (rule group 2) **– Tests for internal inconsistencies; For example, the maximum temperature recorded being less than the minimum, or the maximum being less than the minimum of the previous day.  Internal inconsistency edits include swapping maximum and minimum temperatures, assigning the maximum of today equal to the minimum of the previous day or deleting the datum most likely in error.
** **

## Cooperative Observers Network

**Flatliner Temperature Check (rule group 3)   - **a flatliner is defined to be a run of at least five consecutive days of the same maximum or minimum temperatures.  A five day run is flagged as suspect, 10 day run highly suspect and greater than ten days deleted.  Exceptions to the flatliner rule is made for places where day to day temperature variability is low such as the tropics.
**Excessive Diurnal Range Check (rule group 4) – **tests for diurnal ranges exceeding 75 o  temperatures are examined for continuity.  If discontinuities are found temperature variables may be deleted.
**Temperature Spike Check (rule group 6, rule group 5 pertains to liquid/frozen precipitation checks) **– identifies spikes in the temperature time series.  A spike is based on the comparison of the singular differences of three consecutive days.

## Cooperative Observers Network

**Multiple check (rule group 7 & 8) **– The seventh and eighth rule group checks recognize the coexistence of two or more simultaneous rule failures and assign combination code flags accordingly.  For example, a record maximum temperature may be assigned a suspect flag and later be detected as a discontinuity by the spike test.
**Failed fix check (rule group 9) - **Recognizes elements that are subject to an edit; however, fail a second iteration (i.e., attempted fixes that fail the rules test)

## The Quality Control of the Integrated Surface Hourly (ISH) Global Database

- 2 QC phases thus far
- ISH is being renamed ISD – Integrated Surface Data, to reflect its usage in the Integrated Surface Data Processing System, and inclusion of hourly, daily, etc data
**                     ****ISH Quality Control Phase 1**
- Data comparisons as part of merge process to ensure records are for same station-time
- Inventory system to ensure no data loss from beginning to end of process
- Test data to verify all software is working properly
- Random checks of “Version 1” output data
- Reprocessing as necessary

## ISH Quality Control Phase 2

- 57 quality control algorithms (fully automated)
- Validity checks for all data fields
- Extreme value checks to eliminate gross errors
- Temporal continuity checks
-     (2-sided) for elements such as temperature, dew point, wind,
-     & pressure

## ISH Quality Control Phase 2

- Internal consistency checks between elements
- Checks for several known systematic problems
- Test data verification
- Random checks of “Version 2” output data
**                                ****ISH Lessons Learned**
- Thorough test data are critical, and allow for the establishment of baselines for future comparison
- Peer review is important—don’t be an “island”
- Use phased approach—don’t insist on the “big house” initially; start with the foundation
- Have a long-term plan, but be flexible
- Expect to reprocess, but limit its frequency by following good design and testing principles

## FULLFILLING USER NEEDS

_*Unedited*__ __*Local Climatological Data*_ publication replaced with on-line final _*Climatological Data*_ publication dynamically updated daily
- Today’s end user need data that is processed daily instead of the current end of month time scale
- High quality data that in the past were available 45 to 90 days after the end of the month will be available within days after receipt
- Products based on preliminary or unedited data to be replaced with daily QC’d data on line (automated QC)
- ISDPS will provide capability for dissemination of surface data into legacy data formats (ASOS, HPD, COOP, Global daily, hourly)

## FULLFILLING USER NEEDS

- Complete a distributed network for both data and software
- Integrate real-time and historical data so that *any* data can be provided to users within servicing time constraints