# Presentation: 010690

## Observations, PREPBUFR, and the Interface to PREPBUFR1st GSI User Orientation4-6 January 2005Dennis KeyserNOAA/NWS/NCEP/EMC

## Observational Data Pipeline to NCEP

- GTS and aviation circuit bulletins transferred from the NWS Telecommunication Operations Center (TOC/NWSTG) to NCEP's Central Operations (NCO)
- Networked to one of two interactive nodes on NCEP IBM-CCS (one production, one development) using LDM (Local Data Manager) and DBNet (Distributive Brokered Networking) software packages
- Data are then encoded into WMO BUFR format

## Observational Data Pipeline to NCEP (cont.)

- Most of the NESDIS satellite data are processed in batch mode as they become available from the various NESDIS ftp servers
- Regularly scheduled jobs on the IBM-CCS transfer "new" files from these servers and encode the data into WMO BUFR format
- NCEP receives “Level 3” radial wind data from 154 NEXRAD radar stations via the radar multicast (NIDS format)
- At eight minutes past each hour, a script runs to gather up the most recent radar data into eight pairs of files containing the raw data and a station list
- At thirty minutes past each hour these raw data are superobed and  encoded into WMO BUFR format

## Observational Data Pipeline to NCEP (cont.)

- Mesonet surface data are obtained directly from FSL
- FSL pushes data to NCO server
- NCO converts data from netCDF to WMO BUFR format

## BUFR Database on NCEP IBM-CCS

- All of the encoded BUFR data are appended to the appropriate files (“tanks”) in the database
- The files are organized by the WMO BUFR type and local subtype and most contain information in 24 hour blocks (based on report time)
- Observational files remain on-line for up to 10 days before migration to offline cartridges
- While online, there is open access to them for accumulating late arriving observations, for operational and checkout dumping, and for research and study

## Network Data Dumps

- The various NCEP production suite networks access the observational database at set times each day (i.e., the data cutoff time) and perform a time-windowed and, in the case of regional networks, geographically-filtered dump of requested observations
- Observations of a similar type [e.g., satellite-derived winds (“satwnd”), surface land reports (“adpsfc”)] are dumped into individual BUFR files which maintain the original structure of  reports
- some interactive quality control is applied
- Reports on NCEP reject list are flagged
- NESDIS/National Oceanographic Data Center (NODC) q.c. is applied to marine data
- duplicate reports are removed
- upper-air report "parts" are merged.

## Network Data Dumps (cont.)

- Some of the BUFR data dump files are re-processed into new BUFR files such that they can be used properly by subsequent PREPBUFR processing or analysis programs
- SSM/I wind speed, total precipitable water and rainfall products are superobed onto a one degree lat/lon grid (in most networks)
- quikSCAT scatterometer data are q.c.’d and then superobed onto a one-half degree lat/lon grid
- TRMM TMI rainfall product is q.c.’d and then superobed onto a one-degree lat/lon grid
- Additional function of dump step is to q.c. tropical storm position records (used later in PREPBUFR processing)

## Network Data Dumps (cont.)

- Current data dump (cutoff) times:
- ETA (NAM): T+1:11 (00, 12Z), T+0:50 (06, 18Z)
- EDAS (NDAS): 2240/1040 UTC (t-12, t-09, t-06), 2300/1100 UTC (t-03) for 00/12Z; 0520/1720 UTC (t-12, t-09, t-06, t-03) for 06/18Z
- AVN (GFS): T+2:46 (00, 06, 12, 18Z)
- FNL (GDAS): T+6:05 (00Z), T+8:05 (12Z), T+5:50 (06, 18Z)
- RUC (upper-air): T+0:58 (00, 12Z), T+0:26 (all other hours)
- RUC (surface): T+0:05 (early, all hours), T+0:22 (catch-up, all hours)

## “PREPBUFR” Processing

- The final step in preparing most of the observational data for assimilation by the analysis
- Executes a series of programs designed to assemble observations dumped from the BUFR database
- Encodes information about the observational error for each data type as well the background (first guess) interpolated to each data location
- Performs both rudimentary multi-platform quality control and more complex platform-specific quality control
- Stores the output in a monolithic BUFR file, known as PREPBUFR

## “PREPBUFR” Processing (cont.)

-  Tropical cyclone relocation
- A series of programs that relocates one or more tropical cyclone or hurricane vortices in the global sigma first guess file which is read by the analysis

## “PREPBUFR” Processing (cont.)

- PREPDATA program
- Reads in and consolidates observations dumped from individual BUFR databases
- Performs rudimentary checks on the data, and organizes upper-air data by decreasing pressure
- Adds forecast background (first guess) interpolated to each observation location
- Adds observational error (read in from a look-up table) to each observation
- Converts dry bulb temperature to virtual and dewpoint temperature to specific humidity for surface data
- Generates an initial “pre-q.c.” monolithic PREPBUFR file

## “PREPBUFR” Processing (cont.)

- SYNDATA program
- Reads in quality controlled tropical storm position records file valid at the current time and uses them, along with other observations in the PREPBUFR file, to generate synthetic (bogus) wind mandatory level profile reports (throughout the depth of the storm) in the vicinity of the storm(s) to better define tropical systems for the analysis
- Runs only if previous Tropical Cyclone Relocation processing was unable to “locate” a vortex center (i.e., cases where the vortex is weak)

## “PREPBUFR” Processing (cont.)

- CQCBUFR program
- Performs complex quality control on rawinsonde height and temperature data to identify or correct erroneous observations that arise from location, transcription or communications errors
- Attempts to correct commonly occurring types of errors
- Data that cannot be corrected are flagged and will not be considered by the analysis
- The checks used are: hydrostatic, increment, horizontal statistical, vertical statistical, baseline and lapse rate and are based upon differences from the first guess

## “PREPBUFR” Processing (cont.)

- CQCBUFR program  (cont.)
- Also applies intersonde (radiation) corrections to the quality controlled rawinsonde height and temperature data
- The degree of correction is a function of the rawinsonde instrument type, the sun angle and the vertical pressure level.
- Converts rawinsonde and dropwinsonde dry bulb temperature to virtual and rawinsonde and dropwinsonde dewpoint temperature to specific humidity

## “PREPBUFR” Processing (cont.)

- PROFCQC and CQCVAD programs
- Perform complex quality control on wind profiler and on Vertical Azimuth Display (VAD) wind data from WSR-88D radars, respectively in order to identify erroneous data and remove it from consideration by the analysis
- The checks used are: increment, vertical statistical, temporal statistical, and combined vertical-temporal and are based upon differences from the first guess
- CQCVAD includes a bird migration algorithm

## “PREPBUFR” Processing (cont.)

- PREPACQC program
- Performs quality control on conventional AIREP, PIREP and AMDAR (Aircraft Report, Pilot Report, Aircraft Meteorological Data Relay) aircraft wind and temperature data
- The flight tracks are checked, with bad reports flagged and duplicate reports removed
- In addition, AIREP and PIREP reports are quality controlled in two ways:
- isolated reports are compared to the first guess with outliers flagged
- groups of reports in close geographical proximity are inter-compared using both a vertical wind shear check and a temperature lapse check

## “PREPBUFR” Processing (cont.)

- ACARSQC
- Performs quality control on MDCRS ACARS aircraft wind and temperature data
- Currently only performs simple data bounds checks
- PREPACQC and ACARSQC will soon be replaced with a new q.c. module used at FNMOC
- Includes track checking on MDCRS ACARS reports as well as all other aircraft types

## “PREPBUFR” Processing (cont.)

- Structure of the PREPBUFR file
- Reports are grouped into unique BUFR Message Types
- ADPUPA [upper-air (RAOB, PIBAL, recco, drops) reports]
- AIRCAR (MDCRS ACARS aircraft reports)
- AIRCFT (AIREP, PIREP and AMDAR aircraft reports)
- SATWND (satellite-derived wind reports)
- PROFLR (wind profiler reports)
- VADWND [VAD (NEXRAD) wind reports]
- ADPSFC [surface land (synoptic, METAR) reports]
- SFCSHP [surface marine (ship, buoy, C-MAN platform) reports]
- SPSSMI [SSM/I re-prococessed wind speed, tpw, rainfall)
- SYNDAT (synthetic tropical cyclone bogus reports)
- GOESND (NESDIS GOES sounding, retrieval, radiance data)
- QKSWND [quikSCAT re-processed wind speed]

## “PREPBUFR” Processing (cont.)

- Structure of the PREPBUFR file (cont)
- Future BUFR Message types
- MSONET (Mesonet surface reports)
- RASSDA (RASS virtual temperature reports)
- GPSIPW (GPS-Integrated Precipitable Water Reports)
- Individual reports in a BUFR message are further defined by a PREPBUFR report type
- Reports are split into mass and wind pieces
- All mass reports contain report types in the range 100-199
- All wind reports contain report types in the range 200-299
- Observation errors are defined according to report type
- The assimilation uses the PREPBUFR report type to identify the reports in the PREPBUFR file

## “PREPBUFR” Processing (cont.)

- Structure of the PREPBUFR file (cont)
- Most of the observation types in the PREPBUFR file are associated with quality markers
- These are used by the analysis to place a weight on the data
- Each PREPBUFR processing step which changes a datum (either the observation itself, or its quality marker) records the change as an "event” with a program code and a reason code
- Each time an event is stored, the previous events for the datum are "pushed down" in the stack, thus the PREPBUFR file contains a complete history of changes to the data throughout all of the PREPBUFR processing
- The most recent changes are always at the top of the stack and are thus read first by any subsequent data decoder routine

## “PREPBUFR” Processing (cont.)

- Operational Report Types in PREPBUFR file:
- 120/220: ADPUPA - rawinsonde virtual temperature, specific humidity, station pressure / wind
- 130/230: AIRCFT - AIREP and PIREP aircraft sensible temperature / wind
- 131/231: AIRCFT - AMDAR AIRCRAFT sensible temperature / wind
- 132/232: ADPUPA - flight-level reconnaissance and profile dropwinsonde virtual temperature, specific humidity, station pressure / wind
- 133/233: AIRCAR - MDCARS ACARS aircraft sensible temperature / wind
- 150: SPSSMI - SSM/I superobed FNOC rainfall
- 151: GOESND - NESDIS SFOV GOES cloud top pressure
- 152: SPSSMI - SSM/I superobed total precipitable water

## PREPBUFR” Processing (cont.)

- Operational Report Types in PREPBUFR file (cont):
- 156: GOESND - NESDIS GOES 4-layer PW over land – clear
- 174: GOESND - NESDIS GPES brightness temperatures over ocean – clear
- 180/280: SFCSHP - surface marine (ship, buoy, C-MAN) virtual temperature, specific humidity, station pressure / wind
- 181/281: ADPSFC - surface land synoptic and METAR station pressure / wind
- 182: SFCSHP – Dropwinsonde splash level virtual temperature, specific humidity, station pressure
- 183: ADPSFC, SFCSHP - surface marine (ship, buoy, C-MAN), land synoptic and METAR virtual temperature, specific humidity (estimated station pressure not used)
- 187/287: ADPSFC - surface METAR virtual temperature, specific humidity, station pressure (estimated from altimeter) / wind

## “PREPBUFR” Processing (cont.)

- Operational Report Types in PREPBUFR file (cont):
- 210: SYNDAT - synthetic tropical cyclone wind
- 221: ADPUPA - PIBAL wind
- 223: PROFLR - wind profiler
- 224: VADWND - NEXRAD VAD wind
- 242/252: SATWND - JMA IR & visible cloud drift wind below/above 850 mb
- 243/253: SATWND - EUMETSAT IR & visible cloud drift wind below/above 850 mb
- 245: SATWND - NESDIS IR cloud drift wind
- 246: SATWND - NESDIS imager water vapor wind (motion)
- 282: SFCSHP – ATLAS buoy wind
- 283: SPSSMI - SSM/I superobed wind speed
- 285: QKSWND – quikSCAT wind

## Operational data that do not pass through PREPBUFR processing

- BUFR dumps for the following data types are read directly by the analysis
- NESDIS 1B satellite brightness temperatures from TOVS (HIRS-2, MSU) and ATOVS (HIRS-3, AMSU/A, AMSU/B)
- NESDIS solar backscatter ultraviolet radiance (SBUV) data (ozone)
- NASA Tropical Rainfall Measuring Mission (TRMM) rainfall data
- NEXRAD Level 3 radial winds

## BUFR

- International WMO standard
- Efficient means for transferring data
- Flexibility for adding new observation elements
- Table driven
- “Table A” defines the data category associated with a particular BUFR message containing report data
- “Table B” classifies and defines data elements, or descriptors, according to scale, reference value, number of bits and units (includes code and flag tables)
- “Table C” defines data description operators
- “Table D” defines the list of common sequences

## BUFR (cont.)

- A special application has been designed at NCEP to provide user-friendly access to the BUFR files through a series of FORTRAN subroutines in a machine independent BUFR library (i.e., BUFRLIB)
- Allows data to be encoded into or decoded from BUFR using mnemonics to represent the data
- The mnemonics are associated with BUFR descriptors in a special version of the BUFR Tables A, B, C and D
- When a BUFR file is first created, the mnemonic table is read in from an external location and is itself encoded into BUFR messages at the top of the output file
- This allows each BUFR file to be “self defined” (i.e., no external tables are needed to decode data out of the file)

## Simple Example to Interface to PREPBUFR

**CHARACTER*8 SUBSET, STNID**
**REAL*8 HDR(5), OBS(7,255)**
**EQUIVALENCE (STNID,HDR(1))**
**...****.**
**CALL DATELEN(10) ! Y2K change – all dates expressed as 10-digit (YYYYMMDD)**
**! Open BUFR file in unit LUBFR (1****st**** arg) for input (read) (2****nd**** arg) using BUFR table internal to LUBFR (3****rd**** arg)  **
**CALL OPENBF(LUBFR,'IN',LUBFR)**
**! Loop through all of the messages in the PREPBUFR file**
**! SUBSET returns message type (e.g., “ADPUPA”); IDATE returns message date YYYYMMDD (same for all**
**!    messages) **
**DO WHILE (IREADMG(LUBFR,SUBSET,IDATE).EQ.0)**
**! Loop through all reports (subsets) in this BUFR message**
**        ****DO WHILE (IREADSB(LUBFR).EQ.0)**
**! Read header info out of rpt {HDR(x) where x: 1- id (STNID), 2-rpt type, 3-D-time, 4-lon, 5-lat}, ILVL returns as 1**
**           ****CALL UFBINT(LUBFR,HDR,5,1,ILVL,‘SID TYP DHR XOB YOB')**
**! Read level info out of a report  {OBS(x,y) where on level y, x: 1-level category, 2-pressure, 3-temp obs, 4-temp q.m.,**
**!     5-u-wind obs, 6-v-wind obs, 7-wind q.m.}, ILVL returns the number of levels in the report**
**           ****CALL UFBINT(LUBFR,OBS,7,255,ILVL,‘CAT POB TOB TQM UOB VOB WQM')**
**...****.**
**        ****ENDDO**
**ENDDO**
**CALL CLOSBF(LUBFR) ! Close BUFR file**
**...****.**
**STOP**
**END**

## Web Site

- http://www.emc.ncep.noaa.gov/mmb/data_processing/data_processing/