# Presentation: 548866

## ICOADS: Data, Products, and Access

- Steven J Worley
- Scott D. Woodruff
- Zaihua Ji
- Richard W. Reynolds

## Outline

- New reference document
- Status of Release 2.2
- Access

## New Reference Document

- ICOADS Release 2.1 Data and Products
- Worley et al.* ,Int. J. Climatol.* **25**: 823-842 (2005)
- Special Issue: Advances in Marine Climatology
- Copies are available
- Release History
- 1st, Release 1, 1985 coverage 1854-1979
- 6th (last), Release 2.1, 2003 coverage 1784-2002
- Data Composition
- 185 million records
- Emphasize record tracking - DCK (Table II), SID (Table III)
- Development Combination
- Delayed mode (DM) processing - GTS plus other sources
- Real-time (RT) mode processing - exclusively GTS

**Notes:**
60-80 entries each for DCK and SID 
NODC WOD = DCK 780, and SIDs = 48,49 (WOA94), 88,89 (WOD98)

## New Reference Document

- How DM and RT Compare?
- Use overlap period 1997  1998

- Bars = DM
- Lines = RT

- DM increases records for ships, moored buoys, and C-Man

- DM decreases the number of drifting buoy observation.  MEDS, composites and QA checks.

**Notes:**
60-80 entries each for DCK and SID 
NODC WOD = DCK 780, and SIDs = 48,49 (WOA94), 88,89 (WOD98)

## New Reference Document

- Data Usage Considerations
- Changing observing systems - temporally

- decreasing ships since 1989
- increasing moored and drifting buoys

**Notes:**
Increasing buoys and decreasing ships (1989)

## New Reference Document

- Data Usage Considerations
- Changing observing systems, spatially

- 45% of possible 2° boxes have 5 samples/month in the 70-80’s

- Decreasing low counts due to reduced ships observations

- Increasing high counts due to high frequency sampling from buoys

**Notes:**
Percentage of ocean area sampled, monthly, ranked by number of samples in 2 degree boxes

## New Reference Document

- Data Usage Considerations
- Other changes
- Instrument types and sampling procedures
- Codes for measurements and visual observations
- Instrument package placement on platforms
- Sampling frequency and time of day
- More ...

**Notes:**
Increasing buoys and decreasing ships
Early period all ships and notable gaps

## Status of Release 2.2

- Focus time period, 1998 - 2004
- Reprocessing of 1998-2002 from Release 2.1
- Add on two years, 2003-2004
- Processing Status
- Observations
- Complete and available from NCAR  (12 October)
- Further analysis of result to be done
- Don’t expect a rerun, but ....
- Monthly Summary Statistics (MSG)
- Prepare after the observations are fixed (~ 1-2 months)
- LMRF (old binary format) will be generated for Release 2.2
- We need LMRF to compute MSG, will transition later
- Some past users may want this also - access by special request
- Warning: LMRF will be discontinued

## Status of Release 2.2

- Availability
- First from NCAR
- All products
- Following soon after from ESRL
- The former NOAA/CDC is now part of the NOAA Earth System Research Laboratory (ESRL)
- Only Monthly Summary Statistics in netCDF format
- Future at NCDC
- Observations; retrospective and near real-time

## Status of Release 2.2

- Data Composition
- New DM Data, 1998-2004
- TAO/TRITON, PIRATA, JAMSTEC Moored Buoy Research Archives
- Including RH » DPT
- NDBC Moored Buoy Data
- DPT in DM and frequency not known in RT stream
- MEDS Drifting Buoy QC’d Archive
- WMO Pub 47 metadata **(1973 - 2004)**
- Thanks to Liz Kent and Dave Berry
- Kent, et al,  2005: WMO Pub. No. 47 Metadata and an Assessment of Observation Heights in ICOADS. *Submitted JAOT*
- http://www.soc.soton.ac.uk/JRD/MET/eck_pubs.php

## Status of Release 2.2

- Data Composition
- Carry forward RT Data, 1998-2002
- NCEP GTS with DM sources filtered out
- New RT Data, 2003-2004
- NCEP GTS with filtered

## Status of Release 2.2

- Challenges and Misc. Notes
- Meshing DM tropical moored buoys
- Sampling frequency, hourly and 10-minute
- 10-minute reduced to hourly
- Clean insert with accurate exclusion of GTS
- Daily resolution and matching WMO numbers
- Removed buoys with erroneous ID=SHIP from NCEP ship GTS
- Eight month period (10/99 - 5/2000)
- Fill some gaps in MEDS with NCEP buoy GTS
- Made some QA(QC) flag fixes for full period of record
- Minor errors transitioning from LMRF to IMMA format
- Further checking will be done and documentation prepared
- Many details are not told here
- www.cdc.noaa.gov/coads/e-doc/other/<dupelim_1998>
- Ask questions offline

## Status of Release 2.2

- Moored buoy coverage
- NDBC (approximate)
- PMEL/NOAA (TAO and PIRATA)
- JAMSTEC
- Note: new array beginning in Indian Ocean
- European buoys not shown

## Status of Release 2.2

- Resource issue for US ICOADS team
- Current personnel

| Person Time Commitment | Role | Name  | Organization |
| --- | --- | --- | --- |
| .8 | Coordinator | Scott Woodruff | NOAA/ESRL |
| .8 | Programmer | Sandy Lubker | NOAA/ESRL |
| .1 | Coordinator | Steven Worley | NSF/NCAR |
| .3 | Programmer | Zaihua (Hua) Ji | NSF/NCAR |
| .1 | Science Advisor | Dick Reynolds | NOAA/NCDC |
| .1 | Marine Operations |  | NOAA/NCDC |
| 2.2 | Total |  |  |
| Bad News: FY05 lost 20% of programmer @ESRL |  |  |  |
| Good News: 
 Received international contributions (e.g. Kent et al, metadata)
 NCDC/CDMP (Elms) supporting data and metadata rescue |  |  |  |

## Access

- Links to all data and information
-  www.cdc.noaa.gov/coads  (under Data and Metadata)
- See Table VI (Worley et al. 2005)

| Product | Format | Partner | Subset | Comments |
| --- | --- | --- | --- | --- |
| Observations: |  |  |  |  |
| Release 2.2 | ASCII  | NCAR | t,s,v | Binary (LMRF) by request |
| Release 2.1 | ASCII  | NCDC | t,s | Currently not available |
| Auxiliary  | ASCII, other | NCAR | none | GUI (IMMA), and datasets |
| NRT | ASCII | ESRL | none | NCEP GTS, min. variables, update cycle end of month + 5 days |
| Monthly Statistics: |  |  |  |  |
| Release 2.1 | binary, ASCII | NCAR | t,s,v | GUI and bulk download |
| Release 2.1 | netCDF, ASCII, other | ESRL | t,s,v | GUI or LAS |
| NRT | netCDF, ASCII, other | ESRL | t,s,v | GUI or LAS, min. variables |

- t,s,v = subsetting by time, space, and variable

## Access

- User profiles from NCAR for the past 14 Months
- Who is using ICOADS?

## Access

- User profiles from NCAR for the past 14 months
- What quantities of ICOADS are requested?

## Access

- User profiles from NCAR for the past 14 months
- What time domains of ICOADS are requested?

## Access

- User profiles from NCAR for the past 14 months
- What space domains of ICOADS are requested?

## Access

- Review IMMA structure

- IMMA “core”,  basic data, 48 fields

- Mandatory Record Segment

- Optional Record Attachments

- ICOADS, historical ID (e.g. DCK, SID), QA(QC) flags, etc

- IMMT-2/FM 13, non-core fields from IMMT and FM-13

- Model QC,   e.g. UK model comparison with VOSCLIM

- Ship Metadata,  WMO Pub. 47 ship metadata

- Historical (proposed, primarily for pre-1854 data)

- Supplemental Data – original data in original format

- Recommended Attachment

- Others (e.g. moored/drifting buoys?)

- In use

- To be activated

## Access

- New interface to IMMA observations @ NCAR
- Put online about two weeks ago
- Verify and improve linkages throughout ICOADS portal
- Builds on previous successes
- Provide ASCII output
-  Previously from LMRF (binary) now IMMA
- All data are available for bulk file download (FTP, Web)
- Users select time and space domains
- User select how QA(QC) flags are applied
- Added services
- Access to all attachment data
- Auxiliary data sources

## Access

- Highlights from Interface
- Filtering application

## Access

- Highlights from Interface
- Core variable selection

## Access

- Highlights from Interface
- Attachment data – e.g. ship metadata

## Access

- Highlights from Interface
- Auxiliary Data – two types
- New, likely unique
- Replacement, destine to eventually replace other data

- New

- Replacement

- Note: Can switch off ICOADS request
-  Just acquire auxiliary data
-  If you provide IMMA data we can serve it
-  Incorporate in ICOADS later

## Access

- New interface - Level of automation
- Request submission detected automatically by daemon, every 2 minutes
- Processing begins immediately, with max. processes = 3
- Processing may take several hours for long temporal requests
- IMMA is not efficient for processing speed
- Users are notified by email and download via Web or FTP

## Access

- Sample with ship metadata from WMO Pub 47

-  Type of Barometer
-  Aneroid (AN)
-  Digital Aneroid (DA)
-  Length of Vessel (m)
-  Depth of SST  measurements (m)
-  Height of anemometer (m)

## Access

- Accessories to go with the data
- IMMA documentation
- Sample FORTRAN code created dynamically to match data requested
- Release ID
- Variable definitions

## Access

- Current strategy
- Delayed mode processing
- File based
- Future possibility
- DB with core, attachments, position, date
- Enable near real-time processing
- Processing, O(minutes)
- Push or pull data delivery
- Could be easily updated (monthly) with GTS

## Conclusions

- New reference document with many details about ICOADS (Worley et al. 2005)
- Release 2.2, 1998-2004 (plus other fixes), observations are released to the public.
- Monthly Summary Statistics will be computed soon
- Broaden access in different forms from the partners soon
- Data services are going well (judged from NCAR), especially internationally.
- IMMA permits more comprehensive access; to attachment and auxiliary data
- Supporting resources in the US are thin.  International collaboration has been helpful and more is welcomed.  Project recognition from the science community can help sustain support for ICOADS

## Questions?

- http://www.cdc.noaa.gov/coads/