# Presentation: 111863

## Ensemble Data Dissemination – Current and Future Challenges

- 01 Nov 2006
- Brent Gordon
- NCEP Central Operations

## TOPICS

- Current Status of Ensemble Data Availability
- Network Issues Surrounding Ensemble Data
-  What is OUR Road Ahead?

## NCEP Global Ensembles

- Currently produce 75 GB of ensemble data per cycle (300 GB/day)
- Up from 10.5 GB per cycle (42 GB/day) in 2004
- Major Upgrade in May 2006
- Addition of Climate anomaly and bias corrected forecast data products for each member
- 4 new members
- File structure overhauled

## Public Data Locations

- NCEP FTP server (ftpprd.ncep.noaa.gov)
- 10 members + 1 control
- Both GRIB1 and GRIB2 data
- 6 month conversion period to GRIB2 to begin November 8, 2006
- Forecasts only – no anomalies or bias corrected
- 15 GB per cycle

## Public Data Locations

- NWS FTP server (tgftp.nws.noaa.gov)
- 10 members + 1 control
- Only GRIB1 data (GRIB2 data to be available Nov 8)
- Forecasts only – no anomalies or bias corrected
- 7.5 GB per cycle
- Fewer high resolution data sets due to space limitations
- Issue to be corrected soon with a replacement server, however bandwidth limitations remain
- GRIB2 Data to be available on the new public server

## Available on NOAAPORT

- 10 members + 1 control
- Only GRIB1 data Forecasts only – no anomalies or bias corrected
- Only two cycles per day
- 35 MB per cycle
- Soon to be 125 MB with the addition of another grid and an extension to 240 hours.
- Project underway to determine NWS ensemble requirements

## Experimental NOMADS System

- NOMADS servers
- These servers are fed via non-operational data flows from NCEP/EMC.
- Data sets put on that server have been at the discretion of EMC

## Availability Summary

- NCO dataflow has been working hard all year long to determine how to make more ensemble data available.
- We got the May 2006 changes accommodated (up to 10 members)
- We are still struggling with how to move the new products off of our system without major latency.

## Availability Summary

- NCEP network currently saturated during global ensemble delivery times.
- Impacts delivery of other models as well
- RUC, Hurricane, and Wave models
- Any increase in volume of ensemble data dissemination has had a noticeable impact on the latency of these data to the servers
- >30 minute delays to NCEP FTP servers
- >10 minute delays to NWS FTP servers

## Availability Summary

- NCEP already sending GRIB2 data sets ONLY
- Data are converted back to GRIB1 on the ftp servers
- We have effectively “played our GRIB2 card”
- No new ensemble data sets can be added to the NCEP or NWS ftp servers until current bandwidth bottleneck is addressed.
- Future Changes that could help:
- Remove redundant data feed to NWS (Feb 2007 – tentative)
- Network upgrade (Q4 FY2007)
- Change what global ensemble data are delivered.

## Network Costs

- Currently NCEP spends $1.2 M annually on its networks
- In FY07 this cost will jump to over $2.0 M
- This price increase will allow an increase in bandwidth as well as add the necessary redundancy needed to all of NCEP’s remote interests.
- Represents a very large portion of NCO’s total operating budget
- In an era of flat funding it also represents a significant challenge for the future as well.

## Network Costs

- Even after NCEP bandwidth is increased it may only push the problem further downstream
- NCEP ftp server traffic accounts for over 65% of the total traffic hitting the NOAA internet gateway
- >2 TB of NCEP data served daily to the world
- NOAA now requiring NCEP to pay for internet bandwidth.
- Added costs to NCEP!

## Road Ahead

- Ensemble data volumes are already growing exponentially.
- The future holds more expansions as well
- Additional members
- Higher resolutions
- More individual ensemble output products
- Additional ensemble members from external organizations
- How do we deliver ensemble information to end users?

## Possible Solutions

- Throttle back the delivery of ensemble data
- Pros: Possibly deliver more data
- Cons: No more real time access; same volume of data impacting our internet gateway.
- Use lossy compression techniques with GRIB2
- Pros: compression ratio’s 20:1, 50:1, or as high as 100:1 over GRIB1
- Cons: Some loss of information (smoothing)
- Generate ensemble products centrally
- Pros: Smaller package to distribute
- Cons: Loss of user input on the product

## Comments?