# Presentation: 129799

## Ensemble Hydrologic Forecasting

**Prepared by**
**D.-J. Seo, J. Schaake, E. Welles, H. Herr,**
**M. Mullusky, J. Demargne, L. Wu, X. Fan, and**
** ****S. Cong**
**for**
**NRC AHPS Science Review**
**National Weather Service**
**Office of Hydrologic Development**
**Hydrology Laboratory**
**Jan 28, 2004**

## Content

**Introduction**
***Overview of Ensemble Streamflow Prediction (ESP)***
**Short-term ESP**
***Overview***
***Preprocessing***
**Precipitation**
**Temperature**
***Post-processing***
***Parametric uncertainty***
***Multimodel ensembles***

**Long-term ESP**
***Overview***
**Medium-range ESP**
**Data assimilation**
**Verification**
**Collaboration and Partnerships**
**Science infusion**
**Summary**
**References**

## Introduction

**Probabilistic prediction is the underpinning of AHPS (see Slide 4)**
**Ensemble prediction (see Slide 5) offers a scientifically sound and practical methodology for hydrologic forecasting across space-time scale**
***The time scale is currently tied to the lead times of available meteorological forecasts (see Slide 6);***
**1 to 5 days**
**6 to 14 days**
**Two weeks and beyond**
***These divisions in time scale distinguish short-/medium-range and long-term ESP as currently practiced***
***The spatial scale ranges from a few km******2****** to the continental (see Slide 7)***
***Attaining seamlessness and consistency across scale remains an outstanding science issue***

## Slide 4

## Ensemble Streamflow Prediction (ESP)

## Slide 6

## Spatial Scale of Hydrologic Modeling

**Spatial Scale of Hydrologic Modeling**

- National

- Major River System

- River Basin with Forecast Points

- Forecast Group

- Headwater Basin and Radar Rainfall Grid

- High Resolution Basins

## Short-term ESPOverview

**Meteorological forcing used;**
***Day 1-5 Quantitative Precipitation Forecast (QPF) and smoothed precipitation climatology for day 2-5 (if no QPF available)***
***Day 1-7 Quantitative Temperature Forecast (QTF)***
**As an alternative to the Probabilistic QPF (PQPF) strategy (see Slide 13 ), a pre-processor is used in ESP to generate ensembles from the existing single value forecasts (see Slides 10, 11, 12, 15, 16)**
**An ensemble post-processor is used to account for all hydrologic uncertainties collectively (see Slides 17, 18)**
**See Welles et al. (2003) for status and plans**

## Schematic of Short-Term ESP

**Reflects both uncertainties**

**Ensemble traces of precipitation, temperature**

**Ensemble traces of streamflow**

**Ensemble traces of streamflow**

**QPF, QTF**

**ESP Pre-Processor**

**Hydrologic model**

**ESP Post-Processor**

**Corrects bias, accounts for meteorological uncertainty**

**Schematic of Short-Term ESP**

## Short Term Ensemble Precipitation Forecasts from Deterministic Forecasts

**Analyze joint distribution of historical forecasts and corresponding observations**
**Estimate joint distribution parameters**
**Derive conditional distribution given a deterministic forecast – varies spatially**
**Create synthetic ensemble forecasts consistent with conditional distribution from climatology**
**Evaluate results**

## Ensemble Pre-Processor1.  Calibration: at each time step, compute the conditional distribution of future observed precipitation given the current QPF

**Ensemble Pre-Processor****1.  ****Calibration****: at each time step, ****compute the conditional distribution of ****future observed precipitation given the current QPF**

**From Welles et al. (2004)**

## Ensemble Pre-Processor2.  Distribution mapping for PQPF derivation: at each time step, generate adjusted ensemble points given the conditional distribution of future observed precipitationfrom climatology time series

**From Welles et al. (2004)**

## Ensemble Precipitation Processor (EPP)

**A PQPF-based approach to generating precipitation ensembles (see Slide 14)**
**Initially pursued in parallel with the Bayesian approach (Krzysztofowicz 1998a,b, 1999a,b,c, Kyzysztofowicz and Herr 2001) as a part of the NWS/Eastern Region PQPF-PRSF (Probabilistic River Stage Forecast) demonstration project**
**Experimental implementation at OHRFC proved very labor- and computer-intensive**
**May hold potential for direct down-scaling of NWP ensembles**
**For details, see Seo et al. 1999, 2000**

## An example PQPF field (PoP, cond. Mean, cond. CV)

**An example PQPF field (PoP, cond. Mean, cond. CV)**

**An example ensemble trace from EPP**

**From Seo et al. 2000**

## Short-Term Quantitative Temperature Forecast (QTF)

**From FMAT (6-hour future mean areal temperature) derive FMAX (daily maximum FMAT) and FMIN (minimum daily FMAT)**
** ****Derive MAX/MIN conditional forecast distributions given the current FMAX and FMIN (cf Slides 11)**
** ****Using Distribution Mapping Technique create an ensemble of maximums and an ensemble of minimums (cf Slide 12)**
** ****Temporally disaggregate maximum and minimum to six hourly temperature ensembles with user determined parameters (see Slide 16)**
**For details, see Fan et al. 2003**

## Example of Temperature EnsemblesShirleysburg, PA

**From Fan et al. 2003**

## ESP Post Processor

**Corrects for systematic model biases**
**Collectively accounts for hydrologic uncertainties; the uncertainty in initial conditions, parametric uncertainty and model structural errors (see Slide 26)**
**The current post processor is based on probability matching (between the marginal cdf’s of observed and model-simulated flows) and linear regression in the normal space (Perica et al. 1999, Seo et al. 2004)**

## From Seo et al. 2004

**Post-processed ensembles are very reliable if there is little long-term change in distribution of streamflow**

**From Seo et al. 2004**

**The reliability diminishes significantly, however, if significant long-term changes occur in the cdf of streamflow**

## Explicit Accounting of Parametric Uncertainty

**Explicit modeling of parametric uncertainty (see Slide 20) is needed to capture propagation of long-memory errors and extremely nonlinear errors (e.g., rain on snow) and to simplify statistical modeling of residual errors**
**Quantification and characterization of parametric and combined (parametric and input) uncertainties in lumped and distributed hydrologic models (Carpenter and Georgakakos 2004)**
**Comparative assessment of uncertainty in soil-based SAC parameters with and without local optimization (Kuzmin et al. 2003)**

## Schematic of future short-term ESP with data assimilation and explicit uncertainty processors

## Multimodel ensembles

**A part of the super ensemble strategy (see Slides 22)**
**Unique opportunities provided by the Distributed Model Intercomparison Project (DMIP)**
***Encouraging results from analysis of multimodel ensembles from DMIP (see Slide 23, Georgakakos et al. 2004)***
**Single-model multi-parameter ensemble strategy (cf parametric uncertainty processor in Slide 20) also pursued**
***The Model Parameter Estimation Experiment (MOPEX) to compare multi-parameter ensembles from different models***

## Ensemble Hydrologic Forecasting:

- Future
- Super Ensembles

- Initial
- Conditions

- Hydrologic
- System

- Parameters

- Hydrographs

- Meteorological
- Forcing

## From Georgakakos et al. 2004

**From Georgakakos et al. 2004**

**Mean of multimodel ensemble (arrow) is superior to the best single model simulation (cross hair)**

**Probabilistic prediction using multimodel ensemble has larger economic value than single model results**

**Illinois river at Savoy**

**All DMIP basins**

## Long-Term ESPOverview

**Integrates long-lead (14 days and beyond) meteorological forecasts/climate outlooks from NCEP/CPC (see Slide 20) via a pre-processor**
**The pre-processor adjusts historical mean areal precipitation (MAP) and temperature (MAT) time series with respect to the current meteorological forecasts/climate outlooks (see Slide 25)**
**The marginal exceedence probabilities of the adjusted time series are consistent with the CPC forecasts (see Slide 27)**
**For details, see Perica 1998**

## NCEP/CPC

**NCEP/CPC**

## Long-Term ESPAdjustment of historical time series based on CPC forecasts

**Distribution is assumed to be normal**
**Historical MAT time series are adjusted according to;**
****** ****= T****fcst**** – T****hist**
**	****where ******** is the adjustment to historical MAT, T****fcst**** is the distribution-average temperature of the forecast, and T****hist**** is the average temperature from climatology**

**Conditional distribution is assumed to be Gamma**
**Historical MAP time series are adjusted according to;**
****** ****= P****fcst**** / P****hist**
**	****where ******** is the adjustment factor to historical MAP, P****fcst**** is the (unconditional) distribution-average precipitation amount in the forecast, and P****hist**** is the average precipitation amount from climatology**

**Temperature**

**Precipitation**

## Goodness of distribution approximation in precipitation adjustment

**Goodness of distribution approximation in precipitation adjustment**

**From Schaake **

## Medium-Range ESPMRF-ESP Project

**Collaboration among CIRES, Univ. of Colorado, CDC, and CBRFC**
**Assimilates medium-range meteorological ensemble forecasts into ESP**
**Ensembles are from the ‘frozen’ MRF for days 1 through 14**
**The raw ensembles are post-processed and downscaled to obtain MAP and MAT at the basin scale**
**Forcing ensembles for Days 15-265 are from historical data and blended with Days 1-14 ensembles**
**For details, see Brandon (2003a)**

## Data Assimilation

**A critical component in ensemble prediction**
***To reduce and to quantify uncertainty in the initial conditions***
***To effect automatic run-time modification***
***To move toward ESP with explicit accounting of individual sources of hydrologic uncertainties (see Slide 20)***
**To be operationally implemented in ESP (see Demargne and Mullusky 2003)**
**Hydrologic data assimilation/state updating**
***State-Space Sacramento (SS-SAC, Sperflage and Georgakakos 1996) implemented in NWSRFS***
***Variational assimilation-based technique (VAR, Seo et al. 2003) experimentally implemented at WGRFC (see Slide 31, Seo et al. 2003d)***
**Assimilates streamflow observations at the headwater basin outlet, potential evaporation and precipitation in real time**
**Addresses some key issues with SS-SAC**

## Data Assimilation (cont.)

**Snow**
***National Snow Analysis (NSA) at the National Operational Hydrologic Remote Sensing Center (NOHRSC, see Slides 31, 32, 33)***

## National Operational Hydrologic Remote Sensing Center (NOHRSC) 
The NOHRSC provides remotely-sensed and modeled hydrology products for the coterminous U.S. and Alaska for the protection of life and property and the enhancement of the national economy. 

**National Snow Analyses**

**National Operational Hydrologic Remote Sensing Center (NOHRSC) **
***The NOHRSC provides remotely-sensed and modeled hydrology products for the coterminous U.S. and Alaska for the protection of life and property and the enhancement of the national economy. ***

- National Snow Analyses (NSA)
- As part of this mission, the NOHRSC develops, maintains, and operates the experimental* NOAA National Weather Service NSA.
- Interactive NSA map, text, and data products comprehensively describe snow conditions throughout the coterminous U.S. :
- water content of the snow pack (SWE) (hourly)
- snow depth (hourly)
- snow pack temperature (hourly and previous 24-hr average)
- snow melt (hourly and previous 24-hr total)
- snow and non-snow precipitation (hourly and previous 24-hr total)
- water losses through sublimation and blowing snow (hourly and 24-hr total)
- NSA support operational flood forecasting, water resource management, hydrologic research.

**Water content (SWE) analysis of the Nation’s snow pack (in cm) for 6Z January 17, 2004. **

**Average snow pack temperature analysis (C) for the 24-hour period ending 6Z January 17, 2004. **

- * The NSA are not sufficiently supported by NOAA/NWS to maintain an operational status. Gaps in NSA products and services can occur at any time.

**Notes:**
CLPX Title Slide – should mention:
Two year experiment
AMSR Validation a significant component of the experiment design
Many agencies and universities involved in making this happen

## NSA are Based on Modeled and Observed Snow Properties
Land Surface Model
5-layer energy- and mass-balance model for snow and soil with comprehensive snow physics
1-km spatial resolution for coterminous U.S.
Hourly temporal resolution
Forced by physically downscaled numerical
	weather	analyses (RUC2)
Daily assimilation of all operationally available 
     snow observations into model

**National Snow Analyses**

**NSA are Based on Modeled and Observed Snow Properties**
***Land Surface Model***
**5-layer energy- and mass-balance model for snow and soil with comprehensive snow physics**
**1-km spatial resolution for coterminous U.S.**
**Hourly temporal resolution**
**Forced by physically downscaled numerical**
**	****weather****	****analyses (RUC2)**
***Daily assimilation of all operationally available ***
***     ******snow observations into model***

- Satellite-derived areal extent of snow cover
- Ground-based and airborne observations of snow water equivalent and snow depth

- Cartoon depicting physical processes represented in NSA snow model.

- In Situ Observations
- Approximately 4000 snow depth and 1000 SWE observations are evaluated each day.
- Majority of observations in eastern U.S. are snow depth.
- Majority of SWE observations are in western U.S.
- Sources include NWS and FAA stations, NWS cooperative observers, NWS airborne snow surveys, NRCS SNOTEL, and several federal and state agency snow survey networks.

- Map showing differences between observed and modeled snow water equivalent and snow depth, eastern U.S.

**Notes:**
CLPX Title Slide – should mention:
Two year experiment
AMSR Validation a significant component of the experiment design
Many agencies and universities involved in making this happen

## NSA Modeling and Data Assimilation Development
Major support for development of the NSA land surface model and data assimilation framework was provided by the NASA Earth Science Enterprise Terrestrial Hydrology Program.
Support for development of geostatistical spatial analysis techniques for snow observations (related to snow data assimilation) was provided by the GEWEX Americas Prediction Project (GAPP, formerly GCIP) through NOAA’s Office of Global Programs.
Additional support for NSA development was provided by the NWS Office of Climate, Water, and Weather Services and by the Susquehanna River Project.

NSA Applications
NSA data products are currently being used to support operational river and flood forecasting as part of the NWS Advanced Hydrologic Prediction System (AHPS).
e.g. Ohio River Forecast Center (OHRFC)
 In 2002, OHRFC tested NSA products to update snow states in their NWS River Forecast System (NWSRFS) model.
Results indicated up to 70% improvement in forecast accuracy using NSA products.
In 2003, OHRFC began using NSA products routinely to support their river and flood forecasting operations.

**NSA Development and Applications**

**NSA Modeling and Data Assimilation Development**
***Major support for development of the NSA land surface model and data assimilation framework was provided by the NASA Earth Science Enterprise Terrestrial Hydrology Program.***
***Support for development of geostatistical spatial analysis techniques for snow observations (related to snow data assimilation) was provided by the GEWEX Americas Prediction Project (GAPP, formerly GCIP) through NOAA’s Office of Global Programs.***
***Additional support for NSA development was provided by the NWS Office of Climate, Water, and Weather Services and by the Susquehanna River Project.***
**NSA Applications**
***NSA data products are currently being used to support operational river and flood forecasting as part of the NWS Advanced Hydrologic Prediction System (AHPS).***
**e.g. Ohio River Forecast Center (OHRFC)**
*** ******In 2002, OHRFC tested NSA products to update snow states in their NWS River Forecast System (NWSRFS) model.***
**Results indicated up to 70% improvement in forecast accuracy using NSA products.**
***In 2003, OHRFC began using NSA products routinely to support their river and flood forecasting operations.***

**Notes:**
CLPX Title Slide – should mention:
Two year experiment
AMSR Validation a significant component of the experiment design
Many agencies and universities involved in making this happen

## Examples of hydrologic data assimilation at WGRFC

- The improvement due to parameter refinement is often significant for high flows
- This improvement translates directly into improved performance envelope for VAR

**Examples of hydrologic data assimilation at WGRFC**

## Hydrologic data assimilation research with distributed hydrologic models

**The goal is to develop a data assimilation system for the Hydrology Laboratory-Research Modeling System (HL-RMS, Koren et al. 2004) and its (experimental) operational version, the Distributed Modeling System (DMS)**
**Estimation of distributed kinematic-wave routing parameters in HL-RMS via variational assimilation (Seo et al. 2003c)**
**Real-time updating of distributed model states of soil moisture using radar-based precipitation, potential evapotranspiration and streamflow data via variational assimilation (Seo et al. 2003b)**

## Verification of ESP forecasts

**Must consider;**
***Local climatology, dependence on lead time, season, flow regime, basin size and characteristics***
***Quantification of informativeness, skill, reliability***
**The ESP Verification System (ESPVS) currently under redevelopment**
***Based on Franz and Sorooshian (2002) and others***
***Includes Talagrand diagram, ranked probability score (RPS), ranked probability skill score (RPSS), discrimination diagrams, and reliability diagram***
***For details, see Brandon (2003b), Demargne and Mullusky (2003), Welles et al. (2003), Wick (2003)***

## Collaboration and Partnerships

**OHD-NCEP**
***Global ensemble (EMC)***
***Regional ensemble (EMC)***
***Specification of uncertainty factor in single-value forecasts (HPC)***
**GCIP/GAPP**
***MRF-ESP (CDC, Univ. of Colorado, CIRES, CBRFC)***
***Verification (Univ. of Iowa, UNC)***

## Collaboration and Partnerships (cont.)

**MOPEX**
***Parameter uncertainty***
***Single-model multi-parameter ensembles among different models***
***Development of data sets***
**DMIP**
***Multimodel ensembles***
***Development of data sets***
**See Schaake (2002) for details**

## Science infusion

**AHPS Science Infusion Strategy (Schaake 2002)**
**Hydrologic Ensemble Prediction Experiment (HEPEX: see Slide 40)**
**National Hydrologic Long-Range Prediction System (NHLPS)**
***An OHD-led effort to develop a test bed for large-scale long-range hydrologic ensemble prediction***
***To produce probabilistic hydrologic forecasts for lead times a month to a year***
***To focus initially on experimental forecasts of natural runoff***

## HEPEXHydrologic Ensemble Prediction Experiment

**Goal**
***Develop “engineering quality” hydrologic ensemble prediction procedures for time scales (flash-flood to 1-yr) and space scales (1-km to continental)***
**Organization**
***IAHS (PUB), GEWEX (WRAP), WMO, UNESCO?***
**Initial Workshop**
***ECMWF, March 2004***
***Develop science plan***

## Summary

**The current ESP methodology in AHPS;**
***Integrates major scientific components for a state-of-the-art end-to-end ensemble hydrologic prediction system***
***Integrates available meteorological forecasts from day 1 to a year***
***Reflects balance among maturity of the science, practicality, and benefit-cost effectiveness in meeting the service goals***
***Allows generation of probabilistic products across a wide range of space-time scale***
***Accounts for both meteorological and hydrologic uncertainties***

## Summary (cont.)

**Areas of improvement include;**
***seamless and consistent assimilation of meteorological forecasts, including post processing and downscaling***
***uncertainty processors***
***data assimilation/state updating***
***reservoir modeling***
***verification***
***see Demargne and Mullusky (2003) for details***
**To expedite science advances and transfer to operations, NWS is actively engaged and playing leadership roles in;**
***collaboration and partnerships***
***science infusion***
***See Schaake (2002) for details***

## References

**Brandon, D., 2003a: MRF/ESP Project. HIC Innovation Meeting, NWS, Oct 27-28, Kansas City, MO.**
**Brandon, D., 2003b: Prototype ESPVS. HIC Innovation Meeting, NWS, Oct 27-28, Kansas City, MO.**
**Carpenter, T. M., and K. P. Georgakakos, 2004: Impacts of parametric and radar rainfall uncertainty on the ensemble streamflow simulations of a distributed hydrologic model. Accepted for publication in DMIP Special Issue of J. Hydrol.**
**Cong, S., J. C. Schaake, and E. Welles, 2003: Retrospective verification of ensemble streamflow forecasts: A case study. AMS meeting. Long Beach, CA.**
**Demargne, J., and M. Mullusky, 2003: Ensemble activities at the OHD Hydrology Laboratory, internal document, NWS/OHD.**
**Fan, X., M. G. Mullusky, L. Wu, E. Welles, J. Ostrowski, E. Pryor, and J. C. Schaake, 2003: Short Term Ensemble River Stage Forecasts: Application, J5.5, AMS annual meeting, Long Beach, CA.**
**Franz, K. J., and S. Sorooshian, 2002: Verification of National Weather Service probabilistic hydrologic forecasts. Final Report, Depart. Of Hydrol. And Water Resour., The Univ. of Arizona, Tucson, AZ.**
**Hashino, T., A. A. Bradley, and S. S. Schwartz, 2002: Verification of probabilistic streamflow forecasts. IIHR Report No. 427, IIHR-Hydroscience & Eng. And Depart. Of Civil and Environ. Eng., The Univ. of Iowa, Iowa City, IA.**

## References (cont.)

**Herr, H., E. Welles, M. Mullusky, L. Wu, and J. Schaake, 2002a: Simplified short term precipitation ensemble forecasts: Theory. J1.17, AMS annual meeting, Orlando, FL.**
**Herr, H., E. Welles, M. Mullusky, L. Wu, J. Schaake, and D.-J. Seo, 2002b: Probabilistic hydrologic forecasting: An ensemble approach. 2nd Federal Interagency Hydrologic Modeling Conf., Las Vegas, NV.**
**K. P. Georgakakos, D.-J. Seo, H. Gupta, J. Schaake and M. B. Butts, 2003: Towards the characterization of streamflow simulation uncertainty through multimodel ensembles. Accepted for publication in DMIP special issue of J. Hydrol.**
**Koren, V., M. Smith, S. Reed, Z. Zhang and D.-J. Seo, Hydrology Laboratory Research Modeling System of the National Weather Service, 2004: To appear in Catchment Modeling special issue of J. Hydrol. **
**Kyzysztofowicz, R., 1998a: An integrated probabilistic hydrometeorological forecast system. Preprints, Special Symp. On Hydrol., AMS, Phoenix, AZ, J45-J50.**
**Kyzysztofowicz, R., 1998b: Probabilistic hydrometeorological forecasts: Toward a new  era in operational forecasting, AMS Bulletin, 79(2), 243-251.**
**Krzysztofowicz, R. 1999a: Probabilities for a period and its subperiods: Theoretical relations for forecasting. Mon. Wea. Rev., 127(2), 228-235.**
**Krzysztofowicz, R., 1999b: Bayesian theory of probabilistic forecasting via deterministic hydrologic model. Water Resour. Res. 35(9), 2739-2750.**
**Krzysztofowicz, R., 1999c: Point-to-area rescaling of probabilistic quantitative precipitation forecast, J. Appl. Meteor. 38(6), 786-796.**

## References (cont.)

**Kyzysztofowicz, R. and H. Herr, 2001: Hydrologic uncertainty processor for probabilistic river stage forecasting: Precipitation-dependent model, J. Hydrol., 249, 46-68.**
**Kuzmin, V., V. Koren, and D.-J. Seo, 2003: Robust and efficient estimation of hydrologic model parameters using a priori information. OHD seminar.**
**Mullusky, M., J. Schaake, S. Perica, D. J. Seo, H. Herr, 2001: Accounting for Uncertainty in Short Term (1-5 day) Deterministic Precipitation Forecasts. EGS.**
**Mullusky, M., L. Wu, H. Herr, E. Welles, J. C. Schaake, J. Ostrowski, and N. Pryor, 2002: Simplified short term precipitation ensemble forecasts: Application. JP1.19, AMS annual meeting, Orlando, FL.**
**Mullusky, M. G., M. DeWeese, E. Welles, and J. Schaake, 2003: Information and products derived from ensemble streamflow forecasts, JP3.17, AMS annual meeting, Long Beach, CA.**
**Mullusky, M., J. Demargne, E. Welles, L. Wu, and J. Schaake, 2004: Hydrologic applications of short and medium range ensemble forecasts in the NWS Advanced Hydrologic Prediction Services (AHPS), J11.5, AMS annual meeting, Seattle, WA.**
**Perica, S., 1998: Integration of meteorological forecasts/climate outlooks into Extended Streamflow Prediction (ESP) System, 78th Annual AMS Meeting, Phoenix, AZ.**

## References (cont.)

**Perica, S., J. C. Schaake, and D.-J. Seo, 1999a: Accounting for hydrologic model errors in ensemble streamflow prediction. preprint volume, 14th Conf. on Hydrol., Dallas, TX, Jan 10-15.**
**Perica, S., J. Schaake, and D.-J. Seo, 1999b: National Weather Service River Forecast System (NWSRFS) operational procedures for using short and long range precipitation forecasts as input to Ensemble Streamflow Prediction (ESP), 14th Conf. On Hydrol. Dallas, TX.**
**Perica, S., J. Schaake, and D.-J. Seo, 2000: Hydrologic application of global ensemble precipitation forecasts. EGS.**
**Schaake, J. C., 2002: AHPS science infusion strategy, Draft, revised 03/01/02, NWS/OHD.**
**Schaake, J. C., 2003a: Flood forecasting practices in the USA. Int. Conf. on Advances in Flood Forecasting in Europe. Rotterdam.**
**Schaake, J.C., 2003b: Diagnosis of uncertainty in atmospheric ensemble model precipitation forecasts. AGU/EGS, Nice.**
**Schaake, J.C., 2003c: Ensemble approach to hydrologic forecasting in the U.S. Advanced Hydrologic Prediction Services (AHPS). AGU/EGS, Nice.**
**Schaake, J.C., 2003d: Infusion of GEWEX science into hydrologic forecasting for the United States Advanced Hydrologic Prediction Services (AHPS). GEWEX-IAHS. Water Resources Management Workshop, Sapporo, Japan.**

## References (cont.)

**Schaake, J.C., 2003e: Hydrologic applications of ensemble precipitation forecasts. Water Management Information Workshop. Montreal.**
**Schaake, J.C., and Q. Duan, 2003: National Long-range Hydrologic Prediction System (NLHPS). OHD Seminar.**
**Schaake, J., M. Mullusky, E. Welles, and L. Wu, 2003a,b,c: Short-range ensemble precipitation forecasts for NWS Advanced Hydrologic Prediction Services (AHPS): Parameter estimation issues. JP3.7, AMS annual meeting, Long Beach, CA.**
**Schaake, J.C., Z. Toth, D. Reynolds, M. Antolik, J. Maloney, J. Du, B. Zhou, M. Halpert, R. Martin, P. Dallavalle, E. Danaher. and K. Lynott, 2003d: Toward a science infusion strategy toward NWS Probabilistic Quantitative Precipitation Forecasting (PQPF). AMS meeting. Long Beach, CA.**
**Schaake, J., S. Perica, M. Mullusky, J. Demargne, E. Welles, and L. Wu, 2004a,b: Pre-processing of atmospheric forcing for Ensemble Streamflow Prediction. AMS annual meeting, 5.2, Seattle, WA.**
**Schaake, J. C., A. Henkel, and S. Cong, 2004c: Application of PRISM climatologies for hydrologic modeling and forecasting in the Western U.S. AMS annual meeting. Seattle, WA.**
**Seo, D.-J., 2003: Ensemble forecasting as a means of reducing forecast error. Workshop on dealing with certainties in the hydroelectric energy business. Montreal.**
**Seo, D.-J., S. Perica, E. Welles, and J. Schaake, 1999: An Ensemble Precipitation Processor (EPP) for generating precipitation ensembles for the next 24 hours. 14th Conf. On Hydrol., AMS, Jan 10-15.**

## References (cont.)

**Seo, D.-J., S. Perica, E. Welles, and J. Schaake, 1999: Space-Time Simulation of Ensemble Precipitation Traces from Probabilistic Quantitative Precipitation Forecast (PQPF). AGU fall meeting.**
**Seo, D.-J., S. Perica, E. Welles, and J. Schaake, 2000: Simulation precipitation fields from Probabilistic Quantitative Precipitation Forecast, J. Hydrol., 239, 203-229.**
**Seo, D.-J., V. Koren and L. Cajina, 2003a: Real-time variational assimilation of hydrologic and hydrometeorological data into operational hydrologic forecasting. J. Hydrometeorol., 4, 627-641.**
**Seo, D.-J., V. Koren, and N. Cajina, 2003b: Real-time assimilation of radar-based precipitation data and streamflow observations into a distributed hydrologic model. In Tachikawa et al., eds., Weather Radar Information and Distributed Hydrological Modeling, Proceedings of Symp. HS03 held during IUGG2003 at Sapporo, No. 282 in IAHS Publ., 138-142.**
**Seo, D.-J., V. Koren, and S. Reed, 2003c: Improving a priori estimates of hydraulic parameters in a distributed routing model via variational assimilation of long-term streamflow data. In Tachikawa et al., eds., Weather Radar Information and Distributed Hydrological Modeling, Proceedings of Symp. HS03 held during IUGG2003 at Sapporo, No. 282 in IAHS Publ., pages 109-113.**
**Seo, D.-J., V. Koren, L. Cajina, R. Corby, B. Finn, F. Bell and T. Howieson, 2003d: Real-Time Variational Assimilation of Streamflow and Radar-Based Precipitation Data into Operational Hydrologic Forecasting. AGU-EGS, EAE03-A-14671; NH4-1MO4P-1416. **

## References (cont.)

**Seo, D.-J., H. Herr, and J. C. Schaake, 2003e: A hydrologic error model for ensemble streamflow prediction. JWH02, IUGG, Sapporo, Japan.**
**Seo, D.-J., H. Herr and J. C. Schaake, 2004: Accounting for Hydrologic Uncertainty in Short-range Ensemble Streamflow Prediction (ESP), submitted to J. Hydrol.**
**Sperflage, J. A., and K. P. Georgakakos, 1996: Implementation and testing of the HFS operation as part of the National Weather Service River Forecast System (NWSRFS). HRC Tech. Rep. 1, Hydrologic Research Center, San Diego, CA, 213pp.**
**Welles, E. and M. Markus, 1996: A verification system for probabilistic hydrograph forecasts. ASCE Int. Conf. On Natural Disaster Reduction. Washington, D.C.**
**Welles, E., and L. Larson, 1998: A summary of the National Weather Service Advanced Hydrologic Prediction System Demonstration in Des Moines, Iowa. 1st Federal Interagency Modeling Conf. Apr 19-23.**
**Welles, E., T. Adams, and J. Schaake, 1999: Operational experience with Ensemble Streamflow Prediction in the Upper Monongahela Probabilistic Forecast Demonstration Project, 14th Conf. On Hydrol, AMS, Jan 10-15.**
**Welles, E. and others, 2003: Short Term Ensemble Forecasts: An Ongoing End to End Project. OHD seminar.**
**Wick, G., 2003: Evaluation of potential accuracy performance measure for the Advanced Hydrologic Prediction Service. draft internal report.**