# Presentation: 653781

## Verification of Daily CFS forecastsHuug van den Dool & Suranjana Saha

- CFS was designed as ‘seasonal’ system
- Hindcasts 1981-2005, 15 ‘members’ per month
- Here we look at CFS(T62L64) as NWP (never mind the delayed ocean analysis)
- 25 years of forecasts (4500 forecasts out to 9 months) by a ‘constant’ T62L64 model !!!!!.   Climatology of daily scores
- No ensemble average, no time average
- Compare to CDAS  & CDC’s MRF

*Acknowledgement**: Cathy Thiaw (reruns)**               **and on CDAS info Bob Kistler/Fanglin Yang/Pete Caplan*

## Day 5 AC-scores, using the harmonically smoothed model and observed climatologies (which are more ‘competitive’ than the old-old-old climo used on Pete Caplan’s pageVariables: Extra-tropical Z500 (NH, SH)PSI200 and CHI200

- Z500:  72.3 (4500 cases, grand mean)

- Congratulations with a constant system!

- 180 forecasts per dot

## Grand Mean NH over 1984-2005: 70.5 (CDAS1)

- 0.7

- 0.7

- ((CDAS statistics has several problems))

- From Pete Caplan’s EMC website:

## CDAS

## CFS

## Next Two Slides: Excursion to SH Z500

## SH (CFS)

- Scores go UP and UP! “Congratulations” to whom ???

- Scores ‘volatile’ in SH (still 180 per dot)

## SH (CDAS)

## Doing the best we can:Comparing CFS to CDAS day 5 Z500 scores) 

|  | NH | SH |
| --- | --- | --- |
| CFS 1981-2005
T62L64, coupled, R2 | 0.723 | 0.629 |
| CDAS 1984-2005
T62L28, 
R1 | 0.705 | 0.623 |

- Warning: Number of forecasts per month differ. Climos differ!!! and change in ’96 for CDAS

- Very flawed

## Prelim Conclusion

- CFS is (slightly) better than CDAS.
- NH much better than SH (typical for pre-2000 technology) – this will change completely in next CFS
- Trend-issues and non-constancy of system (will get worse)

## Next Two Slides: Excursion to TROPICS

## PSI200:  63.0 (4500 cases grand mean)

- No increase???

- 180 forecasts per dot

- ENSO?

## CHI200:  45.9 (4500 cases grand mean)

- This may be lowish for MJO type prediction.

- Definitely no increase over time. Perhaps a decrease!!!

- 180 forecasts per dot

- ENSO?

## How about Bias Correction???One of the claimed usages of hindcasts

- Indisputable but very small improvement.
- Is Z500 incorrigeable?

- Based on 375 forecasts.

## The gain due to bias correction in a few selected months. Day 5 scores Z500 1981-2005		

|  | Raw | Bias Corrected |
| --- | --- | --- |
| NH DEC | 73.0 | 74.5 (+1.5) |
| NH JUL | 65.8 | 68.3 (+2.5) |
| SH FEB | 61.8 | 63.5 (+1.7) |
| SH AUG | 62.2 | 63.7 (+1.5) |

## Is Z500 incorrigeable? Largely Yes, because the systematic error is small.Wait till you see CHI200

## Chi200 improves tremendously from cleaning up the bias, especially early on.

- 375 forecasts

*Can we do MJO forecasts ??*

## PSI200 in Tropics does not improve very much from bias removal. !!!!

*PSI200 in Tropics does not improve very much from bias removal. !!!!*

## Distribution of Skill in space is importantand largely unexplored and unexplaind

## Slide 22

## Slide 23

## Slide 24

## Slide 25

## Skill as a function of (EOF) mode is interesting

## Slide 28

## Now: OUT TO 270 DAYS !!

## NH

- Detail, first month.

- Detail: Month 2 to Month 8

## SH

## FOCUS

- Week 3
- Week 4
- = days 15-21 and 22-28
- Physical basis of wk3/wk4
- Ocean interaction is as much a liability as a promise at this point in time.

## DAY   15

|  | CFS | Pers |  | CFS | Pers |
| --- | --- | --- | --- | --- | --- |
| NH | 15.2 | 4.2 |  | 6.1 | 3.9 |
| TR | 36.2 | 29.3 |  | 29.6 | 27.5 |
| SH | 9.6 | 6.2 |  | 2.0 | 3.4 |

- DAY  28

- Anomaly correlation of CFS and ‘Persistence’ Z500 prediction in Feb 1981-2005

##  NOISE

- ______________________     ratio
-  NOISE

- Improving signal to noise ratio:  (often cosmetic)  by reducing noise by applying some operator and, hopefully, not hurting the signal by this operator
- Take a time mean (7 days)
- Ensemble mean  (not done here)
- EOF filters (or better (maximally predictable modes))

## Anomaly correlation of CFS Z500 prediction, daily as well as weekly, in Feb 1981-2005

|  | Day 15 | Wk3 | Wk4 | Day 28 |  |
| --- | --- | --- | --- | --- | --- |
| NH | 15.2 | 15.7 | 9.2 | 6.1 |  |
| TR | 36.2 | 46.7 | 42.9 | 29.6 |  |
| SH | 9.6 | 10.6 | 5.2 | 2.0 |  |

- Effect of a 7 day mean.........

## So, overall, we went from daily scores (15.2....6.1) to weekly scores (15.7 and 9.2) to filtered weekly scores (22.0 and 23.9 at best)

| Wk3  | Wk3
EOF1 | WK3
EOF1-10 | WK4 | WK4
EOF1 | WK4
EOF1-10 |  |
| --- | --- | --- | --- | --- | --- | --- |
| 15.7 | 22.0 | 18.7 | 9.2 | 23.9 | 12.0 |  |

- Effect of EOF truncation......

## 15.7

## 9.2

## Conclusions

- CFS is (slightly) better than CDAS (Z500).
- Over time (1981-2005), the CFS “system” appears quite constant with various qualifiers
- If we need an as-constant-as possible in-house system, look no further (better than CDAS)
- Bias correction: small +ve impact on Z500 and Ψ200 (1-2pnts)
- Bias correction: large +ve impact on CHI-200 (15 pnts)
- CFS loses 5-20% in terms of SD and eDOF in the first few days, then, admirably, stays nearly constant out to 270 days

## Conclusions

- (Very) modest skill in wk3 and wk4
- Even with 2 of 3 signal to noise improvements in place AC is only 0.20-0.25. (No ensemble averagehere)
- Waiting for the next CFS and CFSRR (higher Res, consistent IC)

## Conclusions:

- The day 1 – 3 forecasts appear to be too damped, and damp faster than a regression would. Increasing anomaly amplitude as a postprocessor (undoing the sd decrease) actually improves the rms error early on. A curiosity?
- Probably: initial conditions are damped as well.

## THE REST IS EXTRA

- Does the eDOF variation explain the AC variation????

## It is not just resolution....

*It is not just resolution*....

- Member 5

- hurricane

- Only 25 forecasts

## T382 used GDAS

*Initial error=6.1 gpm ; Systematic error growth s=13.5 gpm/day ; Small error amplification a=0.181/day ; *
*e-infinity=157.7gpm ; NOSEC*

## Savijarvi’s equation 5:

*Initial error=4.7 gpm ; Systematic error growth s=13.1 gpm/day ; Small error amplification a=0.181/day ; *
*e-infinity=155.7gpm ; SEC*

*Initial error=4.7 gpm ; Systematic error growth s=12.6 gpm/day ; Small error amplification a=0.200/day ; *
*e-infinity=158.3gpm ; SECSD*

## Example:

- Systematic errors for mid-January at day 5.

## Slide 54

## Climatological Annual Cycle of day-5 scores

## NH

- Forecasts verifying in month shown

- NH: Best month is Feb (76.4), Worst month is July (67.7). Near sinusoidal variation. *Range=8.7pnts*

- 375 forecasts per month

- SH: Best month is  Aug (64.9), Worst month is March (59.0). Typically 62-64, except Feb-Apr. *Range=5.9pnts*

*Curious: March is best/worst in NH/SH. Aug is best/worst in SH/NH.*

- SH

## The NH has an annual cycle in skill, every year.

- (Each color curve is a different year.)

- The SH does NOT have a clear annual variation in skill each year.

- 15 forecasts per month

## forecasts verifying in month shown

- 375 forecast per dot(month)

- Large annual variation in the tropics! But volatile.

## Day 5 scores

- AC (obviously; already shown),
- But also:
- SD
- eDOF

## CFS misses 5-20 % of variability

- Loss of variance does not increase beyond 10-15 days and is never more than 20%

## CFS misses several degrees of freedom

- Loss of dof does not increase beyond 10-15 days and is never more than shown above

## Compare the SD reduction in the 

- Compare the SD reduction in the
- NH
- To that in the
- SH

## Compare the DOF reduction in the 

- Compare the DOF reduction in the
- NH
- To that in the
- SH