Considerations for using observations
to guide numerical model development
Presenter: Katia Lamer
Brookhaven National Laboratory, Upton NY, USA
2nd ESA EarthCARE Validation Workshop
25-28 May 2021 (online)
ESA UNCLASSIFIED – For ESA Official Use Only

1

Main challenges for comparing model and observations
1. Hydrometeor properties defined in terms of different moments (i.e., mass vs. radar reflectivity)
2. Scale differences between model output and observations
3. Discrepancies in performance (i.e., ability to detect/simulate small amounts of liquid or ice)
4. Ill-defined evaluation metrics
What does one mean by cloud layer depth in mixed-phase clouds?
How much liquid is required for a hydrometeor population to be labeled as “mixed-phase”?
5. What is a suitable evaluation spatiotemporal scale?
6. What level of accuracy between model and observations is acceptable?

Forward simulators can help overcome some of these barriers
21. Hydrometeor properties defined in terms of different moments
1- Models simulate each hydrometeor species separately, remote sensors cannot readily isolate hydrometeor species=> model output information about all species must be combined.
2- Observations are sensitive to hydrometeor moments that do not necessary strongly affect process rates in models.
=> missing key hydrometeor properties need to be retrieved or assumed.

For all simulated species:
Stratiform & Convective
Clouds&kgPrecipitation
meteor mixing
ratios (kg) Hydrometeor fall velocity (m s )1010
0.5 1.0 1.5 2.0
Liquid & Ice &0.0Graupel-1-10

Hydrometeor mixing(kg(mm)kg-1) Hydrometeor
effectiveratios
radiusHydrometeor
fall fraction
velocity(%)(m s-1)-10 0.1 10-51
10-150.001 10
0.01
0.01
50 1.5 100
0.0 0.5 1.0
2.0-1-5℃
PSD informationb)• mass• size)• shape of theb distribution
Particle properties• densityb)• shape
08/12
08/16 08/08
08/12
fall speed
Date•(mm/dd)
Date (mm/dd)• water coating

1284
012840ca2)d2b)2)

ac3)d3b)3)

0234-40℃d1b)1)

0℃ca4)
1284008/16 08/08

d4b)4)

08/12

Date (mm/dd)

08/16 08/08

08/12

Date (mm/dd)

c1)

d1)

c2)d2)c3)d3)

SGP KAZRc4)

08/16 08/08d4)

08/12

Date (mm/dd)

08/16 08/08

08/12

Date (mm/dd)

08/16

Linear depolarization ratio (dB)

ac1)

Hydrometeor fraction (%)
0.01
50
100

cloud liq. cloud ice precip. liq.. precip. ice

12840b1)

Hydrometeor effective radius (mm)0.001 0.01 0.1
1cloud liq. cloud ice precip. liq.. precip. ice-40℃

Height (km)-15

Two types of observations

GCM ModelE3 output

3

1. Hydrometeor properties defined in terms of different momentsÞScattering calculations and empirically derived relationships allow us to convert simulated hydrometeors
properties (~D3) to observables (~D6 and ~D2). Weather/climate model complexity influences the choice ofscattering models
COMPLEX
Discrete Dipole Approximation for non-spherical particles (Draine et al 1994)
Mie theory for spherical particles (e.g., QuickBeam (Haynes et al. 2007) for COSP-CloudSat and for ARM CRS)
Ensemble of empirical relationship between water content – radar reflectivity (e.g., (GO)2-SIM (Lamer et al. 2018))(GO)2-SIM scattering modelSIMPLEa)

Lamer (2018)

Observable

Observableb)

ModelModel4

2. Scale differences between model output and observations
1- While models may provide 4D global or limited domain
simulations, sensors collect dimensionally challenged measurements
either at a fixed location or along an orbit/swath.
ÞInstrument forward simulators allow us to extract the model gridsthat match the sensor sampling geometry. Those require knowledgeof:• Platform geolocation information (i.e., orbit model)• Viewing geometry (i.e., angle, swath)• Revisiting time (i.e., diurnal sampling)2- Sensor sampling volume often differs from model resolution. This
scale gap must be addressed. Techniques to address the scale gap
generally vary with model resolution.

5

2. Scale gap – Model grid >> Observation volumeLamer (2019)

When model doesn’t resolve cloud and precipitation features but observations do
we need to decide how should hydrometeor fraction be distributed in space.
Why: Vertical overlap between cloud and precipitation layers strongly affects remotesensing measurements because signal attenuation generally increases with layer depth.
How: For example:
NASA/GISS ModelE3 horizontal resolution ~100 km
Ground-based ARM/KAZR observations: ~30 s
Approximately 12 hrs of observation within a model gridÞCannot average without convoluting diurnal cycle and horizontal variability.
ÞGenerate subcolumns that adheres to all overlap assumptions within the modeland distribute each species. (e.g., SCOPS (Klein and Jakob 1999; Webb et al. 2001),
PRECIP-SCOSPS (Zhang et al. 2010), Raisanen et al. (2004))
But what if overlap assumptions for different water phase species are not given?
But what if overlap assumptions differ across schemes within the same model…

Noise/clutterfiltering

62. Scale gap – Model grid << Observation volume
When model resolve cloud and precipitation features at a resolution higher thanwe can observe them=> an option is to apply an instrument forward simulator.
Requires knowledge of:• Radar pulse range weighting function• Radar pulse along track weighting function• Footprint at the ground

500m

200m-3km

Battaglia et al. (2020)

Lamer et al. (2020)7

3. Sensor performance – Effects other than scattering
Models can simulate even minute hydrometeor amounts. Insteadof applying a threshold on model output to match instrumenta) CloudSatfperformance we can reproduced more realisticheightdependenta) CloudSatfsensor detection limitations in forward space.

e) ACCP250e) ACCP250b)
CloudSatfa (500 m range res., -27 dBZ MDS)a) CloudSata)a) CloudSatb)
CloudSatfb) CloudSb)b) CloudSCloudSd)
CloudSata+esc) EarthCAREc)
EarthCARE(500 m range res., -35 dBZ MDS)a)
CloudSata+esfc) CloudSat
EarthCAREd)d) CloudSd)b) CloudSd)CloudSf) ACCPe)
ACCP100250e)
ACCPc)
EarthCARE
250 (250 m range res., -26 dBZ MDS)
f)
e)
ACCP
a)ACCP
CloudSatRadar
100250 ff) ACCP10f)
d)
CloudS
10f)
ACCPb)ACCP
CloudS1e)250c) ACCP
EarthCAREf)10d)ACCP
CloudSafe) ACCP250

Lamer et al. (2020)

Radarreflectivity(dBZ)
Radar
reflectivity(dBZ)

Þ Reproduce:• Range dependent sensitivity lossc) EarthCARE• Signal attenuation by liquid (e.g., Chepferet al. 2008)c) EarthCARE• Blind zone cause by outgoing pulse or surface echo

Forward simulated radar reflectivity
for different spaceborne radars8f) ACCP1

10-3a1) Lidar total copol backscattering (m-1sr-1)

10-4

3. Sensor performance – Effects other than scattering

10-5
10-6
10-3a2) Lidar total copol backscattering attenuated detected (m-1sr-1)

Models can simulate even minute hydrometeor amounts. Insteadof applying a threshold on model output to match instrument
performance we can reproduced more realistic height dependent
sensor detection limitations in forward space.

Total attenuation-40℃

Forward simulated0℃a ) Lidar total copol backscattering (m sr )a ) Lidar lidar
linear depolarization
ratio detected( ) radar reflectivity
backscatterandwith and without instrument model
-1

1
3

-1

-3
10-6

10-4
10-5b1) Radar total co-pol
Totalreflectivity
attenuation(dBZ)-40℃0℃a ) Lidar total copol backscattering attenuated detected (m-1sr-1)a3)2 Lidar linear depolarization ratio detected ( )Totalreflectivity
attenuationattenuated detected (dBZ)
b2) Radar total co-pol
Loss of sensitivity with range-51010-6-6
1010-3-40℃

10-4-40℃

10-50℃a3) Lidar linear depolarization ratio detected ( )10-4-510------ Lidar -------31010-4a2) Lidar total copol backscattering attenuated detected (m-1sr-1)

Total attenuation

10-5

1010-6-30.0a1) Lidar total copol backscattering (m-1sr-1)

Þ Reproduce:• Range dependent sensitivity loss
• Signal attenuation by liquid (e.g., Chepfer et al. 2008)• Blind zone cause by outgoing pulse or surface echo

10-410-60.00℃bb13))Radar
co-pol(dBZ) detected (m s-1)
Radartotal
co-polmeanreflectivity
Doppler velocityb1) Radar
Radar total
total co-pol
co-pol reflectivity
reflectivity (dBZ)
detected-1)bb42)) Radar
co-polDoppler spectral attenuatedwidth detected(m s(dBZ)
Total attenuation
Loss of sensitivity with range

0.0-40℃------ Radar ------

Within (GO)2-SIM, we a stratiform clouds andprecipitation case we found that only ~78 % of
simulated grid boxes could beobserved/assessed by real sensors.

1.00.00℃b2) Radar total co-pol reflectivity attenuated detected (dBZ)
bTotalco-pol mean Doppler Lossvelocity
detected (ms-1)range
attenuation3) Radarof sensitivity
with-40℃
Lamer et al. (2018)

91.00℃b ) Radar co-pol mean Doppler velocity detected (m s-1)b4)3 Radar co-pol Doppler spectral width detected (m s-1)

0.0
1.00.0b4) Radar co-pol Doppler spectral width detected (m s-1)

3. Sensor performance – Non-meteorological echoes
Lamer et al. (2020)

One must remain mindful that forward simulators do not represent
instrument noise, surface clutter, insects and other nonmeteorological targets.
Surface clutter
Signal to noise ratio (dB)ÞAs such it is essential that observations be rigorously quality
controlled, filtered and calibrated before they can be used toguide model development.

ENA KAZR

ArtifactInsects

Linear depolarization ratio (dB)

SGP KAZR

10

4. Evaluation metrics
Quality controlled observations and forward simulated model output can be compared to estimate model performance, but it
may be difficult to track back the cause of any discrepancy
A more informativewayisVerticaltoapplythe same retrieval to observations and forward simulated model output and compare theExtended
Cloud
Structure
Regimes
Hydrometeor
verticallayering
regimes
retrieved property. Useful for ill defined properties like hydrometeord)layerH MxLdepth or liquid fraction in mixed-phasesystems.
Water phase
HM
HxM
HxM L
HxMxL

HHL

HML

H MxL

MLMxLLM

Clear sky

HxMxL

Ice Mixed-phase Liquid

Height [m]High-altitude
530 hPa
delimiting level

MxL• GOCCP approach (Cesana and Chepfer 2013; Chepfer et al 2010),
Mid-altitude
M

Low-pressure altitude
790 hPa
delimiting level• ISCCP approach (Rossow and Schiffer 1991)
Low-altitude
LL• CVS approach (Remillard and Tselioudis 2013, 2015)

3938m

M
1980m

Clear
Sky
100

• HVL approach (Lamer 2019)H
Non-dominantÞFor cloud layering, a possibility is to first establish if cloud are distributed correctly across 3 fundamental level
High-altitude
HL

Lamer (2019)

Relative frequencies of occurrence of liquid, ice and mixed-phase within each dominant regime [%]Relative frequencies of occurrence of dominant hydrometeor vertical layering regime [%]a) Time-height map of cloud and precipitation water phase.

Water phase
Ice Mixed-phase Liquid

HxMxL
HxMxL
HxMxL

HeightHeight[m][m]Height[m]

440 mb3938 m

MxL
HHMxL

MxL
MxL

2985 m/680 mb
1980 mHH
Non-dominantNon-dominant

Height [m]Water phase
Ice Mixed-phase Liquidd)c)

7116 m

LL

ClearClearSkySky

3938m
3938m

MM

1980m
1980m

LL100

Relative
Relativefrequencies of occurrence of liquid, ice and mixed-phase within each dominant regime [%]b) Time series of hydrometeor vertical layering regimes.
HLLMxL

H MxL

HxMxL

MxL

HxM L M L

Relative
Relativefrequencies of occurrence of dominant hydrometeor vertical layering regime [%]

Lc)

H MxL

1980 mH
MxL

Non-dominant

3938 mHeight [m][m]Height

HxMxL
HxMxL

Height [m]113938m

M
1980m

Katia12/04/2018
12
Lamer – PhDmodel
defense evaluation of ModelE3 against ARM obs.
Preliminary

at the North Slope of Alaska
Relative frequencies of occurrence of dominant regimes [%]First metric: Dominant regimes typesa) Observations December 2012

HxM
HxMxLDecember 2012 ARM observations
Height [m]MxL

Non-dominant

5 dominant regimes

Water phase
Ice Mixed-phase Liquid

HM

LL

Clear sky100

December 2011 NASA ModelE3 simulationb) ModelE3 simulation December 2011

HxMxL

Height [m]✓ 4 dominant regimes were simulated
X Missing deep elevated layers (HxM)

MxL

Non-dominant

Water phase
Ice Mixed-phase LiquidHMLClear skyL100

Lamer (2019)

Relative frequency of occurrence of liquid, ice and mixed-phase within each dominant regime [%]12Katia
12/04/201813
Lamer – PhDmodel
defense evaluation of ModelE3 against ARM obs.
Preliminary

at the North Slope of Alaska
Relative frequencies of occurrence of dominant regimes [%]Second metric: Dominant regimes thicknessa) Observations December 2012

HxM
HxMxL

December 2012 ARM observations
Height [m]MxL

Non-dominant

Deep systems (HxMxL) extend to 7266m

Water phaseIce Mixed-phase LiquidHMLL

Clear sky100

December 2011 NASA ModelE3 simulationb) ModelE3 simulation December 2011

HxMxL

Height [m]X Deep systems (HxMxL) are too deep
MxL

Non-dominant

Water phase
Ice Mixed-phase Liquid

HML
Clear skyL

100

Lamer (2019)

Relative frequency of occurrence of liquid, ice and mixed-phase within each dominant regime [%]13

Katia12/04/201814
Lamer – PhDmodel
defense evaluation of ModelE3 against ARM obs.
Preliminary

at the North Slope of Alaska
Relative frequencies of occurrence of dominant regimes [%]Third metric: Dominant regimes occurrencea) Observations December 2012

HxM
HxMxL

December 2012 ARM observations
Height [m]5 % deep elevated layers (HxM)
7 % deep low-level layers (MxL)

MxL

Non-dominant

7 % deep systems (HxMxL)

Water phase
Ice Mixed-phase LiquidHML

33 % low-level layers (L)L

Clear sky100

26 % clear skyb) ModelE3 simulation December 2011

HxMxL

Height [m]OK deep systems (HxMxL)- deep elevated layers (HxM)Clear sky+ 16 % low-level layers (L)HML✓ deep low-level layers (MxL)
OK clear sky

MxL

Non-dominant

Water phase
Ice Mixed-phase Liquid

December 2011 NASA ModelE3 simulation

L100

Lamer (2019)

Relative frequency of occurrence of liquid, ice and mixed-phase within each dominant regime [%]14

Main challenges for comparing model and observations
1. Define hydrometeor properties in terms of the same moments (i.e., mass vs. radar reflectivity)
Þ Apply scattering forward simulator to model output (mass => radar reflectivity, lidar backscatter)
2. Account for the scale differences
Model output scale << Measurement scale => Average model output using instrument IFOV weighting functions
Model output scale >> Measurement scale => Create model sub-columns that follow model assumptions
3. Objectively filter data for performance differenceÞ Forward simulate effects other than scattering (attenuation, sensitivity drop with range, blind-zone)Þ Filter observations for non meteorological echoes (e.g., noise, surface clutter, insects) and calibrate them
4. Define an objective evaluation metricÞ Apply the same retrieval on forward simulated model output and observations(e.g., hydrometeor layer depth, hydrometeor population phase)
5. Identify a suitable evaluation timescale
Þ Case study vs. statistical analysis
6. Define what level of discrepancy is acceptable or not given the uncertainty in the process
15