Looking Back: Lessons Learned
from CALIPSO Validation
Dave Winker,
NASA Langley Res Ctr

EarthCARE Science and Validation Workshop, 13-17 November 2023

1

Validation Objectives• Verify instrument performance (w.r.t. Level 1 data)- calibration, SNR, linearity, transient recovery• Verify geolocation- pointing, altitude registration• Quantify the accuracy and precision of Level 2 science data products- identify sources of random errors and biases• Examine underlying assumptions in retrieval algorithms- Sa, Sc, spectral independence of cirrus backscatter- Validate uncertainty estimates !

2

Validation Resources– Ground-based networks• Aeronet• Earlinet, ADnet– Satellite comparisons• MODIS, MLS, AIRS
• SAGE-III– Targeted, aircraft campaigns• LaRC HSRL (King Air)Jun 2006 - 20012• NOAA ESRL (Cessna)

Jul-Aug 2006-2009• NASA ER-2

Aug 2006, ORACLES– Large field campaigns• NASA AMMA (Cape Verde)

Aug 2006• SAMUM

2006, 2008

• CIRCLE-2

May 2007• NASA TC4 (Costa Rica)

Jul-Aug 2007

• ASTAR/PAM-ARCMIP

April 2007/09/11/12• ARCTAS/PolarCatApril, July 2008

• SEAC4RS

Aug-Sep 2012• etc3

Lessons Learned from On-orbit Checkout• Discovered ranging error– Speed of light ≠ 3.000E+8 m/s– Required a change to payload software (~ 1 week)• Discovered intra-orbit drifts in 1064 channel calibration– Not fixed until 2014

• Discovered excess PMT noise in South Atlantic Anomaly– Not seen during LITE (260 km orbit altitude)

532n m dark noise– Required modification of 532 nm calibration algorithm

4Initial Level 1 Validation• Early validation of Level 1 profiles was critical• Dedicated airborne campaign in Aug 2006 (CC-VEX)

11 August 2006

• Payload on NASA ER-2:– Goddard Cloud Profiling Lidar (CPL)CALIOP– JPL W-band radar (CRS)– MODIS Airborne Simulator (MAS)• Initial CALIOP Level 1 validation objectives:

CPL– Sanity check on Level 1 lidar profiles• Do they ‘look right’? Unexpected artifacts?

– Verify predicted detection sensitivity– A first check on calibration
CALIOP (black)
CPL (blue)5

Long term HSRL campaign: Level 1, Level 2HSRL(~ 2018)

Mean extinction profiles from HSRL
underflights (mostly in eastern US)

Bias =
3.4%

6
CALIOP (US)HSRL (US)CALIOP (Caribbean)
HSRL (Caribbean)

5Altitude (km)CALIPSO

432101E-30.01

0.1-1532 nm extinction (km )6From HSRL aggregated statistics~ 1 km

Sa: HSRL vs. CALIOP:(Rogers et al, 2012)~ 40 km~ 40 km

Characterization of CALIOP detection sensitivity:• Average HSRL to CALIOP resolution• Aggregate measurements of similar aerosol type
• Compare histogram of HSRL bins inside CALIOPlayer with histogram of all HSRL bins
HSRL
CALIOP7

Ground Networks (example): Earlinet (circa 2011)•One of several lidar networks– > 20 stations•

Many 355 nm Raman lidars•A few 532 nm Raman lidars•

Routine observations over many years•

Issues:– Requires time to acquire a sufficient collection ofmatched observations• Spatial mismatches with CALIPSO• Blockage by clouds– Raman is (was?) mostly night-only– Comparison methodology: Perrone et al (2011)8

Ground Networks: Aeronet
Even with > 400 Aeronet sites, need to accumulate matched samples over several years

1.5

Dhadnah (all - 22)

CALIOP AOD

1.0

0.5

0.0
0.0

0.2

0.4

0.6

0.8

1.0

Aeronet AOD
0.6

1.5

0.5

Abu Al Bukhoosh (all - 26)

0.4

CALIOP AOD

1.0

CALIOP AODOmar et al, 2012: 1081 samples over 4 years (600 cloud-free)
Typically, only one or two usable samples/station/year
In addition to CALIOP retrieval errors, AOD biases could be due to:
Spatial mismatch
Topography

Sede Boker (all - 36)

0.5

0.3

0.2

0.1

0.0
0.0

0.5

1.0

Aeronet AOD

1.50.00.0elevation at Aeronet site higher or lower than CALIOP groundtrack
0.20.4

Aeronet AOD

0.6

Aeronet cloud contamination
9

Validation ‘truth’ is not always available⚫

The goal is ‘global validation’⚫
⚫

⚫

Satellite intercomparisons⚫

⚫

Field campaigns are expensive and rare
Ground networks have limited coverage
Depending on parameter and sensor, varies from true validation to 'sanity check'

Consistency checks (does it 'look right’)⚫

Can be useful, especially in the early days

10

Validation using other satellite sensors: CALIOP vs IIR• IIR has been very useful for validating CALIOP cirrus retrievals:• Constraints on cirrus lidar-ratio LUT(daytime, 2008 over oceans)unconstrained (QC=0)
constrained (QC=1)• Identified biases in daytime constrained cirrus retrievals• ‘Validated’ extinction retrieval uncertainty estimates• Resolved MODIS-CALIOP cirrus OD discrepancy:• IIR visible OD more accurate than MODIS (till C6)

cirrus OD retrievals
IIR vs CALIOP constrained(nighttime)

Green: Mean CALIOP-IIR COD differenceRed:
RMS CALIOP-IIR COD difference
Purple: RMS CALIOP COD uncertainty estimates
Light Blue: RMS of IIR COD uncertainty
11

Consistency Checks: Examples
Visual inspection can identify
non-physical results••

Cloud ice-water fraction vs. temperature relations should
be stable over time
A drift in phase vs temperature relationship may indicate
drifts in polarization calibration

12

Calibration of Polarization Gain Ratio (perp/par channel)
Unscaled daytime PGR from “solar background above opaque cirrus”

Difficult validation problems:
latitudinal (intra-orbital) biases
seasonal (intra-annual) biases13

Summary• Never as much high quality validation data as you want (need)– Must be creative• Co-located, accurate, airborne profiles are ideal, but:
– Airborne campaigns are rare, and regional– Also need long-term measurements (ground sites, ‘dedicated’ aircraft)• Often no validation data available for remote regions, so must rely on:– Consistency checks– Statistical comparisons against previous datasets (CALIOP)– Evaluation using models (if they are good enough)• Validation of uncertainty estimates is also important– Value of assimilation improves with good estimates of uncertainties

14

Summary• Never as much high quality validation data as you want (need)– Must be creative• Co-located, accurate, airborne profiles are ideal, but:
– Airborne campaigns are rare, and regional– Also need long-term measurements (ground sites, ‘dedicated’ aircraft)• Often no validation data available for remote regions, so must rely on:– Consistency checks– Statistical comparisons against previous datasets (CALIOP)– Evaluation using models (if they are good enough)• Validation of uncertainty estimates is also important– Value of assimilation improves with good estimates of uncertainties• Don’t forget to intercompare results from different algorithms– Single-instrument vs. Synergistic– European vs Japanese15

We’re looking forward to the launch of EarthCARE …

Good Luck!

16