NOVEL RETRIEVALS OF AEROSOLMICROPHYSICAL AND RADIATIVE PROPERTIESFROM LIDAR OBSERVATIONS IN FUTURE
SATELLITE MISSIONSIan Chang
Research Scientist

Rachel EmeryUG Student

Abiola Fakoya
Grad Student

Brad Lamkin
Grad Student

Emily Lenhardt
Grad Student

Logan Mitchell
Grad Student

Connor Flynn
Lan GaoSen. Research Scientist Research Scientist

Jens Redemann1, L. Gao, F. Xu, C. Flynn, E.
Lenhardt, L. Mitchell, T. Thorsen, J. Yorks, S. Burton,
J. Hair, A. Nehrir, R. Ferrare, C. Hostetler
1School of Meteorology – OU

jredemann@ou.eduJens Redemann
SoM Faculty

Emily West
Grad Student

Affiliated SoM group leads:

EarthCARE workshop, Frascati, Italy - Nov. 13, 2023
Marcela Loria
PM2.5/Aerosol satellite obs

Greg McFarquhar
Cloud m-physics

Feng Xu
Radiative Transfer

Motivation: Vertically resolved CCN⮚ Column-effective aerosol quantities may not be relevant to aerosolcloud interaction.

⮚ The uncertainty of CCN-AOD parameterization is large, depending on:▪ Aerosol Type▪ Vertical distribution▪ Humidity response of light scattering▪ Spatiotemporal variability

Rosenfeld, et al. Science 2008Stier, 2016: “…71 % of the area of the globe shows
correlation coefficients between CCN0.2 % at cloudbase and aerosol optical depth (AOD) below 0.5, i.e.
AOD variability explains only 25 % of the CCNvariance” – model-based, self-consistent.

Physics-based retrieval of CCN using lidar and polarimeter observations (Gao et al, AGU 2021)⮚ For NASA AOS retrieval simulations, we developed aphysics-based optimal estimation (OE) approach forlidar+polarimeter retrieval of speciated aerosol profiles.
⮚ Used aerosol reanalysis product to estimate the bulk
hygroscopicity parameter for aerosol mixture.
⮚ Applied κ-Köhler theory to calculate CCN concentration.

(Xu, F., Gao, L., Redemann, J., et al.,2021)

Limitations – AOS retrieval simulation “lessons learned”▪ Great dependence on a priori information (aerosol size
distribution and chemical composition) to retrieve CCN▪ Computationally very expensive.
3

The Machine Learning alternative⮚ Collocate HSRL-2 and in-situ measured CCN
from multiple campaigns.
✔ ACTIVATE, CAMP2EX, DISCOVER-AQ, ORACLES⮚ Train neural networks for different sets of lidar
observables (e.g., ATLID, NASA AOS).
✔ HSRL-2: 3β + 2α + 3δ✔ HSRL-1: 2β + 1α + 2δ✔ EarthCARE/ATLID: 1β + 1α + 1δ✔ Simulated-Elastic-Backscatter (SEBL): 2β + 2δ⮚ Evaluate model prediction using in-situ
measured CCN or ABS✔ Correlation coefficient (R)✔ Mean absolute error (MAE)✔ Mean relative error (MRE)Training the ML model▪ Aircraft observations (lidar and in situ) from multiple campaigns
Lidarobservables
Ancillary
In situEXT355, EXT532,BSC355, BSC532, BSC1064,DEPO355, DEPO532, DEPO1064

Relative humidity (RH),
Temperature (T)CCN concentration
at 0.4% SS (~9,900)
Absorption, ABS (~2,700)

MACHINE LEARNING - HOW TO DEAL WITH THE BLACK BOX…
Before machine learning:”‘Garbage in, garbage out’
– computer science▪ Remove data that has large uncertainties▪ Lidar✔ Negative lidar observables✔ Aerosol depolarization ratio greater than 1▪

In situ✔ CCN below 10 cm-3
✔ ABS below 0.1 Mm-1

Algorithm selection:▪
▪

Supervised regression learning problem with large number ofnumerical features.
Fully-Connected Neural Network (FCNN) regression model

Architecture setup:▪
▪
▪
(https://xkcd.com/1838/)

Training data: 70%, Testing data: 30%
10-fold cross validation
Hyperparameters are tuned iteratively during the training using
Bayesian optimization

Simulation of ML retrievals: CCN/ABS for full set of HSRL-2 observables (3β + 2α + 3δ )WITHOUT Reanalysis

CCNABS

WITH Reanalysis

Simulation of ML retrievals: CCN/ABS for EarthCARE/ATLID observables (1β + 1α + 1δ )WITHOUT Reanalysis

CCNABS

WITH Reanalysis

Mean relative error and fraction of predictions within 50% uncertainty

Significant Boostin retrieval
performance for
ATLID fromaddingreanalysis data
(T, RH) asconstraints

Simulation of ML retrievals: CCN/ABS for EarthCARE/ATLID observables (1β + 1α + 1δ )WITHOUT Reanalysis

CCNABS

WITH Reanalysis

WITH 50% noise & Reanalysis

Test for ATLID observables with incomplete in situ data: limited range of training data

Provide only center 80% of CCN for training , but attempt prediction for full range of CCN pdf

Grey-shaded areas
excluded in trainingTest for ATLID observables with incomplete in situ data: Discontinuous training data

Provide only intermittent of CCN for training, but attempt prediction for full range of CCN pdf

Grey-shaded areas
excluded in trainingCCN/ABS for HSRL-2, HSRL-1 and EarthCARE/ATLID observables without reanalysis
HSRL-2: 3β + 2α + 3δ

CCN

ABS

HSRL-1: 2β + 1α + 2δATLID: 1β + 1α + 1δ

CCN/ABS for HSRL-2, HSRL-1 and EarthCARE/ATLID observables with reanalysis

CCNABS

HSRL-1: 2β + 1α + 2δATLID: 1β + 1α + 1δWill be submitted to Science this week

HSRL-2: 3β + 2α + 3δ

CCN predicted by ML model for ER-2 flight across Southeast Atlantic - Aug. 26, 2016

Mean Absolute (Relative) Error of CCN and ABS predictions for all and pristine conditions

Predictor Data
set →ATLID observables

Predictor
Indicator →ATLID observables+ Reanalysis Data

ATLID observables + 50% noise+ Reanalysis Data

Mean Absolute Error (Relative)
Pristine
0<CCN<100
0<ABS<0.5

All conditions

Pristine
0<CCN<100
0<ABS<0.5

CCN [1/cm3]

213.1 (33.8%) 192.5 (345%)

94.2 (15.0%)

79.6 (142.5%) 148.5 (23.4%) 146.1 (268.4%)

ABS [10-6 m-1]

0.55 (32.0%)

0.43 (24.9%)

0.26 (93.1%)

All conditions

0.29 (104.4%)

All conditions

0.5 (28.3%)

Pristine
0<CCN<100
0<ABS<0.5

0.31 (109.3%)

Novel Retrievals of Aerosol Microphysical and Radiative
Properties from Lidar
Conclusions
1. We trained ML algorithms using airborne HSRL-2 observations collocated with in situ CCN (N≈9,900)
and ABS (N≈2,700) measurements to predict CCN/ABS from lidar observables.

2. ML models have been adapted to many sets of future spaceborne lidar obs, incl. EarthCARE/ATLID,and tested with high-accuracy HSRL-2 data as input (overly optimistic, but necessary).
3. For ATLID observables, ML models predict CCN and ABS with mean relative errors of 30-35%.

4. Adding reanalysis data (T, RH) boosts CCN errors to ~15% and ABS to ~25%.
5. Performance depends greatly on completeness of training data.
6. For pristine conditions, CCN/ABS retrieval errors are much higher (partly due to sparse training data).

7. Actual retrieval errors for spaceborne systems will depend on error characteristics.
8. Philosophically, the paradigm should use the airborne HSRL-trained ML models, as the low
uncertainties provide maximum likelihood for ML models to discover non-linear and multi-variate
correlations between lidar observables and other aerosol properties.

Fig.S1: flowchart
Collocation of lidar and in situ data

Predictors:

Responses:

Lidar observables,
reanalysis

Training data

MLtraining(FCNN)

Optimal model

Aerosol properties

70% 30%

Testing data

Modelevaluation
using in situ
data

Table1: List of lidar systems and aerosol variables used in this study

Aerosol variables and wavelengths (nm)Lidar system

Depolarization
ratio

Extinction

Backscatter

HSRL-2 (3β + 2α + 3δ)355, 532

355, 532, 1064 355, 532, 1064

HSRL-1 (2β + 1α + 2δ)532

532, 1064

532, 1064

Elastic-backscatter-lidar (2β + 2δ )-532, 1064

532, 1064

EarthCare-like-lidar (1β + 1α + 1δ)355

35535519

New paradigm for aerosol
retrievals from lidar
General ML model

Optimized ML model• Trained with collocatedsuborbital HSRL-2 (alll’s) & in situ data
• Augmented withreanalysis data
• Tested with HSRL/in situ
data

• Retrained for specificlidar observables• Augmented withreanalysis data
• Tested with HSRL/in situ
data

Application to ANY
Lidar• Applied to• EarthCARE• AOS• Ground-based• Airborne• Tested with in situ ?

“Best case”

Lidar-type specific, e.g.

Lidar-system specific, e.g.

3β + 2α + 3δ

1β + 1α + 1δ

1β + 1α + 1δAccuracy of aerosol
retrievals depends on
HSRL uncertainties &information content

Accuracy of aerosol
retrievals depends on
HSRL uncertainties &information content

Accuracy of aerosol
retrievals depends on
system-specific uncertainties& information content

Some deficiencies in climate models
Black/brown carbon and associated absorption→ Treatment ofcarbonaceous
aerosol lifecycle(and its impact onabsorption) is vastlydifferent between
modelsAerosol Single Scattering Albedo (SSA)
𝑘𝑠𝑐𝑎𝑡𝑡𝑒𝑟𝑖𝑛𝑔
𝑘𝑠𝑐𝑎𝑡𝑡𝑒𝑟𝑖𝑛𝑔
𝑆𝑆𝐴 ==
𝑘𝑒𝑥𝑡𝑖𝑛𝑐𝑡𝑖𝑜𝑛
𝑘𝑠𝑐𝑎𝑡𝑡𝑒𝑟𝑖𝑛𝑔 + 𝑘𝑎𝑏𝑠𝑜𝑟𝑝𝑡𝑖𝑜𝑛
𝑆𝑆𝐴 =𝑘𝑒𝑥𝑡𝑖𝑛𝑐𝑡𝑖𝑜𝑛 −𝑘𝑎𝑏𝑠𝑜𝑟𝑝𝑡𝑖𝑜𝑛
𝑘𝑒𝑥𝑡𝑖𝑛𝑐𝑡𝑖𝑜𝑛=1-𝑘𝑎𝑏𝑠𝑜𝑟𝑝𝑡𝑖𝑜𝑛
𝑘𝑒𝑥𝑡𝑖𝑛𝑐𝑡𝑖𝑜𝑛

Brown et al., 2021