# Presentation: 420699

## How Well Do OMI’s UV-Based Retrievals Sample the Lowermost Troposphere?

- 	Methods using the ultraviolet absorption of ozone in the Huggins bands have been extremely successful in producing estimates of the total ozone column, providing accuracies to 1%–2% in situations where they can be compared to the best ground-based solar occultation methods.   The scattering, reflection, and absorption of UV radiation have been the basis of the TOMS, GOME, SCIAMACHY, and OMI instruments.  Until recently, only very coarse resolution of the vertical structure of ozone distribution in the vertical has been attempted. Recently, methods using multi-wavelength optimal estimation have shown promise with both GOME and now OMI (X. Liu, et al., JGR, 2005  X. Liu et al., Aura Sci. Team Mtg.)  Alternatively, a lower-atmospheric ozone partial column has been estimated using other measurements.  A very promising development has been the continued development of “OMI minus MLS” (Ziemke et al., 2006) method by using sophisticated trajectory modeling to spread the limited MLS observational opportunities forward in time and space (Schoeberl et al., 2008).
- 	On the other hand, air pollution forecasters and ozone regulatory bodies have insisted that they would like very detailed description of ozone concentrations.  They would prefer useful measurements describing the first ca. 0.8–1.5 km (daytime mixed layer depth); for remote sensing to be relevant. A more achievable goal is to map ozone within the lowest 2–3 km (including an upper ozone-transport layer, one that is frequently mixed, albeit over a few days, by fair-weather clouds.  This is a demanding requirement for  UV techniques, since these partial column are the similar to, sometimes slightly larger, than the current ozone column precision.
- 	Furthermore, the basic TOMS-style retrieval technique — which has been improved over decades — is based on fundamentals of radiative transfer that suggest sensitivity of UV methods to smog ozone is much lower than other ozone.  An often-shown graphical characterization of the retrieval sensitivity (in the vertical) of the OMI/TOMS technique, as estimated  by the forward-calculation theory (P.K. Bhartia, Aura Science Mtg., 2007 ) is ***shown*** ***to the right***.  This is valid for clear-sky conditions.  Sensitivities are ca. 0.2 for 0–1 km,   ca. 0.27 for 1–2 km, and ca. 0.35 for 2–3 km.  This is particularly discouraging when one seeks of < 10 Dobson units (DU) of integrated ozone partial column in the lower atmosphere within a background of 350 DU total column (mostly in the stratosphere).  OMI-MLS Tropospheric Ozone Residual techniques have settled for the easier task of tropospheric ozone, ~35 DU, in hopes that such estimates will be of some use in describing ozone’s pollution and climate impacts.
- 	Despite these theoretical estimates, investigators have repeatedly found variations in ozone column (TCO) or tropospheric ozone residuals (TOR) that correlate surprisingly well with surface ozone (Fishman, Vukovich, ..., J. Appl. Meteor., 1987, Fishman et al., AGU Spring 2008) or lower-tropospheric ozone soundings (e.g., Chatfield et al., GRL, 2004).

*Methods*
*Previous Work, Basic Concepts, and Difficulties*
- 	At the 2007 Aura Science Meeting, we reported a bit-too-simple technique that tried to compare the SHADOZ and IONS ozonesonde datasets with the then-current version of the Schoeberl OMI–MLS technique (***figures to the right***).  The basic idea carries through to the technique presented below.  For Schoeberl TOR, one forms a regression with *TOR**i* as the dependent variable (l.h.s) and information summed up from the sonde  reports (partial ozone columns for atmospheric layers defined by pressure difference (***p**l*) on the right-hand side multiplied by an averaged mixing ratio for that layer ***il* (in fact one characterizes the layers with the appropriate integration over sounding points.)  One seeks a set of sensitivity parameters *a**l* that portray empirically the way that the layer DU small partial-column estimates co-vary with the total TOR estimate.  Since soundings quite variable in shape, there is information with which to inform the empirical estimate of the sensitivity terms *a**l*.  *Note:* *The estimates are independent of any radiative transfer theory, retrieval theory, or practical expedients employed in dealing with clouds, aerosols, or subtle, remaining instrument effects. *Symbolically,
- Where  *i*  indexes sonde/OMI pairs,  *l*  indexes pressure layers  *p**l*  and associated layer-mean pressures  *p**i *,  *il*  indicates mixing ratios from sounding measurements,  *t**i*  indicates an above-burst estimated ozone partial column,  *C*  is a constant for the whole regression, and  ***i*  are the errors associated with the match-up.  The sensitivity coefficients are the  *a**l* , *each estimated separately*.
- 	Several problems were identified with this estimate: first it needlessly conflated effects of the TCO estimation algorithm with the effects of the stratosphere-elimination (TOR) algorithm.  It was clearly desirable to work both with and without the stratospheric-elimination.  Secondly, it became clear that estimation for adjacent layers *a**l* was correlated, possibly leading to over-fitting and very scattered layer-sensitivity estimates. Thirdly, it seemed clear that there must be more information for certain levels than others, but it was not clear how to estimate the information content and therefore the appropriate layer choices.  Fourthly, there were  clear indications of different effects associated with different stations, possibly due to the operation and instrumental character of the ozonesondes, or conceivably due to meteorological and physical effects on the OMI retrieval — somehow related to the ozonesonde launch site. Certain stations like Sable Island often seemed to stand out. Lastly, the error terms were most directly associated with the TOR estimates (lhs), not errors in the ozonesonde measurements (consistently high or low sensitivity to ozone throughout the sounding, or associated with individual layers).
*Cross-Validated, Statistically Guided, Smoothing Technique.*
- 	Recent developments in statistical theory and available software have solved many of these problems, and  so we have begun to address the difficulties.  The “general linear model” software *gam* contained in the *mgcv* package devised and written by Simon Wood (Chapman and Hall Press, 2007, *R* documentation, 2008) allows us to estimate a “variable coefficients model” that defines constrains the parameters  *a**l*  to be evaluated values of a ***smooth spline function *** *s*(*p**l*) , with the smoothness and the evaluation of error contributions to be synthesized in one statistical estimation.  The sensitivity is estimated as a function of layer; the fit is over many soundings.
- 	The formula that was used to seek a best rationalization of the sondes vertical information to the estimate by OMI of a total column was
- Here *i* stands for the total column ozone (TCO), or, as we shall see, other partial columns describing tropospheric ozone.  The term *s*(*p**l*) is constrained in its curvature (second derivative), not its deviation from any starting or preferred value (sensitivity = 0, sensitivity = 1, sensitivity estimated from an asymptotically appropriate forward model, ... or whatever).  If correlation does not establish a sufficient regression effect, the statistical model prefers a constant term *C**i*.  This means that sensitivities normally lie between 1 (excellent correlation with insignificant noise) and 0 (no information is provided by the variability found in the sounding).   The method is essentially linear regression, and so there should (see below) be ***one*** ***solution***, directly achievable, albeit by an iterative method. The restriction on curviness, not primarily on discrepancy from an *a priori*, is also an important difference from Rodgers (2000) and similar (Maddy and Barnet’s linearization points, TGRS, 2007) retrieval methods.  The constant term has *some* similar role to the *a priori*. While optimal estimation analyzes sensitivity considering the *a priori*, this method analyses sensitivity as improvement over a constant mean estimate. Low sensitivities may still allow for useful estimation Each view has its benefits, and the views may be synthesized with further research.  Bayesian methods could be implemented.
- 	To repeat, the new Wood technique for general linear modeling implements a comprehensive strategy in which the estimation of parameters and the degree of smoothing in the spline can be estimated from the data.  Second derivatives of the spline functions are penalized in a uniform way within an existing statistical tradition; the penalty be decreased if one allows a greater chance that random error affects the parameters  Cross-validation compares many subsets of the model data to estimate the uncertainties and error-bounds, so that over-fitting can be avoided; incremental increases in curviness can be seen to have minor influence on explained variance, and rejected.  The *mgcv* package employs the popular “generalized cross-validation” technique; this avoids a minor pitfall of a naïve use of many subsets.

- Robert B. Chatfield, NASA Ames Research Center
-     	*Robert.B.Chatfield@nasa.gov  MS 245-5 Moffett Field, CA 94305 USA*
- P.K. Bhartia, NASA GSFC
- Mark Schoeberl,NASA GSFC
- Xiong Liu, NASA GSFC, UMBC, Smithsonian CfA

**How Well Do OMI’s UV-Based Retrievals Sample the Lowermost Troposphere?**

***Conclusions***
- 	We have made an empirical statistical analysis of the response of two complete retrieval procedures of OMI (ultra-violet) ozone column by exploiting co-variation of ozone at different layers in the atmosphere and the OMI retrieved products.  Similarly, we have analyzed one derived product OMI-MLS tropospheric ozone TOR (surface to 200 hPa version) by its correlation with ozonesonde variations to 200 hPa.  Despite the suspected ~10% variability of ozonesondes’ determination of the ozone column (to balloon burst), the large number of sondes can provided useful information.   There may be surprisingly good detection of ozone down to the lowest kilometers, better than we have expected from some analyses of the forward model for radiation transfer.  There is some additional support suggesting the precision OMI TCO to  be 1–2%.
- 	The use of the generalized linear model in a form that supports “variable-coefficient models” has allowed us to use regression to estimate response of OMI-based retrievals,  with a set of coefficients smoothly varying with altitude that multiply narrow sonde-derived layer ozone partial column amounts.  The statistical technique itself usefully provides information about the degree of fitting that appears to be appropriate. The sensitivity measure differs from Bayesian methods and other retrievals based on forward radiation models, but it does not disagree and appears to provide a useful, separate, empirical view of sensitivity.
- 	There are reasons to be cautious: other analyses have strongly suggested that there are variations in the typical match-up of sondes to OMI from station to station, with some stations particularly at variance.  These are probably more easily attributed to ozonesonde procedures and sonde properties than they are to special observing situations affecting the station location. More detailed and varied analysis is needed for the effects of clouds.  Additionally, there are reasons to suspect that variations in sensitivity of the sondes down as far as even 50 hPa, perhaps affected by pump degradations or their corrections.  However, variance at these levels is small.  For these and other regions (e.g., the middle troposphere), it may be that the true variation is sufficiently low that, in the presence of various errors, the general linear model regression has little ability to “prefer” high sensitivity  at those levels to a constant value.  Sensitivity *s*(*p**l*) at these levels is nevertheless poorly defined in a relative sense.  These results make more precise ideas about sensitivity based on situation analysis (Chatfield et al., 2006) and from estimates based on unconstrained simple linear regression (Chatfield et al., Aura Science Mtg., 2007).
- 	UV methods do seem to sense variations in tropospheric ozone closer to the surface than we might have expected, although perhaps still not so well.  We do not yet understand when such sensitivities seem to be high or seem to be low. UV methods do not appear yet to give much PBL (0 to 1-2.5 km) information for typical clear-sky significant smog episodes. No better than 6 km resolution, at the best, appears to be currently available from the optimal estimation technique.  There is much more to be understood about sondes, clouds, and retrievals as they actually affect estimates.

*Schoeberl OMI–MLS Tropospheric Ozone Residual Sensitivities*
- 	A new form of estimate of tropospheric ozone residual (TOR) was released by Mark Schoberl in September, 2008.  It was based on OMI-TOMS Version 3 and with several improvements including better mapping of MLS stratospheric ozone.  The estimate provides a partial ozone column for that part of the troposphere *above the equivalent cloud-top pressure*, or equivalently, an appropriate layer-averaged ozone mixing ratio.   Maps of the quantity must be used with *care for the variation of effective cloud top*.  We have analyzed the surface–200 hPa version of these recently, and display them for comment. The large number of sondes used reflects the reduced need for screening.  (***Mid-latitude sondes, above left;  tropical sondes, right***) Note the very small intercept estimates, –2 to 2 DU, and high sensitivities, ~0.5–1.0.  The zero-sensitivity line is off the graph to the left in each case.  Twenty-five tropospheric levels were used to estimate *s(p**l**).*
- 	These seem to indicate relatively good accuracy for the product. We would like to examine the accuracy of the new TOR, distinguishing mixing-ratio variation from effective-cloud-top variation, as resources may permit.  The large variation of estimated sensitivity for the mid-latitude sondes (left panel) caused us to seek fits with less curvature than was justified by generalized cross-validation. The low-sensitivity estimates in the upper troposphere for both mid-latitude and tropical analyses seem to occur near the climatological levels of upper-troposphere cloud tops for the respective cases.

*X. Liu Many-**** Optimal Estimation of the Distribution Tropospheric O**3*
- 	The X. Liu total ozone analysis is broken into ***layers on the right***.  (***Leftmost***) Total ozone column (TOC), followed by (***Proceeding to the right***) sensitivity estimates for a sequence of layers. These are *layers that are significant for the study of smog and tropospheric ozone*, not originating from an analysis of degrees of freedom for signal.  We composed these layers using the resolving information in the Liu dataset for each sounding and our own estimate, from the sondes, for the same layers.  Clearly, analysis for PBL ozone should not be derived from a 1015-900 hPa layer estimate.  More information for the boundary layer comes in the estimates for layers up to 500 hPa, or better, thicker layers reaching to approximately 500 hPa.  The 500-350 hPa layer estimate seems to provide a good *empirical* portrayal of the vertical resolution of the method.  The 350-200 hPa region shows the same complex, problematic behavior as the OMI-MLS method using OMI/TOMS.

*Sensitivity of Version-3 OMI/TOMS to Ozone at Differing Levels of the Atmosphere*
- The basic question of sensitivity to the lower troposphere in present distributed products begins here.  The version-3 ozone  product is reported with an effective cloud pressure and an amount of ozone that can be filled “below cloud” to provide a map-able “old-standard”product TCO. These quantities take into account partial cloudiness and sensed UV radiation from within clouds with simple linear approximations. We analyzed “observed-only” ozone to cloud tops, and used only appropriate level contributions computed for IONS and SHADOZ sondes, under restrictive conditions of burst at or above 10 hPa. (***Above left***) Sensitivity estimates, depicted on the horizontal axis,  for the “observed-only” ozone have a decrease towards the surface similar to the the forward radiative-transfer calculation, but the statistics find enough information to describe a curve upward towards the surface! 150-hPa ozone is sampled with excellent sensitivity (probably limited by the sonde accuracy), but sensitivity above ~50 hPa decreases (we speculate that pumps and pump-pressure corrections vary significantly).  Above-burst ozone, estimated using the McPeters et al. (JGR, 2005) climatology, does not show useful correlated variation, but see concluding comments below.  With care, we may add more sondes and categorize groups of sondes to highlight the most important variables .
- (***Above right***) Surprisingly, the inclusion of climatological (*a-priori*) ozone near the surface increases “sensitivity” in this sample. Ozone “invisibility” may correlate with meteorological variability and high or low ozone.  Note, however, the large spread between the two dashed lines indicating two standard error estimates. Ozone column is typically estimated with *R*2 > 0.94, and a standard deviation ~2% of the mean.  *This study is preliminary, and we would prefer a bit more feedback and experience before quoting more on accuracies.*  Direct solar occultation measurements are no doubt a more direct measure of accuracy for OMI retrievals in the special circumstances of clear skies. Note for these cases that clear skies seem to be correlated with ozone in the lowermost troposphere. Nevertheless, the relative success of the fitting at this stage points to a useful role for sondes.
- (Interesting tidbit) When we omit the estimation of a constant (like 130.8 or 141.6 DU) above, the sensitivity of the near-burst region increases!	This forces higher sensitivities in some regions: in this case the linear regresion “reaches” for the relatively low-variance layer above ~50 hPa.  Similar considerations may or may not explain the dips  in “sensitivity” in the mid-troposphere: an analysis of variances should be carried out.

- 	The mean estimate of sensitivity is high near the surface, ~0.85, but the standard error bands are broad.  In this analysis we used both tropical and midlatitude sondes, but *did not take into account the presence of clouds*.  As we are able to collocate more samples, we expect to distinguish more effects.  As mentioned with the TOMS technique, the low sensitivity near burst need not affect error estimates ot the total strongly: these regions may be relatively more constant in mixing ratio, so that a constant serves well.  More extensive study is required to make the lower-tropospheric estimates firm.

*Sensitivity of the Observed Portion of the Atmosphere “Above Cloud”*

*Sensitivity of the Common Total-Ozone Product Including Climatology “Below Cloud”*

*Sensitivity of the Common Total-Ozone Product Including Climatology “Below Cloud”*

- Samuel Oltmans, NOAA GMD
- Robert Esswein,  NASA Ames and BAER Inst
- David Tarasick, AQRB, Environment Canada
- Anne Thompson, Pennsylvania State Univ

- 1015-900 hPa

- 900-700 hPa

- 700-500 hPa

- 500-350 hPa

- 350-200 hPa

- 200-0 hPa

- 1015-0 hPa

- Mid-latitude OMI-MLS

- Tropical OMI-MLS

*The Goal: 0-1.5 km Smog O**3**:** *A difficult goal is to map air pollution in the lowermost troposphere with detail exceeding this model simulation, performed using a 36-km grid.  Better detail, showing details of 5–10 km within cities is useful to improve local understanding, but this scale is useful in understanding and predicting regional impacts 150–2500 km downwind.  Courtesy: Georg Grell, NOAA.

*Sensitivity  of X. Liu Many-**** Optimal Estimation of Total Column O**3*
- 	Xiong Liu (with Kelly Chance, Mike Newchurch, and P.K. Bhartia) have estimated total ozone column and specific height-distributed sub-columns (discussed below) using the spectral distribution of absorption in the Huggins bands.  A limited number of sondes satisfied our criteria of coincidence: one degree latitude and one degree longitude away and on the same day. (Very likely the hour was appropriate, as Liu has concentrated on retrievals at instances when there were sondes. We can likely extend the number of coincidences.)

*a**l*

*s(p**l**)*

*s(p**l**)*

*s(p**l**)*

*s(p**l**)*

* **compare to s(p**l**)*

*Thanks to Nathaniel Livesey and the whole MLS team for the MLS products, *
*...**to the many field scientists launching ozone sondes,*
*...**to Jaquelyn Witte and the WCRP for collating the information,*
*...**to Vance Fong and Region-9 US EPA for stimulating greater interest in making remote sensing relevant to health and pollution problems,*
*... **to Johan de Haan for insight and useful suggestions,*
*... **to Simon Wood for communicating his newest refinement of mgcv,*
*... **and to many scientists referenced and unreferenced for discussions. *
*Supported primarily by the NASA ACMAP/Aura program.*

*s(p**l**)*