dataesr/scientific-paragraphs-categorization huggingface dataset:
By implementing PCA over a training dataset, the original fields can be transformed to superpositions of a set of mutually orthogonal vectors, or principal components (PCs). Then, the original aerosol fields can be approximately reconstructed using a linear combination of a certain number of PCs, with accuracy dependent on the number of PCs employed. effective radii of fine ( r eff, fine ) and coarse ( r eff, coarse ) mode aerosols, column effective real part of refractive indices ( n r, 1-4 ) at 0. 439, 0. 675, 0. 870, and 1. 018 m, respectively, imaginary part of refractive indices ( n i, 1-4 ), and spherical particle volume concentration ( C v,sphere ). Though not specified by use of legend, each color is associated with an independent set of retrieval parameters. Top right panel: spatial and temporal mean of all retrievals and the first four principal components (PCs).
The scatter plot is colored by the density of the points. Each panel corresponds to a specific aerosol property indicated in the title. This empirical correlation among aerosol parameters motivates reduction of the dimensionality of the retrieval parameter space and acceleration of the radiative transfer (RT) computations. Benefiting from the empirical aerosol correlations and orthogonality of PCs, increased computational efficiency is achieved by developing a PC-based fast multi-pixel RT computation scheme, as described in Section 4. A straightforward way to reduce dimensionality of the retrieval space is to first derive PCs from a training dataset, such that the retrieval problem reduces to determination of the pixel-dependent PC weights. To account for possible errors in the precomputed PCs, the correlation-based inversion should allow adjustment of the elements of the PC vectors.
This is not straightforward, as the vector elements vary significantly in magnitude. A methodology for handling this issue is discussed in Section 3. PCA has been used to significantly accelerate hyperspectral RT computations  [7] [8] [9]. Following a determination of representative components from a training dataset of trace-gas and solar-induced chlorophyll fluorescence, the hyperspectral inversions of these quantities are sped up by retrieving PC weights  [10] [11] [12]. In another retrieval application, Multi-angle Imaging SpectroRadiometer (MISR) operational aerosol retrieval over land uses spatial contrasts to derive PCs of the surface related contribution to the top-of-the-atmosphere (TOA) radiances  [13, 14]. The similarity in the angular shapes of surface bidirectional reflectance factors among MISR's four spectral bands in the visible and near-infrared  [15]  is an example of a prior empirical constraint used to stabilize the aerosol retrievals.
The individual fields are then constructed from the PC weights and PC vectors. The smoothness constraints on those fields, as implemented in the original "multi-pixel algorithm" used with airborne data at JPL  [18, 19] , are adapted to the current correlated multi-pixel inversion approach. Both "multi-pixel" and "correlated multi-pixel" retrieval approaches utilize a one-dimensional (1D) code based on the independent pixel approximation  [20]  to ensure forward modeling efficiency. This paper is organized as follows. Following an algorithm overview in Section 2, the correlated multi-pixel retrieval algorithm and error analysis are formulated in Section 3. A faster forward vector (polarized) RT model for a coupled atmosphere-surface system is presented in Section 4. In Section 5, retrieval tests are performed using 27 datasets acquired by the Airborne Multiangle SpectroPolarimetric Imager (AirMSPI) during multiple NASA field campaigns.
An overview of the correlated multi-pixel approach and its mapping to different sections of this paper is shown in Figure  3. Symbols and abbreviations used in this work are found in Appendix A (Table  A1 ). A training dataset of correlated parameters is used to derive the spatial and temporal mean of correlated parameters over a targeted area, along with PCs of the correlated departures from the mean. Together with the uncorrelated retrieval parameters, a state vector is initialized with elements specified in Section 3. 1. In addition to observational constraints provided by the remote-sensing observations used in the retrievals (Section 3. 2. 1), convergence and robustness retrieval are optimized by imposing additional constraints. For correlated fields, the additional constraints include: (a) the spectral variation of certain correlated fields (e.
6. In PC space, the smoothness constraints can be applied to the PC weights w and vectors v separately. Application to across-pixel weights w, discussed in Section 3. 2. 3, forms the sixth type of constraint. 7. Similarly, application of the smoothness constraints to certain type of parameters within a PC vector v k , also discussed in Section 3. 2. 3, forms the seventh type of constraint. Although it appears that the sixth and seventh types of constraints applied to "w" and "v" separately are redundant with the fifth type of constraint that ensures a smooth variation of overall correlated field constructed from PCs; they are helpful when poor initial guesses of "w" and "v" are provided by a training dataset. 8. The eighth type of constraint, formulated in Section 3. 2. 5, imposes a zero sum of the PC weights.
9. Determination of Lagrange Multipliers The Lagrange multipliers reflect the strength of smoothness constraints for a given parameter to be retrieved. Each multiplier is defined as, γ i = ε 2 i=1 /ε 2 i ( 93 ) where ε 2 i are the first diagonal elements of the covariance matrices corresponding to i-th type of constraints. To estimate ε 2 i for a given parameter to be retrieved (f = x), which is a function of z, the most unsmooth known solution f us (z) over the target domain is used, as suggested by Dubovik and King  [25] , namely, ε 2 = z max z min d m [ f us (z)] d m z 2 dz ( 94 ) where z min and z max specify the lower and upper bound of z. For i = 3, 4, 5, γ i is evaluated using least smooth known solution of correlated parameter x over the target domain. For i = 6 and 7, γ i is evaluated using the least smooth known solution of PC weights and PC vectors, respectively, which are derived from a training dataset over the target domain.
Retrieval Options A priori or first guess of PCs can be obtained from running PCA on a climatology database, field measurements, or model output. In the case of PCs for which confidence in their contents is low, simultaneous retrieval of PC weights and PC vectors has to be performed. For high confidence PCs, a two-step retrieval can be used to speed up the retrieval. That is, we run PC weight retrieval using the predetermined PCs first and then relax PCs during the retrieval to refine the solution. This relaxation is one way to capture unexpected variance that exceeds that represented by the predetermined PCs. In the extreme situation, where PCs are not representative of the actual aerosol properties, the scene model can be reproduced by using N PCs, where N equals to the number of aerosol parameters.
1. 1); (2) coupling reflection/transmission matrices of the atmosphere with surface reflection pixel-by-pixel to model the TOA observations (Section 4. 1. 2). For retrieval use, evaluation of the Jacobian matrix that contains the derivatives of TOA observations with respect to all aerosol and surface parameters is described in Section 4. 2. To further ensure modeling efficiency, the correlated aerosol microphysical properties derived from their PCs are input to a light scattering database  [30]  for determination of aerosol scattering properties using interpolation. The modeled scattering properties include aerosol optical depth (AOD), single scattering albedo (SSA), and phase matrix. To model surface reflection, the Rahman-Pinty-Verstraete (RPV) model  [31]  is used to calculate the unpolarized surface bidirectional reflectance distribution function (BRDF).
Inversion of Aerosol and Surface Properties Section 3 provided a general algorithm formulation to retrieve correlated and uncorrelated parameters. This section describes practical specifics in using AirMSPI data to retrieve aerosol and surface properties. In doing so, we assume correlations in aerosol properties, no correlation in surface properties, and no correlation between aerosol and surface properties. The first guesses of PCs, PC weights, and mean aerosol fields are derived from a training dataset, which contains all historical AirMSPI retrievals for selected scenes with the criteria of well-calibrated measurements, clear-sky conditions, and collocated ground observations. The aerosol fields include spectrally dependent real and imaginary parts of the aerosol refractive index, volume concentrations of multiple aerosol size components, nonspherical particle fraction, and Gaussian profile-based parameterization of aerosol layer height and standard deviation  [19].
Though not noted here, the angular shape parameter "g" with the RPV model needs to be offset by a constant before being transformed into logarithmic space. An overview of correlated multi-pixel inversion of aerosol properties and surface reflection algorithm flow is shown in Figure  7. As part of the state vector, the PCs of the correlated aerosol microphysical properties are derived from a training dataset from climatology or other sources. As another part of the state vector, uncorrelated parameters (such as surface reflection properties) are initialized to be static  [19]. The inversion is stabilized by applying a priori constraints on smooth variations of certain aerosol and surface properties in spatial and spectral dimensions. Iterations repeat until convergence is achieved. For the retrieval tests in this study, it takes five to seven iterations for the solution to converge.
As demonstrated in earlier POLDER and AirMSPI retrievals  [6, 19] , imposition of temporal constraints on the variation of surface reflectance can improve aerosol and surface retrievals. In this case, multiple scenes acquired from revisits of a target have to be used. This functionality is not turned on in the following retrieval tests to simplify the demonstration. An overview of correlated multi-pixel inversion of aerosol properties and surface reflection algorithm flow is shown in Figure  7. As part of the state vector, the PCs of the correlated aerosol microphysical properties are derived from a training dataset from climatology or other sources. As another part of the state vector, uncorrelated parameters (such as surface reflection properties) are initialized to be static  [19]. The inversion is stabilized by applying a priori constraints on smooth variations of certain aerosol and surface properties in spatial and spectral dimensions.
AirMSPI Datasets The retrieval algorithm is designed to retrieve column aerosol and surface reflection properties from observations by AirMSPI, which flies on NASA's ER-2 aircraft at an altitude of 20 km and operates in eight spectral channels: 355, 380, 445, 470*, 555, 660*, 865*, and 935 nm, with the asterisk denoting polarimetric bands, in which the Stokes parameters Q and U are measured in addition to radiance I. Images of each targeted area were obtained at 9 viewing angles: 0° (nadir), ±29°, ±48°, ±59°, and ±66° in AirMSPI's step-and-stare mode. At nadir, the imaged area covers a 10 km x 11 km region and the data are mapped to a 10-m spatial grid. Without using the water-vapor influenced band (935 nm), a total of 117 signals per pixel are used, which include 63 radiances (transformed to logarithmic space in retrieval) at nine angles and seven spectral bands, and 27 signals of q=Q/I and Forward RT model Jacobian matrix (K q ) Construct constraints on a): (x̄q ,corr , w q , v q ) and x q,surf regarding smooth aerosol and surface property variations b): sum-to-zero of PC weights and orthogonality&unity of PCs x q = [x q,aer ; x q,surf ], where x q,aer is derived from (x̄q ,corr , w q , v q ) via Eq.
PCA of correlated fields in a training dataset from climatology or other sources Update x̄q +1,corr ,v q+1 and w q+1 in x q+1 by reapplying PCA to the constructed correlated fields Solve Eq. (  87 ) for x q Spectral, angular & polarimetric observations from a groups of pixels x q = x q+1 t q =1 Evaluate cost function  q+1 2 Evaluate modeled data and cost function  q 2 No Yes Output x retrieved = x q+1  q+1 2 <  q 2 - q+1 2 / q 2 < Forward RT model t q = t q /2 Yes Evaluate uncertainty for all retrieval output.
AirMSPI Datasets The retrieval algorithm is designed to retrieve column aerosol and surface reflection properties from observations by AirMSPI, which flies on NASA's ER-2 aircraft at an altitude of 20 km and operates in eight spectral channels: 355, 380, 445, 470*, 555, 660*, 865*, and 935 nm, with the asterisk denoting polarimetric bands, in which the Stokes parameters Q and U are measured in addition to radiance I. Images of each targeted area were obtained at 9 viewing angles: 0 • (nadir), ±29 • , ±48 • , ±59 • , and ±66 • in AirMSPI's step-and-stare mode. At nadir, the imaged area covers a 10 km × 11 km region and the data are mapped to a 10-m spatial grid. Without using the water-vapor influenced band (935 nm), a total of 117 signals per pixel are used, which include 63 radiances (transformed to logarithmic space in retrieval) at nine angles and seven spectral bands, and 27 signals of q = Q/I and another 27 signals of u = U/I in the three polarimetric bands.
Retrievals for all pixels of a surface area viewed from all 9 angles are performed simultaneously. The measurement errors are adopted as 4% for radiance (to account for angle-to-angle and pixel-to-pixel uncertainties) and 0. 005 for DOLP. A wide range of atmospheric conditions and terrestrial environments have been covered by AirMSPI during over a hundred flights in several airborne campaigns. In this paper, we use AirMSPI data from the Polarimeter Definition Experiment (PODEX) (January to February 2013), Studies of Emissions, Atmospheric Composition, Clouds, and Climate Coupling by Regional Surveys (SEAC 4 RS) (August to September 2013), CalWater-2 (January to March 2015), and Imaging Polarimetric Assessment and Characterization of Tropospheric Particulate Matter (ImPACT-PM) (July 2016) campaigns.
The retrievals are performed over the area viewed in common at all 9 AirMSPI view angles, outlined by the yellow box. Figure  8b  shows TOA BRF at 445, 555, and 660 nm in the left, middle, and right panels, respectively, for the retrieval area with spatial resolution ~1. 0 km. Figure  8c  shows maps of retrieved AOD, SSA, and surface albedo (A surf ) at 555 nm in the left, middle, and right panels, respectively. The retrieved AOD, SSA, and volume weighted aerosol size distribution for the atmosphere above the super-pixel closest to the Baskin site is compared to the AERONET reference data in the left, middle, and right panels of Figure  8d. Good agreement (quantified below) is obtained for all of these quantities, except for the coarse particle size distribution, likely due to the lack of bands longward of 1000 nm in AirMSPI.
Generally, the difference between AirMSPI retrievals of AOD, SSA, and size distribution and AERONET reference data is within their retrieval uncertainties. The AirMSPI uncertainties plotted in Figure  8d  are estimated as the root mean square of the retrieval uncertainties of these aerosol quantities and the standard deviation of their variations over the whole image. The AERONET uncertainties consist of two parts: temporal variation within the ±~1-h window centered on the AirMSPI nadir overpass time, and aerosol measurement and retrieval error  [34]. A temporally closest AERONET reference data was identified compare to AirMSPI retrieval at the spatially closest pixel. To account for airmass change during the measurements, the ±~1-h window centered on AirMSPI nadir overpass time is used to calculate the AERONET uncertainty from temporal variation.
The AOD results for seven AirMSPI spectral bands are plotted in different colors: pink (355 nm), purple (380 nm), dark blue (445 nm), light blue (470 nm), green (555 nm), red (660 nm), and brown (865 nm). Linear regression is performed to obtain slope a, intercept b, as well as the coefficient of determination R 2. The mean absolute difference (MAD) in AERONET and AirMSPI results is also calculated to measure the overall deviation. Strong correlation and low bias (R 2 ≥ 0. 88, a ~0. 90, b ≤ 0. 05, and MAD < 0. 02) are observed. It can also be observed that a variation of aerosol loading by ~30% around the mean (namely in the range 0. 28 ≤ AOD 445nm ≤ 0. 40 with mean value 0. 32) across the retrieval area is captured by the correlated multi-pixel inversion. Implementation of our approach using several datasets with even higher (~90%) variation of aerosol loading over several smoke scenes acquired by AirMSPI during the recent Aerosol Characterization from Polarimeter and Lidar (ACEPOL) campaign showed the algorithm to be capable of capturing this variation.
Implementation of our approach using several datasets with even higher (~90%) variation of aerosol loading over several smoke scenes acquired by AirMSPI during the recent Aerosol Characterization from Polarimeter and Lidar (ACEPOL) campaign showed the algorithm to be capable of capturing this variation. The regression in Figure  9b  shows correlations and low bias of SSAs ( R 2 > 0. 40, a > 0. 60, b < 0. 030, and MAD ≤ 0. 004) from the two inversions as well. Figure  9c  shows basic consistency in the retrieved aerosol size distributions: both algorithms find the peaks of fine and coarse mode aerosol size to be around ~0. 15μm and ~2μm, respectively. Due to the lower sensitivity of AirMSPI's longest wavelength 865 nm to coarse mode aerosols, some differences in coarse mode aerosol size can be observed.
This indicates the impact of insufficient observational information about certain aerosol properties. Comparisons of pixel-scale AOD, SSA, and size distribution at other retrieval cases show a similar quality of agreement. (a)  against the previous retrievals using multi-pixel inversion (MPI,  [19] ). The AirMSPI dataset used for retrieval is the same as in Figure  5      against the previous retrievals using multi-pixel inversion (MPI,  [19] ). The AirMSPI dataset used for retrieval is the same as in Figure  5  over the AERONET Baskin site. The results for seven AirMSPI spectral bands are plotted in different colors: pink (355 nm), purple (380 nm), dark blue (445 nm), light blue (470 nm), green (555 nm), red (660 nm), and brown (865 nm). (b) The same as Figure  9a  but for SSA. (c) Comparison of image-mean volume-weighted aerosol size distribution retrieved from correlated multi-pixel inversion and multi-pixel inversion.
Figures 10 and 11 compare AirMSPI and AERONET retrievals of AOD and SSA respectively. Since the temporal variation of aerosol loading and properties is not constant, non-symmetric AERONET error bars can be observed in Figures  10  and 11. Under the circumstance of no AERONET reference data before or after AirMSPI measurement, only AERONET measurement/retrieval error is reported. Figures  12  and 13  compare real and imaginary parts of aerosol refractive index, respectively. The AERONET spectral aerosol products were linearly interpolated in wavelength to match the AirMSPI band centers. Comparisons of fine and coarse mode effective radii are shown in Figure  14. To facilitate the comparison of aerosol size, an effective radius was calculated for fine and coarse mode aerosols from AirMSPI retrievals using Equation  (35)  in a previous study  [19].
These properties include AOD, SSA, real and imaginary parts of refractive index (n r , n i ), and effective radii of fine and coarse particles (r eff, fine , r eff, coarse ). The bias is evaluated by taking the mean of the absolute difference between AirMSPI and AERONET retrievals at collocated pixels. The correlated multi-pixel inversion has slightly higher MAD than that of original multi-pixel retrieval, namely by ~0. 01, 0. 015, 0. 001, 0. 002, and 0. 05 for AOD and SSA in visible, n r , n i , r eff, fine , and r eff, coarse , respectively. The deviation of both correlated and original multi-pixel retrievals from AERONET reference data are mostly within the retrieval uncertainties from our algorithms (see Sections 3. 3 and 4. 2 of the previous study  [19] ) and those estimated for AERONET, as observed in Figures 10-14 of this work, as well as Figures  4 5 6 7 8 in the previous study  [19].
It is anticipated that correlated multi-pixel inversion will save 90% CPU time if an image of 100 × 100 pixels is retrieved simultaneously and all 52 parameters are correlated with each other. Table  6. Mean absolute bias (MAD) in key aerosol properties from correlated multi-pixel inversion approach using two PCs (this paper) and from original multi-pixel inversion approach adapted to AirMSPI  [19]. The bias is calculated by taking the mean of the absolute difference between AirMSPI and AERONET retrievals at collocated pixels. A total of 27 AirMSPI datasets are used. Interpolation is used to calculate some AERONET parameters at central wavelengths of the AirMSPI spectral bands. By performing regression against AERONET reference AOD, the coefficients of determination (R 2 ) are given in columns 3 and 5.
The algorithm makes use of multiple types of constraints, including across-pixel smoothness constraints transformed to be imposed on the PCs to retrieve zero sum of the PC weights, and orthogonality and unit norm conditions on PC weights and PC vectors, respectively. While applying smoothness constraints to the PCs instead of the individual aerosol parameters, the regularization (smoothness) remains faithful to both aerosol correlations and smooth spatial and spectral variation within the scene. To accelerate the multi-pixel retrieval, a PC-based RT model is developed, which capitalizes on the aerosol correlations and mutual orthogonality of the PC vectors. The retrieval methodology was tested by comparing aerosol retrievals from 27 AirMSPI datasets acquired between 2012 and 2016 with collocated aerosol reference data reported by AERONET.
To derive a priori of the correlations in surface properties, PC analysis can be performed over a surface reflectance dataset, such as the one based on the Multi-Angle Implementation of Atmospheric Correction (MAIAC) algorithm  [36, 37] , the ASTER spectral library  [38] , and the U. S. Geological Survey (USGS) digital spectral library  [39]. To impose the correlation between aerosol and surface, however, one might encounter a complex surface condition over a targeted area (e. g. , with co-existence of multiple types of surfaces such as soil, water, snow, forest, etc. ). Under such circumstances, surface pixel classification can be performed to identify surface types first, either from existing surface climatology or from various indices for vegetation, snow, soil, water, etc. based on their spectral difference.
Figure9. (a) Regression of pixel-scale AOD retrieved from correlated multi-pixel inversion (CMPI) against the previous retrievals using multi-pixel inversion (MPI, [19] ). The AirMSPI dataset used for retrieval is the same as in Figure5over the AERONET Baskin site. The results for seven AirMSPI spectral bands are plotted in different colors: pink (355 nm), purple (380 nm), dark blue (445 nm), light blue (470 nm), green (555 nm), red (660 nm), and brown (865 nm). (b) The same as Figure9abut for SSA. (c) Comparison of image-mean volume-weighted aerosol size distribution retrieved from correlated multi-pixel inversion and multi-pixel inversion.
Acknowledgments: We are grateful to  Carol J. Bruegge ,  Michael J. Garay ,  Gerard van Harten ,  Veljko M. Jovanovic ,  Olga V. Kalashnikova ,  Brian E. Rheingans ,  Felix C. Seidel ,  Irina N. Tkatcheva , and  Mika Tosca  of  JPL  for their support of AirMSPI data acquisition and calibration in multiple NASA field campaigns. We also thank  Xu Liu  at  NASA Langley Research Center and Vijay Natraj at JPL  for helpful discussions on principal component analysis to accelerate radiative transfer modeling and atmospheric remote sensing.