# Using Shannon's entropy to sample heterogeneous and high-dimensional atmospheric datasets

M. Paul, F. Aires

► To cite this version:

M. Paul, F. Aires. Using Shannon's entropy to sample heterogeneous and high-dimensional atmospheric datasets. Quarterly Journal of the Royal Meteorological Society, 2015, 141 (687), pp.469-476. 10.1002/qj.2367 . hal-02400762

HAL Id: hal-02400762

https://hal.science/hal-02400762v1

Submitted on 23 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# Using Shannon's entropy to sample heterogeneous and high-dimensional atmospheric datasets

M. Paul<sup>a*</sup> and F. Aires<sup>a,b</sup>

*Correspondence to: M. Paul, LERMA, Observatoire de Paris, 61 avenue de l'Observatoire, 75014 Paris, France.
E-mail: maxime.paul@obspm.fr

The construction of diverse and synthetic datasets of atmospheric situations, used as first guesses or training bases for remote-sensing algorithms, is still a challenge. Numerical constraints require the use of datasets with a limited number of representative situations, but keeping, as much as possible, the full diversity observed in nature. This study presents an innovative sampling method that allows extraction of a new, more limited, dataset from a large database of atmospheric situations. One major issue of such sampling concerns the heterogeneity of the input space variables: different units and ranges of temperatures and specific humidities, for instance, or locations from the lower troposphere to the higher stratosphere, can hardly be compared. We illustrate the fact that sampling using only one variable type is not optimal, since erroneous features can be observed in the other variables not used for the sampling. The use of Shannon's entropy can help to develop a sampling technique able to deal with very heterogeneous variables. A dataset of 10 000 situations is built from *EUMETSAT* satellite atmospheric retrievals: it includes temperature and water-vapour profiles, four integrated ozone layers and surface temperature. The sampling increases the entropy of the original dataset from 22 to 28 (about 20% increase).

Key Words: sampling; entropy; atmospheric dataset

Received 19 August 2013; Revised 6 March 2014; Accepted 12 March 2014; Published online in Wiley Online Library 8 May 2014

## 1. Introduction

Atmospheric datasets describing the variability of the climate system are necessary for multiple applications:

(1) validating climate models or remote-sensing algorithms (e.g. statistics are performed on a dataset in order to quantify some quality criterion);

(2) studying the state variability of the climate system;

(3) providing first guesses for inversion methods such as optimal interpolation (Rodgers, 1990) or neural networks (Aires et al., 2001); or

(4) calibrating retrieval schemes (see Aires (2011) for the training of neural networks or Kummerow et al. (2001) for Bayesian applications).

An important example of such an atmospheric dataset is the Thermodynamical Initial Guess Retrieval (TIGR) dataset (Chedin et al., 1985; Escobar, 1993; Chevallier et al., 1998). It was originally composed of atmospheric radio-sounding measurements of the atmosphere and was sampled iteratively, one situation at a time. This sampling is performed for temperature, water vapour and ozone. This dataset has evolved to improve some statistical parameters such as the surface temperature (Aires et al., 2002a) and the European Centre for Medium-Range Weather Forecasts (ECMWF)'s re-analyses were included (Chevallier et al., 2000) to improve its coherency with the ECMWF

assimilation system. Chevallier et al. (2000) stress that sampling multivariate variables is important (e.g. simultaneous sampling of temperature and water vapour). A solution is proposed, but its generalization to multiple variables is difficult (first stratification into low/medium/high contents, difficulty of having a distance for heterogenous variables, representative standard deviation for the normalization of the variables, choice of sampling threshold parameters in high-dimensional and heterogenous spaces). A similar dataset has been built by the ECMWF (Chevallier et al., 2006). To face (i) the increasing number of variables to sample (e.g. the addition of cloud or surface properties) and (ii) the consequent higher heterogeneity between the variables, the sampling at ECMWF is not performed independently for each variable. Five independent sampled datasets have been built for temperature, water vapour, ozone, liquid water and ice. These five datasets can be used more or less independently, depending on the application. For instance, the learning of a statistical retrieval scheme for temperature, water vapour and ozone would require the pooling together of the three corresponding datasets. It will be shown in the following that this is not an optimal strategy.

Contrarily to other fields, e.g. in pharmacology where samples are rare and expensive, the amount of information about the atmosphere is increasing exponentially, so the number of available samples is generally not an issue. We will refer, in the following, to a large sample of atmospheric situations (typically a few million situations) as the 'original (atmospheric) database'. However,

<footnote><sup>a</sup>LERMA/CNRS, Observatoire de Paris, France</footnote>

<footnote><sup>b</sup>Estellus, SAS, Paris, France</footnote>
most applications require a dataset with a limited and practical number of samples. For example, the training of statistical remote-sensing algorithms requires a training dataset that includes both the atmospheric properties and associated radiative transfer simulations. The simulations are computationally expensive for instruments such as the Infrared Atmospheric Sounding Interferometer (IASI) or the next generation of high-spectral resolution instruments. As a consequence, the challenge is to build a new dataset by sampling the original database with a limited number of representative samples.

The sampling method can have two objectives. Firstly, the statistics of the sampled dataset are required to reproduce the statistics of the original database: the mean, standard deviation and cross-correlation of the input variables are preserved in this process. Secondly, the sampling strategy can also aim for a uniform sampling of the input space. This approach emphasizes extreme events. It is also particularly interesting when the resulting sampled database is used to find first guesses for a retrieval scheme. In this way, the first-guess statistics are more uniform: this facilitates the specification of the first-guess statistics and better ensures inversion-algorithm convergence in all situations. A discussion on these sampling strategies can be found in Aires and Prigent (2007). Both approaches are legitimate, but we will focus here on uniform sampling, since the main application of the resulting sampled dataset will be the training of remote-sensing algorithms and extreme cases are recommended in the training of statistical models (Bishop, 1995).

There are two main difficulties as regards these sampling methods. Firstly, the atmospheric samples are in a high-dimensional space. If multiple atmospheric profiles are considered (e.g. temperature, water vapour, ozone, cloud properties, etc.), this input space typically has a dimension equal to a few hundred. Secondly, the input space is generally extremely heterogeneous, with atmospheric variables such as temperature and water vapour concentration that have very different units and variability ranges. Even for the same atmospheric quantity, e.g. water vapour, the variables can scarcely be compared in the lower and highest parts of the atmospheric profile. The objective of this article is to perform this sampling simultaneously on all the variables considered. We will refer to this constraint as the 'multivariate aspect' of the problem. Sampling simultaneously on all variables will avoid some spurious behaviours and allow the sampling of multivariate extremes (i.e. extremes of two or more variables at a time).

Entropy is a measure of randomness and is maximum for uniform distributions. The entropy has been used, for example for channel selection (Rodgers, 1990), to maximize the information of an observing system. One of its advantages lies in the fact that entropy can be estimated on very heterogeneous variables, since it is unitless. This article seeks to use the entropy to develop a new sampling algorithm able to deal with high-dimensional data and very heterogeneous variables.

The original atmospheric dataset to be sampled is described in section 2. Section 3 presents the methodology developed in this article, including an introduction to the concept of entropy and the sampling algorithm. The results of the sampling of an atmospheric dataset are shown in section 4. A conclusion and perspectives are drawn in section 5.

## 2. Original atmospheric database

The original atmospheric database to be sampled is a one year extraction from the IASI Level 2 products from EUMETSAT (Eumetsat Project, 2012). These atmospheric profiles are generated by the European Polar System (EPS) Core Ground Segment of the IASI measurements. Advanced Television and Infrared Observational Satellite (TIROS) Operational Vertical Sounder (ATOVS) instrument observations are also used in the retrieval scheme, together with meteorological forecasts. The retrieval is made in three steps. In the first one, the cloud characteristics

are retrieved. In the second step, a linear or neural-network regression is used to obtain a first guess. If the result is outside some fixed bounds, it is replaced by a climatology. The third and last step is an iterative modification (i.e. optimal interpolation) of this first guess to obtain a retrieval that fits the IASI observations.

The atmospheric profiles are represented on fixed pressure levels. For demonstration purposes, this study focuses only on some of the available atmospheric variables. The method is very flexible and can be used on any type of variable. In this article, the following variables have been studied:

* the atmospheric profile of temperature;

* the atmospheric profile of specific humidity;

* the surface temperature; and

* the ozone amount for four layers.

Ozone amounts are available on four integrated layers. They represent the amount of ozone integrated between the surface and the following pressure levels: 0.005, 132.49, 222.94 and 478.54 hPa. Depending on the surface pressure, the atmospheric profiles can be represented at a different number of levels. In order to address this problem, only the profiles for 86 pressure levels are kept in this study, from 0.005–940 hPa. Similar results have been obtained for the other pressure levels, but they are not presented here for conciseness of the presentation. Furthermore, the variability of water vapour in the higher layers is very low, so they will not be used in the following either.

The instrument is providing measurements in its footprint, continuously in time, at a very high temporal resolution. Cloudy situations are filtered out, using the EUMETSAT cloud flag product. The extraction is for year 2010 and the resulting original database includes nearly 4000 000 clear situations. The use of the sampled atmospheric dataset for the learning of retrieval algorithms typically requires a few thousand situations. In this study, the goal is to obtain a dataset of 10 000 samples from the 4000 000 situations in the original database.

## 3. Sampling methodology

### 3.1. Shannon's entropy

In order to sample the dataset, the concept of entropy will be used. It was introduced in information theory by C. Shannon (Shannon et al., 1949; Shannon, 1951). The entropy of a variable represents the quantity of information that is needed to determine it fully. A fixed variable has an entropy of zero and a random one has an entropy ≥ 0. It is possible to compute the entropy of continuous variables, but in practice the entropy is estimated on a discretized variable.

Let X be a random variable that can take values $(x_1, x_2, \dots, x_n)$ with a probability $(p_1, p_2, \dots, p_n)$. Shannon's entropy of X is estimated by

$$E(X) = \sum_{i=1}^{n} \left[ p_i \log \left( \frac{1}{p_i} \right) \right]. \qquad (1)$$

This discretized entropy is a lower-biased estimator of the true entropy, but it converges when the number of bins increases. It should be noted that the estimation of Eq. (1) is dependent on the number of bins. This entropy is maximum if $\forall i \in \{1, \dots, n\}, p_i = 1/n$. In this case, X takes the n different bin values with an equal probability, corresponding to a uniform probability density function (PDF).

The variables considered in this article (temperature, water vapour, ozone and surface temperature) are continuous (section 2). In order to calculate their entropy using Eq. (1), they are transformed into discrete variables just by considering regular bins between their minimum and maximum values.
Let us use a simple example to illustrate the concept of entropy. Four datasets of surface temperatures (TS in K) are considered:

$D_1 = \{275; 275; 275; 275; 275; 275; 275; 275; 275; 275\}$

$D_2 = \{275; 277; 276; 279; 278; 297; 295; 299; 296; 300\}$

$D_3 = \{279; 287; 299; 294; 300; 299; 282; 277; 282; 275\}$

$D_4 = \{282; 287; 296; 282; 276; 289; 297; 275; 294; 292\}$

The maximum TS value is 300 K and its minimum is 275 K. This TS interval is first divided into $N = 5$ bins: [275, 280[, [280, 285[, [285, 290[, [290, 295[ and [295, 300]. The probability of TS belonging to each bin is then computed. The results are shown in Table 1. The entropy $E(x)$ is calculated using Eq. (1). The first dataset $D_1$ concentrates on the first bin and its entropy is equal to zero. In the second dataset $D_2$, only extreme temperatures are present, so the entropy is low, equal to 0.35. For the third dataset $D_3$, the temperatures are well distributed among the different intervals so the entropy is quite high, 0.65. In the fourth dataset $D_4$, the temperatures are equally distributed between the five intervals and the resulting entropy is at its maximum value at 0.7. Thus, the entropy of a variable is a measure of its dispersion. Maximizing the entropy allows us to obtain variables uniformly distributed along their variability ranges.

For a multivariate variable

$$X = \begin{pmatrix} X_1 \\ X_2 \end{pmatrix},$$

the entropy $E(X)$ of $X$ requires the binning of the multidimensional input space. A major difficulty arises with high-dimensional spaces, since the number of bins increases exponentially with the dimension of $X$. For a $m$-dimensional space, the binning of each input variable with 20 subintervals results in $20^m$ bins. This is not manageable in practice for input spaces with dimension $m > 10$, which is the case for atmospheric dataset sampling. In order to solve this practical difficulty, we approximate the total entropy by the sum of the independent entropies. This is of course a crude simplification: $E(X)$ is lower than the sum of the two entropies $E(X_1)$ and $E(X_2)$, its difference being the mutual information between $X_1$ and $X_2$. In fact, $E(X_1, X_2) = E(X_1|X_2) + E(X_2)$ and $E(X_1|X_2) = E(X_1)$ only if $X_1$ or $X_2$ are independent. Therefore, calculating the entropy of a multivariate variable as the sum of the entropies of its variables is valid if variables are not too statistically dependent. It therefore leads to an overestimation of the total entropy in other cases.

In this article, using this simplification is not optimal, since the variables are linked to each other. However, maximizing the independent entropies will increase the global entropy. The following tests will show that this simplification is satisfactory and leads to good results. It will be seen in perspective that a practical solution using the principal component analysis tool should be considered in the future to reduce the dimension of the data and this would allow for the true entropy in the sampling algorithm.

## 3.2. Sampling algorithm

The entropy defined in the previous section can be used to develop a sampling algorithm. The objective is to sample the

<table><caption>Table 1. Probabilities and entropy for a synthetic example of four surface temperature datasets $D_1$, $D_2$, $D_3$ and $D_4$.</caption><thead><tr><th>Dataset</th><th>p<sub>1</sub></th><th>p<sub>2</sub></th><th>p<sub>3</sub></th><th>p<sub>4</sub></th><th>p<sub>5</sub></th><th>E(x)</th></tr></thead><tbody><tr><td>D<sub>1</sub></td><td>1.0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>D<sub>2</sub></td><td>0.5</td><td>0</td><td>0</td><td>0</td><td>0.5</td><td>0.35</td></tr><tr><td>D<sub>3</sub></td><td>0.3</td><td>0.2</td><td>0.1</td><td>0.1</td><td>0.3</td><td>0.65</td></tr><tr><td>D<sub>4</sub></td><td>0.2</td><td>0.2</td><td>0.2</td><td>0.2</td><td>0.2</td><td>0.70</td></tr></tbody></table>

original database described in section 2 (4000 000 situations) into a new dataset $\mathcal{B}$ of 10 000 samples. This new dataset should have a PDF as uniform as possible for the considered input variables: the profiles of temperature (T) and water vapour (WV), the four ozone (O) integrated quantities and the surface temperature (TS). The idea is therefore to maximize the entropy of the sampled dataset.

The entropy $E(\mathcal{B})$ of the sampled database $\mathcal{B}$ can be simplified by adding the entropies of each input variable. The variability range of each variable is divided into $N = 20$ regular bins. For each variable 'var', $p_n^{\text{var}}$ represents the probability of var belonging to bin $n$. In this application, the global entropy of the sampled dataset $\mathcal{B}$ is approximated using

$$E(\mathcal{B}) = w_T \times \sum_{l=1}^{L_T} \sum_{n=1}^{N} p_n^{Tl} \log \left( \frac{1}{p_n^{Tl}} \right) \\ + w_{WV} \times \sum_{l=1}^{L_{WV}} \sum_{n=1}^{N} p_n^{WVl} \log \left( \frac{1}{p_n^{WVl}} \right) \\ + \sum_{l=1}^{4} \sum_{n=1}^{N} p_n^{O_l} \log \left( \frac{1}{p_n^{O_l}} \right) \\ + \sum_{n=1}^{N} p_n^{TS_l} \times \log \left( \frac{1}{p_n^{TS_l}} \right). \quad (2)$$

$L_T = 43$ is the number of atmospheric levels for the temperature profile; only one level out of two is considered for computational speed considerations. $L_{WV} = 20$ is the number of levels for the water-vapour profile; the upper layers are not considered since the WV values have no or almost no variability in this part of the atmosphere. For each variable, $N = 20$ bins are considered. The binned-based entropy is an estimation of the entropy and, as for all entropy estimators, it is negatively biased (Paninski, 2003), so the binning choice is an important factor. The number of bins was chosen here so that each bin has at least a few samples in the original Eumetsat dataset.

A coefficient is used for each one of these four variables (T, WV, O and TS) in order to control the weight of each variable in the overall entropy. This weighting has been set up taking into account an approximation of the degrees of freedom for each variable. There is only one degree of freedom for TS and the four O values, so their coefficients are set to unity. Fifteen degrees of freedom are considered for the temperature profile T ($w_T = 15/43$) and five for WV ($w_{WV} = 5/20$). The degrees of freedom have been estimated using principal component analysis (PCA) on each variable independently. It will be mentioned in the concluding section that PCA could be used simultaneously on all the variables as a pre-processing step for the sampling algorithm. In this case, the weights would be unnecessary.

The sampling algorithm starts by randomly choosing 10 000 situations in the original atmospheric database. Experiments have been conducted using various initial conditions (not shown); the optimization does not seem to be sensitive to this parameter. This represents the sampled dataset $\mathcal{B}$ at iteration zero. Then, at each iteration of the algorithm, a batch of 100 000 new random situations of the original database are considered. For each one of these new situations $x_i$ (for $i = 1, \dots, 100 000$), $E(\mathcal{B})$ is computed $j = 1, \dots, 100 000$ times using Eq. (2), where sample $j$ in $\mathcal{B}$ is replaced by $x_i$. If the entropy is increased when this exchange is made, $\mathcal{B}$ is updated. In order to reduce the computational burden, the entropy $E(\mathcal{B})$ is not reevaluated each time but rather only the change in entropy is estimated. If this change is positive, $\mathcal{B}$ is updated. This iterative process is the input-space search algorithm. The intent is to visit the input space, searching for the global optimal solution. Note that more sophisticated optimization methods can be used instead of this brute-force algorithm. For instance, the simulated-annealing optimization has a probabilistic search approach in the input space; this might
<figure><img src="image_3.png" /><figcaption>**Figure 1.** Convergence of the sampling algorithm: (a) total entropy of the sampled dataset *B* and (b) number of situations changed.</figcaption></figure>

be better suited to identifying the global optimum. However, our simpler approach has been initiated with multiple different initial conditions and the results do not seem to be dependent on this factor, so the solution should be close to the global optimum.

Figure 1 illustrates the convergence of the algorithm. Each iteration in the horizontal axis represents a batch of 100 000 situations. This means that, at the end of the convergence, 300 iterations represent the processing of 30 000 000 situations. As the global database contains about 4000 000 situations, this means that each of its situations has been examined multiple times. The total entropy of the sampled dataset is plotted in the upper part of the figure. This curve shows a fast increase of entropy at the beginning, until the algorithm reaches a stable higher entropy value *E*(*B*) of about 28. The number of situations that have been changed in the sampled dataset *B* for each batch is plotted in the lower part of the figure. For the first batches, nearly all situations are changed (10 000). The number of exchanges then decreases very fast, which means that the algorithm rapidly finds a solution *B* that maximizes the entropy *E*(*B*). The convergence criterion of the algorithm is defined using this number of exchanges for each batch. The algorithm is stopped when this number is lower than 30 for two consecutive batches.

Note that the original random sampling (i.e. the initial 10 000 samples of the optimization algorithm) is already a valid sampling strategy. This means that the initial entropy in Figure 1 is already good. As a consequence, we do not need to improve the entropy much, from 22 to 28 (about 30% increase). However, the effect of this change of entropy in the resulting distributions is significant, e.g. the extreme cases cover a larger range in the optimized sampled database. Furthermore, we cannot optimize the entropies fully because some situations of the input space are not physically possible. The sampling approach cannot build samples that are not in the original input dataset. Whatever the optimization scheme, some parts of the input space will not be represented and the final PDF will not be perfectly uniform, as expected. This of course limits the room for improvement of the final entropy.

### 3.3. *Impact of the multivariate property*

As seen in Eq. (2), the use of the entropy allows us to combine, in a unique criterion, very different variables, where other sampling techniques need to use complex distances in the input space based on some normalization by the variance of the variables (Aires and Prigent, 2007). In this section, it will be shown that performing the sampling independently, for each variable, is not optimal (see Chevallier *et al.*, 2000) and that the ability of entropy to deal with multivariate and heterogeneous variables is an important feature.

A simple way to demonstrate this multivariate property is to use an example. Three different samplings of the original atmospheric database of section 2 are performed and compared with equivalent ECMWF samplings. Each time, 10 000 situations are selected. For the first sampling, only the temperature profile is taken into account in the calculation of the entropy. For the second sampling, the entropy is calculated only on the water-vapour profile. For the third sampling, both profiles are used.

To represent the quality of the three samplings, the minimum and maximum of each profile (*T* and *WV*) are plotted (Figure 2). Between those curves, the colour of the *N* = 20 bins represents the probability of the variable being in the bin (the sum of each probability for a vertical layer is equal to one). White intervals correspond to bins that are not populated in the sampled dataset, red is for bins that are highly populated and blue for less populated bins. The objective is to obtain uniform sampling, so the final figures should be as uniform and blue as possible and with most bins populated.

Figure 2 illustrates five different samplings. In the first row, the result of the sampling on the temperature profile is plotted. The resulting situations present a rather uniform variation of the temperature profiles, meaning that the sampling on the temperature profile has succeeded. However, the water vapour (in relative humidity to facilitate readability) presents a high quantity of profiles with a low level of water vapour. Some high values of the relative humidity, between 450 and 150 hPa, are never reached. This sampling is therefore not satisfactory for water vapour.

The second row of the figure presents the results of the sampling on the water-vapour profiles. The representation of the water-vapour profile is, this time, satisfactory. There is no over-representation of low values of relative humidity and all the bins are populated. However, it can be noted that the temperature profiles are highly concentrated round a mean temperature profile. As a consequence, extreme values of temperature are never reached. Once again this sampling is not satisfactory; the sampling in water vapour is good but the temperature results are not acceptable.

This mono-variable sampling strategy has also been used in the current ECMWF databases (using a different algorithm). The third and fourth rows represent the same quantities and can be compared directly with the first and second rows. The results are equivalent, in particular concerning the peculiar PDF of the un-sampled variables that is the direct consequence of the mono-variable sampling. This shows that a multivariate sampling should improve the quality of the multivariate distributions.

The lower part of the figure presents the dataset sampled by maximizing the entropy of *B* simultaneously for the two profiles. There are no more values unpopulated in the dataset for the temperature or the water-vapour profiles. A little hint of the 'mean' temperature profile linked to the high water vapour values remains, but it is much less pronounced and the sampling appears, overall, to be quite uniform. Figure 2 illustrates how important it is to perform the sampling simultaneously for each variable of the database. If the sampling is performed on one variable only, some unacceptable features can appear in the other variables, such as the over-representation of a mean profile in this example. The entropy-based sampling algorithm developed in this section is, by design, able to deal simultaneously with heterogeneous input variables.

Figure 3 is used to monitor two aspects: (1) the independence of temperature and water vapour at 727.43 hPa and (2) the quality of the joint uniform sampling of both variables. The left column represents the product of probabilities *P*(*T*) × *P*(*WV*) and the right column is the joint probability *P*(*T*, *WV*). The upper part of the graph is for the original dataset representing about 4 million samples, the lower part is for the sampled database of 10 000 samples. When the two variables are statistically independent of each other (i.e. a stronger constraint than decorrelation), the product of probabilities and the joint probability are identical:
<figure><img src="image_4.png" /><figcaption>**Figure 2.** The different plots represent the percentage of profiles belonging to each of the 20 intervals between the minimum and the maximum value at each pressure level. Distributions are given for (a,c,e,g,i) temperature and (b,d,f,h,j) water vapour. From top to bottom rows, the distributions are for (i) temperature-only sampling, (ii) water-vapour-only sampling, (iii) temperature-only sampling from the ECMWF database, (iv) water-vapour-only sampling from ECMWF and (v) the joint sampling of temperature and water vapour.</figcaption></figure>

$P(T) \times P(WV) = P(T, WV)$ (Aires *et al.*, 2002b). The fact that the top left and right graphs are rather similar means that temperature and water vapour at this level are reasonably independent in the original dataset, but the strong predominance of some bins can hide correlations for extreme cases.

The comparison of the upper and lower panels shows that the sampling has resulted in more uniform distributions for both the individual probabilities (on the left) and the joint probabilities (on the right). Furthermore, the lower graph on the right side shows that the joint distribution samples the extreme events better

than the original dataset. In particular, the bivariate extremes are better represented (e.g. warm and humid extreme). The joint distribution is not smooth enough, because it is difficult to sample 20 × 20 = 400 bins with only 10 000 samples; each bin represents only, on average, a joint probability of 0.25%. Using more samples should smooth out this joint distribution. This diagnostic could be used to determine a more optimal number of samples. Note also that not all parts of this two-dimensional space are sampled (e.g. warm and dry situations), because no such samples could be found for this region in the original dataset.
<figure><img src="image_5.png" /><figcaption>**Figure 3.** (a, c) Probability product *P*(*T*) × *P*(*WV*). (b, d) Joint probability *P*(*T*, *WV*), at 700 hPa. The upper part is for the original dataset (4 million samples), the lower part is for the sampled database (10 000 samples).</figcaption></figure>

This limits the potential increase of the total entropy by the
sampling algorithm. Finally, this figure shows how difficult it is to
sample a two-dimensional space and how even more challenging
it is to sample the space of the atmospheric profiles (dimension
43+20+4+1 = 68).

4. Results

In this section, the sampling algorithm previously presented
is applied to the original atmospheric database of section 2
(4000 000 situations) in order to extract 10 000 samples, using
all the variables simultaneously. The results of this sampling are
presented using plots similar to Figure 2: the minimum and
maximum of the variables are plotted for each pressure level
and the colours of the *N* = 20 bins represent the population
probability of each bin. As the surface temperature is a single
variable, a different plot has been made for this variable. The
actual value of each probability is represented for each bin as a
curve, instead of using coloured squares. Such plots are built for
the original database (left) and for the sampled dataset (right).
The results for the profiles of temperature, water vapour, ozone
and surface temperature are presented in Figure 4. The left part
of the figure shows an overpopulation of a ‘mean’ temperature
profile. Situations with low relative humidity, especially at higher
levels, are over-represented. The peak is also higher for the four
layers of ozone. For the second layer (132.49 hPa to surface),

a particular bin contains nearly half of the situations. The surface
temperature PDF also presents a peak near 298 K. It can be seen
that the probability of the ‘mean’ temperature profile is lower on
the right side, for the sampled dataset, and that the blue colours are
lighter overall, meaning that extreme cases are more populated.
Water vapour is more uniformly distributed, except for higher
layers that are still difficult to deal with. The improvement of the
ozone PDF uniformity is even more pronounced. The variations
of the ozone integrated values for the four layers are quite low in
the original database (left), whereas in the sampled one no interval
contains more than 16% of the situations. The peak of the surface
temperature PDF has also disappeared. These plots show that the
sampled distributions have a more uniform distribution in the
sampled dataset B.

Of course, the results are not perfect when considering only
one of the variables, because the sampling algorithm needs to
find a good compromise to obtain a uniform PDF for all input
variables. As stated earlier, some parts of the input space (i.e.
atmospheric profiles) are not physically possible, which means
that no samples can be found in these bins.

The use of the entropy allowed us to sample the original
global atmospheric database into a more uniform dataset for all
the variables considered using only 10 000 situations. Figure 5
represents as grey dots all the situations of the original database.
The higher elevation locations are not covered by the dataset, as
only those situations with a surface pressure higher than 940 hPa
<figure><img src="image_6.png" /><figcaption>**Figure 4.** First row (a, b): percentage of situations belonging to the 20 bins between the minimum and maximum value of the temperature profile. Second and third rows (c–f): similar but for water vapour and the four ozone layers. Bottom row (g, h): probability density function of the surface temperature. The left part of the figure is calculated on the global original database (section 2) and the right part is calculated on the sampled dataset *B*.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 5.** Spatial distribution of the global original database (grey) and the sampled dataset *B* (black). This graph represents the configuration with surface pressure higher than 940 hPa.</figcaption></figure>

are kept (see section 2). All other regions are covered by this database. The black dots correspond to the sampled dataset. Even if only 0.25% of situations are kept in the sampled dataset (10 000 out of 4000 000 situations), the spatial coverage remains similar. Rather homogeneous domains with similar profiles are less represented (the Sahara for example); this is to be expected because of the uniform sampling requirement on temperature and water vapour. In contrast, more variable and complex locations are more represented. In particular, the Northern Hemisphere, which is covered by snow in winter and grass and trees in summer, is more represented in the sampled dataset *B*. This global coverage is satisfactory, in particular since no spatial information was provided to the sampling process. As a conclusion, the sampled dataset is representative of all locations and has a more uniform distribution of the temperature, water vapour, ozone and surface temperature than the original dataset.
## 5. Conclusion and perspectives

An original uniform-sampling algorithm based on entropy has been developed. Its efficiency in sampling high-dimensional spaces of atmospheric variables has been demonstrated. This technique is able to represent the variability of large databases using only a limited number of samples that represent, as much as possible, all situations, in particular the extreme ones. The entropy allowed us to extract 10 000 situations out of 4000 000 with a quite uniform distribution of the 68 variables considered (43 levels of temperature, 20 levels of water vapour, four layers of ozone and the surface temperature).

Sampling using individual variables (e.g. only temperature or only water vapour) has been shown to be non-optimal, as it can yield non-representative distributions of the variables not used in the sampling; see also Chevallier et al. (2000). The entropy-sampling approach is able to deal simultaneously with very heterogeneous variables (temperature, water vapour or ozone concentrations at various vertical levels). Such a feature is a key aspect of the sampling technique, since it provides immediately a dataset optimized for all the desired variables.

The resulting sampled atmospheric dataset will be made available to the community. It could be useful, in particular, to operational centres such as ECMWF or EUMETSAT to provide independent climate first guesses for iterative inversion algorithms, conduct information-content analysis on a limited but representative dataset or train inversion methods such as statistical remote-sensing models.

This new sampled atmospheric dataset will be tested at EUMETSAT and the impact of its use as a first guess will be analyzed. The dataset will also be used as a learning dataset for statistical retrieval of temperature and water-vapour profiles. It has been shown that the synergy of infrared and microwave instruments is very important for the simultaneous retrieval of atmospheric temperature and water-vapour profiles (Aires, 2011; Aires et al., 2011, 2012). The fact that entropy-based sampling is able to build atmospheric datasets for both variables in a coherent way should facilitate the use of temperature/water vapour dependence in the retrieval scheme.

An important perspective will also be the use of a multivariate entropy estimation on very high-dimensional spaces. For example, an atmospheric dataset such as the ones developed in this article should include hydrometeor properties or surface characteristics. This is a difficult task, since the so-called ‘curse of dimensionality’ requires a number of bins and samples that increases with the dimension of the input space, but some sparse space sampling ideas could be tested in future. For instance, the PCA tool could be used to compress a high-dimensional data space (atmospheric profiles) into a lower-dimensional space where coordinates are uncorrelated. This would have two advantages that can be used in two different ways. Firstly, the obtained principal components are uncorrelated, so the approximation of the total entropy with the sum of entropies (section 3.2) would be more realistic. Secondly, the binning of data might be possible in this lower-dimensional space. This might allow us to use the full entropy (not the sum of entropies) in the sampling algorithm. With this PCA pre-processing, both approaches would be able to account for the links between variables in the sampling algorithm.

This innovative entropy-based sampling technique is very general and can address numerous sampling problems. It is a flexible technique that can be easily adapted to other problems, since the entropy is independent of the unit of the variables considered. Such methods are crucial today in a large variety of domains: the exponential use of computer technology increases the volume of available data and new methodologies are required to deal with this amount of data. This new field of research is referred to as the ‘big data’ (Silver, 2012) and sampling techniques

could be one tool to deal with such problems, particularly for the atmospheric sciences.

## Acknowledgements

This work has been financed by the Collège des Ingénieurs and the Université Pierre et Marie Curie in Paris, through a PhD allowance to Maxime Paul. The authors thank EUMETSAT for inviting Maxime Paul to Darmstadt in Germany to discuss this subject and to implement our database and algorithm into their system. We also thank two anonymous reviewers and the QJRMS chief editor, Professor Douglas Parker, for helping to improve the manuscript.

## References

* Aires F. 2011. Measure and exploitation of multi-sensor and multi-wavelength synergy for remote sensing: Part I – Theoretical considerations. *J. Geophys. Res.* **116**: D02301, doi: 10.1029/2010JD014701.

* Aires F, Prigent C. 2007. Sampling techniques in high-dimensional spaces for satellite remote sensing databases generation. *J. Geophys. Res.* **112**: D20301, doi: 10.1029/2007JD008391.

* Aires F, Prigent C, Rossow W, Rothstein M. 2001. A new neural network approach including first-guess for retrieval of atmospheric water vapour, cloud liquid water path, surface temperature and emissivities over land from satellite microwave observations. *J. Geophys. Res.* **106**: 14887–14907, doi: 10.1029/2001JD000085.

* Aires F, Rossow W, Scott N, Chedin A. 2002a. Remote sensing from the Infrared Atmospheric Sounding Interferometer instrument 1. Compression, denoising, and first-guess retrieval algorithms. *J. Geophys. Res. (Atmos.)* **107**: D224,4619, doi: 10.1029/2001JD000955.

* Aires F, Rossow WB, Chedin A. 2002b. Rotation of EOFs by the independent component analysis: Towards a solution of the mixing problem in the decomposition of geophysical time series. *J. Atmos. Sci.* **59**: 111–123.

* Aires F, Paul M, Prigent C, Rommen B, Bouvet M. 2011. Measure and exploitation of multi-sensor and multi-wavelength synergy for remote sensing: Part II – An application for the retrieval of atmospheric temperature and water vapour from MetOp. *J. Geophys. Res.* **116**: D02302, doi: 10.1026/2010JD014702.

* Aires F, Aznay O, Prigent C, Paul M, Bernardo F. 2012. Synergetic multi-wavelength remote sensing versus a posteriori combination of retrieved products: Application for the retrieval of atmospheric profiles using MetOp. *J. Geophys. Res.* **117**: D18304, doi: 10.1029/2011JD017188.

* Bishop M. 1995. *Neural Networks for Pattern Recognition*. Oxford University Press, Inc.: New York, NY.

* Chedin A, Scott N, Wahiche C, Moulinier P. 1985. The improved initialization inversion method: A high resolution physical method for temperature retrievals from satellites of the TIROS-N series. *J. Appl. Meteorol. Climatol.* **24**: 128–143.

* Chevallier F, Cheruy F, Scott N, Chedin A. 1998. A neural network approach for a fast and accurate computation of a longwave radiative budget. *J. Appl. Meteorol.* **37**: 1385–1397.

* Chevallier F, Morcrette J-J, Chedin A, Cheruy F. 2000. TIGR-like atmospheric-profile databases for accurate radiative-flux computation. *Q. J. R. Meteorol. Soc.* **126**: 777–785.

* Chevallier F, Di Michele S, McNally A. 2006. *Diverse Profile Datasets from the ECMWF 91-level Short-Range Forecasts*. European Centre for Medium-Range Weather Forecasts: Reading, UK.

* Escobar J. 1993. 'Base de données pour la restitution de paramètres atmosphériques à l'échelle globales; étude sur l'inversion par réseaux de neurones des données des sondeurs verticaux atmosphériques satellitaires présents et à venir', PhD thesis. Université de Paris Diderot: Paris VII.

* Eumetsat Project. 2012. 'IASI level 2 product guide', Technical Report EUM/OPS-EPS/MAN/04/0033. Eumetsat: Darmstadt, Germany. http://oiswww.eumetsat.org/WEBOPS/eps-pg/IASI-L2/IASIL2-PG-0TOC.htm (accessed 19 July 2012).

* Kummerow C, Hong Y, Olson W, Yang S, Adler R, McCollum J, Ferraro R, Petty G, Shin D, Wilheit T. 2001. The evolution of the Goddard Profiling Algorithm (GPROF) for rainfall estimation from passive microwave sensors. *J. Appl. Meteorol.* **40**: 1801–1820.

* Paninski M. 2003. Estimation of entropy and mutual information. *Neural Comput.* **15**: 1191–1254.

* Rodgers CD. 1990. Characterization and error analysis of profiles retrieved from remote sounding measurements. *J. Geophys. Res.* **95**: 5587–5595, doi: 10.1029/1990JD095iD05p05587.

* Shannon C. 1951. Prediction and entropy of printed English. *Bell Syst. Tech. J.* **30**: 50–64.

* Shannon C, Weaver W, Blahut R, Hajek B. 1949. *The Mathematical Theory of Communication*, **117**: University of Illinois Press: Urbana, IL.

* Silver N. 2012. *The Signal and the Noise: Why so Many Predictions Fail – but Some Don't*. Penguin Press: New York, NY.