dataesr/scientific-paragraphs-categorization huggingface dataset:
(16) One key aspect of our POD implementation is to center and standardize the reduced coefficients obtained in Eq. 14. The standardization step is usually referred to as whitening (e. g. Kessy et al. , 2018)  and guarantees the following properties for k:      E[k i ] = 0 ∀ i = 1,. , L , E[k i k j ] = 1 if i = j, 0 otherwise. ∀ i, j = 1,. , L. (17) The transformation in Eq. 14 may now be applied to the original dataset (expressed in the grid space) to create a new dataset of POD reduced coefficients k for varying input parameters µ. The compressed information is suitable for modeling (through a metamodel) the relationship between the mean tracer concentration field and the uncertain input parameters. This is detailed in next section.
Therefore, one key aspect of the metamodeling stage deals with the choice of the kernel (Sect. 3. 3. 2) and the hyperparameter optimization (Sect. 3. 3. 3) to adapt the Gaussian processes to the variety of length-scales across the POD modes. Because the POD reduced coefficients are decorrelated (Eq. 17), we design L independent GPR models, i. e. we learn the relation between each reduced coefficient k l and the input parameters µ for l varying from 1 to L (Fig. 5 ). Noisy training data We consider the LES data to be somewhat noisy due to time averaging. To account for this noise, the non-interpolating GPR model for the lth reduced coefficient k l is written as:.
Reduced coefficient prediction The GPR framework assumes that the mapping f l is a Gaussian stochastic process such that: f l (µ) ∼ GP (m l (µ), r l (µ, µ * )) ∀ (µ, µ * ) ∈ P × P, ( 19 ) with m l (µ) = E[f l (µ)] the mean function of the Gaussian process and r l (µ, µ * ) = E[(f l (µ) -m l (µ)) (f l (µ * ) - m l (µ * ))] its associated covariance function. GPR starts with a prior distribution over the mean and covariance function. In this work, the prior mean is assumed to be 0 because of POD data transformation (Sect. 3. 2), and the prior covariance is specified by choosing a kernel (Sect. 3. 3. 2). The learning stage consists in updating the mean and covariance by integrating the information from the reduced-coefficient dataset (Sect. 3. 2. 3). We note D l = {(µ (n) , k (n) l ), n = 1,.
, N } the dataset dedicated to the lth reduced coefficient. It is worth noting that we use the raw samples of the Halton's sequence over the interval [0, 1] 4 as the input data to GPR in order to ensure that the samples are adequately distributed over the uncertain space. This is motivated by the fact that the raw Halton's sequence has suitable statistical properties (same parameter range [0, 1], approximately identical distribution), while the input parameters are of very different nature. In particular, the surface roughness z 0 is log-uniformly distributed and orders of magnitude smaller in comparison to the other input parameters. From now on, we make the distinction between the training sample and the test sample: the corresponding N train and N test -collection matrices are denoted by (U, K l ) and (U * , K * l ).
Using these notations, the joint distribution between the training dataset and some new test evaluations is expressed with respect to the kernel as: K l K * l ∼ N 0, r l (U, U) + s 2 l I r l (U, U * ) r l (U * , U) r l (U * , U * ) , (20) where s l is the noise variance (Eq. 18) and I stands for the identity matrix. We can derive the inference formula for the test reduced coefficients from the following conditional distribution: K * l | U, K l , U * ∼ N (m * l , cov(K * l )) , (21) where m * l = r l (U * , U) r l (U, U) + s 2 l I -1 K l cov(K * l ) = r l (U * , U * ) -r l (U * , U) r l (U, U) + s 2 l I -1 r l (U, U * ). ( 22 ) All terms in Eq. 22 are known. The covariance formulation depends on prior variance r l (U * , U * ) over the test dataset refined by information from the training dataset.
Smoothness hyperparameter The stochastic Gaussian process resulting from a Matérn kernel is ν -1 times differentiable in the mean-square sense. The smoothness parameter ν will take the form ν = p + 1/2, p ∈ N, since it is a common choice in machine learning framework (e. g. Elbeltagi et al. , 2021; Mukesh Kumar and Kavitha, 2021). Strong smoothness is irrelevant when dealing with experimental data. It might be hard to distinguish high values of smoothness ν ≥ 7/2 (existence of high-order derivatives) from noisy data  [Williams and Rasmussen, 2006]. In this work, since the training dataset is assumed to be noisy, we consider ν = 5/2.
Training, calibration and test In this study, LES data (made of 750 snapshots -Sect. 2. 5. 2) are split into three subsets following Halton's sequence ordering: i) a training dataset to learn the POD modes and the POD reduced coefficients using GPR (63%, i. e. N train = 472); ii) a calibration dataset to estimate GPR hyperparameter prior distributions (7%, i. e. N calib = 53); and iii) a test dataset to evaluate the capacity to predict LES quantities of interest for new samples of the uncertain input parameters (30%, i. e. N test = 225). Having a calibration dataset that is independent to the proper training dataset is essential to provide unbiased estimation of the Gaussian process prior distributions.
Performance metrics Individual Gaussian process regression model In this work, we quantify the individual GPR model performance using a Q 2 (explained variance) criterion for each POD reduced coefficient. With this criterion, the lth GPR prediction error is weighted by the variance over the coefficients k l of the lth POD mode: Q 2 l = 1 - k l -m * l 2 2 k l -Ê[k l ] 2 2 , ∀l = 1,. , L, (26) where m * l is the lth GPR mean prediction (Eq. 22). In this study, Q 2 l is estimated over the test dataset. Since we use whitening, reduced coefficients have zero-mean and unit-variance, meaning that this per-mode Q 2 metric directly reflects the mean squared error (MSE) on the predicted reduced coefficients (i. e. Q 2 ≈ 1 -MSE). Reduced-order model To quantify the performance of the reduced-order model in the physical space, we also evaluate the Q 2 criterion on each feature.
The reconstruction/prediction error is weighted by the variance over the LES samples for each grid element i: Q 2 i = 1 - K les,i -K rb,i 2 2 K les,i -Ê[K les,i ] 2 2 , ∀i = 1,. , N h. (27) In this study, the Q 2 i criterion is estimated over the LES training dataset for verification as well as over the LES test dataset for evaluating the reduced-order model prediction capacity. To help with the analysis, we also derive a global score from the variance weighted local Q 2 criterion as: Q 2 = N h i=1 ω i Q 2 i , ω i = V(K les,i ) N h j=1 V(K les,j ) , ( 28 ) where V(K les,i ) corresponds to the variance unbiased estimation over the snapshots defined in Eq. 13. This weighted average matches the usual explained variance criterion of the POD, in agreement with the problem of 2 -matrix norm maximization: Q 2 = L l=1 σ l N h l=1 σ l.
Results In the following, we analyze POD modes and quality of representation (Sect. 4. 1), give details about how to inform GPR models using POD (Sect. 4. 2), and analyze the POD/GPR reduced-order model accuracy on mean tracer concentration fields using the full dataset (Sect. 4. 3). We then carry a numerical analysis of the reduced-order model robustness with smaller training dataset (Sect. 4. 4). We end this section with an extension of the reduced-order model to the turbulent scalar flux (Sect. 4. 5).
Analysis of proper orthogonal decomposition This section investigates the quality of representation of the ensemble variance as well as the mode spatial structures for time-averaged quantities of interest. We explain the choices that lead to the selection of the number of POD modes L. The number of POD modes directly determines the number of GPRs to perform (Eq. 8). Cumulative explained variance Several criteria are available in the literature for selecting the number of modes, the vast majority relying on the explained variance metrics evaluated on training data (e. g. the cumulative explained variance, the Kaiser and elbow rules -  Jolliffe, 2002). Figure  6  shows the cumulative explained variance of the training dataset (Sect. 3. 4. 1) decomposed on the POD reduced basis as a function of the L POD modes.
The different streaks are directly related to the different source positions of the LES database since upstream emission source locations induce very refined, horizontally-elongated plume wakes. Finally, the example of the fiftieth mode (Fig. 7e ) highlights that the very high POD modes focus on the remaining ensemble variance heterogeneity. This mode features very localized bubble structures, which match tracer concentration peaks due to emission source locations present in the POD training database. The small structure values also raise the question of the potential noise induced in the POD modes due to the lack of training data volume. Field prediction example Figure  8  illustrates through the example of the nominal snapshot (Fig. 3a ) that is part of the LES test database, the capacity of reconstructing the mean tracer concentration fields from POD for different truncation levels L.
Gaussian process hyperparameter prior distribution GPR requires optimizing hyperparameters θ l = {s 2 l , , λ uz c , λ z0 , λ xsrc , λ zsrc } for the lth reduced-order model, which is carried out by gradient descent on the marginal log-likelihood (MLL) or on the posterior distribution (MAP). In the case of MAP, prior distributions are no longer assumed to be uniform  (Sect. 3. 3. 3). This section presents how to calibrate Gamma and Gaussian distributions for the hyperparameters from POD modes and reduced coefficient features that are required as input to the MAP optimization procedure. Noise prior POD often assumes that the variability of the low-order reduced coefficients is related to systematic behavior among the LES dataset, whereas the variability carried by the high-order reduced coefficients measure noise  [Jolliffe, 2002].
This is equivalent to treating the reduced coefficients on the first modes as unbiased data, which is a very restrictive assumption  (Sect. 3. 3). In this study, GPR accounts for the noise on the lth reduced coefficient through the hyperparameter s 2 l (Eq. 18). In this work, we assume that a prior estimate of s 2 l may be obtained from the noise introduced by the time-averaging process performed on the LES (Sect. 2. 5. 2). Note that this is not the only source of noise introduced during the generation of the LES database, but this provides a lower bound noise estimation that is useful to demonstrate the added value of our methodology. The main parameter involved in the time-averaging process is the length of the time-averaging window used to acquire LES statistics. To evaluate the noise for the lth reduced-order model, we compare the reduced coefficient of converged simulations (k l ) with the non-converged reduced coefficient obtained by averaging over only 50% of the full simulation time window (k l,50% ) as: ŝl 2 ≈ 1 2N N n=1 k l (µ (n) ) -k l,50% (µ (n) ) 2.
For the hyperparameter optimization step, the starting point of the gradient descent for s 2 l is taken as the Gamma distribution mode value that is obtained from a fit on the estimated noise (see thick line in Fig. 9 ). Mean and scaling priors Now we establish prior information on the mean and variance of the Gaussian processes (Eq. 19). In our GPR formalism, the mean is assumed to be constant, and the variance is decomposed into a systematic component and a noise component. From the expression of the noisy GPR models, we obtain for each POD reduced coefficient l: E[k l ] = m l , Var(k l ) = + s 2 l , (31) where the signal variance hyperparameter (Eq. 23) characterizes systematic variability, while s 2 l characterizes noise variability. is linked to the successive transformations associated with POD centered and normalized reduced coefficients on the training dataset.
For this reason, we assume that the total variability is essentially induced by systematic behavior of LES data and carried by the hyperparameter. We set a Gaussian prior distribution for with mean and variance estimated from the calibration data ensemble statistics and set to 1 and 0. 03, respectively. During the optimization step, the starting point of the gradient descent for is set to the Gaussian distribution mean. Correlation length-scale prior We are now interested in the prior distributions of the Matérn kernel correlation length-scales {λ xsrc , λ zsrc , λ Uz c , λ z0 } (Sect. 3. 3. 2). The analysis of the POD modes (Fig. 7 ) reveals an increase in small-scale spatial heterogeneity for increasingly higher-order modes, which results from the ensemble variability driven by the emission source location.
This is consistent with the small structures observed in the high-order modes and associated with tracer concentration wakes (Sect. 4. 1). A large number of POD (L = 100) is necessary here to characterize the high sensitivity of the tracer concentration upstream of the obstacle to the source height and position. Gaussian process regression accuracy We now evaluate the accuracy of the GPR models over the test dataset (Sect. 3. 4. 1) for validation purpose. Figure  12  shows the Q 2 scores obtained for each POD mode for the MLL and MAP procedures. For the very first modes, the Q 2 score is close to one, which means that GPR models perform almost perfectly on large variance structures with limited overfitting. However, they cannot maintain this level of performance when moving to high-order modes that feature more complex and localized structures.
The decrease in performance appears linear, and the higher noisier modes are more difficult to predict (with a Q 2 coefficient evolving between 0. 4 and 0. 6 from the fiftieth mode onwards). Still, this adaptive procedure remains partly successful in predicting fine structures of the LES ensemble variance. Optimizing the hyperparameters mode per mode greatly improves the GPR accuracy compared to simply imposing prior trend on hyperparameters. This gain in Q 2 is more pronounced when moving to high-order modes. Results also confirm that the reduced-order models obtained from MLL and MAP procedures are equivalent as anticipated from the equivalent correlation length-scales in Fig. 11. Globally, the Q 2 score of the reduced-order model evaluated on the test dataset is equal to 96. 8%. This model slightly overfits data since the Q 2 metric is equal to 99.
3% on the training dataset (recall that the reduced-order model cannot achieve a global Q 2 metric beyond 99. 6%, which is the upper limit imposed by the POD reconstruction error for L = 100 -Fig. 6 ). Note that the s 2 l -hyperparameter slightly reduces accuracy on the training dataset but this is not of the same magnitude as the decrease in performance observed on the test dataset. This hyperparameter is therefore not responsible for the observed regression error. Figure  12 : Per mode-Q 2 (Eq. 26) for GPR models with noise and length-scales that are either imposed using prior information (dashed line) or optimized by MLL (solid line) or MAP (dotted line). Field prediction examples Figures 13-15 show the mean normalized tracer concentration fields predicted by the reduced-order model (resulting from MAP optimization) for three snapshots of the LES test dataset (Fig.
The prediction absolute error can reach up to 75% of the LES solution. The fine structures of the plume are hard to predict since information is mostly carried by multiple high-order POD modes in a region where there are only a few samples of the emission source in the LES dataset (this case is located at the boundary of the tracer source area -Fig. 1 ). To complement the analysis, we finally present the prediction result for a case where the emission source is located in the recirculation area downstream of the obstacle (Fig. 15 ). This case is much better predicted by the reduced-order model. The areas of high tracer concentrations form a wide-spread structure in areas carried by the first POD modes. Most of the information can be therefore conveniently recovered from the first reduced coefficients with accurate GPR models, including in the recirculation area downstream of the obstacle.
The largest prediction error is on the order of 10% of the LES solution. These three snapshots highlight the capacity of the reduced-order model to predict the main tracer concentration structures, in particular when the emission source is located in recirculation zones near the obstacle. In this situation, the tracer concentration patterns are smooth as the dispersion is dominated by turbulent diffusion, making the prediction task easier. These results were obtained by considering a large number of modes in the reduced basis with a rich training dataset (N train = 472). Such a large LES database is unaffordable in all realistic atmospheric dispersion cases involving larger computational domains and three-dimensional effects. We now evaluate how much the reduced-order model accuracy degrades when the size of the training dataset is reduced to have a better idea of the minimum required budget for building a multi-query uncertainty quantification framework.
Robustness to training dataset We now reduce the training dataset to one hundred snapshots (N train = 100) and fifty snapshots (N train = 50) to move towards a more realistic multi-query LES framework. Note that 90% of these training snapshots are used to determine the POD reduced basis, with the remaining 10% used for calibration as before. This implies that the maximum number of modes in the reduced basis is directly equal to the POD training size (90 for N train = 100; 45 for N train = 50). The only difference is that here the whole dataset is used to optimize the GPR models since the dataset is of very limited size. Note also that the test dataset remains the same as before (as described in Sect. 3. 4. 1) to avoid introducing bias during the validation stage. Figure  16  compares the evolution of per-mode Q 2 for both full and reduced training datasets.
The reduced-order model accuracy significantly decreases for the reduced database. There is a faster linear decrease towards Q 2 = 0 than with the full training dataset (the threshold Q 2 = 0 is approximately reached for the fiftieth mode for N train = 100 and the the twenty-fifth mode for N train = 50). This suggests that the number of POD modes to consider in the reduced-order models should be reduced due to the too limited size of the training dataset. Figure  16 : Per mode-Q 2 (Eq. 26) for GPR models trained with 472 snapshots (dotted line corresponding to the MAP result already presented in Fig. 12 ), 100 snapshots (dashdotted line) or 50 snapshots (densely dashdotted line). Including higher-order modes in the reduced-order model can even degrade its global performance. Here, the optimal choice for N train = 100 is to keep L = 42 modes and L = 25 modes for N train = 50.
The associated global Q 2 score is equal to 87. 3% and 85. 1%, respectively, and can be compared to 96. 8% for the full training dataset (Sect. 4. 3). These scores may seem satisfactory but when looking at the nominal snapshot prediction, the prediction errors observed before are amplified. Figure  17  presents the prediction result for N train = 100. The shape of the tracer concentration wake is retrieved but the reduced-order model has difficulty to predict the correct tracer concentration magnitude: the tracer concentration is underestimated at the emission source and upstream of the obstacle, and the high tracer concentrations correspond to a much thinner region than for the full training dataset (Fig. 13 ). This highlights that the analysis of the overall Q 2 score can be misleading about the reduced-order model accuracy, since the prediction quality is spatially heterogeneous and can decrease upstream of the obstacle due to the large tracer concentration gradients near the emission source.
Application to turbulent scalar fluxes To better exploit the explicit resolution of turbulence by LES approach (compared to other approaches such as RANS for instance), it is possible to rely on a similar reduced-order modeling approach for higher-order statistical quantities. In the context of dispersion, the turbulent scalar flux is a key quantity that is fully modelled in low-fidelity dispersion models  [Vervecken et al. , 2015b]. From LES data, reduced-order model for this type of quantities can be extracted. This illustrated in this section by extending our reduced-order modeling approach to the resolved vertical turbulent scalar flux defined and normalized as (vK -vK) H 2  Qs , where v is the vertical component of the velocity. This demonstration is carried out with the full training dataset (N train = 472) as in Sect.
4. 1 to Sect. 4. 3. The finer structures make the prediction of these fields slightly more complex than for the mean quantities of interest. The POD variance is indeed spread over a larger number of POD modes: one hundred modes carry 99. 3% of the ensemble variance for the vertical turbulent scalar flux (compared to 99. 6% for the mean quantities of interest); and the Kaiser rule (Sect. 4. 1) suggests to keep around 32 modes instead of 25 modes for the mean quantities of interest. As for the mean quantities of interest, the maximum performance on the test dataset is obtained for L = 100 modes, and achieves Q 2 = 94. 0% globally for the vertical turbulent scalar flux (compared to 96. 8% for the mean quantities of interest). Hyperparameter trends (not shown) are similar to what we described in Sect.
Conclusion The goal of this study was to design and validate a non-intrusive reduced-order model combining POD and GPR for predicting LES field statistics of interest associated with near-source tracer concentration dispersion in a multi-query uncertainty quantification context. A two-dimensional case study corresponding to a turbulent atmospheric flow over a surface-mounted obstacle was considered to generate a large LES ensemble database based on perturbed inflow boundary conditions and emission source location. POD analysis revealed that low-order modes carry concentration information in widely-spread areas and near the obstacle, whereas high-order modes are necessary to include in the POD basis to characterize the concentration high variability near the emission sources. Hundred modes were therefore retained in the reduced-order model, with a mode-per-mode optimization of the Gaussian process hyperparameters to adapt to the wide range of spatial scales across the POD basis.
These high-order modes feature very localized structures associated with perturbed emission location, which are difficult to predict and which are prone to more noise than low-order modes. We also showed that this difficulty of predicting high-order modes increased when the training dataset  is reduced, but the prediction performance remains acceptable when considering at least fifty to hundred LES snapshots in the training database. For all these issues, the mode-per-mode analysis was essential to understand the behavior of the reduced-order model. Future work includes testing alternative compression approaches to POD. Since POD is based on linear algebra, decomposing information with substantial nonlinearity might be tricky. Deep-learning-inspired techniques such as autoencoders offer an intriguing line of study for improving compression performance  [Fukami et al.