# Robust model calibration using determinist and stochastic performance metrics

Paul Lepine, Scott Cogan, Emmanuel Foltete, Marie-Océane Parent

► To cite this version:

Paul Lepine, Scott Cogan, Emmanuel Foltete, Marie-Océane Parent. Robust model calibration using determinist and stochastic performance metrics. International Modal Analysis Conference, Jan 2016, Orlando, United States. pp.185-191, 10.1007/978-3-319-29754-5_18. hal-02131297

HAL Id: hal-02131297

https://hal.science/hal-02131297

Submitted on 9 Jan 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Robust Model Calibration Using Determinist and Stochastic Performance Metrics

P. Lépine, S. Cogan, E. Foltête, and M.-O. Parent

**Abstract** The aeronautics industry has benefited from the use of numerical models to supplement or replace the costly design-build-test paradigm. These models are often calibrated using experimental data to obtain optimal fidelity-to-data but compensating effects between calibration parameters can complicate the model selection process due to the non-uniqueness of the solution. One way to reduce this ambiguity is to include a robustness requirement to the selection criteria. In this study, the info-gap decision theory is used to represent the lack of knowledge resulting from compensating effects and a robustness analysis is performed to investigate the impact of uncertainty on both deterministic and stochastic fidelity metrics. The proposed methodology is illustrated on an academic example representing the dynamic response of a composite turbine blade.

**Keywords** Uncertainty • Model calibration • Info-gap approach • Performance metric • Robust solution

## 18.1 Introduction

Ceramic matrix composite (CMC) turbine blades have been developed by HERAKLES [1]. They consist of ceramic fibers embedded in a ceramic matrix. These textile composites show high resistance to extremely high temperature (1000 °C), low density and a good fracture toughness compared to conventional metallic alloys. Finite element models (FEM) applied to the design of composite parts are under investigation in order to obtain the predictive simulations.

Model calibration is commonly used to improve the correlation between FEM and measured data. Typically, fidelity-to-data is optimized in such approaches even though compensating effects between model parameters may lead to subspaces of fidelity-equivalent solutions [2]. These effects tend to hide missing physics in the model form and create ambiguity in the model selection process. A model is considered as predictive if it remains in a subdomain limited by a satisfying boundary at the required performance level. A satisfying boundary is a family of plausible models displaying the same comparable measure of fidelity [3]. A recent paper has proposed to complete a deterministic fidelity error metric with a robustness criteria quantifying the impact of compensating effects on the fidelity-to-data [3]. This paper will extend that approach to illustrate that it remains relevant for stochastic error metrics as well. Info-gap decision theory is used to provide a formal framework for investigating the impact of unknown compensating effects on both deterministic and stochastic fidelity metrics. A simplified model of a turbine blade is studied to illustrate the proposed methodology using simulated test results.

<footnote>P. Lépine (✉)
Institut FEMTO-ST, Université de Bourgogne-Franche-Comté, 24 rue de l'épitaphe, 25000 Besançon, France</footnote>

<footnote>SNECMA-SAFRAN, Rond-point René Ravaud, Réau, 77550 Moissy-Cramayel, France
e-mail: paul.lepine@femto-st.fr</footnote>

<footnote>S. Cogan • E. Foltête
Institut FEMTO-ST, Université de Bourgogne-Franche-Comté, 24 rue de l'épitaphe, 25000 Besançon, France</footnote>

<footnote>M.-O. Parent
SNECMA-SAFRAN, Rond-point René Ravaud, Réau, 77550 Moissy-Cramayel, France</footnote>
### 18.1.1 Fidelity-to-Data Metrics for Model Calibration

The fidelity of simulated outcomes to experimentally identified outcomes can be quantified with both deterministic and stochastic metrics. For example, a common deterministic metric is the normed Euclidean distance $D_E$:

$$D_E = \sum_{i=1}^{n} \frac{\sqrt{(v_{a_i} - v_{m_i})^2}}{v_{m_i}} \qquad (18.1)$$

where $v_m$ is a vector containing the experimentally identified outcomes ($n$ outputs) and $v_a$ the homologous vector containing the analytical outcomes.

When population data is available for simulations and/or experiments, stochastic metrics can be implemented. In this case, the model now provides uncertain outputs and can be calibrated using stochastic approaches such as covariance adjustment [4] or Gibbs sampling [5] and Metropolis-Hasting algorithm [6]. In this paper, the Bhattacharyya distance $D_B$ will be used to quantify the fidelity-to-data for multivariate features [7] but other metrics could be used as well. The Bhattacharyya metric takes into account the differences between both the means and the covariances between the simulated and experimental distributions:

$$D_B = \frac{1}{8}(\bar{v}_a - \bar{v}_m)^T \Sigma^{-1} (\bar{v}_a - \bar{v}_m) + \frac{1}{2} \ln \left( \frac{\det(\Sigma)}{\sqrt{\det \Sigma_a \det \Sigma_m}} \right) \qquad (18.2)$$

where $\bar{v}_m$ is the vector containing the mean values of the measured outcomes and $\bar{v}_a$ the mean values of the simulated outcomes. The pooled matrix $\Sigma = \frac{\Sigma_a + \Sigma_m}{2}$ combines the covariance matrices of the simulated and experimental outcomes.

### 18.1.2 Info-Gap Theory

Info-gap decision theory, developed by Ben-Haim [8], has been used to study a wide range of problems in various disciplines including climate models [9] and medical research [10]. The purpose of info-gap is to provide tools for decision-makers in order to assess risks and opportunities of model-based decisions under a severe lack of information. The reader is referred to [8] for more detailed information.

In the info-gap approach, a horizon of uncertainty $\alpha$ quantifies the difference between what is known and what needs to be known to make an informed decision. When considering fidelity-to-data $R$, the worst case fidelity $\hat{R}$, for a given horizon of uncertainty $\alpha_i$, is evaluated by solving the following optimization problem [3]:

$$\hat{R}(\alpha) = \max_{\theta \in U(\alpha, \tilde{\theta})} R(\theta) \qquad (18.3)$$

where $\tilde{\theta}$ is the vector of calibrated best-estimate parameters of the simulation. The robustness function expresses the greatest level of uncertainty at which performance remains acceptable.

$$\hat{\alpha} = \max\{\alpha : \hat{R}(\alpha) \geq R_c\} \qquad (18.4)$$

where $\hat{\alpha}$ is the maximum horizon for which the performance function $R \leq R_c$ is satisfied.

The notional diagram in Fig. 18.1 illustrates how the robustness curve is obtained. Let $u_1$ and $u_2$ be the calibration parameters. The point $\alpha_A = 0$ corresponds to the nominal model with no uncertainty and $R(u_{1A}, u_{2A})$ the corresponding performance. As the horizon of uncertainty increases to $\alpha_B$, the worst case performance over the uncertain domain is given by $R(u_{1B}, u_{2B})$, and so on. In practice, the robustness curve is obtained by solving an optimization problem with an appropriate algorithm based on either local or global approaches. For example, in [11], the authors search the space using a factorial design.
<figure><img src="image_2.png" /><figcaption>**Fig. 18.1** Nested input sets (left) and the corresponding robustness curve (right)</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>**Fig. 18.2** 3D tomography reconstruction (a) and the FEM plate studied (b) [15]</figcaption></figure>

## 18.2 Numerical Application

Ceramic composite textiles are made of sets of tows Si-C sewn into a pattern as shown in Fig. 18.2. A ceramic matrix is filled into the fibres of the preform to enhance the mechanical properties. Composite textiles can be considered as periodic mesoscopic structures characterized by their smallest elementary pattern called the representative unit cell. A mesoscopic analysis allows the homogenized properties of the composite [12] to be computed. The mesoscopic structure of the studied material has been probed using an X-ray Micro Tomograph and its behavior through multi-instrumented experiments [13, 14].

These properties are then implemented in a numerical model to simulate the structural responses. It is not uncommon for the nominal test-analysis error to be unacceptably large. Given the uncertainties in the manufacturing process and experiments, stochastic calibration approaches are best adapted to improve model fidelity. Model form error (MFE)—due to poorly modeled boundary conditions, geometry, and material properties—is also an important source of discrepancy between simulated and experimental outcomes and remains a challenge in the calibration process.

The purpose of this section is to illustrate the impact of compensating effects and model form error on model selection with both optimal and robust strategies. A simple two parameters example is studied representing a free-free thin CMC plate with orthotropic properties. The calibration parameters are the Young's modulus $E_1$ and $E_2$ while the other parameters are assumed to be known. The experimental data are simulated based on the nominal model parameters. The fidelity error metric is formulated with respect to the eigenfrequencies of the structure.

The deterministic and stochastic calibration metrics for the first ten eigenfrequencies are calculated over the domain [0.6 : 1.4], discretized into 21 levels, corresponding to the correction coefficients for the parameters $E_1$ and $E_2$. The center point $E_1 = E_2 = 1$ thus defines the nominal design used to simulated the tests. In order to reduce calculation costs for the population based fidelity metric, a neural network surrogate model is constructed from a preliminary full factorial design. The Bhattacharyya metric is then evaluated at each design point based on $10^5$ Monte-Carlo samples.
<figure><img src="image_4.png" /><figcaption>Fig. 18.3 Isocontour performance without model form error. (a) Euclidean metric. (b) Bhattacharyya metric</figcaption></figure>

<table><caption>Table 18.1 Random parameter values for case 1</caption><thead><tr><th>Parameters value</th><th colspan="3">Output means value</th><th colspan="3">Output standard deviation</th><th colspan="3">Output covariance</th></tr><tr><th>(E<sub>1</sub>, E<sub>2</sub>)</th><th>μ&#772;<sub>1</sub></th><th>μ&#772;<sub>2</sub></th><th>μ&#772;<sub>3</sub></th><th>σ(μ<sub>1</sub>)</th><th>σ(μ<sub>2</sub>)</th><th>σ(μ<sub>3</sub>)</th><th>cov(μ<sub>1</sub>, μ<sub>2</sub>)</th><th>cov(μ<sub>2</sub>, μ<sub>3</sub>)</th><th>cov(μ<sub>1</sub>, μ<sub>3</sub>)</th></tr></thead><tbody><tr><td>(1,1)</td><td>1.00</td><td>1.13</td><td>2.52</td><td>1.00</td><td>0.03</td><td>1.13</td><td>0.18</td><td>0.20</td><td>1.12</td></tr><tr><td>(0.7,0.7)</td><td>0.84</td><td>1.12</td><td>2.23</td><td>1.58</td><td>0.00</td><td>5.43</td><td>-0.06</td><td>-0.12</td><td>2.92</td></tr><tr><td>(0.7,1.3)</td><td>0.84</td><td>1.12</td><td>2.27</td><td>1.53</td><td>0.00</td><td>5.42</td><td>-0.07</td><td>-0.12</td><td>2.88</td></tr><tr><td>(1.3,1.3)</td><td>1.11</td><td>1.17</td><td>2.60</td><td>0.53</td><td>0.20</td><td>0.03</td><td>0.33</td><td>0.07</td><td>0.12</td></tr><tr><td>(1.3,0.7)</td><td>1.11</td><td>1.17</td><td>2.58</td><td>0.52</td><td>0.21</td><td>0.03</td><td>0.33</td><td>0.06</td><td>0.10</td></tr></tbody></table>

### 18.2.1 Test Case Without Model Form Error

In this first case, we assume that the numerical eigenfrequencies are obtained with a perfect model, that is to say, without any model form error. The fidelity metrics are plotted in the space of the two parameters (Fig. 18.3). The contours indicate fidelity-equivalent solutions and define the satisfying boundaries. The square marker designates the globally minimum error and represents the optimal solution. As expected, this marker totally coincides with the design point used to generate the experimental data for distance metrics.

In Fig. 18.3a, there is a large area where the error remains below 5%, illustrating that calibrated solutions can be very different for equivalent levels of fidelity. These compensating effects are inevitable even in the absence of bias in the model prediction. In Fig. 18.3b, the isocontours create nested contours centered on [1, 1]. The reference outputs at $E_1 = E_2 = 1$ and some numerical results are presented in Table 18.1.

### 18.2.2 Test Case with Model with Form Error

In this case we assume that the model contains a model form error and is unable to perfectly simulate the observed physics. In this academic example, the shear modulus $G_{12}$ is overestimated about 20%. The experimental eigenfrequencies remain the same as before. Figure 18.4 displays the new satisfying boundary maps.

The shape of the isocontours for the euclidean metric is similar to the previous case (Fig. 18.3a) with a noticeable shift toward the upper left of the domain. Thus the optimal parameters are no longer located in (1, 1) with 2.9% error but (0.72, 1.12) with 1.5%. The Bhattacharyya metric displays more complex isocontour lines and a shifted optimal solution as well. The implication of this shift is that any calibration algorithm based uniquely on optimizing the fidelity will invariably converge to the wrong solution. Although there is clearly no simple solution to this problem, it is possible in some cases to attenuate the impact of compensating effects on model fidelity.
<figure><img src="image_5.png" /><figcaption>**Fig. 18.4** Isocontour performance with model form error. (a) Euclidean metric. (b) Bhattacharyya metric</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Fig. 18.5** Robust isocontour performances of the optimal model. (a) Euclidean metric. (b) Bhattacharyya metric</figcaption></figure>

### 18.2.3 Robustness to Compensating Effects

Given the inevitable presence of compensating effects between calibration parameters—with and without model form error—the approach proposed in [3] has been adopted here with the objective of demonstrating that it remains relevant for both deterministic and stochastic fidelity metrics. In this context, the info-gap uncertainty represents the lack of knowledge in the parameter space due to compensating effects and the goal is to find the most robust sets of parameters $E_1, E_2$ insuring an acceptable level of fidelity.

With respect to the previous test case without model form error, the corresponding robust fidelity metric is shown in Fig. 18.5 for a horizon of uncertainty $\alpha = 10\%$. These contours are obtained by evaluating, for each design point ($E_1^i, E_2^i$), the worst-case fidelity within the uncertain subdomain, a square of sides $0.1 \times 0.1$, centered about the point. The contours have not changed significantly in shape but the overall errors have of course increased. The round marker indicates the best robust calibrated solution which is different from the optimal solution from before. These contours clearly depend on the horizon of uncertainty and increasing $\alpha$ will necessarily lead to larger fidelity errors and even different robust solutions.

The robust fidelity isocontours for the second test case containing model form error are shown in (Fig. 18.6), with markers for the reference solution (cross-shaped), the optimal solution (square), and the robust-optimal solution (round). Table 18.2
<figure><img src="image_7.png" /><figcaption>Fig. 18.6 Robust isocontour performances of the model with form error. (a) Euclidean metric. (b) Bhattacharyya metric</figcaption></figure>

<table><caption>Table 18.2 Optimal-robust parameters comparison</caption><thead><tr><th rowspan="2"></th><th colspan="2">Nominal</th><th colspan="2">Stochastic</th></tr><tr><th>Coordinates</th><th>Robust error (%)</th><th>Coordinates</th><th>Robust error</th></tr></thead><tbody><tr><th>Reference</th><td>(1.00,1.00)</td><td>5.45</td><td>(1.00,1.00)</td><td>1.02 × 10<sup>4</sup></td></tr><tr><th>Optimal</th><td>(0.72,1.12)</td><td>2.81</td><td>(0.80,1.12)</td><td>9.13 × 10<sup>3</sup></td></tr><tr><th>Optimal-robust</th><td>(0.72,1.12)</td><td>2.81</td><td>(0.76,1.08)</td><td>8.76 × 10<sup>3</sup></td></tr></tbody></table>

provides a summary of these solutions. In this example, these solutions are clearly different and although it is generally impossible to know the true solution, it is possible to improve the robustness of the solution to compensating effects for both deterministic and stochastic fidelity metrics.

## 18.3 Conclusion

Model calibration based uniquely on fidelity-to-data metrics fall prey to compensating effects between model parameters due to the presence of model form errors and subspaces of fidelity-equivalent solutions. This paper extends earlier work that includes a robustness criteria in the model selection process and illustrates the relevance of such an approach for both deterministic and stochastic metrics. An academic test case representing a composite turbine blade is studied in a notional way with simulated test data. Two calibration metrics have been investigated, in particular a deterministic Euclidean error and a statistical Bhattacharyya error, and both are seen possess both fidelity optimal and robust optimal solutions which are distinct from one another.

## References

1. SNECMA and HERAKLES: Aube de Turbomachine en Matériau Composite et Procédé pour sa Fabrication - Brevet. WO 2011/080443 A1 (2011)

2. Berman, A.: Multiple acceptable solutions in structural model improvement. AIAA J. 33(5), 924–927 (1995)

3. Atamturktur, S., Liu, Z., Cogan, S., Juang, H.: Calibration of imprecise and inaccurate numerical models considering fidelity and robustness: a multi-objective optimization-based approach. Struct. Multidiscip. Optim. 51, 659–671 (2014)

4. Govers, Y., Link, M.: Stochastic model updating - Covariance matrix adjustment from uncertain experimental modal data. Mech. Syst. Signal Process. 24(3), 696–706 (2010)

5. Smith, A.F.M., Roberts, G.O.: Bayesian computation via the Gibbs sampler and related Markov chain Monte Carlo methods. J. R. Stat. Soc. 55(1), 3–23 (1993)

6. Chib, S., Greenberg, E.: Understanding the Metropolis-Hastings algorithm. Am. Stat. 49(4), 327–335 (1995)

7. Aherne, F.J., Thacker, N.A., Rockett, P.I.: The Bhattacharyya metric as an absolute similarity measure for frequency coded data. Kybernetika 34(4), 363–368 (1998)
8. Ben-Haim, Y.: Info-Gap Decision Theory: Decisions Under Severe Uncertainty. Academic, New York (2006). [ISBN: 978-0-12-373552-2]

9. Hall, J.W., Lempert, R.J., Keller, K., Hackbarth, A., Mijere, C., McInerney, D.J.: Robust climate policies under uncertainty: a comparison of robust decision making and info-gap methods. Risk Anal. **32**(10), 1657–1672 (2012)

10. Ben-Haim, Y., Zacksenhouse, M., Keren, C., Dacso, C.C.: Do we know how to set decision thresholds for diabetes? Med. Hypotheses **73**(2), 189–193 (2009)

11. Kuczkowiak, A.: Modèle hybride incertain pour le calcul de réponse en fonctionnement d'un alternateur. PhD thesis, Université de Franche-Comté (2014)

12. Naouar, N., Vidal-Sallé, E., Schneider, J., Maire, E., Boisse, P.: Meso-scale FE analyses of textile composite reinforcement deformation based on X-ray computed tomography. Compos. Struct. **116**, 165–176 (2014)

13. Couégnat, G.: Approche multiéchelle du comportement mécanique de matériaux composites à renfort tissé. PhD thesis, Université Sciences et Technologies-Bordeaux I (2008)

14. Dupin, C.: Etude du comportement mécanique des matériaux composites à matrice céramique de faible épaisseur. PhD thesis, Bordeaux 1 (2013)

15. Naouar, N., Vidal-Salle, E., Schneider, J., Maire, E., Boisse, P.: 3d composite reinforcement meso F.E. analyses based on X-ray computed tomography. Compos. Struct. **132**, 1094–1104 (2015)