Lessons Learnt from Studies on Sensitivity Analysis Techniques in the EU Project PAMINA - A Benchmark Study

Several EU PAMINA project partners have applied different Sensitivity Analysis (SA) techniques to analytical models as well as to a simplified, though representative, PA model. The aims of the exercise were to investigate the performance of different techniques in the presence of features such as (lack of) linearity, (lack of) monotony, interactions, etc.; to check the importance of sample size; to compare different options within a given SA technique; to cross-compare the results obtained using different techniques; and to get a better understanding of the rationale behind every available technique and about their capabilities and shortcomings. This paper presents and discusses some results on the tests of global sensitivity analysis performed with variance-based sensitivity indicators. Facilia (Sweden), JRC-Petten (The Netherlands), and TU Clausthal (Germany). A diverse range of available algorithms was in use, starting from linear regression over variance-based global sensitivity analysis to screening methods and statistical tests for performing Monte-Carlo Filtering. In order to unify the results and to draw more attention to the variance-based SA indicators a prescribed setting was specified for certain models to be analysed in a second simulation round, of which selected results are presented in this paper. For each model, 25 runs of 100, 300, 1000, 3000, and 10000 samples sizes were requested and for each run the indicators mean, variance, , rank-based , and the variance-based sensitivity indicators (SI) first order effects and total effects (where available) were computed. The choice of the SI algorithms was left to the participants of this second round. Details of the theory behind the different sensitivity analysis algorithms are presented in Badea and Bolado (2008) and in the books Saltelli et al. (2000), Saltelli et al. (2004), Saltelli et al. (2008). For the second round, contributions were received from Facilia (FCL, Sweden), GRS Cologne (Germany), JRC-Petten (The Netherlands), and TU Clausthal (TUC, Germany). An example gathering some of the problems encountered in the analytical benchmarks is the PSACOIN Level E model, Nuclear Energy Agency (1989). As an optional element for the participants, a sensitivity study of the Level-E geosphere transport model was requested. Here Facilia and TUC provided results. 3 SA TECHNIQUES USED IN THE STUDY As we already noted the choice of the SA methods was left to the participants. In this section we concentrate on the methods used in the second phase of the benchmark where the SA techniques were restricted to variance-based sensitivity analysis methods. The sensitivity index of first order effect is given by For total effects analogous formulas apply where in the condition “given ” is replaced by “given all but ”. We can classify the methods for the calculation of the first order and/or total effects into four different groups. Correlation ratio methods. Here the conditional variance in the formula of the sensitivity indices is replaced by for a suitable partition of the range of the th input parameter. To compute higherorder effects or total effects these methods suffer from the curse of dimensionality. But despite all drawbacks, these methods can work with given data; hence model evaluations available prior to the SA (e.g. from Monte Carlo simulations performed in order to carry out uncertainty analyses, i.e. to obtain statistics for the model output) can be reused. Then the choice of different partitions influences the result of such a method. Sobol' methods. For these methods, a special sampling scheme ensures that there are enough realisations available so that statistics for can be efficiently computed. A simple scheme is named after Ishigami/Homma/Saltelli (IHS), another scheme using a hyperconvergent quasiMonte-Carlo sampling scheme is named after Sobol'. These methods can estimate first order and total effects. Fourier-based techniques. For these methods, the input parameter realisations have to fulfil special frequency properties. Then a frequency decomposition of the output maps different frequencies attributed to the input factors to different fractions of the variance of the output. Different frequency selection schemes have been developed, named Fourier Amplitude Sensitivity Test (FAST), Extended FAST (EFAST), and Random Balance Design (RBD). They can estimate first and/or total effects. Other “cheap” methods. Methods working with pre-computed model evaluations are computationally efficient. As straight-forward extensions of a linear regression method a polynomial fit of the data and a conditional linear fit have been tested. Furthermore, in the course of the benchmark an algorithm named EASI (“Effective Algorithm for variance-based Sensitivity Indices”) has been developed that couples given model evaluations with Fourier-based sensitivity analysis techniques, see Plischke (2009). 4 MODELS CONSIDERED IN THE STUDY The four analytical models used in the second round were chosen in such a way that a sensitivity analysis based upon linear regression analysis was bound to fail. These models are non-linear, non-monotonic, discontinuous, multi-parametric, or with input dependencies. Moreover, a time-dependent model which has some of these properties is the PSACOIN Level-E geosphere transport model. Some results on the SA of this complex model are also reported. 4.1 The Ishigami test function The Ishigami test function is a three parameter model. It is in so far interesting as the second and third input factors have a Pearson Correlation Coefficient of zero. A variance-based analysis retrieves a 44% first order effect for the second input factor, but a zero effect for the third factor. Only when estimating total effects the third input factor is attributed 24% of the variance. The Ishigami function is given by where are uniformly distributed in