# Uncertainty Analysis of Method-Based Operating Event Groups Ranking

Authors: Zdenko Šimić; Benoit Zerger; Reni Banov
Journal: Science and Technology of Nuclear Installations
 (2014)

---

## Abstract

Safe operation and industrial improvements are coming from the technology development and operational experience (OE) feedback. A long life span for many industrial facilities makes OE very important. Proper assessment and understanding of OE remains a challenge because of organization system relations, complexity, and number of OE events acquired. One way to improve OE events understanding is to focus their investigation and analyze in detail the most important. The OE ranking method is developed to select the most important events based on the basic event parameters and the analytical hierarchy process applied at the level of event groups. This paper investigates further how uncertainty in the model affects ranking results. An analysis was performed on the set of the two databases from the 20 years of nuclear power plants in France and Germany. From all uncertainties the presented analysis selected ranking indexes as the most relevant for consideration. Here the presented analysis of uncertainty clearly shows that considering uncertainty is important for all results, especially for event groups ranked closely and next to the most important one. Together with the previously performed sensitivity analysis, uncertainty assessment provides additional insights and a better judgment of the event groups’ importance in further detailed investigation.


---

## Body

## 1. Introduction

Collecting and understanding operating experience are an important part of keeping continuous, reliable, and safe operation of any complex industrial facility including nuclear power plants. This operating experience is taken from the facility to the national and international levels (e.g., [1, 2]). More detailed investigation is performed only for selected important events. The selection process is easy for accidents, but not so easy for a large number of events because this requires resources and may imply a degree of subjectivity. One approach to select events for detailed investigation is to develop and apply a method based on event groups ranking. Ranking results are useful to regulators and industry for better resource prioritization in maintaining and improving safety and operation.In [3], four different approaches for event groups ranking were compared with findings about their difference and proposed favorable method. The selected method is based on the application of the analytical hierarchy process in order to allow easier determination of relative importance for all ranking indexes (RI). Sensitivity of the method was analyzed in [4] with findings on how events grouping and selected dataset (i.e., technology and country) influence the results.This paper investigates how the uncertainty of ranking indexes of relative importance (i.e., weightings) influences the ranking results. The described method is applied separately for 20 years of events from the nuclear power plants operation in France and Germany. The focus is only on the uncertainty for each data source, with no aim of comparing them because of significant differences in technology, data collection, and other factors.


## 2. Method

Here, we first briefly present the event groups ranking method and then proceed to analyze uncertainty. The ranking method description focuses on the most critical elements for the uncertainty. As mentioned before, a complete description of the method is available in [3].


### 2.1. Ranking Method and Inputs for Uncertainty Assessment

Ranking is performed at the level of event groups (EG) using selected sets of parameters. For the brevity, only about half of the more important groups are selected and presented in the paper. Table1 presents respected parameters values for selected event groups (Table 3 lists respected description for each EG). In total, 20 event groups are analyzed for French (FR) and 14 for German (DE) datasets. Because of technology and data collection differences, some event groups do not exist in both datasets and there are also differences in parameters. Both datasets have frequency, trend, and precursors parameters. The French dataset has special parameters which present a number of so-called “generic and recurring” events. There is also a difference in the number of events reported to the IAEA/NEA (International Atomic Energy Agency/Nuclear Energy Agency) international reporting system (IRS) because in France it is chosen to report only one representative event in case similar events occur in several plants of the French fleet. The trend value is presenting change of the number of events for a certain period (i.e., 0.5 means that the number of events is not changing, values up to 1 mean increasing trend, and values down to 0 represent a decreasing trend). The trend has been considered, in this analysis, only for the last five years the most relevant. All other parameters are for a whole period of 20 years.Table 1
Selected event groups for ranking uncertainty assessment with respected number of events and trend value.
ID
FR
DE
N
G & R
IRS
Pr.
T5y
U
N
IRS
Pr.
T5y
U
01.2
161
12
5
11
0.98
N
102
0
3
0.33
Y
02.3
271
23
2
15
0.35
Y
6
0
0
0.50
N
03
44
10
7
8
0.20
Y
33
1
1
0.50
Y
04.1
89
89
16
10
0.18
Y
44
1
0
0.76
N
04.2
20
16
4
0
0.83
N
6
1
0
0.23
Y
04.3
70
70
2
5
0.52
N
38
0
0
0.01
Y
05.1
64
35
43
21
0.42
Y
1
0
0
0.50
N
05.2
48
1
1
6
0.18
Y
149
0
8
0.75
Y
07.1
1415
149
43
97
0.97
Y
168
3
19
0.54
Y
08.2
18
3
3
5
0.50
N
104
0
2
0.32
Y
08.3
320
52
31
24
0.25
Y
—
—
—
—
—
09.1
105
2
16
15
0.21
Y
167
1
13
0.81
Y
09.3
130
12
16
35
0.20
Y
8
0
0
0.50
N
10.2
5
0
0
0
0.53
N
33
0
0
0.12
Y
12
236
105
48
65
0.40
Y
4
0
1
0.50
N
13.2
—
—
—
—
—
—
351
5
11
0.93
Y
13.3
—
—
—
—
—
—
458
6
10
0.94
Y
13.4
1054
114
46
190
0.17
Y
—
—
—
—
—
14
470
3
12
48
0.97
N
172
5
22
0.04
Y
15
111
67
6
15
0.00
Y
42
1
2
0.76
N
16.1
1194
803
14
39
0.87
Y
69
3
2
0.97
N
16.2
1681
0
5
34
0.00
Y
68
1
1
0.89
N
16.3
1726
79
28
61
0.02
Y
102
0
1
0.60
N
17.1
393
270
13
22
0.16
Y
—
—
—
—
—
17.2
219
24
9
20
0.47
Y
—
—
—
—
—
17.3
162
162
14
5
0.26
Y
—
—
—
—
—
17.4
—
—
—
—
—
—
119
0
0
0.21
Y
18.1
108
66
22
15
0.25
Y
23
0
1
0.93
N
19.3
35
0
2
6
0.50
N
12
1
3
0.55
Y
21
78
48
25
30
0.40
Y
82
3
0
0.93
N
FR: French dataset; DE: German dataset;  ID: number of the event groups (described in Table3); N: number of events; G & R: subset of events in the FR database comprising generic and recurring events; IRS: subset of events reported to the IRS (for FR, only the first-time event for each type is reported); Pr.: precursor events; T5y: trend parameter value (i.e., 0.5 for unchanged number of events, up to 1 for increasing trend, and down to 0 for decreasing trend); U: event groups considered (“Y,” or not “N”) in the presented uncertainty assessment.The ranking index value is calculated for each parameter and event group as a ratio of the number of events for that parameter in the group and the maximal number of events for the same parameter considering all the event groups:(1)
R
I
EG
.
parameter
=
N
EG
.
parameter
N
EG
.
param
.

max
⁡
,
where RI is ranking index, EG is event group, and N is number of events.For the trend, RI value is calculated to present an unchanged number of events with 0.5, increasing number of events (parameter value up to 1), and decreasing number of events (values down to 0) over a certain period.The final total ranking value for each EG is calculated as the sum of all parameters ranking values multiplied by respect weights:(2)
R
I
EG
=
∑
parameters
w
parameter
R
I
EG
.
parameter
,
where RI is ranking index, EG is event group, and w is weights for respected parameter.These weights could be judged by experts directly or more transparently and consistently by using some methods like the analytical hierarchy process (AHP, [3, 5]) where the only requirement is to make pairwise comparisons for all parameters. Table 2 presents a set of RI weights used in this assessment which was derived using AHP results as the base. (Short description of the method is provided in the appendix.)Table 2
Ranking indexes weights for each dataset.
Country ranking index
Frequency
Generic and recurring
IRS
Precursor
Trend
France (FR)
0.07
0.19
0.15
0.36
0.23
Germany (DE)
0.08
NA
0.20
0.40
0.32
IRS: events reported to the IAEA/NEA international reporting system.Table 3
ID and description for selected event groups with presence in the respected country database.
ID
Description
FR
DE
01.2
Loss of offsite power-loss of electrical bus bar
N
Y
02.3
Reactivity events-transient pressure or temperature changes
Y
N
03
Loss of heat removal system
Y
Y
04.1
Leakages from the primary circuit
Y
N
04.2
Leakages from primary to secondary circuit
N
Y
04.3
Primary coolant leakage outside cont.
Y
Y
05.1
Essential water intake clogging
Y
N
05.2
Failures in the component cooling or service water systems operating trains
Y
Y
07.1
Scram with influence on reactivity
Y
Y
08.2
Other instrumentation and control failures affecting the reactor protection system
N
Y
08.3
Instrumentation and control failures related to other safety systems
Y
—
09.1
Complete emergency diesel generator failure
Y
Y
09.3
Incipient emergency diesel generator failure
Y
N
10.2
Control rod events without influence on reactivity
N
Y
12
Common cause failure events
Y
N
13.2
Failure of 1 safety system redundancy
—
Y
13.3
Failure of a safety system component
—
Y
13.4
Failure of safety features redundancies
Y
—
14
Actuation of safety features
Y
Y
15
Lubricant
Y
N
16.1
Circuit misalignment (human/organization)
Y
N
16.2
Risk analysis (human/organization)
Y
N
16.3
Inadequate maintenance quality (human/organization)
Y
N
17.1
Failure after release from maintenance or design modification
Y
—
17.2
Failure caused by plant design modification
Y
—
17.3
Failure caused by using temporary devices
Y
—
17.4
Failure related to NPP modification
—
Y
18.1
Event external to the site
Y
N
19.3
Other reportable internal fires
N
Y
21
Design/construction
Y
N
Y or N designates existence of the particular event group in the FR and DE country database.Applying (1) and (2) to all EG using data presented in Table 1 and weights in Table 2, total ranking values are quantified. Most important event groups are then considered once which have the highest ranking values. (Selected results are presented in the Table 4 for the FR and Table 5 for the DE datasets under column “AHP point.”)Table 4
Statistics for combined uncertainty results in France data with AHP RI weights point results (with triangle distribution, graphically presented in Figure3).
EG ID
Q1
Median
AHP point
Mean
Q3
FR02.3
11%
19%
20%
19%
22%
FR03
11%
20%
23%
22%
25%
FR04.1
16%
28%
29%
28%
32%
FR05.1
21%
30%
32%
32%
39%
FR05.2
5%
17%
21%
19%
24%
FR07.1
35%
42%
42%
44%
51%
FR08.3
24%
34%
35%
35%
42%
FR09.1
14%
26%
26%
26%
32%
FR09.3
19%
30%
31%
29%
33%
FR12
35%
43%
45%
44%
50%
FR13.4
68%
76%
76%
73%
83%
FR15
13%
26%
30%
28%
33%
FR16.1
30%
38%
39%
41%
47%
FR16.2
22%
37%
38%
40%
47%
FR16.3
45%
53%
52%
54%
61%
FR17.1
28%
35%
36%
36%
37%
FR17.2
14%
20%
21%
20%
22%
FR17.3
20%
27%
27%
27%
29%
FR18.1
17%
28%
29%
28%
35%
FR21
20%
28%
29%
28%
34%
EG: event group (description is available in Table3). Q1 and Q3 are Quartiles 1 and 3. AHP point versus Median of uncertainty results absolute difference is in the range from −1 to 4%.Table 5
Statistics for combined uncertainty results for German data with AHP RI weights point results (with triangle distribution, graphically presented in Figure4).
EG ID
Q1
Median
AHP point
Mean
Q3
DE01.2
18%
27%
29%
27%
34%
DE03
14%
20%
22%
20%
23%
DE04.2
14%
25%
28%
26%
31%
DE04.3
9%
28%
32%
29%
39%
DE05.2
24%
25%
25%
24%
28%
DE07.1
50%
59%
62%
58%
64%
DE08.2
16%
25%
27%
26%
33%
DE09.1
26%
36%
36%
34%
39%
DE10.2
8%
25%
29%
26%
34%
DE13.2
42%
47%
45%
48%
60%
DE13.3
44%
49%
48%
54%
66%
DE14
84%
89%
90%
85%
93%
DE17.4
13%
27%
27%
26%
33%
DE19.3
16%
21%
23%
21%
25%
EG: event group (description is available in Table3). Q1 and Q2 are Quartiles 1 and 3. AHP point versus Median of uncertainty results absolute difference is in the range from −2 to 5%.


### 2.2. Approach to Uncertainty Assessment

Most generally, uncertainty consists of the aleatory and epistemic part, which includes parameter and model uncertainties [6]. The ranking uncertainty, because of the model, was considered separately and this analysis applies to the AHP based ranking method, [3]. Uncertainty of dataset values is considered outside this assessment because it is epistemic and not significant (in comparison to the ranking uncertainties). Therefore, we believe that data source uncertainty is less critical for the ranking. Clearly this assumption depends on the completeness of the data collection program. Considering the central importance of RI weights and expert judgement impact, they are selected as primary uncertainty input to the assessment. The upper described ranking approach is far from the complexity of some deterministic models (e.g., [7]) and the parameters uncertainty consideration is less challenging.The sampling-based approach is selected because of its effectiveness and wide use. The basic idea is that analysis resultsy
(
x
)
=
[
y
1
(
x
)
,
y
2
(
x
)
,
…
,
y
n
(
x
)
] are a function of uncertain inputs x
=
[
x
1
,
x
2
,
…
,
x
n
], [8]. This allows quantification of results uncertainty and determination of input parameters influence.In this uncertainty study, event groups ranking is quantified as a statistical distribution resulting from the ranking indexes weights defined with statistical distributions instead as point value. This will allow examining event groups ranking sensitivity and defining need for further assessment (i.e., better uncertainty method regarding input statistical distribution, etc.).A next critical step for the uncertainty analysis is the determination of distributions for the parameters sampling. Without sufficient base for direct determination of specific distributions from the nature of input parameters, they are selected based on the accepted referenced approach and suitability for this stage of ranking uncertainty study. Two different distributions could be found related to the uncertainty where AHP is applied: triangular [9] and uniform [10]. The triangular distribution is used to represent cases when we know the most likely outcome of sampled RI weights. In this particular case, the most likely outcome is calculated with the AHP process (the calculated RI weights). The second common case is that we do not know enough about sampled RI weights (only smallest and largest values), in which case the uniform distribution can be used with smallest and largest values set to 0 and 1, respectively. Referenced works use distributions applied to the process of AHP weights determination, that is, to the pairwise comparison. For our purpose, we decided to apply both distributions to the already quantified AHP weight. In order to sample all RI weights in a way they always make a sum of 1 (100%) sampling was done for one RI weight in separate simulation with corrections to other RI weights for each sampling and relative to the AHP value. This means, for example, if i RI, with w
i weight, is sampled and new sampled weight is w
s
i
, then correction (i.e., multiplier) for other RI weights is c defined by (3). Then, new weight value for RI j is w
n
j
=
c
·
w
j, where w
j is AHP determined original weight. Consider
(3)
c
=
(
1
-
w
i
)
(
1
-
w
s
i
)
,
where w
s
i is sampled RI weight; w
i is AHP derived (starting) weight; and c is correction (i.e., multiplier) for other RI weights.For the triangular distribution, separate simulation was performed for each RI with minimal sampling values set to 0 and maximum set to 1 keeping the original AHP value as the most likely. Figure1 presents the distribution for the “frequency” RI in German data where AHP weight is equal to 0.08 (Table 2). A resulting dependent distribution for other RIs has also been presented.Sampling distribution of inputs: “Frequency” RI weight and other RI weights distribution (created using (3)) for German data: on the left with 10 and on the right with 100 thousand samples.
(a)
(b)In the case of uniform distribution of the input variable after several iterations ±33% change from the original AHP weights is selected for further analysis. Higher value would be in fact sensitivity analysis and smaller value would not influence the results enough.For this exercise, ranking values were quantified, based on the AHP determined RI weights using Microsoft Excel with add-on Monte Carlo simulation and statistical tool Quantum XL [11]. The number of samplings was determined based on the comparison of sampling distributions of input and output variables for 10 and 100 thousand samples. Figure 1 presents the sampling distribution for one input variable (with the others accordingly adjusted), and Figure 2 presents resulting distributions of four highest ranked event groups from the same simulations. Because a significant statistical difference was not found, further analysis continued only with 10 thousand samplings per simulation.Sampling distribution of results for four highest ranked event groups for German data using input from the “Frequency” RI weight triangle distribution sampling: on the left with 10 and on the right with 100 thousand samples.
(a)
(b)Finally, after all simulations for all RIs are performed, cumulative ranking values distributions are presented with statistical analysis and box plot [8]. This was done in order to simplify the presentation of results and to make clear conclusions about uncertainty. A separate detailed analysis of results for each RI weight simulation is possible in the same way if found important.


## 2.1. Ranking Method and Inputs for Uncertainty Assessment

Ranking is performed at the level of event groups (EG) using selected sets of parameters. For the brevity, only about half of the more important groups are selected and presented in the paper. Table1 presents respected parameters values for selected event groups (Table 3 lists respected description for each EG). In total, 20 event groups are analyzed for French (FR) and 14 for German (DE) datasets. Because of technology and data collection differences, some event groups do not exist in both datasets and there are also differences in parameters. Both datasets have frequency, trend, and precursors parameters. The French dataset has special parameters which present a number of so-called “generic and recurring” events. There is also a difference in the number of events reported to the IAEA/NEA (International Atomic Energy Agency/Nuclear Energy Agency) international reporting system (IRS) because in France it is chosen to report only one representative event in case similar events occur in several plants of the French fleet. The trend value is presenting change of the number of events for a certain period (i.e., 0.5 means that the number of events is not changing, values up to 1 mean increasing trend, and values down to 0 represent a decreasing trend). The trend has been considered, in this analysis, only for the last five years the most relevant. All other parameters are for a whole period of 20 years.Table 1
Selected event groups for ranking uncertainty assessment with respected number of events and trend value.
ID
FR
DE
N
G & R
IRS
Pr.
T5y
U
N
IRS
Pr.
T5y
U
01.2
161
12
5
11
0.98
N
102
0
3
0.33
Y
02.3
271
23
2
15
0.35
Y
6
0
0
0.50
N
03
44
10
7
8
0.20
Y
33
1
1
0.50
Y
04.1
89
89
16
10
0.18
Y
44
1
0
0.76
N
04.2
20
16
4
0
0.83
N
6
1
0
0.23
Y
04.3
70
70
2
5
0.52
N
38
0
0
0.01
Y
05.1
64
35
43
21
0.42
Y
1
0
0
0.50
N
05.2
48
1
1
6
0.18
Y
149
0
8
0.75
Y
07.1
1415
149
43
97
0.97
Y
168
3
19
0.54
Y
08.2
18
3
3
5
0.50
N
104
0
2
0.32
Y
08.3
320
52
31
24
0.25
Y
—
—
—
—
—
09.1
105
2
16
15
0.21
Y
167
1
13
0.81
Y
09.3
130
12
16
35
0.20
Y
8
0
0
0.50
N
10.2
5
0
0
0
0.53
N
33
0
0
0.12
Y
12
236
105
48
65
0.40
Y
4
0
1
0.50
N
13.2
—
—
—
—
—
—
351
5
11
0.93
Y
13.3
—
—
—
—
—
—
458
6
10
0.94
Y
13.4
1054
114
46
190
0.17
Y
—
—
—
—
—
14
470
3
12
48
0.97
N
172
5
22
0.04
Y
15
111
67
6
15
0.00
Y
42
1
2
0.76
N
16.1
1194
803
14
39
0.87
Y
69
3
2
0.97
N
16.2
1681
0
5
34
0.00
Y
68
1
1
0.89
N
16.3
1726
79
28
61
0.02
Y
102
0
1
0.60
N
17.1
393
270
13
22
0.16
Y
—
—
—
—
—
17.2
219
24
9
20
0.47
Y
—
—
—
—
—
17.3
162
162
14
5
0.26
Y
—
—
—
—
—
17.4
—
—
—
—
—
—
119
0
0
0.21
Y
18.1
108
66
22
15
0.25
Y
23
0
1
0.93
N
19.3
35
0
2
6
0.50
N
12
1
3
0.55
Y
21
78
48
25
30
0.40
Y
82
3
0
0.93
N
FR: French dataset; DE: German dataset;  ID: number of the event groups (described in Table3); N: number of events; G & R: subset of events in the FR database comprising generic and recurring events; IRS: subset of events reported to the IRS (for FR, only the first-time event for each type is reported); Pr.: precursor events; T5y: trend parameter value (i.e., 0.5 for unchanged number of events, up to 1 for increasing trend, and down to 0 for decreasing trend); U: event groups considered (“Y,” or not “N”) in the presented uncertainty assessment.The ranking index value is calculated for each parameter and event group as a ratio of the number of events for that parameter in the group and the maximal number of events for the same parameter considering all the event groups:(1)
R
I
EG
.
parameter
=
N
EG
.
parameter
N
EG
.
param
.

max
⁡
,
where RI is ranking index, EG is event group, and N is number of events.For the trend, RI value is calculated to present an unchanged number of events with 0.5, increasing number of events (parameter value up to 1), and decreasing number of events (values down to 0) over a certain period.The final total ranking value for each EG is calculated as the sum of all parameters ranking values multiplied by respect weights:(2)
R
I
EG
=
∑
parameters
w
parameter
R
I
EG
.
parameter
,
where RI is ranking index, EG is event group, and w is weights for respected parameter.These weights could be judged by experts directly or more transparently and consistently by using some methods like the analytical hierarchy process (AHP, [3, 5]) where the only requirement is to make pairwise comparisons for all parameters. Table 2 presents a set of RI weights used in this assessment which was derived using AHP results as the base. (Short description of the method is provided in the appendix.)Table 2
Ranking indexes weights for each dataset.
Country ranking index
Frequency
Generic and recurring
IRS
Precursor
Trend
France (FR)
0.07
0.19
0.15
0.36
0.23
Germany (DE)
0.08
NA
0.20
0.40
0.32
IRS: events reported to the IAEA/NEA international reporting system.Table 3
ID and description for selected event groups with presence in the respected country database.
ID
Description
FR
DE
01.2
Loss of offsite power-loss of electrical bus bar
N
Y
02.3
Reactivity events-transient pressure or temperature changes
Y
N
03
Loss of heat removal system
Y
Y
04.1
Leakages from the primary circuit
Y
N
04.2
Leakages from primary to secondary circuit
N
Y
04.3
Primary coolant leakage outside cont.
Y
Y
05.1
Essential water intake clogging
Y
N
05.2
Failures in the component cooling or service water systems operating trains
Y
Y
07.1
Scram with influence on reactivity
Y
Y
08.2
Other instrumentation and control failures affecting the reactor protection system
N
Y
08.3
Instrumentation and control failures related to other safety systems
Y
—
09.1
Complete emergency diesel generator failure
Y
Y
09.3
Incipient emergency diesel generator failure
Y
N
10.2
Control rod events without influence on reactivity
N
Y
12
Common cause failure events
Y
N
13.2
Failure of 1 safety system redundancy
—
Y
13.3
Failure of a safety system component
—
Y
13.4
Failure of safety features redundancies
Y
—
14
Actuation of safety features
Y
Y
15
Lubricant
Y
N
16.1
Circuit misalignment (human/organization)
Y
N
16.2
Risk analysis (human/organization)
Y
N
16.3
Inadequate maintenance quality (human/organization)
Y
N
17.1
Failure after release from maintenance or design modification
Y
—
17.2
Failure caused by plant design modification
Y
—
17.3
Failure caused by using temporary devices
Y
—
17.4
Failure related to NPP modification
—
Y
18.1
Event external to the site
Y
N
19.3
Other reportable internal fires
N
Y
21
Design/construction
Y
N
Y or N designates existence of the particular event group in the FR and DE country database.Applying (1) and (2) to all EG using data presented in Table 1 and weights in Table 2, total ranking values are quantified. Most important event groups are then considered once which have the highest ranking values. (Selected results are presented in the Table 4 for the FR and Table 5 for the DE datasets under column “AHP point.”)Table 4
Statistics for combined uncertainty results in France data with AHP RI weights point results (with triangle distribution, graphically presented in Figure3).
EG ID
Q1
Median
AHP point
Mean
Q3
FR02.3
11%
19%
20%
19%
22%
FR03
11%
20%
23%
22%
25%
FR04.1
16%
28%
29%
28%
32%
FR05.1
21%
30%
32%
32%
39%
FR05.2
5%
17%
21%
19%
24%
FR07.1
35%
42%
42%
44%
51%
FR08.3
24%
34%
35%
35%
42%
FR09.1
14%
26%
26%
26%
32%
FR09.3
19%
30%
31%
29%
33%
FR12
35%
43%
45%
44%
50%
FR13.4
68%
76%
76%
73%
83%
FR15
13%
26%
30%
28%
33%
FR16.1
30%
38%
39%
41%
47%
FR16.2
22%
37%
38%
40%
47%
FR16.3
45%
53%
52%
54%
61%
FR17.1
28%
35%
36%
36%
37%
FR17.2
14%
20%
21%
20%
22%
FR17.3
20%
27%
27%
27%
29%
FR18.1
17%
28%
29%
28%
35%
FR21
20%
28%
29%
28%
34%
EG: event group (description is available in Table3). Q1 and Q3 are Quartiles 1 and 3. AHP point versus Median of uncertainty results absolute difference is in the range from −1 to 4%.Table 5
Statistics for combined uncertainty results for German data with AHP RI weights point results (with triangle distribution, graphically presented in Figure4).
EG ID
Q1
Median
AHP point
Mean
Q3
DE01.2
18%
27%
29%
27%
34%
DE03
14%
20%
22%
20%
23%
DE04.2
14%
25%
28%
26%
31%
DE04.3
9%
28%
32%
29%
39%
DE05.2
24%
25%
25%
24%
28%
DE07.1
50%
59%
62%
58%
64%
DE08.2
16%
25%
27%
26%
33%
DE09.1
26%
36%
36%
34%
39%
DE10.2
8%
25%
29%
26%
34%
DE13.2
42%
47%
45%
48%
60%
DE13.3
44%
49%
48%
54%
66%
DE14
84%
89%
90%
85%
93%
DE17.4
13%
27%
27%
26%
33%
DE19.3
16%
21%
23%
21%
25%
EG: event group (description is available in Table3). Q1 and Q2 are Quartiles 1 and 3. AHP point versus Median of uncertainty results absolute difference is in the range from −2 to 5%.


## 2.2. Approach to Uncertainty Assessment

Most generally, uncertainty consists of the aleatory and epistemic part, which includes parameter and model uncertainties [6]. The ranking uncertainty, because of the model, was considered separately and this analysis applies to the AHP based ranking method, [3]. Uncertainty of dataset values is considered outside this assessment because it is epistemic and not significant (in comparison to the ranking uncertainties). Therefore, we believe that data source uncertainty is less critical for the ranking. Clearly this assumption depends on the completeness of the data collection program. Considering the central importance of RI weights and expert judgement impact, they are selected as primary uncertainty input to the assessment. The upper described ranking approach is far from the complexity of some deterministic models (e.g., [7]) and the parameters uncertainty consideration is less challenging.The sampling-based approach is selected because of its effectiveness and wide use. The basic idea is that analysis resultsy
(
x
)
=
[
y
1
(
x
)
,
y
2
(
x
)
,
…
,
y
n
(
x
)
] are a function of uncertain inputs x
=
[
x
1
,
x
2
,
…
,
x
n
], [8]. This allows quantification of results uncertainty and determination of input parameters influence.In this uncertainty study, event groups ranking is quantified as a statistical distribution resulting from the ranking indexes weights defined with statistical distributions instead as point value. This will allow examining event groups ranking sensitivity and defining need for further assessment (i.e., better uncertainty method regarding input statistical distribution, etc.).A next critical step for the uncertainty analysis is the determination of distributions for the parameters sampling. Without sufficient base for direct determination of specific distributions from the nature of input parameters, they are selected based on the accepted referenced approach and suitability for this stage of ranking uncertainty study. Two different distributions could be found related to the uncertainty where AHP is applied: triangular [9] and uniform [10]. The triangular distribution is used to represent cases when we know the most likely outcome of sampled RI weights. In this particular case, the most likely outcome is calculated with the AHP process (the calculated RI weights). The second common case is that we do not know enough about sampled RI weights (only smallest and largest values), in which case the uniform distribution can be used with smallest and largest values set to 0 and 1, respectively. Referenced works use distributions applied to the process of AHP weights determination, that is, to the pairwise comparison. For our purpose, we decided to apply both distributions to the already quantified AHP weight. In order to sample all RI weights in a way they always make a sum of 1 (100%) sampling was done for one RI weight in separate simulation with corrections to other RI weights for each sampling and relative to the AHP value. This means, for example, if i RI, with w
i weight, is sampled and new sampled weight is w
s
i
, then correction (i.e., multiplier) for other RI weights is c defined by (3). Then, new weight value for RI j is w
n
j
=
c
·
w
j, where w
j is AHP determined original weight. Consider
(3)
c
=
(
1
-
w
i
)
(
1
-
w
s
i
)
,
where w
s
i is sampled RI weight; w
i is AHP derived (starting) weight; and c is correction (i.e., multiplier) for other RI weights.For the triangular distribution, separate simulation was performed for each RI with minimal sampling values set to 0 and maximum set to 1 keeping the original AHP value as the most likely. Figure1 presents the distribution for the “frequency” RI in German data where AHP weight is equal to 0.08 (Table 2). A resulting dependent distribution for other RIs has also been presented.Sampling distribution of inputs: “Frequency” RI weight and other RI weights distribution (created using (3)) for German data: on the left with 10 and on the right with 100 thousand samples.
(a)
(b)In the case of uniform distribution of the input variable after several iterations ±33% change from the original AHP weights is selected for further analysis. Higher value would be in fact sensitivity analysis and smaller value would not influence the results enough.For this exercise, ranking values were quantified, based on the AHP determined RI weights using Microsoft Excel with add-on Monte Carlo simulation and statistical tool Quantum XL [11]. The number of samplings was determined based on the comparison of sampling distributions of input and output variables for 10 and 100 thousand samples. Figure 1 presents the sampling distribution for one input variable (with the others accordingly adjusted), and Figure 2 presents resulting distributions of four highest ranked event groups from the same simulations. Because a significant statistical difference was not found, further analysis continued only with 10 thousand samplings per simulation.Sampling distribution of results for four highest ranked event groups for German data using input from the “Frequency” RI weight triangle distribution sampling: on the left with 10 and on the right with 100 thousand samples.
(a)
(b)Finally, after all simulations for all RIs are performed, cumulative ranking values distributions are presented with statistical analysis and box plot [8]. This was done in order to simplify the presentation of results and to make clear conclusions about uncertainty. A separate detailed analysis of results for each RI weight simulation is possible in the same way if found important.


## 3. Results and Discussion

Based on the approach described in the previous section for each country dataset, one simulation was performed for every RI (i.e., five for France and four for Germany datasets). The following subsections present the most important results and findings.


### 3.1. Simulation Results

Results from each simulation are generating ranking values distribution for all selected event groups. This is a vast volume of results and for brevity graphical results are presented only for highest ranked event groups. Most of the results are presented for simulations with triangle distribution. For France datasets, Figure5 shows separately for each RI simulation all five RI weights distributions, and Figure 6 presents respected resulting distribution of the ranking values of the eight most important event groups. Similarly for Germany Figure 7 presents all RI weights distributions and Figure 8 presents respected resulting distributions of the ranking values of the four most important event groups. The number and specific event groups are selected according to the country overall results.Final results for the distribution of all ranking values for selected event groups are presented with a box plot in Figure3 for France and Figure 4 for Germany. Values for Quartile 1 (Q1), Mean, Median, and Quartile 3 (Q3) are presented, together with baseline AHP RI weights ranking results in Table 4 for France and Table 5 for Germany.Figure 3
Box plot of the French event groups ranking cumulative uncertainty results with triangular distribution for all ranking indexes weights inputs. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range). Q1 and Q3 (represented by the boxes) with Mean and Median values are also presented in Table4.Figure 4
Box plot of the German event groups ranking uncertainty results with triangular distribution for all ranking indexes weights inputs. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range). Q1 and Q3 (represented by the boxes) with Mean and Median values are also presented in Table5.French input data sampling distribution for all RIs weights (i.e., sampled and corrected using (3)): figures from (a) to (e) for sampling “Frequency,” “G&R,” “IRS,” “Precursor” and “Trend,” RI, respectively (max number of samples for peak RI values is in the range from 800 to 2000). Selected ranking results are presented in Figure 6.
(a)
(b)
(c)
(d)
(e)French data results distribution for eight highest ranked event groups with all RIs weights triangle distribution inputs separate sampling: figures from (a) to (e) for sampling “Frequency,” “G&R,” “IRS,”,” “Precursor” and “Trend,” RI, respectively (max number of samples for peak RI values is from 1400 to around 2200 and in “G&R” case almost 6000). Respected inputs samplings are presented in Figure5.
(a)
(b)
(c)
(d)
(e)German input data sampling distribution for all RIs weights (i.e., sampled and corrected using (3)): figures from (a) to (d) for sampling “Frequency,” “IRS,”,” “Precursor” and “Trend,” RI, respectively (max number of samples for peak RI values is from 800 to 2000). Selected ranking results are presented in Figure 8.
(a)
(b)
(c)
(d)German data results distribution for four highest ranked event groups with all RIs weights triangle distribution sampling: figures from (a) to (d) for sampling “Frequency,” “IRS,”,” “Precursor” and “Trend,” RI, respectively (max number of samples for peak RI values is from 1400 to over 9000). Respected inputs samplings are presented in Figure7.
(a)
(b)
(c)
(d)For uniform distributions, simulations with ±33% sampling of RI weights produce a much smaller influence to the final ranking values for all event groups. Because of brevity only final box plot graphs are presented for each country dataset in Figures9 and 10.Figure 9
Box plot of the French event groups ranking cumulative uncertainty results with uniform ±33% distribution of all ranking indexes weights. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range).Figure 10
Box plot of the German event groups ranking uncertainty results with uniform ±33% distribution of all ranking indexes weights. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range).


### 3.2. Discussion

Resulting combined ranking values uncertainty for both countries shows numerically and graphically how ranking is affected by RI weights uncertainty. For both countries, an overlap (by looking at values between Q1 and Q3) is smaller for higher ranked event groups. For France data the two highest ranked event groups (13.4 and 16.3) are clearly separated and then after that the following five (7.1, 8.3, 12, 16.1, and 16.2). For Germany data, event group 14 is clearly the highest ranked and the following three event groups (7.1, 13.2, and 13.3) are distinctly higher ranked than the rest of the event groups.As a form of simulation results confirmation, it is valuable to point out that the difference between Median, Mean, and baseline AHP RI weights ranking results is very small. The difference for France data is between −1% and 4% and between −2% and 5% for German data (Tables4 and 5).By looking at the resulting ranking values distributions, it is clear that the difference between Q1 and Q3 varies between event groups. The same is true for the spread symmetry (regarding the size and side).In the case of uniform ±33% distribution of the RI weights, Mean and Median results are the same and equal to the baseline AHP RI weights ranking results. Q1 and Q3 values are very close, with the maximal difference ~3% of absolute value which is less than 10% of relative value even for the low ranked event groups. Compared to the triangle distribution case, outliers and suspected outliers are almost nonexistent. These results do not show any significant change and they clearly do not have value compared to the triangular simulations.The presented results, from triangular simulations, seem mostly useful to prove the robustness of ranking results obtained by only using AHP point estimate for RI weights. Based on the two countries datasets, it also seems that uncertainty results are helping find better separation between more important event groups and the rest of them. Finally, for two closely ranked event groups, it seems easier to select more important by using uncertainty results (e.g., FR5.1 versus FR8.3 and DE1.2 versus DE4.2).A comparison between two countries datasets has a lot of limitations due to the various differences (i.e., technology, regulation, safety culture, etc.). This does not prevent making observations about uncertainty results. It seems that uncertainty results are consistent between these two datasets with all mentioned differences regarding the background.


## 3.1. Simulation Results

Results from each simulation are generating ranking values distribution for all selected event groups. This is a vast volume of results and for brevity graphical results are presented only for highest ranked event groups. Most of the results are presented for simulations with triangle distribution. For France datasets, Figure5 shows separately for each RI simulation all five RI weights distributions, and Figure 6 presents respected resulting distribution of the ranking values of the eight most important event groups. Similarly for Germany Figure 7 presents all RI weights distributions and Figure 8 presents respected resulting distributions of the ranking values of the four most important event groups. The number and specific event groups are selected according to the country overall results.Final results for the distribution of all ranking values for selected event groups are presented with a box plot in Figure3 for France and Figure 4 for Germany. Values for Quartile 1 (Q1), Mean, Median, and Quartile 3 (Q3) are presented, together with baseline AHP RI weights ranking results in Table 4 for France and Table 5 for Germany.Figure 3
Box plot of the French event groups ranking cumulative uncertainty results with triangular distribution for all ranking indexes weights inputs. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range). Q1 and Q3 (represented by the boxes) with Mean and Median values are also presented in Table4.Figure 4
Box plot of the German event groups ranking uncertainty results with triangular distribution for all ranking indexes weights inputs. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range). Q1 and Q3 (represented by the boxes) with Mean and Median values are also presented in Table5.French input data sampling distribution for all RIs weights (i.e., sampled and corrected using (3)): figures from (a) to (e) for sampling “Frequency,” “G&R,” “IRS,” “Precursor” and “Trend,” RI, respectively (max number of samples for peak RI values is in the range from 800 to 2000). Selected ranking results are presented in Figure 6.
(a)
(b)
(c)
(d)
(e)French data results distribution for eight highest ranked event groups with all RIs weights triangle distribution inputs separate sampling: figures from (a) to (e) for sampling “Frequency,” “G&R,” “IRS,”,” “Precursor” and “Trend,” RI, respectively (max number of samples for peak RI values is from 1400 to around 2200 and in “G&R” case almost 6000). Respected inputs samplings are presented in Figure5.
(a)
(b)
(c)
(d)
(e)German input data sampling distribution for all RIs weights (i.e., sampled and corrected using (3)): figures from (a) to (d) for sampling “Frequency,” “IRS,”,” “Precursor” and “Trend,” RI, respectively (max number of samples for peak RI values is from 800 to 2000). Selected ranking results are presented in Figure 8.
(a)
(b)
(c)
(d)German data results distribution for four highest ranked event groups with all RIs weights triangle distribution sampling: figures from (a) to (d) for sampling “Frequency,” “IRS,”,” “Precursor” and “Trend,” RI, respectively (max number of samples for peak RI values is from 1400 to over 9000). Respected inputs samplings are presented in Figure7.
(a)
(b)
(c)
(d)For uniform distributions, simulations with ±33% sampling of RI weights produce a much smaller influence to the final ranking values for all event groups. Because of brevity only final box plot graphs are presented for each country dataset in Figures9 and 10.Figure 9
Box plot of the French event groups ranking cumulative uncertainty results with uniform ±33% distribution of all ranking indexes weights. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range).Figure 10
Box plot of the German event groups ranking uncertainty results with uniform ±33% distribution of all ranking indexes weights. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range).


## 3.2. Discussion

Resulting combined ranking values uncertainty for both countries shows numerically and graphically how ranking is affected by RI weights uncertainty. For both countries, an overlap (by looking at values between Q1 and Q3) is smaller for higher ranked event groups. For France data the two highest ranked event groups (13.4 and 16.3) are clearly separated and then after that the following five (7.1, 8.3, 12, 16.1, and 16.2). For Germany data, event group 14 is clearly the highest ranked and the following three event groups (7.1, 13.2, and 13.3) are distinctly higher ranked than the rest of the event groups.As a form of simulation results confirmation, it is valuable to point out that the difference between Median, Mean, and baseline AHP RI weights ranking results is very small. The difference for France data is between −1% and 4% and between −2% and 5% for German data (Tables4 and 5).By looking at the resulting ranking values distributions, it is clear that the difference between Q1 and Q3 varies between event groups. The same is true for the spread symmetry (regarding the size and side).In the case of uniform ±33% distribution of the RI weights, Mean and Median results are the same and equal to the baseline AHP RI weights ranking results. Q1 and Q3 values are very close, with the maximal difference ~3% of absolute value which is less than 10% of relative value even for the low ranked event groups. Compared to the triangle distribution case, outliers and suspected outliers are almost nonexistent. These results do not show any significant change and they clearly do not have value compared to the triangular simulations.The presented results, from triangular simulations, seem mostly useful to prove the robustness of ranking results obtained by only using AHP point estimate for RI weights. Based on the two countries datasets, it also seems that uncertainty results are helping find better separation between more important event groups and the rest of them. Finally, for two closely ranked event groups, it seems easier to select more important by using uncertainty results (e.g., FR5.1 versus FR8.3 and DE1.2 versus DE4.2).A comparison between two countries datasets has a lot of limitations due to the various differences (i.e., technology, regulation, safety culture, etc.). This does not prevent making observations about uncertainty results. It seems that uncertainty results are consistent between these two datasets with all mentioned differences regarding the background.


## 4. Conclusion

Uncertainty assessment is important for enhancing the model results interpretation and usability. Critical parameters uncertainty influencing the model results can be assessed by a sampling-based approach. AHP determined weights for ranking indexes are selected as the most important epistemic source of uncertainty in the event groups ranking method. Triangular distribution with a most likely value determined by the AHP and limits between 0 and 1 is selected as a sampling input. Dependencies between all ranking indexes were treated with simple corrections and separate simulation for each ranking index. Combined distributions for all event groups ranking values are generated from all simulations. The uncertainty assessment approach applied to two countries datasets proves both consistent and valuable results.An alternative uniform sampling distribution proved to be without sufficient merit to be considered for use.Event groups ranking with analytical hierarchy based ranking indexes weights could be improved with sampling-based uncertainty assessment because it confirms most of the results and provides help for better distinguishing closely ranked event groups.



---

*Source: 324710-2014-09-18.xml*