# Effect of the shape of ROC curves on Risk Based
Inspection: a parametric study

Franck Schoefs, Jérôme Boéro

► To cite this version:

Franck Schoefs, Jérôme Boéro. Effect of the shape of ROC curves on Risk Based Inspection: a parametric study. ICOSSAR'09, 2009, Osaka, Japan. hal-01008102

HAL Id: hal-01008102

https://hal.science/hal-01008102

Submitted on 1 Oct 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Effect of the shape of ROC curves on Risk Based Inspection: a parametric study

F. Schoefs

GeM, Nantes Atlantic University, Nantes, France

J. Boéro

OXAND S.A., Avon, France

**ABSTRACT:** Owners of civil infrastructures base their maintenance decisions schemes mainly both on structural integrity assessment and consequence analysis. Some inputs come from information collected by inspections by non-destructive or destructive tools. Uncertainties and error of measurements can lead to bad decisions but are rarely integrated into the process of decision. Nowadays, Risk Based Inspection (RBI) provides the basic concepts for optimizing the maintenance of existing structures. It lies both on reliability computations and probabilistic modelling of inspection results. In a first part, we focus on the benefit of the combination of multiple Non-Destructive Testing tools (NDT) and the role of expert judgement in this process. RBI of steel harbour structures is used for illustration. Second part is dedied to highlight the effect of the shape of ROC curve on the decision process by means of a parametric study.

## 1 INTRODUCTION

Reassessment of existing structures generates a need for up-dated materials properties. In a lot of cases, on-site inspections are needed and in some cases visual inspection is not sufficient. For example Non Destructive Testing (NDT) tools are required for the inspection of coastal and marine structures where marine growth acts as a mask or immersion area gives harsh conditions of inspection. In these fields, the cost of inspection can be prohibitive and an accurate description of the on-site performance of NDT tools must be provided. Inspection of existing structures by a NDT tool is not perfect and it has become a common practice to model their reliability in terms of probability of detection (PoD), probability of false alarms (PFA) and Receiver Operating Characteristic (ROC) curves (Rouhan & Schoefs 2003, Straub & Faber 2003, Pakrashi et al. 2008). These results quantities are generally the main inputs needed by owners of structures in view to achieve Inspection, Maintenance and Repair plans (IMR) (Sheils et al., 2008). The assessment of PoD and PFA is even deduced from inter-calibration of NDT tools or from the modelling of the noise and the signal (Barnouin et al. 1993, Rudlin 1996, Rudlin & Dover 1996).

Theoretical aspects coming from detection theory and probabilistic modelling of inspections results in view to provide inputs in the computation of mathematical expectation of RBI cost models are described in Schoefs et al. (2008b).

First, let us focus in this paper on the benefit of the combination of multiple Non-Destructive Testing (NDT) and the role of expert judgement in this process. In this case, expert judgement acts at two levels:

- The knowledge of ageing laws to provide the probability of defect existence that is needed when computing likelihood of events that govern the cost expectation;

- The way to address the decision after obtaining results from the two inspections.

The methodology is illustrated for the RBI of steel harbour structures.

Then, the effect of the shape of ROC curve on the decision process is highlighted. To this aim, a parametric study is performed to analyze the influence of the polar coordinates of the best performance point of the NDT tool (NDT-BPP) on the expectation of the cost of detection and the cost of no detection. NDT-BPP is defined as the closest point of the ROC curve to the best performance point BBP of coordinates (PFA=0, PoD=1). The paper introduces the polar coordinates of NDT-BPP for characterizing ROC curve. That allows to perform parametric studies based on these two parameters.
## 2 COMBINATION OF MULTIPLE INSPECTIONS: EXEMPLE OF APPLICATION FOR THE RBI OF STEEL HARBOUR STRUCTURES

### 2.1 Context

The objective is here to analyse the combination of the same technique of NDT tools and the role of expert judgement on the minimization of the cost expectation and on the ranking decisions. In the combination, we consider a memory-less inspection: the inspection is carried out by two similar operators that don't remember the previous result at the same location. Then inspections are considered as independent.

The methodology is illustrated for the RBI of steel marine structures submitted to uniform corrosion (steel sheet pile seawalls). In this example, we consider only results provided by ultrasonic residual thickness measurement that is the common technique for identifying zones on steel marine structures which are the most affected by corrosion.

ROC curves obtained for three wharfs by three couples (diver-inspection society) for the same level of corrosion are represented on Figure 1.

<figure><img src="image_2.png" /><figcaption>Figure 1. R.O.C curves used for illustration.</figcaption></figure>

In this study, performance of NDT tool is only represented by the minimal distance $\delta_{NDT}$ between the optimal point of ROC, with cartesian coordinates (0, 1) in PFA, PoD plane and ROC curve number i. The ROC3 curve represents the best performance of the three different NDT tool results and the ROC2 curve the worst. Measurements performed in the immersion area for which inspection's conditions are harsh (Schoefs et al., 2008a).

### 2.2 Combination of identical techniques

Objective is here to combine characteristics of ROC obtained from different couples (diver-inspection society).

The ROC curves obtained by intersection and union methods of NDT tool results 2 and NDT tool results 3 are represented on Figure 2 (Yang & Donath, 1984).

<figure><img src="image_3.png" /><figcaption>Figure 2. R.O.C curves resulting from intersection and union methods in case of different NDT tool results (2) and (3).</figcaption></figure>

In view to perform a cost analysis, points of ROC curves can be considered for the computation of $P_{i[i \in \{1:4\}]}$ where 'i' characterizes the scenario in the decision tree (Rouhan & Schoefs 2003). If the detection threshold is unknown, the performance of the technique is given by the whole ROC curve. Then, computation of $P_{i[i \in \{1:4\}]}$ leads to project ROC curves on $P(E_i)$ surfaces (Schoefs et al., 2008b).

Then $P_i$ are replaced in cost analysis by their geometric mean value. Geometric mean values $m(P_2)|_{ROC}$ and $m(P_3)|_{ROC}$ of NDT tool results 2 and NDT tool results 3 are represented on Figures 3-4.

<figure><img src="image_4.png" /><figcaption>Figure 3. Results of techniques combination for $m(P_2)$.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Figure 4. Results of techniques combination for m(P<sub>3</sub>).</figcaption></figure>

This consequence analysis allows introducing interest of repeating the same or other technique (Schoefs & Clément 2004). On Figure 2, we notice that union method leads to better performances than intersection method. Note that data fusion can be used in this case. As we fusion similar data with known probability the probabilistic method is selected here.

This aspect shows the influence of expert's judgment on the choice of the combination method.

Let us now analyze the role probability of defects presence γ evaluated from expert's judgement or ageing laws. It is studied though the sensitivity of m(P<sub>□</sub>)<sub>ROC</sub> and m(P<sub>□</sub>)<sub>ROC</sub> to this parameter respectively on Fig. 8 and 10 for three likelihoods: low (γ=0.1), medium (γ=0.5) and large (γ=0.9).

## 2.3 Sensibility of expert's judgement on cost analysis

In the following, we focus on the two choices that affect the costs in our methodology both in case of detection E(C)<sub>d</sub> and in case of no detection E(C)<sub>nd</sub>:

- The probability of defects presence γ;

- The decision after two inspections (union or intersection).

Cost model selected for the application in table 1 gathers common values considered for marine structures (Rouhan et al., 2003).

<table><caption>Table 1. Cost model selected for failure, repair, inspection and inspection combination.</caption><thead><tr><td></td><td>Cost</td></tr></thead><tbody><tr><td>Failure</td><td>1.000</td></tr><tr><td>Repair</td><td>0.010</td></tr><tr><td>Inspection</td><td>0.001</td></tr><tr><td>Inspection combination</td><td>0.002</td></tr></tbody></table>

Results are presented on figures 5-6. Cost in case of no detection is much larger than cost in case of detection. It comes clearly from the selected cost

model (huge gap between C<sub>repair</sub> / C<sub>failure</sub> ratio and C<sub>inspection</sub> / C<sub>repair</sub> ratio).

Influence of probability of defects presence γ is in an opposite way for E(C)<sub>d</sub> and E(C)<sub>nd</sub>: cost of detection is more sensitive for small probability of defects presence γ (0.1 to 0.5) and cost of no detection is more sensitive for high probability of defects presence γ (0.5 to 0.9).

<figure><img src="image_6.png" /><figcaption>Figure 5. Influence of probability of defects presence γ on the cost in case of detection E(C)<sub>d</sub>.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 6. Influence of probability of defects presence γ on the cost in case of no detection E(C)<sub>nd</sub>.</figcaption></figure>

Furthermore, cost of detection and no detection increase when performances of ROC curves decrease.

# 3 EFFECT OF THE SHAPE OF ROC CURVES ON THE DECISION PROCESS

## 3.1 Aim and principle

A simple geometric characterization of ROC curves is the distance between the curve and the best performance point BPP of coordinates (PFA = 0, PoD = 1) (Schoefs, 2004): by definition, the more is this distance, the worst is the performance. The corresponding point on the ROC curve is called the performance point of the NDT tool (NDT-BPP).
However as the configuration of ROC curves for the same distance are various, we extend in this paper this characterization by using the polar coordinates of the NDT-BPP. Then NDT-BPP polar coordinates are defined by:

- The radius $\delta_{NDT}$ equal to the performance index (NDT-PI) (distance between the best performance point and the ROC curve) (Schoefs, 2004, 2007);

- The angle $\alpha_{NDT}$ between axis (PFA = 0) and the line (BPP, NDT-BPP).

Objective of this parametric study is to analyse the effect of the shape of ROC curves on the decision process: inspection, maintenance, repair. To achieve this goal, the influence of the performance of ROC curves, represented by $\delta_{NDT}$ and $\alpha_{NDT}$, is appreciated through the expectation of the cost in case of detection $E(C)_d$ and the cost in case of non detection $E(C)_{nd}$.

To this aim the ‘Signal + noise’ and noise amplitude are normally distributed with respective characteristics $N(\mu_{s+n}, \sigma_{s+n})$ and $N(\mu_n, \sigma_n)$. The mean and the standard deviation of the ‘signal + noise’ are modified as well as the standard deviation of the noise with a step of 0.5. Variation range of the statistical parameters of ‘signal + noise’ and noise used during the study are shown on Figures 7-8.

<figure><img src="image_8.png" /><figcaption>Figure 7. Variation range of the statistical parameters of the ‘signal + noise’ normally distributed.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 8. Variation range of the statistical parameters of the noise normally distributed.</figcaption></figure>

Then, the best performance point of the NDT tool (NDT-BPP) is defined for each configuration.

## 3.2 Analysis of the NDT-BPP mapping

Figure 9 presents the mapping of NDT-BPP in the plan (PFA, PoD). We notice that all the NDT-BPP are included between 0.4 and 1.0 on the PoD axis and between 0.0 and 0.6 on the PFA axis, that represents an interesting area to analyse the effect of the shape of ROC curves on the decision process.

<figure><img src="image_10.png" /><figcaption>Figure 9. NDT-BPP mapping in cartesian plan.</figcaption></figure>

Four examples of NDT tools performances represented by ROC curves are plotted on Figure 9. For each one, Cartesian and polar coordinates of NDT-BPP are given on Table 2.

<table><caption>Table 2. Cartesian and polar coordinates of NDT-BPP.</caption><thead><tr><th rowspan="2"></th><th colspan="2">Cartesian plan</th><th colspan="2">Polar plan</th></tr><tr><th>PFA</th><th>PoD</th><th>δ</th><th>α (°)</th></tr></thead><tbody><tr><td>NDT-BPP1</td><td>0.50</td><td>0.50</td><td>0.71</td><td>45.00</td></tr><tr><td>NDT-BPP2</td><td>0.00</td><td>1.00</td><td>0.00</td><td>00.00</td></tr><tr><td>NDT-BPP3</td><td>0.16</td><td>0.79</td><td>0.26</td><td>36.83</td></tr><tr><td>NDT-BPP4</td><td>0.18</td><td>0.88</td><td>0.21</td><td>57.23</td></tr></tbody></table>

NDT-BPP1 corresponds to the worst NDT tool performance, meaning that some noise can be easily detected, even if nothing is to be detected (no presence of defect) and finally leading to a high number of false alarms. At the opposite, NDT-BPP2 is the best NDT tool performance in our numerical data-base. In this case, the probability of detection reaches very quickly values near 1, with small probabilities of false alarms for high values of probabilities of detection.

Performances of NDT tool 1 and NDT tool 2 can be easily compared with the radius $\delta_{NDT}$, i.e. 0.71 for NDT-BPP1 versus 0.00 for NDT-BPP2. But between these two extreme levels of NDT tools performances, there is a multitude of cases where this single parameters doesn't allow to differentiate ROC curves. To illustrate the role of $\alpha_{NDT}$, the maintenance policy, cost functions and cost models are the same that those used in section. The best of four NDT tools is the one which minimize the two cost functions or a combination of them (sum, ...),
related to the cases of detection or no detection. We also considered three values of the probability of defect presence γ, i.e. 0.1, 0.5 and 0.9, that is needed when computing likelihood of events that govern the cost expectation.

Results of cost analysis are presented on Table 3. Let us focused on the comparison of the NDT tool 3 and NDT tool 4 performances. It is interesting to notice that for the three values of γ, the cost in case of detection E(C)d is always minimum for NDT tool 3 and the cost of no detection E(C)nd is always minimum for NDT tool 4. Here cost in case of no detection is much larger than cost in case of detection. It comes clearly from the selected maintenance policy and cost models. With another cost model, this tendency can be reversed (Breysse et al. 2008, O'Connor et al. 2008). In this case, δNDT become not sufficient to characterize the performance of NDT tools. Then, it is necessary to use the two polar coordinates: radius δNDT and the angle αNDT.

<table><caption>Table 3. Performances of the four NDT tools based on cost analysis.</caption><thead><tr><th rowspan="2"></th><th colspan="2">γ = 0.1</th><th colspan="2">γ = 0.5</th><th colspan="2">γ = 0.9</th></tr><tr><th>E(C)<sub>d</sub>E(C)<sub>nd</sub></th><th>E(C)<sub>d</sub>E(C)<sub>nd</sub></th><th>E(C)<sub>d</sub>E(C)<sub>nd</sub></th><th>E(C)<sub>d</sub>E(C)<sub>nd</sub></th><th>E(C)<sub>d</sub>E(C)<sub>nd</sub></th><th>E(C)<sub>d</sub>E(C)<sub>nd</sub></th></tr></thead><tbody><tr><td>NDT tool 1</td><td>0.010</td><td>0.100</td><td>0.006</td><td>0.501</td><td>0.001</td><td>0.901</td></tr><tr><td>NDT tool 2</td><td>0.004</td><td>0.026</td><td>0.002</td><td>0.154</td><td>3.10<sup>-4</sup></td><td>0.372</td></tr><tr><td>NDT tool 3</td><td>0.006</td><td>0.040</td><td>0.002</td><td>0.252</td><td>4.10<sup>-4</sup></td><td>0.713</td></tr><tr><td>NDT tool 4</td><td>0.008</td><td>0.032</td><td>0.003</td><td>0.189</td><td>5.10<sup>-4</sup></td><td>0.457</td></tr></tbody></table>

This last point is conforted by the fact that variation ranges of costs of detection and costs of no detection according to δNDT are relatively important. Results are presented on Figures 10-11 for three probabilities of defect existence γ.

<figure><img src="image_11.png" /><figcaption>Figure 10. Variation range of costs of detection E(C)d according to the radius δNDT.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 11. Variation range of costs of no detection E(C)nd according to the radius δNDT.</figcaption></figure>

Variation ranges of costs of detection and costs of no detection according to αNDT are shown on Figures 12-13.

<figure><img src="image_13.png" /><figcaption>Figure 12. Variation range of costs of detection E(C)d according to the radius αNDT.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 13. Variation range of costs of no detection E(C)nd according to the radius αNDT.</figcaption></figure>

As the radius δNDT, the angle αNDT can't be used alone to characterize performance of NDT tool. Consequently, the following study is dedicated to analyze costs of detection and costs of no detection according to the whole numerical data-base of NDT-BPP obtained during the previous simulations (see Figure 14).
<figure><img src="image_15.png" /><figcaption>Figure 14. NDT-BPP mapping in polar plan.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 16. Mapping of cost of detection E(C)<sub>d</sub> in polar plan for probability of defect existence γ = 0.1.</figcaption></figure>

### 3.3 Influence of NDT-BPP polar coordinates on cost of detection and cost of no detection

Figures 15-22 present mappings of cost of detection and cost of no detection according to the cartesian and polar coordinates of NDT-BPP. Mappings are established for the three probabilities of defect presence γ used in this study, i.e. γ = 0.1, 0.5 and 0.9.

For low probability of defect presence (γ = 0.1), costs of detection are minimum when associated mainly to values of radius δ<sub>NDT</sub> lower than 0.4 and values of angle α<sub>NDT</sub> minor than 40°. Maximum costs of detection are located in zone with high values of δ<sub>NDT</sub> and α<sub>NDT</sub>. It is also interesting to notice that for values of δ<sub>NDT</sub> less than 0.1, the influence of α<sub>NDT</sub> is negligible. Costs of no detection for γ = 0.1 are maximum for values of δ<sub>NDT</sub> more than 0.1 and for values of α<sub>NDT</sub> less than 30°. When values of α<sub>NDT</sub> are higher than 30°, costs of no detection increase essentially with δ<sub>NDT</sub>.

<figure><img src="image_17.png" /><figcaption>Figure 15. Mapping of cost of detection E(C)<sub>d</sub> in cartesian plan for probability of defect existence γ = 0.1.</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Figure 17. Mapping of cost of no detection E(C)<sub>nd</sub> in cartesian plan for probability of defect existence γ = 0.1.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>Figure 18. Mapping of cost of no detection E(C)<sub>nd</sub> in polar plan for probability of defect existence γ = 0.1.</figcaption></figure>

For high probability of defect presence (γ = 0.9), maximum costs of detection are located in zone delimited by values of δ<sub>NDT</sub> higher than 0.3 and α<sub>NDT</sub> and higher than 50°. Outside this area, cost of detection increases mainly with δ<sub>NDT</sub>. Costs of no detection for γ = 0.9 increases when α<sub>NDT</sub> decreases and δ<sub>NDT</sub> increases.
<figure><img src="image_20.png" /><figcaption>Figure 19. Mapping of cost of detection E(C)d in cartesian plan for probability of defect existence γ = 0.9.</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>Figure 20. Mapping of cost of detection E(C)d in polar plan for probability of defect existence γ = 0.9.</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>Figure 21. Mapping of cost of no detection E(C)nd in cartesian plan for probability of defect existence γ = 0.9.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>Figure 22. Mapping of cost of no detection E(C)nd in polar plan for probability of defect existence γ = 0.9.</figcaption></figure>

## 4CONCLUSION

This paper shows the interest of combination methods, i.e. intersection or union, to optimize the maintenance of existing structures and highlights the impact of the expert judgement in the decision process at two levels:

- The way to address the decision after obtaining results from the two inspections (choice of combination methods);

- The estimation of the probability of defect existence that is needed when computing likelihood of events that govern the cost expectation.

These aspects are treated by an illustration of the process for the RBI of steel harbour structures submitted to uniform corrosion.

Finally, the effect of the shape of ROC curves on the decision process - inspection, maintenance, repair- is analyzed. The objective of the parametric study is to assess the influence of the performance of NDT tools, represented by δNDT and αNDT, through the expectation of the costs in case of detection E(C)d and in case of non detection E(C)nd.

## 5REFERENCES

Barnouin, B., Lemoine, L., Dover, W.D., Rudlin, J.R., Fabbri, S., Rebourget, G., Topp, D., Kare, R., Sangouar, D. 1993. 'Underwater inspection reliability trials for offshore structures', ASME NY, editor. Proc. of the 12th International Conference on Offshore Mechanics and Arctic Engineering. Vol. 2 pp. 883-890.

Breysse, D., Sheils, E., Schoefs, F., O'Connor, A. 2008. 'Life cycle cost analysis of ageing structural components based on non destructive condition assessment', to be published in the Australian Journal of Structural Engineering.

O'Connor, A., Breysse, D., Schoefs, F., Sheils, E. 2008. 'Disaster and hazard mitigation for infrastructural elements/networks through provision of a two stage inspection process', Third International Forum on Engineering Decision Making, 'Optimal Strategies for Disaster and Hazard Mitigation', IFED, December 12-15 2007, Port Stephens, New South Wales, Australia.

Pakrashi, V., Schoefs, F., Memet, J.B., O'Connor, A. 2008. 'An Image Analysis Based Damage Classification Methodology', Structure and Infrastructure Engineering: Maintenance, Management, Life-Cycle Design and performance (NSIE), in press.

Rouhan, A., Schoefs, F. 2003. 'Probabilistic modelling of inspections results for offshore structures', Structural safety, vol 25, pp. 379-399, 20 pages, Elsevier Ltd.

Rudlin, J.R. 1996. 'Reliability of inspection for fatigue cracks in offshore structures', Institution of Electrical Engineers, Savoy Place, London WC2R OBL, UK. 6/1-6/3.

Rudlin, J.R., Dover, W.D. 1996. 'The ICON database. Assisting underwater inspections', Offshore technology, vol. 4.

Schoefs, F., Clément, A. 2004. 'Multiple inspection modeling for decision making and management of jacket off-shore platforms: effect of false alarms', First International Forum on Engineering Decision Making (IFED'04), december 5-9 2004, Stoos, Switzerland.
Schoefs, F., Clément, A., Memet, J.B., and Nouy, A. 2007. Spatial dependence of Receiver Operating Characteristic curves for Risk Based Inspection of corroded structures: application to on-pile wharf, *Proc. of 10th International Conference on Applications of Statistics and Probability in Civil Engineering, (I.C.A.S.P'07), July31-August 3 2007, Tokyo, Japan*.

Schoefs, F., Clément, A., Memet, J.B., Nouy, A. 2008a. 'Inspection of Steel structures in Coastal Area with NDT tools: use of polynomial chaos decomposition for building Receiver Operating Curves', *Proceeding of 1st International Conference on Applications Heritage and Constructions in Coastal and Marine Environment, (MEDACHS'08), 28-30 January 2008, Lisbon (LNEC), Portugal*.

Schoefs, F., Clément, A., Boéro, J; Capra, B. 2008b. Expert judgement for combining NDT tools in RBI context: Application to marine structures, *4th ASRANet Colloquium, 25-27 June 2008, Athens, Greece*.

Sheils, E., O'Connor, A., Breysse, D., Schoefs, F., and Yotte, S. 2008. Development of a two stage inspection process for the assessment of deteriorating bridge structures, *Journal of Bridge Engineering (Publication of the American Society of Civil Engineers)*, to be published.

Straub, D., Faber, M.H. 2003. 'Modelling dependency in inspection performance', *Proc. Application of Statistics and Probability in Civil Engineering, ICASP 2003 - San Franccisco, Der Kiureghian, Madanat and Pestana eds.*, Millpress, Rotterdam, ISBN 90 5966 004 8. pp. 1123-1130

Yang, J.R., Donath, R.C. 1984. 'Improving NDE through multiple inspections'.