# Development and evaluation of the accuracy of an indicator of the appropriateness of interventional cardiology generated from a French registry

Florence Francis-Oliviero, Pierre Coste, Emilie Lesaine, Corinne Perez, Francois Casteigt, Jean Marie Clerc, Nicolas Delarche, Akil Hassan, Bernard Larnaudie, Jean Louis Leymarie, et al.

► To cite this version:

Florence Francis-Oliviero, Pierre Coste, Emilie Lesaine, Corinne Perez, Francois Casteigt, et al. Development and evaluation of the accuracy of an indicator of the appropriateness of interventional cardiology generated from a French registry. Archives of Public Health, 2022, 80 (1), pp.132. 10.1186/s13690-022-00885-4. hal-03696536

HAL Id: hal-03696536

https://hal.science/hal-03696536

Submitted on 16 Jun 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
RESEARCH

Open Access

<figure><img src="image_2.png" /></figure>

Check for
updates

# Development and evaluation of the accuracy of an indicator of the appropriateness of interventional cardiology generated from a French registry

Florence Francis-Oliviero<fnref n="1" />,²*, Pierre Coste³, Emilie Lesaine<fnref n="1" />, Corinne Perez<fnref n="1" />, François Casteigt⁴, Jean-Marie Clerc⁵, Nicolas Delarche⁶, Akil Hassan⁷, Bernard Larnaudie⁴, Jean-Louis Leymarie⁸, Louis-Rachid Salmi<fnref n="1" />,² and Florence Saillour-Glenisson<fnref n="1" />,²

**Abstract**

**Background:** Development of appropriateness indicators of medical interventions has become a major quality-of-care issue, especially in the domain of interventional cardiology (IC). The objective of this study was to develop and evaluate the accuracy of an indicator of the appropriateness of interventional cardiology acts (invasive coronary angiographies (ICA) and percutaneous coronary interventions (PCI)) in patients with coronary stable disease and silent ischemia, automated from a French registry.

**Methods:** All ICA and PCI recorded in a Regional IC Registry (ACIRA) and operated for a stable coronary artery disease or silent ischemia from January 1st to December 31st 2013 in eight IC hospitals of Aquitaine, southwestern France, were included.

The indicator was developed to reflect European guidelines. Classification of appropriateness by the indicator, measured on the registry database, was compared to the classification of a reference standard (expert judgment applied through complete record review) on a random sample of 300 interventions.

Accuracy parameters were estimated. A second version of the indicator was defined, based on the analysis of false negative and positive results, and its accuracy estimated.

**Results:** The second indicator accuracy was: sensitivity 63.5% (95% confidence interval CI [51.7–75.3]), specificity 76.0% (95%CI [70.4–81.6]), PPV 43.0% (95% CI [33.0–53.0]) and NPV 88.0% (95% CI [83.4–92.6]). When stratified on the type of act, parameters were better for ICA alone than for PCI.

**Conclusions:** Accuracy of the indicator should raise with improvement of database quality. Despite its average accuracy, it is already used as a benchmark indicator for cardiologists. It is sent annually to each IC center with value of the indicator at the region level to allow a comparison.

**Keywords:** Angiography, Angioplasty, Practice registry, Appropriateness

**Background**

All health systems face challenges in delivering high-quality, effective and safe care at an affordable cost. Improving appropriateness of care, i.e. the adequacy of

<footnote>*Correspondence: florence.francis@u-bordeaux.fr</footnote>

<footnote n="1">¹ Inserm Bordeaux Population Health, U1219, ISPED, Univ Bordeaux, Bordeaux, France
Full list of author information is available at the end of the article</footnote>
care to patient needs in accordance with practice guidelines, is a major challenge, both for clinical care and public health policy [1].

Appropriateness of care has become of particular importance in the field of interventional cardiology (IC) [2]. IC includes all catheter-based procedures for treating congenital heart defects, rhythm disorders, or diagnosing and treating acute or chronic coronary artery disease. The latter indication of myocardial revascularization involves mainly two procedures: invasive coronary angiography (ICA) and percutaneous coronary intervention (PCI). These are invasive and costly interventions, potentially exposing patients to adverse events such as hema-toma, arterial dissection, stroke, or related to toxicity of the contrast agent.

In 2014, major variations in standardized proportions of myocardial revascularization, from 111 to 371 procedures per 100 000 inhabitants, were observed in 12 Organization for Economic Co-operation and Development countries [3]; France, with 213 PCI per 100 000 inhabitants, was at the 8th position. One possible explanation for these variations relates to the appropriateness of IC and their consequences on the quality and safety of care. In the United States, appropriateness of PCI is now one of the inpatient quality indicators measured by the Agency for Healthcare Research and Quality [4].

Using the appropriate use criteria (AUC) for coronary revascularization and diagnostic catheterization, developed by the American College of Cardiology and five other professional organizations, recent studies showed higher proportions of inappropriate ICA and PCI in the non-acute context [5, 6]. For example, Chan et al. identified inappropriate use in 11.6% of non-acute PCIs in the United States National Cardiovascular Data Registry [7]. Bradley et al. also identified, in Washington State, inappropriate use in 17.0% in non-acute against 1.0% in acute PCIs [7]. In New York, Hannan et al. reported a proportion of 24.9% inappropriate ICAs [8]. Less information is available in Europe, and particularly in France, on the frequency of inappropriate ICA and PCI and no indicator of appropriate use has been implemented.

USA, Sweden, Germany, Great Britain, Denmark, Austria, and Japan have implemented regional or national IC registries, recording IC activity, and assessing medical practice to improve quality of care. From the registry databases, indicators may be measured to assess and monitor continuously the level of PCI or ICA appropriateness and to provide feedbacks [9, 10]. Implementation modalities vary between registries [11–13] and most of them only concern PCI. In France, such cardiovascular registries collect exhaustive and prospective data concerning interventional cardiology acts [14, 15]. One of them, the ACIRA registry is running since 2012 and includes all acts of interventional cardiology performed in the Aquitaine region, southwestern France [16].

No appropriateness indicators, to our knowledge, have been yet developed in France. Indeed, the AUC cannot directly be derived from European IC registries for two reasons: i) the criteria were developed from American guidelines for the management of stable coronary artery disease (SCAD) and can hardly be transposed to European guidelines that were develop by European experts considering the specificities of European health care systems; ii) they are based on clinical symptoms and intensity of medical treatment, data that are not available in European interventional cardiology registries information systems, particularly in France.

The objective of this study was to develop and evaluate the accuracy of an indicator of the appropriateness of ICA and PCI in patients with coronary stable disease and silent ischemia, automated from the ACIRA registry.

## Methods

### Study design

A first version of the indicator has been developed from the ACIRA database (Fig. 1). Its accuracy (sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV)) was estimated by comparing, on a random sample of interventions from the ACIRA database, the classification of appropriateness by the indicator and a reference standard, based on experts’ judgment. Reasons for false positives (FP) and negatives (FN) results

<figure><img src="image_3.png" /></figure>
were then investigated to construct a final indicator and improve its accuracy.

## The ACIRA registry

ACIRA is an exhaustive nominative and prospective cohort of patients cared with PCI, ICA or both. Included patients are all adults (≥ 18 years) French residents who underwent ICA or PCI in one of the 11 interventional cardiology hospitals (public university or local and private hospitals) in the Aquitaine region (southwestern France) and agreed to participate [16]. ACIRA registry identifies and describes patients who had an ICA or PCI in Aquitaine, assesses one-year outcomes and IC practices, healthcare pathway, and organization. It is meant to help cardiologists, researchers, healthcare decision-makers and hospitals answer questions about quality, safety, suitability, efficiency of interventional cardiology in the region and facilitate clinical and interventional research projects. ACIRA data are only extracted from existing databases. The data related to hospitalization and procedure is directly extracted from hospital information systems. Readmissions, in-hospital complications, cardiovascular morbidity and in-hospital mortality are collected from the French hospital medical information system database (PMSI), during one year after the initial procedure. An audit process of extractions checking, data harmonization and quality controls are carried out. Each year ACIRA includes about 15 000 IC acts.

## Study population and sample

The study population included all IC acts, registered in the ACIRA registry database, performed for the treatment or diagnosis of SCAD or silent ischemia (SI), from January 1st to December 31st 2013, in eight of the eleven interventional cardiology centers in the region. At the time of the study, three of the eleven centers had not yet

implemented data extraction from their hospital information system. To estimate accuracy of the indicator, a sample of 300 interventions was randomly extracted from the study population. This threshold of 300 consisted in a fair balance between feasibility of data collection, and precision of the sensitivity and specificity estimations. Assuming a proportion of 30% inappropriate acts, estimated from a pilot study, 300 interventions were sufficient to estimate, with a two-sided 3% precision (confidence interval estimated with the normal law), an expected specificity of 95% and two-sided 7% precision around an expected sensitivity of 85%.

## Indicator development

The indicator was developed on the basis of the European Society of Cardiology guidelines [17, 18]. Only IA and IIIA grade recommendations have been considered (Table 1). Indicators were developed for each of three types of coronary disease management: ICA alone, ICA followed by PCI, and PCI alone. For ICA followed by PCI, ICA and PCI appropriateness were independently taken in consideration. If one of the two interventions was identified as inappropriate, the whole strategy was considered inappropriate.

As no data on symptoms or current treatment was available in the ACIRA database, all included patients with a SCAD were considered as having typical symptoms of stable angina and receiving optimal medical treatment. In the same way, calculation of the SYNTAX score has been partially calculated because of missing variables in the registry [19]. It is an anatomical prognostic score of severity, based on a precise description of the nature and complexity of the vascular lesions. Usually, it is calculated based on the vascular heart dominance, the localisation of segments that are diseased, the presence or absence of: total occlusion,

<table><caption>**Table 1** High graded 2014 ESC/EACTS Guidelines on myocardial revascularization for Interventional Coronary Angiography and Percutaneous Coronary Intervention indications</caption><thead><tr><td>Guidelines issues</td><td>Recommendation</td><td>Class/Level</td></tr></thead><tbody><tr><td rowspan="2">Indication of ICA</td><td>LVEF &lt; 50% and either typical angina or PTP &gt; 85% without prior non-invasive tests</td><td>IA</td></tr><tr><td>Results of non-invasive testing were in favor of an ischemia and the PTP was between 15 and 85%</td><td>IA</td></tr><tr><td rowspan="3">Indication of revascularization for prognosis</td><td>A left main disease with stenosis &gt; 50%</td><td>IA</td></tr><tr><td>Any proximal left anterior descending coronary artery stenosis &gt; 50%</td><td>IA</td></tr><tr><td>Two-vessel or three-vessel disease with stenosis &gt; 50% with impaired LV function (LVEF &lt; 40%)</td><td>IA</td></tr><tr><td>Indication of revascularization for symptoms</td><td>Any coronary stenosis &gt; 50% in the presence of limiting angina or angina equivalent, unresponsive to medical therapy</td><td>IA</td></tr><tr><td rowspan="2">Non-indication of a PCI but recommendation of a CABG when a revascularization is recommended</td><td>Left main disease with a SYNTAX score &gt; 32</td><td>IIIB</td></tr><tr><td>Three-vessel disease with a SYNTAX score &gt; 23</td><td>IIIB</td></tr></tbody></table>

CABG Coronary Artery Bypass Graft, ICA Interventional Coronary Angiography, PCI Percutaneous Coronary Intervention
trifurcations, bifurcations, aorto ostial lesion, severe
tortuosity, heavy calcification and length of the disease
superior to 20 mm. In the present study, using data
from ACIRA registry, SYNTAX score has been calcu-
lated using the following items: stenosis grade, exist-
ence of trifurcations and number of segments diseased,
and thrombus. Details concerning occlusion were not
available in the ACIRA database and could not be
implemented in the score calculation. In guidelines,
when the SYNTAX Score is higher than 32 with a left
main disease, or higher than 23 with a three-vessel dis-
ease, PCI is not recommended. Moreover, as the infor-
mation about the dominance side could not be known
from the ACIRA database, we have considered for the
SYNTAX score calculation that all people had a right
dominance.

Functional Flow Reserve (FFR) is a procedure per-
formed during an ICA: if it was more than 0.80, it was
considered that functional impact was insufficient to
allow revascularization by PCI. Only the conduct of FFR
was recorded in the ACIRA database, but not its results.
If an FFR was done and all stenosis were less than 50%,
and a revascularization followed, FFR was considered as
being greater 0.8.

Reference standard

The reference standard was based on experts' judgment. It was conducted in two steps: regional consensus and case review. Experts were referent cardiologists from the registry centers, who accepted to participate in the process.

First step: regional consensus

The nominal group technique (NGT) was used to obtain
a regional consensus on ICA and PCI indications [20].
The objectives of the NGT meeting were to define an
exhaustive and unambiguous classification of any patient
with stable angina or silent ischemia, allowing sorting
them according to the appropriateness of the act they
received. The NGT was organized in four steps: silent
generation of ideas in response to the question, "round
robin", clarification, and voting. The five experts were
asked the following question: "Which clinical and man-
agement elements do determine appropriateness or inap-
propriateness of ICA and PCI in the context of SCAD
and SI?" A proposal was considered as consensual if four
of five cardiologists had voted for it. Finally, consensu-
ally-retrieved items were organized by two of the authors
(FF and FSG) to produce the appropriateness classifica-
tion. This classification allowed carrying out the case
review on a consensual basis.

Second step: case review

During the case review, the experts judged appropriate-
ness of the acts randomized in the study sample. Infor-
mation needed to assess appropriateness of interventions
(clinical data such as medical history, symptoms, type
of ischemia, radiological examinations and results...)
was collected from patient record files and recorded
on a standardized grid. Content of this grid had been
previously validated by the cardiologists. To judge the
appropriateness of the 300 randomly selected interven-
tions, experts were asked to use the previously defined
classification. Each case was independently assessed by
two experts, each pair reviewed approximately 100 files.
Experts assessed appropriateness blindly from each
other, and did not know in which center the interven-
tion had been done. If they disagreed, a third expert was
asked to settle.

Accuracy and discordance analyses

Sensitivity, specificity, PPV, and NPV of the indicators
were estimated, using expert's judgment as the refer-
ence standard. A true positive (TP) was an intervention
defined inappropriate by both experts and indicator. Cor-
responding 95% confidence intervals (CI) were estimated
using normal approximation [21]. Measurement char-
acteristics were calculated first for all 300 interventions,
then stratified for ICA alone, ICA followed by a PCI, and
PCI alone.

Two experts (FC and JLL) analyzed extracted files of
false positives (FP) and negatives (FN) results to under-
stand mechanisms leading to discordances between the
reference standard and the indicator. Then a new indica-
tor (final indicator) was developed with an independent
cardiologist, who was not involved in previous phases,
and its accuracy was estimated.

We conducted robustness analyses on hypothesis made
(approximation of Syntax score) by adding and subtracting
5 points to Syntax score calculated and by imputing
data regarding eight files that had been excluded because
experts were unable to decide on their appropriateness.

Results

Study population and sample

From the 15 233 interventions in the 2013 ACIRA data-
base, 5 678 were selected in the study population (3 612
for SCAD and 2 066 of SI); 35.1% had a previous PCI, and
62.8% had high blood pressure. Clinical characteristics of
the study sample are described in Table 2.

Indicator development and accuracy

Among the 300 interventions, eight were excluded,
for which experts did not succeed in assessing
<table><caption>**Table 2** Clinical characteristics of the randomly selected study sample for the validation of the appropriateness tool (n = 300)</caption><thead><tr><th rowspan="2"></th><th colspan="3">Sample</th></tr><tr><th>MV</th><th>N</th><th>%</th></tr></thead><tbody><tr><td>Male gender</td><td></td><td>229</td><td>76.3</td></tr><tr><td>Ischemia type</td><td></td><td></td><td></td></tr><tr><td>Stable angina</td><td></td><td>191</td><td>63.7</td></tr><tr><td>Silent ischemia</td><td></td><td>109</td><td>36.3</td></tr><tr><td>Antecedent of</td><td></td><td></td><td></td></tr><tr><td>PCI</td><td>4</td><td>104</td><td>35.1</td></tr><tr><td>CABG</td><td>3</td><td>24</td><td>8.1</td></tr><tr><td>Myocardial infarctus</td><td>21</td><td>47</td><td>15.7</td></tr><tr><td>Stroke</td><td>3</td><td>10</td><td>3.4</td></tr><tr><td>LEVF &lt; 50%</td><td>88</td><td>34</td><td>16.1</td></tr><tr><td>Chronic renal failure</td><td>4</td><td>8</td><td>2.7</td></tr><tr><td>Arteriopathy</td><td>4</td><td>38</td><td>12.8</td></tr><tr><td>Cardiovascular risk factors</td><td></td><td></td><td></td></tr><tr><td>Current smoking</td><td>12</td><td>50</td><td>17.4</td></tr><tr><td>Diabetes</td><td>0</td><td>93</td><td>31.0</td></tr><tr><td>Dyslipidemia</td><td>4</td><td>191</td><td>65.0</td></tr><tr><td>Coronary heredity</td><td>10</td><td>61</td><td>21.0</td></tr><tr><td>Obesity (BMI &gt; = 30 kg/m²)</td><td>17</td><td>68</td><td>24.0</td></tr><tr><td>Arterial hypertension</td><td>2</td><td>187</td><td>62.8</td></tr><tr><td>Positive non-invasive tests</td><td>0</td><td></td><td></td></tr><tr><td>Electrocardiogram modifications</td><td></td><td>53</td><td>17.7</td></tr><tr><td>Elevated cardiac enzyme</td><td></td><td>11</td><td>3.7</td></tr><tr><td>Stress test</td><td></td><td>84</td><td>28.0</td></tr><tr><td>Myocardial scintigraphy</td><td></td><td>38</td><td>12.7</td></tr><tr><td>Stress echocardiography</td><td></td><td>35</td><td>11.7</td></tr><tr><td>Cardiac RMI</td><td></td><td>0</td><td>0.0</td></tr><tr><td>Coroscanner</td><td></td><td>3</td><td>1.0</td></tr><tr><td>FFR analysis</td><td>55</td><td>14</td><td>5.7</td></tr></tbody></table>

CABG Coronary Artery Bypass Graft, FFR Fractional flow reserve, MV missing values, LVEF Left Ventricular Ejection Fraction, PCI Percutaneous Coronary Intervention, RMI Resonance Magnetic Imaging

appropriateness, because of missing relevant clinical data. Among the remaining interventions, 88 (30.1%) were classified as inappropriate by the indicator, whereas 63 (21.6%) were judged as inappropriate by the experts (24.2% for ICA, and 11.2% for PCI). Sensitivity was 57.1% (95% CI [44.9–69.3]) and specificity 77.3% (95% CI [71.9–82.7%]) (Table 3).

## False positive and negative results

The 52 FPs and 27 FNs were due to two main reasons. Firstly, missing values in the registry (probably not completed by cardiologists) yet mentioned in the medical files (especially concerning past history, results of tests and exams performed) for 33 (63.5%) FPs and 6 (22.2%) FNs. Secondly, a particular clinical context not taken into account in the indicator algorithm because of a lack of details in the registry, explained 12 (23.1%) FPs and 15 (55.6%) FNs. Particularly, the simplification made by considering only age and symptoms in the first step to evaluate appropriateness of ICA lead to misclassification. Four FPs occurred because of variation among cardiologists of the definition of SCAD in the ACIRA database. Some cardiologists considered as “stable angina” situations of acts performed in the month following an acute coronary syndrome, while others consider it as acute coronary syndrome. Lastly, approximations in the SYNTAX score calculation explained 3 (5.8%) FPs and 1 (3.2%) FN; 4 (7.7%) FPs and 5 (9.6%) FNs remained unexplained.

## Final indicator definition and accuracy

Notion of severe coronary medical history (at least two cardiovascular risk factors among diabetes, hypertension and smoking) was added to age and sex to evaluate the pretest probability and indicate or not an ICA (Fig. 2). To avoid problems linked to stable angina diagnostic definition, patients with an acute coronary syndrome in the month before an interventional act (PCI or CAG) and a SCAD diagnosis in the database were excluded.

<table><caption>**Table 3** Diagnostic performances of the initial and final tool stratified according to the type of act (ICA only, ICA followed by PCI or PCI only)</caption><thead><tr><th rowspan="3">Performances</th><th colspan="2">First version</th><th colspan="2">Final version</th></tr><tr><th>Total (n = 292)</th><th>Total (n = 284)</th><th>ICA only (n = 176)</th><th>ICA followed by a PCI (n = 43)</th></tr><tr><th>% (95% CI)</th><th>% (95% CI)</th><th>% (95% CI)</th><th>% (95% CI)</th></tr></thead><tbody><tr><td>Sensitivity</td><td>57.1 (44.9–69.3)</td><td>63.5 (51.7–75.3)</td><td>76.0 (64.2–87.8)</td><td>100.0 (69.2–100.0)</td></tr><tr><td>Specificity</td><td>77.3 (71.9–82.7)</td><td>76.0 (70.4–81.6)</td><td>73.8 (66.1–81.5)</td><td>57.5 (42.2–72.8)</td></tr><tr><td>PPV</td><td>40.9 (30.6–51.2)</td><td>43.0 (33.0–53.0)</td><td>53.5 (41.9–65.1)</td><td>46.5 (24.6–68.4)</td></tr><tr><td>NPV</td><td>86.8 (82.2–91.4)</td><td>88.0 (83.4–92.6)</td><td>88.6 (82.4–94.8)</td><td>53.5 (40.5–66.5)</td></tr></tbody></table>

ICA Invasive Coronary Angiography, PCI Percutaneous Coronary Intervention, PPV Positive Predictive Value, NPV Negative Predictive Value, CI Confidence interval
(A) Indication of ICA

<figure><img src="image_4.png" /></figure>

(A) Indication of ICA : final version

<figure><img src="image_5.png" /></figure>

(B) Indication of PCI

<figure><img src="image_6.png" /><figcaption>**Fig. 2** Presentation of the appropriateness indicators</figcaption></figure>
Final indicator sensitivity was 63.5% (95% CI [51.7-75.3]) and specificity was 76.0% (95%CI [70.4-81.6]). When stratified on the type of act (Table 3), parameters were better for ICA alone than for PCI.

### Robustness analysis (Table 4)

When adding 5 points to the SYNTAX score, the indicator had a sensitivity of 66.7% (95%CI [55.1-78.3]), and specificity of 72.0% (95%CI [66.1-77.9]). When subtracting 5 points, sensitivity was 65.1% (95%CI [53.3-76.9]) and specificity 75.6% (95%CI [69.9-81.3]). When completing database records with information found in clinical files, all measurement parameters dramatically improved: sensitivity was 76.2% (95%CI [65.7-86.7]), specificity 89.6% (95%CI [84.7-93.6]), PPV 67.6% (95%CI [56.7-78.5]), and NPV of 93.0% (95%CI [84.7-89.5]).

### Discussion

We conducted the first study developing and validating an appropriateness indicator of PCI and ICA in a SCAD or SI context, from a French interventional cardiology database. A detailed analysis of FPs and FNs sources allowed an improvement of the indicator's accuracy. From this perspective, final results suggest that the indicator might be improved through the improvement of ACIRA database completeness (in process). A new evaluation of indicator's accuracy after improvement of database quality should be performed. Despite its average accuracy, it is already used as a benchmark indicator for cardiologists. Each center receives annually the value of the indicator at the center level compared to regional value. The idea is not to conclude on the basis of numerical results, but to have a reference value year after year, which allow cardiologists to question their practices.

The indicator had a very low sensitivity to detect inappropriate PCI, even in its final version, leading us not to consider it as a valid tool for the assessment of PCI appropriateness. Analysis of FNs and FPs allowed us to understand the reasons of poor accuracy. Some hypotheses, guideline approximations and simplifications had

to be done in the indicator construction, due to lack of information in the ACIRA database inappropriate. PCI were difficult to circumscribe with details from guidelines, which include many grade-II recommendations, difficult to implement into the tool, because of their uncertainty. Another reason of the low accuracy to detect inappropriate PCI is linked with the approximation made to calculate the SYNTAX score. This could explain the difficulty of the indicator to detect inappropriate PCIs; moreover, all unexplained cases after discordance analysis were PCI cases. This analysis allowed us also to precise and adapt our indicator to the database.

A robustness analysis assuming a perfectly completed database dramatically improved accuracy. Another element is concordant with filling limits of the ACIRA registry: all randomly selected records of our study sample concerned patients with SCAD or SI (except those secondarily excluded), but eight were recorded with only an enzyme elevation. Thus, the ability of our indicator to predict appropriateness could be improved by improving the database quality by cardiologists. Implementation of feedback and, more broadly, the use of this indicator could encourage cardiologists to improve the filling of the database. An advocacy for improvement and harmonization of ACIRA completion is ongoing and impact on quality of data will be further evaluated.

### Comparison with the literature

Proportions of inappropriate ICA and PCI, as estimated by experts, were comparable with previous reports [7, 8]. However, a Swiss study showed a much higher proportion of inappropriate ICA in a stable clinical context (37.5%) [22]. These results could be explained by the differences in the guidelines used. The authors considered an ICA as inappropriate only when prior non-invasive tests had not been performed, whereas according to our guidelines, some ICA could be done in first line, if the medical history of the patient indicated a pre-test probability of cardiovascular disease superior to 85%

<table><caption>**Table 4** Diagnostic performances after robustness analysis</caption><thead><tr><th>Type of robustness analysis</th><th>Sensitivity<br>Estimate, % (95% CI)</th><th>Specificity<br>Estimate, % (95% CI)</th><th>PPV<br>Estimate, % (95% CI)</th><th>NPV<br>Estimate, % (95% CI)</th></tr></thead><tbody><tr><td>Addition of 5 points to the Syntax score</td><td>66.7 (55.1-78.3)</td><td>72.0 (66.1-77.9)</td><td>40.4 (31-49.8)</td><td>71.9 (65.3-78.5)</td></tr><tr><td>Subtraction of 5 points to the Syntax score</td><td>65.1 (53.3-76.9)</td><td>75.6 (69.9-81.3)</td><td>43.2 (33.2-53.2)</td><td>88.4 (83.8-93.0)</td></tr><tr><td>Addition of the 8 files, considering them all inappropriate</td><td>66.2 (55.2-77.2)</td><td>74.2 (68.4-80.0)</td><td>45.2 (35.6-54.8)</td><td>87.2 (82.4-92.0)</td></tr><tr><td>Addition of the 8 files, considering them all appropriate</td><td>65.1 (53.3-76.9)</td><td>72.5 (66.7-80.3)</td><td>39.4 (30.1-48.7)</td><td>88.3 (83.7-92.9)</td></tr></tbody></table>

CI confidence interval
(men > 70 years old and at least two cardio-vascular risk factors, in the final indicator).

## Strength of the study

The reference standard consisted in a case review by experts, after the implementation of a NGT to improve judgment homogeneity and consensus among experts. The request of a third expert when necessary limited the classification bias regarding the definition of the reference test. The NGT has been implemented using methodological standards [20]. Attention has been paid to the independence and blindness of the experts during the case review. Expressing a consensus of good practices during the NGT allowed us recording relevant variables for the review. Efforts made through the NGT to elicit appropriate indications also reinforced validity of the reference test.

The important involvement of interventional cardiologists from Aquitaine centers was another strong component. They were representative of all types of centers, private, public, and from all around the region. Their involvement in this project makes us confident that the development of such indicators is expected to fuel discussion on clinical practices not only in Aquitaine but also elsewhere in France.

## Limits

The low accuracy of detection of inappropriate PCI is one of the major limits. Moreover, hypotheses made led to decrease indicator's accuracy. For example, all patients diagnosed with a stable angina were considered as having typical angina symptoms and being under optimal medical therapy. We assumed this happens for a large majority of patients undergoing an IC act. We plan to conduct an ad hoc study to evaluate and collect data on symptoms and treatments to confirm this hypothesis. This may have led to underestimate the proportion of inappropriate PCI, which was still higher than expected.

A lack of precision in estimations of measurement characteristics occurred because our first hypotheses on expected values on specificity and sensitivity were too optimistic. We have limited incorporation bias, which occurs when the reference standard includes the index test, by developing the reference standard independently from the indicator: experts were blinded to the indicator during the NGT and were judging appropriateness of the cases review independently of each other. Moreover, the final indicator has been developed with an independent expert. During case review, the experts knew ICA intervention results of each patient case, which could lead to an information bias. Some cardiologists may have been influenced by these results in their judgment of ICA appropriateness and may have overestimated

ICA inappropriateness in case of normal ICA results. Finally, although allowing us to better target the patients to be included in the algorithm, the exclusion of the eight records corresponding to the acute context might have overestimated the accuracy of the final indicator. These eight cases had to be excluded during the review process, because of missing relevant data preventing the experts to assess appropriateness. Once the review sessions were carried out, it was not possible to include a posteriori eight new cases to compensate for the missing data. Notably due the difficulty to re-ask cardiologists to review another batch of 8 files, given their availabilities.

## Conclusion

We developed and validated an ICA appropriateness indicator whom accuracy should improve with the improvement of ACIRA database completeness. Indeed, this study highlighted for investigators and cardiologists, the value of completing the registry to develop more complete and accurate indicators.

Developing such a tool appears as an opportunity for cardiologists to benchmark with other cardiologists at a national and international level; it could also reduce inappropriate acts that expose patients to discomfort, avoidable hospitalizations or complications, and represent heavy costs for healthcare system. By testing its transferability, the indicator could be proposed to other European interventional cardiology registries constituted according to the same scheme and variables. Another perspective is to determine structural, organizational and individual factors associated with inappropriate IC. In summary, developing and validating such computerized quality indicator allowed us to confirm the great interest shown by cardiologists to evaluate and improve their clinical practices and reduce the proportion of inappropriate acts.

## Abbreviations

AUC: Appropriate Use Criteria; CABG: Coronary Artery Bypass Graft; ESC: European Society of Cardiology; FFR: Functional Flow Reserve; FN: False Negative; FP: False Positive; TN: True Negative; TP: True Positive; ICA: Invasive Coronary Angiography; NGT: Nominal Group Technique; NPV: Negative Predictive Value; PCI: Percutaneous Coronary Intervention; PPV: Positive Predictive Value; SCAD: Stable Coronary Artery Disease.

## Acknowledgements

The ACIRA registry acknowledges all patients participating in the registry and all investigators: Dr C.Abadie, Clinique de St Augustin, Bordeaux; Dr P.Ancelin, CH de Mont de Marsan, Mont de Marsan; Dr JL.Banos, GCS de Bayonne, Bayonne; Dr MP.Benetier, ARS Nouvelle Aquitaine, Bordeaux; Dr Y.Blanchard, CH de la côte Basque, Bayonne; Dr S.Boutreux, CH de Périgueux, Périgueux; Dr V.Buhaj, CH de Périgueux, Périgueux; Dr F.Casteigt, Clinique de Caudéran les pins-francs, Bordeaux; Dr JM.Clerc, CH de Périgueux, Périgueux; Pr P.Coste, CHU de Bordeaux, Bordeaux; Dr N.Delarche, CH de Pau, Pau; Dr P.Fournier, Clinique Esquirol St Hilaire, Agen; Dr V.Gilleron, CHU de Bordeaux, Bordeaux; Dr A.Hassan, CH de Mont de Marsan, Mont de Marsan; Dr X.Jacquelin, CH de Libourne, Libourne; Dr B.Karsenty, Hôpital privé St Martin, Pessac; Dr G.Laplace,
Clinique cardiologique d'Aressy, Aressy; Dr B.Larnaudie, Clinique de Caudéran les pins-francs, Bordeaux; Dr JL.Leymarie, Clinique de St Augustin, Bordeaux; Dr N.Marque, Clinique cardiologique d'Aressy, Aressy; Dr F.Perret, Hôpital privé St Martin, Pessac; Dr JM.Perron, CH de Libourne, Libourne; Dr D.Reviron, Clinique Esquirol St Hilaire, Agen; Dr JF.Rivière, CH de Pau, Pau; Dr C.Robin, Clinique cardiologique d'Aressy, Aressy.

**Authors' contributions**

EL and FF developed concept of research, designed methods, led research and edited manuscript. FSG developed concept of research, designed methods, led research and reviewed manuscript. JLL, IJ and LF developed concept of research and designed methods. CP led research and supported data quality assessment. JPL led research and supported data management. LRS developed designed methods and reviewed manuscript. PC developed concept of research, designed methods, oversaw the research and reviewed manuscript. All authors read and approved the final manuscript.

**Funding**

The ACIRA registry is supported by funding from the Aquitaine Regional health agency.

**Availability of data and materials**

The data that support the findings of this study are available from Dr Emilie Lesaine but restrictions apply to the availability of these data, which are not publicly available. Data are however available from the authors upon reasonable request and with permission of Dr Emilie Lesaine.

**Declarations**

**Ethics approval and consent to participate**

According to French regulations applicable at the time of research (Code de la santé publique—Article R1123-21. Code de la santé publique), this study was exempted from examination by an Ethics Committee in France and did not require written or verbal informed consent to participate. ACIRA register had received authorization from the French Commission nationale de l'informatique et des libertés (authorization n° CNILdecision DR-2011-436 and DR-2012-619). According to regulations, patients received oral and written information at ACIRA inclusion about the project, its objectives and data management process and about their refusal rights and process.

**Consent for publication**

Not applicable.

**Competing interests**

The authors declare that there is no conflict of interest regarding the publication of this article.

**Author details**

¹Inserm Bordeaux Population Health, U1219, ISPED, Univ Bordeaux, Bordeaux, France. ²Medical Information Department, Bordeaux University Hospital, Bordeaux, France. ³Hôpital Cardiologique-CHU de Bordeaux, Université de Bordeaux, Pessac, France. ⁴Polyclinique Bordeaux Nord Aquitaine, Bordeaux, France. ⁵Centre Hospitalier de Périgueux, Périgueux, France. ⁶Centre Hospitalier de Pau, Pau, France. ⁷Centre Hospitalier de Mont de Marsan, Mont de Marsan, France. ⁸Clinique Saint Augustin, Bordeaux, France.

Received: 15 June 2021 Accepted: 25 April 2022

Published online: 06 May 2022

**References**

1. Brownlee S, Chalkidou K, Doust J, Elshaug AG, Glasziou P, Heath I, et al. Evidence for overuse of medical services around the world. The Lancet. 2017;390(10090):156-68.

2. Stone GW, Moses JW. How should the appropriateness of PCI be judged?: Interventional cardiology. Nat Rev Cardiol. 2011;8:544-6.

3. OCDE. Geographic Variations in Health Care: What Do We Know and What Can Be Done to Improve Health System Performance?, OECD

Health Policy Studies, Éditions OCDE, Paris. 2014. https://doi.org/10.1787/9789264216594-en.

4. Quality Indicator User Guide: Patient Safety Indicators (PSI) Composite Measures. 2021;7.

5. Patel MR, Dehmer GJ, Hirshfeld JW, Smith PK, Spertus JA. ACCF/SCAI/STS/AATS/AHA/ASNC/HFSA/SCCT 2012 Appropriate Use Criteria for Coronary Revascularization Focused Update. J Am Coll Cardiol. 2012;59:857-81.

6. CORONARY REVASCULARIZATION WRITING GROUP, Patel MR, Dehmer GJ, Hirshfeld JW, Smith PK, Spertus JA. ACCF/SCAI/STS/AATS/AHA/ASNC 2009 Appropriateness Criteria for Coronary Revascularization: A Report of the American College of Cardiology Foundation Appropriateness Criteria Task Force, Society for Cardiovascular Angiography and Interventions, Society of Thoracic Surgeons, American Association for Thoracic Surgery, American Heart Association, and the American Society of Nuclear Cardiology: Endorsed by the American Society of Echocardiography, the Heart Failure Society of America, and the Society of Cardiovascular Computed Tomography. Circulation. 2009;119:1330-52.

7. Chan PS, Patel MR, Klein LW, Krone RJ, Dehmer GJ, Kennedy K, et al. Appropriateness of percutaneous coronary intervention. Jama. 2011;306:53-61.

8. Hannan EL, Samadashvili Z, Cozzens K, Walford G, Holmes DR, Jacobs AK, et al. Appropriateness of Diagnostic Catheterization for Suspected Coronary Artery Disease in New York State. Circ Cardiovasc Interv. 2014;7:19-27.

9. Vachon B, Désorcy B, Gaboury I, Camirand M, Rodrigue J, Quesnel L, et al. Combining administrative data feedback, reflection and action planning to engage primary care professionals in quality improvement: qualitative assessment of short term program outcomes. BMC Health Serv Res. 2015;15:391.

10. Hysong SJ, Teal CR, Khan MJ, Haidet P. Improving quality of care through improved audit and feedback. Implement Sci. 2012;7:1.

11. Ludman PF, on behalf of the British Cardiovascular Intervention Society. British Cardiovascular Intervention Society Registry for audit and quality assessment of percutaneous coronary interventions in the United Kingdom. Heart. 2011;97:1293-7.

12. Jernberg T, Attebring MF, Hambraeus K, Ivert T, James S, Jeppsson A, et al. The Swedish Web-system for Enhancement and Development of Evidence-based care in Heart disease Evaluated According to Recommended Therapies (SWEDEHEART). Heart. 2010;96:1617-21.

13. Anderson HV, Shaw RE, Brindis RG, Hewitt K, Krone RJ, Block PC, et al. A contemporary overview of percutaneous coronary interventions: the American College of Cardiology-National Cardiovascular Data Registry (ACC-NCDR). J Am Coll Cardiol. 2002;39:1096-103.

14. Rangé G, Hakim R, Beygui F, Angoulvant D, Marcollet P, Godin M, et al. Incidence, delays, and outcomes of STEMI during COVID-19 outbreak: Analysis from the France PCI registry. J Am Coll Emerg Physicians Open. 2020;1:1168-76.

15. Caussin C, Escolano S, Mustafic H, Bataille S, Tafflet M, Chatignoux E, et al. Short-term exposure to environmental parameters and onset of ST elevation myocardial infarction. The CARDIO-ARSIF registry Int J Cardiol. 2015;183:17-23.

16. Lesaine E, Saillour-Glenisson F, Leymarie J-L, Jamet I, Fernandez L, Perez C, et al. The ACIRA Registry: A Regional Tool to Improve the Healthcare Pathway for Patients Undergoing Percutaneous Coronary Interventions and Coronary Angiographies in the French Aquitaine Region. Crit Pathw Cardiol. 2020;19:1-8.

17. ESC/EACTS Guidelines on myocardial revascularization. The Task Force on Myocardial Revascularization of the European Society of Cardiology (ESC) and the European Association for Cardio-Thoracic Surgery (EACTS) Developed with the special contribution of the European Association of Percutaneous Cardiovascular Interventions (EAPCI). Eur Heart J. 2014;35:2541-619.

18. Task Force Members, Montalescot G, Sechtem U, Achenbach S, Andreotti F, Arden C, et al. 2013 ESC guidelines on the management of stable coronary artery disease: The Task Force on the management of stable coronary artery disease of the European Society of Cardiology. Eur Heart J. 2013;34:2949-3003.

19. SYNTAX Score Calculator [Internet]. Available from: http://syntaxscore.org/calculator/syntaxscore/frameset.htm. Cited 29 Mar 2022

20. Van de Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health studies. Am J Public Health. 1972;62:337-42.
21. Bouyer J, Hémon D, Cordier S, Institut national de la santé et de la recherche médicale France Unité 822. Épidémiologie principes et méthodes quantitatives. Paris: Éd. Tec & doc; 2009.

22. Chmiel C, Reich O, Signorell A, Tandjung R, Rosemann T, Senn O. Appropriateness of diagnostic coronary angiography as a measure of cardiac ischemia testing in non-emergency patients–a retrospective cross-sectional analysis. PLoS ONE. 2015;10:e0117172.

**Publisher's Note**

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Ready to submit your research? Choose BMC and benefit from:

* fast, convenient online submission

* thorough peer review by experienced researchers in your field

* rapid publication on acceptance

* support for research data, including large and complex data types

* gold Open Access which fosters wider collaboration and increased citations

* maximum visibility for your research: over 100M website views per year

At BMC, research is always in progress.

Learn more biomedcentral.com/submissions

<figure><img src="image_7.png" /></figure>