# Sensitivity and specificity of an algorithm based on medico-administrative data to identify hospitalized patients with major bleeding presenting to an emergency department

Emmanuel Oger, Marie-Anne Botrel, Catherine Juchault, Jacques Bouget

► To cite this version:

Emmanuel Oger, Marie-Anne Botrel, Catherine Juchault, Jacques Bouget. Sensitivity and specificity of an algorithm based on medico-administrative data to identify hospitalized patients with major bleeding presenting to an emergency department. BMC Medical Research Methodology, 2019, 19 (1), pp.194. 10.1186/s12874-019-0841-6 . hal-02363490

HAL Id: hal-02363490

https://univ-rennes.hal.science/hal-02363490v1

Submitted on 10 Jul 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
RESEARCH ARTICLE

Open Access

<figure><img src="image_2.png" /></figure>

Check for
updates

# Sensitivity and specificity of an algorithm based on medico-administrative data to identify hospitalized patients with major bleeding presenting to an emergency department

Emmanuel Oger¹*, Marie-Anne Botrel¹, Catherine Juchault² and Jacques Bouget¹

**Abstract**

**Background:** Validation studies on an ICD-10-based algorithm to identify major bleeding events are scarce, and mostly focused on positive predictive values.

**Objective:** To evaluate the sensitivity and specificity of an ICD-10-based algorithm in adult patients referred to hospital.

**Methods:** This was a cross-sectional, retrospective analysis. Among all hospital stays of adult patients referred to Rennes University Hospital, France, through the emergency ward in 2014, we identified major bleeding events according to an index test based on a list of ICD-10 diagnoses. As a reference, a two-step process was applied: firstly, a computerized request for electronic health records from the emergency ward, using several hemorrhage-related diagnostic codes and specific emergency therapies so as to discard stays with a very low probability of bleeding; secondly, a chart review of selected records was conducted by a medical expert blinded to the index test results and each hospital stay was classified into one of two exclusive categories: major bleeding or no major bleeding, according to pre-specified criteria.

**Results:** Out of 16,012 hospital stays, the reference identified 736 major bleeding events and left 15,276 stays considered as without the target condition. The index test identified 637 bleeding events: 293 intracranial hemorrhages, 197 gastrointestinal hemorrhages and 147 other bleeding events. Overall, sensitivity was 65% (95%CI, 62 to 69), and specificity was 99.0%. We observed differential sensitivity and specificity across bleeding types, with the highest values for intracranial hemorrhage. Positive predictive values ranged from 59% for "other" bleeding events, to 71% (95%CI, 65 to 78) for gastrointestinal hemorrhage, and 96% for intracranial hemorrhage.

**Conclusions:** Low sensitivity and differential measures of accuracy across bleeding types support the need for specific data collection and medical validation rather than using an ICD-10-based algorithm for assessing the incidence of major bleeding.

**Keywords:** Validation study, ICD-10, Algorithm, Major bleeding

<footnote>* Correspondence: emmanuel.oger@univ-rennes1.fr
¹Univ Rennes, EA 7449 REPERES [Pharmacoepidemiology and Health Services Research], Rennes, France
Full list of author information is available at the end of the article</footnote>
## Background

A reimbursement claim database enables large cohorts to be set up, providing comprehensive data at a relatively low cost [1]. Their use in pharmacoepidemiology has considerably increased in recent years [2]. In France, numerous studies have been conducted using the French National Health database (SNDS, previously known as SNIIRAM) [3-7]. The French hospital database (PMSI), part of the SNDS, provides a discharge diagnosis (ICD-10 codes) for all patients admitted to hospital in France. It is considered that hospital-based data, and discharge codes in particular, can be used as valuable sources of information to define patient populations, assess comorbidities [8] or the severity of disease, determine patient outcomes [3] and drug effectiveness [4], and detect adverse events, including major bleeding [5-7].

Major bleeding is the most feared serious adverse reaction when using antithrombotic agents. Estimating the occurrence of major bleeding events is therefore a key issue. Patients presenting major bleeding are mostly referred to hospital, which therefore makes hospital-based data useful. Nonetheless, caution is needed regarding the accuracy of codes in hospital-based data for major bleeding event identification: for instance when assigning primary or secondary discharge diagnoses, a focus on the reimbursement of the care delivered could hide the real reason for admission; coding inaccuracies or inconsistencies can occur across care sites, and bleeding events that are not coded could be overlooked; hence a validated algorithm is crucial.

Emergency wards are obviously optimal settings to observe and report serious adverse drug reactions for drugs prescribed in the community. With regard to major bleeding events, the validation of a hospital-based data algorithm could gain from a confrontation with medical charts from emergency wards.

## Methods

### Aim, design and setting

Our objective was to evaluate the sensitivity and specificity of an ICD-10-based algorithm that has already been used [5] to identify major bleeding events in adult patients referred to hospital in a dedicated dataset.

This was a cross-sectional, retrospective analysis conducted at Rennes University Hospital, a tertiary care facility.

### Study population

All hospital stays of adult patients referred to Rennes University Hospital through emergency ward between 01/01/2014 and 12/31/2014 were identified through the hospital registry and were eligible to participate.

## Index test

We used a list of ICD-10 primary hospital discharge diagnosis codes previously published [5]: hospitalization for bleeding, including intracranial (hospital discharge ICD-10 codes I60, I61, I62, S06.3, S06.4, S06.5, S06.6), gastrointestinal (I85.0, K25.0, K25.2, K25.4, K25.6, K26.0, K26.2, K26.4, K26.6, K27.0, K272, K27.4, K27.6, K28.0, K282, K28.4, K28.6, K29.0, K62.5, K92.0, K92.1, K92.2) and other bleeding (D62, N02, R31, R58, H11.3, H35.6, H43.1, H45.0, H92.2, J94.2, K66.1, M25.0, N92.0, N92.1, N92.4, N93.8, N93.9, N95.0, R04.0, R04.1, R04.2, R04.8, R04.9). See Additional file 1 for details on labeling. Each hospital stay was classified into one of four exclusive categories: intracranial, gastrointestinal, other bleeding or no bleeding event. We chose this list because it was derived from and had already been used on French hospital-based data.

## Standard procedure

To provide a reference, a standard two-step process was applied to all hospital stays: firstly, a computerized request for electronic health records from emergency wards using several hemorrhage-related diagnostic codes (see Additional file 2) and specific emergency therapies (red blood cell transfusion, platelet transfusion, vitamin K, protamin sulfate, prothrombin complex concentrate, and FEIBA*, an anti-inhibitor coagulant complex); this request demonstrated good sensitivity (96, 95% exact confidence limits (CL) 80 to 99%) and specificity (100%, exact 95% CL, 99 to 100%) in a pilot study. The probability of bleeding in the discarded records (i.e. records not identified by this request) was consequently considered to be probably very low. Secondly, a review of the selected records was conducted by a medical expert (JB) blinded to index test results, and each hospital stay was classified in one of two exclusive categories: major bleeding or no major bleeding. Major bleeding was defined on at least one of the following criteria in the review of medical charts: unstable hemodynamic (systolic arterial pressure < 90 mmHg or mean arterial pressure < 65 mmHg) or shock, uncontrollable bleeding, need for transfusions or hemostatic procedure (embolization, endoscopic procedure, surgery), and life-threatening locations such as intracranial, intra-spinal, intraocular, retroperitoneal, pericardial, thoracic, intra-articular, intramuscular hematoma with compartment syndrome, acute gastrointestinal bleeding. We considered major bleeding in case of epistaxis when at least two procedures of nasal packing were needed, and in case of hematuria when bleeding continued for more than 12 h despite bladder washing. An extensive chart review was simply unrealistic, because of the expense involved in reviewing so many charts. Of course, medical review of all charts would have been the true reference standard.
## Statistical analysis

From contingency tables, indicators of diagnostic accuracy, sensitivity, specificity, negative and positive predictive values, and positive and negative likelihood ratios were calculated along with an exact 95% confidence interval, using SAS 9.4 software (SAS Institute, Cary, NC., USA). Briefly, sensitivity or true positive rate is the number of positive index test (true positive) out of all stays with a target condition as defined by the standard procedure (true positive plus false negative). Specificity also called the true negative rate is the number of negative index test (true negative) that are identified as negative by the standard procedure (true negative plus false positive).

## Results

### Study population

Between 01/01/2014 and 12/31/2014 at Rennes University Hospital there were 49,792 emergency ward records and 22,400 hospital stays for adult patients. Out of these, we identified 16,012 hospital stays for adult patients with a hospital admission through emergency ward. Among these, the mean (SD) age was 61.7 (22.7) years and 52.2% were men.

### Index test

Across 16,012 hospital stays, the previously published algorithm [5] based on the hospital discharge main diagnosis, identified 637 bleeding events (Table 1). There were 293 intracranial hemorrhages (46%), 197 gastrointestinal hemorrhages (31%) and 147 other bleeding events (23%).

### Standard procedure

The automated first step identified 1959 records from the 16,012 eligible hospital stays, of which 736 were classified as major bleeding events by a medical expert review. All other stays (n = 15,276) were considered as without the target condition (1223 classified as such by medical expert review and 14,053 discarded by the automated first step).

### Main outcomes

From the contingency table (Table 2) we derived the diagnostic performances (Table 3 and Fig. 1): there were 482 true positive index test results (positive index test - either ICH, GI or other bleeding according to the ICD-10 based algorithm - among those stays classified as having the target condition by the standard procedure), 15,121 true negative (negative index test among those stays

<table><caption>**Table 1** Discharge diagnosis ICD-10 codes for 637 events identified by index test [5]</caption><thead><tr><th>Type</th><th>Code</th><th>Label</th><th>Percent</th><th>Count</th></tr></thead><tbody><tr><td rowspan="4">Intracranial hemorrhage</td><td>I61</td><td>Non-traumatic intra-cerebral hemorrhage</td><td>45.7</td><td>134</td></tr><tr><td>S06</td><td>Intracranial injury</td><td>34.8</td><td>102</td></tr><tr><td>I60</td><td>Non-traumatic subarachnoid hemorrhage</td><td>15.0</td><td>44</td></tr><tr><td>I62</td><td>Other and unspecified non-traumatic intracranial hemorrhage</td><td>4.43</td><td>13</td></tr><tr><td rowspan="8">Gastrointestinal hemorrhage</td><td>K92</td><td>Other diseases of the digestive system</td><td>39.1</td><td>77</td></tr><tr><td>K62</td><td>Other diseases of the anus and rectum</td><td>35.0</td><td>69</td></tr><tr><td>K26</td><td>Duodenal ulcer</td><td>12.2</td><td>24</td></tr><tr><td>K25</td><td>Gastric ulcer</td><td>11.7</td><td>23</td></tr><tr><td>K28</td><td>Gastrojejunal ulcer</td><td>1.01</td><td>2</td></tr><tr><td>K27</td><td>Peptic ulcer, site unspecified</td><td>0.51</td><td>1</td></tr><tr><td>K29</td><td>Gastritis and duodenitis</td><td>0.51</td><td>1</td></tr><tr><td rowspan="12">Other bleeding</td><td>R04</td><td>Hemorrhage in the respiratory tract</td><td>48.9</td><td>72</td></tr><tr><td>R31</td><td>Hematuria</td><td>26.5</td><td>39</td></tr><tr><td>D62</td><td>Acute post-hemorrhagic anemia</td><td>10.9</td><td>16</td></tr><tr><td>M25</td><td>Other joint disorder, not elsewhere classified</td><td>5.44</td><td>8</td></tr><tr><td>H11</td><td>Ocular hemorrhage</td><td>1.36</td><td>2</td></tr><tr><td>K66</td><td>Other disorders of the peritoneum</td><td>1.36</td><td>2</td></tr><tr><td>N02</td><td>Recurrent and persistent hematuria</td><td>1.36</td><td>2</td></tr><tr><td>N95</td><td>Menopausal and other peri-menopausal disorders</td><td>1.36</td><td>2</td></tr><tr><td>R58</td><td>Hemorrhage, not elsewhere classified</td><td>1.36</td><td>2</td></tr><tr><td>H92</td><td>Otalgia and effusion from the ear</td><td>0.68</td><td>1</td></tr><tr><td>J94</td><td>Other pleural conditions</td><td>0.68</td><td>1</td></tr></tbody></table>
<table><caption>**Table 2** Contingency table by type of hemorrhage</caption><thead><tr><th rowspan="2">Index test</th><th colspan="3">Standard procedure</th></tr><tr><th>Positive</th><th>Negative</th><th>Total</th></tr></thead><tbody><tr><td>ICH</td><td>281</td><td>12</td><td>293</td></tr><tr><td>GI</td><td>141</td><td>56</td><td>197</td></tr><tr><td>Other</td><td>60</td><td>87</td><td>147</td></tr><tr><td>Negative</td><td>254</td><td>15,121</td><td>15,375</td></tr><tr><td>Total</td><td>736</td><td>15,276</td><td>16,012</td></tr></tbody></table>

The index test is an ICD-10-based algorithm (previously published [5])

ICH denotes intracranial hemorrhage, GI Gastrointestinal

classified as not having the target condition by the standard procedure), 155 false positive (positive index test among those stays classified as not having the target condition by the standard procedure) and 254 false negative (negative index test among those stays classified as having the target condition by the standard procedure); sensitivity (TP/TP + FN) and specificity (TN/TN + FP) varied across types of major bleeding, with the highest values for intracranial hemorrhage.

### Description of false positives

Out of 155 hospital stays, 141 (91%) were identified by the automated first step, thereby indicating there was potentially a hemorrhage, but seriousness was ruled out by the medical expert review. It can be noted that these cases were mainly “other” bleeding events (62%) and gastrointestinal bleeding (37.4%); only 8 cases had ICD-10 codes for intracranial bleeding, 3 of them had code S06.5 (traumatic subdural hemorrhage) and another code S06.6 (traumatic subarachnoid hemorrhage). For the remaining 14 hospital stays, the ICD-10-based algorithm identified 3 “other” bleeding events, 6 gastrointestinal bleedings and 5 intracranial bleedings. Additional file 3 shows the diagnosis as coded in emergency ward and the main discharge diagnosis in order address the question of in-hospital bleeding as opposed to bleeding as the motive for referral, or no bleeding at all for these stays. For all intracranial bleedings, symptoms as coded in emergency wards were consistent with the main discharge diagnosis; this was not the case for gastrointestinal bleeding and other bleeding events where in-hospital bleeding may have occurred.

<table><caption>**Table 3** Accuracy of a previously used [5] ICD-10-based algorithm for major bleeding</caption><thead><tr><th>N</th><th>True positive</th><th>True negative</th><th>False positive</th><th>False negative</th><th>Sensitivity % (95% CI)</th><th>Specificity % (95% CI)</th><th>PPV % (95% CI)</th><th>NPV % (95% CI)</th><th>Positive LR % (95% CI)</th><th>Negative LR % (95% CI)</th></tr></thead><tbody><tr><td>16,012</td><td>482</td><td>15,121</td><td>155</td><td>254</td><td>65.5 (61.9 to 68.9)</td><td>99.0 (98.8 to 99.1)</td><td>75.7 (72.1 to 78.9)</td><td>98.3 (98.1 to 98.5)</td><td>64.5 (21.8 to &infin;)</td><td>0.35 (0.00 to 2.08)</td></tr></tbody></table>

PPV and NPV denote positive and negative predictive values respectively, LR denotes likelihood ratio, CI denotes Clopper-Pearson's (exact) confidence interval

### Description of false negatives

Diagnoses (ICD-10 codes) as coded in emergency ward are shown in Table 4. Most of them (18 codes, totaling 107 stays, 42%) were codes used by the ICD-10-based algorithm. Additional file 4 shows the main discharge diagnoses retained for these 107 stays: the codes were mostly based on etiology. The main discharge diagnosis codes mostly (75%) related to four chapters of the ICD-10 classification: S (injury, n = 94, 37%), D (n = 34, 13.4%, mostly diseases of the blood rather than neoplasms), I (n = 34, 13.4%), and K (n = 30, 11.8%).

### Discussion

First, we observed overall low sensitivity (65%); a third of the major bleeding events identified by our standard procedure were not detected by the ICD-10-based algorithm (false negatives). Second, we highlighted differential sensitivity and specificity across types of bleeding, with the highest values for intracranial hemorrhage. One hundred fifty-five cases out of 637 (24%) were false positives, with a large majority being non-serious, involving mostly “other” bleeding and GI bleeding events.

Major bleeding is an adverse event common to all types of antithrombotic drugs. An automated approach to identifying major bleeding in real time would be particularly useful. Indeed it would enable continuous monitoring and early signal detection in the area of pharmacovigilance. However, accuracy in coding for major bleeding is required to yield trustworthy results on the basis of hospital databases. To date, validation studies on ICD-10 code-based algorithms are scarce and focused on positive predictive values: only one study [9] identified major bleeding from emergency ward discharges using 35 ICD-10 codes for ICH or GI bleeding; a random sample was independently reviewed by two trained chart reviewers to validate the diagnosis, but no criteria to define major bleeding were applied, except for the location; the analysis showed an overall good positive predictive value of 88% (95%CI 83 to 91), with better estimates for ICH (90%) than for upper GI bleeding (74%). It can be noted that other forms of major bleeding events were not studied. These results based on emergency ward discharges were similar to others from studies using hospital discharge records (ICD-10 codes): Kokotailo et al. reported a positive predictive value of 98 and 91% for ICH and subarachnoid hemorrhage respectively [10]. Cunningham et al. showed
<figure><img src="image_3.png" /><figcaption>**Fig. 1** Accuracy of the previously used [5] ICD-10-based algorithm according to type of major bleeding. PPV and NPV denote positive and negative predictive values respectively</figcaption></figure>

that an algorithm identifying bleeding-related hospitalizations from the primary discharge diagnosis had a positive predictive value of between 89 and 99% in distinguishing specific bleeding sites [11]. The accuracy of upper GI bleeding codes in one Dutch administrative database using the ICD-10 coding system showed a positive predictive value of 77% [12]. In an assessment of four different coding systems in different European countries, it was concluded that positive predictive value is associated not only with the code itself, but is also with the way the code is

used [12]; France was not part of this study. In France, main discharge diagnoses may not reflect the motive for referral, but rather what most impacts hospital resources; indeed the PMSI database has a primarily financial objective, not an epidemiological point of view. Lastly, Delate et al. recommended a manual chart review to validate warfarin-related bleeding events from administrative data [13]. Ruigomez et al. also advocated additional information to prevent misclassification as regards major GI or urogenital bleeding events [14]. Our findings are in line

<table><caption>**Table 4** The most cited diagnoses (> 2%) in emergency ward out of 254 false negatives on the index test (an ICD-10-based algorithm, previously used [5])</caption><thead><tr><td>Codes</td><td>Label</td><td>Percent</td><td>Count</td></tr></thead><tbody><tr><td>K62.5</td><td>Hemorrhage of the anus and rectum</td><td>10.2</td><td>26</td></tr><tr><td>D64.9</td><td>Anemia, unspecified</td><td>7.87</td><td>20</td></tr><tr><td>R57.1</td><td>Hypovolemic shock</td><td>6.29</td><td>16</td></tr><tr><td>S39.0</td><td>Injury of muscles, fascia and tendon of abdomen, lower back and pelvis</td><td>5.90</td><td>15</td></tr><tr><td>K92.0</td><td>Hematemesis</td><td>5.51</td><td>14</td></tr><tr><td>I61.9</td><td>Nontraumatic intracerebral hemorrhage, unspecified</td><td>4.72</td><td>12</td></tr><tr><td>S00.0</td><td>Superficial injury of the scalp</td><td>3.15</td><td>8</td></tr><tr><td>S06.3</td><td>Focal traumatic brain injury</td><td>3.15</td><td>8</td></tr><tr><td>K66.1</td><td>Hemoperitoneum</td><td>2.36</td><td>6</td></tr><tr><td>K92.1</td><td>Melena</td><td>2.36</td><td>6</td></tr><tr><td>R31</td><td>Hematuria</td><td>2.36</td><td>6</td></tr><tr><td>T81.0</td><td>Complications of procedures, not elsewhere classified</td><td>2.36</td><td>6</td></tr></tbody></table>

Codes used by the ICD-10-based algorithm (but on discharge diagnoses) are in italics
with these previous results and have highlighted differen-
tial measures of accuracy between ICH and GI. When
evaluating anticoagulant safety profiles, it would be wise
to perform separate analyses according to these outcomes.
It is well known that non-differential misclassification
biases the risk towards the null, but it can be noted that
NOAC trials reported a lower risk of ICH but no signifi-
cant decrease of GI bleeding; hence differential positive
predictive values could be problematic when evaluating
overall safety profiles.

The reported low sensitivity is a concern. This result
might reflect a discrepancy between the motive for refer-
ral and the discharge coding (false negatives); it is worth
noting that the PMSI does not collect emergency ward
data; in any case, emergency ward data is thought to be
unreliable with considerable inconsistency as a result of
a lack of standardization. Only primary hospital dis-
charge diagnoses were used in the index test to define
major bleeding [5]. The consequence when using this al-
gorithm (index test) will be an underestimation of the
incidence of major bleeding. It has already been ob-
served that when the code is listed as the most likely
diagnosis or the admission diagnosis, a true bleeding
event has occurred 96% of the time [15].

False positives from the algorithm (index test) were mostly non serious gastrointestinal bleeding or non serious other bleeding events. The point here is whether the algorithm catches in-hospital serious bleeding. This is important to consider when the question focuses on serious bleeding as a reason for hospital referral, which means bleeding potentially related to drug delivered on an ambulatory basis. Including in the analysis in-hospital serious bleeding might biased the results because at this time patients may not be exposed to the drug they were prescribed before hospital entry.

Our study has several strengths. Our sample size is by
far the largest among studies testing the validity of ICD
codes. We reviewed a consecutive set of charts irrespective
of the ICD-10 coding allocated. Using negative controls,
we calculated the sensitivity. In contrast, previous studies
have been published without negative controls and have
only reported positive predictive values. The medical chart
review was blinded to the discharge diagnoses.

Our study also has several limitations. Firstly, the
study was carried out by one medical expert reviewer in
a single center. However, the reviewer followed objective
criteria to determine the presence of major bleeding. On
the other hand, inter-rater variability related to different
level of expertise would have been an issue. Secondly,
our definition for major bleeding was conservative, and
it is likely that we underestimated the numbers of cer-
tain major bleeding events, especially with respect to the
ISTH definition [16], which includes a drop in
hemoglobin level of 20 g L⁻¹. The PMSI database does

not include laboratory results. In addition, to take ac-
count of a drop in hemoglobin level, a reference level is
required, which is not straightforward. Our definition
applied only to hospital-based care. Therefore, data from
individuals who do not seek medical attention or who
are only seen in outpatient clinics or surgeries was not
captured. For major bleeding, we thought this would not
lead to a substantial bias, except for bleeding-related
sudden death. Thirdly, we used a two-step approach as
the standard procedure; an extensive chart review was
simply unrealistic, because of the expense involved in
reviewing so many charts. A previous pilot study has
shown good sensitivity and specificity for the first auto-
mated step. Of course, medical review of all charts
would have been the true reference standard.

Conclusion

To conclude, the external validity of bleeding diagnostic
codes has not been previously assessed in the French
PMSI database. To the best of our knowledge, this is the
first report. Our results showed overall low sensitivity,
and, interestingly, different measures of accuracy across
bleeding types. The results therefore provide support for
specific data collection and a medical validation approach
rather than an ICD-10-based algorithm for assessing the
incidence of major bleeding.

**Supplementary information**

**Supplementary information** accompanies this paper at https://doi.org/10.1186/s12874-019-0841-6.

**Additional file 1.** List of ICD-10 primary hospital diagnostic discharge previously published [5].

**Additional file 2.** Codes used by computerized requests made on electronic health records from emergency ward.

**Additional file 3.** Diagnosis as coded in emergency ward and main discharge diagnosis for 14 hospital stays (not identify as bleeding event by computerized request on electronic health records from emergency ward using hemorrhagic-related diagnostic codes and specific emergency therapies) out of 155 false positives.

**Additional file 4.** Main discharge diagnoses compared to emergency ward (EW) ICD-10 code for 107 stays where emergency ward diagnosis was coded as hemorrhage but not the main discharge diagnosis (false negative).

Abbreviations

ICD-10: International Classification of Disease, 10th edition; NPV: Negative predictive value; PMSI: French hospital database; PPV: Positive predictive value; SNDS: French National Health database

Acknowledgements

Not applicable.

Authors' contributions

CJ and MAB gathered data. EO analyzed the data. EO and JB interpreted the data. EO was a major contributor in writing the manuscript. All authors read and approved the final manuscript.

Funding

No external source of funding.
**Availability of data and materials**

The dataset used during the current study is available from the
corresponding author on reasonable request.

**Ethics approval and consent to participate**

This study was authorized by the local ethics committee, named Comité d'éthique du CHU de Rennes (Rennes University Hospital), committee's reference number 13.06. Data used in this study was anonymised before its use.

**Consent for publication**

Not applicable

**Competing interests**

The authors declare that they have no competing interests.

**Author details**

¹Univ Rennes, EA 7449 REPERES [Pharmacoepidemiology and Health Services Research], Rennes, France. ²Department of Medical Information, CHU Rennes, Rennes, France.

Received: 30 April 2019 Accepted: 24 September 2019

Published online: 18 October 2019

**References**

1. O'Malley KJ, Cook KF, Price MD, Wildes KR, Hurdle JF, Ashton CM. Measuring diagnoses: ICD code accuracy. Health Serv Res. 2005;40:1620-39.

2. Gavrielov-Yusim N, Friger M. Use of administrative medical databases in population-based research. J Epidemiol Community Health. 2014;68:283-7.

3. Tuppin P, Cuerq A, de Peretti C, Fagot-Campagna A, Danchin N, Juillière Y, et al. Two-year outcome of patients after a first hospitalization for heart failure: a national observational study. Arch Cardiovasc Dis. 2014;107:158-68.

4. Neumann A, Maura G, Weill A, Ricordeau P, Alla F, Allemand H. Comparative effectiveness of rosuvastatin versus simvastatin in primary prevention among new users: a cohort study in the French national health insurance database: comparative effectiveness of rosuvastatine versus simvastatine. Pharmacoepidemiol Drug Saf. 2014;23:240-50.

5. Maura G, Blotière P-O, Bouillon K, Billionnet C, Ricordeau P, Alla F, et al. Comparison of the short-term risk of bleeding and arterial thromboembolic events in nonvalvular atrial fibrillation patients newly treated with dabigatran or rivaroxaban versus vitamin K antagonists: a French nationwide propensity-matched cohort study. Circulation. 2015;132:1252-60.

6. Bouillon K, Bertrand M, Maura G, Blotière P-O, Ricordeau P, Zureik M. Risk of bleeding and arterial thromboembolism in patients with non-valvular atrial fibrillation either maintained on a vitamin K antagonist or switched to a non-vitamin K-antagonist oral anticoagulant: a retrospective, matched-cohort study. Lancet Haematol. 2015;2(4):e150-9.

7. Bouillon K, Bertrand M, Boudali L, Ducimetière P, Dray-Spira R, Zureik M. Short-term risk of bleeding during heparin bridging at initiation of vitamin K antagonist therapy in more than 90 000 patients with nonvalvular atrial fibrillation managed in outpatient care. J Am Heart Assoc. 2016;5(11).

8. Quan H, Li B, Couris CM, Fushimi K, Graham P, Hider P, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol. 2011;173:676-82.

9. Al-Ani F, Shariff S, Siqueira L, Seyam A, Lazo-Langner A. Identifying venous thromboembolism and major bleeding in emergency room discharges using administrative data. Thromb Res. 2015;136:1195-8.

10. Kokotailo RA, Hill MD. Coding of stroke and stroke risk factors using international classification of diseases, revisions 9 and 10. Stroke. 2005;36:1776-81.

11. Cunningham A, Stein CM, Chung CP, Daugherty JR, Smalley WE, Ray WA. An automated database case definition for serious bleeding related to oral anticoagulant use. Pharmacoepidemiol Drug Saf. 2011;20(6):560-6.

12. Valkhoff VE, Coloma PM, Masclee GM, Gini R, Innocenti F, Lapi F, on behalf of the EU-ADR consortium, et al. Validation study in four health-care databases: upper gastrointestinal bleeding misclassification affects precision but not magnitude of drug-related upper gastrointestinal bleeding risk. J Clin Epidemiol. 2014;67:921-31.

13. Delate T, Jones AE, Clark NP, Witt DM. Assessment of the coding accuracy of warfarin-related bleeding events. Thromb Res. 2017;159:86-90.

14. Ruigómez A, Brobert G, Suzart-Woischnik K, García-Rodríguez LA. Ascertainment and validation of major bleeding events in a primary care database. Pharmacoepidemiol Drug Saf. 2019;28(2):148-55.

15. Arnason T, Wells PS, Van WC, Forster AJ. Accuracy of coding for possible warfarin complications in hospital discharge abstracts. Thromb Res. 2006;118:253-62.

16. Schulman S, Karon C, for the Subcommittee on Control of Anticoagulation of the Scientific and Standardization Committee of the International Society on Thrombosis and Haemostasis. Definition of major bleeding in clinical investigations of antihemostatic medicinal products in non-surgical patients. J Thromb Haemost. 2005;3(4):692-4.

**Publisher's Note**

Springer Nature remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.

Ready to submit your research? Choose BMC and benefit from:

* fast, convenient online submission

* thorough peer review by experienced researchers in your field

* rapid publication on acceptance

* support for research data, including large and complex data types

* gold Open Access which fosters wider collaboration and increased citations

* maximum visibility for your research: over 100M website views per year

At BMC, research is always in progress.

Learn more biomedcentral.com/submissions