# Analysis of applying a patient safety taxonomy to patient and clinician-reported incident reports during the COVID-19 pandemic: a mixed methods study

Thomas Purchase, Alison Cooper, Delyth Price, Emma Dorgeat, Huw Williams, Paul Bowie, Jean-Pascal Fournier, Peter Hibbert, Adrian Edwards, Rhiannon Phillips, et al.

► To cite this version:

Thomas Purchase, Alison Cooper, Delyth Price, Emma Dorgeat, Huw Williams, et al. Analysis of applying a patient safety taxonomy to patient and clinician-reported incident reports during the COVID-19 pandemic: a mixed methods study. BMC Medical Research Methodology, 2023, 23, pp.234. 10.1186/s12874-023-02057-6 . hal-04337839

HAL Id: hal-04337839

https://nantes-universite.hal.science/hal-04337839

Submitted on 4 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Analysis of applying a patient safety taxonomy to patient and clinician-reported incident reports during the COVID-19 pandemic: a mixed methods study

Thomas Purchase, Alison Cooper, Delyth Price, Emma Dorgeat, Huw Williams, Paul Bowie, Jean-Pascal Fournier, Peter Hibbert, Adrian Edwards, Rhiannon Phillips, et al.

► To cite this version:

Thomas Purchase, Alison Cooper, Delyth Price, Emma Dorgeat, Huw Williams, et al. Analysis of applying a patient safety taxonomy to patient and clinician-reported incident reports during the COVID-19 pandemic: a mixed methods study. BMC Medical Research Methodology, 2023, 23, pp.234. 10.1186/s12874-023-02057-6. doi:10.1186/s12874-023-02057-6

HAL Id: hal-04337839

https://nantes-universite.hal.science/hal-04337839v1

Submitted on 4 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
RESEARCH

Open Access

<figure><img src="image_2.png" /></figure>

Check for
updates

# Analysis of applying a patient safety taxonomy to patient and clinician-reported incident reports during the COVID-19 pandemic: a mixed methods study

Thomas Purchase¹, Alison Cooper¹,², Delyth Price¹,², Emma Dorgeat¹, Huw Williams³, Paul Bowie⁴,⁵,
Jean-Pascal Fournier⁶, Peter Hibbert⁷,⁸, Adrian Edwards¹,², Rhiannon Phillips⁹, Natalie Joseph-Williams¹,² and
Andrew Carson-Stevens¹,²*

## Abstract

**Background** The COVID-19 pandemic resulted in major disruption to healthcare delivery worldwide causing medical services to adapt their standard practices. Learning how these adaptations result in unintended patient harm is essential to mitigate against future incidents. Incident reporting and learning system data can be used to identify areas to improve patient safety. A classification system is required to make sense of such data to identify learning and priorities for further in-depth investigation. The Patient Safety (PISA) classification system was created for this purpose, but it is not known if classification systems are sufficient to capture novel safety concepts arising from crises like the pandemic. We aimed to review the application of the PISA classification system during the COVID-19 pandemic to appraise whether modifications were required to maintain its meaningful use for the pandemic context.

**Methods** We conducted a mixed-methods study integrating two phases in an exploratory, sequential design. This included a comparative secondary analysis of patient safety incident reports from two studies conducted during the first wave of the pandemic, where we coded patient-reported incidents from the UK and clinician-reported incidents from France. The findings were presented to a focus group of experts in classification systems and patient safety, and a thematic analysis was conducted on the resultant transcript.

**Results** We identified five key themes derived from the data analysis and expert group discussion. These included capitalising on the unique perspective of safety concerns from different groups, that existing frameworks do identify priority areas to investigate further, the objectives of a study shape the data interpretation, the pandemic spotlighted long-standing patient concerns, and the time period in which data are collected offers valuable context to aid explanation. The group consensus was that no COVID-19-specific codes were warranted, and the PISA classification system was fit for purpose.

Full list of author information is available at the end of the article

<footnote>*Correspondence:
Andrew Carson-Stevens
carson-stevensap@cardiff.ac.uk</footnote>
RESEARCH

Open Access

<figure><img src="image_2.png" /></figure>

Check for
updates

# Analysis of applying a patient safety taxonomy to patient and clinician-reported incident reports during the COVID-19 pandemic: a mixed methods study

Thomas Purchase¹, Alison Cooper¹,², Delyth Price¹,², Emma Dorgeat¹, Huw Williams³, Paul Bowie⁴,⁵,
Jean-Pascal Fournier⁶, Peter Hibbert⁷,⁸, Adrian Edwards¹,², Rhiannon Phillips⁹, Natalie Joseph-Williams¹,² and
Andrew Carson-Stevens¹,²*

## Abstract

**Background** The COVID-19 pandemic resulted in major disruption to healthcare delivery worldwide causing medical services to adapt their standard practices. Learning how these adaptations result in unintended patient harm is essential to mitigate against future incidents. Incident reporting and learning system data can be used to identify areas to improve patient safety. A classification system is required to make sense of such data to identify learning and priorities for further in-depth investigation. The Patient Safety (PISA) classification system was created for this purpose, but it is not known if classification systems are sufficient to capture novel safety concepts arising from crises like the pandemic. We aimed to review the application of the PISA classification system during the COVID-19 pandemic to appraise whether modifications were required to maintain its meaningful use for the pandemic context.

**Methods** We conducted a mixed-methods study integrating two phases in an exploratory, sequential design. This included a comparative secondary analysis of patient safety incident reports from two studies conducted during the first wave of the pandemic, where we coded patient-reported incidents from the UK and clinician-reported incidents from France. The findings were presented to a focus group of experts in classification systems and patient safety, and a thematic analysis was conducted on the resultant transcript.

**Results** We identified five key themes derived from the data analysis and expert group discussion. These included capitalising on the unique perspective of safety concerns from different groups, that existing frameworks do identify priority areas to investigate further, the objectives of a study shape the data interpretation, the pandemic spotlighted long-standing patient concerns, and the time period in which data are collected offers valuable context to aid explanation. The group consensus was that no COVID-19-specific codes were warranted, and the PISA classification system was fit for purpose.

Full list of author information is available at the end of the article

<footnote>*Correspondence:
Andrew Carson-Stevens
carson-stevensap@cardiff.ac.uk</footnote>
**Conclusions** We have scrutinised the meaningful use of the PISA classification system's application during a period of systemic healthcare constraint, the COVID-19 pandemic. Despite these constraints, we found the framework can be successfully applied to incident reports to enable deductive analysis, identify areas for further enquiry and thus support organisational learning. No new or amended codes were warranted. Organisations and investigators can use our findings when reviewing their own classification systems.

**Keywords** COVID-19, Patient safety, Incident reporting, PISA, Taxonomy, Classification, Systems learning

**Background**

The COVID-19 pandemic resulted in major disruption to the delivery of healthcare worldwide [1], causing medical services to rapidly adapt their standard practices. Organisations need to learn from the global body of evidence around the impact of COVID-19 to develop strategies that not only focus on short-term recovery, but on the development of sustainable and resilient healthcare systems equipped to withstand similar future events [2]. Learning how such healthcare system adaptations result in unintended consequences and lead to patient safety incidents is essential to mitigate future healthcare-associated harms [3].

Since the early 2000s, healthcare organisations have increasingly used patient incident reporting and learning systems to learn from patient safety incidents and empirically identify opportunities to improve patient safety [4]. The primary function of these reporting systems is to harness the results of data analysis to guide system improvements [5]. Reporting systems form a vital part of routine monitoring of clinical practice [6] and can

improve the quality of care delivered by healthcare services through the identification of common safety issues, helping to prioritise funding and resources, and the development of safety interventions [7].

Classification systems are important tools for providing structure to meaningful reporting and data analysis [8] and help to make sense of the complex nature of patient safety incidents [9]. A taxonomy is required to standardise these classifications into a hierarchical form to aid categorisation and organisation [10]. Figure 1 illustrates how organisational learning benefits from classification systems.

Previously, inconsistent definitions used in patient safety literature impeded international learning [11]. In 2009, the World Health Organization (WHO) responded to this need to standardise the terms used in classifying patient safety by developing the International Classification for Patient Safety (ICPS) conceptual framework [12]. The ICPS framework allows for greater comparisons to be made between countries and care settings because of a more organised, accepted, and understandable

<figure><img src="image_3.png" /><figcaption>**Fig. 1** Steps to learning from patient safety incidents. Key steps related to our aims and implementation of the PISA classification system are outlined in bold</figcaption></figure>
**Conclusions** We have scrutinised the meaningful use of the PISA classification system's application during a period of systemic healthcare constraint, the COVID-19 pandemic. Despite these constraints, we found the framework can be successfully applied to incident reports to enable deductive analysis, identify areas for further enquiry and thus support organisational learning. No new or amended codes were warranted. Organisations and investigators can use our findings when reviewing their own classification systems.

**Keywords** COVID-19, Patient safety, Incident reporting, PISA, Taxonomy, Classification, Systems learning

**Background**

The COVID-19 pandemic resulted in major disruption to the delivery of healthcare worldwide [1], causing medical services to rapidly adapt their standard practices. Organisations need to learn from the global body of evidence around the impact of COVID-19 to develop strategies that not only focus on short-term recovery, but on the development of sustainable and resilient healthcare systems equipped to withstand similar future events [2]. Learning how such healthcare system adaptations result in unintended consequences and lead to patient safety incidents is essential to mitigate future healthcare-associated harms [3].

Since the early 2000s, healthcare organisations have increasingly used patient incident reporting and learning systems to learn from patient safety incidents and empirically identify opportunities to improve patient safety [4]. The primary function of these reporting systems is to harness the results of data analysis to guide system improvements [5]. Reporting systems form a vital part of routine monitoring of clinical practice [6] and can

improve the quality of care delivered by healthcare services through the identification of common safety issues, helping to prioritise funding and resources, and the development of safety interventions [7].

Classification systems are important tools for providing structure to meaningful reporting and data analysis [8] and help to make sense of the complex nature of patient safety incidents [9]. A taxonomy is required to standardise these classifications into a hierarchical form to aid categorisation and organisation [10]. Figure 1 illustrates how organisational learning benefits from classification systems.

Previously, inconsistent definitions used in patient safety literature impeded international learning [11]. In 2009, the World Health Organization (WHO) responded to this need to standardise the terms used in classifying patient safety by developing the International Classification for Patient Safety (ICPS) conceptual framework [12]. The ICPS framework allows for greater comparisons to be made between countries and care settings because of a more organised, accepted, and understandable

<figure><img src="image_3.png" /><figcaption>**Fig. 1** Steps to learning from patient safety incidents. Key steps related to our aims and implementation of the PISA classification system are outlined in bold</figcaption></figure>
classification inclusive of concepts with standard definitions [13]. In 2014, in response to a lack of granular patient safety classification systems suitable for primary care, our group empirically developed the ‘Patlent Safety (PISA) classification system’ [14] which is ontologically aligned with the ICPS framework. PISA has since been used to analyse more than 75,000 primary care incident reports [14].

The longevity and value of a taxonomy is determined by how well it meets the end-user’s needs [10]. Maintaining this requires the taxonomy to undergo regular review to ensure its coding remains inclusive and comprehensive, yet flexible, allowing for new codes to be added or for existing codes to be amended [15]. The International Classification of Diseases (ICD) provides a common language for healthcare professionals and is a global standard for the systematic recording of morbidity and mortality data [16]. In response to the COVID-19 pandemic, the WHO classification and terminologies unit activated emergency codes for COVID-19 in ICD-10 and ICD-11, regarding its diagnosis, complications, and vaccinations [17]. It is not known whether existing patient safety classifications, like PISA, are sufficient to capture novel safety concepts arising from systemic constraints related to the pandemic in the context of patient incident reporting and learning systems. It is unclear whether they require updating to incorporate COVID-19-specific coding in relation to healthcare-associated harm.

We sought to critically review our experience of using the PISA classification system during the COVID-19 pandemic, with a focus on its ability to classify incidents, enable the initial deductive analysis, and identify areas for further enquiry, thus generating learning from patient- and clinician-reported safety concerns (as shown in Fig. 1). We sought to identify whether modifications to our PISA classification system should be considered to maintain its meaningful use and ensure accessible and timely learning from patients and health-care professionals.

# Methods

## Study design

We conducted a mixed-methods study integrating two phases in an exploratory, sequential design:

1) A comparative secondary analysis of patient safety incident reports from two studies conducted during the first wave of the COVID-19 pandemic: (i) patient-reported incidents from the United Kingdom (UK); (ii) general practitioner (GP)-reported incidents from France.

2) Presentation of findings to a focus group of health services researchers, clinicians, and leaders.

# Description of reference framework (PISA) and its development

The Patlent SAfety (PISA) Research Group at Cardiff University has led an extensive characterisation of patient safety incidents reported by healthcare professionals from primary and secondary care settings. This includes studies to identify priority areas for patient safety across the health and social care continuum, including unsafe discharge from secondary to primary care settings [18], incidents experienced by children in primary care [19], older adults [20], patients receiving palliative care [21], advanced care planning [22] and adults receiving opiate replacement therapy [23]. The PISA classification system has also previously been used to characterise the nature of patient-reported safety incidents in primary care settings from the UK and Australia, enabling the data to be used for service learning and improvement [24]. The PISA classification system was developed using a constant comparative method [25] on an initial sample of 13,600 reports [26] followed by nine years of iterative development (i.e., addition of new codes, refinement of new and existing definitions). The PISA classification system is used by health services researchers in the UK, Canada, Brazil, France, Spain, and Australia. It incorporates multiple coding frameworks with four independent classes describing the incident (‘what happened?’), the contributing factors (‘why it happened?’), the resultant harm outcome and the level of harm. Each code and its definitions within each class are intended to be ‘mutually exclusive’.

Where an existing code is not available to describe a salient feature within a report narrative, our research group discuss whether a new code is needed, or whether the definition of an existing code should be amended to be more inclusive. This is based on analytical memos created by coders during the review and application of codes. Our interdisciplinary group is comprised of clinicians (doctors, nurses, pharmacists, optometrists, dentists), mixed methods researchers, and Human Factors and patient safety experts.

By following the Recursive Model of Incident Analysis (Fig. 2), PISA can capture the chronological sequence of events typically described by reporters as leading to a safety incident. The coding framework is structured into higher-level ‘parent codes’, such as ‘2. Staff factors’ and branches into more nuanced ‘child codes’, such as ‘2.2 Task a piece of work to be done or undertaken’ leading to ‘2.2.1 Failure to follow protocol - failure to adhere to procedures or regulation’. Coders can assign parent or child codes dependent on the explicit content of the incident report.
classification inclusive of concepts with standard defi-
nitions [13]. In 2014, in response to a lack of granular
patient safety classification systems suitable for primary
care, our group empirically developed the ‘Patlent Safety
(PISA) classification system’ [14] which is ontologically
aligned with the ICPS framework. PISA has since been
used to analyse more than 75,000 primary care incident
reports [14].

The longevity and value of a taxonomy is determined
by how well it meets the end-user’s needs [10]. Maintain-
ing this requires the taxonomy to undergo regular review
to ensure its coding remains inclusive and comprehen-
sive, yet flexible, allowing for new codes to be added or
for existing codes to be amended [15]. The International
Classification of Diseases (ICD) provides a common
language for healthcare professionals and is a global
standard for the systematic recording of morbidity and
mortality data [16]. In response to the COVID-19 pan-
demic, the WHO classification and terminologies unit
activated emergency codes for COVID-19 in ICD-10 and
ICD-11, regarding its diagnosis, complications, and vac-
cinations [17]. It is not known whether existing patient
safety classifications, like PISA, are sufficient to capture
novel safety concepts arising from systemic constraints
related to the pandemic in the context of patient incident
reporting and learning systems. It is unclear whether
they require updating to incorporate COVID-19-specific
coding in relation to healthcare-associated harm.

We sought to critically review our experience of using
the PISA classification system during the COVID-19
pandemic, with a focus on its ability to classify inci-
dents, enable the initial deductive analysis, and iden-
tify areas for further enquiry, thus generating learning
from patient- and clinician-reported safety concerns (as
shown in Fig. 1). We sought to identify whether modi-
fications to our PISA classification system should be
considered to maintain its meaningful use and ensure
accessible and timely learning from patients and health-
care professionals.

Methods

Study design

We conducted a mixed-methods study integrating two
phases in an exploratory, sequential design:

1) A comparative secondary analysis of patient safety incident reports from two studies conducted during the first wave of the COVID-19 pandemic: (i) patient-reported incidents from the United Kingdom (UK); (ii) general practitioner (GP)-reported incidents from France.

2) Presentation of findings to a focus group of health services researchers, clinicians, and leaders.

Description of reference framework (PISA) and its
development

The Patlent SAfety (PISA) Research Group at Cardiff
University has led an extensive characterisation of patient
safety incidents reported by healthcare professionals
from primary and secondary care settings. This includes
studies to identify priority areas for patient safety across
the health and social care continuum, including unsafe
discharge from secondary to primary care settings [18],
incidents experienced by children in primary care [19],
older adults [20], patients receiving palliative care [21],
advanced care planning [22] and adults receiving opiate
replacement therapy [23]. The PISA classification system
has also previously been used to characterise the nature
of patient-reported safety incidents in primary care set-
tings from the UK and Australia, enabling the data to
be used for service learning and improvement [24]. The
PISA classification system was developed using a con-
stant comparative method [25] on an initial sample of
13,600 reports [26] followed by nine years of iterative
development (i.e., addition of new codes, refinement of
new and existing definitions). The PISA classification
system is used by health services researchers in the UK,
Canada, Brazil, France, Spain, and Australia. It incorpo-
rates multiple coding frameworks with four independent
classes describing the incident ('what happened?'), the
contributing factors ('why it happened?'), the resultant
harm outcome and the level of harm. Each code and its
definitions within each class are intended to be 'mutually
exclusive'.

Where an existing code is not available to describe a
salient feature within a report narrative, our research
group discuss whether a new code is needed, or whether
the definition of an existing code should be amended to
be more inclusive. This is based on analytical memos
created by coders during the review and application of
codes. Our interdisciplinary group is comprised of clini-
cians (doctors, nurses, pharmacists, optometrists, den-
tists), mixed methods researchers, and Human Factors
and patient safety experts.

By following the Recursive Model of Incident Analy-
sis (Fig. 2), PISA can capture the chronological sequence
of events typically described by reporters as leading to a
safety incident. The coding framework is structured into
higher-level ‘parent codes’, such as ‘2. Staff factors’ and
branches into more nuanced ‘child codes’, such as ‘2.2
Task a piece of work to be done or undertaken’ leading to
‘2.2.1 Failure to follow protocol - failure to adhere to pro-
cedures or regulation’. Coders can assign parent or child
codes dependent on the explicit content of the incident
report.
<figure><img src="image_4.png" /><figcaption>**Fig. 2** Example of the recursive model of incident analysis</figcaption></figure>

## Patient- and clinician-reported safety incidents

### Data sources

The first source of data came from the COVID-19 UK Public Experiences (COPE) study [27]. This prospective, longitudinal study was developed to gain a better understanding of health behaviours, experiences and well-being outcomes throughout the pandemic. The baseline survey was launched on 13th March 2020 and closed on 13th April 2020. The patient cohort was recruited either via social media (using multi-faceted sampling methods) or via two emails sent to participants of Health Wise Wales (HWW), a national population survey and research register of participants who live or receive healthcare in Wales [28]. A three-month follow-up survey was conducted in June/July 2020. This survey included optional, additional modules, one of which was a free-text module for patients to report any 'healthcare experiences (including patient-reported safety concerns)'. A total of 318 people opted to complete this module.

The second source of data came from an exploratory, mixed-methods study of GP-generated patient safety incident reports relating to COVID-19 in France (henceforth referred to as the PSI RECORD study) [29]. The team created a national patient safety incident reporting platform to collect data on patient demographics and a free-text description of the incident. The contact list of the Collège National des Généralistes Enseignants (CNGE, a French scientific society of general medicine) was invited to participate via email on 28th April 2020 (with one reminder email sent on 28th May 2020). The GPs were asked to report patient safety incidents observed since 17th March 2020. Reported incidents were included up to 29th June 2020. A total of 103 GPs submitted between one and four anonymised reports, generating 132 incident reports for inclusion.

All data from both datasets were fully anonymised, with no personal identifiers remaining within the reports, prior to being added to the PISA platform for coding.

## Data coding and comparative analysis

The anonymised reports from the PSI RECORD study were translated from French into English using Google Translate [30]. A French-speaking member of the team (ED) sense-checked all the translations and generated a list of French-specific abbreviations for reference. Trained coders applied the PISA classification system to code the anonymised free-text information within the COPE (AC) and PSI RECORD (AC, TP) incident reports. Using the principles of the constant comparative method, we undertook an exploratory descriptive analysis of the coded data to identify the most frequent primary incidents and created cross-tabulations to explore the relationships between these incidents, their respective contributing factors, and the harm outcomes. A primary incident was defined as 'an issue, complication, or lack of something perceived important during a task or process of care delivery, occurring prior to the outcome and is the incident most proximal to the identified patient outcome' [12]. To ensure validity and reliability of the coding, Cohen's kappa statistics were calculated for the primary incident type, aiming for a kappa>0.7, which is consistent with previous similar studies [31]. Drawing on the established methods used to inductively amend the PISA classification system through regular review, the analysis sought to identify potential areas where the classification could be iterated to incorporate new or amended codes or their definitions.
<figure><img src="image_4.png" /><figcaption>**Fig. 2** Example of the recursive model of incident analysis</figcaption></figure>

## Patient- and clinician-reported safety incidents

### Data sources

The first source of data came from the COVID-19 UK Public Experiences (COPE) study [27]. This prospective, longitudinal study was developed to gain a better understanding of health behaviours, experiences and well-being outcomes throughout the pandemic. The baseline survey was launched on 13th March 2020 and closed on 13th April 2020. The patient cohort was recruited either via social media (using multi-faceted sampling methods) or via two emails sent to participants of Health Wise Wales (HWW), a national population survey and research register of participants who live or receive healthcare in Wales [28]. A three-month follow-up survey was conducted in June/July 2020. This survey included optional, additional modules, one of which was a free-text module for patients to report any 'healthcare experiences (including patient-reported safety concerns)'. A total of 318 people opted to complete this module.

The second source of data came from an exploratory, mixed-methods study of GP-generated patient safety incident reports relating to COVID-19 in France (henceforth referred to as the PSI RECORD study) [29]. The team created a national patient safety incident reporting platform to collect data on patient demographics and a free-text description of the incident. The contact list of the Collège National des Généralistes Enseignants (CNGE, a French scientific society of general medicine) was invited to participate via email on 28th April 2020 (with one reminder email sent on 28th May 2020). The GPs were asked to report patient safety incidents observed since 17th March 2020. Reported incidents were included up to 29th June 2020. A total of 103 GPs submitted between one and four anonymised reports, generating 132 incident reports for inclusion.

All data from both datasets were fully anonymised, with no personal identifiers remaining within the reports, prior to being added to the PISA platform for coding.

## Data coding and comparative analysis

The anonymised reports from the PSI RECORD study were translated from French into English using Google Translate [30]. A French-speaking member of the team (ED) sense-checked all the translations and generated a list of French-specific abbreviations for reference. Trained coders applied the PISA classification system to code the anonymised free-text information within the COPE (AC) and PSI RECORD (AC, TP) incident reports. Using the principles of the constant comparative method, we undertook an exploratory descriptive analysis of the coded data to identify the most frequent primary incidents and created cross-tabulations to explore the relationships between these incidents, their respective contributing factors, and the harm outcomes. A primary incident was defined as 'an issue, complication, or lack of something perceived important during a task or process of care delivery, occurring prior to the outcome and is the incident most proximal to the identified patient outcome' [12]. To ensure validity and reliability of the coding, Cohen's kappa statistics were calculated for the primary incident type, aiming for a kappa>0.7, which is consistent with previous similar studies [31]. Drawing on the established methods used to inductively amend the PISA classification system through regular review, the analysis sought to identify potential areas where the classification could be iterated to incorporate new or amended codes or their definitions.
<table><caption>**Table 1** List of expert group participants (n=6) and facilitators (n=2)</caption><thead><tr><td></td><td>Position and experience</td><td>Interests</td></tr></thead><tbody><tr><td>Participant 1 (P1)</td><td>National programme director for safety and quality improvement</td><td>Chartered Human Factors and Ergonomics specialist and has led reviews into safety taxonomy and incident coding frameworks for healthcare settings</td></tr><tr><td>Participant 2 (P2)</td><td>Academic GP</td><td>Involved in the development of the PISA classification system and its application within an extensive range of health services research projects</td></tr><tr><td>Facilitator</td><td></td><td></td></tr><tr><td>Participant 3 (P3)</td><td>Health services researcher and incident analyst/ investigator</td><td>Has led reviews into safety classification and incident coding frameworks and helped develop the WHO ICPS</td></tr><tr><td>Participant 4 (P4)</td><td>Senior academic with expertise in patient care improvement</td><td>Interest in person-centred and value-based approaches in routine healthcare and patients' experiences of routine care</td></tr><tr><td>Participant 5 (P5)</td><td>Academic GP with patient safety research expertise</td><td>Involved in the development of the PISA classification system and its application within an extensive range of health services research projects</td></tr><tr><td>Participant 6 (P6)</td><td>GP specialty registrar trained in Human Factors</td><td>Has worked on patient safety research projects using incident reporting frameworks to codify data</td></tr><tr><td>Facilitator</td><td></td><td></td></tr><tr><td>Participant 7 (P7)</td><td>Academic GP with patient safety research expertise</td><td>Led the development of the PISA classification system</td></tr><tr><td>Participant 8 (P8)</td><td>Academic GP with patient safety and quality improvement research expertise</td><td>Quality and safety of health care shared decision-making expert</td></tr></tbody></table>

<table><caption>**Table 2** Coding unstructured (free-text) data – Patient and clinician examples</caption><thead><tr><td>Source of incident</td><td>Free text</td><td>Coding</td></tr></thead><tbody><tr><td>Patient-reported</td><td>'I was shocked & outraged to observe that staff with face masks hanging off one ear or under their chin. Totally unprofessional touching their faces with the gloved hands that were worn for whole shift with no use of hand gel or hand washing'</td><td>Incident type: Staff professionalism<br>Contributing factor(s): Staff behaviour and infection control protocols<br>Harm outcome: No outcome described<br>Harm severity: Unclear</td></tr><tr><td>GP-reported</td><td>'The patient called us for vomiting still bloody, despite our repeated advice to call 15 [emergency services for life-threatening conditions] they did not go. The patient died of a digestive hemorrhage 7 h after the first call. The patient's daughter was afraid to take her to the hospital for fear that she would catch COVID 19 there'</td><td>Primary incident: Delayed diagnosis Contributing incident: Environmental hazard<br>Contributing factor: Behaviour – patient/family<br>Harm outcome: Clinical deterioration<br>Harm severity: Death</td></tr></tbody></table>

Expert discussion

Expert review and feedback on the analysis of the COPE and PSI RECORD data were sought through an online focus group. The purpose of the focus group discussion was to identify similarities and differences between codes required to characterise COVID-19 reports and, as a research collaboration experienced in classification development, to recommend whether and how the existing frameworks could be modified to capture relevant learning arising due to unanticipated major systemic constraints like those posed by the COVID-19 pandemic. The participants (Table 1), selected for their expertise in the development and maintenance of classification systems, were invited to attend via email and signed an electronic consent form before taking part.

The participants were presented with the findings of the comparative analysis and seven content-rich safety incident examples with different incident types, including the free-text and coding (such as those presented in Table 2). During data analysis, a list of codes was created that were considered to represent new concepts in the context of COVID-19 and suggested new codes which included COVID-19 as a direct contributing factor in relation to the incident. Examples of suggestions included 'Shielding patient', 'Service unavailable due to

COVID-19', 'Patient fearful of attending healthcare service' and 'System change due to crisis management planning'. Guided by the incident examples, codes generated from the secondary data analysis, and semi-structured questions, the expert group discussed whether the PISA classification system warranted iteration to incorporate COVID-19-specific coding (see Appendix 1 for meeting schedule).

The group was facilitated by two of the researchers (AC, TP). The discussion was recorded via Zoom, lasted 1 h 30 min, and was transcribed verbatim in-house (DP).

We then conducted an inductive thematic analysis [32] of the expert group transcript to identify their perspectives on the ability of the PISA classification system to classify incidents from the context of the pandemic, support the initial deductive analysis and identify areas for further enquiry to derive learning. Researchers (DP, TP) ensured thorough familiarisation with the data and generated initial themes based on the purpose and objective of the analysis. These themes were reviewed both individually and during subsequent team meetings. Once the themes had been agreed upon, they were defined and named, and a focussed literature search was undertaken to determine whether these concepts have been described previously in the literature.
<table><caption>**Table 1** List of expert group participants (n=6) and facilitators (n=2)</caption><thead><tr><td></td><td>Position and experience</td><td>Interests</td></tr></thead><tbody><tr><td>Participant 1 (P1)</td><td>National programme director for safety and quality improvement</td><td>Chartered Human Factors and Ergonomics specialist and has led reviews into safety taxonomy and incident coding frameworks for healthcare settings</td></tr><tr><td>Participant 2 (P2)</td><td>Academic GP</td><td>Involved in the development of the PISA classification system and its application within an extensive range of health services research projects</td></tr><tr><td>Facilitator</td><td></td><td></td></tr><tr><td>Participant 3 (P3)</td><td>Health services researcher and incident analyst/ investigator</td><td>Has led reviews into safety classification and incident coding frameworks and helped develop the WHO ICPS</td></tr><tr><td>Participant 4 (P4)</td><td>Senior academic with expertise in patient care improvement</td><td>Interest in person-centred and value-based approaches in routine healthcare and patients' experiences of routine care</td></tr><tr><td>Participant 5 (P5)</td><td>Academic GP with patient safety research expertise</td><td>Involved in the development of the PISA classification system and its application within an extensive range of health services research projects</td></tr><tr><td>Participant 6 (P6)</td><td>GP specialty registrar trained in Human Factors</td><td>Has worked on patient safety research projects using incident reporting frameworks to codify data</td></tr><tr><td>Facilitator</td><td></td><td></td></tr><tr><td>Participant 7 (P7)</td><td>Academic GP with patient safety research expertise</td><td>Led the development of the PISA classification system</td></tr><tr><td>Participant 8 (P8)</td><td>Academic GP with patient safety and quality improvement research expertise</td><td>Quality and safety of health care shared decision-making expert</td></tr></tbody></table>

<table><caption>**Table 2** Coding unstructured (free-text) data – Patient and clinician examples</caption><thead><tr><td>Source of incident</td><td>Free text</td><td>Coding</td></tr></thead><tbody><tr><td>Patient-reported</td><td>'I was shocked & outraged to observe that staff with face masks hanging off one ear or under their chin. Totally unprofessional touching their faces with the gloved hands that were worn for whole shift with no use of hand gel or hand washing'</td><td>Incident type: Staff professionalism<br>Contributing factor(s): Staff behaviour and infection control protocols<br>Harm outcome: No outcome described<br>Harm severity: Unclear</td></tr><tr><td>GP-reported</td><td>'The patient called us for vomiting still bloody, despite our repeated advice to call 15 [emergency services for life-threatening conditions] they did not go. The patient died of a digestive hemorrhage 7 h after the first call. The patient's daughter was afraid to take her to the hospital for fear that she would catch COVID 19 there'</td><td>Primary incident: Delayed diagnosis Contributing incident: Environmental hazard<br>Contributing factor: Behaviour – patient/family<br>Harm outcome: Clinical deterioration<br>Harm severity: Death</td></tr></tbody></table>

Expert discussion

Expert review and feedback on the analysis of the COPE and PSI RECORD data were sought through an online focus group. The purpose of the focus group discussion was to identify similarities and differences between codes required to characterise COVID-19 reports and, as a research collaboration experienced in classification development, to recommend whether and how the existing frameworks could be modified to capture relevant learning arising due to unanticipated major systemic constraints like those posed by the COVID-19 pandemic. The participants (Table 1), selected for their expertise in the development and maintenance of classification systems, were invited to attend via email and signed an electronic consent form before taking part.

The participants were presented with the findings of the comparative analysis and seven content-rich safety incident examples with different incident types, including the free-text and coding (such as those presented in Table 2). During data analysis, a list of codes was created that were considered to represent new concepts in the context of COVID-19 and suggested new codes which included COVID-19 as a direct contributing factor in relation to the incident. Examples of suggestions included 'Shielding patient', 'Service unavailable due to

COVID-19', 'Patient fearful of attending healthcare service' and 'System change due to crisis management planning'. Guided by the incident examples, codes generated from the secondary data analysis, and semi-structured questions, the expert group discussed whether the PISA classification system warranted iteration to incorporate COVID-19-specific coding (see Appendix 1 for meeting schedule).

The group was facilitated by two of the researchers (AC, TP). The discussion was recorded via Zoom, lasted 1 h 30 min, and was transcribed verbatim in-house (DP).

We then conducted an inductive thematic analysis [32] of the expert group transcript to identify their perspectives on the ability of the PISA classification system to classify incidents from the context of the pandemic, support the initial deductive analysis and identify areas for further enquiry to derive learning. Researchers (DP, TP) ensured thorough familiarisation with the data and generated initial themes based on the purpose and objective of the analysis. These themes were reviewed both individually and during subsequent team meetings. Once the themes had been agreed upon, they were defined and named, and a focussed literature search was undertaken to determine whether these concepts have been described previously in the literature.
<figure><img src="image_5.png" /><figcaption>**Fig. 3** Flowchart of processing clinician- (PSI RECORD) and patient-reported (COPE) incident reports</figcaption></figure>

<table><caption>**Table 3** Key themes derived from the expert group</caption><thead><tr><td>Theme</td><td>Description</td></tr></thead><tbody><tr><td>1. Capitalise on the unique and contrasting lens on patient safety offered by different groups</td><td>Patients and clinicians/healthcare professionals may have complementary and differing perspectives on safety</td></tr><tr><td>2. Existing frameworks should already enable the identification of high-level signals to inform decisions about where to prioritise resources for further analysis or investigation</td><td>Current coding frameworks can capture relevant learning. Be clear on how to generate those signals, e.g., by reviewing the most frequent incident types and severity combinations</td></tr><tr><td>3. Consider the objectives and purpose of the inquiry when using the framework, with a process to enable a timely and insightful analysis</td><td>Study objectives are key to interpreting the data. The need for granular coding approaches depends on the purpose of the question being asked of the different data sources</td></tr><tr><td>4. Be aware of factors that might always have been present but have not been previously captured</td><td>Avoid overlooking new learning from less-cited contributing factors that have been brought to the fore in reports in specific contexts like COVID-19</td></tr><tr><td>5. Consider the temporal relationship between the period of data collection and substantive events/system constraints</td><td>The timeframe in which data have been collected sometimes makes it unique and may offer explanations for new findings. The time periods of major systemic constraints can also be used to contextualise findings rather than generating bespoke labels or new classes (e.g. COVID-19 related) in a classification system</td></tr></tbody></table>

## Results

The breakdown of included and excluded reports from the PSI RECORD and COPE studies are detailed in Fig. 3.

### Thematic analysis of expert group discussion

We identified five themes from thematic analysis of the expert group discussion transcript (Table 3). To help illustrate the context of these themes, they are presented alongside the salient excerpts presented to the expert group from the secondary analysis of the coded data.

### Clinicians and patients may have complementary and differing perspectives on safety

About half of the patient-reported incidents comprised the two most common incident types: (1) environmental hazards (n=39/148, 26%), for example, not being able to socially distance, and (2) difficulty accessing healthcare (n=39/148, 26%), for example cancellation of routine clinics (Table 4). In 21 (14%) reports the primary incident related to staff professionalism and a lack of equipment, such as Personal Protective Equipment (PPE) (example in Table 2).

It was noted by the expert group that these codes have been less frequently used in previous studies, highlighting
<figure><img src="image_5.png" /><figcaption>**Fig. 3** Flowchart of processing clinician- (PSI RECORD) and patient-reported (COPE) incident reports</figcaption></figure>

<table><caption>**Table 3** Key themes derived from the expert group</caption><thead><tr><td>Theme</td><td>Description</td></tr></thead><tbody><tr><td>1. Capitalise on the unique and contrasting lens on patient safety offered by different groups</td><td>Patients and clinicians/healthcare professionals may have complementary and differing perspectives on safety</td></tr><tr><td>2. Existing frameworks should already enable the identification of high-level signals to inform decisions about where to prioritise resources for further analysis or investigation</td><td>Current coding frameworks can capture relevant learning. Be clear on how to generate those signals, e.g., by reviewing the most frequent incident types and severity combinations</td></tr><tr><td>3. Consider the objectives and purpose of the inquiry when using the framework, with a process to enable a timely and insightful analysis</td><td>Study objectives are key to interpreting the data. The need for granular coding approaches depends on the purpose of the question being asked of the different data sources</td></tr><tr><td>4. Be aware of factors that might always have been present but have not been previously captured</td><td>Avoid overlooking new learning from less-cited contributing factors that have been brought to the fore in reports in specific contexts like COVID-19</td></tr><tr><td>5. Consider the temporal relationship between the period of data collection and substantive events/system constraints</td><td>The timeframe in which data have been collected sometimes makes it unique and may offer explanations for new findings. The time periods of major systemic constraints can also be used to contextualise findings rather than generating bespoke labels or new classes (e.g. COVID-19 related) in a classification system</td></tr></tbody></table>

## Results

The breakdown of included and excluded reports from the PSI RECORD and COPE studies are detailed in Fig. 3.

### Thematic analysis of expert group discussion

We identified five themes from thematic analysis of the expert group discussion transcript (Table 3). To help illustrate the context of these themes, they are presented alongside the salient excerpts presented to the expert group from the secondary analysis of the coded data.

### Clinicians and patients may have complementary and differing perspectives on safety

About half of the patient-reported incidents comprised the two most common incident types: (1) environmental hazards (n=39/148, 26%), for example, not being able to socially distance, and (2) difficulty accessing healthcare (n=39/148, 26%), for example cancellation of routine clinics (Table 4). In 21 (14%) reports the primary incident related to staff professionalism and a lack of equipment, such as Personal Protective Equipment (PPE) (example in Table 2).

It was noted by the expert group that these codes have been less frequently used in previous studies, highlighting
<table><caption>**Table 4** Most common Primary Incident Types</caption><thead><tr><td>Patient-reported incidents</td><td>n (%)</td><td>GP-reported incidents</td><td>n (%)</td></tr></thead><tbody><tr><td>Access to healthcare</td><td>39 (27)</td><td>Diagnosis and assessment</td><td>52 (44)</td></tr><tr><td>Environmental hazard</td><td>39 (27)</td><td>Treatment and procedure</td><td>21 (18)</td></tr><tr><td>Staff professionalism and equipment</td><td>21 (14)</td><td>Investigation errors</td><td>17 (15)</td></tr></tbody></table>

the learning opportunity now gained when exploring the patients' perspective. Contrastingly, in the GP-reported incidents, environmental hazards accounted for only one incident and staff professionalism did not feature. This may reflect what patients prioritise as safety concerns, describing events more personal to them or directly related to their immediate surroundings, e.g., PPE and social distancing. The expert group commented on the value gained from these new and varied insights.

P2: It's what's visible to patients as well, so they can see that, but they might not be able to see that their referral has gone missing or they had a badly labelled sample, it's what they can see, what the people in front of them are doing.

The GP-reported incidents found that a delay in diagnosis and assessment accounted for 44% (n=52/117) of the primary incident types. These incident reports specifically related to a delay in the diagnosis in 92% (n=48/52) of the reports. The next most frequent primary incident type concerned treatment and procedures (n=21/117, 18%), illustrating the GP concerns were predominantly centred around management of patient conditions.

## Current coding frameworks can capture relevant learning

Half of the incidents relating to a delayed diagnosis reported by GPs captured 'patient behaviour' as a contributory factor (n=24/48) and most cases (59%) resulted in a delay in patient management. When exploring patient behaviours contributing to the delay in diagnosis, a recurring theme was a fear by patients or their family of presenting to either a GP or hospital setting. This fear stemmed from being exposed to or contracting COVID-19 (example in Table 2).

The underlying contributing factors behind these behaviours are not captured by the current framework. Some GPs postulated that this fear may have been triggered by media coverage in relation to the pandemic, for example. The PISA classification system and its related frameworks potentially does not detect the nuance required, and the expert group agreed that 'we need to acknowledge the limitations of incident reporting... in terms of the depth you can collect' (P3). However, the PISA classification system identified 'patient behaviour' as an area worthy of exploring in more detail. Similarly, when considering the patient-reported examples, the group discussed that the PISA classification system was able to pick out the high-level signals of key broader concepts and concerns. They concluded a meaningful use of

a taxonomy should be to enable timely analysis of existing coding that could prompt more detailed follow-on investigation (i.e. interviews with staff and patients, or encouraging more reporting on a specific salient issue) or an inductive analysis of reports with similar characteristics (i.e. a review of all reports assigned the code 'patient behaviour' for a specific time period).

P1: it needs some investigation into why staff are behaving this way, is it because of the inadequate fitting of face masks, they're too warm, they need to breathe, they feel restricted. These are contributory factors in my head, as to why these types of incidents are taking place. And I don't know whether, in terms of the reporting we want to try and encourage the reflection on some of these contributory factors. It's this whole thing about trying to understand, more deeply, the nature of why incidents occur, and designing the classification system to reflect that.

## Study objectives are key to interpreting the data

About two-thirds (64%, n=23/46) of patient-reported incidents with a primary incident type of environment hazards had 'infection control policy' as a contributing factor, such as the patient example in Table 2. This example only captures the perspective of the patient and the true contributory factors explaining why this incident occurred cannot be fully ascertained from the report. If the aims of this study included identifying the causes for infection control protocols not being followed, or whether an appropriate protocol was in place, then the coding would be inadequate. If the purpose of the study is describing pandemic-related incidents and identifying areas for deeper analysis (as shown in Fig. 1) to derive learning, then the classification is sufficient to accomplish this.

P2: The depth and the granularity of the coding framework really depends on what you want to do with it. If... you're just trying to get a handle on what are the main issues here when we're dealing with a new disease that we don't know how to manage, then you just want high-level [intelligence] to get an idea about what's going on.

## New learning from less-cited contributing factors

Patient factors were also shown to be a contributing factor, but to a lesser extent, in one of the commonest patient-reported incidents concerning environmental hazards (n=7/39, 18%). A recurring theme for these patient-related contributing factors was fear of presenting to a healthcare setting. The expert group discussed
<table><caption>**Table 4** Most common Primary Incident Types</caption><thead><tr><td>Patient-reported incidents</td><td>n (%)</td><td>GP-reported incidents</td><td>n (%)</td></tr></thead><tbody><tr><td>Access to healthcare</td><td>39 (27)</td><td>Diagnosis and assessment</td><td>52 (44)</td></tr><tr><td>Environmental hazard</td><td>39 (27)</td><td>Treatment and procedure</td><td>21 (18)</td></tr><tr><td>Staff professionalism and equipment</td><td>21 (14)</td><td>Investigation errors</td><td>17 (15)</td></tr></tbody></table>

the learning opportunity now gained when exploring the patients' perspective. Contrastingly, in the GP-reported incidents, environmental hazards accounted for only one incident and staff professionalism did not feature. This may reflect what patients prioritise as safety concerns, describing events more personal to them or directly related to their immediate surroundings, e.g., PPE and social distancing. The expert group commented on the value gained from these new and varied insights.

P2: It's what's visible to patients as well, so they can see that, but they might not be able to see that their referral has gone missing or they had a badly labelled sample, it's what they can see, what the people in front of them are doing.

The GP-reported incidents found that a delay in diagnosis and assessment accounted for 44% (n=52/117) of the primary incident types. These incident reports specifically related to a delay in the diagnosis in 92% (n=48/52) of the reports. The next most frequent primary incident type concerned treatment and procedures (n=21/117, 18%), illustrating the GP concerns were predominantly centred around management of patient conditions.

## Current coding frameworks can capture relevant learning

Half of the incidents relating to a delayed diagnosis reported by GPs captured 'patient behaviour' as a contributory factor (n=24/48) and most cases (59%) resulted in a delay in patient management. When exploring patient behaviours contributing to the delay in diagnosis, a recurring theme was a fear by patients or their family of presenting to either a GP or hospital setting. This fear stemmed from being exposed to or contracting COVID-19 (example in Table 2).

The underlying contributing factors behind these behaviours are not captured by the current framework. Some GPs postulated that this fear may have been triggered by media coverage in relation to the pandemic, for example. The PISA classification system and its related frameworks potentially does not detect the nuance required, and the expert group agreed that 'we need to acknowledge the limitations of incident reporting... in terms of the depth you can collect' (P3). However, the PISA classification system identified 'patient behaviour' as an area worthy of exploring in more detail. Similarly, when considering the patient-reported examples, the group discussed that the PISA classification system was able to pick out the high-level signals of key broader concepts and concerns. They concluded a meaningful use of

a taxonomy should be to enable timely analysis of existing coding that could prompt more detailed follow-on investigation (i.e. interviews with staff and patients, or encouraging more reporting on a specific salient issue) or an inductive analysis of reports with similar characteristics (i.e. a review of all reports assigned the code 'patient behaviour' for a specific time period).

P1: it needs some investigation into why staff are behaving this way, is it because of the inadequate fitting of face masks, they're too warm, they need to breathe, they feel restricted. These are contributory factors in my head, as to why these types of incidents are taking place. And I don't know whether, in terms of the reporting we want to try and encourage the reflection on some of these contributory factors. It's this whole thing about trying to understand, more deeply, the nature of why incidents occur, and designing the classification system to reflect that.

## Study objectives are key to interpreting the data

About two-thirds (64%, n=23/46) of patient-reported incidents with a primary incident type of environment hazards had 'infection control policy' as a contributing factor, such as the patient example in Table 2. This example only captures the perspective of the patient and the true contributory factors explaining why this incident occurred cannot be fully ascertained from the report. If the aims of this study included identifying the causes for infection control protocols not being followed, or whether an appropriate protocol was in place, then the coding would be inadequate. If the purpose of the study is describing pandemic-related incidents and identifying areas for deeper analysis (as shown in Fig. 1) to derive learning, then the classification is sufficient to accomplish this.

P2: The depth and the granularity of the coding framework really depends on what you want to do with it. If... you're just trying to get a handle on what are the main issues here when we're dealing with a new disease that we don't know how to manage, then you just want high-level [intelligence] to get an idea about what's going on.

## New learning from less-cited contributing factors

Patient factors were also shown to be a contributing factor, but to a lesser extent, in one of the commonest patient-reported incidents concerning environmental hazards (n=7/39, 18%). A recurring theme for these patient-related contributing factors was fear of presenting to a healthcare setting. The expert group discussed
the likelihood that similar concerns around attending healthcare settings for fear of developing a healthcare-acquired infection have always been an important factor but with the context of an infectious pandemic they have been accentuated.

P2: I would probably suggest that what people end up reporting often reflects some of their existing concerns or worries about healthcare. So, it's interesting that the patients said a lot of stuff around the doctors not washing their hands or not wearing masks properly, whereas the doctors/GPs are saying 'well they can't get access to secondary care in time', or 'there wasn't a service for the patients to come and see us about.' Those are the sorts of things that have been worrying people for a while, before they got an opportunity to report.

The context of 'fear' as a patient behaviour, more specifically fear of contracting COVID-19, posed the question as to whether this warranted an iteration to the existing framework. The expert group considered that generating additional codes to cover multiple eventualities meant that codes may go unused, and may result in a larger and more complex classification system, reducing its usability.

P3: I think that works better than just continually developing more and more codes, because you can just go on forever.

**The timeframe in which data are collected sometimes makes that data unique**

The commonest contributory factor for an environmental hazard incident type in the patient-reported incidents related to infection control policies. The expert group discussed that the type of infection involved (e.g., MRSA, pneumonia) could impact on the incident and outcome severity. The severity of the incident is only captured by the chosen harm outcome and severity, for example, primary incident type: environmental hazard, contributing factor: infection control protocols, harm outcome: infection, harm severity: death. An 'infection control policy' as a contributing factor also does not highlight what the specific concern might be with the policy, or lack thereof. However, an awareness that these data were obtained during the first few months of the COVID-19 pandemic gives a context whereby these concerns may be explainable. For instance, was a new infection control policy being developed or were there policies in place related to crisis management strategies? As the pandemic, our understanding of it, and its impact on healthcare systems was in a state of flux from an organisational perspective, this should be borne in mind when looking at the data.

P5: It's really helpful to put it back into the context of where those PRCs [incidents reports] came from, and the setting and timing of when they were reporting. There were lots of new rules coming out about PPE, social distancing... maybe that's why they were reported, because of the concerns going around at that moment.

Due to the ever-changing response and re-organisation of healthcare systems during the pandemic, it is pertinent to note the point at which the data were collected since the pandemic began. This is likely to play a role in reports made by both patients and clinicians.

P4: The timing, this was three months into the pandemic, obviously services had been delayed already to some extent, but maybe at that point they [patients] weren't really thinking about the impact of that delay on treatment. Or if there was a delay in diagnosis, they probably wouldn't have even been aware of it at the time.

Bringing together the above themes, there was a consensus amongst the group that no new codes needed to be added or amended within the frameworks as a direct consequence of COVID-19-related system constraints.

P5: It's been helpful to think about where we need to add codes... but the feeling I'm getting back from the group is that these shouldn't be COVID-specific codes, but [to continue with] more general codes that can then be analysed in more detail from thematic analysis and inductive analysis.

**Discussion**

**Main findings**

The meaningful use of the PISA classification system has undergone scrutiny following its application to patient safety incident reports generated during a global pandemic, from healthcare systems where resources were constrained and in which both patients and healthcare professionals faced new challenges and uncertainty [33]. A key finding from our study was that despite these significant system changes, the consensus was the PISA classification system and its frameworks could be meaningfully applied to safety data like incident reports without the need for new COVID-19-specific codes to be added or amended. This suggests that our existing codes can describe healthcare-associated harms which might arise from future system constraints generated by other pandemics or major system stressors, as a basis for further enquiry and analysis. Necessary updates to such frameworks through consistent iterative mechanisms are part of standard implementation.

Our results offer the perspectives of both patients and clinicians at the early stages of the pandemic, which may indicate the types of concerns and the capability of patient safety incident reporting and learning systems that may arise from evolving and future pandemics or other major system stressors. We describe five themes arising from the expert group which organisations and investigators can consider when evaluating and using their own patient safety taxonomies in the context of
the likelihood that similar concerns around attending healthcare settings for fear of developing a healthcare-acquired infection have always been an important factor but with the context of an infectious pandemic they have been accentuated.

P2: I would probably suggest that what people end up reporting often reflects some of their existing concerns or worries about healthcare. So, it's interesting that the patients said a lot of stuff around the doctors not washing their hands or not wearing masks properly, whereas the doctors/GPs are saying 'well they can't get access to secondary care in time', or 'there wasn't a service for the patients to come and see us about.' Those are the sorts of things that have been worrying people for a while, before they got an opportunity to report.

The context of 'fear' as a patient behaviour, more specifically fear of contracting COVID-19, posed the question as to whether this warranted an iteration to the existing framework. The expert group considered that generating additional codes to cover multiple eventualities meant that codes may go unused, and may result in a larger and more complex classification system, reducing its usability.

P3: I think that works better than just continually developing more and more codes, because you can just go on forever.

## The timeframe in which data are collected sometimes makes that data unique

The commonest contributory factor for an environmental hazard incident type in the patient-reported incidents related to infection control policies. The expert group discussed that the type of infection involved (e.g., MRSA, pneumonia) could impact on the incident and outcome severity. The severity of the incident is only captured by the chosen harm outcome and severity, for example, primary incident type: environmental hazard, contributing factor: infection control protocols, harm outcome: infection, harm severity: death. An 'infection control policy' as a contributing factor also does not highlight what the specific concern might be with the policy, or lack thereof. However, an awareness that these data were obtained during the first few months of the COVID-19 pandemic gives a context whereby these concerns may be explainable. For instance, was a new infection control policy being developed or were there policies in place related to crisis management strategies? As the pandemic, our understanding of it, and its impact on healthcare systems was in a state of flux from an organisational perspective, this should be borne in mind when looking at the data.

P5: It's really helpful to put it back into the context of where those PRCs [incidents reports] came from, and the setting and timing of when they were reporting. There were lots of new rules coming out about PPE, social

distancing...maybe that's why they were reported, because of the concerns going around at that moment.

Due to the ever-changing response and re-organisation of healthcare systems during the pandemic, it is pertinent to note the point at which the data were collected since the pandemic began. This is likely to play a role in reports made by both patients and clinicians.

P4: The timing, this was three months into the pandemic, obviously services had been delayed already to some extent, but maybe at that point they [patients] weren't really thinking about the impact of that delay on treatment. Or if there was a delay in diagnosis, they probably wouldn't have even been aware of it at the time.

Bringing together the above themes, there was a consensus amongst the group that no new codes needed to be added or amended within the frameworks as a direct consequence of COVID-19-related system constraints.

P5: It's been helpful to think about where we need to add codes... but the feeling I'm getting back from the group is that these shouldn't be COVID-specific codes, but [to continue with] more general codes that can then be analysed in more detail from thematic analysis and inductive analysis.

## Discussion

### Main findings

The meaningful use of the PISA classification system has undergone scrutiny following its application to patient safety incident reports generated during a global pandemic, from healthcare systems where resources were constrained and in which both patients and healthcare professionals faced new challenges and uncertainty [33]. A key finding from our study was that despite these significant system changes, the consensus was the PISA classification system and its frameworks could be meaningfully applied to safety data like incident reports without the need for new COVID-19-specific codes to be added or amended. This suggests that our existing codes can describe healthcare-associated harms which might arise from future system constraints generated by other pandemics or major system stressors, as a basis for further enquiry and analysis. Necessary updates to such frameworks through consistent iterative mechanisms are part of standard implementation.

Our results offer the perspectives of both patients and clinicians at the early stages of the pandemic, which may indicate the types of concerns and the capability of patient safety incident reporting and learning systems that may arise from evolving and future pandemics or other major system stressors. We describe five themes arising from the expert group which organisations and investigators can consider when evaluating and using their own patient safety taxonomies in the context of
other major events resulting in substantive system-wide
constraints.

## Context of existing literature

Collecting patient safety data from distinct groups enables an opportunity to capitalise on their individual perspectives and gain a broader understanding of how events, such as the COVID-19 pandemic, differentially affects them. Whereas patient or family perspectives may have previously been viewed predominantly as a litigation issue, they are increasingly seen as playing an important role in educating healthcare providers on where and how to improve systems and mitigate against further safety incidents. All patient concerns should therefore be treated equally with healthcare provider concerns, to overcome the blind spots apparent if solely relying on healthcare professional reporting [34]. In keeping with our first theme, previous studies have found both primary and secondary care patients identify and report different types of safety incidents when compared with healthcare professionals [24, 35, 36] and may report incidents that would otherwise go undetected [37]. Engaging patients in patient safety can result in positive outcomes (for patients and organisations) and mitigate the risk of future adverse events [38].

The pandemic is a unifying context for interpretation of both patient- and clinician-reported incidents. The significant disruption and delays to healthcare services, such as routine surgery [39] and cancer care [40], account for patient concerns around access to healthcare and GP-reported delays in treatment and procedures. These system disruptions and reduced service availability heighten patient anxiety regarding disease progression [41], therefore an awareness of any shared concern between patients and clinicians could focus resource management in tackling this area as a priority.

The pandemic resulted in increased levels of anxiety amongst patients [42] with significant numbers avoiding routine and emergency medical care due to concerns surrounding COVID-19 [43, 44]. This concern was captured in the GP-reported incident contributing factors. However, creating a new code for 'fear of COVID-19' is likely to be too specific. A trade-off is required between having the right depth to identify important common themes which guide further investigation without requiring too much detail in the coding system. This is consistent with the key concepts that led to the development of ICPS [13]. If the higher-level signals can identify areas to prioritise for further enquiry, then a classification system does not require modification to capture specific incidents related to other system constraints.

Within the scope of our study aims and objectives, the framework did not require modification. However, given the PISA classification system is used across healthcare

settings internationally, there may be specific issues that do warrant additional or amended codes. For example, McFadzean et al. [45] explored safety incidents reported from the prison setting in the UK. During this study, new codes within the contributory factors framework were generated, including 'security priorities' and 'lockdown rules (e.g., situations in which prisoners are not allowed out of their prison cell)', in order to capture learning specific to the context of prison healthcare systems. The new codes were iteratively developed, agreed upon by the research team and will benefit future research investigating prison healthcare.

Substantive events, such as the COVID-19 pandemic, create an opportunity for lesser-known contributing factors to be brought to the forefront of reporting. Such concerns are likely to have affected patients for a significantly longer period without being addressed. For example, codes for incidents relating to a patient's care environment have previously been used [46] and are therefore not specific to the pandemic. Anxiety surrounding hospital attendance for fear of catching a hospital-acquired infection is also not new, and has been reported with other infections, such as MRSA [47].

## Implications for practice, policy, and future research

To realise the full potential of existing patient safety classification systems and related frameworks and how they can be applied to maximise learning from patient safety incidents during significant system disruption, the following recommendations can be derived from the themes established by the expert group.

Multiple sources of patient safety data are available. When collecting data from only one source there is potential for incidents to go undetected [48]. Our results demonstrate the value added of using different data sources (e.g., patient-completed surveys and incident reporting by clinicians) to generate learning from patient safety incidents. Whilst accepting that there are limitations in terms of the variable depth of data collected through some methods, including patient surveys [49] and incident reporting [50], it is important to ensure that these data continue to be collected and used to maximise the learning opportunities from them. Common themes can then be explored in more detail to elicit a deeper, more nuanced understanding through other methods, such as patient interviews, which can identify additional concerns [51].

The higher-level granularity of data gathered in this study enabled learning in relation to safety incidents during the pandemic. Such signals identified in a similar way during future substantive events can be applied to initiate more in-depth, timely investigative processes (e.g., root cause analysis) which can support the design of improvement projects to reduce the frequency of safety
other major events resulting in substantive system-wide
constraints.

## Context of existing literature

Collecting patient safety data from distinct groups enables an opportunity to capitalise on their individual perspectives and gain a broader understanding of how events, such as the COVID-19 pandemic, differentially affects them. Whereas patient or family perspectives may have previously been viewed predominantly as a litigation issue, they are increasingly seen as playing an important role in educating healthcare providers on where and how to improve systems and mitigate against further safety incidents. All patient concerns should therefore be treated equally with healthcare provider concerns, to overcome the blind spots apparent if solely relying on healthcare professional reporting [34]. In keeping with our first theme, previous studies have found both primary and secondary care patients identify and report different types of safety incidents when compared with healthcare professionals [24, 35, 36] and may report incidents that would otherwise go undetected [37]. Engaging patients in patient safety can result in positive outcomes (for patients and organisations) and mitigate the risk of future adverse events [38].

The pandemic is a unifying context for interpretation of both patient- and clinician-reported incidents. The significant disruption and delays to healthcare services, such as routine surgery [39] and cancer care [40], account for patient concerns around access to healthcare and GP-reported delays in treatment and procedures. These system disruptions and reduced service availability heighten patient anxiety regarding disease progression [41], therefore an awareness of any shared concern between patients and clinicians could focus resource management in tackling this area as a priority.

The pandemic resulted in increased levels of anxiety amongst patients [42] with significant numbers avoiding routine and emergency medical care due to concerns surrounding COVID-19 [43, 44]. This concern was captured in the GP-reported incident contributing factors. However, creating a new code for 'fear of COVID-19' is likely to be too specific. A trade-off is required between having the right depth to identify important common themes which guide further investigation without requiring too much detail in the coding system. This is consistent with the key concepts that led to the development of ICPS [13]. If the higher-level signals can identify areas to prioritise for further enquiry, then a classification system does not require modification to capture specific incidents related to other system constraints.

Within the scope of our study aims and objectives, the framework did not require modification. However, given the PISA classification system is used across healthcare

settings internationally, there may be specific issues that do warrant additional or amended codes. For example, McFadzean et al. [45] explored safety incidents reported from the prison setting in the UK. During this study, new codes within the contributory factors framework were generated, including 'security priorities' and 'lockdown rules (e.g., situations in which prisoners are not allowed out of their prison cell)', in order to capture learning specific to the context of prison healthcare systems. The new codes were iteratively developed, agreed upon by the research team and will benefit future research investigating prison healthcare.

Substantive events, such as the COVID-19 pandemic, create an opportunity for lesser-known contributing factors to be brought to the forefront of reporting. Such concerns are likely to have affected patients for a significantly longer period without being addressed. For example, codes for incidents relating to a patient's care environment have previously been used [46] and are therefore not specific to the pandemic. Anxiety surrounding hospital attendance for fear of catching a hospital-acquired infection is also not new, and has been reported with other infections, such as MRSA [47].

## Implications for practice, policy, and future research

To realise the full potential of existing patient safety classification systems and related frameworks and how they can be applied to maximise learning from patient safety incidents during significant system disruption, the following recommendations can be derived from the themes established by the expert group.

Multiple sources of patient safety data are available. When collecting data from only one source there is potential for incidents to go undetected [48]. Our results demonstrate the value added of using different data sources (e.g., patient-completed surveys and incident reporting by clinicians) to generate learning from patient safety incidents. Whilst accepting that there are limitations in terms of the variable depth of data collected through some methods, including patient surveys [49] and incident reporting [50], it is important to ensure that these data continue to be collected and used to maximise the learning opportunities from them. Common themes can then be explored in more detail to elicit a deeper, more nuanced understanding through other methods, such as patient interviews, which can identify additional concerns [51].

The higher-level granularity of data gathered in this study enabled learning in relation to safety incidents during the pandemic. Such signals identified in a similar way during future substantive events can be applied to initiate more in-depth, timely investigative processes (e.g., root cause analysis) which can support the design of improvement projects to reduce the frequency of safety
incidents [52]. Exploration of the multi-faceted nature of patient safety incidents, guided by findings from incident reports, should be routinely assisting organisations to understand where and how to implement appropriate change (as illustrated in Fig. 1).

Better reporting systems for receiving and responding to patient concerns are needed and organisations could employ newer methods of data collection, such as the national platform being developed in Wales to gather patient-reported outcome measures (PROMS) [53]. Patients are then offered valuable, additional opportunities to express their experiences in a more structured manner across multiple healthcare settings.

## Strengths and limitations

The international impact of COVID-19 meant analysing two sources of data allowed us to explore a wider scope of safety concerns from both healthcare professionals and patients from two separate countries experiencing the same crisis. This adds to the external validity of our findings and produced a more thorough examination of the framework.

Limitations within the datasets include the quality of data gathered from the patient-reported concerns, as free-text responses were often brief and lacked detail. This may be a result of the question being an optional module amongst a larger questionnaire and without structured guidance on accurately describing a safety concern. The healthcare professional data reported GP concerns only, thereby missing incidents from other community-based professionals. Fournier et al. [29] did attempt to mitigate against this by conducting an ancillary study of pharmacists, nurses, physiotherapists, and midwives, but participation in this study was extremely low with only one incident reported.

We gained valuable insights from an international group of experts in patient safety with a working knowledge of the PISA classification system and its functionality. This provided consensus feedback from end-users on the application and performance of the PISA classification system amidst the pandemic. However, the expert group did not include a Patient and Public Involvement (PPI) representative. Our results miss the additional expertise and insights from such representatives but follow-on activities with PPI representation are planned.

## Conclusions

The PISA taxonomy can be successfully applied to patient safety incident reports to support the first stages in deriving learning and identifying areas for further enquiry. We identified no incidents that warranted new codes to be added to the PISA classification system, which may extend to other substantive public health crises, negating the need for additional, specific coding within such

classification systems and related frameworks for similar system-wide constraints.

## List of abbreviations

<table><tr><td>WHO</td><td>World Health Organization</td></tr><tr><td>ICPS</td><td>International Classification for Patient Safety</td></tr><tr><td>GP</td><td>General Practitioner</td></tr><tr><td>PISA</td><td>Patient SAfety (PISA) research group (Cardiff University)</td></tr></table>

## Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12874-023-02057-6.

Supplementary Material 1

## Acknowledgements

The authors thank Mr Stuart Hellard, Data Manager, School of Medicine, Cardiff University for his service in managing the PISA Research Group Database.

## Authors' contributions

TP, AC and ACS were involved in the conception of this study. TP and AC were involved in coding and analysing the data. ED sense-checked the reports translated from French to English. DP transcribed the focus group recording. TP and DP were involved in the thematic analysis of the focus group transcripts. TP drafted the first version of the manuscript. All authors read, edited and approved the final manuscript.

## Funding

Sources of funding for the COPE study are described in their published works. Our study was supported by PRIME Centre Wales, funded by Health Care and Research Wales. The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

## Data Availability

The datasets used and/or analysed during the current study are available from previously published studies (COPE/ PSI RECORD). All data generated or analysed during this study are available from the authors upon reasonable request.

## Declarations

### Competing interests

The authors declare no competing interests.

### Ethics approval and consent to participate

Ethics approval for the study was provided by the School of Medicine Research Ethics Committee at Cardiff University. SREC reference: SMREC 20/119. The original consent and participant information for both the PSI RECORD study and the COPE study covered making anonymised data available for analysis by the research team. All participants of the focus group provided informed consent prior to participating. The study was carried out in accordance with Cardiff University's policies, procedures and guidance.

### Consent for publication

Not applicable.

### Author details

¹Division of Population Medicine, School of Medicine, Cardiff University, Cardiff CF14, UK

²PRIME Centre Wales, Division of Population Medicine, School of Medicine, Cardiff University, Cardiff, UK

³Westway Surgery, Ely, Cardiff, UK

⁴Medical Directorate, NHS Education for Scotland, Glasgow, UK

⁵School of Health, Science and Wellbeing, Staffordshire University, Stafford, UK

⁶Département de Médecine Générale, Faculté de Médecine, Nantes Université, Nantes, France
incidents [52]. Exploration of the multi-faceted nature of patient safety incidents, guided by findings from incident reports, should be routinely assisting organisations to understand where and how to implement appropriate change (as illustrated in Fig. 1).

Better reporting systems for receiving and responding to patient concerns are needed and organisations could employ newer methods of data collection, such as the national platform being developed in Wales to gather patient-reported outcome measures (PROMS) [53]. Patients are then offered valuable, additional opportunities to express their experiences in a more structured manner across multiple healthcare settings.

## Strengths and limitations

The international impact of COVID-19 meant analysing two sources of data allowed us to explore a wider scope of safety concerns from both healthcare professionals and patients from two separate countries experiencing the same crisis. This adds to the external validity of our findings and produced a more thorough examination of the framework.

Limitations within the datasets include the quality of data gathered from the patient-reported concerns, as free-text responses were often brief and lacked detail. This may be a result of the question being an optional module amongst a larger questionnaire and without structured guidance on accurately describing a safety concern. The healthcare professional data reported GP concerns only, thereby missing incidents from other community-based professionals. Fournier et al. [29] did attempt to mitigate against this by conducting an ancillary study of pharmacists, nurses, physiotherapists, and midwives, but participation in this study was extremely low with only one incident reported.

We gained valuable insights from an international group of experts in patient safety with a working knowledge of the PISA classification system and its functionality. This provided consensus feedback from end-users on the application and performance of the PISA classification system amidst the pandemic. However, the expert group did not include a Patient and Public Involvement (PPI) representative. Our results miss the additional expertise and insights from such representatives but follow-on activities with PPI representation are planned.

## Conclusions

The PISA taxonomy can be successfully applied to patient safety incident reports to support the first stages in deriving learning and identifying areas for further enquiry. We identified no incidents that warranted new codes to be added to the PISA classification system, which may extend to other substantive public health crises, negating the need for additional, specific coding within such

classification systems and related frameworks for similar system-wide constraints.

## List of abbreviations

WHO World Health Organization

ICPS International Classification for Patient Safety

GP General Practitioner

PISA Patent SAfety (PISA) research group (Cardiff University)

## Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12874-023-02057-6.

Supplementary Material 1

## Acknowledgements

The authors thank Mr Stuart Hellard, Data Manager, School of Medicine, Cardiff University for his service in managing the PISA Research Group Database.

## Authors' contributions

TP, AC and ACS were involved in the conception of this study. TP and AC were involved in coding and analysing the data. ED sense-checked the reports translated from French to English. DP transcribed the focus group recording. TP and DP were involved in the thematic analysis of the focus group transcripts. TP drafted the first version of the manuscript. All authors read, edited and approved the final manuscript.

## Funding

Sources of funding for the COPE study are described in their published works. Our study was supported by PRIME Centre Wales, funded by Health Care and Research Wales. The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

## Data Availability

The datasets used and/or analysed during the current study are available from previously published studies (COPE/ PSI RECORD). All data generated or analysed during this study are available from the authors upon reasonable request.

## Declarations

### Competing interests

The authors declare no competing interests.

### Ethics approval and consent to participate

Ethics approval for the study was provided by the School of Medicine Research Ethics Committee at Cardiff University. SREC reference: SMREC 20/119. The original consent and participant information for both the PSI RECORD study and the COPE study covered making anonymised data available for analysis by the research team. All participants of the focus group provided informed consent prior to participating. The study was carried out in accordance with Cardiff University's policies, procedures and guidance.

### Consent for publication

Not applicable.

### Author details

<footnote n="1">¹Division of Population Medicine, School of Medicine, Cardiff University, Cardiff CF14, UK</footnote>

<footnote n="2">²PRIME Centre Wales, Division of Population Medicine, School of Medicine, Cardiff University, Cardiff, UK</footnote>

<footnote n="3">³Westway Surgery, Ely, Cardiff, UK</footnote>

<footnote n="4">⁴Medical Directorate, NHS Education for Scotland, Glasgow, UK</footnote>

<footnote n="5">⁵School of Health, Science and Wellbeing, Staffordshire University, Stafford, UK</footnote>

<footnote n="6">⁶Département de Médecine Générale, Faculté de Médecine, Nantes Université, Nantes, France</footnote>
⁷Australian Institute of Health Innovation, Macquarie University, Sydney, Australia

⁸Allied Health and Human Performance, IIMPACT in Health, University of South Australia, Adelaide, Australia

⁹Cardiff School of Sport and Health Sciences, Cardiff Metropolitan University, Cardiff, UK

Received: 15 September 2022 / Accepted: 6 October 2023

Published online: 14 October 2023

## References

1. World Health Organization News release. Essential health services face continued disruption during COVID-19 pandemic. 7 Feb 2022. [Accessed 14th Sept 2022]. Available from: https://www.who.int/news/item/07-02-2022-essential-health-services-face-continued-disruption-during-covid-19-pandemic.

2. Reed SJ, Schlepper L, Edwards N. Health system recovery from Covid-19: International lessons for the NHS. Report, Nuffield Trust; 2022.

3. Dallat C, Salmon PM, Goode N. Risky systems versus risky people: to what extent do risk assessment methods consider the systems approach to accident causation? A review of the literature. Saf Sci. 2019;119:266-79. https://doi.org/10.1016/j.ssci.2017.03.012.

4. World Health Organization. Patient safety incident reporting and learning systems: technical report and guidance. 16 September 2020. [Accessed 14th Sept 2022]. Available from: https://www.who.int/publications/i/item/9789240010338.

5. Liang C. Understanding Patient Safety Reports via Multi-label Text Classification and Semantic Representation. 2017. UT SBMI Dissertations (Open Access). 35. [Accessed on 14th Sept 2022]. Available from: https://digitalcommons.tmc.edu/uthshis_dissertations/35.

6. Vincent CA. Analysis of clinical incidents: a window on the system not a search for root causes. Qual Saf Health Care. 2004;13(4):242-3. https://doi.org/10.1136/qshc.2004.010454.

7. Pronovost PJ, Morlock LL, Sexton JB, Miller MR, Holzmueller CG, Thompson DA et al. Improving the Value of Patient Safety Reporting Systems. In: Henriksen K, Battles JB, Keyes MA, Grady ML, editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol 1: Assessment). 2008. Rockville (MD): Agency for Healthcare Research and Quality.

8. Leape LL, Abookire SA. World alliance for patient safety: WHO draft guidelines for adverse event reporting and learning systems: from information to action. World Health Organization. January 2005. [Accessed 14th Sept 2022]. Available from: https://www.who.int/publications/i/item/WHO-EIP-SPO-QPS-05.3.

9. Liang C, Gong Y. Knowledge representation in Patient Safety Reporting: an Ontological Approach. J Data Inform Sci. 2016;1(2):75-91. https://doi.org/10.2039/jdis.201615.

10. Blackburn B, Smallwood R, Earley S. Information Organization and Classification: Taxonomies and Metadata. Information Governance. 2012. p. 355 - 84. [Accessed 14th Sept 2022]. Available from: https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781118433829.app1.

11. Sherman H, Castro G, Fletcher M, Hatlie M, Hibbert P, Jakob R, et al. Towards an international classification for Patient Safety: the conceptual framework. Int J Qual Health Care. 2009;21(1):2-8. https://doi.org/10.1093/intqhc/mzn054.

12. World Health Organization & WHO Patient Safety. Conceptual Framework for the International Classification for Patient Safety Version 1.1: final technical report. World Health Organization; January 2009. [Accessed 14th Sept 2022]. Available from: https://apps.who.int/iris/handle/10665/70882.

13. Runciman W, Hibbert P, Thomson R, Van Der Schaaf T, Sherman H, Lewalle P. Towards an international classification for Patient Safety: key concepts and terms. Int J Qual Health Care. 2009;21(1):18-26. https://doi.org/10.1093/intqhc/mzn057.

14. Carson-Stevens A, Hibbert P, Avery A, Butlin A, Carter B, Cooper A, et al. A cross-sectional mixed methods study protocol to generate learning from patient safety incidents reported from general practice. BMJ Open. 2015;5(12). https://doi.org/10.1136/bmjopen-2015-009079.

15. Nickerson RC, Varshney U, Muntermann J, Isaac H, editors. Taxonomy development in information systems: developing a taxonomy of mobile applications. ECIS; 2009. https://aisel.aisnet.org/ecis2009/388.

16. World Health Organization. ICD-11 Fact Sheet. 9. Feb 2022. (Accessed 22nd July 2023). Available from: https://www.who.int/publications/m/item/icd-11-fact-sheet.

17. World Health Organization. Emergency use ICD codes for COVID-19 disease outbreak. [Accessed 14th Sept 2022]. Available from: https://www.who.int/standards/classifications/classification-of-diseases/emergency-use-icd-codes-for-covid-19-disease-outbreak.

18. Williams H, Edwards A, Hibbert P, Rees P, Prosser Evans H, Panesar S, et al. Harms from discharge to primary care: mixed methods analysis of incident reports. Br J Gen Pract. 2015;65(641):e829-37. https://doi.org/10.3399/bjgp15X687877.

19. Rees P, Edwards A, Powell C, Hibbert P, Williams H, Makeham M, et al. Patient safety incidents involving Sick Children in Primary Care in England and Wales: a mixed methods analysis. PLoS Med. 2017;14(1):e1002217. https://doi.org/10.1371/journal.pmed.1002217.

20. Cooper A, Edwards A, Williams H, Evans HP, Avery A, Hibbert P, et al. Sources of unsafe primary care for older adults: a mixed-methods analysis of patient safety incident reports. Age Ageing. 2017;46(5):833-9. https://doi.org/10.1093/ageing/afx044.

21. Yardley I, Yardley S, Williams H, Carson-Stevens A, Donaldson LJ. Patient safety in palliative care: a mixed-methods study of reports to a national database of serious incidents. Palliat Med. 2018;32(8):1353-62. https://doi.org/10.1177/0269216318776846.

22. Dinnen T, Williams H, Yardley S, Noble S, Edwards A, Hibbert P, et al. Patient safety incidents in advance care planning for serious illness: a mixed-methods analysis. BMJ Support Palliat Care. 2019. https://doi.org/10.1136/bmjspcare-2019-001824.

23. Gibson R, MacLeod N, Donaldson LJ, Williams H, Hibbert P, Parry G, et al. A mixed-methods analysis of patient safety incidents involving opioid substitution treatment with methadone or buprenorphine in community-based care in England and Wales. Addiction. 2020;115(11):2066-76. https://doi.org/10.1111/add.15039.

24. Hernan AL, Giles SJ, Carson-Stevens A, Morgan M, Lewis P, Hind J, et al. Nature and type of patient-reported safety incidents in primary care: cross-sectional survey of patients from Australia and England. BMJ Open. 2021;11(4):e042551. https://doi.org/10.1136/bmjopen-2020-042551.

25. Glaser BG, Strauss AL. The Discovery of grounded theory: strategies for qualitative research. 1st ed. Routledge; 1999. https://doi.org/10.4324/9780203793206.

26. Carson-Stevens A, Hibbert P, Williams H, Evans HP, Cooper A, Rees P, et al. Characterising the nature of primary care patient safety incident reports in the England and Wales National Reporting and Learning System: a mixed-methods agenda-setting study for general practice. Southampton (UK): NIHR Journals Library; 2016 Sep. https://doi.org/10.3310/hsdr04270.

27. Hallingberg B, Williams D, Cannings-John R, Hughes K, Torrens-Burton A, Gillespie D et al. Protocol for a longitudinal mixed-methods study of psychosocial determinants of health behaviour, health and well-being outcomes during the COVID-19 pandemic: The UK COVID-19 Public Experiences (COPE) Study [Internet]. Cardiff Metropolitan University; 2021 (Accessed 18th July 2023). Available from: https://figshare.cardiffmet.ac.uk/articles/preprint/Protocol_for_a_longitudinal_mixed-methods_study_of.psychosocial_determinants_of_health behaviour health_and_well-being_outcomes_during_the_COVID-19_pandemic_The_UK_COVID-19 Study/14184857/1.

28. Hurt L, Ashfield-Watt P, Townson J, Heslop L, Copeland L, Atkinson MD, et al. Cohort profile: HealthWise Wales. A research register and population health data platform with linkage to National Health Service data sets in Wales. BMJ Open. 2019;9(12):e031705. https://doi.org/10.1136/bmjopen-2019-031705.

29. Fournier JP, Amelineau JB, Hild S, Nguyen-Soenen J, Daviot A, Simonneau B, et al. Patient-safety incidents during COVID-19 health crisis in France: an exploratory sequential multi-method study in primary care. Eur J Gen Pract. 2021;27(1):142-51. https://doi.org/10.1080/13814788.2021.1945029.

30. Google Translate. [Accessed 14th Sept 2022]. Available from: https://translate.google.co.uk/.

31. Makeham MA, Stromer S, Bridges-Webb C, Mira M, Saltman DC, Cooper C, et al. Patient safety events reported in general practice: a taxonomy. Qual Saf Health Care. 2008;17(1):53-7. https://doi.org/10.1136/qshc.2007.022491.

32. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3(2):77-101. https://doi.org/10.1191/147808706qp0630a.

33. Koffman J, Gross J, Etkind SN, Selman L, Uncertainty, and COVID-19: how are we to respond? J R Soc Med. 2020;113(6):211-6. https://doi.org/10.1177/0141076820930665.
⁷Australian Institute of Health Innovation, Macquarie University, Sydney, Australia

⁸Allied Health and Human Performance, IIMPACT in Health, University of South Australia, Adelaide, Australia

⁹Cardiff School of Sport and Health Sciences, Cardiff Metropolitan University, Cardiff, UK

Received: 15 September 2022 / Accepted: 6 October 2023

Published online: 14 October 2023

## References

1. World Health Organization News release. Essential health services face continued disruption during COVID-19 pandemic. 7 Feb 2022. [Accessed 14th Sept 2022]. Available from: https://www.who.int/news/item/07-02-2022-essential-health-services-face-continued-disruption-during-covid-19-pandemic.

2. Reed SJ, Schlepper L, Edwards N. Health system recovery from Covid-19: International lessons for the NHS. Report, Nuffield Trust; 2022.

3. Dallat C, Salmon PM, Goode N. Risky systems versus risky people: to what extent do risk assessment methods consider the systems approach to accident causation? A review of the literature. Saf Sci. 2019;119:266-79. https://doi.org/10.1016/j.ssci.2017.03.012.

4. World Health Organization. Patient safety incident reporting and learning systems: technical report and guidance. 16 September 2020. [Accessed 14th Sept 2022]. Available from: https://www.who.int/publications/i/item/9789240010338.

5. Liang C. Understanding Patient Safety Reports via Multi-label Text Classification and Semantic Representation. 2017. UT SBMI Dissertations (Open Access). 35. [Accessed on 14th Sept 2022]. Available from: https://digitalcommons.library.tmc.edu/uthshis_dissertations/35.

6. Vincent CA. Analysis of clinical incidents: a window on the system not a search for root causes. Qual Saf Health Care. 2004;13(4):242-3. https://doi.org/10.1136/qshc.2004.010454.

7. Pronovost PJ, Morlock LL, Sexton JB, Miller MR, Holzmueller CG, Thompson DA et al. Improving the Value of Patient Safety Reporting Systems. In: Henriksen K, Battles JB, Keyes MA, Grady ML, editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol 1: Assessment). 2008. Rockville (MD): Agency for Healthcare Research and Quality.

8. Leape LL, Abookire SA. World alliance for patient safety: WHO draft guidelines for adverse event reporting and learning systems: from information to action. World Health Organization. January 2005. [Accessed 14th Sept 2022]. Available from: https://www.who.int/publications/i/item/WHO-EIP-SPO-QPS-05.3.

9. Liang C, Gong Y. Knowledge representation in Patient Safety Reporting: an Ontological Approach. J Data Inform Sci. 2016;1(2):75-91. https://doi.org/10.20309/jdis.201615.

10. Blackburn B, Smallwood R, Earley S. Information Organization and Classification: Taxonomies and Metadata. Information Governance. 2012. p. 355 - 84. [Accessed 14th Sept 2022]. Available from: https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781118433829.app1.

11. Sherman H, Castro G, Fletcher M, Hatlie M, Hibbert P, Jakob R, et al. Towards an international classification for Patient Safety: the conceptual framework. Int J Qual Health Care. 2009;21(1):2-8. https://doi.org/10.1093/intqhc/mzn054.

12. World Health Organization & WHO Patient Safety. Conceptual Framework for the International Classification for Patient Safety Version 1.1: final technical report. World Health Organization; January 2009. [Accessed 14th Sept 2022]. Available from: https://apps.who.int/iris/handle/10665/70882.

13. Runciman W, Hibbert P, Thomson R, Van Der Schaaf T, Sherman H, Lewalle P. Towards an international classification for Patient Safety: key concepts and terms. Int J Qual Health Care. 2009;21(1):18-26. https://doi.org/10.1093/intqhc/mzn057.

14. Carson-Stevens A, Hibbert P, Avery A, Butlin A, Carter B, Cooper A, et al. A cross-sectional mixed methods study protocol to generate learning from patient safety incidents reported from general practice. BMJ Open. 2015;5(12). https://doi.org/10.1136/bmjopen-2015-009079.

15. Nickerson RC, Varshney U, Muntermann J, Isaac H, editors. Taxonomy development in information systems: developing a taxonomy of mobile applications. ECIS; 2009. https://aisel.aisnet.org/ecis2009/388.

16. World Health Organization. ICD-11 Fact Sheet. 9. Feb 2022. (Accessed 22nd July 2023). Available from: https://www.who.int/publications/m/item/icd-11-fact-sheet.

17. World Health Organization. Emergency use ICD codes for COVID-19 disease outbreak. [Accessed 14th Sept 2022]. Available from: https://www.who.int/standards/classifications/classification-of-diseases/emergency-use-icd-codes-for-covid-19-disease-outbreak.

18. Williams H, Edwards A, Hibbert P, Rees P, Prosser Evans H, Panesar S, et al. Harms from discharge to primary care: mixed methods analysis of incident reports. Br J Gen Pract. 2015;65(641):e829-37. https://doi.org/10.3399/bjgp15X687877.

19. Rees P, Edwards A, Powell C, Hibbert P, Williams H, Makeham M, et al. Patient safety incidents involving Sick Children in Primary Care in England and Wales: a mixed methods analysis. PLoS Med. 2017;14(1):e1002217. https://doi.org/10.1371/journal.pmed.1002217.

20. Cooper A, Edwards A, Williams H, Evans HP, Avery A, Hibbert P, et al. Sources of unsafe primary care for older adults: a mixed-methods analysis of patient safety incident reports. Age Ageing. 2017;46(5):833-9. https://doi.org/10.1093/ageing/afx044.

21. Yardley I, Yardley S, Williams H, Carson-Stevens A, Donaldson LJ. Patient safety in palliative care: a mixed-methods study of reports to a national database of serious incidents. Palliat Med. 2018;32(8):1353-62. https://doi.org/10.1177/0269216318776846.

22. Dinnen T, Williams H, Yardley S, Noble S, Edwards A, Hibbert P, et al. Patient safety incidents in advance care planning for serious illness: a mixed-methods analysis. BMJ Support Palliat Care. 2019. https://doi.org/10.1136/bmjspcare-2019-001824.

23. Gibson R, MacLeod N, Donaldson LJ, Williams H, Hibbert P, Parry G, et al. A mixed-methods analysis of patient safety incidents involving opioid substitution treatment with methadone or buprenorphine in community-based care in England and Wales. Addiction. 2020;115(11):2066-76. https://doi.org/10.1111/add.15039.

24. Hernan AL, Giles SJ, Carson-Stevens A, Morgan M, Lewis P, Hind J, et al. Nature and type of patient-reported safety incidents in primary care: cross-sectional survey of patients from Australia and England. BMJ Open. 2021;11(4):e042551. https://doi.org/10.1136/bmjopen-2020-042551.

25. Glaser BG, Strauss AL. The Discovery of grounded theory: strategies for qualitative research. 1st ed. Routledge; 1999. https://doi.org/10.4324/9780203793206.

26. Carson-Stevens A, Hibbert P, Williams H, Evans HP, Cooper A, Rees P, et al. Characterising the nature of primary care patient safety incident reports in the England and Wales National Reporting and Learning System: a mixed-methods agenda-setting study for general practice. Southampton (UK): NIHR Journals Library; 2016 Sep. https://doi.org/10.3310/hsdr04270.

27. Hallingberg B, Williams D, Cannings-John R, Hughes K, Torrens-Burton A, Gillespie D et al. Protocol for a longitudinal mixed-methods study of psychosocial determinants of health behaviour, health and well-being outcomes during the COVID-19 pandemic: The UK COVID-19 Public Experiences (COPE) Study [Internet]. Cardiff Metropolitan University; 2021 (Accessed 18th July 2023). Available from: https://figshare.cardiffmet.ac.uk/articles/preprint/Protocol_for_a_longitudinal_mixed-methods_study_of.psychosocial_determinants_of_health behaviour health_and_well-being_outcomes_during_the_COVID-19_pandemic_The_UK_COVID-19 Study/14184857/1.

28. Hurt L, Ashfield-Watt P, Townson J, Heslop L, Copeland L, Atkinson MD, et al. Cohort profile: HealthWise Wales. A research register and population health data platform with linkage to National Health Service data sets in Wales. BMJ Open. 2019;9(12):e031705. https://doi.org/10.1136/bmjopen-2019-031705.

29. Fournier JP, Amelineau JB, Hild S, Nguyen-Soenen J, Daviot A, Simonneau B, et al. Patient-safety incidents during COVID-19 health crisis in France: an exploratory sequential multi-method study in primary care. Eur J Gen Pract. 2021;27(1):142-51. https://doi.org/10.1080/13814788.2021.1945029.

30. Google Translate. [Accessed 14th Sept 2022]. Available from: https://translate.google.co.uk/.

31. Makeham MA, Stromer S, Bridges-Webb C, Mira M, Saltman DC, Cooper C, et al. Patient safety events reported in general practice: a taxonomy. Qual Saf Health Care. 2008;17(1):53-7. https://doi.org/10.1136/qshc.2007.022491.

32. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3(2):77-101. https://doi.org/10.1191/147808706qp0630a.

33. Koffman J, Gross J, Etkind SN, Selman L, Uncertainty, and COVID-19: how are we to respond? J R Soc Med. 2020;113(6):211-6. https://doi.org/10.1177/0141076820930665.
34. Gallagher TH, Mazor KM. Taking complaints seriously: using the patient safety lens. BMJ Qual Saf. 2015;24:352–5. https://doi.org/10.1136/bmjqs-2015-004337.

35. Poldervaart JM, van Melle MA, Reijnders LJ, de Wit NJ, Zwart DL. Transitional safety incidents as reported by patients and healthcare professionals in the Netherlands: a descriptive study. Eur J Gen Pract. 2019;25(2):77–84. https://doi.org/10.1080/13814788.2018.1543396.

36. Ward JK, Armitage G. Can patients report patient safety incidents in a hospital setting? A systematic review. BMJ Qual Saf. 2012;21(8):685–99. https://doi.org/10.1136/bmjqs-2011-000213.

37. Vincent C, Davis R. Patients and families as safety experts. CMAJ. 2012;184(1):15–6. https://doi.org/10.1503/cmaj.111311.

38. Park M, Giap TT. Patient and family engagement as a potential approach for improving patient safety: a systematic review. J Adv Nurs. 2020;76(1):62–80. https://doi.org/10.1111/jan.14227.

39. Carr A, Smith JA, Camaradou J, Prieto-Alhambra D. Growing backlog of planned surgery due to covid-19. BMJ. 2021;372:n339. https://doi.org/10.1136/bmjn.339.

40. Riera R, Bagattini ÂM, Pacheco RL, Pachito DV, Roitberg F, Ilbawi A. Delays and disruptions in Cancer Health Care due to COVID-19 pandemic: systematic review. JCO Global Oncology. 2021;7311–23. https://doi.org/10.1200/GO.20.00639.

41. Gultekin M, Ak S, Ayhan A, Strojna A, Pletnev A, Fagotti A, et al. Perspectives, fears and expectations of patients with gynaecological cancers during the COVID-19 pandemic: a pan-european study of the European Network of Gynaecological Cancer Advocacy Groups (ENGAGe). Cancer Med. 2021;10(1):208–19. https://doi.org/10.1002/cam4.3605.

42. Apisarnthanarak A, Siripraparat C, Apisarnthanarak P, Ullman M, Saengaram P, Leeprechanon N, et al. Patients' anxiety, fear, and panic related to coronavirus disease 2019 (COVID-19) and confidence in hospital infection control policy in outpatient departments: a survey from four thai hospitals. Infect Control Hosp Epidemiol. 2021;42(10):1288–90. https://doi.org/10.1017/ice.2020.1240.

43. Czeisler ME, Marynak K, Clarke KEN, Salah Z, Shakya I, Thierry JM, et al. Delay or Avoidance of Medical Care because of COVID-19-Related concerns - United States, June 2020. MMWR Morb Mortal Wkly Rep. 2020;69(36):1250–7. https://doi.org/10.15585/mmwr.mm6936a4.

44. Fung THM, Kuet ML, Patel MK, Puri P. Addressing COVID-19 fear to improve clinic attendance for patients with wet age-related macular degeneration. Acta Ophthalmol. 2021;99(2):e285. https://doi.org/10.1111/aos.14520.

45. McFadzean IJ, Davies K, Purchase T, et al. Patient safety in prisons: a multi-method analysis of reported incidents in England. J R Soc Med. 2023;0(0). https://doi.org/10.1177/01410768231166138.

46. O'Hara JK, Reynolds C, Moore S, Armitage G, Sheard L, Marsh C, et al. What can patients tell us about the quality and safety of hospital care? Findings from a UK multicentre survey study. BMJ Qual Saf. 2018;27(9):673–82. https://doi.org/10.1136/bmjqs-2017-006974.

47. Jolley S. Assessing patients' knowledge and fears about MRSA infection. Nurs Times. 2008;104:32–3.

48. an Melle MA, Erkelens DCA, van Stel HF, et al. Pilot study on identification of incidents in healthcare transitions and concordance between medical records and patient interview data. BMJ Open. 2016;6:e011368. https://doi.org/10.1136/bmjopen-2016-011368.

49. Schöpf AC, Vach W, Jakob M, Saxer F. Routine patient surveys: patients' preferences and information gained by healthcare providers. PLoS ONE. 2019;14(8):e0220495. https://doi.org/10.1371/journal.pone.0220495.

50. Shojania KG. The frustrating case of incident-reporting systems. Qual Saf Health Care. 2008;17(6):400–2. https://doi.org/10.1136/qshc.2008.029496.

51. Nilan J, Doltani D, Harmon D. Assessment of patient concerns: a review. Ir J Med Sci. 2018;187:545–51. https://doi.org/10.1007/s11845-018-1741-1.

52. Ramírez E, Martín A, Villán Y, Lorente M, Ojeda J, Moro M, et al. Effectiveness and limitations of an incident-reporting system analyzed by local clinical safety leaders in a tertiary hospital: prospective evaluation through real-time observations of patient safety incidents. Med (Baltim). 2018;97(38):e12509. https://doi.org/10.1097/MD.0000000000012509.

53. Withers K, Palmer R, Lewis S, Carolan-Rees G. First steps in PROMs and PREMs collection in Wales as part of the prudent and value-based healthcare agenda. Qual Life Res. 2021;30(11):3157–70. https://doi.org/10.1007/s11136-020-02711-2.

## Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
34. Gallagher TH, Mazor KM. Taking complaints seriously: using the patient safety lens. BMJ Qual Saf. 2015;24:352–5. https://doi.org/10.1136/bmjqs-2015-004337.

35. Poldervaart JM, van Melle MA, Reijnders LJ, de Wit NJ, Zwart DL. Transitional safety incidents as reported by patients and healthcare professionals in the Netherlands: a descriptive study. Eur J Gen Pract. 2019;25(2):77–84. https://doi.org/10.1080/13814788.2018.1543396.

36. Ward JK, Armitage G. Can patients report patient safety incidents in a hospital setting? A systematic review. BMJ Qual Saf. 2012;21(8):685–99. https://doi.org/10.1136/bmjqs-2011-000213.

37. Vincent C, Davis R. Patients and families as safety experts. CMAJ. 2012;184(1):15–6. https://doi.org/10.1503/cmaj.111311.

38. Park M, Giap TT. Patient and family engagement as a potential approach for improving patient safety: a systematic review. J Adv Nurs. 2020;76(1):62–80. https://doi.org/10.1111/jan.14227.

39. Carr A, Smith JA, Camaradou J, Prieto-Alhambra D. Growing backlog of planned surgery due to covid-19. BMJ. 2021;372:n339. https://doi.org/10.1136/bmjn.339.

40. Riera R, Bagattini AM, Pacheco RL, Pachito DV, Roitberg F, Ilbawi A. Delays and disruptions in Cancer Health Care due to COVID-19 pandemic: systematic review. JCO Global Oncology. 2021;7311–23. https://doi.org/10.1200/GO.20.00639.

41. Gultekin M, Ak S, Ayhan A, Strojna A, Pletnev A, Fagotti A, et al. Perspectives, fears and expectations of patients with gynaecological cancers during the COVID-19 pandemic: a pan-european study of the European Network of Gynaecological Cancer Advocacy Groups (ENGAGe). Cancer Med. 2021;10(1):208–19. https://doi.org/10.1002/cam4.3605.

42. Apisarnthanarak A, Siripraparat C, Apisarnthanarak P, Ullman M, Saengaram P, Leeprechanon N, et al. Patients' anxiety, fear, and panic related to coronavirus disease 2019 (COVID-19) and confidence in hospital infection control policy in outpatient departments: a survey from four thai hospitals. Infect Control Hosp Epidemiol. 2021;42(10):1288–90. https://doi.org/10.1017/ice.2020.1240.

43. Czeisler ME, Marynak K, Clarke KEN, Salah Z, Shakya I, Thierry JM, et al. Delay or Avoidance of Medical Care because of COVID-19-Related concerns - United States, June 2020. MMWR Morb Mortal Wkly Rep. 2020;69(36):1250–7. https://doi.org/10.15585/mmwr.mm6936a4.

44. Fung THM, Kuet ML, Patel MK, Puri P. Addressing COVID-19 fear to improve clinic attendance for patients with wet age-related macular degeneration. Acta Ophthalmol. 2021;99(2):e285. https://doi.org/10.1111/aos.14520.

45. McFadzean IJ, Davies K, Purchase T, et al. Patient safety in prisons: a multi-method analysis of reported incidents in England. J R Soc Med. 2023;0(0). https://doi.org/10.1177/01410768231166138.

46. O'Hara JK, Reynolds C, Moore S, Armitage G, Sheard L, Marsh C, et al. What can patients tell us about the quality and safety of hospital care? Findings from a UK multicentre survey study. BMJ Qual Saf. 2018;27(9):673–82. https://doi.org/10.1136/bmjqs-2017-006974.

47. Jolley S. Assessing patients' knowledge and fears about MRSA infection. Nurs Times. 2008;104:32–3.

48. an Melle MA, Erkelens DCA, van Stel HF, et al. Pilot study on identification of incidents in healthcare transitions and concordance between medical records and patient interview data. BMJ Open. 2016;6:e011368. https://doi.org/10.1136/bmjopen-2016-011368.

49. Schöpf AC, Vach W, Jakob M, Saxer F. Routine patient surveys: patients' preferences and information gained by healthcare providers. PLoS ONE. 2019;14(8):e0220495. https://doi.org/10.1371/journal.pone.0220495.

50. Shojania KG. The frustrating case of incident-reporting systems. Qual Saf Health Care. 2008;17(6):400–2. https://doi.org/10.1136/qshc.2008.029496.

51. Nilan J, Doltani D, Harmon D. Assessment of patient concerns: a review. Ir J Med Sci. 2018;187:545–51. https://doi.org/10.1007/s11845-018-1741-1.

52. Ramírez E, Martín A, Villán Y, Lorente M, Ojeda J, Moro M, et al. Effectiveness and limitations of an incident-reporting system analyzed by local clinical safety leaders in a tertiary hospital: prospective evaluation through real-time observations of patient safety incidents. Med (Baltim). 2018;97(38):e12509. https://doi.org/10.1097/MD.0000000000012509.

53. Withers K, Palmer R, Lewis S, Carolan-Rees G. First steps in PROMs and PREMs collection in Wales as part of the prudent and value-based healthcare agenda. Qual Life Res. 2021;30(11):3157–70. https://doi.org/10.1007/s11136-020-02711-2.

## Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.