Algorithmovigilance, lessons from pharmacovigilance

Alan Balendran, Mehdi Benchoufi, Theodoros Evgeniou, Philippe Ravaud

► To cite this version:

Alan Balendran, Mehdi Benchoufi, Theodoros Evgeniou, Philippe Ravaud. Algorithmovigilance, lessons from pharmacovigilance. npj Digital Medicine, 2024, 7 (1), pp.270. 10.1038/s41746-024-01237-y. hal-05492065

HAL Id: hal-05492065

https://hal.science/hal-05492065v1

Submitted on 3 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

https://doi.org/10.1038/s41746-024-01237-y

# Algorithmovigilance, lessons from pharmacovigilance

<figure><img src="image_3.png" /></figure>

Check for updates

Alan Balendran<fnref n="1" />✉, Mehdi Benchoufi<fnref n="1" />, Theodoros Evgeniou² & Philippe Ravaud<fnref n="1" />,³,⁴

Artificial Intelligence (AI) systems are increasingly being deployed across various high-risk applications, especially in healthcare. Despite significant attention to evaluating these systems, post-deployment incidents are not uncommon, and effective mitigation strategies remain challenging. Drug safety has a well-established history of assessing, monitoring, understanding, and preventing adverse effects in real-world usage, known as pharmacovigilance. Drawing inspiration from pharmacovigilance methods, we discuss concepts that can be adapted for monitoring AI systems in healthcare. This discussion aims to improve responses to adverse effects and potential incidents and risks associated with AI deployment in healthcare but also beyond.

As artificial intelligence (AI) systems are increasingly used across applications and sectors, ranging from healthcare to energy, financial, transportation, and others, the focus is increasingly shifting to the post-deployment stages of the AI lifecycle to ensure the safe use of these systems<fnref n="1" />. Monitoring of AI systems is becoming increasingly critical², and new tools and practices are being developed for this purpose, such as statistical tools or so-called AI incidents databases³⁻⁵. Indeed, the number of AI incidents is already large and is expected to increase. For example, 10.196 AI-related incidents and hazards have been identified since 2014⁶. While there is significant research on trustworthy AI⁷, there is only limited work on what happens *after* AI systems are deployed. Another limitation of AI monitoring is the lack of consideration of human or organizational factors⁸. For example, stakeholders like physicians who identify errors may lack effective feedback processes for reporting issues, hindering potential improvements to AI systems.

However, large-scale monitoring of technologies, broadly defined, is not a new challenge. Healthcare regulators, such as the US Food and Drug Administration (FDA), use processes and practices to monitor potential negative side effects of drugs in the market despite rigorous clinical trials in their pre-approval stages⁹. For instance, each year, regulators and manufacturers recall, on average, more than 1300 drugs, with more than half potentially causing temporary or serious health problems¹⁰. According to the FDA, a recall is defined as a manufacturer's removal or correction of a marketed product that the FDA considers to be in violation of the laws it administers¹¹.

Given the scale and potential of AI medical devices¹², ensuring their proper monitoring and potential recall or withdrawal is necessary to limit their potential negative impact, similar to drugs. While AI is not the same as drugs, we argue that a lot can be learned from the field of pharmacovigilance¹³.

We argue that drug development can also be seen as a kind of black box in some situations. For some widely prescribed drugs, it has been proven that their precise mechanisms of action sometimes remain unknown or not well understood, and, in some cases, have been secondarily proven to be different from what was initially thought¹⁴. Inspired by pharmacovigilance, which encompasses the assessment, monitoring, comprehension, and prevention of adverse effects associated with the use of drugs, Embi proposed in 2021, the term *algorithmovigilance* to transpose the same concepts for adverse effects associated with algorithms in healthcare¹⁵. Building on this term, we explore five key characteristics from pharmacovigilance that can be relevant for algorithmovigilance—also summarized in Table 1.

## Post-approval evaluation

Current AI monitoring techniques are mostly based on the use of statistical tests from data capturing the behavior (e.g., various error rates) of the AI systems. These methods typically detect potential shifts in the data/ environment, such as covariate and concept shifts¹⁶⁻¹⁸. These tests are often automated (e.g., tests are computed every month or accuracy is aggregated over a couple of weeks long window), aligning with the emergence of MLOps aimed at operationalizing ML monitoring⁹. However, a significant number of FDA-authorized AI medical devices lack multi-site validation, and even fewer have undergone prospective evaluation¹⁹. This approach may inadequately reflect real-world performance—even within the same institution. Moreover, relying solely on algorithmic performance creates the risk of downplaying the importance of human factors who will be in “contact” with the AI system: patients, physicians, caregivers, AI operators, AI researchers, developers, hospitals, regulators, etc. For example, a study of an AI system for detecting diabetic retinopathy across multiple clinics in Thailand evaluated how the AI system affected patients and the way nurses

<footnote n="1">¹Université Paris Cité and Université Sorbonne Paris Nord, Inserm, INRAE, Center for Research in Epidemiology and StatisticS (CRESS), Paris, France. ²INSEAD, Fontainebleau, France. ³Centre d'Epidémiologie Clinique, AP-HP, Hôpital Hôtel-Dieu, Paris, France. ⁴Columbia University Mailman School of Public Health, Department of Epidemiology, New York, NY, USA. ✉e-mail: alan.balendran@u-paris.fr</footnote>
<table><caption>Table 1 | Translating key concepts from pharmacovigilance to AI surveillance and monitoring (namely, Algorithmovigilance)</caption><thead><tr><th>Practice</th><th>Pharmacovigilance</th><th>Algorithmovigilance</th></tr></thead><tbody><tr><td>Post-approval evaluation</td><td><ul><li>Assessment done using observational data after post-market approval.</li><li>Investigates real-world safety in a larger and more representative population (larger population, more representative population, longer follow-up time, different dosages, and different outcomes)</li><li>Assessment done at different points in a drug's lifecycle (e.g. Periodic Update Safety Report (PSUR))</li></ul></td><td><p><strong>What needs to be done:</strong></p><ul><li>Assessment of model generalizability using external data after model approval</li><li>Identify model blind spots such as extreme/hard cases, racial bias, spurious correlations using data collected from different institutions, at different temporality, and for different tasks</li><li>Identify model impact on the clinical workflow (e.g. healthcare professionals and patients dissatisfaction, difficulty to use, perceived benefits)</li></ul></td></tr><tr><td>Case reporting</td><td><p><strong>Who can report:</strong></p><ul><li>Patients, caregivers, healthcare professionals, healthcare institutions, and drug manufacturers.</li><li><strong>Content of report:</strong></li><li>Patient information</li><li>Adverse reactions description</li><li>Information related to the suspected drug(s)</li><li>Information on management of the adverse reactions</li><li>Information about the reporter</li><li><strong>Where to report:</strong></li><li>MedWatch</li><li>EudraVigilance</li></ul></td><td><p><strong>What needs to be done:</strong></p><ul><li>Case reports for adverse AI-system reaction (AAIR):</li><li><strong>Who can report:</strong></li><li>Patients, healthcare professionals, AI operators, AI researchers, hospitals, regulators, and AI system developers.</li><li><strong>Content of report:</strong></li><li>Patient information</li><li>Adverse reactions description</li><li>Information related to the AI systems (version details, hardware, software information)</li><li>Data preprocessing pipelines</li><li>Sequence of events leading up to the error and work environment</li><li>Details on other systems that impacted the model's decision</li><li>Information on the AI developers</li><li>Information about the reporter</li><li><strong>Where to report:</strong></li><li>GitHub/GitLab for non-sensitive data</li><li>Private database for sensitive data</li></ul></td></tr><tr><td>Data standardization</td><td><p><strong>ISO standard (HL7 27953-2:2011)</strong></p><ul><li>Standardized framework for the reporting of case reports</li><li><strong>Medical Dictionary for Regulatory Activities (MedDRA)</strong></li><li>Terminology developed for the evaluation of safety monitoring of medicinal products through all phases of the development cycle (i.e., from clinical trials to post-marketing surveillance).</li><li><strong>Common Data Model</strong></li><li>Standardized vocabularies to harmonize the structure and content of observational data to enable efficient analyses and allow interoperability across various clinical domains.</li></ul></td><td><p><strong>What needs to be done:</strong></p><ul><li>Harmonization of data information (data acquisition devices and protocols, data quality control method, data annotation/labeling procedures, data management, population characteristics, and pipelines)</li><li>Harmonization of vocabulary used for model information (model specification, training information, and evaluation metric(s)) to allow interoperability across different hospitals</li></ul></td></tr><tr><td>Causality assessment</td><td><p><strong>Rechallenge/Dechallenge:</strong></p><ul><li>Rechallenge (giving the drug again) or dechallenge (stopping the drug) to determine likelihood of causal relationship between drug and adverse event(s).</li><li><strong>Causality assessment:</strong></li><li>Classify potential relationships between drugs and suspected adverse drug reaction into different categories:</li><li>Certain, Probable/Likely, Possible, Unlikely, Conditional/ Unclassified, Unassessable/Unclassifiable.</li><li><strong>Concomitant Drugs:</strong></li><li>Check for other concomitant drugs that may cause or be related to the adverse event(s)</li></ul></td><td><p><strong>What needs to be done:</strong></p><ul><li><strong>Reproducibility tests:</strong></li><li>Conduct reproducibility tests to identify the conditions under which the error occurred and to stress-test in alternative settings, identifying potential similar errors.</li><li><strong>Ablation studies:</strong></li><li>Perform ablation studies to pinpoint the component responsible for the incident (e.g., acquisition devices, preprocessing software, interpretability tools, biases in training data, human interpretation)</li><li><strong>Retraining/Redesigning:</strong></li><li>Consider potential retraining or redesigning of the AI system, incorporating new data or adjusting outcome variables in certain cases.</li></ul></td></tr><tr><td>Adverse events dissemination</td><td><p><strong>For healthcare professionals:</strong></p><ul><li>Summary of Product Characteristics (SmPC) accompanies authorized drugs, presenting key information on adverse events updated based on post-market authorization studies:</li><li>Adverse events are categorized based on seriousness and frequency: very common (≥1/10); common (≥1/10 to &lt;1/10); uncommon (≥1/1000 to &lt;1/100); rare (≥1/10,000 to &lt;1/1000); very rare (&lt;1/10,000).</li><li>Adverse events are stratified by specific groups of population (e.g. pediatric, elderly patients, patients with renal or hepatic impairment, and patients with other diseases or specific genotypes)</li><li><strong>For patients:</strong></li><li>Patient Medication Information and Package leaflet intended for patients in an understandable and concise way to ensure safe and effective use of drugs and prevention of side effects.</li></ul></td><td><p><strong>What needs to be done:</strong></p><ul><li>Standardization of documentation for healthcare professionals and AI operators:</li><li>Document that accompanies each AI-system in healthcare describing the different known adverse events in a tabulated format based on seriousness and frequency.</li><li>Indicate model risks and performance, or absence of reliable information, on specific patient subgroups (e.g. by age, race, gender, patients at high/low risks, patients with other comorbidities, etc.)</li><li>Standardization of documentation for patients:</li><li>Easily accessible and understandable document for patients informing them about the AI model's functionality, the population on which the model has been tested, its impact on decision-making, and expected adverse events.</li><li><strong>Database to improve transparency:</strong></li><li>Database compiling the different known adverse events related to AI systems in healthcare. Incidents can be filtered by type of model, medical application, frequency, population at risk, and severity of the incident.</li></ul></td></tr></tbody></table>

made decisions based on the model's predictions²⁰. Current methods lack strategies to address post-deployment scenarios of AI systems.

For drugs, eventual studies conducted to evaluate the tolerance during the development of a drug are insufficient to identify all potential adverse

reactions, especially rare adverse events, due to limited sample sizes (e.g., studies generally do not exceed 3000 participants) and non-representative trial population (e.g., does not include patients with more or other comorbidities, younger and older patients, use of concomitant drugs, etc.)²¹.
In contrast, safety studies investigate drug safety in the real world across diverse conditions, including a larger population, more representative population, longer follow-up time, different dosages, and different outcomes²². For example, the Periodic Update Safety Report (PSUR) is a structured report that must be produced after its approval at different points in a drug's lifecycle with updated information about its safety²³.

Similarly, AI systems require evaluation beyond their pre-approval assessments. This should be carried out using diverse data from various sources, including data from the same institution but collected at different times (temporal generalizability), different institutions (geographic generalizability), and different clinical environments (domain generalizability)²⁴. This is crucial to identify an AI system's blind spots missed during pre-approval evaluation, such as hard or extreme cases, racial bias, or spurious correlations²⁵,²⁶. Human evaluation, assessing the AI system's impact on the clinical workflow and patient (such as user dissatisfaction, difficulty in use, perceived benefits, etc.)²⁷⁻²⁹, should also complement pre-market evaluations. This process should ultimately result in a periodic update of the AI system findings based on the evidence learned from multiple validations, similar to the PSUR.

## Case reporting

When an adverse AI-system reaction (AAIR) occurs, it is important to report information related to the event. Current reports typically consist of automatic logs documenting each step of the AI system execution. However, these logs may lack context regarding the clinical environment in which the error occurred and prove challenging to interpret, hindering error reproduction.

For drugs, reporting of a suspected adverse event can be issued by anyone: patients, caregivers, healthcare professionals, and manufacturers. Reports can take different forms: spontaneous and voluntary reports, or mandatory reports. Each case report is populated not only by information about the drug and patient's characteristics but also other information, such as the environment in which it occurred⁸. For a drug, this corresponds to information related to the suspected drug(s) (name of the drug(s), dosage, therapy date, batch number, reason for administration), description of the adverse reaction, information on the management of the adverse reactions (what assisted for the confirmation of the reaction, criteria for regarding the reaction as serious, use of other treatments after patient experiencing the adverse reaction, outcome of the adverse reaction), use of other concomitant drugs, the work environment, the personnel involved, information about the reporter, and any key contributing factors³⁰,³¹. These reports are then collected, curated, and aggregated in publicly available databases like the FDA Adverse Event Reporting System (FAERS) through MedWatch or the EudraVigilance database from the EMA. Data protection is ensured by providing limited access based on the stakeholders³². These reports are critical as they are then used to make decisions about a drug, such as updating its labeling information or, in some cases, removing it from the market.

Similarly, reporting of incidents involving an AI system in healthcare should be accessible to anyone who will be “in contact” with the AI system: patients, healthcare professionals, AI operators, hospitals, AI researchers, regulators, and AI system developers. Methods to report AI-related suspected errors could be inspired by other mature communities. For instance, the software development community's effective bug reporting, discussion, and tracking, facilitated by platforms like GitHub or GitLab, offer valuable insights. Sensitive data could be reported in private databases where privacy-preserving techniques such as differential privacy or synthetic data could be used to preserve data privacy while enabling robust reporting. Additionally, restricting access to case reports to designated entities responsible for analysis can enhance oversight and accountability. Reports should provide details on both technical errors and the clinical environment, ensuring comprehensive understanding and reproducibility of incidents. This may include specifics on the adverse event, version details of the AI system, hardware and software information, pipelines used to process the data, details on other systems that impacted the model's decision (such as any

acquisition devices, preprocessing tool(s), or interpretability tool(s)), latency, data quality assessment, difficulties or issues encountered when using the AI model, the sequence of events leading up to the error, the work environment, information on AI operators, and any other information.

## Data standardization

Data harmonization is essential in analyzing reports from various sources to reduce misclassification and information bias. For AI, the coding of data and concepts is based on the healthcare system adopted by the hospital or healthcare institutions. Discrepancies in coding standards can occur not only across different institutions but also within the same hospital when transitioning between different healthcare systems. Moreover, the standardization of AI development processes is also lacking. Risks of not having standardized concepts can hinder interoperability, particularly when multiple hospitals with different coding standards encounter the same AI system incident. The lack of uniformity in coding may result in disparate reports, hindering accurate incident assessment.

For drugs, to improve the analysis of drug-related adverse events, different strategies have been adopted to improve consistency across reports. For instance, the ISO/HL7 27953-2:2011 standard offers a common framework of data elements and a messaging format for transmitting adverse event reports. Additionally, initiatives like the Medical Dictionary for Regulatory Activities (MedDRA) propose common terminology for regulatory information exchange concerning medical products, including pharmaceuticals and drug-device combination products³³. MedDRA has been translated into multiple languages to enhance interoperability across different linguistic contexts. Moreover, as discussed in the first section, evidence on drug safety can also be derived from observational studies, therefore, efforts to standardize terminology for observational data have been initiated. For instance, the Common Data Model project aims to standardize the structure and content of observational data, enabling subsequent analysis for drug safety surveillance³⁴.

Similarly, coding information related to AI systems encompasses both data and model aspects. Information related to the data acquisition devices and protocols, data quality control method, data annotation/labeling procedures, data management, population characteristics, and pipelines should be consistent across different hospitals. Initiatives such as *Healthsheets* have started to identify the essential information that needs to be reported for a dataset in healthcare³⁵. Moreover, standardizing information regarding the model development protocol, such as model specification, training details, and evaluation procedures (similar to Clinical Operations “ClinOps” for clinical trials) would enhance interoperability and facilitate comparisons among different AI systems.

## Causality assessment

Existing methods for monitoring AI systems are mainly focused on detecting deviations through statistical approaches. Less emphasis is placed on understanding and correcting the deviations. Errors related to AI systems can originate from various factors, including the data quality, the underlying bias in the data, acquisition devices, other software used in the workflow, medical treatments, and the skills and experience of the AI operators¹⁶,²⁵,²⁶. Undervaluing the importance of correcting AI system deviations may lead to situations where the system is disabled without attempting correction, adversely impacting the clinical workflow in which the AI system is integrated.

For drugs, investigating the relationship between a drug and a potential adverse effect is essential³⁶. While reports of adverse events provide valuable safety signals, they only indicate a potential relationship between a drug and an adverse effect, which is why they are also often referred to as suspected adverse drug reactions. Investigating whether a drug actually caused an adverse event is important for several reasons: it can provide more evidence about the drug-adverse effect relationship, it can prevent a drug from being withdrawn if it is found to actually not be the cause of the adverse event, and it can allow the identification of other drugs that may provide a better explanation for the focal adverse event. Various methods, including clinical
judgment and probabilistic methods, can be used to assess clinical plausibility<sup>37,38</sup>. For instance, one method used to determine the likelihood of a causal relationship between a drug and an adverse event is to conduct a rechallenge (or dechallenge) by giving again (or stopping) the drug to see if the adverse event reappears (or resolves). Reports are then classified into different causality categories: Certain, Probable/Likely, Possible, Unlikely, Conditional/Unclassified, Unassessable/Unclassifiable<sup>39</sup>.

Similarly, for an AI system, after an error has been detected, systematic reproducibility tests should be conducted to identify the exact conditions under which the error occurred and stress-test alternative settings, to identify potential similar errors. Ablation studies should be explored to identify the plausibility of each part of the AI system in the incident. In some cases, this could also motivate the retraining or redesigning of the AI system on a new batch of data or using a different outcome variable while ensuring (using the proper testing method) that the updated model does not result in a performance decline on existing data<sup>16,25,40</sup>. Versioning platforms such as GitHub and GitLab facilitate tracking model changes across versions. However, we note that current regulation mainly permits marketing approval for so-called locked AI systems<sup>2</sup>, although the FDA is investigating new regulations for adaptive AI systems in healthcare<sup>41</sup>. But this should not discourage the use of methods to control various aspects of a model after deployment, for instance, post-hoc explainability or fairness methods to investigate already deployed models without modifying or retraining them<sup>25,26,42,43</sup>.

## Adverse events dissemination

Effective dissemination of adverse events is crucial for preventing and mitigating their impact on end-users of AI systems. Several AI-related incident databases, such as the Algorithmic and Automation Incidents and Controversies Repository (AIAAIC) and the AI Incident Database have emerged<sup>4,5</sup>. However, these databases are not specifically tailored for incidents occurring in healthcare settings. Other efforts focus on presenting and summarizing key information about AI systems for decision-making<sup>44,45</sup>. For example, Sendak et al. introduced the “Model facts” labels, a one-page document summarizing key information about a machine learning model used in clinical settings<sup>46</sup>. Some challenges of those documents are that they primarily focus on model performance or are rarely intended for patients.

For drugs, access to comprehensive and easily readable information on adverse events is crucial to effectively prevent similar occurrences. In Europe, as part of the marketing authorization process, the Summary of Product Characteristics (SmPC) accompanies all authorized drugs, providing physicians with key information on safe and effective drug usage. In addition to general information such as the posology and the intended population, it also presents standardized, tabulated information on expected adverse events observed during clinical trials and after post-market authorization, using the MedDRA. Adverse events are categorized based on their seriousness and frequency: very common (≥1/10); common (≥1/100 to <1/10); uncommon (≥1/1000 to <1/100); rare (≥1/10,000 to <1/1000); very rare (<1/10,000). Moreover, adverse events are also indicated for specific subgroups (e.g., pediatric, elderly patients, patients with renal or hepatic impairment)<sup>47</sup>. Additionally, patient-oriented documents such as patient medication information and package leaflets offer concise information on safe drug usage and adverse effect prevention.

Similarly, dissemination of AAIRs could be done at different levels to effectively prevent future harm caused by AI systems. For example, a document could accompany each AI system in healthcare and include a description of the AI system, when and how to use it, general performance, and a tabulated summary of known adverse events that can be expected when using the AI system with the corresponding frequency and seriousness. The document could also include a section presenting the model performance on specific patient subgroups (e.g. by age, race, gender, patients at high/low risks, patients with other comorbidities, etc.). If the AI system has not been assessed for those subgroups, this should also be stated. These documents would benefit healthcare professionals and AI operators handling multiple systems. Moreover, easily accessible and understandable documents for patients could inform them about the AI system's

functionality, the population on which the system has been tested, its impact on decision-making, and expected adverse events. Both documents would be regularly updated with new validation on new data (see first section) and evidence of adverse events. Additionally, a public database could centralize information on known adverse effects, allowing users to filter healthcare-related incidents by type of AI system, medical application, type of error, frequency, seriousness, and population at risk.

## Differences between drug and AI development

While drug and AI system development shares similarities, we must also recognize their distinctions. First, the current state of AI development has led to a proliferation of models still lacking standardized validation, exacerbated by the absence today of tailored development or reporting guidelines—although this will likely improve over time<sup>18</sup>. This leaves the community with an excess of untested models. Another difference is that, unlike drugs, whose generalizability is mainly affected by variations in patients' characteristics, the generalizability of AI systems depends on many more factors, including operational environments, clinical workflows, and integration with other IT systems. Updating AI systems is also inherently easier than updating drugs: AI systems can potentially be updated at varying frequencies and complexities, with different levels of performance guarantees<sup>40</sup>. Finally, unlike for drug safety studies, which use de-identified data to analyze data that minimizes concerns for privacy and confidentiality, the analysis of AI systems in healthcare requires access to patients' EHR data, posing potential privacy challenges. Methods such as privacy-preserving machine learning, federated learning, or synthetic data generation can offer solutions for data analysis while preserving data privacy.

## Algorithmovigilance, the “pharmacovigilance” for AI

As increasingly more AI systems are being marketed in healthcare—and other industries—further attention may need to be shifted from the pre-marketing approval processes to the post-marketing risk management ones. Existing programs, such as MedWatch, may not be sufficient due to the evolutive nature of AI systems. As algorithmovigilance studies are emerging, lessons from pharmacovigilance may prove valuable and raise new research as well as products and practice directions<sup>19</sup>. While our focus is on AI systems in health, the ideas discussed herein are also applicable to the monitoring and surveillance of AI systems beyond healthcare.

## Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Received: 30 January 2024; Accepted: 27 August 2024;
Published online: 02 October 2024

## References

1. Hwang, T. J., Kesselheim, A. S. & Vokinger, K. N. Lifecycle regulation of artificial intelligence- and machine learning-based software devices in medicine. *JAMA* **322**, 2285–2286 (2019).

2. Babic, B., Gerke, S., Evgeniou, T. & Cohen, I. G. Algorithms on regulatory lockdown in medicine. *Science* **366**, 1202–1204 (2019).

3. Testi, M. et al. MLOps: a taxonomy and a methodology. *IEEE Access* **10**, 63606–63618 (2022).

4. McGregor, S. Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. In *Proc. AAAI Conference on Artificial Intelligence*, **35**, 17, 15458–15463 https://doi.org/10.1609/aaai.v35i17.17817 (2021).

5. AIAAIC. *AI, Algorithmic, and Automation Incidents and Controversies (AIAAIC) Repository* https://www.aiaaic.org/aiaaic-repository (2019).

6. OECD. *AIM: The OECD AI Incidents Monitor, an Evidence Base for Trustworthy AI* https://oecd.ai/en/incidents (2023).

7. High-Level Expert Group on AI (AI HLEG). *Ethics Guidelines for Trustworthy AI* https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai (2019).
8. Gerke, S., Babic, B., Evgeniou, T. & Cohen, I. G. The need for a system view to regulate artificial intelligence/machine learning-based software as medical device. *NPJ Digit. Med.* **3**, 53 (2020).

9. U.S. Food & Drug Administration (FDA). *Postmarketing Surveillance Programs* (U.S. Food & Drug Administration (FDA), 2020).

10. U.S. Food & Drug Administration (FDA). *FDA Dashboards—Recalls* (U.S. Food & Drug Administration (FDA), 2024).

11. Food and Drug Administration, Department of Health and Human Services. *21 Code of Federal Regulations (CFR) Part 7—Enforcement Policy* (Food and Drug Administration, Department of Health and Human Services, 2024).

12. Benjamens, S., Dhunnoo, P. & Meskó, B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. *NPJ Digit. Med.* **3**, 118 (2020).

13. Beninger, P. *Pharmacovigilance: an overview*. *Clin. Ther.* **40**, 1991–2004 (2018).

14. Przybyla, G. W., Szychowski, K. A. & Gmiński, J. Paracetamol—an old drug with new mechanisms of action. *Clin. Exp. Pharmacol. Physiol.* **48**, 3–19 (2021).

15. Embi, P. J. *Algorithmovigilance*—advancing methods to analyze and monitor artificial intelligence-driven health care for effectiveness and equity. *JAMA Netw. Open* **4**, e214622 (2021).

16. Finlayson, S. G. et al. The clinician and dataset shift in artificial intelligence. *N. Engl. J. Med.* **385**, 283–286 (2021).

17. Duckworth, C. et al. Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during COVID-19. *Sci. Rep.* **11**, 23017 (2021).

18. Rabanser, S., Günnemann, S. & Lipton, Z. C. Failing loudly: an empirical study of methods for detecting dataset shift. *In Proc.* 33rd *International Conference on Neural Information Processing Systems*, 1396–1408 (Curran Associates Inc., Red Hook, NY, USA, 2019).

19. Wu, E. et al. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. *Nat. Med.* **27**, 582–584 (2021).

20. Beede, E. et al. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In *Proc. 2020 CHI Conference on Human Factors in Computing Systems*, 1–12 (Association for Computing Machinery, Honolulu, HI, USA, 2020).

21. Strom, B. L. How the US drug safety system should be changed. *JAMA* **295**, 2072 (2006).

22. Etmianan, M. & Samii, A. *Pharmacoepidemiology I: a review of pharmacoepidemiologic study designs*. *Pharmacotherapy* **24**, 964–969 (2004).

23. European Medicines Agency. *Guideline on Good Pharmacovigilance Practices (GVP) Module VII—Periodic Safety Update Report (Rev 1)* (European Medicines Agency, 2013).

24. de Hond, A. A. H. et al. Perspectives on validation of clinical predictive algorithms. *Npj Digit. Med.* **6**, 1–3 (2023).

25. Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. *Science* **366**, 447–453 (2019).

26. Winkler, J. K. et al. Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition. *JAMA Dermatol.* **155**, 1135–1141 (2019).

27. Laï, M.-C., Brian, M. & Mamzer, M.-F. Perceptions of artificial intelligence in healthcare: findings from a qualitative survey study among actors in France. *J. Transl. Med.* **18**, 14 (2020).

28. Esmaeilzadeh, P. Use of AI-based tools for healthcare purposes: a survey study from consumers’ perspectives. *BMC Med. Inform. Decis. Mak.* **20**, 170 (2020).

29. Esmaeilzadeh, P., Mirzaei, T. & Dharanikota, S. Patients’ perceptions toward human-artificial intelligence interaction in health care: experimental study. *J. Med. Internet Res.* **23**, e25856 (2021).

30. Center for Drug Evaluation and Research & U.S. Food and Drug Administration (FDA). *Guidance for Industry Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment* (Center for Drug Evaluation and Research & U.S. Food and Drug Administration (FDA), 2005).

31. Dambal, A., Venkatesh, M. P., Rao, S. B. & Kumar, T. M. P. *Pharmacovigilance system in USA and WHO*. *J. Glob. Trends Pharm. Sci.* **10**, 6669–6684 (2019).

32. European Medicines Agency (EMA). *European Medicines Agency Policy on Access to EudraVigilance Data for Medicinal Products for Human Use* (European Medicines Agency (EMA), 2019).

33. Brown, E. G., Wood, L. & Wood, S. The medical dictionary for regulatory activities (MedDRA). *Drug Saf.* **20**, 109–117 (1999).

34. Overhage, J. M., Ryan, P. B., Reich, C. G., Hartzema, A. G. & Stang, P. E. Validation of a common data model for active safety surveillance research. *J. Am. Med. Inform. Assoc.* **JAMIA** **19**, 54–60 (2012).

35. Rostamzadeh, N. et al. Healthsheet: Development of a Transparency Artifact for Health Datasets. In *Proc. 2022 ACM Conference on Fairness, Accountability, and Transparency*, 1943–1961 (Association for Computing Machinery, New York, NY, USA, 2022).

36. Meyboom, R. H., Hekster, Y. A., Egberts, A. C., Gribnau, F. W. & Edwards, I. R. Causal or casual? The role of causality assessment in pharmacovigilance. *Drug Saf.* **17**, 374–389 (1997).

37. Agbabiaka, T. B., Savović, J. & Ernst, E. Methods for causality assessment of adverse drug reactions: a systematic review. *Drug Saf.* **31**, 21–37 (2008).

38. Pande, S. Causality or relatedness assessment in adverse drug reaction and its relevance in dermatology. *Indian J. Dermatol.* **63**, 18–21 (2018).

39. Edwards, I. R. & Biriell, C. Harmonisation in pharmacovigilance. *Drug Saf.* **10**, 93–102 (1994).

40. Feng, J. et al. Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. *NPJ Digit. Med.* **5**, 66 (2022).

41. U.S. Food & Drug Administration (FDA). *Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD)—Discussion Paper and Request for Feedback* (U.S. Food & Drug Administration (FDA), 2019).

42. Babic, B., Gerke, S., Evgeniou, T. & Cohen, I. G. Beware explanations from AI in health care. *Science* **373**, 284–286 (2021).

43. Zhang, J., Wang, Y., Molino, P., Li, L. & Ebert, D. S. Manifold: a model-agnostic framework for interpretation and diagnosis of machine learning models. *IEEE Trans. Vis. Comput. Graph.* **25**, 364–373 (2019).

44. Mitchell, M. et al. Model Cards for Model Reporting. In *Proc. Conference on Fairness, Accountability, and Transparency*, 220–229 (Association for Computing Machinery, New York, NY, USA, 2019).

45. Arnold, M. et al. FactSheets: increasing trust in AI services through supplier's declarations of conformity. *IBM J. Res. Dev.* **63**, 6:1–6:13 (2019).

46. Sendak, M. P., Gao, M., Brajer, N. & Balu, S. Presenting machine learning model information to clinical end users with model facts labels. *Npj Digit. Med.* **3**, 1–4 (2020).

47. European Medicines Agency (EMA). *A Guideline on Summary of Product Characteristics (SmPC)* (European Medicines Agency (EMA), 2009).

48. Collins, G. S. et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. *BMJ* **385**, e078378 (2024).

49. Hanks, H., Austin, A., Kailasanath, V. & Park, S. *UK NHS Pilots AI Tool Aimed at Reducing Bias in Healthcare Datasets*—a Step Toward ‘algorithmovigilance’? https://technologyquotient.freshfields.com/post/102hisr/uk-nhs-pilots-ai-tool-aimed-at-reducing-bias-in-healthcare-datasets-a-step-towa (2022).
## Acknowledgements

The study was realized in the context of the @Hotel-Dieu project, which was funded by the Banque Publique d'Investissement in France. The funder played no role in the study design, data collection, analysis, and interpretation of data, or the writing of this manuscript. The authors thank Raphaël Porcher and Viet-Thi Tran for their feedback during the revision.

## Author contributions

A.B., T.E., and P.R. were involved in the conceptualization of this study. A.B. and T.E. wrote the first draft of the manuscript. A.B., T.E., M.B., and P.R. contributed to subsequent drafts and revisions of the final manuscript.

## Competing interests

The authors declare no competing interests.

## Additional information

**Supplementary information** The online version contains supplementary material available at https://doi.org/10.1038/s41746-024-01237-y.

**Correspondence** and requests for materials should be addressed to Alan Balendran.

**Reprints and permissions information** is available at http://www.nature.com/reprints

**Publisher's note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

© The Author(s) 2024