Table of Contents 

I. Background and Scope............................................................................................... 1 

II. Current and Potential Uses of AI/ML in Drug Development ....................................... 3 

A. Drug Discovery................................................................................................... 3 

1. Drug Target Identification, Selection, and Prioritization ....................................... 3 

2. Compound Screening and Design....................................................................... 4 

B. Nonclinical Research ......................................................................................... 5 

C. Clinical Research ............................................................................................... 5 

1. Recruitment ......................................................................................................... 6 

2. Selection of Trial Participants .............................................................................. 6 

3. Dose/Dosing Regimen Optimization.................................................................... 6 

4. Adherence ........................................................................................................... 7 

5. Retention ............................................................................................................. 7 

6. Site Selection....................................................................................................... 7 

7. Clinical Trial Data Collection, Management, and Analysis................................... 7 

8. Clinical Endpoint Assessment.............................................................................. 8 

D. Postmarket Safety Surveillance ......................................................................... 9 

1. Case Processing.................................................................................................. 9 

2. Case Evaluation................................................................................................. 10 

3. Case Submission............................................................................................... 10 

E. Advanced Pharmaceutical Manufacturing........................................................ 10 

1. Optimization of Process Design......................................................................... 11 

2. Advanced Process Control ................................................................................ 11 

3. Smart Monitoring and Maintenance................................................................... 12 

4. Trend Monitoring................................................................................................ 12 

F. FDA Experience with AI/ML for Drug Development ......................................... 12 

III. Considerations for the Use of AI/ML in Drug Development..................................... 15 

A. Overarching Standards and Practices for the Use of AI/ML............................. 15 

B. Discussion of Considerations and Practices for AI/ML in Drug Development .. 17 

IV. Next Steps: Engagement and Collaboration ........................................................... 23 

Glossary ........................................................................................................................ 24 

References.................................................................................................................... 27 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

1 I. Background and Scope 

2 

3 To fulfill its mission of protecting, promoting, and advancing public health, the Food and 

4 Drug Administration’s (FDA’s) Center for Drug Evaluation and Research (CDER), in 

5 collaboration with the Center for Biologics Evaluation and Research (CBER) and the 

6 Center for Devices and Radiological Health (CDRH), including the Digital Health Center 

7 of Excellence (DHCoE), is publishing this document to facilitate a discussion with 

8 stakeholders on the use of artificial intelligence (AI)1 and machine learning (ML)2 in 

9 drug development,

3,4 including in the development of medical devices intended to be 

10 used with drugs, to help inform the regulatory landscape in this area. 

11 

12 FDA helps to ensure that drugs are safe and effective while facilitating innovations in 

13 their development. Recent, rapid technological innovations in data collection and 

14 generation tools, combined with robust information management and exchange systems 

15 and advanced computing abilities, may transform the way drugs are developed and 

16 used (ElZarrad, Lee, Purcell, & Steele, 2022). This evolving ecosystem presents 

17 unique opportunities and challenges, and FDA is committed to working across its 

18 medical product centers with partners domestically and internationally to ensure that the 

19 full potential of these innovations is realized for the benefit of the public. 

20 

21 Developers, manufacturers, regulators, academic groups, and other stakeholders are 

22 working to develop a shared understanding of where and how specific innovations, such 

23 as AI and ML, can best be used throughout the drug development process. FDA is 

24 publishing this discussion paper as part of a multifaceted approach to enhance mutual 

25 learning and to establish a dialogue with FDA stakeholders on this topic. AI can 

26 generally be described as a branch of computer science, statistics, and engineering that 

27 uses algorithms or models to perform tasks and exhibit behaviors such as learning, 

28 making decisions, and making predictions.

5 ML is considered a subset of AI that allows 

29 ML models to be developed by ML training algorithms through analysis of data, without 

30 models being explicitly programmed.

6 Additionally, there are a variety of ML methods 

31 and different types of algorithms that may be utilized in a given context. For purposes 

32 of this document, AI and ML will be referenced together as AI/ML, and references to 

1 Words and phrases in bold italics are defined in the Glossary. 

2 There are multiple definitions for AI and ML, and the Glossary includes several definitions from federal 

legislation and agencies. 

3 For purposes of this discussion paper, all references to drug or drugs include both human drugs and 

biological products. 

4 FDA is focusing this discussion paper on drug development. However, many of the AI/ML scientific and 

regulatory science principles outlined in this document may be applicable across all medical products, 

including in the development of medical devices intended to be used with drugs (including, but not limited 

to, combination products, companion devices, and complementary devices). Some medical devices 

intended to be used with drugs are intended for use only in clinical investigations; others are intended to 

be marketed for use outside of clinical investigations. Examples include medical devices that help identify 

side effects of drugs as well as medical devices that assist in drug dosing. 5 See IMDRF/AIMD WG/N67 Machine Learning-enabled Medical Devices: Key Terms and Definitions, 

final document, May 6, 2022. https://www.imdrf.org/documents/machine-learning-enabled-medicaldevices-key-terms-and-definitions 

6 Ibid. 

1 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

33 drug development and the drug development process include a wide scope of activities 

34 and phases, including manufacturing and postmarket drug safety monitoring, among 

35 others.

7,8 

36 

37 This discussion paper, which considers the application of AI/ML in the broad context of 

38 the drug development process, is not FDA guidance or policy and does not endorse a 

39 specific AI/ML use or approach in drug development. Rather, this discussion paper is 

40 an initial communication with stakeholders, including academic groups, researchers, 

41 and technology developers, that is intended to promote mutual learning and discussion. 

42 It is particularly beneficial for those new to drug development and human subjects 

43 research, to recognize some of the initial thinking and considerations involved with 

44 utilizing these technologies, including having familiarity with FDA’s current activities, 

45 initiatives, practices, and potentially applicable regulations. FDA is soliciting feedback 

46 on the opportunities and challenges with utilizing AI/ML in the development of drugs, as 

47 well as in the development of medical devices intended to be used with drugs. This 

48 feedback will provide an additional resource to help inform the regulatory landscape in 

49 this area. 

50 

51 In this discussion paper, three main topics are discussed: 

52 

53 • Landscape of current and potential uses of AI/ML: FDA recognizes the 

54 potential for AI/ML to enhance drug development in many ways, including to help 

55 bring safe and effective drugs to patients faster; provide broader access to drugs 

56 and thereby improve health equity; increase the quality of manufacturing; 

57 enhance drug safety; and develop novel drugs and drug classes, as well as 

58 personalized treatment approaches. Section II provides examples of the use of 

59 AI/ML to highlight the potential impact of AI/ML across the drug development 

60 process and includes a brief description of FDA’s experience with AI/ML in drug 

61 development. The list of examples in this section is not comprehensive of all 

62 AI/ML uses, and it includes uses where FDA oversight may or may not be 

63 applicable. The purpose of this section is to promote shared learning and to 

64 identify areas where future regulatory clarity may be helpful. 

65 

66 • Considerations for the use of AI/ML: FDA is also aware of the potential 

67 concerns and risks with emerging innovations such as AI/ML and will share initial 

68 considerations and solicit feedback on how to help ensure the responsible 

69 utilization of AI/ML in drug development. Section III briefly describes several key 

70 efforts to develop general principles, standards, and practices for the use of 

71 AI/ML across diverse applications and then explores the principles and 

72 considerations that may be particularly applicable when using AI/ML for drug 

73 development activities. FDA understands that AI/ML use in drug development is 

7 See The Drug Development Process, January 2018. https://www.fda.gov/patients/learn-about-drug-anddevice-approvals/drug-development-process 

8 In this discussion paper, the topic of clinical investigations focuses on the drug development process, 

however, many other activities and phases included as part of the drug development process may also be 

part of the development process for other medical products; see footnote 4. 

2 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

75

80

85

90

95

100

105

110

115

74 diverse, and careful assessments that consider the specific context of use are 

needed. Taking a risk-based approach to evaluate and manage the use of AI/ML 

76 can help facilitate innovations and protect public health. 

77 

78 • Next steps and stakeholder engagement: FDA is interested in mutual 

79 opportunities to learn and engage with all stakeholders to establish a shared 

understanding of AI/ML systems and their rapidly evolving potential uses and 

81 considerations in drug development. As part of this ongoing effort, FDA 

82 welcomes feedback on this discussion paper and any AI/ML-related issues 

83 pertaining to drug development. Specifically, to initiate a broader dialogue with 

84 stakeholders, Section III includes several key questions to which interested 

parties can provide perspectives and Section IV outlines opportunities for future 

86 engagement. 

87 

88 II. Current and Potential Uses of AI/ML in the Drug Development Process 

89 

This section provides a high-level overview of the diverse and evolving uses of AI/ML 

91 being employed throughout the drug development process. These examples are not 

92 comprehensive of all AI/ML uses and include uses where FDA oversight may or may 

93 not be applicable.9 Additionally, while some of the uses of AI/ML described in this 

94 section may also have utility in clinical practice, this paper is focused on uses of AI/ML 

in the drug development process. The purpose of this section is to promote shared 

96 learning and to identify areas where future FDA regulatory clarity may be beneficial. 

97 

98 Although the overall drug development process is an iterative continuum of activities 

99 and not strictly linear in nature, for simplicity, this section utilizes different phases of 

drug development to highlight several uses of AI/ML, ranging from drug discovery and 

101 clinical research to postmarket safety surveillance and advanced pharmaceutical 

102 manufacturing. The section also includes references to how AI/ML is being applied to 

103 real-world data (RWD) and data from digital health technologies (DHTs) in support 

104 of drug development. Some of the general challenges and considerations with utilizing 

AI/ML in different drug development use cases are discussed in Section III. 

106 

107 A. Drug Discovery 

108 

109 Early drug discovery is one of the areas with significant interest and activity in utilizing 

AI/ML. Included below is a brief discussion of the current and potential uses of AI/ML 

111 for drug target identification, selection, and prioritization, as well as compound 

112 screening and drug design in drug discovery. 

113 

114 1. Drug Target Identification, Selection, and Prioritization 

9 The examples listed were not necessarily submitted to FDA for review or approval and are not meant to 

suggest an endorsement of any specific approach. The FDA does not endorse any particular use of 

AI/ML. 

3 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

The early stages of drug development generally rely on the initial identification of a 

suitable biological target for drug candidates. As a starting point, the process of 

identifying biological targets and elucidating disease relationships can utilize AI/ML to 

analyze and synthesize significant amounts of information from existing scientific 

research, publications, and other data sources. The growth of available genomic, 

transcriptomic, proteomic, and other data sources from healthy persons and those with 

a specific disease of interest provide a significant opportunity to inform biological target 

selection. These datasets are often complex and originate from disparate sources, 

which can be well-suited for the utilization of AI/ML approaches (Fumagalli et al., 2023). 

Building from existing validated data, AI/ML can be applied to mine and analyze these 

large multi-omics and other datasets to provide information on the potential structure 

and function of biological targets to predict their role in a disease pathway (Vamathevan 

et al., 2019; Weissler et al., 2021). While early target identification and prioritization is a 

critical step where AI/ML could help improve the efficiency and effectiveness of drug 

development, it is important to validate the role of the biological target in the disease of 

interest through subsequent studies (Fumagalli et al., 2023). 

2. Compound Screening and Design 

The discovery of potential drug candidates that modify the function of the identified 

biological targets of interest generally involves significant in silico or experimental 

screening of compound libraries, followed by subsequent refinement of a compound’s 

specificity and selectivity for the biological target. In the area of compound screening, 

potential AI/ML uses include predicting the chemical properties and bioactivity of 

compounds and predicting efficacy and potential adverse events based on the 

compound’s specificity and affinity for a target (Chan, Shan, Dahoun, Vogel, & Yuan, 

2019; Schneider et al., 2020). 

AI/ML approaches used to further elucidate drug-target interactions could also help 

provide predictions about classes of drugs potentially interacting with the same targets 

or having a similar mechanism of action, which may help predict the toxicity of a 

molecule based on specific known features. This strategy can help guide drug 

repurposing efforts that could utilize previously characterized compounds. Drug 

repurposing efforts utilizing AI/ML can also potentially benefit from the increased 

availability of suitable RWD from a variety of sources (e.g., electronic health records 

(EHRs), registries, and DHTs) to identify previously unknown effects of drugs on 

disease pathways (Z. Liu et al., 2022). 

Finally, AI/ML could accelerate advances in de novo drug design (Mouchlis et al., 2021). 

For example, AI/ML may be applied to help predict the 3D structure of target proteins, 

informing chemical synthesis and the potential effect of a drug candidate on the target, 

including predicting affinity and potential toxicity (Chan et al., 2019; Jumper et al., 2021; 

Vamathevan et al., 2019). It is worth noting that one must be cautious with the use of 

AI/ML in 3-D structure prediction, as many proteins that are developed for 

pharmaceutical applications are codon optimized (with many synonymous mutations 

4 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

incorporated), the impact of which on protein structure is still an area of active research 

(Fumagalli et al., 2023; Jumper et al., 2021). 

B. Nonclinical Research 

Nonclinical research refers to in vitro and in vivo studies and is designed to further 

advance potential therapeutics towards clinical research in humans. Nonclinical 

studies, in support of new drug development, can be conducted at all phases of 

development: prior to clinical studies, in parallel with clinical development, and even in 

postmarketing environments. Data from pharmacokinetic, pharmacodynamic, and 

toxicologic studies conducted in animals; exploratory in vitro and in vivo mechanistic 

studies conducted in animal models; organ-on-chip and multi-organ chip systems; and 

cell assay platforms may be leveraged using AI/ML (e.g., computational modeling and 

simulation techniques) for evaluating toxicity, exploring mechanistic models, and 

developing in vivo predictive models (Bulitta et al., 2019; Harrison & Gibaldi, 1977; Hsu 

et al., 2014; Mager, Woo, & Jusko, 2009; Shroff et al., 2022). 

Pharmacokinetics (PK) describes the time course of drug absorption, distribution, 

metabolism, and excretion. Pharmacodynamics (PD) explores the body’s biological 

response to drugs. When PK and PD are integrated in a model, the model can describe 

how the drug effect will change with time when a certain dose or dosing regimen is 

used. Pharmacokinetic/pharmacodynamic (PK/PD) modeling has been used in drug 

development for decades and can be applied at both the nonclinical and clinical stages 

(Daryaee & Tonge, 2019). Along with the advances in computational tools and 

technology and the availability of modeling platforms, use of physiologically-based 

pharmacokinetic (PBPK) and physiologically-based PK/PD (PBPK-PD) modeling is also 

increasing (Sager, Yu, Ragueneau-Majlessi, & Isoherranen, 2015). There are current 

efforts to explore the use of more novel AI/ML algorithms (e.g., artificial neural network 

models and tree-based models) for PK/PD modeling. For example, a recurrent neural 

network, an ML algorithm commonly used for analyzing time series data, may be used 

to complement traditional PK/PD models in the area of highly complex PK/PD data 

analysis, and possibly lead to improved accuracy for nonclinical and clinical 

applications (Liu et al., 2021). 

C. Clinical Research 

Clinical research typically involves a series of phases of clinical trials in increasing 

numbers of human subjects to assess the safety and effectiveness of a drug. One of 

the most significant applications of AI/ML in drug development is in efforts to streamline 

and advance clinical research. For example, AI/ML is being utilized to analyze vast 

amounts of data from both interventional studies (also referred to as clinical trials) and 

non-interventional studies (also referred to as observational studies) to make inferences 

regarding the safety and effectiveness of a drug. Additionally, AI/ML has the potential to 

inform the design and efficiency of non-traditional trials such as decentralized clinical 

trials, and trials incorporating the use of RWD extracted from EHRs, medical claims, or 

other data sources. AI/ML may also have a role in analyzing and interpreting data 

5 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

207 collected from DHTs used in clinical studies. Finally, AI/ML could also be used to 

208 improve the conduct of clinical trials and augment operational efficiency. The following 

209 subsections will highlight some of the uses and potential uses of AI/ML during the 

210 design and conduct of clinical research. 

211 

212 1. Recruitment 

213 

214 AI/ML is increasingly being developed and used to connect individuals to trials for 

215 investigational treatments from which participants may benefit. Specifically, AI/ML is 

216 being used to mine vast amounts of data, such as data from clinical trial databases, trial 

217 announcements, social media, medical literature, registries, and structured and 

218 unstructured data in EHRs, which can be used to match individuals to trials (Harrer, 

219 Shah, Antony, & Hu, 2019). While these algorithms are trained on high volumes of 

220 patient data and enrollment criteria from past trials, it is important to ensure adequate 

221 representation of populations that are likely to use the drug (e.g., gender, race, and 

222 ethnicity) as matching algorithms are created and, when used, to confirm that equitable 

223 inclusion was achieved during the recruitment process. In the future, these 

224 technologies, if properly validated, may continue to play an increasing role in matching 

225 individuals with investigational treatments. 

226 

227 2. Selection and Stratification of Trial Participants 

228 

229 Enrichment strategies can aid participant selection in clinical investigations designed to 

230 demonstrate the effectiveness of drug and biological products.

10 AI/ML has been 

231 explored and used as part of a clinical investigation in the prediction of an individual 

232 participant’s clinical outcome based on baseline characteristics (e.g., demographic 

233 information, clinical data, vital signs, labs, medical imaging data, and genomic data) 

234 (Aerts et al., 2016; Athreya et al., 2019; Dercle et al., 2020; Harrer et al., 2019; 

235 Kawakami et al., 2019). Such predictive models can be used to enrich clinical trials 

236 (e.g., identifying high-risk participants or participants more likely to respond to the 

237 treatment). When these types of AI/ML algorithms are used for patient evaluation and 

238 selection before randomization, it may be possible to reduce variability and increase 

239 study power (Y. Wang, Carter, Li, & Huang, 2022). 

240 

241 In addition to utilization in enrichment strategies, such predictive models can also be 

242 used for participant stratification, for example, if an AI/ML model could predict the 

243 probability of a serious adverse event before an investigational treatment is 

244 administered. Based on their predicted risk for these serious adverse events, 

245 participants can be stratified into different groups and then monitored accordingly (or 

246 excluded depending on predicted severity of the adverse event). 

247 

248 3. Dose/Dosing Regimen Optimization 

249 

10 See the guidance for industry Enrichment Strategies for Clinical Trials to Support Determination of 

Effectiveness of Human Drugs and Biological Products (March 2019). 

https://www.fda.gov/media/121320/download 

6 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

250

255

260

265

270

275

280

285

290

AI/ML can be used to characterize and predict PK profiles after drug administration. It 

251 can also be used to study the relationship between drug exposure and response, taking 

252 into consideration confounding factors. These kinds of models can be used to optimize 

253 the dose/dosing regimen selection for a study (Liu et al., 2021; Lu, Deng, Zhang, Liu, & 

254 Guan, 2021). This could potentially include aiding in dose optimization in special 

populations where there may be limited data (e.g., rare disease studies, pediatric and 

256 pregnant populations). 

257 

258 4. Adherence 

259 

AI/ML can be used to monitor and improve adherence during a clinical trial through 

261 tools, such as smartphone alerts and reminders, eTracking of medication (e.g., smart 

262 pillboxes and tools for visual confirmation) (Mason et al., 2022), and eTracking of 

263 missed clinical visits, which trigger non-adherence alerts. Examples of AI/ML used in 

264 clinical research to improve medication adherence include applications using digital 

biomarkers, such as facial and vocal expressivity, to monitor adherence remotely. 

266 

267 5. Retention 

268 

269 AI/ML has the potential to improve the participants’ access to relevant trial information 

by enabling tools, such as AI chatbots, voice assistance, and intelligent search. AI/ML 

271 can also be used to reduce the burden for participants by using passive data collection 

272 techniques and by extracting more information from available data generated during 

273 clinical practice or by study activities (Weissler et al., 2021). Additionally, data from 

274 DHTs and other systems can be used to develop patient profiles to potentially predict 

dropouts and adverse events to ensure participant retention. 

276 

277 6. Site Selection 

278 

279 Trial operational conduct could also be optimized by utilizing AI/ML to help identify 

which sites have the greatest potential for a successful trial and to aid sites in identifying 

281 process gaps. For example, algorithms can be used to evaluate site performance and 

282 to help determine which sites may have a higher risk of running behind schedule based 

283 on data from other trials at that site. 

284 

7. Clinical Trial Data Collection, Management, and Analysis 

286 

287 a. Data Collection 

288 

289 DHTs, such as wireless and smartphone-connected products, wearables, implantables, 

and ingestibles, are increasingly being used in clinical trials to collect objective, 

291 quantifiable, longitudinal, and continuous physiological data.

11 In addition, many of 

292 these DHTs enable the use of AI/ML, either as embedded algorithms within the DHT or 

11 See the draft guidance for industry, investigators, and other stakeholders Digital Health Technologies 

for Remote Data Acquisition in Clinical Investigations (December 2021). When final, this guidance will 

represent FDA’s current thinking on this topic. https://www.fda.gov/media/155022/download 

7 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

employed upon the data generated after the data are collected from the DHT, and have 

been used to predict the status of a chronic disease and its response to treatment 

(Stehlik et al., 2020) or to identify novel characteristics of an underlying condition 

(Avram et al., 2020). AI/ML can be utilized to analyze the large and diverse data 

generated from the continuous monitoring of persons using these technologies. This 

could include using AI/ML to aid in the evaluation of multimodal data and composite 

measures that may combine individual measures collected through multiple DHTs 

(Cohoon & Bhavnani, 2020). 

b. Data Management 

AI/ML can be used for a range of data cleaning and curation purposes, including 

duplicate participant detection and imputation of missing data values (Zhang, Yan, Gao, 

Malin, & Chen, 2020), as well as the ability to harmonize controlled terminology 

across drug development programs. Use of AI/ML could also significantly enhance data 

integration efforts by using supervised and unsupervised learning to help integrate data 

submitted in various formats and perform data quality assessments. Additionally, AI/ML 

can be used for data curation via masking and de-identification of personal identifiable 

information, metadata creation, and search and retrieval of stored data. These 

applications can potentially increase data accuracy and improve the speed at which 

data are prepared for analyses. 

c. Data Analysis 

AI/ML has been used to analyze high volumes of diverse and complex RWD extracted 

from EHRs, medical claims, and disease registries, among other sources. Additionally, 

the use of AI/ML in predictive modeling and counterfactual simulation to inform clinical 

trial designs is being actively explored. For example, in silico clinical trials utilize 

computational modeling and simulation to evaluate drug candidates using a virtual 

cohort of simulated participants with realistic variability of traits representing the desired 

participant population (Pappalardo, Russo, Tshinanu, & Viceconti, 2019). AI/ML could 

be employed in these situations to aid in evaluating a vast number of counterfactual 

simulations and to predict trial outcomes before human trials. 

At an even more personalized level, AI/ML can also be used in the context of digital 

twins of patients, an emerging method that could potentially be used in clinical research. 

To create digital twins of patients, AI/ML can be utilized to build in silico representations 

or replicas of an individual that can dynamically reflect molecular and physiological 

status over time (European Medicines Agency, 2022; Laubenbacher, Sluka, & Glazier, 

2021; Schuler et al., 2021). In comparison to a participant in a clinical trial that received 

an investigational treatment, the digital twin could potentially provide a comprehensive, 

longitudinal, and computationally generated clinical record that describes what may 

have happened to that specific participant if they had received a placebo. 

8. Clinical Endpoint Assessment 

8 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

340

345

350

355

360

365

370

375

380

339 Clinical endpoint assessment is a key part of evaluating safety and efficacy of medical 

interventions in clinical trials. AI/ML-enabled algorithms could detect clusters of signs 

341 and symptoms to identify a potential safety signal, as well as help detect cases with 

342 safety issues in real time (Pierce et al., 2017; Routray et al., 2020). AI/ML could be 

343 used to assist in the assessment of outcomes captured from diverse sources (e.g., 

344 DHTs, social media) during a clinical trial, including those consisting of large amounts of 

data for which manual review may be impractical. 

346 

347 D. Postmarketing Safety Surveillance 

348 

349 For purposes of this paper, pharmacovigilance (PV) refers to the science and activities 

related to the detection, assessment, understanding, and prevention of adverse events 

351 or any other drug-related problems (including medication errors and product quality 

352 issues).12 Postmarketing safety surveillance, or PV activities in the post-approval 

353 period, includes postmarketing safety reporting of adverse events associated with use 

354 of human drug and biological products. An individual case safety report (ICSR) is used, 

as applicable, for the postmarketing reporting of adverse events to FDA and serves as 

356 an important data source of potential drug safety issues for postmarket safety 

357 surveillance. The clinical information in ICSRs can include suspect product or products, 

358 and temporal information related to use of the product and occurrence of the adverse 

359 event(s) in the patient’s medical history, clinical course, and outcome. Complete and 

accurate reporting of ICSRs is critical to the understanding of a drug’s safety profile. 

361 For reasons including increases in ICSR volume, AI/ML applications are being explored 

362 to help process and evaluate ICSR submissions within regulatory agencies (Ball & Dal 

363 Pan, 2022; Bate & Hobbiger, 2021). 

364 

1. Case Processing 

366 

367 There are potential opportunities to use AI/ML for automation during ISCR processing. 

368 The number and complexity of data sources of adverse events for ICSRs have 

369 increased, including from spontaneous reports, clinical trials, EHRs, social media, 

phone calls, emails, literature, patient registries, claims data, and post-approval safety 

371 studies (Beninger, 2020). The use of AI/ML to detect information from source 

372 documents could help identify adverse events for ICSR submission. For instance, the 

373 use of AI/ML to detect and evaluate drug event associations from literature and to 

374 screen social media for adverse events has been explored (Comfort, Dorrell, Meireis, & 

Fine, 2018; Negi, Pavuri, Patel, & Jain, 2019; S. V. Wang et al., 2017; W. Wang et al., 

376 2011). 

377 

378 After an adverse event is identified from a data source, AI/ML could be used for case 

379 validity, case prioritization, duplicate check, coding, and quality control. The use of 

AI/ML can help identify whether a case is a valid case, which includes determining 

12 See the guidance for industry Good Pharmacovigilance Practices and Pharmacoepidemiologic 

Assessment (March 2005). Accessed September 30, 2022. https://www.fda.gov/media/71546/download 

See also, Council for International Organizations of Medical Sciences (CIOMS) Pharmacovigilance 

definition. Accessed September 29, 2022. https://cioms.ch/pharmacovigilance/ 

9 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

381 whether a case contains the minimum reporting requirements, such as an identifiable 

382 patient, suspect drug or biological product, adverse event(s), and identifiable reporter 

383 (Abatemarco et al., 2018; Schmider et al., 2019). During case intake, to assist in the 

384 prioritization of cases, AI/ML has been used to classify adverse events by expectedness 

385 (whether an adverse event is known and in the product labeling) (Abatemarco et al., 

386 2018; Routray et al., 2020). Automated duplicate checks using AI/ML are being 

387 conducted to identify whether the case is a true duplicate, a follow up version of a prior 

388 case, or a new case (Kassekert 2022). Another area in which AI/ML has been applied 

389 is the coding of adverse events described in ICSRs to structured medical dictionary 

390 terms and for quality control purposes (Ghosh 2020). 

391 

392 2. Case Evaluation 

393 

394 Adverse event cases undergo clinical assessment. Case evaluation includes assessing 

395 the possibility of a causal relationship between the drug and adverse event, as well as 

396 assessing the outcome of the case. An AI model was developed based on relevant 

397 features used in causality assessments; it was trained, validated, and tested to classify 

398 cases by the probability of a causal relationship between the drug and adverse event 

399 (Comfort et al., 2018). AI/ML has also been applied to determine seriousness of the 

400 outcome of ICSRs (Routray, et al., 2020), which not only supports case evaluation, but 

401 also the timeliness of individual case submissions that require expedited reporting. 

402 

403 3. Case Submission 

404 

405 Generally, the final step after case processing is the submission of ICSRs. AI/ML 

406 algorithms have been used to automate reporting rules for submission of ICSRs to FDA. 

407 The reporting of ICSRs is required on an individual basis, as well as in aggregate 

408 (Ghosh et al., 2020). The aggregate reporting of adverse events generally involves the 

409 compilation of safety data for a product that is submitted at regular time intervals as 

410 specified. AI/ML can be used to develop aggregate reports that include multiple 

411 adverse events for particular products that occur within a time period for reporting 

412 purposes (Lewis & McCallum, 2020). 

413 

414 E. Advanced Pharmaceutical Manufacturing13 

415 

416 A critical aspect of drug development includes the methods, facilities, and controls used 

417 in manufacturing, processing, packing, and holding of a drug to help ensure that the 

418 drug meets the requirements of safety and effectiveness, has the identity and strength it 

419 is represented to possess, and meets quality and purity characteristics. Advanced 

13 The examples in this section are based on the review of general published information that projects or 

forecasts how AI/ML may be currently used in the pharmaceutical manufacturing space. In the continued 

spirit of FDA’s recent engagement through the Quality Metrics Feedback Program and CDER’s Emerging 

Technology Program, FDA has been able to solicit valuable feedback demonstrated by industry 

interactions on several AI/ML use cases in the pharmaceutical manufacturing space, such as optimal riskbased supply chain modeling, business forecasting, process optimization, application of natural language 

processing (NLP) algorithms for complaints reduction, use of predictive analytics for non-conformance 

(NC) reduction, and corrective and preventive action (CAPA) effectiveness. 

10 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

analytics leveraging AI/ML in the pharmaceutical manufacturing industry offers many 

possibilities, including, but not limited to, enhancing process control, increasing 

equipment reliability and throughput, monitoring early warnings or signals that the 

manufacturing process is not in a state of control, detecting recurring problem clusters, 

and preventing batch losses. The use of AI/ML to support pharmaceutical 

manufacturing can be deployed together with other advanced manufacturing 

technologies (e.g., process analytical technology, continuous manufacturing) to achieve 

the desired benefits. AI/ML is an enabler for the implementation of Industry 4.0, a term 

that refers to the fourth industrial revolution that brings together rapidly evolving 

technologies, and could result in a well-controlled, hyper-connected, digitized 

ecosystem and pharmaceutical value chain for the manufacturer (Arden et al., 2021). 

AI/ML could also be used to improve the reliability of the manufacturing supply chain 

through forecasting product demand, analyzing production schedules, estimating and 

mitigating the impact of potential disruptions, and optimizing inventory. Use of AI/MLbased approaches in pharmaceutical manufacturing can be broadly grouped into the 

areas outlined below that cover the entire drug manufacturing life cycle, from design to 

commercial manufacturing. 

1. Optimization of Process Design 

Digital twins can also be used in process design optimization. In this context, a digital 

twin of a process is a digital replica of the physical process used to better understand, 

analyze, predict, and optimize process performance. The digital twin could be 

especially beneficial for analyzing manufacturing processes characterized by a limited 

amount of development data, where AI/ML models could potentially leverage prior 

knowledge of the product and process (e.g., from previous studies, development 

programs, and scientific literature) to more quickly identify the optimal processing 

parameters, thus reducing design time and waste. 

2. Advanced Process Control 

Process controls have been implemented in pharmaceutical manufacturing for several 

decades. Traditional process controls maintain input process parameters at set points, 

but are not capable of simultaneously changing multiple input parameters to maintain 

the output parameters at desired levels to optimize the process. On the other hand, 

advanced process control (APC) allows dynamic control of the process to achieve a 

desired output (Huang et al., 2021). AI/ML techniques such as neural networks, with 

real-time process data as inputs, can be used to implement APC. These methods can 

also be used to develop process controls that can predict whether a process is 

performing under a state of control by using AI/ML tools in combination with real-time 

sensor data, including, in conjunction with smart monitoring of production lines, to 

improve existing manufacturing line efficiency and output. In the near term, APC 

approaches that combine physics and chemistry knowledge with AI/ML techniques are 

expected to be increasingly adopted and have already been reported by several 

pharmaceutical manufacturers (National Academies of Sciences, 2021). In these APC 

applications, high quality model inputs inform process understanding and, model 

11 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

structure. These robust inputs, when combined with data-driven modeling, allow 

derivation of model parameters. These models leverage data required for model 

development while improving model robustness. 

3. Smart Monitoring and Maintenance 

Manufacturing processes can be automated and monitored in real time, leading to more 

efficient inventory management with shorter lead times and increased production 

output, without impacting product quality. AI/ML methods can be used to monitor 

equipment and detect deviations from normal performance that can trigger maintenance 

activities, thus reducing process downtime. Another example is the use of computer 

vision-based quality control that uses images (e.g., images of packaging, labels, or 

glass vials) that are analyzed by AI/ML-based software to detect deviations and to 

ensure images match the requirements of a given quality attribute of a product. 

Augmenting human visual inspection of drug products and packaging with such AI/MLbased methods can improve the accuracy and efficiency of visual inspection controls. 

4. Trend Monitoring 

AI/ML can be used in many ways to make manufacturing more effective and efficient 

with faster output, less waste, more informed decision-making, and enhanced quality 

control. Current practice for the analysis of deviations in the process is primarily done 

by quality personnel and relevant subject matter experts. AI/ML could be utilized to 

assist in examination of deviation reports that mostly contain large volumes of data or 

text to analyze manufacturing-related deviation trends, cluster problem areas, and 

prioritize areas for proactive continual improvement. This offers the advantage of 

expediting the process of identifying root causes, as solely manual review of deviation 

trends can be very time-consuming. AI/ML methods integrated with process 

performance (Ppk) and process capability (Cpk) metrics can be used to proactively 

monitor manufacturing operations for trends and out-of-control events, and predict 

thresholds for triggering CAPA effectiveness evaluations. 

F. FDA Experience with AI/ML for Drug Development 

FDA recognizes the increased use of AI/ML throughout the drug development life cycle 

and its potential to accelerate the development of safe and effective drugs. AI/ML is 

increasingly integrated in areas where FDA is actively engaged, including clinical trial 

design, DHTs, and RWD analytics. Over the last few years, FDA has seen a rapid 

growth in the number of submissions that reference AI/ML. Submissions across drug 

and biological product applications that include AI/ML have increased over the last few 

years to more than 100 submissions in 2021 (Q. Liu et al., 2022). These submissions 

cut across a range of therapeutic areas, and the uses of AI/ML within the submissions 

cover the many different areas of the drug development process highlighted in this 

section, from drug discovery and clinical trial enrichment to endpoint assessment and 

postmarket safety surveillance. Inclusion of AI/ML in the clinical development/research 

phase represents the most common stage for AI/ML uses in submissions. 

12 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

512 

513 One of the ways FDA has been supporting the development of innovative and robust 

514 AI/ML is through the establishment of the CDER AI Steering Committee (AISC), which 

515 coordinates efforts around AI/ML uses across therapeutic development. Leveraging its 

516 commitment to advancing innovative approaches and promoting collaborative efforts 

517 across the Agency, CDRH, including the DHCoE, have provided consults for drug 

518 submissions that involve AI/ML, and are developing a framework for AI/ML-based 

519 devices, including predetermined change control plans for devices incorporating 

520 AI/ML,

14 as well as a foundation for Good Machine Learning Practices for medical 

521 device development.

15 In addition, FDA has organized various workshops16,17 and held 

522 a Patient Engagement Advisory Committee (PEAC) meeting on DHT and AI/ML-related 

523 topics18 and has fostered regulatory science research, including on robustness, user524 centered transparency, and bias identification and management, through external 

525 academic and clinical partnerships to evaluate the safety and effectiveness of emerging 

526 AI/ML products.

19 

527 

528 Additionally, CDER has developed the Innovative Science and Technology Approaches 

529 for New Drugs (ISTAND) Pilot Program, which is designed to expand drug 

530 development tool (DDT) types included in the DDT qualification programs, including 

531 tools that leverage DHTs. Applications of AI/ML may represent novel DDTs or could be 

532 used to aid in the interpretation and analysis of traditional DDTs (such as biomarkers 

533 or clinical outcome assessments), potentially speeding novel therapeutics to patients 

534 by enhancing the evidence available for decision-making.

20 In the area of model535 informed drug development (MIDD), FDA’s CDER and CBER have established a MIDD 

536 Pilot Program to facilitate the development and application of exposure-based, 

537 biological, and statistical models derived from nonclinical and clinical data sources.

21 In 

14 Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-

Based Software as a Medical Device (SaMD) – Discussion Paper and Request for Feedback, April 2019. 

https://www.fda.gov/files/medical%20devices/published/US-FDA-Artificial-Intelligence-and-MachineLearning-Discussion-Paper.pdf 

15 Good Machine Learning Practice for Medical Device Development: Guiding Principles, October 2021. 

https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practicemedical-device-development-guiding-principles 

16 See the Virtual Public Workshop – Transparency of Artificial Intelligence/Machine Learning-enabled 

Medical Devices, October 14, 2021. https://www.fda.gov/medical-devices/workshops-conferencesmedical-devices/virtual-public-workshop-transparency-artificial-intelligencemachine-learning-enabledmedical-devices 

17 See the Public Workshop – Evolving Role of Artificial Intelligence in Radiological Imaging, February 

25–26, 2020. https://www.fda.gov/medical-devices/workshops-conferences-medical-devices/publicworkshop-evolving-role-artificial-intelligence-radiological-imaging-02252020-02262020 

18 See the Patient Engagement Advisory Committee Meeting Announcement, October 22, 2020. 

https://www.fda.gov/advisory-committees/advisory-committee-calendar/october-22-2020-patientengagement-advisory-committee-meeting-announcement-10222020-10222020 

19 See CERSI research projects, October 2022. https://www.fda.gov/science-research/advancingregulatory-science/cersi-research-projects 

20 See the guidance for industry and FDA staff Qualification Process for Drug Development Tools 

(November 2020). https://www.fda.gov/media/133511/download 

21 See the Model-Informed Drug Development Paired Meeting Program, October 2022. 

https://www.fda.gov/drugs/development-resources/model-informed-drug-development-pilot-program 

13 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

538 the context of MIDD, AI/ML could be employed to help improve clinical trial simulations, 

539 optimize dose selection or estimations, or enhance predictive or mechanistic safety 

540 evaluations. 

541 

542 In the area of postmarket safety surveillance, the FDA’s Sentinel Initiative, including 

543 CDER’s Sentinel System,

22 CBER’s Biologics Effectiveness and Safety (BEST) 

544 system,23 and CDRH’s National Evaluation System for health Technology (NEST)24 

545 efforts, are exploring AI/ML approaches to improve existing systems. The FDA outlined 

546 its goals for using linked claims and EHR data supported by advanced analytics in the 

547 5-year Sentinel System strategic plan.25 The Sentinel System Innovation Center has 

548 outlined a four-pronged approach to implement this plan by incorporating emerging data 

549 science innovations and EHR data for medical product safety surveillance: (1) data 

550 infrastructure, (2) feature engineering, (3) causal inference, and (4) detection analytics 

551 (Desai et al., 2021). Examples of AI/ML applications in this approach include natural 

552 language processing (NLP) and automated feature extraction from unstructured EHR 

553 clinical notes for computable phenotyping and improved confounding adjustment from 

554 EHR-based variables using advanced statistical and ML approaches, such as 

555 algorithms created to enhance performance or “Super Learner” and targeted maximum 

556 likelihood estimation (Naimi & Balzer, 2018). 

557 

558 CBER’s BEST system is designed to provide better data sources, methods, tools, 

559 expertise, and infrastructure to conduct surveillance and epidemiological studies.

26 Part 

560 of this program is an effort to use AI/ML methods to analyze EHRs to predict or better 

561 understand adverse events associated with the use of biological products and other 

562 products that CBER regulates. This work may also enhance FDA’s understanding of 

563 the use of AI/ML methods for generating real-world evidence about product efficacy. 

564 

565 CDER is also exploring the application of AI to enhance the evaluation of ICSRs 

566 submitted to the FDA Adverse Event Reporting System (FAERS) (Ball & Dal Pan, 

567 2022). The Information Visualization Platform (InfoViP) was developed with AI/ML to 

568 detect duplicate ICSRs, classify ICSRs by level of information quality, and derive 

569 visualization of the timeline of clinical events to aid in analysis of reported adverse 

570 events (Kreimeyer et al., 2022; Kreimeyer et al., 2021; Spiker et al., 2020). AI/ML 

571 methods have been investigated to automate the identification of adverse events in drug 

572 product labeling to support safety reviewers in the triaging of ICSRs to facilitate the 

573 identification of unknown or unexpected safety issues (Bayer et al., 2021; Ly et al., 

574 2018). Another AI-based tool that focuses on drug product labeling and is currently in 

22 See FDA’s Sentinel Initiative, December 2022. https://www.fda.gov/safety/fdas-sentinel-initiative 

23 See the CBER Biologics Effectiveness and Safety (BEST) System, March 2022. 

https://www.fda.gov/vaccines-blood-biologics/safety-availability-biologics/cber-biologics-effectivenessand-safety-best-system 

24 See the National Evaluation System for health Technology (NEST), October 2019. 

https://www.fda.gov/about-fda/cdrh-reports/national-evaluation-system-health-technology-nest 

25 See the FDA Sentinel System Five-Year Strategy, January 2019. 

https://www.fda.gov/media/120333/download 

26 See the CBER BEST System, March 2022. https://www.fda.gov/vaccines-blood-biologics/safetyavailability-biologics/cber-biologics-effectiveness-and-safety-best-system 

14 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

575

580

585

590

595

600

605

610

615

use is the Computerized Labeling Assessment Tool (CLAT), which serves to automate 

576 the review of label and labeling (e.g., prescribing information, carton and container 

577 labeling). NLP and ML are also being explored to classify free-text narratives in FAERS 

578 ICSRs into structured medical dictionary medication error terminologies to support the 

579 human review of coding quality. Additionally, through the FDA Quality Metrics Reporting 

Program,

27 CDER’s Emerging Technology Program, and CBER’s Advanced 

581 Technologies Team (CATT) Program,

28 FDA has been able to engage industry and 

582 gain valuable feedback on AI/ML use cases in pharmaceutical manufacturing. 

583 

584 The FDA also utilizes mechanisms such as a Broad Agency Announcement to solicit 

extramural proposals that address emerging regulatory science priorities, including 

586 leveraging external expertise and infrastructure to provide insight on the methods used 

587 to integrate and evaluate AI/ML in drug development. 

588 

589 III. Considerations for the Use of AI/ML in Drug Development 

591 As shown in Section II, AI/ML has been applied to a broad range of drug development 

592 activities and continues to evolve. The use of AI/ML has the potential to accelerate the 

593 drug development process and make clinical trials safer and more efficient. However, it 

594 is important to assess whether the use of AI/ML introduces specific risks and harms. 

For example, AI/ML algorithms have the potential to amplify errors and preexisting 

596 biases present in underlying data sources and, when the findings are extrapolated 

597 outside of the testing environment, raise concerns related to generalizability and ethical 

598 considerations. Additionally, an AI/ML system may exhibit limited explainability due to 

599 its underlying complexity or may not be fully transparent for proprietary reasons. These 

concerns have resulted in a focus on developing standards for trustworthy AI that 

601 address specific characteristics in areas such as explainability, reliability, privacy, 

602 safety, security, and bias mitigation. This section begins with an overview of 

603 considerations and good practices for the general application of AI/ML and ends with 

604 questions to solicit feedback from stakeholders on these considerations and to further 

identify potential good practices in the context of drug development. This will aid FDA in 

606 further identifying opportunities and challenges with utilizing AI/ML throughout the drug 

607 development process. 

608 

609 A. Overarching Standards and Practices for the Use of AI/ML 

611 There has been an increased commitment by the Federal Government and the 

612 international community to facilitate AI innovation and adoption, which includes 

613 promoting trustworthy and ethical AI (Exec. Order No. 13859, Maintaining American 

614 Leadership in Artificial Intelligence, February 11, 2019; Exec. Order No. 13960, 

Promoting the Use of Trustworthy Artificial Intelligence in the Federal Government, 

616 December 3, 2020; Lander & Nelson, October 22, 2021; Notice of Request for 

27 See the Quality Metrics for Drug Manufacturing, October 2022. 

https://www.fda.gov/drugs/pharmaceutical-quality-resources/quality-metrics-drug-manufacturing 

28 See the CBER Advanced Technologies Team (CATT) Program, June 27, 2019. 

https://www.fda.gov/vaccines-blood-biologics/industry-biologics/cber-advanced-technologies-team-catt 

15 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

617 Information on Public and Private Sector Uses of Biometric Technologies, October 8, 

618 2021; Organisation for Economic Co-operation and Development, 2019; Vought, 2020). 

619 As a result, efforts for the development of cross-sector and sector-specific standards to 

620 facilitate the technological advancement of AI have rapidly increased in both domestic 

621 and international forums. For example, in August 2019, the National Institute for 

622 Standards and Technology (NIST) released “U.S. Leadership in AI: A Plan for Federal 

623 Engagement in Developing Technical Standards and Related Tools” to help ensure the 

624 use of technical standards and to advance innovation, trust, and confidence in the use 

625 of AI (National Institute of Standards and Technology, 2019). The plan identified 

626 several areas of focus for AI standards development, including data and knowledge, 

627 performance testing and reporting methodology, risk management, and trustworthiness, 

628 among others. Other standards organizations, such as the International Organization 

629 for Standardization (ISO), the Institute of Electrical and Electronics Engineers (IEEE), 

630 and the International Electrotechnical Commission (IEC), are also developing relevant 

631 AI/ML standards and work products addressing fundamental issues of data quality, 

632 explainability, and performance, in addition to examining applications that are specific to 

633 certain industries. The Verification and Validation (V&V 40) risk-informed credibility 

634 assessment framework was initially developed by the American Society of Mechanical 

635 Engineers (ASME) for the assessment of credibility of computational models used for 

636 medical devices (American Society of Mechanical Engineers, 2018) and was later 

637 adopted into model-informed drug development29 (Kuemmel et al., 2020; Viceconti et 

638 al., 2021). As AI/ML is also used for computational models, the V&V 40 framework 

639 potentially serves to inform whether the AI/ML model is credible for use in drug 

640 development.30 The V&V 40 Standard, which is not specific to AI/ML and does not 

641 specify activities or define criteria required to establish model credibility for a particular 

642 context of use or application, has been adapted for medical devices and for model643 informed drug development.

31,32 

644 

645 In addition to the V&V 40 Standard for evaluating the predictive capability of 

646 computational models for medical devices, FDA, Health Canada, and the United 

647 Kingdom’s Medicines and Healthcare products Regulatory Agency (MHRA) jointly 

648 published 10 guiding principles to inform the development of Good Machine Learning 

649 Practices (GMLP) for medical devices that use AI/ML.

33 The guiding principles include 

29 Promoting Innovation in Medical Product Assessment: A Risk-based Framework for Evaluating 

Computational Models for Regulatory Decision-Making, October 2020. https://www.fda.gov/drugs/newsevents-human-drugs/promoting-innovation-medical-product-assessment-risk-based-frameworkevaluating-computational-models 

30 A V&V 70 Subcommittee has been established for Verification and Validation of Machine Learning. 

31 See the draft guidance for industry and FDA staff Assessing the Credibility of Computational Modelling 

Simulation in Medical Device Submissions (December 2021). When final, this guidance will represent 

FDA’s current thinking on this topic. https://www.fda.gov/media/154985/download 

32 Promoting Innovation in Medical Product Assessment: A Risk-based Framework for Evaluating 

Computational Models for Regulatory Decision-Making, October 2020. https://www.fda.gov/drugs/newsevents-human-drugs/promoting-innovation-medical-product-assessment-risk-based-frameworkevaluating-computational-models 

33 Good Machine Learning Practice for Medical Device Development: Guiding Principles, October 2021. 

https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practicemedical-device-development-guiding-principles 

16 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

650 adopting a total product life cycle approach in which multidisciplinary expertise is 

651 leveraged throughout product development, with an in-depth understanding of how the 

652 model is integrated into the clinical workflow. The principles also emphasize the 

653 importance of adequate representation of age, gender, sex, race, and ethnicity within 

654 the clinical study population to manage bias, improve generalizability, and provide 

655 sufficient transparency with clear and essential information, such as the product’s 

656 intended use and indications, the data used to test and train the model, and known 

657 limitations. Finally, these GMLP highlight the importance of monitoring deployed 

658 models for performance while managing the risk of model retraining. FDA’s CDRH had 

659 previously discussed the role of GMLP for medical devices, and in 2019 issued a 

660 proposed framework for modifications to AI/ML-based SaMD. The framework proposed 

661 a predetermined change control plan mechanism—whereby a sponsor can proactively 

662 specify intended modifications to device software incorporating AI/ML and the methods 

663 that will be used to ensure their safety and effectiveness—thereby laying the foundation 

664 for AI/ML-enabled devices with improved capacity for adaptation.

34 

665 

666 Although the standards and practices described in this section were not tailored 

667 specifically for drug development, the utility and applicability of these standards to drug 

668 development and the development of medical devices intended to be used with drugs, 

669 will be explored to ensure alignment and consistency. 

670 

671 B. Discussion of Considerations and Practices for AI/ML in Drug Development 

672 

673 Informed by the diverse applications of AI/ML in drug development (see Section II), 

674 FDA is considering approaches to provide regulatory clarity around the use of AI/ML in 

675 drug development, supported by an expanding body of knowledge and a clear 

676 appreciation of the opportunities and challenges with utilizing AI/ML in drug 

677 development. While certain standards and practices outlined in Section III.A can 

678 potentially be adapted to address the use of AI/ML in the context of drug development, 

679 the use of AI/ML in drug development may raise specific challenges that could highlight 

680 additional considerations. As noted above, this document is not FDA guidance or policy 

681 and does not endorse any specific approaches for the use of AI/ML in drug 

682 development. However, the feedback and future discussions with stakeholders can 

683 help inform future regulatory activities. 

684 

685 Adapting the overarching principles of the General Accountability Office AI 

686 accountability framework35 below, FDA’s CDER, CBER, CDRH, including DHCoE, aim 

687 to initiate a discussion with stakeholders and solicit feedback on three key areas in the 

688 context of AI/ML in drug development: 

689 

34 Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-

Based Software as a Medical Device (SaMD) – Discussion Paper and Request for Feedback, April 2019. 

https://www.fda.gov/files/medical%20devices/published/US-FDA-Artificial-Intelligence-and-MachineLearning-Discussion-Paper.pdf 

35 See Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities (June 

2021). https://www.gao.gov/assets/gao-21-519sp.pdf 

17 
 

 

 

 

 

 

 

 

690 (1) human-led governance, accountability, and transparency; 

691 

692 (2) quality, reliability, and representativeness of data; and 

693 

694 (3) model development, performance, monitoring, and validation. 

695 

696 In each of these areas, a risk-based approach could include measures commensurate 

697 with the level of risk posed by the specific context of use for AI/ML. 

698 

(1) Human-led governance, accountability, and transparency 

Human-led AI/ML governance can help ensure adherence to legal and ethical values, 

where accountability and transparency are essential for the development of 

trustworthy AI. Such governance and clear accountability may extend across the 

spectrum of planning, development, use, modification, and discontinuation (as 

applicable) of AI/ML in the drug development process. 

As part of governance, a risk management plan that considers the context of use may 

be applied to identify and mitigate risks. This approach can help guide the level of 

documentation, transparency, and explainability, with tracking and recording of key 

steps and decisions, including the rationale for any deviations and procedures that 

enable vigilant oversight and auditing. Transparency and documentation can provide 

critical insight on the initial planning, development, function, and any modifications of 

the AI/ML in the specific context of use, while explainability can provide 

accompanying evidence or reason for the outputs. 

Questions: 

• In what specific use cases or applications of AI/ML in drug development are 

there the greatest need for additional regulatory clarity? 

• What does transparency mean in the use of AI/ML in drug development (for 

example, transparency could be considered as the degree to which appropriate 

information about the AI/ML model—including its use, development, 

performance, and, when available, logic—is clearly communicated to 

regulators and/or other stakeholders)?36 

• In your experience, what are the main barriers and facilitators of transparency 

with AI/ML used during the drug development process (and in what context)? 

• What are some of the good practices utilized by stakeholders for providing riskbased, meaningful human involvement when AI/ML is being utilized in drug 

development? 

36 Adapted from ISO/IEC JTC1/SC42 DIS 25059 (draft). 

https://www.iso.org/standard/80655.html?browse=tc 

18 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

• What processes are in place to enhance and enable traceability and 

auditability? 

• How are pre-specification activities managed, and changes captured and 

monitored, to ensure the safe and effective use of AI/ML in drug development? 

(2) Quality, reliability, and representativeness of data 

AI/ML is particularly sensitive to the attributes or characteristics of the data used for 

training, testing, and validation. Although not unique to AI/ML, missing data, bias, and 

data drift are typically important considerations. Ensuring data quality, reliability, and 

that the data are fit for use (i.e., relevant for the specific intended use and population) 

can be critical. Potential data-related issues to consider include: 

Bias: AI/ML can potentially amplify preexisting biases that exist in the underlying 

input data. NIST published a document characterizing three categories of bias 

(human, systemic, and statistical/computational) and “how they may occur in the 

commission, design, development, and deployment of AI technologies that can be 

used to generate predictions, recommendations, or decisions (e.g., algorithmic 

decision systems), and how AI systems may create societal harms.”37 

Integrity: The completeness, consistency, and accuracy of data.38 

Privacy and security: The protection and privacy of data, linked to data 

classifications and the technical features of the system. 

Provenance: Record trail that accounts for the origin of a piece of data (in a 

database, document, or repository) together with an explanation of how and why it 

got to the present place.39 Provenance describes “the metadata, or extra 

information about data, that can help answer questions such as who created the 

data and when.”40 

Relevance: Adequate data are available and are appropriate for the intended use. 

Replicability: Obtaining consistent results across studies aimed at answering the 

same question, each of which has obtained its own data.41 It is important to clarify 

data access early in the process. 

37 NIST Special Publication 1270, March 2022. https://doi.org/10.6028/NIST.SP.1270 

38 For additional considerations related to data integrity see the guidance for industry Data Integrity and 

Compliance with Drug CGMP (December 2018). https://www.fda.gov/media/119267/download 

39 Encyclopedia of Database Systems, definition of data provenance. 

https://link.springer.com/referenceworkentry/10.1007%2F978-0-387-39940-9_1305 

40 21st Century Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification 

Program (March 2019). https://www.federalregister.gov/documents/2019/03/04/2019-02224/21st-centurycures-act-interoperability-information-blocking-and-the-onc-health-it-certification 

41 Ibid. 

19 
 

 

 

 

 

 

 

 

 

 

 

 

Reproducibility: Obtaining consistent results using the same input data, 

computational steps, methods and code, and conditions of analysis42 (while not 

confirming validity, the transparency required to demonstrate reproducibility 

permits evaluation of the validity of design and operational decisions (S. V. Wang 

et al., 2017)). 

Representativeness: Confidence that a sample from which evidence is generated 

is sufficiently similar to the intended population. In the context of patient 

experience data, representativeness includes the extent to which the elicited 

experiences, perspectives, needs, and priorities of the sample are sufficiently 

similar to those of the intended patient population.43 

Questions: 

• What additional data considerations exist for AI/ML in the drug development 

process? 

• What practices are developers, manufacturers, and other stakeholders 

currently utilizing to help assure the integrity of AI/ML or to address issues, 

such as bias, missing data, and other data quality considerations, for the use of 

AI/ML in drug development? 

• What are some of the key practices utilized by stakeholders to help ensure 

data privacy and security? 

• What are some of the key practices utilized by stakeholders to help address 

issues of reproducibility and replicability? 

• What processes are developers using for bias identification and management? 

(3) Model development, performance, monitoring, and validation 

The use of the model may be important to consider in evaluating AI/ML model 

development and performance, including through practices of pre-specification steps 

and clear documentation of criteria for developing and assessing models. It may also 

be important to consider the model risk and credibility; model risk drives the selection 

of credibility goals and activities.

44 Model risk is determined by two factors, which are 

42 National Academies of Sciences, Engineering, and Medicine, 2019, Reproducibility and Replicability in 

Science. https://doi.org/10.17226/25303 

43 See discussion document for Patient-focused Drug Development Public Workshop Collecting 

Comprehensive and Representative Input, December 2017. 

https://www.fda.gov/media/109179/download 

44 Credibility refers to trust in the predictive capability of a computational model for a particular context of 

use (Kuemmel et al., 2020). This includes steps to document performance and approaches to measure 

uncertainty at the component level (e.g., model and non-level components, including metrics and 

20 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

shaped by the context of use: model influence (the weight of the model in the totality 

of evidence for a specific decision) and decision consequence (the potential 

consequences of a wrong decision). 

In balancing performance and explainability, it may be important to consider the 

complexity of the AI/ML model. In situations where complex models (e.g., artificial 

neural network models) are determined to have similar performance, there may be 

overall advantages to selecting the more traditional and parsimonious (i.e., fewer 

parameters) model. 

It may also be important to monitor and document monitoring efforts of the AI/ML 

model to ensure it is reliable, relevant, and consistent over time. This includes 

documentation of the results of monitoring and any corrective action taken to ensure 

that the AI/ML produces intended results. Subsequent assessments (e.g., postmarket 

safety monitoring, surveillance) can provide valuable feedback on processes and realworld model performance. Real-world model performance includes applications that 

may be supported by collection and monitoring of RWD (e.g., electronic health 

records, product and disease registries). Potential re-training based on real-world 

performance could provide important insights to model performance, and following 

such re-training, it may be important to monitor and document the AI/ML model to 

appropriately manage risks. 

Data considerations also include providing the details of the training dataset utilized to 

develop the AI/ML model, along with the performance, when employing independent, 

external testing data to support verification and validation (“external validity”). It is 

generally important for data of sufficient quality for the particular context of use to be 

representative of the population where the AI/ML method will be utilized. It is 

important to help ensure AI/ML models are validated to produce results that are 

credible for the model’s use. Credibility activities include verification of the software 

code and calculations, validation of the model, and evaluation of the applicability of 

validation assessments to the context of use. These activities include considerations 

of measuring the level of uncertainty of the model predictions. Upon completion of 

credibility activities, an assessment can be made to determine whether the model is 

sufficiently credible for its use and whether the model may be acceptable for a given 

regulatory purpose. 

Questions: 

• What are some examples of current tools, processes, approaches, and best 

practices being used by stakeholders for: 

assessing performance and outcome of each component) and system level (e.g., methods for 

assessment, performance metrics, and outcomes), where feasible. Demonstration of credibility often 

includes a risk-based approach, where uses presenting the highest risk generally require the greatest 

standard of evidence, with a gradient of evidence needed based on the associated risk (i.e., informing 

early-stage drug development for non-serious medical condition versus evaluating drug safety and 

effectiveness for critical medical condition). 

21 
 

 

 

 

 

 

 

 

- Documenting the development and performance of AI/ML models that can 

be applied in the context of drug development (e.g., CONSORT-AI (Liu et 

al., 2020) and SPIRIT-AI (Cruz Rivera et al., 2020))? 

- Selecting model types and algorithms for a given context of use? 

- Determining when to use specific approaches for validating models and 

measuring performance in a given context of use (e.g., selecting relevant 

success criteria and performance measures)? 

- Evaluating transparency and explainability and increasing model 

transparency? 

- Addressing issues of accuracy and explainability (e.g., scenarios where 

models may provide increased accuracy, while having limitations in 

explainability)? 

- Selecting open-source AI software for AI/ML model development? What 

are considerations when using open-source AI software? 

- The use of RWD performance in monitoring AI/ML? 

• What practices and documentation are being used to inform and record data 

source selection and inclusion or exclusion criteria? 

• In what context of use are stakeholders addressing explainability, and how 

have you balanced considerations of performance and explainability? 

• What approaches are being used to document the assessment of uncertainty 

in model predictions, and how is uncertainty being communicated? What 

methods and standards should be developed to help support the assessment 

of uncertainty? 

699 

700 As outlined above, many of the overarching principles and standards related to the 

701 characteristics of trustworthy AI can help inform considerations or key practice areas for 

702 the application of AI/ML in the context of drug development. In addition to meeting 

703 current requirements to support regulatory decision-making regarding a drug’s safety 

704 and effectiveness, the use of AI/ML in drug development raises challenges related to 

705 human-led AI/ML governance, accountability, and transparency; data considerations; 

706 and model development, performance, monitoring, and validation. Transparency and 

707 documentation across the entire product life cycle can help build trust in the use of 

708 AI/ML. In this regard, it may be important to consider pre-specification and 

709 documentation of the purpose or question of interest, context of use, risk, and 

710 development of AI/ML. While not unique to the use of AI/ML in drug development, there 

711 are also a broad range of data quality, relevance, and reliability-related considerations. 

22 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

712 Related to the area of model development, performance, monitoring, and validation, the 

713 V&V 40 risk-informed credibility assessment framework may be a helpful guide when 

714 considering the specific use for AI/ML. In general, use of a risk-based approach may 

715 guide the level of evidence and record keeping needed for the verification and validation 

716 of AI/ML models for a specific context of use. Engagement with the FDA early in the 

717 process can also help inform and address these considerations. 

718 

719 IV. Next Steps: Engagement and Collaboration 

720 

721 The release of this initial discussion paper is part of a broader effort to communicate 

722 with a range of stakeholders and to explore the relevant considerations for the use of 

723 AI/ML in the development of human drugs and biological products. Coupled with this 

724 document, FDA has included a series of questions for feedback, and a workshop with 

725 stakeholders is planned to provide an opportunity for further engagement. The FDA will 

726 also provide several other mechanisms to engage with stakeholders, sponsors, and 

727 developers on this topic, and these can be utilized to address questions before 

728 conducting a study that utilizes AI/ML. In addition to formal meetings where these 

729 methods can be discussed, the Critical Path Innovation Meetings (CPIM),

45 ISTAND 

730 Pilot Program,

46 Emerging Technology Program,47 and Real-World Evidence Program48 

731 meetings are examples of additional avenues for communicating and discussing a 

732 relevant AI/ML methodology or technology and improving efficiency and quality in drug 

733 development. Additionally, communication and engagement with patients and the 

734 public regarding considerations for AI/ML in drug development is critical to ensure 

735 patient-centered approaches and policies. 

736 

737 Building on this discussion paper, FDA will continue to solicit feedback and engage a 

738 broad group of stakeholders to further discuss considerations for utilizing AI/ML 

739 throughout the drug development life cycle. These discussions and future 

740 collaborations with stakeholders may provide a foundation for a future framework or 

741 guidance. 

45 See CPIM, November 11, 2022. https://www.fda.gov/drugs/new-drugs-fda-cders-new-molecularentities-and-new-therapeutic-biological-products/critical-path-innovation-meetings-cpim 

46 See the ISTAND Pilot Program, February 10, 2021. https://www.fda.gov/drugs/drug-development-toolddt-qualification-programs/innovative-science-and-technology-approaches-new-drugs-istand-pilotprogram 

47 See Emerging Technology Program, February 22, 2022. https://www.fda.gov/about-fda/center-drugevaluation-and-research-cder/emerging-technology-program 

48 See Framework for FDA’s Real World Evidence Program, April 14, 2020. 

https:/fda.gov/media/120060/download 

23 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

742 Glossary 

743 

744 Accuracy: The level of agreement between the measured value and the true value of 

745 the clinical event or characteristic. 

746 

747 Artificial Intelligence (AI): A branch of computer science, statistics, and engineering 

748 that uses algorithms or models to perform tasks and exhibit behaviors such as learning, 

749 making decisions, and making predictions.49 

750 

751 Biomarker: A defined characteristic that is measured as an indicator of normal 

752 biological processes, pathogenic processes, or biological responses to an exposure or 

753 intervention, including therapeutic interventions. Biomarkers may include molecular, 

754 histologic, radiographic, or physiologic characteristics. A biomarker is not a measure of 

755 how an individual feels, functions, or survives.50 

756 

757 Clinical Outcome Assessment (COA): A measure that describes or reflects how a 

758 patient feels, functions, or survives. There are four types of COAs: patient-reported 

759 outcome, observer-reported outcome, clinician-reported outcome, and performance 

760 outcome.

51 

761 

762 Context of Use: A statement that fully and clearly describes the way AI/ML is to be 

763 used and the drug development-related purpose of the use.

52 

764 

765 Controlled Terminology: A finite set of values (e.g., codes, text, numeric) that 

766 represent the only allowed values for a data item. Generally, controlled terminology 

767 standards specify the key concepts that are represented as definitions, preferred terms, 

768 synonyms, and code systems.53 

769 

770 Decentralized Clinical Trial: A clinical investigation where some or all of the trial771 related activities occur at a location separate from the investigator’s location.54 

772 

773 Digital Health Technology (DHT): A system that uses computing platforms, 

774 connectivity, software, and/or sensors for health care and related uses. These 

775 technologies span a wide range of uses, from applications in general wellness to 

776 applications as a medical device. They include technologies intended for use as a 

49 See IMDRF/AIMD WG/N67 Machine Learning-enabled Medical Devices: Key Terms and Definitions, 

final document, May 6, 2022. https://www.imdrf.org/documents/machine-learning-enabled-medicaldevices-key-terms-and-definitions 

50 See BEST (Biomarkers, EndpointS, and other Tools) Resource Glossary, 2016. 

https://www.ncbi.nlm.nih.gov/books/NBK338448 

51 See Clinical Outcome Assessment (COA), December 2020. https://www.fda.gov/about-fda/clinicaloutcome-assessment-coa-frequently-asked-questions 

52 CDISC Glossary, 2022. https://evs.nci.nih.gov/ftp1/CDISC/Glossary/CDISC%20Glossary.html 

53 Ibid. 54 See the draft guidance for industry, investigators, and other stakeholders Digital Health Technologies 

for Remote Data Acquisition in Clinical Investigations (December 2021). When final, this guidance will 

represent FDA’s current thinking on this topic. https://www.fda.gov/media/155022/download 

24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

777 medical product, in a medical product, or as an adjunct to other medical products 

778 (devices, drugs, and biologics). They may also be used to develop or study medical 

779 products. Data captured by DHTs can often be transmitted directly to investigators, 

780 sponsors, and/or other authorized parties, with the capability to maintain blinding or 

781 masking when appropriate. The ability to transmit data remotely increases opportunities 

782 for patients to participate in clinical investigations at locations remote from the 

783 investigator’s site.55. 

784 

785 Digital Twins: An integrated multi-physics, multiscale, probabilistic simulation of a 

786 complex system that uses the best available data, sensors, and models to mirror the 

787 behavior of its corresponding twin. A fully developed digital twin consists of a physical 

788 component (e.g., unit operations), a virtual component, and automated data 

789 communications between the two. The development and application of digital twins are 

790 now being extended to manufacturing and complex products to assess sensitivities of 

791 material attributes and process parameters, reliability of control strategies, and 

792 effectiveness of mitigation plans for potential disturbances.56 

793 

794 Drug Development Tool (DDT): A biomarker, COA, or any other method, material, or 

795 measure determined to aid drug development and regulatory review. Animal models 

796 developed to be used for product development under the Animal Rule57 have been 

797 determined by FDA to be DDTs under section 507 of the FD&C Act.58 

798 

799 Endpoint: A precisely defined variable intended to reflect an outcome of interest that is 

800 statistically analyzed to address a particular research question. A precise definition of 

801 an endpoint typically specifies the type of assessments made, the timing of those 

802 assessments, the assessment tools used, and possibly other details, as applicable, 

803 such as how multiple assessments within an individual are to be combined.59 

804 

805 Machine Learning (ML): A subset of AI that allows ML models to be developed by ML 

806 training algorithms through analysis of data, without being explicitly programmed.60 

807 

808 Natural Language Processing (NLP): The branch of computer science, specifically 

809 the branch of AI, concerned with giving computers the ability to understand text and 

810 spoken words in much the same way human beings can.61 

55 Ibid. 56 See Modeling & Simulation at FDA, November 16, 2022. https://www.fda.gov/science-research/aboutscience-research-fda/modeling-simulation-fda 

57 See Animal Rule Approvals, June 2022. https://www.fda.gov/drugs/nda-and-bla-approvals/animal-ruleapprovals 

58 See the guidance for industry and FDA staff Qualification Process for Drug Development Tools 

(November 2020). https://www.fda.gov/media/133511/download 

59 See BEST (Biomarkers, EndpointS, and other Tools) Resource Glossary, 2016. 

https://www.ncbi.nlm.nih.gov/books/NBK338448 

60 See IMDRF/AIMD WG/N67 Machine Learning-enabled Medical Devices: Key Terms and Definitions, 

final document, May 6, 2022. https://www.imdrf.org/documents/machine-learning-enabled-medicaldevices-key-terms-and-definitions 

61 “What is natural language processing?” Accessed September 8, 2022. 

https://www.ibm.com/cloud/learn/natural-language-processing#toc-what-is-na-jLju4DjE 

25 
 

 

 

 

 

 

 

 

 

 

 

 

 

811 

812 Neural Network: A commonly used form of AI/ML that is used for categorization 

813 applications and has been loosely likened to the way that neurons in the brain process 

814 signals. Neural networks typically consist of at least three layers of neurons: input layer 

815 (which receives information), hidden layer (responsible for extracting patterns and 

816 conducting the internal processing), and output layer (produces and presents the final 

817 network output).62 

818 

819 Real-World Data (RWD): The data relating to patient health status and/or the delivery 

820 of health care routinely collected from a variety of sources. Examples of RWD include 

821 data derived from electronic health records (EHRs); medical claims and billing data; 

822 data from product and disease registries; patient-generated data, including from in823 home-use settings; and data gathered from other sources that can inform on health 

824 status, such as mobile devices.63 

825 

826 Real-World Evidence (RWE): The clinical evidence about the usage and potential 

827 benefits or risks of a medical product derived from analysis of RWD. RWD sources 

828 (e.g., registries, collections of EHRs, administrative and medical claims databases) can 

829 be used for data collection and, in certain cases, to develop analysis infrastructure to 

830 support many types of study designs to develop RWE, including, but not limited to, 

831 randomized trials (e.g., large simple trials, pragmatic clinical trials) and observational 

832 studies (prospective or retrospective).64 

833 

834 Recurrent Neural Network: A type of artificial neural network that uses sequential 

835 data or time series data to exhibit temporal dynamic behavior. These algorithms are 

836 commonly used for ordinal or temporal problems, such as language translation, NLP, 

837 speech recognition, and image captioning.65 

62 See the Executive Summary for the Patient Engagement Advisory Committee Meeting: Artificial 

Intelligence and Machine Learning in Medical Devices, October 22, 2020. 

https://www.fda.gov/media/142998/download 

63 See the draft guidance for industry, investigators, and other stakeholders Real-World Data: Assessing 

Electronic Health Records and Medical Claims Data to Support Regulatory Decision-Making for Drug and 

Biological Products (September 2021). https://www.fda.gov/media/152503/download 

64 Ibid. 

65 Adapted from https://www.ibm.com/cloud/learn/recurrent-neural-networks 

26 
 

 

 

 

 

 

 

 

 

 

 

 

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

References 

Abatemarco, D., Perera, S., Bao, S. H., Desai, S., Assuncao, B., Tetarenko, N., . . . Mingle, E. 

(2018). Training Augmented Intelligent Capabilities for Pharmacovigilance: Applying 

Deep-learning Approaches to Individual Case Safety Report Processing. Pharmaceut 

Med, 32(6), 391-401. doi:10.1007/s40290-018-0251-9 

Aerts, H. J. W. L., Grossmann, P., Tan, Y., Oxnard, G. R., Rizvi, N., Schwartz, L. H., & Zhao, B. 

(2016). Defining a Radiomic Response Phenotype: A Pilot Study using targeted therapy 

in NSCLC. Sci Rep, 6, 33860. doi:10.1038/srep33860 

American Society of Mechanical Engineers. (2018). Assessing Credibility of Computational 

Modeling through Verification and Validation: Application to Medical Devices. In. 

Arden, N. S., Fisher, A. C., Tyner, K., Yu, L. X., Lee, S. L., & Kopcha, M. (2021). Industry 4.0 for 

pharmaceutical manufacturing: Preparing for the smart factories of the future. Int J 

Pharm, 602, 120554. doi:10.1016/j.ijpharm.2021.120554 

Athreya, A. P., Neavin, D., Carrillo-Roa, T., Skime, M., Biernacka, J., Frye, M. A., . . . Bobo, W. 

V. (2019). Pharmacogenomics-Driven Prediction of Antidepressant Treatment 

Outcomes: A Machine-Learning Approach With Multi-trial Replication. Clin Pharmacol 

Ther, 106(4), 855-865. doi:10.1002/cpt.1482 

Avram, R., Olgin, J. E., Kuhar, P., Hughes, J. W., Marcus, G. M., Pletcher, M. J., . . . Tison, G. 

H. (2020). A digital biomarker of diabetes from smartphone-based vascular signals. Nat 

Med, 26(10), 1576-1582. doi:10.1038/s41591-020-1010-5 

Ball, R., & Dal Pan, G. (2022). "Artificial Intelligence" for Pharmacovigilance: Ready for Prime 

Time? Drug Saf, 45(5), 429-438. doi:10.1007/s40264-022-01157-4 

Bate, A., & Hobbiger, S. F. (2021). Artificial Intelligence, Real-World Automation and the Safety 

of Medicines. Drug Saf, 44(2), 125-132. doi:10.1007/s40264-020-01001-7 

Bayer, S., Clark, C., Dang, O., Aberdeen, J., Brajovic, S., Swank, K., . . . Ball, R. (2021). ADE 

Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug 

Labels for Pharmacovigilance. Drug Saf, 44(1), 83-94. doi:10.1007/s40264-020-00996-3 

Beninger, P. (2020). Signal Management in Pharmacovigilance: A Review of Activities and 

Case Studies. Clin Ther, 42(6), 1110-1129. doi:10.1016/j.clinthera.2020.03.018 

Bulitta, J. B., Hope, W. W., Eakin, A. E., Guina, T., Tam, V. H., Louie, A., . . . Hoover, J. L. 

(2019). Generating Robust and Informative Nonclinical In Vitro and In Vivo Bacterial 

Infection Model Efficacy Data To Support Translation to Humans. Antimicrob Agents 

Chemother, 63(5), e02307–02318. doi:10.1128/AAC.02307-18 

Chan, H. C. S., Shan, H., Dahoun, T., Vogel, H., & Yuan, S. (2019). Advancing Drug Discovery 

via Artificial Intelligence. Trends Pharmacol Sci, 40(8), 592-604. 

doi:10.1016/j.tips.2019.06.004 

Cohoon, T. J., & Bhavnani, S. P. (2020). Toward precision health: applying artificial intelligence 

analytics to digital health biometric datasets. Per Med, 17(4), 307-316. doi:10.2217/pme2019-0113 

Comfort, S., Dorrell, D., Meireis, S., & Fine, J. (2018). MOdified NARanjo Causality Scale for 

ICSRs (MONARCSi): A Decision Support Tool for Safety Scientists. Drug Saf, 41(11), 

1073-1085. doi:10.1007/s40264-018-0690-y 

Cruz Rivera, S., Liu, X., Chan, A. W., Denniston, A. K., Calvert, M. J., & and the SPIRIT-AI and 

CONSORT-AI Working Group; SPIRIT-AI and CONSORT-AI Steering Group; SPIRIT-AI 

and CONSORT-AI Consensus Group. (2020). Guidelines for clinical trial protocols for 

interventions involving artificial intelligence: the SPIRIT-AI extension. Nat Med, 26(9), 

1351-1363. doi:10.1038/s41591-020-1037-7 

Daryaee, F., & Tonge, P. J. (2019). Pharmacokinetic-pharmacodynamic models that incorporate 

drug-target binding kinetics. Curr Opin Chem Biol, 50, 120-127. 

doi:10.1016/j.cbpa.2019.03.008 

27 
 

 

 

 

 

 

 

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

Dercle, L., Fronheiser, M., Lu, L., Du, S., Hayes, W., Leung, D. K., . . . Zhao, B. (2020). 

Identification of non-small cell lung cancer sensitive to systemic cancer therapies using 

radiomics. Clin Cancer Res, 26(9), 2151-2162. doi:10.1158/1078-0432.CCR-19-2942 

Desai, R. J., Matheny, M. E., Johnson, K., Marsolo, K., Curtis, L. H., Nelson, J. C., . . . 

Schneeweiss, S. (2021). Broadening the reach of the FDA Sentinel system: A roadmap 

for integrating electronic health record data in a causal analysis framework. NPJ Digit 

Med, 4(1), 170. doi:10.1038/s41746-021-00542-0 

ElZarrad, M. K., Lee, A. Y., Purcell, R., & Steele, S. J. (2022). Advancing an agile regulatory 

ecosystem to respond to the rapid development of innovative technologies. Clin Transl 

Sci, 15(5), 1332–1339. doi:10.1111/cts.13267 

European Medicines Agency. (2022). DRAFT Qualification opinion for Prognostic Covariate 

Adjustment (PROCOVATM). European Medicines Agency Retrieved from 

https://www.ema.europa.eu/documents/other/draft-qualification-opinion-prognosticcovariate-adjustment-procovatm_en.pdf 

Exec. Order No. 13859, Maintaining American Leadership in Artificial Intelligence. (2019-

02544). (February 11, 2019). 84 Fed. Reg. 3967 Retrieved from 

https://www.federalregister.gov/documents/2019/02/14/2019-02544/maintainingamerican-leadership-in-artificial-intelligence 

Exec. Order No. 13960, Promoting the Use of Trustworthy Artificial Intelligence in the Federal 

Government. (2020-27065). (December 3, 2020). 85 Fed. Reg. 78939 Retrieved from 

https://www.federalregister.gov/documents/2020/12/08/2020-27065/promoting-the-useof-trustworthy-artificial-intelligence-in-the-federal-government 

Fumagalli, S. E., Padhiar, N. H., Meyer, D., Katneni, U., Bar, H., DiCuccio, M., . . . KimchiSarfaty, C. (2023). Analysis of 3.5 million SARS-CoV-2 sequences reveals unique 

mutational trends with consistent nucleotide and codon frequencies. Virol J, 20(1), 31. 

doi:10.1186/s12985-023-01982-8 

Ghosh, R., Kempf, D., Pufko, A., Barrios Martinez, L. F., Davis, C. M., & Sethi, S. (2020). 

Automation opportunities in pharmacovigilance: An industry survey. Pharmaceut Med, 

34(1), 7-18. doi:10.1007/s40290-019-00320-0 

Harrer, S., Shah, P., Antony, B., & Hu, J. (2019). Artificial Intelligence for Clinical Trial Design. 

Trends Pharmacol Sci, 40(8), 577-591. doi:10.1016/j.tips.2019.05.005 

Harrison, L. I., & Gibaldi, M. (1977). Physiologically based pharmacokinetic model for digoxin 

disposition in dogs and its preliminary application to humans. J Pharm Sci, 66(12), 1679-

1683. doi:10.1002/jps.2600661206 

Hsu, V., de, L. T. V., M., Zhao, P., Zhang, L., Zheng, J. H., Nordmark, A., . . . Huang, S. M. 

(2014). Towards quantitation of the effects of renal impairment and probenecid inhibition 

on kidney uptake and efflux transporters, using physiologically based pharmacokinetic 

modelling and simulations. Clin Pharmacokinet, 53(3), 283-293. doi:10.1007/s40262-

013-0117-y 

Huang, J., O'Connor, T., Ahmed, K., Chatterjee, S., Garvin, C., Ghosh, K., . . . Warman, M. 

(2021). AIChE PD2M Advanced Process Control workshop-moving APC forward in the 

pharmaceutical industry. Journal of Advanced Manufacturing and Processing, 3(1), 

e10071. doi:10.1002/amp2.10071 

Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., . . . Hassabis, D. 

(2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873), 

583-589. doi:10.1038/s41586-021-03819-2 

Kawakami, E., Tabata, J., Yanaihara, N., Ishikawa, T., Koseki, K., Iida, Y., . . . Okamoto, A. 

(2019). Application of atificial intelligence for preoperative diagnostic and prognostic 

prediction in epithelial ovarian cancer based on blood biomarkers. Clin Cancer Res, 

25(10), 3006-3015. doi:10.1158/1078-0432.CCR-18-3378 

28 
 

 

 

 

 

 

 

 

 

 

 

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

Kreimeyer, K., Dang, O., Spiker, J., Gish, P., Weintraub, J., Wu, E., . . . Botsis, T. (2022). 

Increased Confidence in Deduplication of Drug Safety Reports with Natural Language 

Processing of Narratives at the US Food and Drug Administration. Frontiers in Drug 

Safety and Regulation, 2. doi:10.3389/fdsfr.2022.918897 

Kreimeyer, K., Dang, O., Spiker, J., Munoz, M. A., Rosner, G., Ball, R., & Botsis, T. (2021). 

Feature engineering and machine learning for causality assessment in 

pharmacovigilance: Lessons learned from application to the FDA Adverse Event 

Reporting System. Comput Biol Med, 135, 104517. 

doi:10.1016/j.compbiomed.2021.104517 

Kuemmel, C., Yang, Y., Zhang, X., Florian, J., Zhu, H., Tegenge, M., . . . Zineh, I. (2020). 

Consideration of a credibility assessment framework in model-informed drug 

development: Potential application to physiologically-based pharmacokinetic modeling 

and simulation. CPT Pharmacometrics Syst Pharmacol, 9(1), 21-28. 

doi:10.1002/psp4.12479 

Lander, E., & Nelson, A. (October 22, 2021). ICYMI: WIRED (Opinion): Americans Need a Bill of 

Rights for an AI-Powered World. https://www.whitehouse.gov/ostp/newsupdates/2021/10/22/icymi-wired-opinion-americans-need-a-bill-of-rights-for-an-aipowered-world/ 

Laubenbacher, R., Sluka, J. P., & Glazier, J. A. (2021). Using digital twins in viral infection. 

Science, 371(6534), 1105-1106. doi:10.1126/science.abf3370 

Lewis, D. J., & McCallum, J. F. (2020). Utilizing advanced technologies to augment 

pharmacovigilance systems: Challenges and opportunities. Ther Innov Regul Sci, 54(4), 

888-899. doi:10.1007/s43441-019-00023-3 

Liu, Q., Huang, R., Hsieh, J., Zhu, H., Tiwari, M., Liu, G., . . . Huang, S. M. (2022). Landscape 

analysis of the application of artificial antelligence and machine learning in regulatory 

submissions for drug development from 2016 to 2021. Clin Pharmacol Ther. 

doi:10.1002/cpt.2668 

Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., & the SPIRIT-AI and 

CONSORT-AI Working Group. (2020). Reporting guidelines for clinical trial reports for 

interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med, 

26(9), 1364-1374. doi:10.1038/s41591-020-1034-x 

Liu, X., Liu, C., Huang, R., Zhu, H., Liu, Q., Mitra, S., & Wang, Y. (2021). Long short-term 

memory recurrent neural network for pharmacokinetic-pharmacodynamic modeling. Int J 

Clin Pharmacol Ther, 59(2), 138-146. doi:10.5414/CP203800 

Liu, Z., Chen, X., Carter, W., Moruf, A., Komatsu, T. E., Pahwa, S., . . . W., T. (2022). AIpowered drug repurposing for developing COVID-19 treatments. In Reference Module in 

Biomedical Sciences: Elsevier. 

Lu, J., Deng, K., Zhang, X., Liu, G., & Guan, Y. (2021). Neural-ODE for pharmacokinetics 

modeling and its advantage to alternative machine learning models in predicting new 

dosing regimens. iScience, 24(7), 102804. doi:10.1016/j.isci.2021.102804 

Ly, T., Pamer, C., Dang, O., Brajovic, S., Haider, S., Botsis, T., . . . Ball, R. (2018). Evaluation of 

Natural Language Processing (NLP) systems to annotate drug product labeling with 

MedDRA terminology. J Biomed Inform, 83, 73-86. doi:10.1016/j.jbi.2018.05.019 

Mager, D. E., Woo, S., & Jusko, W. J. (2009). Scaling pharmacodynamics from in vitro and 

preclinical animal studies to humans. Drug Metab Pharmacokinet, 24(1), 16-24. 

doi:10.2133/dmpk.24.16 

Mason, M., Cho, Y., Rayo, J., Gong, Y., Harris, M., & Jiang, Y. (2022). Technologies for 

medication adherence monitoring and technology assessment criteria: Narrative review. 

JMIR Mhealth Uhealth, 10(3), e35157. doi:10.2196/35157 

29 
 

 

 

 

 

 

 

 

 

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

Mouchlis, V. D., Afantitis, A., Serra, A., Fratello, M., Papadiamantis, A. G., Aidinis, V., . . . 

Melagraki, G. (2021). Advances in de Novo Drug Design: From Conventional to Machine 

Learning Methods. Int J Mol Sci, 22(4). doi:10.3390/ijms22041676 

National Academies of Sciences, E., and Medicine. (2021). Innovations in Pharmaceutical 

Manufacturing on the Horizon: Technical Challenges, Regulatory Issues, and 

Recommendations. Retrieved from https://doi.org/10.17226/26009. 

National Institute of Standards and Technology. (2019). U.S. Leadership in AI: A Plan for 

Federal Engagement in Developing Technical Standards and Related Tools. Retrieved 

from 

https://www.nist.gov/system/files/documents/2019/08/10/ai_standards_fedengagement_ 

plan_9aug2019.pdf 

Negi, K., Pavuri, A., Patel, L., & Jain, C. (2019). A novel method for drug-adverse event 

extraction using machine learning. Informatics in Medicine Unlocked, 17, Article 100190. 

doi:10.1016/j.imu.2019.100190 

Notice of Request for Information on Public and Private Sector Uses of Biometric Technologies. 

(2021-21975). (October 8, 2021). 86 Fed. Reg. 56300 Retrieved from 

https://www.federalregister.gov/documents/2021/10/08/2021-21975/notice-of-requestfor-information-rfi-on-public-and-private-sector-uses-of-biometric-technologies 

Organisation for Economic Co-operation and Development. (2019). Recommendation of the 

Council on Artificial Intelligence. Retrieved from 

https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449 

Pappalardo, F., Russo, G., Tshinanu, F. M., & Viceconti, M. (2019). In silico clinical trials: 

concepts and early adoptions. Brief Bioinform, 20(5), 1699-1708. 

doi:10.1093/bib/bby043 

Pierce, C. E., Bouri, K., Pamer, C., Proestel, S., Rodriguez, H. W., Van Le, H., . . . Dasgupta, N. 

(2017). Evaluation of Facebook and Twitter monitoring to detect safety signals for 

medical products: An analysis of recent FDA safety alerts. Drug Saf, 40(4), 317-331. 

doi:10.1007/s40264-016-0491-0 

Routray, R., Tetarenko, N., Abu-Assal, C., Mockute, R., Assuncao, B., Chen, H., . . . Mingle, E. 

(2020). Application of augmented intelligence for pharmacovigilance case seriousness 

determination. Drug Saf, 43(1), 57-66. doi:10.1007/s40264-019-00869-4 

Sager, J. E., Yu, J., Ragueneau-Majlessi, I., & Isoherranen, N. (2015). Physiologically based 

pharmacokinetic (PBPK) modeling and simulation approaches: A systematic review of 

published models, applications, and model verification. Drug Metab Dispos, 43(11), 

1823-1837. doi:10.1124/dmd.115.065920 

Schmider, J., Kumar, K., LaForest, C., Swankoski, B., Naim, K., & Caubel, P. M. (2019). 

Innovation in pharmacovigilance: Use of artificial intelligence in adverse event case 

processing. Clin Pharmacol Ther, 105(4), 954-961. doi:10.1002/cpt.1255 

Schneider, P., Walters, W. P., Plowright, A. T., Sieroka, N., Listgarten, J., Goodnow, R. A., . . . 

Schneider, G. (2020). Rethinking drug design in the artificial intelligence era. Nat Rev 

Drug Discov, 19(5), 353-364. doi:10.1038/s41573-019-0050-3 

Schuler, A., Walsh, D., Hall, D., Walsh, J., Fisher, C., & for the Critical Path for Alzheimer's, D., 

the Alzheimer's Disease Neuroimaging Initiative and the Alzheimer's Disease 

Cooperative Study. (2021). Increasing the efficiency of randomized trial estimates via 

linear adjustment for a prognostic score. Int J Biostat. doi:10.1515/ijb-2021-0072 

Shroff, T., Aina, K., Maass, C., Cipriano, M., Lambrecht, J., Tacke, F., . . . Loskill, P. (2022). 

Studying metabolism with multi-organ chips: new tools for disease modelling, 

pharmacokinetics and pharmacodynamics. Open Biol, 12(3), 210333. 

doi:10.1098/rsob.210333 

30 
 

 

 

 

 

 

 

 

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

Spiker, J., Kreimeyer, K., Dang, O., Boxwell, D., Chan, V., Cheng, C., . . . Botsis, T. (2020). 

Information visualization platform for postmarket surveillance decision support. Drug Saf, 

43(9), 905-915. doi:10.1007/s40264-020-00945-0 

Stehlik, J., Schmalfuss, C., Bozkurt, B., Nativi-Nicolau, J., Wohlfahrt, P., Wegerich, S., . . . 

Pham, M. (2020). Continuous Wearable Monitoring Analytics Predict Heart Failure 

Hospitalization: The LINK-HF Multicenter Study. Circ Heart Fail, 13(3), e006513. 

doi:10.1161/circheartfailure.119.006513 

Vamathevan, J., Clark, D., Czodrowski, P., Dunham, I., Ferran, E., Lee, G., . . . Zhao, S. (2019). 

Applications of machine learning in drug discovery and development. Nat Rev Drug 

Discov, 18(6), 463-477. doi:10.1038/s41573-019-0024-5 

Viceconti, M., Pappalardo, F., Rodriguez, B., Horner, M., Bischoff, J., & Musuamba Tshinanu, F. 

(2021). In silico trials: Verification, validation and uncertainty quantification of predictive 

models used in the regulatory evaluation of biomedical products. Methods, 185, 120-

127. doi:10.1016/j.ymeth.2020.01.011 

Vought, R. T. (2020). Guidance for Regulation of Artifical Intelligence Applications. Retrieved 

from https://www.whitehouse.gov/wp-content/uploads/2020/11/M-21-06.pdf 

Wang, S. V., Schneeweiss, S., Berger, M. L., Brown, J., de Vries, F., Douglas, I., . . . the joint 

ISPE-ISPOR Special Task Force on Real World Evidence in Health Care Decision 

Making. (2017). Reporting to improve reproducibility and facilitate validity assessment for 

healthcare database studies V1.0. Pharmacoepidemiol Drug Saf, 26(9), 1018-1032. 

doi:10.1002/pds.4295 

Wang, W., Haerian, K., Salmasian, H., Harpaz, R., Chase, H., & Friedman, C. (2011). A drugadverse event extraction algorithm to support pharmacovigilance knowledge mining from 

PubMed citations. AMIA Annu Symp Proc, 2011, 1464-1470. Retrieved from 

https://www.ncbi.nlm.nih.gov/pubmed/22195210 

Wang, Y., Carter, B. Z., Li, Z., & Huang, X. (2022). Application of machine learning methods in 

clinical trials for precision medicine. JAMIA Open, 5(1), ooab107. 

doi:10.1093/jamiaopen/ooab107 

Weissler, E. H., Naumann, T., Andersson, T., Ranganath, R., Elemento, O., Luo, Y., . . . 

Ghassemi, M. (2021). The role of machine learning in clinical research: transforming the 

future of evidence generation. Trials, 22(1), 537. doi:10.1186/s13063-021-05489-x 

Zhang, X., Yan, C., Gao, C., Malin, B. A., & Chen, Y. (2020). Predicting Missing Values in 

Medical Data via XGBoost Regression. J Healthc Inform Res, 4(4), 383-394. 

doi:10.1007/s41666-020-00077-1 

31