AMED Research on Regulatory Science ofPharmaceuticals and Medical Devices

Title of Research Project: Regulatory science research to achieve
efficiency of clinical research and developmentwith the utilization of a patient registry[Grant Number: JP21mk0101154]

Subproject Title: Consideration of biostatistical issues and linkage withother databases

Biostatistical Considerations When Utilizing RWD/RWE in
Pharmaceutical and Medical Device Development

March31, 2022
Ver1.0

Representative, Subgroup for “Consideration of biostatistical issues
and linkage with other databases” :Yukari Uemura (National Center for Global Health and Medicine)Principal Investigator:
Taro Shibata (National Cancer Center Japan)

Table of Contents
1.

Background ......................................................................................................................02.

Purposes and scope of application ...................................................................................11

3.

Securing transparency for using RWD ............................................................................15

4.

Single-arm clinical trial using RWD as an external control .............................................18
4.1.

Assumed situations ..................................................................................................18

4.2.

Points to consider for implementing the design and analysis ....................................19

4.3.

Issues to consider regarding statistical methods .......................................................26

4.3.1.

Discussion of statistical methods for evaluating efficacy and safety using RWD 26

4.3.2.

Representative biases and how to deal with them..............................................28

4.3.3.

Bayesian method ..............................................................................................35

4.3.4.

Assessment of validity .......................................................................................38

4.4.

Points to consider for reporting................................................................................39

4.4.1.

Necessity for uniform format ............................................................................39

4.4.2.

Points to consider when reporting according to the CONSORT checklist .........40

4.5.

Cases .......................................................................................................................44

4.5.1.

Single-arm clinical trials using the SCRUM JAPAN registry as external controls
44

4.5.2.

Points to consider when performing statistical analysis from the perspective of a

new drug application .......................................................................................................45
5.

6.

7.

Randomized clinical trial using RWD as hybrid control ..................................................48
5.1.

Hybrid control .........................................................................................................48

5.2.

Possibilities and statistical issues of hybrid control ..................................................49

5.2.1.

Advantages and disadvantages .........................................................................50

5.2.2.

Issues in the future ...........................................................................................52

Use of RWD as complementary information ...................................................................57
6.1.

Introduction.............................................................................................................57

6.2.

Complementation of RWD in the regulatory application of data from pivotal trials.58

6.3.

Use of RWD as a clinical trial platform ....................................................................59

6.4.

Conclusion ...............................................................................................................61

Co-analysis of registry data and data from other databases.............................................64
7.1.

Example of co-analyzing registry data and data from other databases in each NC ...65

7.1.1.

Efforts by National Center for Child Health and Development .........................65

7.1.2.

Efforts by National Center for Geriatrics and Gerontology ..............................671

7.1.3.

8.

Efforts by National Center of Neurology and Psychiatry ..................................68

7.2.

Efforts in the world ..................................................................................................70

7.3.

Issues in conjunction with other databases ...............................................................72

Issues to consider for implementation .............................................................................76
8.1.

Data extraction ........................................................................................................76

8.2.

Conjunction with other databases ............................................................................76

Appendix． ............................................................................................................................78
List of authors and reviewers .................................................................................................842

1.

Background
Randomized controlled trials (RCTs) are an established scientific methodology for evaluating theefficacy and safety of drugs and medical devices in humans and are generally recognized as the mostpowerful means to evaluate their effectiveness1,2. However, conducting clinical trials with a sufficientnumber of patients for appropriately estimating the therapeutic effect is difficult for rare diseases, rare
fractions, regenerative medicine, and gene therapies. In addition, in certain serious disease areas, theuse of placebo is ethically unacceptable. Thus, an increasing number of issues are difficult to addressin the framework of conventional RCTs3. Furthermore, there is usually a gap between the target
population in a clinical trial for marketing approval and the actual population in the medical practiceto which the evidence of the clinical trial is applied. Hence, this discrepancy is likely to limit the extent
to which the findings of a clinical trial can be generalized from the population of participants in thattrial to the broader actual patient population and broader medical practice4.
Under these circumstances, various efforts have been made by the industry, the government, andacademia to utilize real-world data (RWD) for the development and evaluation of pharmaceuticals andmedical devices. One reason for the recent increase in these activities is the 21st Century Cures Act5
enacted in the United States (US) in 2016, which recommends the use of real-world evidence (RWE)
for new drug development, additional indications, and post-marketing safety evaluation. Under theAct, the Food and Drug Administration (FDA) has released a framework for the RWE program in
2018,6 as well as guidance on the use of RWE in medical device development and guidance on points
to consider for submitting documents using RWD and RWE to the FDA in the development of drugs
and biologics7,8. Additionally, in collaboration with Brigham and Women’s Hospital in the USA, theRCT DUPLICATE project, a verification program for predicting the results of an RCT using RWE,
has been undertaken since 20179. Furthermore, in Europe, a GetREAL project of the Innovative
Medicines Initiative (IMI) has launched the initiative to utilize RWE for drug development,10 and theinfrastructure for using RWE for regulatory approval is being developed. The FDA defines RWD as“data relating to patient health status or the delivery of health care routinely collected from a variety
of sources” and RWE as “the clinical evidence regarding the usage and potential benefits or risks of a
medical product derived from analysis of RWD.” In this report, RWD and RWE are referred toaccording to these definitions. Additionally, this report discusses the use of clinical trial databases.
In Japan, a conditional early approval system has been implemented since 2017 and enshrined into
law in 2019 with an amendment of the Act on Securing Quality, Efficacy and Safety of
Pharmaceuticals, Medical Devices, Regenerative and Cellular Therapy Products, Gene Therapy
Products, and Cosmetics11. In line with this, in post-marketing surveillance (PMS) that is mandated asa condition of approval and that the sponsor must additionally conduct, medical information databases
and patient registries are allowed to be utilized as necessary, and the revised Good Post-marketingStudy Practice (GPSP) Ministerial Ordinance was promulgated in February 2018, allowing PMS using
medical information databases 12. With regard to the use of RWD, the Ministry of Health, Labour andWelfare (MHLW) has enacted ministerial ordinances and notifications regarding: Documents to be
attached to reexamination applications for new ethical drugs, basic principles on utilization of healthinformation databases for PMS of medical products,13 and securing the reliability of post-marketing
database surveys of drugs and medical devices14,15,16. Furthermore, in 2019, the Pharmaceuticals andMedical Devices Agency (PMDA) sought comments regarding the “Basic principles in conducting avalidation study of outcome definitions used for post-marketing database study” 17 and issued adocument based on its results in July 202018. In the fiscal year (FY) 2020, guidelines were issued forbasic principles on utilization of registry for applications19, and points to consider for ensuring
reliability in the utilization of registry data for applications20. In addition, the PMDA newly establisheda new consultation segment in FY 2019 regarding clinical trial design based on the utilization of theregistry for improving the quality and ensuring the reliability of the registry21.
As described, the Japanese government and regulatory authorities are providing notifications andguidance on the use of RWD for regulatory approval and re-evaluation submissions. The scope of
specific utilization of RWD in clinical research is summarized in a slide presented by the FDA at aworkshop held at Duke-Margolis Center22. Figure 1 presents its outline in Japanese translation. Study
designs utilizing RWD can be classified into three types: RCTs, non-randomized interventional trials,
and observational studies (Figure 1). Numerous efforts have been made for referring to RWD for
conducting an RCT such as for setting enrollment criteria, assessing the feasibility of implementing
the trial, and selecting sites. Alternatively, the framework of a conventional RCT can include the useof data directly collected from electronic health records (EHR), which is RWD, as the source data ordata obtained by incorporating an electronic data capture (EDC) application into a smartphone
terminal. Likewise, by combining RWD with analysis technologies such as artificial intelligence,
which has shown rapid progress in recent years, endpoints can be judged as auxiliary, and theutilization of data obtained from wearable device terminals as endpoints is also being considered. 23.
In addition, pragmatic RCTs using RWD can be conducted; in several cases, approvals have been
granted by the FDA for addition of indications24 and pragmatic RCTs have been conducted using
registries as a platform.
The use of RWD as an external control in a single-arm trial is attracting attention as a method ofutilizing RWD in the development of drugs for rare diseases, which is difficult under the traditional
RCT framework25. Its main purpose is to evaluate hypotheses that single-arm trials alone do notprovide answers to and to find scientific knowledge and evidence that is useful in real-world medical
practice, by actively combining and integrating RWD as an external control with data obtained froma single-arm trial. Additionally, as a relatively new challenge, in a proposed study design, RWD has
been supplemented in a subset of the control group of an RCT rather than in all external RWD controls.
1

Observational studies have been generally known to use RWD, i.e., cohort studies that prospectively
collect data and retrospective cohort studies using data from existing databases.

Figure 1. Cited from the slide of the framework for the FDA’s RWE program
RWD may be potentially used for a wide range of study designs; however, when utilizing RWD, it
is crucial to consider whether the research setting is suitable for the use of RWD and whether thecontents and quality of RWD are fit for the purpose, as well as to select a study design that provides
RWE that can test the hypothesis and to conduct appropriate analyses26. In this report, we will focus
primarily on study designs that use RWD for part or all of a control group in clinical trials and outline
statistical considerations. Notably, the term external control is not recently emerged; it has been
mentioned in the “Choice of Control Group and Related Issues in Clinical Trials” [E-10 guideline of
the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for
Human Use (ICH)], which states that “the principal methods of determining who will be in the control
group are by randomization or by selection of a control population separate from the population treatedin the trial (external or historical control)27.” In addition, in the revised version of the ICH-E8 guideline(R1)28 endorsed to be part of Step 3 (Regional regulatory consultation, Discussion of regional
consultation comments, and Experts Draft Guideline) in May 2019, the external control is newlyreferred to with the keyword “internal control”. In the near future, the use of RWD collected outside2

of a clinical trial environment may become more of an option for regulatory authorities in regulatory
submissions and PMS.
As previously mentioned, notifications and guidance have been issued by the Japanese
government and regulatory authorities on the use of RWD for regulatory approval applications andreexamination applications, and studies on the use of RWD/RWE are underway. Nevertheless, inseveral cases in Japan and abroad, RWD/RWE has already been used in regulatory applications. For
example, Table 1 presents cases where RWE was used for evaluating the efficacy of a drug
application submitted to the FDA for approval29,30.
Table 1. Examples of the use of RWD for regulatory applications to FDA

In Japan, several drugs and medical devices have received regulatory approved using RWD as anexternal control; however, the term "RWD" was not used at that time. For example, an investigatorinitiated clinical trial evaluated the efficacy and safety of tacrolimus hydrate and corticosteroid
combination therapy in patients with polymyositis/dermatomyositis and interstitial pneumonitis in a
retrospective cohort study31. Specifically, the planned trial design involved a experimental treatmentgroup that was compared with a historical control group (i.e., external control) established byretrospectively collecting data on initial treatment with corticosteroid alone. For the development of
medical devices, a single-arm clinical trial was conducted for a new approval application for Najuta,
a thoracic stent graft system to repair aortic aneurysm, and enrollment in its control group was3

conducted through the Japan Adult Cardiovascular Surgery Database (JACVSD)32. Short-term
therapeutic effect and safety were compared between the stent graft and surgical treatments. The
results of this study were submitted as part of an approval application, and approval was granted in
201233,34. Likewise, for the marketing approval for EXCOR Pediatric, an extracorporeal ventricularassist system for children, the results of a single-arm trial were compared with the treatment resultsof extracorporeal circulation, a conventional therapy, and approval was granted to EXCOR Pediatric.
For the conventional therapy, registry data separately accumulated in foreign countries were used35.
In other cases, reexamination applications have been filed using registries established throughcooperation among academic societies, medical device companies, and the PMDA (ventricular assist
device registry, registry for transcatheter aortic valve replacement with prosthetic valve, and registry
for cerebral aneurysm treatment devices). Thus, although the use of RWD in approval review has
been reported in several cases, the predictability of the conditions under which RWD can be used forexternal control is poor, which may act as a hurdle for using RWD in the development of drugs andmedical devices.
Under the abovementioned circumstances, examination of the utilization of RWD for thedevelopment of drugs and medical devices has been initiated, and the clinical innovation network(CIN) concept was proposed for promoting the building of an infrastructure for efficient clinical trials,
post-marketing surveys, and clinical research with the use of disease registry data 36. The National
Research Center for Advanced and Specialized Medical Care (National Center) and academic societies
have built their disease registries and established systems, organized environments, and developed
guidelines to efficiently use data obtained from the registries in clinical trials, clinical researches, andpost-marketing surveys. In particular, in response to the recommendations from an FY 2015 Healthand Labour Sciences Special Research Project, “A Study on Measures to Promote New Clinical Trials
and Clinical Researches on the Basis of Disease Registration Systems (Patient Registries) Establishedby the National Research Center for Advanced and Specialized Medical Care (National Center), etc.
(H27-Special-Designated-018: Research representative, Shinichi Takeda, M.D., National Center of
Neurology and Psychiatry),” seven research groups were established under the CIN concept37. Of theseven research groups, four have established and maintained registries (registries for muscular
dystrophy, amyotrophic lateral sclerosis, rare fractions of cancer, and neurosurgery inpatients), andthree have examined common issues in each disease registry in a cross-sectional manner. The crosssectional research groups are Development of New Clinical Research Designs to Utilize Patient
Registries (Hayashi Group)38, CIN promotion measures (Takeda group)39, and Establishment of aComprehensive Center for Patient Registry Information (Kokudo group)40. Among these researchgroups, the group for Japan Agency for Medical Research and Development (AMED) Research on
Regulatory Science of Pharmaceuticals and Medical Devices “Utilization of Real World Evidence
Using Patient Registry Data to Support Regulatory Decision‒Making” (FY 2016 to 2018) [Research
4and development representative: Kunihiko Hayashi, M.D. (Gunma University)] summarized papers
and reports of evaluation on points to consider when using registries as data for obtaining approval for
drugs from the perspectives of reliability and statistics for so-called difficult areas investigated using
RCTs such as rare diseases, rare fractions, and pediatric diseases41. In addition, the Japan
Pharmaceutical Manufacturers Association has published many reports on the methods for using
RWD/RWE and points to consider for these reports42–47.
As a successor to the AMED Hayashi group, the following group was launched: AMED: Research
on Regulatory Science of Pharmaceuticals and Medical Devices “A Regulatory Science Research on
the Use of Patient Registry Data to Facilitate Efficient Clinical Development” (July 2019 to March
2022) [Research and development representative: Taro Shibata (National Cancer Center Japan)]. As
previously mentioned, while the registries of each National Center (NC) are being organized under the
CIN concept, the reliability of the data and the utilization of the data to enhance its value appeared to
be in a trial-and-error stage. Furthermore, the ICH GCP renovation provides multiple trial design anddata source options; thus, the development of drugs and medical devices using RWD is anticipated tofurther advance in the future. Under such circumstances, “Biostatistical considerations and discussionsof database linkage with other data sources,” a sub-study group of the AMED Shibata groupsummarizes and reports the results of our study on the feasibility and difficulties of linkage between
multiple databases, as well as the consideration of various databases and data utilization methods froma biostatistical perspective, while taking into account the continuity of the results of the preceding
AMED Hayashi Group.

References
1. Zelen M. (1979). A new design for randomized clinical trials. N Engl J Med, 300, 1242-1246.
2. Bothwell LE, Greene JA, Podolsky SH, et al. (2016). Assessing the Gold Standard--Lessons fromthe History of RCTs. N Engl J Med, 374(22): 2175-2181.
3. Dreyer NA. (2018). Advancing a Framework for Regulatory Use of Real-World Evidence: When
Real Is Reliable. Ther Innov Regul Sci, 52(3): 362-368.
4. Cerreta F, Eichler HG, Rasi G.(2012). Drug policy for an aging population--the European
Medicines Agency's geriatric medicines strategy. N Engl J Med, 367(21) : 1972-1974.
5. United

States

Food

&

Drug

Administration.

21st

Century

Cures

Act.

https://www.fda.gov/regulatory-information/selected-amendments-fdc-act/21st-century-curesact(Accessed 7/20, 2020)
6. United States Food & Drug Administration. Framework for FDA's Real-World Evidence Program.
https://www.fda.gov/media/120060/download (accessed July 2020)
7. United States Food & Drug Administration. (2017). Use of Real-World Evidence to Support
5Regulatory Decision-Making for Medical Devices.
https://www.fda.gov/media/99447/download (Accessed 7/20, 2020)
8. United States Food & Drug Administration. (2019). Submitting Documents Using Real-World
Data and Real-World Evidence to FDA for Drugs and Biologics．
https://www.fda.gov/media/124795/download (Accessed 7/20, 2020)
9. RCT DUPLICATE. https://www.rctduplicate.org/ (Accessed 7/20, 2020)
10. New methods for RWE collection and synthesis. https://www.imi-getreal.eu/ (Accessed 7/20,
2020)
11. Pharmaceutical Evaluation Division, Pharmaceutical Safety and Environmental Health Bureau,
Ministry of Health, Labour and Welfare. (2017). Implementation of a Conditional Early Approval
System for Pharmaceutical Products. (PSEHB/PED Notification No. 1020-1 dated October 20,2017).https://www.pmda.go.jp/files/000220723.pdf (accessed July 2020)
12. Pharmaceutical Safety and Environmental Health Bureau, Ministry of Health, Labour and Welfare.
(2017). Ministerial Ordinance for Partial Revision of the Ministerial Ordinance on Good Postmarketing Study Practice (GPSP).

(PSEHB/PED Notification No. 1020-1 dated October 26,2017). https://www.pmda.go.jp/files/000220767.pdf (Accessed July 2020)
13. Pharmaceutical Evaluation Division and Safety Division, Pharmaceutical Safety andEnvironmental Health Bureau, Ministry of Health, Labour and Welfare. (2017). Basic Conceptson the Use of Medical Information Databases for Post-marketing Pharmacovigilance(PSEHB/PED Notification No. 0609-8/PSEHB/PSD Notification No. 0609-4 dated June 9, 2017).
(Accessed 5/20, 2021)
14. Pharmaceutical Evaluation Division, Pharmaceutical Safety and Environmental Health Bureau,
Ministry of Health, Labour and Welfare. Documents to Be Attached to Reexamination
Applications for New Ethical Drugs [PSEHB/PED Notification No. 1128-2 from the Director of
the Pharmaceutical Evaluation Division, Pharmaceutical Safety and Environmental Health Bureau,
Ministry of Health, Labour and Welfare dated November 28, 2017 (revised on March 25, 2020)].
(Accessed 5/20, 2021)
15. Pharmaceutical Evaluation Division, Pharmaceutical Safety and Environ mental Health Bureau,
the Ministry of Health, Labour and Welfare. Points to Consider for Ensuring the Reliability of
Post-marketing Database Study for Drugs. (MHLW/PSEHB/PED Notification No.

0221‒1

dated February 21, 2018). https://www.pmda.go.jp/files/000223003.pdf (Accessed 7/20, 2020)
16. P Medical Device Division, Pharmaceutical Safety and Environmental Health Bureau, theMinistry of Health, Labour and Welfare. Points to Note regarding Reliability Assurance inPostmarketing Database Surveys for Medical Devices (PSEHB/MDED Notification No. 1219-4
dated December 19, 2018). https://www.pmda.go.jp/files/000229088.pdf (Accessed 7/20, 2020)
17. Pharmaceuticals and Medical Devices Agency. (2019). Invitation of Opinions on the “Basic
6

principles in conducting a validation study of outcome definitions used for post-marketing
database study.” https://www.pmda.go.jp/safety/pub-comments/0019.html (Accessed 7/20, 2020)
18. Pharmaceutical Evaluation Division, Pharmaceutical Safety and Environmental Health Bureau,
Ministry of Health, Labour and Welfare, Pharmaceutical Safety Division, Pharmaceutical Safety
and Environmental Health Bureau, Ministry of Health, Labour and Welfare. (2020). Establishmentof the Basic Concepts on the Implementation of Validation of Outcome Definitions Used in Postmarketing Database Surveys (Notification No. 0731001 issued by the Director of Office ofRegulatory Science, Pharmaceuticals and Medical Devices Agency and Notification No. 0731001
issued by the Director of the Center for Product Evaluation, Pharmaceuticals and Medical Devices
Agency, dated July 31, 2020).
https://www.mhlw.go.jp/hourei/doc/tsuchi/T200803I0020.pdf (Accessed 7/20, 2020)
19. Pharmaceutical Evaluation Division, Pharmaceutical Safety and Environmental Health Bureau,
Ministry of Health, Labour and Welfare, Medical Device Evaluation Division, Pharmaceutical
Safety and Environmental Health Bureau, Ministry of Health, Labour and Welfare. (2021). “Basicprinciples on Utilization of Registry for Applications.” (PSEHB/PED Notification No. 0323-1/
PSEHB/MDED Notification No. 0323-1 dated March 23, 2021).
https://www.mhlw.go.jp/hourei/doc/tsuchi/T210324I0010.pdf. (Accessed 3/29, 2021)
20. Pharmaceutical Evaluation Division, Pharmaceutical Safety and Environmental Health Bureau,
Ministry of Health, Labour and Welfare, Medical Device Evaluation Division, Pharmaceutical
Safety and Environmental Health Bureau, Ministry of Health, Labour and Welfare. (2021). “Points
to Consider for Ensuring the Reliability in Utilization of Registry Data for Application.”(PSEHB/PED Notification No. 0323-2/ PSEHB/MDED Notification No. 0323-2 dated March 23,2021).
21. Pharmaceuticals and Medical Devices Agency. Consultation for the Use of Regenerative Medicine
Product

Registries.

https://www.pmda.go.jp/review-services/f2f-pre/consultations/0101.html(Accessed 7/20, 2020)
22. United States Food & Drug Administration & Duke-Margolis Center for Health Policy. (2019).
Public Workshop Developing Real-World Data and Evidence to Support Regulatory DecisionMaking. https://healthpolicy.duke.edu/events/developing-real-world-data-and-evidence-supportregulatory-decision-making.

(Accessed 7/20, 2020)

23. Ali Z, Zibert JR, Thomsen. SF. (2020). Virtual Clinical Trials: Perspectives in Dermatology.
Dermatology, 236(4): 375-382.
24. United States Food & Drug Administration (2017).
https://www.accessdata.fda.gov/drugsatfda_docs/nda/2017/022264Orig1s015.pdf(Accessed

12/5, 2020)
25. Franklin JM, Glynn RJ, Martin D, et al. (2019). Evaluating the Use of Nonrandomized Real-World
7

Data Analyses for Regulatory Decision Making. Clin Pharmacol Ther, 105(4): 867-877.
26. Simon R, Blumenthal GM, Rothenberg ML, et al. (2015). The role of nonrandomized trials in
the evaluation of oncology drugs. Clin Pharmacol Ther, 97(5): 502-507.
27. ICH-E10 “CHOICE OF CONTROL GROUP AND RELATED ISSUES IN CLINICAL TRIALS”.
https://www.pmda.go.jp/files/000156803.pdf(Accessed 7/20, 2020)(Accessed 7/20, 2020)

28. ICH-E8 （ R1 ） “GENERAL CONSIDERATIONS FOR CLINICAL STUDIES (Step4).
https://database.ich.org/sites/default/files/E8-R1_Guideline_Step4_2021_1006.pdf(Accessed

10/14, 2021)
29. United States Food & Drug Administration. (2019).Regulatory Use of Real World Evidence:
Expectations, Opportunities, and Challenges
https://isctm.org/public_access/Feb2019/Presentation/Payers-Stein.pdf (Accessed 7/20, 2020)
30. Andre EB, Reynolds R, Caubel P, et al. (2019). Trial designs using real-world data: The changing
landscape of the regulatory approval process. Pharmacoepidemiol Drug Saf, 29(10):1201-1212.
31. Pharmaceuticals and Medical Devices Agency. (2013). “Review Report of Tacrolimus Hydrate.”
http://www.pmda.go.jp/drugs/2013/P201300087/800126000_20500AMZ00157000_A100_2.pdf(Accessed 7/20, 2020)
32. Japan Cardiovascular Surgery Database Organization. http://jcvsd.umin.jp/ (Accessed 7/20,
2020)
33. Pharmaceuticals and Medical Devices Agency. (2011). Review Report of Aortic Stent Graft
https://www.pmda.go.jp/medical_devices/2013/M201300001/200162000_22400BZX00516000_A100_2.pdf (Accessed 7/20, 2020)
34. Tomotaki A, Miyata H, Okubo S, et al. (2012). The Efficiency of Data Collection Using the Japan
Adult Cardiovascular Surgery Database (JACVSD) as a Historical Control in Clinical Trials.
Japanese Journal of Cardiovascular Surgery. 41(1): 1–7.
https://www.pmda.go.jp/medical_devices/2015/M20150727001/200638_22700BZX00179000_
A100_1.pdf (Accessed 7/20, 2020)
35. Pharmaceuticals and Medical Devices Agency. (2015). Review Report of Extracorporeal
Ventricular Assist Pump for Single Use.
https://www.pmda.go.jp/medical_devices/2015/M20150727001/200638_22700BZX00179000_
A100_1.pdf (Accessed 7/20, 2020)
36. Health Sciences Council. (2018). Documents of the FY 2018 Second meeting of the Medical
Device System Subcommittee.
https://www.mhlw.go.jp/file/05-Shingikai-10601000-DaijinkanboukouseikagakukaKouseikagakuka/02_2.pdf (Accessed 7/20, 2020)
37. Health and Labour Sciences Special Research Project. 2015. Research and Substudy Report: A
Study on Measures to Promote New Clinical Trials and Clinical Researches on the Basis of8

Disease Registration Systems (Patient Registries) Established by the National Research Center
for Advanced and Specialized Medical Care (National Center), etc. (Research Representative,
Shinichi Takeda).
38. AMED: Research on Regulatory Science of Pharmaceuticals and Medical Devices. 2019.
Utilization of Real World Evidence Using Patient Registry Data to Support Regulatory Decision‒
Making (Research and Development Representative, Kunihiko Hayashi, Grant number,
JP18mk0101068).．
39. Clinical Research and Trial Promotion Study Project. A Study on Measures to Promote the ClinicalInnovation Network Concept with the Effective Use of Disease Registry Systems (Research andDevelopment Representative, Shinichi Takeda).
https://www.amed.go.jp/content/000037331.pdf (Accessed 7/20, 2020)
40. Clinical Innovation Network Promotion Support Project. Establishment of a Comprehensive Basefor Registry Information to Accelerate and Promote the Clinical Innovation Network Concept(Research and Development Representative, Norihiro Kokudo)https://cinc.ncgm.go.jp/ (Accessed 7/20, 2020)
41. Shibata T. (2019). Proposal on assuring the reliability of patient registry data for use in application
dossiers of pharmaceuticals and medical devices. Japan Society of Clinical Trials and Research,
47(Suppl 1): 13–22.
42. Japan Pharmaceutical Manufacturers Association (2019). FY2018 Task Force 2, Clinical
Evaluation Subcommittee, Drug Evaluation Committee. How Much Existing Domestic Realworld Data Can Be Used for Drug Development?
http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/rwd2.pdf (Accessed 7/20, 2020)
43. Japan Pharmaceutical Manufacturers Association. (2016). FY2015 Task Force 3, Data Science
Subcommittee, Drug Evaluation Committee. Examination on “data quality.”
http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/rwd_quality.pdf (Accessed 7/20,
2020)
44. Japan Pharmaceutical Manufacturers Association. (2019). “FY2019 Task Force 2, Clinical
Evaluation Subcommittee, Drug Evaluation Committee. Toward the Promotion of Use of RWD
by Pharmaceutical Companies –current Status, Topics, Solving Issues, Future View–.”
http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/bd_rwd_sg3.pdf （ accessed July
2020）
45. Japan Pharmaceutical Manufacturers Association. (2019). “FY2019 Task Force 2, Clinical
Evaluation Subcommittee, Drug Evaluation Committee. The First Step Toward Approval
Applications Using External Control Groups –Tips and Issues from Disease Registries–.”
http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/bd_rwd_sg1.pdf
46. Togo K, Kawamatsu S, Kiguchi R, et al. (2019). “Utilizing Real-world Data in Drug Development
9

–Expectations From Pharmaceutical Companies–.” Japanese Journal of Pharmacoepidemiology.
24(1), 19-30.
47. Office of Pharmaceutical Industry Research, Japan Pharmaceutical Manufacturers Association.
(2018). “Report of the Study Group for the Use of Big Data From Medical and Health Care Vol.
3.”
http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/rwd_quality.pdf (Accessed 7/20,
2020)

10

2.

Purposes and scope of application
The effective use of RWD in the development of pharmaceuticals and medical devices is expected

to play an important role in supporting the decision-making process of regulatory authorities. In
utilizing RWD, biostatistical considerations from a scientific perspective are considered essential, in
addition to regulatory, data reliability, and ethical perspectives. This document provides an overview
of the biostatistical considerations when using RWD/RWE in the development of drugs and medical
devices, outlines trial designs that use RWD/RWE for enhancing the clinical evidence obtained, anddiscusses the biases that need to be considered and strategies to address them, while considering thecontinuity of results obtained by the AMED Hayashi group. This document is also intended to discuss
the feasibility and difficulties of linkage among multiple databases.
While there are no specific disease areas in which RWD, including registry data, are used in clinical
trials of drugs and medical devices, this report focuses on the use of RWD in rare diseases, pediatricdiseases, diseases with serious life-threatening effects (fatal diseases), and diseases with high medical
needs for which no fundamental treatment has been found, so-called "disease areas for which RCTs
are difficult to conduct”. This report does not deny the possibility of using RWD for disease areas
other than the abovementioned ones; these areas are briefly addressed in the report. In this report, wesummarize the results of our review of statistical considerations, focusing on disease areas where
conducting RCTs is challenging.
Specific scenes of using RWD/RWE for regulatory applications can be classified into four types: (1)new approval applications, (2) expansion of indications and target populations, (3) post-marketing
efficacy and safety evaluations of drugs to which conditional early approval is granted, and (4) use forpost-marketing efficacy and safety evaluations (see Figure 2)1. While the report by AMED Hayashi
group does not especially identify scenarios for using RWD, this report outlines situations related toregulatory approval as the main scope. The following is an overview of the situations that could be
utilized at each timepoint.
（1）New approval applications
Drug or medical-device data for approval are collected from clinical trials, while the control data may
be obtained from external controls. Alternatively, part of the control data in the RCT could be data forthe external control. RWD may be utilized in disease areas where a traditional RCT poses challenges;
examples of these areas are rare diseases or a small number of cases in a subpopulation defined byrare genetic abnormalities. This approach can be particularly effective in evaluating new drugs andmedical devices that are expected to have a significant therapeutic effect on diseases where the degreeof deterioration in health status is significant and can be quantified in a highly objective manner2.
（2）Use for expansion of indications and target populations
RWD may be used in cases where approval has been initially granted on the basis of a surrogate
endpoint, following which an approval application is submitted with a true endpoint to add an
11

indication. It is possible to conduct a single-arm trial using the approved drug, enrolling an indicatedpopulation for the purpose of evaluating the true endpoint, using RWD as an external control, as in (1)new drug application. Additionally, this approach may be used for expanding the indications ofapproved drugs, that is, adding indications for other diseases or for pediatric use. In this case, a trial
design may be assumed for clinical trials in an off-label population, and it may utilize RWD as anexternal control. Notably, while not within the scope of this document, RWD could be used inobservational studies, utilizing data from cases used off-label; however, in this case, caution should betaken regarding the possibility of a unique population (which may be difficult to extrapolate).
（3）Post-marketing efficacy and safety evaluations of drugs to which conditional early approval is
grantedThe conditional early approval system3 is applicable to drugs for serious diseases with a lack ofeffective therapy for which confirmatory trials are difficult to be conducted because only a small
number of patients are available or the study period is long. This system aims to put drugs with hightherapeutic effect for serious diseases into practice early by reviewing the efficacy and safety to someextent in clinical trials, other than a confirmatory trial (e.g., a single-arm trial to compare thresholdvalues), at the time when an approval application is submitted and by attaching conditions for approvalsuch as implementing a survey necessary for reevaluating the efficacy and safety after marketing. In
reevaluating the efficacy and safety required as the condition for approval, it may be possible toconduct a single-arm trial using approved drugs and utilizing RWD as an external control.
（4）Use for post-marketing efficacy and safety evaluations
Patients enrolled in clinical trials for regulatory submissions are a limited target population: Among
patient populations in whom the treatment be actually used, the study population is defined byinclusion and exclusion criteria, and populations in whom restrictions are placed on the use ofconcomitant therapies and the use of concomitant or rescue therapies that may affect the evaluation oftreatment efficacy. Therefore, it may be possible to use RWD as a platform for clinical trials for
conducting pragmatic clinical trials for evaluating efficacy and safety. In addition, an efficacy andsafety survey can be performed through a post-marketing database study using a medical information
database provided by a company dealing with medical information databases. Likewise, RWD may be
used for detecting, enhancing, or validating signals as a source of information for covering a wider
range of patients for the purpose of safety assessment of patients who will be used in daily practiceafter marketing. However, the extent of generalizability should be carefully considered, according to
the type of RWD used and its study design.
In the above-mentioned situations in which RWD is utilized for regulatory purposes, we focus mainly
on study designs in which RWD is used as part or all of the control group in a clinical trial, and outline
the statistical issues that need to be considered in Chapters 4 and 5. Also, in considering the use of
RWD for regulatory purposes, it is possible to adopt a policy of not only replacing elements of clinical12

data packages established in existing clinical trials with RWD, but also supplementing limited clinical
trial data with RWD from various perspectives. We will outline the use of RWD as complementary
information in Section 6.

Figure 2. Use of RWE in regulatory application1(Partially modified by the author)
Although not detailed in this report, RWD can be used for planning clinical trials and interpreting
results, mainly in the situations described in (1), (2), and (3). For instance, RWD can be used for
estimating a sample size when planning a clinical trial, for evaluating the usefulness of a drug or
medical device to be assessed with existing databases by performing a network meta-analysis, and forassessing the appropriateness of a surrogate endpoint (surrogacy). Furthermore, RWD may be usefulin predicting the details of intercurrent events and the frequency of missing data/mechanisms inlongitudinal data. In these applications, RWD is not used simply as a substitute of an RCT but forcompensating evidence when clinical trial data are limited. When considering the use of RWD for
regulatory purposes, RWD can be used not only for replacing the elements of a clinical data package
conducted in existing clinical trials but also for supplementing the limited clinical trial data with RWDfrom various perspectives. The use of RWD as supplemental information is explained in Section 6.
Furthermore, clinical trials utilizing RWD can be broadly classified into Exploratory Treatment
Effectiveness Studies and Hypothesis-Evaluating Treatment Effectiveness Studies4,5. This reportcovers the latter study types. The latter is defined as a hypothesis-testing study for evaluating
whether a therapeutic effect exists in a specific population, while the former is intended to gain
knowledge about a drug of interest through exploratory studies of therapeutic effect. Such
exploratory studies are important in studies using RWD. However, in exploratory studies, statisticalinferences may be performed using the same data as for hypotheses selected exploratory from thedata, especially for rare diseases. Thus, there are statistical considerations such as multiplicity. A
13statistical method used in these situations is a relatively new framework called Selective Inference.6,7
In addition to using this method, it will be necessary in the future to clarify the statistical issues insuch situations and to develop and evaluate statistical methodologies.
Finally, in addition to statistical issues, this report will focus on registry data and discuss issues inlinking it with other databases. Even if an appropriate trial design and statistical methods are employed,
data possessed by a registry database alone may not be adequate to test hypotheses regarding theefficacy and/or safety of a drug. In that case, linkage with other databases such as electronic medical
records, individual vital statistical data, and resident information enables collection of necessary andsufficient data. However, several issues need to be considered when linking the databases, such as the
granularity and format of the information in both databases. In this report, we introduce some examplesof registry data and other database linkages that have been implemented in other countries and NCs
and summarize the issues that need to be addressed in the linkages.
References
1. Franklin JM, Glynn RJ, Martin D, et al. (2019). Evaluating the Use of Nonrandomized Real-World
Data Analyses for Regulatory Decision Making. Clin Pharmacol Ther, 105(4): 867-877.
2. Simon R, Blumenthal GM, Rothenberg ML, et al. (2015). The role of nonrandomized trials in theevaluation of oncology drugs. Clinical pharmacology and therapeutics, 97(5), 502–507.
3. Pharmaceutical Evaluation Division, Pharmaceutical Safety and Environmental Health Bureau,
Ministry of Health, Labour and Welfare. (2017). “Implementation of the Conditional Early DrugApproval System. (PSEHB/PED Notification No. 1020-1 dated October 20, 2017).”
https://www.pmda.go.jp/files/000220723.pdf (Accessed 7/20, 2020)
4. Berger ML, Sox H, Willke RJ , et al . (2017). Good practices for real-world data studies oftreatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE
Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol
Drug Saf, 26(9): 1033-1039.
5. International Society for Pharmacoeconomics and Outcome Research. (2019). Improving
Transparency in Non-Interventional Research for Hypothesis Testing—WHY, WHAT, and
HOW: Considerations from The Real-World Evidence Transparency Initiative.
https://www.ispor.org/docs/default-source/strategic-initiatives/improving-transparency-in-noninterventional-research-for-hypothesis-testing_final.pdf. (Accessed 7/20, 2020)
6. Yoav B, Ruth H, Daniel Y. (2009). Selective inference in complex research. Phil Trans R Soc A,
367: 4255-4271.
7. Barber RF, Candes EJ. (2019). A knockoff filter for high-dimensional selective inference. Ann
Statist. 47(5): 2504-2537.

14

3.

Securing transparency for using RWD
Transparency is important for assessing trial quality. The transparency of clinical trials is usuallyachieved by documenting the protocol and statistical analysis plan (SAP), registering them on public
registration sites before trial initiation, and appropriately tracking the created and revised dates of these
documents. Especially for regulatory applications, defining and documenting the protocol, statisticalanalytical methods, and other relevant matters in advance and prespecifying is crucial for ensuringtransparency. In contrast, clinical studies using RWD present potential challenges for establishingprospective pre-specifications compared to traditional clinical trials. Thus, there are concerns
regarding “data dredging”, and securing the transparency of the trial and reliability of the results is notstraightforward. For such data dredging, the International Society for Pharmacoeconomics andOutcomes Research (ISPOR) and International Society for Pharmacoepidemiology (ISPE) jointly
formed a task force and reported an examination on good practices for clinical research using RWD1–
3. One of the several recommendations in this report is to pre-register the protocols and SAPs to public

registration websites. It may also be preferable to preregister the selected database(s) and the contents(data and persons in charge) of the feasibility survey. As a matter of course, even if transparency isadequately ensured, trials that are not appropriately designed and conducted will be of low quality;
hence, transparency by itself does not guarantee a high study quality. Nonetheless, transparency will
enable a deeper understanding of how the research was conducted and whether the results obtainedreflect the hypotheses and methods planned in advance and will allow third parties to appropriately
evaluate the quality and validity of the research. In addition, if different conclusions have been drawn
for similar research hypotheses, it becomes possible to examine the factors that may be the cause forthe differences. Thus, poor transparency makes an objective assessment of study quality difficult.
Moreover, RWD often are not collected for the primary purpose of research and are intended forsecondary use; they may be more likely to be revised with respect to the pre-defined statistical analysis
plan, compared to traditional clinical trials. For example, if the target is a rare disease with small
number of patients, the planned analysis adjusting for background characteristics may not beappropriate; thus, the original analysis plan may need to be amended. In that case, the rationale forand timing of the amendments are required to be documented as part of the transparent research
process. We consider that at the start of a study using RWD, it is important to clarify the study design,
including the study population and analysis methods, and to document any changes that occur during
the study period, so that concerns about "data dredging" can be addressed in a flexible manner2.
However, to examine the possibility of the use of RWD, it is essential to confirm the data contained in
the database (such as the number of cases of target diseases and background factors), and it is probably
difficult to specify all plans in advance. Therefore, for ensuring transparency, it is important to report
not only what was planned in advance at the beginning of the study but also the analysis methods
actually implemented3,4. Since the actual contents required to be included in the analysis plan overlap
15with the contents to be reported, please refer to Section 5.4 for the items to be included.
Additionally, in RCTs, data is fixed at the end of the trial, whereas in RWD, there is generally noend date for case enrollment or follow-up, and data is collected and updated continuously. Although
there may be a possibility that the data for the period used in the analysis may be updated in the future
because of queries about test items, it is important to ensure traceability by clarifying the cut-off date
for the data used in the analysis and preserving the relevant data sets as necessary.
By providing clinical trials on a registered site, other researchers may search for the existence ofclinical studies being conducted with hypotheses similar to their own research hypotheses; thisapproach has the benefit of reducing the waste of funds and time. Registering clinical research using
RWD may limit the above benefits because it utilizes pre-existing RWD. Nonetheless, presenting theresults of studies using databases with different data sources may contribute to examining thegeneralization. We believe that this approach would make it possible to visualize the enrollment of thesame patients in multiple clinical trials. For example, when conducting a clinical trial for rare diseases
or rare fractions, the data for the same patient in a registry database may be repeatedly used in multiple
clinical trials as control group data. Particularly, this might be a concern when constructing evidence
using meta-analyses, and the issues associated with using the same patients may need to be discussed
in the future. Although avoiding the repeated use of the same patients for inference is difficult andunlikely to be appropriate, it would be possible to visualize this actual situation by including such
issues in the way results are reported and presented and by registering clinical studies using RWD.
References
1.

Berger ML, Sox H, Willke RJ, et al . (2017). Good practices for real-world data studies oftreatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE
Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol
Drug Saf , 26(9): 1033-1039.

2.

International Society for Pharmacoeconomics and Outcome Research. (2019). Improving
Transparency in Non-Interventional Research for Hypothesis Testing—WHY, WHAT, and
HOW: Considerations from The Real-World Evidence Transparency Initiative.
https://www.ispor.org/docs/default-source/strategic-initiatives/improving-transparency-in-noninterventional-research-for-hypothesis-testing_final.pdf. (Accessed 7/20, 2020)

3.

Wang SV, Schneeweiss S, Berger ML, et al; joint ISPE-ISPOR Special Task Force on RealWorld Evidence in Health Care Decision Making. (2017). Reporting to Improve
Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0.
Pharmacoepidemiol Drug Saf, 26(9) : 1018-1032. Erratum in: Pharmacoepidemiol Drug Saf,
26(12), 1570.

4.

Duke-Margolis center. (2019). Understanding the need for non-interventional studies using
16

secondary data to generate Real-World evidence for regulatory decision making, anddemonstrating their credibility. https://healthpolicy.duke.edu/sites/default/files/2020-08/NonInterventional%20Study%20Credibility.pdf. (Accessed 7/20, 2020)

17

4.

Single-arm clinical trial using RWD as an external control

4.1. Assumed situations
In a clinical trial comparing the intervention treatment with the control treatment, it is desirable todefine the internal controls with randomization and blinding after appropriately identifying thetreatment effect to be estimated. This is because comparability is maintained by minimizing theimbalance of known and unknown prognostic factors and the impact of selection bias such that any
treatment is conducted in a similar way other than the test treatment or the control treatment. On theother hand, it is difficult to control the bias due to these factors without establishing internal controls.
However, a single-arm trial including external controls can be considered when it is difficult to set
internal controls and when the advantages of the treatment comparison are deemed to overweigh thedisadvantages despite minimizing potential biases. In this section, the following situations in particular
are assumed:
・The quality and quantity of clinical trials that have been conducted before are insufficient.
・The population of patients with a serious disease is targeted.
・The disease is rare.
・The test treatment is expected to have substantial treatment effects.
・There is no promising treatment option other than the test treatment.
In the presence of such factors, it is difficult, or considered inappropriate from an ethical viewpoint,etc., to conduct a study using the data of previous clinical trials as external controls. A randomized
clinical trial using a hybrid control and a registry-based RCT (R-RCT) are described in Section 6.3. In
such cases, a trial design using RWD as the external control will be given priority consideration.
The common biases expected in conducting a clinical trial include confounding variables, selection
bias, and information bias. Even in clinical trials using RWD, it is necessary to reduce these biases to
the greatest possible extent and to consider the presence of biases that cannot be eliminated in thediscussion of the results. There are multiple points to consider in the design and analysis to estimate
the efficacy of drugs or medical devices based on the data obtained from a single-arm trial using RWD
as an external control while reducing biases to the fullest possible extent. In the report of the Duke
Margolis Center1 and the researches by Franklin2,3, the points to consider in constructing the evidence
for regulatory application by using RWD from non-RCTs are summarized. It has also been pointed
out that, when deducing a causal effect related to the efficacy and safety of a treatment using
observational study data, it is possible to examine the points to consider related to the design andanalysis beforehand for the purpose of avoiding any pitfalls that can lead to bias, in view of statisticallyemulating an RCT (target trial) that allows for the evaluation of the hypothesis of interest4–6. In thefollowing sections, the points to consider in a single-arm trial using RWD as external controls are
outlined in reference to these issues.

184.2. Points to consider for implementing the design and analysis(1) Identification of the scientific question of interest and estimands.
Prior to conducting an RCT, it is important to appropriately define the treatment effect to be
estimated in the relevant trial based on the scientific question of interest, and this also applies to
conducting single-arm trials using RWD as the external control. In the guidelines of the ICH E9(R1) document, a concept termed the “estimand” has been proposed as a basis for discussing
different treatment effects7, and it encompasses elements such as the treatment condition, thepopulation, the variable, the intercurrent events, and a population-level summary for the variable.
Among these, the intercurrent events refer to the events occurring after the beginning of treatmentthat include the use of additional treatment or alternative treatment and treatment discontinuation,and they prevent the observation of the variables or the effective interpretation of the variables.
Clearly identifying the estimands allows us to discuss which data should be collected after anintercurrent event, that is, the missing data to be addressed in case there is a failure in collecting
them. In particular, in the case of RWD, details of the control treatment are not specified in the
protocol, and intercurrent events, such as treatment switching and treatment discontinuation, may
be more likely to occur compared with conventional RCTs. For this reason, carefully discussing
the appropriate estimand in advance and designing the analysis plan to include data handling are
required.
(2) Biases


Selection of an appropriate control group and target patients and selection bias

When using RWD, the control treatment may be defined as the absence of treatment because it isdifficult to designate a placebo treatment for the control group. In real-world clinical settings, however,
the absence of treatment may result from a situation in which the severity of the disease is high suchthat it is very unlikely that it will be cured by any treatment or the patient cannot be treated due toother health-related problems, etc. In such cases, it may be difficult to sufficiently minimize biases
based on the confounding variables measured at baseline1. Depending on the research hypothesis, thebias can be reduced by using an active drug or standard treatment as the control in many cases2.
Although it goes without saying, RWD need to include the data to select the subjects with the same
eligibility criteria as those for the single-arm trial. For example, if one of the criteria for the singlearm trial is “no steroid treatment in the past 2 years,” the RWD also should include these relevant data
to evaluate the same criterion. In addition, it is difficult to appropriately compare two populations if
they differ in the definition or in the diagnostic method for the disease and its severity that specify the
population. Thus, in addition to collecting appropriate data that are necessary for specifying the
population, the following two points should also be well-considered in selecting the control group andselecting the subjects.
19

The first point is that, when a standard therapeutic drug is defined as the control treatment, somebias may be introduced if the control group includes patients who have already been administered thedrug. When the patients who have been administered the standard therapeutic drug for a long duration
are included in the control group, they may have experienced the event prior to being included in the
study, and they might have tolerated adverse reactions occurring immediately after the treatment withthe drug was initiated. Furthermore, bias may be introduced if the observation of the endpoints(baseline values) and covariates observed at a timepoint different from the beginning of the treatmentin the control and interventional groups of the RCT. For such bias, which is called “prevalent user bias”
in the epidemiological field, using a new user design that limits the subjects to new users or defining
the period of non-use of the relevant drug in the eligibility criteria has been proposed2,8,9.
The second point is that selection of the subjects may introduce an immortal time bias. Assumingthat the intervention treatment is started immediately after the enrollment period in the single-arm trial,
while the control treatment is started at a certain period after the enrollment of patients in the registry,a bias may be introduced if only the patients who actually started the control treatment are included in
the external control population because these are patients who “survived” without experiencing theevent before the beginning of treatment10,11. Although it can be considered that no immortal time bias
exists in RCTs in which randomized patients constitute the study population and the time ofrandomization is defined as the starting timepoint, attention is required when using RWD, becausesome immortal time bias can be introduced, depending on the definition of the treatment anddetermination of the starting timepoint. Some analytical methods have been proposed to mitigate theimmortal time bias, such as landmark analysis, handling the treatment as a time-dependent variable,and performing a statistical adjustment by the inverse probability censoring weighted (IPCW) method
5,6,12.

The abovementioned biases are two types of selection biases that are generated by conditioning on
colliders, which are factors affected by both the treatment and the outcome (or their causal factors)12–18. However, the immortal time bias can also be classified as an information bias in that it consists ofthe misclassification of the information time11. To avoid selection bias in selecting patients, it isnecessary to be careful to not select the subjects based on the variables measured after baseline. For
example, if the endpoint is the test value at 6 months, it is not appropriate to limit the inclusion ofsubjects to those who could be followed up until 6 months because this generally introduces selectionbias.


Measurement items to be collected and information bias

RWD may contain some form of information bias related to the patient’s disease, exposure totreatments including the control treatment, and outcome. The measurement error associated withdiscrete variables is called “classification error (or misclassification),” and the degree and direction of
the bias affecting the estimation of the treatment effect depend on the distribution of such error. For20

example, when the presence/absence of treatment has been measured with error, a bias for averagely
underestimating the treatment effect is generated if certain conditions are met (e.g., the distribution oferror is independent from the outcome, and the treatment variable is binary)19,20. On the other hand, it
is known that, in the case of systematic classification error related to the outcome or risk factors, thebias can lead to either an overestimation or underestimation. When using RWD, it is necessary toadjust for confounding variables. In this case, attention is needed because it has been reported that
adjusting for confounding variables with error may paradoxically increase bias when the confounding
variable has an effect on the outcome in different directions between the intervention and control21.
Different variables may be associated with a high risk of classification error, depending on the type of
RWD used as the external control. For example, when a disease registry is used as an external control,
the risk of classification error regarding the diseases in patients, which is a key eligibility criterion forthe control group, is considered to be low compared with other RWD.
In addition, to estimate the treatment effect with good accuracy and without bias, it is important thatthe outcome has been measured precisely and with high accuracy. For example, if the endpoint is abinary variable consisting of the presence or absence of an event, it is important to define the event
precisely and comprehensively. If the event cannot be defined directly from the information recordedin the RWD, the outcome is defined using other data, etc., and it consequently becomes important toassess its validity. For the method used for validating the definition of the outcome, refer to the “Basicprinciples in conducting a validation study of outcome definitions used for post-marketing databasestudy,” issued on July 31, 202022.
In addition, these discussions should be considered together with the reliability of the data.
Regarding the concept of quality management system introduced following the revision of ICH-E6
guidelines, it is necessary to discuss how to define and guarantee a “regulatory grade” when using
RWD for regulatory purposes, which is not mentioned in detail here because it is out of the scope of
the present report. This requires ideas of quality control that match the nature of RWD, such as
quantitatively discussing quality control by considering it from the viewpoint of error control indecision-making and simplifying and clarifying the establishment and evaluation of the quality target
by focusing the quality assessment on the data processing method instead of the data. The
establishment, operation, and management of patient registry data with guaranteed reliability required
for the utilization of data for regulatory purposes, including this “regulatory grade,” are discussed inanother assigned task of this project, in the “Operation and management for ensuring the datareliability in utilization of patient registry data as approval application documents for pharmaceuticalproducts, etc.”
Moreover, one of the points to consider when using RWD as an external control for a single-arm
trial is that the measurement schedule for the endpoints in clinical trials, in general, largely differsfrom that in routine clinical practice. In real-world clinical settings, data are observed according to the21visit schedule, which is determined as a function of the patient’s condition and clinical course, etc. For
example, if loss-to-follow-up of the patients and the presence/absence of missing data in the RWD
depend on the treatment effects and adverse reactions, making a comparison using only the observed
data does not produce appropriate results. Therefore, it is necessary to perform evaluations consideringsuch differences. In addition, the treatment period for the control group may be different from that for
the intervention group. If the observation method presents such differences in the observation period
or evaluator, as well as those due to the purpose (e.g., real-world clinical settings versus clinicalresearch settings), the results that depend on them are expected to possess bias. Moreover, in relation
to the selection of the patient population, differences in the definitions of the disease and its severity,
diagnosis method, etc. introduce biases in the specified population and in the results.
Confounding bias

In a single-arm trial using RWD as an external control, it is necessary to adjust for confounding
variables in the trial design and analysis because randomization is not performed and there is thus nocomparability between the intervention and control groups. For this purpose, it is necessary that theconfounding variables that are risk factors and related to treatment have been appropriately collectedin both the single-arm trial and in the RWD. It is useful to use directed acyclic graphs for determining
whether the variables should be adjusted as confounders23–25. However, it should be noted that
unknown or unmeasured confounding variables cannot be controlled due to the lack of performing
randomization. For the adjustment method, refer to Section 4.3.
(3) Data sources used as an external control
There are various possible candidate data sources to constitute the external control. The selectionof which data source(s) to use and the decision on whether to combine multiple data sources largely
depend on the compatibility with the trial objective in the clinical trial and the objective of the overallclinical development and whether the same variables are evaluated as those in the clinical trial areobserved with equivalent quality based on that objective. Whether the biases expected from thecharacteristics of the data source can be identified beforehand is also an important element guiding
the decision. As mentioned above, to constitute the external control, it is necessary that the datarequired for selecting the patients appropriately and adjusting for the confounding variables is obtained,
and it should be carefully examined whether the final data obtained can sustain the achievement of thestudy objective. Even when the study population appears, at first glance, to be appropriately selected,
differences in the enrolled patients between the different study sites and the resulting sample size ofthe external control population may be insufficient for making conclusions. If it is expected that
deducing a conclusion from a single data source will be difficult, it may be beneficial to link multiple
databases. The experience of linking databases at the NC and the points to consider, etc. are describedin Chapter 7.
22

Among the trials that have been submitted for approval using external control data thus far, there
are multiple cases in which the regulatory agencies did not approve the results of the comparison withan external control. These cases also demonstrate that the examination of these points during theplanning of the trial may largely influence the interpretation of the results and the conclusion. The
relevant cases are described below.

Case 1 (related to immortal time bias):
For XPOVIO® (non-proprietary name: selinexor), which is a therapeutic drug for recurrent or
refractory multiple myeloma, the US FDA concluded that the external control data submitted for in
the application, the overall survival obtained from the data from EHR were insufficient from thefollowing points26: [1] correspondence with the prespecified analysis cannot be guaranteed; [2] largely
different eligibility criteria caused selection bias, classification error, and confounding bias; [3] there
was the presence of immortal time bias due to a problematic definition of the starting point formeasuring the overall survival time; and [4] while there is no comparability due to missing data inimportant clinical characteristics and failure to meet the assumptions required for the analysis using
the propensity score, the differences between the populations were not resolved by the analysis.
Point [1] refers to the content mentioned in Section 3, and points [2], [3], and [4] are related to thebiases described in (2). This case suggests that the results of the comparison with the external control
cannot be appropriately interpreted due to these problems.
Case 2 (other biases related to the data source):
For the application of BALVERSA® (non-proprietary name: erdafitinib), which has been approved asa therapeutic drug for locally advanced or metastatic urothelial carcinoma with fibroblast growth factor
receptor mutations, the human epidermal growth factor receptor 2 data and genetic panel data were
submitted as external control data in the single-arm trial. The US FDA considered that these data wereinsufficient as an external control from the following points27: [1] important confounders have missing
data; [2] the eligibility criteria are different between the treatment groups; [3] the characteristics of theparticipating centers differed between the treatment groups, suggesting that the characteristics of thepatient populations were also different; and [4] the number of patients (with gene mutations, as
indicated in the RWD) is not sufficient for concluding its efficacy.
Points [1] and [2], which were in common to case 1, should always be examined at the time of theplanning of the trial because they are considered to be particularly likely to occur when using RWD
as an external control. In contrast, points [3] and [4] are related to the selection of data sources
described in (3) and should be carefully assessed at the time of the planning of the trial because these
problems may not appear beforehand.
23

References
1. Duke Margolis Center for Health Policy. (2019). Understanding the need for non-interventional
studies using secondary data to generate real-world evidence for regulatory decision making, anddemonstrating their credibility.
https://healthpolicy.duke.edu/sites/default/files/2020-08/Non-Interventional Study Credibility.pdf(Accessed February 09, 2021).
2. Franklin, J.M., Schneeweiss, S. (2017). When and how can real world data analyses substitute for
randomized controlled trials? Clin Pharmacol Ther, 102:924-933.
3. Franklin JM, Glynn RJ, Martin D, et al. (2019). Evaluating the use of nonrandomized real-world
data analyses for regulatory decision making. Clin Pharmacol Ther, 105(4):867-877.
4. Hernán MA, Robins JM. (2016). Using big data to emulate a target trial when a randomized trial
is not available. Am J Epidemiol, 183(8):758-764.
5. Maringe C, Benitez Majano S, Exarchakou A, et al. (2020). Reflections on modern methods: trial
emulation in the presence of immortal-time bias. Assessing the benefit of major surgery for elderlylung cancer patients using observational data. Int J Epidemiol, 49(5):1719-1729.
6. Hernán MA, Alonso A, Logan R, Grodstein F, et al. (2008). Observational studies analyzed likerandomized experiments: an application to postmenopausal hormone therapy and coronary heartdisease. Epidemiol, 19:766-779.
7. ICH-E9 (R1): Addendum on estimands and sensitivity analysis in clinical trials to the guideline
on statistical principles for clinical trials (Step 4). https://database.ich.org/sites/default/files/E9R1_Step4_Guideline_2019_1203.pdf (Accessed February 09, 2021).
8. Ray WA. (2003). Evaluating medication effects outside of clinical trials: new-user designs. Am J
Epidemiol, 158(9):915-920.
9. Yoshida K, Solomon DH, Kim SC. (2015). Active-comparator design and new-user design in
observational studies. Nat Rev Rheumatol, 11(7):437-441.
10. Suissa S. (2007). Immortal time bias in observational studies of drug effects. Pharmacoepidemiol
Drug Saf, 16(3):241-249.
11. Suissa S. (2008). Immortal time bias in pharmaco-epidemiology. Am J Epidemiol, 167(4):492499.
12. Hernán MA, Sauer BC, Hernández-Díaz S, et al. (2016). Specifying a target trial prevents
immortal time bias and other self-inflicted injuries in observational analyses. J Clin Epidemiol,
79:70-75.
13. Hernán MA, Hernández-Díaz S, Robins JM. (2014). A structural approach to selection bias.
Epidemiol, 15(5):615-625.
14. Cole SR, Platt RW, Schisterman EF, et al. (2010). Illustrating bias due to conditioning on a collider.
24

Int J Epidemiol, 39(2):417-420.
15. Hernán MA, Alonso A, Logan R, et al. (2008). Observational studies analyzed like randomized
experiments: an application to postmenopausal hormone therapy and coronary heart disease.
Epidemiol, 19(6),766-779.
16. Danaei G, Tavakkoli M, Hernán MA. (2012). Bias in observational studies of prevalent users:
lessons for comparative effectiveness research from a meta-analysis of statins. Am J Epidemiol,
175(4), 250-262.
17. Greenland S. (2003) Quantifying biases in causal models: classical confounding vs colliderstratification bias. Epidemiol, 14:300-306.
18. Mansournia MA, Nazemipour M, Etminan M. (2021). Causal diagrams for immortal time bias.
Int J Epidemiol, Invited Editorial, in press.
19. Dosemeci M, Wacholder S, Lubin JH. (1990). Does nondifferential misclassification of exposure
always bias a true effect toward the null value? Am J Epidemiol, 132(4):746-748.
20. Jurek AM, Greenland S, Maldonado G, et al. (2005). Proper interpretation of non-differential
misclassification effects: expectations vs observations. Int J Epidemiol, 34(3):680-687.
21. Ogburn EL, VanderWeele TJ. (2012). On the nondifferential misclassification of a binary
confounder. Epidemiol, 23(3):433-439.
22. Basic concepts on the implementation of validation of outcome definitions used in post-marketing
database surveys [Notification No. 0731001 issued by the Director of the Office of Regulatory
Science, Pharmaceuticals and Medical Devices Agency (PMDA), dated July 31, 2020;
Notification No. 0731001-1 issued by the Director of the Center for Product Evaluation, PMDA;
Notification issued by the Pharmaceutical Evaluation Division, Pharmaceutical Safety andEnvironmental Health Bureau, Ministry of Health, Labor and Welfare, and the Pharmaceutical
Safety Division, Pharmaceutical Safety and Environmental Health Bureau, Ministry of Health,
Labor and Welfare].
https://www.pref.fukuoka.lg.jp/uploaded/attachment/118817.pdf (Accessed July 20, 2020).
23. Greenland S, Pearl J, Robins JM. (1999). Causal diagrams for epidemiologic research. Epidemiol,
10(1):37-48.
24. Hernán MA, Hernández-Díaz S, Werler MM, et al. (2002). Causal knowledge as a prerequisite forconfounding evaluation: an application to birth defects epidemiology. Am J Epidemiol,155(2):176-184.
25. Schisterman EF, Cole SR, Platt RW. (2009). Overadjustment bias and unnecessary adjustment in
epidemiologic studies. Epidemiology. 20(4):488-495.
26. United States Food and Drug Administration. NDA/BLA multi-disciplinary review andevaluation (NDA 212306) XPOVIO (selinexor).
https://www.accessdata.fda.gov/drugsatfda_docs/nda/2019/212306Orig1s000MultidisciplineR.p
25

df (Accessed October 02, 2021).
27. United States Food and Drug Administration. NDA/BLA multi-disciplinary review andevaluation (NDA [NME] 212018) BALVERSA (erdafitinib).
https://www.accessdata.fda.gov/drugsatfda_docs/nda/2019/212018Orig1s000MultidisciplineR.p
df (Accessed October 02, 2021).

4.3. Issues to consider regarding statistical methods
4.3.1.

Discussion of statistical methods for evaluating efficacy and safety using RWD

A considerable portion of Section “4.2. Points to consider in implementing the design and analysis”
is devoted to “bias”; therefore, close attention needs to be focused on potential biases deriving fromvarious causes when estimating the efficacy of products including pharmaceuticals and medical
devices in a clinical trial using RWD as external controls. Several factors may lead to biases. Selection
biases, information biases, and confounding biases are mainly described in 4.2-(2). Selection bias andconfounding bias, which frequently occur when conducting a clinical trial using RWD, can beaddressed by statistical techniques to a certain extent, as explained in the next section, “4.3.2
Representative biases and how to deal with them.” A separated section is dedicated to the immortal
time bias, because this bias can be induced in different structures according to the purpose of analysis(effect to be determined/estimand) and the analysis method; it can be interpreted as a selection bias
depending on the analytical method [as discussed in the paragraph on selection bias in 4.2-(2)], while
it can be classified into information biases with respect to misclassification of the information time1,2.
Although measurement bias is not discussed in the next section (4.3.2), the impact of themisclassification (classification error) of explanatory variables and outcome can be assessed using
sensitivity analysis, which is introduced in Section 4.3.4.
Selection bias is typically induced by conditioning on a variable that represents a common outcomeof the two variables (collider) or that is directly or indirectly influenced by the collider. The bias
appears when these two variables (or other variables that are influenced by them) show an associationthat differs from the association without conditioning on a collider 3. A “selection bias” may occur
even without conditioning on any collider4. However, in this report, we discuss the modification of theestimand described in the previous section by the change in the population, the efficacy in which isestimated by conditioning. The selection bias induced by conditioning on a collider may occur
specifically in cases such as loss to follow-up, missing values (although loss to follow-up can be
categorized as missing, other reasons are also included here), subject’s self-choice to participate in theinvestigation, and choice of treatment assignment by the treating doctor. Such selection biases canoccur in both RCTs and observational studies, including studies using RWD. In particular, certain
concomitant drugs irrelevant to the target treatment could have beenprescribed or perhaps the target
26

treatment could havebeen suspended in RWD, which includes data from clinical practice. Otherwise,note that selection biases can occur when using RWD for external controls. If a treatment is initiatedor suspended depending on the subject’s clinical status, informative censoring is considered to haveoccurred. Since informative censoring causes a selection bias, appropriate adjustment is required
according to the effect to be estimated (estimand).
Next, we consider confounding biases. It cannot be excluded that a bias may occur in the results of
estimation of indices for efficacy and safety of the treatment due to the impact of confounders, unlessa RCT is conducted. Thus, since there is no comparability between treatment groups, it is necessary
to adjust the confounders. It may generally be possible to address these biases to a certain extent bytraditional methods, such as matching and stratified analysis, using confounders. However, thesemethods have limited ability to deal with the bias when the number of confounders is too large
compared to the number of subjects or events. Although regression methods allow the considerationof a large number of confounders with limited data, they are not perfect techniques, in terms ofsensitivity to the deviation from the assumptions of the model, a sparse data bias in the data containing
a stratum with a small number of events, and an assessment for comparability after adjustment. In
addition, care must be taken when simply interpreting regression coefficients as an “efficacy index
after adjustment of confounders”. For example, the regression coefficients of a logistic regression
model can be interpreted as the estimates of the “log odds ratio conditioned on confounders.” Even
when the log odds ratio can be assumed to be shared between confounders, it is not strictly possible
to interpret it as “a marginal odds ratio in the overall population.” By defining estimands (efficacy
indices to be estimated), it may be possible to determine the results of regression models using
standardization (g-computation) or a doubly robust method in addition to the regression coefficients
In recent years, various techniques have been proposed that are based on assumptions different from
regression models or providing more stable statistical features for certain data, and they have been
applied in clinical trials aimed at regulatory approval in some cases. Although it is possible to adjust
the impact of confounders by using statistical analytical methods detailed later, they do not necessarily
eliminate the impact of confounders (residual confounding) completely.
Finally, we treat immortal time biases. Immortal time is a period in which no outcome occurs
during the observation or follow-up period, because of a definition of treatment exposure.
Consequently, immortal biases can occur as a result of the immortal time 5. Consider comparing
outcomes in terms of mortality due to presence/absence of surgical interventions as an example. It
generally takes time to progress from diagnosis to surgery. While survival until surgery is guaranteedin the surgery group, deaths may occur in this interval of time in the non-surgery group. The immortal
time refers to this interval from diagnosis to surgery in the surgery group. In this case, if the mortality
rate is simply compared between the two groups, a bias occurs in favor of the surgery group becauseof the presence of the immortal time only in the surgery group. When estimating the effect of an27

intervention using RWD such as registry data, inadvertently setting the follow-up period may cause
similar problems, which need to be addressed. For instance, consider the following case: in single-arm
trials, patients tend to be registered shortly after diagnosis and administered the experimental drug
immediately. Meanwhile, in the control group, according to the registry data, patients are incorporated
long after diagnosis. Consider disease onset as the initiation of observation. If the treatment and control
groups are compared in a straightforward manner, a bias occurs and the effect in the control group isoverestimated. The reasons are that (i) the period until the initiation of a treatment is included as theimmortal time, because of which the event rates are underestimated in both groups, and (ii) the
immortal time becomes heterogeneously distributed between the groups.
Certain points listed in 4.2 that are not discussed in this section can be addressed to a certain extent
using statistical analysis; nevertheless, many problems remain because no effective countermeasurehas been developed that cannot, in principle, be dealt with by statistical methodology. An elaborate
study design can be used, if it addresses these problems. However, in single-arm clinical trials using
RWD as external controls, which is the focus of this report, it is impossible to expect a perfect method,
because existing RWD are used in most cases, or it is difficult to change the collected data and/or
method for obtaining data in the middle of the study. In such cases, it is important to examine thevalidity of the study and evaluate whether there are intractable problems.
4.3.2.

Representative biases and how to deal with themIn this section, statistical techniques are introduced for addressing representative biases: selection
bias, confounding bias, and immortal time bias.

Figure 3: An example of a causal relation graph representing the selection bias caused by loss tofollow-up
The analysis limited to cases without censoring corresponds to conditioning on censoring. As shownin Figure (b), in which a conditioning variable is surrounded by a square frame, a path connecting
28

between X and Z is opened by conditioning on the censoring, which is a common outcome between
X and Z. A backdoor path from X to Y is open, and an association between X and Y, which should not
originally exist, is observed in consequence. This illustrates an example of selection bias. X, Y, Z, andU may represent treatment intervention, death, pain associated with disease progression, and grade ofdisease progression, respectively. Accordingly, “X → censoring”, “Z → censoring”, and “U → Y”
correspond to dropouts due to side effects of the treatment, dropout due to pain caused by diseaseprogression, and increase in the risk of death due to disease progression, respectively.
We explain selection bias considering as an example a case of loss to follow-up, which frequently
occurs in clinical trials. Figure 1(a) shows an example of causal relation graphs representing theoccurrence of loss to follow-up owing to censoring. If censoring occurs in a different manner between
treatment groups and if censoring is caused by a variable related to the outcome, the censoring variable
meets the criteria for being a collider. Therefore, selection bias is caused by conditioning on the
censoring variables (Figure 1(b)). A methods for dealing with the bias caused by such “informative
censoring” is the IPCW method, which is performed by (i) estimating the observation probability (theprobability of non-censoring) conditioned on the variables for each data, and then (ii) weighting thenon-censored patients by the inverse of the probability6. For example, in Figure 1, the IPCW method
using the observation probability conditioned on X and Z allows to estimate without bias the treatment
effects that would be observed in the absence of censoring. In general, since censoring can occur
throughout the follow-up period, and there are a large number of variables to be conditioned on theobservation probability, it is necessary to model the observation probability for each timepoint using
the patient profile observed for each timepoint. The observation probability estimated for each
timepoint is multiplied throughout the timepoints and weighted by its inverse for constructing the
IPCW estimator. The IPCW method can be considered as a method for adjusting confounding after
the beginning of treatment (time-dependent confounding) by deeming “elimination of censoring” as a
type of intervention.
The countermeasures for selection bias due to missing values include weighting by the inverse of
the observation probability and multiple imputation7,8. In the former method, which uses a similar
concept as the IPCW method, the observation probability is estimated using the factors expected toaffect missing, such as covariates representing the characteristics of the subjects, and its inverse isused as a weight for each of them. In the latter method, after modeling the missing values using othervariables, multiple data sets without missing values are created by complementing them using theestimates from the models, and the results separately analyzed are merged. To obtain results withoutbias using these methods, it is necessary to assume that the mechanism of generating missing values
depends on the observed data, but it is independent of the values themselves (missing at random:MAR]. However, these methods cannot eliminate the bias, if the occurrence of missing values is29

affected by the values themselves (missing not at random: MNAR).
Next, we explain approaches for addressing confounding. Confounding refers to incomparability
between groups. It is often explained as a state in which the estimated causal effect is biased becauseof the presence of a variable that constitutes a shared cause for the two variables (confounder, in the
narrow sense). Confounding may be possibly addressed to some extent using traditional methods suchas matching on confounders and use of stratified analysis. However, when performing a single-arm
clinical trial using RWD as external controls for evaluating efficacy and safety, it is necessary to adjust
many confounders for the number of patients and events, and it is not realistic to employ matchingand stratified analysis. Regression analysis incorporating confounders into the explanatory variables
is also rational. However, additional techniques need to be considered in cases containing strata withfew events or requiring assessment of comparability.
In such situations, methods based on a propensity score are commonly employed and applied inmany cases. Specifically, these methods are considered to be useful in the field of rare diseases with alimited number of patients and events, and they have been applied for evaluation of efficacy and safety
in clinical trials aimed at regulatory approval. External data, such as disease registries, have alreadybeen used in clinical trials in some (although few) cases. The guidance by the Hayashi group9 has adetailed discussion regarding methods for adjustment of confounding when using registry data as
external controls.
Propensity score-based methods are implemented by two-step estimation. The first step iscalculating the propensity score, which is the conditional probability of receiving the test treatment or
not after stratification of confounding variables, for each subject. The next step is estimating thetreatment effect with balancing of the confounding variables using the calculated propensity scores10.
A common problem encountered in actual analysis using this procedure is the handling of missing datafor calculating the propensity scores. Nevertheless, there is no generally applicable method for the
missing data problem, and the countermeasure should be examined according to the mechanism ofoccurrence of missingness. For this purpose, methods such as complete-case analysis, missingness
pattern approach (pattern-mixture model-based approach), missing indicator method, and multiple
imputation method have been proposed. For estimating the effect using the propensity score, thefollowing approaches are mainly used: matching on propensity score11,12, stratified analysis13,14,
application of regression models with the propensity score as a covariate15, and the inverse probability
weighted (IPW) estimator16,17. All these approaches generally employ regression models (e.g., logistic
model) for estimating propensity scores, and the model should be correctly identified for avoiding any
bias in the estimator. Any of these approaches can be extended to a doubly robust estimator, whichbecomes a consistent estimator if either the model calculating propensity scores or the regression
model of the outcome variable conditioned on the same confounding variable is correctly specified18,19.
The same causal effects are not always estimated by these approaches. For example, in the case of30

matching on propensity score, since the patients who failed to form a pair on the matching (off-support
cases) are excluded from subsequent analysis, the causal effect will be estimated in the post-matchingpopulation and not in the pre-matching population. This is not necessarily undesirable; on the contrary,
it may be rather desirable for overlapping propensity scores, that is, we are interested in the causal
effect in a population where a clinical choice regarding the intervention may be involved. Thus,
selection of an appropriate method depends on the definition of the causal effect of interest. For thisreason, it is important to clearly define the causal effect of interest. For this discussion, an article by
Kurth et al.20 is informative.
As an advantage of the propensity score method, for which the analysis is performed with two steps,
it is possible to reduce confirmation biases caused by tweaking the analysis based on the definitive
estimation results, because estimation of the propensity score and outcome comparison can be
assigned to different analysts. To exclude the possibility that the result of repeated ex-post analysis bychanging the propensity score model is reported, it is important to specify a whole analysis plan before
collecting the outcome data21–24. It could be unrealistic to use this scheme depending on an individual
situation. In such cases, a possible scheme is fixing the propensity scores before starting the analysisof the outcome and reporting the calculation method.
In addition, the propensity score method requires measurement of the confounder to be adjusted.
Unknown/unmeasured confounders are almost always present in real situations. In the presence of
unknown/unmeasured confounders, rather than immediately concluding that the causal effect has notbeen estimated without biases, it would be more practical and productive to evaluate and discuss how
impactful unknown/unmeasured confounders affect the conclusion by using the sensitivity analysis
described in Section 4.3.4.
In the following section, we discuss methods that are effective even for unknown/unmeasured
confounders (except for RCT). These methods include the instrumental variables methods andregression discontinuity design. The instrumental variables method is explained in detail in theguidance by the Hayashi group9. These approaches are highly advantageous in terms of costs andfeasibility, because they allow estimation of treatment effects even without measurement ofconfounding variables. However, although they do not need complete measurement for theconfounders, they require different assumptions. Moreover, these methods are not available at any
time, but each can be applied to a limited range of situations. A difficulty faced in the instrumental
variables method is the determination of appropriate “instrumental variables”; additionally, evaluationof the appropriateness is difficult, because the appropriateness of instrumental variables itself contains
an assumption that cannot be directly verified from data. Furthermore, when only a weak relationshipexists between instrumental variables and the treatment, and the unmeasured confounding variables
are not so many, the mean squared error of the estimated instrumental variables is not reduced
compared to the least squares estimator without adjustment of unmeasured confounding except when31the sample size is quite large, because of small variance despite the presence of bias23. Considering
the above points, it may be even more difficult to apply this method when conducting an RCT is notrealistic because of the sample size, such as in the case of rare diseases, which is the focus of thisreport. Even the regression discontinuity design cannot be applied if there is no cutoff criterion todetermine whether the intervention is to be administered.
Finally, we explain the immortal time bias. The immortal time bias is a bias caused by the period inwhich no outcome occurs during the observation period or the follow-up period, because of a time-lag
between an initiation of the period and treatment exposure. An illustration of the immortal time bias
is shown in Figure 2. Proposed methods for addressing immortal time bias25,26 include the exclusion
method, conditional landmark analysis, extended Cox model with time-varying covariates (MantelByar method), and IPW method. To make a conclusion first, the exclusion method should not be used.
Each of the other methods except for the exclusion method utilizes different assumptions and therefore
exhibits different advantage and disadvantages. The following paragraphs briefly explain each of thesemethods and presents a brief summary of articles comparing these different methods.
The exclusion method is an analysis method in which the immortal time is simply excluded fromthe target period of analysis. Although it is the easiest method, it has been reported to overestimate theeffect compared to the landmark method and the extended Cox model28. In conditional landmark
analysis, an arbitrary timepoint during the follow-up (or observation) period is set as the landmark
timepoint, and only the subjects who have survived till the landmark are incorporated. Specifically,
the subjects who survived till the landmark timepoint extracted from the target population are includedin the analysis set, and the treatment groups are classified on the basis of the treatment status at thelandmark timepoint. Even if the state changes after the landmark time point, the classification of thetreatment group does not change. Subsequently, the occurrence of the outcome/event and the time toits occurrence are compared by defining the landmark timepoint as the starting timepoint. Although
the conditional landmark analysis is relatively easy to perform, it has some drawbacks, such as
information regarding the outcome occurring before the landmark timepoint is not used, and thelandmark timepoint is arbitrarily set. Regarding the arbitrariness of setting of the landmark timepoint,
it should be possible to evaluate its effect by sensitivity analysis. The extended Cox model with timevarying covariates addresses the immortal time bias by treating the exposure or intervention status asa time-dependent covariate. Data for subjects whose exposure/intervention status changed during theobservation period will be divided into multiple records based on their status. The records from theperiod when there was no exposure/intervention will be used as the non-exposure (non-intervention)group, and the records from the period when there was exposure/intervention will be used as the
exposure (intervention) group for analysis. Details on analyzing the Extended Cox model can be foundin the book by Therneau et al.29. Similar to the IPCW method for censoring and the IPW estimators
using the propensity score, the IPW method for immortal time bias constructs a model predicting the32

probability of receiving exposure/intervention during the observation period based on the informationin the database including demographic characteristics and clinical information of the subjects.
Subsequently, using the predicting model to calculate the probability that each subject receives
exposure/intervention during the observation period at each timepoint. By cumulatively multiplying
the probability of receiving the exposure/intervention at each time point and weighting it by its
reciprocal, we estimate the difference in the occurrence of the outcome. Similar to the propensity score
methods, attention should be paid to the following point: provided the information about the subject(e.g., demographic characteristics and clinical information) used in the prediction model at a given
timepoint, it is required that whether the subject shifts to the exposure (intervention) group at that
timepoint does not depend on other outcome risk factors. It is also possible to deal with immortal timebias using the IPCW method introduced in section 4.3.2. This method imitates the target trial (seesection 4.1), treating it as a censoring once the patient deviates from the treatment regimen of
hypothesis to be assessed, and adjusting for selection bias via the IPCW method. Please see Hernan etal. (2016) for details.

Figure 4. Presentation of the immortal time bias, cited from Lévesque, L. E., et. al., 201015
The upper panel illustrates an example of immortal time, where the interval from the start ofobservation to the start of intervention becomes the immortal time. It indicates that all subjects who
received the intervention have survived until the start time of intervention. When observation is
initiated from the same time-point for the two groups, a misclassification occurs in "the intervention
33

group," if these subjects are used as those with intervention. The bottom panel shows an example of
selection bias, where the immortal time from diagnosis (i.e., the start of observation in the nonintervention group) to the start of intervention is excluded because the start of observation is notaligned with the start of intervention in the intervention group.
References
1.

Mansournia MA, Nazemipour M, Etminan M. (2021). Causal diagrams for immortal time bias.
International journal of epidemiology, Invited Editorial, 50(5), 1405–1409.

2.

Hernán MA, Sauer BC, Hernández-Díaz S, et al. (2016). Specifying a target trial prevents
immortal time bias and other self-inflicted injuries in observational analyses. Journal of clinicalepidemiology, 79, 70-75.

3.

Hernán MA, Hernández-Díaz S, Robins JM. (2004). A structural approach to selection bias.
Epidemiology, 15(5), 615-625.

4.

Hernán MA. (2017). Invited Commentary: Selection Bias Without Colliders. Am J Epidemiol,
185(11), 1048-1050. https://doi.org/10.1093/aje/kwx077

5.

Suissa S. (2008). Immortal time bias in pharmacoepidemiology. American journal of
epidemiology, 167(4), 492-499.

6.

Robins JM, Rotnitzky A, Zhao LP. (1995). Analysis of semiparametric regression models for
repeated outcomes in the presence of missing data. J Am Stat Assoc, 90(429), 106-121.

7.

Little RJ, Rubin D B. (2019). Statistical analysis with missing data (Vol. 793). John Wiley &
Sons.

8.

Rubin DB. (2004). Multiple imputation for nonresponse in surveys (Vol. 81). John Wiley & Sons.

9.

AMED Research on Regulatory Science of Pharmaceuticals and Medical Devices. (2019)
Utilization of Real World Evidence Using Patient Registry Data to Support Regulatory Decision‒
Making (research and development representative, Kunihiko Hayashi, Grant number,
JP18mk0101068).

10. Rosenbaum PR, Rubin DB. (1983). The central role of the propensity score in observational
studies for causal effects. Biometrika, 70(1), 41-55.
11. Abadie A, Imbens GW. (2006). Large sample properties of matching estimators for averagetreatment effects. Econometrica, 74(1), 235-267.
12. Abadie A, Imbens GW. (2016). Matching on the estimated propensity score. Econometrica, 84(2),781-807.
13. Rosenbaum PR, Rubin DB. (1984). Reducing bias in observational studies using subclassificationon the propensity score. J Am Stat Assoc, 79(387), 516-524.
14. Desai RJ, Rothman KJ, Bateman BT, Hernandez-Diaz S, Huybrechts KF. (2017). A Propensity
score based fine stratification approach for confounding adjustment when exposure is infrequent.
34

Epidemiology, 28(2), 249-257.
15. Vansteelandt S, Daniel RM. (2014). On regression adjustment for the propensity score. Stat Med,
33(23), 4053-4072.
16. Robins JM, Hernan MA, Brumback, B. (2000). Marginal structural models and causal inferencein epidemiology. Epidemiology, 11(5), 550-560.
17. Shinozaki T, Suzuki, E. (2020). Understanding marginal structural models for time-varying
exposures: pitfalls and tips. J Epidemiol, 30(9), 377-389.
18. Bang H, Robins JM. (2005). Doubly robust estimation in missing data and causal inference
models. Biometrics, 61(4), 962-973.
19. Kang JD, Schafer JL. (2007). Demystifying double robustness: A comparison of alternative
strategies for estimating a population mean from incomplete data. Stat Sci, 523-539.
20. Kurth T, Walker AM, Glynn RJ, et al. (2006). Results of multivariable logistic regression,
propensity matching, propensity adjustment, and propensity-based weighting under conditions of
nonuniform effect. Am J Epidemiol, 163(3), 262-270.
21. Li H, Yue LQ. (2007). Statistical and regulatory issues in nonrandomized medical device clinicalstudies. J Biopharm Stat, 18(1), 20-30.
22. Yue LQ. (2012). Regulatory considerations in the design of comparative observational studies
using propensity scores. Journal of biopharmaceutical statistics, 22(6), 1272-1279.
23. Yue LQ, Campbell G, Lu N, Xu Y, et al. (2016). Utilizing national and international registries toenhance pre-market medical device regulatory evaluation. Biopharm Stat, 26(6), 1136-1145.
24. Yue LQ, Lu N, Xu Y. (2014). Designing premarket observational comparative studies using
existing data as controls: challenges and opportunities. Biopharm Stat, 24(5), 994-1010.
25. Boef AG, Dekkers OM, Vandenbroucke JP, et al. (2014). Sample size importantly limits theusefulness of instrumental variable methods, depending on instrument strength and level ofconfounding. J Clin Epidemiol, 67(11), 1258-1264.
26. Lévesque LE, Hanley JA, Kezouh A, et al. (2010). Problem of immortal time bias in cohort
studies: example using statins for preventing progression of diabetes. BMJ, 340.
27. Giobbie-Hurder A, Gelber RD, Regan MM. (2013). Challenges of guarantee-time bias. J Clin
Oncol, 31(23), 2963.
28. Mi X, Hammill BG, Curtis LH, et al. (2013). Impact of immortal person-time and time scale incomparative effectiveness research for medical devices: a case for implantable cardioverterdefibrillators. J Clin Epidemiol, 66(8), S138-S144.
29. Therneau TM, Grambsch PM. (2000). Modeling survival data: extending the Cox model.
Springer.
4.3.3.

Bayesian method35

When using external data in a clinical trial aimed at regulatory approval, the external data can beused as prior information in the framework of Bayesian statistics. In the framework of Bayesianstatistics, it is possible to quantify the treatment effect by accounting for the results of pastobservational studies and clinical trials. For example, the Bayesian approach is applied to clinicaldevelopment and marketing authorization approval for medical devices in the US. In this section, we
explain the methods for utilizing external data by means of the Bayesian approach.
The Bayesian approach quantifies the treatment effect by statistically integrating prior information
regarding the treatment effect and the data observed in the clinical trial. Prior information regarding
the treatment effect refers to the treatment effect estimated from the results of past observational
studies and clinical trials and the doctor’s clinical experience, as well as its grade of uncertainty. Since
external data sources, such as registry data, allow the use of individual-level and summary data, they
are expected to be utilizable as prior information of higher reliability than the summary data at
population level alone. Thus, the essential difference from the frequentist approaches is that priorinformation regarding the treatment effect is directly incorporated into estimation of the treatment
effect instead of being used as reference.
It may be useful to use the external data as prior information by means of the Bayesian approach inseveral situations. We present two examples here.
The first example is when efficacy is evaluated on the basis of the response rate in a single-arm
phase II cancer trial. The sample size of the single-arm phase II cancer trial is often calculated on thebasis of the frequentist approach by assuming the expected and threshold response rates. When thesample size is established on the basis of the frequentist approach, it is necessary to continue the trial
until the sample size is accumulated, in principle. Therefore, it may be necessary to greatly extend thestudy period to accumulate the remaining several patients. Because of the difficulties associated with
development based on the frequentist approach in the field of rare diseases, in addition to the problem
of mere extension of the period, it is difficult to estimate the length of extension required in manycases of rare diseases, and the impact of the information obtained by waiting for those several patientson the statistical deduction may not be significant. In the presence of such concerns, the trial may beefficiently conducted using the Bayesian approach for intermediate monitoring. For this approach, it
is necessary to define the prior information regarding the expected response rate. Noninformative priordistribution is used in many cases, because the prior information regarding the expected response rateis often insufficient. In addition, the Bayesian approach considering the uncertainty of the thresholdresponse rate has been proposed1. For the threshold response rate, utilization of external data is
expected to be useful, allowing to define more objective prior information.
The second example is when efficacy is evaluated in an RCT with limited sample size. For example,
in the field of rare diseases, it is difficult to conduct a comparative trial ensuring a sufficient power
because of limitations in the accumulation of patients, and the sample size may be established based36

on feasibility. After completion of the trial, efficacy is evaluated by the frequentist approach, such as
hypothesis testing; however, the results thus obtained are difficult to interpret. In such trials, thetreatment effect can be evaluated from a viewpoint different from the frequentist approach by applying
the Bayesian approach to external data. For example, the treatment effect can be estimated byintegrating an RCT with limited sample size and external data in a Bayesian manner. In particular, it
is possible to use external data (e.g., prior clinical trials) for the control group. For example, it ispossible to integrate the data obtained from the placebo group in the placebo-controlled RCT of theexisting drug A as the prior distribution into the data of the placebo group in the placebo-controlled
RCT of the new drug B. However, in the presence of heterogeneity between the external data and thedata observed in a new clinical trial, a bias may occur in the estimation of the treatment effect if theamount of information is treated as equivalent between the external data and the new clinical trial data.
Therefore, such bias may be attenuated by discounting the amount of information contained by theexternal data in a Bayesian manner.
When external data such as registry data are available, considering the possibility of directly using
individual patient data instead of the summary data presented in the literature, it may be possible todefine the prior information by extracting a population similar to the target patients for a clinical trial
under planning. However, since quality control of the external data differs from that for the clinical
trial, the quality of the external data must be assessed to determine whether it is acceptable from theregulatory viewpoint before using these data as the prior information. Application of the Bayesian
approach using external data should be limited to cases where validity of such application is approvedby consulting the regulatory agencies in advance. The Bayesian approach uses a simulation-based
method to estimate the posterior distribution. However, there are several simulation methods, and it isnecessary to diagnose the extent of convergence of the posterior distribution. It may take enormous
time to perform these assessments. Moreover, it is necessary to perform clinical trial simulation forevaluating design features, such as the error probability and expected sample size. For clinical trial
simulation, refer to the FDA guidance for the adaptive design (2019)2.
Conventionally, application of Bayesian methods tended to be avoided, because definition of theprior distribution and choice of the timing of analysis leave a lot of room for subjectivity, unlike thefrequentist methods. Although these points should be taken into account, the Bayesian approach,however, contributes to the transparency of the decision-making process in the sense that theframework of inference is made visible in situations where the frequency approach is difficult toimplement, such as in aspects related to the development of rare disease areas. We consider a case inwhich a clinical trial was planned so as to require the frequentist deduction but this plan was modified
and the results are submitted as application documents following early discontinuation or a case inwhich the threshold to be used for frequentist deduction is not clearly defined in a single-arm clinical
trial; some aspects of a lack of transparency are noted even in the frequentist deduction currently
37

applied for rare diseases. Bayesian technique has room to contribute not only from the viewpoint ofcontrol of the type I and II errors but also from the viewpoint of better clarification of the rationale fordecision in the process of development/approval examination and visualization of the decision-making
process. This approach may be beneficial in situations where unclear decision process represents oneof the development risks at the stage of regulatory approval.
References
1.

Thall PF, Simon R. (1994). Practical Bayesian guidelines for phase IIB clinical trials. Biometrics,
337-349.

2.

United States Food and Drug Administration. (2019). Adaptive Design Clinical Trials for Drugs
and Biologics: Guidance for Industry. https://www.fda.gov/regulatory-information/search-fdaguidance-documents/adaptive-design-clinical-trials-drugs-and-biologics-guidance-industry(Assessed 10/14, 2021)

4.3.4.

Assessment of validity

It is necessary to evaluate the validity and robustness of the results, regardless of which statisticalmethods were used. Methods for confounding adjustment work well under certain assumptions, someof which cannot be shown to be met on the basis of the data. Analysis of the data from observational
studies, such as RWD, usually suffers from unadjustable biases due to unmeasured confounders,selection of subjects, or misclassification of the exposure or outcome. Therefore, assessment isrequired regarding the extent to which deviations from the assumptions lead to changes in the results
and affect the conclusion. Evaluation of the impact of biases on conclusions is called sensitivity
analysis or quantitative bias analysis. While the sensitivity analysis in the narrow sense, according to
ICH E9 (R1), is defined under the same estimand, this report defines the sensitivity analysis in a
broader sense as the general term of analyses for evaluating the impact of biases.
The basic principle of sensitivity analysis is to model as bias parameters the factors causing a bias
in the target estimator and to obtain the corrected estimates by assigning plausible values to theparameter. Bias parameters represent the quantitative relationship between the factors and theestimator. For example, to evaluate the impact of biases due to unmeasured confounders, the outcome
incidence rate in the presence or absence of such confounders can be used as parameters. The simplest
method of sensitivity analysis consists of assigning a single set of values to the bias parameters toevaluate its impact on the estimates. “Recommendation of sensitivity analysis in observational study:
Introduction” shows in detail the concept and procedure of basic sensitivity analysis including
determination of the bias parameter for each type of bias (i.e., biases due to unmeasured confounders,selection of subjects, and misclassification of the exposure or outcome)1. In most cases, to evaluate
the bias caused by the factors for consideration, it may be insufficient to obtain a single corrected38

estimate by assigning a single set of values to the bias parameters. For this reason, it is better to obtaincorrected estimates by assigning multiple sets of values to the bias parameters, reflecting multiple
scenarios. This method is called multidimensional sensitivity analysis. This allows us to assesschanges in the estimates according to the range of changes in the bias parameters. However, only
assigning value sets to the parameters leads to lack of consideration of the uncertainty related withbias, and the evaluation of the combined effect of multiple biases is difficult. Therefore, sensitivity
analysis methods with Monte Carlo simulation have been developed. These processes include
assigning probability distributions to the bias parameters and performing Monte Carlo simulation. Thismethod is called probabilistic sensitivity analysis. The model incorporating distributions of bias
parameters in probabilistic sensitivity analysis can be interpreted in Bayesian frameworks and easily
extended to the technique to introduce multiple bias parameters corresponding to multiple biases. Sucha technique is called multiple bias modeling. See “Recommendation of sensitivity analysis inobservational study: Practice”2 for detailed practices of multidimensional sensitivity analysis andprobabilistic sensitivity analysis. Further theoretical issues are shown in “Sensitivity analysis forbiases in observational studies”3.
References
1.

Japan Pharmaceutical Manufacturers Association. Recommendation of sensitivity analysis inobservational study: Introduction,
http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/sensitivity_analysis.pdf(Accessed 7/27, 2020)

2.

Japan Pharmaceutical Manufacturers Association. Recommendation of sensitivity analysis inobservational study: Practice.
http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/sensitivity_analysis_practice.zip(Accessed 7/27, 2020)

3.

Takeuchi A. and Noma H. (2014). Sensitivity analysis for biases in observational studies.
Proceedings of the Institute of Statistical Mathematics, 62(1): 77–92.

4.4. Points to consider for reporting
4.4.1.

Necessity for uniform format

When RWD is used as external controls in combination with clinical trial data, it is necessary to
precisely report the study. In clinical research, it is necessary to report the results ensuring a certain
quality from the viewpoint of completeness, clarity, and transparency, and international guidelines,
such as Consolidated Standards of Reporting Trials (CONSORT)1,2 and Strengthening the Reportingof Observational Studies in Epidemiology (STROBE),3,4 are widely utilized. However, although some
papers have described points to consider regarding the statistical analysis method and trial design when
39

using RWD5,6, there is no review article that can serve as a guideline for reporting on a single-arm trial
using RWD as external controls. For the purpose of approval application, ICH E37 and the guideline
regarding common technical document (CTD) required as documents to be submitted for theapplication [Notification issued by the Director of the Pharmaceutical Evaluation Division,
Pharmaceutical Safety and Environmental Health Bureau, MHLW (2017). “Annex 5 (CTD –Guideline for preparation of clinical documents)8”] could be available. However, since these
documents should be referred to at the time of reporting the clinical trial results, none of them provides
direct reference as a guideline for reporting the results of a single-arm trial using RWD as external
controls.
This chapter focuses on reporting results of single-arm trials using RWD as external controls aimed
at new drug application and outlines the points to consider for reporting the study results on the basisof CONSORT and STROBE, which are commonly used for clinical trials and observational studies,and the Notification issued by the Director of the Pharmaceutical Evaluation Division, Pharmaceutical
Safety and Environmental Health Bureau, MHLW (2017). “Annex 5 (CTD – Guideline for preparationof clinical documents),”8 which is consulted for the preparation of approval application documents.
The proposal based on the STROBE is presented in the Appendix.
4.4.2.

Points to consider when reporting according to the CONSORT checklist

Assuming the use of RWD as external controls in a single-arm trial, the items that may require
additional description are summarized below. These are organized by section/topic of the CONSORT
checklist.

1) Methods(1) “Trial Design” section
In the CONSORT checklist, description of the trial design, patient eligibility criteria, and post-hoc
amendment of intervention are listed.
Reports on results of studies using RWD as external controls should include descriptions regarding
the background for using RWD as external controls and details of the selected database (e.g., method
and period of data collection, centers performing data collection, handling of personal information,
quality assurance management system, and the presence/absence of informed consent for secondary
use of the data). From the viewpoint of comparison between the single intervention arm and RWD, it
is desirable to mention the unified codes, such as the drug code and adverse event terms, if they may
be used in the analysis. In addition, from the viewpoint of reproducibility of data analysis, it isnecessary to describe the date of RWD extraction and the format at the time of data integration after
extraction, as well as the presence/absence and method of data conversion. Additionally, it is necessary
to specify the category of controls, such as external controls, internal controls collected as the natural
history follow-up group in the same trial as the test treatment group, historical and simultaneous40

controls, and hybrid control between external and internal controls.
(2) “Participants” section
The CONSORT checklist includes the patient eligibility criteria and data collection andmanagement.
When the results of the study using RWD as external controls are reported, the inclusion criteria forpatients in the RWD should be described in detail, because it is considered that patients are screenedon the basis of criteria different from that of a single-arm trial. It is desirable that potential biases
associated with comparison between the single intervention arm and RWD are also mentioned in thisdescription. On the basis of the viewpoint of transparency of the study, it is necessary to describe therole of the person in charge of the data extraction (e.g., the presence/absence of involvement in the
single-arm trial and access authority to the outcome data).
(3) “Interventions” section
In the CONSORT checklist, details of the planned intervention are listed.
Even when reporting the results obtained using RWD as external controls, details of the treatment
performed in external controls should be reported. The treatment intervention (including diagnostic
imaging and various tests such as laboratory test) for the internal control group, in case of a RCT,
should be specified whenever possible, and the difference from the treatment actually performed forthe external controls should be discussed in detail especially considering the impact of this difference
on the interpretation of the analysis results.
(4) “Statistical methods” section
The CONSORT checklist lists statistical analysis methods including comparison between groups,
as well as the exploratory analysis, such as subgroup analysis and analyses for confounding adjustment.
When the results of a study using RWD as external controls are reported, the endpoints of interestin the integrated analysis, as well as the statistical hypothesis for each endpoint and the treatment effectto be estimated, should be defined in detail from the viewpoint of the estimand. Thereafter, thestatistical analysis methods performed should be detailed. The following points should be paid special
attention:(a) For setting the sample size to the external controls, it is crucial to assess the error probability(type I error rate, power) by considering the multiplicity of testing with the primary analysisof the single-arm trial.
(b) The items of predefined analysis should be specified. However, for the methods used forhandling missing data and validation of confounding adjustment method, the details of theitems for post-hoc analysis is desirable to be described, because the methods may be
determined after the data collection.
(c) If the confounding adjustment is performed, the report should specify that the analysis policy
was defined without accessing the outcome data and should prescribe the variables used for41

confounding the adjustment, as well as the methods for confounding adjustment and methodof the validation for the statistical assumptions.
(d) Even the use of multiple methods is associated with bias compared with the use of a RCT.
Additionally, it is important to describe in detail the method for sensitivity analysis forquantitatively evaluating potential biases and the size of their impact, considering thecharacteristics of the external controls. The characteristics of collected data in terms of
period, place, and target of analysis is expected to differ from that in the RCT. For this reason,
the following differences, for example, are considered as factors that may cause potential
biases, and it is desirable to quantitatively evaluate the impact of these differences:①

Difference in the content of treatment in the collected control group
Extraction of patients with the same treatment as in the RCT from RWD, and the same
statistical method is supposed to be utilized②

Difference in the collected outcome data, including definition
The statistical method using outcome data from RWD, which is recalculated orrecategorized on the basis of definition in the RCT, is supposed to be performed.

2) Results(1) “Participant flow” section
In the CONSORT checklist, it is required to describe the process of selection of the analysis set
from the enrolled patients in the trial. This additionally applies to reporting the results of a study using
RWD as external controls. It is effective to insert visual information, such as a patient flow chart, inthe detailed description, if necessary.

3) Discussion(1) “Limitations” section
The CONSORT checklist describes the limitations and causes through the data collection and data
analysis, such as the presence of potential biases and deterioration of estimation accuracy.
When the results of the study using RWD as external controls were reported, it is necessary to pay
especial attention to the following points:(a) When not all of other available data sources/data were used, it would be necessary to explain
the reasons why those databases were not used and discuss the impact that would be
generated if they were used and the interpretation of the results.
(b) Validity of the algorithm for extraction of the external controls should be evaluated. For
example, discussion on how the analysis results might change if the extraction conditions
were modified is suggested. If the external source has missing data for an important variablein patient selection, it may be worth evaluating the impact of this situation if the relevant
variable was measured and if it could be considered in the statistical analysis, from theperspective of discussing the limitations of the integrative analysis.
42

(c) It is also important to discuss the differences in treatment method and period of datacollection, as well as their impact.
(d) If the definition of the endpoints differs between the single-arm trial and external controls,
the impact on the analysis results should be detailed.
(e) On the basis of the results of the sensitivity analysis, for evaluating expected potential biases,
the size of the impact of such biases should also be discussed, including interpretation of thestudy.
References
1.

Schulz KF, Altman DG, Moher D, CONSORT Group.. (2010). CONSORT 2010 Stateme
nt: Updated Guidelines for Reporting Parallel Group Randomised Trials. BMJ, 340:c332.

2.

Tsutani K (2010). “CONSORT 2010 Statement: The Latest Guidelines for Reporting Randomized Parallel-group Comparative Studies.” Japanese Pharmacology & Therapeutics, 38(11): 939–949.

3.

Vandenbroucke JP, von Elm E, Altman DG, et al. (2007). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration. Epide
miology, 18(6): 805-35.

4.

Kamioka H (2008). The Strengthening the Reporting of Observational Studies in Epidem
iology (STROBE) Statement: Guidelines for Reporting Observational Studies. International Rules for Clinical Research, 202-9.

5.

Wang SV, Schneeweiss S, Berger ML, et al. (2017). Reporting to Improve Reproducibili
ty and Facilitate Validity Assessment for Healthcare Database Studies V1.0. Pharmacoepi
demiol Drug Saf, 26(9): 1018-1032.

6.

Duke-Margolis center. (2019). "Understanding the Need for Non-Interventional Studies U
sing Secondary Data to Generate Real-World Evidence for Regulatory Decision Making,and Demonstrating Their Credibility." https://healthpolicy.duke.edu/sites/default/files/2019-1
1/non-internvetional_study_credibility_0.pdf. (Accessed 7/15, 2020)

7.

European Medicines Agency. (1996). "ICH E3 Structure and Content of Clinical Study
Reports." https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e-3-structure-content-clinical-study-reports-step-5_en.pdf. (Accessed 1/31, 2021)

8.

Notification issued by the Director of the Pharmaceutical Evaluation Division, Pharmace
utical Safety and Environmental Health Bureau, Ministry of Health, Labour and Welfare.
(2017). “Annex 5 (CTD – Guideline for preparation of clinical documents), PSEHB/PE
D Notification No. 0202-1.” https://www.pmda.go.jp/files/000216164.pdf. (Accessed 1/20,
2020)

43

4.5. Cases
4.5.1.

Single-arm clinical trials using the SCRUM JAPAN registry as external controls

In recent years, clinical trials of anti-tumor drugs have often focused on genetic alterations that help
predict the efficacy. When a gene mutation is considered to be very promising for predicting thetreatment effect, an enrichment design enrolling only the subpopulation presenting the gene mutationis adopted in the clinical trials. However, it may be difficult to conduct a large-scaled RCT for a very
few patients with certain genetic alterations, such as lung adenocarcinoma with the ROS1 fusion gene.
In many cases where the development is focused on such rare cancers, new drug application is
performed using the data of the phase II single-arm trial. However, the appropriateness of external
controls in the single-arm trial (e.g., whether the threshold response rate established from external
information is appropriate in a single-arm trial using response rate as primary endpoint) raises aquestion during approval review in many cases. In the last few years, with growing interest regarding
the utilization of RWD, expectations for the possibility of using external sources as control group data
are increasing, especially in single-arm clinical trials for cancer patients with a rare genetic alteration(rare fraction) and rare cancers. This section presents examples including the integrate analysis planof a single-arm trial data and the control group data extracted from RWD, using SCRUM Japan, an
industry–academia collaborative genome screening project for the development and supply of optimal
therapeutic drugs using gene panel testing.
SCRUM Japan is a genome screening research performing complementary gene panel tests inmedical institutions in Japan. If patient recruitment is ongoing in clinical trials of a new drug for theidentified genetic alteration, the patient is recommended to be enrolled in related investigator-initiatedor sponsor-initiated clinical trials. For the patients enrolled in SCRUM Japan, clinical information,
genomic information, treatment details, and efficacy data including antitumor effect and survival
outcome are entered in the EDC system at each center (*a part of genomic information is centrally
managed at the National Cancer Center Hospital East). The cancer patients with genetic alterations
that are targeted by the agents under development are advised to be enrolled in the disease registry(SCRUM Japan registry) for collection of deeper and wider data regarding patient background,
treatment, and efficacy, which will be registered in the EDC system of SCRUM Japan registry.
One project of SCRUM Japan is GOZILA, a screening research in patients with solid tumor except
for lung cancer1. Under this screening research, many clinical trials (almost all of them are clinical
trials for new drug applications) have been planned, and an umbrella trial integrating screening
research and clinical trials is being conducted. Among these trials, in the EPOC1602 (TRIUMPH) trial(under new drug application, as of December 2021), external sources including the SCRUM Japan
registry will be used as the control group. Note that this trial result is published in November 2021 in
Nature Medicine2, and the contents of this section are based only on the limited published information.

44

Table2. Trial examples including the integrate analysis plan of a single-arm study data and thecontrol group data extracted from RWD
Trial name(acronym)
EPOC1602(TRIUMPH)

Trial registration
numberTarget population

Primary
endpoint

External controls

UMIN000027887

HER2-positive metastatic
colorectal cancer

Objective
response rate

EPOC1703(BIGBANG)

UMIN000031857

BRAF non-V600E mutatedmetastatic colorectal cancer

Objective
response rate

EPOC1708(METBEIGE)

JapicCTI-194677

MET-amplified metastatic
colorectal cancer

Objective
response rate

・ Natural history
follow-up data(internal controls)
・ SCRUM Japan
registry・Data platform
provided by
Flatiron Health
・MyPathway trial(NCT02091141)
data
・ Natural history
follow-up data(internal controls)
・ SCRUM Japan
registry・Natural history
follow-up data(internal controls)
・SCRUM Japan
registry

EPOC1805(TiFFANY)

JapicCTI-194624

Advanced solid malignancies
with FGFR alterations incirculating tumor DNA

Objective
response rate

SCRUM Japan
registry

EPOC1806(HERALD)

JapicCTI-194707

HER2-amplified solid tumors
identified by circulating tumor
DNA analysis

Objective
response rate

SCRUM Japan
registry

The TRIUMPH trial is a multicenter phase II trial that evaluates the efficacy and safety of
combination therapy with trastuzumab and pertuzumab in patients with human epidermal growth
factor receptor 2 (HER2)-positive metastatic colorectal cancer (UMIN000027887). The planned
sample size is 25 patients for each of the following cohorts: HER2-positive patients diagnosed using
tumor tissue samples and HER2-positive patients diagnosed using blood samples. The primary
endpoint is the objective response rate, which was reported to be statistically higher than the thresholdvalue of 5% in the two cohorts. Since a high response could not be expected in the standard treatments
for the patients with metastatic colorectal cancer, the threshold response rate was estimated as 5% on
the basis of previous trial results. To examine validity of this setting, an integrate analysis comparing
the control group data extracted from external sources including SCRUM Japan registry and theefficacy data in the TRIUMPH trial was planned at the beginning of the trial. (As of January 2021, an
integrated analysis is underway with the cooperation of the applicant companies). The findings andpoints to consider from TRIUMPH trial are shown in the next section.
4.5.2.

Points to consider when performing statistical analysis from the perspective of a new45

drug application
In the previous section, we introduced the overview of the SCRUM Japan project, including
construction of SCRUM Japan registry and several single-arm clinical trials using the registry as
external controls. In this section, we overview on the findings and points to consider obtained fromthe integrative analysis of TRIUMPH trial and the points to consider in proceeding with the integrative
analysis with patient registry data the RWD is intended for use in a new drug application.
Before proceeding with the integrative analysis, we first discussed how to extract the control groupdata from the registry. While the procedure for data extraction was predefined as possible to ensure
reproducibility and objectivity, detailed statistical analysis plan was documented after enrollment andbefore implementation of the primary analysis. This is because the SCRUM Japan registry was underplanning when planning the TRIUMPH trial. To avoid arbitrariness of extraction of the control groupdata owing to access to the outcome data of the TRIUMPH trial, the handling of analysis set and data
is finalized on the basis of a repeated discussion with the registry administrator, data managers,
biostatisticians, and the sponsor company. Since the TRIUMPH trial was planned concomitantly withthe launch of the SCRUM Japan registry, it was unavoidable to document an SAP and analysis
procedures without detailed information regarding the items and schedule of data collection, data
management system, and specification of the database. Even if these details could be precisely
collected, the number of target patients of the SCRUM Japan registry cannot be estimated until before
data analysis. In addition, the SAP must be prepared considering an uncertainty, since the decision on
the handling of missing data and efficacy data is also unknown until data analysis. On the assumptionthat efficacy data and covariates data can be obtained ensuring a certain level of quality by data
management and monitoring, it was written in the SAP that the confounding adjustment method using
the propensity score should be performed if the sample size in the external control group reached a
specified number, and the characteristics of the control group data should be descriptively summarized
if the number was not reached. According to the process proposed in recent years, an independent
statistician not involved in the analysis of this trial should finalize the SAP by reviewing the sample
size calculation and details of the confounding adjustment method to be applied after estimating thepropensity score without viewing the outcome data, so that the confounding adjustment is performed
under conditions as similar as possible to those of the RCT3. However, this was not considered in thistrial.
In the SCRUM Japan registry, monitoring (central monitoring and sampling source data verification(SDV)) and pruning audit are performed, although not frequently as in clinical trials. Under suchactivities to assure data quality and reliability, deviations of imaging examinations and missing of date
information (e.g., missing information on the day of the date) occurred, which resulted in a difficultyof determining how to handle such data and whether such patients should be included in the analysis
sets. From the viewpoint of preparation of the application documents to the regulatory agencies,46

compliance with the Clinical Data Interchange Standards Consortium (CDISC) standards is also an
issue. Fortunately, the sponsor company with rich knowledge and experience about CDISC standards
took charge of the task of converting EDC raw data of the registry into the CDISC standards. Even inthis case, education regarding the CDISC standards and close cooperation with the sponsor company
are indispensable in academic research organization (ARO) for ensuring smooth progress ofoperations. Moreover, the presence/absence of consent to secondary use of the registry data to be
utilized and handling of personal information must be considered. When using foreign country data,
it is necessary to additionally consider international ethical aspects. Including collection of these data,
it is necessary to proceed with analysis while arranging sharing of information and adjusting thedivision of work assignment with the sponsor company.
References
1.

Nakamura Y, Taniguchi H, Ikeda M, et al. (2020). Clinical Utility of Circulating Tumor
DNA Sequencing in Advanced Gastrointestinal Cancer: SCRUM-Japan GI-SCREEN and
GOZILA Studies. Nat Med, 26(12): 1859-1864.

2.

Nakamura Y, Okamoto W, Kato T, et al. (2021). Circulating Tumor DNA-Guided Treat
ment With Pertuzumab Plus Trastuzumab for HER2-amplified Metastatic Colorectal Canc
er: A Phase 2 Trial. Nat Med, 27(11): 1899-1903.

3.

Yue LQ, Campbell G, Lu N, Xu Y, et al. (2016). Utilizing national and international registries to enhance pre-market medical device regulatory evaluation. J Biopharm Stat, 26(6):1136-1145.

47

5.

Randomized clinical trial using RWD as hybrid control
This chapter explains a concept termed “hybrid control”. This concept attracts considerable

attention as a plausible study design and analysis method for integrating concurrent and/or historicalcontrol data sources into current clinical trials. Some papers denote this concept as hybrid design or
hybrid approach; however, here, we uniformly use the term “hybrid control”.
5.1. Hybrid control
Hybrid control is a generic term for analysis methods that are used for integrated analysis, whichuses an external data source and RCT data. External data sources include both data routinely collectedin daily practice and aggregated data from individual patient level datasets from past clinical trials.
Data from patient registries would serve as a plausible external data source. Regardless of how andwhen a dataset is collected, an external data source includes efficacy and/or safety data from patients
who received or will potentially receive the same or similar treatment as the treatment in the control
group of a current RCT. If one could abstract valid control datasets for RCTs from external data sources
using scientifically valid methods and subsequently estimate a scientifically reasonable treatment
effect using a statistically valid approach, the hybrid control concept is expected to result in an
efficiency gain in clinical trials, such as reduction in sample size and study periods. For example, if
the external control data from external data sources can act as a supplemental control dataset of RCT
with unequal allocation ratio (e.g., 1:3), the required sample size for the current RCT is smaller thanthat for an RCT with equal allocation ratio, which results in a great advantage in efficiency. In
particular, in settings where there are ethical and feasibility concerns regarding the conduct of RCTs
(e.g., clinical trials for patients with rare diseases), cost-unrelated merits can be expected. The othermerit of using hybrid control in RCTs with unequal allocation ratios is that one could collect more
safety information on investigational treatments. Notably, the concept of the hybrid control ismentioned in the draft version of ICH-E8 (R1): “It may be possible for a single clinical study to use
both internal and external control subjects. For example, the conduct of a study may be facilitated bysupplementing the internal control group with additional data on the external control group.”1 The ideaof the hybrid control is pioneering; thus, it has been rarely used in actual clinical trials. As of November
2020, it has been reported that an external data source from the Medidata synthetic control database isplanned to be used as a hybrid control in an ongoing phase 3 RCT (sponsor, Medicenna), allocating
cancer patients into MDNA55 and control groups in a 3:1 ratio2,3. Although the statistical analysis planand the complete clinical trial information was not provided, the press statement stated that an “Endof phase 2” meeting with the FDA was completed and that the FDA expressed a favorable view. Other
examples include (i) a case where past eight placebo-controlled RCTs were used as external data
sources in an RCT (allocation to a ratio of 4:1) for patients with ankylosing spondylitis4 and (ii) a casewhere a past RCT for adult patients was used in an RCT for pediatric patients with multiple sclerosis5.
The latter is a trial under the US FDA’s Complex Innovative Designs pilot program6. For the above48

two cases, the meta-analytic predictive (MAP) prior approach7,8 was used. In the MAP approach, an
external source is used to form informative prior distributions for the concurrent control group in acurrent RCT. Several ongoing pilot studies are attempting to extract external control datasets fromexisting data sources and evaluate the replicability of treatment effect estimates of RCTs by using
external observational study data9–13.
In external data sources, the target population, treatments (dose/dosing schedule), and method ofcollecting efficacy/safety data (definition and schedule) are likely to be different from that in clinical
trials. It is inappropriate to perform a simple integrated analysis using individual patient data from anexternal control dataset and that from a current RCT. Extracting a subset from the external control
dataset, so that comparability with the current RCT is guaranteed, by applying statistical methods
adjusting for a potential heterogeneity between the two data sources that may lead to a bias in the
result. As discussed above, cases that have adopted hybrid control appear to be very few at thistimepoint. In the current stage, discussions on hybrid control are in progress as a research theme ofbiostatistics, rather than application to actual RCTs.
Many statistical issues are common to the settings where external sources are used as the control
group in a single-arm trial. However, there are issues specific to the use of hybrid control. The next
section discusses statistical issues in the use of hybrid control and outlines future perspectives.
5.2. Possibilities and statistical issues of hybrid control
In RCTs using hybrid control, data analysis will be planned and implemented in the following
process. From here on, we mainly focus on statistical issues under the following premises: appropriate
external sources that fit the purpose of the analysis, appropriate consent obtained regarding thesecondary data usage, and appropriate agreement signed regarding the scope of data use and handlingof obtained knowledge between the owner and the user of external source.
First, for the integrated analysis, a population that can be considered as the control group data of an
RCT must be strictly defined. It is desirable that inclusion/exclusion criteria should be as close as
possible to that of the RCT. In some cases, it may be necessary to examine the treatment method that
was actually administered and method and schedule for collecting the outcome data and to establish acertain rule for these parameters before extracting the control group data. After defining how to
identify patients who were treated with the same control treatment as in the control arm of the current
RCT (as explained above), one can initiate an operation to extract a population whose baselinecharacteristics were similar to that of patients assigned to the control group in the RCT so that thecomparability with treatment groups after integration into RCT data is not compromised. A possible
option is a propensity score method, which enables data extraction for a patient subset whose key
confounders are similar to that of the control group data in the RCT. The whole process should be
performed by personnel who is blinded to the outcome data. An example of a plausible approach is49

pre-specification of implementation system, analysis methods, and procedure to estimate a propensity
score and its adjusting methods. In addition, after the analyses, submission of the entire process
together with software codes to regulatory authorities will be helpful. If necessary, it may be beneficial
to outsource the operation to an analyst independent of the research organization. The abovementioned
processes can be used for identifying the external control data that is to be integrated into RCT data.
There are various proposals for techniques for integrating the control group data into RCT data14,15.
Proactive discussions on the methodology are ongoing. A method for simple integrated analysis at anindividual level ignores heterogeneity between external sources and an RCT; thus, it may lead to awrong conclusion. From this point, there exist some proposed methods that data-adaptively determine
how much the external control data can be borrowed16. Major techniques include the test-then-pool
method (a method17 to test the null hypothesis that an outcome is equal between the external control
data and the control group data in an RCT and to merge both if the null hypothesis is accepted); themodified power prior18,19 and the commensurate power prior,20,21 which are an extension of power
prior16; and the MAP prior model7,8 that assumes several external information sources. Refer to theliterature22–30 for recent statistical methodology for hybrid control.
It is desirable to strictly prespecify the procedures for data analysis and statistical analysis methods
discussed above in a SAP. It is desirable that the SAP provides a detailed definition regarding thetreatment effect to be estimated from the viewpoint of so-termed estimand, and the SAP should detailthe target analysis set and intercurrent events. The main statistical approaches will include theconfounding adjustment method and Bayesian approach, which are used for observational studies ingeneral rather than for clinical trials. In the SAP, a sensitivity analysis is also prespecified (e.g., amethod for validating the required assumptions for performing confounding adjusting methods). If
analyses using a hybrid control is confirmatory in nature, multiplicity adjustments may be necessary,considering multiple hypothesis testing on the analysis using RCT data only (decision-making basedon the posterior distribution in case of the Bayesian approach). It is additionally important to examine
whether the power of analyses using hybrid control is maintained at a certain level in terms of therequired sample size. If power calculation is performed using simulations, an auxiliary document (e.g.,a document that summarizes the simulation plan, results, and discussions)/analysis program, aside
from the SAP and statistical analysis reports should be prepared and ready for submission if necessary,so that a third-party organization such as regulatory authorities can reproduce and interpret the resultscorrectly.
The next section summarizes the advantages and disadvantages of analyses using hybrid control in
the above processes and discusses the possibilities in the future.
5.2.1.

Advantages and disadvantages

Hybrid control has the following potential advantages:
Supporting new drug development for populations neglected in previous studies
50Addressing ethical concerns for randomization

Acquisition of RCT results, which are different from that obtained through a single-arm trial
alone



Owing to frequent missing and extensive crossovers, in some RCTs, interpretation of theefficiency (power) of the planned statistical analysis and the estimated treatment effect may
be difficult. If missing crossovers are infrequent in the external data sources, the problems
mentioned above may be prevented or mitigated.



Since the control group data of RCT exist, the possibility of extracting a population that canbe compared with the test group in the RCT is higher than in the case where an external control
is extracted in a single-arm trial.



By comparing the control group data in an RCT with the external control data, it may bepossible to investigate in detail the problems and validity of the process of extraction fromexternal sources. Notably, since the control group data in an RCT do not exist in a single-arm
trial, there are limitations related to the problems and validity of the process of extraction.



Cost reduction to perform clinical trials



Expediting the clinical trial owing to a reduction in sample size and rapid enrollment, andresulting mitigation of the burden on the support staffs

These advantages are considered to take effect in clinical development, especially for rare diseases.
Although this is not the main objective of this document, it appears that the hybrid control has great
advantages in terms of cost efficiency, even in disease areas where a large-scale RCT is possible (e.g.,
circulatory diseases and malignant tumor in a large number of patients).
However, the concept of hybrid control presents the following disadvantages:
General problems when using external sources as the control group data (refer to Sections 6.3
and 6.4)



In cases where RCT with unequal randomization ratio is planned with a small sample size, onewill face statistical problems (e.g., a high variability in treatment effect estimates and a loss inpower).

An RCT with unequal allocation has an advantage of increasing the number of patientsassigned to the experimental group. However, because a certain number of patients needs tobe assigned to the control group, ethical concerns regarding disadvantages of this approachremain, compared to the concerns for a single-arm clinical trial.

In light of comparability, the result of RCTs should be mainly reported. The SAP will become
more complex when an analysis plan using the hybrid control is incorporated. This will resultin difficulty in interpreting the results.
For example, opinions may be divided on which should be the primary or supportive
analysis, “RCT analysis only” or “integrated analysis of using hybrid control.” One can51easily imagine that the two analysis results will be different (e.g., a statistical significancein the former disappears in the latter). It is important to explore the reasons behind sucha difference. However, the possible causes are diverse, e.g., different distribution of
unmeasured confounders between the RCT and the external control group, a statisticalassumption to perform the analysis cannot be examined. In the worst case, the reasons
itself cannot be identified because of the limited information in the external data source.
Additionally, multiplicity is a concern to be tackled when multiple confirmatory analyses
are planned. Careful discussion is required regarding the requirement for a sample size
larger than the sample size for RCTs owing to excessive multiplicity adjustments.



For most of the abovementioned analytic approaches for integrating the external control into
RCT, the strength of borrowing external data is accommodated according to the similarity of asingle primary endpoint between the RCT and external sources. However, such anaccommodation can be decided on the basis of the similarity of other endpoints, data collection
methods, difference in data reliability, and the sample sizes. It is necessary to discuss withregulatory authorities the amount of information to be integrated from external sources.



Many of the above-mentioned methods utilize the Bayesian approach. At present, there is nointernational consensus on what and how informative prior distribution should be constructedin confirmatory trials. For using a propensity score for the integration, several methods can beutilized for estimating a target treatment effect. Even for these candidates, there is nointernational consensus on what method should be primarily selected; thus, careful discussions
with regulatory authorities is indispensable.



For Bayesian approaches, Markov chain Monte Carlo (MCMC) method is routinely used toderive posterior distributions of interest. In addition, a simulation is also routinely required toevaluate the operating characteristics and estimate the required sample size. At the planningphase, it is highly recommended to discuss with regulatory authorities regarding a check forconvergence in MCMC sampling and the content and interpretation of simulation studies.



Recently, there is an increasing trend of multi-regional clinical trials. There is a room to discussapproaches to evaluate ethnic factors in RCTs using hybrid control. In many cases, it is difficult
to collect patient data from every country participating in a multi-regional RCT. Even if sucha collection is possible, it is highly likely that the time and method of data collection differ
among countries. This issue should also be discussed with regulatory authorities in advance.

5.2.2.

Issues in the future

Several experiences and lessons are achieved among several related parties, such as regulatory
authorities and sponsors, in situations where external control data is used in a single-arm trial. In
particular, in the US, after the enactment of the 21st Century Cures Act, several organizations
52

submitted opinion papers,1,29-32 and several applicational examples are reported (e.g., the control groupdata extracted from the pooled database on adult B-cell precursor acute lymphocytic leukemia
collected by Amgen35 in Table 4 of literature 15, the control group data on lung cancer and breast
cancer extracted from the database of Flatiron Health9,37,38, and the control group data extracted fromthe clinical trial database established by Medidata12). In the US, the FDA has announced multiple
guidances and the summary of various efforts to solve problems using RWD 38,39. The move towardcommercial utilization of high-quality external sources40–43 is also active. It is expected that the
number of cases where external control data are used in single-arm trials will increase steadily in the
future.
Nevertheless, there is no certain prospect for the hybrid control approaches. The performance of thestatistical method to be applied has not been thoroughly evaluated. There has been practically no case,where this approach has been utilized for an approval application, as far as we know. It is unknown
whether the hybrid control is favorably accepted in the decision of approval. The wide recognition andapplication of the new concept of hybrid control will depend on the examination results of various
projects led by the FDA such as RCT duplicate44 that examines whether the results of RCTs can be
reproduced from RWD. At this stage, we only recognize it as a new concept in the stage of examination.
However, regardless of whether it is applied for a rare disease or a disease for which an RCT is feasible,
hybrid control has great cost advantages for study sponsors because the number of patients and RCT
period can be reduced. This means that the possibilities of research and development of new medical
products for diseases, neglected in past drug-and-device developmental programs, may increase, eventhough they might be small. In addition, it has potential for obtaining added values such as
improvement of ethical concerns about randomization and bias in treatment effect owing to missing
data and crossovers. We hope that a more thorough discussion on statistical methods is conductedamong stakeholders and that the applicational cases for actual RCTs will gradually increase.

References
1.

Pharmaceutical Evaluation Division, Pharmaceutical Safety and Environmental Health Bur
eau, Ministry of Health, Labour and Welfare. (2019). ICH E8 (R1) General Policies/guidelines for Clinical Studies (Draft), Published Version for Collecting Opinions in ICH P
rocess in Step 2 (Version of May 8, 2019). https://search.e-gov.go.jp/servlet/PcmFileDow
nload?seqNo=0000187470. (Accessed 1/20, 2020)2.

Medicenna Therapeutics Corp. (2020). Medicenna provides mdna55 rgbm clinical progra
m update following positive end of phase 2 meeting with the U.S. Food and Drug Ad
ministration (FDA). https://ir.medicenna.com/news-releases/news-release-details/medicenna-p
rovides-mdna55-rgbm-clinical-program-update-following. (Accessed 10/15, 2020)
53

3.

Zacks Small Cap Research. (2020). MDNA: planning for innovative hybrid registration t
rial for MDNA55…. https://scr.zacks.com/News/Press-Releases/Press-Release-Details/2020/
MDNA-Planning-for-Innovative-Hybrid-Registration-Trial-for-MDNA55-article/default.aspx.
(Accessed 10/20, 2020)

4.

Baeten D, Baraliakos X, Braun J, et al. (2013). Anti-interleukin-17a Monoclonal Antibo
dy Secukinumab in Treatment of Ankylosing Spondylitis: A Randomised, Double-blind,
Placebo-controlled Trial. Lancet, 382(9906): 1705-1713.

5.

Schmidli H, Häring DA, Thomas M, et al. (2020). Beyond Randomized Clinical Trials:
Use of External Controls. Clin Pharmacol Ther, 107(4): 806-816.

6.

United States Food and Drug Administration (FDA). (2019). Complex Innovative Trial
Designs Pilot Program. https://www.fda.gov/drugs/development-resources/complex-innovativ
e-trial-designs-pilot-program. (Accessed 10/20, 2020)

7.

Neuenschwander B, Capkun-Niggli G, Branson M, et al. (2010). Summarizing Historical
Information on Controls in Clinical Trials. Clin Trials, 7(1): 5-18.

8.

Schmidli H, Gsteiger S, Roychoudhury S, et al. (2014). Robust Meta‐analytic-predictive
Priors in Clinical Trials With Historical Control Information. Biometrics, 70(4): 1023-10
32.

9.

Carrigan G, Whipple S, Capra WB, et al. (2020). Using Electronic Health Records to
Derive Control Arms for Early Phase Single-arm Lung Cancer Trials: Proof-of-concept in Randomized Controlled Trials. Clin Pharmacol Ther, 107(2): 369-377.

10. Bartlett VL, Dhruva SS, Shah ND, et al. (2019). Feasibility of Using Real-world Data to Replicate Clinical Trial Evidence. JAMA Netw Open, 2(10): e1912869.
11. Menefee ME, Gong YT, Mishra-Kalyani PS, et al. (2019). Project Switch: Docetaxel asa Potential Synthetic Control in Metastatic Non-small Cell Lung Cancer (mNSCLC) Tria
ls. J Clin Oncol, 37(15).
12. Davi R, Chandler M, Elashoff B, et al. (2019). Non-small Cell Lung Cancer (NSCLC)
Case Study Examining Whether Results in a Randomized Control Arm are Replicated b
y a Synthetic Control Arm (SCA). J Clin Oncol, 37(15).
13. Khozin S, Zhi JZ, Jun M, et al. (2019). Real-world Characteristics and Outcomes of Pa
tients With Advanced Non-small Cell Lung Cancer (aNSCLC) Receiving Immune Check
point Inhibitor. J Clin Oncol, 37(15).
14. van Rosmalen J, Dejardin D, van Norden Y, et al. (2018). Including Historical Data in
the Analysis of Clinical Trials: Is It Worth the Effort?. Stat Methods Med Res, 27(10):3167-3182.
15. Lim J, Walley R, Yuan J, et al. (2018). Minimizing Patient Burden Through the Use of
Historical Subject-level Data in Innovative Confirmatory Clinical Trials: Review of Met
54

hods and Opportunities. Ther Innov Regul Sci, 52(5): 546-559.
16. Ibrahim JG, Chen MH. (2000). Power Prior Distributions for Regression Models. Stat S
ci, 15(1): 46–60.
17. Viele K, Berry S, Neuenschwander B, et al. (2014). Use of Historical Control Data for
Assessing Treatment Effects in Clinical Trials. Pharm Stat, 13(1): 41–54.
18. Duan YY, Ye KY, Smith EP. (2006). Evaluating Water Quality Using Power Priors to I
ncorporate Historical Information. Environmetrics, 17(1): 95-106.
19. Neuenschwander B, Branson M, Spiegelhalter DJ. (2009). A Note on the Power Prior. S
tat Med, 28(28): 3562–3566.
20. Hobbs BP, Carlin BP, Mandrekar SJ, et al. (2011). Hierarchical Commensurate and Pow
er Prior Models for Adaptive Incorporation of Historical Information in Clinical Trials.
Biometrics, 67: 1047–1056.
21. Hobbs BP, Sargent DJ, Carlin BP. (2012). Commensurate Priors for Incorporating Histori
cal Information in Clinical Trials Using General and Generalized Linear Models. Bayesi
an Analysis, 7: 639–674.
22. Psioda MA, Soukup M, Ibrahim JG. (2018). A Practical Bayesian Adaptive Design Inco
rporating Data From Historical Controls. Stat Med, 37(27): 4054-4070.
23. Lin J, Gamalo-Siebers M, Tiwari R. (2019). Propensity‐score‐based Priors for Bayesian
Augmented Control Design. Pharm Stat, 18(2): 223-238.
24. Lin J, Gamalo-Siebers M, Tiwari R. (2018). Propensity Score Matched Augmented Controls in Randomized Clinical Trials: a Case Study. Pharm Stat, 17(5): 629-647.
25. Yuan J, Liu J, Zhu R, et al. (2019). Design of Randomized Controlled Confirmatory Tri
als Using Historical Control Data to Augment Sample Size for Concurrent Controls. J
Biopharm Stat, 29(3): 558-573.
26. Wang C, Rosner GL. (2019). A Bayesian Nonparametric Causal Inference Model for Sy
nthesizing Randomized Clinical Trial and Real‐world Evidence. Stat Med, 38(14): 25732588.
27. Lewis CJ, Sarkar S, Zhu J, et al. (2019). Borrowing From Historical Control Data in C
ancer Drug Development: a Cautionary Tale and Practical Guidelines. Stat Biopharm Res, 11(1): 67-78.
28. Chen WC, Wang C, Li H, et al. (2020). Propensity Score-integrated Composite Likeliho
od Approach for Augmenting the Control Arm of a Randomized Controlled Trial by In
corporating Real-world Data. J Biopharm Stat, 30(3): 508-520.
29. Frients of Cancer Reseach (2019). " Characterizing the Use of External Controls for Au
gmenting Randomized Control Arms and Confirming Benefit." from https://www.focr.org/
events/friends-cancer-research-annual-meeting-2019. (Accessed 1/20, 2020)55

30. Eichler HG, Koenig F, Arlett P, et al. (2019). Are novel, nonrandomized analytic metho
ds fit for decision making? the need for prospective, controlled, and transparent validati
on. Clin Pharmacol Ther, 107(4): 773-779.
31. Duke-Margolis Center. (2019). "Understanding the need for non-interventional studies usi
ng secondary data to generate real-world evidence for regulatory decision making, and demonstrating their credibility." https://healthpolicy.duke.edu/sites/default/files/u31/non-interv
entional_study_credibility_0.pdf. (Accessed 1/20, 2020)
32. European Medicines Agency (2019). ICH E8(R1) Guideline Draft version (document dat
ed 25 March 2019). https://www.ema.europa.eu/en/documents/scientific-guideline/draft-ichguideline-e8-r1-general-considerations-clinical-studies-step-2b_en.pdf. (Accessed 1/20, 2020)
33. Gökbuget N, Dombret H, Ribera JM, et al. (2016). International Reference Analysis of
Outcomes in Adults With B-precursor Ph-negative Relapsed/refractory Acute Lymphoblas
tic Leukemia. Haematologica, 101(12): 1524-1533.
34. Huang Bartlett C, Mardekian J, Cotter MJ, et al. (2020). Concordance of Real-world Ve
rsus Conventional Progression-free Survival From a Phase 3 Trial of Endocrine Therapy
as First-line Treatment for Metastatic Breast Cancer. PLoS One, 15(4), e0227256.
35. Pfizer Inc (2019). U.S. FDA Approves Ibrance® (Palbociclib) for the Treatment of Men
With HR+, HER2- Metastatic Breast Cancer: Approval of Expanded Indication Based P
redominately on Real-world Data. https://www.pfizer.com/news/press-release/press-release-d
etail/u_s_fda_approves_ibrance_palbociclib_for_the_treatment_of_men_with_hr_her2_metasta
tic_breast_cancer. (Accessed 8/31, 2020)
36. United States Food and Drug Administration (2017). "Use of Real-World Evidence to S
upport Regulatory Decision-Making for Medical Devices" https://www.fda.gov/media/9944
7/download. (Accessed 8/31,2020)
37. United States Food and Drug Administration (2018). "Framework for FDA’s Real-world
Evidence Program" https://www.fda.gov/media/120060/download. (Accessed 8/31, 2020)
38. Yin PT, Desmond J, Day J. (2019). Sharing Historical Trial Data to Accelerate ClinicalDevelopment. Clin Pharmacol Ther, 106(6): 1177-1178.
39. Target RWE. Target RWE https://www.targetrwe.com/. (Accessed 8/31, 2020)
40. Optum, Inc. (2020). "Real-world Data Centralized Teams." https://www.optum.com/busine
ss/resources/library/rwd-centralized-teams.html. (Accessed 8/31, 2020)
41. MEDABLE. "MEDABLE." https://www.medable.com/. (Accessed 8/31, 2020)
42. RCT DUPLICATE. "RCT DUPLICATE." https://www.rctduplicate.org/. (Accessed 8/31, 2
020)

56

6.

Use of RWD as complementary information

6.1. Introduction
This chapter reports the use of RWD to complement data that will be used for regulatory purposes
and a clinical trial platform in a context different from that in the previous two chapters. In drug
development, RCTs play a major role in the collection of data that will be used in a regulatory context.
In the regulatory application, RCTs mainly evaluate the efficacy of the drug. In daily clinical settings
where a physician selects an optimal treatment method for a patient, the evidence from RCTs isinsufficient from the perspective of generalizability. In contrast, if the information from an intervention
that is performed in a real-world clinical setting can be used as RWD, the evaluation of theeffectiveness of therapeutic drugs, including the influence of concomitant drugs and post-treatment
effects, can be determined. Specifically, while RCTs mainly evaluate the efficacy of drugs(explanatory evaluation), RWD enables the evaluation to obtain useful information for determining
treatment policies (pragmatic evaluation). Thus, using RWD will enable the evaluation to be
performed from a perspective different from that of the conventional regulatory application of data(complementation of data). RWD can also be used to efficiently accumulate cases to conduct an RCT(complementation of the platform).

Similar ideas can be applied to medical product development in disease areas in which it isdifficult to implement RCTs, which is the scope of this report. The use of RWD is considered
for the following reasons:(1) To add knowledge obtained from RWD to the regulatory application of data that focuses onpivotal trials, and(2) To use RWD as a clinical trial platform.
Regarding (1), there have been several cases in Japan and in countries overseas in which RWD
collected from non-pivotal trials have been used with a regulatory application in efficacy and safety
evaluations and for indication expansion purposes. For (2), the use of RWD in the conventional
regulatory application was limited. However, in the investigator-initiated clinical trial on viltolarsen,
Remudy, a Japanese registry of neuromuscular disorders established by the National Center of
Neurology and Psychiatry, was used to enroll patients, and this clinical trial achieved an efficient
subject registration1. Judging from such an example of the improvement in patient registries, anincrease of this type of utilization is expected to be observed in the future.
In the subsequent sections, (1) and (2), in the utilization of RWD, are summarized. Regarding (1),cases of utilization are introduced, and regarding (2), the advantages and disadvantages of R-RCTs
that use registries as a data platform are summarized. R-RCTs deviate from the scope of this report,
and it is rather special to handle them as (2). However, the actual use of R-RCTs has recently drawn
attention, and the advantages and disadvantages of R-RCTs are summarized systematically. Part of the
R-RCT methodology can be extrapolated into the settings where RWD is used as a clinical trial
57

platform for therapeutic areas in which RCTs are difficult to implement. For the above reasons, RRCTs are introduced hereafter.
6.2. Complementation of RWD in the regulatory application of data from pivotal trials
This section introduces the cases in Japan and in countries overseas in which the findings obtained
from RWD were added to data with a regulatory application, mainly consisting of data from pivotal
trials. Furthermore, this section explains that the importance of such an approach may increase intherapeutic areas in which the implementation of RCTs is difficult.
An example in Japan is a case of the indication expansion for a granulocyte colony stimulating
factor filgrastim (genetic recombination) regarding the mobilization of healthy donor hematopoietic
stem cells in the peripheral blood during hematopoietic stem cell transplantation2. In the clinical trial
on this drug, the long-term safety was not evaluated sufficiently. In the drug approval review process,the results of a post hoc survey administered to donors of “the research on establishment andfamiliarization of the radical cancer treatment system by allogeneic hematopoietic stem celltransplantation mainly from unrelated donors” from The MHLW of Japan cancer grant were classified
as RWD. Of 95 patients who underwent allogeneic hematopoietic stem cell transplantation in Japan
from August 1997 to December 1999, 37 completed a follow-up survey, the results from which
confirmed that there were no patients with particular clinical abnormalities. The Pharmaceuticals andMedical Devices Evaluation Center, which was in charge of the approval of the review asked the
Office for Organ Transplant whether the unrelated donors were controlled by the Japan Marrow Donor
Program in the same manner as those who underwent allogeneic bone-marrow transplantation. The
following facts were confirmed: the system was being established; regarding the allogeneic bonemarrow transplantation in Japan, the Japan Marrow Donor Program and the Japan Society for
Hematopoietic Cell Transplantation jointly prepared the system to implement a long-term follow-up
for the healthy donors (related and unrelated) of allogeneic peripheral blood stem cells; and the Officefor Organ Transplant announced that the Japan Society for Hematopoietic Cell Transplantation
decided to disclose the follow-up data of the donors while protecting their privacy. Such information
has led to the understanding that the system that will be used to implement the long-term follow-up
after the drug is approved has been established.
As shown in the cases above, RWD was used as a substitution of RCT data in the application ofdata and may also be used to complement the data from clinical trials serving as interventional researchdata. In therapeutic areas in which the implementation of RCTs is difficult, information rationalizing
the efficacy of a drug from RCTs is not obtained, the scale of the trial is smaller, and the content of theinvestigation in the trial is restricted. Therefore, such conditions result in only a small volume of data.
For this reason, if an evaluation of medical products is conducted from the perspective of whether theproper use is possible in the clinical setting, not just limited to a confirmation of the efficacy and safety58

in the narrow sense, complementing data from studies with RWD in regulatory approval applications
will become more important in therapeutic areas in which RCTs are difficult to implement than inthose in which the implementation of RCTs is feasible. More specifically, not just limiting thediscussions to the inclusion of RWD as a replacement of RCT data, the significance and necessity of
performing a comprehensive evaluation by combining information from various evidence levels
should be examined.
Furthermore, as in the case of using RWD when data from a pivotal trial does not exist, the additionof indications of palbociclib to locally advanced or metastatic breast cancer in men who are estrogen
receptor (ER)-positive and HER2-negative has been reported in the US3. Palbociclib was approved by
US FDA in 2015 as a combination drug with endocrine therapy for advanced breast cancer in postmenopause females who are ER-positive and HER2-negative. In 2019, it gained an indication foradvanced male breast cancer in combination with endocrine therapy (aromatase inhibitors or
fulvestrant). Because it was estimated that there were 2,670 male breast cancer patients in 2019, which
accounts for less than 1% of the total breast cancer patient population, it is difficult to perform an RCT,
let alone a clinical trial, with only male breast cancer patients. Data from a database of insurance
claims owned by IQVIA, data from a database from an EHR system owned by Flatiron, and data froma global safety database owned by Pfizer were used as RWD to retrospectively investigate the use ofpalbociclib in male breast cancer patients and evaluate its response rate, study extension period, andsafety. Although the application of data without including data from a clinical trial is very unlikely tobe accepted by the pharmaceutical affairs system in Japan, complementing data with RWD, as shownin the previous case, is the most extreme case of a policy for which a comprehensive evaluation isconducted by combining information from multiple evidence levels. However, in medical productevaluations in ultra-orphan therapeutic areas, whether such an approach should be acceptable is worth
discussing.
For the other cases, refer to a document “Use the real-world data for approval application –A
proposal to promote the utilization”4 issued by the Clinical Evaluation Expert Committee, Drug
Evaluation Committee of the Japan Pharmaceutical Manufacturers Association, and refer to the study
by Andre et al. (2019)5.
6.3. Use of RWD as a clinical trial platform
This section summarizes R-RCTs regarding the significance of RWD as a clinical trial platform andintroduces that this same concept can be used in types of clinical trials other than RCTs.
An R-RCT is defined as a pragmatic trial that uses registries as a platform for performing patient
enrollment, data collection, randomization, and follow-up surveying. For data from pragmatic trials,
refer to the “Recommendation of Pragmatic Trials,” issued by the FY2017 Task Force 5 of the Data
Science Expert Committee, Drug Evaluation Committee of the Japan Pharmaceutical Manufacturers
59

Association6.
The scoping review on R-RCTs by Karanatsios et al. (2020)7 and research by Li et al. (2016)8
mention the following as being advantages of R-RCTs compared with RCTs in which a conventional
explanatory evaluation is performed.
Instead of reestablishing an electronic record and other infrastructures for each new trial,
existing registries can be used to identify patients, collect baseline patient data, and identify
the outcomes of interest. This can decrease the cost of additional study visits and training forthe staff and clinical research coordinators at the sites.



The results from R-RCTs have a higher level of generalizability than do those from RCTs
because the rigidity of inclusion and exclusion criteria is low, and the monitoring and followup of patients are much closer to that observed in the real world compared with that in intensive
monitoring, such as in RCTs, in which an explanatory evaluation is performed.



By using registries that have existing clinical information, eligible patients can be promptly
identified for enrollment. More specifically, data are already available in the registry, andentering in case report forms is not required for evaluating the eligibility of patients, and thissaves a huge effort in the patient enrollment process.

In northern Europe and Canada, since patients can be followed between registries almost
perfectly by using identification numbers specific to individual patients, the linkage ofregistries enables a broad range of patient clinical information to be obtained.

On the other hand, it should also be considered that R-RCTs have the following disadvantages andissues.
Procedures for ethical consideration, such as the necessity of consent at two stages (i.e., atenrollment in patient registries and at participation in RCTs), require further discussion.
Thus, it is not easier to obtain consent regarding randomization using this method than it is in
RCTs with an explanatory evaluation.

To ensure the generalizability of the trial results, the selection of outcomes should not disrupt
the practice of treatment in the actual clinical setting. Therefore, to ensure the pragmatic
evaluation for that purpose, the outcomes collected on a routine basis should be used. R-RCTs
are usually conducted in an open-label setting. If many subjective outcomes are adopted, biases
are more likely to be introduced. The method for minimizing the bias or enhancing theaccuracy of outcomes, such as standardization or blinding of the outcome evaluation, should
be considered if the risk of bias is high or the variation in measurement data is large. However,outcomes requiring detailed rules, such as imaging test results, cannot be easily collected in
the registry and are thus difficult to use. Since blinding is dissociated from the daily clinicalsetting and may disrupt the pragmatic evaluation, whether to perform blinding or not is adifficult decision to make10.

The quality of registries varies greatly depending on the original purpose of each individual
registry. Therefore, the following concerns are considered: the access to outcome data is60

difficult in existing registries; missing data and important prognostic factors that affect the
research results cannot be captured; and the original data of the disease cannot be identified in
the registry when the patients frequently change hospitals. It should also be noted that the sizeof the registry used in the trial restricts the number of patients.
Compared with that in RCTs that have an explanatory evaluation, the patient compliancewith the protocol in R-RCTs is not monitored as strictly, and more deviations from theprotocol may occur. Since few physicians are familiar with clinical trials, the need to include
conventional-level monitoring increases the cost of the clinical trial11.



CONSORT12 for pragmatic trials13 may be a useful resource for achieving this. However, atpresent, there are no guidelines or recommendations on the reporting of R-RCTs. Since the RRCT is a relatively new type of clinical trial, the guidelines and recommendations for it may
be limited. However, to improve the understanding and evaluation of R-RCTs and facilitate
decision-making of their feasibility and applicability, the development of systematic andstandardized tools would be useful. Furthermore, there are no useful tools or guidance for thequalitative evaluation for R-RCTs. Although the Cochrane Handbook14 and the Grading ofRecommendations Assessment, Development, and Evaluation system15 were developed toevaluate the quality of evidence of conventional RCTs, similar standards for R-RCTs are stilllacking.

The above discussions are about therapeutic diseases that can be investigated with R-RCTs, andthey can also be referred to for the therapeutic areas in which it is difficult to conduct RCTs. In
therapeutic areas in which it is difficult to conduct RCTs, there is no issue in the cost reduction for alarge-scale trial, but there is an issue in the need to efficiently accumulate subjects using the limited
financial and temporary resources. As a solution to this issue, the use of registry data as a clinical trial
platform can be a meaningful approach.
6.4. Conclusion
The advancement of research on pragmatic evaluations may affect the methods of constructing
clinical data for regulatory approval applications because it may indicate the need to supplement thefindings obtained from clinical trials with an explanatory evaluation or those from small-scale clinical
trials. More specifically, there may be an application for RWD beyond conventional discussions inthat RWD are used to replace a component of clinical data for explanatory evaluations. Although the
R-RCT methodology cannot be used in therapeutic areas in which it is difficult to implement RCTs, it
may be utilized in these areas if the data from registries are used as the trial platform.
References1.

Website of Japan Agency for Medical Research and Development. (2020). Marketing ap
proval for therapeutic drug for Duchenne muscular dystrophy (NS-065/NCNP-01, viltolar
61sen).
2.

https://www.amed.go.jp/news/release_20200327.html (Accessed 1/21, 2021).

Director, National Institute of Health Sciences. (2000). Review report, issued by NIHS
No. 2083. Retrieved January 21, 2021, from http://www.pmda.go.jp/drugs/2000/g000308/2
0repo01.pdf. (Accessed 1/21, 2021)

3.

Wedam S, Fashoyin-Aje L, Bloomquist E, et al. (2020). FDA Approval Summary: Palbo
ciclib for Male Patients with Metastatic Breast Cancer. Clinical Cancer Research, 26(6):1208-1212.

4.

Task Force 2, Clinical Evaluation Expert Committee, Drug Evaluation Committee, Japan
Pharmaceutical Manufacturers Association. (2020). Use the real-world data for approval
application –A proposal to promote the utilization. Retrieved January 21, 2021, from htt
p://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/bd_rwd_sg2.pdf. (Accessed 1/21,
2021)

5.

Baumfeld Andre E, Reynolds R, Caubel P, et al. (2020). Trial designs using real-world
data: The changing landscape of the regulatory approval process. Pharmacoepidemiol Dr
ug Saf, 29(10): 1201-1212.

6.

FY2017 Task Force 5, Data Science Expert Committee, Drug Evaluation Committee, Jap
an Pharmaceutical Manufacturers Association (2018). Recommendation of pragmatic trial
s. http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/pragmatic_trial_final_2.pdf.
(Accessed 1/21, 2021)

7.

Karanatsios B, Prang KH, Verbunt E, et al. (2020). Defining key design elements of registrybased randomised controlled trials: a scoping review. Trials, 21(1): 552.

8.

Li G, Sajobi TT, Menon BK, Korngut L, et al. (2016). Registry-Based Randomized Controlled
Trials- What are the Advantages, Challenges, and Areas for Future Research? J Clin Epidemiol,
80: 16-24.

9.

Hayashi K. (2017). Tips for effectively utilizing real-world data. Regulatory Science of Medical
Products, 7(3): 197-203.

10. Welsing PM, Oude Rengerink K, Collier S, et al. (2017). Series: Pragmatic trials and real world
evidence: Paper 6. Outcome measures in the real world. J Clin Epidemiol, 90: 99-107.
11. Irving E, van den Bor R, Welsing P, et al. (2017). Series: Pragmatic trials and real world evidence:
Paper 7. Safety, quality and monitoring. Journal of Clinical Epidemiology, 91: 6-12.
12. Schulz KF, Altman DG, Moher D. (2010). CONSORT 2010 statement: updated guidelin
es for reporting parallel group randomised trials. BMJ, 340: c332.
13. Zwarenstein M, Treweek S, Gagnier JJ, et al. (2008). Improving the reporting of pragm
atic trials: an extension of the CONSORT statement. BMJ, 337: a2390.
14. Higgins JPT, Thomas J, Chandler J, et al. (2019). Cochrane Handbook for Systematic R
eviews of Interventions version 6.1 (updated September 2020). www.training.cochrane.org
62

/handbook (Accessed 1/21, 2021).
15. Guyatt G, Oxman AD, Akl EA, et al. (2011). GRADE guidelines: 1. Introduction-GRA
DE evidence profiles and summary of findings tables. Journal of Clinical Epidemiology,
64(4): 383-394.

63

7.

Co-analysis of registry data and data from other databases
Data linkage is effective when the data necessary to evaluate a hypothesis are not collected in theregistry being utilized but are obtained from other databases. In other words, data linkage can
supplement information that is lacking in the registry alone. For example, if registry data and a longterm traceable receipt database can be linked, more information on treatment and history prior to thestarting date of treatment, etc. can be captured. Another advantage of using a registry (which does nothave a high degree of completeness) is that it is possible to evaluate the characteristics of the
subpopulation of cases enrolled in the registry in the context of actual clinical practice.
In addition, it is important for the flexible operation of the approval system that post-marketing
information and corresponding risk management actions can be implemented to supplement the dataat the time of regulatory approval of a drug. For example, information contained in the clinical data
package at the time of regulatory approval tends to be relatively limited for drugs for rare diseases,
vaccines, and products that are subject to the conditional early approval system, and development
strategies should be formulated with the future post-marketing data collection system taken into
consideration before approval or from the stage of clinical trial planning. For example, in light of therecent trend in vaccine development against COVID-19, a system that will enable early regulatory
approval is being considered in Japan as part of future risk management measures. As one factor inestablishing such a system, it is considered necessary to consider facilitating the collection andutilization of data after marketing authorization. However, there is currently no registry in Japan thatcan be used to evaluate the efficacy and safety of vaccines. For example, if there is a registry (e.g.,
National Drug Use Surveillance) conducted by a pharmaceutical company that follows only vaccine
recipients for a certain period of time, and if the registry can be linked to a cross-facility receipt
database (e.g., National Database), information on long-term follow-up after vaccination and medical
history before vaccination will be available. However, in this case, only data on the vaccination groupis available and there are no comparison controls. In Japan, because routine vaccines are not coveredby insurance, information on their vaccination is not registered in the receipt database, but only
recorded in data managed and administered by each local government and the national government.
Therefore, in order to evaluate the efficacy and safety of vaccination, it is essential to link thevaccination information managed by the national and local governments with the long-term traceable
receipt database. Such linkage would not only be beneficial from the perspective of post-marketing
safety measures, but would also lead to the promotion of development and early approval of new
vaccines and other products from the viewpoint that it would enable the construction of a framework
for integrated evaluation of data before and after obtaining regulatory approval from the development
stage.
In this chapter, we will introduce examples of collaboration among NCs and other databases in theworld, and summarize considerations to be taken into account.
647.1. Example of co-analyzing registry data and data from other databases in each NC

7.1.1.
Efforts by National Center for Child Health and Development

Efforts for co-analyzing official statistics and clinical registry data
Based on the research studies for which personnel of the National Center for Child Health andDevelopment (NCCHD) serve as the chief researchers [“A research on perinatal care evaluation by
the linkage between clinical database and government statistics data (2014–2017)” and “A researchon linkage of perinatal period-related medical database (2016–2018)”], the reports of the vital statisticsof the population and the database owned by the perinatal-related academic societies/nonprofit
organizations (NPOs) were linked by the unit of the report form and subsequently analyzed.
Specifically, two linkage databases were prepared, and a secondary analysis was performed on each.
Firstly, a database was prepared by linking the birth reports from the vital statistics of the population
regarding the births in 2004–2012 (approximately 6,000,000 cases) with approximately 50,000 cases
from infant death reports (aged <1 year), approximately 600,000 cases from the Japan Society of
Obstetrics and Gynecology Perinatal Period Registration database, and approximately 30,000 cases
from the Neonatal Research Network Database. In both the birth and death reports, the nationality,
date of birth, sex, gestational age (in weeks) and birth weight of the baby, number of fetuses, birthorder, and date of birth of the mother are entered for each subject. However, while the birth reports arecompleted by a physician witnessing the birth, the death reports are completed by a physician
diagnosing and certifying the baby’s death. Accordingly, the descriptions in the death reports may bedifferent from those in the birth reports. Actually, only 88% of the death reports were completely
consistent with the corresponding birth report. In the database operated by the NPO, the items
consistent with the birth reports were fewer than those consistent with the death reports (e.g., dates ofbirth of the mother and baby were not included), and the entries of erroneous digits were also noted.
Even in the reports that were seemingly for the same patients, all of the redundant items werecompletely consistent only in 70–80% of the patients. To improve the consistency between the
databases, the birth reports were linked to each database based on the probability linkage advocatedby Fellegi and Sunter et al. By this method, based on the µ-probability (frequency of a specific value)
generated from the distribution of values that can occur in the data for each item and the accuracy ofthe prediction of the data (m-probability), the probability that all patients in both databases are identicalis calculated, and the probability and data are linked by the probabilistic linkage based on apredetermined cutoff value. Programs that enable an easy data linkage include Record Linkage(Python, The Python Software Foundation), Freely Extensible Biomedical Record Linkage, and
LinkPlus (US Centers for Disease Control and Prevention). Subsequently, a linkage database wasprepared to identify 357 deaths within 1 year after childbirth in the vital statistics of the population
birth reports regarding births and stillbirths from 2014–2016 and in the vital statistics of the population
65

death reports for women aged 12–60 years from 2015–2017. In the preparation of these databases,
because there was an expectation that the information of the subject’s name (in Chinese characters)
and date of birth is highly specific and accurate, a method linking patient data between databases byidentifying the identical date of birth/stillbirth and mother’s age and name was used. This researchused the data on the name of the mother. Access to this information was limited to the researchers
involved in the database linkage process. After the linkage was performed, the information was deleted
and fixed as an analysis of data.
By using these two linked databases above, obstetricians, pediatricians, and social epidemiologists
collaboratively conducted the analyses and published four articles on the results. The analysis of therelationship between the social background characteristics of the mothers and the mortality of thechildren indicated that the mortality before and after very premature birth increased as the averageannual income of the municipality in which the residence was located decreased. The research on theinfant mortality statistics and the social and medical background characteristics of the mothers
revealed that both the social background characteristics and medical backgrounds of the mothers arerisk factors for death due to abuse or disease, e.g., babies with high medical risk factors, such aspremature babies and others, are more likely to experience abuse and accidents, and mothers with highsocial risk factors, such as single mothers, are associated with higher infant mortality due to the burdenof disease. Based on the incidence of a premature delivery, incidence of stillbirth, method of childbirth,and complications from pregnancy, the optimal increase in body weight during pregnancy wascalculated by the body type of pregnant Japanese women. Furthermore, an investigation on the heightof the mother and the prognosis after childbirth demonstrated that the risk of pregnancy-induced
hypertension was higher in short mothers compared with tall mothers, and this seemed to be the reason
for the high incidence of premature delivery that was observed. Regarding the death of mothers within
1 year after childbirth, postpartum suicide accounted for a relatively high proportion of the deaths,
which was a finding that drew attention from the government. Going forward, through thecollaboration of several databases and with cooperation among field of study for which cooperation
was not previously common, the results from various research studies will contribute to theimprovement of perinatal medical care in Japan.
Overseas perinatal data are separated into the data from the mothers and data from the babies thatcan be controlled. In contrast, the dates of birth for the mother and child, birth weight, and gestational
age are registered redundantly for the mothers and children. Research using the population-based vitalstatistics linked with clinical data extracted from a database by the probabilistic linkage method
without using unique identification numbers has been performed on occasion. For example, in thesoutheastern area of the state of Ohio and the northern area of the state of Kentucky in the US, there
is a research study on the primipara who gave birth from 2007–2010, which linked their birth reports
from population-based vital statistics with information from the medical record of the mothers on
66

admission and discharge and information on home visits after childbirth using the probabilistic linkage
method. In the state of California in the US, there is a research study in which the birth and deathreports from population-based vital statistics from 1990–2000, information from the medical recordsof the mothers on admission and discharge, and information and records of the admission of the infants
within 14 days after birth were linked. The research reported that most of medical expenses forperinatal care were used for babies delivered prematurely and babies with a low birth weight and that
increasing the prevention of premature deliveries will decrease the medical expenses in the state.
Analyses on annual changes in neonatal jaundice requiring hospitalization in terms of treatment trends
and risk factors are being conducted.
An effort to co-analyze medical receipt information with clinical registry data
Using funding obtained for the “research on elucidation of the cause, prevention, and treatment ofinfertility,” for which personnel of the NCCHD serve as chief researchers, a program was created to
extract and aggregate the data to perform the linkage of individual medical information from
hospitalization in the neonatal intensive care unit from 2011–2018 with clinical registry data. The data
used in this study are the medical data from FY 2019 from 192 general/regional perinatal care centers
in Japan that provided its medical care information to the Diagnosis of Population Classification
Research Institution for research by 2018, and these data were permitted by the executives of thecenters to use for this research.
A program was specifically created to summarize information on the major procedures (e.g.,
duration of using an artificial ventilator, duration of intubation, duration of using a peripherally
inserted catheter, duration of hypothermia therapy, and duration of nitrous oxide therapy) andinformation on the dose and duration of treatment (e.g., steroids and catecholamines) from the medical
data via the EF files of the children.
In the future, the validity of this program will be evaluated, and a system for collecting patient
medical data more easily and more accurately will be established.
7.1.2.


Efforts by National Center for Geriatrics and Gerontology

Organized Registration for the Assessment of dementia on Nationwide General consortium
toward Effective treatment (ORANGE) registry (Mild Cognitive Impairment [MCI] registry)
The ORANGE registry (MCI registry) is a prospective research project that aims to analyze thelong-term follow-up of patients with MCI. The subjects are recruited from cooperative research
institutions, including our center.
This registry is linked with an electronic medical record system (SS-MIX2, file maker) at our center.
Our center installed the entry form of the electronic medical record and provides the system thattransfers the data into the EDC of the registry. The data center is established at our center. The system
is available at other cooperative institutions. However, the other facilities, except for the National
67

Center for Geriatrics and Gerontology (NCGG), did not install the system directly into the EDC due
to security issues.


Regional cohort
In the NCGG Study of Geriatric Syndromes (NCGG-SGS), a large-scale regional cohort study,
health examinations for the elderly were conducted, and the data from these examinations werecollected in cooperation with the municipality. These health examination data are combined withhealth insurance data (medical insurance receipt data and data on the certification of the need for care,etc.), which are anonymized. These data are regularly updated using the reports by each municipality
and the relevant institutions. Although the NCGG-SGS targets all of the elderly residents in the area,other types of information, such as the certificates of the need for caring and death certificate, etc. isalso obtained after receiving the list of names prior to administering the survey. With these types of
information, this cohort was selected based on the eligibility of participants. Necessary information isappropriately provided for the investigation and with the support of the municipalities.
For constructing such a regional cohort, collaboration and cooperation between the municipalities
are vital to its success. Our center made a strong effort to establish a reliable relationship with themunicipalities for this study. For example, we hold regular (monthly or more frequently) meetings
with the relevant authorities, we hold meetings with the practitioners, we perform lectures, and weoffer various types of information on health promotion.
7.1.3.


Efforts by National Center of Neurology and Psychiatry

Introduction

To promote clinical development for hereditary neuromuscular diseases, a registry of patients withhereditary neuromuscular diseases Remudy (REgistry of MUscular DYstrophy) was initiated in 2009
in cooperation with TREAT-NMD (Network of Excellence for Neuromuscular Diseases), aninternational clinical research network1. A registration office is established at the National Center of
Neurology and Psychiatry. In close collaboration with the patient organization Japan Muscular
Dystrophy Association, a registry was established where patients register themselves; Remudy started
registration for dystrophinopathy, and it runs registries for GNE myopathy, myotonic dystrophy, andcongenital muscular diseases. Till date, it has conducted various epidemic surveys and surveys on
clinical trial feasibility, contributed to patient recruitment, and transmitted information for patients. Its
future tasks include collection of natural history information that can be used and referred to for control
groups in clinical trials and establishment of registries contributing to PMS. They are being promoted
under the CIN.
International Collaboration
Under TREAT-NMD (Headquarters located in New Castle University in the UK), we havepursued the establishment of TREAT-NMD Global Registry. Since the inauguration, TREAT-NMD
68

has been promoting the establishment of a global-scale patient registration database on Duchenne
muscular dystrophy (DMD) and spinal muscular atrophy (SMA). Since TREAT-NMD started its
activities, the establishment of new patient registration databases has been promoted in Europeanand Asian countries including Japan2. With operation of patient databases in relevant countries as thecore activity, through administration under the charter (Charter for TREAT-NMD Patient
Database/Registry), annual conference with the conference body consisting of the representative ofdisease registry in relevant countries [TREAT-NMD Global Database Oversight Committee(TGDOC)], unification of registration items, and training seminar for those involved in database in
relevant countries (TREAT-NMD Curator’s Training Course), disease registration has beenpromoted. Remudy, the national registry in Japan, has been collaborating with TREAT-NMD for its
activities since its establishment, dispatching TGDOC members to serve as members.


Use of registry under the pharmaceutical affairs system
Remudy has fulfilled its functions, e.g., conduct of medical researches, collaboration with patients,

and provision of Trial-Ready Cohort (cohort to promote clinical trials/clinical research, which is usefulfor field survey, formulation of research plan, and patient recruitment)2. While the use of RWD and
RWE for drug development is being discussed in recent years, TREAT-NMD is actively examining
the registries comparable to Trial-Ready Cohort at the beginning of establishment, following natural
history study, and PMS. At the TREAT-NMD conference in December 2019 and TGDOC conference,the common data set was discussed and the pilot study was introduced, these activities are based onearly collaboration with pharmaceutical companies. Regarding DMD in Japan, prospective researchon natural history3 is ongoing since a few years, expecting utilization as the control group of clinical
trials. For DMD, Viltepso® Injection 250 mg (non-proprietary name: viltolarsen) was approved under
the conditional early approval system. Risk Management Plan (RMP) for Viltepso describes the survey
using a registry in the pharmacovigilance plan and surveys and trials regarding efficacy. Recently, a
registry was initiated for PMS, under the name of Remudy-DMD4. As a prospective multicenter
observational study in which all the DMD patients can be registered, registration items that can becollected in post-marketing clinical settings are established, and the reliability sought by possessors in
the utilization of registry as medical information database in GPSP is ensured, taking quality
management into consideration. A registry suitable for the purpose should be established on the basisof the fit-for-purpose quality control and design according to the planning of quality control (quality
by design). These new efforts include reconstruction of Remudy based on the quality control under
the pharmaceutical affairs system, in addition to the patient-based registration so far.
Collaboration with other medical information
In

Japan,

the AMED

ClinicalResearch/Clinical Trial

Promotion

Research

Project(18lk0201058h0003), on the electronic chart system of medical institutions participating in the
research group (4 institutions), created the Remudy registration form template on a trial basis. Entry69

in the template enabled storage of Remudy registration information in medical charts and extractionof registration information in SS-MIX2 extension storage. However, it appeared difficult to expand
this system into the institutions participating in Remudy in Japan at this time owing to maintenance ofregistration template, challenges in making changes to registration form, maintenance of data server,
and the associated cost, as well as the difference in policy regarding medical information among
institutions.
References
1.

Bladen CL, Salgado D, Monges S, et al. (2015). The TREAT-NMD DMD Global Database:analysis of more than 7,000 Duchenne muscular dystrophy mutations. Human mutation, 36, 395-402.
2.

Nakamura H, Kimura E, Mori-Yoshimura M, et al. (2013). Characteristics of Japanese Duchenne

and Becker muscular dystrophy patients in a novel Japanese national registry of muscular dystrophy(Remudy). Orphanet journal of rare diseases, 8, 60.
3. The natural history study on ambulatory patients with Duchenne muscular dystrophy(UMIN000030562)
4. New National Patient Registry for Duchenne muscular dystrophy (Remudy-DMD)(UMIN000040768).
7.2. Efforts in the world
In the field of medical devices, the Medical Device Epidemiology Network public-private
partnership (MDEpiNet-PPP) published a proposal in August 2015. In this proposal, eight priority
issues were identified for consideration in order to shift from “clinical trial + post-marketing safetysurveillance” to “continuous evaluation through the total product life cycle (TPLC)”1. These are (a)continuous evaluation, (b) minimum essentials, (c) strategic collaboration, (d) nurturing reliability, (e)step-wise practice, (f) sustainability, (g) methodology development, and (h) international
reconciliation. Toward solution of these examination items, establishment of a strategic registrycollaboration network was proposed, and several research studies have been initiated.
The subsequent International Medical Device Regulators Forum (IMDRF), Patient Registry
Working Group prepared the written guidance regarding the (1) definition of medical device registry
and collaboration and (2) methodology of medical device registry collaboration. In a document, known
as Document N332, that describes definitions, the definition and requirement for medical devices aredescribed and the medical device registry that was addressed back then is evaluated. The following
are cases of best practice that results in the maximum in both patient registration and patients in the
future: Beyond Compliance (https://www.beyondcompliance.org.uk/) and percutaneous aortic-valve
replacement registry (US-TVTR) by American College of Cardiology and the Society of Thoracic
Surgeon. Both were assessed as serving as a bridge between pre-approval and post-marketing safety
70monitoring.
As an item of (2), listed above, so-called Document N423 was published in March 2017. In thisdocument, the methodology is expected to contribute to discovery of important evidence through longterm evaluation/rare event evaluation and research by comparing the efficacy, by viewing the entire
TPLC and recognizing frequently mentioned problems such as biases and confounding inobservational studies. In particular, the document specified the following approaches for discovering
signals detected from several data sources: (a) cumulative sum (CUSUM), (b) cumulative rereplacement ratio (CRRT), and (c) funnel plot. This indicates the importance of improvement in thequality of safety evaluation. Document N464, published in March 2018, indicated evaluation
requirements for regulatory authorities to decide on a registry as RWD, according to the examination
purposes.
The document lists the Japanese Registry for Mechanically Assisted Circulatory Support (J-MACS)
as a global registry collaboration; this registry was established in 2010 through industry–government–
university cooperation in collaboration with the global Interagency Registry for Mechanically Assisted
Circulatory Support (INTERMACS)5. J-MACS uses evaluation criteria and survey items comparable
to those in the INTERMACS in the US, taking future integration with INTERMACS into
consideration. Consequently, it can contain sufficient information for evaluating the efficacy ofdevices for patients with artificial heart, as well as the safety of artificial heart using competing risk
analysis. The results were summarized as the first report in 20176. In Japan, the duration of use ofdevices is different from that overseas because of the difference in the frequency of transplantation
between Japan and other countries; however, “safety issues due to problems such as device failure aresimilar between Japan and overseas countries,” and “the devices function without problems duringlong-term use.” This example indicates that medical devices can be used across the world despite thedifference in transplantation frequency.
Regarding medical products, FDA and Brigham and Women’s Hospital in the US published in April
2019 a confirmatory project7 using RWE to predict the results of an ongoing phase IV trial (RCT).
This project predicted the trial results by using RWD such as insurance database and electronic medical
records before completion of the phase IV trial (RCT) and before a conclusion was reached. The results
were verified against the RCT results that were obtained later for comparing the accuracy of theestimation. The interim results were to be reported in mid-2019, and the final results were to bereported at the end of 2020. Through the use of RWD in collaboration with commercial companies,
FDA proactively verifies whether RWD can be used as substitution for RCT data. If favorable results
are obtained, the active use of RWE under a specific condition for approval application may becomea trend, including the use of hybrid control in an RCT.
References71

1.

Krucoff MW, Brindis RG, Mack MJ, Holmes DR, Medical Device Innovation. (2012).

Prospective Solutions for an Ecosystem in Crisis. Adding a Professional Society Perspective. J Am
Coll Cardiol Intv, 5: 790-796.
2.

IMDRF. Principles of International System of Registries Linked to Other Data Sources and Tools.

http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-160930-principles-system-registries.pdf.
(Accessed 03/15, 2021)
3.

IMDRF. Methodological Principles in the Use of International Medical Device Registry Data.

http://www.imdrf.org/docs/imdrf/final/consultations/imdrf-cons-methodological-principlesimdrd.pdf. (Accessed 03/15, 2021)
4.

IMDRF. Tools for Assessing the Usability of Registries in Support of Regulatory Decision-

Making.

http://www.imdrf.org/docs/imdrf/final/consultations/imdrf-cons-registries-n46-pd1-

170817.pdf (Accessed 03/15, 2021)
5.

Ishii K, Tawaragi T. (2015). Post-marketing safety measures of medical devices –Post-marketingsurveillance and registry–. Regulatory Science of Medical Products, 5(1): 73–80.
6.

Nakatani et.al. (2017). Japanese registry for Mechanically Assisted Circulatory Support: First

report, Heart Lung Transplant, 36(10):1087-1096.
7.

RCT DUPLICATE project https://www.rctduplicate.org/ (Accessed 09/10, 2020)

7.3. Issues in conjunction with other databases
Problems in granularity of information
The granularity of information refers to the extent of detail within the information. For example,

considering a disease name, “lung cancer” is classified into “non-small cell lung cancer” and “small
cell lung cancer,” and “non-small cell lung cancer” is further classified into “adenocarcinoma,”“squamous cell carcinoma,” and “large-cell carcinoma.” Information classified into small categories
is called fine granular information, while information with rough categorization such as “lung cancer”
is called coarse granular information.
In the case of general disease databases that collect information from CRF entry and manual entry in
EDC WEB forms, considering the balance between the quality of collected data and burden ofcollection, data specific to the disease area are often collected with fine granularity, and data positioned
as associated information have coarse granularity.
If considering integration and collaboration of several databases, data with coarse granularity are
selected, and fine granular data information is lost. Considering the example of lung cancer, to convert
fine granular data into coarse granular data, knowledge of the classification of lung cancer, as
mentioned earlier, is required. To automatize this process, a thesaurus is needed that describes thisknowledge in a form that a computer can handle.
Problems in the form of information72

For collaboration of different databases, it is necessary to understand in what form the same data
are stored in the individual databases. Considering blood pressure, for instance, the definition ofindividual databases has to be confirmed as follows: whether the values are entered in one field,
divided with a slash (/), for example, “126/80” (other delimiters may be used) or each value is enteredin a different field, for example, “126” and “80” and whether the value “126” is handled as a numericalvalue (integer) or a letter (string) in the database. The same data in a different form of information
require the introduction of a neutral conversion system according to the difference, for datacollaboration.
Problems in codes
To integrate or link data in databases, the code of individual data is usually used as the key.

Conjunction and integration are easy if the same code is used for the databases to be linked. However,if the codes are different, a conversion table should be prepared. Regarding the code of medical careinformation, extra precaution is required for the use of electronic chart data. Different prescription
codes and testing codes, even with different digit numbers, are used among individual medical
institutions. Even if the electronic chart is provided by the same vendor, different codes are usuallyused for the same drug or the same testing. Under such a circumstance, the MHLW has established thestandard specification for the field of health and medical information and designated the standard
codes to be used1. However, they are not widely used in many medical institutions.


Problems in measurement method and reference values
When test data are collected from several institutions for establishing a disease registry, the samemeasurement method is not always adopted by all institutions. Conversion of values from different
measurement methods should be considered. For integrating databases, it is necessary to confirm themeasurement method (or whether several different measurement methods coexist) that each database
adopts as the criteria.
Even if the same measurement method is adopted, the reference values are different depending on thetype and characteristics of the testing device. For collecting test data from multiple institutions, theprocedure for handling these reference values needs to be considered. Similarly, for integrating
different databases, it is necessary to examine the handling including whether to take the reference
values into consideration.
Problems in conjunction with electronic charts
Medical care data stored in electronic charts can be retrieved by various methods2. To link them, it

is necessary to deal with several tasks.
Electronic chart data are stored in the granule size optimized for ordering function of the order entry
system and the access function of the electronic chart. The granule size is usually smaller than CRF
and Web-based EDC and is often too small for use in general clinical research. In case of drugs, forexample, different codes are used for each non-proprietary name and brand name, and even for the73

same brand name, different codes are used if the dosage forms and specifications (e.g., 5 mg and 10mg) are different. Using these codes for clinical research is difficult unless these codes are collected
and units converted.
By paying attention to the structure of data used in electronic charts, the data can be classified into
structured and unstructured data. The structured data are mainly computerized information that is used
as “slips” in the clinical settings. The information included in the “slips” is distinctively itemized.
Examples are the basic information, prescription information, and testing information of patients. In
contrast, clinically important data that represent unstructured data are descriptions in medical charts.
Generally, descriptions in medical charts form sentences (atypical, natural sentences), and the analysisof these sentences requires such processing as natural language processing. Database compilation anddata analysis of structured data are easier than that of unstructured data. However, because theircharacteristics are specific to each electronic chart vendor, it is impossible to link electronic chart
databases in their original forms. For this reason, as a scheme for utilizing data commonly in all
medical institutions without depending on electronic chart vendors, a specification unique to Japan,
which is called SS-MIX24, was developed on the basis of the international standard specifications3 for
HL7, communication between medical information systems (comparable to electronic transmission ofslips), and it is being introduced in many medical institutions. Most structured data in electronic charts
are stored in the SS-MIX standard storage. Among them, medical care data are stored in a common
granularity, structure, and code system*1, and it is possible to use and link medical care information
without depending on electronic chart vendors.
In contrast, two methods are used for unstructured data such as chart descriptions in electronic charts.
One method is, as previously stated, extracting necessary information from the descriptions inelectronic charts by natural language processing and using the information. Owing to the recent
progress of artificial intelligence, natural language processing is advancing rapidly, and its application
is expected. However, in reality, sentences in medical charts contain a large volume of technical terms
and abbreviations with more than one meaning that is different depending on the clinical department(or medical institutions in some cases). Additionally, the sentences are prosaic and more difficult than
general sentences, requiring a background of medical knowledge. The second method is framing theentry of the medical records into a structure. Similar to EDC entry via the Web, predetermined entry
form is prepared as a template and installed in electronic charts, and a medical record is completed byusing the template. Since each item in the template is stored in structure in the database, data utilizationand conjunction are easy. A typical example of a disease registry adopting this method is the Japan
Diabetes compREhensive database project based on an Advanced electronic Medical record System(J-DREAMS)5,6, a large-scale, multicenter, diabetic mellitus registry.
*1: However, as explained in the section on problems in codes, the standard codes designated by the
MHLW are not adopted by many medical institutions, and data collectors may need some arrangement.
74

References
1.

Reference, Ministry of Health, Labour and Welfare. Promotion of information technology inmedical field.
https://www.mhlw.go.jp/stf/seisakunitsuite/bunya/kenkou_iryou/iryou/johoka/index.html(Accessed 09/20, 2021)

2.

Miyo K. (2018). Sharing electronic health record data with clinical researches: Present status andissues. Journal of Clinical and Experimental Medicine, 264(6): 549–554.

3.

HL7 Japan. http://www.hl7.jp/index.html (Accessed 09/20, 2021)

4.

Consortium for SS-MIX Dissemination and Promotion. http://www.ss-mix.org/cons/ (Accessed
09/20, 2021)

5.

Japan Diabetes compREhensive database project based on an Advanced electronic Medical
record System. http://jdreams.jp/ (Accessed 09/20, 2020)

6.

Miyo K. (2020). J-DREAMS (Japan Diabetes compREhensive database project based on an
Advanced electronic Medical record System). Precision Medicine, 3(9): 80–807.

75

8. Issues to consider for implementation
8.1. Data extraction
In the analysis of registries and RWD that surrounds them, there are two phases, i.e., extraction oftarget population from the data and statistical analysis on extracted population. This section describes
issues to consider for databases. Statistical analysis software is described in Appendix.
Data are submitted in various forms. Comma-separated value (CSV) file is used for gener
al purposes because this file form is accessible in many types of software. However, as indicated, “there is no formal specification in existence” for the CSV format in RFC4180 (https://datatracker.ietf.org/doc/html/rfc4180), note that the file format includes several variations. In
the case of database using Japanese language, mis-conversion or missing data may occur bec
ause of the letter code. The analysis data should be prepared after paying attention to thesepoints.
Regarding the statistical analysis stated in this report, cases where none of the target registry data
are used can be expected. For statistical analysis on part of data extracted according to the purpose of
research, it is more efficient to use languages specific to the database such as SQL language and thelatest Python language for data extraction. For ensuring the reliability of analysis results, storing the
program used before analysis is necessary, and recording the version of the language used for theoperation is desirable.

8.2. Conjunction with other databases
Linkage between databases is considered to be effective when data for a given purpose can beobtained from other databases. Since registries are defined in various ways, it is obvious that the
granularity and precision of information are different1. However, there is a possibility that owing to
the rapid advancement of digital technology, signals on performance and safety of medical device andsignals of long-term prospect on patient prognosis can be detected from several data sources. For
example, receipt data, the use of which is increasing recently, can be used for obtaining information
on insurance subscribers such as testing, procedures, and materials in a specific period (visiting morethan one hospital). Limitations of databases are variation in the nature of insurance associations (suchas occupation) that are the data source and unregistered laboratory data. Attention needs to be paid
regarding whether the purpose is different among registries, such as between registries targeting
patients and registries of devices and procedures.
In case of conjunction, the inconsistency of precision and granularity of the information among
registries or with other databases must be evaluated in advance, and the research questions according
to the evaluation must be formulated. For example, the following are expected:
・Target of registry, database (type, purpose of registry, and age range)
・Use of standardized data dictionary (ICD-9/10, …, UDI)
76

・Precision/granularity of information (even with the same item name, the content may be different)
・Possibilities for accessing the source of data (even partial)
If an identifiable code such as insurer number (medical ID at a specific hospital, if provision ispossible) is available, although personal information needs to be considered, direct access to data may
become possible. If it is not possible, a probabilistic method of collation with common items is being
proposed; however, a standardized methodology has not been established.
In recent years, as reported by Dhruva et al (2020), cases of analysis have used both receipt data
and registry data, and progress is seen regarding experience in supplementing information from datawith different accuracy and linking it to decision making.
References
1.

Guide for preparation and use of registry. https://cinc.ncgm.go.jp/?p=67. (Accessed 2021/07/27)

2.

Dhruva SS, Parzynski CS, Gamble GM, et al. (2020). Attribution of adverse events following
coronary stent placement identified using administrative claims data. J Am Heart Assoc, 9(4),e013606.

77

Appendix．
A1. Proposal based on STROBEThe STROBE checklist outlines the contents to be included in reports on observational studies. If
RWD are used as external controls in a clinical trial aiming at approval application, the primary partof the study is an interventional study, and it is typical to use CONSORT as the reporting method. In
contrast, data used as external controls may be collected in a form of non-interventional research.
Since contention specific to observational studies should also be considered in reporting, contents to
be included, in addition to 4.4.2, have been summarized by referring to the STROBE checklist.
1) Methods(1) “Study design” section
The STROBE checklist presents important factors in study design. Similar to the proposal based on
CONSORT, the reason for the use of external control of RWD should be described. Additionally, it isimportant to discuss the difference in characteristics of these data and their impact regarding whether
the RWD collection method is prospective or retrospective and the time of data generation andacquisition.
(2) “Setting” section
The STROBE checklist clearly shows the content of data collection, location of data collection anddata on reference dates such as period of registration, exposure, and follow-up/period of data collection.
To discuss from the viewpoint of an observational study, it is necessary to indicate the purpose ofcreation of the RWD database used as an external control, the process of creation, and its record. It is
desirable to describe in detail the extraction algorithm for the data used in analysis and the data
integration method including data conversion, while discussing the EDC used for database
construction and the software used for data integration.
(3) “Participants” section
The STROBE checklist presents the patient eligibility criteria and parent population of the study
and reports the matching criteria for research involving matching. These contents are indicated bycohort study and by case-control study, and a checklist for each study design is prepared. Required
additional discussions include the analysis set. Regarding the use of RWD as external controls, cases
where it is difficult to adopt the definition of analysis set prescribed by the protocol of a single-arm
trial as it is (e.g., variables required to identify the analysis set were not measured in RWD) are
expected. If such a difference in analysis sets between a single-arm trial and the external control occurs,
it is necessary to describe the details of the difference and the impact on the analysis results.

78

(4) “Variables” section
The STROBE checklist is characterized by detailed descriptions such as all analysis methods
including the analyses considering potential biases and handling of missing data, which are presented
in more detail than the CONSORT checklist. Required additional discussions include the definition ofevaluation variables that are necessary for the calculation of endpoints (definition of events and cutoff).
As explained in the section, “Participants”, when RWD is used as external control, cases where it isdifficult to adopt the definition of evaluation variables prescribed by the protocol of a single-arm trial
as it is (e.g., efficacy data are not collected in line with the same criteria/schedule as those of theclinical trial) are expected. As previously mentioned, it is also highly expected that data necessary forevaluation of endpoints matched with those in the single-arm trial do not exist in the RWD database
to be compared. It is also expected that the method and period of data follow-up for evaluation
variables of interest are different between the single-arm trial and external control. If such differences
between the single-arm trial and external control occur, it is necessary to describe the details of andthe impact on the analysis results.

2) Results(1) “Participants” section
The STROBE checklist includes details of the number of patients at each stage of research (thenumber of potential target patients, actual target patients, eligible patients, patients to be followed, andpatients in analysis set) and a visual depiction of the patient flowchart, indicating that the content of
the list is more substantial than the CONSORT checklist. When RWD are used as an external control
for a single-arm trial, it is necessary to prepare a flowchart and describe in detail the definition ofindividual flows since the generation process of individual data (purpose of collection, data generation,and timing of collection) is different.

3) Discussion(1) “Limitations” section
In the STROBE checklist, the checklists are created by focusing on the potential biases and the
problem of deterioration in the accuracy of estimation during analysis. Although the content ispractically the same as that in the CONSORT checklist, it is necessary to consider the possible
limitations and to discuss separately clarified issues and the ambiguous issues clarified in the research,on the basis of the issues discussed in sections such as Methods and Results.

A2. Points to consider for reporting in approval application documents
In previous sections, we explained the form of reporting with respect to academic papers. At the stageof preparation of application documents for regulatory approval, more detailed information is79

generally required than that at the level of academic papers. The points to consider in the descriptions
in each section of CTD are basically picking up the limitations of the less frequently used study design
and adding discussions on the uncertainty of the conclusion to be led. For the summary of individual
trials, and items to be included in individual sections of CTD, it is important to present the informationthat can be evaluated by a third party regarding the discussions of these limitations indicated above.
We explain them according to the chapters of CTD in the Notification from the Director of thePharmaceutical Evaluation Division, Pharmaceutical Safety and Environmental Health Bureau,
MHLW (2017), “Annex 5 (CTD – Guidelines for preparation of clinical documents).”6 It should beconsidered that the chapter or section that indicates the results of integrated analysis may be changed
depending on whether the integrated analysis on external controls extracted from RWD and the singlearm trial was conducted under almost the same environment as the clinical trial or conducted asassociated academic research.
[1] “2.5.4 Overview of efficacy” and “2.5.5 Overview of safety”
These sections comprise chapters stating the summary results on efficacy and safety, respectively,on the basis of the analysis results. The results of integrated analysis with RWD, in addition to theresults obtained from a usual clinical trial (single-arm trial in this context), are to be described in these
sections. The purpose of the use of RWD and study design should be described clearly. The details
regarding treatment in the control group that would have been set if a RCT was conducted, should also
be mentioned, and it is necessary to discuss the difference in actual treatment in the external control
extracted from RWD, the difference in patient background, time of data collection, and items forcollection. It is necessary to discuss the impact of these differences on the analysis results. Additionally,
the above points should be described in detail in “2.7.3 Summary of clinical efficacy” and “2.7.4
Summary of clinical safety.”[2] “2.7.6 Synopses of individual trials”
This section, in addition to the summary described in the clinical trial, requires a discussion on themethod for selecting target patients, method for selecting variables, method for ensuring comparability,
statistical analysis policy, and their decision process in the use of registry data.
[3] “5.3.5 Reports of efficacy and safety studies”
This section, in addition to the results obtained from the clinical trial, similar to other clinical study
reports, should describe the existence of trials other than the trials related to the indication, such asreports on the results of interim analysis and ongoing trials.

A3. Implementation procedure for the introduced methods
Section 5.3 described issues to consider for statistical analysis in RWD analysis and introduced several
methods. Statistical analysis is usually performed using software. This section presents documents on
specific procedures for implementing the statistical methods introduced in Section 5.3.
80

First, the implementation of confounding adjustment method in observational studies is
introduced. Representative methods use the propensity score. The propensity score can becalculated by SAS, using the LOGISTIC procedure.
Section 2.6.6 of the “Introduction of pharmacoepidemiology: the design and analysis” discusses the
specific SAS program and examples of analysis1. The PSMATCH procedure may be useful for
propensity score matching. For IPW analysis by the propensity score, the CAUSALTRT procedure
may be helpful. These procedures are demonstrated in Section 4.1.7 of the “Statistical method toestimate treatment effect by using data of a non-RCT.”2 Furthermore, the CAUSALTRT procedureenables implementation of not only the simple IPW method but also adjustment of confounders usinga regression model, as well as the combination of these methods, the augmented IPW (AIPW) method.
“Causal inference by SAS: Introduction of CAUSALTRT procedure.”3 explains in detail theimplementation of these procedures. The author presents an example of implementation for theestimated risk ratio adjusted by a regression model for outcome, adjusted by the IPW method using
the propensity score, or adjusted by both, which is introduced in page 67 of “Causal inference by SAS:
Introduction of CAUSALTRT procedure”: https://github.com/t-yui/causaltrtRiskRatioCI. A method
for adjusting the impact of unmeasured confounders, the instrumental variables method, was presented
in Section 5.4. Regarding the implementation of instrumental variables method by the two-stage leastsquares method, the implementation method by SAS or R is explained in Section 4.2.2 of the“Statistical method to estimate treatment effect by using data of a non-RCT.”2 The implementation ofgeneralized

methodsofmoments(GMM)

isdescribed

intheofficial

homepage(https://www.lightstone.co.jp/stata/stata11_gmm.html).
In the analyses of data from observational studies, including RWD, there are usually unadjustablebiases such as those due to unmeasured confounders, sampling of subjects, and misclassification of
exposure and outcome. The evaluation of impact of these biases is called sensitivity analysis orquantitative bias analysis. Simple examples of sensitivity analysis are introduced in “Recommendationof sensitivity analysis in observational study: Introduction”4. Furthermore, the “Recommendation ofsensitivity analysis in observational study: Practice” discusses theoretical supplementations forsensitivity analysis. It additionally demonstrates probabilistic sensitivity analysis, which is frequently
performed in actual settings5.
References
1.

Japan Pharmaceutical Manufacturers Association. Introduction of pharmacoepidemiology: thedesign and analysis –for biostatisticians working for clinical development divisions ofpharmaceutical companies.
http://www.jpma.or.jp/information/evaluation/publishing_center/pdf/015.pdf (Accessed 07/27,
2020)
81

2.

Japan Pharmaceutical Manufacturers Association. Statistical method to estimate treatment effectby using data of a non-randomized clinical trial.
http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/non-randomization_01.pdf(Accessed 07/27, 2020)

3.

Taguri M. Causal inference by SAS: Introduction of CAUSALTRT procedure.
https://www.sas.com/content/dam/SAS/ja_jp/doc/event/sas-user-groups/usergroups2017-a-04.p
df (Accessed 07/27, 2020)

4.

Japan Pharmaceutical Manufacturers Association. Recommendation of sensitivity analysis inobservational study: Introduction.
http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/sensitivity_analysis.pdf(Accessed 07/27, 2020)
Japan Pharmaceutical Manufacturers Association. Recommendation of sensitivity analysis inobservational study: Practice.
http://www.jpma.or.jp/medicine/shinyaku/tiken/allotment/pdf/sensitivity_analysis_practice.zip.
(Accessed 07/27, 2020)

82

The authors express their sincere gratitude to Mr. Teruhiko Miyazaki, Ms Yuko Iwata, and Ms. Mayumi
Tsukiyama for their excellent assistance in performing the JOINT 04 protocol and preparing the manuscript.

83

List of authors and reviewers
Corresponding author (index based on the kana syllabary order)
Name

Affiliation

Area of responsibility

Asakura Koko

National Cerebral and Cardiovascular Center

Section 4.1, 4.2

Ishiguro Chieko

National Center for Global Health and Medicine

Chapter７introduction

Uemura Yukari

National Center for Global Health and Medicine

Chapter１,２, 3

Ohtsu Hiroshi

National Center for Global Health and Medicine

Section 7.2, 8.1, 8.2

Omae Katsuhiro

National Cerebral and Cardiovascular Center

Section 4.1, 4.2

Kinoshita Tomoyasu

National Center for Geriatrics and Gerontology

Section 7.1.2

Shibata Taro

National Cancer Center Japan

Chapter 7 introduction

Shinozaki Tomohiro

Tokyo University of Science

Section 4.3.1, 4.3.2, 4.3.4

Tachimori Hisateru

National Center of Neurology and Psychiatry

Section 4.3.1, 4.3.2, 4.3.4, A3

Nakamura Harumasa

National Center of Neurology and Psychiatry

Section 7.1.3

Nomura Shogo

The University of Tokyo

Section 4.4, 4.5, 5, A1, A2

Hirakawa Akihiro

Tokyo Medical and Dental University Hospital

Section 4.3.3

Masashi Mikami

National Center for Child Health and Development

Chapter 6

Miyo Kengo

National Center for Global Health and Medicine

Section 7.3

Naho Morisaki

National Center for Child Health and Development

Section 7.1.1

Wakabayashi Masashi

National Cancer Center Japan

Section 4.4, 4.5, A1, A2(Affiliation as of March 2022)
Author Collaborators (Review) (index based on the kana syllabary order)
Name

Affiliation

Oi Hideki

National Center of Neurology and Psychiatry

Oba Mari

National Center of Neurology and Psychiatry

Kawashima Takahiro

National Center of Neurology and Psychiatry

Tohru Kobayashi

National Center for Child Health and Development

Suzuki Keisuke

National Center for Geriatrics and Gerontology

Shimizu Yosuke

National Center for Global Health and Medicine

Tsujimoto Masashi

National Center for Geriatrics and Gerontology

Pak Kyongsun

National Center for Child Health and Development(Affiliation as of March 2022)

84

Acknowledgments
We would like to express our sincere appreciation to the members of the Japan Pharmaceutical
Manufacturers Association and the members of Pharmaceuticals and Medical Devices Agency for theirvaluable comments and advice in the process of preparing this report.

85