# Clinical Trial Validity Guidance from the HTACG: Looking for Chicken Teeth

Mondher Toumi, Bruno Falissard, Asma Jouini, Samuel Aballéa, Laurent Boyer

► To cite this version:

Mondher Toumi, Bruno Falissard, Asma Jouini, Samuel Aballéa, Laurent Boyer. Clinical Trial Validity Guidance from the HTACG: Looking for Chicken Teeth. Journal of Market Access & Health Policy, 2025, 13 (2), pp.15-15. 10.3390/jmahp13020015. hal-05059750

HAL Id: hal-05059750

https://hal.science/hal-05059750v1

Submitted on 20 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Editorial

# Clinical Trial Validity Guidance from the HTACG: Looking for Chicken Teeth

Mondher Toumi <fnref n="1" />*, Bruno Falissard <fnref n="2" />, Asma Jouini ³ID, Samuel Aballéa ⁴ID and Laurent Boyer ¹ID

The Member State Coordination Group on Health Technology Assessment (HTACG) guidance on the validity of clinical studies [1] presents an overly simplistic view of the relationship between internal and external validity in clinical trials. This approach overlooks the well-established trade-off between these two dimensions of validity, which has been extensively documented in the literature [2–4]. The guidance’s suggestion that internal validity, external validity, and statistical precision are independent dimensions that can all be maximized simultaneously in a single trial is problematic [1]. This view contradicts the widely accepted understanding in clinical research methodology, as illustrated by tools like PRagmatic Explanatory Continuum Indicator Summary-2 (PRECIS-2) [5,6], which conceptualizes trials along a continuum from explanatory (high internal validity) to pragmatic (high external validity).

Several factors may have contributed to this misunderstanding:

1. The epistemological debate on the internal–external validity trade-off or prerequisite [2–4].

2. A narrow conceptualization of external validity [7].

3. The misinterpretation of internal validity as restricted to the absence of systematic error, which only represents one facet (although critical) of internal validity [3,8–10].

The guidance's approach fails to acknowledge the reality of clinical trial design, where researchers must often sacrifice external validity to achieve high internal validity [2,6,11,12], particularly in regulatory trials. This is evident in the high exclusion rates seen in many randomized controlled trials (RCTs), which can significantly limit their generalizability. For example, clinical trials in asthma exclude 95% of patients [13], while trials in Chronic Obstructive Pulmonary Disease (COPD) exclude 83% [14]. A large-scale analysis of external validity, covering 43,895 trials and 5,685,738 individuals across 989 unique drugs and 286 conditions, found that the median RCT population exclusion proportion was 81.5%, leaving RCTs with very low external validity [15]. Moreover, the HTACG guidance overlooks more nuanced approaches to balancing internal and external validity, such as the estimand framework recently adopted by the European Medicines Agency (EMA) [16]. This framework offers a more sophisticated way to reinforce internal validity and address external validity concerns in trial design and analysis, particularly by considering intercurrent events from different perspectives. However, it was only briefly mentioned in the guidance on multiplicity issues without reference to external validity [17].

The absence of discussion on pragmatic trials and estimands in the guidance is surprising, especially given the historical contributions of French academics to the field of

<figure><img src="image_2.png" /></figure>

check for
updates

Received: 19 February 2025

Accepted: 25 February 2025

Published: 16 April 2025

**Citation:** Toumi, M.; Falissard, B.; Jouini, A.; Aballéa, S.; Boyer, L. Clinical Trial Validity Guidance from the HTACG: Looking for Chicken Teeth. *J. Mark. Access Health Policy* **2025**, *13*, 15. https://doi.org/10.3390/jmahp13020015

**Copyright:** © 2025 by the authors. Published by MDPI on behalf of the Market Access Society. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

<footnote n="1">¹ CEReSS/UR3279—Health Services Research and Quality of Life Center, Aix-Marseille University, 13385 Marseille, France; laurent.boyer@univ-amu.fr</footnote>

<footnote n="2">² CESP, INSERM U1018, Université Paris-Saclay, 94800 Villejuif, France; bruno.falissard@gmail.com</footnote>

<footnote n="3">³ Clever Access, Tunis 1053, Tunisia; asma.jouini@clever-access.com</footnote>

<footnote n="4">⁴ Inovintell, 3023 GJ Rotterdam, The Netherlands; sab@inovintell.com</footnote>

<footnote>* Correspondence: mto@inovintell.com</footnote>
pragmatic trials [18], while less surprising for estimand, given the German resistance to integrating estimand in the HTA framework [19]. A more comprehensive literature review, as recommended by evidence-based medicine principles referenced in the EU-HTA regulation, could have helped avoid these oversimplifications and provided a more accurate representation of the complexities involved in assessing clinical trial internal and external validity [20,21].

A review of the guidance is needed to

1. Acknowledge the need to balance internal and external validity.

2. Ensure that the estimand framework enhances external validity, given regulators' preference for internal validity in new interventions.

3. Require the use of PRECIS-2 to assess the balance between internal and external validity.

4. Introduce pragmatic trials and discuss their associated uncertainty due to lower internal validity [22].

5. Address adaptive design trials, as they allow starting with a very narrow population with high internal validity and expanding to a broader population, thus enhancing external validity [23].

**Author Contributions:** M.T.: Conceptualized the content and wrote the first draft of the manuscript. The co-authors: B.F., A.J., S.A. and L.B. Challenged the concept, edited the manuscript, and refined arguments for clarity and coherence. All authors have read and agreed to the published version of the manuscript.

**Conflicts of Interest:** S.A. is employee of company Inovintell. A.J. is employee of Clever Access Tunisia. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

## Abbreviations

The following abbreviations are used in this manuscript:

COPD Chronic Obstructive Pulmonary Disease

EMA European Medicines Agency

EU-HTA European Health Technology Assessment

HTA Health Technology Assessment

HTACG Member State Coordination Group on Health Technology Assessment

PRECIS Pragmatic Explanatory Continuum Indicator Summary

RCT Randomized Controlled Trial

## References

1. HTA Coordination Group (HTACG). Guidance on the Validity of Clinical Studies for Joint Clinical Assessments. V1.0. 4 July 2024. Available online: https://health.ec.europa.eu/document/download/9f9dbfe4-078b-4959-9a07-df9167258772_en?filename=hta_clinical-studies-validity_guidance_en.pdf (accessed on 30 December 2024).

2. ASH Clinical News. Who's in and Who's Out? Available online: https://ashpublications.org/ashclinicalnews/news/2774/Whos-In-and-Who-s-Out (accessed on 14 January 2025).

3. Busch, C. The relationship between internal and external validity. *Rerum Causae* **2017**, *9*, 71–91.

4. Jimenez-Buedo, M.; Miller, L.M. Why a trade-off? The relationship between the external and internal validity of experiments. *Theoria. Rev. Teoría Hist. Fundam. Cienc.* **2010**, *25*, 301–321. [CrossRef]

5. Loudon, K.; Treweek, S.; Sullivan, F.; Donnan, P.; Thorpe, K.E.; Zwarenstein, M. The PRECIS-2 tool: Designing trials that are fit for purpose. *BMJ* **2015**, *350*, h2147. [CrossRef] [PubMed]

6. Oregon Health & Science University (OHSU). Optimizing Inclusion and Exclusion Criteria in Clinical Research. 2019. Available online: https://www.ohsu.edu/sites/default/files/2019-07/2019-04-18%20Incl%20Excl%20Criteria%20and%20Pragmatic%20clinical%20trial%20slides.pdf (accessed on 3 January 2025).

7. Roe, B.E.; Just, D.R. Internal and external validity in economics research: Tradeoffs between experiments, field experiments, natural experiments, and field data. *Am. J. Agric. Econ.* **2009**, *91*, 1266–1271. [CrossRef]
8. Cahit, K. Internal validity: A must in research designs. *Educ. Res. Rev.* **2015**, *10*, 111–118.

9. Slack, M.K.; Draugalis, J.R., Jr. Establishing the internal and external validity of experimental studies. *Am. J. Health Syst. Pharm.* **2001**, *58*, 2173–2181. [PubMed]

10. Collaboration for Environmental Evidence (CEE). Critical Appraisal of Study Validity. Available online: https://environmentalevidence.org/information-for-authors/7-critical-appraisal-of-study-validity/ (accessed on 2 January 2025).

11. Food and Drug Administration (FDA). Evaluating Inclusion and Exclusion Criteria in Clinical Trials. 2018. Available online: https://www.fda.gov/media/134754/download (accessed on 3 January 2025).

12. Scribbr. Inclusion and Exclusion Criteria | Examples & Definition. 2022. Available online: https://www.scribbr.com/methodology/inclusion-exclusion-criteria/ (accessed on 5 January 2025).

13. Herland, K.; Akselsen, J.P.; Skjønsberg, O.H.; Bjermer, L. How representative are clinical study patients with asthma or COPD for a larger “real life” population of patients with obstructive lung disease? *Respir. Med.* **2005**, *99*, 11–19. [CrossRef] [PubMed]

14. Scichilone, N.; Basile, M.; Battaglia, S.; Bellia, V. What proportion of chronic obstructive pulmonary disease outpatients is eligible for inclusion in randomized clinical trials? *Respiration* **2014**, *87*, 11–17. [PubMed]

15. Tan, Y.Y.; Papez, V.; Chang, W.H.; Mueller, S.H.; Denaxas, S.; Lai, A.G. Comparing clinical trial population representativeness to real-world populations: An external validity analysis encompassing 43,895 trials and 5,685,738 individuals across 989 unique drugs and 286 conditions in England. *Lancet Healthy Longev.* **2022**, *3*, e674–e689. [CrossRef] [PubMed]

16. European Medicines Agency (EMA). ICH E9 (R1) Addendum on Estimands and Sensitivity Analysis in Clinical Trials to the Guideline on Statistical Principles for Clinical Trials. 2020. Available online: https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e9-r1-addendum-estimands-and-sensitivity-analysis-clinical-trials-guideline-statistical-principles-clinical-trials-step-5_en.pdf (accessed on 7 January 2025).

17. HTA Coordination Group (HTACG). Guidance on Reporting Requirements for Multiplicity Issues and Subgroup, Sensitivity and Post Hoc Analyses in Joint Clinical Assessments. 2024. Available online: https://health.ec.europa.eu/document/download/f2f00444-2427-4db9-8370-d984b7148653_en?filename=hta_multiplier_jca_guidance_en.pdf (accessed on 8 January 2025).

18. Schwartz, D.; Lellouch, J. Explanatory and pragmatic attitudes in therapeutical trials. *J. Chronic Dis.* **1967**, *20*, 637–648. [CrossRef] [PubMed]

19. The Institute for Quality and Efficiency in Healthcare (IQWiG). 2018: Do the New “Estimand” Strategies Compromise the Standards of Benefit Assessments? 2018. Available online: https://www.iqwig.de/en/events/iqwig-in-dialogue/2018-do-the-new-estimand-strategies-compromise-the-standards-of-benefit-assessments.html (accessed on 16 January 2025).

20. European Access Academy (EAA). Open Letter to DG Santé and the Member State Coordination Group on HTA. 2024. Available online: https://irp.cdn-website.com/e52b6f19/files/uploaded/Open_Letter_Methods_EU_HTA.pdf (accessed on 28 January 2025).

21. European Commission. Regulation (EU) 2021/2282 of the European Parliament and of the Council of 15 December 2021 on Health Technology Assessment and Amending Directive 2011/24/EU. 2021. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32021R2282 (accessed on 18 December 2024).

22. Godwin, M.; Ruhland, L.; Casson, I.; MacDonald, S.; Delva, D.; Birtwhistle, R.; Lam, M.; Seguin, R. Pragmatic controlled clinical trials in primary care: The struggle between external and internal validity. *BMC Med. Res. Methodol.* **2003**, *3*, 28. [CrossRef] [PubMed]

23. Burnett, T.; Mozgunov, P.; Pallmann, P.; Villar, S.S.; Wheeler, G.M.; Jaki, T. Adding flexibility to clinical trial designs: An example-based guide to the practical use of adaptive designs. *BMC Med.* **2020**, *18*, 1–21. [CrossRef] [PubMed]

**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.