@How health science is done:
the good, the bad and the ugly
Nerd Nite Tokyo — 2023年12月8日
Catherine BEAUCHEMIN
カトゥリン · ボシゥメン

C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 1/14

Negative impressions about health research• experts disagree / keep changing their minds• experts have an agenda (profit, politics, social engineering, self-promotion, etc.)

Key issues with health research• health research has for-profit components (pharma co., private hospitals/insurance)• urgency to cure a disease conflicts with understanding how it works• poor mathematical/statistical training in health sciences– disregarding data in favour of one’s own expectation– studies designed to prove rather than disprove hypothesis– routinely major flaws in study design or analysis

C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 2/14

International standards for clinical trials
In 2008, the World Medical Association’s (WMA) Declaration of Helsinki stated:• Clinical trials must be registered in public database before recruiting 1st subject• All studies should be published, regardless of statistical significance of outcome
In 2015, World Health Organization (WHO) added:• Main findings of clinical trials are to be published within 2 years of completion, key
outcomes publicly available within one year in result section of clinical trial registry
Adherence to these 3 recommendations is voluntary.
ClinicalTrials.gov(2015–2020)

completedw results funded by industry
United States
20,102 7,462 (37%)
8,677 (43%)
Canada3,247 1,055 (32%)
1,471 (45%)∗United Kingdom
3,746 1,071 (29%)
1,959 (52%)
Japan∗
1,025547 (53%)
945 (92%)→ Lack of transparency reduces future participation & trust in results.
C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 3/14

Irregularities in registered clinical trials
When registering a clinical trial, you need to identify:• therapy or intervention to be evaluated (e.g., drug)• target population (e.g., sex, age, race or genetic traits)• inclusion/exclusion criteria (e.g., people with other underlying diseases)• primary outcome indicating success (e.g., reduction in # of pimples)• (optionally) secondary outcomes (e.g., make hair more curly, lower itchiness score)

Moving the goalposts
A study of trials conducted in Germany, 2009–2017 [doi:10.1371/journal.pmed.1004306]:• 8% changed primary outcome bwn registration & latest entry.
• 18% changed primary outcome bwn latest entry & paper (1% said so in paper).
Changing ‘measure of success’ = bias reporting + statistical need to account for multiple
hypothesis testing → misconduct (can be caught by peer-reviewer thanks to registry)
C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 4/14

Research misconduct → paper retractions
To get grants/job/promotion, researchers need to publish peer-reviewed papers→ extra points for: more papers, fancier journals, more citations
Rank. Name
# retracted
FieldCountry
01. J Boldt
194
Medicine
Germany
02. Y Fuji
172
MedicineJapan
03. H Ueshima
124
MedicineJapan
From
04. Y Sato
113
MedicineJapan
05. A Nazari
100
Engineering
Iran
Retraction Watch
06. J Iwamoto
88
MedicineJapan
Leaderboard
07. D Stapel
58
Psychology Netherlands
08. Y Saitoh
56
MedicineJapan
09. A Maxim
48
Engineering
USA
10. AS Elahi
44
Engineering
Iran
11. CY Chen
43
Engineering
Taiwan12. F Sarkar
41
Medicine
USA
13. S Shamshirband
41Comp. Sci.
Malaysia
14. H Zhong
41
Medicine
China
15. S Kato
40MedicineJapan
9/15 (60%) in Medicine! Sadly 6/15 (40%) from Japan, all Medicine.
C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 5/14

Issues with the peer-review system[Fauci] So that [hydroxychloroquine] study is a flawed study and I think anyone who
examines it carefully sees that it is not a randomized, placebo-controlled trial.
[Luetkemeyer] It’s been peer-reviewed...
[Fauci] It doesn’t matter, you can peer-review something that’s a bad study.
[Wait... Why?!?]US House Oversight & Reform Select Subcommittee
on Coronavirus Crisis, July 31, 2020
C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 6/14

Peer-reviewed results inform the standard of careStandard of careguidelines generally accepted by
medical community in treatmentof disease or condition.
Failure to meet it =medical negligence

C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 7/14

Peer-reviewed results inform the standard of careStandard of careguidelines generally accepted by
medical community in treatment
of disease or condition.
Failure to meet it =medical negligence

C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 8/14

Or in the words of Mr. Spock to Dr. McCoy[Star Trek (Original Series), Season 2, Epis. 25 “Bread and Circuses” © 1968, Desilu Prod.]

C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 9/14

Bar graphs are sneaky & p-values are misleading

C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 10/14

Bad graphs: when a picture is not worth 1,000 words
Figure from peer-reviewed article published in Nature Medicine (top 10 medical journal)
Primary endpoint: increase in [y -axis value] from Baseline to 12 months.
Key finding:‘[...]levels after 12 months werehigher in [treatment] groupcompared with [placebo](average marginal effect(AME)=0.057, p=0.037)’
Why is AME used rather thansimply increase from baseline?
Do we expect this marginalchange to result in significant
health improvement?
C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 11/14

Ongoing efforts to improve practices

C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 12/14

Some encouraging signs of improvement

C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 13/14

Main take home message• Health research has issues: profit motive, reporting bias, misconduct, innumeracy• I have seen encouraging progress over the last 5–7 years
bar→dot plots, less p-value obsession, more honest titles/conclusions, open data
• There is still a long way to go...

Where does that leave you?
• Avoid thinking in absolute terms (vaccines are good/bad), be more moderate (somevaccines are essentials, others so-so)• Avoid trusting results just because they agree with your expectations

The END.
Catherine Beauchemin (cbeau@riken.jp)
https://phymbie.physics.ryerson.ca
C.Beauchemin (ボシゥメン) — RIKEN/TorontoMetroU — Slide 14/14