Risk Literacy Among Portuguese Medical and Dentistry Students - a Cohort Study
Conditions: Medical Education, Decision Support Techniques
Sponsor: Harding Center for Risk Literacy

 
Study Protocol Cover Page 
 
 
 
 
 
Official Study Title:   
Risk Literacy among Portuguese medical and 
dentistry students - a cohort study 
 
Clinicaltrails.gov-ID: 
 
NCT04380428 
Protocole-ID:  
 
 
HC2019/11 
 
 
Current Revision: B (10.05.2020) 
 
Revision History: 
 
Revision ID 
Date 
Initiator(s) 
A 
01.05.2020 
Maxim Benz, Alexej Zhogov 
B 
10.05.2020 
Maxim Benz, Alexej Zhogov 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
1 
Protocol Outline 
 
 
Protocol Title: Risk Literacy among Portuguese medical and dentistry students - a cohort study 
Short Title: RiskommPT  
 
Protocol Version:  
 
1.4 
 
Protocol Date:  
 
24.10.2019 
Principal Investigators:  
Maxim Benz, Alexej Zhogov  
Research Team: 
Maxim Benz, Alexej Zhogov, Henrique Proença da Cunha, 
Mirjam Jenny, Gerd Gigerenzer, Mário Pereira Pinto,  
Niklas Keller 
 
I. 
Abstract 
Provide a summary of the study background, aims, and design.  
 
Risk literacy is the competence to deal with probabilistic information in an informed 
way. In medicine and dentistry (in the following text both referred as medicine), this is 
crucial to accurately interpret the results of diagnostic tests and to make risk-benefit-
assessments for treatment options. Without it, applying medical evidence to clinical 
practice remains illusory. Previous research showed that there is still a lack of risk 
literacy among medical personnel in different countries [Garcia-Retamero, Galesic & 
Gigerenzer, 2011]. It was also shown that this deficit is easy to overcome with simple 
teaching interventions [Keller, Jenny & Gigerenzer, 2018]. With this investigation we 
want to assess the current state of risk literacy among Portuguese medical students in 
order to assess a baseline. This allows measurement of improvement of risk literacy in 
Portugal in the future and generates data for comparisons of risk literacy between 
European countries.                             
 
We are planning to publish our study in an open access journal, thereby promoting its 
dissemination. With our findings we want to encourage a dialogue with relevant 
stakeholders in the Portuguese healthcare system.  We have simplified the design of the 
study, facilitating reproduction and replication through other research groups in the 
future and in other countries, thereby generating more data for epidemiological mapping 
of risk literacy in the world. 
 
      
II. 
Background and Significance/Preliminary Studies 
Describe the current environment that is the basis for the proposed research, 
including a presentation of the problem (with references) and a review of current 
literature.  Include a critical evaluation of current knowledge and preliminary 
studies related to the proposed research and describe how this proposal will 
enhance this knowledge. 
                       
Medical risk literacy encompasses the skills to understand medical statistics and 
numerical medical evidence. It is central to the practice of evidence-based medicine. In 
diagnostics, it affects the interpretation of test results, such as being able to assess the 
probability of a lab-result being false-negative or false-positive. In treatment planning, 
it affects the assessment of the effectiveness of different interventions. Without risk 
literacy, doctors can be influenced in their treatment decisions for example by framing 
effects (Gigerenzer & Wegwarth, 2013). Further, risk literacy helps immunize patients 

 
2 
against the influence of non-evidence-based treatment suggestions by physicians 
(Wegwarth, Wagner & Gigerenzer, 2017). It was shown that a quick, 90-minute 
intervention can significantly boost medical students’ skills in understanding medical 
evidence and its effect on medical practice (Keller et al., 2017). A big part of the 
research on risk literacy has been done in Germany and some medical facilities in 
Germany are beginning to integrate the findings into their teaching curricula. 
 
To give a positive example of this change associated with the research, risk literacy was 
integrated into the curriculum of the biggest European medical university-clinic, Charité 
Universitätsmedizin Berlin, in 2014. It is taught in the 9th semester. Due to a student 
initiative, students also have the opportunity to learn about risk literacy already from 
the second semester onwards. This voluntary course has received positive feedback 
from the students. Our opinion is, that risk literacy must be taught earlier, since a lot of 
probabilistic information is already contained in the curriculum which potentially can 
be misinterpreted. 
 
Portugal has a very broad medical educational landscape. During our preparatory 
research, we found out that several Portuguese Universities already offer statistical 
courses as part of the curriculum for the students and some elements of risk literacy are 
already taught in medical curricula in Portugal. During our preparatory searches we 
were not able to identify any course with similar content composition and knowledge 
packages as proposed and evaluated by Keller et al., 2017. 
 
Several tests have been developed that assess for risk literacy, of which the three most 
widely used ones are the Quick Risk Test (QRT,  Jenny, Keller & Gigerenzer, 2018), 
the Critical Risk Interpretation Test (CRIT, Caverly et al., 2015) and the Berlin 
Numeracy Test (BNT, Cokely & Galesic, 2012). The QRT has been specifically 
developed for physicians and other health professionals. 
 
 
 
 
      
III. 
Study Aims 
Describe the purpose of the study, including identification of specific primary 
objectives/hypotheses. Secondary objectives/hypotheses should be described as 
necessary. 
 
The primary objective of the study is the measurement of the risk literacy among all 
participating Portuguese medical students. We intend to obtain a representative 
demographic sample from the participating institutions, to assess the baseline in order 
to be able to document an improvement in the future and to compare these results with 
students of other European countries. 
 
The secondary objectives of the study are to compare students of different years of 
study with each other to see if there are some differences.  This information could help 
determine the best possible time point(s) for intervention(s). 
 
We will also explore whether students, who have already received statistics training at 
some point prior to testing receive higher scores in risk literacy. 
 
 

 
3 
IV. 
Administrative Organization 
Describe the participating units, including other participating study sites, 
laboratories, data management center, and coordinating center as applicable. 
● Harding Center for Risk Literacy (Max Planck Institute for Human 
Development), Berlin, Germany 
● We hope to obtain the participation of the following Portuguese faculties: 
○ Instituto Universitário de Ciências da Saúde 
○ Instituto Universitário Egas Moniz 
○ Universidade Católica Portuguesa - Centro Regional das Beiras 
○ Universidade da Beira Interior 
○ Universidade de Coimbra - Faculdade de Medicina 
○ Universidade de Lisboa - Faculdade de Medicina 
○ Universidade de Lisboa - Faculdade de Medicina Dentária 
○ Universidade do Minho 
○ Universidade do Porto - Faculdade de Medicina 
○ Universidade do Porto - Faculdade de Medicina Dentária 
○ Universidade do Porto - Instituto de Ciências Biomédicas Abel Salazar 
○ Universidade Fernando Pessoa 
○ Universidade Nova de Lisboa - Faculdade de Ciências Médicas 
● Presidencia (High Patronage requested) 
      
      
V. 
Study Design 
a. Experimental design of the study (e.g., single-blind, double-blind)                     
This exploratory, epidemiological investigation is conducted as an 
observational, cross-sectional cohort study. 
b. Study population general description                                                                     
The studied population are medical and dentistry students of Portuguese 
faculties. 
c. Sample size determination and power analyses    
 
 
 
      
Not applicable, due to the exploratory character of the study.        
d. Study outcomes/endpoints.                                                                                          
The primary Outcome is the quantitative measurement of "medical risk 
literacy" among Portuguese medical students measured with the Quick Risk Test 
(QRT) [Jenny, Keller & Gigerenzer, 2018] and the multiple choice version of 
the Berlin Numeracy Test (BNT) [Cokely & Galesic, 2012]. We do not use the 
CRIT (Caverly et a., 2015) in this study, as it tests for more advanced statistical 
understanding and is a very extensive test, which would require at least 30 to 45 
minutes for completion.  
 
Secondary outcomes is to investigate, whether study year, faculty, and prior 
statistical education have an impact on the level of risk literacy. 
 
 
 
VI. 
Study Procedures 
a. Subject selection procedures 
i. Sampling plan including Inclusion/Exclusion criteria (subject and 
disease characteristics)  
 
 
 
                              
The participants can be included if they have successfully enrolled into 
medical or dentistry faculty, but have not yet completed the final exam.  

 
4 
 
 
ii. Recruitment procedures 
1. Where will recruitment occur? 
We will approach the faculties to determine the local 
mechanisms to approach the students.  
2. Where and when will consent be obtained?                                   
Informed consent will be obtained prior to the initiation of the 
online-questionnaire. No written consent can be given due to 
anonymity.     
3. Who will obtain consent?                                                                  
The investigators and the Harding Center for Risk Literacy                
4. What is the advertising plan, if applicable? 
University mailing lists and if necessary students’ associations 
and social media. 
5. What recruitment materials will be provided to the potential 
participant (brochures/information sheets/video presentation)? 
The invitational email will contain some introductory 
information as to the purpose, duration and incentivization of the 
study. The complete participant information will be displayed on 
the introductory page of the online-questionnaire.  
 
iii. Screening procedures 
Not applicable 
1. What procedures are required for screening?                        
What is the screening schedule (number of visits, length of 
visits)? 
2. Which screening tests/procedures are part of standard care and 
which are for research purposes only? 
3. What happens with screen failures (including any data gathered 
during screening)? 
b. Randomization procedures (if applicable)                      
Not applicable 
(Exploratory study) 
 
c. Study Intervention 
Not applicable 
i. For Drug/device studies: 
1. Active study agents 
2. Placebo study agents 
3. Blinding/labeling/preparation of agents 
4. Storage 
5. Administration 
6. Toxicities and guidelines for adjustments 
ii. For Other types of intervention studies: 
1. Active intervention description 
2. Control group, if applicable 
d. Study Assessments and Activities 
i. Describe all study procedures, assessments, and subject activities 
Students receive an email with some introductory information on the 
study (goal of the study, expected duration, incentivization). Students are 

 
5 
then led via URL to the online questionnaire, the first page of which 
contains the full participant information. Students declare their consent 
to participation by clicking a button that leads begins the questionnaire 
containing the QRT, BNT and additional information. Participants can 
abort the questionnaire at any time by closing the browser. The time of 
the completion of the questionnaire will be recorded. After completion 
of the questionnaire, on the final page, participants receive an individual 
code which  which they can use to receive a free 2-week subscription to 
the AMBOSS medical learning platform. A link to the AMBOSS website 
will also be displayed. They may also use this code to enter into a lottery 
to win one of three six-month-subscriptions. Note that registration with 
AMBOSS is only possible with provision of a valid email address of the 
faculty. Each code can only be entered once. There are no mechanisms 
that allow the exchange of study data with the personal information 
provided during the AMBOSS registration process. 
ii. Provide a schedule of all study assessments and subject activities, 
including a tabular representation or timeline as applicable 
 
VII. 
Safety Monitoring Plan 
a. Definition of adverse events, serious adverse events 
not necessary 
b. What procedures will be used to monitor subject safety?   
not necessary 
c. Who (list names) will identify, document, and report adverse events? 
not necessary 
d. What is the frequency for review of summarized safety information and who 
will perform the review (e.g., safety monitoring board)? 
not necessary 
e. What are the stopping rules with regard to efficacy and safety? 
not necessary 
 
VIII. Analysis Plan 
 
Describe statistical analysis methods as appropriate.  For example, will intention-to-
treat methodology be used in the analysis?  Will there be any sample stratification?. 
We will conduct an exploratory analysis of the current level of risk literacy among 
Portuguese medical students as a baseline for international comparison and 
measurement of improvement in the future. We will only perform descriptive analyses 
as well as exploratory correlational analyses. No hypotheses are tested. 
Representativeness of the sample to the population of Portuguese medical students will 
be assessed after data collection is completed.  
 
 
IX. 
Literature Cited 
         
1.     Garcia-Retamero R, Galesic M, Gigerenzer G. Enhancing understanding and 
recall of quantitative information about medical risks: a cross-cultural 
comparison between Germany and Spain. Span J Psychol. 2011;14(1):218-226. 
http://www.ncbi.nlm.nih.gov/pubmed/21568179. Accessed May 1, 2016. 
3.     Gigerenzer G, Wegwarth O. Five year survival rates can mislead. BMJ. 
2013;346:f548. http://www.ncbi.nlm.nih.gov/pubmed/23360815. Accessed May 

 
6 
1, 2016. 
4.     Jenny MA, Keller N, Gigerenzer G. Assessing minimal medical statistical 
literacy using the Quick Risk Test: a prospective observational study in 
Germany. BMJ Open. 2018;8(8):e020847. doi:10.1136/bmjopen-2017-020847 
5.     Keller N, Feufel MA, Kendel F, Spies CD, Gigerenzer G. Training medical 
students how to extract, assess and communicate evidence from an article. Med 
Educ. 2017;51(11):1162-1163. doi:10.1111/medu.13444 
6.     Wegwarth O, Wagner GG, Gigerenzer G. Can facts trump unconditional trust? 
Evidence-based information halves the influence of physicians’ non-evidence-
based cancer screening recommendations. PLoS One. 2017;12(8):1-12. 
doi:10.1371/journal.pone.0183024 
7.     Cokely ET, Galesic M, Schulz E, Ghazal S, Garcia-Retamero R. Measuring risk 
literacy: The berlin numeracy test. Judgm Decis Mak. 2012;7(1):25-47. 
8.  
Caverly TJ, Prochazka A V., Combs BP, et al. Doctors and numbers: An 
assessment of the critical risk interpretation test. Med Decis Mak. 
2015;35(4):512-524. doi:10.1177/0272989X14558423