JMIR FORMATIVE RESEARCH

Gnant et al

Original Paper

Assessing Knowledge, Competence, and Performance Following
Web-Based Education on Early Breast Cancer Management:
Health Care Professional Questionnaire Study and Anonymized
Patient Records Analysis
Michael Gnant1, MD; Khatijah Lim Abdullah2, PhD; Frances Boyle3, MD; Chiun-Sheng Huang4, MD; Katie Bickford5,
BSc; Sola Neunie5, MSc; Alexander Noble5, MSc; Anne Nunn5, PhD; Caroline Sproat5, PhD; Nadia Harbeck6, MD;
Carlos Barrios7, MD
1

Comprehensive Cancer Center, Medical University of Vienna, Vienna, Austria

2

Department of Nursing, School of Medical and Life Sciences, Sunway University, Selangor, Malaysia

3

Patricia Ritchie Centre for Cancer Care and Research, University of Sydney, Sydney, Australia

4

Department of Surgery, National Taiwan University Hospital, Taipei City, Taiwan

5touch Independent Medical Education Limited, Stockport, United Kingdom

6

Breast Center, Department of Obstetrics & Gynecology and Comprehensive Cancer Center Munich, University Hospital of Munich Ludwig Maximilian,
Munich, Germany
7

Latin American Cooperative Group and Oncoclínicas Group, Hospital São Lucas da Pontifical Catholic University of Rio Grande do Sul, Porto Alegre,
BrazilCorresponding Author:
Michael Gnant, MD
Comprehensive Cancer Center
Medical University of Vienna
Spitalgasse 23
BT86/E01
Vienna, 1090
Austria
Phone: 43 1 40160 35602
Email: mgnant@icloud.com

Abstract
Background: Web-based learning activities are key components of continuing medical education (CME) for health care
professionals (HCPs). However, the published outcomes of web-based educational interventions for early breast cancer (EBC)
are limited.
Objective: This study aims to objectively assess knowledge, competence, and performance among HCPs following participationin 2 EBC-focused CME activities and to identify the remaining educational gaps.
Methods: We developed 2 CME-accredited web-based educational activities addressing high-risk EBC, including integrationof shared decision-making to optimize patient care (touchMDT) and stratification for early identification of high-risk patientsand novel treatment strategies (touchPANEL DISCUSSION). Knowledge, competence, and performance were assessed before
and after the activities against an expanded outcomes framework (levels 1-5) using self-reported questionnaires and an analysisof anonymized data extracted from patient records.
Results: Six months after the launch of the activity, 7047 and 8989 HCP participants engaged with touchMDT and touchPANELDISCUSSION, respectively. The overall satisfaction was 82% (a total score of 20.6 out of 25) for the touchMDT and 88% (atotal score of 21.9 out of 25) for the touchPANEL DISCUSSION. For the evaluation of knowledge and competence (50 respondents
before the activity and 50 learners after the activity), there was a significant increase in the mean number of correctly answeredquestions from pre- to postactivity (touchMDT: median 4.0, IQR 3.0-5.0 to median 5.5, IQR 4.0-7.0; mean 4.00, SD 1.39 to mean
5.30, SD 1.56 and touchPANEL DISCUSSION: median 4.0, IQR 4.0-5.0 to median 6.0, IQR 5.0-7.0; mean 4.32, SD 1.30 tomean 5.88, SD 1.49; both P<.001). A significant improvement in self-reported performance (50 respondents before the activity
and 50 learners after the activity) was observed in a combined analysis of both activities (median 3.0, IQR 2.0-3.0 to median 4.0,
https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e50931 | p. 1
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Gnant et al

IQR 3.0-5.0; mean 2.82, SD 1.08 to mean 4.16, SD 1.45; P<.001). Patient record analysis (50 respondents before the activity and
50 learners after the activity) showed that the HCPs used a range of measures to determine EBC recurrence risk and revealed nosignificant differences in adjuvant therapies used before and after the activity (P=.97 and P>.99 for Ki-67 <20% and Ki-67 ≥20%
tumors, respectively). The remaining educational gaps included strategies for implementing shared decision-making in clinicalpractice and the use of genetic and biomarker testing to guide treatment selection.
Conclusions: Brief, web-based CME activities on EBC were associated with an improvement in HCP knowledge, competence,and self-reported performance and can help identify unmet needs to inform the design of future CME activities.
(JMIR Form Res 2024;8:e50931) doi: 10.2196/50931
KEYWORDS
continuing medical education; early breast cancer; performance; risk stratification; shared decision-making

Introduction
Clinical Decision-Making in High-Risk Early Breast
CancerClinical decision-making in cancer can be complex, requiring
health care professionals (HCPs) to consider multiple clinicalfactors, patient preferences, and environmental factors [1-3]. A
multidisciplinary team approach can improve the overall qualityof decision-making [4-7], and there is evidence that shareddecision-making (SDM) with patients can lead to better
adherence to treatment and outcomes [1,8,9]. However, obstacles
such as lack of resource availability and patient factors (eg,cultures and health literacy) have prevented the complete
integration of SDM into cancer care [1,10-14]. HCPs need tounderstand these obstacles to deliver high-level care tailored tothe individual.
The complexity of treating high-risk early breast cancer (EBC)
has increased further with the emergence of complex new dataon risk stratification and treatment. These include data on the
prognostic and predictive value of biomarkers and the benefitsof cyclin-dependent kinase 4 and 6 inhibitors, second-generation
selective estrogen receptor antagonists and degraders, andprotein kinase B (Akt) inhibitors [15-20]. Furthermore, theguidelines on the management of EBC have been updated
recently to reflect recent data, which can be interpreted as
influencing clinical practice [5,21-28]. Because of this rapid
influx of complex information, HCPs who treat EBC arechallenged to keep up with practice changes; consequently, it
can take time for new and emerging data to be effectively
integrated into practice, and patients with EBC may not receive
optimal care [29-32].

Effectiveness of Web-Based HCP Education in Breast
CancerEffective continuing medical education (CME) programs are
crucial to ensure that HCPs implement up-to-date,evidence-based practices for their patients. Although CME for
HCPs has traditionally been delivered using face-to-faceformats, web-based educational activities now offer a viable
alternative approach [33]. To date, several studies havedemonstrated the effectiveness of web-based HCP education
for breast cancer [34-36]. However, gaps remain in the provisionof independent web-based medical education programs forhigh-risk EBC treatment and effective SDM. Although a small
number of programs have been developed in EBC [37-39], thesehttps://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

have typically been nonaccredited, therefore not providing HCPs
with continuing professional development credits for attendance;
pharmaceutical company–led, therefore with the potential to beperceived as promotional; and restricted or limited in terms ofaccess; therefore only accessible to those with the means to doso. To address these gaps and complement the currently
available programs, accredited, independent, expert-led
education available on free-to-access platforms is required tosupport the knowledge, competence, and performance of HCPs.

Study Rationale, Objectives, and Aims
An expanded 7-level framework for assessing the outcomes of
CME programs was developed by Moore et al [40] in 2009.
This framework evaluates participation (level 1), satisfaction(level 2), knowledge (level 3), competence (level 4),
performance (level 5), patient health status (level 6), andcommunity health status (level 7). These levels are now widelyused and included in many consensus documents and practicerecommendations [41]. To date, published Moore’s level 5 [40]outcomes analyses in medical education have largely usedsubjective self-reported outcomes [42-44]. Although these canprovide valuable insights, they can be open to bias, for example,
recall bias or social desirability bias [45,46]. An analysis ofanonymized patient records provides a complementary objective
measure of performance that may enhance the understandingof physician behavior changes in practice.
In this study, we developed and implemented 2 faculty-led,
CME-accredited web-based educational activities in high-risk
EBC. The objectives of this study were to (1) measure changes
in knowledge, competence, and self-reported performance afterparticipation in the activities; (2) objectively evaluate changes
in performance using anonymized patient records; and (3)identify the remaining educational gaps.

Methods
Study Design
This was a web-based study that analyzed the impact of 2
educational activities in high-risk EBC through HCP
questionnaires and deidentified patient records. The target
audience comprised oncologists (including breast cancer
surgeons), oncology nurse specialists, pathologists, andradiologists from Asia, Brazil, and Europe (excluding the United
Kingdom). Satisfaction, knowledge, competence, andperformance were assessed according to levels 2 to 5 of the
JMIR Form Res 2024 | vol. 8 | e50931 | p. 2
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
expanded framework for assessing the outcomes of CME
programs by Moore et al [40] using self-reported questionnaires
and anonymized data extracted from patient records.
Questionnaire distribution and data collection were carried out
by an independent third party (nuaxia Limited, United
Kingdom).

Ethical Considerations
All consents and ethical confirmations for this study wereobtained before the completion of both outcome questionnaires.
The personal data of HCPs and patients were anonymized.
General Data Protection Regulation [47] consents were obtainedin the European Union and North America for eachquestionnaire; adherence to additional requirements in local
markets was required to be confirmed in line with European
Pharmaceutical Market Research Association professional
guidance [48]. For respondents to the patient records
questionnaire, consents and ethical confirmations were obtained
as part of the questionnaire in accordance with the British
Healthcare Business Intelligence Association professionalguidelines [49] to ensure patient confidentiality. As part of theirwritten consent to participate in the questionnaires, the HCPs
were required to agree to the statement, “As with all research,
your identity and personal data are strictly confidential and will
not be revealed without your explicit further consent. You havethe right to withdraw your consent to participate in market
research at any time.” Complete patient records were notobtained by touch Independent Medical Education (touchIME)or any third party; the HCPs were instructed to extract therequired anonymized information into a questionnaire. Ethics
approval was not applicable for this study because this analysis
was not classified as “health research” according to the UK
Research and Innovation definition that guides whether researchrequires Research Ethics Committee review and Health Research
Authority approval [50,51]. The study was defined as “market
research” per rule 1.1 of the European Pharmaceutical Market
Research Association code of conduct, which states that market
research does not require a Clinical Research Ethics Committee
or independent review board approval [48].

Educational Activities
Educational gaps and learning objectives were identified before
the start of activity development in October 2021 by touchIME(a provider of IME for the global HCP community) through areview of the published literature. The expert faculty members
further inputted into the educational gaps and learning objectives
at the start of activity development. Expert faculty memberswith a background in breast cancer and SDM were identified
by touchIME medical directors through searches of literatureindexed in the PubMed database, congress websites, andweb-based educational videos. Patient faculty were identified
using social media searches across multiple channels. The
faculty members (CB, FB, MG, NH, CSH and KLA) wereblinded to the plans to assess participation and satisfaction withthe education and to publish the outcomes. All the expert faculty(CB, FB, MG, NH, CSH and KLA) involved in the educational
activities are authors of this manuscript.
A total of 2 web-based educational activities, touchMDT: “howcan shared decision-making be successfully integrated tohttps://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

Gnant et al
optimize care of patients with high-risk early breast cancer?”and touchPANEL DISCUSSION: “new horizons in high-risk
HR+ HER2- EBC: Risk stratification for early identificationand novel treatment strategies” were developed by touchIME
in collaboration with the faculty members (CB, FB, MG, NH,
CSH and KLA; complete details and learning objectives areincluded in Multimedia Appendix 1). The touchMDT activity
comprised three 13- to 15-minute videos, providing 42 minutesof education in total, and the touchPANEL DISCUSSION
activity comprised three 11- to 15-minute videos, providing 39
minutes of education in total. The activities were CME
accredited by the Accreditation Council for Continuing Medical
Education and the American Nurses Credentialing Center at the
University of South Florida Health. Both activities weretranslated (video subtitles and downloadable slides) fromEnglish into Brazilian Portuguese, Chinese Mandarin, French,
German, Italian, and Spanish by an independent third party(UnBabel, United States), and both activities were free to accessfor 1 year (the permissible lifetime of an accredited educational
activity) on the touchONCOLOGY website from March 30,
2022, to March 30, 2023 (touchMDT) and from June 15, 2022,
to June 15, 2023 (touchPANEL DISCUSSION). The 2 activities
featured together on the host website, but participation in bothactivities was not mandatory.
Various communication channels were used to reach the target
audience, including direct publicity emails to thetouchONCOLOGY database within the first 12 weeks of theactivity launch, with a further reminder at around 6 months;
display banners on the touchONCOLOGY website;
advertisements in the peer-reviewed journal touchREVIEWS
in Oncology and Haematology; publicity via various relevant
medical society partnerships; and social media partnerships on
Facebook (Meta Platforms, Inc), LinkedIn (Microsoft Corp),
and Twitter (rebranded as X; X Corp) targeted at the HCPs
throughout the lifetime of the activity.

Assessment of Educational Outcomes
Levels 1 and 2 (Participation and Satisfaction)
Levels 1 (participation) and 2 (satisfaction) were assessed
separately for each activity (Multimedia Appendix 2). Level 1
included the number of HCPs who engaged in the web-based
educational activities and the average time they spent viewing
the videos, and it was the only assessment not evaluated using
a questionnaire.
Level 2 (satisfaction) was assessed by scoring 5 statements ofsatisfaction (using a Likert scale) in a postactivity questionnaire.

Levels 3 to 4 and Level 5 (Knowledge, Competence, andPerformance)
Levels 3 to 4 and level 5 were assessed using questionnaires
completed by relevant respondents (HCPs completing thequestionnaire before the activity) and learners (HCPs completingthe questionnaire following the education). No pretesting was
performed. The target audience was predefined by specialty(eg, oncologists [including breast cancer surgeons], oncology
nurse specialists, pathologists, and radiologists) and by region(Brazil, France, Germany, Italy, and Spain). To avoid any
pre-exposure bias and to obtain a statistically representative
JMIR Form Res 2024 | vol. 8 | e50931 | p. 3
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
sample size, data were collected using an independent samples
model for each activity. For levels 3 to 4, separate questionnaires
were developed for each activity (each comprising 7 questions),
whereas for level 5, a combined questionnaire was developedfor both activities (3 questions each for touchMDT andtouchPANEL DISCUSSION). All questions were developed
by touchIME medical directors and writers and were approved
for medical accuracy by the faculty. An overview of the topics
and the complete questionnaires are provided in Multimedia
Appendices 3 to 6.
For the touchPANEL DISCUSSION activity, changes in
performance were also assessed through an evaluation of
redacted patient records. The HCPs were sent a web-basedquestionnaire that captured data on (1) general patient
demographics and history, (2) risk assessment and stratification
to predict disease recurrence, and (3) treatment. Each HCP was
requested to extract all relevant nonidentifiable informationfrom a single anonymized patient record into the questionnaire.

Self-Reported Confidence and Intention to Change
Practice
As part of the questionnaires, respondents and learners wereasked, “How confident are you in treating patients with breast
cancer?” (mutually exclusive responses: “not confident,” “alittle confident,” “somewhat confident,” “moderately confident,”
and “extremely confident”) and “As a result of your participationin this session, will you make a change in your practice?”(mutually exclusive responses: “yes,” “uncertain—moreeducation needed,” “uncertain—practical limitations,”“no—more education needed,” and “no—practical limitations”).

Identification of Outstanding Educational Gaps
A total of 5 potential educational gaps were included in thelevels 3 to 4 and level 5 questionnaires (Multimedia Appendix
7), and the learners were asked to rank them by importance.
The results were analyzed using a single transferable vote system(Multimedia Appendix 8 [52]). In addition, questions that wereanswered incorrectly by ≥30% (15/50) of the learners in all
postactivity questionnaires were identified as educational gaps.
The 30% cutoff was determined by touchIME as an indicatorof the requirement for further education based on analyses ofdata from previous outcomes.

Fielding of the Questionnaires
Preactivity levels 3 to 4 questionnaires were fielded 1 to 2 weeks
before launch, whereas postactivity levels 2 to 4 questionnaires
were fielded immediately after launch to a different set of HCPs
who had participated in the education. The questionnaires were
distributed to a database of 20,420 HCPs and then “closed”
once a prespecified number had responded (n=50).
The level 5 questionnaire and first patient records questionnaire
were fielded 1 to 2 weeks before launch—to a different set of
HCPs than those who answered the levels 3 to 4
questionnaires—and closed after completion by a prespecified
number of respondents (n=50). A total of 26 weeks after launch,
the postactivity level 5 questionnaire and second patient records
questionnaire were distributed to the same set of HCPs whocompleted the level 5 questionnaire and patient records
https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

Gnant et alquestionnaire before launch. Because of drop out over time,
additional HCPs who were matched to the dropouts weresurveyed postactivity, if needed.
The HCP participants’ specialties were all validated by nuaxia
Limited against third-party data with the assistance of artificial
intelligence. This comprised any combination of data fromprofessional associations, hospitals, professional publications,and prescribing and insurance databases.
The participants were required to answer “yes” to the screening
question “Do you treat patients with early breast cancer?” to beeligible to participate in the questionnaire study. The HCPs
eligible to participate received an honorarium in the form of adigital reward, which could be redeemed via a number of reward
partners. In accordance with industry regulations andprofessional body guidance, the honorarium received waslimited to the fair market value for an HCP of that seniority.

Statistical Analyses
Data were analyzed using SPSS Statistics software (version
28.0.1; IBM Corp). The sample size (50 respondents and 50
learners for each activity) was predetermined by the level offunding, and by applying a power analysis, we determined a
resulting margin of error of 10% for both the touchMDT andtouchPANEL DISCUSSION. Knowledge, competence, andself-reported outcomes were compared for the overall population
using an independent samples 2-tailed t test. Data were
subdivided by region and years of experience, and a 2-way
ANOVA was used to assess the variation in results by thesedemographics. Individual questions were analyzed with a paired
samples 2-tailed t test and a 1-way ANOVA, followed by acluster analysis, to provide insights into the overall change incorrect answers across the questions asked and the specificchanges in all the answers given. Data on the number ofquestions answered correctly were measured as a continuous
variable. At the individual question level, the answers werecategorical. For performance, measured by the redacted patient
records questionnaire, pre- and postactivity comparisons weremade using paired and independent samples 2-tailed t tests,
chi-square tests, and cluster analysis. These tests were used toassess patient history, use of risk assessment and stratification
strategies, and patient treatment pathways.

Results
Assessment of Educational Outcomes
Levels 1 and 2 (Participation and Satisfaction)
By 6 months after launch, 7047 and 8989 participants had
engaged with the touchMDT and touchPANEL DISCUSSION
activities, respectively. The participants were predominantly
oncologists, and the majority were based in Italy for the
touchMDT and Brazil for the touchPANEL DISCUSSION
activities (Multimedia Appendix 9). Overall satisfaction was
82% (a total score of 20.6 out of 25) for touchMDT and 88%(a total score of 21.9 out of 25) for touchPANEL DISCUSSION(Multimedia Appendix 10). Multichannel publicity reach andimpact for the touchMDT and touchPANEL DISCUSSION
activities are reported in Multimedia Appendix 11.
JMIR Form Res 2024 | vol. 8 | e50931 | p. 4
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Gnant et al
A total of 50 respondents and 50 learners were questioned toassess levels 3 to 4 outcomes (Table 1).

Levels 3 and 4 (Knowledge and Competence)

Table 1. Demographics of participating health care professionals (HCPs) for the touchMDT and touchPANEL DISCUSSION activitiesa.
Demographics of HCPs

Levels 2 to 4 questionnaire
touchMDT

Level 5 questionnaire
touchPANEL DISCUSSION

Both activities

Respondents

Learners

Respondents

Learners

Respondents

Learners

50

50

50

50

50

50

Oncologist

40 (80)42 (84)44 (88)47 (94)43 (86)42 (84)

Radiologist

0 (0)

0 (0)

6 (12)

3 (6)

4 (8)

3 (6)

Pathologist

0 (0)

0 (0)

0 (0)

0 (0)

3 (6)

5 (10)

8 (16)

0 (0)

0 (0)

0 (0)

0 (0)

Participants, N
Specialty, n (%)

Radiologist and pathol- 10 (20)ogist
Years of practice, n (%)<1 to 56 (12)

5 (10)

N/Ab

N/A

N/A

N/A

>5 to 2029 (58)25 (50)

N/A

N/A

N/A

N/A

<10

N/A

N/A

10 (20)11 (22)

N/A

N/A

10 to 20

N/A

N/A

27 (54)27 (54)

N/A

N/A

0 to 20

N/A

N/A

N/A

N/A

30 (60)32 (64)>20

15 (30)20 (40)13 (26)

12 (24)20 (40)18 (36)

Brazil14 (28)11 (22)16 (32)15 (30)

0 (0)

0 (0)

France5 (10)

9 (18)

5 (10)

5 (10)

7 (14)

7 (14)

Germany

14 (28)12 (24)16 (32)10 (20)16 (32)18 (36)Italy

8 (16)

9 (18)

7 (14)

10 (20)13 (26)

11 (22)Spain

9 (18)

9 (18)

6 (12)

10 (20)14 (28)14 (28)Country, n (%)

aData were collected 6 months after the launch of the touchMDT and touchPANEL DISCUSSION activities on September 29, 2022, and November
21, 2022, respectively. The questionnaire response categories differed between questionnaires and activities, meaning that some options were notapplicable. The respondents and learners were the HCPs who completed the pre- and postactivity questionnaires, respectively.
b

N/A: not applicable.

For the touchMDT and touchPANEL DISCUSSION activities,
respectively, only 12% (6/50) and 18% (9/50) of the respondents
answered at least 6 (86%) of the 7 questions correctly before
the activity. This increased to 50% (25/50) and 70% (35/50) in
the learners, respectively, after the activity (Figure 1). In Figure
1A, the heat maps show the proportion of respondents (N=50)
and learners (n=50) who answered a specific number ofquestions correctly, as displayed by colors ranging from white(the lowest proportion of respondents and learners) to dark red(the highest proportion of respondents and learners). In Figure
1B, the box and whisker plots show the distribution of thenumber of questions correctly answered by all respondents andlearners. In both plots, the horizontal red line within the box
indicates the median, the “x” symbol represents the mean, theboxes indicate the IQR, and the vertical lines (whiskers) extend
to the range of values, excluding outliers. The outliers aredefined as values that fall outside a distance of 1.5 times thehttps://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

IQR from the upper and lower quartiles and are represented byempty circles. The respondents and learners were the HCPs whocompleted the pre- and postactivity questionnaires, respectively.
For both activities, there was a statistically significant increasein the number of correctly answered questions from pre- to
postactivity for all participants (touchMDT: median 4.0, IQR
3.0-5.0 to median 5.5, IQR 4.0-7.0; mean 4.00, SD 1.39 to mean
5.30, SD 1.56; P<.001 and touchPANEL DISCUSSION: median
4.0, IQR 4.0-5.0 to median 6.0, IQR 5.0-7.0; mean 4.32, SD
1.30 to mean 5.88, SD 1.49; P<.001; Figure 1). In subgroup
analyses, significant increases in the number of correctly
answered questions were observed irrespective of specialty or
country for both activities and irrespective of years of experience
for the touchPANEL DISCUSSION (Multimedia Appendices
12 and 13). Refer to Multimedia Appendix 14 for the responses
to individual topics for the levels 3 to 4 questionnaires.

JMIR Form Res 2024 | vol. 8 | e50931 | p. 5
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Gnant et al

Figure 1. Summary of the number of correct responses for the levels 3 and 4 outcomes questionnaire before and after the launch of touchMDT andtouchPANEL DISCUSSION. (A) Heat maps show the proportion of respondents (n=50) and learners (n=50) who answered a specific number ofquestions correctly, as displayed by colors ranging from white (the lowest proportion of respondents and learners) to dark red (the highest proportionof respondents and learners). (B) Box and whisker plots show the distribution of the number of questions correctly answered by all respondents andlearners. In both plots, the horizontal red line within the box indicates the median, the “x” symbol represents the mean, the boxes indicate the IQR, andthe vertical lines (whiskers) extend to the range of values, excluding outliers.

Level 5 (Performance)
A total of 50 respondents and 50 learners completed the level
5 questionnaire and the patient records questionnaire (Table 1).
The dropout rate was 16% (8/50); therefore, 8 of the 50 learners
were de novo HCPs who were matched to the dropouts.
Self-Reported
Preactivity, in total, 6% (3/50) of respondents answered at least
5 (83%) out of 6 questions with the best clinical option. This
increased to 44% (22/50) of the learners after the activity (Figure2). In Figure 2A, the heat maps show the proportion of
respondents (n=50) and learners (n=50) who answered a specificnumber of questions correctly, as displayed by colors ranging
from white (the lowest proportion of respondents and learners)
to dark red (the highest proportion of respondents and learners).

https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

In Figure 2B, the box and whisker plots show the distributionof the number of questions correctly answered by all respondents
and learners. The horizontal red line within the box indicatesthe median, the “x” symbol represents the mean, the boxes
indicate the IQR, and the vertical lines (whiskers) extend to therange of values, excluding outliers. The outliers are defined asvalues that fall outside a distance of 1.5 times the IQR from theupper and lower quartiles and are represented by empty circles.
The respondents and learners were HCPs who completed thepre- and postactivity questionnaires, respectively. There was astatistically significant increase in the number of correctly
answered questions from pre- to postactivity for all participants(median 3.0, IQR 2.0-3.0 to median 4.0, IQR 3.0-5.0; mean
2.82, SD 1.08 to mean 4.16, SD 1.45; P<.001). Refer toMultimedia Appendix 15 for the responses to individual topics
for the level 5 questionnaires.

JMIR Form Res 2024 | vol. 8 | e50931 | p. 6
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Gnant et al

Figure 2. Summary of the number of correct responses for the level 5 outcomes questionnaire before and after the launch of touchMDT and touchPANELDISCUSSION. (A) Heat maps show the proportion of respondents (n=50) and learners (n=50) who answered a specific number of questions correctly,
as displayed by colors ranging from white (the lowest proportion of respondents and learners) to dark red (the highest proportion of respondents andlearners). (B) Box and whisker plots show the distribution of the number of questions correctly answered by all respondents and learners. The horizontal
red line within the box indicates the median, the “x” symbol represents the mean, the boxes indicate the IQR, and the vertical lines (whiskers) extend
to the range of values, excluding outliers.

In subgroup analyses of data by specialty, years of experienceand country (Multimedia Appendix 16), increases in the numberof questions with the correct clinical response were observed
irrespective of specialty or years of experience, and in thesubgroups of participants from Germany, Spain, and Italy.
Patient Record Data
The patient characteristics reported by the respondents andlearners were broadly similar (Multimedia Appendix 17). The
respondents and learners used a wide range of measures todetermine the patient risk of recurrence (30 and 26 different
combinations of measures, respectively; Multimedia Appendix
18). There was a significant difference between the number of
respondents and learners selecting each measure (χ249=508.474;
P<.001) and the number of measures selected (χ256=203.125;
P<.001; Multimedia Appendix 19). The Ki-67 index use was>90% for both groups, and most of those who used the Ki-67
index reported patients who had tumors with a Ki-67 score of≥20% (respondents: 30/48, 62.5% and learners: 30/47, 64%;
Figure S1 in Multimedia Appendix 20). There was no significant
difference between responders and learners regarding adjuvant
therapy when analyzing patient record data by Ki-67 index,
menopausal status, germline BRCA status, or the number of
positive lymph nodes (Figure S2 in Multimedia Appendix 20).

Self-Reported Confidence and Intention to Change
Practice
Results from the levels 3 to 4 and level 5 questionnaires
indicated modest increases in confidence in treating patientswith EBC from pre- to postactivity, with the proportion ofparticipants reporting that they felt moderately or extremely
confident increasing from 72% (36/50) to 78% (39/50; P=.82)
for touchMDT and 74% (37/50) to 86% (43/50; P=.49) for
touchPANEL DISCUSSION and from 72% (36/50) to 80%
https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX(40/50; P=.90) for the level 5 questionnaire (Figure S3 in
Multimedia Appendix 20).
At least half of all learners stated that they would make a change
to their practice following their participation in the activities(levels 3 to 4 questionnaire: touchMDT—25/50, 50% andtouchPANEL DISCUSSION—29/50, 58%; level 5questionnaire: 26/50, 52%). The reasons given for not making
a change were split between the need for more education (levels
3 to 4 questionnaire: touchMDT—12/50, 24% and touchPANELDISCUSSION—7/50, 14%; level 5 questionnaire: 13/50, 26%)
and practical limitations (levels 3 to 4 questionnaire:
touchMDT—13/50,
26%
andtouchPANELDISCUSSION—14/50, 28%; level 5 questionnaire: 11/50, 22%).

Identification of Outstanding Educational Gaps
In ranking the unmet educational needs, touchMDT learners
highlighted “strategies for implementing SDM in clinicalpractice” and “effective communication strategies in SDM to
optimize patient outcomes” as key areas for future education,and touchPANEL DISCUSSION learners highlighted “using
genetic and biomarker testing to guide therapy choice” and“individualization of treatment choice in high-risk hormone
receptor–positive human epidermal growth factor receptor 2
negative (HR+ HER2–) EBC.” From the level 5 questionnaire,learners highlighted “using genetic and biomarker testing to
guide therapy choice for patients with high-risk HR+ HER2EBC” and “understanding the latest data on emerging treatments
for high-risk HR+ HER2- EBC” as their most important unmet
educational needs (Table S1 in Multimedia Appendix 20).
For the touchPANEL DISCUSSION, high levels of knowledge
and competence were achieved for all 3 learning objectives.
However, some unmet needs remained after participation intouchMDT, with 60% (30/50) and 36% (18/50) of the learners,
respectively, being unable to demonstrate declarative knowledge
JMIR Form Res 2024 | vol. 8 | e50931 | p. 7
(page number not for citation purposes)

JMIR FORMATIVE RESEARCHof the current consensus on the use of patient decision aids tosupport SDM in breast cancer and being unable to demonstratecompetence in supporting patients in understanding data andreaching a shared decision regarding neoadjuvant systemic
therapy. In the level 5 questionnaire, ≥36% (18/50) of thelearners were unable to provide the best clinical responses tothe questions on reducing the patient burden (18/50, 36%), riskof recurrence (20/50, 40%), and chemotherapy treatment
decisions (26/50, 52%).

Gnant et al

Discussion

address knowledge gaps in complex and rapidly evolving
medical fields. Although published data on outcomes following
educational activities with similar methodologies remain limited,
the results of this study are consistent with the results of a small
number of previous studies in other disease areas [55-57].
Notably, the field of breast cancer is evolving rapidly, and it isessential for oncologists, particularly nonspecialists, to remain
informed of the latest findings and recommendations [19-26].
Therefore, focused web-based learning activities may also be
valuable for general oncologists to enable them to keep
up-to-date with changes in specialist fields.

Principal Findings

Strengths and Limitations

This study evaluated the impact of 2 faculty-led,
CME-accredited web-based learning activities on EBC.
Participation in both activities resulted in high satisfaction scoresand statistically significant improvements in self-reported
knowledge, competence, and performance. Subgroup analyses
indicated that the education was broadly beneficial, regardlessof region, specialty, and experience. Patient record data analysis
showed statistically significant differences in how the risk ofrecurrence was determined by responders and learners. The
impact of the activities on therapeutic choice was limited,
although this may be because of factors such as local guidelines,
access to treatments, and individual patient characteristics.
Approximately half of the learners (80/150, 53%) stated thatthey would change their practice in response to the educational
material, further suggesting that the activities were valuable
learning tools.

There were several strengths associated with this study:

Participation in these web-based activities was considerably
higher than could be achieved with traditional face-to-faceprograms and may also reflect the involvement of amultinational panel of recognized experts and the availabilityof translations. The latter is a major difference from traditional
conference-based activities, as they are usually only available
in 1 language, typically English.
Although many learners stated that they would change theirpractice, there remained a considerable number who wereuncertain or who would not change their practice at the current
time. The reasons for not changing practice were largely split
between the requirement for more education and practicallimitations. Although learners were not canvassed for specificdetails regarding practical limitations, we can speculate thatthey may relate to regional or national differences in institutional
resources and structure or regulatory and reimbursement
approvals of treatment. Indeed, documented barriers to change
in clinical practice include institution-related issues such as
flawed leadership and communication and insufficientorganizational culture shift, and practical restrictions such as
recently updated guidelines and access to emerging drugs [53].
In addition, a 2022 study on factors that influenced HCPs’ intent
to put newly acquired learning into practice suggested that alack of belief in one’s capabilities and the potential
consequences of adopting new clinical behaviors can prevent
HCPs from translating education into practice [54].
These data demonstrate the potential value of succinct
web-based learning activities that are specifically designed tohttps://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

1.
2.

3.

4.

5.

6.

7.

The activities provided a holistic curriculum and unbiased
education focused on clearly defined learning objectives.
The activities were translated into several languages andwere available on a free-to-access website, which likely
increased their global accessibility.
A broad target audience was successfully reached,highlighting the importance of the multichannel publicity
strategy and the inclusion of faculty from often
underrepresented geographies (Latin America andAsia-Pacific).
The multidisciplinary team approach and inclusion of asurvivor of breast cancer ensured that the activities weredirectly applicable to the participants’ daily practice; theemotive element can also aid learning and retention [58].
The format used in the touchPANEL DISCUSSION allowed
the latest evidence to be put into context for practicing
physicians.
The development of the education in a short time frame
allowed for emerging data to be communicated to the HCPs
potentially ahead of publications and conventional
educational formats.
A major strength was using an objective measure of
performance to complement the more subjective
self-reported data.

This study also has several limitations:
1.

2.

3.

4.

The conclusions that could be drawn from the main andsubgroup analyses were speculative and limited because of
the relatively small sample size.
All medical educational studies are affected by inevitableself-selection bias; that is, the HCPs who feel they lackknowledge on a specific topic are more likely to participate
and, therefore, show a bigger impact of the education thanmight be the case for a wider audience.
This study only evaluated outcomes following 2 specific
CME activities in EBC, and the conclusions may not be
generalizable to other web-based learning formats.
An independent samples method was used for the levels 3
to 4 questionnaires and the self-reported level 5questionnaire to avoid pre-exposure bias; however, thismeant that the direct impact of education on each learner
could not be determined. Therefore, we cannot rule out thepossibility that other sources of education may havecontributed to the reported outcomes.
JMIR Form Res 2024 | vol. 8 | e50931 | p. 8
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
Despite these limitations, this study offers the firstcomprehensive assessment of such activities using an accepted
methodology.

Identification of Needs for Further Education in
High-Risk EBC
Notably, only 30% (15/50) and 48% (24/50) of the touchMDT
and touchPANEL DISCUSSION learners, respectively,
answered all questions correctly in the levels 3 to 4
questionnaires. Similarly, only 20% (10/50) of the learners gave
the best clinical response for all questions in the level 5questionnaire. This may, in part, reflect a mixed audience but
suggests that further education may be warranted to reinforce
the content of these activities. To refine future educational
activities, we identified several unmet needs during this study
from the questions that were incorrectly answered after theactivities. In addition, several self-reported educational gaps
were identified, including implementing SDM in clinicalpractice, using genetic and biomarker testing to guide therapy
choices, and understanding the latest data on emerging
treatments.

Future Directions
Limitations of traditional CME approaches in affecting changes
in physician performance have been previously reported [59];however, the optimal format for continuing educational activities
in both breast cancer and the wider medical field remains to be
fully defined. This study and other similar studies show evidence
that more engaging approaches have the potential to improvethe effectiveness of CME [55-57,60,61]. Evaluation of thelong-term impacts of CME on HCP performance, alongside

Gnant et alassessment against Moore’s Levels 6 and 7 [40] to determine
the effect of education on patient (level 6) and community (level7) health will add further value to CME programs in the future.
Activity formats that may be particularly relevant include
experts sharing insights drawn from experience and their own
clinical practice, conversational and panel discussion formats,and expert interviews supplemented by vignettes of consultation
strategies. Furthermore, web-based delivery may offer scopefor developing more individualized approaches. Web-based
activities can also be made available as enduring materials,
providing a resource designed for optimal HCP accessibility.
Future studies will need to be designed to evaluate thecomparative effectiveness of these different approaches to CME.
In addition, in future studies, a larger sample size will increase
the statistical power of the analyses.

Conclusions
Our study demonstrated that focused, free-to-access web-based
CME activities on EBC were associated with improvements in
HCP knowledge, competence, and self-reported performance.
This study addressed the gaps in the provision of HCP-focused
web-based IME programs that encompass recent advances in
the high-risk EBC treatment landscape, the need for andapproaches to effective SDM, and the gaps within the literaturein terms of reporting outcomes from web-based CME activities.
The identified unmet needs should be used to inform the designof future educational activities for this disease area. Further
studies evaluating the long-term impact of education on HCP
performance and on patient and community health will bevaluable in defining the clinical impact of CME and the mosteffective channels for its delivery.

Acknowledgments
Medical writing support was provided by Steph Carter, and editorial support was provided by Cheryl Brant, both from touch
Independent Medical Education (touchIME), Cheshire, United Kingdom. The educational activities were jointly provided by the
University of South Florida Health and touchIME. The authors would like to thank the touchIME audience outreach team (Joel
Turner, Hannah Morton-Fishwick, and Louise Jarman) for the collection and analysis of the level 1 data. The authors would liketo thank former employee of touchIME Sonia Mills for additional editorial support. Eli Lilly provided an independent medical
education grant to fund the educational activity, medical writing, and editorial support but did not contribute directly to manuscript
development, the decision to submit, or the submission process.

Data Availability
The data sets generated during and/or analyzed during this study are available from the corresponding author on reasonable
request.

Authors' Contributions
All authors contributed to the conceptualization, formal analysis of the data, and writing (review and editing) of the manuscript.
KB, SN, A Noble, A Nunn and CS contributed to the investigation, methodology, project administration, visualization of thestudy, and writing (original draft) of the manuscript. SN contributed to the funding acquisition for and supervision of the manuscript.
A Noble contributed to data curation.

Conflicts of Interest
CB reports receiving advisory board fees from AstraZeneca, Bayer, Boehringer Ingelheim, Eisai, GlaxoSmithKline, Lilly, MSD,
Novartis, Pfizer, Genentech (Roche), Sanofi, and Zodiac Pharma; consultancy fees from AstraZeneca, Bayer, Boehringer Ingelheim,
Eisai, GlaxoSmithKline, Lilly, MSD, Novartis, Pfizer, Genentech (Roche), Sanofi, and Zodiac Pharma; grants and researchsupport from AbbVie, Abraxis Biosciences, AB Science, Amgen, Asana Biosciences, Astellas Pharma, AstraZeneca, BioMarin,
Boehringer Ingelheim, Bristol Myers Squibb, Daiichi Sankyo, Exelixis, GlaxoSmithKline, ImClone Systems, LEO Pharma, Lilly,
https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e50931 | p. 9
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Gnant et al

Medivation, MSD, Merck KGaA, Merrimack, Millennium, Mylan, Novartis, Pfizer, PharmaMar, Polyphor, Genentech (Roche),
Sanofi, Shanghai Henlius Biotech, and Taiho Pharmaceutical; and holding stock and shares (self-managed) in MEDSIR andTummi. FB reports receiving advisory board fees from Eisai, Eli Lilly, Gilead, MSD, Novartis, Pfizer, and Roche and consultancy
fees from Eli Lilly. MG reports receiving personal fees and travel support from AstraZeneca, Daiichi Sankyo, Eli Lilly,
Menarini-Stemline, MSD, Novartis, Pierre Fabre, and Veracyte, and his immediate family member is employed by Sandoz. NH
reports receiving consultancy fees from Gilead, Roche, Sandoz, and Seagen and speaker bureau fees from AstraZeneca, Daiichi
Sankyo, Gilead, Lilly, MSD, Novartis, Pierre Fabre, Pfizer, Roche, and Seagen. CSH reports receiving advisory board fees from
AstraZeneca, Lilly, Pfizer, Novartis, and Roche; grants and research support from AstraZeneca, Daiichi Sankyo, EirGenix, Lilly,
MSD, Novartis, OBI Pharma, Pfizer, and Roche; and speaker bureau fees from AstraZeneca, Novartis, Pfizer, and Roche. KLA
has no interests or relationships or affiliations to disclose. KB, SN, A Noble, A Nunn and CS are employees of touch Independent
Medical Education Limited.

Multimedia Appendix 1
Activity development process.
[DOCX File , 25 KB-Multimedia Appendix 1]

Multimedia Appendix 2
Assessment of participation (level 1) and satisfaction (level 2).
[DOCX File , 22 KB-Multimedia Appendix 2]

Multimedia Appendix 3
Topics included in levels 3 and 4 and level 5 outcomes questionnaires.
[DOCX File , 13 KB-Multimedia Appendix 3]

Multimedia Appendix 4
Questions included in the levels 3 and 4 outcomes questionnaire for the touchMDT activity.
[DOCX File , 16 KB-Multimedia Appendix 4]

Multimedia Appendix 5
Questions included in the levels 3 and 4 outcomes questionnaire for the touchPANEL DISCUSSION activity.
[DOCX File , 16 KB-Multimedia Appendix 5]

Multimedia Appendix 6
Questions included in the level 5 outcomes questionnaire for the touchMDT and touchPANEL DISCUSSION activities.
[DOCX File , 16 KB-Multimedia Appendix 6]

Multimedia Appendix 7
Unmet educational need options included in levels 3 to 4 and level 5 questionnaires.
[DOCX File , 13 KB-Multimedia Appendix 7]

Multimedia Appendix 8
Single transferable vote system methodology.
[DOCX File , 12 KB-Multimedia Appendix 8]

Multimedia Appendix 9
Engagement results and demographics for the touchMDT and touchPANEL DISCUSSION activities.
[DOCX File , 14 KB-Multimedia Appendix 9]

Multimedia Appendix 10
Mean satisfaction scores of levels 2 to 4 questionnaire (out of a maximum of 5) completed by touchMDT and touchPANELDISCUSSION learners.
[DOCX File , 13 KB-Multimedia Appendix 10]

https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e50931 | p. 10(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Gnant et alMultimedia Appendix 11
Multichannel publicity reach and impact for the touchMDT and touchPANEL DISCUSSION activities.
[DOCX File , 13 KB-Multimedia Appendix 11]

Multimedia Appendix 12
Mean number of correct responses for the levels 3 and 4 outcomes questionnaire before and after the launch of touchMDT by(A) country, (B) years of experience, and (C) specialty subgroups of the respondents and learners.
[DOCX File , 71 KB-Multimedia Appendix 12]

Multimedia Appendix 13
Mean number of correct responses for the levels 3 and 4 outcomes questionnaire before and after the launch of touchPANELDISCUSSION by (A) country, (B) level of experience, and (C) specialty of the respondents and learners.
[DOCX File , 68 KB-Multimedia Appendix 13]

Multimedia Appendix 14
Summary of correct responses for individual topics for the levels 3 and 4 outcomes questionnaire before and after the launch of(A) touchMDT and (B) touchPANEL DISCUSSION.
[DOCX File , 60 KB-Multimedia Appendix 14]

Multimedia Appendix 15
Summary of correct responses for individual topics for the level 5 outcomes questionnaire before and after the launch of touchMDT
and touchPANEL DISCUSSION.
[DOCX File , 31 KB-Multimedia Appendix 15]

Multimedia Appendix 16
Mean number of correct responses for the level 5 outcomes questionnaire before and after the launch of touchMDT and touchPANELDISCUSSION by (A) country, (B) level of experience, and (C) specialty of the respondents and learners.
[DOCX File , 102 KB-Multimedia Appendix 16]

Multimedia Appendix 17
Patient characteristics from patient records data (level 5) submitted by responders and learners for the touchPANEL DISCUSSION
activity.
[DOCX File , 15 KB-Multimedia Appendix 17]

Multimedia Appendix 18
Combinations of measures used to determine patient risk for recurrence from patient records data (level 5) submitted by responders
and learners for the touchPANEL DISCUSSION activity.
[DOCX File , 15 KB-Multimedia Appendix 18]

Multimedia Appendix 19
Measures used to determine patient risk for recurrence reported by respondents and learners in the level 5 patient records
questionnaire.
[DOCX File , 42 KB-Multimedia Appendix 19]

Multimedia Appendix 20
Ki-67 use and Ki-67 index scores reported by respondents and learners in the level 5 patient records questionnaire (Figure S1);
adjuvant or planned adjuvant therapy reported by respondents and learners in the level 5 patient records questionnaire (FigureS2); changes in self-reported confidence in (A) levels 3 to 4 questionnaires and (B) level 5 questionnaires following participationin the touchMDT and touchPANEL DISCUSSION activities (Figure S3); unmet educational needs identified by the touchMDT
and touchPANEL DISCUSSION learners (Table S1).
[DOCX File , 423 KB-Multimedia Appendix 20]

References
https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e50931 | p. 11(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
1.
2.

3.
4.

5.

6.

7.
8.

9.

10.

11.

12.

13.
14.

15.
16.

17.

18.

19.
20.

21.

Katz SJ, Belkora J, Elwyn G. Shared decision making for treatment of cancer: challenges and opportunities. J Oncol Pract.
May 2014;10(3):206-208. [FREE Full text] [doi: 10.1200/JOP.2014.001434] [Medline: 24839284]
Gustafson A. Reducing patient uncertainty: implementation of a shared decision-making process enhances treatment quality
and provider communication. Clin J Oncol Nurs. Feb 01, 2017;21(1):113-115. [doi: 10.1188/17.CJON.113-115] [Medline:
28107332]
NQP shared decision making action brief. National Quality Forum. Oct 2017. URL: https://www.qualityforum.org/
Publications/2017/10/NQP_Shared_Decision_Making_Action_Brief.aspx [accessed 2024-01-11]
Barrios C, Sánchez-Vanegas G, Villarreal-Garza C, Ossa A, Lombana MA, Monterrosa-Blanco A, et al. Barriers andfacilitators to provide multidisciplinary care for breast cancer patients in five Latin American countries: adescriptive-interpretative qualitative study. Lancet Reg Health Am. Jul 2022;11:100254. [FREE Full text] [doi:10.1016/j.lana.2022.100254] [Medline: 36778924]
Cardoso F, Kyriakides S, Ohno S, Penault-Llorca F, Poortmans P, Rubio IT, et al. Early breast cancer: ESMO Clinical
Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol. Oct 01, 2019;30(10):1674. [FREE Full text] [doi:10.1093/annonc/mdz189] [Medline: 31236598]
Cardoso F, MacNeill F, Penault-Llorca F, Eniu A, Sardanelli F, Nordström EB, et al. Why is appropriate healthcare
inaccessible for many European breast cancer patients? - The EBCC 12 manifesto. Breast. Feb 2021;55:128-135. [FREEFull text] [doi: 10.1016/j.breast.2020.12.010] [Medline: 33461060]
National Comprehensive Cancer Network home page. National Comprehensive Cancer Network. URL: https://www.
nccn.org/ [accessed 2024-01-11]
Spronk I, Burgers JS, Schellevis FG, van Vliet LM, Korevaar JC. The availability and effectiveness of tools supporting
shared decision making in metastatic breast cancer care: a review. BMC Palliat Care. May 11, 2018;17(1):74. [FREE Fulltext] [doi: 10.1186/s12904-018-0330-4] [Medline: 29747628]
Stacey D, Légaré F, Lewis K, Barry MJ, Bennett CL, Eden KB, et al. Decision aids for people facing health treatment orscreening decisions. Cochrane Database Syst Rev. Apr 12, 2017;4(4):CD001431. [FREE Full text] [doi:10.1002/14651858.CD001431.pub5] [Medline: 28402085]
Ankolekar A, Dekker A, Fijten R, Berlanga A. The benefits and challenges of using patient decision aids to support shareddecision making in health care. JCO Clin Cancer Inform. Dec 2018;2:1-10. [FREE Full text] [doi: 10.1200/CCI.18.00013][Medline: 30652607]
Savelberg W, Boersma LJ, Smidt M, Goossens MF, Hermanns R, van der Weijden T. Does lack of deeper understandingof shared decision making explains the suboptimal performance on crucial parts of it? An example from breast cancer care.
Eur J Oncol Nurs. Feb 2019;38:92-97. [FREE Full text] [doi: 10.1016/j.ejon.2018.12.004] [Medline: 30717943]
Covvey JR, Kamal KM, Gorse EE, Mehta Z, Dhumal T, Heidari E, et al. Barriers and facilitators to shared decision-making
in oncology: a systematic review of the literature. Support Care Cancer. May 2019;27(5):1613-1637. [doi:10.1007/s00520-019-04675-7] [Medline: 30737578]
Pourgholam N, Shoghi M, Borimnejad L. Patients' lived experiences of the paternalistic care behavior: a qualitative study.
J Caring Sci. Aug 2022;11(3):163-171. [FREE Full text] [doi: 10.34172/jcs.2022.10] [Medline: 36247035]
Thompson GA, Segura J, Cruz D, Arnita C, Whiffen LH. Cultural differences in patients' preferences for paternalism:
comparing Mexican and American patients' preferences for and experiences with physician paternalism and patient autonomy.
Int J Environ Res Public Health. Aug 26, 2022;19(17):10663. [FREE Full text] [doi: 10.3390/ijerph191710663] [Medline:
36078378]
Gil-Gil M, Alba E, Gavilá J, de la Haba-Rodríguez J, Ciruelos E, Tolosa P, et al. The role of CDK4/6 inhibitors in earlybreast cancer. Breast. Aug 2021;58:160-169. [FREE Full text] [doi: 10.1016/j.breast.2021.05.008] [Medline: 34087775]
Harbeck N, Rastogi P, Martin M, Tolaney SM, Shao ZM, Fasching PA, et al. Adjuvant abemaciclib combined with endocrine
therapy for high-risk early breast cancer: updated efficacy and Ki-67 analysis from the monarchE study. Ann Oncol. Dec
2021;32(12):1571-1581. [FREE Full text] [doi: 10.1016/j.annonc.2021.09.015] [Medline: 34656740]
Gnant M, Dueck AC, Frantal S, Martin M, Burstein HJ, Greil R, et al. Adjuvant palbociclib for early breast cancer: the
PALLAS trial results (ABCSG-42/AFT-05/BIG-14-03). J Clin Oncol. Jan 20, 2022;40(3):282-293. [FREE Full text] [doi:10.1200/JCO.21.02554] [Medline: 34874182]
Loibl S, Marmé F, Martin M, Untch M, Bonnefoi H, Kim SB, et al. Palbociclib for residual high-risk invasive HR-positive
and HER2-negative early breast cancer-the Penelope-B trial. J Clin Oncol. May 10, 2021;39(14):1518-1530. [doi:10.1200/JCO.20.03639] [Medline: 33793299]
Harbeck N. Risk-adapted adjuvant therapy of luminal early breast cancer in 2020. Curr Opin Obstet Gynecol. Feb 01,
2021;33(1):53-58. [doi: 10.1097/GCO.0000000000000679] [Medline: 33337614]
Turner N, Howell S, Jhaveri K, Gomez H, Toi M, Hu X, et al. 350TiP A phase III trial of capivasertib and fulvestrant versus
placebo and fulvestrant in patients with HR+/HER2− breast cancer (CAPItello-291). Ann Oncol. Sep 2020;31:S388-S389.
[doi: 10.1016/j.annonc.2020.08.452]
Ditsch N, Wöcke A, Untch M, Jackisch C, Albert US, Banys-Paluchowski M, et al. AGO recommendations for the diagnosis
and treatment of patients with early breast cancer: update 2022. Breast Care (Basel). Aug 2022;17(4):403-420. [FREE Fulltext] [doi: 10.1159/000524879] [Medline: 36156915]

https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

Gnant et al

JMIR Form Res 2024 | vol. 8 | e50931 | p. 12(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
22.

23.

24.

25.

26.
27.
28.

29.

30.
31.
32.
33.

34.

35.

36.

37.
38.
39.
40.
41.

42.

43.

Burstein HJ, Curigliano G, Thürlimann B, Weber WP, Poortmans P, Regan MM, et al. Customizing local and systemic
therapies for women with early breast cancer: the St. Gallen International Consensus Guidelines for treatment of earlybreast cancer 2021. Ann Oncol. Oct 2021;32(10):1216-1235. [FREE Full text] [doi: 10.1016/j.annonc.2021.06.023] [Medline:
34242744]
Moy B, Rumble RB, Carey LA, Chemotherapy and Targeted Therapy for Endocrine-Pretreated or Hormone
Receptor–Negative Metastatic Breast Cancer Expert Panel. Chemotherapy and targeted therapy for endocrine-pretreatedor hormone receptor-negative metastatic breast cancer: ASCO guideline rapid recommendation update. J Clin Oncol. Feb
20, 2023;41(6):1318-1320. [doi: 10.1200/JCO.22.02807] [Medline: 36626701]
Gradishar WJ, Moran MS, Abraham J, Aft R, Agnese D, Allison KH, et al. Breast cancer, version 3.2022, NCCN clinicalpractice guidelines in oncology. J Natl Compr Canc Netw. Jun 2022;20(6):691-722. [doi: 10.6004/jnccn.2022.0030][Medline: 35714673]
Gnant M, Fitzal F, Rinnerthaler G, Steger GG, Greil-Ressler S, Balic M, et al. Duration of adjuvant aromatase-inhibitor
therapy in postmenopausal breast cancer. N Engl J Med. Jul 29, 2021;385(5):395-405. [doi: 10.1056/NEJMoa2104162][Medline: 34320285]
Gnant M, Frantal M, Pfeiler G, Steger GG, Egle D, Greil R, et al. Long-term outcomes of adjuvant denosumab in breast
cancer. NEJM Evidence. Nov 22, 2022;1(12). [FREE Full text] [doi: 10.1056/evidoa2200162]
Harbeck N, Gnant M. Breast cancer. Lancet. Mar 18, 2017;389(10074):1134-1150. [doi: 10.1016/S0140-6736(16)31891-8][Medline: 27865536]
Thomssen C, Balic M, Harbeck N, Gnant M. St. Gallen/Vienna 2021: a brief summary of the consensus discussion on
customizing therapies for women with early breast cancer. Breast Care (Basel). Apr 2021;16(2):135-143. [FREE Full text][doi: 10.1159/000516114] [Medline: 34002112]
Kristensen N, Nymann C, Konradsen H. Implementing research results in clinical practice- the experiences of healthcare
professionals. BMC Health Serv Res. Feb 10, 2016;16(1):48. [FREE Full text] [doi: 10.1186/s12913-016-1292-y] [Medline:
26860594]
Beatty PA. Coping with abundance: the burden of progress in medical oncology. Oncologist. 2012;17(2):294-295. [FREEFull text] [doi: 10.1634/theoncologist.2011-0230] [Medline: 22367600]
Charlton M, Schlichting J, Chioreso C, Ward M, Vikas P. Challenges of rural cancer care in the United States. Oncology(Williston Park). Sep 2015;29(9):633-640. [Medline: 26384798]
Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational
research. J R Soc Med. Dec 2011;104(12):510-520. [FREE Full text] [doi: 10.1258/jrsm.2011.110180] [Medline: 22179294]
Ngenzi JL, Scott RE, Mars M. Information and communication technology to enhance continuing professional development(CPD) and continuing medical education (CME) for Rwanda: a scoping review of reviews. BMC Med Educ. Apr 29,
2021;21(1):245. [FREE Full text] [doi: 10.1186/s12909-021-02607-w] [Medline: 33926419]
Meani F, Kovacs T, Wandschneider W, Costa A, Pagani O. Multidisciplinary blended learning to build a breast cancer
specialist career: survey on the perspective of the first 2 cohorts of the ESO-ULM Certificate of Competence in Breast
cancer (CCB). BMC Med Educ. May 05, 2022;22(1):344. [FREE Full text] [doi: 10.1186/s12909-022-03414-7] [Medline:
35513883]
Dorkhom N, Middleton D, Guedj P, Chen P, Lucero K, Curigliano G. An outcomes summary of the clinical advances
curriculum on CDK4 and 6 inhibition in breast cancer. Ann Oncol. May 2020;31:S77. [FREE Full text] [doi:10.1016/j.annonc.2020.03.271]
Walling BM, Totzkay D, Silk KJ, Boumis JK, Thomas B, Smith S. Evaluating the feasibility of continuing medical education
for disseminating emerging science on the breast cancer and environment connection. J Health Commun. Jun 03,
2021;26(6):391-401. [FREE Full text] [doi: 10.1080/10810730.2021.1946728] [Medline: 34292846]
Loibl S, de Azambuja E, Johnston S, Dawood S. Raising the bar for patients with HR+/HER2- early breast cancer. ESMO.
May 25, 2022. URL: https://tinyurl.com/2cb3scha [accessed 2024-01-11]
Disease education resources: oncology. Lilly. URL: https://tinyurl.com/mup6jche [accessed 2024-01-11]
Johnston SR. Assessing risk in early HR+/HER2- breast cancer management: a patient clinic. Medscape. May 26, 2021.
URL: https://www.medscape.org/viewarticle/951798 [accessed 2024-01-11]
Moore DEJ, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment
throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1-15. [doi: 10.1002/chp.20001] [Medline: 19288562]
Bannister J, Neve M, Kolanko C. Increased educational reach through a microlearning approach: can higher participationtranslate to improved outcomes? J Eur CME. Oct 20, 2020;9(1):1834761. [FREE Full text] [doi:10.1080/21614083.2020.1834761] [Medline: 33354408]
Milgrom ZZ, Severance TS, Scanlon CM, Carson AT, Janota AD, Vik TA, et al. An evaluation of an Extension forCommunity Healthcare Outcomes (ECHO) intervention in cancer prevention and survivorship care. BMC Med Inform
Decis Mak. May 17, 2022;22(1):135. [FREE Full text] [doi: 10.1186/s12911-022-01874-x] [Medline: 35581580]
Ivanova A, Baliunas D, Ahad S, Tanzini E, Dragonetti R, Fahim M, et al. Performance change in treating tobacco addiction:
an online, interprofessional, facilitated continuing education course (TEACH) evaluation at Moore's level 5. J Contin Educ
Health Prof. Jan 01, 2021;41(1):31-38. [doi: 10.1097/CEH.0000000000000328] [Medline: 33433131]

https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

Gnant et al

JMIR Form Res 2024 | vol. 8 | e50931 | p. 13(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
44.

45.
46.
47.

48.
49.
50.
51.
52.
53.
54.

55.

56.
57.
58.
59.

60.

61.

Gnant et alBednář J, Buchanan J, Michels NR, Kalda R, Vajer P, Miftode R, et al. Current state of the evidence-base for effective
continuous medical education for family medicine and general practice. European Academy of Teachers in General Practiceand Family Medicine (EURACT). 2021. URL: https://tinyurl.com/4469rddt [accessed 2024-02-20]
Rosenman R, Tennekoon V, Hill LG. Measuring bias in self-reported data. Int J Behav Healthc Res. Oct 2011;2(4):320-332.
[FREE Full text] [doi: 10.1504/IJBHR.2011.043414] [Medline: 25383095]
Althubaiti A. Information bias in health research: definition, pitfalls, and adjustment methods. J Multidiscip Healthc.
2016;9:211-217. [FREE Full text] [doi: 10.2147/JMDH.S104807] [Medline: 27217764]
Regulation (EU) 2016/679 of the European parliament and of the council of 27 April 2016. Official Journal of the European
Union. May 4, 2016. URL: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679 [accessed2024-01-19]
Code of conduct. European Pharmaceutical Market Research Association. Sep 2023. URL: https://tinyurl.com/242pp57k[accessed 2024-01-26]
Guidelines and legislation. British Healthcare Business Intelligence Association. URL: https://www.bhbia.org.uk/
guidelines-and-legislation [accessed 2024-01-26]
Understanding health research. UK Research and Innovation. URL: https://tinyurl.com/4u32dw35 [accessed 2024-01-26]
Research Ethics Committee review. National Health Service Health Research Authority. URL: https://tinyurl.com/mswuu6c6[accessed 2024-01-26]
Tideman N. The single transferable vote. J Econ Perspect. Feb 01, 1995;9(1):27-38. [doi: 10.1257/jep.9.1.27]
Barach PR. Addressing barriers for change in clinical practice. In: Guidet B, Valentin A, Flaatten H, editors. Quality
Management in Intensive Care: A Practical Guide. Cambridge, UK. Cambridge University Press; 2016.
Bergeron L, Décary S, Djade CD, Daniel SJ, Tremblay M, Rivest LP, et al. Factors associated with specialists' intention
to adopt new behaviors after taking web-based continuing professional development courses: cross-sectional study. JMIR
Med Educ. Jun 02, 2022;8(2):e34299. [FREE Full text] [doi: 10.2196/34299] [Medline: 35476039]
Harris SB, Idzik S, Boasso A, Neunie SQ, Noble AD, Such HE, et al. The educational impact of web-based, faculty-led
continuing medical education programs in type 2 diabetes: a survey study to analyze changes in knowledge, competence,and performance of health care professionals. JMIR Med Educ. Oct 14, 2022;8(4):e40520. [FREE Full text] [doi:10.2196/40520] [Medline: 36102282]
Larkin A, Schrand J, Le A. Success of CME at improving knowledge of clinical use of real-time CGM. Diabetes.
2021;70(Supplement_1):614P. [FREE Full text] [doi: 10.2337/db21-614-P]
Rohani-Montez C, Gunesekera D, Coppo P. Effectiveness of interactive case-based online education in improving Attp
diagnosis and treatment selection. Blood. Nov 2020;136(Supplement 1):36-37. [doi: 10.1182/blood-2020-134292]
Tyng CM, Amin HU, Saad MN, Malik AS. The influences of emotion on learning and memory. Front Psychol. Aug 24,
2017;8:1454. [FREE Full text] [doi: 10.3389/fpsyg.2017.01454] [Medline: 28883804]
Davis D, O'Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical
education: do conferences, workshops, rounds, and other traditional continuing education activities change physician
behavior or health care outcomes? JAMA. Sep 01, 1999;282(9):867-874. [doi: 10.1001/jama.282.9.867] [Medline: 10478694]
Diehl LA, Souza RM, Gordan PA, Esteves RZ, Coelho IC. InsuOnline, an electronic game for medical education on insulin
therapy: a randomized controlled trial with primary care physicians. J Med Internet Res. Mar 09, 2017;19(3):e72. [FREEFull text] [doi: 10.2196/jmir.6944] [Medline: 28279950]
Wiecha J, Heyden R, Sternthal E, Merialdi M. Learning in a virtual world: experience with using second life for medical
education. J Med Internet Res. Jan 23, 2010;12(1):e1. [FREE Full text] [doi: 10.2196/jmir.1337] [Medline: 20097652]

Abbreviations
Akt: protein kinase B
CME: continuing medical education
EBC: early breast cancer
HCP: health care professional
SDM: shared decision-making
touchIME: touch Independent Medical Education

https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e50931 | p. 14(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Gnant et al

Edited by A Mavragani; submitted 24.07.23; peer-reviewed by J Lapointe, N Than, S Ganesh; comments to author 09.01.24; revised
version received 06.02.24; accepted 07.02.24; published 21.03.24
Please cite as:
Gnant M, Abdullah KL, Boyle F, Huang CS, Bickford K, Neunie S, Noble A, Nunn A, Sproat C, Harbeck N, Barrios C
Assessing Knowledge, Competence, and Performance Following Web-Based Education on Early Breast Cancer Management: HealthCare Professional Questionnaire Study and Anonymized Patient Records Analysis
JMIR Form Res 2024;8:e50931
URL: https://formative.jmir.org/2024/1/e50931
doi: 10.2196/50931
PMID: 38512328©Michael Gnant, Khatijah Lim Abdullah, Frances Boyle, Chiun-Sheng Huang, Katie Bickford, Sola Neunie, Alexander Noble,
Anne Nunn, Caroline Sproat, Nadia Harbeck, Carlos Barrios. Originally published in JMIR Formative Research(https://formative.jmir.org), 21.03.2024. This is an open-access article distributed under the terms of the Creative Commons
Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction
in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete
bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license
information must be included.

https://formative.jmir.org/2024/1/e50931

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e50931 | p. 15(page number not for citation purposes)