JMIR FORMATIVE RESEARCH

Beccia et al

Original Paper

Exploring the User Acceptability and Feasibility of a Clinical
Decision Support Tool Designed to Facilitate Timely Diagnosis of
New-Onset Type 1 Diabetes in Children: Qualitative Interview
Study Among General Practitioners
Chiara Beccia1,2, BBiomed (Hons), DLang; Barbara Hunter1, BA (Hons), PhD; Jo-Anne Manski-Nankervis1,2,3, BSci
(Hons), MBBS (Hons), CHIA, PhD; Mary White4,5,6, MB BCh BAO, MD
1

Department of General Practice and Primary Care, The University of Melbourne, Melbourne, Australia

2

National Health and Medical Research Council Centre for Research Excellence in Digital Technology to Transform Chronic Disease Outcomes,
Melbourne, Australia
3

Primary Care and Family Medicine, Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore

4Royal Children's Hospital, Melbourne, Australia

5

Melbourne School of Population and Global Health, The University of Melbourne, Melbourne, Australia

6

Health Services and Economics Research Unit, Murdoch Children's Research Institute, Melbourne, Australia

Corresponding Author:
Chiara Beccia, BBiomed (Hons), DLang
Department of General Practice and Primary Care
The University of Melbourne
Medical Building (181)
Melbourne, 3010
Australia
Phone: 61 0468480327
Email: cbeccia@student.unimelb.edu.au

Abstract
Background: Up to half of the children with new-onset type 1 diabetes present to the hospital with diabetic ketoacidosis, alife-threatening condition that can develop because of diagnostic delay. Three-quarters of Australian children visit their generalpractitioner (GP) the week before presenting to the hospital with diabetic ketoacidosis. Our prototype, DIRECT-T1DM(Decision-Support for Integrated, Real-Time Evaluation and Clinical Treatment of Type 1 Diabetes Mellitus), is an electronicclinical decision support tool that promotes immediate point-of-care testing in general practice to confirm the suspicion of diabetes.
This avoids laboratory testing, which has been documented internationally as a cause of diagnostic delay.
Objective: In this investigation, we aimed to pilot and assess the feasibility and acceptability of our prototype to GP end users.
We also explored the challenges of diagnosing type 1 diabetes in the Australian general practice context.
Methods: In total, 4 GPs, a pediatric endocrinologist, and a PhD candidate were involved in conceptualizing the DIRECT-T1DM
prototype, which was developed at the Department of General Practice and Primary Care at the University of Melbourne.
Furthermore, 6 GPs were recruited via convenience sampling to evaluate the tool. The study involved 3 phases: a presimulationinterview, simulated clinical scenarios, and a postsimulation interview. The interview guide was developed using the Consolidated
Framework for Implementation Research (CFIR) as a guide. All phases of the study were video, audio, and screen recorded.
Audio recordings were transcribed by the investigating team. Analysis was carried out using CFIR as the underlying framework.
Results: Major themes were identified among three domains and 7 constructs of the CFIR: (1) outer setting—time pressure,difficulty in diagnosing pediatric type 1 diabetes, and secondary care considerations influenced GPs’ needs regarding
DIRECT-T1DM; (2) inner setting—DIRECT-T1DM fits within existing workflows, it has a high relative priority due to its
importance in patient safety, and GPs exhibited high tension for change; and (3) innovation—design recommendations included
altering coloring to reflect urgency, font style and bolding, specific language, information and guidelines, and inclusion of patient
information sheets.

https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e60411 | p. 1
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Beccia et al

Conclusions: End-user acceptability of DIRECT-T1DM was high. This was largely due to its implications for patient safety
and its “real-time” nature. DIRECT-T1DM may assist in appropriate management of children with new-onset diabetes, which isan uncommon event in general practice, through safety netting.
(JMIR Form Res 2024;8:e60411) doi: 10.2196/60411
KEYWORDS
type 1 diabetes; digital health innovation; clinical decision support tool; diabetes; acceptability; feasibility; diagnosis; child;children; youth; qualitative and simulation study; hospital; diabetic ketoacidosis; diagnostic delay; Australian; Australia; video
recorded; audio recorded; screen recorded; video; videos; patient; patients

Introduction
Type 1 Diabetes and Diabetic Ketoacidosis
Diabetic ketoacidosis (DKA) is a life-threatening emergency
that can occur in up to half of pediatric patients at the diagnosisof type 1 diabetes (T1D) [1]. DKA occurs due to absolute insulin
deficiency, a state of metabolic deterioration that can be aconsequence of undiagnosed or untreated T1D [2]. A single
episode of moderate DKA confers the risk for cognitive
impairment in a child’s critical developmental years [3]. It also
confers the risk for coma and death [2]. An episode of DKA is
also a traumatic event for both the patient and their caregivers,
who report higher levels of posttraumatic stress and anxiety
following a DKA admission [4].
Diagnosis of T1D before the point at which the child reaches
this critically unwell state is challenging. Early symptoms arenonspecific and can often be attributed to more commonchildhood illnesses, such as an upper respiratory tract infection,or to general childhood growth and development [5]. It also
requires a level of public awareness of the signs and symptomsof T1D to facilitate timely presentation to either a generalpractitioner (GP) or tertiary care to receive life-saving insulin
treatment [5].

Diagnostic Delay Exacerbates DKA Frequency andSeverity
Timely diagnosis and treatment are critical, as a diagnostic delayof even 24 hours following symptom recognition confers a2-fold risk for the development of moderate to severe DKA[6,7]. It has been identified that three-quarters of Australianchildren with T1D presented to their GP the week before
presenting to the hospital [8,9]. The standard guidelines
recommend that GPs conduct on-site point-of-care testing and
immediately refer a child with suspected T1D to the hospitalfor a specialist review [10,11]. Diagnostic delay following avisit with a GP can occur due to the provision of an alternate
diagnosis [7] and can be compounded by waiting to receive
pathology test results before referral to confirm suspicions of
T1D [9].

Our Intervention: Decision-Support for Integrated,
Real-Time Evaluation and Clinical Treatment of Type
1 Diabetes Mellitus

T1D as a potential diagnosis but have elected to request
pathology tests instead of direct referral for specialist care. In
an Australian setting, it has been identified that 56% of referral
delay can be accounted for due to awaiting pathology test results[9]. Electronic clinical decision support tools have shownpromise in serving to promote patient safety, leading to changes
in clinician behavior that are guideline based and prevent
adverse clinical outcomes [12].
In response to diagnostic delay arising from the wait associatedwith receiving confirmatory pathology results, we have devised
DIRECT-T1DM (Decision-Support for Integrated, Real-Time
Evaluation and Clinical Treatment of Type 1 Diabetes Mellitus).
DIRECT-T1DM is an electronic clinical decision support tool,
which provides a real-time alert triggered by the request for adiabetes-related pathology test in a child aged >18 years without
an established diabetes diagnosis. The alert advises GPs
suspecting T1D as a possibility to consider point-of-care testing
instead of venous blood tests in a laboratory, facilitating thepatient’s timely referral and safety.
DIRECT-T1DM was designed to work within Future HealthToday (FHT), a clinical decision support system (CDSS) thathas been developed by the University of Melbourne’s
Department of General Practice and Primary Care. FHT is
installed in select general practice clinics across Victoria [13]and is compatible with Best Practice and Medical Director, theelectronic medical record (EMR) software used by GPs tomanage patient care [13]. FHT currently works as anafter-the-fact analysis by reading EMRs on a centralized server
within the practice overnight, applying coded algorithms toidentify patients who may benefit from further review. An
onscreen pop-up activates when the patient file is opened(usually when the patient returns for a visit) and displays
recommendations for patient care. In contrast, DIRECT-T1DM
is designed to activate in real time using a combination of
information stored in the FHT server and information enteredinto the EMR during a consultation.
The DIRECT-T1DM pop-up has 3 components. The initial partof the pop-up (Figure 1) asks the question: “Do you think that
this child has new onset type 1 diabetes?” Depending on the
response, clinical advice is provided to the GP regarding therecommended next steps.

Hospital audits examining pathology test practices among GPs
demonstrate that in some children, GPs have correctly identified

https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e60411 | p. 2
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Beccia et al

Figure 1. DIRECT-T1DM (Decision-Support for Integrated, Real-Time Evaluation and Clinical Treatment of Type 1 Diabetes Mellitus) clinicaldecision support pop-ups. This figure is a flowchart of the alert that appears on the computer screen when DIRECT-T1DM is triggered. The initial alert(A) reads “STOP! Review for possible type 1 diabetes” in a green header, with a gray textbox below it asking, “Do you think this child has type 1
diabetes?” The general practitioner (GP) has the option to either select yes or no. When the GP selects yes (B), a new alert comes up describing theclinical action to take. When the GP selects no (C), a different alert appears.

Acceptability of electronic clinical decision support tools by
GPs is a strong facilitator of their use within the community[14]. Elements that need to be designed effectively to increaseusability and acceptability include user-friendliness andcompatibility with workload and workflow, and decreased
relevance of information are barriers to the use of clinicaldecision support [14]. Ascertaining whether these elements are
acceptable to GPs is therefore an important aspect of our
implementation process.
The aims of this investigation were to assess the acceptabilityof the DIRECT-T1DM software program by end users, by
testing the program in a simulated clinical environment, and to
explore the additional factors influencing T1D diagnosis that
could be used to optimize DIRECT-T1DM.

Methods
Overview
We conducted qualitative interviews with GPs to evaluate theacceptability of the DIRECT-T1DM software program. Content
analysis underpinned our methodological orientation. The
qualitative interviews were conducted in a simulated
environment in 3 steps. First, a presimulation interview was
conducted to understand the prior experience of the GPs. Next,
simulated clinical scenarios were tested so that GPs could
evaluate the software in the context of which they would use
it. Finally, a post simulation interview took place to evaluate
the tool and provide feedback. The COREQ (ConsolidatedCriteria for Reporting Qualitative Research) framework wasused and can be found in Multimedia Appendix 1.

Research Team and Reflexivity
CB conducted the interviews with all participants. CB is afemale PhD candidate, and the results of these interviews will
form part of her thesis. She has prior experience in conducting
qualitative interviews with health professionals and works withhttps://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

GPs frequently in her research. She had preexisting professional
relationships with 3 out of the 6 GPs before beginning theinterview process. One GP interviewee has collaborated with
CB in prior research in the same field of DKA prevention and,
therefore, may have had prior knowledge of existing guidelines
and reasoning for the implementation of DIRECT-T1DM. Steps
were taken in the design of the interview guide (Multimedia
Appendix 2) to manage any preexisting relationships with theparticipants by delineating clear roles during the interview andby an initial briefing and debriefing about the research (refer tothe before and after interview schedules). The influences ofthese participants’ responses were considered during analysis
and are acknowledged in the Strengths and Weaknesses of This
Study section. BH, JAMN, and MW were CB’s PhD supervisors,
who collaborated in the development of DIRECT-T1DM,
construction and editing of the interview guides, recruitment of
the participants, discussion, and provision of feedback oninterviews as they were conducted. They also provided feedbackon the results and analyses that emerged from the study andedited this publication. BH is a qualitative and implementation
researcher, JAMN is an academic GP, and MW is a pediatricendocrinologist and health services researcher.

Software Development
Business requirements and algorithm design documentation for
the DIRECT-T1DM prototype were submitted in February 2023
to the FHT technical team at the University of Melbourne. One
GP, a pediatric endocrinologist, and a PhD student were theinvestigators on this project and were involved in the early-stage
development of the tool. The tool was then presented to an
additional 3 GPs for their feedback on where to improve thedesign aspect. The design requirements were then passed on to
the software developer within the FHT technical team forproduction of the prototype.

JMIR Form Res 2024 | vol. 8 | e60411 | p. 3
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Research Design: Qualitative Interviews andSimulation

Beccia et alpathology and,
DIRECT-T1DM.

therefore,

neverinteract

with

GPs were invited to evaluate DIRECT-T1DM at the Digital
Health Validitron SimLab at Melbourne Connect, a Universityof Melbourne facility designed to conduct user acceptability
testing of digital solutions in near-live settings [15]. This
involved a presimulation interview, providing them context
about the purpose of the investigation and the prototype; thesimulation itself, involving subset of possible 5 clinical scenarios
where the pop-up would be triggered, facilitating on-the-spot
feedback for the tool; and a postsimulation interview discussing
their experience using the prototype. Each session lasted
approximately 60 minutes.

The postsimulation interview was designed using theConsolidated Framework for Implementation Research (CFIR)[17], evaluating GPs’ appraisal of the tool based on several key
domains and settings: inner setting, outer setting, innovation,
roles, individual characteristics, and implementation process[17]. CFIR is a commonly used, comprehensive frameworkused to ascertain key contextual determinants of success or
failure of an intervention as well as for appraisal of theimplementation process, facilitating optimization ofinterventions before deployment into the broader community[17].

The presimulation interview included questions surrounding
the GP’s experience of practicing, practice location, and an
estimation of the proportion of pediatric patients that they would
see on an average day. It also involved a discussion around
T1DM, whether they had diagnosed it before, and any challenges
involved with this. An explanation of the prototype, its purpose,and the simulation were then provided.

Participant Selection and Recruitment

The simulation session involved a “think-aloud” protocol [16],where GPs were instructed to be as verbose as possible
throughout the clinical scenarios and their interactions with the
pop-up. This was done with the intention of collecting
on-the-spot feedback about the pop-up tool from each GP. GPs
would engage with 2 allocated clinical scenarios and were
presented with a third if time permitted. All clinical scenarios
were designed with input from GPs. JAMN, a member of thestudy, contributed to the design of the clinical scenarios, and 3
GPs, independent to the study, were briefed on the study aims
and provided brief clinical scenarios for feedback. This feedback
was then used to optimize the scenarios to ensure that we could
meet our study aims. In total, 2 out of 5 scenarios were related
to a child presenting with symptoms suggestive of T1DM, andthe remaining 3 scenarios were related to presentations
suggestive of type 2 diabetes, polycystic ovary syndrome, orwhere a family sought to discuss the screening for T1D in thesetting of a positive family history. Clinical scenarios were
designed to warrant the request of a diabetes-related pathology
test, such as HbA1c or fasting blood glucose, thereby triggering
DIRECT-T1DM. Clinical scenarios were presented by theinterviewing researcher, not by a patient actor. The reasoning
for this was three-fold:
1.

2.

3.

The intention of the interviews was to assess theacceptability of DIRECT-T1DM, not to test whether GPs
were aware of the guidelines. It allowed triggering of thetool so that participants were guaranteed to interact with it,
and having the researcher deliver the clinical scenarios
allowed GPs to provide instant feedback as they interactedwith the tool.
Pediatric T1D symptoms before diagnosis are often vague
and nonspecific; therefore, a simulated consultation with apatient actor may lead to T1D never being suspected by theGP.
If GPs suspected diabetes, they may have elected tocomplete the point-of-care testing instead of referring forhttps://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

GPs were recruited via email through the personal networks of
the investigators as well as through the Victorian Practice Based
Research Network [18]. We aimed to recruit GPs of different
genders and various levels of experience by ensuring ourapproach involved practices with an adequate representativenessof junior and senior GPs as well as gender balance. Some
nonparticipants were present during the interviews, including
staff running the SimLab as well as additional researchers on
the investigating team. Participants were made aware of theirpresence and introduced to the nonparticipants.

Data Collection
The interview guide designed for all 3 stages of the study wasdevised, pilot-tested, and optimized by CB and BH. Video,
audio, and screen recording were completed using SimLab
resources. Brief field notes were collected by the interviewing
researcher to reflect on her interviewing technique. Copies oftranscripts and summaries of the results of this study were
offered to participants upon request. The interviews lasted
between 40 and 60 minutes.

Data Analysis
Analysis was facilitated by NVivo (version 14, release 1.6.1;
QSR International). CB initially familiarized herself with thedata by transcribing the audio recordings verbatim and readingeach transcript line by line. The domains and constructs within
CFIR (5 domains and 39 constructs) were transposed into the
NVivo codebook. CB coded the data, grouping codes
deductively into the constructs within CFIR. No domain orconstruct within the codebook was removed before initial
analysis. Following initial analysis, themes within CFIR
constructs were inductively generated. Following this, domains
and constructs that did not pertain to our simulation study were
removed. The decision made for this involved charting theexisting codes to determine whether they were major or minor
themes and whether they could be assigned to a different CFIR
domain. The domains and their related constructs that were
removed were individual characteristics and roles. This isbecause they did not map to the aims of our investigation, which
heavily involved appraising the “innovation” domain. It was
identified that CFIR constructs within domains did not exist in
isolation when appraising our tool; thus, visualization of how
the themes link from one to the next was undertaken, as
demonstrated by Sarkies et al [19]. No double coding was
JMIR Form Res 2024 | vol. 8 | e60411 | p. 4
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
conducted on the data, and all data analyses were conducted by
the primary author CB.

Ethical Considerations
The ethics application for this project was approved by the
Human Research Ethics Committee at the University of
Melbourne (ethics ID: 2023-25185-45082-4). Informed writtenconsent was obtained for each of the participant involved in thisstudy. Transcripts were deidentified, and published results
attribute responses to a deidentified code representing each

Beccia et al
participant. GPs were compensated with an Aus $200 (US$1=Aus $1.48) VISA Gift Card for their participation in the
study. No participants withdrew consent, and no interviews
were repeated.

Results
Figure 2 outlines a summary of the key results across 3 CFIR
domains and 7 constructs. The results summarized here are
expanded upon in subsequent sections.

Figure 2. Summary of the key results across all Consolidated Framework for Implementation Research (CFIR) domains. This figure describes the 3
CFIR domains that were included in this study: innovation domain, outer setting, and inner setting. This is a flowchart that starts with the words “CFIR
Domains” at the top, which splinters off into innovation, outer setting, and inner setting categories. In each category, there is a summary of important
points derived from relevant CFIR constructs. DIRECT-T1DM: Decision-Support for Integrated, Real-Time Evaluation and Clinical Treatment of Type
1 Diabetes Mellitus; GP: general practitioner.

Participants
In total, 6 GPs were recruited to take part in the simulation
study. Participants were diverse in gender, years of GP
experience, and exposure to CDSS. Two were academic
registrars completing their general practice specialty training
with <2 years of experience, 2 had 3 to 5 years of experience,and 2 had >30 years of experience working as a GP. Three had
experience of using FHT in their practice. All were familiar
with the Best Practice medical software. Table 1 summarizes
the participant demographic information.
Of the 6 participants interviewed, only 1 (16%) GP had
previously diagnosed a child with new-onset T1D before theseinterviews. As such, the scenarios were not instances theparticipants had encountered in practice, and their initial
responses were based on how they believed they would have
responded in practice. Participants were also explicitly asked
to interact with the scenarios in a manner that would trigger the
prompt (potentially not following best clinical practice), to
explore the acceptability and feasibility of the tool. Some GPs
reflected on what they would have done in response to eachhttps://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

clinical scenario in practice and whether T1D would havecrossed their mind in this case. One GP stated that preexistingawareness of the investigation before engaging in the interviews
would skew a participants’ response to the scenarios, especiallygiven the scarcity of T1D presentations in practice. The
think-aloud protocol was beneficial for on-the-spot feedback,appraisal, and reflection regarding DIRECT-T1DM, meetingthe aims of our investigation. However, it should be noted thatthe burden on GP participants was increased, when compared
to “near-live” scenarios or real life:
I didn’t read too much about [T1D] or anything,because sometimes I think it’s better walking into
something like this a bit fresh... [GP5]Yeah, I think as a participant, you probably overthink
things and second guess yourself more than you would
do in real life. I think even just taking your history
felt really painful just now compared to real life. But
that’s, I guess that’s part of the process. I think though
as far as participating in something it’s very easy.
[GP3]

JMIR Form Res 2024 | vol. 8 | e60411 | p. 5
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Beccia et al

Table 1. Participant demographic information. This table describes the GPsa interviewed, their gender, years practicing, familiarity with Best Practice,
FHTb, and whether they had diagnosed pediatric T1Dc in the past.

aGP

GenderYears practicing

Familiarity with Best
Practice

Uses FHT in practiceDiagnosed pediatric T1D (number of patients diagnosed)

GP1

Male

>30

Yes

No

No

GP2

Male

<2

Yes

No

No

GP3

Female

<2

Yes

No

No

GP4

Male

3-5

Yes

Yes

No

GP5

Female

3-5

Yes

Yes

No

GP6

Female

>30

Yes

Yes

Yes (n=1, 16%)

GP: general practitioner.

b

FHT: Future Health Today.

cT1D: type 1 diabetes.

Innovation Domain
Acceptability of DIRECT-T1DM
The CFIR construct we aimed to assess in our interviews
primarily involved the innovation domain, appraising
DIRECT-T1DM itself, its acceptability, and proof-of-concept.
We found that DIRECT-T1DM was highly acceptable to GPs.
This is because DIRECT-T1DM appeared because of an action
the GP had taken during the consultation. The GPs did not
anticipate encountering this tool often, as pediatric patients do
not constitute a sizeable proportion of their consultations, andof those pediatric consultations, they do not anticipate requesting
many pathology tests. GPs recognized the need for this tool,
stating its importance as a safety measure, to prevent potentiallyfatal sequelae. Design recommendations were provided to
optimize DIRECT-T1DM by addressing specific GP needs athttps://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

the point of care, enhancing compatibility with GP workflow
and workload, and alleviating pressures arising from both theinner and outer settings. Major constructs that emerged fromthe innovation domain were design and relative advantage. A
summary of the key results across all CFIR domains can befound in Figure 2.

Construct 1: Design
In CFIR, design refers to the degree to which the intervention
is well packaged and presented [17]. Major themes within thisconstruct were surrounding the language, structure, available
links, coloring, and emphasis within DIRECT-T1DM.
Language and Content
Overview
Table 2 illustrates the key responses to DIRECT-T1DM as they
pertain to language and content.

JMIR Form Res 2024 | vol. 8 | e60411 | p. 6
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Beccia et al

Table 2. Key responses to DIRECT-T1DMa as they pertain to the language and content used within each of the pop-ups.
Section of DIRECT-T1DM and key feedback Representative quotes
Initial question: Do you think that this child has T1Db? (Figure 1A).
Pose the question of T1D as a possibility •not a certainty•

GPs appreciated the succinctness of theinitial alert••“It’s pretty obvious in this case, but what if it wasn’t so obvious? What if I said I’d been tired andlost a few kilos, not sure, but nothing is acute though, so it could be just anything else but diabetes,
then you do a sugar. It’d be, I wouldn’t say I think he had it, I think, is it possible? Yeah, it’s
possible. So that might prompt me to do the fingerprick test.” [GPc6]“It’s almost as if the pop up needs to say, instead of saying, do you think this child has? Because
that’s kind of what’s most likely. Yeah, it could be like, is there a chance this child has type one?
Something a bit more, not on the balance of probabilities, but more on, you don’t want to misstype 1, so is there a chance this child has type 1?” [GP5]“It’s good. It’s a simple question. Because it’s giving you a differential that you may or may not
have.” [GP3]“That’s why it’s good to be reminded. Hang on, stop, clinical point here. We want you to you
know it’s really good to check the sugars now because this person at high risk who might needurgent action.” [GP6]Instructions that appear after you select yes (Figure 1B).
Information was necessary, actionable, •and specific enough for the point of care“It’s really clear—you know what’s normal and what’s abnormal and when the child needs to go
to emergency and how urgently.” [GP3]Reference ranges were helpful•“I like the fact that it reminds you of the reference ranges for [blood glucose levels].” [GP2]Make the clinical action “stand out”•

“I think I’d change it to make [a point of care blood test] really stand out. Yeah. Yeah, maybe
even like, a box like. ‘Please check your pinprick sugar and urine ketones.’ I would make thosestand out, like bang and bang. That’s the important information. That’s the action. This is what
to do with information.” [GP6]Instructions that appear after you select no (Figure 1C) .
Specific phrasing: “routine paediatricscreening” was confusing••

Information in this part of the alert was
too restrictive for general practice••

Explain osmotic symptomsa•“[Reading DIRECT-T1DM recommendation] ‘These tests are not indicated as part of routine
paediatric screening.’ I’m a little confused by that.” [GP1]“I guess it’s a bit confusing because it’s saying you shouldn’t use them as a routine screening.
There’s not really any, not really much routine paediatric screening. Yeah, like there’s not bloods
that we order on every child. The routine immunization and, you know, once they get to a certain
age, cycle screening all that, but I’m not quite sure what that means.” [GP5]“I would word it differently. ‘This is only to be requested as part of targeted screening.’ I think
this sounds too prescriptive and that it doesn’t... Because that doesn’t take into account experience,
what the patient might want, etc, their baseline, you know... So, I suppose, I don’t want that. I
would just click out of it and order the test... It would be another annoying, that’d be an annoying
pop up for me. Wouldn’t be helpful.” [GP6]“Yes, I think less words... So, if you’re trying to make people more aware of missing type one
diabetes, I would just focus on that rather than sort of talking about routine screening.” [GP5]“Is this bit about risk? So regardless of BMI status, especially if osmotic symptoms are present.
Osmotic symptoms. Would be more helpful if that was a bit more specific.” [GP2]DIRECT-T1DM: Decision-Support for Integrated, Real-Time Evaluation and Clinical Treatment of Type 1 Diabetes Mellitus.

b

T1D: type 1 diabetes.

c

GP: general practitioner.

Positing T1D as a Possibility Rather Than a Certainty
During the delivery and think-aloud response to the clinicalscenarios, it became clear that often, even when “typical”
symptoms of T1D were present, the clinical suspicion of T1D
was low. Therefore, this indicated that in general, GPs may berequesting diabetes-related pathology as 1 part of a panel ofseveral different tests to ascertain what condition the child haspresented with. All GPs participating in our study requested apanel of different pathology tests alongside a blood glucose test
https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

when demonstrating what they would do in practice. This led
to a design recommendation surrounding an alteration in thelanguage used in the initial pop-up. GPs suggested that in mostcases, it was likely that they would be uncertain that the child
had T1D, as children may not appear emergently unwell.
Consequently, they believed that adjusting the language in theinitial question from “Do you think this child has type 1
diabetes?” to “Is it possible that this child has type 1 diabetes?”would change their immediate reaction to DIRECT-T1DM.
They emphasized a shared belief that it is important to still
JMIR Form Res 2024 | vol. 8 | e60411 | p. 7
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Beccia et alinteract with DIRECT-T1DM, even in cases where there is alow degree of suspicion of pediatric T1D, and they believedthat an alteration in language will prompt them to exclude thepossibility of DKA through a point-of-care test before moving
on to requesting further pathology.

should the child have hyperglycemia was clear and helpful.
More emphasis could have been placed on the initial part of themessaging to emphasize the purpose of the point-of-care testingand decrease the cognitive load required to interpret themessaging within this part of DIRECT-T1DM.

Increasing Language Specificity to Tailor the Tool Optimallyfor the Point of Care
There was consensus regarding increasing the specificity of thelanguage within DIRECT-T1DM. In sum, time pressure inpractice necessitates pop-up alerts with information that is short,sharp, and directly actionable at the point of care. Increasing
specificity of language within the tool will also decrease thecognitive load required of the GP to interpret it midconsultation.
This is important with our tool, as it is disruptive by design.
The initial question “Do you think that this child has type 1
diabetes?” was appraised as fitting within the needs of GPs
during that point of the consultation, as it was concise andinterrupted their clinical decision-making at an appropriate point
in time.

When GPs select “No” in response to DIRECT-T1DM’s initial
prompt (“Do you think this child has type 1 diabetes?”), GPs
were confused by the phrase “not indicated as part of routine
paediatric screening processes.” Some pointed out that there
are no routine pediatric screening processes in general practice.
Others believed that the text was too restrictive and prescriptive
for the context of general practice. The consensus
recommendation was rephrasing of the text to centralize andemphasize its key purpose. One GP also stated that due to its
restrictive nature, it would be a recommendation that would beresented and not used in the community. Several GPs also
reported that they would prefer a brief explanation of what
osmotic symptoms are or to replace this phrasing with more
specific language delineating these symptoms instead.

When GPs selected “Yes” to the initial question, they areprompted to complete a point-of-care test instead of pathology
testing to prevent deterioration to DKA. GPs agreed that the
information provided within this tool was necessary and specificenough for actioning within the time available. The provisionof reference ranges and instructions for further management

Positioning on Screen, Visual Structure, Links andGuidelines, Font Colors, and Emphasis

https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

Overview
Table 3 summarizes the key feedback provided to us regarding
positioning on screen, visual structure, links and guidelines,
fonts, colors, and emphasis.

JMIR Form Res 2024 | vol. 8 | e60411 | p. 8
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Beccia et al

Table 3. Key responses to the positioning on screen, visual structure, links and guidelines, fonts, colors, and emphasis in each of the pop-ups.
Theme and key feedback

Representative quotes

Positioning on screen
There is limited time available in the clinic to review
pop-ups in their entirety, and some as a result are relegated to a lower priority.

••

DIRECT-T1DMb has an attention-grabbing central po- •sition on screen, which makes them less likely to missthe pop-up. They felt that, due to its implications forpatient safety, that this pop-up’s central location is
warranted.
••“Just with that, it is just time pressure. So, I’m guilty of it. Like if I’m running behind and the pop up will just appear in the background and I probably won’t lookat it, even though I’m I know I’m supposed to.” [GPa4]“I just feel sad that I just don’t even notice the Future Health Today ones as muchnow... and I wish I did, but I just don’t because, as I said, I think there’s so muchhappening in consultation and it’s just another thing, you know. Yeah, it just that’s
one thing I can ignore and pull my energy into the patient.” [GP6]“As a registrar, I’m acutely aware of the limitation of just the less experience you
have, the more likely you are to miss things. So, if you have something to remind
you or to make you think naturally, you’ll remember the differential that you had,and particularly when you’re struck for time, running late, all that, all those factorsthat make us, rush or forget things and that sort of a safety net for us to remember
and to not under- or over-investigate them.” [GP3]“I like that it’s, you know, big bold letters. Stop. Consider type one diabetes. I
think it’s just the right level of sort of intrusive. You know it, it should pop up andbe in the middle of the screen. It’s appropriate.” [GP2]“It’s also something you could even just show the patient or the parents as well,because if it’s a sort of clinical decision-making thing, a lot of people don’t likethe idea of having to go to emergency... So, this I think is helpful that it’s right on
the screen there. I use it for drug interactions as well.” [GP5]Visual structure and design•“...maybe with the first part just have the first line. So please check for hyperglycaemia now and then point of care test and then maybe have another to pop up afterwards...so I think it just breaks it up. That’s just probably me, because I think,just I don’t like to read too much text on my screen.” [GP4]Links provided to documents and guidelines should be •succinct and focused for the clinical action that is relevant.

“If you select the type 2 or you don’t think that it’s type 1 diabetes, that one I founda little bit more confusing... And that the link didn’t really give me any good specific information on a, you know, a child, a child with suspected type 2 diabetes...”[GP1]“...I think it’s a little bit confusing the way it hooks between the type 2 and type 1
and refers to sort of very long guidelines and things. And I think the focus should
just be on diagnosis and screening, like management is a sort of tomorrow problem...
You’re probably not even going to not to find the section in the time frame thatyou have to make a decision...” [GP5]The simpler the design and structure, the easier it is at
the point of care.

Links and guidelines•

Include patient communication material should a GP
have to make a referral to emergency.

•“But when you send the hospital, the other pop up which would be useful would
be probably a fact sheet for parents as well...What diabetes is, what the management
is, why we’re worried about it and why we have to send to the ED.” [GP2]•“Oh, actually I like the colour, it’s calming. But I wonder if red makes it more
like...more like a health alert or something, you know. You want it to sort of bang!”[GP6]

Color, font style, and emphasis
Make the design of the alert more conspicuous.

a

GP: general practitioner.

b

DIRECT-T1DM: Decision-Support for Integrated, Real-Time Evaluation and Clinical Treatment of Type 1 Diabetes Mellitus.

Positioning on Screen
GPs reported that the available time in clinic to review pop-ups
while seeing patients is often limited, and as a result, pop-ups
that appear may be relegated to lower priority of items theymust address during a consultation. However, GPs felt that
DIRECT-T1DM’s attention-grabbing central screen position
with bold font, which was different from the location of other
FHT prompts, assisted them in engaging with the directive in
https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

the pop-up, even when this time pressure existed. They felt as
though they were less likely to miss this pop-up, even when
under pressure to undertake a consultation in a short amount oftime or impacted by alert fatigue and could use the alert to also
engage with patients about the importance of referral, if required.
Visual Structure and Design
GPs indicated that optimizing DIRECT-T1DM to have a more
simplistic visual structure will increase their engagement withJMIR Form Res 2024 | vol. 8 | e60411 | p. 9
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
the tool and contribute significantly to decreasing the cognitive
load required to appraise the information held within the tool,
allowing them to keep the rest of their attention to the patient
during the consultation. All GPs agreed that the clinicalrecommendations had too many words. Some GPs recommended
checkboxes, while others suggested that the information wasprovided in a two-step structure: (1) complete a point-of-care
test now to prevent deterioration into DKA and (2) reference
ranges and further action depending on the point-of-care results.
Links and Guidelines
GPs who suspected a diagnosis of T1D were directed to theRoyal Australian College of General Practitioners “Managementof Hyperglycaemic Emergencies” document; GPs would havepreferred it if the link immediately directed them to theflowchart for management. The links providing guidance when
T1D was not suspected (Management of type 2 diabetes: A
handbook for general practice [20] and the Australian PaediatricEndocrine Group type 2 diabetes screening guidelines [21])were too long—198 and 14 pages, respectively—and did notcontain direct and focused instructions that could direct
treatment at the point of care. Several GPs suggested also adding
a patient information sheet to DIRECT-T1DM to aid in

Beccia et al
explaining the child’s situation to parents who go to an
emergency department (ED) following a visit with the GP.
Color, Font Style, and Emphasis
Red coloring was suggested in place of the green color scheme
embedded within DIRECT-T1DM due to a desire for this toraise alarm among GPs regarding the safety of the child and theurgency of their situation. A more consistent style of font was
desired across all parts of DIRECT-T1DM. Other signifiers ofemphasis on our tool were desired, such as exclamation marks,
bold font, and underlining.

Construct 2: Relative Advantage
Overview
Relative advantage refers to the extent to which the innovation
is better than existing innovations targeting the same issue orcurrent practice [17]. Two themes emerged when discussing
relative advantages with interviewees: (1) access to CDSS
compared to GP education or public awareness campaigns and(2) relative advantage in cases where T1D may not be initiallyrecognized by GPs. Textbox 1 summarizes the key responses
to DIRECT-T1DM as they relate to relative advantage.

Textbox 1. Key themes related to relative advantage.
Education programs alongside Decision-Support for Integrated, Real-Time Evaluation and Clinical Treatment of Type 1 Diabetes Mellitus(DIRECT-T1DM)•“I think [DIRECT T1DM’s] one part of the solution. I think probably awareness raising both with the general public and in general practice.
Both of those factors can complement each other.” [general practitioner; GP1]

Risk prediction tools: what do we do when type 1 diabetes (T1D) is too difficult to recognize?
•“It would be great if we had this artificial [intelligence] system that can [read] through the patient history and detect type one [diabetes] for thechildren and then just [tell the GP] consider [diabetes] or something.” [GP3]•“Just trying to think. Like whether it would also be useful and maybe future health today already does this, but like even just weight loss insomeone under 18. Like it would almost be good if that could trigger [DIRECT-T1DM] alone.” [GP5]Other issues to consider due to the pediatric setting•“I think the challenges initially are going to be because they’re a child. So, the younger they are, the less capacity they have to give you a history,
and you’re relying on parents or carers for information. I think the other difficulty with children is that they probably present, maybe less typicallyand perhaps with symptoms that you might not necessarily associate with you know, diabetes, new onset diabetes.” [GP1]

DIRECT-T1DM reminds you to check for T1D in cases that may appear mild or where suspicion may be low.
•

“Yeah. So, it prompts you to do the point of care testing. Yeah, which a barrier often is time. Yeah, I think, yeah. If I’m honest, it’s the rightthing to do. And we should always do it. But if I’m meeting this child the first time, they’re just taking a real long history... maybe it’s easier forme to just order the [test] and the blood test form and send along.” [GP3]•“Because this guy is, I’m worried about. OK, I’ll probably send to hospital. Do you see what I’m saying? So, it’s more the grey, the grey area
thing. So, I think I think it’s a good idea to do it to prompt the GP to do a pinprick at a lower level of suspicion.” [GP6]

Education Programs May be Desired Alongside
DIRECT-T1DM
Some GPs felt that an awareness campaign or GP education
may also be helpful in the prevention of DKA and that these
preferences would likely vary by GP. Implementation of
DIRECT-T1DM could benefit from pairing with an awarenesscampaign.

https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

DIRECT-T1DM Should Be Used in Conjunction With Other
Methods in Cases Where T1D Is Hard to Recognize
GPs were additionally concerned about whether they were
keeping pediatric T1D in focus during their day-to-day
consultations. Some GPs expressed that this could result indelays that would not be addressed by the DIRECT T1D tool,
which focused on potentially inappropriate pathology requests.
Many GPs stated that they would like a prompt that would help
them consider T1D based on the child’s prior medical history
or triggered based on the symptom input at that consult. This
JMIR Form Res 2024 | vol. 8 | e60411 | p. 10(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Beccia et alway, diabetes is kept in focus and they can complete afingerprick and direct future management as required.
DIRECT-T1DM’s trigger point, following the ordering of
diabetes-related pathology, may not catch all children who
attend general practice with new-onset T1D.

area,” and the urgency of a diagnosis may not be apparent, as
children may appear well or not any less well than children withother common conditions, such as viral illnesses.

GPs felt that diagnosis was also made difficult due to thepediatric setting, as children are unable to verbalize their
symptoms as well as an adult may be able to, limiting theircapacity to give an accurate history. Children may also vary
more widely in their presentation, with symptoms lessattributable to T1D. GPs attributed the relative obscurity of
T1D-related symptoms, combined with the lack of time available
in practice as another potential reason why they may elect toorder the pathology test externally rather than complete
point-of-care testing. Often, symptoms of T1D sit in a “greyOverview

https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

Inner Setting Domain
The inner setting domain is the setting within which theintervention is implemented [17]. In our case, this would be thegeneral practice clinic. The 5 constructs within the “inner setting
domain” predominantly assessed in our interviews were tension
for change, relative priority, compatibility, learning
centeredness, and IT infrastructure. Table 4 summarizes the
responses from GPs to each construct within the inner setting
domain.

JMIR Form Res 2024 | vol. 8 | e60411 | p. 11(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Beccia et al

Table 4. Constructs and themes discussed for the inner setting domain.
Construct and key themes

Representative quotes

Tension for change
All GPsa recognized a need for the current sit- •uation to improve.
•“You hear about people attending emergency and being diagnosed [with T1D]. So, if
someone were to sort of walk into my room with [T1D], I wouldn’t mind a sort of slightly
alarmist pop up.” [GP5]“I’d prefer to do some sort of point of care test whether it’s a, you know, a pinprick or aurine, at least to get an idea of where we’re at. Yeah, on the spot rather than doing thepathology. So, I think that recommendation is very reasonable.” [GP1]Relative priority•“So, I think yeah, based on that because it’s a paediatric age group and cohort then probably
wouldn’t, you wouldn’t get it often, so that when you did get it you would take it seriously.”[GP3]

DIRECT-T1DM appears at the right time dur- •ing workflow.

“And I think it comes up at the right time, like when you’re sort of, you know, maybe you’resort of trying to decide do I order bloods and bring them back or do I actually just need tosend them now? Like it’s a good prompt at that time, or you’ve got the file open and everything, yeah.” [GP5]DIRECT-T1DMb takes high priority because
it would appear rarely in practice and its purpose in enhancing patient safety.
Compatibility

Culture — learning centeredness
GPs appreciated the retraining in pediatric-re- •lated medicine as they recognize that you may
deskill as you see fewer pediatric patients“Like it’s not that often that I’m ordering a sugar test, in a young person, that it would,
you’re not going to sort of get pop up fatigue from it. I think it would be generally, especiallymaybe because we don’t see a huge proportion of paediatric patients [in this clinic]... it’salmost more necessary I think, because you have to think of what is going to affect themdifferently as to a 60 year old male or whatever, but they’re almost a different species in a
lot of ways. So yeah, you kind of deskill a little bit if you’re seeing less of them.” [GP5]Teaching practices have an emphasis on evi- •dence-based, guideline-driven care. DIRECTT1DM fits within this culture.

“From my experience with talking with other clinicians about this, the people who seemthe most interested and accepting of it are property more junior or doctors who have maybe
graduated, you know, maybe within the last five to 10 years, I think because they’re a little
less fixed in their in their ways and the way they practice, very guideline driven, very evidence driven and sort of. I think would appreciate these tools whereas I think maybe someolder clinicians who have like vast amount of personal experience probably might not be
as accepting of it.” [GP2]“I think most people would find it helpful, even those that are very experienced, you know,
for the most part. Lifelong learners as doctors and really open to changes in guidelines and,you know, being prompted to consider things that may not have. I think teaching practices
in particular, and obviously as a registrar, you’re in a teaching practice. So yes, most people
are pretty open to it.” [GP3]•IT infrastructure
A potential barrier to clinical decision support •is convincing clinics to install the software thatsits outside the EMRc, rather than the clinicaldecision support tool itself.
a“That would be the barrier, yes. Not so much the usefulness of the pop up, but actually
getting additional software or software that engages with the EMR and has the potential tohave privacy issues. I think other general practices are wary of because of data breachepisodes...” [GP1]GP: general practitioner.

b

DIRECT-T1DM: Decision-Support for Integrated, Real-Time Evaluation and Clinical Treatment of Type 1 Diabetes Mellitus.

c

EMR: electronic medical record.

Tension for Change
GPs welcomed the implementation of DIRECT-T1DM, thus
recognizing the need for a change to the current practice. Two
GPs were also aware of children attending the ED with DKA
in part due to delay from pathology test requests, emphasizing
the need for the implementation of DIRECT-T1DM because ofits benefits to patient safety. The other GPs still recognized aneed for change to current practice, even if they were not awareof any specific patient circumstance relating to DKA
https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

presentation because of delayed diagnosis of T1D. They valued
the introduction of DIRECT-T1DM because of its purpose asa safety mechanism to ensure that patients in urgent need ofcare receive it in a timely manner.

Relative Priority
Despite receiving numerous electronic pop-ups and alerts eachday, GPs stated that DIRECT-T1DM would have a higher
relative priority when compared to other interventions. Thiswas because they did not anticipate encountering it often, andJMIR Form Res 2024 | vol. 8 | e60411 | p. 12(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Beccia et al

its purpose in preventing deterioration to DKA heightened thelikelihood that they would engage when it does appear.

nonteaching practices that may not have such a strong emphasis
on learning.

Compatibility

IT Infrastructure

GPs stated that DIRECT-T1DM integrated well with theirworkflow, as it interrupts their clinical decision-making at theexact right point in time, when they go to press “print” on apathology test, giving them the chance to reverse their choice,ascertain the safety of continuing with the pathology testing,and move on from there. GPs found the flow of the instructions
within DIRECT-T1DM to be compatible with their workflow,
as the provision of reference ranges for blood glucose levels
enabled them to direct future management on the spot, saving
time that would have been spent looking for the appropriate
guidelines. As previously stated, when appraising intervention
design, the structure and visual aspect of the tool could be
improved to decrease the cognitive workload placed upon theGP.

DIRECT-T1DM requires the installation of FHT clinicaldecision support software. Not all GPs interviewed practiced
in clinics with FHT installed. GPs without adequate IT
infrastructure reflected that it may be harder to convince
practices to install CDSS that sits outside their EMR systemthan to adopt the DIRECT-T1DM intervention. Therefore, thisis a barrier that would decrease the use of DIRECT-T1DM.
Some GPs mentioned that DIRECT-T1DM was similar toadverse drug interaction alerts that were already embedded
within their EMRs (without the use of FHT), and they believedthat DIRECT-T1DM could be implemented into community
general practice this way. For example, GPs saw this support
tool as being a potential integration within EMR Software Best
Practice, which is used by 65.8% of Australian general practices[22].

Culture: Learning Centeredness
GPs had a positive attitude toward the implementation of thistool. Most GPs acknowledged that because they do not see alarge proportion of pediatric patients in their day-to-day practice,they might lose proficiency in treating these patients. As a result,they welcomed DIRECT-T1DM as a learning opportunity.
Younger doctors recognized that in their teaching practices,
there is an emphasis on evidence-based, guideline-driven tools
for learning. Hence, they acknowledged that their practices may
be more open to DIRECT-T1DM in comparison to other

Outer Setting Domain
Overview
The outer setting in CFIR refers to the broader context in which
the inner setting resides [17], which, in this case, is the broader
health care system. DIRECT-T1DM could be optimized to fit
considerations that exist due to the outer setting. Table 5summarizes the additional considerations due to externalpressure in the outer setting.

Table 5. External pressures in the outer setting.
Construct and key themes

Representative quotes

External pressure
Overreferral to the emergen- •cy department“I guess there’s a risk that, and maybe an acceptable risk, that you over refer to ED, and that you’re sending
people to ED that just have a viral illness...So yeah, I guess that’s a real, it’s a consideration right now. It is•so swamped.” [GPa3]“Yeah, I’m probably just going to send them to emergency, but this is kind of good, I think, to back up yourdecision making or if you’re not quite sure you did the right thing.” [GP5]•“I guess this is worth bringing to people’s attention. Just to discuss with the paediatric specialists. However,in my current clinic access to paediatric specialist is really challenging. For a variety of reasons: wait times,
financial barriers. Yeah. So, I feel like that’s good, that’s nice, yeah. But in reality, as a GP, you gotta dosomething while [the patient’s] waiting to be seen.” [GP3]Societal stigma surrounding •diabetes“Definitely, if you come in, and...you’ve got some something on the screens telling you’ve got diabetes. Yeah,
you’d be pretty distressed. And I think that there’s a lot of stigma associated with diabetes. And if you’re
getting that as a child, it’s amplified.” [GP3]“I think it would take a bit of explanation to try and give the parent the idea that there’s a couple of types ofdiabetes, type 1 diabetes, yes, they need regular insulin. But if they don’t get their insulin, they get very sick,very quick. Yeah, so, you’d have to sort of really, you know, explain, or emphasize the need for urgent care.”[GP1]

Accessibility of pediatricspecialists•a

GP: general practitioner.

External Pressure: Secondary and Tertiary Care
Considerations
Some GPs felt that overreferring to the ED was a consideration
when assessing children with potential T1D. GPs were consciousof the exacerbated wait times that exist within EDs currently;
however, they believe that the risk of overreferral was an
https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

acceptable risk given the consequences of undiagnosed T1D.
DIRECT-T1DM was helpful in reinforcing and encouraging
the clinical decision to refer to the ED. When T1D wassuspected, GPs found the clinical decision support
recommendations to be thorough, including clear instructions
to refer the patient to the nearest ED. This is because not all
JMIR Form Res 2024 | vol. 8 | e60411 | p. 13(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
practices will have access to pediatric endocrinologists, which,for some, is related to the location of practice in relation totertiary settings. Instead, such practices would refer the childto the nearest ED where they are also equipped to treat DKA.
In contrast, participants identified that the clinical decision
support recommendations for treatment of type 2 diabetes may
not be inclusive of all practices where access to pediatricians
for the treatment of type 2 diabetes may be limited due to wait
times and financial barriers. GPs recommended tailoring theinformation in this section of the pop-up.

External Pressure: Societal Stigma
Societal stigma surrounding diabetes and preconceived
expectations regarding the consultation may play a role in how
the instructions in the DIRECT-T1DM intervention arecommunicated. Parents or patients may not expect that they will
be referred to the ED following a visit to the GP for relatively
nonspecific symptoms.
Multimedia Appendix 3 summarizes how the context provided
in the interviews shaped design recommendations pertaining to
DIRECT-T1DM.

Discussion
Principal Findings
Major themes were identified within 7 constructs across 4 CFIR
domains: outer setting, implementation process, inner setting,and innovation. Acceptability of the initial DIRECT-T1DM
clinical prompt and recommendations (when T1D wassuspected) was strong because of its straightforward design, thefact that GPs did not anticipate seeing it often, and the severityof the consequences associated with DKA. Design
recommendations were driven primarily by the desire for
information to be highly specific to management at the pointof care as well as to adjust the design packaging to reflect theurgency of the prompt. Acceptability of recommendations in
DIRECT-T1DM when the GP did not suspect T1D was mixed:
some GPs found that the information provided was not specificenough to children, others appreciated the reminder regarding
the risk of T1D despite BMI status. One GP found thisrecommendation not to be useful and would ignore it in practice.

Scarcity of Time in General Practice Shapes Design
Needs for Clinical Decision Support
Prior research has demonstrated that facilitating GP needs within
the context of the inner and outer settings is beneficial tointervention adoption [23]. Having a short amount of available
time in general practice is commonly reported amonginterviewees, not only in our study, but across the country [24].
This influences the amount of time that GPs would like to spend
using a clinical decision support tool, emphasizing the need to
tailor it for usability at the point of care. Secondary careconsiderations of ED overreferral and patient perception have,to our knowledge, not been explored within the research
involving early T1D recognition. However, these issues havebeen explored in the broader general practice context, whereconcerns about overreferral [25], the tool’s influence on patient
perceptions, and its impact on communication have been
documented [26]. Our study supports that design considerations
https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

Beccia et alshould involve appraising the time spent understanding anddeciding about the information within the tool.

Situating T1D Within a GP’s Caseload
Difficulty in diagnosing pediatric T1D stemmed from a
combination of the lack of available time in consultation toconsider it as a possibility, nonspecific presentations, and therelative scarcity of children with T1D in the population. Thisis consistent with existing qualitative interviews assessing thechallenges in receiving a T1D diagnosis [5,27,28]. The influenceof new-onset T1D on a GP’s workload was exemplified in ourstudy, as in a combined approximately 80 years of GP
experience, only 1 patient was diagnosed with new-onset T1D.
This is different from existing qualitative interviews, whichtargeted GPs who had previously diagnosed children with
new-onset T1D [5,28]. This informs an existing research gap,
as the perceptions of GPs who have not previously seen
new-onset T1D in practice may be more reflective of most GPs’
experiences and attitudes when encountering a clinical decision
support tool, such as DIRECT-T1DM. Including GPs without
prior experience of diagnosing T1D at new onset increases therepresentativeness of GPs’ perceptions of the challenges andbarriers to diagnosing T1D and may demonstrate the differencesin perceptions of how a child may present at new onset whencompared to GPs who have diagnosed T1D in the past.

Pathology Testing Is Conducted at a Lower Level ofSuspicion, Which May Be Common Among Patients
With New-Onset Type 1 Diabetes
Existing research studies assessing pathology referrals fromgeneral practice are quantitative in nature [6,9,29,30]. Our study
provides clinical context as to why this may be occurring in
general practice. We found that the burden of time pressure maylead GPs to prefer laboratory testing over point-of-care testing.
In addition, GPs may have elected to conduct the pathology test
instead of point-of-care testing in cases where their suspicionof T1D is low, as part of a broader panel of tests to narrow down
the child’s diagnosis. Prior qualitative interviews with GPs who
have diagnosed a child with new-onset T1D have demonstrated
that often, the child may present “well” and not at a level ofseverity that would induce suspicion of T1D or evoke a senseof urgency to refer to the ED [5]. Combining our findings withthis existing research may demonstrate that pathology tests thatdelay diagnosis of T1D may be requested in general practicebecause symptom presentation is not specific enough to diabetes
or severe enough to evoke a sense of urgency from the GP’s
perspective. This, in turn, modifies our design considerations
for the tool, embedding language that encourages GPs to exclude
pediatric T1D as a possibility, even at a lower level of suspicion.

Integration With Clinical Decision Support Tools at
the Appropriate Time Within a GP’s Workflow Is
Critical to Acceptability
Acceptability of DIRECT-T1DM as a concept was high. Thiswas because of its capacity to increase patient safety and because
GPs did not anticipate encountering the tool often. This is inline with existing research evaluating clinical decision support
tools, as the timing of advice is important to whether GPs engage
with them in practice [14,31]. Design considerations provided
JMIR Form Res 2024 | vol. 8 | e60411 | p. 14(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
by GPs were intended to improve the ease of use, fit into
workflow, increase the involvement of patients, and improve
the presentation and specificity of advice. Existing research has
demonstrated that these are important facets of innovation
design, particularly with clinical decision support tools,
influencing the degree to which they are used in practice [32].

Strengths and Weaknesses of This Study
The think-aloud protocol was advantageous as it allowed forthe recognition of instantaneous response to the tool. A lot of
the feedback about design packaging, relative priority, andusefulness of the tool was collected upon on-the-spot interaction
with DIRECT-T1DM. Participants then had a chance toelaborate on their feedback in the postsimulation interviews.
This is in line with what previous comparisons between“think-aloud” and “near-live” simulation studies have identified[16]. However, it has also been identified that “think-aloud”
protocols are not as beneficial when compared to “near-live”
protocols when assessing compatibility with workflow andrelative priority [16]. As a result, our conclusions regarding
workflow compatibility and priority may be weakened. Future
simulation studies involving GPs can consider the “think-aloud”
protocol as a tool for evaluating clinical decision support tools,
especially in cases where it is unfeasible to adopt a “near-live
setting.” Times where GPs can pause and take a break should
be considered within the interview design, as the “think-aloud”
process may induce a higher than usual level of burden on theparticipant.
We elected to recruit a small sample of GPs (N=6). This sample
size may pose limitations toward the application of ourconclusions to the wider population. Furthermore, theinterviewer (CB) had preexisting professional relationships with
3 of the GPs taking part in the simulations before the initiationof the study. One GP had collaborated on a different study
involving DKA prevention research. We recognize that thismay skew perceptions favorably toward DIRECT-T1DM andthat there is limited generalizability of our findings as a result.
Further investigations following implementation within thecommunity involving GPs, such as qualitative interviews andsurveying, will be conducted to assess the acceptability of thistool with a larger sample of clinics. Our study aim was toevaluate the usability and acceptability of DIRECT-T1DM,
rather than to assess GP performance when encountering patient
scenarios that involve possible T1D diagnoses. Therefore, wecannot draw conclusions regarding the awareness of bestpractice guidelines or GP performance in such scenarios. In the
future, we plan to investigate whether the tool effectively
optimizes the management of suspected T1D in the community.
Adoption of CFIR [17] allowed for in-depth analysis of theeffect of different settings, contexts, and GP needs on design
considerations and optimizations for our tool. We were able to

Beccia et al
identify critical clinical context surrounding the challenges indiagnosing T1D and how we can improve our tool to meet theneeds that arise because of these challenges. We also identified
a need for embedded risk prediction tools in general practicefor the development of pediatric T1D to respond to the challengeof recognizing T1D in children with nonspecific symptoms inbusy general practice.

Implications for Future ResearchThis study informs the optimization of DIRECT-T1DM andsuggests that there is possible clinical utility as well as end-user
acceptability. As a result of this investigation, changes will bemade to DIRECT-T1DM, including the use of more alarmingcolors, specific language and guidelines, and the developmentof a patient information handout. We anticipate that these
optimizations to the tool will enhance its usability andacceptability. DIRECT-T1DM could also be integrated into
EMR systems, such as Best Practice, rather than being embedded
within FHT, and this important preliminary work informs theintegration of our tool, ensuring that it is acceptable and useful
before broader implementation.
Once DIRECT-T1DM is implemented in the community
settings, further evaluation will take place for all stakeholders.
Qualitative interviewing as well as surveying of GPs using thetool will be undertaken to ensure that the tool is usable andacceptable. Auditing the rate of pathology test requests ingeneral practice, as opposed to point-of-care testing that is taking
place, will also be undertaken. Rates of DKA and referral
following a point-of-care testing can be measured by our local
EDs at the Royal Children’s Hospital, and retrospective
identification of patients and their caregivers that were referredto the ED because of our tool can be interviewed for insights
into their experiences.
All GPs within our study identified a need for an additional
clinical decision support tool for the context of new-onset
pediatric T1D where early symptoms are nonspecific and may
be underappreciated by the GP and caregivers alike. It was notedin this study and in the existing literature that usuallyT1D-related symptoms are not deemed serious enough by thepatient to be reported as the presenting complaint when attending
general practice [5,27,28]. As DIRECT-T1DM is triggered uponrequest of a diabetes-related pathology test, it will not addresscases where T1D is not “front of mind” and, therefore, notaddress the cases where T1D is misdiagnosed. Future research
can consider the development of risk prediction tools for the
purposes of alerting GPs when a child is at elevated risk ofdeveloping T1D before the consultation. Our investigation also
demonstrates that a “think-aloud” protocol is suitable for
qualitative evaluation of CDSS tools when it is unfeasible toadopt a near-live simulated protocol.

Acknowledgments
The authors acknowledge the Validitron Centre for Digital Transformation of Health at Melbourne Connect. In particular, theauthors acknowledge Kit Huckvale and Hasan Ferdous for their support in facilitating the simulation sessions. The authors
acknowledge the National Health and Medical Research Council Centre for Research Excellence in Digital Technology toTransform Chronic Disease Outcomes for their support in funding the publication of this manuscript and for funding thehttps://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e60411 | p. 15(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Beccia et al

reimbursement of our general practitioners for their participation. The authors also thank associate professors Roy Rasalam andRalph Audehm for their involvement and feedback on the initial prototypes of DIRECT-T1DM (Decision-Support for Integrated,
Real-Time Evaluation and Clinical Treatment of Type 1 Diabetes Mellitus). The authors would also like to acknowledge Sean
Lo and the Future Health Today team, who built the DIRECT-T1DM software and readied it for evaluation. Funding from the
National Health and Medical Research Council Centre for Research Excellence in Digital Technology to Transform Chronic
Disease Outcomes supported the reimbursement of the general practitioners for their participation and for the publication of thismanuscript. Software development was funded by the 2021 Murdoch Children’s Research Institute Population Health Theme
Funding; Aus $20,000 (US $1=Aus $1.48) was awarded for the integration of a clinical decision support system to optimize thediagnosis and early management of pediatric type 1 diabetes mellitus in the primary care setting.

Conflicts of Interest
None declared.

Multimedia Appendix 1
COREQ (Consolidated Criteria for Reporting Qualitative Research) checklist.
[PDF File (Adobe PDF File), 374 KB-Multimedia Appendix 1]

Multimedia Appendix 2Interview script, including the clinical scenarios.
[DOCX File , 26 KB-Multimedia Appendix 2]

Multimedia Appendix 3
Summary of the relationship between the outer setting, implementation process domain, and innovation domain.
[DOCX File , 16 KB-Multimedia Appendix 3]

References
1.

2.
3.
4.

5.

6.

7.

8.

9.
10.

11.

Atlas G, Rodrigues F, Moshage Y, Welch J, White M, O'Connell MA. Presentation of paediatric type 1 diabetes in Melbourne,
Australia during the initial stages of the COVID-19 pandemic. J Paediatr Child Health. Oct 2020;56(10):1654-1655. [FREEFull text] [doi: 10.1111/jpc.15081] [Medline: 32959935]
McMorrow R, Beccia C, Kilov G, White M, Manski-Nankervis J. Diabetes emergencies: identifying and managing HHS
and DKA. Diabetes Manage J. Nov 23, 2023:26-27. [FREE Full text]
Aye T, Mazaika PK, Mauras N, Marzelli MJ, Shen H, Hershey T, et al. Impact of early diabetic ketoacidosis on thedeveloping brain. Diabetes Care. Mar 2019;42(3):443-449. [FREE Full text] [doi: 10.2337/dc18-1405] [Medline: 30573652]
Moffett MA, Buckingham JC, Baker CR, Hawthorne G, Leech NJ. Patients' experience of admission to hospital withdiabetic ketoacidosis and its psychological impact: an exploratory qualitative study. Pract Diabetes. Jun 13,
2013;30(5):203-207. [FREE Full text] [doi: 10.1002/pdi.1777]
Usher-Smith JA, Thompson MJ, Walter FM. 'Looking for the needle in the haystack': a qualitative study of the pathway
to diagnosis of type 1 diabetes in children. BMJ Open. Dec 03, 2013;3(12):e004068. [FREE Full text] [doi:10.1136/bmjopen-2013-004068] [Medline: 24302510]
Rugg-Gunn CE, Dixon E, Jorgensen AL, Usher-Smith JA, Marcovecchio ML, Deakin M, et al. Factors associated withdiabetic ketoacidosis at onset of type 1 diabetes among pediatric patients: a systematic review. JAMA Pediatric. Dec 01,
2022;176(12):1248-1259. [doi: 10.1001/jamapediatrics.2022.3586] [Medline: 36215053]
Usher-Smith JA, Thompson MJ, Sharp SJ, Walter FM. Factors associated with the presence of diabetic ketoacidosis at
diagnosis of diabetes in children and young adults: a systematic review. BMJ. Jul 07, 2011;343:d4092. [FREE Full text][doi: 10.1136/bmj.d4092] [Medline: 21737470]
Craig ME, Wong CH, Alexander J, Maguire AM, Silink M. Delayed referral of new-onset type 1 diabetes increases therisk of diabetic ketoacidosis. Med J Aust. Feb 16, 2009;190(4):219. [FREE Full text] [doi:10.5694/j.1326-5377.2009.tb02356.x] [Medline: 19220193]
Rodrigues F, O'Connell MA, White M. A point-of-care test increases same-day referral and reduces DKA in children with
new-onset type 1 diabetes. Aust J Gen Pract. Apr 1, 2021;50(4):246-251. [doi: 10.31128/ajgp-04-20-5348]
Mayer-Davis EJ, Kahkoska AR, Jefferies C, Dabelea D, Balde N, Gong CX, et al. ISPAD clinical practice consensus
guidelines 2018: definition, epidemiology, and classification of diabetes in children and adolescents. Pediatr Diabetes. Oct
2018;19 Suppl 27(Suppl 27):7-19. [FREE Full text] [doi: 10.1111/pedi.12773] [Medline: 30226024]
Emergency management of hyperglycaemia in primary care. The Royal Australian College of General Practitioners. URL:
https://www.racgp.org.au/clinical-resources/clinical-guidelines/key-racgp-guidelines/view-all-racgp-guidelines/
emergency-management-of-hyperglycaemia [accessed 2024-08-29]

https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e60411 | p. 16(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
12.

13.

14.

15.
16.

17.

18.

19.

20.

21.

22.

23.

24.
25.

26.

27.

28.

29.

30.
31.

Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support
systems: benefits, risks, and strategies for success. NPJ Digit Med. Feb 06, 2020;3:17. [FREE Full text] [doi:10.1038/s41746-020-0221-y] [Medline: 32047862]
Hunter B, Alexander K, Biezen R, Hallinan CM, Wood A, Nelson C, et al. The development of future health today: piloting
a new platform for identification and management of chronic disease in general practice. Aust J Prim Health. Feb
2023;29(1):8-15. [doi: 10.1071/PY22022] [Medline: 36318973]
Westerbeek L, Ploegmakers K, de Bruijn GJ, Linn AJ, van Weert JC, Daams JG, et al. Barriers and facilitators influencing
medication-related CDSS acceptance according to clinicians: a systematic review. Int J Med Inform. Aug 2021;152:104506.
[FREE Full text] [doi: 10.1016/j.ijmedinf.2021.104506] [Medline: 34091146]
The validitron. The University of Melbourne. URL: https://mdhs.unimelb.edu.au/digitalhealth/our-work/validitron [accessed2024-08-29]
Richardson S, Mishuris R, O'Connell A, Feldstein D, Hess R, Smith P, et al. "Think aloud" and "near live" usability testingof two complex clinical decision support tools. Int J Med Inform. Oct 2017;106:1-8. [FREE Full text] [doi:10.1016/j.ijmedinf.2017.06.003] [Medline: 28870378]
Damschroder LJ, Reardon CM, Widerquist MA, Lowery J. The updated consolidated framework for implementation
research based on user feedback. Implement Sci. Oct 29, 2022;17(1):75. [FREE Full text] [doi: 10.1186/s13012-022-01245-0][Medline: 36309746]
VicREN: Victorian primary care practice-based research and education network. The University of Melbourne. URL: https://medicine.unimelb.edu.au/school-structure/general-practice-and-primary-care/engagement/primary-care-community/research/vicren [accessed 2022-06-10]
Sarkies M, Long JC, Pomare C, Wu W, Clay-Williams R, Nguyen HM, et al. Avoiding unnecessary hospitalisation forpatients with chronic conditions: a systematic review of implementation determinants for hospital avoidance programmes.
Implement Sci. Oct 21, 2020;15(1):91. [FREE Full text] [doi: 10.1186/s13012-020-01049-0] [Medline: 33087147]
Management of type 2 diabetes: a handbook for general practice. The Royal Australian College of General Practitioners.
2020. URL: https://www.racgp.org.au/getattachment/41fee8dc-7f97-4f87-9d90-b7af337af778/
Management-of-type-2-diabetes-A-handbook-for-general-practice.aspx [accessed 2024-08-29]
Peña AS, Curran JA, Fuery M, George C, Jefferies CA, Lobley K, et al. Screening, assessment and management of type 2
diabetes mellitus in children and adolescents: Australasian Paediatric Endocrine Group guidelines. Med J Aust. Jul
2020;213(1):30-43. [doi: 10.5694/mja2.50666] [Medline: 32578226]
Practice incentives program quality improvement measures: annual data update 2022-23. Australian Institute of Health andWelfare. URL: https://www.aihw.gov.au/reports/primary-health-care/practice-incentives-program-quality-improvement/
contents/about [accessed 2024-08-29]
Breimaier HE, Heckemann B, Halfens RJ, Lohrmann C. The consolidated framework for implementation research (CFIR):
a useful theoretical framework for guiding and evaluating a guideline implementation process in a hospital-based nursing
practice. BMC Nurs. Aug 12, 2015;14:43. [FREE Full text] [doi: 10.1186/s12912-015-0088-4] [Medline: 26269693]
Schattner PL, Coman GJ. The stress of metropolitan general practice. Med J Aust. Aug 03, 1998;169(3):133-137. [doi:10.5694/j.1326-5377.1998.tb116013.x] [Medline: 9734508]
Conlon C, Nicholson E, Rodríguez-Martin B, O'Donovan R, de Brún A, McDonnell T, et al. Factors influencing generalpractitioners decisions to refer paediatric patients to the emergency department: a systematic review and narrative synthesis.
BMC Fam Pract. Oct 16, 2020;21(1):210. [FREE Full text] [doi: 10.1186/s12875-020-01277-9] [Medline: 33066729]
Meunier PY, Raynaud C, Guimaraes E, Gueyffier F, Letrilliart L. Barriers and facilitators to the use of clinical decision
support systems in primary care: a mixed-methods systematic review. Ann Fam Med. 2023;21(1):57-69. [FREE Full text][doi: 10.1370/afm.2908] [Medline: 36690490]
Townson J, Gallagher D, Cowley L, Channon S, Robling M, Williams D, et al. "Keeping it on your radar"-assessing thebarriers and facilitators to a timely diagnosis of type 1 diabetes in childhood: a qualitative study from the early detectionof type 1 diabetes in youth study. Endocrinol Diabetes Metab. Dec 18, 2017;1(1):e00008. [FREE Full text] [doi:10.1002/edm2.8] [Medline: 30815545]
Hardikar N, Power T, Leong GM, Liu A, Bhurawala H. Primary care providers' experience in the management of paediatrictype 1 diabetes in Western Sydney, New South Wales. Aust J Gen Pract. Jul 2023;52(7):464-471. [FREE Full text] [doi:10.31128/AJGP-03-22-6381] [Medline: 37423244]
Wersäll JH, Adolfsson P, Forsander G, Ricksten SE, Hanas R. Delayed referral is common even when new-onset diabetes
is suspected in children. A Swedish prospective observational study of diabetic ketoacidosis at onset of type 1 diabetes.
Pediatr Diabetes. Sep 2021;22(6):900-908. [doi: 10.1111/pedi.13229] [Medline: 33978305]
Iyer D, Choudhary D, Agwu JC. Timeliness of referral of children with new onset type 1 diabetes. Postgrad Med J. May
2017;93(1099):242-244. [doi: 10.1136/postgradmedj-2016-134023] [Medline: 27489372]
Westerbeek L, de Bruijn GJ, van Weert HC, Abu-Hanna A, Medlock S, van Weert JC. General Practitioners' needs andwishes for clinical decision support systems: a focus group study. Int J Med Inform. Dec 2022;168:104901. [FREE Fulltext] [doi: 10.1016/j.ijmedinf.2022.104901] [Medline: 36279654]

https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

Beccia et al

JMIR Form Res 2024 | vol. 8 | e60411 | p. 17(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
32.

Beccia et al

Manski-Nankervis JA, Biezen R, Thursky K, Boyle D, Clark M, Lo S, et al. Developing a clinical decision support tool
for appropriate antibiotic prescribing in Australian general practice: a simulation study. Med Decis Making. May
2020;40(4):428-437. [doi: 10.1177/0272989X20926136] [Medline: 32507028]

Abbreviations
CDSS: clinical decision support system
CFIR: Consolidated Framework for Implementation Research
DIRECT-T1DM: Decision-Support for Integrated, Real-Time Evaluation and Clinical Treatment of Type 1
Diabetes Mellitus
DKA: diabetic ketoacidosis
ED: emergency department
EMR: electronic medical record
FHT: Future Health Today
GP: general practitioner
T1D: type 1 diabetes

Edited by A Mavragani; submitted 10.05.24; peer-reviewed by D Chua, J Townson, H Bhurawala; comments to author 27.05.24;
revised version received 21.06.24; accepted 06.07.24; published 23.09.24
Please cite as:
Beccia C, Hunter B, Manski-Nankervis J-A, White M
Exploring the User Acceptability and Feasibility of a Clinical Decision Support Tool Designed to Facilitate Timely Diagnosis of
New-Onset Type 1 Diabetes in Children: Qualitative Interview Study Among General Practitioners
JMIR Form Res 2024;8:e60411
URL: https://formative.jmir.org/2024/1/e60411
doi: 10.2196/60411
PMID:©Chiara Beccia, Barbara Hunter, Jo-Anne Manski-Nankervis, Mary White. Originally published in JMIR Formative Research(https://formative.jmir.org), 23.09.2024. This is an open-access article distributed under the terms of the Creative Commons
Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction
in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete
bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license
information must be included.

https://formative.jmir.org/2024/1/e60411

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e60411 | p. 18(page number not for citation purposes)