JMIR FORMATIVE RESEARCH

Dougherty et al

Original Paper

Obstetric Emergency Supply Chain Dynamics and Information
Flow Among Obstetric Emergency Supply Chain Employees: Key
Informant Interview Study
Kylie Dougherty1, RN, PhD; Abebe Gebremariam2,3,4,5, MD; Heran Biza4, MPH; Mulusew Belew3, MSc; Natalie
Benda6, PhD; Yihenew Tesfaye5, PhD; John Cranmer2,3,4,5,7, MPH, MSN, DNP; Suzanne Bakken6, PhD
1

Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, IL, United States

2

Woodruff Health Sciences Center, Emory University, Atlanta, GA, United States

3

Emory-Ethiopia Partnership, Bahir Dar, Ethiopia

4

Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA, United States

5

College of Health Sciences, Bahir Dar University, Bahir Dar, Ethiopia

6

School of Nursing, Columbia University, New York, NY, United States

7

Center for the Study of Human Health, Emory University, Atlanta, GA, United States

Corresponding Author:
Kylie Dougherty, RN, PhD
Department of Medical Social Sciences
Northwestern University Feinberg School of Medicine
633 N St Clair St
Chicago, IL, 60611
United States
Phone: 1 (312) 503 1725
Email: kylie.dougherty@northwestern.edu

Abstract
Background: For the past several decades, the Ethiopian Ministry of Health has worked to decrease the maternal mortality
ratio (MMR)—the number of pregnant women dying per 100,000 live births. However, with the most recently reported MMRof 267, Ethiopia still ranks high in the MMR globally and needs additional interventions to lower the MMR to achieve thesustainable development goal of 70. One factor contributing to the current MMR is the frequent stockouts of critical medications
and supplies needed to treat obstetric emergencies.
Objective: This study describes the obstetric emergency supply chain (OESC) dynamics and information flow in Amhara,
Ethiopia, as a crucial first step in closing stockouts and gaps in supply availability.
Methods: Applying qualitative descriptive methodology, the research team performed 17 semistructured interviews with
employees of the OESC at the federal, regional, and facility level to describe and gain an understanding of the system in theregion, communication flow, and current barriers and facilitators to consistent emergency supply availability. The team performed
inductive and deductive analysis and used the “Sociotechnical Model for Studying Health Information Technology in Complex
Adaptive Healthcare Systems” to guide the deductive portion.
Results: The interviews identified several locations within the OESC where barriers could be addressed to improve overall
facility-level readiness, such as gaps in communication about supply needs and availability in health care facilities and regional
supply hubs and a lack of data transparency at the facility level. Ordering supplies through the integrated pharmaceutical logistics
system was a well-established process and a frequently noted strength. Furthermore, having inventory data in one place was abenefit to pharmacists and supply managers who would need to use the data to determine their historic consumption. The greatest
concern related to the workflow and communication of the OESC was an inability to accurately forecast future supply needs.
This is a critical issue because inaccurate forecasting can lead to undersupplying and stockouts or oversupplying and waste of
medication due to expiration.
Conclusions: As a result of these interviews, we gained a nuanced understanding of the information needs for various levelsof the health system to maintain a consistent supply of obstetric emergency resources and ultimately increase maternal survival.
This study’s findings will inform future work to create customized strategies that increase supply availability in facilities and thehttps://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e59690 | p. 1
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Dougherty et al

region overall, specifically the development of electronic dashboards to increase data availability at the regional and facility
levels. Without comprehensive and timely data about the OESC, facilities will continue to remain in the dark about their true
readiness to manage basic obstetric emergencies, and the central Ethiopian Pharmaceutical Supply Service and regional hubs will
not have the necessary information to provide essential emergency supplies prospectively before stockouts and maternal deaths
occur.
(JMIR Form Res 2024;8:e59690) doi: 10.2196/59690
KEYWORDS
basic emergency obstetric care needs; BEmOC; supply chain; Ethiopia; Sociotechnical Model

Introduction

to meeting required needs and ensuring that the system matches
the environment in which it is deployed [6].

Background

Objective

For the past several decades, the Ethiopian Ministry of Health(MOH) has worked to decrease the maternal mortality ratio(MMR)—the number of pregnant mothers dying per 100,000
live births [1]. However, with the most recently reported MMRof 267, Ethiopia still ranks high in the MMR globally and needs
additional interventions to lower the MMR to the sustainable
development goal of 70 [2]. Therefore, the Ethiopian MOH isfocusing on improving “the health systems capacity to offer
quality care that meets women’s needs (the supply side)” [3,4].
Toward this goal, the Amhara Regional Health Bureau in
Amhara and the Ethiopian MOH have identified the need for areal-time obstetric emergency readiness system, which willassist in measuring and monitoring facility-level readiness tomanage the 6 most common basic emergency obstetric careneeds (BEmOC), including a set of core drugs; commodities;
and resources to identify emergencies, treat them, andmonitor-modify therapy as clinically indicated [5].

This study aimed to describe the obstetric emergency supply
chain (OESC) dynamics and information flow throughsemistructured qualitative interviews with key informants fromdifferent levels of Ethiopia’s health care system. This
information would be used to guide the design and eventual
implementation of electronic dashboards as a component of thereal-time obstetric emergency readiness system.

Formative work to articulate the specific needs, barriers, andfacilitators for the development and implementation of areal-time obstetric emergency readiness system is foundational

https://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

Methods
Theoretical Model
The “Sociotechnical Model for Studying Health Information
Technology in Complex Adaptive Healthcare Systems” guided
this study (Figure 1) [7]. The Sociotechnical Model is a dynamic
and interconnected model that provides a thorough picture of
the process for designing, implementing, and evaluating healthinformation technology (HIT) through 8 interconnecteddimensions: hardware and software, clinical content,
human-computerinterface,people,workflowandcommunication, internal organization features, external rules
and regulations, and measuring and monitoring (Textbox 1)[7].

JMIR Form Res 2024 | vol. 8 | e59690 | p. 2
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Dougherty et al

Figure 1. Sociotechnical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems.

Textbox 1. Sociotechnical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems dimensions and definitions.
Model dimension and definitions•

Hardware and software••

Clinical content••

The policies, procedures, and culture within the specific organization using the HITExternal rules and regulations••

The necessary steps that a user must accomplish to successfully and effectively complete the task at handInternal organizational features••

The application users from the developers of the health information technology (HIT) to the end users

Workflow and communication••

All aspects of the digital application that the user can see, touch, hear, or manipulate

People••

The categorical or numerical data and images that make up the “language” of the clinical application

Human-computer interface••

The digital infrastructure and equipment that is used to operationalize the clinical application being explored

The policies, procedures, and culture within the larger geographic location where the HIT is locatedMeasuring and monitoring•

The evaluation process and method of measuring the effectiveness of the HIT change, including both intended and unintended consequences

https://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e59690 | p. 3
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Dougherty et al

Guiding Methodology

Ethical Considerations

Guided by qualitative descriptive methodology, we conducted
semistructured interviews to further describe and contextualize
the gaps in emergency supply availability discovered in an
earlier quantitative analysis of facility-level obstetric emergency
readiness in the region [8,9]. The interviews exploredmultifactorial causes of emergency commodity stockouts, thecurrent health system approaches to restocking commodities(from the federal, regional, and facility level), and information
flow gaps and dynamics that could contribute to emergency
supply stockouts. Data collection and analysis occurredconcurrently [10].

The study was approved by institutional review boards atColumbia University (IRB-AAAU2006) and Emory University(MOD005-STUDY00005335) and received ethics approval
from Amhara Public Health Institute (NoH/RfftT1D1o7144).
All participants provided verbal consent before participation.
All data were deidentified before data analysis. Participantsreceived no compensation for their participation in this study.

Recruitment and Sample
We recruited a purposive sample of OESC experts from acrossthe federal, regional, and facility level of care in Ethiopia. This
included federal MOH government officials working in supply
chain management and maternal survival in Addis Ababa,
regional health bureau officials with obstetric expertise, regional
pharmaceutical supply system employees, and supply managers
and pharmacists from individual health care facilities. To
participate in this study, individuals had to be at least 18 yearsof age, speak English and Amharic, and be full-time employees
in one aspect of Amhara’s OESC.

Data Collection: Semistructured Interviews
We created an interview guide with open-ended questions to
explore the flow of information through the OESC, including
barriers and facilitators (factors that enhance or impede accessand use) with paper-based systems and the computer-based
integrated pharmaceutical logistics system (IPLS; Textbox 2and Multimedia Appendix 1). We conducted interviews in
English (federal level) and Amharic (regional and facility level).
The interviews were audio recorded and lasted approximately
60 minutes. Amharic is the national language, and individuals
working in Ethiopia’s health care field received their training
and education in English. During the interviews, we performed
member checks by summarizing the main concepts and ideas
the participants voiced and asked for confirmation to ensure weaccurately captured their perspectives [11].

Textbox 2. Sample interview guide questions for semistructured qualitative interviews with obstetric emergency supply chain employees in Amhara,
Ethiopia.
Questions•

What do you see as the major challenges to having the right supplies on hand to deal with obstetric emergencies when they occur?
(Communication breakdowns, frequently unable to obtain certain items, computers available, or consistent Wi-Fi)•

Can you describe your process for ordering and shipping obstetric emergency supplies?
(Barriers and facilitators in the process, how often it occurs, who does this task, was there any training, and decision-making process)•

What are your impressions of the paper-based supply request system?
(Likes, dislikes, barriers, and facilitators for use, and areas of improvement)•

Can you describe your experience using the integrated pharmaceutical logistics system?
(Likes, dislikes, barriers, and facilitators for use, areas of improvement, and training if it occurred)•

Is there anything else that you’d like me to know about the topics that we’ve discussed today that I didn’t ask about?

Data Management and Preparation
We stored audio files in OneDrive (Microsoft Corporation), an
encrypted, password-protected electronic site accessible only
to the research team members [12]. All interviewees wereidentified by a participant number linked to the OESC level inwhich that individual worked (ie, federal, regional, or facility
level). The participant numbers were linked to their names andstored behind an encrypted, password-protected site. We
checked the transcripts for accuracy against the audio recording
in English or Amharic and deidentified them before sharing
them with other research team members. We used ATLAS.ti(ATLAS.ti GmbH version 9) to manage, code, and analyze all
qualitative data [13].

https://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

Data Analysis
We performed directed content (ie, deductive) and inductive
analysis of the interview transcripts. The directed content
analysis was guided by a codebook developed by KD and SB(Multimedia Appendix 2) based on the a priori constructs of
the Sociotechnical Model dimensions [7,14]. During data
analysis, we added codes to the human-computer interface
dimension of the codebook to reflect the emergence of additional
themes. KD independently coded the transcripts, generated
themes, and documented the coding. SB reviewed the
documentation and refined codes with KD to establish theconfirmability of the coding decisions. We created saturationtables using a priori constructs of the Sociotechnical Model,
concurrently with data analysis to document if and when data
saturation, a measure of data adequacy that signals that nofurther data collection is needed, had occurred (Multimedia
JMIR Form Res 2024 | vol. 8 | e59690 | p. 4
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Dougherty et al

Appendix 3) [15]. We specifically used a priori thematic
saturation to guide the decision on when to stop data collection[16].

Results
Overview
We conducted 17 interviews (federal level: n=5, 29%; regional
level: n=5, 29%; facility level: n=7, 41%) from February 17 toMarch 17, 2023. We relied on the qualitative research experienceof the data collectors (KD and YA) and data saturation(Multimedia Appendix 3) to guide the decision on when to stop
participant recruitment. The sample was predominantly male
(14/17, 82%), with an age range of 30 to 60 years, and with 2
to 38 years of experience working within the OESC (Multimedia
Appendix 4). While we analyzed all 8 dimensions of theSociotechnical Model, in the main paper, we focused on the 5(62%) dimensions most pertinent to the development of
dashboards for monitoring facility-level readiness to manage
obstetric emergencies and reported on the other 3 (38%) in
Multimedia Appendix 5.

Hardware and Software
In 2014, Ethiopia launched IPLS which allows facilities to placeorders for medical supplies either electronically or through paper
forms. IPLS also enables the central Ethiopian Pharmaceutical
Supply Service (EPSS) and regional hubs to see reports on
supply availability at the regional and national levels.
Two software systems are used within the OESC to monitor the
movement of medical supplies, namely Vitas and Dagu. Vitas
is used by central EPSS, which supplies the country withmedications and medical supplies. The director general of EPSS
and input officers have access to Vitas dashboards, which allows
them to check supply availability and track commodities at boththe federal and regional hub levels. Dagu is the software used
at individual health care facilities. Dagu is used at the facility
level to monitor human resources and service availability, but
it does not have dashboards for medical supply logistics andmovement. Some software in the OESC have components thatfunction offline while other features are fully dependent on live
internet access. Barriers (eg, unstable internet access andfrequent electric power outages) and facilitators (eg, some IPLS
components available offline) and related quotes are summarizedin Table 1.

Table 1. Reported barriers and facilitators for the hardware and software dimension (the digital infrastructure and equipment that is used to operational
the clinical application being explored) of the Sociotechnical Model for Studying Health Information Technology in Complex Adaptive Healthcare
Systems extracted from semistructured interviews with obstetric emergency supply chain employees in Amhara, Ethiopia.
ThemesQuotesBarriers
Lack of or outdated laptops, tablets, and •phones•“Most health facilities lack a computer, a printer, or the training required to actively utilize the[current electronic system(s)].” [Regional #1]“Updating our computers and laptops would be a good idea since we’ve been using them for a
decade! It’s tough to get them to work in this setting.” [Regional #4]

Unstable and inaccessible internet•“In general, 75% of the facilities have no internet access.” [Federal #2]

Technology is not tailored for use on
mobile phones or offline•“There are many things that need to be updated. Many systems have changed, but they are notkeeping up with the change. For example, we have to find information on our mobile phones. But
there is not much like that. It should be transparent.” [Facility #3]“I want the IPLS system to be mobile application-based; if it is, then EPSS or other health offices
should be checking the... hospital system.” [Facility #7]•

Facilitatorsa

Tablets provided to some facilities•“We distributed tablets to all health posts to track the workflow of health posts as well as any
commodity-related issue activities.” [Federal #2]

Most employees have access to mobile
phones•“Almost all health extension workers have a mobile phone and can easily see any piece of information.” [Federal #2]

Some IPLSa components can work offline if a source of electrical power isavailable•“In some areas, the facilities are using solar panels.” [Federal #2]

IPLS: integrated pharmaceutical logistics system.

Clinical Content
Respondents at all levels of the supply chain agreed that several
pieces of information are critical to ensure facilities are readyto provide BEmOC and to prevent emergency supply stockouts.
In many health care facilities, pharmacists hand-calculate thehttps://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

last month’s consumption, average monthly consumption, andthey forecast future monthly and annual supply demands. In
contrast, some health care facilities used components of theelectronic IPLS to calculate expiration and current quantity ofsupplies automatically. These respondents stated that the
automatic calculations assisted in their consumption calculations.
JMIR Form Res 2024 | vol. 8 | e59690 | p. 5
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Dougherty et al

Overall, participants reported a desire for an automated system
at all health care facilities that will calculate consumption, waste,
and medication expiration in real time.
Upon receiving orders from hubs, health care facilities must
know and report the number of items they originally ordered,
the number of items received, the number of items missing fromthe order, who sent and received the shipment, and the date thesupplies arrived at the health care facility. These pieces of
information are stored in various data sources, such as bin cards
and stock record cards resulting in a fragmented, nontrackable,
and nonintegrated reporting system.
Furthermore, when individual health care facilities are
determining how much medicine and medical supplies to order,

they must also know the disease burden in the population thatthey serve, and which items are available for ordering. For
example, in obstetric emergencies, facilities should know theestimated size of their catchment population, the number ofmothers who give birth at their facility for antenatal care service,
the number of pregnant women who attend the facility, and thenumber of mothers who are typically referred to higher facility
levels for more intensive treatment. These complex variables
are implicitly factored into health care facility orders, but there
is no system to formally account for those variables in creatingorders from facilities. Respondent perceptions of barriers (eg,lack of reliable, high-quality, real-time consumption data) andfacilitators (eg, integration of medical supplies into one place)as well as exemplar quotes are displayed in Table 2.

Table 2. Reported barriers and facilitators for the clinical content dimension (the categorical or numerical data and images that make up the languageof the clinical application) of the Sociotechnical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems extracted
from semistructured interviews with obstetric emergency supply chain employees in Amhara, Ethiopia.
ThemesQuotesBarriers
Lack of accurate, high-quality •consumption data
•

•
•
•
•

“When we go to [look at] the health facility level, especially the primary centers like health posts and healthcenters, it is very difficult to visualize what they have or do not have.” [Federal #2]“The expert tries to send information by speculation; this is also another problem because you may takemore medication and supplies than you need, which will lead to expired medication and supplies andsometimes the institution also faces a shortage of medication and supplies because they send false information.
Thus, using guessing information will create overstock and understock problems.” [Regional #1]“Accurate and on time information is highly needed and very vital to get the right supplies on hand to deal
with problems, but unfortunately, we don’t gather reliable information and we are making mistakes.” [Regional #1]“We are using the issue data as a proxy data [for] consumption and we usually use this data for the national
quantification so it might exaggerate or decrease the national quantification.” [Federal #3]“The IPLS does not take the average monthly consumption component into account, necessitating a reviewof these issues.” [Facility #7]“There are also issues with the quality of data because there is no way for us to make sure the person sent avalid report with real information or not.” [Regional #2]

Facilitators
High integration of medical
supplies into 1 place•••“Compared to the previous IPLS, the current IPLS manual system is highly integrated. The previous system
was very scattered because the pharmacy sent requests by themselves, the laboratory sent requests bythemselves, and others did the same thing, and it was very uncoordinated. As of right now, the integrated
system is very good, and all requests are sent in the same direction and in a coordinated manner.” [Facility#2]“Most of our facilities are integrated into IPLS, particularly most maternal commodities, which were integrated into IPLS after the 2009 Ethiopian calendar (2017 Gregorian Calendar). Therefore, everyone cancheck what they have, what they do not have, what is nearing its expiration date.” [Federal #2]“Following EPSS’s establishment, the fragmented administration system completely stopped. For example,
when we deliver medicines such as TB, HIV medicines, HIV examination kits, and mother-and-child-related
medicines, we do so in an integrated way, not in fragments. When we deliver, we take all the HIV, TB, andmedicines through one route because it is an IPLS system, which means we deliver integrated.” [Regional#4]

Workflow and Communication
In the Amhara region, there are 918 health centers, 103 hospitals,
and 19 supply hubs. To determine the type and quantity of
medical supplies to order, a facility must first determine its
monthly consumption and average monthly consumption toforecast future needs. When requesting for supplies, facilities
can request up to a maximum of 4 months of stock. The headof each department or unit at the health care facility plansmonthly consumption every 2 weeks and submits an internal
request and reporting forms to the central store or pharmacy inhttps://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

the facility. The pharmacist reviews and fulfills these biweekly
requests from the facility’s stock and also completes and submits
requesting and reporting forms (RRFs) to their designatedregional supply hub to refill facility-level supplies. Pharmacists
submit these forms every 1 to 2 months depending on the sizeof the population they serve and may also submit an emergency
RRF if they have less than 2 weeks’ supply of an item orexperience a stockout. The RRFs are either submitted in paper
form or electronically through the IPLS. A health carefacility–level respondent describes the critical role of IPLS
saying, “It acts as the system’s brain and is crucial for having
JMIR Form Res 2024 | vol. 8 | e59690 | p. 6
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
the right medication in the right quantity and of the right quality
at the right time.”
After receiving the RRFs through the IPLS, regional hubs review
the forms and if they have the stock on hand in the hub, theycreate a stock transfer voucher and fulfill the request byproviding the commodity to the facility. In addition, the hub
consolidates the consumption reports and RRFs for theindividual health care facilities under their jurisdiction andestimates overall consumption and forecasting for the upcoming
months, which they report in an RRF and send to the central or
national office of the EPSS for fulfillment every 1 to 2 months.
Central EPSS reviews the RRFs from their hubs and fulfills theorders based on consumption reports and local and national
forecasting numbers. The hubs receive the products from central

Dougherty et al
EPSS and review the orders to see if they received the full or
partial order. The hubs compare this information to the requests
they have received from their facilities and send the supplies tothe facilities every 1 to 2 months. After receiving the order fromtheir designated hub, health care facilities then review what they
have received and dispense it throughout the various
departments or units at the facility. They track the new deliveries
and existing inventory with bin cards and stock record cards
that are paper-based. A summary of the various data sources
and communication systems present in this supply chain ispresented in Multimedia Appendix 6. Figure 2 illustrates the
procurement process. The barriers and facilitators related toworkflow and communication are presented in Table 3 withsample quotes from participants at various levels of the system.

Figure 2. Procurement process for ordering and receiving medical supplies in Amhara, Ethiopia, from the facility level through regional hubs and at
the central hub level. EPSS: Ethiopian Pharmaceutical Supply Service; RRF: requesting and reporting form.

https://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e59690 | p. 7
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Dougherty et al

Table 3. Reported barriers and facilitators for the workflow and communication dimension (the necessary steps that a user must accomplish to successfullyand effectively complete the task at hand) of the Sociotechnical Model for Studying Health Information Technology in Complex Adaptive Healthcare
Systems extracted from semistructured interviews with obstetric emergency supply chain employees in Amhara, Ethiopia.
ThemesQuotesBarriers
Inaccurate forecasting

•
•
•

•

“The forecasting error at Ethiopia’s health facility is not better than 35%, and you see 40%, very goodfacilities are forecasting at 50%. What does this mean, they are either over-quantifying or underquantifying.” [Federal #1]“The knowledge gap about capacity and forecasting error is evident.” [Federal #1]“There is a [communication] barrier, the hub knows only the stock on hand issued data, so as a logisticians, the issue data is the proxy data [for] consumption, that is not the actual consumption but we
use it as a proxy data as consumption and we usually use this data for the national quantification so it
might exaggerate or decrease the national quantification and the forecasting that might occur.” [Federal#3]“If we request medical equipment, we have to consider consumption, but currently, we request theamount of medical equipment by guessing, not based on consumption data, so this exposes us to
medical equipment scarcity.” [Facility #2]•“There is something that needs to be improved. I think it [IPLS] has not changed since the first format
design. Therefore, the format must be adjusted, now the information network has grown.” [Facility#3]

Overreliance on paper-based meth- •ods•“The other challenge is the system problem. Most of our health facilities use a manual method of requesting and ordering medical supplies and equipment.” [Regional #1]“If it is paper based, it is like to be blinded. It’s going to be difficult to find out the medicine that weare running out of stock, so it must be digital to analyze the data. Paper-based has nothing to show theexpert how much demand there is today and what demand there will be tomorrow. Paper-based is not
good. I think it should be updated.” [Facility #3]“Paper-based systems are not appropriate for the 21st century; they are archaic, tedious, difficult toaccess, and unsuitable for auditing.” [Facility #7]“In the current situation, the system between health facilities and us [regional hub] is still manual, so
it’s also possible for us to make several mistakes when it comes to requesting medicines due to thismanual system.” [Regional #4]“The paper-based or hard-copy system of supplying or requesting the information is very helpful, but
it is very difficult to share information immediately.” [Regional #5]

Outdated forms (both paper andelectronic)•••

Facilitators
There is an established process forordering medical supplies through•“The integrated system is very good, and all requests are sent in the same direction and in a coordinatedmanner.” [Facility #2]

IPLSa
a

IPLS: integrated pharmaceutical logistics system.

Human-Computer Interface
Individuals interact with the OESC at the federal, regional, andindividual facility levels of the system through a variety ofhuman-computer interfaces. Multimedia Appendix 7 describes
the needs of the 3 types of users as well as how they interact
with the interfaces available at their level of the health system.
In addition to the current interfaces, some respondents wanted
the electronic IPLS to be accessible via a mobile app or on
mobile devices so that they have greater access to the data,
particularly when desktop or laptop computers were notavailable at the health facilities and alternative mechanisms
were needed to use the electronic components of IPLS.
Multimedia Appendix 8 summarizes the barriers and facilitators
identified for human-computer interfaces.

https://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

Discussion
Principal Findings
We used the Sociotechnical Model to explicate participants’
perceptions of barriers and facilitators to securing and managing
supplies within the OESC in Amhara, Ethiopia, and to identify
intervention targets to increase BEmOC supply availability at
the facility level as a key step for reducing Ethiopia’s MMR[7]. Changing one component within the OESC can influence
the success of the system, but current barriers may impede orinteract with the system in unique and unintended ways. An
example of this would be if the EPSS increased the number ofcomputers and tablets available to health care facilities but failed
to explore users’ usability and ease of use concerns. If thetechnology is still viewed as difficult to use and users continue
to prefer paper-based forms, then EPSS as a whole will not see
the full benefits of increasing access to the technology. This
type of outcome is highlighted in a recent scoping review inlow- and middle-income countries, which included Ethiopia,
JMIR Form Res 2024 | vol. 8 | e59690 | p. 8
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
and it identified lack of training and user involvement as barriers
to successful digital health interventions [17]. To ensure thesuccess of the electronic components of IPLS the researchers
and individuals working within this field can build on thepreexisting strengths found in the IPLS to rectify barriers andconcerns for HIT-based real-time monitoring of BEmOC supply
provision and availability.
The lack of access to computers and tablets led to lower use of
the electronic components of IPLS; this is often the case at small
and rural health posts. If facilities do not have the equipment
to access IPLS, then they will continue to rely on thepaper-based method [18]. When the MOH provided tablets toindividual facilities, they reported they were more likely to use
the electronic components of IPLS. However, even if facilities
had the hardware available, if they did not have consistent accessto the internet, the electronic systems were still of little use
because some features of the technology are unable to workoffline. These findings, while novel, as they relate to technology
specifically tailored for OESCs, also affirm what is found in
the current literature surrounding digital health research in lowand middle-income countries, which found that increased accessto technology and consistent reliable internet led to increased
use of the digital health technology [17,18]. Creating dashboards
and electronic components of the IPLS that work offline is
crucial because internet access is not always consistent in allparts of Amhara [19]. To encourage users to transition frompaper-based forms and ensure facilities can leverage the benefitsof the electronic system, the central EPSS and MOH should
ensure the hubs and individual facilities have the physicalsupplies necessary to access the government’s technology
infrastructure.
Users report the traditional siloed process for ordering medical
supplies for each health program or clinical condition wastedious and time-consuming. Having all the supplies needed to
manage obstetric emergencies in one place, along with supplies
for other medical situations, makes the ordering process morestreamlined. Furthermore, having all this information available
in one place is useful when pharmacists and supply managers
are trying to view inventory levels, determine their consumption,and forecast future needs. Pharmacists stated they enjoy having
this information available to them electronically. As adevelopmental next step, the Ethiopian MOH could take steps
to automate the electronic components of IPLS (internet-based
and hardware devices) at all health care facilities, which will
help with the calculation of monthly consumption, waste, andexpiration so that pharmacists can have these current, real-timedata available when they are making supply requests to theregional hubs. This recommendation is based on participant
reports of calculation errors and is supported by the preexisting
literature which found forecasting errors to be a concern with
Ethiopia’s medical supply chain [20]. In addition, knowing whatis available at various hubs can assist in making supply
distribution decisions and advancing supply accessibility acrosshubs and the facilities they service, which was reported in astudy conducted in South Africa [21]. Without the availabilityof rich inventory data, facilities will continue to remain in thedark about their true readiness to manage basic obstetricemergencies, and the central EPSS and regional hubs will nothttps://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

Dougherty et al
have the necessary information to provide essential emergency
supplies prospectively, before stockouts and maternal deaths
occur [20,22].
Facilities reported using the same process for ordering andobtaining supplies was incredibly helpful. Using one system toorder all basic emergency obstetric supplies created a uniform
communication flow between individual facilities, regionalhubs, and central EPSS. However, as the IPLS ages, it will
benefit from ongoing updates and adaptation to the evolving
needs of the system’s users at multiple levels, which aligns withbest practices stating that technology should be routinely
modified and updated to both align with user feedback andcomply with recent regulatory standards [23]. For example,
providing space to justify seasonal differences in supply request
quantities would be a direct response to user requests, allowing
facilities to explain the change in demand, helping hubs andcentral EPSS to both forecast future needs based on seasonal
trends, and also helping central planners make informed,
strategic decisions even in cases where they must distribute a
smaller number of supplies to multiple hubs [24]. Furthermore,similar work conducted in Zambia identified seasonal changes
to be a driver of malaria medication stockouts in the country;
therefore, Ethiopia’s modification to include justification forseasonal differences would also align with the preexisting
literature [25].
The greatest concern related to the workflow and communicationof the OESC was an inability to accurately forecast future supply
needs. This is a critical issue because inaccurate forecasting canlead to undersupplying and stockouts or oversupplying andwaste of medication due to expiration. This same issue was
identified in a similar study of hospital laboratories in Gambela,a region in the southwest of Ethiopia, which emphasizes that
this may be a countrywide issue and not only seen Amhara [26].
When individuals know their true consumption, they can analyze
the trend in their past use patterns and forecast their future needs.
Increasing data visibility is not only a concern at facilities forpreventing maternal deaths but also crucial at every stage of the
supply chain to ensure consistent, predictable emergency supply
availability for reducing facility-based mortality at the
population level in Amhara. Previous research in the UnitedStates demonstrated the utility of using multilevel dashboards(ie, facility and national levels) to monitor the distribution andinventory status of COVID-19 vaccines [27]. Currently, thegreatest communication breakdown occurs between regional
hubs and individual facilities. Bridging the gap between these
2 areas will not only assist with forecasting but also provide an
additional level of surveillance at the regional hubs which canmonitor inventory levels at facilities and prospectively send out
supplies if they notice a facility is reaching critically low levels
to prevent stockouts in advance.
Technology updates and changes must be reviewed andmonitored not only to measure their success but also toinvestigate if there are any positive or negative unintended
consequences of the technology updates [23]. This is seen in
the Dagu 1 to Dagu 2 updates which improved data transparency
within the system but also introduced barriers to efficient useof the system at the facility level. On the basis of user complaints
from this update, it is imperative to conduct a formative
JMIR Form Res 2024 | vol. 8 | e59690 | p. 9
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
assessment with end users to ensure the technology updates
cohere with the HIT’s intended purpose and existing user
workflow [28]. Furthermore, the Dagu 1 to Dagu 2 updates
underscore the notion that usefulness does not always equal
usability, and updates to technology should be reviewed bytargeted end users to ensure they will not have unintended
negative consequences. This same problem occurred whenelectronic health records were widely integrated into hospitals
in the United States in the 2010s [29]. Before implementingchanges within the electronic components of IPLS, thoseindividuals who are leading these updates should establish aprocess to monitor and measure the impact of the changes.
The findings from these interviews highlight the need to bridge
the data visibility gap between facilities and regional hubs.
Furthermore, facility-level participants stated that they would
appreciate the ability to view inventory data electronically andhaving that information available would assist in forecasting
decisions and supply requests. The discussions with individuals
at all levels of the supply chain underscore the importance of
having data views that are tailored to the different job types andare supported by the current literature. Previous healthtechnology research has found curated technology displays fordifferent end users or job tasks to be beneficial [30]. Providing
health technology, such as dashboards with different views forthe various stakeholders in the supply chain is helpful because
it can provide a granular view for each stakeholder dependingon their information needs and job functions [31]. Different
types of dashboards that could be useful would include
dashboards for individuals who work at facilities and regional
hubs. The individual facility employees can use the dashboards
to assist with performing accurate forecasting and supply
requests, as well as monitoring their day-to-day inventory levels.
In contrast, the dashboards for region hub employees could
assist with the hub’s ability to monitor the inventory levels of
the facilities under their jurisdiction and provide supplies inadvance of stockouts.
One facilitator of the current human-computer interfaces is that
there is already strong buy-in from the users because they
believe the technology is both useful for their jobs and easy touse. While some reported ease of use issues, most respondents
indicated the existing EPSS system and IPLS were usable. New
dashboards must maintain these existing usability strengths.
Future steps can be taken to curate additional individualized
dashboards that provide the information that is most pertinent
for each stakeholder across the health system from facilities toregional hubs and the central EPSS. In addition, if individuals
continue to use the technology on tablets or mobile devices,then the EPSS could ensure the technology is tailored for use

Dougherty et alon those devices instead of only computers. This incremental
adaptation of the IPLS and EPSS interfaces to mobile technology
can ensure that the supply monitoring technology follows
usability heuristics for mobile health, which is critical to ensure
success of the technology on the new platform [32].

Limitations
One potential limitation of the study is that when transcripts
were translated from Amharic to English, there may have been
unintended translation errors. However, to mitigate this concern,a medical anthropologist, who lives in Amhara and has >14
years of experience performing qualitative data collection andtranslating data from Amharic to English, performed thetranslations. In addition, while this work was designed for healthcare facilities in the Amhara region of Ethiopia, all regionaland facility-level participants worked in Bahir Dar city, thecapital of the region. This limitation on sampling outside theregional capital was due to civil unrest which prevented teammembers from safely visiting other locations [33]. Despite thispotential limitation, the study sample represents different healthcare facility levels and regional positions and still provides arich understanding of the current OESC dynamics in Amhara.
Finally, the sample was predominately male, which may haveunintentionally excluded the opinions and experiences of female
individuals working within the OESC. However, thismale-dominated sample matches preexisting HIT user staffing
in the region.

Conclusions
This study captured insight into the information andcommunication flow within Amhara’s OESC. Participant
responses identified several strengths and barriers related to thesuccess of the current IPLS. A frequently noted strength was
the high degree of integration of medications and supplies thatcan be purchased through the system. Common barriers include
a lack of data transparency at the facilities and also between
facilities and regional hubs. The findings offer several
recommendations for how future technology can be designed
and tailored to meet the needs of current OESC employees. In
addition, the results of the interviews underscored theimportance of conducting qualitative research early in thedevelopmental process of HIT to ensure a rich understandingof the current environment and user tasks that the technology
will need to accomplish. Although these data come from Amhara
and national MOH informants, the EPSS system and IPLS arecommon across Ethiopia’s regions. Consequently, HIT-based
innovations resulting from this research can be readily adapted
to the hubs and facilities in multiple Ethiopian regions.

Acknowledgments
This work was supported by the National Institute of Nursing Research (F31NR020569) and a Fulbright US student research
grant to Ethiopia.

Data Availability
The data sets generated during and analyzed during this study are available from the corresponding author on reasonable request.

https://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e59690 | p. 10(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Dougherty et al

Conflicts of Interest
None declared.

Multimedia Appendix 1
Qualitative interviews guide for semistructured interviews with obstetric emergency supply chain employees in Amhara, Ethiopia:
English version.
[DOCX File , 15 KB-Multimedia Appendix 1]

Multimedia Appendix 2
Qualitative codebook for semistructured interviews with obstetric emergency supply chain employees in Amhara, Ethiopia.
[DOCX File , 18 KB-Multimedia Appendix 2]

Multimedia Appendix 3
Thematic saturation table for qualitative semistructured interviews with obstetric emergency supply chain employees in Amhara,
Ethiopia; tracking codes obtained during each data collection session.
[DOCX File , 20 KB-Multimedia Appendix 3]

Multimedia Appendix 4
Study participant demographics from semistructured qualitative interviews with obstetric emergency supply chain employees inAmhara, Ethiopia.
[DOCX File , 14 KB-Multimedia Appendix 4]

Multimedia Appendix 5
Additional Sociotechnical Model dimensions not included in the main manuscript from the qualitative interviews with facility-,
regional-, and federal-level obstetric emergency supply chain employees during qualitative semistructured interviews in Amhara,
Ethiopia.
[DOCX File , 22 KB-Multimedia Appendix 5]

Multimedia Appendix 6
Data sources and communication channels that were identified as currently available to facility-, regional-, and federal-levelobstetric emergency supply chain employees during qualitative semistructured interviews in Amhara, Ethiopia.
[DOCX File , 16 KB-Multimedia Appendix 6]

Multimedia Appendix 7
Human-computer interfaces that were reported as currently available to facility-level, regional-level, and federal-level obstetricemergency supply chain employees during qualitative semistructured interviews in Amhara, Ethiopia.
[DOCX File , 18 KB-Multimedia Appendix 7]

Multimedia Appendix 8
Reported barriers and facilitators for the human-computer interface dimensions (all aspects of the digital application that the user
can see, touch, hear, or manipulate) of the Sociotechnical Model for Studying Health Information Technology in Complex
Adaptive Healthcare Systems extracted from semistructured interviews with obstetric emergency supply chain employees inAmhara, Ethiopia.
[DOCX File , 23 KB-Multimedia Appendix 8]

References
1.
2.
3.

Maternal, neonatal and child health. United States Agency for International Development. URL: https://www.usaid.gov/
ethiopia/global-health/maternal-and-child-health [accessed 2024-04-29]
Ethiopia. The World Bank. URL: https://genderdata.worldbank.org/countries/ethiopia/#:~:text=267%20women%20die%20per%20100%2C000,2000%20to%20267%20in%202020 [accessed 2024-01-10]
Austin A, Langer A, Salam RA, Lassi ZS, Das JK, Bhutta ZA. Approaches to improve the quality of maternal and newborn
health care: an overview of the evidence. Reprod Health. Sep 04, 2014;11(2):S1. [doi: 10.1186/1742-4755-11-s2-s1]

https://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e59690 | p. 11(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
4.
5.

6.

7.
8.

9.
10.
11.
12.
13.
14.
15.
16.

17.

18.

19.

20.

21.

22.

23.
24.

25.

26.

Ethiopia: demographic and health survey. Central Statistical Agency Addis Ababa, Ethiopia. URL: https://dhsprogram.
com/pubs/pdf/FR328/FR328.pdf [accessed 2024-04-29]
Cranmer JN, Young S, Spangler S, Gebremariam A. Survive and thrive: transforming labor-related emergency care in
Ethiopia’s health system by linking novel assessment with real-time data-to-action solutions. Emory Global Health Institute.
2020. URL: https://globalhealth.emory.edu/faculty/seed-grants-awardees.html [accessed 2024-04-29]
Cresswell KM, Bates DW, Sheikh A. Ten key considerations for the successful implementation and adoption of large-scale
health information technology. J Am Med Inform Assoc. Jun 2013;20(e1):e9-13. [FREE Full text] [doi:10.1136/amiajnl-2013-001684] [Medline: 23599226]
Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare
systems. Qual Saf Health Care. Oct 2010;19 Suppl 3(Suppl 3):i68-i74. [doi: 10.1136/qshc.2010.042085] [Medline: 20959322]
Dougherty K, Gebremariam Gobezayehu A, Lijalem M, Alamineh Endalamaw L, Biza H, Cranmer JN. Comparison ofobstetric emergency clinical readiness: a cross-sectional analysis of hospitals in Amhara, Ethiopia. PLoS One. Aug
2023;18(8):e0289496. [FREE Full text] [doi: 10.1371/journal.pone.0289496] [Medline: 37535678]
Doyle L, McCabe C, Keogh B, Brady A, McCann M. An overview of the qualitative descriptive design within nursing
research. J Res Nurs. Aug 18, 2020;25(5):443-455. [FREE Full text] [doi: 10.1177/1744987119880234] [Medline: 34394658]
Sandelowski M. Whatever happened to qualitative description? Res Nurs Health. Aug 2000;23(4):334-340. [doi:10.1002/1098-240x(200008)23:4<334::aid-nur9>3.0.co;2-g] [Medline: 10940958]
Guba EG. Criteria for assessing the trustworthiness of naturalistic inquiries. Educ Technol Res Dev. Jun 1981;29(2):75-91.
[FREE Full text] [doi: 10.1007/bf02766777]
Securely save and share what’s important with OneDrive. Microsoft. URL: https://www.microsoft.com/en-us/microsoft-365/
onedrive/online-cloud-storage [accessed 2024-04-29]
ATLAS.ti. URL: https://atlasti.com/atlas-ti-desktop [accessed 2024-04-29]
Ancker JS, Benda NC, Reddy M, Unertl KM, Veinot T. Guidance for publishing qualitative research in informatics. J Am
Med Inform Assoc. Nov 25, 2021;28(12):2743-2748. [FREE Full text] [doi: 10.1093/jamia/ocab195] [Medline: 34537840]
Guest G, Bunce A, Johnson L. How many interviews are enough?: an experiment with data saturation and variability. Field
Methods. Jul 21, 2016;18(1):59-82. [doi: 10.1177/1525822X05279903]
Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, et al. Saturation in qualitative research: exploring its
conceptualization and operationalization. Qual Quant. Mar 2018;52(4):1893-1907. [FREE Full text] [doi:10.1007/s11135-017-0574-8] [Medline: 29937585]
Kaboré SS, Ngangue P, Soubeiga D, Barro A, Pilabré AH, Bationo N, et al. Barriers and facilitators for the sustainabilityof digital health interventions in low and middle-income countries: a systematic review. Front Digit Health. Nov 28,2022;4:1014375. [FREE Full text] [doi: 10.3389/fdgth.2022.1014375] [Medline: 36518563]
Manyazewal T, Woldeamanuel Y, Blumberg HM, Fekadu A, Marconi VC. The potential use of digital health technologies
in the African context: a systematic review of evidence from Ethiopia. NPJ Digit Med. Aug 17, 2021;4(1):125. [FREE Fulltext] [doi: 10.1038/s41746-021-00487-4] [Medline: 34404895]
Tegegne MD, Wubante SM. Identifying barriers to the adoption of information communication technology in Ethiopian
healthcare systems. a systematic review. Adv Med Educ Pract. 2022;13:821-828. [FREE Full text] [doi:10.2147/AMEP.S374207] [Medline: 35959138]
Boche B, Mulugeta T, Gudeta T. Procurement practice of program drugs and its challenges at the Ethiopian pharmaceuticals
supply agency: a mixed methods study. Inquiry. Mar 22, 2022;59:469580221078514. [FREE Full text] [doi:10.1177/00469580221078514] [Medline: 35316108]
Mabizela S, Nakambale HN, Bangalee V. Predictors of medicine redistribution at public healthcare facilities in King
Cetshwayo District, KwaZulu-Natal, South Africa. BMC Health Serv Res. Oct 17, 2023;23(1):1108. [FREE Full text] [doi:10.1186/s12913-023-10096-4] [Medline: 37848899]
Agarwal S, Glenton C, Henschke N, Tamrat T, Bergman H, Fønhus MS, et al. Tracking health commodity inventory andnotifying stock levels via mobile devices: a mixed methods systematic review. Cochrane Database Syst Rev. Oct 28,
2020;10(10):CD012907. [FREE Full text] [doi: 10.1002/14651858.CD012907.pub2] [Medline: 33539585]
Alotaibi YK, Federico F. The impact of health information technology on patient safety. Saudi Med J. Dec 03,
2017;38(12):1173-1180. [FREE Full text] [doi: 10.15537/smj.2017.12.20631] [Medline: 29209664]
Cranmer JN, Dettinger J, Calkins K, Kibore M, Gachuno O, Walker D. Beyond signal functions in global obstetric care:using a clinical cascade to measure emergency obstetric readiness. PLoS One. Feb 23, 2018;13(2):e0184252. [FREE Fulltext] [doi: 10.1371/journal.pone.0184252] [Medline: 29474397]
Leung NZ, Chen A, Yadav P, Gallien J. The impact of inventory management on stock-outs of essential drugs in sub-SaharanAfrica: secondary analysis of a field experiment in Zambia. PLoS One. May 26, 2016;11(5):e0156026. [FREE Full text][doi: 10.1371/journal.pone.0156026] [Medline: 27227412]
Boche B, Temam S, Kebede O. Inventory management performance for laboratory commodities and their challenges inpublic health facilities of Gambella Regional State, Ethiopia: a mixed cross-sectional study. Heliyon. Nov 2022;8(11):e11357.
[FREE Full text] [doi: 10.1016/j.heliyon.2022.e11357] [Medline: 36387489]

https://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

Dougherty et al

JMIR Form Res 2024 | vol. 8 | e59690 | p. 12(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
27.

28.
29.
30.

31.
32.

33.

Dougherty et al

Shumunov A, Fina P, Martin J, Barrett A, Echevarria K, Cunningham F. Emergency use authorization medication distribution,utilization, and monitoring in a national integrated healthcare system. Am J Health Syst Pharm. Apr 08, 2023;80(8):518-525.
[doi: 10.1093/ajhp/zxac386] [Medline: 36566503]
Usability 101: introduction to usability. Nielsen Norman Group. URL: https://www.nngroup.com/articles/
usability-101-introduction-to-usability/ [accessed 2021-10-10]
Coiera E, Ash J, Berg M. The unintended consequences of health information technology revisited. Yearb Med Inform.
Mar 06, 2018;25(01):163-169. [doi: 10.15265/iy-2016-014]
Rabiei R, Bastani P, Ahmadi H, Dehghan S, Almasi S. Developing public health surveillance dashboards: a scoping review
on the design principles. BMC Public Health. Feb 06, 2024;24(1):392. [FREE Full text] [doi: 10.1186/s12889-024-17841-2][Medline: 38321469]
Kruglov A, Strugar D, Succi G. Tailored performance dashboards-an evaluation of the state of the art. PeerJ Comput Sci.
2021;7:e625. [FREE Full text] [doi: 10.7717/peerj-cs.625] [Medline: 34712785]
Galavi Z, Norouzi S, Khajouei R. Heuristics used for evaluating the usability of mobile health applications: a systematic
literature review. Digit Health. May 15, 2024;10:20552076241253539. [FREE Full text] [doi: 10.1177/20552076241253539][Medline: 38766365]
Alene GD, Ayenew GM, Bantie GM, Zeru T, Yismaw G. Impact of war on maternal health services in the war affectedareas of the Amhara region, Ethiopia: disruption of antenatal care, institutional delivery and postnatal services. Ethiop J
Transl Sci. 2023;1(1):29-41. [FREE Full text]

Abbreviations
BEmOC: basic emergency obstetric care needs
EPSS: Ethiopian Pharmaceutical Supply Service
HIT: health information technology
IPLS: integrated pharmaceutical logistics system
MMR: maternal mortality ratio
MOH: Ministry of Health
OESC: obstetric emergency supply chain
RRF: requesting and reporting form

Edited by A Mavragani; submitted 19.04.24; peer-reviewed by EE Egger; comments to author 13.06.24; revised version received
17.06.24; accepted 03.07.24; published 05.09.24
Please cite as:
Dougherty K, Gebremariam A, Biza H, Belew M, Benda N, Tesfaye Y, Cranmer J, Bakken S
Obstetric Emergency Supply Chain Dynamics and Information Flow Among Obstetric Emergency Supply Chain Employees: Key
Informant Interview StudyJMIR Form Res 2024;8:e59690
URL: https://formative.jmir.org/2024/1/e59690
doi: 10.2196/59690
PMID: 39235860©Kylie Dougherty, Abebe Gebremariam, Heran Biza, Mulusew Belew, Natalie Benda, Yihenew Tesfaye, John Cranmer, Suzanne
Bakken. Originally published in JMIR Formative Research (https://formative.jmir.org), 05.09.2024. This is an open-access article
distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which
permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR
Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on
https://formative.jmir.org, as well as this copyright and license information must be included.

https://formative.jmir.org/2024/1/e59690

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e59690 | p. 13(page number not for citation purposes)