The CommCare
Evidence Base for
Frontline Workers
Overview
March 2021
More than 75 peer-reviewed studies
have been conducted exploring themobile data collection platform
CommCare’s impact on frontline
programs globally.
Collectively, these studies provide
strong evidence that equipping
Frontline Workers (FLWs) with
CommCare improves FLW
performance, client behaviors, andclient outcomes.

1OVERVIEW
Around the world, Frontline Workers (FLWs) provide a key point of contact between health systems andhouseholds as the first point of care in their communities. They play a critical role in their communities’
health and wellbeing, especially in low-resource and remote communities. To capitalize on this opportunity,
both governments and non-governmental organizations are increasingly equipping FLWs with digital tools
to improve their effectiveness in delivering community services.
Dimagi (www.dimagi.com) is a social enterprise that builds digital products and provides services for
programs across several sectors, including health, agriculture, and education. Dimagi’s aim is to support
wide scale use of digital products in order to improve global development efforts and help accelerate theend of extreme poverty.
Dimagi’s primary product, CommCare (www.commcarehq.org), is a user-configurable, open source
platform that enables anyone to build mobile applications to support data collection, counseling, behavior
change, and a variety of other functions. FLWs use CommCare to track and support their clients withregistration forms, checklists, SMS reminders, and multimedia. Dimagi built CommCare based on the
belief that it has the potential to achieve wide-scale use by frontline programs, and to help dramatically
improve those programs.
The purpose of this document is to synthesize available evidence to understand CommCare’s effect on
FLWs, and its subsequent impact on FLW’s clients. In particular, we want to understand what the available
evidence is on the following hypothesis:

Equipping FLWs with CommCare can improve the service delivery provided bythose FLWs and lead to improved client outcomes and behaviors

2

To view a comprehensive list of research studies evaluating CommCare, please refer to this online repository.
CommCare has been extensively studied, with new studies being shared and published frequently. The
CommCare Evidence Base is an online list of studies about CommCare maintained by Dimagi. Dimagi adds
new studies to the online list as they are discovered. The Overview of the CommCare Evidence Base (thisdocument) provides an overview of key learnings we have drawn from these studies. The Evidence Baseincludes studies that meet both of the following criteria:
1

The study helps assess the effect of FLWs using CommCare in terms of FLW service delivery,client behaviors, or outcomes.

2

Either the study is described in a peer-reviewed publication or is a “grey literature” study that, in
our opinion, contributes substantially to the understanding of the value of FLWs using CommCare
relative to peer-reviewed publications about CommCare.

We interpret the first criteria relatively broadly. For example, we would exclude a study that happened
to use CommCare for data collection in lieu of paper surveys, because that use of CommCare (while
important) is not relevant to how CommCare affects or improves the work of FLWs. In previous versionsof the Evidence Base, we included studies that were conducted on precursors of CommCare. We now
exclude these studies, though do reference them in the discussion below.
For the second criteria, our intention is to include all relevant peer-reviewed studies and a subset of theunpublished (often called grey literature) studies that we think are worth including. As the Evidence Basehas grown, we have removed grey-literature studies that no longer add substantially to the peer-reviewed
ones. Figure 1 shows the number of studies in the CommCare Evidence Base by year and category:

Figure 1: Number of studies by category in the CommCare Evidence BaseIn the above graph, the “Client” category shows all peer-reviewed papers that investigate the effect ofequipping FLWs with CommCare on client outcomes. The “FLW” category shows all peer-reviewed
papers that look at how CommCare impacts FLW service delivery. The “Feasibility” category shows peerreviewed papers that don’t fit the criteria of the “Client” or “FLW” categories, but rather demonstrates overall
acceptability of CommCare or describes some important conceptual frameworks.

3

The “Data” category (in which there are only three papers) focuses on demonstrating value that has been
derived from the data collected by CommCare. Finally, the “Grey lit” category (referring to “grey literature”)
includes studies that are not peer-reviewed but that we deemed important in understanding CommCare.
We are very excited to see the continued growth of the CommCare Evidence Base, as well as increasing
evidence in support of our original hypothesis.

FLW Performance
In a randomized controlled trial (RCT) in
India, clients of FLWs using
CommCare saw a 73% increase inantenatal care visits, a 58% increase in
the consumption of iron tablets, and a 36%
increase in the use of contraception[Borkum, 2015].
An RCT in Tanzania documented a 74%
institutional delivery rate for clients whose
FLWs use CommCare, compared to a 63%
institutional delivery rate in the control group.

FLW Performance
In India, a study found that after
introducing CommCare, the number of
FLW visits per pregnant women nearly
doubled [Murless, 2015].

Equipping
FLWs with
CommCareimproves

Quality of Care

In Nigeria, FLWs combating Ebola
increased their knowledge of the disease,with statistically significant improvements[Otu, 2016].

FLW Performance

In Burkina Faso, a study found that FLWs
that used CommCare completed 79% of theirtasks in adhering to IMCI protocols
compared to 54% in the control areas, from abaseline of 48% [Sarrassat, 2019].

An RCT in India showed a 24% increasein FLW home visits via performance
feedback. [DeRenzi, 2016].
An RCT in South Africa showed a 75%
reduction in average FLW training
time and a 41% reduction in average
screening time compared to paper
protocols [Surka, 2014].

In Nigeria, a study found that CommCare
increased the frequency at which
healthcare providers provided HIV tests forantenatal care, from 67.5% to 82.2%[McNabb, 2015].

4

FINDINGSTHEME 1 : CLIENT HEALTH OUTCOMES AND BEHAVIORS
The field of digital health is often criticized for lacking rigorous impact evaluations [Agarwal, 2015]. Thissection describes several studies about CommCare that have addressed this gap, including two RCTS.
This section also describes other evidence that equipping FLWs with CommCare can improve theirclients’ behaviors and outcomes.

Bihar, India
Improved client behaviours and outcomesA study conducted by Mathematica Policy Research provides statistically significant evidence of substantive
change in client behaviors from equipping FLWs with CommCare [Borkum, 2015]. The study was conductedin the Saharsa district in Bihar, India—a district that has had persistently low health outcomes in one ofIndia’s poorest and most populous states. The study reports on data from surveys of 1,500 women on theirhealth-seeking behaviors. The 1,500 women were sampled from 70 sub-centers in Saharsa district, half ofwhich were randomized to be in the intervention group and the other half in the control group. All 70 subcenters were already receiving the benefit of extensive health system strengthening services implemented
by CARE International. These services included client mapping, home visit trainings and resources for
FLWs, reproductive and family health counseling tools, sub-center-based interventions to improve thequality of deliveries, and an Interactive Voice Response (IVR) system developed by BBC Media Action andGrameen Foundation to deliver health messages via interactive voice messages to client households. The
only difference between the 1,500 women surveyed in the control and intervention areas was that CARE
International equipped FLWs in the intervention with a CommCare application that features a scheduler
supported by MOTECH (a separate digital health platform).
The intervention area (i.e. equipping FLWs with a CommCare application) resulted in statistically significant
impacts on FLW interactions, including antenatal care, delivery and newborn care, child nutrition, andreproductive health. The only health domain that didn’t see a statistically significant impact was child
immunizations. All four of these statistically significant impacts were positive, i.e., improved by the
CommCare intervention. Figure 2 below shows the improvement in the intervention group as percentage
increase relative to the control group among the statistically significant primary outcomes. These results areespecially notable given the strong (mostly non-ICT) health system strengthening services implemented
by CARE International in the control arm. This study shows the value-add benefits of ICT in addition toextensive support for the community health system.

5

Figure 2 shows statistically significant improvements across antenatal care, delivery and newborn care,child nutrition, and reproductive health. Most notably, the number of women who attended at least threeantenatal care visits was 73% higher among those whose FLWs used CommCare as compared to thecontrol group (whose FLWs did not use CommCare). Interestingly, the study found that increases in client
health outcomes were not consistently accompanied by increases in client knowledge of such behaviors,
which indicates that the use of CommCare addressed other barriers to client health-seeking behaviors,such as cultural norms.

Figure 2: The statistically significant improvements in antenatal care, delivery and newborn care, child nutrition, andreproductive health in the CommCare intervention group.

The report also suggests several areas for improvement in implementing CommCare. There were a
few indicators where behaviors decreased, though these were not statistically significant results. On a
programmatic level, the study reported challenges with data connectivity and broken mobile phones, whichsometimes resulted in poor coordination between FLWs and supervisors. FLWs also experienced anincreased workload due to the continuation of paper documentation required by the government.
Despite these logistical challenges and the lack of improvements in supervision, equipping FLWs with
CommCare created a significant impact on client health outcomes and behaviors.

Tanzania
Increased facility-based delivery rates
The results from an RCT in rural Tanzania report that mothers tracked by FLWs using CommCare had
increased facility-based delivery rates, especially among first-time mothers with low antenatal care uptake[Hackett, 2018]. As with the study in Bihar, both the control and intervention groups received health systemstrengthening services – in this case, these services were administered by World Vision. The FLWs in the6intervention group were also equipped with CommCare applications (implemented and supported by D-tree
International) to assist with data collection, education, danger sign identification, and referrals.
The study included 32 villages that were cluster-randomized to control or intervention. The study surveyeda total of 572 women and found that “The smartphone intervention was associated with significantly higher
facility delivery: 74% of mothers in intervention areas delivered at or in transit to a health facility, versus
63% in control areas. The odds of facility delivery among women counseled by smartphone-assisted healthworkers were double the odds among women living in control villages (OR, 1.96; CI, 1.21±3.19; adjusted
analyses).” First-time mothers with low antenatal care uptake were impacted the most by the CommCareintervention – their facility-based delivery rates were 32% higher than the control group.

Uttar Pradesh, India
Improvements in antenatal and postnatal care
In India, Catholic Relief Services (CRS) deployed the ReMiND (‘Reducing Maternal and Newborn Deaths’)project developed on the CommCare platform. The intervention required development and implementationof a mobile application used as a job aid by ASHAs for registering pregnant women and for providing real‐time guidance through key counselling points, decision support, timely alerts and referral algorithms forvarious maternal and child health issues.
A study of the effectiveness of ReMiND found a statistically significant increase in coverage of iron–folic acidsupplementation (12.58%), self‐reporting of complications during pregnancy (13.11%) and after delivery(19.6%). The coverage of three or more antenatal care visits increased significantly in the intervention area
(10.3%) [Prinja, 2017].

Guatemala
Reduced maternal and infant mortality
A study on TulaSalud’s implementation of CommCare in Guatemala provides evidence of reduced maternalmortality rates (MMR) and infant mortality rates (IMR), as compared to the control areas and the provincial
average [Martinez-Fernandez, 2015].
The ICT intervention took place in Alta Verapaz, a predominantly rural region of northern Guatemala withhigh maternal and infant mortality rates. 125 FLWs in Alta Verapaz were equipped with mobile phones andthe Kawok system developed by TulaSulud, which was built on CommCare, to assist with making client
consultations, collecting epidemiological data, receiving continuous training, and performing community
health promotion and prevention activities. After five years (2008 to 2012) of this intervention that included
CommCare, an observational study was conducted to compare the MMR and IMR between the districts
with FLWs using CommCare, and those without it. Both the intervention and control areas had similar
hospital access, racial/ethnic makeup, age, education, etc.
The study found that in intervention areas, MMR decreased by 18% from 309 to 254 maternal deaths per
100,000 live births (p<0.05), and IMR decreased by 48% from 25 to 13 infant deaths per 1,000 live births(p=0.054). Figure 3 describes the IMR rates over time in the areas with and without the ICT intervention, as
well as the entire region between 2008 and 2012.

7

Figure 3: This figure maps the decrease in IMR rate from 2008 to 2012, dropping from 25 infant deaths to 13
infant deaths per 1,000 live births within the intervention as compared to the control areas and the entire region
over the course of the five-year CommCare intervention [Martinez-Fernandez, 2015].

Other studies
Client health outcomes and behaviours
Several other studies also showed client health-seeking behavior change in the areas of antenatalcare, institutional delivery, and postnatal care. While the above studies are each more rigorous thanthe ones described in this section, these additional studies suggest that the findings are generalizable
to other contexts.
Several other studies reported high institutional delivery rates among clients of FLWs using CommCare[Amoah, 2016] [Battle, 2015] [Hoy, 2015] [World Vision, 2013]. For example, D-tree International
implemented a project using CommCare in Zanzibar to increase the institutional delivery rates,
especially in cases of complicated pregnancies [Hoy, 2015]. Traditional birth attendants (TBAs) were
equipped with CommCare to identify danger signs, refer clients, record family members’ permission totransport the women to a health facility in case of emergencies, and facilitate transportation payment
to local vehicle owners. The intervention reported a 71% institutional delivery rate, compared to theregional average of 32%.
These studies demonstrate that the effects of CommCare hold over a wide range of geographies andmaternal and child health programs. The results show that women with previously lower antenatal
care uptake [Hackett, 2018] and lower castes [Borkum, 2015] experience a higher positive impact in
CommCare interventions. Taken together, there is a strong pool of evidence indicating that pregnant
women who are tracked by FLWs using CommCare have improved health outcomes and behaviors in
the areas of antenatal care, institutional deliveries, and child health.

8

THEME 2 : FLW PERFORMANCE
There is a large body of evidence evaluating how equipping FLWs with CommCare affects FLWs’
performance. This section presents findings in four areas: FLW legitimacy and client interaction, FLW
knowledge, FLW Activity and FLW performance

FLW legitimacy and client interaction
Several studies have shown that CommCare improves FLWs’ personal credibility and the credibilityof the health messages they deliver [Bhavsar, 2014] [Medhi, 2012] [Schwartz, 2013] [Braun, 2016][Gopalakrishnan, 2020]. These findings have emerged from qualitative interviews with FLWs, who havereported that CommCare enhances their credibility in their communities, and that clients and clients’families perceive recorded messages as more trustworthy. CommCare is widely viewed as an independent,
objective source of information, which greatly benefits FLWs’ ability to deliver sensitive messages. FLWs
and clients in Tanzania reported that CommCare is a highly acceptable counseling tool, particularly for its
improved sense of privacy and trust with clients [Braun, 2016].
A study of 50 FLWs in India found that home visits with pregnant women were more inclusive and interactive
when CommCare was used. The client’s husband and mother-in-law were 60% and 110% more likely
to participate during the visit, respectively, and the client was 33% more likely to ask questions when
CommCare was used by the FLW [Mohamed, 2014]. The Mathematica study in Bihar also found that
FLWs who use CommCare were 37% more likely to report a high level of confidence in their skill andability to do their job, and 20% more likely to run regular FLW meetings by themselves [Borkum, 2015]. In a
qualitative study examining both FLWs’ and beneficiaries’ perceptions of a CommCare intervention in Bihar
and Madhya Pradesh, FLWs reported being better able to engage and build trust with influential household
members beyond their direct clients, such as mothers-in-law, which in turn increased direct clients’ ability
to interact with FLWs and receive necessary care. FLWs also reported that features of this intervention
intrigued other community members and encouraged them to gather around FLWs to receive information,
thus allowing FLWs to engage with more clients [Gopalakrishnan, 2020].

9

FLW knowledge
FLWs who use CommCare for maternal and child health interventions have been shown to be more
knowledgeable about the health topics and services they provide. A study on CommCare in India showed
improvements in FLW knowledge of at least three to five pregnancy danger signs by 22% [Kumar, 2012]. In
Nigeria, FLWs combating the Ebola Virus Disease improved their knowledge of the disease, with statisticallysignificant improvements (p<.05) on questions about human transmission of the virus, common symptoms,
and whether Ebola fever is preventable. The study also noted reinforcement against risky behaviors suchas contact with Ebola patients, eating bush meat, and risky burial practices [Otu, 2016].

FLW activity
FLWs whose tasks are tracked through CommCare have been found to perform more efficiently andconsistently. A study in South Africa observed an increase in adverse event form submission from 5% to
27% when switching from paper forms to CommCare [Chaiyachati, 2013]. Two studies in India found that
CommCare improves FLW performance during home visits, particularly in their frequency and timeliness[Borkum, 2015]. In particular, CRS found that, after introducing CommCare, the percentage of women
ever visited by a FLW increased by 15%, the number of FLW visits per pregnant women nearly doubled,
and the percentage of women receiving counseling from their FLW increased by 28% [Murless, 2015]. In
Bihar, India, Mathematica surveyed clients about FLW home visit consistency and found that FLWs using
CommCare were more likely to conduct visits at critical times throughout pregnancy and early childhood
than in the control group (Figure 4) [Borkum, 2015].

Figure 4: FLW Visit Frequency (reported by clients). This figure reports increases in FLW visit frequency, particularlyin the final trimester and within one week of delivery, among women in the CommCare intervention group ascompared to the control [Borkum, 2015].

Home visit consistency improved at every recorded stage of maternal and newborn care among FLWs who
used CommCare in the Mathematica study in Bihar. In a study conducted in Indonesia, the ability to track
FLW tasks through CommCare was cited as a mechanism for motivation and engagement [Walton, 2020].
These findings reinforce the notion that CommCare’s capacity to track when and how FLWs perform tasks
in the field increases performance and accountability.

10

FLW performance
Two RCTs have been conducted to measure the added impact of supplemental features to the CommCareapplication on FLW performance. The first study uses SMS reminders while the second uses web- andvoice-based performance feedback to evaluate the added impact of the CommCare additions on FLW
performance.

SMS feedback
An RCT in Tanzania found that SMS feedback generated from data collected by CommCare increased
FLW visit frequency. The approach hinges on the fact that FLWs’ visits are reported in near real-time to
CommCare’s web portal, where visit data is monitored by FLW supervisors. The study found that SMS
reminders that were escalated to a supervisor in the case of a missed visit improved FLW visit timelinessby 86%, compared to CommCare-using FLWs who did not receive SMS reminders [DeRenzi, 2012].

Self-Tracking Tool
An RCT in India was conducted to measure the impact of phone-based motivational messages on FLW
performance [DeRenzi, 2016]. The intervention was randomized into two groups, one who received genericadvice and encouragement messages irrespective of their performance, and the intervention group who
was given a self-tracking tool allowing FLWs to monitor their own performance through visual graphs andaudio messages.
The study found that the intervention group made 24% more visits than the control group within the
12-month intervention period. The study also found a correlation between FLW performance and usage ofthe self-tracking tool. Within the intervention group, most FLWs used both web- and voice-based feedback
channels, highlighting the demand for diverse feedback mechanisms in FLW programs.

11

THEME 3 : QUALITY OF CARE
Inconsistent health worker trainings, high patient loads, and minimal resources to perform medical tasks
hamper many health facilities. CommCare has addressed some of these pain points by guiding healthworkers through clinical processes to improve adherence to protocols.

Antenatal care and delivery
CommCare has proven effective in improving antenatal care and delivery. In Nigeria, Pathfinder Internationalfound that CommCare increased the antenatal care visit quality score from 13.3 at baseline to 17.2 (outof 25) at end-line in antenatal care clinics. The study found that CommCare improved the quality of healthcounseling during ANC, and the frequency at which healthcare providers performed more technical aspectsof care during antenatal care visits, particularly the increased provision of HIV tests from 67.5% to 82.2%[McNabb, 2015]. In India, a mobile partograph (mLabour) built on the CommCare platform was createdto overcome barriers to partograph use in under-resourced health systems. mLabour provides labor unit
staff with decision support, automatic graphing to replace the paper partograph, and reminders prompting
clinicians to conduct patient exams [Khalid, 2015]. A preliminary study found that data collected throughmLabour was more complete than the corresponding paper charts, and users reported that the application
reduced patient neglect [Schweers, 2015].

Adherence to protocols
CommCare has also been used to improve adherence to protocols by FLWs in low-resource settings. An
RCT in South Africa found that FLWs using CommCare for cardiovascular disease screenings had no errors
in calculating risk scores, compared to 3.8% error when using the paper tool [Surka, 2014]. Clinicians usinga CommCare precursor for Integrated Management of Childhood Illness (IMCI) classification completed
20% more of the required steps on average [DeRenzi, 2008]. They also were able to more accurately
classify diseases (90.9% versus 82.7%) that were more prevalent across clinics [Mitchell, 2013].
More recently, Terre des Hommes and partners implemented the Integrated eDiagnosis Approach(IeDA) interventions at large scale in Burkina Faso, which includes non-technology components andtechnology components, including using CommCare to guide health providers through the IMCI protocols.
An independent, stepped-wedge cluster randomized study was conducted to assess whether the IeDA
increased adherence to the IMCI guidelines during under-five child consultations [Sarrassat, 2019]. Theintervention districts showed a much higher adherence rate, including that the FLWs in the intervention
completed 79% of their tasks compared to 54% in the control areas, from a baseline of 48% (p=.0002).

12

FLWs were compared to nurses who classified the children separately. The FLWs in the intervention
completed 79% of their tasks compared to 73% in the control areas, from a baseline of 75% (p=.04),
showing a smaller but still important improvement in classification accuracy than the improvement infollowing protocols.
Additionally, significant improvements were found while classifying and prescribing malnutrition anddysentery. Though the sample size was small, the data also showed consistency around improvement indanger sign identification, correct referrals/hospitalisations and management of severe malaria or severe
febrile illness.
CommCare has also been used to improve the accuracy of disease screenings for HIV in South Africa[Mitchell, 2012], Acute Malnutrition in India [Chanani, 2015], nutrition-related stunting and anemia in children
under the age of 2 in Indonesia [Htet, 2019] and Rheumatic Heart Disease (RHD) in Zambia [van Dam,2015]. Two studies have found that CommCare improves medicine dosing [Segal, 2015] [Palazuelos,
2013]. In Guatemala, CommCare was used to improve efficiency in calculating prescription dosages by
20%, and decreased consultation time [Segal, 2015]. In Mexico and Guatemala, CommCare use resulted
in a higher medicine dosing accuracy compared to paper-based tools [Palazuelos, 2013].

13

THEME 4 : PROGRAM EFFICIENCYAs CommCare empowers FLWs and improves clinical processes, the effects of the technology inturn improves the entire FLW program. The following studies report the effects of CommCare that are
not specific to either client behaviors, client outcomes, or FLW performance, but rather improve FLW
processes at a programmatic level in terms of FLW productivity, FLW communication and supervision,and FLW program cost-effectiveness.

Program productivity
Several studies discuss the impact of CommCare on data collection and transmission in projects acrosswater, sanitation, and hygiene, malaria, and HIV/AIDS programs. A water, sanitation, and hygiene study
across Vietnam, Cambodia, and Mozambique found that CommCare improved the efficiency of water
data quality transmission from water supply structures to upper administrative levels [Ball, 2013]. Researchon the use of algorithms to evaluate data submitted by FLWs found that CommCare is able to identify
false data with 80% sensitivity and 90% specificity, validating CommCare as a monitoring and evaluation(M&E) and data collection tool for frontline programs [Birnbaum, 2012]. Two studies discuss the impact
of CommCare’s improved data collection on infectious disease control efforts. In Zimbabwe, improved
transmission of malaria test data through CommCare resulted in faster and more accurate diagnoses[Dell, 2014]. A study by MEASURE Evaluation in Mozambique found CommCare to be a more efficient,effective, and cost-effective tool for monitoring HIV/AIDS patient adherence to treatment programs andappointments than paper-based systems [Nascimento, 2014]. An RCT in South Africa found that FLWs
using CommCare for cardiovascular disease screenings took 75% less time to be trained than FLWs
using paper-based screening tools, and 41% less time to diagnose patients for cardiovascular disease
than those using paper tools [Surka, 2014].

Program communication and supervision
As a result of improved data collection and accessibility, several studies have found that CommCare
can impact supervision and communication within frontline programs. In one project in India, improved
data completeness via CommCare resulted in a reduction in the average data transmission time from
FLWs to supervisors from 48 days to 8 hours [Medhi, 2012]. The Mathematica study in Bihar found that
FLWs who used CommCare were 31% more likely to communicate about coordinating home visits intheir catchment area than FLWs who did not use CommCare (p=.018). It also highlighted challenges withsupervision, such as low understanding and usage of a specialized CommCare application for supervisors,
resulting in no substantial increase to FLW supervision (although none of the supervision indicators were
statistically significant) [Borkum, 2015]. Both studies in India saw improved communication with the useof CommCare by FLWs, although only minor improvements in direct supervision were observed.

14

Program cost-effectiveness
USAID commissioned an independent cost-effectiveness analysis for the ReMiND project in Uttar Pradesh(described above). The purpose of the study was to estimate the incremental cost per disability adjusted life
year (DALY) averted and the cost of each death averted as a result of ReMiND intervention, as compared
to routine maternal and child health programs without ReMiND [Prinja, 2018]. The researchers built an
Excel model to estimate change in DALYs and cost from both a health system and societal perspective,
assuming ReMiND over a 10 year period if ReMiND were scaled to all of Uttar Pradesh. Probabilistic
sensitivity analysis was undertaken to account for parameter uncertainties. The analysis found that overa 10-year time period, implementing the ReMiND intervention in Uttar Pradesh averted 312 maternal and149,468 neonatal deaths. This implies that the ReMiND program led to a reduction of 0.2% maternal and
5.3% neonatal deaths. From a health system perspective, ReMiND incurs an incremental cost of USD $205
per DALY averted and USD $5,865 per death averted. The cost-effectiveness of the program is higher thanthat of vaccines for cholera, typhoid, and haemophilus influenza type ‘b’ at current market rates in India.
A study by USAID’s Health Finance and Governance Project evaluated the cost-effectiveness of a
CommCare-based Integrated Management of Childhood Illness (IMCI) application implemented by D-tree
International [Kukla, 2015]. The study evaluated the ratio of the additional cost per FLW to the improved
effectiveness per FLW of a pilot project involving 50 FLWs. The study concluded that “compared withthe existing paper-based system, the mobile tool costs an additional $10.43 per annum for an HSA to
improve his/her diagnostic and treatment accuracy by 1 percent.” Given the low marginal cost of increasing
the number of FLWs using the tool, and assumed consistency in relative effect per FLW, the authors
also estimated a cost-effectiveness of $1.07 per 5,000 FLWs, concluding that a much higher return on
investment is achieved when taking mobile tools to scale.

15

THEME 5 : CHALLENGES IN IMPLEMENTING DIGITAL HEALTH PROGRAMS
Several studies evaluate the challenges of implementing CommCare. Challenges cited include technicalissues such as broken phones and programmatic issues that limit CommCare’s impact.

There are various studies that describe challenges with implementing CommCare. Several studies citedtechnical issues, such as broken phones that remained unfixed, lack of convenient power source to charge
devices, and inconsistent connectivity which limited data sharing and synchronization among FLW teams[Chaiyachati, 2013] [Borkum, 2015] [Adler, 2020] [Gopalakrishnan, 2020].
Other studies also identified programmatic issues that limited CommCare’s impact. One challenge withlarge-scale digital health implementations is that adopting organizations may have limited IT capacity toadequately maintain technology [DeRenzi, 2012] [Chaiyachati, 2013]. Low adoption rates among users ora decrease in usage over time—often referred to as the ‘novelty effect’ of new technology—has also been
observed in CommCare projects [Borkum, 2015] [Chaiyachati, 2013] [DeRenzi, 2012] [DeRenzi, 2016][Segal, 2015]. Chaiyachati attributed low adoption rates to a disconnect between the application design
and the actual value the application provided to FLWs in their everyday work. FLWs reported that they did
not find the primary functionality of the application useful, although using the mobile phones for SMS andphone calls did improve communication between FLWs and their clients and coordination among FLWs
[Chaiyachati, 2013].
It is also worth noting that what may be perceived as beneficial to FLWs through the use of digital health tools
may not always be perceived the same way by or be most impactful for the client if careful considerations
are not made. For example, the study examining FLW and client perspectives on a digital health intervention
in Bihar and Madhya Pradesh, India found that although the majority of interviewed FLWs and clients saw
the tool as a facilitator to engagement and communication, some clients felt that the use of certain featuresintended to improve healthcare delivery, such as video, ultimately limited interpersonal communicationand led to rushed interactions, which may discourage clients from being comfortable enough to share
health history or ask questions. Additionally, there may be socio-cultural factors that could impact service
delivery when implementing digital health interventions (such as customs related to leaving the home, or
the practice of women returning to their natal homes during pregnancy and the postpartum period); these
may be seen as barriers that can limit uptake or FLW engagement with intended clients if not consideredahead of time [Gopalakrishnan, 2020].

16

The Mathematica study also reported that CommCare’s supervisory functionality had several challenges(though it has since been improved), and that CARE had to provide extensive training sessions (163-hour sessions over the course of 8 weeks) to achieve an adequate level of impact [Borkum, 2015].
Other programmatic issues highlighted in CommCare projects include delayed top-up payments to FLWs
[Borkum, 2015], limited stock of medical supplies (in this case, Tetanus Toxoid injections) [McNabb, 2015],
concerns about long-term sustainability due to the cost of devices or desired features (i.e. centralized SMS
delivery) [Adler, 2020], and the risk of inaccuracy in self-reported data [DeRenzi, 2012].
Additionally, a study of mWellcare did not find positive results. The mWellcare system leveraged CommCare
as a clinical decision support system and an e-health record storage system to drive integrated managementof five chronic conditions. The trial found no statistical difference in this setting vs. primary care setting,which had only enhanced usual care for patients with diabetes mellitus and hypertension [Prabhakaran,
2019]. While no improvement was found across secondary outcomes of fasting blood glucose, depression
score, total cholesterol, predicted 10-year risk of cardiovascular disease, or tobacco and alcohol use;
however, the mWellcare arm had higher self-reported adherence to medications. The null result highlights
the potential value of leveraging nonphysician providers and improving access to needed medications
which can be used while drafting public health interventions by policy makers.

17

CONCLUSION

The collective findings from these studies are encouraging. They demonstrate the potential for organizations
to use CommCare in an effort to strengthen frontline programs through improved client health outcomes
and behaviors, FLW performance, quality of care, and program efficiency. We are particularly encouragedby the upward trend in literature evaluating CommCare’s impact on client health outcomes and behaviors.
The evidence demonstrates that CommCare can strengthen frontline programs, but it is the FLWs
themselves who deliver the critical services to underserved populations. Organizations must continually
train and support their FLWs to leverage the vast potential of integrating mobile technology into service
delivery systems, both for health and non-health sectors.
As the CommCare Evidence Base expands, we hope to see more rigorous studies of this kind with a focus
on CommCare’s impact on client health outcomes. As the most direct measure of public health impact,these studies continue to play a vital role in evaluating and improving CommCare as a tool for FLWs.
If you have questions or would like to learn more about CommCare or the CommCare Evidence Base,please email info@dimagi.com

18

References[Adler, 2020] Adler, A.J., Laar, A.K., Kotoh, A.M. et al. Barriers and facilitators to the implementation of a community-based hypertension improvement
project in Ghana: a qualitative study of ComHIP. BMC Health Serv Res 20, 67 (2020). https://doi.org/10.1186/s12913-019-4774-x[Agarwal, 2015] Agarwal, S., et al. Evidence on Feasibility and Effective Use of mHealth Strategies by Frontline Health Workers in Developing
Countries: Systematic Review. Tropical Medicine and International Health, Vol 20 No 8, doi: 10.1111. 2015.
[Amoah, 2016] Amoah, Benjamin et al. Boosting Antenatal Care Attendance and Number of Hospital Deliveries among Pregnant Women in Rural
Communities: A Community Initiative in Ghana Based on Mobile Phones Applications and Portable Ultrasound Scans.” BMC Pregnancy andChildbirth 16 (2016): 141.
[Ball, 2013] Ball, M., Rahman, Z., Champanis, M., Rivett, U., Khush, R. Mobile Data Tools for Improving Information Flow in WASH: Lessons fromThree Field Pilots. 2013. IRC Symposium 2013: Monitoring Sustainable WASH Service Delivery. Addis Ababa, Ethiopia.
[Battle, 2015] Battle JD, Farrow L, Tibaijuka J, Mitchell M. mHealth for Safer Deliveries: A mixed methods evaluation of the effect of an integrated
mobile health intervention on maternal care utilization. Healthc (Amst) 2015 Dec;3(4):180-184.
[Bhavsar, 2014] Bhavsar, M., Grijalva, K. From paper to Mobile: Design Considerations for Field Level Worker Programs. Mobile Communication
For Development Conference. April 2014. Dakar, Senegal.
[Birnbaum, 2012] Birnbaum B.E., DeRenzi B., Flaxman A.D., & Lesh N. Automated quality control for mobile data collection. ACM DEV, 1-1, 2012
[Borkum, 2015] Borkum, E., et al. Evaluation of the Information and Communication Technology (ICT) Continuum of Care Services (CCS)Intervention in Bihar. Mathematica Policy Research , 2015.
[Braun, 2016] Braun, R. et al., An evaluation of a family planning mobile job aid for community health workers in Tanzania, Contraception. 2016
Jul;94(1):27-33.
[Chaiyachati, 2013] Chaiyachati KH, Loveday M, Lorenz S, Lesh N, Larkan L-M, et al. (2013) A Pilot Study of an mHealth Application for Healthcare
Workers: Poor Uptake Despite High Reported Acceptability at a Rural South African Community-Based MDR-TB Treatment Program. PLoS ONE
8(5): e64662.
[Chanani 2015] Chanani, S., Jayaraman, A., Wacksman, J. 101: Use of Mobile Technology for Improving Screening Accuracy of Acute Malnutrition
in a Community-Based Management of Acute Malnutrition Program in Mumbai Informal Settlements. BMJ Open, vol. 5, doi:10.1136, 2015.
[Dell, 2014] Dell, N., Francis, I., Sheppard, H. and Borriello, G. 2014. Field Evaluation of a Camera-Based Mobile Health System in Low-Resource
Settings. International Conference on Human-Computer Interaction with Mobile Devices and Services.
[DeRenzi, 2008] DeRenz, B. et al. E-IMCI: Improving pediatric health care in low-income countries. ACM Conference on Computer-Human
Interaction (CHI). 2008.
[DeRenzi, 2016] DeRenzi, B. et al. Closing the Feedback Loop: A 12-month Evaluation of ASTA, a Self-Tracking Application for ASHAs. ICT4D
Conference ‘16. Ann Arbor, Michigan, USA[Gopalakrishnan, 2020] Gopalakrishnan L, Buback L, Fernald L, Walker D, Diamond-Smith N, et al. (2020) Using mHealth to improve health caredelivery in India: A qualitative examination of the perspectives of community health workers and beneficiaries. PLOS ONE 15(1): e0227451. https://doi.org/10.1371/journal.pone.0227451[Hackett, 2018] Hackett K, Lafleur C, Nyella P, Ginsburg O, Lou W, Sellen D (2018) Impact of smartphone assisted prenatal home visits on
women’s use of facility delivery: Results from a cluster-randomized trial in rural Tanzania. PLoS ONE 13(6): e0199400.
[Htet, 2019] Htet MK, Fahmida U, Do TT, Dibley MJ, Ferguson E. The Use of Tablet-Based Multiple-Pass 24-Hour Dietary Recall Application(MP24Diet) to Collect Dietary Intake of Children under Two Years Old in the Prospective Cohort Study in Indonesia. Nutrients. 2019; 11(12):2889.
doi:10.3390/nu11122889[Hoy, 2015] Hoy, R., Rubin, J., Mitchell, M. Using mobile technology to address the three delays’ to reduce maternal mortality in Zanzibar.
International Journal of Use-Driven Healthcare, doi: 10.4018. 2014.
[Khalid, 2015] Khalid, M., et al. Designing a Mobile Partograph for Real-Time Decision Support for Safer Conference on Social Implications ofComputers in Developing Countries, Negombo, Sri Lanka, 201519

[Kukla, 2015] Kukla, Matt, Pamela Riley, and Sarah Dominis 2015. Evaluating the Cost-Effectiveness of a Mobile Decision Support Tool in Malawi.
Bethesda, MD: Health Finance and Governance Project, Abt Associates Inc.
[Kumar, 2012] Kumar, A. and Bora, G. Msakhi: Putting Technology Into the Hands of Community Health Workers. In: 13th World Congress on
Public Health; April 26, 2012; Addis Ababa, Ethiopia.
[Martínez-Fernández, 2015] Martínez-Fernández, A., Lobos-Medina, I., Días-Molina, C.A., Chen-Cruz, M.F., Prieto-Egido, I. TulaSalud: An
m-health System for Material and Infant Mortality Reduction in Guatemala. Journal of Telemedicine and Telecare, vol. 21 no. 5 283-291, 2015.
[McNabb, 2015] McNabb, M., Chukwu, E., Ojo, O., Shekhar, N., Salami, H., Jega, F. Assessments of the quality of antenatal care services provided
by health workers using a mobile phone decision support application in northern Nigeria: a pre/post-intervention study. Pathfinder International.
2014.
[Medhi, 2012] Medhi I., Jain M., Tewari A., Bhavsar M., Matheke-Fischer M., & Cutrell E. Combating rural child malnutrition through inexpensive
mobile phones. Nordic Conference on Human-Computer Interaction, 2012
[Mitchell, 2012] Mitchell, M. et al. Electronic decision protocols for ART patient triaging to expand access to HIV treatment in South Africa: a crosssectional study for development and validation. International Journal of Medical Informatics, 81(3), doi: 10.1016. 2012.
[Mitchell, 2013] Mitchell, M. et al. Using electronic technology to improve clinical care – results from a before-after cluster trial to evaluate
assessment and classification of sick children according to the Integrated Management of Childhood Illness (IMCI) protocol in Tanzania. BMC
Medical Informatics and Decision Making 2013, 13(95), doi: 10.1186.
[Mohamed, 2014] Mohamed, N., Lesh, N., Conte, F., & Findlater L. Using ICT4CHW To Influence Decision Makers. Mobile Communication For
Development Conference. April 2014. Dakar, Senegal.
[Nascimento, 2014] Nascimento, N., Cannon, M., Perales, N., Chariyeva, Z. Assessment of an mHealth initiative to improve patient retention.
USAID, PEPFAR, Measure Evaluation. August 2014.
[Otu, 2016] Otu A, Ebenso B, Okuzu O, Osifo-Dawodu E. Using a mHealth tutorial application to change knowledge and attitude of frontline healthworkers to Ebola virus disease in Nigeria: a before-and-after study. Human Resources for Health. 2016;14:5. doi:10.1186/s12960-016-0100-4.
[Palazuelos, 2013] Palazuelos D., Diallo A., Palazuelos L., Carlile N., Payne J., & Franke M. User Perception of an mHealth Medicine Dosing Tool
for Community Health Workers. JMIR MHealth and UHealth. 2013; 1(1):e2.
[Prabhakaran, 2019] Prabhakaran et. al, Effectiveness of an mHealth-Based Electronic Decision Support System for Integrated Management ofChronic Conditions in Primary Care, Circulation, Vol 139 No 3 , 2019
[Prinja, 2018] Prinja S, et al. Cost effectiveness of mHealth intervention by community health workers for reducing maternal and newborn mortality
in rural Uttar Pradesh, India. Cost Effectiveness and Resource Allocation : 25 Jun 2018.
[Schwartz, 2013] Schwartz, A., Bhavsar, M., Cutrell, E., Donner, J., Densmore, M. Balancing burden and benefit:
non-prescribed use of employer-issued mobiles devices. Proceedings of the Sixth International Conference on Information and Communication
Technologies and Development. 2013. DOI: 10.1145/2517899.2517911[Schweers, 2015] Schweers, J. et al. mLabour: Design and Evaluation of a Mobile Partograph and Labor Ward Management Application.
Humanitarian Technology: Science, Systems and Global Impact 2015, HumTech2015.
[Segal, 2015] Segal, J.B., Arevalo, J.B., Franke, M.F., Palazuelos, D. Reducing dosing errors and increasing clinical efficiency in Guatemala: firstreport of a novel mHealth medication dosing app in a developing country. BMJ Innovations, 0:1–6, doi:10.1136, 2015.
[Surka, 2014] Surka S, Edirippulige S, Steyn K, Gaziano T, Puoane T, Levitt N. Evaluating the use of mobile phone technology to enhancecardiovascular disease screening by community health workers. Int J Med Inform.2014;83(9):648–654. doi: 10.1016/j.ijmedinf.2014.06.008.
[van Dam, 2015] Van Dam J, Tadmor B, Spector J, et al. An open-access mobile compatible electronic patient register for rheumatic heart disease(“eRegister”) based on the World Heart Federation’s framework for patient registers. Cardiovascular Journal of Africa. 2015;26(6):227-233.
doi:10.5830/CVJA-2015-058.
[Walton, 2020] Walton M, Arsyad DS, Alimuddin S, et al. Implementing a One Health village volunteer programme in West Sulawesi, Indonesia: A
pilot study. Global Public Health. 2020:1-16. doi:10.1080/17441692.2020.1836247[World Vision, 2013] Mobile Technology Strengthens Behavior Change Communication and Referrals by Community Health Workers for Maternal,
Newborn, and Child Health in Rural Afghanistan. Operations Research Brief. USAIDand World Vision.

20