Kilkari Impact Evaluation
Condition: Breast Feeding; Family Planning; Essential Newborn Care; Training of Frontline Health Workers
Study type: Interventional

Summary: 
      British Broadcasting Corporation (BBC) Media Action is presently implementing two large scale
      mobile health (mHealth) initiatives in India: Kilkari and Mobile Academy. Kilkari is an
      outbound service that delivers weekly, time-appropriate audio messages about pregnancy,
      childbirth, and childcare directly to families on their mobile phones, starting from the
      second trimester of pregnancy until the child is one-year-old. Mobile Academy (MA) is an
      Interactive Voice Response (IVR) in-service audio training course for Accredited social
      health activists (ASHAs) in India designed to refresh their knowledge of life-saving
      preventative health behaviors and improve their interpersonal communications skills. Both
      programs were initiated in Bihar in 2012, and have been scaled widely in a number of states
      with support from Ministry of Health and Family Welfare (MOHFW) at the national level,
      National Health Missions (NHM) at the state level, and an alliance of donors (Gates
      Foundation, USAID, Barr Foundation, and UK Department for International Development (UKAid)).

      The Johns Hopkins School of Public Health, Oxford Policy Management, and University of Cape
      Town are supporting BBC Media Action and the Ministry of Health and Family Welfare (MOHFW)
      are conducting an external evaluation of both Kilkari and Mobile Academy (MA). The evaluation
      spans through April 2020.
    

Detailed description: 
      The Johns Hopkins Global mHealth Initiative, Oxford Policy Management, and University of Cape
      Town proposes supporting BBC Media Action and the Ministry of Health and Family Welfare
      (MOHFW) to conduct an external evaluation of Mobile Academy (MA) and Kilkari. In the below
      text, we outline 6 proposed activities which comprise the evaluation, which are planned from
      mid 2018 to April 2020.

      These studies will help inform program modification and scale-up plans, and will help better
      understand the effects of maternal mobile messaging in India and globally.

        1. Secondary analysis of cross-site trends in key performance indicators (Duration:
           throughout study):

           Mobile Academy (MA): We will use system generated data to estimate the proportion of
           ASHAs who have completed MA training across 12 states. We will additionally aim to
           understand linkages between ASHA's exposure to the MA training content and performance
           on MA knowledge assessments.

           Kilkari: We estimate that Kilkari users will have differing levels of exposure to the
           program. For example, some women may not be present when the Kilkari calls come and thus
           miss out on key content. Analyses of system generated data will aim to measure exposure
           to Kilkari messages by content area (e.g. nutrition, family planning, etc) across 12
           states where implementation has occurred to date. As part of this analysis we'll seek to
           understand reasons why calls are not received, including phones being out of network,
           changes in SIM cards, etc.

        2. Summative evaluation of Kilkari (Period: April 2018 to December 2019) Objective: To
           determine the effectiveness of mobile maternal health information messages on key health
           outcomes in 4 districts of Madhya Pradesh (MP).

           This objective will be carried out through an individually randomized controlled trial
           initiated in early 2018. We propose conducting household surveys in all 4 study
           districts among the cohort of women identified during conception, starting in March
           2018. Women identified within the 5-7 month of their pregnancy, who have access to a
           mobile phone, and consent to participate will be interviewed as part of baseline survey
           and recruited for participation into Kilkari by 8 months of conception (this is intended
           to mirror timings observed in Kilkari outreach in other states). Among individuals not
           randomized to receive messages, we would propose delaying implementation for Kilkari for
           a 12-month window to allow for evaluation activities to take place and additionally
           glean information which is anticipated to improve the efficiency of implementation /
           rollout in other areas.

        3. ASHA Survey Objective: Identify factors underpinning differences Reproductive Maternal
           Newborn and Child Health (RMNCH) knowledge and motivation among frontline health
           workers, including exposure to the Mobile Academy training program.

           As part of the baseline surveys described under activity 2 in MP, we will conduct face
           to face interviews with all ASHAs (n=1,200) across the four study districts in MP. A
           sub-sample will also be contacted for a follow-on phone survey a week later, to pilot
           modules and validate a phone survey tool on knowledge and motivation, which can then be
           applied to other ASHAs to measure key outcomes as part of MA deployments in other
           states, including Rajasthan.

        4. Post-natal care phone survey Objective: Measure respectful maternity care (RMC) during
           childbirth, essential newborn care, and infant feeding practices within 0-3 months
           postpartum As part of baseline survey activities in MP, we propose additionally
           identifying women in 1-2 districts who are 0 to 3 months postpartum (n=880). Face to
           face interviews with postpartum women will provide a forum to measure key infant feeding
           and essential newborn care outcomes. Phone surveys, a week later, with a subset will
           help develop a validated phone survey tool for their measurement amongst individuals
           enrolled into Kilkari.

        5. Qualitative interviews of both Kilkari and MA in MP and Rajasthan

           Objectives:

           Mobile Academy

           • Understand stakeholder (ASHA, ASHA supervisor and beneficiary) perceptions of MA,
           including barriers / facilitators underpinning uptake and completion of MA training and
           enacting its content, and how it affects interpersonal communication between
           stakeholders.

           Kilkari

             -  Understand stakeholder (ASHA, ASHA supervisor and beneficiary) perceptions of
                Kilkari, including different message content areas, barriers / facilitators to the
                adoption of Kilkari's recommended RMNCH practices and careseeking behaviors

             -  Identify barriers / facilitators for pregnancy and birth registration

           The qualitative research component will complement the summative evaluation and
           secondary data analysis components. Qualitative research will include in-depth
           interviews and focus group discussions in both MP and Rajasthan. Differentials in uptake
           of MA across these settings provides a unique opportunity to understand the
           characteristics of ASHAs and the larger health systems environment which contribute to
           MA completion. Among Kilkari beneficiaries, qualitative research can help better
           understand user perception of content, the extent and mechanism of impact of the
           messages on health-related knowledge, behavior and relationships.

        6. Economic evaluation of Kilkari Objective: To determine the incremental
           cost-effectiveness of Kilkari as compared to a null scenario of no messages To estimate
           the value for money of Kilkari, we will estimate the costs of program to all key
           stakeholders involved-including BBC Media Action, MOHFW, and end-users. To estimate the
           effects of the program on health outcomes, we will draw from household survey data at
           baseline and endline amongst individuals exposed and not exposed to messages. Data on
           changes in careseeking for antenatal care as well the adoption of practices around
           breastfeeding and family planning will be inputted into the Lives Saved Tool to generate
           a measure of lives saved associated with program activities. Lives saved will be used to
           yield an estimate of the incremental Disability-Adjusted Life Years (DALY) averted and
           ultimately a cost per DALY averted.
    

Intervention name(s): Kilkari
Intervention type(s): Other
Intervention description(s): Kilkari is an outbound service that delivers weekly, time-appropriate audio messages about pregnancy, childbirth, and childcare directly to families on their mobile phones, starting from the second trimester of pregnancy until the child is one-year-old.
Allocation: Randomized
Intervention model: Parallel Assignment
Masking: Single (Participant)
Primary purpose: Health Services Research
Keywords: mobile health, Pregnant and postpartum women, Frontline health workers, India