A proposed model to conduct process and outcome evaluations and implementation research of child health programs in Africa using integrated community case management as an example

Evaluation and implementation research of the delivery of maternal, newborn and child health interventions that focus on impact, specifically those assessing changes in morbidity or mortality, have been deemed critical to determine the effectiveness of programs being implemented [1]. However, unlike an efficacy trial, in reality evaluators and researchers cannot control programs implemented by governments, international or bilateral agencies, and private voluntary organizations, nor can they control contextual factors that affect the program [2,3]. In sub–Saharan African countries and many other developing countries, multiple organizations support different programs and may work in different areas of the same country, implement their programs differently and may implement in areas that were designated to be the control arm of an evaluation without the evaluator being aware. In addition, achieving full implementation at scale can take much longer than anticipated and other contextual factors not under the control of the evaluator, such as national stock out of medications, can negatively impact the programs. Without documenting these factors and other details of implementation, the impact studies are limited in their ability to explain why programs do or do not produce expected impact. Therefore measuring impact should not be the only focus of evaluations or implementation research of maternal, newborn or child health programs that are delivering proven interventions, but rather it should be supplemented with measures of process and outcomes coupled with contextual information to better understand if and how (or if not and why not) programs are providing the interventions to those in need. Process evaluations measure the internal dynamics of implementing organizations, their policy instruments, their service delivery mechanisms, and their management practices [4]. Specifically, they determine what is done by the program, and for whom these services are provided [5]. Outcome evaluations measure the likely or achieved short– and medium–term effects of an intervention’s outputs [4] such as behavior change (eg, what proportion of those who might need the service(s) sought care) and coverage (eg, what proportion of those who might need the service received them). Contextual factors can include factors that may impact implementation such as the functioning of the overall health system or factors that may affect the impact of the intervention such as the socioeconomic factors and status or underlying health status of the population [3]. The problem with evaluations that focus on impact, without examining process, outcomes and context can be highlighted with one particular child health program, integrated community case management (iCCM). Around 2008–9, funding levels to support iCCM expansion in sub–Saharan Africa increased substantially. However, a recent review of published evaluations in Africa concluded there is no evidence of mortality impact of community–based pneumonia treatment [6]. This conclusion does not make sense given that earlier impact evaluations of iCCM, mostly in Asia, have shown that using community health workers (CHWs) to deliver treatments can reduce pneumonia specific and overall mortality in children and the fact that that provision of antibiotics for pneumonia is effective at reducing mortality in children [7–9]. In actuality, many of the reviewed studies in sub–Saharan Africa focused on specific aspects of the program, and not the entire process and outcome. Most of the included studies concentrated on measuring CHWs adherence to established guidelines [10]. Complete evaluations that included process, outcomes and context were missing. Several researchers have proposed evaluation methods that may be able to take into account process, outcome and context. Realist evaluations, which is a form of theory driven evaluation that is context–specific, represent one example [11]. In this type of evaluation, interventions work (or not) because people make particular decisions in response to what is provided by the intervention (or not) in a particular context. Their response to the resources, or opportunities provided by the intervention is what causes the outcomes. Measuring contextual factors matters because these factors influence their response; measuring process is important to understand how and why decisions were made. Another more recent suggestion for evaluations of large–scale programs and initiatives in middle– and low–income countries is a national platform approach [12]. This evaluation approach uses a geographic unit (usually district) as the unit of analysis. Relevant information from existing databases would be integrated in a continuous manner into one data repository. New information about program implementation by different agencies (government, bilaterals, multilaterals, non–governmental organizations [NGOs]) would also be included in the database. It would also include contextual information. Although the focus would be on using existing data, additional data on program management and data quality may need to be collected and added. Comparisons of different geographic units could be done either based on a score or dose–response analyses of program implementation strength and coverage. For both these types of evaluations, additional data to assess process, outcome and contextual factors need to be collected. With the increase in funding for iCCM there has been a great demand from donors for evaluations and implementation research largely focused on measuring the impact, specifically mortality, of these programs. Despite the pressure to focus on mortality impact many evaluators and implementers recognized the importance of documenting and measuring implementation, and therefore examined process, outcomes and context, but used different methods and data elements. Our objective was to develop and apply a newly devised set of criteria to review the study design and scope of collection of process, outcomes and contextual data for evaluations and implementation research of iCCM) in Sub–Saharan African and to propose an evaluation study design(s) based on gaps identified. We examined 24 evaluations and/or implementation research studies of iCCM that were completed in the last 5 years (2008–2013).