Learning by Design : Lessons From Designing a Baseline Study in the NAMWASH Small Towns Programme, Mozambique

This paper examines the design and application of a baseline study for a comprehensive water, sanitation, and hygiene (WASH) intervention in Mozambique. The study was developed to investigate the relationships among key parameters of interest both for comparison to post-implementation data and to contribute to planning the WASH intervention itself. We use this study to discuss key issues surrounding baseline studies. This includes providing guidelines for designing a WASH baseline survey, determining an appropriate sample size, and highlighting key considerations in analysing the survey data, such as incorporating the study design in statistical analyses, post-stratifying and utilising geospatial data. We also show how statistical analyses from a baseline survey can be used to inform subsequent surveys. For example, results from this study suggest that in future WASH studies, selfreporting by households should be supplemented by observational or population data to remove or quantify reporting bias, and care must be taken to reduce respondent fatigue. Introduction This article examines the design of a baseline study, including monitoring of non-target communities, for a comprehensive water, sanitation and hygiene (WASH) intervention program in Mozambique. It explains how the study itself was designed and implemented and discusses some of the potential improvements identified following data collection. It is not intended to report the results of this study but instead provides practical insights into the design and evaluation of future WASH baseline studies. This baseline study was conducted in Mozambique as part of the Nampula Province Water, Sanitation and Hygiene (NAMWASH) Programme. The Programme is aimed at accelerating the achievement of the WASH Millennium Development Goals (MDGs), particularly Goal 7c: “Halve, by 2015, the proportion of the population without sustainable access to safe drinking water and basic sanitation” (United Nations, 2013) in Nampula Province. The project aims to significantly increase access to safe drinking water and sanitation, the adoption of appropriate hygiene practices, improved water safety management and WASH facilities in schools between 2012 and 2016. The program also aims to strengthen the technical and managerial capacity of government, private sector and civil society in small towns and create a WASH intervention model for small towns that can be further applied throughout Mozambique. A baseline study was required to aid in the design of the WASH interventions as well as to provide information on the efficacy of the program post-intervention. It was also intended that the design and assessment of this baseline study would contribute to methodological knowledge on how to undertake impact evaluation of WASH activities in small towns across the globe. The baseline study consisted of seven towns, where five target towns are expected to receive WASH interventions as part of the NAMWASH program, and two non-target towns are not expected to receive WASH interventions within 3-5 years following the baseline year. Several key players were involved: • UNICEF Mozambique, who designed the baseline study as the first stage of a multi-year longitudinal study of WASH interventions in towns in Nampula province, • A national consultant who carried out data collection, • Nampula Provincial Directorate of Health, who spearheaded a multi-sectorial team collecting water quality data, and • Murdoch University, who carried out statistical analysis of data from the baseline study, including post-survey quality control. In total, the baseline study took five weeks to complete (early September to early October of 2012), and the impact study comprised approximately 2% of the total budget. Baseline studies and WASH interventions After several decades of water, sanitation and hygiene (WASH) interventions in developing communities, it has become apparent that the best results are achieved when technical and educational programs relating to each of the WASH subthemes are integrated (e.g. Aziz et al., 1990). It has also become apparent that progress towards individual MDGs is closely linked to, and may assist in the attainment of, other MDGs. As an example, several MDGs, including those related to WASH, have been shown to significantly contribute to the goal of reducing child mortality (Gakidou et al., 2007). As such, integrated WASH interventions are being implemented across the globe. The collection and analysis of community-level data can assist in the development of appropriate integrated WASH interventions (Huber and Mosler, 2013), and an appropriate baseline study can provide such useful data. A well-designed baseline study allows for the planning of an effective WASH intervention program by identifying areas of concern as well as allowing for the examination of relationships among various WASH indicators. For example, the World Bank’s Water and Sanitation Program has collected baseline data to inform designs for more effective hand washing interventions in various communities across the globe (Perez et al., 2011). It is inherently difficult to design a baseline study for a multi-arm intervention that will include not only aspects of the various WASH subthemes (water, sanitation and hygiene) but also range in methodology from infrastructural improvements to education programs. Such projects require investigators to monitor physical parameters (such as water quality), behavioural changes and attitudes towards WASH. It is thus important to design a baseline study that incorporates methods which will assess all of the indicators that will both contribute to WASH intervention design and determine its efficacy post-implementation. With sound statistical design and analysis, such baseline studies can uncover valuable relationships which can inform design of both the current and future WASH interventions. Guidelines for good WASH baseline studies are included in Table 1, and suggestions for good WASH follow up studies in Table 2. Table 1: Guidelines for good WASH baseline studies. Include non-target communities Despite Blum and Feachem’s seminal paper on the weaknesses of inferring health outcomes of WASH interventions from inadequate baseline studies (Blum and Feachem, 1983), many WASH studies do not include baseline sampling of communities not expected to receive interventions (non-target communities). However, monitoring non-target communities is important for post-implementation analysis. Their inclusion in both the baseline study and post-implementation monitoring allows for analysis of the efficacy of WASH interventions by measuring changes in WASH indicators which occurred without WASH interventions. (for a Bangladeshi example of a thorough WASH study including monitoring of nontarget communities see Aziz et al., 1990). Conduct pre-, postand interim monitoring during similar timeframes or seasons of the year Many WASH baseline studies suffer the inherent flaw of temporal effects. Where studies cannot be performed over a long period (as is almost always the case), reporting of WASH factors may only represent their value at that point in time. For example, the volume of water collected may differ significantly between wet and dry seasons, and the incidence of diseases, such as the many which cause diarrhea, may vary throughout the year (Alexander et al., 2013). Although it is likely impossible to increase the period of data collection, especially when conducting household surveys where data is collected at only a certain point in time, seasonal effects can be accounted for to some extent by keeping sound records of climate in the weeks or months prior to, and performing post-intervention monitoring during the same time of the year as, the baseline study. More useful would be to collect data on seasonally affected WASH variables in both the target and non-target clusters regularly throughout the intervention period to determine seasonal patterns and the overall change in the variables. Such patterns of variables may not be obvious from collecting data from single points in time preand post-intervention. Pay careful attention to the selection of impact variables Variables need to be chosen that can realistically be measured in the development context. Those which cannot be measured directly during monitoring and instead rely on government records may prove difficult or impossible to gain access to and may not be of the quality required to perform meaningful analyses. Additionally, some variables may be more accurate than others. For instance, using reported handwashing practices as a measure of hygiene may be a far less reliable measure than observed handwashing practices. Provide relevant information for sample size calculations for follow-up studies In determining an appropriate sample size for the baseline study in this work, the percentage of households using improved water supplies was used as the key indicator. Post-intervention studies will almost certainly be interested in a variety of other key indicators (e.g. prevalence of diarrhea, water quality at the source or household, percentage of people using soap when washing hands), and the baseline study can provide current measures of those key indicators that will inform appropriate sample sizes for the next study. Table 2: Suggestions for good WASH follow-up studies. Where possible, using a randomized controlled trial When designed appropriately, these provide the best assessment of the effect of an intervention by essentially eliminating confounding. In the context of a WASH study, a truly randomized controlled trial at the household level would be difficult to implement across all sub-themes, especially water supply, where introduced improved water sources may in many cases be public standposts, not private taps. Additionally, Cairncross and Valdmanis (2006) argue that such a design at the household level poses potential ethical and poli