IJERD – International Journal of Environmental and Rural Development (2013) 4-2

Research article
erd

An Evaluation of Evaluation Systems for Rural Water
Supply and Sanitation Systems
TUNG NGUYEN NHU*
Vietnam National University, Ho Chi Minh City, Vietnam
Email: nntung@hcmiu.edu.vn
Received 14 December 2012

Accepted 10 June 2013(*Corresponding Author)

Abstract An evaluation of evaluation systems applied to water supply and sanitation
systems in rural areas would contribute to improving the validity and quality of theevaluation results. An evaluation system is supposed to be based on latest indicators in thefield, theory-based evaluation principles, in consideration of threats to the validity of theevaluation. A case study of the Mekong Delta Rural Water Supply and Sanitation Project(AUSAID Project) is used for a review of its evaluation system. This project was
implemented in five provinces in Vietnam for the period from 2000 to 2007. Informationon this evaluation is collected from regular project monitoring and evaluation reports andthe Activity Completion Report. The author’s work as an evaluation officer for the project
improves the understanding of the project’s evaluation approaches. The project evaluation
reports serve the purpose of measuring the effectiveness and sustainability of the project’s
activities. Several potential threats to the validity of the evaluation include some missing
assumptions and intermediate outcome for the project’s impacts. Notwithstanding theabove threats, this evaluation system of rural water and sanitation was the first in this field
in Vietnam, and its indicators satisfy the evaluation standards. This evaluation ofevaluation, based on theory-based evaluation principles, for a specific case of rural watersupply project in Mekong Delta has never been conducted before.
Keywords threats to validity, effectiveness, evaluation systems, water supply,
sustainability, participatory

INTRODUCTION
Because of its importance, an evaluation report must be a useful document for learning. However,
Busby (1999) has stated that evaluation reports are not used effectively because their contents are
too shallow: they fail to explicitly identify the true causes of problems, lack of objective outcomeof data and misinterpret data. The validity of an evaluation report depends on the quality of its
evaluation system design and implementation. Therefore, it is necessary to determine whether an
evaluation system is free of technical errors, misinterpretation, and bias. This type of auditing helps
to improve an ongoing evaluation process. A review of post-project evaluation also audits thecompleteness and soundness of final evaluation reports to provide information to future project
managers. Because of this importance, international donors conduct reviews of evaluations. For
example, AUSAID audited 162 evaluation reports of its funded activities from July 2006 to June2010 and found that approximately one-quarter of these evaluation reports were of “insufficientquality to be published”, and that only 11% of them were of excellent quality (Bazeley, 2011).
An evaluation of an evaluation is defined as a way “to aggregate findings from a series of
evaluations. It is also used to denote the evaluation of an evaluation to judge its quality and/or
assess the performance of the evaluators” (glossary for M&E terms, OECD-DAC 2002). Other
names for an evaluation of an evaluation are a secondary evaluation, a meta-evaluation and anevaluation audit. This evaluation category includes professional critiques of evaluation reports,
reanalysis of data and external evaluations of internal audits. In this paper, the term “metaevaluation” is used to refer to the evaluation of evaluations.

© ISERD
167

IJERD – International Journal of Environmental and Rural Development (2013) 4-2

OBJECTIVE
The purpose of this study is to compile meta-evaluation criteria for a Rural Water Supply andSanitation (RWSS) evaluation based on an intensive literature review and to assess the evaluation
system used by an AUSAID-funded project, the Mekong (Cuu Long) Delta Rural Water Supply
and Sanitation (CLDRWSS) Project. Specifically, the research questions are as follows: 1) Whatshould be reviewed in an evaluation of a RWSS evaluation system? 2) What are the evaluation
methodologies used by the CLRWSS Project? 3) What are some threats to the validity of theevaluation results in the CLRWSS Project?
This research will contribute by providing a proposed framework of meta-evaluation to which
future evaluation efforts in the relevant sector may refer. Evaluators will be able to determine
which aspects of their evaluation content will be checked to allow their auditing to be more
complete, accurate, accountable, and usable.
The limitation of this paper is that it ignores considerations of some meta-evaluation
indicators, including cost-benefit or cost-effectiveness issues (e.g., was the evaluation conducted ata reasonable cost?) and the extent of learning from evaluation results (what did people learn andhow well did people learn from these lessons?). It is important that future meta-evaluations include
these contents.
METHODOLOGY
For the first research question, literature review is used to compile a set of relevant meta-evaluation
indicators to answer the first question. Substantial research has been conducted to include relevant
meta-evaluation indicators in the rural water and sanitation sector. The sources of this information
are mainly evaluation books, journal articles and websites. I also refer to the guidelines publishedby the World Bank, OECD, AUSAID and UNICEF for evaluation standards.
To answer the second and third research questions, desk research is used to retrieve the
CLDRWSS Project’s Monitoring and Evaluation (M&E) reports and the M&E Summary Reportfor the Activity Completion Report to provide evidence of the evaluation system. The above
evaluation reports are scrutinised through the lens of theory-based evaluation model to understandhow and why the project activities would lead to outcomes (Weiss, 1995; Fitz Gibbon et al., 1996;
Rogers et al., 2000). Theory-based evaluation is more and more popularly used in designingevaluation of community-based interventions and the CLDRSS project evaluation system is not an
exception. The theory-based evaluation normally uses a form of logic framework (Logframe) to
guide the collection and analysis of data for evaluation reports.
RESULTS AND DISCUSSION
What should be reviewed in an evaluation of a RWSS evaluation system?
Stufflebeam (1974) proposed eleven specific criteria to determine the quality of an evaluation(Table 1). Weiss (1995) and Rogers (2007) suggested that theory-based evaluation should include alogical framework, or Logframe, which has increasingly been applied in health community-based
interventions such as water and sanitation projects. According to these authors, the outcomes of anintervention are based on theories of how and why this intervention will work or through a pathway
in a particular context. Therefore, a theory-based evaluation must be designed based on a logicalframework or Logframe, and its focus should be on collecting and analysing data as required by theframework. Chelimsky (1995) regarded the political dynamics or environment as one of theconstraints to the quality of an evaluation, which may lead to “the total restriction or classificationof information”. Furthermore, according to the World Bank, three major criteria for evaluating
rural water supply and sanitation systems are sustainability, effective use, and replicability.
Capacity building is central to the progress of these dimensions. Narayan (1993) has emphasised
the importance of a participatory approach to the evaluation of Rural Water Supply and Sanitation© ISERD
168

IJERD – International Journal of Environmental and Rural Development (2013) 4-2

community-based (RWSS) interventions for the purpose of sustainability (Table 1).
Table 1 Summary of Meta-Evaluation Considerations/Indicators
Authors/
Organisations
Stufflebeam (1974)

Weiss (1995) and
Rogers (2007)
Narayan (1993)

Meta-Evaluation Considerations/Indicators
1) Internal validity, 2) External validity, 3) Reliability, 3) Objectivity, 4) Relevance,5) Importance, 6) Scope, 7) Credibility, 8) Timeliness, 9) Pervasiveness, 10) Cost/
effectiveness
Theory-based evaluation (logical framework, input-outcome model, assumptions, context)

Rural Water Supply and Sanitation (RWSS) indicators (participatory evaluation,effectiveness, sustainability, capacity building, replicability)
OECD Development
Effectiveness (e.g., Independent Completion Report effectiveness, gender, M&E system,
Assistance Committee
lessons learned); relevance (theory of change/intervention logics, context); sustainability
Evaluation Criteria(use of government systems, transactional vs. transformational); efficiency (e.g., size andscale, expenditure)
AUSAID criteria
Gender equality, monitoring and evaluation, analysis and learning, overall quality
Chelimsky (1999)
Political dynamics as a constraint
Source: Compilation from literature

What are the evaluation methodologies used by the CLRWSS project?
Evaluation Indicators: A set of indicators, with checklists, were established to review the major
components (Table 2).
Table 2 Major Components of CLDRSS Evaluation Indicators
Project Indicators

Component
A. WATER SUPPLY ANDSANITATION SYSTEMS

1.
2.
3.
4.

System Quality (4 sub-indicators)
Coverage and Access (2 sub-indicators)
Costs and Affordability (3 sub-indicators)
Operation – Maintenance – Management (3sub-indicators)
B. RWSS AGENCY
1. Capacity to promote water supply and
CAPACITY
sanitation
2. Capacity to support institutional capacity
building
3. Capacity to implement RWWS investments
4. Capacity for project management
C. INFORMATION,
1. Focus group analysis of key IEC activities,
EDUCATION AND
including clean water sources, canal water
COMMUNICATION (IEC)treatment, looking after piped water sources
2. Focus group analysis of IEC healthy schools
model
3. Reported cases of diarrhoea (Department of
Health data)
D. COMMUNITY
1. Activity is an appropriate solution, matches
ENVIRONMENTAL
design and is well implemented
SANITATION ACTIVITIES
2. Number of direct beneficiaries disaggregatedby activity type
Source: CLDRWSS Project M&E Report

Relevance to literature
indicators
Sustainability
Effective Use
Efficiency

Capacity building
Replicability

Effective use

Effective use

The indicators, the first ones applied in this field in Vietnam, were designed by an Australian
M&E Specialist and project consultants in compliance with the Vietnam Monitoring and
Evaluation Manual developed by the Vietnam Australia Monitoring and Evaluation Project 2005.
The measurements of sub-indicators for these components are based on 5-point, 4-point or 3-point
scale with scoring guidelines and checklists.
Theory-based Evaluation: Figure 1 shows reporting hierarchical structure. Different evaluation
reports from four teams, i.e. household survey team, technical evaluation team, IEC evaluators and© ISERD
169

IJERD – International Journal of Environmental and Rural Development (2013) 4-2

school IEC evaluators, are compiled and summarized into an M&E Summary Report. Eachevaluation team performed a different function using different data-collecting methods. Household
surveys conducted by external consultants for rating customer-based water and service quality;
technical site visits for standard-based system quality inspection; household surveys and interviews
for IEC outcome auditing; school IEC team for school hygiene and water using checklists andinterviews.
M&E Summary
ReportHousehold
SurveysTechnical
Evaluations

Household IEC
Evaluations

School IEC
Evaluations

Fig. 1 The CLDRWSS Project’s M&E Document Hierarchy
Source: CLDRWSSPThe above indicators and function structure are guidelines for conducting post-project
evaluation. The Project also formed a core M&E team, comprised of an Australian M&E specialist,an M&E project officer, an IT officer and five provincial M&E liaison officers for both processesof evaluation and post-project evaluation.
Table 3 Potential Threats to Evaluation Results
Component
Logic
Framework(Logframe)

Summary Description
Logframe consists of narrative summary
at project, component outcome andoutput levels; verifiable indicators andmeans of verification; achievements (i.e.
inputs).

Threats to Evaluation Reliability
Possible missing inputs: other healthpromotion interventions outside of project
influence water use behaviour during 5 projectyears; such factors as social norms, behaviour
control, service quality, price, etc would
impact intention to use.
Missing assumptions: food bacterial
contamination controlled; diarrhoea
seasonability
Unidentified intermediate outcome: would
the reduced incidence of diarrhoea be an
impact or an outcome? The answer to it is notclearly found in the Project M&E IEC reports.

Process
Evaluation

Indicators: investment information,
IEC activities, performance progress,
beneficiary number
Methods: synchronized data files
Time: monthly
Tools: RUWASS Management
Information System (MIS)
Indicators: See Table 2 for major
indicators
Methods: Site visits, surveys,
interviews
Time: 3 months upon completion
Tools: checklists, questionnaires,
records
Sampling: for household survey, about
10% beneficiary households.

None
Post-projectEvaluation
© ISERD170

 No weighted factor scoring system.
Specifically, the importance weight forsystem quality is 0.67 while that forcoverage and access is only 0.04.
 Three months is too short to see a behaviourchange in a context with deeply-rooted
health practices
 Sample size (about 20 per commune) may
be small as total beneficiary population percommune is 200.
 Good reports on achievements are more
likely to be appreciated (political dynamics).

IJERD – International Journal of Environmental and Rural Development (2013) 4-2

What are some threats to the validity of the evaluation results of the CLRWSS project?
In consideration of the theory-based evaluation model and other literature, the following tablesummarizes some threats to the evaluation results as perceived by the author.
Most of uncovered failure to mention assumptions falls on the information-educationcommunication (IEC) impacts. For example, one note is that prior to the Project, local people
already had high demand for improved water hence the Project IEC delivery programs might beunlikely to have much impact on increasing the demand. Take another example of assumption.
Safer drinking water is assumed to produce better health because it eliminates the risks of waterborne diseases (e.g., diarrhoea). In fact, some health studies conclude that other factors that cause
diarrhoea in poorer countries include flies, poor hand-washing practices, food bacterial
contamination, and diarrhoea seasonality (Blogg, 2005). Another point for justification is weighing
importance to each evaluation indicators (Table 4).
Table 4 shows that the weighted factor of “Coverage and Access” component is only 0.04
(5/135), which is too low compared to that of system quality (0.67). This component should
deserved a higher score because it is important to have more water users to prove that the water
scheme is effectively used. Also, some sub-indicators are on 5-point measurement scale whereas
the others on 3-point measurement scale. Some justification is needed to explain why.
Table 4 Score structure for evaluation in a rural piped water system
Components

Max Score (points)

Weighted Factor

90
5
14
26
135

0.67
0.04
0.10
0.19
1.00

System Quality
Coverage and AccessCosts and Affordability
Operation, Maintenance & Management
Source: CLDRWSS Project M&E Report

CONCLUSION
An evaluation report should be read and applied hence its quality needs to meet some standards.
This study proposes a new model for evaluation systems in a rural water supply and sanitation
project. Firstly, it should measure the degree of capacity building achievement as a result ofeffective use, replicability and sustainability of the intervention. The evaluation system should
stipulate that six months and even some years after water supply construction, site visits should be
conducted to inspect if the water scheme is well functioning, if more villagers change their
behaviour to the use of improved water and if the model of the project success could be copied toother rural locations. This inspection is critical because it is a waste of investment money if thewater supply system breaks down just a short period of time after being constructed due to limited
capacity in management. Previous studies have shown that a rural water supply activity is more
likely to be sustainable with active participation of community members, especially women.
Therefore, evaluation indicators should include the measurements of community participation, withconsiderations of gender issues, in the process of project implementation. Secondly, evaluation
methodology should be based on participatory approach. Community members, as direct
beneficiaries, would have more exact answers to evaluators’ questions on the impacts of the waterproject. Lastly, a review on an evaluation report sticks to the logic framework of the project.
Therefore, the design of an evaluation should include an adequate and clear logic framework. A
logical framework, with clear description of inputs, outputs, outcomes and impacts, with statedassumptions and contexts would be critical in an evaluation system as they would be used as
guidelines to steer evaluators to fairly assess the achievements of a project. Future evaluations
should use these indicators to verify the robustness and soundness of evaluations in this sector.
Generally, the evaluation reports of the CLDRWSS Project served the purpose of measuring theeffectiveness and sustainability of the Project’s activities. Several potential threats to the validity of© ISERD
171

IJERD – International Journal of Environmental and Rural Development (2013) 4-2

the evaluation, as described above, include the lack of some assumptions and intermediateoutcomes, timing for mature outcomes and assigning more resources for larger sample size.
ACKNOWLEDGEMENTS
I owe a debt of gratitude to the following specialists for their inputs to this paper: Mr. Ray Miles,
the former Australian Team Leader; Mr. Bruce Bailey, Australian Evaluation Specialist; Mr. Chris
Trewethey, the M&E coordinator; and Mr. Vince Keogh, Senior Engineer. Without them, I would
have been unable to produce this study.
REFERENCES
Bazeley, P. 2011. Study of independent completion reports and other evaluation documents. Ausaid.,
Australia.
Bailey, B. 2007. Monitoring and evaluation report. Cuu Long Delta Rural Water Supply and Sanitation
Project. Ausaid., Australia.
Busby, J.S. 1999. An assessment of post-project reviews. International Journal of Project Management. 30(3), 23-39.
Blogg, J.R. 2005. Health impact of Cuu Long Delta. Cuu Long Delta Rural Water Supply and Sanitation
Project. Ausaid., Australia.
Fitz-Gibbon, C.T. and Morris, L.L. 1996. Theory-based evaluation - What do we mean by theory-based
evaluation? Evaluation Practice. 17 (2), 177-184.
Chelimsky, E. 1995. The political environment of evaluation and what it means for the development of the
field. Evaluation Practice, 16, 215-225.
Cuu Long Rural Water Supply and Sanitation Project. 2007. Monitoring and evaluation report for activity
completion report. Ausaid, Australia.
Narayan, D. 1993. Participatory evaluation – Tools for managing change in water and sanitation. World Bank
Technial Paper No. 207. World Bank.
OECD-DAC, 1991. Glossary of terms used in evaluation, in: Methods and procedures in aid evaluation.
OECD, 2000. Glossary of evaluation and results based management (RBM) terms. The 32nd Meeting at
OEDC, Paris, France. February 10-11, 2000.
Prosavac, E.J. and Carey, R.G. 1997. Program evaluation. Methods and case studies. Prentice-Hall, Inc. New
Jersy, USA.
Rogers, P.J. 2007. Theory-based evaluation. Reflections Ten Years On. New Directions for Evaluation, 6465.
Stufflebeam, D.L. 1974. Meta-evaluation. Evaluation Center, College of Education, Western Michigan
University. USA.
Weiss, C.H. 1995. Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive
community initiatives for children and families. New approaches to evaluating community initiatives.
Concepts, methods, and contexts, 65-92.
UNICEF.(nd). Evaluation standards. (from;http://www.unicef.org/evaluation/files/Evaluation_standards.pdf.)
Vietnam Australia Monitoring and Evaluation Project. 2005. Monitoring and evaluation manual.
FERD/Ministry of Planning and Investment of Vietnam. Vietnam.
World Bank. (nd.) Glossary of M&E terms. (from; http://www.worldbank.org/ieg/ecd/docs/annex_e.pdf)© ISERD
172