Beyond Logframe;
Using Systems Concepts
in Evaluation

Foundation for Advanced Studies on International Development

The Foundation for Advanced Studies on International Development(FASID) was established in April 1990. FASID and its affiliate,
International Development Research Institute (IDRI), conduct research,facilitate interaction among researchers and practitioners, and offer training programs to development specialists. These activities are aimed for
improvement in the quality of development programs and policies.

Copyright© 2010 by FASID
Published in 2010 in Japan by
the Foundation for Advanced Studies on International Development,
1-6-17 Kudan-minami, Chiyoda-ku, Tokyo 102-0074, Japane-mail: pub@fasid.or.jp
URL: http://www.fasid.or.jp

Editor’s Note

The 2010 Issues and Prospects of Evaluations for International Development
employs systems concepts as clues to re-assess the conventional ways of conducting evaluations and to explore how development evaluation can potentially be made more useful.
In Japan, development evaluation predominantly relies on the Logical
Framework (logframe) when conducting evaluations. Evaluations based on alogframe often face difficulties. One such difficulty arises from the futile
attempt to develop an evaluation framework based on a logframe, which, inmany cases, was prepared as part of the early-stage planning of the project
and which then does not necessarily reflect a project’s real situation at thetime of evaluation. Although a logframe can be utilised initially as a tentative
project plan, logframes are rarely revised even when the situation haschanged. By the end of the project, the original logframe may not be an accurate embodiment of what the project is about and therefore logframes do notparticularly help in terminal or ex-post evaluations.
Still, having been institutionalized by clients, logframe-based evaluations
are common practice and in extreme cases, evaluators face the danger of
evaluating the logframe instead of the actual project. Although widely used
for its simplicity, logframes can end up becoming a cumbersome tool, oreven a hindrance to evaluation.
Various attempts have been made to overcome the limitations of thelogframe and some aid organizations such as USAID 1, UNDP 2, CIDA 3 andthe World Bank have shifted from the logframe to Results-BasedManagement (RBM). Now GTZ 4 is in the process of shifting to a new project
management approach designed on RBM and systems ideas.
In the first article, “Beyond logframe: Critique, Variations andAlternatives,” Richard Hummelbrunner, an evaluator/consultant from
Austria, sums up the critique of logframe and the Logical Framework
Approach (LFA), and explores some variations employed to overcome specific shortcomings of LFA. He then outlines a systemic alternative to logframe
12
3
4

United States Agency for International Development
United Nations Development Programme
Canadian International Development Agency
Deutsche Gesellshaft für Technische Zusammenarbeit (The German Technical Cooperation)and introduces the new GTZ management model for sustainable development called “Capacity WORKS.” Richard has dealt with LFA and possible
alternatives to LFA at various points along his career, and he is currently
involved in GTZ’s rollout of Capacity WORKS as it becomes the standard
management model for all BMZ 5 projects and programmes.
What does he mean by “systemic alternative”? In the second article,“Systems Thinking and Capacity Development in the International Arena,”
Bob Williams, a consultant and an expert in systems concepts, explains what“thinking systemically” is about and how it might help evaluation. He boils
down systems ideas into three core concepts (inter-relationships, perspectives, and boundaries), and relates these concepts to various systems methods.
In December 2009, FASID offered a training course and a seminar on this
topic in Tokyo. Through the exchange of numerous e-mails with the instructors prior to the seminar, it occurred to me that the concepts might be moreeasily understood presented as a conversation. That is what we tried to do in
the third article, “Using Systems Concepts in Evaluation – A Dialogue withPatricia Rogers and Bob Williams –.” These two instructors of the FASID
training course and workshop explain in simple conversational style where
and how we can start applying systems concepts in development evaluation.
This issue also carries a report of two collaborative evaluations ofJapanese Official Development Assistance (ODA) projects. The first case presents an innovative joint evaluation conducted collaboratively with
Vietnamese stakeholders. The evaluation took place in 2009 – 2010 as the lastyear of a three-year evaluation capacity development project coordinated by
the Japan International Cooperation Agency. The second case covers a joint
evaluation study of another Japanese ODA project in Lao PDR with a local
Lao administration for which neither logframe nor OECD DAC five criteria
was used. Instead, an evaluation framework was developed from scratch,based entirely on the beneficiaries’ interests and perspectives.
In both cases, a partner country’s participation in the evaluation necessitated considerable changes in perspectives of evaluation practice. I hope they
provide examples of how boundaries and perspectives, as discussed theoretically in the first three articles, relate to development evaluation in practice.

5

The German Federal Ministry for Economic Cooperation and Development

I would like to express my sincere gratitude to the contributors. Eacharticle or report is based on the opinions of each author and does not represent the opinions of the organizations to which the author belongs. It would
be my pleasure if this publication contributes to the ongoing discussions in
the development evaluation practice.
March 2010
Nobuko Fujita
FASID

Issues and Prospects of Evaluations for International Development – Series IV

Beyond Logframe;
Using Systems Concepts
in Evaluation
Contents
Editor’s Note
ArticlesI. Beyond logframe: Critique, Variations and Alternatives
Richard Hummelbrunner ………………………………………… 1
II. Systems Thinking and Capacity Development in the International
Arena
Bob Williams …………………………………………………… 35
III. Using Systems Concepts in Evaluation– A Dialogue with Patricia Rogers and Bob Williams –Patricia Rogers, Bob Williams, FASID ………………………… 55
ReportEnriching Evaluation with Perspectives of Local People– Two Cases of Collaborative Evaluation –Nobuko Fujita…………………………………………………… 75
Annex:The Socio-Economic Impacts of the Second MekongInternational Bridge
Nobuko Fujita, Eriko Takahashi ………………………………… 87
About the Contributors ………………………………………………… 119

1
Beyond logframe:
Critique, Variations and Alternatives
Richard Hummelbrunner 1

Over the last decades, the Logical Framework Approach (LFA) has become
universally known and has assumed a key role for planning and managing
development interventions 2. LFA, however, is not uncontroversial and theapproach has been subject to criticism, concerning both its theoretical foundations and practical use. Despite these criticisms LFA’s position has notbeen fundamentally weakened and while many donors acknowledge its limits
and weaknesses, they maintain (some would say impose) its use as a planning and monitoring tool.
This chapter reviews some of the experience gained with LFA and outlines major attempts to develop variations – or move beyond it altogether.
The first section briefly describes the LFA concept and summarizes the main
points of critique. Section 2 then explores some variations which have beendeveloped in response to this critique and to improve LFA as a management
tool. Section 3 proposes a systemic alternative to logframe and Section 4 outlines alternatives to LFA which have recently been introduced in German
development aid.

1. The Logical Framework Approach and its limitations
Over the past few decades LFA has come to play a central role in planning,
managing and monitoring development interventions. Its origins lie in a US
military planning approach, which USAID adopted for development projects
sometime in the late 1960s. In the 1980s it was used by several international

1
2

Senior Associate, ÖAR Regionalberatung, Austria
Intervention is used here as a summary term for projects and programmes alike.

1CHAPTER 1

development organizations (e.g. UNDP, ILO) and also taken on board by several European development organizations. In the late 1980s it was introduced
as a planning and management tool for European development aid. And by
the end of the 1990s it had become the standard approach required by many
donors and directorates of the European Commission.
1.1. The Logical Framework - concept and rationale
Originally the logframe was a tool for intra-organisational management inmilitary and business contexts of the 1960s, which were marked by strong
central authority and control around a relatively clear set of goals. It reflects amanagement style which demands precisely structured and quantifiableobjectives (“management by objectives”), assuming that the actors dispose of
all relevant information and operate in rather stable environments. The focus
is on the delivery of activities and outputs, and on the achievement of intended effects through intended routes.
It is useful to distinguish the Logical Framework (logframe) – the matrix
which summarises the main elements of an intervention and connects themto each other – from the Logical Framework Approach (LFA) – the process bywhich these elements are formulated:• Although there are variations in terminology and structure, the Logical
Framework as a matrix has a reasonably standard form which integrates
two types of logic:i) a vertical logic as a hierarchy of objectives – activities deliver outputs,
which contribute to outcomes, which help bring about the overall goal;ii) a horizontal logic showing how progress against each objective can beassessed (indicators and means of verification) and the external factors(assumptions and risks) which might affect the achievement of objectives at the next level.
• The Logical Framework Approach is concerned with the procedures ofproblem analysis, the development of objectives and indicators, and theidentification of risks and assumptions, which feed into the matrix. In
general, this process should be a participatory one, involving key stakeholders in order to reach consensus on an intervention, which is thensummarised in a logical framework.
Thus the LFA is more than the logframe, but there are many variations
using (or combining) these two aspects. Bakewell and Garbutt (2005) haveconcluded that the term’s connotations fall into three broad categories: the
LFA as a formal system, the LFA as a way of thinking and the LFA as a2

Beyond logframe: Critique, Variations and Alternatives‘brand’ to comply with donor requests. In practice, LFA can be used to produce a logframe (i.e. the ideal type described in the literature), but there arealso cases where logframes are established without LFA (i.e. a participatory
process) or LFA takes place without a logframe (i.e. fitting the agreements
reached into a matrix format).
LFA definitely has a series of advantages: It helps to think about and conceptualize interventions in a structured manner. It imposes a uniform way of
thinking and demands a formalized way of representing an intervention thateveryone who has been initiated into the method can read and understand.
By providing a common terminology, it can facilitate discussion andexchange between the various stakeholders. The logframe provides a convenient overview of the main features of an intervention, as well as the information needed for monitoring and evaluation. This overview is particularly useful for (busy) senior officials, funders or supervisors, who require – and prefer – clear and simple descriptions of an intervention.
However, as with any model, logframes are inevitably simplifications,which become dangerous when not seen as such; they can help logical thinking, not substitute for it. This danger particularly holds true when thelogframe (matrix) is mistaken for the design of an intervention – and not as avisual aid that summarises its most important aspects.
1.2. Critique of the Logical Framework Approach
In public interventions (e.g. development aid) it is difficult and dangerously
misleading to assume – or attempt imposing – the same sort of clarity andorder for which LFA was originally designed: differences in opinion among
stakeholders are likely to exist, which makes the assumption of consensual
objectives difficult and often highly problematic. A single centre of authority
rarely exists and – in practice – clearly agreed to objectives are rare to find.
When pursuing these objectives in situations in which there are multiple anddiverse stakeholders or changing or uncertain context conditions, the LFA
tends to over-specify objectives and to over-emphasize control as opposed toflexibility.
In his critique of LFA, Gasper concluded that “it has been a practice withrelatively little accompanying theory...while awareness of alternative possible
formats and styles within LFA has increased in the 1990s, understanding of thenature of what LFA attempts, what it achieves and where it fails has been limited, compared to the remarkable spread in its use” (Gasper, D. 2000, p.18).
Based on an analysis of LFA use, he has pointed to three recurrent fail3

CHAPTER 1

ings which inhibit the potential of the approach from being fully exploited(Gasper, D. 2000, p. 21f):• “logic-less frame”: When the use of logframes is imposed by donors, they
are often invented after a project has been prepared. Thus only an illusionof logic is provided because the logframe format is used to accommodatea pre-existing design, rather than to help create a logical design in an
appropriate format.
• “lack-frame”: The logframe is frequently too simple and omits vital
aspects of a project, as not everything of importance can be captured in
one table. Many users have underestimated that a “frame” includes somethings and leaves others out, and that a “frame-work” is to “frame”(=help) the required work, not substitute for it.
• “lock-frame”: after an LFA has been prepared, it tends to be fixed and not
up-dated, thus blocking learning and adaptation. Therefore oversimplified
plans (matrices) become treated as blueprints that dictate outcome, andas a control tool to ensure that the required outcome is achieved.
As a consequence, LFA is inherently easy to misuse. Its balance of advantages vs. disadvantages will depend on the characteristics of an intervention,
its context conditions, the nature of the problem(s) faced and the characteristics of involved actors. Therefore an essential skill in dealing with logframes
will be to know when not to use them – or when to supplement them withother methods.
The use of LFA in development aid has revealed some further problems,
which have added to and aggravated the shortcomings inherent to theapproach (Gasper 1997, Chambers 1997):• Often LFA is imposed externally (e.g. by donors), and then tends to be
applied in an over-standardized, rigid and top-down manner. Donors inparticular tend to consider the logframe (matrix) as something definite
against which aid recipients can be held accountable. LFA turns into arigid directive and unbendable administrative rule, turning “lack-frames”
into “lock-frames” – against all logic.
• Power imbalances, low trust and existential distances between “partners”
in aid programmes have contributed towards the lock-frame syndrome,
fearing a loss of accountability and control if receivers are allowed tomodify what has earlier been agreed.
• Use of LFA in a cross-cultural context has often led to the domination ofan external concept and the development ideology on which it is based.
Local management traditions or skills have been neglected and participa4

Beyond logframe: Critique, Variations and Alternatives

tion in the use of the approach has been severely hindered due to culturalalienation. And the use of LFA often ends with the funding, undermining
attempts to induce sustainable learning and capacity building.
• LFA’s has a mechanistic rationale, assuming a “linear” progression ofeffects which takes place quasi-automatic, i.e. irrespective of the actors
involved or contextual conditions. But for instance, carrying out activities
as planned is by no means a guarantee that expected results let alone
impacts will be achieved, as there are many other important (external orinternal) factors.
• It is very tempting to claim observable effects, whether or not an intervention has actually contributed to their achievement. This is often the casewith higher-level objectives, in which contributions of single factors areeasy to claim – but difficult to (dis)prove. Or in the case of long impact
chains, where causes and effects are rather distant from each other,either in time or in their functional relations.
As a result, LFA often fails to reflect the messy realities facing development actors, thus producing confusion rather than clarity. In addition, context dependent and far from universally applicable terminology and language
aggravates the situation. It is often difficult for different actors to agree onterms like output or outcome because they are not naturally occurring divisions in time. It is particularly difficult to communicate this way of thinking toactors with different logic or cultural backgrounds.
LFA was introduced as the core management tool of the EU’s PHARE
Programme (Poland and Hungary: Assistance for Restructuring their
Economies) in the early 1990s and was later expanded to provide financial
and technical assistance to the countries of Central and Eastern Europe andprepare them for EU membership. In fact, during the 1990s it became theworld’s most important and voluminous aid programme.
Having worked as consultant and manager of PHARE funded projects, I
gained much hands-on experience with LFA at the time. It had been introduced in one of its earlier formats, and had not taken into account LFA’s conceptual limits as well as the experience gained with its use up to that date.
Therefore it was little surprising that a lot of the deficiencies and problems
outlined above (re)appeared when LFA was used with PHARE.
LFA’s deficiencies were aggravated by the fact that it was applied in aperiod for which this tool is not well suited: intense economic and socialtransformation in Central and Eastern Europe, i.e. contexts of high dynamic
and complexity. In addition, LFA’s usefulness was further dampened because5CHAPTER 1

implementation of the PHARE programme was notoriously slow, leading toconsiderable delays between the planning and implementation of projects.
As the cumulative result of these factors, some of the most significant
effects were (Hummelbrunner 2001):– The tendency of the responsible authorities to enforce strict implementation of predetermined plans (as defined in the Terms of Reference – ToR)
led to a “tunnel vision” of projects. Projects were dissociated from theircontext and prior assumptions, regarding their implementation, were
often neglected. Important developments in the implementation context
were either overlooked or downplayed as “undesired” effects.
– Even though periodic reviews of plans were foreseen, the requirements
and approval procedures had the effect that modifications were only considered at periodic, pre-defined review steps: if at all. This was often inappropriate, either too late or – as in the case of inception reports – tooearly. Given the rather short implementation periods for projects,
approval times for project changes were particularly excessive (up to ayear).
– Strict separation between planning and implementation of projects contributed to a “mechanistic” style of implementation, seen as merely therealisation of previously defined plans. This separation was aggravated by
the fact that – due to tendering procedures and considerations of faircompetition – planning and implementation were usually carried out bydifferent actors.
– Due to this mechanistic style, the assumption side of logframes was often
superficial, if not neglected altogether. In particular when project documents needed to be drawn up quickly, these external factors were considered almost irrelevant, further reinforcing the tendency for “tunnel
vision.”– LFA was implemented in a context of cultural alienation and poor understanding. External experts who were usually not sufficiently familiar with
Iocal circumstance and culture dominated its use. Moreover, logframes
were often grafted upon projects in retrospect by (foreign) experts andparticipation in this exercise by local actors was rather poor.
– Despite the rhetoric of partnership, implementation of PHARE often took
place in a climate of dominance by EU authorities and mistrust towards
national “partners.” Furthermore this led to excessive rules and guidelines, which created a climate that diminished the chances for a more
flexible use of LFA and became an obstacle for its application.
6

Beyond logframe: Critique, Variations and Alternatives

As a consequence of this implementation framework, conformity to preestablished plans – instead of performance in relation to changing needs andcircumstance – became the central criteria against which EU officials, national administrators and experts were judged and also ‘rewarded’ each other. It
was more advantageous for all of these actors to implement interventions
according to original plans (as exemplified by the ToR), than to seek modifications when new insights or changes in context conditions warranted them.
Collective behaviour patterns based on these values led to satisfactory
reports, but hardly to better – and more sustainable – projects.
Since the LFA is intended for assessing progress towards objectives, it isoften used as a tool for monitoring and evaluation: However, due to conceptual flaws and limitations, it is not well suited for these purposes(Hummelbrunner 1997):– Since the focus of LFA is on achieving intended effects via intendedroutes, its utility for monitoring & evaluation is rather limited. By restricting attention to these elements, the “tunnel vision” aspects are reinforced
and effects other than those specified are systematically neglected. Andeven if unforeseen routes or unintended effects are taken into account,they are usually treated in an isolated manner and not linked to the original intervention logic, thus they have little explanatory value.
– This ‘accountability bias’ also reflects LFA’s fundamental assumptions ofperfect advance knowledge and full control of implementation - despite
the existence and conceptual importance of the Assumptions & Risks column in a logframe. But this part is usually taken the least seriously, especially if compared to outcomes and indicators. When it is not envisaged
that events can develop in unforeseen ways, no need is seen for a learning attitude and a corresponding evaluation style, which could help infine-tuning implementation and would need to pay attention to a broader
range of effects.
– LFA can become a serious hindrance in situations where unintended
effects or routes are important for understanding – and assessing – interventions. Or when the relations between an intervention and its context
are entangled and hard to foresee, intended routes are not well understood or based on prior experience. In short, when situations become
increasingly complex or require innovative routes.
Gasper (1997) noted that the LFA’s audit focus is an obstacle for learningand reinforces a climate of control. Under such conditions care is taken that
7

CHAPTER 1

interventions are made auditable: structured in line with the need to be monitored. This leads to a preference for measurable variables and short-term
effects and to the detriment of information which is qualitative and can only
be captured in the longer term. It can also turn into a veritable obsession
with indicators (‘indicator-itis’ 3) and their quantification, forgetting that indicators are only observation tools and should not be confused with what they
are intended to measure. But that’s exactly what happens when indicatorsare (mis)used as substitutes for stated objectives.
In recent years there has been less critical writing on LFA, or at least no
essentially new arguments were brought forth. In their review ofInternational Development NGOs’ Experiences with LFA, Bakewell andGarbutt (2005) have concluded that “the individual’s attitude to the LFA isrelated to how useful it is for their work: managers dealing with multiple projects like it as a useful summary to help simplify the complexity they face.
However, those closer to the messy realities of development are less convinced.”
LFA’s promise of improved management often remains unfulfilled andrespondents in this survey lamented LFA’s dominant role as a management
tool. People operate with a much higher level of complexity than can possibly
be included in a logframe, so the neat logic does not work in reality.
Moreover, the LFA requires reaching a consensus on one theory of change,
one logical path from activities to goals, which ignores that there may bealternative paths to get to the same goal. And it is usually the path that ismost acceptable to donors which dominates.
Where NGOs have invested in participatory LFA processes, they ironically found that the more participatory the approach, the more difficult it is torevise this consensus later on during implementation (‘lockframe’).
Whatever attempts were made to make the LFA participatory and flexible, it
seemed unable to change as quickly as the environment within which it has
been placed. And again: the more people participate in developing the logic
and completing the matrix, the more difficult it is to adapt the programme toa rapidly changing environment. But despite such difficulties, the blame isoften placed on those who use the LFA, rather than the approach itself.
“There is a common chorus...that the problem with the LFA is not the framework itself, but the way it is used” (Bakewell and Garbutt, 2005 p. 12).

3

8

A word coined by Gasper – the disease of being obsessed with indicators.

Beyond logframe: Critique, Variations and Alternatives

2. Variations of the Logical Framework Approach
Awareness of the need for different formats and styles within LFA has
increased throughout the 1990s and attempts were made to adapt it in various ways. At the most basic level, the terminology was adapted to fit betterwith the specific conditions. Some have made changes to the matrix, forinstance by adding additional rows to reduce the size of steps between levelsin the objective hierarchy. Others have used nested logframes to break downlarge programme frameworks into smaller parts. But while such adaptations
make it easier to map the reality of development programmes, they do nottackle any of the more profound objections to LFA.
Another, more radical adaptation of the LFA is to abandon the matrix altogether, while retaining the basic elements. This has the possible advantagesof avoiding the ‘box’ format or allowing for better visualization, as is the casewith Logic Models, for instance, which moved from a tabular (matrix) structure to diagrams. Using Logic Models, which already began in the 1970s, has
become quite popular with the rise of theory-based evaluations. Although
there are some methodological differences, programme logic models use asimilar hierarchy of objectives and linear sequence of events.
Three variations are outlined in this section: One, “PCM” which wasdeveloped early on as a reaction to initial critique of LFA, and two others
related to “Outcome Mapping and Social Network Analysis” which link LFA
with methods that have been developed more recently to deal appropriately
with complicated or complex situations.
2.1. Project Cycle Management (PCM)
In 1992 the Directorate General for External Aid (DG VIII) of the European
Commission adopted “Project Cycle Management” (PCM) as its primary setof project design and management tools. PCM, which was developed at theend of an extensive evaluation exercise, was adopted with a rather soberingoutcome. In his account on the making of PCM, the Director of DG VIII at
the time, H. Eggers (1998), pinpoints the key reasons:– There was a general tendency to confuse the project with the people thatwere meant to profit from its implementation and functioning;– In many cases, vitally important aspects were overlooked in project preparation, implementation and follow-up; and– All too often, decisions were taken without being subject to the required
9CHAPTER 1

decision-making discipline appropriate to each phase in the cycle.
These three root causes of project failure were then transformed into
future guidance in the form of three principles, which remain the cornerstones of PCM today:
1. Express the project purpose in terms of sustainable benefits for the target
group: This is the most basic principle of PCM, which should allow making a clear distinction between the project and the people affected by it.
The Project Purpose is also called ‘Specific Objective’ and its place in a
log-frame is between ‘Results’ and ‘Overall Objective(s)’.
2. Devise a ‘basic format’ setting out the vital aspects of a project: These are
the criteria that must be observed if a project is to achieve sustainable
benefits. The Basic Format is PCMs first technical tool and structures thecriteria under the following headings:– Background: presenting the policy guiding the project, the beneficiaries
concerned and the problems to be solved;– Intervention: spelling out the objectives of the project, notably the kind ofsustainable benefits to be created, and the activities designed to achieve
the objectives;– Assumptions: alluding to the risks to be faced and identifying the circumstances or events considered essential for the project’s success but that it
cannot influence;– Implementation: containing the concrete project description – means tobe employed, their cost, procedures to be applied, the implementation
timetable;– Sustainability factors: policy, technological, environmental, socio-cultural,management and economic aspects that are designed to ensure projectsurvival after completion;– Monitoring/evaluation: allowing the project to be kept on track and be
reoriented, if necessary.
3. Devise a mechanism to guide sound decision-making throughout the project cycle: This principle requires the application of the same criteria (asexpressed in the ‘Basic Format’) throughout the entire project cycle. The
cycle of operations for managing the EU’s external assistance projects
has five phases, as shown in Figure 1 below 4.
Whereas the duration and importance of each phase will vary for different
410

In the first version, the project cycle included six phases, but later ‘Financing’ was removed as a single
stage in the cycle, given that the financing decision is taken at different times depending on the EC
Regulation under which projects are financed.

Beyond logframe: Critique, Variations and Alternatives

Figure 1
Programming

Evalution & Audit

Identification

Implementation
Formulation

projects (depending on their scale, scope and operating modalities), threemain principles are to be followed:– Decision making criteria and procedures are defined at each phase(including key information requirements and quality assessment criteria);– The phases in the cycle are progressive – each phase should be completed for the next to be tackled with success; and– New programming and project identification draws on the results of monitoring and evaluation as part of a structured process of feedback andinstitutional learning.
The ‘Format of Phases and Decisions’ is PCM’s second technical tool. It
provides guidance for the production of good-quality key document(s) ineach phase (with commonly understood concepts and definitions) and therelated decision-making mechanisms which they support.
The logframe is PCM’s third technical tool. But it is only part of themethod and was integrated as an analytical tool, also taking into account its
weaknesses. Therefore PCM tries to limit the use of the logframe for project
design, and emphasizes the importance of the LFA (as a participatory
process) as much as the matrix product. Logframe analysis is to be carriedout during the Identification phase and involves three main elements:
Problem Analysis, Analysis of Objectives and Analysis of Strategies(Stakeholder Analysis was added later on as a forth element). And thelogframe is to be drawn up as a planning tool during the Formulation phase.
The ‘Basic Format’ and the logframe do overlap in parts and should be
used complimentary: Whereas the Basic Format is employed in structuring
the Terms of Reference for studies or reports along the project cycle, the11CHAPTER 1

logframe is a means for summarising the project design as well as a ‘checking mechanism’ for the internal logic of the project and each document.
PCM has been conceived as an instrument for improving development
projects and it differs – at least in theory – from LFA due to a mutual learning
philosophy, a more participatory approach and a positive debating culture. It
should facilitate the management of larger, complex interventions and allow
implementing them in a more flexible, learning-oriented manner. But in PCM
practice the logframe is often given a (too) dominating role, which leads tooverly rigid applications and overrides some of the benefits expected fromPCM.
The ‘father’ of PCM, Eggers (1998), was very clear on this threat, when
he formulated the following warning after several years of PCM practice:“Don’t be tempted to fall back into the routine of the logframe tradition. LFA
and PCM are not the same thing. Applying only the logframe would, in myview, be a recipe for failure”.
Unfortunately, later versions of PCM (e.g. European Commission 2004)
did not correct this flaw: They did take new aid delivery modalities into
account, introduced institutional and organisational capacity assessments
and provided a set of quality attributes, criteria and standards (the Quality
Frame). But they did not fundamentally alter the role and function of LFA –and the logframe in particular – within PCM.
As consequence, when applying PCM (especially with larger, long terminterventions and/or under more dynamic context conditions) some of the
weaknesses associated with LFA become even more visible and acute:– Rigid planning: Although reviews are foreseen in principle, the procedural requirements often prohibit modifications (especially at higher levels),thus original plans turn into rigid tracks which are hard to leave or
bypass, despite changes or better insights.
– Problem orientation: The focus is on problems and their solutions, as
exemplified by the Problem (Tree) Analysis in the Identification phase.
This is not only a retrospective leaning, but also an emphasis on obstaclesor threats – and neglects prospective creativity.
– Linear causality: The intervention logic (of the first logframe column)
assumes a linear means-end causality between the various levels and asequence of stages over time. Although there can be several elements ateach level, there is no meaningful way of showing how they interact within or between levels. The assumed process of causation is also one way
and therefore circular causality (feedback) and connections between ele12

Beyond logframe: Critique, Variations and Alternatives

ments or delays are neglected. But these interrelationships are essential
for understanding – and taking account of – the dynamic features of interventions.
2.2. From Logical to Social Frameworks
Social Network Analysis (SNA) is a set of techniques for analyzing social systems. It can be used to understand networks and their participants, i.e. grasp
and describe the organization of the network as a whole as well as the position of individual actors. It offers a variety of techniques (e.g. matrices, network graphs) for visualizing, measuring and simulating relationships andallows for analyzing these relationships in visual as well as mathematicalterms.
Since SNA lends itself well for capturing complex relationships, it canalso be used as an alternative to prevailing linear stage models for representing theories of change. Programmes involving various actors and operating
at different levels of scale can be conceptualized as networks, in terms thatare easy to understand and possible to verify.
The UK based evaluation and monitoring consultant Rick Davies suggests using an SNA perspective to overcome some of the problems withlogframes. “It is useful to think of events described in the Logical Framework in
an actor-centred way, not simply as events involving disembodied and abstract
processes of change. When this is done most “theories of change” documented
within a Logical Framework becomes clearer, along with their plausibility orlack there of ” (Davies 2005, p.3).
He points out that the connections between each of the adjacent levels of
the Logical Framework (and the associated types of actors) are far from linear. They can often be quite complex and the various linkages may overlap
leading to networks of relationships. The same holds true for the indicatorsin the Means of Verification column. It is useful to conceive them as interconnected networks, rather than as a series of parallel one-to-one linkages
because there are usually several indicators found at each level,.
He has also developed several potential applications for SNA-based models of change (see his website for corresponding material): 5
The first and foremost is to use SNA for moving from Logical to Social
Frameworks. A network perspective replaces the stages in time of Logical5

His website http://mande.co.uk/special-issues/network-models/ has two special sections for SNA and
LFA, each of them containing a range of documents and other material.

13

CHAPTER 1

Frameworks (e.g. from Activities to Goals) by a sequence of actors, connected by their relationships (e.g. intended beneficiaries, partner organization,and project staff). This can be seen as a potential impact pathway along
which influence, money and material objects can pass – in both directions.
The difference between the two views is captured in the figure below,
which was developed while working on a Monitoring and Learning Plan for
an NGO (ALNAP). It shows how the traditional rows in a logframe can be
refocused to describe the expected change in each actor within a larger
chain of actors (a ‘Social Framework’). The other columns of the matrix,
describing Objectively Verifiable Indicators, Means of Verifications andAssumptions, can still be used to describe what should happen with eachactor in a Social Framework.
Figure 2. Relationships between rows in Social vs. Logical Framework viewof ALNAP
New view: Actors who are part of a pathway
Intended beneficiaries
of humanitarian actions

Other organisations

Old view: Stages in time

Super-Goal

Goal

ALNAP members

Purpose

ALNAP
Secretariat

Outputs

Steering Committee

Activities

Nodes = Actors, or categories of actors
Lines = Relationships (to other actors and amongst members of a category of actors
Thick line = Main relationship documented in the Social FrameworkThin line = Other relationships that are expected to be important, described under
Assumptions
Source: Davies, R. (2009), p. 9

Such a social framework view has several advantages: Responsibility forachieving results is distributed along the whole chain of actors and therespective changes expected from each actor can be described. This form
allows the building of several response levels into each row and the delega14

Beyond logframe: Critique, Variations and Alternatives

tion and the clarification of details of the framework to the actors concerned
at each level. Since this pathway takes place within a wider network of actors,
the map can show the range of possible relations between actors. This inturn can serve to identify alternative pathways for achieving desired changes
that can then be used for defining viable options and guiding subsequentmonitoring and evaluation efforts.
Secondly, SNA can also be used to move beyond overly simple logic models,e.g. by connecting events which take place at different levels. For instance,complex causal relationships can be represented by linking output (indicators) and results (respectively objectives) and by assembling the expected
contributions of each output. This is best done in a participatory manner, byusing matrices projected on screens in workshops, and progressively filled inby participants, line by line, and revised, as discussion of each relationshipproceeds. If the matrix becomes too difficult to handle, it could be simplified
by focusing only on the most important linkages (those above a specificthreshold value) and transforming the information into a network diagram.
In this way, SNA tools can be used alongside logframes or logic models andapplied to aspects that cannot be easily captured by the logframes or logic
models.
Whereas the two uses outlined above are best applied within a single
organisation, SNA can also be used in situations where multiple organizations
work together without a joint plan or theory of change. Here network structures can be considered emergent outcomes, resulting from the decision
making of individual actors. For example, individual organizations often
make their own decisions about which issue coalitions to support, but these
individual decisions can result in an aggregate structure that may or may not
be optimal from the network point of view. The process of mapping those
networks, and feeding back the results to the network members, has thepotential to facilitate decentralized planning and evaluation processes, without impinging on each actor’s autonomy.
These uses exemplify ways in which SNA tools can be used for developing new methods of participatory and decentralized planning or evaluation, in
situations which involve a diversity of objectives and actors. Strategies can bebuilt bottom-up, by aggregating the views of individual actors, identifying
their consequences, and resolving gaps through reiteration. Such a process
can also provide useful information about overall objectives to all as well as
guidance to individual actors about who they need to work with about what.

15

CHAPTER 1

2.3. LFA and Outcome Mapping
In response to the weaknesses and criticism of other monitoring frameworks
in complex change processes, Outcome Mapping (OM) was developed in2001 by the International Development Research Centre (IDRC) in Canada
with research partners in Asia, Africa and Latin America. By too narrowly
focusing their planning, monitoring and evaluation activities, results-based
management frameworks (e.g. logframe) have been found to be counterproductive: hindering interventions. This is an obstacle for innovative approaches, learning and flexibility – and a strong marker for failure in complex situations.
OM is an approach to planning, monitoring, and evaluating social change
initiatives. It is a set of tools and guidelines that steer project or programme
teams through an iterative process to identify their desired change and towork collaboratively to bring it about. Its originality lies in the shift awayfrom assessing the development impact of a programme – and towardchanges in the behaviours, relationships, actions or activities of the people,groups, and organizations with whom a development programme is working
directly and seeking to influence – and of the programme being influenced
by these interactions.
OM differs from other logic models in several ways: Foremost, it recognizes the importance of perspectives, i.e. that actors operate within different
logic and responsibility systems. It is not based on a linear cause-effectframework but assumes that multiple (often nonlinear) causes lead tochange. And it departs from the notion of attributing that change to specificintervention(s), but assumes that only contributions are made – and tracks
these by looking at the logical links between interventions and behaviouralchange.
The focus of OM is on one specific type of outcome, changes in the
behaviour of ‘boundary partners,’ with whom a programme works directly
and anticipates opportunities for mutual influence. These outcomes should atleast be logically linked to programme activities (but not necessarily directly
caused by them). While recognizing that challenges beyond the scope of the
programme exist, OM limits performance assessment to a programme’s
direct sphere of influence. It monitors and evaluates whether a programme
has contributed to intended behaviour changes – and does so in ways which
can be sustained in the future. In that sense it is also one of the most boundary-aware monitoring and evaluation approaches.
In operational terms OM defines three distinct but interrelated sets of16

Beyond logframe: Critique, Variations and Alternatives

activities and changes, and provides tools to monitor them: changes in partners, programme strategies and organizational practices. By linking the programme’s organizational assessment with monitoring the changes of theirpartners, OM assumes that a programme needs to grow and develop in orderto make a significant contribution. Programmes are encouraged to think ofthemselves as dynamic organizations, improving their ability to work with
their partners by reviewing and adjusting their own goals and methods.
OM assumes that change is controlled by the partners of an initiative, andthat development (aid) programmes, as external agents, can only facilitate
the process for a certain period of time by providing access to new resources,
ideas, or opportunities. A focus on partner’s behaviour emphasizes the needto effectively devolve power and responsibility to endogenous actors, as acondition for success.
Although data is gathered on the programme’s actions and on changes in
its partners, OM does not attempt to imply a causal relationship between the
two. The programme can make a logical argument regarding its contributions to change, but cannot claim sole credit. By combining information on
external outcomes with data on internal performance, a programme will beable to tell learning stories illustrating how the programme has improved its
work in order to encourage the transformation of its partners. And it can document the extent to which the partners have moved in the desired directions.
Lately some attempts were made towards a fusion of LFA and OM, inorder to address the gaps that are perceived in each approach. Ambrose andRoduner (2009) postulate that such a fusion can integrate LFA’s results-oriented focus with OM’s process-oriented learning pathways. In their paper
they define four cornerstones of the fusion model:• Focus on different scales of results (behavioural changes, capacity building, and impact): Orientation on capacity building and on results are to be
seen as complementary approaches but not as mutually exclusive options(unfortunately this is often the case).
• Looking for the common denominator: Instead of seeing LFA and OM as
competing approaches, the intention is to combine them and thus use
their respective strengths, i.e. the most convincing and effective elementsof each approach.
• Capacity building: Since the OM approach was originally conceived for
projects in which the focus is on changes in behaviours and capacities ofpartners, a fusion model will be most applicable for projects where capacity building plays a major role.
17

CHAPTER 1• Fusion = LFA+ or OM+: The goal of the fusion is to combine the advantages and strengths of both approaches, so that it is applicable in different
institutional contexts in the most multifaceted way.
The details of a fusion model should be defined in response to context
and stakeholders, either by enhancing the LFA or OM. The focus consists oforientation towards an overall goal and explicit consideration of changes inbehaviour of project partners. The fusion model should make it possible todetermine and display the distribution of roles and responsibilities of development actors directly in the logic model. So far as it is reasonable and possible, this should be presented in synoptic form (e.g. as a table or matrix), outlining the summary of core elements of the intervention. A document containing detailed descriptions of all elements is needed for a more profoundunderstanding.
Key requirements and demands of a fusion model are (see also fig.
below):• Programme Goal: In a fusion model these goals are defined as results ofthe behaviour changes of the partners and – following the logic of OM –the responsibility for verifying whether these goals are achieved rests primarily with the programme partners.
• Outcome Challenges: They are formulated for each partner and describe
the tasks, responsibilities and activities that they must carry out in order
to contribute to programme goals within their system, including what
they must do beyond programme support.
• Progress markers: Qualitative and quantitative indicators are defined foreach partner for monitoring changes in their practice or behaviour.
Progress markers may also be defined for several partners at once andneed to be monitored at specific stages or times.
• Strategy maps and outputs: Clear and concrete description of the strategies (activities and outputs), roles and responsibilities. Outputs musthave a plausible relation to outcome challenges and progress markers
and should be verifiable through indicators.
• Mission of the external change agent: This is useful for defining theintended support to the partners and clarifying the change agent’s role(which is limited in time and scope).
A fusion model can lead to con-fusion and more work, therefore it should
only be applied under conditions where it can provide added-value. For
instance in situations where there is a need to harmonise between various
levels (i.e. from ministries to communities), when information for various
18Beyond logframe: Critique, Variations and Alternatives

Figure 3: The Fusion Model at a Glance
Formulation of the desired
situation as well as thepractices and behaviourof project partners

Overall Goal
(Beneficiaries)

Impact indicators

Description of concretechanges to be attainedby the project (situationalchanges)

Program Goal

Goal indicators

Description of tasks andresponsibilities ofprogram partners

Program Partner A
Outcome Challenge

Program Partner A
Outcome Challenge

Definition of tasks androles of the project team &outputs that are provided
to partners

OutputOutputStrategy map

Mission: Defining theintended overall support
by the external change
agent

Mission of the Change Agent &Project Management (Budget, HR, Organisational Practices)

Progress Markers(Outcome Measurement)

Output indicators
Strategy map

Source: Ambrose and Roduner (2009)

partners or for different needs and accountabilities is needed, or if a programme should be improved in an LFA environment.

3. Systemic Project Management
The approaches described in the previous section are variations of LFA,
which may be needed to overcome some LFA’s shortcomings and render it amore effective management tool. A systemic management perspective, on
the other hand, is an alternative approach that is based on profoundly different concepts and uses instruments other than the logframe.
Since systems concepts and systems thinking are explored in other chapters of this volume, focus will be placed here on the implications for managing interventions. The systemic approach to project management outlined
here is largely based on ideas developed at the University of St. Gallen,
Switzerland, and the writings of Prof. Schwaninger and his colleagues at theInstitute for Management Science in particular (2006, 2001 a, b). They havetaken systems ideas, notably Stafford Beer’s ‘Viable System Model’ (VSM),
and elaborated an ‘evolutionary’ management model, which has influenced
management thinking in German speaking countries and beyond.
A central aspect of the Viable System Model is the principle of recursion.
19

CHAPTER 1This principle states that, in a constellation of systems which are themselvescomposed of sub-systems, each system, regardless of the level on which it issituated, will display the same structure. This means that systems that are
structured according to this principle will not display the pyramid-type structure familiar from organizational theory. Rather, they will be like mutuallyinterlocking and “nested” systems that resemble Russian “Matryoshka
dolls.”
Systemic Project Management consists of three main elements and corresponding tools:
At the core are Primary (“value creating”) Processes, i.e. those activities
that are directly responsible for producing desired outputs and outcomes. In
development interventions these primary processes usually consist of activities, which are implemented by (public or private) actors/owners for whom
an external agent (donor) provides resources, mostly in the form of funding,but also advisory services, provision of information, co-ordination etc.
These processes are conceived as networks or loops, which connect thevarious elements (or variables) that are considered decisive for producing
the desired effects. Causal Loop Diagrams (CLDs) are an appropriate – andwidely used – representational tool. They are based on the concept of “feedback” (originally developed in cybernetics), and their building blocks are
feedback loops, i.e. closed sequences of causes and effects: variable X isaffecting Y and Y in turn affecting X, due to the relations between them.
Feedback loops can appear in two types, reinforcing (positive) or balancing(negative) feedback.
CLDs visualize variables and their relationships over time, thus permitting the analysis of relational patterns of a primary process. But they also
allow looking beyond current states and reaching a dynamic – and systemic –level of understanding, by mapping the structure that is responsible for producing recurring patterns of events over time.
On the other hand, the primary processes can also be conceived as the
product of interacting social systems, which are involved in producing thedesired outputs or outcomes (e.g. donors, project staff, beneficiaries, external experts). These social systems are semi-autonomous agents who act
according to their own logic and values. In cooperation systems (without acentral command entity) their interactions are essentially self-organized.
Thus the behaviour patterns do not only depend on the properties of the individual actors, but also on their relationships. However, self-organized behaviour is neither predictable nor determined by the starting conditions alone,20

Beyond logframe: Critique, Variations and Alternatives

but will evolve over time.
Therefore the management task should be conceived as the ‘steering’ ofinterlinked value-creating processes, which needs to take into account thelogics of the actors involved. Communication and managing relationships areimportant ingredients of such steering processes, which should also aim atreconciling the various logics. If the logic of one actor (e.g. the donor)
becomes too dominant, sub-optimal outputs or outcomes are likely – andmight even jeopardize the achievement of objectives altogether.
Although this steering task is essentially fulfilled by those who are in
direct contact with beneficiaries (e.g. project staff), they act within a framework established by others. Such a framework can be regarded as a set ofnested systems, and steering should be done in line with the principles of
Recursive Management, which is the second element of the method. Thismeans that the interfaces between a system and its subsystem are managed
in similar ways. Therefore the flow of information follows the same ‘self-similar’ (fractal) pattern at all organisational levels. The following figure illustrates the management levels of a cooperation system (e.g. Donor,
Programme and Project)
Figure 4: Recursive management modelInterface
Donor

Programme(e.g. Country,
Sector)Interface
Project
Source: Adapted from Schwaninger, M. (2001a)

The interfaces between these levels can be designed according to theprinciples of “management by objectives”: Contracts or agreements determine what other systems (e.g. donors, partners) expects from a project, forinstance the objectives to be met or the quality criteria to be achieved; fur21

CHAPTER 1

thermore it defines the most important framework conditions for implementation (e.g. rules, milestones, issues to be taken into account).
What should not be specified, however, is how this contract is to be fulfilled, namely the activities and operations envisaged. This is entrusted to thesole responsibility of the project and its respective cooperation system, i.e.
external agents and their partners. Thus the agreement is not concluded any
more at the level of outputs and outcomes (as in a logframe), but instead at
the “meta-level” of objectives, core processes and rules. Such an approach
not only drastically reduces the workload for planning and monitoring, it also
makes a project more adaptive, flexible and realistic.
If interventions are designed and their implementation is organised insuch a recursive manner, monitoring and control requirements will be substantially reduced and their structure considerably altered. Instead of a hierarchic top-down manner, control will be organized systemically, whereby
each level monitors its own performance, but within the framework definedby the level(s) above. Systemic Control is the third element of the method andshould follow similar principles as management, essentially linking self-control processes carried out at various levels in a recursive manner.
Thus responsibility for the entire system is assumed jointly and is “translated” into rules for self-control of the various sub-systems, to be supervisedby the corresponding higher level. But standardizing these processes is difficult, which essentially require sensitivity for change. To this end, control
tasks must be carried out in a decentralized manner and by the same actors
who are responsible for planning and implementation. They have the best
knowledge on planning assumptions or implementing conditions and are
therefore well placed to detect relevant changes, including weak or ‘earlywarning’ signals. These can foreshadow important changes, but often remain
unnoticed in centralized control systems.
In more general terms, the tasks of Systemic Control can be associatedwith different management levels. These are not to be confused with the subsystems of the recursive management model, but the tasks can be distributed across the subsystems.
Systemic control is carried out at three levels of management (derived
from the VSM concept), which act in support of the primary processes:– The task of the normative level is to assure that things are done at the service “of the larger whole”: It establishes policy in light of competing
demands between the present and the future, and between internal andexternal perspectives. It ensures long term viability and development,
22

Beyond logframe: Critique, Variations and Alternatives

Figure 5: Model of Systemic Control and corresponding management levels
Normative
Management

Environment/
Context

Strategic
Management

Operative
Management

Primary (Valuecreating) Processes

Source: Adapted from Schwaninger M. (2001a)

identity and vision.
– The task of the strategic level is “to do the right things”: It decides on
resources for the primary processes and controls their effective use. To
this end it acts as interface between the normative and operative level,
but also monitors the environment and assures organizational stability
through adaptation. It is carried out through supervisory entities (e.g.
steering committees, boards) and makes use of intelligence tools likeforesight or scans.
– The task of the operative level is “to do things right”: It provides information and communication which are useful to coordinate the various primary processes, assuring stability and conflict resolution. This is usually carried out through coordination teams, information systems or internal service providers.
In spite of rising complexity from the operative to the normative level, thesteering tasks at each level follow similar patterns and are essentially relying
on the self-organising capacity of these sub-systems. In this model, hierarchy
is a logical construct and does not imply a hierarchy of command and control. Therefore authority is not rooted in chains of command, but in the relevance of information. And continuous, real time feedback should ensure thatthe information is kept relevant during the entire process duration.
Schwaninger and Körner have tested elements of this model with development projects (2001b). And together with colleagues from ÖAR
Regionalberatung I have applied this model to the management challenges
23

CHAPTER 1

posed by the EU’s Structural Fund Programmes (Baumfeld,
Hummelbrunner, Lukesch 2002). These programmes are characterized bymultiple objectives and involve several implementing agents located at different levels. They normally consist of a set of support measures with specificobjectives and budget that are implemented through a large number of projects, with a given time frame and pre-defined funding conditions. Morerecently, we have also applied these ideas to the management tasks of territories (“regional governance”) and development programmes implemented in
multi-level, multi-actor contexts.

4. Alternatives to LFA in German Development Aid
The German Association for Technical Co-operation (GTZ) has been a
European “pilot user” of LFA. After initial experiments with the logical framework in the 1970s GTZ have developed their own variant called ZOPP (objectives-oriented project planning). Some of the criticism on earlier LFA versions has been taken into account and lead to the incorporation of new analytical elements (e.g. stakeholder and problem analysis). ZOPP was also conceived as a participatory process, which consisted in a series of stakeholder
workshops that were facilitated by trained moderators and involved the useof metaplan (‘cards’) technique.
ZOPP was formerly adopted in 1983 as a compulsory method and rapidly
became GTZ’s trademark. It was also received quite favourably by the international aid community and judging by how many donors and NGO havetaken it up, GTZ is probably the most widespread 2nd generation version of
LFA today. Despite this positive response, however, the appropriateness anddelivery modes of the method, particularly for cross-cultural work, haveattracted increasing discussion.
These criticisms lead to substantial modifications and eventually to a‘downgrading’ of ZOPP. From the mid-90s it was treated as just one set oftools among many others, which required flexible and selective use. From1995 onwards GTZ relaunched LFA within the broader perspective of PCM.
Later on, other formats were tested, which conceptually departed from LFA
and used a systemic perspective. This notably involved the work of
Schwaninger (see section 3) and SINFONIE, a systemic planning method forstrategy development in complex situations.

24Beyond logframe: Critique, Variations and Alternatives

4.1. Managing for Results (‘Orientation on Impacts’)With the formulation of the Millennium Development Goals and the Paris
Declaration, the discussion on quality and effectiveness of development cooperation has taken on new importance. New approaches for measuring andattributing results were introduced. This greater emphasis on results concerned not only the procedures and instruments used but also their correct
application, calling for a change in attitude and modes of conduct. The paradigm had shifted to an understanding that development cannot be precisely
planned and risks are not always foreseeable.
In the past GTZ’s quality assurance was based on detailed, goal oriented
and logical project planning (quality at entry) combined with performancemonitoring based on comparisons of the actual situation with targets. The
assumption was that detailed situation analysis and goal-oriented planning
automatically led to quality and success. But this assumption was increasingly contradicted by project practice. And it became evident that this kind ofrather input-oriented project planning and implementation did not necessarily correlate with the economic and social development processes in the partner countries. This insight has led to a stronger focus on development
results (quality at exit). Consequently managing for development results has
been enhanced at all levels of project work (GTZ 2004).
Fig 6: Managing for development results at GTZ
Before:Input/Output
Orientation
focus onoutputInput/Activities

less on impact

Outputs

Outcome/Results

Impact/Results

Today: Managing
for Development
Results(Outcome, Impact)
focus onoutcomeand on impact

Input/Activities Outputs Use of Outputs Outcome/Results Impact/Results
Source: Reuber, M. and Haas, O. (2009)

All GTZ monitoring & evaluation activities are therefore now geared
towards results. The project environment is examined from two perspectives:
25

CHAPTER 1

first, the effect the environment has on the project, and second, the extent towhich the project contributes to the changes in the environment. Typicalquestions in results-based monitoring & evaluation are:
– Which areas may be affected by the project’s activities and outputs?
– Is the project about to achieve its intended results?
– Are there any unintended (positive or negative) results that need to beaddressed?
– What changes can be observed in the project environment and can beplausibly attributed to the project?
Fig. 7: The GTZ results model (results chain taking the example of vocational
training)
Impact / overarching indirect
results

Poverty reduction

graduates find employment
and income

Attribution gap
Outcome (direct results)
Intermediate outcome(use of outputs)

graduated have the knowledge and skills
damanded by the labour market

the application of new curricula at vocational training
institutes

Outputs
Activities

demand-orientated curricula are available
development and proving of new curricula

Inputsmaterials and equipment, personnel, funding
Source: Reuber, M. and Haas, O. (2009)

The planning of a project is usually based on cause/effect hypotheses, i.e.
assumptions of the relationship between intervention and result. The GTZ
results model (‘orientation on impacts’) represents the results of project outputs on different levels:– Use of outputs: This level describes whether and how the outputs of a
project are used by the beneficiaries (e.g. the application of new curricula
at vocational training institutes).
– The direct results describe the achievements through the use of outputs(e.g. graduates have the knowledge and skills demanded by the labour
market). The overall objective of the project is located at this level (out26

Beyond logframe: Critique, Variations and Alternativescome).
The overarching indirect results describe the changes of higher aggregated development progress (impact) to which GTZ’s work aims to contribute via direct results (e.g. graduates find employment and income
possibly leading to poverty reduction).
It is generally difficult to attribute development progress at higher aggregated levels because this concerns comprehensive and complex changes inthe environment of the project, in which several actors are often involved. It
is methodologically challenging to clearly attribute the contribution of anindividual project to changes at higher levels. GTZ therefore acknowledges
the existence of an “attribution gap.” Although it is difficult to establish an
unambiguous causal relationship between the project and the changes in its
environment, a plausible attribution should be attempted. After all the real
purpose of project work lies in achieving development progress beyond theattribution gap (e.g. poverty reduction).
In 2002 a new contractual framework (named ‘AURA’ after its German
acronym) has incorporated this ‘orientation on impacts’ also in the contractual relations between GTZ and the Federal Ministry for Cooperation (BMZ).
These contracts (and the corresponding documents) need to specify thedevelopment objectives and results to be achieved. They must be jointly
agreed with the partners, who are – together with the external agent (e.g.
GTZ) – responsible and accountable for their achievement. But the operational details are deliberately left out of these contracts and delegated to thelevel of the supported project or programme (in line with a recursive management philosophy). In practice this has lead to much shorter documents,
less prescriptive designs and more flexibility in implementation.
I have incidentally taken the basic ideas of this ‘orientation on impacts,’
blended them with elements from Outcome Mapping and adapted them tosuit the needs and requirements of EU Structural Fund programmes. I havelabelled this impact-led monitoring approach “Process Monitoring of
Impacts,” as it is essentially about identifying processes considered relevant
for the achievement of effects, and monitoring whether these processes arevalid and actually take place. It is a theory-based approach that makes use oflogic models, transforms them into “circular” ones and deals with them in asystemic manner (Hummelbrunner 2007).
–4.2. Capacity WORKS
The ‘orientation on impacts’ approach specifies what should be achieved, but
27

CHAPTER 1

leaves open how this is to be done. To this end GTZ has developed a new
management model called Capacity WORKS 6. This model was also intended
as a response to changes in the aid architecture and stakeholder landscapes,
i.e. the delivery via programmes (instead of projects) and the need to steer
them in supra-organisational cooperation systems (beyond managing theGTZ contributions). In short, to better handle the increasing complexity ofdevelopment work.
An important role of GTZ is to assist in steering development projects, i.e.
establish and secure functioning cooperation frameworks. Projects and programmes are conceived as cooperation systems involving many partners. But
in supra-organisational cooperation systems none of the participating entities
is legitimated to “lead” these co-operations in the traditional sense, sincedecisions are not based on hierarchies. Rather these systems are based onnegotiation processes and take place within political processes. Therefore
Capacity WORKS should help to identify, focus and work through the relevant processes to achieve negotiated, agreed and measurable results.
In elaborating this new model, GTZ could build on its long tradition of
methodological development, using the experience gained and incorporating
elements from its past repertoire of methods. In addition, a survey was carried out on a series of programmes that were particularly successful in thepast. In-depth interviews with key staff identified several success factors thatrenowned experts and researchers further validated. These empirical success factors were to form the core of the model, which were then complemented by a set of tools to assist in working with them in practice.
Capacity WORKS is a management model to support contract and cooperation management especially during the implementation phase, though it isalso suitable for the project appraisal and preparation phases, as well as theconcluding phase. The model is designed to guarantee the necessary coherence with existing procedures and tools such as project preparation, approvalof the offer concept, project progress review, etc. Moreover, Capacity
WORKS is GTZ’s management model for sustainable development and“translates” this corporate policy concept into practical questions.
The key points of reference for Capacity WORKS are the objectives andresults jointly agreed with partners. The agreement of objectives and results
should therefore be understood as a fundamental process of political negotiation, and structured accordingly. Results chains are vital tools in this process.
628“Capacity WORKS” is protected as a word mark both in Germany and internationally.

Beyond logframe: Critique, Variations and Alternatives

They describe how activities based on inputs provided by the partners,
including the German ones, are to generate outputs and, ultimately, sustainable results. Subsequently all decisions should be measured against theseobjectives, and monitored accordingly during implementation.
Capacity WORKS operates with five success factors. These are central tothe approach and serve as a methodological guide for contract and cooperation management. The success factors are also instrumental in project andprogramme design. They provide the framework for negotiating the projector programme with partners. The success factors applied in Capacity
WORKS are based on the European Foundation for Quality Management(EFQM) model, adapted to meet the specific demands inherent in steering
complex projects and programmes:
The five success factors of Capacity WORKS and their key messages are(GTZ 2009):
1. Strategy: Negotiate and agree on the strategic orientation. A clear andplausible strategic orientation leads to positive results.
2. Cooperation: Network people and organisations to facilitate change. A
clear definition of who the project/programme will be cooperating withand how, leads to positive results.
3. Steering Structure: Negotiate the optimal structure. An effective steering
structure leads to positive results.
4. Processes: Manage processes for social innovation. A clear understanding of the key strategic processes leads to positive results.
5. Learning and Innovation: Focus on learning capacity from the outset.
Individual and organisational learning capacities in all success factors
lead to positive results.
The concept and action in each success factor are guided by key questions. And the model includes an extensive management toolbox, which is
described in detail in an internal Manual. Each of the 40 tools contained in
the toolbox is assigned to one of the five success factors and their key questions.
Incidentally, the logframe (called ‘Project Overview Plan’ in the ZOPP tradition) is also part of this toolbox. The logframe has been retained because it
is still used quite widely within the GTZ community and continues to be amethodological ‘anchor’ for many (older) staff members. And because it isuseful for planning, provided it is applied with care and embedded in a rangeof complementary tools. It is therefore included as a planning tool, intended
for translating strategic orientations into implementation plans.
29

CHAPTER 1

Most of the tools originate from practice in organisational development,
process or (systemic) management consulting. Although all of the tools havebeen tried and tested, they need to be adapted to the context and the stakeholders concerned. Thus Capacity WORKS provides a structuring framework, but does not relieve the user of the need to perform some conceptualand creative work. The model does not provide blueprints, but is intended tocreate scope for action – and provides a menu which is proven and based on
experience.
One of the expected benefits is to promote communication on the level of‘how’ the GTZ works, both internally (e.g. in teams, with partners) and externally (e.g. with other donors, in cooperation forums). It is expected that each
officer responsible for contracts and cooperation will be able to provide
soundly articulated and differentiated responses to the questions raised in
the context of the respective success factors. This should support and guidethe dialogue on the quality of GTZ services.
The experience gained in a pilot phase (2007 and 2008) in which Capacity
WORKS was tested in more than 60 projects, was used to further refine themodel into its present form. Since 2009 Capacity WORKS has been gradually
introduced in all GTZ projects and programmes worldwide. Initially by training all expatriate staff and selected national counterparts and then by training
external experts/consultants who can then use Capacity WORKS only after
they have been licensed to do so. Capacity WORKS will eventually become
the standard management and steering model of GTZ, at least for all BMZfunded projects and programmes.
OEAR Regionalberatung has been involved in the pilot phase and is alsopart of the training team for the roll-out phase of Capacity WORKS, which isstill going on. Having worked on this myself as a trainer, I was able to experience the enormous interest of the trainees and their overwhelmingly positive
response to the new model. To my knowledge this is, to date, the most thorough move of any major international aid organisation beyond the LFA tradition and certainly a massive investment for ‘retooling’ GTZ towards a management style which is expected to cope more effectively with the complexities of future aid delivery.
However, in order to unleash its full potential it will be important thatpartners as well as other donors will also buy into this model, and thus resonate – and reinforce – GTZ’s pilot efforts. Although Capacity WORKS has
been tailor-made to suit the needs of GTZ and German development aid, its
underlying principles, success factors and tools could be transformed and30Beyond logframe: Critique, Variations and Alternatives

adapted to fit with other contexts or needs, as well.

Concluding remarks
LFA is an analytical and management tool which is now used (in one form or
another) by most donors, aid agencies and NGOs. Despite formidable arguments which have been raised against its use, when intelligently applied (i.e.
taking into account its conceptual limits and conducive context conditions),
LFA can be an effective tool and the logframe remains the most common format for planning interventions. Apparently LFA continues to be useful formany people and organisations in development aid, especially those underpressure to demonstrate performance, assure effectiveness and accountability.
Although deeply flawed, LFA offers a middle path, as it is a component of
results based management but also allows for intense stakeholder participation, at least at the planning stage. And it seems to provide some degree ofcertainty amid the messy realities and insecurity of aid delivery. Despite
numerous attempts no other approaches have been sufficiently convincing to
aid managers to replace LFA as alternatives for planning and monitoring.
Until now few organisations (GTZ being the most thorough and promising
exception) have managed to make a radical departure from LFA.
The use of LFA is least controversial in simple situations, where interventions can rely on proven tracks and can be modelled in a linear, mechanisticmanner – although even under such conditions it should not be handled in ablue-print style. In complicated situations, involving multiple actors or
dynamic contexts, however, LFA can only be applied with much caution andshould be complemented with other methods that are better suited for these
conditions. Furthermore, in complex situations, displaying recursive causality or emerging outcomes, LFA is best not used at all, as its fundamental
assumptions are not appropriate and LFA will not work under such conditions.
Therefore it should be possible to base the utilization of LFA on rational
choice and sound evidence. And nowadays sufficient variations - and evenalternatives – are available to avoid inappropriate application. In other words:
a frame exists for a more ‘logical’ use of logframe – provided this frame isnoticed and used.

31

CHAPTER 1

References
GermanGTZ (2004) Wirkungsorientiertes Monitoring: Leitfaden für Vorhaben derTechnischen Zusammenarbeit. Eschborn: GTZ Stabsstelle 04,
Unternehmensentwicklung.
GTZ/BMZ (2003) Handreichung für die Bearbeitung von AURA Angeboten.
Eschborn.
Hummelbrunner, R. (1997) Die Logik hinter dem “Logical Framework” –Stärken und Schwächen des Planungs- und Evaluierungskonzepts der EU.
Paper presented at ÖIR Plattform “Evaluation der Evaluationen”, Feb. 26
1997, Vienna.
Schwaninger, M. and Körner, M. (2001a) Systemisches Projektmanagement:
Ein Instrumentarium für komplexe Veränderungs- und
Entwicklungsprojekte. St. Gallen: Universität St. Gallen – Institut für
Betriebswirtschaftslehre.
＿＿＿＿ (2001b) Komplexitätsbewältigung in Entwicklungsprojekten: Ein systemisches Instrumentarium auf der Grundlage des St. GallenerManagement – Konzepts. Unpublished manuscript.

EnglishAmbrose, K. and Roduner, D. (2009) A conceptual fusion of the Logical
Framework Approach and Outcome Mapping, OM Ideas Paper No. 1.
Bakewell, O. and A. Garbutt (2005) The use and abuse of the Logical
Framework Approach A Review of International Development NGOs’
Experiences. Sida.
Baumfeld, L., Hummelbrunner, R., Lukesch R. (2002) Reforming StructuralFunds: Towards a New Model of Policy Making based on Systems Thinking.
Vienna: ÖAR Regionalberatung.
Beer, S. (1979) The Heart of Enterprise. New York: John Wiley.
Chambers, R. (1997) Whose Reality Counts? London: Intermediate
Technology Publications.
Davies, R. (2005) Moving from Logical to Network Frameworks: A modular
matrix approach to representing and evaluating complex programmes＿＿＿＿ (2009) The Use of Social Network Analysis Tools in the Evaluation ofSocial Change Communications. Communication for Social Change
Consortium.
Earl, S., Carden, F., Smutylo, T. (2001) Outcome Mapping: Building Learning
and Reflection into Development Programs, Ottawa: IDRC.
32

Beyond logframe: Critique, Variations and Alternatives

Eggers, H. (1998) Project Cycle Management Revisited. The Courier, MayJune 1998, Brussels: European Commission.
European Commission (2004) Project Cycle Management Guidelines. Aid
Delivery Methods, Vol. 1. EuropeAid Cooperation Office, Development
DG
Gasper, D. (1997) ‘Logical Frameworks’: A Critical Assessment. Working
Paper no.278, The Hague: Institute of Social Studies.
＿＿＿＿ (2000) Evaluating the “logical framework approach” – towards learning-oriented development evaluation. Public Administration andDevelopment, vol. 20, 1728.
GTZ (2009) Capacity WORKS: The Management Model for Sustainable
Development. Eschborn.
Hummelbrunner, R. (2001) Use and misuse of the LFA in PHARE and PreAccession Programmes. Unpublished manuscript.
＿＿＿＿ (2007) Process Monitoring of Impacts – Applied Study for theEuropean Territorial Cooperation Programmes, Vienna: INTERACT(http://www.interact-eu.net/interact_studies/63)
Reuber, M. and Haas, O. (2009) Evaluations at the GTZ. Eschborn.
Schwaninger, M. (2006): Intelligent Organizations: Powerful Models for
Systemic Management. Springer Verlag.

33

Systems Thinking and Capacity Development in the International Arena

2
Systems Thinking andCapacity Developmentin the International Arena
Bob Williams
Drawing on an earlier OECD/DAC paper, the EuropeAid discussion document, “Institutional Assessment and Capacity Development: Why, what andhow?” proposes the following definitions of capacity and capacity development 1:“Broadly taken, capacity can be defined as the ability to perform tasks
and produce outputs, to define and solve problems, and make informed
choices. Capacity development is the process by which people and organizations create and strengthen their capacity over time.”
An uncontroversial and common sense statement, you might think.
In contrast, here is something I wrote several years ago:
Think of a water tank whose purpose is to store water for irrigation purposes. The capacity for irrigating the land is stored in that tank. The fuller
it is the more capacity it has for providing sustenance. However, the capability to irrigate successfully depends on more than the capacity to store
water. It depends on climatic conditions, historical rainfall, the presenceof something or someone to turn the tap on and off at the right moment,
the quality of the tap and water distribution networks. That’s what affects
the capability of that amount of water to achieve its objectives.
All the skills, knowledge, technical expertise and experience in the world1

Institutional Assessment and Capacity Development Why, what and how? (2007) Office for Official
Publications of the European Communities, Luxembourg.
http://ec.europa.eu/europeaid/multimedia/publications/documents/tools/europeaid_adm_concept_
paper_en.pdf, accessed 2 June 200935

CHAPTER 2

won’t help an evaluation if the capability of the program, community,organisation or environment cannot sustain and nurture those skills andabilities.
What I generally see at the moment, is capacity building (i.e. building
large storage containers) without a great deal of capability building (i.e.
the complex and strategic business of getting adequate water on the
ground at the appropriate time). If this continues, I believe we are in danger of being all dressed up and nowhere to go.
Not everyone distinguishes between “capacity” and “capability” as above.
Some use the distinction but switch the definitions and some use completely
different words for “capacity” and “capability.” My point is, however, that distinguishing between the theoretical ability to do something and the actual
ability to do something is useful.

What is thinking systemically?
Although a single, widely agreed upon definition of “thinking systemically”
does not exist, a working definition could be, “using systems concepts inaddressing situations.” This definition, however, raises the question of what
constitutes “systems concepts?” and although no single understanding exists
this paper provides a framework that accommodates most of the accepted
systems concepts. In some ways identifying what “systems thinking” is not –or not entirely is easier. It is not, for example, a wiring diagram with lots ofboxes with even more arrows and lines drawn between those boxes. It is also
not holistic in the sense that it includes everything. Most systems thinkers
acknowledge that nobody can think about everything and even if they could
it would be of little practical use. You simply can’t take everything into
account, so don’t begin to try.

Systems thinking
So if systems thinking is not primarily about boxes, lines and arrows and notabout thinking about everything then what is it about? And how can it help us
think about capacity and capability?

36

Systems Thinking and Capacity Development in the International Arena

From systems to thinking systemicallySome history will help. The systems field as we know it today developed inand around the Second World War. That war posed some very tricky, seemingly intractable problems at the individual, team, organisational and institutional levels. Most of these had to do with capacity and capability – theachievement of complex strategies with very limited resources. The pressurefor increased capacity and capability not only promoted the development ofsystems ideas, it also spurred important innovations in organisational development, group dynamics, and action research. Indeed these ideas often overlapped and intermingled with the development of the systems field; this is
nicely illustrated in a recent compendium of system thinkers (Ramage &Shipp, 2009). The history of the systems field is thus rooted in addressing
complicated and complex problems with limited time and with restricted
information. Over the past 50 years the systems field has expanded from its
relatively modest beginnings into a suite of perhaps 1,000 or more methods
and methodologies. Its orientation towards addressing capacity and capability issues however, has remained throughout.
So what, at core, might thinking systemically be about? This article identifies one way based on three core concepts. All three concepts have deeproots that stretch back many years.

Three core systems concepts: Relationships,
Perspectives and Boundaries
During the 1960s and 1970s the focus of the systems field was very much oninter-relationships. In many ways this was the wiring diagram stage ofthinking systemically. By the mid-1970s it was clear that the inter-relationships, whilst important, were not neutral concepts. The relative importance ofparticular inter-relationships depended on the different purposes you could
ascribe to any single situation. Thus thinking systemically began to address
the implications of applying different perspectives to the same situation.
However, by the mid-1980s it was clear that these perspectives were also notneutral. Perspectives determined what was seen to be relevant or not relevant and what lay “in” the system and what lay outside it. Whoever defined
the dominant perspective controlled the system’s boundary. Thus the importance of studying boundaries and critiquing boundary decisions (and those37

CHAPTER 2

who made them) became the third core concept underpinning systems
approaches.
These three concepts are essential both for understanding systemic interventions and for distinguishing systemic interventions from other approaches for dealing with complex situations. These three concepts underpin all themodels, metaphors, methodologies and methods used in the systems field.

Inter-relationships
Although the systems field draws on many methods that focus on interrelationships it tends to address five main questions:• What is the nature of the inter-relationships within a situation?
• What is the structure of these inter-relationships?
• What are the processes between them?
• What are the patterns that emerge from those processes, with whatconsequences and for whom?
• Why does this matter? To whom? In what context?
Partly because “inter-relationships” is the oldest systems concept it is themost familiar. Questioning how things are connected and with what consequence stems from the earliest thinking about systems. Inter-relationships isalso the concept most strongly embedded in the popular imagination. For
example, when we talk about the filing system, or the healthcare system, theimage we have in our minds is of a set of objects and processes that are interconnected in some way. The popularity of system dynamics, with its boxes
and arrows, further cements the notion that inter-connection is an importantsystems concept.
The study of inter-relationships is central to any systemic inquiry. In particular, systems approaches look at the following aspects of inter-relationships:• dynamic aspects (where the way the inter-relationships affect behaviourof a situation over a period of time)• non-linear aspects (where the scale of “effect” is apparently unrelated tothe scale of the “cause”; often but not always caused by “feedback”)• the sensitivity of inter-relationships to context (where the same intervention in different areas has varying results, making it unreliable to translate a “best” practice from one area to another)• massively entangled inter-relationships (distinguishing the behaviour of38

Systems Thinking and Capacity Development in the International Arena“simple,” “complicated” and “complex” inter-relationships)

Two systems methods that highlight inter-relationships
System Dynamics is a method that seeks to explore the consequences of nonlinear relationships and delay. It is usually, although not always, used in conjunction with computer simulation.
System Dynamics addresses the following questions:• How does “delay” impact the performance of the situation?
• How do patterns of feedback affect the behaviour of a situation?
• What controls the way in which resources flow through the situation? How does this affect performance?
Results chains and process models often assume cause and effect relationships that are relatively sequential, linear and non-recursive (i.e. A leads
to B leads to C). For example, we might understand “capacity” development
as being the consequence of training (A), which leads to increased knowledge (B), which leads to employment (C).
A

B

C

In contrast, System Dynamics acknowledges that the relationship is circular and the arrows may go in either direction.
A

?

?

B

C
?

So training (A) might increase knowledge (B), and this knowledge may
increase the demand for further training (A) which leads to greater knowledge (B). In this relationship capacity will increase rapidly. However, it also
allows for the possibility that knowledge (B) may lead to people gaining
employment in the field (C), which might reduce their ability to engage infurther training (A) because they are now out in the field. On top of thisbehaviour there may be delays in response that further complicate the picture. So whilst A might affect B quite quickly, B may affect the demand for A
39

CHAPTER 2

more slowly than C affects the demand for A. Thus in these scenarios, whilst
the capacity of the situation may be enhanced – at least initially (i.e. moretraining, knowledge, employment), over time the capability of the situation
reduces.
Viable System Modelling (VSM) is based on the idea that different “levels”
within a situation need different kinds of information to function effectively.
Viable System Modelling addresses the following questions:• What are the operational, co-ordination, management, strategy andgovernance needs of the situation in order to deliver on its purpose?
• What information is needed at each level of the situations to achieve
the purpose?
• How does information flow through the situation?
• Is the right information available at the most appropriate level of asituation’s hierarchy of tasks?
The model has five levels or “systems”:
System 1

The entire collection of interacting operational units within an organisation.

System 2

The system responsible for stability/resolving conflict between operational units.

System 3

The systems responsible for optimisation/generating synergy between operational
units.

System 4

Future plans and strategies. Adaptation to a changing environment. The overallsystem’s link to the environment.

System 5

Policy. Sets system rules and direction.

In diagram form:

5
4
3
2

2
1

1

1

1

1

1

E
n
v
i
r
o
n
m
e
n
t

Source: Williams & Hummelbrunner (2010)40

Systems Thinking and Capacity Development in the International Arena

Jon Walker (see references) describes each of these categories in the following way:“First of all you need the working bits. This is System 1, the operational
units. System 1 is the bit which actually does something. It’s the muscles, theengine room, the machines, the producers. Secondly you must ensure that
there are ways of dealing with conflicting interests which are inevitable in theinteractions which occur as the parts of System 1 interact. Conflict resolution
is the job of System 2. System 2 is also given the job of ensuring stability.
Once the interactions of the System 1 units are rendered stable, it becomes
essential to look at ways of optimising these interactions. This is the job of
System 3. System 3 works with an overview of the entire complex of interacting System 1 units and thinks ‘If this one does this and that one does that,then the whole thing will work more effectively.’ The extra efficiency is calledsynergy. System 3 is there to regulate System 1 – its function is optimisation.
Once you have a stable, optimised set of operational units, then you mustensure that it can survive in a changing environment. This is the job of
System 4. System 4 looks at the outside world [environment], considers what
it sees, and looks for threats, opportunities and schemes. System 4 is there to
produce plans to ensure long term viability. And finally, the whole thing must
function within some sort of overall context. Everyone must be pulling in thesame direction. This is System 5’s job. It provides the ground rules and themeans of enforcing them to ensure that the system in complete. System 5
provides the ultimate authority.”
VSM is commonly used to understand why monitoring and information
management procedures are ineffective. An organisation’s capacity is thesum total of all information possessed by an organisation. Each of the fivesystems needs information to function. However not all of the information
required by one system is generated inside that system – it makes demands
on other systems for information. So each system has to handle its own information needs as well as the information needs of other systems. While people in each system will understand the information needs of their own systemthey may not understand the information needs of the other systems.
Furthermore they will often lack the motivation to ensure that information
required by other systems is relevant and timely.
For instance, take a refugee agency. Workers in a refugee camp providing emergency food supplies (System 1), will be committed to keeping
records of the amount of rice in store, but may not see the relevance of information requests for the ethnicity of food recipients. Yet at a strategic level41

CHAPTER 2

this information may be useful to the refugee agency as a means of assessing
the political conditions that will determine their approach to refugee camp
placement planning (i.e. System 4)
Bad information sloshes around the organisation and thus threatens its
viability. In other words, capacity does not match capability. VSM is thus
used as a tool to help organisations balance capability and capacity and toensure that the right information is generated in the right place, supplied to
the right people at the right time to allow them to take the right decisions.
For instance, once upon a time there was a small development agency
dealing with local health issues. The staff, clients and funding agencies werein close contact with each other. Generally speaking everyone was part of all
five systems, although it was probable that System 4 was largely ignored.
Suddenly, along comes HIV/AIDS and large sums of money from an international foundation. The organisation expands from 5 people to 50 people, thedemands from the donors become more complex, the response from theenvironment: less predictable. The old way of management by discussion
and consensus breaks down, everyone no longer can know everything that isgoing on, generalist tasks become specialist. At this point organisations often
collapse, unable to respond to the new dynamics and needs. VSM provides ameans of addressing these issues, by exploring, for instance, who needs tobe trained to do what; how do you meet the competing information demands
without overwhelming the task?

Perspectives
The systems field draws on a number of approaches for exposing andexploring perspectives:• What are the different ways in which this situation can be understood?
• How are these different understandings going to affect the way inwhich people judge the success of an endeavour?
• How will it affect their behaviour, and thus the behaviour of the system, especially when things go wrong from their perspective? With
what result and significance?
A systemic approach is more than a study of how boxes and arrows fit
together or how information networks operate. Just looking at interconnections does not make an inquiry “systemic.” What makes it systemic is how
42Systems Thinking and Capacity Development in the International Arena

you look at the picture, big or small, and explore interconnections. Whenpeople observe inter-relationships they will “see” and interpret those interrelationships in different ways.
People participate in a project for many different reasons. Think of yourown involvement in the international capacity development field. How manydifferent ways of seeing your involvement are there, and how do they affectthe kinds of decisions you make? What you may regard as a situation able todeliver adequate locally resourced economic initiatives (i.e. large capacity,
medium capability for economic development), someone else may regard ascompletely ignoring women’s rights (i.e. large capacity, low capability forsocial development). These interpretations, these motivations and the behaviours that flow from these different perspectives may have little or nothing to
do with the formal goals or objectives of a project or program. The program
may have indeed been primarily about economic development. Yet the
expectation by some key players for social development will affect how the
program performs and ultimately affect the results.
Thus we cannot comprehend the behaviour of a program, in this casecapacity development, without identifying and understanding a wider rangeof perspectives. Perspectives help to explain and predict unanticipated behaviours because they give us a window into motivations. They also draw ourattention to consequences: unplanned and unintended. Towering above thisis the need to acknowledge that people make programs work, not someimagined “logic” like a logframe dreamed up by a funding agency.
The introduction of “perspectives” as a core systems concept was profound. First, it highlighted the notion that a situation can be “seen” in different ways, and this will affect how you understand the system. Whilst thatidea isn’t inherently radical, most of the attention outside the systems fieldhas been how different stakeholders hold perspectives that are different fromeach other. However, the systems field holds that different stakeholders may
share the same perspective and most importantly any one stakeholder willhold several different perspectives, not all of which will be compatible witheach other. For instance, I’ve never held a single unified view on any project
I’ve been involved in. How I handle a situation – shall I give my $5 donation
to this person on the street – will be the result of a complex set of internalarguments and trade-offs that can change in the time it takes for me to reach
into my pocket. In other words, understanding perceptions is important at abehavioural level. Perceptions are more than how people look at a situation;these perceptions generate behaviours that impact on the way the situation
43

CHAPTER 2

operates. Yet the management theories that dominate the international development world tend to force us to pick one and pretend that it’s the one thatshould motivate everyone. And then we wonder why things don’t quite workout as we planned. In terms of capacity and capability, the key question thatflows from this discussion on perspectives is not whether there is capacity
within a situation, but what that capacity has the capability of doing.
Second, perspectives importantly drew the focus away from the “system”
as it supposedly exists in “real life” and allowed us to consider alternatives:
what it might be like, could be like, or even should be like. Or how different
people imagine how it might be like. This opened up the systems world,
because not only could you draw conclusions based on a study of the world
as it is, but you could also compare alternative perceptions of what they think
it is with what actually is, or with perceptions of what is or with what might be.
The similarities and differences between what is and what might be create
puzzles and contradictions which can stimulate deeper learning.

Two systems methods that highlight perspectives
Soft Systems is a methodology that first forces you to consider alternative perspectives (e.g. development as “aid,” development as “patronage,” development as a “tool of foreign policy,” development as “empowerment”). It thenasks a series of questions that help you work out what the structure, function
and logical consequences are of each perspective.
Soft Systems poses the following questions of a situation:• What are the different ways in which the situation can be viewed?
• How does each of these ways express and give meaning to whatsome people within the situation value?
• How are these perspectives reflected in people’s motivations andbehaviours within the situation?
• How does each of these ways affect the performance of the situation?
For instance, take an HIV/AIDS program in India. If you perceived development primarily as “aid” then you may well focus primarily on ensuring theavailability of skilled medical care sourced from wherever that skill may be.
Your understanding of capacity will be how well that skill is coordinated withthe local situation. However, if you perceived development from an “empow44

Systems Thinking and Capacity Development in the International Arenaerment” perspective, you would be more concerned about progressively
building up the skill from the local base. Your use of externally sourced
expertise would be on a different basis; your notion of capacity would bequite different.
Of course “reality” is always a mixture of perspectives. Soft Systems does
not force you to pick one perspective over the other, or even prioritise them.
Instead what it encourages you to do is separate out these perspectives, workout the implications of each and then assess how best to integrate them in a
way that they work for rather than against each other.
Activity Systems is an approach closely associated with the idea of communities of practice. A community of practice is a set of mutually agreed uponand shared activities directed towards a common purpose. Activity Systems
add two other factors: the tools we use and the motivations that drive us
towards a goal. It seeks to address problems associated with different motivations towards goals and how to constructively handle shocks to the system.
Activity Systems addresses the following questions:• What fundamentally are the motivations underpinning the achievement of a goal?
• What tools, rules and roles are necessary for that motivation to betranslated into goal directed activities?
• How does the system handle contradictions in tools, rules, roles
and motivations so that the goal is achieved?
For instance, both of us could be providing HIV/AIDS counselling to fulfil a goal about reducing the incidence of HIV/AIDS. However, your motivation for providing counselling may be women’s empowerment, whereas mymotivation may be to complete my counselling qualification accreditation.
Most of the time these two motivations will not contradict each other.
However, something might occur – perhaps a tricky ethical issue – that bringus into conflict. How does one handle such a situation constructively or creatively rather than destructively? Activity Systems provides a framework forassisting that process. Activity Systems approaches enable people to engage
constructively in resolving the tensions that arise when circumstances
expose situations where people are engaged in the same activities but for different reasons. So whilst a conventional capacity development perspective
might focus on the quality of the actual counselling, an Activity Systems
approach would be in developing the capability to handle conflicts construc45

CHAPTER 2

tively.

Boundaries
A systemic approach to boundaries will help you navigate through complexity by posing the following general questions:• How is a situation being framed?
• What does this imply about the way in which the situation can be
managed or investigated?
Boundary setting is not optional. We make situations manageable by settingboundaries.
Every endeavour has to make a choice between what it includes and what
it excludes. A boundary differentiates between what is “in” and what is “out,”
what is deemed relevant and irrelevant, what is important and what is unimportant, what is worthwhile and what is not, who benefits and who is disadvantaged. Boundary setting also determines: how we approach a situation;
what we expect from it and what methods we might use to manage it; what isdeemed relevant and what is not; and which perspectives are honoured andwhich perspectives are marginalised. Boundaries drive how we “frame” situations. We manage something “framed” as a simple situation quite differently
from a complex one; our definitions of what constitutes a simple situation andwhat constitutes a complex situation are core boundary decisions.
In evaluation terms, boundaries are the sites where values get played out
and disagreements are highlighted. A lot of power issues are wrapped up inboundaries – just as the person with the magic marker controls what goes on
the whiteboard, the person whose perspective dominates a project decides
the boundaries. Capacity development in the international arena is full ofboundary decisions: who gets what kind of resources for what purpose andwhose interests are marginalised. Indeed, as we shall see in a couple of paragraphs even the notions of “capacity” and “capability” are themselves boundary choices.
As suggested earlier, during the mid-1980s, questions were being asked
within the systems field about how boundaries are set, who sets them andwhat the consequences of setting boundary are. Whilst it’s fine to map relationships and it may be fine to acknowledge that there will be different perspectives on those relationships, those relationships and perspectives are notneutral – someone somewhere decides which relationships and perspectives
46

Systems Thinking and Capacity Development in the International Arena

are the most important.
Once it was fully acknowledged that thinking systemically about perspectives and inter-relationships involved boundary choices, many in the systems
field started taking a deliberate and often debated approach to boundary
identification and selection.

Two systems methods that highlight boundaries
Critical Systems Heuristics (CSH) is a set of questions that helps draw out
debates on boundaries.
Critical Systems Heuristics addresses the following questions:• Who or what is being excluded, marginalised or made a victim by
the way in which this situation is being viewed or is operating?
• What does this say about what is “valued,” by whom, in this situation?
• What are the consequences of boundary setting decisions? How cannegative effects of such decisions be mitigated?
Critical Systems Heuristics involves exploring four aspects of boundaries:• Purpose/Values: Whose interests are being served and whose interests should be served?
• Resources/Control: Who controls what resources and who should
control what resources?
• Knowledge/Expertise: What expertise is required? Who do we trust
as experts and what expertise should be required; what’s the risk ofassuming this is all the expertise needed?
• Legitimacy: Whose interests are being excluded, marginalized or
harmed by the way we are framing the situation and whose interests
should be excluded, marginalized or harmed?
Let’s consider an HIV/AIDS prevention project that used peer education
methods to increase the degree of control sex workers have over theirclients by improving their knowledge about HIV/AIDS and its transmission.
In parallel, the project used microfinance methods to allow sex workers todevelop alternative income streams, and to become less economically dependent on clients (and thus able to be more assertive with their knowledge).
The evaluation of this project documented some deep systemic problems
that could be more clearly understood by considering “boundary” issues. In
47

CHAPTER 2

terms of “purpose/values,” the project seemed in many ways to be driven
more by the interests of the religious organisation that funded the project
than by the interests of the sex workers. In terms of “resources/control” theproject was dominated by an external agency that controlled most of theresources but did not have the internal capacity to process and monitor theloans and repayments. Thus loans were slow to be made and largely unrepaid – violating two of the basic tenets of microfinance. In terms of “knowledge/expertise” – the project critically misunderstood the expertise needed
to negotiate with clients. The sex workers actually knew much more than the
educational model assumed, in fact the key issue was a resource issue (lackof condoms) not an educational issue. Finally, in terms of “legitimacy,” keyinterests were excluded (e.g. hotel owners and traditional money lenders),
who actively worked against the project.
Although capacity development touches on all four of these categories(purpose, resources, expertise, legitimacy), issues of capability focus particularly on expertise. Capacity and capability are intricately bound up with
notions of expertise – and Critical Systems Heuristics poses some very challenging notions about what assumptions are being made about expertise;what expertise is regarded as relevant (or irrelevant) and who should havethat expertise.
Cynefin (a Welsh word roughly meaning “place of multiple belongings”)
is a systems-based framework that distinguishes (i.e. draws a boundary)
between four aspects of a situation.
These four aspects are:• simple situations that deal with the “known” (i.e. the aspect of the situation is self- evident and uncontested)• complicated aspects that are about the “knowable” (the aspect of thesituation is knowable with sufficient investigation and expertise)• complex (the aspect of the situation is unknowable in advance but
knowable in the present)• chaotic aspects of situations (the aspect of the situation is completely
unknowable)
Each situation implies a different set of knowledge and management
capacities, and suggests different ways to make those capacities available.

48

Systems Thinking and Capacity Development in the International Arena

COMPLEX
Probe
Sense
RespondCOMPLICATED
Sense
AnalyseRespond

CHAOTIC
Act
Sense
Respond

SIMPLE
Sense
Categorise
Respond

Source: Williams & Hummelbrunner (2010)

Simple aspects of a situation imply very little capacity development –things can just be borrowed from elsewhere and applied with very little modification (i.e. sense, categorise, respond). Complicated aspects of a situationneed capacities that involve research (to find out where an intervention has
been used before) and adaptation (i.e. ability to understand how context willaffect application). In Cynefin shorthand – sense, analyse, respond. Complex
aspects require capacities that are close to action research – an ability toreflect critically on practice, identify patterns and modify processes (i.e.
probe, sense, respond). Chaotic aspects need immediate action (e.g. triage)
to stop the patterns and then consider what to do (i.e. act, sense, respond)
Thus the capacity required to manage a “simple” aspect of a situation is
likely to be inappropriate for managing a complex aspect of a situation. In
terms of capacity development, Cynefin challenges the idea that there is a listof capacities and capabilities that can enable all aspects of a situation.
Cynefin thus addresses the following questions:• How are we framing the situation?
• What are the implications of this framing for how we manage a situation?
• What are appropriate ways of managing a situation on the basis of
this framing?
Referring again to HIV/AIDS projects, a simple aspect of a project may be
the HIV testing process itself, whereas a more complicated aspect may be
the treatment of those who have moved to an AIDS diagnosis. A complex
aspect may be the medium to long term management of those with AIDS
within their own communities, or the process of encouraging people to come49

CHAPTER 2

forward for tests. Activities in the complex zone are likely to be highly sensitive to local conditions and relatively unpredictable in outcome.

Thinking systemically about capacity development
The reassessment of international development over the past decade, thestruggles over the Millennium Development Goals (MDGs), the acknowledged failure of results-based management methods (United Nations, 2008) 2
and the shift toward capacity development at a national rather than at projectlevels poses many challenges to existing capacity development concepts. In
particular it highlights the need to distinguish carefully between capacity andcapability. This distinction raises many inter-relationship, perspective andboundary questions. These include at what scale (national, trans-national,
local) should capacity development interventions be conceived and assessed,
who should be the primary beneficiaries, who or what could be harmed bythat choice, can tension be resolved, what expertise is considered necessary
to provide adequate capability and who should control what resources. Theseare questions that the systems field is well equipped to address.

Systems thinking and systems methods
Choosing systems methods ...
There will be times when the high level of abstraction in the three basic
dimensions of systems thinking is inadequate. You need to dig deeper. That’s
when selecting from the wide range of systems methods becomes important.
In which case, how do you decide between the methods?
In many ways it is personal judgement or deciding which approach feels
right. In many ways it is a pragmatic call, which methods do I know best, do I
feel I can learn the quickest or can I find someone with expertise.
However, there are more systematic ways.
One way is to consider the kinds of questions that a systems methodaddresses. For each of the six examples in this document there are a set ofquestions most closely associated with that method. Which of these questions most closely relate to the situation that concerns you?
Another way relates to the nature of the situation. For instance, this2

50

United Nations (2008) Review of results-based management at the United Nations. Report of the Officeof Internal Oversight Services. 22 September 2008.
[http://www.un.org/ga/search/view_doc.asp?symbol=A/63/268].

Systems Thinking and Capacity Development in the International Arena(much debated) framework by Mike Jackson, Robert Flood and Paul Keys
(for a summary see Midgley, 2000) focuses on the structure, dynamics andpurpose of the situation.
The dynamics of the situation and its context
Is the situation unitary?: There is agreement between key stakeholders.
In which case methods that are essentially technical and oriented towards
inter-relationships are appropriate. These would include System
Dynamics and Viable System Modelling.
Is the situation pluralist?: There are disagreements between stakeholders.
In which case methods that are focused on perspectival issues are appropriate. These would include Soft Systems, Activity Systems and Cynefin.
Is the situation coercive?: There are power dynamics that seek to control
the way in which the situation is perceived and handled. In which casemethods that – with care – can be used to expose that dynamic. Critical
Systems Heuristics is claimed by some to be an appropriate approach.
Activity Systems might also be a contender as well as Cynefin.
The purpose of the intervention
If you wish to predict or seek to bring a situation under control: use
approaches such as System Dynamics.
If you wish to involve people in reflective debate so that learning may befacilitated: use approaches such as Soft Systems or Cynefin.
If you wish to expose underlying assumptions to ethical critique or otherforms of non-technical critique: use methods such as Activity Systems or
Critical Systems Heuristics.
Midgley (2000) and colleagues developed a mixed methodology
approach. Rather than picking one method or methodology to address a situation, this approach uses a range of different systems approaches. For
instance, at the start of an intervention you may wish to understand the
dynamics of a situation, so use System Dynamics, or you may want to know
how information is flowing around a system, so use Viable System
Modelling, or how the situation has handled sudden shock and contradictions (i.e. Activity Systems). Once you have got this you may want to set
some boundaries around the intervention you are intending for this situation– and thus use Critical Systems Heuristics as a means of working out how
best to frame that intervention. This may result in a set of different framings
that Soft Systems may help form the platform for discussion around possible
51

CHAPTER 2

resolutions problems associated with the situation. Finally in terms of intervention you may want to determine whether to treat some aspects of the situation as complex or complicated or simple. At this point you are beginning toclose the gap between thinking about systems methods and thinking systemically.
... or thinking systemicallyThinking systemically is a matter of capacity development itself. There isknowledge to be acquired, skills to be gained, learnings to be acquired – andopportunities to be sought to apply all of these. However, it is also critically
about capability. Over the years running workshops on systems thinking, acommon (and realistic) comment has been that the ideas are great, but theorganisation people work with would never tolerate the kind of questions systems thinking poses. Under such conditions the question arises about whereto start.
Generally speaking the best place to start with is where you are right now(Midgley in Williams and Imam, 2007). Do the notions of focusing on interrelationships, perspectives and boundaries help you improve your own
understanding of capacity development? Will addressing the questions undereach of the main headings of this paper help you puzzle your way through aproblem, or bring constructive light onto an issue? If you have answered“yes” to any of these questions, then you have probably found the best place
to start.

＊＊＊
This article is based on a shorter article in Capacity.Org (http://www.capacity.org/en/journal/feature/thinking_systemically) as well as previous writings by the author and contributions from Gerald Midgley, Richard
Hummelbrunner, Amy La Goy, Iraj Imam, Martin Reynolds and Glenda
Eoyang. In various forms it has formed the basis of workshops, lectures andarticles, including the American Evaluation Association, the University ofWageningen(http://portals.wi.wur.nl/navigatingcomplexity/?Report) and The Broker(http://www.thebrokeronline.eu/en/articles/Bucking-the-system).

Useful References
Checkland, P. and Poulter, J. (2006) Learning For Action: A Short Definitive
Account of Soft Systems Methodology, and its use for Practitioners, Teachers
and Students. Wiley.
52

Systems Thinking and Capacity Development in the International Arena

Flood, R. and Jackson, M. (1991) Creative Problem Solving: Total Systems
Intervention. Wiley.
Irwin, J.S. (2000) Business Dynamics: Systems Thinking and Modelling in aComplex World. McGraw-Hill.
Kurtz, C. F. and Snowden, D. J. (2003) The new dynamics of strategy: Sensemaking in a complex and complicated world. Cognitive Edge: http://www.
cognitive-edge.com/
Midgley, G. (2000) Systemic Intervention: Philosophy, Methodology, andPractice. New York: Kluwer Academic/Plenum Publishers.
Ramage, M. and Shipp, K (2009) Systems Thinkers, Springer.
Reynolds, M. (2008) Reframing expert support for development management. Journal of International Development. 20.
Williams, B. and Hummelbrunner, R. (2010) Systems Concepts in Action : A
Practitioners’ Toolkit. Stanford University Press.
Williams, B. and Imam, I. (eds) (2007) Systems Concepts in Evaluation. An
Expert Anthology. EdgePress.

Some On-line Resources
Flood, R. Rethinking the Fifth Discipline: Learning Within the Unknowable.
http://www.ebookmall.com/ebook/132488-ebook.htm
Systems Dynamics : http://wwwu.uni-klu.ac.at/gossimit/linklist.php
VSM : Walker, J. An introduction to the Viable System Model as a diagnostic& design tool for co-operatives & federations, see
http://www.esrad.org.uk/resources/vsmg_3/screen.php?page=1qguide(checked 28/6/09)
Soft Systems Methodology – http://www.bobwilliams.co.nz.
Go to the section titled “systems stuff” in the sidebar menu.
Centre for Activity Theory and Developmental Work Research; University of
Helsinki
http://www.edu.helsinki.fi/activity/pages/chatanddwr/activitysystem
Critical Systems Heuristics: Werner Ulrich’s homepage:
http://www.geocities.com/csh_home/
Cynefinhttp://www.cognitive-edge.com

53

Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –3
Using Systems Concepts in Evaluation– A Dialogue with Patricia Rogers
and Bob Williams –Edited by FASID (1)As Mr. Williams explained in the previous article, more than 1,000 systems
tools exist which can make it even more difficult for newcomers in this field
to understand how systems concepts can actually be applied to development
evaluation. Therefore, in this section we have asked veterans Dr. Patricia
Rogers and Mr. Bob Williams to explain how we can use systems approachesin evaluation.

1. Thinking about Systems and Evaluation
FASID: Thank you both very much for taking your time to participate in this
dialogue. I read Dr. Rogers’ fascinating article (2) about using program theory
to evaluate complicated and complex interventions which also suggested possibilities for using systems thinking to tackle some difficulties in development evaluation. I have also learned from Dr. Rogers that Mr. Williams is
now writing a book on this topic with Dr. Richard Hummelbrunner. I am sureit will be a great resource for evaluators and we are very pleased that we are
getting a preview of it today. But, before getting into the details, would you
mind telling us how you encountered this field of systems concepts and evaluation?
Bob: I was originally trained as a biologist, and worked as a research ecologist for a number of years. I became mixed up in the environmental movement in the early 1970s, and ended up as a community worker. During thattime I became associated with the “Systems Group” at the Open University in
the UK. So I went from an environmental focus to a social focus via a systems
focus. After a number of years working as a community worker, I began asking myself the question, “Is this doing any good?” What I did not know at the55

CHAPTER 3

time was that I had wandered into an area that I did not even know existed –evaluation. So here I was in the mid 1990’s posing evaluation questions on
the basis of my physical and social science backgrounds with a systems orientation.
Patricia: My background was in political science, with a Ph.D. in evaluation.
I started teaching in RMIT (3) University’s Evaluation Program in 1989. It wasan extremely unusual evaluation program set up by Jerome Winston whose
disciplinary background was in physics. He had studied general systems theory at MIT and had found that the theory provided a useful framework forevaluating human service programs. He developed a graduate program in
evaluation that was, I believe, unique in having several compulsory subjects
in systems approaches and systems thinking. Although the systems field wasan area I had not formally studied, I discovered that a key feature of systems
approaches was developing and using conceptual models appropriately,
remembering that “the map is not the territory.” This resonated with mytraining in political science.
Bob

Community Work
Environment
Ecology

Patricia
Systems
Political Science
Approaches
Evalution Models
toQuantitative& Qualitative
Evaluation
Research Models

FASID: So, you are working together in this combined field of systems andevaluation. Why?
Bob: We are not doing this just because the systems approach is interesting,but because we believe that systems concepts provide something that canmake evaluation better.
FASID: When we talk about “systems,” what exactly are we talking about?
Bob: I’m afraid you will not find an agreed definitive definition of systems in
the literature. It’s a bit like evaluation. In the evaluation field, you will neverfind a single agreed definition of what is and what is not evaluation. People
argue all the time over what it is; it’s the same in the systems field. That’s not
to say that people haven’t tried to define what a system is, but in more recent
times the emphasis is less on what a system is and more on what taking asystemic viewpoint implies. Derek Cabrera distinguishes between “thinking
about systems” and “thinking systemically.” I think that is a useful distinction.
56

Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –FASID: Would you explain the difference between “thinking about systems”
and “thinking systemically”?
Bob: OK, here’s an example. Peter Checkland defines a system in the following way. It is a situation that displays the following properties:• An ongoing purpose (that may be determined in advance – purposeful, or assigned through observation – purposively)• A means of assessing performance• A decision-taking process• Components that are also systems (i.e., the notion of sub-systems)• Components that interact• An environment (with which the system may or may not interact)• A boundary between the system and the environment (that may beclosed or open)• Resources• Continuity
So thinking about systems, would be exploring only situations that display allof these properties. The problem is that relatively few situations display allthese properties. And even when they do, you find that people often draw different conclusions about them. In contrast, thinking systemically doesn’t
require agreement about whether or not a situation is a “system,” but
explores a situation using principles that underpin systems ideas, revealing
insights that would otherwise be hidden from view.
FASID: Then, what are those principles?
Bob: Many people have attempted to identify these. What we conclude fromour different experiences within the systems field and within evaluation isthat the principles of systems thinking can be expressed by three concepts.
One of them addresses “inter-relationships.” The second is the concept thatdifferent people see different things when you explore inter-relationships –this concept is known as “perspectives.” Finally, the third concept, which I
personally think has enormous implications in terms of evaluation, is that it isfine to have multiple perspectives, but isn’t it interesting that somebody’s perspective always seems to win out – that there is always a dominant perspective? This leads to a focus on “boundaries” the third distinction that can bemade in the systems field.
FASID: So, you are saying that by applying these three concepts – inter-relationships, perspectives and boundaries – we can improve evaluation from its
planning to its final analysis?
Bob: Using these three concepts can improve evaluations – and for any sin57

CHAPTER 3

gle evaluation they can also help to make its limitations clear. These systems
concepts highlight certain aspects of a situation which an evaluation could
then examine. Taking a systems approach to evaluation is not about looking
at everything, its about being aware of what to look at, what can be ignored
and what are the implications of looking at one aspect and ignoring another.
Systems thinking provides a powerful framework for making those decisions.
Patricia: These systems principles don’t only have implications for a single
evaluation. They raise questions about how we organize evaluations within
and across organizations. There are real limits to what you can do in any single evaluation. For example, requiring lots of small project evaluations can bea tremendous waste of resources especially if they are done too soon or too
quickly to adequately report on results. It would be better, in my opinion, toreduce the scale of the evaluations required of all projects or programs, andput more evaluation resources into better understanding a few key projects
or programs. People working in development evaluation need to think morestrategically about the protocols and requirements. Systems thinking canhelp with this as well.

2. Three Core Concepts – (1) Inter-relationships
FASID: You mentioned that the systems approach has three core concepts;
would you please explain them briefly one by one?
Patricia: Sure, I’ll start with “Inter-relationships.” Evaluators love inter-relationships. But while we draw lots of inter-relationship diagrams, many of our
methods such as program logic and logframe don’t actually explore the real
depth of those inter-relationships and what affects them.
For example, if we were evaluating a program based around the proverb “anapple a day keeps the doctor away,” many evaluators would develop a logic
model that went something like this:INPUTS – Apples

PROCESSES – Deliver apples

OUTPUTS – Apples eaten

OUTCOME/IMPACT – Improved healthIf we measured each inter-relationship, and addressed the issue of causal
58

Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –attribution, we might be able to conclude whether or not the program had
worked, but we would have no idea how it worked, or why it didn’t work, andwe would not have enough information to be able to transfer this evidence toa new situation or improve a program that wasn’t working.
If we were taking a systems approach, we would explore the relationships
between eating and better health – for whom, in what circumstances andhow does it work? For example, the program might work very well for young
children who don’t have access to fresh fruit, by increasing their vitamin andfibre intake, or it might work well for overweight people by encouraging
them to substitute an apple for a high-calorie snack, or it might increase theintake of Quercetin which may reduce inflammation, including influenza andheart disease. Each of these possible explanations would lead to impacts ofdifferent types of health improvements, and would have implications aboutother situations to which it could be generalized, and possible ways of adapting it. If it works through vitamin C, then oranges would be effective if apples
were not available. Alternatively, if it works through Querecetin, red onions
would be a better substitute than oranges.
FASID: I see. That means inter-relationships are something we should think
about at the stage of framing the evaluation?
Patricia: Yes, and we can also use it at the analysis stage. Thinking about
inter-relationships should be considered an analytical tool to make sense ofexisting data, and to work out in advance what data can be collected in thetime available for adequate analysis.
FASID: Do you differentiate between intended and unintended paths from
input to goal? And what about unintended goals?
Patricia: I think you should at least consider or investigate these different
paths, as this may also help you to understand any unintended outcomes,
especially negative ones.
Bob: This actually leads onto the next concept about “perspectives.” If I poseyour question from a “perspectives” viewpoint, I would ask “Unintended bywhom?” So even the language we use like “intended” and “unintended” consequences, reveals something about the perspective taken about a particularintervention. An unintended consequence from your perspective may very
well be a specifically intended consequence for somebody else. I sometimes
say there is no such thing as an unintended consequence – someone, somewhere, intended it. I know my rhetorical sleight of hand is not universallytrue, but it does challenge us to think about perspectives – and especiallythose we use in the design and analysis of evaluations.
59

CHAPTER 3

3. Three Core Concepts – (2) Perspectives
FASID: So, the next core concept is “perspectives.”
Bob: Understanding a variety of perspectives is useful for both practical reasons and ethical reasons. Often it’s not practical to understand what is really
going on by seeing things from only one point of view. A soccer match ending in a riot cannot be explained if you see a soccer match solely from theperspective of a game of skill between two teams. An ethical reason forunderstanding perspectives is the possibility that someone or a certain group
could get harmed if you don’t see things through an alternative perspective.
Seeing a particular development issue solely through the lens of women’s
empowerment, for instance, may mean the negative consequences on social
cohesion are ignored. Therefore, it is important to see things not justthrough one pair of glasses.
FASID: We all understand the importance of considering different perspectives. But, isn’t it a challenge to come up with all the possible perspectives
within a given time span?
Patricia: Yes, that is a very good point. We can’t identify or investigate all
possible perspectives, but we can identify some important perspectives. Then
we can use these different perspectives when deciding what data to collect,
or at least when interpreting the data we have. I know it is sometimes hardwhen we are very constrained not just by time but also in terms of what are
seen as legitimate perspectives. But if I am looking at something with different perspectives, I can get a better understanding of why something works ordoesn’t work – and how it can be fixed.
Bob: I would also say this is not about getting things right. You can’t look at
everything and we will get to boundary setting in a minute to further addressthat. What we are trying to do is to introduce you to a new approach that,given the resources you have, can enable you to make more powerful judgements of worth. Perspectives and worth are very closely linked. You mightjudge the worth of a soccer match differently if you perceive soccer as agame of skill or a means of entertainment. A soccer game played by incompetents could be judged hugely entertaining, whereas a game played skillfully
could be judged very dull. So, it is not about going through an enormous
process to get things right. It is about being skilled on knowing the rightquestions to ask.
FASID: How should we find different perspectives?
60

Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –Bob: Well, there are several strategies to think about multiple perspectives.
One is to think about the motivations of those involved in a project. Thisimplies a focus on stakes as well as stakeholders in an evaluation. Clearly different stakeholders have different perspectives. However, one single stakeholder can have multiple perspectives depending on their <<stakes>> andtheir motivations. If we were evaluating the workshop we’ve just finishedfrom a multi-perspective point of view, most evaluators would put Patricia andI in the <<facilitators>> stakeholder grouping. However looking at things froma <<stake>> point of view opens up other ways of understanding how the workshop went. Our stakes in the workshop included our reputation as evaluatorsand the workshop as a source of income, as an opportunity to visit Tokyo, as
an opportunity to work together, and as a means of rolling out the idea of systems ideas into the international development arena. How we managed thosedifferent stakes influenced the seminar in a range of ways. And those stakes
were driven fundamentally by our motivations. So a good place to startchoosing perspectives to explore is to ask <<what are the motivations thatstakeholders are bringing to the project and which motivations will play a
key role in how the project performs?>>
Patricia: Another strategy is to work backwards from a result, whether goodor bad. And the third strategy is to think about something from a formal
philosophical viewpoint, such as a Marxist view, a feminist view, a Buddhist
view, an environmentalist view, and a social capital view.
For example, an HIV/AIDS prevention program in Africa could be seen froma human rights perspective, a public health perspective or a poverty reduction perspective. Suppose the improvement of economic conditions (by providing alternative sources of income) is considered to lead to the empowerment of commercial sex workers (who can then negotiate with clients overcondom use) and which can lead to the reduction of HIV transfer in the community. If poverty reduction is achieved but the program does not lead to a
reduction of HIV transfer, the project might be considered a success from an
economic perspective but not from a public health perspective.
Bob: This I think also helps us understand that perspectives are a way inwhich we <<frame>> things, and how we frame things will influence our ability
to resolve issues. For instance, an American manufacturing company thatbuilds machinery for large infrastructure projects was in constant industrial
relations strife because of the regular fluctuations in their workforce. They
often had to make large numbers of people redundant when orders dried up
and then cope with issues of hiring new workers when orders were high.
61

CHAPTER 3This meant they were constantly in conflict with their trade unions overredundancy payments and training issues. Framed as an industrial relationship problem it seemed unsolvable. They tried hard for a number of years
and failed. Finally they saw the issue through a different lens – as an economic cycle issue. As we know, the world economy goes up and down, and infrastructural projects tend to go up and down with local economic conditions.
By framing the problem as an economic cycle issue, rather than a labour
relations issue, the company came to realize that their product range was too
closely associated with one phase of the economic cycle. Framing the problem as an economic cycle issues allowed them to ask an important question –with the current workforce skills and the plant can we manufacture something that is counter-cyclical, so that when demand for infrastructure construction is low, we can switch to an alternative product? They did find an
answer to this question; creating a stable workforce on the way and resolving
their industrial relations problems.
Patricia: That’s a nice example of how reframing a problem can help to solve
it. It may be interesting to think about this case study when you are involvedin evaluation; those who appear to be negatively affected may not necessarily
be so depending on how the project is framed.
FASID: It is always useful to look at things from a different perspective. How
can we incorporate this in our work?
Bob: For example, you could include groups of people who have different
viewpoints in your interview list. And when you include them, some question
techniques are very helpful.
For example, seek an overall judgement of whether or not the projectworked, and then explore what they mean by “worked.” You will discoverthat they might have different ideas about what the project was trying to do.
Ask them directly about their motivation. Why are you involved in this? What
are you hoping to achieve? You are enlarging your interview group a little bit,and you are making sure there are open-ended questions in there. Thatwould be great.
Patricia: Another point to think about is, what are the consequences of notconsidering a certain perspective? You cannot take into account every single
perspective. It is just not possible. You are going to have to draw a line somewhere. That boundary may be drawn by your client or it may be drawn byyou. When you draw that boundary, when you choose to have only one or
two perspectives or perhaps just the client’s perspective, you need to think
through the possible consequences of that decision. If you think the conse62

Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –quences of leaving out other perspectives are significant, then you should
see if it is possible to include them. If they are not, why bother to write themin a report? But one thing I have learned is that I take personal responsibility
if I was aware of possible consequences but did not mention them. It is a professional judgement call you take on the basis of having at least explored or
begun to explore some of the possible consequences.

4. Three Core Concepts – (3) Boundaries –FASID: This brings us nicely to the third core concept – boundaries.
Bob: Right. Boundaries determine what is “in” and what is “out,” what is considered important or unimportant, or relevant or irrelevant. As I said earlier,
it is good to explore different perspectives but it is not possible to look at
everything. There is no such thing as holism; every human endeavor has tohave a boundary around it. Now many in the systems field say very strongly
that simply setting boundaries is not enough; when you set boundaries, youhave an obligation, an ethical obligation, to consider the impact of settingthat boundary.
Patricia: This is particularly important in the development field where there
is a constant struggle over setting boundaries and determining and identifying any negative consequences. For example, if you only include women in a
micro-credit scheme, men who have been excluded from the project might
be unsupportive and seek to destroy it.
FASID: Indeed. By the way, in terms of the boundary of an evaluation, it isoften set by the client so don’t we have to work within that boundary?
Bob: Well yes and no. I tend to get clients to think about the boundaries
they’ve set for the evaluation and the practical consequences (positive andnegative) of that decision. I generally find that although clients have often
thought about these things, the conversation and decisions could have been
better thought through. Once that’s been done then the key is doing the best
we can within whatever boundary has been set. Of course this process is justas relevant for setting project boundaries as for setting evaluation boundaries.
FASID: How do you set the boundaries? If, for example, we are thinking
about a certain group of people who are negatively affected by a development
program, from whose perspective do we set a boundary? From one perspective, we may say “they are negatively affected,” but that may not be true fromanother viewpoint...
63

CHAPTER 3

Bob: Absolutely. So, the argument is that if you are trying to understand how
a program works, you have to understand that tension. There will be people
in the program who may not even consider, for instance, “girls” or “the environment” an important aspect of a rural economic development program.
Thinking about boundaries is a way of getting a bit deeper into the understanding of how a particular project works.
Going back to your question about how you <<set>> boundaries, I think it isalso important to understand how you consider boundaries. What are you
looking for and what do you do when you find one? At this point I have tomention Critical Systems and its role in identifying and exploring boundaries.
I think Critical Systems deals with issues constantly confronting evaluation
but which evaluators have few ways of addressing and therefore I’d like it tobe much better known in the evaluation field. Churchman who developed
Critical Systems firmly believed three things.
Firstly, that every world-view is terribly restricted, thus systems thinking
starts when you see the world through the eyes of another. Secondly that any
judgement of human activity erects a boundary between “worthwhile” (i.e.
“in”) and “marginalised” (i.e. “out”). Finally because, as systems thinkers, weare obliged to see the world through the eyes of others, then with every
boundary we set we have to consider the implications of setting that boundary for others. This has enormous implications for evaluation in the developing world, since evaluators often take the judgements of “worth” of donors as
the only basis for judgement. At least formally they do. The reality is different. When I talk to evaluators working in the development field I find they
are acutely aware of these boundary judgements – but there are few established approaches in evaluation to help them work this through. The systems
field, which has a strong bias towards the resolution of opposites, has quite a
lot of approaches to address this. Critical System is one of them. A guy called
Werner Ulrich took these ideas and developed them into a tool – Critical
Systems Heuristic – which dives very deeply into these issues. It’s a tool thatmany evaluators and their clients could benefit from using – perhaps morethan any other tool from the systems field.
Patricia: The important point that Bob just made is that you have to draw
the boundaries. We can’t look at everything, but what we must do is to consider the consequences of drawing boundaries and whether we need to mitigate the consequences of the boundaries we have chosen.
Bob: In the paper on capacity building (4), I explore several important boundary issues. These include the purpose of a project and who is likely to benefit
64Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –from it, the amount of resources available to a project, the way in which necessary expertise is framed and what you do about the inevitable decision to
exclude certain interests (such as “girls” or “the environment” from a rural
economic development project). These are all extremely important boundary
decisions that profoundly affect whether a project ends up being worthwhileor not (and to whom). By surfacing these boundary decisions you are able tosubmit them to critical analysis. If they are submerged you never do. For
instance, how many projects have you been involved in where assumptions
about the necessary expertise required within the project proved to be woefully wrong? Or the fact that excluding certain interests created resistancethat in turn reduced the impact of the project?

5. Applying Systems Approaches to Development
Evaluation
FASID: So, we can start “thinking systemically” using these three concepts.
Patricia: Yes, and for application to development evaluation, different systems tools and methods can be utilized. For instance, “Critical Systems
Heuristics” is oriented towards thinking about boundaries, “Soft Systems”
about perspectives, “Systems Dynamics” about inter-relationships, and“Cynefin” which blends all three core concepts.
FASID: How do you know which systems tools to use for what kind of evaluation?
Patricia: In the paper on capacity development, Bob has laid out a very helpful guide indicating which tool can be used under what circumstances. This
guide includes some specific questions that relate to how to determine therelevant systems tool to use. For example, specific questions for the systems
method “Cynefin” would be the following;• Does the situation display simple, complicated, complex or chaotic
inter-relationships? From whose perspective? With what consequences?
• What patterns have developed within the situation? What have been
the likely generators of those patterns?
Bob: So identifying the questions you want to ask may be a way of identifying which tool to use. You may think “Here is a list of questions – which tool
will allow me to address those questions best?” The capacity paper lists thosequestions. It also suggests a couple of other approaches to selecting methods, but in the end it is really about which approach you feel you understand65CHAPTER 3

best. Even though we have said that Soft Systems is strongest when dealing
with perspectival issues, it doesn’t mean that it is only good at that.
Remember in order for a method to be considered a systems method it has
to address all three components; inter-relationships, perspectives and boundaries. The questions I’ve linked with the systems methods came out of myexperience in using these methods.
Patricia: And the questions may be specific to a particular situation.
Bob: Absolutely. So reading about and understanding the tools may give you
the set of questions. Or you can work at it the other way around; being confronted with the set of questions will help you select the specific kinds oftools that are the most relevant. For instance, I think questions about “legitimacy” and “expertise” highlight two areas where evaluation is relatively
weak. Historically these are areas that evaluation does not go, they are justnot part of the evaluation tradition. Deeper exploration of some of theseboundary issues (e.g. what expertise is considered relevant and what is not)may lead evaluation into new and tough areas. This idea of exploring the consequences of ignoring those who may be disadvantaged by a project is something that I’m increasingly hearing in the evaluation field. As evaluators
become more accountable for what they do to a wider range of people,
accountability becomes absolutely evident. And boundary questions raise allkinds of accountability issues – such as an evaluation’s (or project’s) legitimacy. Legitimacy to whom?
FASID: What do you mean by legitimacy in association with evaluation?
Bob: It is really asking, in whose eyes is this project (or indeed its evaluation) a legitimate thing to do? Who may not be convinced about that andwhat are the consequences of ignoring those viewpoints? That is the sort ofpragmatic practical side of asking those questions. But there is also the ethical side; is this project – or this evaluation – the right thing to do?
Patricia: In evaluation, we often focus on intended outcomes, whether wewere able to accomplish the objectives. As you can see from the diagram
below, unintended outcomes and negative outcomes are two separate concepts. Sometimes we can have unintended positive outcomes. Unanticipatedoutcomes are a subset of unintended outcomes, and I think an important partof evaluation is to try to have enough openness to be able to see these outcomes that were not expected, as well as the “intended but negative fromsome perspectives” outcomes.

66Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –Unintended＋ (Positive)

Intended＋ (Positive)

Unanticipated−(Negative)− (Negative)

FASID: So, evaluators are there to set appropriate boundaries under timeand resources limitations. Is it a judgement call?
Bob: An informed judgement call. As an evaluator I do the best I can withinthose boundaries. I’m informed by an awareness of the consequences of setting those boundaries, and attempt to mitigate any negative consequences of
those boundaries. So the issue is, “What are the consequences of thoseboundaries and how can we make the best within those boundaries?” Take“time” for instance. There are also various techniques that we can use towork around a time limit. For example, many of our jobs take place in highly
complex and contested areas. In such cases getting adequate representationof those issues in our data set is a big challenge even before we start thinking
systemically. For instance, traditional unstructured interview-based representative sample surveys are hugely resource and time intensive. So to ensurethat I am able to gain as many relevant perspectives but in a short space oftime, I may consider using interviewing methods that allow respondents to
get to core issues very quickly. For instance, there is a method called“Convergent interviewing,” developed by an Australian called Bob Dick (5)that can generate and assess a wide range of perspectives very quickly andwith limited resources.
Patricia: Now, the traditional way of doing evaluation is that we spend a longtime planning, then we collect data, do the analysis and then we report.
However, very often, the planning part, the data collection part and the analysis part take longer time than we think, and the most important parts ofanalysis and reporting get squeezed in at the end. So, another way of getting
around this time boundary, introduced in Michael Patton’s UtilizationFocused book (6), is “Reality Testing,” where the process of “planning-datacollection-analysis-reporting” is done in a series of very short cycles. This is
another useful method in conducting evaluation time-efficiently.
FASID: When you are deeply involved in one particular perspective, thenevaluation seems to be extremely difficult. Especially in the case of internal
evaluation by donor agencies.
67

CHAPTER 3

Bob: That’s a risk, but again just keep with the mantra – inter-relationships,
perspectives and boundaries, just keep posing those dozen or so core questions and you shouldn’t go too wrong.
FASID: You say that in evaluation, evaluators have to draw boundaries andthey should look into different perspectives as much as they can, thinking
systematically and sometimes using systems tools. Could participatory evaluation play some role in it?
Bob: Many in the systems field would argue that systems approaches are
inherently participative. How else can you ensure that you are dealing withthe appropriate and most important inter-relationships, perspectives andboundaries? Participatory evaluation raises the issue of stakeholders, and I’dlike to remind you of the comments made earlier about the distinction
between stakes and stakeholders. As I said earlier, every person has multiple“stakes” in a project no matter what “stakeholder” grouping they fall into.
Evaluation generally tends to assume that stakeholders have only one“stake,” a very substantial over-generalisation and often an erroneous one.
A focus on stakes as well as stakeholders is a core part of the Critical
Systems Heuristic tool I described earlier.
FASID: The Japanese translations for both “complex” and “complicated” arethe same. Could you elaborate on the difference in meaning as they relate to
Cynefin?
Bob: I know this will sound familiar by now, but different people will give
you different answers. For me the key difference relates to knowability ofoutcomes. You know you are dealing with a complicated situation if theresult of an intervention is difficult to predict but fundamentally knowable inadvance. The situation is a complex one if, no matter how much research you
do and no matter how much you try to follow good practice, the result isn’tknowable in advance. In a complex setting you can only really understandthe result of an intervention after the intervention. One of the big mistakesthat evaluators make is to try to and understand and evaluate complex settings in which the results of the intervention are not knowable using toolsthat are more suited for complicated settings, in which the results of theintervention are knowable – at least in a general way. Obviously context matters. What’s knowable in one context may not be knowable in another.
Patricia: My take is similar. If an intervention can be adequately represented
by a logic model, showing a predictable sequence of cause and effect, thenthe intervention is being understood as either simple or just complicated. If
an intervention is truly complex, we can’t predict in advance the cause and68Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –effect relationships, but need ongoing processes of learning and adaptation.
Some people use the example of raising a child as a complex activity becauseeven when you think you have done it successfully before, you might have achild that is really different to your first child, and you are going to have to
see what works with them and adapt and respond. These are specific uses of
the word ‘complex’ – it doesn’t just mean “difficult.” Sometimes people use
the word “complex” as a fancy way of saying that something is “difficult”: thiscan be confusing.
Bob: The reality is that most projects that we work on have aspects in bothzones (complicated and complex) and they are very much intertwined. Thinkof Patricia’s child raising example. Some aspects of raising a child are complicated and some are complex – and as a parent you handle these aspects indifferent ways. So one of the things that I often find useful to do is to untangle the two – often using a framework known as Cynefin. Then in evaluation
terms we can say – how well did a project handle the complex parts of theintervention and how well did it handle the complicated aspects.
FASID: Using systems tools, how do you strive to make a good evaluation?
Patricia: I guess one way I do this is to use some of the ideas from Critical
Systems to examine who is making the key decisions about evaluation andwhat are the consequences of those decisions. You can’t just end up representing your evaluation client, and excluding other perspectives that havenot been looked at. Another way is to consider simple, complicated and complex aspects of evaluation practice. There are some simple rules in evaluationthat are true everywhere – like keeping commitments to your client, keeping
them informed, being truthful. There are some complicated aspects, wherecertain methods are appropriate in certain contexts, and where expertise isneeded to be able to match the methods to the situation. And there are sometimes complex aspects, which cannot be totally planned in advance, where
there needs to be a process of doing something (in the Cynefin frameworkthat Bob mentioned this process is called a ‘probe’) and seeing the response.
There needs to be a process for documenting and reflecting on the evaluation itself and making adjustments if needed. This requires considerable
trust and partnership with the evaluation client, and is inconsistent with acontractual process where you have to set out the details of your evaluation
plan in advance and cannot change them later.

69

CHAPTER 3

6. Systems Thinking and the DAC Evaluation Criteria
FASID: As you know, Japanese ODA adopted the specific evaluation criteria
set by the OECD Development Assistance Committee (DAC) – relevance,
effectiveness, efficiency, impact and sustainability. And under each of these
five criteria, more detailed evaluation criteria are set. For example, “policy”
relevance is the foremost important issue in regards to the “relevance” criteria.
Bob: So, let’s unpick the boundary issue about policy relevance. A purely
policy orientation can exclude other factors that might help us understandhow a project works. I recall an evaluation where the focus of the evaluation
was supposed to be about the impact of a new funding policy on the management of community organisations. What we found was that whilst the newpolicy improved the management of community organizations, the greatest
impact was on the relationship between the funding body and the organizations. The relationship got a lot worse. We couldn’t get the policy people to
be interested in this, even though down the track it was likely to be far more
important to the sustainability of the projects than dodgy management. The
policy people were only interested in the intended change in management
practice; they had set a policy boundary and had no interest in either looking
over the boundary or even considering the implications of setting that boundary.
Patricia: We have to think also about non-participants in terms of relevance.
One of the issues is obviously that there are participants and “intended” participants, and one of the reasons intended beneficiaries do not participate inprojects is that they do not see the project as being relevant to them.
Bob: Certainly. And that’s one of the areas that I have become interested in.
If I’m working in a project then clearly that project is terribly important tome. It’s what I care about, it’s what I get paid to do, and it may even define
my role in the community. Because my focus is on that project, I care about
people benefiting from that project, and I look for them benefiting from the
project. Implicitly I’m assuming the project is important because people arebenefiting from it, so therefore it must be relevant to them. But we must ask
the question whether I have things out of proportion. Whilst I may think thebenefits are important and indeed people may be benefiting, I still need toconsider just how important they are in relation to all the other things thatare going on in beneficiaries’ lives. We have a very project-centric way of70

Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –looking at the world. Going back to Churchman’s comment – just steppingout and looking back at the situation from the participants’ point of view iscritical. Just how worthwhile is the project to them in the grand scheme ofthings? Could we have done something more worthwhile with our money
and their time?
Patricia: Yes, so the participants might see the project as being relevant but
not in the way that you expected it to be; what they are getting out of the project may not have been what you had anticipated. Asking some of the systemic perspectives questions may help you a lot for this criterion.
FASID: Another DAC criterion is sustainability. How can systems ideas help
here?
Bob: “Sustainability” is a concept that crops up frequently in the systems literature, although it is often framed as “adaptation” or “emergence.” I’m very
interested in “sustainability,” how would you assess that within your evaluation framework?
FASID: We look at how the effects of the project continue to be sustained
after the end of the project, usually looking at different aspects of sustainability such as policy aspects, organizational aspects, financial aspects, technicalaspects, socio-cultural aspects and environmental aspects.
Bob: I see. So here are some things to think about from a systemic perspective. One of the things that concerns a number of people in the systems fieldis that sustainability is often framed as meaning “staying the same.” The
underpinning assumptions are that the context is both known or is knowable
and is relatively stable. In other words once you’ve got the project right you
keep going and know enough about the environment to keep things on track.
However, especially in developing countries, not all environments are either
knowable or stable. In which case sustainability in terms of keeping things
the same is neither feasible nor arguably desirable. How many times in yourevaluations have you felt people were addressing last year’s context? The systems world tends to think of things more in terms of adaptability rather thansustainability – this may be what your evaluation criteria of “sustainability”
actually means. There is an entire field of systems called “complex adaptive
systems,” which asks “how do we understand how situations evolve overtime?” and “how do they duck and weave through the confusing and complicated environments that they work in?”Patricia: I think one of the big issues for us in the work we do, is for us to bevery clear about what it is that we want to sustain. Sometimes it is the project
but that is not always the case. Sometimes a strategy to sustain the effects of71

CHAPTER 3

the project would be to institutionalize its activities, but that will of course
need ongoing resources. Other times you may want it to be actually incorporated within a business as an upfront investment, or as an idea that people
can pick up and share easily, or some technical expertise.
Bob: Another sustainability aspect is the sustainability of the capacity to do
things. For example, although the original project to control malaria may
have disappeared you may now have the capacity to undertake other healthpromotion activities related to HIV/AIDS that you didn’t have before themalaria project. In this case it’s not the sustainability of the project, but thesustainability of the organization or the community capacity and capability toactually do something.
FASID: How can we incorporate the systems approach when using the logical framework (LogFrame) or project design matrix? Let’s take the “impact”
criteria for example. The impact is measured mainly by the overall goal, so
although we do look at other unexpected positive or negative impacts, thefirst place we need to look is the overall goal within the logical framework, so
we tend not to place so much importance on other impacts.
Patricia: Yes, but sometimes these unintended impacts can actually be more
important than the goal, whether positive or negative.
Bob: That’s right. Take my example of the policy intervention. Exploring
inter-relationships, perspectives and boundaries can help in assessing
whether these unintended impacts are likely to occur. And, no, you don’t
have to explore every possible inter-relationship, perspective and boundary.
What you are looking for are the likely impacts that are sufficiently big to beof concern.
Patricia: So, you are trying to include unintended outcomes, but how do you
do that in your design, within the framework of your evaluation? There’s amethod that Carol Weiss used called “Negative Program Theory (7),” where
instead of saying “Here is what we want to achieve and here is what we aredoing – how do we think it will work?” she asks “Here is what we are doing what is something bad that might happen and how might that happen?” Forexample, if we’re implementing a policy to increase teachers’ salaries so thatchildren will learn better, and if we apply negative program theory, we might
ask “What is a way that we could increase teachers’ salaries but end up witha result that children learn worse? How could that happen? ” So you set up acausal chain that is plausible and identify things that you might want tocheck on. You can redesign your intervention to reduce the risk or you mightjust want to track it and get early warnings. Negative program theory may
72

Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –help you to get evidence about unintended outcomes, if you can anticipate
them. However, it will not help you with the totally unanticipated outcomes.
We therefore need to build into our evaluations some ways of collecting
things we don’t expect – for example, by talking to different people with different perspectives, and just generally keeping our eyes open. As I said earlier, when you have a very limited viewpoint it is much harder, but just havingsome open-ended questions in the interview schedule may help you to catch
unanticipated outcomes.
Bob: It is certainly one of the traps within the project design matrix or
logframe. In a logframe for instance, the “means of verification” and the “indicators” are very much framed around the desired impact. You will therefore
be shining your data collection torch in a relatively narrow zone. Once you’ve
designed a logframe, then it starts to have a life of its own, and changing alogframe as they go, I know, is very difficult. I’m undecided whether
logframe is inherently unsystemic, but I think the way it’s used often is.
Patricia: Systems thinking may be effective here, when defining the outcome as well as when you are designing a logframe. Some questions to ask
may be: What does success require? What standard of performance would beconsidered a success? What would be credible evidence?
FASID: So summing up what would you say is a key <<take-away>> from thisconversation?
Patricia: If you genuinely ask questions, it will transform evaluation tremendously. Explicitly or implicitly, you should think about complicated and complex issues at key decision points in planning and conducting an evaluation.
Bob: Explore inter-relationships, perspectives and boundaries. And do itwith local people. That will increase legitimacy.
FASID: Thank you Dr. Rogers and Mr. Williams.

(1) This dialogue was composed jointly by Patricia Rogers, Bob Williams,
Nobuko Fujita, and Eriko Takahashi, drawing from material developed
over the course of numerous conversations and e-mail exchanges; material was also taken from the Q&A sessions held in December 2009 in connection with a FASID seminar: “Using Systems Concepts in Evaluation”.
(2) Rogers, P.J., 2008, Using Programme Theory to Evaluate Complicatedand Complex Aspects of Interventions, Evaluation, 2008;14;29, Sage
Publications.
(3) Royal Melbourne Institute of Technology
73CHAPTER 3(4) Ch.2 of this issue(5) Dick, R., Convergent Interviewing( http://www.scu.edu.au/schools/gcm/ar/arp/iview.html#a_iv_intro)(6) Patton, M. Q. 1997, Utilization-Focused Evaluation 3rd Edition. Sage
Publications, CA.
(7) Weiss, C. H., 1997, Evaluation 2nd Edition. Prentice-Hall, NJ.

74

ReportEnriching Evaluation with
Perspectives of Local People:

Two Cases of Collaborative Evaluation 1
Nobuko Fujita

Both the Paris Declaration (2005) and its follow up in the Accra Agenda for
Action (2008) address issues in the field of international development assistance evaluation and stress the importance of working through partner countries’ systems. Currently, however, so much evaluation work is still donor-driven and designed to meet the donors’ evaluation needs that the current stateof evaluation falls far short of the ideals as stated in international commitments (O’Brien 2009).
The situation is unfortunate since, once the project is completed, responsibility lies with the partner countries to make positive impacts sustainable
and to minimize any negative impacts. Providing partner countries with relevant information, therefore, should be one of the primary goals of conducting
evaluations. Donor-centered evaluation tends to “de-motivate those involved
from the partner side” (Lundgren & Kennedy 2009). Minimal partner
involvement, in turn, lowers their interest in utilizing evaluation results andconsequently, such evaluations become of little use to local decision-makers,
implementers and beneficiaries.
In addition, for partner countries accountability is an issue. The donor’s
purpose in conducting evaluation is two-fold: to learn from the experienceand apply any lessons learned to current and future projects and to show
accountability and fulfill their responsibility to explain to taxpayers how effectively their tax money was spent. Likewise, partner countries should utilize
evaluation to fulfill their responsibility to be accountable to their own citizens.

1

The author gratefully acknowledges the insightful comments made by Takako Haraguchi and Keishi
Miyazaki who have been working for evaluation capacity building in Vietnam for the last three years.

75Moreover, donor-driven evaluations might overlook important impacts
because they are generally geared towards the needs of a project’s funders
and follow pre-set evaluation criteria and checklists in guidelines. In the end,
international development evaluation comes down to determining what cooperation brought and what the people, especially local people or beneficiaries,
think about it. No matter how extensive and comprehensive a donor-driven
evaluation is, outsiders cannot feel the same way local people do. Conducting
evaluation with local people; therefore is effective to make sure that the project is evaluated from local people’s perspectives and to avoid overlooking
important aspects or groups of people who may be affected by the project.
For some time now, so-called “joint evaluations” have been ongoing.
Most joint evaluations, however, are still primarily donor-driven, and partnercountries’ participation in those attempts is limited 2. How can we move tomore collaboratively conducted evaluations which genuinely incorporate
beneficiaries’ perspectives and which enrich evaluations so that they cantruly help developing countries tackle the issues they are facing and makethem useful for future cooperation?
This report introduces two recent examples at more genuine collaboration in evaluation to see how such collaboration is useful for partners anddonors alike. Section one presents a joint ODA evaluation in Vietnam.
Section two explains an evaluation study conducted jointly with a Lao PDR
local government. In both cases, local participants’ perspectives enrichedevaluation considerably.

1. Innovative Joint Evaluation Approach in Vietnam
1. 1. Summary and process
The first case is a joint ex-post evaluation of the Japanese concessional loan
projects being conducted in Vietnam 3. It is different from conventional joint
evaluations in that the Vietnamese side (developing country) is playing amajor role in its implementation.
The 34 member evaluation team consists of four JICA consultants (two
2

3

76

Typical form of developing country’s participation in joint evaluation is that either reviewing/commenting on report drafted by donor side or some government officials joining an “evaluation committee” established by a donor.
Ex-post evaluation of the three Official Development Assistance (ODA) loan projects by Japan
International Cooperation Agency (JICA). The three projects are Hanoi Drainage Project forEnvironment Improvement (1) (2), Da Nang (Tien Sa) Port Improvement Project, and Hai Van Pass
Tunnel Construction project (1) (2) (3). Evaluation period is from September 2009 to August 2010.

Enriching Evaluation with Perspectives of Local People: Two Cases of Collaborative Evaluation

Vietnamese, two Japanese), three local evaluation experts, two local advisors
who were active participants in joint evaluations in previous years, and 25
members form three working groups (one group for each project). Amongthe 34 members, only two are from the donor side (consultants). The remaining members come from various Vietnamese Ministries, local implementing
agencies/companies, and local consulting firms.
The Vietnamese members developed the evaluation frameworks and survey questionnaires, and conducted the field study. From time to time, theJapanese consultants accompanying in the field study gave some advice, but
generally left Vietnamese members in charge of interviews and survey. The
team members are now drafting the evaluation report in English: the common language chosen for the report.
1. 2. Merit of this approachThis joint evaluation of Japanese ODA projects in which stakeholders from apartner country fully participate in the evaluation work is the first of its kind.
The team accepted me as an observer for the field studies, therefore I joined
two out of the three projects 4 and observed the following merits of thisapproach.
First, the team’s motivation in conducting evaluation was clear andstrong. When trying to collect data the local team members stressed that“their own government: not just the donor” needs to know the result of the
project. In a sense, without such motivation, the essence of collaborative evaluation may have been lost, and the evaluation might have become pro forma.
Also, when the interviewees were approached in that way, a slight change of
expression and attitude could be observed: interviewees’ typical defensiveness towards evaluators seemed to have faded.
Second, interviewees could respond in a more straightforward manner
because most of the team members were from their own country, and interviews were conducted in their local language. Asked what the City of Da
Nang would have been like now without all these projects, city officials
answered that the municipality would have invested their own funds
anyway 5.
Third, the interviewees’ frankness was reinforced by the local team mem4
5

The Hai Van Pass Tunnel Construction Project and the Da Nang Port Improvement Project. Only one
Japanese consultant was present in the field study of these two projects.
This would force the team to look for some particular benefits of investing in it with cooperation fromJapan. For example, technology transfer, capacity building during the projects.

77bers’ precise knowledge of the projects and the background of the project. In
terms of knowledge of the local context, donors’ consultants cannot competewith their local partners: no matter how competent they may be. For example, the local team members had detailed background information concerning sensitive issues such as relocating residents to build a port access road,therefore they could more naturally direct interviewees to reveal what theyknew. Gone was the tension or defensiveness experienced in typical ODA
evaluation, instead, openly discussing such situations among the different
stakeholders to see if they could do something better was just “business as
usual.”
Fourth, interview time was better utilized by, for example, minimizing theneed for courtesy calls which can sometimes eat away precious time during
interviews conducted by donor side evaluators.
Fifth, none of the Vietnamese team members was passive and held
almost complete ownership. On the last day of the field trip, when all thescheduled interviews were over, they knew exactly what they had to do
before starting to write the report: follow up data collection, undertake additional interviews and make confirmations.
Overall, this approach seemed to be quite successful in conducting evaluation in local context.
1. 3. Success Factors
First, this joint evaluation followed two years of evaluation capacity building
projects by JICA in Vietnam. Therefore, the Vietnamese side was ready toengage in full-fledged joint evaluation. The two Japanese consultants who
were in charge of the first two years of capacity building kept working in thethird year and by that time they had built-up a network and trust among
stakeholders.
Second, in Vietnam, institution building for monitoring and evaluation
had been enhanced through, among others, a two-year capacity building project as mentioned above and therefore Ministries and implementing institutions have generally been more and more conscious of the necessities of conducting project evaluations.
Third, the composition of the team was well thought out, possibly due tothe continuity of the Japanese members. The Vietnamese consultants who
participated in the evaluation capacity building projects in the first two years
joined the team as core members (among them, there were experts in social
research). All the concerned Ministries and project implementing institutions
78

Enriching Evaluation with Perspectives of Local People: Two Cases of Collaborative Evaluation

also joined the team. They could share all sorts of background knowledge
regarding the projects and their individual field of expertise.
Fourth, good teamwork also contributed to the success of the study.
Prior to the start of this joint evaluation, Vietnamese consultants and evaluation advisors participated in five days of evaluation training organized by theJapanese consultants. On top of that, four to five days training was offered tothe core members of the team. Through these processes, the purpose andthe significance of evaluation, evaluation methods and evaluation tools were
understood and ownership of evaluation was developed. Also, by working ina group throughout the training, they naturally were able to build a teamwhich exhibited excellent teamwork. (Luckily, this team even included ateam-building expert.)
1. 4. Challenges in Implementation
Of course, not everything was rosy and challenges abounded: both practicaland substantial.
First, the cost of sharing information is higher in these types of joint evaluations. An increase in the number of the team members increases the timeneeded to share the information collected every day as, ideally, all the data
gathered should be shared before proceeding to the next day. That meansincreasing the number of internal meetings required during the already-compressed field study schedule. However, this happens to any large-scale evaluation team. And as far as this particular case was concerned, the merits of alarger team composition seemed to have outweighed the demerits since datacollection work was efficiently divided among team members.
Second, language matters. Luckily, almost all the members of this particular team were English speaking. Had it been otherwise, the process of evaluation planning, making, using, and analyzing questionnaires, and drafting
the report, translating and revising it back and forth each time would havebeen time consuming and would have tested everyone’s patience. Actually,last year (during the second year of capacity building), according to the teammembers who conducted joint evaluations in two languages, too much timeand energy was spent translating everything back and forth.
Third, cost maters. The evaluation team had members from both Hanoi
and the project sites in Da Nang which is about 750km away from Hanoi.
Some members had to travel back and forth for the field study and meetingsnecessary for finalizing the draft report. For the Vietnamese entities thesecosts had to be justified legally and also practically.
79

Fourth, difficulties exist fulfilling donor’s requirements. For donor-funded
development projects, numerous requirements exist in regards to its evaluation. Explaining each requirement and ensuring all the team members pay
attention to them probably improves the quality of evaluation. On the other
hand, some of the detailed requirements might not even be relevant to theVietnamese side.
1. 5. Implications
The last point actually leads to a fundamental issue: how much deviationfrom guidelines can donors, or clients of ODA evaluations accept? Different
perspectives could change the direction of the study, the evaluation questions, and even the conclusion. Would a donor accept whatever evaluation
results were turned out or would the donor’s consultants have to rewrite thereport to strictly adhere to the donor’s guidelines?
Compared to conventional donor-driven evaluation, the standards andguidelines of joint evaluations may be more lenient. (This could mean thatthe purpose of a joint evaluation may have to be slightly adjusted).

(Interview at Da Nang People’s Committee. Vice Mayor (in front) is
giving authorization to respective person in charge (on the left) toprovide information required by the evaluation team (on the right)).

2. A Collaborative Evaluation Study in Lao PDR
2. 1. Summary and Process
The second case is a socio-economic impact study of the Second MekongInternational Bridge (hereafter FBII) in Lao PDR 6. This evaluation study was
conducted not as an official ODA evaluation but as a research program by
FASID 7. The purpose of the study was to gain a comprehensive understanding of the impact of the FBII by collaborating with a local partner: theDepartment of Planning and Investment, Savannakhet Province, Lao PDR
80

Enriching Evaluation with Perspectives of Local People: Two Cases of Collaborative Evaluation(hereafter DPI). It was free from the donor’s checklists and pre-set project
goals, and it tried to look at project-related issues and concerns from the local
people’s perspectives.
Although DPI was the only implementation partner of the study, evaluation questions were derived from the preparatory study conducted in the previous year which consisted of interviews and questionnaire surveys from central and local government officials, business leaders, and think-tanks fromboth sides of the FBII (Savannakhet, Lao PDR and Mukdahan, Thailand).
This preparatory study found that in addition to direct economic benefits the
local people were very much concerned with the social impacts of the FBII,
and instead of the outcomes of a single project, they were concerned with the
combined impact of other projects and government policies 8. Based on the
prep-study, DPI and FASID jointly developed the evaluation questions for the
impact study.
Because of the language difference and DPI’s time-availability, a report
was drafted in English, translated into Lao, reviewed and more information
was added by DPI (the report in the Annex).
2. 2. Merit of this approach
In this case, evaluation was conducted collaboratively which involved bothsides taking ownership in the evaluation process and jointly deciding thedetails of the study (such as what issues to be studied; what research methods should be used; who should be interviewed; how the questionnaireshould be phrased; and how the findings would be communicated and used).
When compared to a donor-centered evaluation, the following significant
changes resulted.
First, it made the evaluation’s coverage more extensive. This is natural
since the local people often consider what outsiders fail to. The FBII impact
study covered the farmers whose land was confiscated but who were notincluded in the initial evaluation framework.
Second, it made the information source far richer since the local partner
6

7
8

The Second Mekong International Bridge, which opened in December 2006, was built over the MekongRiver between Mukdahan, Thailand, and Savannakhet, Lao PDR. It was built to promote trade between
Lao PDR and Thailand and to promote economic development in the region. It was one of a number of
infrastructure projects on the East-West Corridor (a transnational road connecting Vietnam, Lao PDR,
Thailand, and Myanmar.). The Japan Bank for International Cooperation provided a concessional loan
for construction and consulting services to both Thailand (4,079 mil. yen) and Lao PDR (4,011 mil. yen).
The evaluation team consisted of a representative from DPI, two local consultants/interpreters, and two
FASID staffs.
Fujita 200981

acted proactively in finding the best information sources which in this caseincluded brothel owners, commercial sex workers, and a human trafficking
victim. Furthermore instead of focusing on the town by the FBII, the local
partner stressed the impact on the areas adjoining the national road connected to the FBII.
Third, it provided a more holistic interpretation by finding various explanations for one phenomenon. For example, the causes for the sharp decline
in Tuk-Tuk (bike taxi) business and in the number of Tuk-Tuk drivers using
the ferry port as their base were explained by not only the ferry customers
shifting to using the bridge, but also the government policy to shift short-distance transportation means from Tuk-Tuks to minibuses, and the change inlifestyle such that more Lao people were using privately-owned bikes andcars rather than Tuk-Tuks as their means of transportation.
Fourth, compared to donor-taken initiative, information collection doneby the partner under its own initiative and using its own networks and knowledge base is more efficient. Altogether, these factors provided greater context to the evaluation, and consequently made it more useful to the local partner.
An additional result was that joint evaluation greatly motivated the local
partner. Towards the end of the field study, the provincial government wasquite confident in running similar evaluations and requested training for evaluators for future projects. After all, partner countries’ ownership is critical;
public policy evaluation is important beyond international development cooperation programs as there are accountability and information needs to be
met throughout the public sector in partner countries.
2. 3. Challenges in implementation
Some of the challenges were similar to the Vietnam case. First, communicating within the team. The fact that communication between DPI and FASID
had to go through interpreters all the time sometimes restricted mutual
understanding. It may work better in this case to nominate a local teamleader who can communicate well with the donor side and can assume theresponsibilities of running the field study in close consultation with the donor
side.
Second, the trade-off between cost-bearing and ownership. Unlike thefirst case, no long travel was involved for DPI in this study. Still, for thesekind of donor initiated studies, the donor side customarily pays local partners
a per diem . Such compensation increases the risk that activities are done for82

Enriching Evaluation with Perspectives of Local People: Two Cases of Collaborative Evaluation

financial incentives and not for the utility of a study.
Third, a prior understanding of how much of the results will be madepublic may be necessary since some sensitive issues could come up unexpectedly during the study.
Fourth, certain compromise may have to be made in terms of evaluation
methods. For example, in discussions with DPI during the process of shaping up the evaluation methods, PRA and random sampling were excluded. As
for PRA, DPI had prior experience in which they felt researchers intentionally directed the opinions of uneducated group of people 9. Random sampling
was excluded because DPI claimed that uneducated people do not express
their opinions for this kind of surveys and that obtaining the opinions of thepeople most positively/negatively affected is more important. Such logic
makes sense to some extent but may not be persuasive to a donor.

The Second Mekong International Bridge (FBII)Interviewing a basket vendor

3. Boundaries and Perspectives
As an official ODA evaluation, the joint evaluation of the project in Vietnam
had to fulfill all of the donor’s requirements. The boundary of evaluation,
therefore, was restricted and the initial evaluation framework may have been
more or less the same as a donor-driven evaluation. Perspectives, however,naturally differed from a donor-driven evaluation since it was mostly local
people who implemented the evaluation and this may have changed the evaluation’s boundaries accordingly (the report will be made public by the end of2010).
In the Lao PDR evaluation study, no donor guideline was used and theteam set the evaluation questions entirely under their own discretion. The
9Therefore, in this evaluation study in Lao PDR, instead of targeting particular group of people, each
respondent’s educational background was asked and later responses were compared according to theireducation level.

83evaluation’s perspectives changed from that of the donor’s to that of the local
government and the local people. The boundary of the evaluation waschanged from “whether or not the pre-set goals of the bridge project wereachieved” to “what actually happened after the FBII was constructed.” Theinter-relationships of various factors were considered instead of the directcause-effect relationships.
In both cases, the evaluations’ perspectives and boundaries clearlychanged from the donor-driven evaluation and for this reason the evaluations
could help bring out the real impact of the project, and shed light upon truly
important aspects.
Compared to 20 years ago, huge progress has been made ensuring thatthe evaluation of ODA projects has become standard practice. However, inthe process, too much emphasis may have been placed on the process ofstandardization and on meeting the formal requirements of the donors. The
goal of ODA is to help development and the goal of evaluation is to help
ODA. Evaluation of ODA projects can be used to help development in partnercountries in a number of important ways in addition to improving a project’s
management or providing accountability to taxpayers.

References
Bryk, A., Stakeholder-based Evaluation, New Directions for Program
Evaluation (17), 1983.
Cousins, J.B.and Whitmore, E., Framing Participatory Evaluation, New
Directions for Evaluation (80), 1998, pp.5-23.
Dabelstein, N., Kliest, T., and Wood, B., Evaluating the Paris Declaration: A
Joint Cross National Evaluation, Presented at Annual Conference of theAmerican Evaluation Association, 2008.
Daigneault, P., Jacob, S., Toward Accurate Measurement of ParticipationRethinking the Conceptualization and Operationalization of Participatory
Evaluation, American Journal of Evaluation 30(3), 2009.
Fujita, N., Evaluating a Project from Beneficiaries’ Perspectives – A Case
Study of the Second Mekong International Bridge –, Review ofDevelopment and Cooperation, 2009.
Haraguchi, T., and Miyazaki, K., “A Practical Approach of Assistance in
Evaluation Capacity Development Experience through Evaluation of
ODA Loan Projects,” Trends in Development Assistance 5, pp.53-95, 2009.
http://www.fasid.or.jp/english/publication/research/evaluating.html
King, J., Making Sense of Participatory Evaluation Practice, New Directions
84

Enriching Evaluation with Perspectives of Local People: Two Cases of Collaborative Evaluation

for Evaluation (80), 1998, pp.56-68.
Lundgren, H., and Kennedy, M., “Supporting Partner Country Ownershipand Capacity in Development Evaluation. The OECD DAC Evaluation
Network,” in UNICEF, 2009, Country-led monitoring and evaluation systems, pp.77-95.
O’Brien F., “Preface,” in UNICEF, ibid., pp.2-3, 2009.
OECD Development Assistance Committee, Paris Declaration on Aid
Effectiveness
http://www.oecd.org/dataoecd/11/41/34428351.pdf,2005a. Accessed on
2008. 11. 23rd .
＿＿＿＿ Joint Evaluations: Recent Experiences, Lessons Learned and Options
for the Future, DAC Evaluation Network Working Paper, 2005b.
＿＿＿＿Guidance for Managing Joint Evaluations, DAC Evaluation Series,
2006.
Segone, M., “Enhancing evidence-based policy-making through country-led
monitoring and evaluation systems,” in UNICEF, 2009, op.cit., pp.17-31.
Stake, R.E., Standards-Based and Responsive Evaluation, presented at an
IPEN conference on Development and Evaluation, Tbilisi, Georgia, 29th
September, 2006.
Weaver, L., Cousins, J.B., Unpacking the participatory process. Journal ofMultidisciplinary Evaluation, I, pp.19-40. Retrieved 21st September, 2009,from
http://survey.ate.wmich.edu/jmde/index.php/jmde_1/article/view/
144/159.
World Bank, The Accra Agenda for Action, 2008. Retrieved 23rd November,
2008, fromhttp://siteresources.worldbank.org/ACCRAEXT/Resources/47007901217425866038/AAA-4-SEPTEMBER-FINAL-16h00.pdf
Worthen, R., Sanders, J.R., Fitzpatrick, J.L., Program Evaluation: Alternative
approaches and practical guidelines, Longman, White Plains. NY, 1997,
pp.153-169.

85

AnnexThe Socio-Economic Impacts of theSecond Mekong International Bridge
Evaluation Study Conducted by:
The Department of Planning and Investment,
Savannakhet Province, Lao PDR (DPI) and
The Foundation for Advanced Studies on
International Development (FASID)1

Nobuko Fujita, Eriko Takahashi

1. Introduction
The Second Mekong International Bridge (FBII 2, Figure 1) opened in
December 2006. Three years after its opening, in order to comprehensively
understand its socio-economic impact, the Department of Planning andInvestment (DPI), Savannakhet Province, and Foundation for Advanced
Studies on International Development (FASID) jointly conducted the evaluation study which follows.

1

2

The evaluation team consisted of the following members: Keokhonsy Thonchankhame (DPI), Vinata
Sayavong and Vannasao Soumpholphakdy (Civilise Consulting Co. Ltd.), Nobuko Fujita and Eriko
Takahashi (FASID). The report was compiled by FASID (sections 1-5, 7: Fujita, section 6: Takahashi,
section 8:Fujita and Takahashi) and reviewed by DPI who added more detailed information.
FBII stands for the Second Friendship Bridge.

87

Figure 1. Map of the Site
Mekong River

China
Vietnam

Myanmar◎HanoiLao PDR

Fourth Int’l Bridge◎Naypyidaw◎Vientiane Third Int’l Bridge
First Friendship Bridge
Yangon○Second Int’l Bridge○Mawlamyine
East-West Economic Corridor○ ○Savannakhet○ Da Nang
MukdahanThailand
Bangkok◎Cambodia
Phnom Penh◎

(Source: Author)

2. Criteria of Evaluation
The criteria of evaluation were initially derived from the preparatory study(evaluation needs study) conducted in August 2008 3. Evaluation needs studies are conducted to determine what developing countries expect from evaluations of development assistance projects. For the FBII needs study a total of
60 interviews were conducted between July 9th and 16th, 2008 4.
Interviewees were asked: “If an evaluation of the construction of the FBII
were conducted, what aspects would you like to look at?” or “What results oreffects of the bridge construction would you like to know more about, or are
you concerned about?” Responses were grouped into six categories: actual
economic benefits, social impacts, impacts on other countries, immediate
issues (such as immigration services), cost-benefit analysis, and prescriptions. Among responses interviewees showed a pronounced interest in thesocial impacts of the bridge. Interviewees’ interests covered a wide range ofissues: from labor migration, prostitution, HIV/AIDS infection, human traf-

3
4

88

Fujita 2009
Interviewees included high-ranking central and local government officials; leaders of business associations; researchers at a University and two think-tanks; residents and business people (living or runningbusinesses near the bridge, ferry terminal, in the markets, alongside National Highway No.9, or inside
the town of Kayson Phomvihane).

The Socio-Economic Impacts of the Second Mekong International Bridge

ficking, traffic accidents, environmental issues, to an increase in crime (murder, robbery, drug trafficking, and smuggling).
These issues are prioritized according to the frequency of responses.
Some of the issues were already examined to some extent by the AsianDevelopment Bank’s evaluations and were therefore excluded 5. After discussing the rest of the issues with the DPI, they chose the issues to study
according to their sense of urgency. In the end, they selected almost all thesocial impacts along with the cultural, economic impacts (Table 1).
Table 1. Criteria of Evaluation
Category
Social Impact

IssuesLabor immigration
Remittances
Traffic accidents
HIV/AIDS
Drugs
Human trafficking
CrimesSmuggling
Prostitution
Social exchange with Thai-side

Cultural Impact

Change in fashion, music, food, language.

Environmental Impact 6 Environmental changes in town
Environmental changes along the corridor
Economic Impact

Feeling of affluence
Economic growth, trade, investment, tourism
Change of occupation

OthersBenefit of the bridge
Other changes

3. Methods used
Key informant interviews and semi-structured interview-style questionnaire
surveys were conducted during the period September 4 th-11 th, 2009, in
5

6

Criteria discussed in the 2008 preparatory study and determined to have been covered in evaluations by
the Asian Development Bank and therefore excluded in this study included immigration services andthe impact on minority villagers near the Vietnamese border.
Environmental issues (such as deforestation, noise from vehicles and people, dust, garbage, and exhaustgas from increased traffic, dump trucks with toxic substances travelling at high speeds,etc.) were later
excluded from the evaluation criteria of the study since it was deemed to be too much to cover in the
short period of time and given the expertise needed in that area was lacking.

89

Kayson Phomvihane, Outhoumphone, and Champhone Districts in
Savannakhet Province (Table 2, Figure 2, and Figure 3). Official statistics
were collected wherever available.
Table 2. List of Interviewees
Key Informant Interviews
Interviewees

NumberSavannakhet Province Dept. of Health3

Savannakhet Province Dept. of Labor & Social
Welfare1

Savannakhet Province Dept. of Public Workand Transport

1

Savannakhet Province, Women’s Union

1

Residents who moved to Savannakhet Province
within three years

4

Truck drivers from Thailand

2

Service women (Kayson Phomvihane)

5

Human trafficking victim

1
Total

Notes

18

Interview-style Questionnaire Survey 7
Interviewees

NumberResidents in Nong Dern Village

3

Village where the casino is locatedResidents in Phonsavan Neu Village

1

Near the FBII, along the by-pass

Udom Viley Village

5

Near the FBII, along the highway

Champhone District

7

56 km from central Kayson
Phomvihane

Near km 352

35 km from central Kayson
Phomvihane

Service women (Beer stand along Route 9)

4

Xay Ya Poom District (Kayson Phomvihane)

7

Residents whose land was confiscated for the
FBII construction

6

TukTuk drivers
・Near the Ferry Terminal
・The FBII Terminal

7

Residents near That Inghang

5

90Hotel, restaurants and shops

4
1

Residents north side of the FBII (within 1 kmof the FBII)

7

Notes

Total

50

Grand Total

70

The most popular tourist spot in
Savannakhet

Respondents were chosen from residents who had been living in Savannakhet for more than three
years.

The Socio-Economic Impacts of the Second Mekong International Bridge

Figure 2. Area of Survey (1) Kayson Phomvihane
Udon Vilay

That IngHang

Kayson Phomvihane City

Km35
N

CasinoDinosaur Junction

TempleChurchNong Dern
Phonsavan Neu
By-pass
Provincial
Administration
OfficesMarketImmigration Office
AirportBus Terminal

Second Friendship Bridge
Mekong River

XayYaPoom
Ferry Terminal

0500 1000 1500m(Source: Author)

Figure 3. Area of Survey (2) Savannakhet Province
Route 13

Thailand

Vietnam
Outhoumphone
District

KaysonPhomvihane
District
FBII

Route 9Champhone
Distirict

Surveyed area

(Source: Author)

4. Major Changes in the Year 2008-2009
Major changes in the town of Kayson Phomvihane in 2008-2009, included,
the opening of a casino near the FBII, the reopening of Savannakhet airport,
and an investment boom associated with both of these new developments.
The casino (Savan Vegas) opened in August 2008 8, and as of September
8

An adjacent hotel opened in April 2009.

91

2009, has been attracting more than 1,000 tourists (mostly Thai) every day 9.
The majority of Thai visitors are picked up by vans provided by the casino
and offered a free lunch buffet; some tourists opt to visit local restaurants taking Tuk-Tuks, or private cars. According to the Department of Labor andSocial Welfare, the casino has also generated a total of 1,200 jobs 10.
Savannakhet airport reopened in August 2008. Three flights per week are
now available from Vientiane and travel time from Vientiane has been shortened from an eight-hour car drive to a one-hour flight. The Bangkok route
started in October 2008 has also improved access to Savannakhet tremendously for international businessmen and tourists.
The FBII and related factors as mentioned above have helped activate theSavannakhet economy. New resort hotels and modern shops opened and thetown has become cleaner than it was before. Other investments such as
sugar cane factories and manufacturing have been on the increase as well.
Despite an improved economy which has helped create a labor shortage in
Savannakhet, local workers still go to Thailand for higher wages, and thenumbers are increasing.

5. Social Impact
5.1. Results of the Questionnaire Survey
A summary of the questionnaire survey results are shown in Figure 4 (n=50).
The respondents were asked about the changes that have occurred since theopening of the FBII. According to the results, the largest change has been
seen in labor migration.
To address the concerns of the DPI, the results of the questionnaire survey were broken down into respondents with a middle school education or
above (Figure5, n=36), and respondents with a high school education or
above (Figure 6, n=27). The results show that differences in respondents’
educational levels did not affect the answers. 5.2 through 5.9 below explain
the changes in each item.

9

On September 9, 2009, the so-called “999 (lucky) day,” 3,000 Thai crossed the FBII to come to thecasino, according to a local newspaper.
10 Out of the 1,200 workers, 200 were foreigners.

92

The Socio-Economic Impacts of the Second Mekong International Bridge

Figure 4. Perception of the Changes after the Opening of the FBII (All Respondents)
Perception of the changes after FBII (All respondents. n=50)
labor migration
More often
than before

prostitutiondrugs

unchanged

crimesless than
beforetraffic accidents
HIV / AIDS

don’t knowhuman trafficking

othersmuggling
0%

20%

40%

60%

80%

100%

(Source: Questionnaire Survey, 2009)

Figure 5. Perception of the Changes after the Opening of the FBII (Respondents
with Middle School Education)
Perception of the changes after FBII (Respondents with education level:
middle school or above. n=36)
labor migration
More often
than before

prostitutiondrugs

unchanged

crimesless than
beforetraffic accidents
HIV / AIDS

don’t knowhuman trafficking

othersmuggling
0%

20%

40%

60%

80%

100%

(Source : Questionnaire Survey, 2009)

Figure 6. Perception of the Changes after the Opening of the FBII (Respondents
with High School Education)
Perception of the changes after FBII (Respondents with education levelhigh-school or above, n=27)
labor migration
More often
than before

prostitutiondrugs

unchanged

crimesless than
beforetraffic accidents
HIV / AIDS

don’t knowhuman trafficking

othersmuggling
0%

20%

40%

60%

80%

100%

(Source : Questionnaire Survey, 2009)93

5.2. Labor Migration
Although the exact numbers are not available, since the completion of the
FBII, labor migration to Thailand has been increasing 11. The questionnaire
survey result shows that 56% of the respondents think that more people fromSavannakhet work in Thailand since the construction of the FBII, 30% thinkthat there has been no change, and 7% think that fewer people work in
Thailand as compared to before.
This seems to be a combined effect of the FBII and visa exemption. Many
Lao people used to enter Thailand with a border pass (valid for three days)
and work illegally in Thailand. Now, many hold a passport which costs
30US$ per applicant and allows them to stay in Thailand for up to threemonths. The Ministry of Foreign Affairs’ service to process passport applications at the FBII facilitated this move. Outgoing Lao travelers fromSavannakhet by passport increased from 35,749 in 2006 to 92,283 in 2008,
while those with a border pass increased in 2007 possibly due to the openingof the bridge, but then decreased again in 2008 to 2006 standards (Figure7).
Although carrying a passport does not allow people to work legally in
Thailand, many Lao workers understand that just carrying a passport makes it
legal. Although Savannakhet now has many employment opportunities suchas working in gold and copper mines, sugarcane and textile factories, the low
minimum wage of 25,000 kip/day makes attracting workers difficult 12.

travelers (1000s)

Figure 7. Out-going Lao Travelers from Savannakhet Province
100
90
80
70
60
50
40
30
20
10
0

Passport
Border Pass

2006

2007

2008

(Source: Lao National Tourism Administration, 2008)

11 ILO reports that 27,000 Savannakhet people worked in Thailand in 2004. In 2008, the number available
was 42,069; however it is not accurate according to the Department of Labor and Social Welfare(Interviewed on Sept.11, 2009).
12 1US$= about 8,500kip as of Sept.10, 200994

The Socio-Economic Impacts of the Second Mekong International Bridge

5.3. Prostitution
The next biggest change since opening of the FBII perceived by the interviewees regarded the issue of prostitution. Although data on the number ofbrothels 13 or prostitutes is unavailable, 43.8% of the respondents answeredthat prostitution has been increasing since the FBII opened (33.3% answered“unchanged,” 0% answered “decreased,” and 18.8% said “don’t know.” See
Figure 4).
Among key informant interviews, respondents’ perceptions showed
mixed results. At a beer house located more than 30 kilometers away fromthe FBII on Route 9, an interviewee stated that her business has declined
since the casino opened. However, more beer houses have been establishedalong Route 13 in Champhone District recently. Of the twenty-three respondents who said that prostitution had increased: fourteen said that theincrease in prostitution came from the demand side due to the increase in
the number of international tourists (including casino guests); and nine people suggested that the increase came from the supply side citing that it is aquick way to make money, especially for local people who are having difficulty finding a job or who simply want to purchase fashionable Thai clothes andcell phones.
One of the authorities mentioned that because of the advocacy campaign,
many of the service women 14 changed their jobs by learning various skills
through training provided by Women’s Union, international organizations
and NGOs. One source claimed that many small beer restaurants shut down
due to this fact, but available statistics could not confirm this claim. On theother hand, another official suggested that one negative impact of the bridge
was that young girls living in the countryside (Louangphabang, Vangviang,
Xiangkhouang) who are attracted to Savannakhet end up working as service
women 15. Their main customers are Thai nationals who can now cross the
FBII anytime between 6:00 A.M. to 10:00 P.M.
5.4. Human Trafficking
Not all women moved into Savannakhet on their own free will. Officials incharge of public security, labor and social welfare, and women’s issues, all

13 Since beer restaurants are all registered as “restaurants,” the official number of brothel-type beer restaurants is not available.
14 Commercial sex workers
15 Out of the 50 questionnaire survey interviewees, 6 moved in to Savannakhet in the last 3 years; 2 cameto go to college, 1 came to open a store, and the remaining 3 were service women.

95said that human trafficking has been increasing since the FBII opened. Lao
immigrant workers in Thailand return home and take friends and relatives
back with them to Thailand, exposing them to human traffickers.
According to Savannakhet Women’s Union, the number of cases ofhuman trafficking reported in Savannakhet Province was four in 2007 andtwo in 2008. Data was not collected in 2006 according to the Women’s
Union 16. In 2009, although none of the cases reported involved Savannakhet
residents, reported cases did involve two Vietnamese women, four womenfrom Attapeu Province and eight women from Khammouane Province. Manyof the traffickers use small boats and cell phones to cross the Mekong River
at night. Additional factors such as an increase in labor migration and adesire for high income also had a combined impact on the increase in humantrafficking. The anti-trafficking campaign might also have made people
acknowledge cases of human trafficking that previously would not have beenrecognized as such.
5.5. HIV/AIDS & Other Communicable Diseases
Cases of people with HIV reported in Savannakhet Province decreased in2008. However, as of the first half of 2009, 90 cases have already been reported, which indicates that an increase is expected in 2009 (Figure 8). In 2008,
86% of all cases reported were in Kayson Phomvihane; 94% were along Route
9 and all the cases reported in 2008 were along either Route 9 or 13 (Figure9). Immigrant workers and truck drivers accounted for 55 percent of allreported HIV/AIDS cases, and housewives who were most likely infected bytheir immigrant worker husbands accounted for 22 percent (Figure 10).
The 2008 study revealed that the lower HIV infection rates in Savannakhet,
Lao PDR (0.8%) compared to Mukdahan, Thailand (2〜3%) was considered alikely reason that Thai go to Laos to obtain services. The difference in infection
rates was about the same in 2009 (Mukdahan: 2〜3%, Savannakhet: 0.9%)
according to Savannakhet Department of Health (DOH) 17.

16 Interviewed on September 9, 2009.
17 Interviewed on September 7, 2009.

96

The Socio-Economic Impacts of the Second Mekong International Bridge

Figure 8. HIV Positive Cases Reported in Savannakhet Province

Number of report

200
150
100
50
0
2006

2007

2008

2009 (Jan.-June)(Source: Savannakhet Province Department of Health, 2009)

Figure 9. HIV Positive Cases Reported in 2008 (by Districts 18)(Savannakhet Province, Jan. to Dec.2008)
Khammouane Province

Route 13

VIETNAMTHAILAND

5
XBL

6
APT

113
KAY

FBII

1

3

OUT

Route 9

PHA

1
CHA

2
SON

Saravane Province(Source: Savannakhet Province Department of Health, 2009)

18 XBL: Xayboury Province, KAY: Kayson Phomvihane, OUT: Outhoumphone, CHA: Champhone,
SON: Songkhone, APT: Atsaphangthong, PHA: Phalanxay Districts.

97Figure 10. HIV Positive Cases Reported in 2008 (by Occupation)(Savannakhet Province, Jan. to Dec. 2008)
60
48

50

Blood Tested: 2.986;
HIV +: 131; Female: 62

Numbers

40
29

30

21
20
10

4

4

6

5

3

1

3

1

ck
d

riv
er

lice
tr u

po

sty
l

ist

en
wo
m

ty
au
be

se
rvi
ce

rch
a

nt

n
me

dre
ch
il

plo
ye
e
em

tw
ork
er

gra
n

mi

er

nw
ork
er

rm

cti
o

t fa

gra
n

ns
tr u
co

mi

Ho
u

se
wif
e

0

(Source: Savannakhet Province Department of Health, 2009)

The H1N1 flu is a recent concern in the Savannakhet Province wheremore than 20 cases have been reported as of September 7, 2009. The first
patient from Savannakhet was infected in Thailand, where he worked (reported on June 26, 2009). On July 12, the first patient in the casino was reported.
The casino, which was crowded with gamblers, their families, spectators, andfree lunch takers, became a hot spot and 16 staff members were diagnosed
as contracting H1N1 19. DOH and the casino immediately took counter-measures such as introducing thermographs and masks, and according to DOH,
no further patients have been reported since August 15, 2009 (as of
September 7, 2009) 20.
Regional surveillance which was conducted for 10 infectious diseases in2008 increased to 19 diseases in 2009. In the case of H1N1, coordination
among hospitals in other provinces and Thailand was done smoothly. DOH
constantly updates information regarding infections and makes such information readily available.
5.6. Drugs, Crimes and Smuggling
Official data recording increases/decreases in drug confiscations, the num-

19 Many of the casino guests come in families, including grandparents and small children.
20 DOH interview on September 7, 2009.

98

The Socio-Economic Impacts of the Second Mekong International Bridge

ber of crimes, and the incidences of smuggling was not available. Therefore,
the following information has been drawn from key person interviews andthe questionnaire surveys.
Eighteen out of 50 respondents (36%) of the questionnaire survey
responded that drugs have been on the rise because of easier access by the
FBII and increased traffic. Ten respondents (20%) said unchanged, and thesame number of respondents (20%) said drug use has decreased due totighter control. Either way, drug use is a two-way issue, i.e., drugs coming infrom Thailand and going out from Lao PDR.
As for general crimes, 15 respondents (30%) said, “increased.” Four ofthem were either victims of burglaries or neighbors of victims; items stolen
included bikes, helmets, cows, shoes and canopies. Some people mentionedthat crimes have increased because Thai criminals have joined forces withlocal criminals. Casino customers in particular have been made prime targetsof robbery and snatching since it first opened. Furthermore, the losers ofgambling can become potential robbers. Twenty-two respondents (44%)
answered that the number of crimes has not changed, and four (8%) said thatcrimes have decreased due to tighter security (initiated by the government
campaign “Village without crimes”). One said the newly built road (accessroad to the casino) also contributed to a decrease in the number of crimes.
As for smuggling, only three respondents (6%) said, “increased”, they
explained that they had witnessed gasoline and food being smuggled by passenger cars and trucks. Twenty-one (42%) said “unchanged.” Eight respondents (16%) said, “decreased,” due to tighter control and regulations. One ofthem said that easier transportation by the FBII made smuggling unnecessary. The respondents who answered “unchanged” said that smuggled goods(in this case, small daily items such as detergents and candy) are brought infrom Thailand by small boats, therefore the construction of the FBII is not
related in any way to cases of smuggling.
5.7. Traffic Accidents
According to the Provincial Police Department, safety campaigns conducted
over the last three years have helped decrease traffic accidents in Kayson
Phomvihane (Figure 11). However, according to the questionnaire survey, 16
people (32%) responded that traffic accidents have increased as opposed to
two respondents (4%) who said “decreased.” Respondents that believe traffic
accidents have increased mentioned that “There have been many motorbike
accidents recently” (18%), “The number of cars on the road have increased”
99(12%), “Many second-hand Korean cars are imported” (2%), and “Drunk drivers coming out of the casino get involved in accidents” (2%). According to
the respondents, the causes of accidents include drunk driving, the absenceof a license, and higher accessibility to motor bikes especially for young people due to cheap bikes imported from China. Fifty-eight percent said“unchanged” implying that the FBII is not a direct cause of the increase in
accidents since most of the accidents involve motorbikes, and motorbikes
are prohibited from crossing the bridge.
Figure 11. The Number and Cause of Traffic Accidents in Kayson Phomvihane
Speeding
Drunk driving
Change lanes
No waiting
Wrong lane
Pass
Fall over
Others2004-2005

2007-2008

0

100

200

300
400
500
number of accidents

600

700

(Source: Kayson Phomvihane Provincial Police, 2009. Figures are for fiscal year, October to September)

5.8. Access to Health/Medical Services in Thailand
According to the questionnaire survey, 22 respondents (45.8%) said theywould seek health/medical care in Thailand, of which 16 (33.3%) said theywould use the FBII to do so. The same number of respondents (22 or 45.8%)
said they would not seek such services. Respondents who answered thatthey would seek medical attention in Thailand offered the quality of available
services as a reason. Respondents who said that they would not go to
Thailand for medical care gave reasons which included, “My family is very
healthy” (7 respondents), “No money” (6), “It is more convenient to use local
services (ID or passport is not required)” (2), and “Local hospitals are getting better” (2). Consequently, the people who seek Thai medical services
may be limited to those people who can afford it, and the FBII may facilitatesuch people to obtain medical services. In terms of emergency care, the FBII
benefited the whole population of Savannakhet since Category I patients 21
are transferred to hospitals in Mukdahan, and the DOH stated that the FBII
has undoubtedly facilitated emergency medical care.
21 Patients with life-threatening conditions

100

The Socio-Economic Impacts of the Second Mekong International Bridge

5.9. Negatively-affected People
5.9.1. Farmers:

The questionnaire survey showed that the respondents think farmers (whose
land was expropriated for the FBII) benefited least by the FBII. For some ofthese people, the process for receiving expropriation compensations is not
over, even though their land was expropriated as early as 2000.
According to the interviews of six farmers who use to own paddy, fruitand vegetable fields and farmhouses at the construction site, alternative land
was offered which was too rocky to make suitable for farming, and was one
hour away by car. (Back then, their farms were in walking distance.) All six
farmers therefore chose monetary compensation instead. The first payment
was made in 2005 and small-scale farmers were all compensated. Large-scale
farmers were to be paid in several payments but as of September 2009, thefinal payment had not yet been made 22. It is noteworthy that, this issue
seems to have some influence in land expropriation for the Savan-Seno
Economic Zone (SEZ). A farmer whose paddy field is located in one of the
SEZ sites said that he would not stop planting rice until compensation is completed and that his neighbors have the same view.
Table 3. Who Benefited the Most and the Least from the FBII
Who Benefited the Most

Responses

Who Benefited the Least

Responses

Transportation companies

13 Farmers (whose land was
expropriated)

12

Traders/Merchants

13 Ferry company owners andemployees

3

Government officials/Immigration
officers

6 Restaurants near the ferry terminal

3

Travel agencies

4 Tuk-Tuk drivers

2

Hotels

3 Small store owners

1

Restaurants

3 Students (*because they don’t use
the FBII)

1

Casino3 Government officials

1Markets

3

Tuk-Tuk drivers

3

Investors

2

Ordinary people

1(n=50)(Source: Questionnaire Survey, 2009)

5.9.2. Ferry Companies:

According to the study conducted in 2008, ferry companies lost half of their
22 Note: Time constraints precluded conducting interviews with government officials to clarify this issue.

101

customers after the FBII opened. This is despite the fact that many people
still prefer to use the ferry due to the ferry terminal’s more convenient location downtown and because it is closer to the market or their residence.
Restaurants by the ferry terminal also experienced a downfall in customers.
5.9.3. Tuk-Tuk Drivers who Used to Work for Ferry Passengers:

Tuk-Tuk drivers who operate near the ferry terminal have seen a decline intheir customers; some say half of the pre-FBII customers have gone.
According to the remaining active drivers, many of their colleagues lost theirjobs and now work in construction and other industries. According to the
Department of Public Work and Transport, the number of registered TukTuks is as follows 23.
Figure 12. Number of Tuk-Tuk in Kayson Phomvihane
450400
Tuk-Tuk# of vehicles

350

Sky-lab type
Tuk-Tuk

300
250
200
150
100
50
0
2005

2006

2007

2008

2009

(Source: Savannakhet Province Department of Public Work and Transport, 2009)

Tuk-Tuk drivers have not left for other jobs simply due to the decrease inferry passengers. Another factor is the Department of Public Work andTransport’s policy to shift from Tuk-Tuks to mini-buses as a means of safe,
short distance transportation. New registrations of the Sky Lab-type TukTuks (Figure 13) are no longer admitted, due to termination of the production and difficulties in proper maintenance. In addition, in line with a booming economy, the mode of transportation is shifting from Tuk-Tuks to private
motor bikes or cars. Therefore, in addition to the FBII, a number of other factors can be attributed to the decrease of Tuk-Tuks and drivers.

23 There was no data available for the number of drivers. However, number of drivers and vehicles are
more or less the same, according to the Department of Public Work and Transport.

102

The Socio-Economic Impacts of the Second Mekong International Bridge

Figure 13. Sky-Lab and Tuk-Tuk
Sky-Lab

Tuk-Tuk(Source: Savannakhet Province Department of Public Work and Transport, 2009)

6. Economic Impact
6.1. Gross Provincial Product
Savannakhet Province has maintained a high Gross Provincial Product(GPP) rate for the past couple of years. As the graph below (Figure 14)
demonstrates, Savannakhet’s real GPP growth rate (the provincial growthrate) greatly exceeded the real GDP growth rate of Lao PDR (the country’s
growth rate). Particularly in the years prior to the FBII construction,
Savannakhet’s GPP growth rate was increasing at a rapid pace. Although
anticipation of the bridge’s completion may have stimulated the province’s
economy (for example, by increasing investment) the growth can more likely
be attributed to the development of the province’s gold and copper mining
industry (IDE-JETRO, 2008).
Full-fledged development of the mining industry began in 2003 with
Australian company – Oxiana’s 30 million dollar investment in the Sepon
Mines (IDE-JETRO, 2008). Since then, an increase in gold and copper
exports has been evident; 2005 exports of gold and copper doubled in comparison to the year before (IDE-JETRO, 2008). Consequently, although the
Lao PDR’s real GDP growth rate has decreased slightly since 2006,
Savannakhet has maintained a high real GPP growth rate.

103

Figure 14. Real Growth Rates of Savannakhet GPP and Lao PDR GDP
14

Growth Rate (%)

12

6

11.9

8.1

7.9

7.5

2006
Year

2007

2008

10.0

10
8

12.5

11.9

7.8
7.1
6.4

4
2
0

2004

2005

Savannakhet GPP

Laos PDR GDP(Source for Savannakhet GPP Real Growth Rate: Savannakhet Province Department for
Planning and Investment, 2007, 2008, 2009. Source for Lao PDR GDP Real Growth Rate:
World Bank, 2009)

Breaking down Savannakhet Province’s GPP by sectors (agriculture,industry and services), Figure 15 illustrates that, for the years 2004 through2008 (at current market prices), all three sectors appear to be enjoying a positive trend.
Table 4 breaks down the growth rate of the province’s GPP by sector.
The GPP of the agriculture sector throughout the observed years has maintained a growth rate of 7-9%. The services sector, likewise, has been growing
at a rate of approximately 13% (both before and after the completion of thebridge), although growth slowed slightly in 2008. As for the industry sector,
although the growth rate was lower than that of the services sector at approximately 12.6% in 2005, for 2006 (the year the FBII was completed) and for theyears following, the growth rate has exceeded 18%, which largely surpasses
the growth rates of the other sectors. Although drawing any conclusions ispremature, it is possible that the FBII has had a positive effect on
Savannakhet’s sectoral GPP growth rates particularly on the industry sector.

104

The Socio-Economic Impacts of the Second Mekong International Bridge

Figure 15. Savannakhet Current GPP by Sector
6,000Billion Kips

5,000
4,000
3,000
2,000
1,000
0

2004

2005
Agriculture

2006
Year

2007

2008

Industry

Services(Source: Savannakhet Province Department for Planning and Investment,
2007, 2008, 2009)

Table 4. Savannakhet Current GPP Growth Rate by Sector
2005

2006

2007

2008

Agriculture Growth Rate (%)

7.73

7.31

7.53

8.92

Industry Growth Rate (%)

12.62

19.94

18.73

19.80

Services Growth Rate (%)

13.22

13.77

13.71

11.76

(Source: Savannakhet Province Department for Planning and Investment, 2007, 2008, 2009)6.2. Trade and Investment
Savannakhet’s principal imported goods for the period 2006 to 2008, areshown in Table 5 below (expressed in US$ thousands). In general, most of
Savannakhet’s principal imported goods increased after 2007 although data is
insufficient to draw any definite conclusions that those increases werebrought about by the completion of the FBII. The large increase in imported
fuels in 2008, for example, probably reflects the soaring oil prices that year.
Table 5. Principal Imported Goods (US$ thousands)
2006

2007

2008

Food

4,853

6,376

Clothes

1,249

1,409

N/A

528

369

686

Agricultural tools and machines
Medicine

6,366

373

369

686

Construction materials

1,452

1,413

3,992

Fuel

29,646

26,78347,897

167

426

971

Vehicles and small parts

8,897

12,271

17,322

Spirits and tobacco

25.4

48

3,597

Electric appliances(Source: Savannakhet Province Department for Planning and Investment, 2007, 2008, 2009)105

Figure 16. Inbound/Outbound Commercial Trucks and Freight Volume at the Border
between Savannakhet (Lao PDR) and Mukdahan (Thailand)

Vehicles (1000s)

20

17.85

15
146
10

19.17

21.21

20.32

19.46

260

225

158

132

126

6.24

6.02

190

195

157

6.80

173
7.10

200
150

7.41

5
0

250

100
50

2004

2005

2006
Year

Commercial Truck (Inbound)
Freight volume (Inbound)

2007

2008

Freight Volume (1,000 tons)

300

25

0

Commercial Truck (Outbound)
Freight volume (Outbound)(Source: Data from the Savannakhet Province Department of Public Work and Transport,
2009)

Figure 16 illustrates the number of inbound and outbound commercial
trucks taken at the border between Savannakhet (Lao PDR) and Mukdahan(Thailand), also represented in terms of freight volume. The graph shows
the steady inflow of both the number of commercial trucks as well as freight
volume both prior to and subsequent to the inauguration of the bridge, hence
the effects of the bridge remain unclear. Furthermore, although the bridge isconsidered advantageous especially for land transportation and freight forwarding businesses, at present, the bridge is being used merely as a transshipment point for Thai and Vietnamese cargo trucks (IDE-JETRO, 2008). To
improve the current situation, investment in key industries in Savannakhet is
crucial and, therefore, further improvements need to be made concerning
the investment climate of the province (IDE-JETRO, 2008).
Trends are also not so apparent in regards to Savannakhet’s principal
exported goods. Table 6 below illustrates Savannakhet’s principal exports interms of US$ thousands for the period 2006 to 2008. Apart from the plunge ingold and copper exports in 2007, the probable result of a large decrease in
the production growth rates of the two minerals, no particular trends concerning exported goods can be seen overall (ADB, 2009). Moreover, Figure
16 clearly shows that both the number of inbound commercial trucks and thetotal freight volume largely exceeds that of the outbound number and volume; most likely due to the fact that a considerable portion of Savannakhet’s
GPP is accounted for by its primary (agriculture, mining) industry.
106

The Socio-Economic Impacts of the Second Mekong International Bridge

Consequently, the economic impact of the FBII on Savannakhet’s exports islikely to be a mid- to long-term effect, largely depending on the developmentof the province’s secondary and tertiary industries.
Table 6. Principal Exported Goods (US$ thousands)
2006

2007

2008

Wood and wooden articles

17,317

19,972

10,238Agricultural products

4,708

1,900

4,175

Manufactured products

524

1,711

1,487

Gypsum

708

N/A

N/A

Gold

92,603

19,467

82,346

Copper385,404

118,412479,198

Clothes

1,193

N/A

N/A

324

42

353

Forest products(Source: Savannakhet Province Department for Planning and Investment, 2007,2008, 2009)

The following graph (Figure 17) presents the inflow of foreign investment
in Savannakhet Province, in terms of capital costs in US$ millions, and thenumber of approved investment projects, for the years 2006 through 2009. It
can be seen from the graph that in 2006, the year the FBII was completed,large-scale investments exceeding a total of US$400 million were made in
Savannakhet. Investments approved that year included a US$250 million
multinational project conducted by an Indian company that made investments in a Eucalyptus tree plantation and the construction of paper factories.
This project, as well as many other investment projects that have been implemented in anticipation of the completion of the bridge are long-term projects(e.g. tree-planting projects) and therefore will require a relatively longamount of time before the effects are observable (IDE-JETRO, 2008).
Consequently, although investment may have been enhanced due toimproved access to the province and increased investment opportunities,
supporting evidence is insufficient to draw any definite conclusions.

107

Figure 17. Foreign Investment – Capital Costs and Number of Approved Projects

400
20

US$ million

350
300

15

250
200

10

150
100

5

50
0

2006

2007

2008

2009

Number of Approved Projects

25

450

0

Year
Capital Cost

Projects(Source: Savannakhet Province Department for Planning and Investment, 2007,2008, 2009)

Initially planned by the government in 2002-2003, the SEZ (a special economic zone located in Savannakhet Province) is currently undergoing development. It consist of four zones covering an area of approximately 800
hectares and is expected to include an export processing zone, a free tradezone and a logistics center. It is anticipated that establishment of the SEZ will
attract additional foreign investment.
6.3. Tourism
The construction of the FBII has had a significant impact on the tourism sector of Savannakhet Province and thus on the tourism sector of Lao PDR as awhole. Figure 18 describes the number of international tourist arrivals (inthousands) in Lao PDR by different ports of entry – Savannakhet Province,
the First Friendship Bridge in Vientiane, Vientiane Wattay International
Airport and other entry points. In 2007 and 2008, a sharp rise in tourist
arrivals via Savannakhet can be seen. As a port of entry, Savannakhet
accounted for almost 25% of all Lao PDR tourist arrivals in 2007 and 2008 – apotential indication of the positive impact of the FBII.

108

The Socio-Economic Impacts of the Second Mekong International Bridge

Tourists (1000s)

Figure 18. Lao PDR Tourist Arrivals by Port of Entry
2,000
1,800
1,600
1,400
1,200
1,000
800
600
400
200
0

Other
Wattay Airport
Friendship Bridge I
Savannakhet

2003
174.0872.22
357.0433.02

2004
242.9986.33
451.10114.392005
313.08105.53541.02135.692006
361.34106.23582.18165.36

2007
462.13105.67656.47399.67

2008
628.46151.94547.59408.80Year(Source: Lao National Tourism Administration, 2008, 2009)

Figure 19. Inbound/Outbound Passenger Cars at the Border Between Savannakhet(Lao PDR) and Mukdahan (Thailand)
16,00013,35414,000
Vehicles

12,00011,192

10,000
8,000
5,371
4,438

6,000
4,000
2,000
0

2,161 1,760
350 277352 190

2004

2005

Passenger Car (Inbound)

2006
Year

2007

2008

Passenger Car (Outbound)(Source: Savannakhet Province Department of Public Work and Transport, 2009)109

Figure 20. Inbound/Outbound Passengers at the Border Between Savannakhet(Lao PDR) and Mukdahan (Thailand)
120109.04Passengers (1000s)

100
82.49
80
62.49
60
45.12
40 35.87

80.15

69.35
60.09

48.12
38.95

20
0

2004

2005

Passengers (Inbound)

2006
Year

2007

2008

Passengers (Outbound)(Source: Savannakhet Province Department of Public Work and Transport, 2009)

The number of passenger cars and passengers respectively that haveentered or exited Savannakhet Province also support this assumption as evident in Figures 19 and 20. From Figure 19, it can be seen that the number ofboth inbound and outbound passenger cars have increased dramatically ataround the same rate. Figure 20 shows that the number of inbound and outbound passengers has been on the rise since completion of the bridge in2006. Since completion, the number of incoming passengers has exceeded
the number of outgoing passengers, with passenger inflow growing at a
noticeably higher rate than passenger outflow 24. It can be concluded that theconstruction of the bridge has played a positive role in attracting tourists tothe province.
The development of Savannakhet Province’s tourism sector, which has
been further enhanced with the inauguration of the FBII, can be observed
from Figure 21. The graph shows recent trends (2003 through 2008) in thenumber of accommodation facilities located in Savannakhet Province (including hotels and guesthouses) and the number of available rooms in those facilities. Since completion of the bridge in 2006, approximately 10 new hotels
and guestrooms were constructed, and nearly 400 more rooms were madeavailable, to help accommodate the increase in incoming travelers to the

24 It should be noted that Figure 20 accounts for passengers traveling in regular passenger cars only; passengers who cross the bridge by bus are not represented in the number of inbound/outbound passengers.

110The Socio-Economic Impacts of the Second Mekong International Bridge

province.
Figure 22 presents the room occupancy rates for all accommodation facilities in Savannakhet Province for the years 2003 to 2008. The room occupancy rates have shown a gradual increase throughout the period; after the inauguration of the bridge, the occupancy rates have remained in the high 60s,
supporting the assumption that there has been a boost in tourism and traveler arrivals.
Figure 21. Number of Accommodation Facilities & Rooms in Savannakhet Province
90
80

1,800

72

1,758

67

70

1,627

60
60

2,000

1,600
1,400

53
1,257

50

1,200

1,236

1,000

40

979

995

Rooms

Accomodation Facilities

83

81

800

30

600

20

400

10

200

0

2003

2004

2005

2006

2007

Accomodation Facilities

Rooms

2008

0

(Source: Lao National Tourism Administration, 2008, 2009)

Figure 22. Room Occupancy Rates for Accommodation Facilities in Savannakhet
Province

Room Occupancy Rate (%)

80
68

70
60
50

58

60

2004

2005

61

65

49

40
30
20
10
0

2003

2006

2007

2008

Year
Room Occupancy Rate
(Source: Lao National Tourism Administration, 2008, 2009)111

Recreational facilities, including Lao PDR’s first casino hotel, which
opened in 2009, have also been developed. The casino (a US$56 million joint
venture project Laos (40%) and a Macau-based gaming investment company(60%)), specifically targets the Thai population living in the vicinity of
Mukdahan Province which is the region in Thailand linked via the FBII to
Savannakhet. Since its opening, the casino has been attracting many Thai
customers and has provided employment opportunities 25. Consequently, therevitalization of Savannakhet’s tourism sector has contributed to theenhancement of the local economy and the creation of employment opportunities.

7. Cultural Impact
As for fashion, food, music and language, interviewees were asked if theinfluence from Thailand has become stronger since the FBII opened. The
answers are as follows (Figure 23). Only for fashion did respondents thatanswered “Yes” exceed those that answered “No.” Changes in style of fashion include young girls wearing revealing clothes such as mini-skirts andshort pants instead of the traditional wrap skirt (Sin). This type of change infashion was considered favorable by some people (7 respondents) and unfavorable by others (7 respondents). On the other hand, as for the influx ofThai cuisine, only positive opinions were given, such as the increase of a variety of different tastes. For music and language, more influence comes fromTV programs and other media, and physical improvement of access does notseem to contribute much.
Figure 23. Increase in Cultural Influence from Thailand since the Opening of the
FBII
FashionYes
Cuisine
No
Music
don’t
know

Language
0%

10%

20%

30%

40%

50%

60%

70% 80%(Source: Questionnaire Survey, 2009)25 Please refer to section 4, “Major Changes in the Year 2008-2009.”

112

90% 100%

The Socio-Economic Impacts of the Second Mekong International Bridge

According to some respondents, the FBII has facilitated visiting Thailand
for cultural events (attending festivals, visiting temples, etc). Thirty % of thesurvey respondents said that thanks to the FBII they cross the Mekong Riverfor such occasions more often than before. However, 46% answered“unchanged,” because they use small boats, rather than ferries or the FBII,
to cross the river depending on the accessibility (location, means of thetransportation to the access point) 26. Thirteen % said they have never visitedThailand for cultural events (Figure 24).
Figure 24. Cultural Visits to Thailand
other
10%
More often
than before
28%

Never gone forcultural events
13%
don’t know
2%
less than
before4%n=45unchanged
43%(Source: Questionnaire Survey, 2009)

8. Conclusion
8.1. Benefits from the Bridge for the People of Savannakhet
The completion of the FBII has brought about evident changes to
Savannakhet. The people of Savannakhet have clearly benefited from thebridge, in terms of the positive impacts it has made on the local economy. As
stated in section 6, the FBII has had a significant impact particularly on the
tourism sector of the province. The inauguration of the FBII has resulted in
an enormous influx in the number of travelers coming into Savannakhet, an
increase in the number of accommodation establishments, and an improvement in the availability of employment opportunities.
The FBII has also brought convenience to the people of Savannakhet interms of shopping, accessing Thai health/medical services, and many other
26 A small boat to cross the river costs 100 Baht/person (round trip), a ferry costs 14,000 kip/ person (oneway) and the cross border bus to cross the FBII costs 50 Baht (one-way). Cost-wise, there is no difference. (1Baht=250 kip as of Sept.9th, 2009)113

cross-border activities. On the other hand, not all the people use the FBII andbenefit from it. Some of the locals that live right by the FBII have never used
it, and instead cross the Mekong River by ferry or small boats from nearby
cross border points as they did before. Savannakhet residents who directly
benefit from the bridge are limited to people with a degree of financial status
and can afford Thai medical/health care services and enjoy shopping, dining,and sightseeing in Mukdahan and beyond. The questionnaire survey asked if
respondents have benefited from the FBII; 66% answered “Yes,” 18%answered “No,” 16% answered “don’t know.”
Table 7. “Have you Benefited from the FBII?” – Questionnaire Survey Results (n=50)
don’t know#

To cross the Mekong RiverYes

17 Never crossed the FBII

5 don’t know

8

For shopping in Thailand

8 Increase of competition,
3 loss of customers(Tuk-Tuk drivers,
2 restaurant owners)
1

4

Good business due to customer increase
Convenient to go to hospitals on Thai side
Convenient to go to festivals
Family member lives in Thailand andconvenient to come and go

#

No

#

1

ATM of Thai banks are now accessible

1

TOTAL

33 TOTAL

9 TOTAL

8(Source: Questionnaire Survey, 2009)8.2. Relationships of Various Social Issues
Economic activities in Savannakhet have been activated by the FBII and the
airport. Investments, which include the casino, hotels, restaurants and sugar
cane factories, have contributed to greater employment opportunities.
Nevertheless, it appears that the establishment of the FBII and the visa
exemption agreement has become a push factor for labor migration to
Thailand as higher wages in Thailand entice more local workers. The FBII
has also promoted the movement of people, particularly young women fromother regions in Lao PDR who are attracted to Kayson Phomvihane andtowns along National Routes 9 and 13. In some cases, they have fallen susceptible to human trafficking. As stated before, there may be no direct causal
relationship between the construction of the FBII and the increase in humantrafficking; the influx of people and the activation of the local economy may
have acted as a pull factor attracting young women to towns nearby the FBII.
The decision to open the casino in Savannakhet was inseparable from theopening of the FBII and the casino has undoubtedly generated employment
and income for Savannakhet. At the same time, however, it has created an
114

The Socio-Economic Impacts of the Second Mekong International Bridge

extremely crowded spot in a once sparsely populated province which has
increased the risk of spreading globally infectious diseases such as H1N1. In
response, a global surveillance system was reinforced and the Department of
Health of Savannakhet Province is now ready to take actions against suchoutbreaks of disease and infections.
Figure 25. Changes after the FBII
Casinoopen
Airportreopens
Convenience intransportation

Increase ininvestment

Increased
employment

Activation of
economyImproved access toThai medical services

Increase in humantrafficking

Increased risk ofinfectious diseases
FBII open

Increase intourists
Convenience
for travelingVisa
exemptions

Increase inlabor migration
Longing forhigh income

Increase inservice women(Source: Author)

8.3. Implications
In conclusion, since the inauguration of the FBII, some signs of economic
benefits to Savannakhet are evident but before the province can reap fullfledged benefits such as the development of local industry, more time isneeded. Nonetheless, Savannakhet is evidently changing at a rapid pace andfurther changes are anticipated as the SEZ further develops and as other economic activities increase. In light of such rapid change, continuously assessing the social impacts of economic development and being able to take swift
measures will be important.
As illustrated in a previous section, when the H1N1 flu struck the casino,
the close cooperation and sharing of information between the relevant parties
successfully prevented the virus from spreading, demonstrating the benefitsof quick and efficient cooperation and transparency between the public sector and the private sector. If Savannakhet can better foster such partnerships
not just within Savannakhet but also domestically with other regions andeven internationally, the people of Savannakhet will almost certainly enjoy
115

further benefits of the FBII as the province’s economy improves.

References
Asian Development Bank, Reviewing the Poverty Impact of Regional Economic
Integration in the Greater Mekong Sub-Region, Regional Synthesis Report,
2006.
＿＿＿＿, Guidelines for Preparing Performance Evaluation Reports for Public
Sector Operations, 2006.
＿＿＿＿, Lao PDR and Viet Nam: Greater Mekong Subregion: East-West
Corridor Project, Completion Report, November 2008a.
＿＿＿＿, Lao People’s Democratic Republic and Socialist Republic of Viet Nam:
Greater Mekong Subregion: East-West Corridor Project, Performance
Evaluation Report, December 2008b.
＿＿＿＿, Greater Mekong Subregion: Maturing and Moving Forward, Regional
Cooperation Assistance Program Evaluation, December, 2008c.
＿＿＿＿, HIV Prevention and Infrastructure: Mitigating Risk in the Greater
Mekong Subregion, May 2008d.
Department for Planning and Investment of Savannakhet Province, Statistical
Yearbook 2006 Savannakhet Province, 2007.
＿＿＿＿, Statistical Yearbook 2007 Savannakhet Province, 2008.
＿＿＿＿, Statistical Yearbook 2008 Savannakhet Province, 2009.
Department of Public Work and Transport in Savannakhet, Statistic of Freight,
Passenger, and Vehicle for crossing the border between Savannakhet (Laos)& Mukdahan (Thailand), 2009.
Foundation for Advanced Studies on International Development, Public Sector
and Private Sector Perception towards Tourism in Savannakhet, 2006.
Fujita, N., Evaluating a Project from Beneficiaries’ Perspectives – A Case
Study of the Second Mekong International Bridge –, Review ofDevelopment and Cooperation, 2009.
IDE-JETRO. 2008 Yearbook of Asian Affairs, 2008.
Lao National Tourism Administration, 2007 Statistical Report on Tourism in
Laos, 2008.
＿＿＿＿, 2008 Statistical Report on Tourism in Laos, 2009.
Sanum Investments Ltd., Executive Summary. Retrieved 17th December,
2009, fromhttp://www.sanuminvestment.com-a.googlepages.com/Sanum_
Investments_Summary.pdf
Savannakhet Province, Savannakhet Investment & Tourism Magazine, No.01,
116

The Socio-Economic Impacts of the Second Mekong International Bridge

October 2007.
Suzuki, M., Keola, S., Perspective of Savan-Seno Special Economic Zone andVientiane Industrial Park in the Lao PDR. In The Formation of Industrial
Clusters in Asia and Regional Integration (Chapter 4). Retrieved 17th
December, 2009, fromhttp://www.ide.go.jp/Japanese/Publish/Download/Report/pdf/2008_
0111_ch4.pdf
World Bank, World Development Indicators 2009 Database. Retrieved 6th
January, 2010.

117

About the Contributors(at the time of writing)Richard Hummelbrunner
Richard Hummelbrunner is a freelance consultant in the field of local andregional development. He has more than 30 years of professional experience
both in his native Austria and internationally, particularly in Europe and SubSaharan Africa. His most recent specialization is in policy consultancy as well
as evaluation and he has been active in promoting the use of systems concepts in these fields – as a practitioner, trainer and author.
Bob Williams
Bob Williams is a freelance consultant, specializing in empirical fieldresearch and evaluation technique, systems theory and practice, organizational development and action research. He works with evaluators of variousorganizations in New Zealand, Australia, Asia, USA and Europe, and is internationally well-known for his contributions in promoting systems approaches
and concepts in the evaluation field.
Patricia Rogers
Patricia Rogers is Professor of Public Sector Evaluation at RMIT University(Royal Melbourne Institute of Technology), Australia. For more than 25
years she has worked in public sector evaluation and research across a wide
range of programs and levels of government. She was the inaugural Chair of
the Systems in Evaluation Topical Interest Group of the American Evaluation
Association and her particular field of interest is in evaluation methods forcomplicated and complex programs and policies.
Nobuko Fujita
Nobuko Fujita is the Deputy Director of the International Development
Research Institute (IDRI) at Foundation for Advanced Studies on International
Development (FASID), where she has been involved in evaluation-related
research, training, and consulting.
Eriko Takahashi
Eriko Takahashi is a Junior Program Officer of the IDRI at FASID, where she
has been involved in evaluation-related projects and developmental research.
119