Measuring systems transformation: towards a preliminary framework
By: Undp Innovation
Tags: Sustainable Development, Evaluation, Systems Thinking, Monitoring And Evaluation

By Søren Vester Haldrup

At UNDP Innovation we are on a journey to shift our approach to innovation to help tackle complex development challenges. In short, we are moving away from single point solutions, and instead we are trying to figure out how to develop interventions that are more coherent with the nature of complex systemic challenges. On this blog, I’m sharing (in “real time”) my reflections, insights and ramblings emerging from this exciting and immensely difficult journey. You can find the past blogs in this series here, here, here and here.

How can we measure, monitor, and evaluate change in complex systems? What do intermediate results look like and how do we learn about whether and when these lead to more substantive change?

Everyone working on complex problems and systems transformation are (or should be) asking themselves these questions. The Center for Evaluation Innovation described the challenges well in a Forum on Systems Change Evaluation.

The problem is that measuring systems transformation and the path towards it is really difficult. It takes a long time for substantive change (higher level results) to materialize and we don’t necessarily know exactly what it will look like, how to measure it and, indeed, how to capture our contribution to it. Furthermore, the paths towards deep and broad change in a system are rarely clear (and never linear), and this makes it difficult to know if we are on the right track and whether activities and early results (e.g. tangible products and new skills) may generate substantive change further down the road.

There is a lively debate and growing literature on these challenges and how to tackle them. Evaluation experts offer useful insights. For instance, Michael Quinn Patton outlines six criteria for evaluating transformation, noting that OECD-DAC evaluation criteria are adequate for business as usual summative and accountability evaluations, but are inadequate for addressing major system transformations. Organizations are also doing interesting experiments, the Global Alliance for the Future of Food has recently launched new guidance on measuring food systems and the Open Government Partnership (OGP) is in the middle of an ambitious two-year developmental evaluation where they are trying to understand complex change processes and the OGPs contribution to these. As part of these efforts, new evaluation approaches are offered as more appropriate for complex change processes, in particular Developmental Evaluation (see here, here and here), and interesting methods are being used for measurement, including contribution tracing, qualitative comparative analysis and various forms of political economy analysis.

At UNDP Innovation we are also trying to tackle these challenges. In a series of so-called Deep Demonstrations we are collaborating with a range of UNDP country offices and partners to learn about how we can transform real-world systems in developing countries (external transformation) as well as transform how UNDP itself works to tackle these types of challenges (internal transformation) — for details see here and here. We know that our Deep Demonstrations alone won’t bring about system transformation, but we hope they can generate a lot of learning and allow us to develop a stronger blueprint for action. In this context, we’ve been trying to figure out what early results look like, how we can measure these, as well as whether early and inter-mediate results may contribute to more substantive change inside UNDP and out in the real world.

I’ve developed a (draft) framework to help us do this. The intention is to enable us to differentiate between different types of results, to specify change processes that we are contributing to as well as the factors that obstruct and has the potential to drive wider and deeper transformation.

I think about the path of change towards transformation along two dimensions: depth and breadth. Depth of change draws inspiration from the iceberg model, and it distinguishes between the tangible “artefacts” that we produce (such as a visual representation of a system, a report or briefing note), surface-level capabilities (skills and knowledge), deep-level capabilities such as mindsets, and finally actual behavioral change (I know it is at odds with the iceberg model to have real world behaviors at the deepest level, but it works for us).

Breadth of change distinguishes between change within different stakeholder groups: within a core team (each of our Deep Demonstrations are implemented by a core team situated within a UNDP country office), among the wider country office and UNDP, as well as among governments, development partners and other external change agents. Think of this type of change as the way a droplet creates concentric waves in a body of water.

We are interested in both types of change processes, and if we combine the two dimensions, we begin to get a more nuanced view of how early results may relate to more substantive change. The framework is presented in the visual below (bear in mind this is a work in progress).