Do They Work? Assessing the Impact of Transparency and Accountability Initiatives in Service Delivery

Anuradha Joshi\* Do They Work? Assessing the Impact of Transparency and Accountability Initiatives in Service Delivery

Transparency and accountability initiatives (TAIs) have emerged as a key strategy for improving public services, but the links between transparency and accountability and their impact on service delivery are often largely assumed. This article reviews a range of TAIs to assess their impact. It finds a mass of evidence suggesting that a range of accountability initiatives have been effective in their immediate goals, and that there is also strong evidence of impact on public services in a range of cases, but that evidence of impact on the quality and accessibility of services is more mixed.

Key words:

1 Introduction

In the past decade, strengthening public accountability has emerged as a key strategy for improving public services and making progress towards attaining the Millennium Development Goals (MDGs) (World Bank, 2004; Deverajan and Widlund, 2007). Increasingly, debates about strengthening accountability have focused on two types of initiatives: (a) increasing government transparency (bringing previously opaque information or processes into the public domain) and b) social accountability (citizen-led action for demanding accountability from providers). The number of publications that attempt to conceptualise, describe and assess social accountability has burgeoned (Arroyo and Sirker, 2005; Claasen and Alpin-Lardiés, 2010; Malena et al., 2004; O’Neil et al., 2007; Rocha Menocal and Sharma, 2008; Sirker and Cosic, 2007).

This enthusiasm for TAIs has overtaken an assessment of impact. Does increasing transparency or supporting social accountability initiatives lead to the desired outcomes desire? What are the assumed links through which these impacts are expected to occur? While there is a wide range of anecdotal evidence suggesting that such initiatives are successful, which seems to be the basis for them being replicated and capturing scholarly and popular imagination, there is little careful examination of these questions. Only a few recent studies have started this important task. This paper adds to this relatively small body of work and attempts to systematically examine the evidence on the impact of such TAIs in the field of public service delivery. The main argument of the paper is that despite the popularity of such initiatives, there is little evidence to make emphatic claims about the conditions under which TAIs will lead to effectiveness and impact. The reasons for this are several: vagueness about what an initiative means, the fragmented nature of the evidence, a lack of systematic attention to impact, and few comparative studies that focus on the identification of key enabling factors.

The paper is structured in the following manner. The next section briefly traces the diverse roots of the interest in accountability, particularly social accountability. Section Three discusses the concepts of transparency and social accountability and defines the boundaries of the evidence reviewed and Section Four outlines the impacts that might be expected and will be traced. Section Five forms the bulk of the paper and comprises of the empirical evidence which is largely organised around different accountability tools. In Section Six, I attempt to identify some themes that emerge from the review of the case material that are common across the cases which seem to have had the most impact. Finally, Section Seven concludes with some pointers for future research.

2 Ideological Origins

The impact of transparency and accountability on service delivery has always been an underlying motif in the literature on service delivery. Accountability took root as a central theme in debates after the World Development Report (WDR) of 2004, which identified failures in service delivery squarely as failures in accountability relationships (World Bank, 2004). By showing how the ‘long route’ of accountability - via elected politicians and public officials through to providers - was failing the poor, the WDR argued in favour of strengthening the ‘short route’ - direct accountability between users and providers. The WDR sparked off a spate of work that examined ways of strengthening the short route: from amplifying voice and increasing transparency to enhancing accountability (Sirker and Cosic, 2007; McNeil and Mumvuma, 2006).

What is interesting is that the importance of accountability, and related transparency, comes from two quite different ideological streams. On the one hand, New Public Management (NPM), which emerged in the 1990s, emphasised the use of market mechanisms in the public sector to make managers and providers more responsive and accountable (Batley, 1999). While many of the NPM reforms for accountability focussed on vertical accountability within organisations, such as performance-based pay, a sub-set, including citizen charters and complaint hotlines, related to downward accountability to citizens. In keeping with NPM’s intellectual roots, most of these downward accountability mechanisms were oriented to users as individual consumers who could either choose to use these mechanisms or exit in favour of other providers.

On the other hand, and at the same time, the failure of democratic institutions to deliver for the poor also resulted in calls for deepening democracy through the direct participation of citizens in governance (Fox, 2007). Innovative institutions such as governance councils in Brazil or village assemblies in India were viewed as embodying this spirit (Cornwall and Coelho, 2006; Manor, 2004). In parallel, social movements were arguing that governments had an obligation to protect and provide basic services as ‘rights’ that were protected under constitutions, rather than ‘needs’ which were at the discretion of officials to interpret and fulfil. Advocates of rights-based approaches to basic services identified ways in which rights - for example to health or education - could be legislated and progressively achieved. The rights-based, direct democracy approaches were distinct from NPM in that they emphasised the collective and public good dimensions of accountability.

While this double-branched provenance was timely in uniting practitioners and scholars in the importance of understanding and enhancing of transparency and accountability, it has simultaneously led to some looseness in what different people mean by the core concepts. Consequently, in the service delivery sub-sector, the literature which can be classified as ‘efforts to improve service delivery, increase citizen engagement, voice and accountability’ is vast. In order to establish criteria for including or excluding specific initiatives in this review, the first step has been to clarify the conceptual terrain and define what we mean by accountability and transparency initiatives.

3 Defining the Terrain How can we define TAIIs, our field of analysis? Transparency initiatives in service delivery are relatively easy to define: any attempts (by states or citizens) to place information or processes that were previously opaque in the public domain, accessible for use by citizen groups, providers or policy makers. Initiatives for transparency can be either proactive or reactive disclosure by government. Although freedom of information laws often play an important part in state or citizen-led transparency initiatives, this paper does not address attempts to legislate freedom of information or the overall impacts of such a law (see Calland and Bentley, this volume, for a discussion of freedom of information).

Accountability initiatives are more difficult to define. The clearest and most basic exposition of the concept of accountability is provided by Schedler (1999) in which public accountability comprises a relationship between the power holder (account-provider) and the delegator (account-demand). There are four elements to this accountability relationship: setting standards, getting information about actions, making judgements about appropriateness and sanctioning unsatisfactory performance. If one takes this conceptualisation as a benchmark, then a social accountability initiative ought to combine attempts to agree standards, gain information, elicit justification, render judgement and impose sanctions. Yet in the literature on accountability, there is considerable ambiguity about which of these elements are essential for a particular initiative to be considered robust. Often initiatives combining some but not all of these four components can be found to have an impact on public services.

Moreover, accountability for service delivery can be demanded from a range of stakeholders: politicians (over adopting inappropriate policies); public officials (over failing to deliver according to rules or entitlements, or to monitor providers for appropriate service levels); or providers (over not maintaining service levels in terms of access and quality). Further, initiatives to hold these multiple actors to account can be state-led or citizen-led.

To limit the vast task of assessing all relevant initiatives, I make several reductive cuts. First, I assess initiatives that are largely citizen-led and fall into the realm of social accountability. This is partly because the recent literature on service delivery has highlighted the failures of traditional accountability mechanisms and placed greater faith in demand-led accountability initiatives from below. The range of such social accountability initiatives is also relatively new and has not been examined closely for evidence of impact.

Second, I faced the question of whether to include initiatives meant to reform policies or establish new entitlements (e.g. collective action for a right to education law). If one takes a broad view of accountability, particularly accountability to citizens who expect elected legislators to deliver policies they have promised to pass, then such attempts to change policy from below would fall within the purview of accountability initiatives. We have many examples of social movements mobilising and succeeding in reforming laws (Gaventa and McGee, 2010). However, for the purposes of this paper, such cases are excluded because they are part of complex political processes in which citizen action forms only part of the story. Fortuitous political circumstances, leadership by particular individuals and other contextual factors are often critical parts of successes in unpredictable ways. While process tracing can show how citizen action contributed to particular outcomes, we cannot treat them as pure accountability initiatives in the strict sense of passing judgement on the conduct of public officials who have been delegated powers. They fall within the remit of the normal politics of policy-making.

Third, the emerging literature on social accountability is muddy on the issue of whether participation in policy-making is part of social accountability (Joshi, 2008). Some limit the term social accountability to citizen groups monitoring the use of public authority (Peruzzotti and Smulovitz, 2006). Others include participation in policy-making, policy advocacy and deliberation as part of the social accountability terrain (Arroyo and Sirker, 2005; Malena et al., 2004). Yet others treat the question as an empirical one, asking whether particular institutional spaces are used for certain kinds of engagement, inclusion and accountability (Cornwall and Coelho, 2006).

This paper restricts itself to examining initiatives that are explicitly oriented towards monitoring and demanding accountability for performance in services that are widely accepted as entitlements - either ‘hard’ entitlements through laws, or ‘soft’ ones through government rules or widely accepted norms. In doing so, we need to differentiate what we call accountability initiatives, which involve monitoring and sometimes sanctions, from the broader literature on participation and citizen engagement. While participatory approaches might be part of accountability initiatives or accompany them, they go beyond accountability work. Further, the literature often discusses both ‘voice’ and accountability initiatives together - raising the issue that ‘voice’ could be raised in the interest of participation in policy formulation as well as accountability (Rocha Menocal and Sharma, 2008; Green, 2008).

Thus, we explicitly exclude attempts by citizen groups to link users with government services (e.g. encouraging women to go to public hospitals for child deliveries) or attempts by providers to engage citizens in the delivery of services or participate in decision-making (e.g. involving households in spreading messages about hygiene and sanitation). We also exclude examples where citizen groups are mobilising and self-providing services, or helping access government services (e.g. community health insurance groups for paying for access to public health care).

On the one hand narrowing down our focus in this fashion makes the evidence to be reviewed more manageable. On the other hand, however, the problem of attribution remains: in many cases, accountability initiatives are one part of a package of strategies that citizen groups use to gain better services which include mobilisation, political advocacy, intermediation, self-provisioning and participation. This makes the task of isolating the impact of accountability initiatives difficult; and harder because a large part of the evidence comes from case studies involving narrative descriptions of the impact of citizen-led initiatives that do not separate out the contribution of different strategies. For example, HakiElimu, a CSO in Tanzania that works on public education, appears to have made some impact on the education system - for example, improved teacher-pupil ratios - through a strategy of budget analysis, research, media dissemination, policy analysis, monitoring and advocacy, but it is not clear specifically how its social accountability activities contributed to that impact (IBP, 2008). It also raises the important question that further research needs to examine: what is the relationship between transparency, accountability and participation in improving public services?

A few further qualifications and parameters are in order. For one, the evidence on the impact of many accountability initiatives is oriented around the tools of accountability such as Public Expenditure Tracking Surveys (PETS), citizen report cards, social audits and community monitoring. We have followed the categorisation in the literature on this issue, although many experiences combine different tools. Another issue is one that is of relevance to health and education specifically: that of uptake by the poor. When public health or education services are not strong, the poor often choose to either go elsewhere, seeking out private practitioners of uncertain quality, or to opt out completely, for example by not sending their children to school. Thus accountability initiatives targeting health and education are often attempting to both improve the quality of services, and increase uptake so that accountability mechanisms can come into play. Separating out the impact of these different strategies can be difficult as we shall see in the cases reviewed. Finally, it should be clarified that this review is not exhaustive, but illustrative. The next section reviews the expected impacts of accountability and transparency initiatives and the theories of change that underpin them.

4 What Impact Should We Expect?

The links between transparency and accountability and their impact and effectiveness in the service delivery arena are often largely assumed in the literature rather than explicitly articulated. Most generally, the assumed link leads from awareness (through transparency and information) to empowerment and articulating voice (through formal and informal institutions) and ultimately accountability (changing the incentives of providers so that change their behaviour and respond in fear of sanctions). Yet, this chain of causation is seldom explicitly examined. In fact, many initiatives are focussed at increasing transparency and amplifying voice, without examining their link with accountability and ultimately responsiveness.

There is also lack of clarity in what the expected impacts actually are. Some studies look at the strengthening of the media as expected impacts (CommGAP, 2007) while others consider an active and independent media to be a factor in other impacts such as improved responsiveness. This confusion arises partly because studies of impact rarely look at the impact of accountability and transparency alone, looking instead at the impact of a range of governance interventions such as changing incentives for public officials or improved management processes. Moreover, different studies identify a wide range of expected impacts, from improving the quality of governance (Malena et al., 2004) to increased empowerment of citizens (Gaventa and Barrett, 2010). Examining this diverse literature however, one can classify expected impacts into three broad categories.

The first and often strongest set of claims in relation to service delivery is that accountability and transparency initiatives expose and reduce corruption. Transparency in particular is expected to help in exposing corruption, through highlighting discrepancies in public accounts and triggering more formal accountability mechanisms such as audits and investigations. In this narrative of the role of transparency however, there is an underlying assumption: that the information made public through transparency initiatives will be used by concerned citizens through exercising voice and expressing outrage at misconduct. There is also an assumed relationship between increased voice and improved accountability but, as Fox (2007) puts it, transparency will not always lead to accountability. Even when citizens protest against misconduct, there needs to be a pressure for public authorities to respond and sanction those responsible. Certain transparency and accountability mechanisms, especially when supported by the threat of credible sanctions, are expected to shift the incentives of public officials by increasing the probability of exposure and the cost of being found guilty.

The second, related set of claims is that transparency and accountability lead to increased responsiveness on the part of providers, in the form of improved access and quality of services, and consequently to better developmental outcomes. These claims are premised on a number of changes happening at intermediate levels including improved policy, practice, behaviour and power relations (Rocha Menocal and Sharma, 2008). Underlying them are a number of assumptions: that the exposure of poor performance will lead to greater responsiveness; that failures in service delivery are due to poor motivation on the part of public officials and not lack of resources or capacities; or that the existence of accountability and transparency mechanisms will have a deterrent effect on errant officials and make them behave better. Yet, there is no clear reason why all of these assumptions will hold true in specific cases. Public providers may be immune to exposure of poor performance, increased citizen voice may be met with backlash and reprisals, lack of resources may constrain the capacity of public officials to respond, and accountability mechanisms may not be enough of a deterrent. In addition, there is an assumption that the outputs of public services, such as increased enrolment, will lead to improved developmental outcomes in health and education. Yet outcomes may be contingent on other factors unrelated to quality or access, and might need complementary interventions. Finally, a related, often unstated assumption is that effective institutions are transparent and accountable - in other words, that ‘all good things go together’. Yet we know that there is a tension between effectiveness and accountability and need to tease out the conditions under which the two move in parallel (Mainwaring, 2003). Thus the claim of better accountability and transparency systems leading to improved outcomes in service delivery is based on a series of step-by-step assumptions that are subject to question in specific cases.

The final set of claims is that TAIs lead to greater empowerment of poor people, greater awareness of rights by users, and the construction of citizenship through greater engagement in service delivery. The logical chain linking transparency to empowerment is clear: information is power. When better information about rights and processes is disseminated, awareness about entitlements is likely to increase. In the case of accountability initiatives however the logic is less straightforward: does the active practice of holding public providers to account lead to citizens getting empowered and more likely to engage with other processes related to citizenship? Or might the causal relationship be the other way around, with citizens who are mobilised and already participating in other ways being more likely to engage in accountability activities? While we simply do not know much about when citizen groups engage in social accountability activities, to the extent they are collective and aggregate citizen voice, they can be said to be empowering of the poor.

The various TAIs reviewed in this paper have different underlying theories of change about impact. For example, citizen report cards and community score cards are based on the assumption that providers care about their rankings either because of their reputation or potential loss of users. Community monitoring implies more of a watchdog role that can pitch community members in an adversarial relationship vis-à-vis providers. PETS are largely meant to expose blocks in fund flows and corruption, improving provider behaviour through fear of exposure. Thus not all accountability initiatives are expected to deliver on all three categories of impact, and the evidence reviewed suggests that their impacts do vary on the three dimensions.

5 Assessing the Evidence

In the past decade, there has been a lively debate in the field about the best methods of evaluating accountability and transparency initiatives (Foresti et al., 2007; O’Neil et al., 2007; Holland and Thirkell, 2009). Despite this, the actual empirical evidence on impact of TAIs on service delivery is extremely fragmented and limited. The studies examined for this paper use a variety of approaches ranging from the strictly quantitative to the highly qualitative, and from external ex-post evaluations to participatory, practitioner-based assessments of impact. The evidence comprises largely of four kinds of studies - qualitative case studies, randomised controlled trials, ex-post quantitative/qualitative evaluations, and participatory evaluations - and each has advantages and disadvantages. For example, while participatory evaluations are useful for highlighting impacts that are important for users, they are often critiqued for bias in reporting successes. Randomised Control Trials (RCTs) are often statistically sound, but can be narrowly focussed and often do not address seriously the question of causality. Most of the evidence is qualitative in the form of case studies of particular initiatives. There are relatively few quantitative studies that examine the impact of TAI s through ex-post evaluations, and only a small but growing number of RCTs. Moreover, and importantly, the quality of the evidence varies considerably. While RCTs and similar evaluations may be methodologically robust, and some qualitative comparative case studies are carefully constructed and explored, other case material can be more descriptive than analytical and requires the extraction of evidence on impact, rather than being impact-focused.

Most of the evidence, rather than being focussed on sectors (e.g. what works in health or education) is focused on specific interventions (community scorecards or PETS) and we follow this categorisation in this review. Further, although most initiatives surveyed here have been effective in that information was disseminated, or the monitoring happened, the evidence on impact is more mixed. Initiatives targeted at exposing corruption have been fairly successful, whereas interventions intended to improve service outcomes and responsiveness have not always demonstrably had the intended impacts.¹

From the existing assessments, there are few attempts to compare the impacts of different mechanisms or reach broader conclusions about the factors that contribute to success in specific strategies. We briefly survey these recent comparative overview studies before examining impacts by intervention type.

Rocha Menocal and Sharma (2008), evaluating the impact of five donor-led voice and accountability initiatives, conclude that donor expectations of such initiatives in terms of poverty alleviation goals is too high. None of the interventions studied could clearly demonstrate impact towards the MDGs. Rather, they conclude that the contribution of these initiatives was in terms of more intermediate changes such as changes in the behaviour and practice of public officials and some changes in policy. They find that when voice and accountability interventions are targeted directly at women and marginalised groups, there is some impact on empowerment, although it is not clear what indicators of

¹ There is a growing interest in this area and several projects are underway which explicitly attempt to evaluate impact. For example, Global Integrity along with the World Bank is developing indicators to assess the impact of access to information in health and education services (Global Integrity, 2010). The DFID-funded Governance and Transparency Fund (GTF) projects have institutionalised baseline data collection and are developing indicators in order to assess impact of their work in the future. Although almost all the 38 projects funded under the GTF have completed their midterm reports, they still do not offer enough evidence to assess impact. empowerment are used in these studies. In general, there appears to be limited evidence of impact on broader developmental outcomes.

In a report evaluating 100 case studies that mapped the outcomes of citizen engagement, Gaventa and Barrett (2010) find over 30 cases in which significant impacts were made on service delivery, including in the health and education sectors. For example, in Brazil, participatory governance councils have been significant in improving access to and quality of health care services. In Bangladesh, parents of girls in schools mobilised to monitor teacher attendance and discourage absenteeism. While the methodology used to synthesise comparative findings advances the ways in which qualitative case material can be analysed, the cases cover all forms of citizen engagement and the study does not isolate the impact of TAIs.

There are a few examples of studies examining whether top-down accountability initiatives work better than bottom-up initiatives and the evidence seems mixed. In an interesting examination of whether top-down or bottom-up accountability mechanisms work better, Lassibille et al. (2010) report on a random experiment in which different approaches were compared in schools in Madagascar. The findings showed that demand-led interventions led to significantly improved teacher behaviour, improved school attendance and test scores when compared with top-down interventions which had minimal effects. It appears that although managers had better tools to hold lower-level staff accountable, they were unlikely to do so without greater incentives.

The overall evidence suggests that TAIs score higher on effectiveness - in that they are often well implemented and reach first order goals such as the use of complaint mechanisms or the exposure of corruption - than on impact, such as improving responsiveness of providers or of services themselves.

The rest of this section is organised around the new mechanisms of accountability and transparency focussing on social accountability, starting with simpler efforts to increase transparency and ending with more complex initiatives meant to improve accountability.

5.1 Information Dissemination There have been a number of recent studies attempting to assess the impact of transparency and information on citizen engagement and service provision, and the evidence seems mixed. An RCT examining the impact of a community-based information campaign on school performance in three Indian states found that the intervention had an overall positive impact (Pandey et al., 2009). The most effect was on teacher presence and effort, whereas the impact on pupil learning was more modest. By contrast, in another RCT study of the impact of information on the ability of communities to engage in accountability mechanisms and subsequent impacts on quality of services in India, Banerjee et al. (2010), show that providing information - about the education programme as well as the level of child achievement in literacy and numeracy - had little impact on engagement with the school system or demanding accountability. Rather, when community volunteers were trained to carry out remedial classes outside the classroom, it had a greater impact on children’s literacy and numeracy skills. The paper concludes that communities face serious constraints in engaging to improve the public school system even when they have information and a desire to improve education. In another study, Björkman and Svensson (2009) found that information dissemination of the quality of health services in Uganda led to reduced absenteeism and better outcomes. As Khemani (2008) points out in her comparative paper of the Indian and Ugandan cases, these different studies of community engagement with information came to two strikingly different conclusions.

5.2 Public Expenditure Tracking Surveys

Public Expenditure Tracking Surveys (PETS) have been used in many countries to highlight leakages and gaps in the delivery of funds to the local level. In a survey of PETS in Africa Gauthier (2006) notes that in almost all cases, they have highlighted the leakage of resources reaching facility levels.

Reinikka and Svensson’s (2005) pioneering examination of education expenditures in Uganda using PET surveys showed that on average only 13 per cent of the actual expenditure meant for schools actually reached them. When this information was made public through an experimental information campaign, the funds reaching schools increased substantially up to 90 per cent. The Ugandan government has made resource information at each tier of facilities public. Although this widely cited case has been questioned by subsequent research (see Hubbard, 2007), the broad findings of the study still stand. In Malawi, the Civil Society Coalition for Quality in Basic Education (CSCQBE) has used PETS three times to achieve impact, improving its methodology each time (IBP, 2008). PETS survey information was used to successfully resist the closure of teacher training colleges, get teacher salaries paid on time and make budget allocations for students with special needs. In 2004, the government started conducting its own tracking survey following CSCQBE’s success. Early indications of PETS in Tanzania for health and education spending carried out over two periods (1999 and 2001) suggest that corruption has reduced considerably (Gauthier, 2006).

These cases however are exceptions. Despite their success in identifying leakages and publicising them, the evidence suggests that PETS have led to reforms in only a few countries, mainly due to lack of political will (Gauthier, 2006).

5.3 Complaints Mechanisms

Another popular measure for increasing accountability of providers comes from various complaint mechanisms including complaint hotlines and complaint management systems. In combination with citizen charters which lay out service delivery norms for basic services, these are intended to bring problems quickly to the attention of relevant personnel and set up standards for addressing complaints, which are monitored by senior managers. Such technology-based mechanisms are usually limited to urban areas.

Complaint mechanisms have been initiated both by citizen groups and public organisations. In Hyderabad, Metro Water started a complaint hotline which offered a formal accountability mechanism for citizens. By using this direct link with citizens, managers were able to hold frontline providers accountable. The findings of the evaluation suggested that the performance of frontline workers improved and corruption was considerably reduced (Caseley, 2003). In Mumbai, a citizen group initiated the Online Complaint Management System which streamlined all complaints on urban public services into an online database which could be used to compile data on time taken to address complaints compared to set norms. An early World Bank study found that the system was successful in putting pressure on public officials to deal with complaints on time. In another initiative, Lok Satta, a citizen group in Andhra Pradesh worked with municipal authorities to publicise citizen charters for forty common public services in one hundred municipalities, which were combined with efficient complaint mechanisms, training of citizens to monitor services and a compensation clause that pays citizens Rs. 50 per day of delay in public services. A review of this experience suggests that the charters have worked better in urban areas than in rural areas because of greater awareness. It also found that the compensation clause to be recovered from the salary of the employee at fault has been ‘properly implemented’ (Sirker and Cosic, 2007). There is other research showing that citizen dissatisfaction with services at the local level often takes the form of individuals complaining loudly and publicly about their treatment by frontline providers, what Hossain (2009) calls ‘rude accountability’. Such naming and shaming might be the only option for very marginalised groups and seems to work particularly well for women; however the broader impact and potential for scaling up such a strategy remains to be researched.

5.4 Citizen Report Cards

Citizen Report Cards (CRCs) follow the practice of consumer satisfaction surveys in the private sector. Such surveys can be carried out by citizen groups or independent bodies. The expectation is that public exposure of comparative poor performance will spur lagging public agencies to perform better. The distinguishing characteristic of CRCs is that they are based on individual opinion and usually done at the macro level, but evidence of their impact is mixed.

A positive review of the CRCs in Bangalore (Ravindra, 2004), where they were first pioneered by a citizen group called the Public Affairs Centre, shows that they have had considerable impact on improving public services. As a UN evaluation report indicated, not only did public satisfaction with services improve, but the incidence of corruption appeared to have declined (UN, 2007). Further, the evaluation identified that citizen mobilisation and awareness had increased as a result of the report cards, and more interestingly, public agencies had become more transparent and willing to share information with citizen groups. Two factors seem to be critical in influencing the impact of report cards as identified by this assessment: the presence of an active and independent media and civil society organisations that were willing to use information to press for accountability and reforms, and the presence of public officials who were catalysed by evidence of the poor performance of their agencies and willing to reform.

A more mixed assessment of provider-based report cards is provided by McNamara (2006) who has assessed their use in the health sector in the United States. She finds that the impacts depend to a large extent on the indicators that are actually used in evaluating providers. In some cases, providers have improved services in response; in others, providers have worked towards improving rankings by using strategies that improve scores but might undermine overall services. What is interesting is that uniformly, it appears, publicly generated performance data has not influenced citizens’ choice of facilities, even though realistic choice is available. In the developing world, report cards have been used to rank hospitals in Uganda (Uganda DISH, 2003). Although no systematic studies of their impact on services have been done, it appears that the average score of providers climbed substantially in the two report card periods.

Yet the findings of impacts on service delivery based on report card type initiatives have to be interpreted cautiously. As Deichmann and Lall (2007) show, citizen satisfaction is in part determined by factors unrelated to actual service quality experienced by the households. More recent efforts to use CRCs are moving away from satisfaction surveys to more objective indicators of the actual quality of services received, as is evidenced by the Delivering Services Indicators proposed for education and health services in Africa (Bold et al., 2010).

5.5 Community Monitoring Community monitoring is slightly different from the CRCs in that it aims to monitor ongoing activities of public agencies rather than rate outcomes. Often, community monitoring is used as a way of ensuring that ongoing performance maintains normal standards and is focussed on observable features, for example teacher or doctor attendance, quality of construction in facilities or appropriate procedures being followed. In particular, community monitoring has been useful in bringing to light instances of corruption or diversion of public resources.

Community monitoring by the Uganda Debt Network (UDN) has been successful in improving facilities at the local level. UDN trained community workers in monitoring, which led to the identification of ‘shoddy work’ by contractors in the construction of classrooms and health posts (Renzio et al., 2006). In several cases community monitoring reported some of the equipment allocated to a health post as missing, and official investigation led to recovery of the missing material. A random experiment in Kenya found that hiring teachers on short contracts combined with community monitoring had significant impacts on student achievements (Duflo, Dupas and Kremer, 2008). By contrast, a widely cited study on citizen monitoring of road projects in Indonesia found that citizen monitoring had little average impact compared to increasing government audits (Olken, 2007).

Community monitoring can often improve the quality of services. In an experiment in Uganda, Björkman and Svensson (2009) found that when local NGOs encouraged communities to engage with local health services, they were more likely to monitor providers. As a result, provider absenteeism declined and responsiveness increased in terms of shorter waiting times and greater efforts to respond to community needs. Usage of public health services also increased, and was reflected in better health outcomes such as reduced child mortality. These findings reflect a vicious cycle in some public services, whereby poor service quality leads to lack of uptake and interest, which in turn results in further worsening of quality and lack of accountability. When uptake increases, however, accountability demands are also likely to increase: as a corollary, when accountability exists, uptake will also increase, leading to a virtuous cycle.

Duflo et al. (2010) found that improving incentives for teachers combined with strong accountability mechanisms improved teacher attendance rates in schools in India. In an RCT, cameras were given to schools to take digitally dated pictures of teachers at the beginning and end of each day. Teachers were guaranteed a base pay with additional increments linked to attendance rates. Absence rates in treated schools dropped to 21 per cent - compared to a little over forty at baseline and in comparison schools - and stayed constant even after fourteen months of the programme. This study illustrates that accountability mechanisms alone may not be sufficient to result in provider responsiveness and subsequently better services. Greater capacity and incentives on the part of providers may be necessary accompaniments to accountability.

5.6 Public Hearings and Social Audits

In India, the NGO Mazdoor Kisaan Shakti Sangathana (MKSS) pioneered the strategy of using jan sunwais (public hearings) to hold public officials accountable for local level implementation of programmes. Jan sunwais operate by first gathering information about budgets and expenditure in public programmes and presenting and verifying these in a public gathering at which all relevant stakeholders - public officials, elected leaders, private contractors and workers - are present. These early public hearings had significant impact in exposing corruption in public works programmes, and in some instances even getting public officials to return the money that they had appropriated. Apart from the widely-publicised work of MKSS, a grassroots Delhi organisation called Parivartan has also held public hearings on the implementation of the Public Distribution System (PDS), a large food subsidy programme intended for the poor. The depth of corruption exposed through the process led to improvements in the operation of PDS as well as institutionalisation of a system of monthly ‘opening of the books’ for public scrutiny (Pande, 2008). Public hearings have also been held by the Right to Health movement in India in an attempt to expose the poor access to healthcare for the poor and provide an evidence base for advocating reforms. There has been no clear study of their impact (Duggal, 2005). While initially such public hearings were informally organised, due to their success and widespread credibility, they have been institutionalised in the form of social audits in some national programmes, most prominently the National Rural Employment Guarantee Scheme. In Andhra Pradesh, where the state has taken a lead in institutionalising social audits, a study found that they have led to a statistically significant increase in employment generated, as well as an increase in the exposure of corruption within the programme and a significant amount of programme funds being recovered (Singh and Vutukuru, 2010).

5.7 Community Score Cards

Several groups are now using Community Score Cards (CSCs) to assess the performance of local public services. CSCs are a hybrid of CRCs, community monitoring and social audits. Besides assessing levels of service satisfaction by users, a CSC process involves community meetings in which performance of public services is discussed among providers, users and other stakeholders and includes self-evaluation of performance by providers, as well as the formulation of an action plan based on scorecard outputs. A key feature distinguishing CSCs is the collective engagement of both providers and users in designing and using the cards.

Analysis of the use of CSCs in primary health care services in Andhra Pradesh found that there were stark discrepancies between the self-evaluation of providers and the evaluation of communities (Misra, 2007). Subsequent discussion resulted in an action plan in which providers agreed to undergo training to improve their interactions with users, to change timings of the health centre to better meet community needs, to institutionalise a better grievance redressal system and to display medicine stocks publicly. Overall the process resulted in increased user satisfaction levels and better understanding of the constraints providers face. In Madagascar, assessing services using the Local Governance Barometer, a similar process that involved local officials and communities, found that there were very low levels of perception of accountability by citizens (Dufils, 2010). The resultant action plan had several positive impacts: effective channels of collaboration and communication were developed; complaint processes were improved; and recruitment procedures for municipal staff were improved, with more women being hired at senior levels.

To summarise, there is a sufficient mass of evidence now suggesting that the new accountability mechanisms have been effective in their immediate goals: citizen report cards have been implemented and disseminated, community monitoring has been carried out and information has been publicised. There is strong evidence of impact on public services in a range of cases. Mechanisms helping to expose corruption have had the clearest impact in terms of bringing to light discrepancies between official accounts and the reality of practice. Initiatives have also been quite successful in increasing awareness of entitlements, empowering people to demand accountability and claim rights as well as increase the practice of active citizenship. Where the evidence is more mixed however is the impact on actual quality and accessibility of services themselves. Despite demands for accountability and exposure of corruption, experience suggests that the kinds of direct social accountability mechanisms discussed above have little traction unless they are able to trigger traditional accountability and impose formal sanctions. Factors such as these that have an impact on the success of social accountability initiatives are taken up in the next section.

6 Emerging Themes

The main finding of this review is that the wide range and diversity of initiatives in the service delivery sector make it very difficult to establish conclusions about key factors that matter in achieving impact, even within similar initiatives. The initiatives themselves vary widely even within the same broad subtype, for example within community monitoring of services. Caution is advised, as not all initiatives will result in the same kind of impact (Khemani, 2008). In fact, most studies conclude that there is an urgent need to examine why certain TAIs succeed and what factors seem to matter.

The overarching lesson seems to be, not surprisingly, that the context matters. Political economy factors, the nature and strength of civil society movements, the relative political strength of service providers (e.g. teacher unions), the ability of cross-cutting coalitions to push reforms, the legal context, and an active media all appear to have contributed in varying degrees to the successful cases. Despite these constraints, some general themes that are common across several cases can be drawn from the existing review.

First, several studies highlight that citizen-led initiatives have impact when there is willingness from the public sector to support attempts to improve accountability. This could be in the form of combined top-down and bottom-up approaches (Lassibille et al., 2010) or in the form of sympathetic reformists within government (Pande, 2008). In some cases, successful demands for accountability from below were accompanied by changing the incentives of public providers through carrots (Duflo et al., 2010) or sticks (Sirker and Cosic, 2008).

Second, most available evidence of impact is based on collective action rather than individual action. This could be because collective accountability mechanisms are better suited to use by the poor and vulnerable and are more likely to result in improved public good benefits as opposed to the private benefits that can be the outcomes of individual action (Joshi, 2008). In particular, collective accountability is more likely to result in reduced corruption and increased empowerment of people as citizens. It is possible that this conclusion arises from a bias in the literature that has privileged collective action over individual voice and accountability measures. A research question that remains is whether individual action is effective and impactful beyond the individual benefit derived, for example from personally getting better attention from the doctor or accessing one’s entitlement to school textbooks.

Third, accountability or transparency mechanisms that have the potential to trigger strong sanctions are more likely to be used and be effective in improving responsiveness by providers. Without the threat of effective sanctions and the resulting impacts, citizen mobilisation is difficult to sustain in the long run. When repeated exposure of misconduct is met with inaction, continued use of publicity as an accountability strategy is likely to die. Social accountability mechanisms have impact when they can trigger traditional accountability mechanisms such as investigations, inspections and audits. Fourth, information and transparency are a necessary but not sufficient condition for desired outcomes to be realised. Contextual factors shape whether information will be used by citizen groups to demand accountability. However, an active and independent media seems to be a critical part of several of the successful cases.

Finally, and most importantly, accountability and transparency initiatives without corresponding support for increasing the capacity to respond can lead to inaction and frustration on the part of providers (Gaventa and Barrett, 2010). Often successful initiatives have constructive engagement and dialogue between providers and users about potential reforms as part of the process of demanding accountability (Björkman and Svensson, 2009; Fung, 2001; George, 2003). The CSC approach seems to encapsulate the best of this strategy, by attempting to surface discrepancies between provider and user perceptions of service quality and working towards solutions through collective discussion and debate. The evidence to date suggests that there is a balancing of tension between demanding accountability and engaging with providers to understand the constraints they face. Information, dialogue, negotiation and compromise are key elements of such engagement. What this points to is that conceptually we need to understand the impact of accountability on its own, but also tease out its links with other forms of participation.

7 Future Research This review finds that there are serious gaps in our understanding of the impacts and effectiveness of TAI s in service delivery which will have to be addressed to meaningfully support transparency and social accountability work. The gaps are both conceptual and empirical.

The conceptual gaps are critical, because they make comparability of the available evidence difficult. Although there seems to be a consensus about the importance of social accountability in improving service delivery, there is little consensus about what it exactly means - whether it includes all citizen engagement including participation in policy formulation, or is limited to citizen monitoring. Moreover, as discussed earlier, while definitions of accountability usually include four elements (standards, information, justification and sanctions) there is some vagueness as to which of these form a core part of social accountability. Without a clarification of the conceptual terrain, assessing the evidence systematically remains a challenge. In parallel, there is little conceptual exposition in the form of theories of change that underpin descriptions of accountability and transparency initiatives. Because the assumptions behind specific initiatives are not made explicit, it becomes difficult to judge the extent to which initiatives were successful in the intermediate steps. There are many normative assumptions about impact and confusion about means and ends. There needs to be more explicit investigation of impact on outcomes of services rather than simply outputs. What are the kinds of interventions that are likely to improve quality of education and learning outcomes, rather than simply deal with teacher absenteeism?

The empirical gaps add to the conceptual ones. As shown throughout, there are few studies that look explicitly at impact or effectiveness; evidence has to be culled from existing accounts that are not oriented to evaluating impacts.

One gap in the service delivery area arises from the narrow ‘object’ of citizen-led accountability activities, namely the state. Most of the evidence on social accountability comes from citizen-led action that targets the state or state providers. As a first cut, this state focus is useful. However, we know that increasingly the state is only one of an array of legitimate actors who exercise public authority and provide services. Privatisation, decentralisation and varieties of co-production increase the disjuncture between traditional accountability mechanisms and the new forms of pluralistic governance. We have unfortunately little understanding of how social accountability initiatives fare when they target a diverse set of non-state actors.

Further, despite the growing literature on the wide range of social accountability initiatives reported in this paper, there is little attempt to analyse these comparatively. How do specific contexts influence the potential for success of particular types of initiatives? For example, are CRCs more likely to succeed in contexts where there is perceived competition among public agencies? Is the community scorecard methodology more appropriate to places where democracy has not established roots?

Neither are they assessed comparatively for their durability or scalability(^2). Are the kinds of initiatives that encourage constructive engagement between citizens and public

(^2) Thanks to Rakesh Rajani for drawing my attention to this point. agents more likely to be sustainable in the long run compared to those that take a more confrontational stance? Are certain kinds of initiatives more amenable to scaling up than others? Comparative evidence on the alternatives to particular interventions in various contexts and service sectors, along different dimensions of success, seems to be essential in order to build a body of knowledge that will be useful for donors, practitioners and public agencies.

A significant problem is that the case material reported here tends to take a snapshot view of social accountability initiatives, limiting analysis to a specific intervention and its subsequent unfolding of outcomes. Part of the reason for this is many of these initiatives are externally driven and circumscribed by project cycles (or research timeframes in the case of RCTs). Thus most studies do not examine a longer trajectory of citizen-state relationships or civil society networks that underpin the outcomes in specific social accountability initiatives, neither do they examine the influence of citizen-led activities outside the narrow scope of the initiative. Other research has shown that the history and trajectory of citizen-state interaction and informal relationships between societal groups and state actors matters in understanding outcomes (Gaventa and McGee, 2010; Unsworth, 2010; Booth, 2011).

This lack of attention to histories and patterns of citizen-state relationships hides a more substantial gap: we do not have robust understandings of the origins of social accountability initiatives. We do not have systematic evidence or propositions for why citizen groups engage in social accountability in some settings and not others, over some issues and not others, or at some points in time and not others. For example, emerging research suggests that participation of citizen groups in policy formation processes ‘upstream’, will increase the likelihood of their engagement in social accountability activities ‘downstream’ (Houtzager, Joshi and Lavalle, 2008). The answers to these questions is important because it enables us to understand the triggers of social accountability activities and the likelihood that institutions created to encourage social accountability will be occupied.

References Arroyo, D. and Sirker, K. (2005) 'Stocktaking of Social Accountability Initiatives in the Asia and Pacific Region', World Bank Institute Working Paper 255, Washington DC: World Bank.

Banerjee, A., Duflo, E., Glennerster, R., Banerji, R. and Khemani, S. (2010) 'Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation of Education in India', American Economic Journal: Economic Policy 2 (1): 1-30.

Batley, R. (1999) 'The New Public Management in Developing Countries: Implications for Policy and Organizational Reform', Journal of International Development 11 (5): 761-765.

Björkman, M. and Svensson, J. (2009) 'Power to the People: Evidence from a Randomized Field Experiment on Community-based Monitoring in Uganda', Quarterly Journal of Economics 124 (2): 735-69.

Bold, T., Gauthier, B., Svensson, J. and Wane, W. (2010) 'Delivering Service Indicators in Health and Education in Africa: A Proposal', World Bank Policy Research Working Paper 5327, Washington DC: World Bank.

Booth, D. (2011) 'Working with the Grain and Swimming Against the Tide', Paper for presentation to the panel 'The Politics and Governance of Public Services in Developing Countries,' International Research Society for Public Management Conference, Dublin 11-13 April 2011.

Caseley, J. (2003) 'Blocked Drains and Open Minds: Multiple Accountability Relationships and Improved Service Delivery Performance in an Indian City', IDS Working Paper 211, Brighton: Institute of Development Studies.

Claasen, M. and Alpin-Lardiés, C. (eds) (2010) Social Accountability in Africa: Practitioners Experiences and Lessons, Pretoria: Idasa and Affiliated Network for Social Accountability.

CommGAP (2007) 'Evaluation Framework for Governance Programmes: measuring the contribution of communication', Communication for Governance and Accountability Program, Washington DC: World Bank.

Cornwall, A. and Coelho, V. (eds) (2006) Spaces for Change? The Politics of Participation in New Democratic Arenas, London: Zed Books.

Deichmann, U. and Lall, S. (2007) 'Citizen Feedback and Delivery of Urban Services', World Development 35 (4): 649-662.

Devarajan, S. and Widlund, I. (eds) (2007) Politics of Service Delivery in Democracies, Stockholm: Ministry for Foreign Affairs. Dufils, J. (2010) 'Local Governance Barometer: Measuring Governance in Madagascar' in Claasen, M. and Alpin-Lardiés, C. (eds) (2010) Social Accountability in Africa: Practitioners Experiences and Lessons, Pretoria: Idasa and Affiliated Network for Social Accountability.

Duflo, E., Hanna, R. and Ryan, S. (2010) 'Incentives Work: Getting Teachers to come to School', Cambridge, MA: Massachusetts Institute of Technology, http://econ-www.mit.edu/files/5582 (Accessed October 2011).

Duflo, E., Dupas, P. and Kremer, M. (2008) 'Peer Effects, Teacher Incentives and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya', National Bureau of Economic Research Working Paper 14475, Cambridge, MA: NBER.

Duggal, R. (2005) 'Public hearings and litigations: the new medicine for policy reform' Express Healthcare Management January 2005: 1-15.

Foresti, M., O’Neil, T. and Hudson, A. (2007) 'Evaluation of Citizens’ Voice and Accountability: evaluation framework', London: Overseas Development Institute.

Fox, J. (2007) 'The Uncertain Relationship between Transparency and Accountability', Development in Practice 17 (4): 663-71.

Fung, A. (2001) 'Accountable autonomy: toward empowered deliberation in Chicago schools and policing', Politics and Society 29 (1): 73–103.

Gauthier, B. (2006) 'PETS –QSDS in Sub-Saharan Africa: A Stocktaking Study', Washington DC: World Bank.

Gaventa, J. and Barrett, G. (2010) 'So What Difference Does it Make? Mapping the Outcomes of Citizen Engagement', IDS Working Paper 348. Brighton: Institute of Development Studies.

Gaventa, J. and McGee, R. (2010) Citizen Action and National Policy Reform: making change happen, London: Zed Books.

George, A. (2003) 'Using Accountability to Improve Reproductive Health Care', Reproductive Health Matters 11 (21): 161-170.

Global Integrity (2010) 'Access to Information in Health and Education Project—Terms of Reference' http://commons.wikispaces.com/Access+to+Information+in+Health+and+Education+Service+Delivery. (Accessed 30 June 2010).

Green, C. (2008) 'Strengthening Voice and Accountability in the Health Sector', PATHS Technical Brief, Abuja: Partnership for Transforming Health Systems. Holland, J. and Thirkell, A. (2009) 'Measuring Change and Results in Voice and Accountability Work', *DFID Working Paper* 34, London: Department for International Development.

Hossain, N. (2009) 'Rude Accountability in the Unreformed State: informal pressures on frontline bureaucrats in Bangladesh', *IDS Working Paper* 319, Brighton: Institute of Development Studies.

Houtzager, P., Joshi, A. and Gurza Lavalle, A. (eds) (2008) 'State Reforms and Social Accountability' *IDS Bulletin* 38 (6).

Hubbard, P. (2007) 'Putting the Power of Transparency in Context: information’s Role in Reducing Corruption in Uganda’s Education Sector', *Centre for Global Development Working Paper* 136, Washington DC: CGD.

International Budget Partnership (2008) 'Our Money, Our Responsibility: a Citizen’s guide to Monitoring Government Expenditures', Washington DC: IBP.

Joshi, A. (2008) ‘Producing Social Accountability? The Impact of Service Delivery Reforms,’ *IDS Bulletin* 38 (6): 10-17.

Khemani, S. (2008) ‘Does Community Monitoring Improve Public Services? Diverging Evidence from Uganda and India’, *Research Brief*, Human Development and Public Services Research, Washington DC: World Bank.

Lassibille, G., Tan, J., Jesse, C. and Nguyen, T. (2010) 'Managing for Results in Primary Education in Madagascar: evaluating the impact of selected workflow interventions', *World Bank Economic Review* 24 (2): 303-329.

Mainwaring, S. (2003) 'Introduction: Democratic Accountability in Latin America', in Mainwaring, S. and Welna, C. (eds) *Democratic Accountability in Latin America*, Oxford: Oxford University Press.

Malena, C., Forster, R. and Singh, J. (2004) 'Social Accountability: An Introduction to the Concept and emerging Practice', *Social Development Paper* 76. Washington DC: World Bank.

Manor, J. (2004) 'User Committees: A Potentially Damaging Second Wave of Decentralization?' *European Journal of Development Research* 16 (1): 192-213.

McNamara, B. (2006) 'Provider-specific report cards: a tool for health sector accountability in developing countries' *Health Policy and Planning* 21 (2): 101-109.

McNeil, M. and Mumvuma, T. (2006) 'Demanding Good Governance: A Stocktaking of Social Accountability Initiatives by Civil Society in Anglophone Africa', *World Bank Institute Working Paper* 261, Washington DC: World Bank. Misra, V. (2007) ‘Pilot Study 1, Andhra Pradesh, India: Improving Health Services through Community Scorecards’, Learning Notes, Social Accountability Series, Washington DC: World Bank.

Olken, B. (2007) 'Monitoring Corruption: Evidence from a Field Experiment in Indonesia', *Journal of Political Economy* 115 (2): 200-249.

O’Neil, T., Foresti, M. and Hudson, A. (2007) 'Evaluation of Citizens’ Voice and Accountability: Review of the Literature and Donor Approaches', London: Department for International Development.

Pande, S. (2008) 'The Right to Information and Societal Accountability: the Case of the Delhi PDS Campaign', *IDS Bulletin* 38 (6): 47-55.

Pandey, P., Goyal, S. and Sundararaman, V. (2009) 'Community Participation in Public Schools: Impact of Information Campaigns in three Indian states', *Education Economics* 17 (3): 355-375.

Peruzzotti, E. and Smulovitz, C. (2006) 'Social Accountability: An Introduction', in Peruzzotti, E. and Smulovitz, C. (eds) *Enforcing the Rule of Law: Social Accountability in the New Latin American Democracies*, Pittsburgh: University of Pittsburgh Press.

Ravindra, A. (2004) 'An Assessment of the Impact of Bangalore Citizen Report Cards on the Performance of Public Agencies', *Evaluation Capacity Development Working Paper* 12, Operations Evaluation Department, Washington DC: World Bank.

Reinikka, R. and Svensson, J. (2005) 'Fighting Corruption to Improve Schooling: Evidence from a Newspaper Campaign in Uganda' *Journal of the European Economic Association* 3 (2-3): 259-267.

Renzio, P., Azeem, V. and Ramkumar, V. (2006) ‘Budget Monitoring as an Advocacy tool: Uganda Debt Network’, Case study Prepared for the research project Lessons from Civil Society Budget Analysis and Advocacy Initiatives, London School of Economics Non-Governmental Public Action Programme.

Rocha Menocal, A. and Sharma, B. (2008) 'Joint Evaluation of Citizen’s Voice and Accountability Report', London: Department for International Development.

Schedler, A. (1999) 'Conceptualizing Accountability', in Schedler, A., Diamond, L. and Plattner, M. (eds) *The Self Restraining State: Power and Accountability in New Democracies*, Boulder CO: Lynne Rienner.

Singh, R. and Vutukuru, V. (2010) ‘Enhancing Accountability in Public Service Delivery through Social Audits: A Case Study of Andhra Pradesh’, Accountability Initiative, New Delhi: Centre for Policy Research. Sirker, K. and Cosic, S. (2007) ‘Empowering the Marginalized: Case Studies of Social Accountability Initiatives in Asia’, World Bank Institute Research Paper 266, Washington DC: World Bank.

Uganda DISH (2003) http://www.ugandadish.org (Accessed July 2004).

United Nations (2007) ‘Auditing for Social Change: A Strategy for Citizen Engagement in the Public Sector Accountability’, Department of Economic and Social Affairs, Division for Public Administration and Development Management, New York: United Nations.

Unsworth, S. (2010) An Upside Down View of Governance, Centre for the Future State, Brighton: Institute of Development Studies.

World Bank (2004) World Development Report 2004: Making Services Work for Poor People, Washington DC: World Bank.