HomeBeneficiary Feedback in the Department’s Evaluations

# 									Beneficiary Feedback in the Department’s Evaluations							

Office of Foreign Assistance

July 25, 2019

There has been a growing interest in the development community in the subject of “beneficiary feedback.” Many bilateral and multilateral agencies such as DFID, the World Bank, UNHCR, SIDA and International Development Agency of Norway have embraced the concept and have been grappling to introduce it. UNHCR has also emphasized a beneficiary feedback approach and has conducted evaluations that have systematically sought the views and opinions of refugees and internally displaced persons. International NGOs such as World Vision and InterAction have been pushing for beneficiary feedback. However, the published material on the subject remains extremely limited. Evaluations that have used feedback mechanisms are not publicly available and remain buried in the bookshelves of the sponsoring organizations. Nor have the evaluators who have used them codified their experiences and explained the problems and challenges that they have faced. As a result, it will take time before the issues surrounding the beneficiary feedback approach are adequately conceptualized and tested, with practical guidance provided to practitioners.

There are two reasons why the Department should start a conversation on this subject.  First, it should not overlook the ethical principle that the beneficiaries in whose name projects and programs are designed and implemented must have input into evaluations. They should be more than a source of information to evaluators. Second, and more importantly, feedback by beneficiaries can improve the quality of findings, conclusions and recommendations. Beneficiaries can often correct the flaws in findings and recommendations and provide a different perspective that can enhance the quality and credibility of evaluations. The purpose of this brief article is to start a conversation on the subject.

## Conceptualizing Beneficiary Feedback in Evaluation

Although the expression is used in development literature, there is no agreed upon definition of beneficiary feedback. Some equate it with sharing evaluation findings and recommendations with beneficiaries to get their responses while others equate it with participatory evaluations in which beneficiaries are full partners in the entire evaluation enterprise. As conceptualized here, it is a systematic approach to give beneficiaries a voice in the evaluation process. The concept of beneficiary feedback can be explained as follows:

***Beneficiary Feedback in Evaluations***

We can visualize a continuum from no inputs to full participation of beneficiaries. On the one extreme lie evaluations in which beneficiaries are only providers of information and have no say in the evaluation process. On the other extreme are participatory evaluations in which beneficiaries are equal partners. Beneficiary participation falls between these two extremes. The difference between participatory evaluation and a beneficiary feedback approach is that while in the former beneficiaries are represented in evaluation teams and are therefore engaged in planning, data collection and analysis, in the case of beneficiary feedback there is communication between the two, but the ultimate responsibility for the evaluation rests with the evaluators. Beneficiaries only provide feedback.
> 
 A beneficiary approach to development evaluation involves a one-way or two-way flow of information between beneficiaries and evaluators for the purpose of improving the evaluation process, findings and use. It is a structured and systematic approach that cuts across all stages of evaluation- from design to dissemination.1 
**Leslie Groves**

## Who Are Beneficiaries?

Despite its wide use, the term beneficiary has been differently used in development discourse. In fact, those who see poverty alleviation as a “right” question the very use of the term and regard it as patronizing. Some experts would prefer to use the term only for direct beneficiaries while others like to include both direct and indirect beneficiaries. Moreover, the question has been raised as to whether the term should include groups that were not originally targeted in projects and programs but also benefited from interventions. It will take time before a consensus will emerge about the precise meaning of the term. For the purpose of this article, beneficiaries refer to (a) people for whose benefit a project or program is designed, such as refugees, internally displaced persons, patients suffering from HIV/AIDS, vulnerable women, farmers or others (called individual beneficiaries);  (b) organizations, governmental or non-governmental, that receive assistance to enhance their organizational effectiveness, performance and delivery systems (called institutional beneficiaries).  Examples include media organizations to which assistance is provided to strengthen and consolidate independent media, human rights NGOs that mobilize public opinion for basic civil and political rights or the organizations that are engaged in promoting policies and programs for climate change. In all these examples, while people benefit in the long run, there are no specifically targeted individuals for whose benefit the project or program is launched. Most experts do not treat the second category as beneficiaries in their analysis. This omission is not justifiable in the case of USG foreign assistance programs, however, as the bulk of them focus on institution building.

## Beneficiary Feedback at Different Stages of Evaluation

Ideally, all projects and programs should have monitoring systems that gather information about beneficiaries and their perceptions, views and responses to the interventions. Such information can be gathered for both individual and institutional beneficiaries through a wide variety of data collection methods. Even when the information is not comprehensive, it will undoubtedly help managers in improving the performance and impacts of their projects and programs.

Beneficiary feedback can take place at four stages of evaluation, namely planning and designing; data collection; formulation of findings, conclusions and recommendations; and finally dissemination of evaluation reports. We discuss the feedback process for two categories of beneficiaries.

Exponents of the beneficiary approach suggest that at the planning stage, evaluation managers can seek the views and suggestions of *individual beneficiaries* about the nature and focus of the evaluation and the questions it seeks to answer. They can do it through interviews, group meetings and focus group discussions with beneficiaries. The problem is that this is not a realistic proposition for the Department’s evaluations. As projects and programs are located in foreign countries, evaluation managers do not have the time and resources to visit abroad and discuss the planned evaluations with a select sample of *individual beneficiaries*. Even the feedback on draft SOWs through written or electronic media is not practical in most cases because it is extremely difficult to reach many beneficiaries. Moreover, it is doubtful beneficiaries will have the time, technical expertise and incentive to provide inputs at the initial stage. The situation is different when there are well organized beneficiary organizations managers can rely on to communicate with beneficiaries at the planning stage.

In the case of institution building projects and programs, evaluation managers can share the evaluation questions and approach with the concerned organizations and seek their feedback. For example, if a project provided technical and financial assistance to a select number of democracy NGOs, a brief document containing the evaluation questions and approach may be shared with them and their comments and suggestions may be considered. In such cases, feedback is possible because the number of beneficiaries is limited and they can be approached through electronic communication. Moreover, given the high likelihood that these organizations focus on evaluating their own performance, they are likely to possess sufficient skills and expertise to make useful and relevant comments and suggestions.

Beneficiary feedback is more plausible at the data collection stage because most evaluation teams meet or should meet a selection of beneficiaries at this point.  They usually conduct key informant interviews, surveys, group discussions or focus groups to obtain necessary data, information and opinions. Unfortunately, in most cases it remains a one-way communication; beneficiaries do not have opportunities to ask questions, challenge the focus of inquiry or review the data and information gathered. The beneficiary feedback approach requires that the beneficiaries be able to do so. There should be two-way communication. Evaluators not only seek information and opinions from the beneficiaries but also provide them with opportunities to ask questions and make suggestions.
> 
Between September and October 2014, Cordaid with the assistance of Turkana Natural Resource Governance Hub carried out a qualitative baseline assessment  in Turkana County to gain a better understanding of the perceived community effects and benefits of oil exploration. After the preliminary report was drafted, they presented their preliminary findings in three validation workshops. Workshops that were attended by beneficiaries gave opportunities to these stakeholders to critically examine the preliminary findings and make suitable suggestions. Participants also provided recommendations for improvement of community engagement. In total, 177 people participated in the workshops.
**Validation of Findings by Stakeholders**

How evaluators can obtain beneficiary feedback at this stage is bound to differ from intervention to intervention, by resources, by time available to evaluation teams and by teams’ technical expertise. In the case of *institutional beneficiaries*, evaluators can conduct qualitative interviews, conduct group interviews or even undertake an informal survey to discuss data collection. For example, a team examining the performance of a project providing training and technical assistance to women entrepreneurs in war-torn societies may call a meeting of the representatives of women getting training or technical assistance and discuss their data collection plans. Similarly, evaluators may call a meeting of NGOs receiving financial and technical assistance to seek advice about appropriate data collection methods. The team should be earnest in its approach and willing to make suitable changes if necessary. As far as *individual beneficiaries* are concerned, their views can be sought during interviews and group discussions. Questionnaires can include approaches that directly or indirectly provide beneficiaries’ views about the evaluation approach.

The third and most critical stage when beneficiary feedback can take place is when evaluators analyze collected data and information and formulate findings, conclusions and recommendations. At this stage, beneficiary feedback from *institutional beneficiaries* can be secured by holding workshops and focus group discussions during which tentative findings are presented and discussed. In an evaluation of a USAID media assistance program in Russia, evaluators used workshops as well as group meeting models for a program that had been providing training and technical assistance to nascent media outlets. They convened a meeting of media outlets that received or were receiving USAID assistance and explained that the main purpose of the evaluation was to learn lessons that can be used for improving USAID’s media assistance programs in the countries. The participants, who were highly educated, asked many pertinent questions and made useful suggestions about the sites to be visited and the economic and political contexts that should be taken into consideration.  Evaluators also held a workshop at the end of fieldwork and shared findings, conclusions and lessons during which several participants provided useful explanations and suggestions.

Getting feedback from a large number of *individual beneficiaries* about findings, conclusions and recommendations is much more difficult. Often beneficiaries are not forthcoming. It requires time, patience and communication skills. One possible approach is to hold meetings with a small number of beneficiaries to get feedback, but this needs careful planning.  For instance, in an evaluation of an area development project in Malawi, the evaluator met with the farmers about their use of improved variety of corn and the assistance they were receiving from extension workers. Focusing on five key findings and presenting them with the help of a translator, the evaluator encountered language and cultural barriers as well as a reluctance to criticize aspects the program in public. The evaluator might have gleaned more information by relying on a local person to hold the meeting. Evaluation teams may find it easier to seek information from individual beneficiary organizations, however. Evaluators will need to recognize that beneficiaries, as with all stakeholders, are an interested party whose views should be triangulated and validated. Socio-cultural context should be considered while seeking feedback from individual beneficiaries.

The last stage is the dissemination of evaluation reports. Practically, most funding agencies or their implementing partners do not share the evaluation reports with beneficiaries. The exponents of the beneficiary feedback approach view this current practice as unjustifiable. Since the beneficiaries have provided inputs into the evaluation, it is still more important they should be informed of the evaluation findings and even involved in the dissemination process. Therefore, evaluators should not only provide the beneficiaries with the evaluation reports but also seek their suggestions as to with whom else the report may be shared.

Table 1 gives examples of beneficiary feedback at different stages of the Department’s evaluations.