Reporting and Using Evaluation Results

Presented on 6/18/15

Facilitator notes: This presentation describes how to report on and use results from a program evaluation, and is intended for grantees approaching the final stages of the program evaluation cycle.

1

Learning objectives

By the end of this presentation, you will be able to:

Understand the importance of communicating and disseminating evaluation results to stakeholders. 

Write an evaluation report and become familiar with other key reporting tools. 

Determine  meaningful programmatic changes based on evaluation findings, and learn how to implement them

Facilitator notes: For this presentation, we have identified a few learning objectives.  

By the end of this presentation, you will be able to: 

Understand the importance of communicating and disseminating evaluation results to stakeholders. 

Write an evaluation report and become familiar with other key reporting tools. And,

Determine meaningful programmatic changes based on evaluation findings, and learn how to implement them.

2

What you’ve done and what’s left to do

Evaluation is:

Part of the culture of a learning organization

An investment in program improvement

A tool for building a program’s evidence base

Facilitator notes: 

Before discussing how to report and use the results generated by an evaluation, it is helpful to remember where you are situated in the evaluation process and your options moving forward.  

Remember that evaluations are conducted to gain valuable information about how your program is working.  When you first began the evaluation planning process, you spent time refining your theory of change and logic model, thinking through exactly how your program operates and how it proposes to address a problem. Perhaps you then engaged stakeholders in discussion about what you wanted to learn about that theory of change, and narrowed down a set of processes or outcomes that you wanted to learn more about. You spent time creating research questions that clearly identified those information needs, and then matched an evaluation design to those questions to generate valuable data. That data was then analyzed to produce your evaluation’s results. 

At this point in the evaluation process, you could simply write up a summary report of the evaluation and its results, file it with the relevant funders and stakeholders, and proceed with business as usual. But you will have spent a good deal of resources- including money, staff and volunteer time, and evaluation expertise- just to engage in a compliance exercise. You also won’t have spread your learning to benefit others operating similar programs, and you haven’t utilized a valuable opportunity to demonstrate accountability to stakeholders. In addition, you likely haven’t learned too much about your program, because you haven’t completed the final step in the evaluation process, which is making meaning and sense of those evaluation results. This critical step will allow you to make your program stronger. 

Completing this final evaluation step, where meaning is made of data and information, is a hallmark of a learning organization. A learning organization is an organization that creates, acquires, and transfers knowledge, and modifies its behavior to reflect new insights. These organizations remain relevant and effective because they are able to continuously improve their operations, using data to identify areas for improvement. Learning about and improving your program requires data, and those data are readily available through program evaluation. Evaluating your program, and then using the results to making meaningful programmatic improvements, helps your organization to serve more people and to serve them better. It also allows you to remain competitive in a scarce funding environment, justify increases in scale, scope, and reach, and demonstrates accountability to your stakeholders. Continuous assessment and regular evaluation work will enable you to become a learning organization, and will potentially produce a sizable evidence base that demonstrates your program’s efficacy.

The bottom line, then, is that evaluation is an important tool to improve your program’s ability to address a particular problem most effectively. Completing the evaluation cycle by reporting and using the results from your evaluation to make meaningful improvements sets your program apart as a forward thinking, impact oriented organization, and brings you closer to achieving your mission.

Before going further into the material, it’s important to note that this webinar will not be covering analysis in depth. While it is an important step in the evaluation process that you must complete to transform your raw data into evaluation results, it is too detailed a topic to be sufficiently covered within this presentation. This course picks up after you have analyzed your data. 

3

Planning

Implementation

Analysis and Reporting

Action and Improvement

Purpose of reporting

Reporting:

Completes the evaluation process by documenting work done and lessons learned

Provides an opportunity for reflection and learning, generating ideas for program improvement

Monitors and tracks progress in strengthening program

Demonstrates accountability to stakeholders

Communicates accomplishments and what the program does 

Facilitator notes:

Reporting is the final stage of the evaluation process. It can be thought of as having two broad components, the first of which is simply recording and communicating out what you did during the course of your evaluation. The second component involves making meaning from your results and generating next steps for program improvement. 

Both broad components of reporting serve several complementary purposes: 

First, reporting documents the steps you took as a part of the evaluation process, including the formulation of research questions, implementation of the evaluation design, and analysis of data to produce results.

Second, it provides an opportunity for reflection on results, giving you a chance to bring program stakeholders together to identify areas of success or those that need improvement or changes.  

Third, it enables monitoring and tracking progress your program makes towards improving its intervention. This is important for demonstrating accountability to beneficiaries, funders, and community partners. 

Finally, it enables your program to demonstrate and communicate the effect it is having on beneficiaries to a variety of audiences, including other programs and practitioners, potential funders, policymakers, researchers, and members of the community.  

4

Basic steps of reporting

Facilitator notes:

Reporting has four key steps. These are often completed sequentially:

First, assess information needs and reporting requirements; 

Then, develop reporting products, such as an evaluation report, briefs, or memos, based on those needs and requirements; 

Next, distribute those products to the various audiences you identified in the first step; 

And finally, support the use of the products, paying particular attention to ensuring that those stakeholders who need to take action based on the evaluation’s results do so. 

5

Assess information needs and reporting requirements

Develop reporting products

Disseminate your products

Ensure and support use of findings by stakeholders

Step one: assess needs and requirements

Facilitator Notes:

With those steps in mind, let’s dig into Step One, assessing information needs and reporting requirements. 

6

Assess information needs and reporting requirements

Develop reporting products

Disseminate your products

Ensure and support use of findings by stakeholders

Assess needs and requirements

How will reporting meet the needs of intended users?

Are there reporting requirements stipulated by funders?

Facilitator notes:

When you begin assessing needs and requirements, first, consider how reporting will meet the information needs of your program’s stakeholders. Think back to the planning phase of the evaluation process. You likely engaged in an intentional process to identify areas of your program that you or specific stakeholder groups wanted to learn more about, and selected research questions to reflect that. If you engaged stakeholders systematically, such as through an evaluation working group or committee meeting, they will have shaped these questions with their needs. The input provided in the planning phase and your research questions determine the information needs to be addressed through the evaluation. Different individuals or groups may need slightly different information reported back to them. 

Second, consider the reporting requirements stipulated by funders. Funders often require reports or similar products in order to ensure accountability, and to encourage grantees to engage in a thoughtful reflection process on what is working about their program and what isn’t. Ideally, you will already be engaged in this reflection process and can build funder’s requirements into that broader conversation. Nevertheless, if your funder has clearly stated that a reporting product should be provided, you will obviously want to produce that product and ensure that it includes all required information or content areas. 

7

Dissemination plans

Develop a dissemination plan

Organizational tool that visually lays out information needs, products, timelines, priorities, and roles and responsibilities

Begin development early on in the evaluation process!

Can use to build products into an external evaluator’s contract

Facilitator Notes:

An easy way to organize all of these needs is by creating a dissemination plan. A dissemination plan is an organizational tool that visually lays out who needs what information, and ties those needs to desired reporting formats, timeframe, priority of each need, and roles and responsibilities for producing reporting products.

You can begin developing your dissemination plan when you are putting together your evaluation plan, since you’ll already be in the process of figuring out what you and other stakeholders want to know about your program. It is really never too early to create your dissemination plan, and if you get a jump start on its creation, you can proactively build products into an external evaluator’s contract. 

8

Create a dissemination plan

Elements of a dissemination plan:

Who (stakeholders)

Needs what (information)

Is it required?

By what time (timeframe)

In what order of importance (priority)

In what format (product/mode)

Who is responsible (creator/distributor)

Think proactively about what information stakeholders could use! 

Facilitator notes:

A dissemination plan need be no more complicated than a spreadsheet or word document, but it does have several key components you shouldn’t leave out. 

First, list out all the stakeholders or groups that need to see information or results from your evaluation. Then, match them with their specific information need, clearly marking which pairings are requirements, and recording any deadlines by which time the product needs to be delivered to the stakeholder. Next, you can assign a ranking or mark the order of importance for that product to be produced and distributed. You can then list out appropriate reporting products for each stakeholder based on the information need. Finally, assign a staff member or evaluation partner to have responsibility for creating and/or distributing each product to the right stakeholder. Be as specific as you need to be about the information in each category; for example, does a funder require a final evaluation report? Or does a community partner need their reporting product printed and shipped to them rather than being delivered electronically? You might also choose to record the amount and type of resources being dedicated to each reporting product, as well as follow up actions that should be taken to ensure products are used by stakeholders. 

Just a cautionary note: A program must think creatively about what information might be helpful to different groups, even if they haven’t specifically requested it as a “Need.” In many cases it is incumbent upon the program to push information out, proactively, whether or not a stakeholder actually thinks they need it.

9

Facilitated example #1: dissemination plan

*See pre-work handout or annotated example impact evaluation plan on the Evaluation Resources page for context for the example. 

Facilitator notes: 

Let’s clarify all that with an example. Since the text on our example plan is so small here, if you’d like to see this in bigger text, you can go to tab “E1.1 Partial ex.” in the excel spreadsheet that was sent with your pre-work, and you can follow along from there. You can also follow along with the program description by turning to pg. 2 of your pre-work word document. 

In your pre-work, you read about a fictional AmeriCorps program that engages members in delivering a financial education curriculum to eligible clients. The program has 10 sites in the greater Cincinnati region that places members in credit unions and local financial institutions to provide financial counseling to low‐income individuals, retirees, and young people. Members also assist with financial seminars and informational fairs, and recruit experienced financial professionals to serve as volunteers in a credit counseling program. Its major funders are AmeriCorps and a small foundation, and it partners with local banks and a local social services agency. This program has the AmeriCorps evaluation requirements for grantees receiving over $500,000 in grant funds per year, which means they need to complete an externally conducted impact evaluation. The evaluation must cover at least 1 program year, and needs to have findings ready to report upon recompetition. The evaluation will only focus on the program’s financial counseling component, and will explore the following research questions: First, Do low‐income clients exit the program with increased knowledge of personal finance concepts relevant to their needs? Do they know how to use those concepts to address their financial challenges? And second, are low‐income clients that participate in member‐led financial counseling through the program able to better manage their personal finances than low‐income individuals who did not participate?

Clearly this program has quite a few stakeholders that will be interested in the results of their impact evaluation. Let’s walk through an example dissemination plan for this evaluation, remembering the questions from the last slide: Who (stakeholders), Needs what (information), In what format (product/mode), By what time (timeframe), and In what order of importance (priority)? 

We’ll fill in one row for our example, and take AmeriCorps as our stakeholder of interest to complete the example. Again, if you’d like to see this in bigger text, you can go to tab “E1.1 Partial ex.” in the excel spreadsheet that was sent with your pre-work. 

The first column identifies who needs the information generated by the evaluation, in other words, who are the target audiences or stakeholders. 

For our example here, the target audience is going to be AmeriCorps/CNCS. Remember that when complete, the program will want their dissemination plan to list each key stakeholder that needs information reported back.

The next column identifies the specific information needs of the stakeholder, and records what they need to know. 

For AmeriCorps, remember the evaluation requirements we mentioned in the program description: as a grantee receiving more than $500,000 per year in grant funds, the program must conduct an independent impact evaluation using a valid comparison group. In this case, CNCS wants to know about the effectiveness of the service intervention being implemented, so the information need here could be classified as “effectiveness of the intervention” or “evidence for intervention”.   

The next column calls out whether this information is part of a requirement, which is important in helping prioritize the order in which the program produces their reporting products. 

We’ve noted here that the reporting for AmeriCorps is part of a requirement. 

The fourth column identifies the timeline for reporting back to the stakeholder, or the deadline by which the reporting product is due.

CNCS’ evaluation requirements for all grantees stipulate that the final evaluation report must be submitted at the end of the 3 year grant cycle at the time of recompetition for funds. It’s important for the program to note this so they can ensure there is sufficient time to complete and edit the reporting product before submission. 

Related to deadlines, the next column notes the priority level of the product. Here, the program has marked it a high priority since CNCS is a major funder and the product is tied to a grant requirement. 

The next two columns identify the type of product and the medium they will use to disseminate findings.

Since CNCS’ information needs concern “evidence for the intervention”, this leaves a few different media open for reporting. But considering that CNCS stipulates that grantees must submit a final evaluation report when they recompete, a written, final evaluation report is the best product for to develop for this audience. 

Note that the program has also included columns in this plan to record who is responsible for preparing the product (an external evaluator), the type and amount of resources available for the product ($5,000). There is also a column noting who is responsible for distributing the product (staff member who writes grants). If you are a larger organization, you might have a communications staff member that is responsible for all of this, or who will manage coordination. If you are a smaller organization or have a smaller number of staff, you may have to split up reporting tasks amongst staff responsible for regular programming. Any number or combination of staff can be responsible for completing reporting tasks, and keep in mind that it may look different for any given organization depending on size and resource availability. 

The program has also included a space to record follow-up activities that should take place after the products are distributed. Follow up to ensure evaluation results are reviewed and used by others maximizes your investment, helps others make informed decisions, and demonstrates accountability and transparency, so consider investing some time connecting back with your stakeholders. In this example, you can see that the program plans to discuss the results of the evaluation with their program officer. They also are going to work with CNCS’ office of research and evaluation to complete a summary brief of their evaluation, which will be posted on CNCS’ newly created Evidence Exchange. 

On your handout, we’ve included a complete example dissemination plan for the program so you can see what this might look like fully fleshed out. This is on tab “E1.2 Complete ex.”. A template is also available for your use on tab “E1.3 Blank Template”.

[Pause for questions]

10

Step two: develop reporting products

Facilitator Notes:

Now that you understand the why and how of assessing information needs and requirements, and are familiar with a way to organize the process, let’s talk about Step Two in the reporting process, the various reporting products you can develop. 

11

Assess information needs and reporting requirements

Develop reporting products

Disseminate your products

Ensure and support use of findings by stakeholders

Types of reporting products

The basic types of reporting products are:

Written products

Presentations

Visuals and graphics

Supplementary social media and creative products (e.g. photo-reporting)  

Facilitator notes: 

There are four basic types of reporting products you can develop.

First are written products, which many of you are already developing based on your evaluation work. At a minimum, every evaluation you conduct should have an evaluation report. This should be written by the individual or team that conducts the evaluation, so if you’re using an external evaluator, that is something that should be a task and deliverable listed in the contract. This can serve as a base for creating additional “derivative” products. 

You can also create presentations to report on your evaluation, or you might choose to create visuals or graphics. Supplementary social media, and other creative works are other types of products you can create once you’ve completed your final evaluation report. 

Some reporting products lend themselves to communicating certain types of information better than others. For example, a memo may describe a programmatic change based on your evaluation results in a way most accessible to program staff, due to the nature of a memo’s length and brevity. An executive summary, on the other hand, might better address a potential funder’s information needs by giving a comprehensive yet succinct description of the evaluation’s process and results.

In this webinar we are focusing more on written products, particularly the evaluation report. The rest are important but you can investigate them more on your own. At the end of the presentation we’ve suggested some sources that might be helpful to consult. 

12

Written products

Written products are composed of textual descriptions and recommendations about the evaluation and its findings

At minimum, you should create a comprehensive evaluation report 

Derivative products can be created from this report

Facilitator notes: 

Written evaluation products are composed of narrative and textual descriptions, explanations, and recommendations regarding the evaluation and its findings. At a minimum, you should plan to write an evaluation report. The purpose of a written evaluation report is to be an objective documentation and presentation of the evaluation’s methods and findings. Derivative products can then be tailored for other communications purposes. 

13

Written products (cont.)

Other written products may be derived from the evaluation report or interim reports. These include: 

Executive summaries

Memos and short communication pieces

Briefs: technical, practitioner focused, policy, etc. 

Academic papers or conference papers, white papers, working papers

Other: interim reports, press releases and media documents, newsletters, blog posts

Facilitator Notes:

Depending on the particular stakeholder, their information needs, and available resources as outlined in your dissemination plan, you could create a number of products based on your evaluation report. For example, these could include executive summaries, which are very short summaries of the evaluation’s purpose, methods, and findings. It could also include memos, or other written pieces intended to quickly communicate specific findings or actions to take based on the evaluation’s results. Briefs focus in-depth on a particular topic covered in your evaluation, and can, for example, be technically focused, practitioner focused, or policy focused. Academic papers, conference papers, white papers, and working papers can connect your evaluation and its findings to a relevant scholarly community. Finally, you can create other supplementary communications products like press releases and media documents, newsletters, or blog posts based on the contents of the evaluation report. Note that you can also develop some of these derivative products from interim reports, but be sure to gauge whether such preliminary results will meet the information needs of the consumers receiving the specific product. 

Be wary that short communications like memos and email blasts can lead to misinterpretation of findings due to lack of contextualizing information. Pay careful attention to wording to avoid misleading or misstating results. It’s always a good idea to offer stakeholders easy access to the full, final evaluation report so they can investigate details of the evaluation’s methodology and implementation for themselves. However, you should not necessarily expect the final report to be read in its entirety, so any information presented in other formats should be accurate and complete on its own.  

14

Elements of an evaluation report

A complete evaluation report contains:

Introduction to the project

Description of evaluation design and methods

Statement of findings and results

Facilitator notes: 

A complete evaluation report contains the following three elements: an introduction to the project, a description of the evaluation design and methods, and a statement of the findings and results. You can use some of the information from your evaluation plan to populate the evaluation report. For example, the program description, evaluation design, and methods can be pulled from the plan and enhanced with a level of detail appropriate for the report. 

15

The evaluation report: introduction

Introduction to the evaluation project: 

Describe the evaluation’s purpose and research questions 

Give background on the program and its theory of change, with extra detail about the component being evaluated

Identify the target population under study

Review past relevant research and evaluation

Outline the structure of the following report sections

Facilitator notes: 

The report should begin with a comprehensive but succinct introduction to the evaluation project. The introduction frames the purpose and goals of the evaluation, and explains why the project is being undertaken. It also provides the necessary contextual information to understand subsequent sections of the report. The introduction should begin with a description of the evaluation’s purpose and research questions, as these set the stage for the entire project and clarify the choice of evaluation approach and design. Next, you should give background on the program and its theory of change and include detailed information on the program component being evaluated. Do not assume that the reader will be familiar with your program or the component under study. At this point, it may also be helpful to mention historical context or any environmental influences that affect the way the program works or that might have bearing on the evaluation. The next step is to identify and describe the target population being studied. You may also choose to identify stakeholders and audiences for the evaluation’s results. This should be followed by a brief review of relevant past research or evaluation work done by your program. You can also include past research and evaluation done by other researchers or similar programs if it is relevant to the program component being evaluated. Finally, you’ll conclude the introduction by outlining the structure of the remainder of the report. 

16

The evaluation report: design and methods

Description of the evaluation design and methods

Describe the evaluation approach being used (i.e. process, impact), and explain rationale for choice

Describe the evaluation design (e.g. pre/post-test with comparison group, case study)

List and describe your data sources (e.g. administrative records) 

Describe the data collection methods used

Describe study limitations and strengths

Facilitator notes: 

The next section of the evaluation report describes the evaluation design and methods. You can begin by restating the research questions to remind the reader of the purpose of the evaluation, and then proceed to describe the evaluation approach being used (i.e. process or impact). You should also explain why that particular design is being used. This is an appropriate place to list funder requirements or other important constraints that shaped the choice of evaluation approach and design. Remember that your evaluation approach and design flow from your research questions, and it is a good idea to point out in your report the strengths of the particular evaluation design in addressing a particular research question. 

Next, you will need to describe the data collection methods used, including details about how data collection was implemented, the timing of the activities, and any noteworthy challenges encountered during the process that could impact the data gathered. You should clearly list and describe your data sources (e.g. administrative records) and, if studying multiple populations, match data collection activities to their relevant populations. Finally, you should state study limitations, especially those that arise from the evaluation design and its implementation. You should also mention the strengths of your study. 

17

The evaluation report: findings

State findings and results

Describe the results of all analyses conducted

Use technical appendices for additional analyses, results, and supplemental data 

Label all charts, tables, and graphs 

Facilitator notes:

The final section of the evaluation report should state the findings and results of the evaluation. This section should begin by presenting the results of analyses conducted on the data you collected. 

A technical appendix can help highlight the most important results in the body of the report by moving additional, important technical information to the back of the report. This will allow you to present a few key graphs, tables, or charts in the body of the report, and preserve additional information for the reader who wishes to seek it out. It is very important that you present all findings, positive, null, and negative, in the report. It is not appropriate to only present the most favorable results.

Be sure to label all charts, tables, and graphs that appear in this section. Check that all axes on graphs and charts are labeled, that tables have appropriate headers, and that all items have titles and captions. 

18

Important considerations when writing a quality evaluation report

Ensure credibility

Discuss study limitations

Discuss generalizability 

Discuss negative or null findings

Hold interpretations, recommendations until end of report

Facilitator notes:

Whether commissioning an external evaluation or conducting one internally, credibility is a key component of a quality evaluation report. Damaging the objectivity of an evaluation means that findings lose integrity and believability. Because your program has an interest in seeing positive results, if efforts to obscure, change, or eliminate findings are detected in the report, you risk compromising the report’s validity. Loss of trust in reported findings can seriously damage relationships with stakeholders, and can jeopardize funding or opportunities to partner or scale up. 

Fortunately, there are steps you can take to ensure the credibility of your evaluation report. First, in your evaluation report, discuss the limitations of your study, including those related to methodology and implementation. This provides critical context as to why a particular result may have come about, and helps readers appropriately interpret findings. Methodological limitations are present in every evaluation design and relate to the specific shortcomings of a given design. For example, an outcomes study without a comparison group cannot demonstrate causality. Limitations that arise from how the evaluation was executed are implementation limitations, and can occur during data collection or analysis. Implementation limitations related to data quality and access to data sources are especially important to discuss. Examples could include how the randomization procedure was applied if a randomized control trial was conducted, or presence of low response rates on surveys.

Generalizability, or the ability for findings from the evaluation to be applied or extrapolated beyond the program being studied, is also important to report. Depending upon your evaluation design and implementation, findings may or may not be appropriate to generalize to the whole program, to another site, to the population served, or to other, similar populations. Again, this affects how results can be interpreted by others, and importantly, how findings may be used to improve the program or shape other programs. 

Don’t shy away from discussing negative or null findings. Null findings are findings of “no effect” or no change. They do not always indicate that your program is not working.  Sometimes null findings are simply a result of a weakness in the evaluation’s design; for example, the sample may have been too small. Again, negative or null findings from one program evaluation do not mean that your program isn’t working or should be shut down! They simply point to areas where positive change can or should be made. Remember that withholding or obscuring such findings invites suspicion and jeopardizes the objectivity of your evaluation. 

Finally, hold recommendations until the end of the report. This allows the reader to come to their own conclusions based on your objective presentation of the analysis and results. You should also maintain a neutral tone throughout the report to avoid the appearance of shaping or influencing conclusions. 

Overall, maintaining objectivity and transparency in reporting will ensure that readers can trust the report and rely on the findings. 

19

Other products

Presentations- poster, PowerPoint, and webinar

Community meetings, conference presentations, and panels

Visuals and graphics 

Social media and creative products

Public use datasets   

Facilitator notes: 

As we’ve mentioned before, once you’ve written a solid evaluation report, you can create additional products to communicate results and findings to different stakeholders. 

Presentations are effective ways to communicate key points about your evaluation and its findings, and may use posters, slide decks such as PowerPoint, or webinar technology, depending on the target audience and venue. They are also a great way to interact with stakeholders about the results and give an opportunity for discussion. It is important to remember that an effective presentation balances the use of speaking, visuals, and text, and does not overwhelm an audience with too much information at too fast a pace. Community meetings, conference presentations, community groups or meetings, or panels are other types of presentations to consider, depending on the types of stakeholders you are trying to reach, and the amount and type of information you wish to present.

Visuals and graphics can be stand alone products or can be part of written reports or presentations. They are especially effective at highlighting trends or patterns, especially when you have a large amount of data. Use visuals and graphics selectively, as too many can distract readers or viewers from your broader message. 

Social media, such as Twitter and Facebook are excellent tools to generate interest in your evaluation or communicate key findings. Be especially wary of wording when posting results through social media, as these modes do not accommodate the context or background information on the evaluation that is necessary to properly interpret findings. You can also consider using other creative products like photo reports, poetry, or other visual arts to communicate about your evaluation. Just be sure that the method you choose makes sense for your target audience and the findings you wish to present. 

Depending on the type of evaluation you conduct, you may wish to create a public use dataset, which has been stripped of any personally identifiable information. These datasets can contribute to further research by other programs or scholars on your intervention or a related topic. 

Finally, when producing any written or visual product, keep in mind that potential consumers with visual or auditory disabilities will want to be able to access findings. Be sure to format reporting products to work with visual or auditory accessibility software, especially if posting online.  

[Pause for questions]

20

Step three: disseminate your products

Facilitator Notes:

As you’ll recall, Step Three in the reporting process involves disseminating your reporting products. Once you’ve completed your evaluation report and any derivative products you’ve decided to create, you can distribute them to program stakeholders according to your dissemination plan. 

21

Assess information needs and reporting requirements

Develop reporting products

Disseminate your products

Ensure and support use of findings by stakeholders

Step four: ensure and support use of findings

Facilitator Notes:

Since we’ve already reviewed what this plan should cover and what an example might look like, let’s proceed to Step Four in the reporting process, ensuring and supporting use of findings by stakeholders.

22

Assess information needs and reporting requirements

Develop reporting products

Disseminate your products

Ensure and support use of findings by stakeholders

Using evaluation results for action and improvement

You’ve completed your evaluation report, but what do the results mean in practice? How do these results translate into actions?

Take your findings and make them actionable!

Identify program components that are working well

Identify program components that need to be improved

Develop and implement an action plan for improvement

Facilitator notes:

While reporting serves the purpose of documenting what was done during the evaluation, it also is the natural juncture in the evaluation cycle for figuring out what your evaluation results suggest in terms of meaningful program improvement. It is a convenient time because you will have all of the information you need to review and make decisions with in one central and easily accessible location. As we mentioned at the beginning of the presentation, purposefully examining results and creating an action plan to implement improvements based on those insights is the critical step that enables continuous improvement and that makes a program a learning organization. Using evaluation results for action and improvement means that a program is being accountable to its theory of change, is examining decision-making on policies and program components, and is drawing lessons for program improvement. 

To put this into practice, it’s helpful to divide the process into two stages: first, you will identify the program components that need to be improved or that are working well based on your evaluation results, and second, you will develop and implement a plan for enacting appropriate programmatic changes. 

Again, keep in mind that this is all taking place after the evaluation report is finalized. Even if you are conducting the evaluation internally, it is important not to reflect on the meaning of results or begin brainstorming changes within the final evaluation report. While it’s fine to suggest areas for future research or state new questions that have arisen during the completion of the evaluation, remember that making conjectures about findings within the evaluation report can diminish objectivity and credibility. In the next few slides, we’ll suggest some ways to go about documenting and tracking these ideas and associated improvements. 

23

Identify program components to be improved

Pair results to the relevant research question:

Did anything surprise you?

Any interesting or confusing patterns and trends?

Revisit logic model and theory of change

Conduct additional analyses of the data if necessary

Decide whether or not enough evidence exists to justify a program improvement

Suggest possible improvements, actions, or changes

Facilitator notes:

As we just mentioned, in stage one, you will identify the program components that need to be improved based on your evaluation results. So how do you find program components that need to be improved? A good first step is to look back to your research questions. At the beginning of the evaluation process, research questions were created to guide the evaluation and ensure that it produced the information you needed about your program. You can try pairing each research question to relevant results from your analysis. These results are the “answers” to your research questions. Consider what these answers are saying: given what you’ve recorded in your theory of change and logic model, are the results expected? Or is there something surprising or unusual you are finding that deviates from the logic model? Did the evaluation identify interesting patterns or trends? If the result you are seeing is not in line with what is expected from your logic model and theory of change, you should flag that component for possible improvement. This includes unexpected positive results, as they could indicate something that should be expanded, scaled, or further developed. We will present a detailed example in a few slides to make this process even clearer. 

Throughout this process, keep in mind how the evaluation was implemented. If there were challenges or flaws that mitigate the finding, the specific results suggesting change may be due to how the evaluation was implemented rather than how the program is actually performing. In any case, if multiple sources support or corroborate a flagged finding, you will have conclusive evidence that a change needs to be made. Regardless, do bear in mind that one high quality evaluation finding can be all that is needed to support or acknowledge a programmatic improvement. 

Once you’ve identified the program components that need to be improved you will need to articulate how, specifically, they will be improved. This could involve a change to the program design or implementation, to the services that are delivered, to the internal staffing, or to partnering, for example. This is an appropriate time to engage relevant stakeholders in discussing the results, and obtaining input into the improvement process. Options for convening stakeholders include conference calls, in-person meetings, or online presentations. Keep in mind that you may need to create specific derivative products from the final evaluation report to facilitate these discussions. Memos or PowerPoint presentations are often helpful for focusing conversation and documenting proceedings. When brainstorming possible improvements, actions, or changes, it is important to set discussion parameters to ensure that realistic, actionable solutions are generated. Any change that could reasonably be implemented should be actionable, specific, and able to be accomplished within a realistic timeframe. Relatedly, make sure that the scale and scope of the change suggested matches that of the evidence generated by the evaluation findings. Again, we’ll go over an example in a few slides to make this more clear. 

24

Developing and implementing an action plan for program improvement

Develop an action plan for implementing change

Changes may include: the program design; how a program is implemented; how services are delivered; the staff, etc.

  Specify the logistics

Who will carry out these improvements?

By when they will take place, and for how long?

What resources (i.e., money, staff) are needed to carry out these changes?

Who can be an advocate or partner in change?

Facilitator notes:

Once you have identified the parts of your program you want to improve, you can ensure follow through by developing an action plan. While this doesn’t have to be a formal document, it helps to have a record of the decisions made. This plan should include an “owner” or “owners” that will carry out the changes. You should set milestones and identify key dates for when certain actions must take place or be complete. It also helps to identify the types and amount of resources, such as money or staff time, you will need to carry out these activities. Once this plan is developed, you can invite stakeholders to review it. This will help hold your organization accountable, and demonstrates transparency to current and potential funders. Along with creating a feeling of stakeholder ownership in the program, it can help you recruit partners and additional resources needed to successfully implement your plan for change. 

25

Thinking about the future

Evaluations pay dividends long after they are completed. An evaluation will:

Build your program’s evidence base

Contribute to a long term research agenda

Facilitate continuous improvement and develop as a learning organization

Facilitator notes:

Developing and implementing your action plan for improvement demonstrates accountability and builds stakeholder trust in your organization. It also completes the evaluation process, bringing a close to the particular evaluation project. But, there is much more to be gained from your evaluation findings. An evaluation can pay dividends long into the future in a few key ways.  

First, every time your program conducts and completes an evaluation, information about program effectiveness compounds, providing a robust picture of how and why your program is or is not achieving its goals as detailed in your theory of change. This accumulation of programmatic knowledge gathered systematically from evaluation is known as an evidence base. This evidence base is valuable to your organization as it strives to improve services for constituents, to similar organizations as they improve their own program models, and to funders and policymakers as they attempt to direct scarce resources to the most effective organizations. 

Second, evaluations can be used strategically as part of a long term research agenda. A long term research agenda is simply a plan for a series of evaluations, completed over a longer time horizon, that build progressively towards an evaluation goal or broad research question about a program’s effectiveness. It requires moving beyond thinking of evaluations as one-time projects that exist in isolation. Instead, evaluations produce information that inform subsequent evaluations, or increase the organization’s capability to conduct a more rigorous evaluation. This change in thinking can be difficult, and often requires an intentional culture change within organizations. This effort is worthwhile, though, as a robust long term research agenda could save your organization a significant amount of time and resources, and can provide the right types of information at the right times in the life cycle of the program. 

Finally, in the long term, an evaluation also contributes critical information for continuous improvement. Reviewing the information provided by evaluation can help an organization set goals, measure progress towards meeting those goals, and make decisions based on the data gathered. It also naturally raises additional questions about program performance, which feeds back into the cycle of evaluation as newly proposed research questions. This feedback cycle makes possible the continuous program improvement that is characteristic of a learning organization. 

26

Facilitated example #2: using results of an impact evaluation

Facilitator notes:

Let’s review what we’ve covered in the last few slides with an example. We’ll use the fictional financial education program from your pre-work, and use some hypothetical findings from their impact evaluation to identify areas for change and develop an action plan for improvement. For the purposes of our example, we’re going to suppose that the program conducted a randomized control trial supplemented with a review of program documents.  

27

Positive RCT findings: clients improve compared to comparison group

Plan for and enact improvements

Compare results to logic model, theory of change to determine improvements

Consult document review, performance measures data

Facilitated example #2: using results of an impact evaluation

Positive, significant difference between treatment and control group

Clients are making progress in solving financial problem 6 mos. post program

Facilitator Notes:

Let’s suppose that the program ran their RCT and just received their final evaluation report from their external evaluator. In the findings section of the report, it is reported that there was a positive, significant difference between the treatment and control groups with regards to management of personal finances six months after the end of their participation in the program. The program is pleased, since according to their logic model and theory of change, they expect that a medium term outcome of the program is that clients will apply the knowledge gained through the counseling program to make significant progress towards solving a financial challenge. Since the findings show that clients are in fact making progress towards solving a financial challenge, as compared to those not in the program, the answer to the research question posed at the beginning of the evaluation supports the intervention as described in the logic model and theory of change. 

28

Positive RCT findings: clients improve compared to comparison group

Plan for and enact improvements

Compare results to logic model, theory of change to determine improvements

Consult document review, performance measures data

Facilitated example #2: using results of an impact evaluation

Member activity logs show high level of consistency in type, duration, and quantity of service being provided by members

Findings triangulated with performance measures data

Facilitator Notes:

This finding is definitely something the program wants to investigate further, because it could have implications for additional programming they could provide in the future. Specifically, the positive results indicating an effective intervention at the current sites helps make a case for increased program scale and reach. They look back to the evaluation report to see if the evaluator noted any important findings from the document review portion of the evaluation which could speak to what could be driving the positive result. The evaluator has reviewed the program’s member activity logs, and has noted that there is a high level of consistency amongst the type, duration, and quantity of service being provided by members. 

Since the program also has performance measures data available on these member activities, they consult the data to triangulate, or corroborate, that finding. The performance measures data reveals that members are completing the targeted number of sessions with each client, and are spending an appropriate amount of time with each client, as is defined in the program’s curriculum. 

29

Positive RCT findings: clients improve compared to comparison group

Plan for and enact improvements

Compare results to logic model, theory of change to determine improvements

Consult document review, performance measures data

Facilitated example #2: using results of an impact evaluation

Findings align with program as described in logic model and theory of change

Facilitator Notes:

With this contextualizing information, the program re-examines their theory of change and logic model. Because there is evidence that outcomes are being achieved, and that the curriculum is being implemented as designed, the program feels confident that the intervention is working as specified. In fact, it seems to be working well enough and with enough consistency across sites that they could expand the program to other sites with characteristics similar to those in the study. 

Even though the study revealed evidence of an effective intervention, there are still some positive improvements that can be made. First, the program can use the results to emphasize to sites the importance of following the curriculum with fidelity, and the importance of ensuring that members maintain a certain level of consistency in service provision across clients. The results also support the program’s efforts to maintain rigorous data collection and performance measurement systems, so program management will want to continue to invest in these capabilities. And finally, program management now has solid evidence to make a case for expanding the program and for obtaining additional staff and funds to do so. 

30

Positive RCT findings: clients improve compared to comparison group

Plan for and enact improvements

Compare results to logic model, theory of change to determine improvements

Consult document review, performance measures data

Facilitated example #2: using results of an impact evaluation

Think about next evaluation: fidelity study

Enhance dissemination plan

All site training day

Facilitator Notes:

Implementing these suggested improvements could be a lot of work, so again, it’s important that the program puts together an action plan for improvement. Remember, the plan would need to specify who will carry out the improvements; by when they will take place, and for how long; what resources (i.e., money, staff) are needed to carry out these changes; and who can be an advocate or partner in change. So, if the program decides it wants to emphasize to sites the importance of ensuring consistency of service provision across clients, they may decide to hold an all-site training day for members and staff. In this example, that part of the plan for improvement would note that the program management would be in charge of arranging and managing the meeting, and developing meeting materials; a convenient date would be set aside for everyone to come together; money for meeting space and materials, plus a sufficient number of staff would be allocated; and site staff would be recruited as partners in the training. 

Given the implications of the evaluation findings, the program may want to enhance their dissemination plan presented earlier in the webinar. A good addition might be a presentation at the all-site training day covering the evaluation results and implications, or wider distribution of the executive summary to members, potential new site partners, and additional funders and individual donors. This is also the time for the program to think about their next evaluation, and consider what new questions were raised as a result of this current evaluation. With their desire to expand services to other sites, an appropriate follow up evaluation could be an implementation study measuring fidelity to the program model at the new sites. The program will want to think strategically about future use of evaluation resources, and would benefit from compiling a long term research agenda to organize their future evaluation work around an ultimate evaluation goal. 

31

Positive RCT findings: clients improve compared to comparison group

Plan for and enact improvements

Compare results to logic model, theory of change to determine improvements

Consult document review, performance measures data

Facilitated example #2: using results of an impact evaluation

Continuous improvement

Increase evidence base

Serve more people, better

Facilitator Notes:

At this point, you can see how our hypothetical financial education program has used its evaluation results to make an improvement to their program, and to move towards serving more people, better. Additionally, the study has also enhanced the program’s evidence base, increased accountability to stakeholders, improved program operations, and contributed important learnings to other programs operating similar interventions. 

32

Positive RCT findings: clients improve compared to comparison group

Plan for and enact improvements

Compare results to logic model, theory of change to determine improvements

Consult document review, performance measures data

Important points to remember 

Reporting should include a variety of different products, for different audiences.

Create a dissemination plan early on in the evaluation process

Ensure credibility and objectivity in the evaluation report

Review results to identify areas for improvement, and create a plan for implementing change

Facilitator notes: 

To wrap up, let’s reiterate a few key points to remember as you report and use evaluation results. 

First, reporting should include a variety of different products, for different audiences. Be sure to match the right reporting product with each audience’s information needs and requirements. 

Next, use a dissemination plan to organize the different reporting activities you’ll need to engage in during the course of reporting your evaluation results. Start this plan early on in the evaluation process to be sure resources are invested effectively, and that all important reporting deadlines are met. 

Next, be sure to maintain credibility and objectivity in the evaluation report by discussing study limitations, generalizability, and negative or null findings if they exist. Also, hold interpretations and recommendations until the end of the report, and avoid inserting speculation and conjecture regarding results. 

Finally, review results against your research questions, logic model, and theory of change to identify areas for improvement or scale up. Organize the subsequent improvement activities in a plan for implementing change.

33

Resources on evaluation

Go to the Evaluation Resources Page on Nationalservice.gov/resources for more information:

http://www.nationalservice.gov/resources/evaluation

Community Toolbox: Using Evaluation to Understand and Improve the Initiative http://ctb.ku.edu/en/table-of-contents/evaluate/evaluation-to-understand-and-improve

UNEG Quality Checklist for Evaluation Reports https://www.iom.int/jahia/webdav/site/myjahiasite/shared/shared/mainsite/about_iom/eva_techref/UNEG_Eval_Report.pdf

Disseminating Evaluation Findingshttp://www.cdc.gov/healthyyouth/evaluation/pdf/brief9.pdf

Using Evaluation to Understand and Improve the Initiative http://ctb.ku.edu/en/table-of-contents/evaluate/evaluation-to-understand-and-improve

Using Policy Evaluation Resultshttp://www.cdc.gov/injury/pdfs/policy/Brief%207-a.pdf

Facilitator notes: For more information on evaluation, please go to the National Service Resource Page on Nationalservice.gov/resources. We also highly recommend visiting the Community Toolbox, which was used extensively in preparing the first half of this presentation. 

34

Questions?

Facilitator notes: Does anyone have any questions?

35

# image3.jpeg

# image4.jpeg

# image5.jpeg

# image6.jpeg

# image2.png

# image13.emf

# image1.jpeg

# image7.jpeg

# image8.png

# image9.jpeg

# image10.png

# image11.jpeg

# image12.jpeg