# Presentation: 155967

## Communicating Evaluation Data

## Workshop Description

- This session will provide an overview of how evaluation questions and data collection can be implemented using a communications for sustainability framework.
- Participants will learn how to use marketing techniques and strategies to promote evaluation data effectively as well as to discover how to use this data to allow for the greatest impact with various target audiences in their communities.

## Session Objectives

- At the end of the session, participants will be able to
- Identify organizations within their community for potential sustainability partnerships
- Understand how to work with their evaluator to collect and disseminate data based on communications needs
- Create meaningful communications using evaluation data that is developed and gathered for specific audiences

## Sustainability Communications Mindset

- Look down the road not only for sustaining SS/HS initiatives but also for creating an environment that integrates the goals and mission of the initiatives into the fabric of your community
- Who needs to be at the table?
- What do we need to say to get them there?
- What data do we need to provide to make the information relevant to a particular audience?

## Who Needs To Be at the Table?

- Organizations with similar priorities as yours
- Think outside of the typical audiences
- Current grant partners
- Other community agencies/organizations
- Civic organizations
- Others?

## Who Needs To Be at the Table?

- Things to think about
- What are the top three priorities of our current partner agencies?
- Where is there natural alignment without money as a motivator?
- Within the community, what other organizations/agencies have priorities/vision/mission that align with one or more initiatives of SS/HS?
- Follow the money
- How does funding funnel into your community?
- How are existing youth initiatives funded?

## What Do We Need To Say to Potential Partners?

- Messages must be relevant to your audience
- Messages must be in a format that is easy for your audience to read and understand
- Messages must be delivered in a timely way by a trusted source

## Questions that you can use to help decide how to present your evaluation data to potential partners
What’s their definition of success?
What are their priorities?
What are they interested in?
What information do they trust?
What level of technical detail do they demand?
What data do you already have?
What data do you need to tell your story?

- What’s their definition of success?
- What are their priorities?
- What are they interested in?
- What information do they trust?
- What level of technical detail do they demand?
- What data do you already have?
- What data do you need to tell your story?

**What Do We Need To Say to Potential Partners?**

## Primary and Secondary Audiences for Communicating Evaluation Data

*Primary Users *are the individuals that you want to use your results; these people are your chief audience and often provide funding or other support
*Secondary Users *are individuals who may be associated with your program or have an interest in what you are doing (e.g., city council, neighborhoods, and service recipients)

## What Information Is Most Relevant to Your Audience’s Priorities?

- Description of
- Programs
- Cost
- Students
- School climate
- Services

- Change in
- Expenditures
- Student referrals
- School climate
- Academic performance
- Consumer satisfaction

## What Information Do They Trust?

- Caregiver-reported data
- Teacher-reported data
- Student-reported data
- Community member-reported data
- Administrator-reported data
- Agency staff-reported data
- Management information system data

## 
Once you have completed your evaluation and analysis, 
you should have information that provides an accurate
picture of your program as well as information to make
decisions about future program implementation. So, how
do you share your findings? Results can be reported not
only in a text document, but also in

Oral reports and presentations
Videos
Posters
Press releases
Newsletters
Other forms of communication 

- Once you have completed your evaluation and analysis,
- you should have information that provides an accurate
- picture of your program as well as information to make
- decisions about future program implementation. So, how
- do you share your findings? Results can be reported not
- only in a text document, but also in
- Oral reports and presentations
- Videos
- Posters
- Press releases
- Newsletters
- Other forms of communication

**What Format Do We Use?**

## 
Exercise:  

Understanding Your Audience’s 
Data Needs

- Exercise:
- Understanding Your Audience’s
- Data Needs

## Understanding Your Audience

- TO USE: Decide if each audience would prefer **quantitative** or hard data (QT); **qualitative**, self-reported, or anecdotal data (QL); or **both** (B). Then, for each audience, enter a **QT, QL, **or **B **in each of the boxes that represent the types of data your target audiences will most want to see.

| Audience Segments
Use space under each segment to specifically identify (e.g., does district leadership include superintendent? School board? Principals?) | Types of Data |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Descriptive |  |  |  |  | Outcome |  |  |  |  |  |  |
|  | Students | Program | Service | Satis-faction | Climate | Student Behavior | Academic Perform. | Service Array | Costs | Satis-faction | School Climate | Academic Indicators |
| District Leadership |  |  |  |  |  |  |  |  |  |  |  |  |
| Law Enforcement |  |  |  |  |  |  |  |  |  |  |  |  |
| Mental Health Providers |  |  |  |  |  |  |  |  |  |  |  |  |
| Policymakers |  |  |  |  |  |  |  |  |  |  |  |  |
| Business Leaders |  |  |  |  |  |  |  |  |  |  |  |  |
| Faith-Based Leaders |  |  |  |  |  |  |  |  |  |  |  |  |
| Community Members |  |  |  |  |  |  |  |  |  |  |  |  |
| Other |  |  |  |  |  |  |  |  |  |  |  |  |

## Inventory Your Data

- What type(s) of data do you have?
- Where did your data come from?
- What is the quality of your data?
- What message does the data convey?

## What Data Do You Need?

- What do you currently have?  What are you currently gathering?
- Depending on the status of your program evaluation you can
- Work with your evaluator to ensure that evaluation instruments that will provide you with relevant data for communications goals are developed
- Use existing evaluation data in a way that is relevant to your audience
- Some of both

##  Qualitative Data

- Provides answers to open-ended questions
- Derives from notes from observations
- Uses words not numbers
- Is obtained through focus groups and unstructured interviews
- Includes contextual detail about feelings and perceptions in an effort to gain a holistic understanding
- Includes relatively few participants

-  Quantitative Data

- Provides answers to close-ended questions
- Uses numbers not words
- Is obtained through structured surveys and interviews, existing records
- Often includes a large number of participants
- Provides less contextual detail
- Is amenable to common statistical procedures

**What Type(s) of Data Do You Have?**

## What Type(s) of Data Do You Have?

- Descriptive Data
- Is cross sectional
- Describes characteristics of people, organizations, or programs at a single point in time

- Outcome Data
- Is longitudinal
- Tracks people, organizations, or programs over time by using the same measures
- Assesses change in people, programs, or organizations
- Has an established baseline

## 
Exercise:  

Do You Have What You Need?

Data Inventory Worksheet

- Exercise:
- Do You Have What You Need?
- Data Inventory Worksheet

## Data Inventory Worksheet

- TO USE: Check off the boxes that best represent the types of data you currently are collecting. Note where you might need to collect additional data, based on your findings from the Audience Data Preferences Worksheet. Discuss with your evaluator how you might fill in those gaps.

| Data Source | Type and Quality of Data |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Qualitative Descriptive |  |  | Qualitative Outcome |  |  | Quantitative Descriptive |  |  | Quantitative Outcome |  |  |
|  | Good | Fair | Poor | Good | Fair | Poor | Good | Fair | Poor | Good | Fair | Poor |
| Agencies |  |  |  |  |  |  |  |  |  |  |  |  |
| Parents |  |  |  |  |  |  |  |  |  |  |  |  |
| Students |  |  |  |  |  |  |  |  |  |  |  |  |
| Teachers |  |  |  |  |  |  |  |  |  |  |  |  |
| Systems Generated |  |  |  |  |  |  |  |  |  |  |  |  |
| Community Members |  |  |  |  |  |  |  |  |  |  |  |  |
| Academic Indicators |  |  |  |  |  |  |  |  |  |  |  |  |
| External Comparison |  |  |  |  |  |  |  |  |  |  |  |  |
| Other_____________ |  |  |  |  |  |  |  |  |  |  |  |  |

## So Far We’ve - 

- Identified potential partners/audiences
- Analyzed the type of data and information that is relevant to them
- Determined the data that you currently have and/or are collecting and identified where there are data needs
- So how do we take all of this information and put pen to paper?

## Planning Your Project: Things To Decide Before Anyone Writes a Single Word

- Who is going to read this?
- What do they have in common?
- When they’re done reading, what do you want this audience to do or know?
- What materials do you have to work from? (See Data Inventory Worksheet)

## Planning Your Project: Things To Decide Before Anyone Writes a Single Word

- What kind of research and/or interviews are necessary? (See Audience Worksheet)
- Will this piece be printed only, or will it be posted on the Web?
- Do you need to design the piece, or must you fit the text into one that’s already designed?
- What “extras” do we need, such as sidebars, charts, or photos?

## Planning Your Project: Things To Decide Before Anyone Writes a Single Word

- Internal factors
- Who will have input, and who has the final say?
- If a lot of people will have input, who will coordinate their comments and resolve any disputes?
- Does everyone agree on the number of drafts allowed and when the draft will become final, with no more changes?
- What resources are available to help you complete this project?

## Planning Your Project: Things To Decide Before Anyone Writes a Single Word

- Internal factors
- Do you have someone who can put in the time necessary for organizing, researching, thinking, drafting, re-writing, and polishing over the life of the project?
- What is the budget for this project?
- How will your materials be distributed?

## What Information Should Be Included When Sharing Evaluation Data?

- State your program objectives
- This is important because, eventually, you will discuss how your evaluation data contributes to your stated objective.

## What Information Should Be Included When Sharing Evaluation Data?

- Describe progress toward achieving your objectives
- Describe your activities as they relate to your objectives
- Identify your beneficiaries—the number and characteristics of people you serve
- Describe your desired results—talk about the indicators of success that you have chosen. These are the factors that are most important to your program

## What Information Should Be Included When Sharing Evaluation Data?

- Describe your evaluation activities
- Identify the types of instruments you used
- Report on the information you collected
- Detail who administered your instruments, how they were administered (e.g., by mail, in person, or by telephone), and who, as well as how many people, completed them
- Note how you measured your desired result
- Discuss any problems encountered during the evaluation

## What Information Should Be Included When Sharing Evaluation Data?

- Describe the results of your evaluation
**This is the most important section!**
- Report your findings—statistics and qualitative information
- Employ graphs, charts, and tables to help convey your message
- Share stories from people who are directly involved in your program and/or your evaluation

## What Information Should Be Included When Sharing Evaluation Data?

- Compare the results of your evaluation with your objectives
- Describe how your services addressed the existing need. You can relate the need to baseline data if you have it
- Refer to previous evaluation findings (such as those reported in earlier reports) to provide a more detailed picture of the program progress

## What Information Should Be Included When Sharing Evaluation Data?

- State ideas for partner support and ask for commitment
**This is your “ask”**
- What do you want the audience to do?
- Show them a clear path to doing it
- Incorporate message development work to ensure success

## Pulling It All Together

**Harrisburg, Pennsylvania **
**Student Survey Data**
- Goal:  To create meaningful communications using evaluation data that is developed and gathered for specific audiences

## Harrisburg, PAStudent Survey Data

- Purpose
- Evaluate prevention programs at the secondary level
- Life Skills Training
- TimeWise
- HealthWise
- School Transitional Environment Project (STEP)
- Teen Outreach Program
- Establish a monitoring system for social, emotional, and behavioral outcomes

## Harrisburg, PAStudent Survey Data

- Methods
- Conducted annually (beginning in October 2003)
- All students in 6th–12th grade
- Paper/pencil administration during a required class
- 77 percent response rate

## Harrisburg, PAStudent Survey Data

- Communications Goals
- Give the data back to students before the 2004 survey administration
- Integrate data into agency accountability systems
- Drive program decisions during grant period

## Harrisburg, PAStudent Survey Data

- Target audience:  Project director
- Communications goal
- Drive program decisions during grant period
- Messaging
- Straightforward report of all data
- Not bad vs. good, but pointing how to improve quality
- Channel
- Bound report

## Harrisburg, PAStudent Survey Data

- Target audience:  Students
- Communications goal
- Give data back to students before the 2004 survey to increase participation
- Message
- Show students that survey data is used and relevant
- Channels
- Student newsletter
- Article from student perspective in school newspaper

## Harrisburg, PAStudent Survey Data

- Target audience:  Steering committee
- Communications goal
- Drive program decisions during grant period
- Messaging
- How do our students compare to national and State averages?
- These students are all our responsibility, how can we collaborate to provide the best mix of programs given these data?
- Channel
- PowerPoint presentation

## Harrisburg, PAStudent Survey Data

- Target audience:  Superintendent
- Communications goal
- Integrate data into agency accountability systems
- Messaging
- The climate of schools is important; we care about the whole student (socially, emotionally, behaviorally, and academically)
- Channel
- Talking points in PowerPoint of survey highlights for Superintendent’s annual “state of the union”

## Other Audiences

- How would you use this same data for other audiences?
-  Businesses?
-  Community organizations?
-  Parents?
-  Others?
- What different information might you need for these audiences?

## 

Questions?

- Questions?