# Presentation: 236769

## Making the Most of Partnerships  through Evaluation

**Making the Most of Partnerships  through Evaluation**

**Susan Ladd**
- HDSP State Management and Evaluation Training
- September 18, 2007

## Objectives for Today

- To identify a process for evaluating partnerships.
- To identify tools for partnership evaluation.
- To begin work on evaluating a state partnership.

## Partnership

- “Group of groups with a shared goal and some awareness that united we stand and divided we fall.”*  *(Labonte, 1993)
- A partnership by any other name.....
- Network, Alliance, League, Consortium, Partnership, Advisory Committee, Task Force, Commission, Coalition

## Partnership Continuum*

- Continuum
- based on:

- Commitment
- Change required
- Risk involved
- Levels of interdependence
- Power
- Trust
- Willingness to share turf

**Networking**
**Coordinating**
**Cooperating**
**Collaborating**

- *AT  Himmelman, Collaboration for a Change

**Notes:**
) NETWORKING is defined as exchanging information for mutual benefit.  
Networking is the most informal of the inter-organizational linkages and often reflects an initial level of trust, limited time availability, and a reluctance to share turf.

(2) COORDINATING is defined as exchanging information and altering activities for mutual benefit and to achieve a common purpose.  
Coordinating requires more organizational involvement than networking and is a very crucial change strategy.  Coordinated services are "user-friendly" and eliminate or reduce barriers for those seeking access to them.  Compared to networking, coordinating involves more time, higher levels of trust yet little or no access to each other's turf.

(3) COOPERATING is defined as exchanging information, altering activities, and sharing resources for mutual benefit and to achieve a common purpose.  
Cooperating requires greater organizational commitments than networking or coordinating and, in some cases, may involve written (perhaps, even legal) agreements.  Shared resources can encompass a variety of human, financial, and technical contributions, including knowledge, staffing, physical property, access to people, money, and others.  Cooperating can require a substantial amount of time, high levels of trust, and significant access to each other's turf.

(4) COLLABORATING is defined as exchanging information, altering activities, sharing resources, and enhancing the capacity of another for mutual benefit and to achieve a common purpose.  

The qualitative difference between collaborating and cooperating in this definition is the willingness of organizations (or individuals) to enhance each other's capacity for mutual benefit and a common purpose.  In this definition, collaborating is a relationship in which each organization wants to help its partners become the best that they can be at what they do.  This definition also assumes that when organizations collaborate they share risks, responsibilities, and rewards, each of which contributes to enhancing each other's capacity to achieve a common purpose.  Collaborating is usually characterized by substantial time commitments, very high levels of trust, and extensive areas of common turf.  A summary definition of organizational collaboration is a process in which organizations exchange information, alter activities, share resources, and enhance each other's capacity for mutual benefit and a common purpose by sharing risks, responsibilities, and rewards.

## What Partnerships Do

- What Partnerships Do

- Network & share information
- Plan & coordinate
- Implement specific projects/activities
- Provide technical assistance & training
- Advocate

## Why Evaluate Partnerships?

- Builds evaluation capacity within partnership
- Determines whether objectives are achieved
- Improves partnership activities and operations
- Provides accountability to funding agencies & stakeholders
- Increases community awareness & support
- Required

## Establish evaluation plan from onset of program
Obtain buy-in from community & coalition partners to build commitment to evaluation
Allot staff time to make evaluation a priority
Match methods to your evaluation questions 
Engage priority population to help create measures that generate reliable data 
Report results clearly & often to community
Be flexible & creative

- Obtain buy-in from community & coalition partners to build commitment to evaluation
- Allot staff time to make evaluation a priority
- Match methods to your evaluation questions
- Engage priority population to help create measures that generate reliable data
- Report results clearly & often to community
- Be flexible & creative

- Seven Steps to SuccessfulPartnership Evaluation*

- *Butterfoss (June 2006)

## Framework for Partnership Evaluation *

- Determine use and users
- Engage stakeholders
- Describe your partnership
- Focus the evaluation (questions & design)
- Gather credible evidence (collect data)
- Justify conclusions (analyze data)
- Share results and lessons learned

- *CDC Evaluation Framework

## Planning Decisions

- Determine what you want your evaluation to accomplish – why you are conducting an evaluation (use and user)
- Stage of development of partnership
**Formation (assessment & partner selection)**
**Building**
**Maintenance**
- Expected outcomes** **** **
- Identify questions you want your evaluation to answer

**Notes:**
Partnership Stage
 FORMATION Assessment and Partner Selection 
 Capacity/Operations Context,   Resources,   Purpose - defined vision and mission Identified need 
Member characteristics (skills and expertise),  Recruitment strategy (interview protocols, member orientation, identified expectations) 
Outcomes: Sphere of influence or reach  

BUILDING 
Capacity/Operations Resources, Training, Processes and structures in place and functioning (communication, defined work, etc.)
Plans for operation 
Outcomes Engaged partners, Committed partners, Change in relationships  

MAINTENANCE 
Capacity/Operations Changing needs for training and staffing, Member contributions/participation,  Sustainable resources, Information feedback loop, Accountability and reporting, Member longevity, 
Outcomes Policy and systems change, Expansion or spread, Outreach effects, Progress in achieving goals, Sustainability

## Use and Users

- Who are the users of the evaluation?
- Funders
- Partner members
- Other agencies/organizations
- What will evaluation be used for?
- Quality improvement
- Impact
- Continuation, expansion, termination

## How can you use evaluation findings to improve you partnership?

## Discussion:Who are your stakeholders and how have you engaged them? 

## Example Partnership Logic Model

**INPUTS**

**ACTIVITIES **

**OUTPUTS**

**OUTCOMES**

**Staff time**
**Resources**
**Partner time and resources**

**Convene partners**
** ****Draft state plan**
**Advocate**
**Evaluate**

**State plan**
**Advocacy plan**
**Evaluation plan**

**Increased collabora-tion**
**Policy change  **

**Context:   Political climate favorable**
**                 ****Multiple chronic partnerships**

## Levels of Partnership Evaluation

- Process
- Capacity
- Environment
- Membership Characteristics
- Operations and Structure
- Communication
- Purpose
- Resources

## Levels of Partnership Evaluation

- Outcomes
- From specific projects implemented by partnership
- Related to changes in policies, practices & environment (e.g., capacity, collaboration, institutionalization of programs)
-  Health status indicators (contribution)

## Discussion:What products and outcomes do you expect from your partnership?

## Data decisions

- Indicators – what will you measure to answer your evaluation questions?
- Data Source – Where will you find data to answer questions?
- Data Collection - How will data be collected?
- Data Analysis – How will data be analyzed/interpreted?
- Timeframe - When will data be collected?

## Indicators

- Question:
- Participation?
- Organization?
- Communication?
- Maintenance?
- Outcomes?

**Indicator **

**?**

## Discussion:How have you collected data? Or how could you collect data? 

## Methods to Collect Data

- Written or telephone surveys
- Structured interviews
- Activity logs
- Document review & abstraction – minutes, records, rosters, medical records
- Focus groups
- Observation
- Existing data

**FROM      **          Partners, stakeholders, community

## Methods to Analyze Data

-   Organizing or tabulating data for meaningful interpretation
- Are there groups that should be considered?
- How will narrative data be analyzed?

- Percents, numbers

- Themes

## Interpreting Data

- Giving meaning to the data
- What does it mean that 45% of members feel that the Partnership is effective?
- Is this a high or low percentage?
- What does it means in terms of how the Partnership operates?

## Who and How Decisions

- Staff Responsible – Which staff member(s) will be responsible for overseeing the evaluation?
- Communication – To whom and how will results be communicated?

## Discussion:How have you communicated evaluation results? 

## Communicating Results  what, to whom, and how?

**  **The communication methods you use will depend on your audience** **

**Written report**
**Oral presentation**
**Short summary statements**
**Film or videotape**
**Pictures, displays, etc.**
**Graphs and visuals**
**Medial releases**
**Internet postings**

**Respondents**
**Partner **** leadership**
**Partners**
**Funders**
**Community**

## Next Steps

- Share the data
- Use the data
- What, if anything, should be done next?

## Write It Down

| Evaluation
Questions | Indicators | Data Source | Data Collection | Timeframe | Data Analysis | Communication Plan | Staff Responsible |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |

## Parting Thoughts

- Start small – plan 1 or 2 evaluation activities/year
- You may be collecting evaluation data & not know it
- Use & adapt existing tools
- Reduce respondent burden by using short instruments/interview frameworks
- Verify partner roster to maximize response rates

## Parting Thoughts

- Enlist partners to help: builds their buy-in and cooperation for evaluation
- You may recruit colleagues or contractors for various parts of the evaluation

## Tools

- Coalition Effectiveness Inventory (CEI)
- Meeting Effectiveness Inventory (MEI)
- Member Satisfaction Surveys
- Member & Leader Interviews

## Social Network Analysis

- SNA provides both a visual and a mathematical analysis of human relationships between people, groups, organizations, computers or other information/knowledge processing entities.
- SNA is focused on uncovering the patterns of people's interaction.

## Tools

- _Collaboration: What Makes It Work_, 2nd Edition. Mattessich, PW, et al. Amherst H. Wilder Foundation, 2004.  This is an up-to-date and in-depth review of collaboration research.  The edition also includes _*The Collaboration Factors Inventory*_.
- _Evaluating Collaboratives_, University of Wisconsin Cooperative Extension.  _[http://cecommerce.uwex.edu/pdfs/G3658_8.PDF](http://cecommerce.uwex.edu/pdfs/G3658_8.PDF)_ The site also includes an organizational assessment tool.
-  Introduction to program Evaluation for Public Health Programs (CDC Evaluation Framework) _[http://www.cdc.gov/eval/evalguide_pt01.pdf](http://www.cdc.gov/eval/evalguide_pt01.pdf)_

## Discussion

- What has gone well with your evaluation efforts involving partnerships?
- What have been your significant challenges?
- What would you do differently the next time you evaluate?
- How have you involved members of the priority population (statewide stakeholders in evaluation?)

## Worksheet

## Susan Ladd

- Applied Research and Evaluation Branch, Evaluation Team
- 770-488-5448
- _[sladd@cdc.gov](mailto:sladd@cdc.gov)_
*The findings and conclusions in this presentation are those of the author and do not necessarily represent the views of the CDC. *