# Presentation: 393776

## Performance Measures, Outcomes, and Impacts

- Sharon Stout
- Office of Planning
- and Accountability
- April 24, 2007

## Objectives 

- To discuss CSREES use of performance measures in the recent past and in the present;
- To review what CSREES OPA hopes to find in future performance measures, outputs, outcomes, impacts; and
- To encourage ongoing dialogue on performance measurement, measures and indicators within the CSREES Land-Grant University System.

## Fundamental Principles 

**Axiom 1**:  What is measured can be improved.
**Axiom 2**:  The above statement is true – only if we are measuring the right things.

**Notes:**
Key  word here is “improve”. 

We want to move beyond the standard quotation “What gets measured gets done” and think about HOW what gets measured influences what gets done.    

If we are measuring the WRONG things – and taking steps to improve our score, we can actually make things worse...  
Challenge to the listener – if you get bored while I am speaking – suggest you think of examples of counter-productive measures .

Numerous examples of poor choice of measures/indicators in government and business ... 

“ ... Among the hundreds of managers with whom I have discussed this matter, there is a widespread consensus that they measure too much or too little, or the wrong things, and that in any event they don’t use their metrics effectively.”  
Ittner, Christopher D. and David L. Larcker.  “Coming Up Short on Nonfinancial Performance Measurement.”  Harvard Business Review On Point Reprint 5380, reprinted from HBR, November, 2003.  Research based on study of 60 companies – supplemented with survey responses from 297 senior executives.

Hamner, Michael.  “The 7 Deadly Sins of Performance Measurement and How to Avoid Them.”  MIT Sloan Management Review, pp. 19-28, Spring 2007. 
Seven Deadly sins:  
Vanity – using measures that will make the program, staff, and especially managers look good
Provincialism – letting organizational boundaries and concerns dictate performance measures
Narcissism – measuring from the program’s point-of-view, rather than the public’s or customer’s point-of-view 
Laziness – assuming that one knows what is important to measure without giving the question adequate effort or thought... 
Pettiness – measuring only a small component of what matters, and forgetting the bigger picture.
Inanity [a/k/a perversity]– selecting measures with counter-productive results 
Frivolity – not being serious about measurement in the first place... 

## Critical Elements for Success  

- Select the right things to measure – both controllable and important to achieving success;
- Measure those things in the right ways – with useable indicators and targets;
- Employ measures in a framework to improve performance; and
- Create an organizational culture valuing and encouraging use of measures.

**Notes:**
Presentation today will basically follow this outline, and we will return to a number of important questions, such as 
1)  Controllable – by whom?  This is a really critical question
Useable indicators and targets – we will all need much more information on these, and you are welcome to raise or send on your questions
Frameworks – of what kind?  Both logic models – and for some, log frame analysis.  Also, consider systems – at least intuitively...
4)	Culture – more on this later... 

## Outline 

- CSREES Past Use of Performance Data
- Using a Performance Measures Framework, including Logic Models, to Improve
- CSREES Examples
- Developing and Using Performance Indicators and Targets
- Using and Valuing Measures

**Notes:**
These are the five section to my talk. 
They are not all of equal length – so I will go in more depth on some than others. 
Also, I will cycle back through a couple of slides more than once. 

**CSREES Past Use **
**of Performance Data**** **

## CSREES Program Leadership and Showcasing Results 

**CSREES and the Land-grant System are**
- Addressing the national problems identified in the Strategic Plan;
- Choosing the right actions to respond;
- Achieving results; and
- Communicating with stakeholders, including funders, the public, and the Land-grant System.

## CSREES Accountability

- Defined as “the state of being accountable; liability to be called on to render an account; the obligation to bear the consequences for failure to perform as expected.”
- CSREES is accountable to the American people, the Administration, and Congress for the use of public funds.
- Webster’s Dictionary, 1913

**Notes:**
We will come back to the concept of accountability at the end of this talk. 
For now, it is enough to note that the American people are listed first here, because 
  We’re using their money – Federal tax dollars and we are especially conscious of this at the Federal level
  We are addressing their priorities – even as we recognize that these can be expressed through Federal, state, and local governments. 

## Outputs, Outcomes and Impacts

**State Plans of Work and Annual Reports were used to provide examples for:**
- Budget justification and narrative;
- PART self-assessments;
- USDA Reports of Accomplishments;
- Communication with Partners, other agencies, and the public.

**Notes:**
Ooops.  I realized that my list of uses is shorter than the one Bart [Hewitt] offered the other day. 
So what are the additional items that were on his list and not on mine, above?
Congressional inquiries
OIG
Any others? 

## Success Extracting Results? 

- CSREES has not been as successful as we would like to be in presenting a ***systematic report*** of results from State POW annual reports.
- External panels brought in to review CSREES accomplishments for the Portfolio Review Expert Process noted the lack of an adequate data system – and results.

**Notes:**
As mentioned earlier, not an uncommon failure – including in both the private and public sector.  
These are difficult issues – both in the private and the public sector.  

I searched for some recent literature in the private sector, to see what we could learn from them, checking both the Harvard Business Review, and the Management Journals of the Sloan School at MIT.   

Sample quotation -- 
“ ... Among the hundreds of managers with whom I have discussed this matter, there is a widespread consensus that they measure too much or too little, or the wrong things, and that in any event they don’t use their metrics [measures] effectively.” 
Ittner, Christopher D. and David L. Larcker.  “Coming Up Short on Nonfinancial Performance Measurement.”  Harvard Business Review On Point Reprint 5380, reprinted from HBR, November, 2003.  Research based on study of 60 companies – supplemented with survey responses from 297 senior executives.

Hamner, Michael.  “The 7 Deadly Sins of Performance Measurement and How to Avoid Them.”  MIT Sloan Management Review, pp. 19-28, Spring 2007.   
Examples of the seven Deadly sins:  
Vanity – using measures that will make the program, staff, and especially managers look good
Provincialism – letting organizational boundaries and concerns dictate performance measures
Inanity [a/k/a perversity]– selecting measures with counter-productive results 

## Representative Comments by Experts in Reports on PREP...  

- “The panel was disappointed with the evidence provided...”
- “The portfolio failed to present a complete picture of all the inputs, outputs and outcomes.”
- “There is a strong need to improve accountability showing measurable impacts, not just in CSREES, but throughout the system and down to individual investigators.”

## Improved POW System is  Raising Panelists’ Expectations

- “... *Plan of Work *and *One Solution *are expected to improve the documentation of significant findings.”
- “In 5 years, [the panel] expects to see the consistent information across knowledge areas necessary to evaluate the portfolio properly.”

## Expectations of Panelists for CSREES OPA and NPLs 

- CSREES OPA will work ...”to improve data collection, performance measurement and reporting.”
- “CSREES and NPLs should have better communication with state partners in order to get significant evidence of outputs and impacts.”

## What is Happening With CSREES POW and 1Solution?   

- Exciting developments – as reported in other presentations here by Bart Hewitt – and
- 	CSREES ISTM.
- “Management dashboards” will enable CSREES deputies, NPLs, and P&A to examine trends – and drill down to specific projects.
- States can access information from other states.

**Notes:**
Refer to ISTM presentations, development of management dashboards.  But realize that as access to data improves, attention to the quantity and quality of the data will also increase.  

Dilemma – Complaint of a private sector management – 
“Now I have this management dashboard that can call up 300 indicators – 
I just want the 20 that matter.”  
Paraphrased quotation, taken from 
Ittner, Christopher D. and David L. Larcker.  “Coming Up Short on Nonfinancial Performance Measurement.”  Harvard Business Review On Point Reprint 5380, reprinted from HBR, November, 2003.  Research based on study of 60 companies – supplemented with survey responses from 297 senior executives.

Also, possibility of calling up responses to POW annual reports in software such as 
I-XReveal and being able to ‘eyeball’ across the range of indicators reported by similar programs?  More on this as events unfold?  

## Present Status, Future Hopes   

- Exciting developments refers to changes in our information systems
- CSREES OPA, ISTM, and Budget Office work together to make better use of our information
- How can we jointly improve the information in our systems?

## Reducing Burden, Increasing Use

- CSREES is no longer collecting ‘‘impacts” through a separate system.
- CSREES is relying on Plans of Work and annual reports for *both* “stories and statistics” for planning, reporting, and budget justification.

## Using 

**Performance Measurement **
**Frameworks to Improve **

## Performance Measurement Definition and Use 

- Ongoing, regular collection of information for monitoring how a policy, program or initiative is doing at any point in time.
- Designed and used to report on the level of attainment of planned results and on performance trends over time.

## Performance Measurement Framework  

- Begins with design of a policy, program or initiative and evolves over time.
- Always is engaged as part of the ongoing management of a policy, program or initiative.
- Continues from the initial choices of performance measures and indicators, through performance monitoring to formative and summative evaluation.

**Notes:**
Formative evaluation – defined as evaluation for improvement, and frequently considered analogous to process evaluation. 
Summative evaluation – defined as evaluation to assess merit, significance, or worth. 
Not simply analogous to outcome or impact evaluation (though could be considered as in some cases as including outcome evaluation) but more generally incorporates pre-selected criteria by which to judge merit, significance, or worth. 

[See Michael Scriven, Evaluation Thesaurus, 4th ed., for more on definitions – and/or check the EVALTALK archives, www.eval.org, for further discussion.]  

## Performance Measurement  Framework Cycle

**5**

**Notes:**
Feedback is the most critical word on this slide – which shows how the same data can be used for performance monitoring AND for evaluation. 
Ongoing performance measurement provides regular snapshots of the performance of a policy, program or initiative. Through this monitoring, it serves as a descriptive tool about how the policy, program or initiative is doing. 
It is important to realize that ongoing performance measurement does not address the issues of how an outcome was achieved or why a strategic outcome was or was not realized. Explanations of how or why outcomes were achieved comes from evaluation, which is discussed in greater detail in a later section of this guide.
Performance measurement information provides managers with knowledge they require in order to manage their policy, program or initiative on an ongoing basis. It can provide reassurance that outcomes are unfolding as expected, or can serve as an early warning that the planned results are not occurring (which could lead to a decision for additional research, such as through evaluation, to determine why). Performance measurement and evaluation are also inextricably connected - ongoing performance measurement could be considered to be part of evaluation. The ongoing performance monitoring information that has been regularly collected can be used in periodic evaluations (which focus more in-depth on explaining the outcomes achieved) of a policy, program or initiative.
Figure adapted from XX

## Criteria for Assessing PM Frameworks   

**Useful for results management and accountability  **
**Shared ownership **
**Transparent  **
**Decision- and action-oriented **
**Credible and realistic  **
**Flexible **

## CSREES Use of PM Framework

- Planning tools (‘strategic thinking’) -- to focus attention on desired outcomes
- Model to identify and track measures  of interest -- inputs, outputs, and outcomes.
- Evaluability assessment – does this program on its face make sense? (Identify questions to be addressed in an implementation plan)

**Notes:**
Joseph Wholey and Midge Smith have addressed evaluability assessment – reviewing the “logic” of a program in the planning stages and again begin commencing any formal evaluation to ask whether the program has a reasonable shot (as designed, and then as implemented) in solving the original problem.  
Look carefully at the problem first...  
Could the program be delivered, carried out as planned – and still not work? 
Wholey – “program plausibility” – what has to happen for the program to be solved, and the desired end achieved?
Midge Smith’s example, in discussion on EVALTALK (AEA listserve, through www.eval.org) on a food preservation program to reduce hunger/food insecurity among migrant workers by teaching them canning.  But the cheapest containers are glass – not suitable for migrant  workers and their families who have to move around.  

## PM Framework in CSREES Context ... 

**Performance measurement framework is designed with organization**
- Mission;
- Vision;
- Strategic Goals;
- Objectives; and
- Organizational and functional linkages.

**Notes:**
What are organizational and functional linkages? 
Yesterday at the Administrative Officer’s Conference, in the Eval mini-series, we got a detailed look at how some states approach developing their Plans of Work.  
At the Federal level, we do not yet know enough about variations in State level planning. 
[Encourage people to go to Mary McPhail Gray’s session on Building Consensus on Indicators ...] 

## CSREES Mission and Vision... 

**Mission  **
- To advance knowledge for agriculture, the environment, human health and well-being, and communities
**Vision **
- To improve the lives of people ***worldwide ***through an agricultural knowledge system  sustained by the innovation of scientists and educators.

**Notes:**
The mission of CSREES is to advance knowledge for agriculture, the environment, human health, and well-being, and communities ... 

which CSREES fulfills by leading the advancement of knowledge through its vital linkages with the components of a broad-based, national agricultural higher education, research, extension system utilizing the partner resources of the US Department of Agriculture, with land grant and other colleges and universities, and public and private laboratories. 
Message from the Administrator, Colien Hefferan, 2007-2012 Strategic Plan.  

Worldwide – examples?  Lane Rawlins in his presentation mentioned Malawi, and what the World Bank calls “bimodal development” (i.e., addressing the need for low-tech innovation as well as high tech innovation in developing nations).   

## Types of Strategies

**Organizational Stimuli**

**Intended Strategy**

**Deliberate Strategy**

**Realized Strategy**

**Unrealized**
**Strategy**

- Source: Mintzberg, H. Ahlstrand, B. Lempel, J. (1998)

**Emergent Strategy**

**Notes:**
Formal planning alone is not the best way for managers to develop strategy. Facts, figures, and forecasts are necessary; but managers also need an intuitive understanding of the organization, a feel for the business not unlike a potter's feel for the clay. Strategy is not just a plan for the future but also a pattern out of the past. Strategies are not always deliberate--they also emerge over time as organizations innovate and respond to their markets. By seeing patterns take shape in their environments, the best strategists find strategies as well as create them. McKinsey Award Winner. 

Strategic Thinking - Future is only vaguely discernable
Strategic Planning - Contingencies are relatively predictable

Formulation:
Strategic Thinking 
Interactive strategy development
Lower level managers have input to strategy development
Emphasis on flexibility in developing strategies
Strategic Planning
Often imposed from above – “top-down”

Implementation:
Strategic Thinking
All Mgrs. understand the larger system and their role in it
	 Greater self-organization – Mgrs. act independently but consistent with plan
Implemented from lower down in hierarchy
Strategic Planning
Mgrs. not expected to understand the larger system
Mgrs. directed to impose plan from above 

## CSREES Strategic Goals 

- Enhance international competitiveness of American agriculture;
- Enhance competitiveness and sustainability of Rural and Farm economies;
- Support increased economic opportunity and improved quality of life in rural America;
- Enhance protection and safety of the Nation’s agriculture and food supply;
- Improve the Nation’s nutrition and health; and
- Protect and enhance the Nation’s Natural Resource Base and Environment.

## CSREES Logic Model as “Roadmap”

**Describe**
**the problem, challenge,  and opportunities**

- Situation

- Inputs

- Outputs

**Occurs **
**when **
**there **
**is a **
**change **
**in **
**knowledge**
**or the**
**participants**
**actually**
**learn**

**Actions**

**Conditions**

- Outcomes

**$$$**
**Other **
**Partners**
**NIH, ARS**

**Describe**
**what comes out from funded activities**

- Activities

**What we do in Research**
**Extension**
**Education**

**Knowledge**

**Occurs **
**when there **
**is a change **
**in **
**behavior **
**or the**
**participants**
**act on what**
**they have **
**learned**

**Occurs **
**when a **
**societal**
**condition**
**is improved**
**due to**
**a participant**
**action taken**
**in the **
**previous **
**column**

**EXTERNAL FACTORS**

**Notes:**
... a logic model serves as a "roadmap", showing the chain of results connecting activities to the final outcomes and, thus, identifying the steps that would demonstrate progress toward their achievement. 
Design – 
Uses – 
 1)  Tabulating, calculating, recording activities or effort – expressed in qualitative or quantitative manner
Assessment of results of  program activity compared to intended purpose – defining data to be collected or evaluated 
 CSREES OPA uses “impact” as synonymous with farthest right hand column, changes in conditions. 
Experience has shown that some groups may find it useful to start their logic model work with the identification of final outcomes, that is, to begin with this last row of the model... 
In a strategic thinking exercise – using the Will and Must, journalist’s question’s 
exercises from Dewar, 2002, Assumption-Based Planning, we started with 
the assumptions and external factors – and THEN returned to listing activities, and then list outputs, outcomes.  Comment from CSREES deputy – “subtle but good to go through this exercise in a group”.    

## Water Quality Program: Logic Model

**Water Quality Program: Logic Model**

**Inputs**

-                    **Outputs**
**Activities****	****         Participants**

-                 **Outcomes- Impact**
**Knowledge         Actions      Conditions**

**S**
**I**
**T**
**U**
**A**
**T**
**I**
**O**
**N**

**P**
**R**
**I**
**O**
**R**
**I****T**
**I**
**E**
**S**

**Staff**
**Money**
**Materials**
**Partners**

**On-farm visits**

**Set up record keeping systems to track phosphorus**

**Educational workshops**

-  **Farmers**
** ****at risk of overfeeding phosphorus**

**Increased knowledge of link between cattle diet and water quality**

**Increased understanding of recommended phosphorus levels**

**Make appropriate adjustments to cattle feed**

**Monitor phosphorus levels in feed, manure, and soil**

**Improved water quality**

**Feed cost savings**

**Reductions in phosphorus use**

**Increased knowledge of tracking phosphorus levels**

**Notes:**
Use the logic model to identify changes – and then work to answer questions about changes, by examining the linkages (arrows). 

Consider how scaling up affects the model – e.g., in a water quality context, what does it mean to change from the field to the watershed level?  How do linkages change?     

## Water Quality Program: Performance Measurement, Evaluation Questions    

| Were the inputs sufficient, timely? Did they meet the program goals?  Were they sufficient given the situation? | Did all activities occur as intended?  Quality of intervention: appropriate content; usable record keeping system; well planned visits? | Did the targeted farmers participate?  Who did not?  What were their reactions?  Who else was reached? | Did knowledge increase?  Did know-ledge  of P levels increase?  What else happened? | Are farmers monitoring P levels?  
Are they adjusting cattle feed?  Anything else?
 | Is there a reduction in P use? 
How much $ saved?  Other benefits?  Negative consequences? |
| --- | --- | --- | --- | --- | --- |

**Water Quality Program: Performance Measurement, Evaluation Questions    **

**Inputs**

-                    **Outputs**
**Activities****	****         Participants**

- 	            **Outcomes**
**Knowledge            Actions                Conditions**

**Notes:**
For actual data, generally three sources: 
Administrative data, collected routinely in the course of the program
2)	Primary data, specific collections (e.g., surveys, focus groups, observation, etc.)
3)  Secondary data, “canned data” collected by other agencies (e.g., national data on changes in conditions?).   

## Water Quality Program: Indicators

**Indicators: How will you know it?**

| # of staff; $ invested; delivery timetable  | # of workshops data systems, on-farm visits, Farm self-reports of quality | Actual vs. desired attendance. #, % attended per session | #, % with increased knowledge of ...  Additional outcomes: + & -  | #, % monitoring P levels; making adjustments to cattle feed Additional outcomes: + & - | #, % reducing P use; level of reduction.  # dollars saved/farmer. Additional outcomes: + & - |
| --- | --- | --- | --- | --- | --- |

**Water Quality Program: Indicators**

**Inputs**

-                    **Outputs**
**Activities****	****         Participants**

-                                   **Outcomes**
**Knowledge            Actions                Conditions**

**Notes:**
When collecting/reporting data, try to include both numbers and percentages.
Increased knowledge? Needs to be specified in a way that is meaningful to the program – and to other similar programs. 
In the example above – #,% 	knowing how to monitor P levels
		how to adjust cattle feed
		how to assess reduced P levels   

## Uses of the Logic Model

- Clarifies the linkages between activities, outputs and expected outcomes of the policy, program or initiative
- Communicates externally about the rationale, activities and expected results of the policy, program or initiative;

## Uses of the Logic Model

- Tests whether the policy, program or initiative "makes sense" from a logical perspective; and
- Provides the fundamental framework on which the performance measurement and evaluation strategies are based (i.e., determining what would constitute success).

## Performance Measures vs.  Performance Indicators  

- Performance measures are conceptual, and need to be operationalized
- Performance indicators are operationalized, to specify actual data (qualitative or quantitative) to be collected and used

**Notes:**
As indicated by quotations from Panel members earlier – this is a area of real need for further development of the POW reporting system.

As results are reported in the new POW system (April 2008?), it will be possible for States to see easily what other States are reporting.  

## What is a Performance Indicator?

-  	An indicator is a quantitative or qualitative amount used to determine the extent to which progress is made toward an outcome.
_*Quantitative Example*_:  percent of firms exporting 10 percent of their produce
_*Qualitative Example*_:  distance between mother and child pairs in video of play activity

**Notes:**
Performance measures will provide information regarding the progress of the program and its activities.  Performance indicators identify the particular factor being measures to provide precise detail regarding the program’s progress. 
A performance indicator possesses special attributes to which particular attention needs to be paid.
Validity – is the measure capturing the right thing? 
Reliability – is the measure consistent over time, across different raters (e.g., inter-rater reliability).  
Unambiguous – ??? 
Disaggregratable (sp?) – can the data be disaggregrated, aggregated? 
Distance between mother/child pairs comes from a parenting program offered in a residential treatment facility for women and their children.  At the beginning of the program, in some cases, women when asked to play with their kids were so far away that mother and child could not be captured in the same video image.  Trained raters assessed the videos pre- and post- program and rated this and other measures. 

## Outputs

**Products, services and events that are:**
- Intended to lead to outcomes; and
- Linked to problems or issues to be addressed – through the logical or causal chain of events depicted in the model.

## Performance MeasuresExamples -- Outputs

- Number of training sessions held
- Number of participants trained
- Number of instructional hours
- Number of publications
- Number of patents

## Outcomes

**Planned results or changes for individuals, groups, communities, organizations or systems, including: **
- Changes in knowledge;
- Changes in behavior; and
- Changes in conditions (impacts) -- *resulting in solution of the original problem or issue.***    **

**Notes:**
As used by CSREES, “impact” is synonymous with “change in condition.” 
In the State POW and Annual Reports, we hope to find stories and statistics, giving us both quantitative indicators and qualitative information.

## Examples of Outcomes 

**Knowledge**: Change in level of knowledge regarding plant production
**Behaviors**: Change in farming practices
**Conditions**: Increased food security

**CSREES Examples**
**Performance Measures **
**and Indicators **

## Examples ...  

- Soybean Rust
- CEAP
- Cryptosporidium
- Nutrition and Obesity
- Family Financial Security

**Notes:**
Examples above were selected because most (if not all) have nationally collected data on social problems of interest – collected by other Federal agencies in some cases – that are of interest to CSREES.  

At the Federal level, we do try to partner with some of these agencies in planning programs, etc. and we want to be cognizant of how they are tracking data on conditions.  

## What are We Seeking? 

**Quantitative and qualitative evidence that**
- Researchers are identifying possible solutions to national problems;
- Educators and extension faculty and staff are helping the public learn; and
- The public is applying new knowledge and  addressing national problems, thus
- Producing changes in conditions.**    **

**Notes:**
From the Federal CSREES view, this is my single best slide – because what we are trying to show is that we can get research out and working in the field consistently.
Comparisons?
It took the British navy 264 years – from the discovery that limes cure scurvy to adopt a policy and put it into practice.
National Academy of Science (NAS) study found that it takes 17 years (on average) for innovations from clinical trials to find their way into practice– with uneven resluts.
Last weeks New York Times mentioned that hand-washing by doctors – identified 200 years ago as a major health issue?– is still an issue     

Explain that we know that this is hard – and why it is so... 
E.g., time frames
Post-award management 
Scale 
Etc.  
Nonetheless, it is what we are seeking... evidence that the CSREES research, education, extension system as applied to a problem produces consistent results in a reasonable (?) time frame.      

## Soybean Rust

- Devastating disease worldwide – some regions lost 60-80% of their soybean crop
- First found in November, 2004 by an Extension Specialist at Louisiana State University trained by a CSREES program, SPDN
- Rapid response by USDA agencies and Land Grant partners
- CSREES and its National Plant Diagnostic Network (NPDN), provided disease recognition and pathogen diagnostic tools.
- Helped save $11 to $299 million in 2005 ...

## Penn State Soybean Rust Model

**Penn State Soybean Rust Model**

## Soybean Rust in 2005

**Soybean Rust in 2005**

- Zooming to a state brings up county level resolution and specific guidelines for that state.

**Notes:**
The map may be zoomed in to any part of the covered area with a zoom tool (click). The most appropriate state commentary will be displayed along with the magnified area.
Crop insurance assistance is built in, and helps insure that producers are consulting the site/seeking advice from Cooperative Extension agents.  

##   Producer Access to Information

**  ****Producer Access to Information**

**Notes:**
Information obtained from Kitty Cardwell, CSREES NPL, indicates lots of hits during the growing season.  
Thus a reasonable inference – that possibly could be bolstered with additional data – that cost savings were at the higher end of the initial ERs estimates.  

## Soy Bean Rust Outputs 

- Research papers modeling spread of soy bean rust;
- Training for extension agents in identifying and treating soy bean rust;
-  Development of Internet Pest Information Platform.

## Soy Bean Rust Output Indicators  

- # of research papers modeling spread of soy bean rust;
- # of training hours for extension agents
- # of participants trained
- Completed development of Pest Information Platform

## Soy Bean Rust Outcomes 

**Knowledge** – producers aware of spread of soy bean rust and access Pest Information Platform for advice on treatment;
**Behavior** – producers reduce fungicide use (cease treating in advance of spread);
**Conditions** – crops appropriately treated, cost savings, less fungicide in ground water.

## Indicators

**Outcomes:**
**Knowledge** – number of producer consultations with Extension agents;
**Behavior** – amounts and locations of applied fungicide; claims filed for crop damage
**Conditions** – TBD – ERS model of cost savings

**Notes:**
In this case – 
Could use  (if one had the resources) surveys or interviews with producers to check on actual use of fungicides.  
E.g., I interviewed farmers on the Eastern Shore of Maryland about their use of chicken manure as fertilizer on crops, as part of a study trying to identify sources of run-off into the Chesapeake Bay (thought to be implicated in outbreaks of pfisteria in fish).  

## Water

- Basic need and pressing national problem
- Directly related to public health, food safety, and other critical public concerns
- Requires research, education, and extension
- Model exemplar for CSREES and the Land Grant System results under the new reporting system?

## 	“Surf your watershed...”   

- http://cfpub.epa.gov/surf/locate/map2.cfm

**Notes:**
This website is funded by EPA/USDA – offers a clickable map to allow US residents access to information about their local water supply.  

 This is an example of where we’re going with data on conditions – trying to get Federally coordinated data on local conditions – so as to be more accountable to the public. 

## Slide 53

## CSREES Actions

- Listening sessions
- Cross-agency planning
- Partnerships
- CSREES White Papers
- Coordinated programming
- Funding, including Formula and Competitive

## Water Situation ... 

- Of 147 countries ranked for water efficiency, the United States ranked last ... (NCSE, 2004).
- Growing populations and changing values have increased demands, increasing water use and management conflicts.
- Agricultural needs conflict with urban needs and with demands to sustain or improve ecosystem services, recreation, and tourism.
- Overuse of water resources  adversely affects water quality.

## Research

**Research**

**Program Development**

**Education/**
**Outreach**

**Policy Development**

**Sustainable Water Resources Management**

**Water Users: Industrial, Ag Sector, Rural/Urban Communities**

**Existing WQ**
**Network**

**Desire for Formula Funds**

**Limited Human Resources**

**Status Quo**

**Funding**

**Global Visibility**

**Oversight**

**Behavioral Science**

**Biological Science**

**Physical Science**

**Water Education Foundations**

**Outreach and Demonstrations**

**Adoption Education K-12**

**Water Security **

## Slide 57

## CEAP – Research Outputs Indicators

- Publications
- Water quality and agricultural management parameters and data linking water quality and agricultural management trends.
- Land use/water quality model (AnnAGNPS) of Rock Creek.

**Notes:**
“Science-based decision making and management practices that improve the quality of the Nation’s surface water and groundwater resources in agricultural, rural, and urbanizing watersheds”
The project has two purposes. The first is to analyze an extensive water quality database for Rock Creek (Seneca County, OH) to determine trends in water quality, and compare them with trends in the application of management practices, to evaluate whether water quality trends can be explained by agricultural management trends. The second purpose of the project is to develop a land use/water quality model (AnnAGNPS) of Rock Creek, show that it reproduces the observed water quality, and then use it to explore a series of questions related to the effectiveness of different BMPs, and how effectiveness is affected by when and where in the watershed these practices are applied.... 
Manuscripts detailing these findings are in preparation, and more than 20 talks have been given on these topics at professional meetings and university seminars. An AnnAGNPS model has been constructed for Rock Creek and successful preliminary runs have been made. Comparison of model output with historical data from the mouth of the Creek will be made as the first step in the calibration/validation process. A survey to explore farmer attitudes and decisions about BMPs is under development and will be carried out during the coming year.

## CEAP – Research and Extension Outcomes

- Knowledge of best management practices for agricultural producers
- Actions:  Agricultural producers adopt best management practices
- Conditions:  Reduced contamination of water and food supply by livestock pathogens

**Notes:**
“Science-based decision making and management practices that improve the quality of the Nation’s surface water and groundwater resources in agricultural, rural, and urbanizing watersheds”
Important findings are: 
Sediment concentrations and loads have generally declined over this period of time in the agricultural watersheds, in response to conservation tillage and other conservation practices in the watersheds. The relationship between sediment concentration and stream flow has changed systematically during this time, demonstrating that the reductions in sediment are due to management measures and not to weather. 
 Total phosphorus concentrations have generally declined during the study period, though the reductions are less dramatic. Particulate phosphorus concentration decreases are very similar to those for sediment, indicating that the phosphorus content of the sediment has not changed significantly during this time. 
Dissolved reactive phosphorus concentrations decreased dramatically until about 1995, but have increased equally dramatically since that time. This is an important but undesirable change that corresponds in time with increasing phosphorus concentrations in Lake Erie and accompanying problems of excess algal growth and hypolimnetic hypoxia in the Central Basin. 
We hypothesize that one cause of the observed DRP trends is nutrient stratification associated with conservation tillage, leading to high P concentrations in the surface horizon that can be readily leached during surface runoff. 
An AnnAGNPS model has been constructed for Rock Creek and successful preliminary runs have been made. Comparison of model output with historical data from the mouth of the Creek will be made as the first step in the calibration/validation process. A survey to explore farmer attitudes and decisions about BMPs is under development and will be carried out during the coming year.

## CEAP – Research and Extension Outcome Indicators 

**Knowledge**:  Feedback from seminars for producers (e.g., evaluations, surveys, etc.)
**Actions**:  Documentation of adoption of best practices
**Conditions**:  Fewer pathogens identified in surface and ground water, reduced number of contaminants found in water and food  supply

**Notes:**
“Science-based decision making and management practices that improve the quality of the Nation’s surface water and groundwater resources in agricultural, rural, and urbanizing watersheds”
... Comparison of model output with historical data from the mouth of the Creek will be made as the first step in the calibration/validation process. A survey to explore farmer attitudes and decisions about BMPs is under development and will be carried out during the coming year.

... Further use of this information? 
[XX Source – for this excerpts? Believe it was the report in CRIS on this project....]

## CEAP, Outcomes, Conditions

- Discovered increased dissolved phosphorus concentrations in Lake Erie tributaries, associated with conservation tillage -- enriching the surface layer of the soil in phosphorus, which then leaches out into the water.
- Ohio EPA convened a phosphorus task force to examine the extent and severity of the issue, and impact on Lake Erie.

## Water Quality Outcomes Conditions...  

- “... improve the quality of the Nation’s surface water and groundwater resources in agricultural, rural, and urbanizing watersheds.”

**National Integrated Water Quality Program, **
**FY 2007 Request for Applications**

## Cryptosporidium, Situation

- Cryptosporidium is one of the most common causes of waterborne disease within humans in the United States
- May be found in drinking water and recreational water in every region of the United States and throughout the world
- Parasite may be carried by agricultural animals, wildlife, and humans

**Notes:**
. In 1993, an estimated 403,000 residents of the greater Milwaukee, Wisconsin, area (population, approximately
1.61 million) became ill when an ineffective filtration process led to the inadequate removal of Cryptosporidium
oocysts in one of two municipal water treatment plants (6). We assessed direct medical costs and productivity losses from diarrheal illness during the Milwaukee outbreak to estimate the total cost of illness and the average cost per person with mild, moderate, and severe illness.

CDC investigators conducted a retrospective cost-of-illness analysis using data from 11 hospitals in the greater Milwaukee area and epidemiologic data collected during the outbreak. The total cost of outbreak-associated illness was $96.2 million: $31.7 million in medical costs and $64.6 million in productivity losses. 

Most treatment plants that take raw water from rivers, lakes, and reservoirs for public drinking water production use conventional filtration technologies. This involves a series of processes including coagulation, flocculation, sedimentation, and filtration. Direct filtration, which is typically used to treat water with low particulate levels, includes coagulation and filtration but not sedimentation. Other common filtration processes are slow sand, diatomaceous earth, membranes, and bag and cartridge filters. Conventional, direct, slow sand and diatomaceous earth technologies will remove 99% of Cryptosporidium[3]. Membranes and bag and cartridge filters remove Cryptosporidium on a product-specific basis. With the proper concentrations and contact time, Cryptosporidium inactivation will occur with chlorine dioxide and ozone treatment. Additionally, ultraviolet light treatment at relatively low doses will inactivate Cryptosporidium.

 If found, may require public health warnings – boiling drinking water, etc. 

Cryptosporidiosis is typically an acute short-term infection but can become severe and non-resolving in children and immunocompromised individuals such as AIDS patients. The parasite is transmitted by environmentally hardy cysts (oocysts) that, once ingested, excyst in the small intestine and result in an infection of intestinal epithelial tissue. The genome of Cryptosporidium parvum was sequenced in 2004 and was found to be unusual amongst Eukaryotes in that the mitochondria seem not to contain DNA 

## Cryptosporidium, Situation

- In 1993, an estimated 403,000 residents of Milwaukee, Wisconsin became ill with cryposporidiosis
- Ineffective filtration process led to the inadequate removal of *Cryptosporidium *oocysts from the municipal water supply
- CDC investigators, in a retrospective cost-of-illness analysis,  found the total cost of illness was $96.2 million: $31.7 million in medical costs and $64.6 million in productivity losses.
- EPA now requires screening for cryptosporidium.

## Cryptosporidium Activities 

**Research** – genome sequencing; transmission, survival and spread of oocysts;
- 	analysis of spread through water shed and into water supply.
**Extension** – methods to prevent transmission through agricultural animals into surface and ground water

## Cryptosporidium, Research Outputs    

- Developed a stochastic model of pathogen transport over the soil surface
- Developed a deterministic model including environmental parameters
- Compared modeled oocyst transport data to experiments
- Applied engineering risk assessment to expand model to analysis of an entire water supply consisting of multiple watersheds.

## Cryptosporidium, Output Indicators    

- Stochastic model itself
- Deterministic model, and its environmental parameters
- Oocyst transport data, as predicted by models and collected experimentally
- Risk assessments, parameters and data values  from multiple watersheds.

**Notes:**
Developed an accurate method for quantitative determination of the effects of agricultural and other watershed management practices and environmental conditions on transport of cryptosporidium pravum oocysts into water supply

## Cryptosporidium, Research Outcomes  

**Knowledge** – Provided information on how environmental factors in a watershed affect transport of oocysts before they reach a water supply system.
**Behavior **– Enabled water treatment managers to better determine risks of pollution in extreme hydrological events
**Conditions** – TBD, Prevention of treatment failures, Prevention of outbreaks of cryptosporidiosis.

**Notes:**

Guidelines for the management of biological contaminants in watersheds developed by city planners and environmental regulators
Resource provided to water treatment managers to better determine risk of pollution in extreme hydrological events. 

## Cryptosporidium, Research Outcome Indicators  

**Knowledge** – Parameters and data on  environmental factors in watershed affecting transport of oocysts, as provided to agricultural producers and water treatment managers
**Behavior** – Changes in best management practices by producers and water treatment managers
**Conditions** – TBD, Cryptosporidium identified in watersheds, water supply

**Notes:**
Guidelines for the management of biological contaminants in watersheds developed by city planners and environmental regulators
Resource provided to water treatment managers to better determine risk of pollution in extreme hydrological events. 

## Cryptosporidium, Outcomes,Extension 

**Knowledge **– Farmers and ranchers learn of best management practices (BMPs) for adequate set back distances and buffers to safeguard leafy greens from livestock pathogens and contaminated rangeland runoff
**Behavior **– Producers adopt BMPs, with cost-share programs as needed
**Condition **– Reduce pathogens in water supply

**Notes:**
The question of adequate set back distances and buffers to safeguard leafy greens from livestock pathogens and contaminated rangeland runoff is a central issue now being debated here in California. 

## Cryptosporidium, Outcomes,Extension Indicators

**Knowledge** – Feedback from seminars for producers
**Behavior** – Documented adoption of BMPs, participation in cost-share programs
**Condition** – Reductions in pathogens found in in water supply, cost savings in water treatment

**Notes:**
The question of adequate set back distances and buffers to safeguard leafy greens from livestock pathogens and contaminated rangeland runoff is a central issue now being debated here in California. 

## Obesity Trends* Among U.S. AdultsBRFSS, 1990, 1995, 2005

**1995**

**Obesity Trends* Among U.S. Adults****BRFSS,**** ****1990, 1995, 2005**

**(*BMI ********30, or about 30 lbs overweight for 5’4” person)**

**2005**

**1990**

## Nutrition and Obesity Situation

**Nutrition and Obesity Situation**

## Nutrition and Obesity 

- Between 1970-1980 & 2003-2004, overweight among adolescents (12-19 years) increased from 5% to 17%.

**Notes:**
NHANES shows  that between 1970-1980 & 2003-2004, overweight among adolescents (12-19 years)  increased from 5.0% to 17.4%.  

## Nutrition and Obesity Among Adolescents, Outputs  

- Completed 1st study to apply dietary pattern methodology to systematically evaluate the diets of adolescents.
- Presentation: National Nutrient Databank Conference, Wash DC 2003
- Publication: Journal of Nutrition, Feb 2007

**Notes:**
Completed 1st study to apply dietary pattern methodology to systematically evaluate the diets of adolescents. 

Presentation: National Nutrient Databank Conference, Wash DC 2003

Publication: Journal of Nutrition, Feb 2007

## Nutrition and Obesity 

**Outcomes, Knowledge**
- Four distinct dietary patterns were identified.  Healthy pattern -- high intake of fruit, vegetables, dairy, grains &  low intake of sweets and fried foods --was followed by 12% of White girls.
- Healthy pattern was associated with more favorable nutrient intakes &  a smaller waist circumference.
- None of the dietary patterns followed mitigated  adiposity (reduced body fat) among Black girls.

**Notes:**
4 distinct dietary patterns were identified.

A healthy pattern was characterized by a high intake of fruit, vegetables, dairy, grains without added fats, mixed dishes & soups, & a low intake of sweetened drinks, other sweets, fried foods, burgers & pizza. 

This pattern was associated with more favorable nutrient intakes &  a smaller waist circumference.

This pattern was followed by 12% of White girls.

Among Black girls, none of the dietary patterns were associated with mitigating adiposity 

## Nutrition and Obesity,Outcome Indicators 

- Four distinct dietary patterns were identified among adolescent girls.
- Healthy pattern -- high intake of fruit, vegetables, dairy, grains &  low intake of sweets and fried foods --was followed by 12% of White girls.
- Healthy pattern was associated with more favorable nutrient intakes &  a smaller waist circumference.
- Among Black girls, none of the dietary patterns were associated with mitigating adiposity (increased body fat).

**Notes:**
4 distinct dietary patterns were identified.

A healthy pattern was characterized by a high intake of fruit, vegetables, dairy, grains without added fats, mixed dishes & soups, & a low intake of sweetened drinks, other sweets, fried foods, burgers & pizza. 

This pattern was associated with more favorable nutrient intakes &  a smaller waist circumference.

This pattern was followed by 12% of White girls.

Among Black girls, none of the dietary patterns were associated with mitigating adiposity 

## Nutrition and Obesity, Outcomes 

- Food intake pattern consistent with guidance for healthy eating (like the USDA/HHS Dietary Guidelines for Americans) appears to help prevent overweight – *among those who followed it in a longitudinal study.*
*Only a minority of adolescent girls followed healthy food intake pattern.    *

**Notes:**
Based on this study, it can be concluded that a food intake pattern consistent with 
guidance for healthy eating like the USDA/HHS Dietary Guidelines for Americans
appears to help prevent overweight. However, this pattern was followed by only a minority of adolescent girls.

## Nutrition and Obesity, Outcomes, Behavior   

- Food intake pattern consistent with guidance for healthy eating (like the USDA/HHS Dietary Guidelines for Americans) appears to help prevent overweight – *among those who followed it in a longitudinal study.*
*Only a minority of adolescent girls followed healthy food intake pattern.**    *

## Nutrition and Obesity, Outcomes, Conditions    

- Results of this study can be used to set goals for effective obesity prevention strategies for adolescent girls.
- Reductions in adiposity reduce risks for heart disease, diabetes, etc. in later life.
- Reductions in health care costs associated with obesity.

**Notes:**
Results of this study can be used to set goals for effective obesity prevention strategies for adolescent girls. 
Results of this study can be used to set goals for effective obesity prevention strategies for adolescent girls. 

## Financial Security 

- Creating ability to meet future needs while keeping pace with day-to-day obligations.
- Preparing for retirement and potential long-term care costs
- Requiring planning, saving, and debt control.

**Notes:**
What is Financial Security in Later Life (FSLL)?
Financial security is the ability to meet future needs while keeping pace with day-to-day
obligations. Preparing for retirement and potential long-term care costs takes planning, saving,
and debt control. This cooperative extension initiative seeks to help people improve personal
finance behaviors leading to financial security in later life, enhance the capacity of local educators
and their partners to deliver effective programs, and increase economic vitality and quality of life
for families and communities. This educational initiative offers a toolkit of web-based and face-to face
programs designed to encourage participants to:
PLAN for retirement and potential long-term care costs (such as Planning for a Secure
Retirement, Financing Long Term Care: A Resource Center for Families, and Legally
Secure Your Financial Future),
ACT to save and invest (such as America Saves, offered in cooperation with the Consumer
Federation of America; Guidebook to Help Late Savers Prepare for Retirement, developed in
cooperation with the National Endowment for Financial Education®; and Investing for Your
Future), and
EVALUATE to assure actions are on track to achieve financial goals.

**Notes:**
Whats up (down) with the savings rate? Posted by John Irons at July 14, 1999 02:54 PM 
The personal savings rate in the US has been falling steadily over the past several years; and, recently, it has even dipped below zero. (See graphs). This overall long-term trend has been a bit worrisome to many economists, but it is the most recent part of the trend that leaves many of us scratching our heads as to why the rate has taken such a dramatic dive since it's local peak in 1992. 
Here's my list of possibilities.    
Possibilities: 
Credit cards have become more common. In econospeak the we say that the "liquidity constraints" are looser. The argument goes that 20 or 30 years ago people with little or no savings would have liked to have spent more, but couldn't go into debt. The ubiquitous credit card has made this easier to do today. The growth of personal debt seems to be the other side of the coin. 
Stock market is booming. With a booming stock market, those people lucky (or smart) enough to have put money into the market have seen their net worth rise at phenomenal rates. These people are thus wealthier and are spending a portion of this accumulated wealth. The measured savings rate compares savings to current income - missing this "wealth effect". (Savings out of total income - including the change in asset values - could remain the same, while measured savings would decline). 
People are expecting higher incomes in the future. The decision to save is a decision of how to allocate spending over time; therefore, what people expect to happen in the future will impact how they behave today. If you win the lottery today, chances are you will start spending before the check arrives in the mail. If people are expecting the economy to keep booming, and bring with it a higher paycheck or bonus at the end of the year, they won't necessarily wait to go out and splurge on that new DVD player today. Since their current income hasn't changed, and spending has increased, the savings rate will dip. 
People are expecting higher prices in the future. It's hard to watch any economic or financial market news these days without hearing worries about increasing inflation. If consumers are thinking that prices might start to increase soon, they'll go out and buy that Ford today while they think the prices are still low. (Creating expected inflation is what some are arguing Japan needs to do to get out of their slump; maybe they should follow our example.) This shift in consumption from the future to the present will lower savings rates (but will increase them down the road). 
People are expecting higher interest rates. If the prospect of higher inflation is giving Alan Greenspan the jitters, consumers may worry that interest rates may go up in the future. This would give people an incentive to purchase those big item purchases now in order to lock in low interest rates. Again, this would shift purchases from the future to the present. 
Simple demographics Grandma spends much more than she earns. As the population ages, the number of people who are actively spending down their wealth is increasing. This is a natural result of the demographic trends in the US. 
People are expecting a tax cut Washington is all a buzz with talk of how to spend the surplus. Tax cuts are at the top of the Republican agenda, and there doesn't seem to be much opposition (provided some of the money is put towards paying down the debt - i.e. "saving" social security.) Again, if people are expecting a gift from Washington next April, why not spend some of that now?   

## Financial Security Outputs 

- Toolkit of web-based and face-to-face programs;
- Guidebook to Help Late Savers Prepare for Retirement,
- Training of educators, and
- Development of eXtension program,
- 	http://www.extension.org/personal+finance

## Financial Security Outcome  Indicators, Behavior 

- 41% percent reported using one or more recommended financial management practices from an initiative program;
- 46% reported they developed plans to achieve retirement and/or future income goals; and
- 57% reported that they had increased their financial security.

## Financial Security Outcome Indicators, Knowledge  

- 89% of program participants increased their financial knowledge related to later life issues
- 64% planned to use recommended financial management practices
- 56% planned to manage their use of credit, reduce debt, and/or reduce household spending in light of their long-term goals for later life

**Notes:**
• 89% of program participants increased their financial knowledge related to later life issues
• 64% planned to use recommended financial management practices
• 56% planned to manage their use of credit, reduce debt, and/or reduce household
spending in light of their long-term goals for later life
• Of those who also were surveyed using additional follow-up survey techniques (33,580
individuals), 41% percent reported using one or more recommended financial
management practices from an initiative program; 46% reported they developed plans to
achieve retirement and/or future income goals; 57% reported that they had increased their
financial security; and 74% reported that the program was valuable to them.
The FSLL initiative has also had a direct economic impact on those who have completed initiative
programs. A group of 7,663 individuals who completed initiative-related programs reported
a total of $6,501,945 of annual financial impact (such as dollars saved, debt reduced, new
dollars invested). This is an average of $848 per person.

## Financial Security Outcome  Indicators, Conditions 

- 7,663 individuals reported a total of $6,501,945 in dollars saved, debt reduced, and/or new dollars invested--an average of $848 per person.

## Developing and  Using Performance      Indicators and Targets 	

**Notes:**
What?  2 X 2 table?  Which combinations are possible?  

## Slide 88

## Performance Measurement  Framework Cycle (revisited)   

**1**

**0**

**2**

**3**

**4**

**5**

**6**

**7**

**8**

**Specify**
**Measures**

**Establish**
**Data collection**
**Procedures**

**Develop**
**Information System**

**Measure**
**& Report **
**Performance **

**Performance **
**Measurement**
**Understanding**

**Formative**
**Evaluation**

**Summative**
**Evaluation**

**Program**
**Profile**
**Logic model**

**Review **
**  ****And**
**Modify**

**FEEDBACK**

**Notes:**
Ongoing performance measurement provides regular snapshots of the performance of a policy, program or initiative. Through this monitoring, it serves as a descriptive tool about how the policy, program or initiative is doing. 
It is important to realize that ongoing performance measurement does not address the issues of how an outcome was achieved or why a strategic outcome was or was not realized. Explanations of how or why outcomes were achieved comes from evaluation, which is discussed in greater detail in a later section of this guide.
Performance measurement information provides managers with knowledge they require in order to manage their policy, program or initiative on an ongoing basis. It can provide reassurance that outcomes are unfolding as expected, or can serve as an early warning that the planned results are not occurring (which could lead to a decision for additional research, such as through evaluation, to determine why). Performance measurement and evaluation are also inextricably connected - ongoing performance measurement could be considered to be part of evaluation. The ongoing performance monitoring information that has been regularly collected can be used in periodic evaluations (which focus more in-depth on explaining the outcomes achieved) of a policy, program or initiative.

## Critical Steps –Using Indicators to Improve

- Create a target performance level for each indicator
- Assign someone to that indicator
- Derive targets carefully
- Check indicators against targets periodically
- Anticipate shortfalls and correct them
- Shortfalls generally of two types –
- Design flaws – redesign process?
- Execution issues -- training, better equipment, etc.

## Performance Measure Example 

- Cumulative number of expanded commercially adaptable processes that convert biomass to fuels through the development of cost effective biochemical or thermochemical technologies, and used commercially

**Notes:**
Performance Monitoring Plan is a critical tool for planning, managing, and documenting how performance data is collected and used

## Performance Indicator Example  

- Long Term Outcome Measure -- Cumulative number of expanded commercially adaptable processes that convert biomass to fuels through the development of cost effective biochemical or thermochemical technologies, and used commercially

## Performance Indicator Criteria 

**Precision** – How well-defined is the indicator?
**Accuracy** – Actual indicators are only approximations of ideal measure, but how good an approximation?
**Simplicity** – Easily measured and interpreted?
**Cost**
**Robustness **(balance) – use multiple indicators, especially for potentially competing criteria

**Notes:**
Seven Deadly sins:  
Vanity – using measures that will make the program, staff, and especially managers look good
Provincialism – letting organizational boundaries and concerns dictate performance measures
Narcissism – measuring from the program’s point-of-view, rather than the public’s or customer’s point-of-view 
Laziness – assuming that one knows what is important to measure without giving the question adequate effort or thought... 
Pettiness – measuring only a small component of what matters, and forgetting the bigger picture.
Inanity (a/k/a perversity) – selecting measures with counter-productive results 
Frivolity – not being serious about measurement in the first place... 

Talk about perversity – 
Find a non-fast food example ... 

## Challenges and Trade-offs (1) 

- In the ***private*** sector, competing dimensions of performance to be captured by indicators –
- Cost
- Quality
- Speed

## Challenges and trade-offs (2) 

- In the ***public*** sector, competing dimensions of performance to be captured by indicators –
- Adequacy
- Cost
- Equity
- Efficiency
- Effectiveness

**Notes:**
Temped to put 
Relevance
Quality 
Performance 
What examples can people think of – tradeoffs?  

E.g., 
Efficiency in grants processing – 
Give fewer and larger grants 
But what is the relationship between that and knowledge development? 

Peer review vs. targeted giving?  
New NIH Road map to speed up scientific progress, adoption of innovation?

## Challenges and trade-offs (3)  

- For Federal programs, including CSREES, competing dimensions of performance to be captured by indicators –
- Relevance
- Quality
- Performance

**Notes:**
Are these competing criteria?  I think so ... 
Relevance is 

Size, efficiency, and processing time for competitive awards?  Not the best example?  

What about peer review vs. targeted giving?  
New NIH Road map to speed up scientific progress, adoption of innovation?

## Slide 97

| LOGIC MODEL 
ELEMENT  |  | Indicator | Data Source/
Method
Used  | Collected 
By ...  | Timing/Frequency of Measurement |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  | Ongoing Measure | Process 
Evaluation | Outcome
Evaluation  |
| Outputs | Output 1 |  |  |  |  |  |  |
|  | Output 2 |  |  |  |  |  |  |
|  | Output X
 |  |  |  |  |  |  |
| Outcomes | Knowledge Outcome 1 |  |  |  |  |  |  |
|  | Knowledge  Outcome 2 |  |  |  |  |  |  |
|  | Behavior Outcome 1
 |  |  |  |  |  |  |
|  | Behavior  Outcome 2
 |  |  |  |  |  |  |
|  | Condition  Outcome 1 |  |  |  |  |  |  |
|  | Condition  Outcome 2 |  |  |  |  |  |  |
|  | Condition  Outcome X |  |  |  |  |  |  |

**Notes:**
1)  Use multiple indicators for each output, outcome ... 
To avoid perverse consequences... 

2)  Consider using indicators to track 
Activities
Assumptions 
External Factors 
Why?  Examples?  

## Using and Valuing Measures

## Critical Steps – Using Performance Measures

- Using PM to enhance learning and improve performance
- Cycling between and among research, education, and extension
- Better use of limited funding
- Stimulating new and better questions, approaches to programs – and performance measurement and evaluation...

## Multiple Uses of PM and Indicators

- Communicating Success – ‘stories and statistics’
- Modeling
- Performance monitoring, including indicators and/or milestones
- Analysis and planning
- Evaluation (formative, definitely, and possibly summative)

## Rolling up results...  

**Strategic Objective**

**Intermediate **
**Result 1**

**Intermediate**
**Result 2**

**Sub IR 1.1**

**Sub IR 1.2**

**Activity B**

**Why?  So What?**

**How?**

**Assuming **
**What?**

**What Else?**

**What Else?**

**Activity A**

**Activity C**

**Notes:**
The relationship between results should not be categorical, definitional, or chronological.  In other words, lower level results should not merely describe component parts of a related “higher” level result

Critical to look at assumptions within and across levels... 

## Cryptosporidium example...   

**Improve water quality **

**Farmers adopt BMP**

**Improved ability **
**to detect and treat **

**Sub IR 1.1**

**Sub IR 1.2**

** ****Green payments **

**Why?  So What?**

**How?**

**Assuming **
**What?**

**What Else?**

**What Else?**

** ****Farm demo **

** ****Improved models**

**Notes:**
The relationship between results should not be categorical, definitional, or chronological.  In other words, lower level results should not merely describe component parts of a related “higher” level result

Critical to look at assumptions within and across levels... 

## Wetland Restoration for Sediment Yield Reduction:A Conceptual Model

**Benefit of Wetland Restoration for Sediment Yield Reduction**

- Marginal Change in Total Downstream Sediment Yield per Restoration Dollar

**Marginal Change in Restored Wetland Area per Restoration Dollar**

**Marginal Change in Hydrologic Response per Restored Wetland Area**

**Marginal Change in Sediment Yield per Change in Hydrologic Response**

- Key

- Concept

- Vellidis et al., 2003. Prioritizing wetland restoration for sediment yield reduction: A conceptual model. Environmental Management 31(2): 301-312.

## Wetland Restoration for Sediment Yield Reduction:A Conceptual Model

**Benefit of Wetland Restoration for Sediment Yield Reduction**

- Marginal Change in Total Downstream Sediment Yield per Restoration Dollar

**Marginal Change in Restored Wetland Area per Restoration Dollar**

**Community Willingness**

**Purchasing Efficiency**

**Restorability**

**Presence of: Watershed Protection Activities; Conservation Programs**
***Watershed & environmental protection groups; conservation easements; protected areas***** **

**Restorability**
***Wetland characteristics; disturbance***** **

**Property Value**
***Land Cost***

- Key

- Concept

- Descriptor

**Indicator**
***Measurement Point***

## Wetland Restoration for Sediment Yield Reduction:A Conceptual Model

**Benefit of Wetland Restoration for Sediment Yield Reduction**

- Marginal Change in Total Downstream Sediment Yield per Restoration Dollar

**Marginal Change in Hydrologic Response Wetland Area**

**Headwater Response**

**Floodplain Response**

**Watershed Yield; Runoff Delivery; Runoff Interception**
***Precipitation; topography, permeability; hydrologic connectivity; flow friction***

**Upstream Velocity; Flood Frequency; Interception**
***Stream order; return period; structural modifications; hydrologic connectivity***

- Key

- Concept

- Descriptor

**Indicator**
***Measurement Point***

## Wetland Restoration for Sediment Yield Reduction:A Conceptual Model

**Benefit of Wetland Restoration for Sediment Yield Reduction**

- Marginal Change in Total Downstream Sediment Yield per Restoration Dollar

**Marginal Change in Hydrologic Response Wetland Area**

**Headwater Sediment Sources**

**Floodplain Sediment Sources**

**Upland Runoff; Unpaved Road Runoff; Development Site Runoff**
***Land use; RUSLE; unpaved road density; conversion rate***

**Channel Sources; Upland Runoff; Unpaved Road Runoff; Development Site Runoff**
***Total flow; bank materials; channel slope; contribution watershed area; sinuosity; land use; RUSLE; unpaved road density; conversion rate***

- Key

- Concept

- Descriptor

**Indicator**
***Measurement Point***

## Summary 

## “Doing it right...”

- Develop a logical/causal framework – model
- Gather data
- Turn data into information – use it for analysis
- Continually refine the model
- Base actions on findings
- Assess outcomes

## Back to Accountability...  

**“... ****The old saw that one cannot be held responsible for what one does not fully control has outlived its usefulness; it is both reasonable and necessary to hold everyone with some influence over a metric [measure] jointly accountable for it.”**

## Next Steps? 

- Questions?
- Where do we go from here?
- Increased communication...
- Increased synthesis...

## Contact Information 

**Sharon Stout**, Social Scientist, OPA, CSREES   202-401-0183, sstout@csrees.usda.gov_ _
**Kitty Cardwell**, kcardwell@csrees.usda.gov
**Mike O’Neill**, moneill@csrees.usda.gov
**Mary Ann Rozum**, mrozum@csrees.usda.gov
**Etta Saltos**, esaltos@csrees.usda.gov
**Jane Schuchardt**, jschuchardt@csrees.usda.gov
**Office of Planning and Accountability Website**
- http://www.csrees.usda.gov/about/strat_plan.html