# Presentation: 793054

## Developing Consensus on Common Models and Standard Set of Indicators: FCS and 4-H Case Studies

**2007 CSREES ADMINISTRATIVE Officers Conference **
**Seattle, WA**

**Mary McPhail Gray**
**Deputy Administrator**
**Families, 4-H and Nutrition**
**800 9th Street, SW**
**Washington, DC  20024**
**202-720-2908**
_**[mgray@csrees.usda.gov](mailto:mgray@csrees.usda.gov)**_

## CES Human Resources are Extensive 

- In 2005 U.S. CES staffing totaled 14,650 (FTE)
- Of which:
- 8,925 county agents/advisors/educators
-     (B. Sc. or M. Sc.; in 3,038 counties)
- 4,050 specialists
-     (Ph.D. faculty; many with joint appointments)
- 715 directors/supervisors
- 960 administrative support

## Federal Partner Interests (Goals)

- Impact Reporting to Administration (White House) and Congress
- Clarity of Program Marketing/Justification
- Identification of New Public and Private Funding Sources

## New Plan of Work Reporting 

- Bart Hewitt OPA summarized the wealth of information being reported in new electronic Plan of Work system
- 85 2007 Plans of Work received, 50 were combined submissions and 35 were single entity
- 1018 Planned Programs, with from 1 to 76 programs reported per plan (with median of 9 planned programs)

## Knowledge Areas Reported by Planned Programs 

- For 2007 Plans of Work, each program included up to 10 knowledge areas (KAs) and outcomes were ***not linked to specific KAs ***
- For 2008 Plans of Work, each program may include up to 20 knowledge areas
- Beginning with the 2008 Plans of Work, outcomes are linked to specific KAs, making it easier to drill down and locate specific outcomes

## Top Ten Program Themes by Frequency in 2007 POW 

- 1. Youth Development and 4-H
- 2. Rural and community Development
- 3. Natural Resources and Environment
- 4. Health
- 5. Plant Production
- 6. Animal Systems
- 7. Nutrition
- 8. Agricultural Systems General
- 9. Sustainable Agriculture
- 10. Food Safety

## Many programs, many outcomes

- Wealth of data -- Plans of Work and Annual Reports, Planned Programs, KAs
- Wealth of outcomes associated with plans, programs, KAs
- Can outcomes be associated with meaningful and measurable indicators?
- How many different indicators will we have?
- Who will combine them, how, and when?

## 2007 Top KAs in 4-H or FCS  By Full-Time Equivalent Staff

- 802, Human Development and Family Well-Being, 4.2%
- 806, Youth Development, 10.5%
- 703, Nutrition Education and Behavior, 3.4%
- 801, Individual and Family Resource Management, 2.7%

## Benefits of Consensus Indicators (from Silliman, 2007 review)

- Monitoring programs consistently
- Mobilizing public opinion
- Tracking goal achievement
- Focusing accountability
- Informing program improvements
- Standardizing program evaluation

## 4-H and FCS Program Impact Reporting Challenges

- Breadth of program activity
- Job stress of State and County Faculty
- Identification with constituents for rewards
- Lack of evaluation resources (training, time, budget, commitment)
- Persuasion to use the Logic Model

## University of Wisconsin Logic Model

- _[http://](http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html)__[www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html](http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html)_

## Federal Impact Reporting Response

**Persuasion and Motivation** regarding goals
- Deductive (national) identification of desired program outcomes
- Inductive identification of state program outcome statements

## Federal/LGU System Process of Building Consensus

- An FCS case
- A 4-H case

## Indicators for 4-H Youth Development Program 

- State 4-H Program Leaders identified need for quality and accountability indicators at 2006 annual meeting
- National Program Leader and representative from National 4-H Council charged with forming a committee of State 4-H Program Leaders to develop indicators
- National 4-H Council committed private funds to support the committee’s efforts
- Review of the literature was commissioned and completed by Dr. Ben Silliman, NC State

**Notes:**
Ben focused his review on the 3 mission mandate areas and developed recommendations for youth outcome level indicators in each of those 3 areas as well as indicators of program quality

Ben used the following definitions:

Outcomes—Results for youth, families, communities
Short-term: Knowledge, Attitudes, Skills, Aspirations (KASA)
Long-term: Behavior Change, Application

Indicators--Measurable traits tracked over time, participants, geography (Performance Criteria)

Targets--Degree of change expected

## 4-H National Mission Mandates

**Science, Engineering, and Technology (SET)**: agriculture and life sciences, family resource management, human development and family
**Youth as Citizens (YAC)**: civic involvement, community service, leadership in youth and community organizations
**Healthy Living (HL)**: nutrition and physical activity, safety, mental health

## Most Frequent 4-H Programs by Theme

- Youth Development (General)
- Science, Engineering, Technology (SET)
- Youth Citizenship (YC)
- Healthy Living (HL)
- Other (OT)
- Ranked in order of frequency, except ‘Other’

## Examples of 4-H Outcomes, Youth Development 

- Youth become caring and contributing members of society through positive experiences in a diverse 4-H Youth Development program
-  Youth demonstrate mastery and competencies needed to become engaged citizens by
- 	-- assuming leadership positions in communities
- 	-- developing and implementing action plans to 	address community needs
- 	-- becoming productive members of the work force

## Examples of 4-H Outcomes, SET Program

- Knowledge:  Young people will understand role of science, engineering, technology in the contemporary world
- Behavior:  ... will develop skills and abilities for success in education and careers that rely on application of science and technology
- Condition:  4-H program is contributing to the long-term economic and social stability of a technologically advanced workforce

## Examples of 4-H Outcomes, Youth Citizenship  

- More youth are involved in leadership roles in their communities
- Youth governmental board developed to influence legislations affecting youth
- Youth are involved in community organizations as partners

## Examples of 4-H Outcomes, Healthy Living   

- Improve the lives of citizens through positive human development, healthy lifestyles and nutrition, and lifelong wellness.
- Increase the number of youth who can maintain a healthy lifestyle – learning how to select a balanced, nutritious diet
-  ... developing skills to prepare food themselves
- ... developing skills and positive attitude to make lifelong physical activity a lifelong habit

## Science, Engineering, & Technology Research (Silliman, 2007)

- National Science Standards: Thinking, not just memorizing facts
- 4-H SET Abilities needed for academic, career success:
**Science Anchors** (content: biology, chemistry, geology, physics, etc.)
**Science Abilities** (processes: observation, hypothesizing, experimentation, analysis, etc.)

## Youth-as-Citizens Research (Silliman, 2007)

**Civics**: Impact/engagement depends on knowledge of how government and organizations work
**Service**: Positive attitudes, experiences in service increase adult service; increased academic, life skills; reduced risk behaviors
**Leadership**: Engagement increases leadership and service activities (more as duration and responsibility increase)

## Healthy Living Research (Silliman, 2007)

**Nutrition/Dietary Habits**: Unhealthy dietary habits and limited activity placing youth at risk into the future
**Safety**: Unintended injuries declining but practical activities require training
**Mental Health**: Stress management critical to varied physical and emotional health outcomes

## Example: 4-H SET Outcomes

- Short-Term Outcome Indicators
- Number increasing knowledge/skills
- Number improving attitude/aspirations
- Number increasing science process skills (observation, comparison, hypothesis, analysis, problem solving)
- Long-Term Outcome Indicators
- Number demonstrating changed behavior in science learning (e.g., school success, pursue additional learning)
- Number applying science process skills (advanced project, mentoring others, young adult education and career choices)

## Example: 4-H SET Program Quality Indicators

- Essential Elements of Inquiry: 1) asking questions; 2) framing hypotheses; 3) gathering and interpreting evidence; 4) revising explanations to fit data
- Effective environment: Learner-centered... Knowledge-centered...Assessment-centered...Community-centered
**Science-by-Inquiry**: guided content and process explanations
**Science-as-Inquiry**: shared discovery and reflection

## Next Steps for 4-H Indicators Work

- Evaluating for Impact Committee co-chaired by Dr. Suzanne Le Menestrel and Dr. Mary Arnold, Oregon State U., will use commissioned paper to recommend indicators in each of the three mission mandate areas
- Evaluating for Impact Committee will develop recommendations for measuring indicators
- Recommendations will be reviewed by original committee of State 4-H Program Leaders

## FCS Impact Reporting Strategic Planning

- 1999 Annual Meeting of FCS State Leaders identified national program impacts as priority
- Committee of State Leaders identified six major program themes supported by 1/3 or more of states by reviewing 2000-2004 POW’s
- Six committees of state specialists led by Program Leaders built consensus for program outcomes known from research literature

## FCS Foundational work to build upon

- Southern Regional work on Outcome Evaluation
- National Extension Cares childcare initiative
- National Extension Parenting Education Model
- Healthy People-Healthy Communities national initiative
- Financial Security Specialists’ decade of work
- CSREES Reporting System, eXtension and One Solution

## FCS Named Program Themes

- Childcare and Afterschool Programs
- Community Nutrition and Food Safety
- Financial Security
- Health
- Housing and Environment
- Parenting

## FCS Program Outcome Examples: Childcare

- Childcare personnel increase their knowledge of developmentally appropriate guidance
- Childcare personnel show an improvement in practice after CE education
- Parents demonstrate greater knowledge in selecting appropriate child care for their family

## FCS Program Outcome Indicators: Financial Security

- Program participants increase their ability to achieve financial self-sufficiency and stability through increasing savings, reducing debt, setting goals, etc.
- Program participants increase their ability to achieve financial security through regular savings, retirement planning, establishing record systems, managing health care costs, etc.

## FCS Program Outcome Indicators:  Community Nutrition, Food Safety and Security

- Program attendees acquire the knowledge to change their diet toward the recommendations of the Dietary Guidelines.
- Program attendees learn to avoid cross contamination in food preparation
- Program attendees make a shopping list, compare prices, use sales fliers, etc.

## Next Steps in Building Consensus

- Leadership from 4-H Evaluating for Impact Committee
- Leadership from FCS State Leaders
- Leadership from F,4-H & N and Planning and Accountability Units at CSREES

## How Can you Help?

- What is the public value of our work in youth and in family?
- How can you evaluate: the increased access to meaningful information, the increased equity or justice achieved, and the short or long term savings accrued?
- Everyone needs an “elevator speech!”

## Thank You for Your Interest and Commitment!

-         Mary McPhail Gray, Ph.D.
-         _[mgray@csrees.usda.gov](mailto:mgray@csrees.usda.gov)_
-         202-720-2908