# Presentation: 943751

## Some Practical Tips for Measuring Financial Success Dr. Angela LyonsUniversity of Illinois

**Evaluating Financial Education **
**in Troubled Times**
**Dr. Lance Palmer**
**University of Georgia**
**Dr. Angela Lyons**
**University of Illinois – Urbana Champaign **
**AFCPE Extension Pre-Conference**
**November 2008**

## Difficulty paying debts 
Low saving rates
Rising home foreclosures
Mounting bankruptcies
Higher unemployment
An increased awareness and impact of natural disasters 

- Difficulty paying debts
- Low saving rates
- Rising home foreclosures
- Mounting bankruptcies
- Higher unemployment
- An increased awareness and impact of natural disasters

**Consumers are in financial trouble!!!**

- Motivation:Recent Financial  Challenges

**Notes:**
Look up current debt levels – is it soaring debt levels or an inability to refinance. 
Low savings rates.
Rising home foreclosures.
Increasing bankruptcies.
Rising unemployment.
Increasing awareness and occurrences of natural disasters. 

## The Result....

- A number of financial education efforts have been developed to address consumers’ financial needs in these troubled times.
- 3 key programming areas:
- Bankruptcy
- Homeownership issues
- Natural Disasters

**Notes:**
Look up natural disaster programming and give some specific examples.

## 
    At the end of the day,             are these financial education    efforts making a difference?

**    ****At the end of the day,             are these financial education    efforts making a difference?**

**Notes:**
Well we know they are making a difference because we see the difference they are making.  We all know which programs were abismall, because we experienced it first hand. But how do others know that our programs are making a difference? How do people in far away places determining the funding for our programs know that our program helped people? How do we objectively correcting or discontinue that program that is or has been a failure. 

## On the front lines with the evaluation experts....

- 	“What do we mean by evaluation?
- 	“What are we really trying to measure?”
- 	“How do we define program success?”
- 	“How do we know if participants are *improving*?”
- 	“What financial outcomes and indicators should we be using?”
- 	“What constitutes a *successful*, or even *acceptable*, evaluation?”

## 1.	Identifying the “ideal” approach to evaluation.

	Evaluation methods and measures vary widely across programs and academic disciplines.

	Wide variation in financial outcomes across programs.

	Significant differences in financial needs across 
	individuals and families. 

	Some participants unable to 
	implement certain financial behaviors.

	Shift towards more one-on-one individualized education.

- 	Evaluation methods and measures vary widely across programs and academic disciplines.
- 	Wide variation in financial outcomes across programs.
- 	Significant differences in financial needs across
- 	individuals and families.
- 	Some participants unable to
- 	implement certain financial behaviors.
- 	Shift towards more one-on-one individualized education.

- “The Big Picture” 5 Biggest Evaluation Challenges in General

**Notes:**
Just as programs vary substantially, evaluation methods SHOULD also vary substantially across programs. Some programs may be best suited for traditional pre / post / and follow-up evaluation methods. Other programs may need more exploratory methods such as focus groups, interviews, and open-ended questions. Furthermore, some evaluation will be focused on behavior change whereas other programs will focus on knowledge and attitude changes and some programs may only seek to assess tangible outcomes. Outcomes will be based on our objectives for our program, and there are many different objectives depending on the type of audience our program is reaching out to.  While I would guess that the majority of our programs are targeting low to moderate income, there are many different aspects of personal finance that may or may not need to be addressed with these audiences.  The heterogeniety of individuals in a similar income group can be difficult in designing programs and evaluating programs because everyone is not starting at the same point. 
Furthermore, given certain family situations or legal circumstances such as bankruptcy, some individuals may truly want to save more, but short term finances would deem otherwise. However, non-financial behavior may be changed that will lead to financial changes down the road.

Furthermore, some education is becoming more counseling rather than “class-room” style education. As a result of this, the consistency of education and programing may be disappated. However, the opportunity to document outcomes and impact is likely greater. One-on-one counseling and also coaching can provide opportunities and pitfalls in conducting evaluations.

Speaking of outcomes, there are a wide variety of financial outcomes that programs desire, and some programs don’t have any immediate financial outcomes, their purpose is to provide knowledge and education. 

## 2.	Defining “program success.”

	Setting realistic expectations for program participants.

	Choosing appropriate outcomes and indicators based on participants’ financial situation or other external constraints.

	Identifying participants’ individual financial desires and applying appropriate educational interventions and evaluation.

	Finding “the teachable moment” or motivated learner.

	Recognizing that “one size does not fit all.”

- 	Setting realistic expectations for program participants.
- 	Choosing appropriate outcomes and indicators based on participants’ financial situation or other external constraints.
- 	Identifying participants’ individual financial desires and applying appropriate educational interventions and evaluation.
- 	Finding “the teachable moment” or motivated learner.
- 	Recognizing that “one size does not fit all.”

**Notes:**
Sometime evaluations fall short of expectations because the expectations were simply set too high. The best evaluations are those that are designed around the most clearly stated and measurable program objectives.  The desired outcomes associated with the objectives need to be realistic. In one a one hour group presentation it will be difficult to affect someones socioeconomic status within a six month period. More realistic, some welfare programs have asset and income restrictions. Depending on the clients situation, there may be a real disinsentive to save, and objective measures of savings may be fail to document program success, even though the participant significantly altered their spending behavior in anticipation of saving at some future point (prepaying bills, paying down debt, stocking up on food or other items).  Also, savings as we describe them, bank account or highly liquid and marketable securities, may be a tertiary savings methods. Instead groups that we work with may have a primary savings method the purchase of objects, sometimes having lasting value, sometimes not.  Furthermore, some programs are mandatory versus voluntary. In a voluntary program, it can be reasonably assumed that all participants are there because they are motivated to be there, that is good. In a mandatory program, such assumptions are just not true. A significant portion of the program may need to focus on motivating the client and raising their awarenss of negative behaviors. Similarly, some adults will come to the program looking for help and solutions, such as in foreclosure and natural disaster recovery, these may or may not be teachable moments for these individuals because of the extreme stress and need for immediate resolution or protection.  At other times they may be teachable moments you will know your audience better than anyone and in anticipating their needs, you will also be able to anticipate the appropriate objectives of the programs and inturn utilze the best indicators of program success. 

## 3.	Collecting impact data from                    program participants.

	Little incentive to complete evaluations (like “pulling teeth”).
	Reluctance to divulge personal information (surveys “too personal”; lack of trust).
	High drop out rates, low response rates, and difficult to track.
	Literacy levels (i.e., ESL, reading level).
	Collecting sensitive data and information.
	Tradeoff between participation and evaluation rigor.

- 	Little incentive to complete evaluations (like “pulling teeth”).
- 	Reluctance to divulge personal information (surveys “too personal”; lack of trust).
- 	High drop out rates, low response rates, and difficult to track.
- 	Literacy levels (i.e., ESL, reading level).
- 	Collecting sensitive data and information.
- 	Tradeoff between participation and evaluation rigor.

**Notes:**
Another major challenge of program evaluation is getting the COMLETED evaluations back. I was in a very small session recently and a participant was an Asian Female.  She looked around the room and said “I am not going to complete this survey because you will know exactly who I am.” When asking personal information we need to make sure that it is anonymous in word and deed. Pre tests are especially difficult since many individuals don’t know you and have limited trust in you and they are already being asked to divulge personal information. Literacy levels also present a challenge. Evaluation take time away from programming.  Generally the more intensive the evaluation the more time it takes. Also, low income households can be very transient until they settle down making followup evaluations difficult, often email addresses may remain more stable than other locater information.    

## 4.	Designing and implementing effective   program evaluations.

	Evaluation process is cumbersome.

	Lack of time, staff, and financial resources.

	The “PUSH” for increased rigor and “the rush to the finish line.”

	Rigor vs. Reality (e.g., measurement issues)

	The limitations of “one-shot” evaluations.
		(pre- and post-tests; intended vs. actual behavior change)

	The reality of conducting longitudinal studies with control groups. 
		(follow-ups and tracking of program participants)

- 	Evaluation process is cumbersome.
- 	Lack of time, staff, and financial resources.
- 	The “PUSH” for increased rigor and “the rush to the finish line.”
- 	Rigor vs. Reality (e.g., measurement issues)
- 	The limitations of “one-shot” evaluations.
- 		(pre- and post-tests; intended vs. actual behavior change)
- 	The reality of conducting longitudinal studies with control groups.
- 		(follow-ups and tracking of program participants)

**Notes:**
There are also many challenges specific to evaluations. Can you have a rigorous evaluation program as part in emergency response, is it appropriate? What is the purpose of the response, what is an appropriate evaluation? Reliability of attitudinal or behavioral scales is important, but the reality is that most programs do not have the funding to test their instruments with a large sample before using their evaluation instrument. Response shift bias, is also a problem when doing evaluations. The timing of when the evaluation is given is important. 

## 5.	Data analysis and measurement issues

- 	Measurement error and validity of indicators.
- 	Self-reports are subject to bias.
- Social desirability
- Norms and “rules of thumb”
- Misperceptions and over-optimism
- Memory distortion and recall bias
- 	Samples may not be representative.
- Non-response bias
- Program attrition
- Self-selection
- Low response rates (e.g., follow-ups)

**Notes:**
Measurement error
Low response rate
Environmental factors
Psychological factors (high stress and impact)

## Other measurement issues

- Unexpected life events
- Program incentives (e.g., rewards, special benefits, enrollment programs)
- Individualized financial advice or “coaching”
- Psychological factors.
- Inherent motivation
- Ability
- Attitudes
- Stress

**Notes:**
One good example of this is that recently we completed a program for two departments on our campus, one was the Athletic department and the other was our home college.  We had about 5 to 8 people in the room for our own college, but for the Athletic department we had 20 to 30 people in the room.  One key difference between the two programs was that the Athletic department ordered a box lunch for everyone.  The act of coaching may be very effecitve, but sometimes coaches want to get involved in the game also.  This will blur the effectiveness of the program and bias results in favor of programatic success, however the individual has not learned anything.  Some may say that is great, the person is better off. Maybe so, but just make sure your program indicators are not being biased by the “help.” 
Inherent motivation is also a key issue when talking about mandatory versus voluntary effects. 

## Bankruptcy Counseling and Education

- Educational impacts are difficult to disentangle from the actual bankruptcy process.
- Difficult for debtors to change certain behaviors when haven’t yet discharged debts and still under court supervision.
- Conducting a follow-up is critical.
- Dealing with a very heterogeneous population that is _***required***_ to complete the counseling and education.

## Bankruptcy Counseling and Education Cont’d

- Selecting indicators that are independent of the debtors’ current financial situation and the bankruptcy discharge.
- Constraints associated with curriculum content and timing of delivery.

**Notes:**
Behavior change is difficult in discharge process. Confounding effect of process and education, but both can lead to reduced education.
Counseling is first, education effect may be reduced because of the preliminary counseling. 
Select indicators that our independent of the discharge. 
Read Angela’s document. Reductions in debt is not a good indicators. 
Providers are dictated to by the trustees department regarding what should be covered and the timing of that coverage. The topics are too basic at times and different programs would be better but the flexibility is not there. 

## Homeownership programs

- Transitional housing programs
- Homeownership education
- Post purchase counseling
- Default and delinquency counseling
- Reverse mortgage counseling
- Less emphasis on group education and more focus on one-on-one models of financial counseling or “financial coaching.”
- Significant grant dollars are available to conduct these programs, but evaluations can be constrained by funders’ requirements.

**Notes:**
Follow-up on the post counseling. 
Tracking them down – must have a good relationship with individual or lender (USDA loans). 
HUD requires substantial feedback. 
Foreclosure counseling (figuring out what the problem is). 
Do you use their address or emails or phone numbers. 
The people stay closely with the people who have the money. Use these people to contact the participants. Email is stable.
Did they stay in their home. Did they go to another organization. Continuing to work with some timeline to purchase. Bringing bills current.
HUD ‘s indicators
Client data is reported online through a houseing counseling system. Online systems will begin talking with each other. Providers include Freddie Mac, Fannie Mae, other groups will have their own systems as well.  Standardizing information gathering.  HUD is very demanding on reporting information and logic models.

Topics are required. Lots of grant dollars available for these programs, but the grantor may have specific data collection requirements regarding evaluation. 

## Post-Natural Disaster Programs

- Informational and financial needs may vary widely across disaster victims.
- Psychological factors are likely to significantly affect motivation, attitudes, and perceived abilities which will influence evaluation responses.
- Disaster victims can be very transient – difficult to track and conduct follow-ups.
- Very time and resource intensive.

**Notes:**
Katrina—post disaster programing evaluation. 
Qualitative methods would be very helpful because they are open.  

Specificity of needs and outcomes is key—differs dramatically.  

Is this a teachable moment, or a service moment.  

How will you track these people? Very time and resource intensive. 

## Overcoming the ChallengesPlanning, planning, and more planning....

- Take stock of who you are – What is “your vision?”
- Be strategic – What are you trying to accomplish?
- Conduct a thorough needs assessment.
- Spend time understanding your target audience(s).
- Define program success for your audience. Be realistic!
***Create an evaluation action plan***
- Identify resources and overcome constraints.
- Take Action!

**Notes:**
Lots of upfront planning.  Evaluation is part of the overall programing efforts.  How will this evaluation be included in the program delivery. 
“There is a demand so let’s create the resources” at the same time build in the evaluation. 

## 4 Steps to Creating An Effective “Evaluation Action Plan”

**Step 1:  Define objectives of the program.**
- 	At the end of the day, what do you want to show?
- 	Who will be the target audience?
- 	What will be the primary delivery method?
- 	Who will use the evaluation and how?
**Step 2:  Select appropriate methods for data collection.**
- 	What is the most appropriate evaluation format?
- 	What types of questions will the evaluation seek to answer?
- 	What types of indicators will be used to show impact?
- 	Are these indicators appropriate for the target audience?

**Notes:**
These are the steps to the evaluation action plan. 

## 
Step 3:  Identify and overcome evaluation challenges.

	What are your biggest implementation challenges?
	What steps can you take to overcome the challenges?
	What financial and non-financial resources are available?
	Are there others who can help?
	What is the program timeline?                                      
	Given constraints, what can you realistically do? 

Step 4:  Analyze and disseminate findings.

	How will data be analyzed?
	What do you hope to learn from findings?
	What are the potential impacts?
	How will the results be used and disseminated?

**Step 3:  Identify and overcome evaluation challenges.**
- 	What are your biggest implementation challenges?
- 	What steps can you take to overcome the challenges?
- 	What financial and non-financial resources are available?
- 	Are there others who can help?
- 	What is the program timeline?
- 	Given constraints, what can you realistically do?
**Step 4:  Analyze and disseminate findings****.**
- 	How will data be analyzed?
- 	What do you hope to learn from findings?
- 	What are the potential impacts?
- 	How will the results be used and disseminated?

## Where do we go from here?Online resources at your fingertips

## NEFE Financial Education Evaluation Toolkit ® http://www2.nefe.org/eval/intro.html 

**NEFE Financial Education Evaluation Toolkit**** ****®**** ****http://www2.nefe.org/eval/intro.html**** **

## NEFE Financial Education Evaluation Toolkit® 

- Database
- Post evaluation only with option for follow-up
- Pre and post evaluation with option for follow-up
- Stages to Change Evaluation
- Train-the-Trainer
- Testing Knowledge
- Building Skills
- Taking Charge
- Manual
- How-to-guide for grass-roots level organizations
- Examples (survey instruments, executive summary, reports)
- Guidance on how to organize and present impact data

- Manual
- http://www2.nefe.org/eval/manual.html
- Part I: Financial Education Overview
- Part II: Understanding Program Evaluation
- Part III: The Evaluation Planning Process
- Part IV: Using the Evaluation Database
- Part V: Reporting Program Impact

- Appendix:
- Sample Evaluation Instruments

- Database
- http://www2.nefe.org/eval/index.php

- Knowledge  Indicators

- Behavior Indicators

- Qualitative Info

- Demographics

- University of Wisconsin-Extension
- http://www.uwex.edu/ces/pdande/evaluation/index.html

- Cornell University Extension
- http://staff.cce.cornell.edu/administration/program/evaluation/evalrefs.htm

- Penn State Extension
- http://www.extension.psu.edu/evaluation/

## Contact Information

- Dr. Lance Palmer
- Assistant Professor
- Department of Housing and Consumer Economics
- University of Georgia
- Phone:  (706) 542-4916
- E-mail:  _lpalmer@uga.edu_

## Questions