# Presentation: 122333

## Session 3 – Training Program Evaluation

- Eric Makela

## Program Evaluation

**Add Peach Pie to the Menu**

## Training Program Evaluation 

- Based on Donald Kirkpatrick’s “Evaluating Training Programs:  The Four Levels”
- Reaction
- Learning
- Behavior
- Results
- Each of the levels answers a different question and builds upon the preceding level

## Reaction

- Qualitative Measurements of Customer Satisfaction to the Training
- Customer believes/or comes to believe the training is important
- Customer will return for future trainings
- Customer will recommend training to others

**Notes:**
Provides valuable feedback and helps to evaluate the program
Tells trainees that the trainers are there to help them do a better job and that they need feedback to determine how effective they are.

## Reaction

- Quantitative Measurements of Customer Satisfaction to the Training
- Provide input to trainee’s managers, DOE, etc.
- Provide feedback to trainers
- Establish standards for performance for future programs
- Businesses won’t call the restaurant for dessert catering if the pie is bad and will not recommend the restaurant to others, Mother Makela wants to improve quality of her pie

**Notes:**
Provides valuable feedback and helps to evaluate the program
Tells trainees that the trainers are there to help them do a better job and that they need feedback to determine how effective they are.

## Reaction

- End of Class Evaluations – Qualitative and Quantitative
- “Smile Forms” or “Happiness Sheets”
- Examples of information that can be collected during evaluation
- Assess quality of training session
- How did the instructor do?
- How are the handout materials?
- How was the overall quality of the training session?
- Did the information that you received meet your expectations?
- Future planning
- What additional courses would you like to see?
- What additional information would you like to have?
- Can be asked on paper or at end of training session as part of the discussion

## Reaction

- Question Format
- Rating System - Quantitative
- Rate the training based on a scale of 1 to 5
- Answers will vary based on participants desire to leave the training site or stay
- Open ended questions - Qualitative
- Force the participant to answer the question and not just circle a number.
- Typically more thought put into the completion of the evaluation

## SampleReaction Form

## Learning

- Used to evaluate increase in knowledge or change in attitude
- Did the participants learn anything from the training?
- Was their attitude toward energy conservation more favorable when they left the training
- Example:  Energy conservation is important
- 			agree   disagree
- Example:  It’s Ok to serve berry pie in the winter
- 			agree	   disagree

## Learning

- Used to evaluate increase in knowledge
- Standardized test administered at beginning of training with same test given at the end of training
- Difference measures the increase in learning

## Learning

- Skills Evaluation
- Did the participants learn the desired skill?
- Example:  Can the trainees successfully input a residence into the RES*check* software?
- Example:  Can the trainees successfully plan review a set of commercial plans for energy?
- Example:  Can the participants successfully make Mother Makela’s pie crust?

## Behavior

- What happens when the trainees leave the classroom and return to their job?
- Did what they learn in training transfer to their:
- Plan review and inspection practices?
- Building design?
- Construction practices?
*For Energy Code Training– This is What It’s All About!*

*Construction Practice – Air Sealing*

*Construction Practice – Effective Insulation*

## Behavior

- Must be evaluated after trainee has an opportunity to put into practice what was learned at the training
- Immediate in some cases (plan review and inspection)
- Long term in other cases (building design)

*Appropriate Design and Equipment Selection*

## Behavior

- Barriers to change
- Trainee does not like the new practice and may not implement
- Trainee may like the new practice but management or time constraints may prevent the change

## Behavior

- So – How Do You Measure a Change in Behavior?
- Possible Alternatives:
- Follow-up surveys to the participants
- Paper or phone
- Have you been able to implement what you learned from the training?
- Why or why not?

## Behavior

- Look at change in construction or design practice for those that attended training
- Design changes can be measured during plan review
- Fewer plan review comments
- More complete documentation

## Behavior

- Construction practice can be measured during inspection or onsite visit
- Fewer correction notices
- Increased level of quality of construction

## Behavior

- Other Indirect Methods of Measuring Impact
- Technical assistance requests to local jurisdiction or BECP Help Desk
- Increase in calls following a training usually means something changed

## Results

- $64,000 Question:
- Is the change in construction practice a direct result of the training????
- $ 74,000 Question:
- Do we care as long as practice is changed in a positive direction??

## Results

- For Energy Code Program Trainers
- How much energy did the training really save????
- Factors that could influence this measurement:
- Code is mandatory
- Current practice already above code
- Other outreach/training programs
- Above code programs in the region
- Knowledgeable workforce

## Results

- Need to develop a baseline to measure results
- How much energy was used in a population of building prior to the training occurring?
- How much energy was used in a similar population of buildings following the training effort?
- Difference is the “return on investment” for the training program
- ROI = Btus Saved / Training Dollars Spent

## Evaluation Feedback Loop

- Results Should Feedback into Program Design
- For modifying individual program
- Results Should Feedback into Needs Assessment
- Design of future programs

## Evaluation Feedback Loop

- Invite inspectors, plan reviewers and builders, add clarification on weathersealing and duct sealing