# Presentation: 267645

## Bringing Your Program Full Circle: Evaluation Tips and Techniques2008 National Air Quality ConferenceElizabeth Schmitz, KY Division for Air QualityJulie Ernst, University of MN Duluth

**Notes:**
Thank Julie, mention workshop, “expert disclaimer”, ask if folks have experience with evaluationn

## Overview of      Program Evaluation

## What is Program Evaluation?

-  The systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.               (Patton, 1997).

**Notes:**
At the heart of program evaluation is a process involving the systematic collection of information about a program to make improvements, judgments, and decisions about future programming.  With the current emphasis on accountability and a lot of buzz about evaluation, it’ s not uncommon for an organization to decide to evaluate their program because they feel they should, even if they don’t have a clear reason or focus.  Program evaluation, however, is not merely accumulating and summarizing information about your program.  The idea of using the evaluation data to improve a program or make decisions is central.  Thus, program evaluations are conducted with a specific use for and user of the evaluation in mind.

## Why evaluate?

- Understand what is working and what isn’t (program improvement)
- Make sound decisions (should the program be continued, scaled back, discontinued, enhanced?)
- Justify programs, explain accomplishments
- Promote programs, products, and services
- Gain funding
- Guide program development

**Notes:**
Why care about evaluation? 
Understand what is working and what isn’t (program improvement)
Make sound decisions (should the program be continued, scaled back, discontinued, enhanced? When there is not enough time and money to do EVERYTHING, evaluation helps us decide which things are worth doing!)
Justify programs, explain accomplishments - GPRA (Government Performance Results Act of 1993 – holds federal entities accountable for using resources wisely and achieving program results); 
Promote programs, products, and services
Gain funding
Guide program development (know where you are going)

## Importance of “Use”

- Idea of using the evaluation data to improve a program or make decisions is central to evaluation
- Evaluations should be conducted with a specific **use for** and **user of** the evaluation in mind

## Types of Evaluation based on Purpose (Intended Use)

**Front-End: **to guide program development (Is this program needed?  How should it be designed?  What should the program outcomes be?); Used by those developing the program
**Formative:** to guide program improvement (What is working? What needs to be improved? How it can be improved?); Generally used internally; *Often* occurs in early stages of program development
**Summative:** to guide decisions about the program’s future; Used internally and externally by key decision-makers (program staff, supervisors, funders); O*ften* occur later in program development

**Notes:**
Front-end evaluation often is used to guide program development, and involves studying a problem, situation, or issue to find out if a program is needed or how it should be framed. Front-end evaluation can be used to identify gaps between your audience’s current level of knowledge, attitudes, skills, and/or behaviors and the level that is desired.  It can also be used to guide the development of new programs by helping to determine the characteristics or needs of an audience, define program goals and objectives, and identify potential stakeholders.  A front-end evaluation usually takes place prior to or in the very early stages of developing a program, and is generally conducted for or used by those who will be developing the program.

Formative evaluation is used primarily to guide program improvement.  Often, formative evaluations are conducted along with a pilot test of the program or materials before they are formally implemented or produced.  Information gathered about an audience’s reaction to and learning from this pilot test can be used to make changes to the program or materials. Formative evaluations also are used to gather information about how a program is delivered.  A formative evaluation can help identify areas of a program that are working and/or areas of a program that need to be improved.

## Formative v. Summative:

*“**When the cook tastes the soup, that’s formative evaluation; when the guest tastes it, that’s summative evaluation.”*
- (Robert Stake in Patton, 1997)

**Notes:**
Determining the purpose for (type of) evaluation  at the beginning of an evaluation helps focus the evaluation

## Check for Understanding

- What type of evaluation do the following examples describe?
- After developing a series of potential messages, the APCD held focus groups to determine what message was most likely to move Louisville residents to take action that would benefit air quality.

**Notes:**
Formative

## Check for Understanding

- Pre- and post-tests are routinely used to evaluate the learning gains of workshop participants, and tallied at the end of a year for an overall picture of program efficacy.

**Notes:**
Front-end
Summative

##         The Evaluation Process

- Focus your evaluation
- Develop your evaluation plan
- Develop data collection tools
- Collect data
- Analyze data and interpret results
- Communicate and use the results to improve the program or make decisions

## Focusing your Evaluation

## Focusing Your Evaluation

- A.  Identify the purpose for your evaluation (clarify uses and users)
- For example:
*  **The purpose of this evaluation is to* *determine which EE programs support the mission of the KY DAQ, in order for the Director to make summative decisions regarding which programs to continue and which to suspend.*

**Notes:**
User? Uses?

## Focusing Your Evaluation

- Questions to consider:
- Who are your program stakeholders?
- Why are you considering evaluating your program?
- Who are your evaluation stakeholders?
- Who is the primary intended user of your evaluation?
- Specifically, how will the results be used?

## Focusing Your Evaluation

- Identify the purpose for your evaluation (clarify uses and users)
- Describe your program
- Introducing the logic model!

**Notes:**
Who here is familiar with logic models?

## What is a Logic Model?

- Diagram that summarizes key elements of a program in a way that shows the relationships among program elements
- Relationship between what we put in, what we do, and outcome); describes the sequence of events thought to bring about benefits or change

**Notes:**
A tool for describing a program
A framework for thinking about WHAT can be evaluated
A tool for linking program development and evaluation

## Everyday Logic Model

- H
- U
- N
- G
- R
- Y

- Gather
- Ingredients

- Cook
- and Eat

- Feel
- Satisfied

## Logic Model

- S
- I
- T
- U
- A
- T
- I
- O
- N

- INPUTS

- OUTPUTS

- OUTCOMES

**Notes:**
Hungry – gather ingredients – cook and eat – feel satisfied

Review the logic model handout

## SITUATION

*The conditions that give rise             to the program*
- What needs to be done?
- What do our stakeholders want done?
- What are our priorities?

## INPUTS

*What we invest*
- Resources and contributions
- Staff, volunteers, time, money, materials, equipment, technology, partners, facilities

## OUTPUTS

*What we do and who we reach*
- Activities (training, recruitment, workshop etc.) and Products (activity guide, exhibit, curriculum, poster, etc.)
- People we reach (visitors, citizens, participants, students)

## OUTCOMES

*The results *
- Learning: Awareness, Knowledge, Attitudes, Skills, Opinions, Motivations
- Action: Behavior, Decision-making, Social action, Policies
- Ultimate Impact: Social, Economic, and Environmental Conditions

## Outputs v. Outcomes

- Output (Activity) driven:
- Teens volunteered an average of 10 hours over the summer in community service projects.
- _Outcome (Impact) driven: _
- Teens learn how to identify and solve a community need.
- Teens feel more responsible for their community.
*Outcomes answer: SO WHAT? What difference does the program make?*

## Logic Model – 2 other pieces

- S
- I
- T
- U
- A
- T
- I
- O
- N

- INPUTS

- OUTPUTS

- OUTCOMES

- External Factors

## Context in which the program is situated and external conditions which influence the success of the program, such as:
		Politics
		Policies
		Demographics				
		Economics					
		Culture			
		Biophysical environment

- 		Politics
- 		Policies
- 		Demographics
- 		Economics
- 		Culture
- 		Biophysical environment

- External Factors

## Logic Model – 2 other pieces

- S
- I
- T
- U
- A
- T
- I
- O
- N

- INPUTS

- OUTPUTS

- OUTCOMES

- External Factors

- Assumptions

## Beliefs we have about the program and the way we think it will work
   ~ the participants
   ~ the way the program will operate
   ~ how resources will be used
Faulty assumptions lead to poor results: Are your assumptions realistic and sound?

-    ~ the participants
-    ~ the way the program will operate
-    ~ how resources will be used
- Faulty assumptions lead to poor results: Are your assumptions realistic and sound?

- Underlying Assumptions

## Check for Understanding

- Teachers applied their new air quality understanding in the classroom.
- The Office of Energy Policy awarded a grant of $350 for the purchase of CFL’s.
- The Clean Air for KY program reached 4,000 students.
- Increasing students’ knowledge about air quality through school-based outreach will encourage students to take action, like turning off the lights, at home.

**Notes:**
Situation
Outcome
Input
Output
Assumption

## Focusing Your Evaluation

- Identify the purpose for your evaluation (clarify uses and users)
- Describe your (EE) program
- Consider logistics
- Available staff for the evaluation
- Timeframe
- Money/other resources available
- Contextual or other external factors that may affect the evaluation process

## Developing your Evaluation Plan

## Evaluation Plan

| Evaluation Questions What do you want to know? | Indicators How will you know it?  | Sources of Info. Who can provide the info? | Data CollectionTools
What will you use to gather the info? | Design/ Sampling
From whom and when will info be gathered? |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |

**Notes:**
Ensures that the remaining steps in the evaluation process are deliberate and purposeful
Helps make sure that you are collecting the right information at the right time from the right sources

| Evaluation Questions | Indicators  | Sources of info | Tools | Design/ Sampling
 |
| --- | --- | --- | --- | --- |

- Evaluation Questions: 2 Phases
- 1. Generate a list of potential evaluation questions.
- 2. Narrow your list - would the evaluation question:
-  Be of interest to primary intended user?
-  Provide information that addresses the intended use 	for the evaluation results?
-  Contribute information that is not already known?
-  Be of continuing interest?
-  Be feasible, in terms of time, money, and skill?
- Issues can emerge that require new or revised questions; be flexible, yet do not chase every new or interesting direction that emerges

**Notes:**
Would the evaluation question:
	- Be of interest to primary intended user?
	- Provide information that addresses the intended use for the evaluation results? Provide information that can be acted upon?
	- Contribute information that is not already known?
	- Be of continuing interest? (v. personal interest or curiosity)
	- Be feasible, in terms of time, money, and skill?

## Slide 32

| Evaluation Questions What do you want to know? | Indicators How will you know it?  | Sources of Info. Who can provide the info? | Tools
What will you use to gather the info? | Design/ Sampling
From whom and when will info be gathered? |
| --- | --- | --- | --- | --- |
| Have fire fighters’ understanding of 401 KAR 63:005 and the health effects of illegal burning increased? |  | 

 |  |  |
| Has the training changed their attitude towards and/or under-standing of the issue? | 

 | 
 |  |  |

**Notes:**
Evaluation Plan: As a planning worksheet/brainstorming and then as a finalized plan

Evaluation Questions: Provide the direction and foundation for the evaluation

## Indicators 

- Evidence or information that represents the phenomena of interest: What would indicate this program objective/outcome has been achieved?  What does “success” look like?
- Help you know something; they are usually specific and measurable
- For each aspect you want to measure, ask: What would it look like? What kind of information is needed?

| Evaluation Questions | Indicators  | Sources of info | Tools | Design/ Sampling
 |
| --- | --- | --- | --- | --- |

## Slide 34

| Evaluation Questions What do you want to know? | Indicators How will you know it?  | Sources of Info. Who can provide the info? | Tools
What will you use to gather the info? | Design/ Sampling
From whom and when will info be gathered? |
| --- | --- | --- | --- | --- |
| Have fire fighters’ understanding of 401 KAR 63:005 and the health effects of illegal burning increased? | Can list what can/can’t be burned legally; can list health effects, identify routes of exposure and sensitive populations | 

 |  |  |
| Has the training changed their attitude towards and/or under-standing of the issue? | Discussion indicates attitude/understanding shift; more illegal burns referred to DAQ by F.D.s | 
 |  |  |

**Notes:**
more coming in next session

## Determine Sources of Information 
(Who will provide the data?)

Participants
Non-participants
Key informants (parents, teachers, previous participants)
Program staff, administrators, or partners
Program documents (logs, records, minutes of meetings)

- (Who will provide the data?)
- Participants
- Non-participants
- Key informants (parents, teachers, previous participants)
- Program staff, administrators, or partners
- Program documents (logs, records, minutes of meetings)

| Evaluation Questions | Indicators  | Sources of info | Tools | Design/ Sampling
 |
| --- | --- | --- | --- | --- |

**Notes:**
Who can provide the information needed – which are most feasible

## Slide 36

| Evaluation Questions What do you want to know? | Indicators How will you know it?  | Sources of Info. Who can provide the info? | Tools
What will you use to gather the info? | Design/ Sampling
From whom and when will info be gathered? |
| --- | --- | --- | --- | --- |
| Have fire fighters’ understanding of 401 KAR 63:005 and the health effects of illegal burning increased? | Can list what can/can’t be burned legally; can list health effects, identify routes of exposure and sensitive populations | Fire fighters

 |  |  |
| Has the training changed their attitude towards and/or under-standing of the issue? | Discussion indicates attitude/understanding shift; more illegal burns referred to DAQ by F.D.s | Fire   fighters

Regional office staff |  |  |

**Notes:**
Look at our matrix – possibilities (or wait until the end and go back across all if time; if not homework)

## Determine Data Collection Tools

| Evaluation Questions | Indicators  | Sources of info | Tools | Design/ Sampling |
| --- | --- | --- | --- | --- |

- Determine Data Collection Tools

- Tool = what you use to collect data
- Choice of tool dependent on:
- Intended users of and use for evaluation
- Evaluation question, indicator, and source of information previously identified
- Amount of time and money
- Skill and philosophy of evaluator
- Weighing of advantages and disadvantages

**Notes:**
reference sheet on tools; course going into detail in future session on several of the tools

## Determine Data Collection Tools

| Evaluation Questions | Indicators  | Sources of info | Tools | Design/ Sampling |
| --- | --- | --- | --- | --- |

- Determine Data Collection Tools

- Types of Data (Information) that Can Be Collected:
- Qualitative** ****–** Descriptive, narrative, rich in explanation.  Often collected using a smaller set of participants.  Depth
- Quantitative** ****–** Numerical measurement.  Often collected through larger set of participants.  In some cases, can be generalizable to a population.  Breadth

**Notes:**
reference sheet on tools; course going into detail in future session on several of the tools

## Slide 39

| Evaluation Questions What do you want to know? | Indicators How will you know it?  | Sources of Info. Who can provide the info? | Tools
What will you use to gather the info? | Design/ Sampling
From whom and when will info be gathered? |
| --- | --- | --- | --- | --- |
| Have fire fighters’ understanding of 401 KAR 63:005 and the health effects of illegal burning increased? | Can list what can/can’t be burned legally; can list health effects, identify routes of exposure and sensitive populations | Fire fighters

 | Pre-Post test
(Quant.) |  |
| Has the training changed their attitude towards  the issue? | Discussion indicates attitude/understanding shift; more illegal burns referred to DAQ by F.D.s | Fire   fighters

Regional office staff | Focus Group
(Qual.)
TEMPO reports
(Quant.) |  |

## Design
When would you need to collect data & from whom if you want to show:
gain or change in participants’ knowledge? 
participants “outperform” another group?
participants have the desired characteristic, behavior, or knowledge after your program? 
changes in participants over time? 
results can be attributed to your program in a causal sense? 

- When would you need to collect data & from whom if you want to show:
- gain or change in participants’ knowledge?
- participants “outperform” another group?
- participants have the desired characteristic, behavior, or knowledge after your program?
- changes in participants over time?
- results can be attributed to your program in a causal sense?

| Evaluation Questions | Indicators  | Sources of info | Tools | Design/ Sampling |
| --- | --- | --- | --- | --- |

**Notes:**
*See Figure 2.4 (p. 158)
think of them as logistics – research more rigorous attention needs to be paid; here, practical questions of who are you going to collect data (treatment and control, treatment only) and when (before and after, after, multiple times?) see reference sheet
(time series/repeated measures or pretest-posttest)
(posttest-only control-group or one-shot with treatment)
(repeated measures/ time series)
(posttest-only equivalent control group or pretest-posttest nonequivalent control group)
Then pass out handout – what is the name for these designs?
Phrasing of evaluation question indicates this

## Sampling (Who/how many?)
A sample is a subgroup of a larger group (population)
Sampling refers to the method used to select the people, classrooms, counties, etc. to study
Sampling decisions are based on population size, what you want to know, and the resources available.
First questions to ask: What is the population of interest and is sampling needed?  If the population is small, you likely will include all its members.  

- A sample is a subgroup of a larger group (population)
- Sampling refers to the method used to select the people, classrooms, counties, etc. to study
- Sampling decisions are based on population size, what you want to know, and the resources available.
- First questions to ask: What is the population of interest and is sampling needed?  If the population is small, you likely will include all its members.

| Evaluation Questions | Indicators  | Sources of info | Tools | Design/ Sampling |
| --- | --- | --- | --- | --- |

**Notes:**
think of them as logistics – research more rigorous attention needs to be paid; here, practical questions of who are you going to collect data (treatment and control, treatment only) and when (before and after, after, multiple times?) see reference sheet

Sampling cont.
Assuming sampling is needed, is generalization from a sample to a larger population desired? (Do you want to say that findings hold for participants beyond the participants from whom data was collected?) Or is it more important that you are including those who are “information rich?”
	If yes - Probability (Random) Sampling: information gained from the sample can be generalized back to the larger population from which the sample is taken
	If no - Nonprobability Sampling: when generalization is undesirable or unnecessary or when probability sampling isn’t possible; sample is not intended to represent the population

## Slide 42

| Evaluation Questions What do you want to know? | Indicators How will you know it?  | Sources of Info. Who can provide the info? | Tools
What will you use to gather the info? | Design/ Sampling
From whom and when will info be gathered? |
| --- | --- | --- | --- | --- |
| Has fire fighters’ knowlege of 401 KAR 63:005 and the health effects of illegal burning increased? | Can list what can/can’t be burned legally; can list health effects, identify routes of exposure and sensitive populations | Fire fighters

 | Pre-Post test | All training participants for each session; pre- and post- test |
| Has the training changed their attitude towards and/or under-standing of the issue? | Discussion indicates attitude/understanding shift; more illegal burns referred to DAQ by F.D.s | Fire   fighters

Regional office staff | Focus Group

TEMPO reports | Volunteer sub-groups

All regional offices; Bi-annual reports |

## Developing Data Collection Tools

## Developing Data Collection Tools

- Interview: Open-ended questions designed to elicit thoughts, feelings, experiences, and stories from respondents; no response provided; Qualitative
- Survey: a list of stable and primarily closed questions; Quantitative

- Developing Data Collection Tools

## Interviews

- Create an Interview Guide
- How Many Respondents?
- Iterative Process

**Notes:**
Create an Interview Guide

Pay attention to wording- Avoid technical jargon or “loaded” words
Put questions into the right sequence –
Establish a comfortable atmosphere; sequence so that people can easily move from one concept to the next
Come up with ideas to keep the conversation going – Prompts are important (i.e. Do you mean...)
Pilot test your interview guide
Make sure that the questions are not biased, the vocabulary is appropriate, interview right length
How many respondents?
A handful to 40 is reasonable
Make sure you are talking to the right people!
Iterative process
After a few interviews, you may change the guide a bit; length of interview may influence number of participants, etc.

## Conducting Interviews

- Be consistent with questions and cues!
- Reassure participants that you will protect their identity
- Begin and end by thanking participants
- Request permission to tape
- Hold in neutral, private territory where the respondent will be comfortable

## Focus Groups

- Group interviews that encourage participants to build on each others’ responses
- Guidelines for developing interviews are true for focus groups as well
- Last roughly 1-2 hours and involve 6-10 people

**Notes:**
Relatively quick and easy way to build a richer understanding
Purpose: not to obtain agreement, but to engage people in exploring their perceptions, feelings, and opinions
Good for eliciting experiences or ideas, especially regarding a draft program or concept; ideal tool for a needs assessment 

## Surveys

- Process large amounts of data
- Best for generalizing results to a larger population
- Can be used for planning, formative, and summative types of evaluation
- Always pilot test your surveys!

## Collect Data

## Collect Data

- First, review your logic model, evaluation focus, and planning matrix to ensure that you are on target with the data you collect
- Look for existing data that may meet your needs before designing new data collection protocol

## Set Data Collection Schedule

| What data collection tools are you using? | What are the sources for your data? | Who will collect the data? | Who can confirm the schedules? | When should data be collected? |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |

## Data Collection

- Maintain consistency!
- Data collection methods and instruments need to be pilot tested
- Standardize data collection
- Ethics of Conducting an Evaluation
- Ask only needed information
- Confidentiality and anonymity
- Informed consent

**Notes:**
Anonymity: an individuals answers are permanently separated from the person’s name or other identifying information

Confidentiality: link participants to their responses through a code sheet (i.e. number on sign in sheet, birthdays or fave color on responses)

May need to create alias for interview participants

Informed consent: Ensure that evaluation participants know, in advance, what they are agreeing to do and voluntarily give their permission.

## Implementing a Survey

- How will (or will you) entice people to complete your survey?
- Explain why the survey is important
- Thank people for their participation
- Send a postcard reminder after 3 weeks
- Train surveyors to be gracious, polite, consistent, and friendly

## Focus Groups

- If respondent types vary on 2 or more important dimensions, hold separate meetings
- Have at least 2 sessions for each group.
- Continue to schedule focus groups as long as you continue to get different types of information.

## Focus Group Planning

- Location – pleasant, neutral (i.e. library, extension office, school)
- Time – convenient for participants
- Incentives
- Thank and confirm location, time, date with participants in advance

## Focus Group Implementation

- Moderator, note-taker, recording device, “quiet” snacks!
- If possible, seat dominant personalities near moderator and shy participants across from the moderator, let participants know about the recorder in a non-threatening way
- Transcribe tapes within 3-5 days

## Analyze Data and Interpret Results

## Surveys

- When the surveys come back:
- Code them
- Enter data on a spreadsheet
- Reduce the data to summary statements and relate these back to your evaluation questions
- Analyze the data
- Frequencies
- Means
- Correlate one variable to another
- Look for significant differences

**Notes:**
Frequency – “plain count” the number of people who respond to each item in a scale or ranked set of responses
Mean – average number of numerical responses (sum / total number)
Median – middle value of the responses
Mode – most common response
Correlate – look for relationships between responses
Significant differences: Compare groups – i.e. pre-test/post-test

## Focus Groups

- Transcription not always necessary – notes are often sufficient
- Identify similarities and patterns
- Group themes into meaningful categories

**Notes:**
Transcription not always necessary – notes are often sufficient

Identify similarities and patterns – You may want to organize the data by question or by respondent – for example, you can read everyone's’ response to the first question and formulate several themes that summarize them. Or you can read all the responses that a certain type of respondent provides, such as all the teenagers or teachers that attended a program.  Code the data using colors, numbers, or key words.

Group themes into meaningful categories - Summarize the data into categories and subcategories so that the information is coherent and complete. All relevant viewpoints should be included somewhere. As you summarize themes and interpret the data, you may find direct quotes that speak to the themes (be sure to protect anonymity).  It may be helpful to make an outline of the themes that deserve explanation and the relationships between them, particularly those themes expressed by several people or those conveyed strongly by a few respondents.

## Communicate and Use Evaluation Results

## Organize and synthesize your data

- Tie specific results to your evaluation questions, using your coded responses.
- Present your findings without interpretation – save that for conclusions
- Use qualitative data to describe quantitative data, if available

**Notes:**
You now need to examine the data and consider how it helps you answer your evaluation questions.  Revisit your logic model and evaluation plan one more time.  This review will help you organize your data –the first step!

## Slide 62

| Evaluation Questions What do you want to know? | Indicators How will you know it?  | Sources of Info. Who can provide the info? | Tools
What will you use to gather the info? | Design/ Sampling
From whom and when will info be gathered? | Evaluation Results: 
What did you find? |
| --- | --- | --- | --- | --- | --- |
| Have fire fighters’ knowledge of 401 KAR 63:005 and the health effects of illegal burning increased? | Can list what can/can’t be burned legally; can list health effects, identify routes of exposure and sensitive populations | Fire fighters

 | Pre-Post test | All training participants for each session; pre- and post- test | Average test score increased from 60% to 75% |

## Draw Conclusions

- Explain results
- Review open ended questions for insights
- Don’t ignore unexpected results
- Consider unsolicited comments
- Avoid claiming causality (use “the data suggest” or “the results indicate”)

## Make Recommendations

- Focus on the most important, data-based conclusions
- Be practical
- List specific action items
- Request stakeholder review

## Sharing Your Findings

- Formal written report
- PowerPoint
- Press Release
- Web Site
- Customize for audience and occasion

## Resources

- Earnst, J.A., Monroe, M.C., Simmons, Bora.  Evaluating Your Environmental Education Programs: A Workbook for Practitioners, 2007 (Draft), to be published by Earth Island Press
- Education Program Evaluation (Course OUT 8102): 4 day course through USFWS National Conservation Training Center – _[www.nctc.org](http://www.nctc.org/)_
- Applied EE Program Evaluation (NRES 410/610): online course through University of Wisconsin Stevens Point – www.uwsp.edu/natres/rwilke/eetap/Website2006/AEEPE.htm

## Thanks!

- Dr. Julie Ernst, U of MN Duluth
- US Fish & Wildlife Service, Division of Education and Outreach, National Conservation Training Center
- National Oceanic and Atmospheric Agency
- North American Association for Environmental Education