page 137

Evaluating Your Program

Considerations in Planning Evaluation

Setting the Stage

Building the Evaluation Team

Using Logic Models to Build the Evaluation 

Framework

Assuring High Quality Data

Best Practices

Common Problems

Measuring Success

Sources of Qualitative and 

Quantitative Data

Process Evaluation Measures

Outcome Evaluation Measures

Disseminating Findings

Summary

6
page 138 Evaluation of Your Program’s Implementation
page 139

6

E very healthy homes program faces the challenge of showing that its activities made a difference in the lives of the clients it serves. 

Similarly, all programs seek to improve the quality of their activities, 

whether in targeting, service delivery, or efficient use of resources. 

CDC’s Evaluation Working Group website 

(http://www.cdc.gov/eval/index.htm) and its 

Asthma Program website (http://www.cdc.gov/

asthma/program_eval/default.htm) contain other 

important evaluation tools.

Ongoing evaluation can help programs engage 

in continual quality improvement and enhance 

the effectiveness of their interventions. 

As noted in Chapter 3, evaluation should 

be incorporated into every step of your 

program’s operations; it should not be 

considered a separate or end-stage activity. 

In fact, much of the data needed for program 

evaluation are collected as part of your 

program’s daily operations. The important 

lesson is to organize, analyze, and use this 

information effectively. 

This chapter provides an overview of evaluation 

issues pertinent to healthy homes programs. 

It relies heavily on the HUD Office of Healthy 

Homes and Lead Hazard Control’s 2010 Draft 

Evaluation Guide for Healthy Homes Grantees 

(Draft Evaluation Guide).1 The Centers for 

Disease Control and Prevention (CDC) also 

issued a Framework for Program Evaluation 

in Public Health in 1999 that contains useful 

guidance on selecting evaluation measures, 

data collection strategies, and dissemination of 

findings.2 Please refer to both documents for a 

more in-depth discussion of specific program 

evaluation issues.

Key Messages

• Evaluation is a continual process in order 

to achieve quality improvement. Planning 

for evaluation needs to begin during the 

program design phase.

• Planning for evaluation includes 

considerations of who should be part of 

the team, how to secure good quality 

data, and ways to measure qualitative and 

quantitative accomplishments.

• Logic models can serve as an important 

program planning and evaluation tool.

• Evaluation measures include process 

(outputs), outcomes, and costs.

• Disseminating evaluation findings is critical 

to program sustainability.

Evaluation of Your Program’s Implementation
page 140

Considerations in Planning 

Evaluation

All healthy homes programs have some 

evaluation activities and audiences in 

common, but there are important differences. 

New programs tend to focus on whether 

they reached their recruitment targets, if 

their staffing, activities, and infrastructure 

operated as planned, and whether program 

deliverables were met. Mature programs often 

set more ambitious goals for evaluation, such 

as determining the comparative costs and 

benefits of alternative strategies to reach target 

populations, and whether to expand service 

areas or implement new intervention strategies. 

Established programs may also want to look 

in a more nuanced way at the outcomes of 

their activities, such as how long the effects of 

interventions or behavioral change strategies 

last, or which intervention has the most impact 

on health or housing conditions. 

Ultimately, the specific questions that your 

program seeks to answer will guide the 

development of your evaluation plan. Broad 

questions that every healthy homes program 

should answer as a part of their evaluation are:

• Was the program implemented as planned?

• Were program participants representative of 

the target populations? Who was excluded 

and why? 

• Were the services provided implemented 

consistently with program protocols?

• How many participants received each of the 

project’s services or interventions?

• Was there a meaningful improvement in the 

condition of the target housing units?

• Was there an improvement in the health of the 

target housing units’ occupants?3

Operationally, evaluation assesses a program’s 

reach, processes (practices) and outcomes 

(effects). Healthy homes programs should 

engage in a combination of process and 

outcome evaluations. Along with these broad 

questions, every program faces decisions about:

• Who should be part of the evaluation team?

• How can we assure high quality evaluation 

data?

• How will we measure success?

• How can we track our costs?

• How will we ensure that evaluation findings are 

used internally to enhance our effectiveness 

and ability to use resources more efficiently?

• How do we plan to communicate the 

evaluation findings to various audiences?

Finally, programs need a framework for 

assessing the overall quality of the evaluative 

effort. General standards for assessing the 

overall quality of an evaluation effort include: 

• Utility: Are the needs of the intended users 

being met?

• Feasibility: Is the evaluation effort practical 

and achievable?

• Propriety: Is the evaluation effort conducted 

with regard for the rights and interests of 

those involved or affected by the program?

• Accuracy: Are the findings correct and 

reported with impartiality?4

Setting the Stage

Several factors are important to setting the 

stage for evaluation:

Planning. A well-designed program is the 

result of rigorous planning. Programs that lack 

a documented program plan are difficult to 

evaluate. Logic models are especially useful 

in developing both a well-designed program 

and a well-executed evaluation. (Logic model 

development will be addressed in more detail 

later in the chapter.)

Piloting. For an evaluation to be accurate, 

the intervention itself, the measurement tool, 

and data collection forms (e.g., instrument 

or questionnaire) need to remain the same 

throughout program implementation (such 

as between pre- and post-intervention data 

collection). While mid-course corrections may 

be needed, changing a protocol during program 

implementation may compromise interpretation 

of evaluation data. For this reason, it is 

important to pilot services, interventions and 

tools before conducting a full-scale evaluation. 

Monitoring. Regular discussion among staff 

delivering services and those responsible for 

Evaluation of Your Program’s Implementation
page 141

evaluation is important to continually adapt 

program activities and data collection in 

accordance with the evaluation design. These 

meetings can identify and solve problems 

arising from data collection, implementation of 

treatments or services, or program participants.

Commitment. Meaningful evaluation requires 

organizational and programmatic dedication. 

Programs that value evaluation view it as an 

integral part of their day-to-day work. This is 

demonstrated by program leadership action to 

ensure the capacity and resources to carry out 

the evaluation. Leadership must also commit to 

applying the lessons learned.

Protection of clients’ interests. All evaluation 

staff should be familiar with and mindful of basic 

principles governing the protection of human 

subjects. Appendix 6.1 includes an overview of 

federal protections.

Building the Evaluation Team

The organizational structures of healthy homes 

programs vary. As a result, evaluation systems—

staffing, communication, documentation, data 

review—will look different within each program. 

It is important that programs plan and budget 

for this effort appropriately. Unless restricted 

by grant or other agency policy, a rule of thumb 

is to dedicate ten percent of costs to evaluation 

efforts.

A key decision is to determine who needs to be 

involved in the evaluation and what information 

they need. In most programs, the bulk of 

the evaluation will be conducted in-house in 

response to grant reporting or internal agency 

requirements. Common staff positions and their 

responsibilities include the following:

Senior Program Staff/Principal Investigators.

An organization’s leadership team needs to 

be briefed regularly on the program’s ability 

to meet deliverables, lessons learned, and 

implementation challenges faced by the 

program. Senior staff members fulfill a critical 

role in disseminating findings to elected and 

non-elected officials, the media, potential 

funders, and community representatives. These 

stakeholders need quantitative data to answer 

questions about scope, effectiveness, and cost 

of services. But they also need qualitative or 

testimonial data in order to “put a face on the 

problem” or show the more intangible benefits 

of program activities. If research grant funding is 

part of the project, the Principal Investigator—

whether part of the agency or a third party—

needs to be engaged directly throughout 

the course of the evaluation in briefing senior 

agency staff on the progress and findings of the 

evaluation. 

Program Managers. Program managers need 

timely and accurate data on program outputs 

(measurable activities), program outcomes 

(changes in health and housing conditions), 

and costs. Program managers need to be able 

to track and project performance measures 

throughout the life of the project, usually 

through the use of spreadsheets or database 

systems. They especially need to be able to 

identify deviations from what was expected 

or planned. This means that evaluation data 

should be available to the program manager on 

a real-time basis, and reviewed at least monthly 

to ensure program deliverable dates are met 

according to the planned time frames. 

Program Service Delivery. Any staff member 

who is engaged in direct services (intake, 

outreach, home visits, assessment, and 

intervention) produces evaluation data and 

needs to be kept aware of the progress of the 

program. If these staff do not understand the 

importance and the impact of their work, the 

quality of service delivery may suffer. Weekly or 

biweekly team meetings, regular staff briefings, 

case review, and periodic staff retreats help 

to build a shared understanding and can also 

identify unexpected consequences or problems 

with implementation. Written protocols, regular 

chart reviews and case conferences are also 

Evaluation of Your Program’s Implementation

Tour guide plays a post-tour game with youth.
page 142

ways to evaluate and standardize the delivery 

of services. Research assistants and data entry 

staff need to be engaged in this process in 

order to understand the relationships between 

data quality and program implementation and 

evaluation. (A more detailed discussion of data 

quality issues follows later in this chapter). 

Clients of the program. Some healthy homes 

programs establish an advisory board composed 

of stakeholders to participate in program 

planning and provide feedback on program 

progress and outcomes. Community-based 

partners and recipients of program services 

should be part of this advisory effort. Genuine 

community involvement is critical to assuring 

that the program stays true to its mission and 

services actually meet community needs. Client 

satisfaction surveys can also be an important 

tool. Moreover, the qualitative data that come 

from focus groups and community meetings 

are important for understanding the context in 

which services were provided.

Data analysts. Advanced expertise (such as 

epidemiology, statistics, or economics) may 

be required to determine the degree to which 

observed program effects occur by chance or 

are directly related to the program’s activities. 

Some agencies have the capacity to conduct 

this analysis in-house, either directly or through 

partnership with other offices. These analysts 

need to be engaged early in program decision 

making so they can determine whether proposed 

evaluation measures have the validity, reliability, 

and sensitivity to meet evaluation needs. They 

also need access to high-quality statistical 

analysis tools, such as SAS®TM, SPSS®TM, 

STATATM®, etc. Finally, they need dedicated time 

to the project throughout the life of the initiative, 

not just at the conclusion of the program when an 

evaluation report is being produced.

Agency information management or 

information technology (IT) staff. Lack of 

computing capacity or the appropriate statistical 

analysis program can hinder in-house evaluation 

efforts. It is important that decisions about 

data collection and security be made early and 

with the involvement of IT staff. In most cases, 

access to a current version of word-processing, 

spreadsheet and database software is sufficient 

for most members of the evaluation team. It is 

important, however, that all members of the 

team have the same version of these programs 

and are trained to use them appropriately.

Third-Party Evaluators. While most healthy 

homes programs do the bulk of their evaluation 

in-house, sometimes outside evaluators are 

engaged if programs lack the necessary inhouse resources. Independent evaluation 

consultants are often located in academic 

institutions and may already be program 

partners. However, selecting an appropriate 

outside evaluator involves finding an individual 

or team experienced in both housing and 

health evaluation, understands your project’s 

activities, structure, and the target population, 

communicates with your program in ways 

that you and your staff understand, is willing 

to spend time on site, and focuses on the 

evaluation questions or hypotheses that 

are important to you and your audience for 

evaluation. For more details on selecting a 

third party, see Draft Evaluation Guide and 

Project Planning and Evaluation Guidebook: a 

Manual for Practitioners and Managers of SelfSufficiency Demonstration Projects.

5

Using Logic Models to Develop an 

Evaluation Framework

Logic models link a program’s framework to 

the evaluation plan. They are a visual method of 

describing the relationships among program 

elements. Some HUD and most CDC and EPA 

grant programs require the development of a logic 

model as part of the grant application, as do many 

private funders. While there are many guidelines 

for building a logic model (see Draft Evaluation 

Guide, Sundra et al, 20036; Kellogg, 20047; Project 

Planning and Evaluation Guidebook8), at their most 

basic level, logic models serve several purposes: 

1. To identify short-term, intermediate, and longterm outcomes for the program.

2. To link expected outcomes to the 

program’s intended activities and inputs 

(staff, resources, behavioral and physical 

interventions). Logic models challenge 

program designers to articulate assumptions 

about cause and effect. They also help to 

specify program milestones such as what 

activities must be completed before certain 

outcomes can be expected.9

Evaluation of Your Program’s Implementation
page 143

Figure 6.1 Elements of a Logic Model

• Resources or inputs can be financial, human, organizational, system-oriented or communitybased—the factors needed to support program activities.

• Activities include services such as education, home visits, environmental assessment and home 

intervention.

• Outputs are counts of activities related to recruitment/intake, education, completed housing 

interventions and case management. 

• Outcomes and impact are changes in short, intermediate and long term measures, such as health 

or housing conditions.

Figure 6.2 A Proposed Logic Model Related to Healthy Homes

Program Inputs Activities Outputs Short Term Mid-Term Long-Term

Focus Outcomes Outcomes Outcomes

Asthma

Healthy 

Homes 

Pilot

Health 

Department 

Staff

Home Visiting 

Programs

Home 

Inspectors

Community 

Organizations

Advocacy 

Organizations

Rental 

Property 

Owners

Pest 

Management 

Professionals 

Contractors

Elected 

Officials

Clinicians

Health 

Insurers

Foundations

Funding

Equipment

Supplies

Educate 

families about 

environmental 

triggers in the 

home

Conduct visual 

assessments

Provide 

supplies for 

dust control 

and pest 

management 

Interventions 

including 

integrated pest 

management 

(IPM), moisture 

control, 

lead hazard 

reduction, etc.

Refer families 

to smoking 

cessation 

programs

Refer families 

to housing 

rehab services 

to address 

issues beyond 

program scope

Refer housing 

units to code 

enforcement

Number of 

home visits 

completed

Number 

of referrals 

to partner 

organizations

Counts of 

supplies 

delivered

Number 

of visual 

assessments 

for pests, 

mold, and 

moisture 

completed

Number 

of homes 

receiving 

specific 

interventions 

such as IPM

Number 

of housing 

inspections for 

housing code 

violations 

Increased use 

of mattress and 

pillow covers, 

IPM supplies 

after one 

month

Improvement 

in family 

Knowledge, 

Information, 

and Behavior 

(KIB) scores in 

one month

Increase 

number of 

units where 

family limits 

smoking in the 

home

Reduction in 

counts of pests 

in units after 

three months

Reduction in 

the reported 

number of 

symptom days 

after 3 months

Reduction in 

the number of 

asthma triggers

Reduction 

in ER and 

hospitaliztions 

at 12 months

Increased 

number of units 

enrolled in 

housing rehab 

programs

Reduction 

in mold and 

moisture 

conditions 

observed at 

12-month visual 

assessment

Families show 

long term 

improvement 

on KIB scores

Health insurers 

reimburse 

or pay for 

home visits 

and low cost 

environmental 

interventions

Property 

owners adopt 

preventive 

policies

Evaluation of Your Program’s Implementation
page 144

3. To establish program boundaries to prevent 

“mission creep.” Knowing what cannot be 

accomplished through project activities is 

often as important as specifying what should 

be accomplished, especially if community 

expectations for a program are high.

Sundra et al. suggest that building a logic 

model can start from left-to-right (specification 

of activities → outcomes) or right-to-left 

(specification of outcomes → identification 

of inputs and activities). In the left-to-right 

approach, each link in the model is accompanied 

by the question, “Why?” (such as Why is this 

input needed for a planned activity? or Why will 

this activity produce the expected short-term 

change?).10 In the right-to-left approach, the 

critical question is “How?” (such as How would 

asthma rates be influenced by the intermediate 

outcomes expected from a particular activity?). 

By working left-to-right and right-to-left, program 

evaluators begin to identify potential weaknesses 

in program design, as well as activities that may 

be extraneous to accomplishing program goals. 

(See Appendix 6.2 for CDC’s proposed logic 

model for Healthy Homes Programs.)

Assuring High Quality Data

Quality improvement and credible evaluation 

rely on accurate, precise, and reliable data. 

Data quality management tools can range from 

simple checklists to detailed Quality Assurance 

Plans (QAP) that outline staff responsibilities for 

program oversight, data collection methods, 

quality control procedures, maintenance 

of records to support reporting and fiscal 

administration, and a data analysis plan.

Two data quality documents—a data collection 

plan and data analysis plan—are especially 

important to develop early in the program 

design and implementation stage. A written 

data collection plan provides direction to staff 

or program partners responsible for collecting 

information needed to evaluate the program. 

Input from field staff is almost always needed to 

produce these plans as these individuals collect 

most of the raw data and have unique insights 

from the field into the target population. Data 

collection policies and procedures need to be 

simple and clear so that project staff understand 

why they are collecting the information and how 

it will be used. Their involvement in piloting data 

collection tools and procedures is also highly 

recommended. 

A data analysis plan keeps the program on track 

by identifying critical data elements that the 

evaluators will study and qualitatively evaluate 

for program impact. Data analysis plans should 

be developed in the program design phase to 

minimize collection of extraneous data. Similarly, 

preliminary data analysis needs to be conducted 

early in the process of implementation, once all 

data collection instruments are finalized. Outcome 

data (change in health or housing conditions) 

associated with interventions conducted in the 

first units enrolled should be analyzed as soon as 

possible. Waiting to analyze data until the end of 

the program impairs the program’s ability to make 

mid-course corrections when needed.

Best Practices 

Strategies for good data collection, data entry, 

and data management include:

Evaluation of Your Program’s Implementation
page 145

• Clearly written protocols, policies, and 

procedures.

• Thorough training of staff to assure high 

levels of accuracy in data entry, protection 

of confidential client information, and data 

security.

• Periodic refresher training for all staff on 

protocols, policies and procedures. Crosstraining and periodic assessment of interrater reliability of data collected by different 

staff members to assure consistency is 

recommended.

• Use of data collection instruments with known 

validity and reliability whenever possible.

• Implementation of double data entry (in which 

two individuals enter the same data and then 

reconcile discrepancies). Data entry staff 

should also be trained to inspect their work for 

missing data and errors.

• Routine checks for data quality and 

completeness by program managers.

• Site visits by supervisors who periodically 

accompany staff on home visits to assess 

whether services are being delivered 

according to program protocols.

• Regular data cleaning by running simple 

statistical reports (e.g., counts, frequencies) 

and correcting out-of-range values. 

• Familiarity with the laboratory quality control 

measures and chain of custody requirements 

if environmental sampling is conducted. It 

is important to assure that the laboratory 

used meets certification requirements, 

such as the National Lead Laboratory 

Accreditation Program (NLLAP—http://www.

epa.gov/oppt/lead/pubs/nllap.htm) and the 

Environmental Microbiology Laboratory 

Accreditation Program (EMLAP—http://www.

aihaaccreditedlabs.org/AccredPrograms/

EMLAP/Pages/default.aspx). 

• Security of physical data (i.e., locked and 

secured files).

• Security of electronic data (i.e., password 

protection or limited access to data, regular 

backups of data).

Common Problems

Some data management and analysis practices 

to avoid:

• Failure to use available data management 

tools appropriately, such as handcounting data rather than using electronic 

spreadsheets.

• Poor record management, such as storage 

of records in multiple locations and lack of 

version control on data collection instruments.

• Failure to clearly state criteria to interpret 

a result, such as what constitutes “high” or 

“low.”

• Failure to consider and report alternative 

explanation of findings.

• Failure to limit conclusions to the situations, 

contexts, and period for which the data are 

applicable.

In-depth discussion of data quality and a QAP 

template can be found at http://portal.hud.gov/

hudportal/documents/huddoc?id=DOC_36504.

doc, and in the Evaluation Guide. 

Figure 6.3 Standard Program 

Forms Used for Evaluation

• Program intake forms (household 

characteristics, housing characteristics, and 

housing conditions)

• Environmental assessment forms

• Resident interviews (also referred to as 

Environmental Questionnaire)

• Environmental sampling 

• Construction forms (documenting 

treatments and costs)

• Health indicator questionnaires

• Health measurements (physiologic 

measures)

• Program tracking tools (reports of 

community outreach efforts, trainings, and 

number of attendees) 

Source: Draft Evaluation Guide, p. 23.11

Evaluation of Your Program’s Implementation
page 146

Figure 6.4 Sources of Qualitative 

and Quantitative Measures

Program evaluation information and data can 

come from documents routinely produced by 

the program:

• Grant proposals and quarterly reports

• Newsletters, publicity materials and press 

releases

• Meeting minutes and administrative 

records 

• Registration and enrollment forms

• Publications and journal articles

• Prior evaluations

• Asset and needs analyses

• Client satisfaction surveys

• Databases

• Reports held by funders or partner 

agencies

• Websites

• Graphs, maps, charts, photos, and videos

• Feedback collected at meetings or 

interviews with key individuals, including 

clients and non-participants, staff, general 

public, key informants, critics, staff of other 

agencies, representatives of advocacy 

groups, policy-makers, funders, and 

federal, state, and local health and housing 

officials.12

To implement the Healthy Homes Surveillance 

System at CDC, the Healthy Homes and Lead 

Poisoning Branch developed the Healthy 

Homes and Lead Poisoning Surveillance System 

(HHLPSS) application. HHLPSS is a web-based 

surveillance system used primarily in state health 

departments. Local health departments will be 

able to access the application through a web 

browser. State and local health departments will 

use HHLPPS to track home-related risk factors 

and interventions, and to report to CDC. Other 

programs may also use HHLPPS, but it may require 

upgrades to their systems’ hardware and software. 

The application is provided at no charge.

Measuring Success

Defining the intended audiences of the 

evaluation helps to determine what measures 

to use and how to best to report the findings. If 

the audience is interested in program activities 

and efficiency in service delivery, a process 

evaluation may be appropriate. If the audience is 

interested in change in behaviors, or health and 

housing conditions, more outcome measures are 

needed. Increasingly, all programs are required 

to demonstrate that program funds are used 

efficiently and effectively. Most healthy homes 

programs use a combination of evaluation 

strategies and measures.

As described in Chapter 4, there are numerous 

validated assessments and tools that can be 

used to evaluate the effectiveness of healthy 

homes interventions. Appendix 4.1 adapted from 

the Evaluation Guide provides links to these tools, 

describes the tools’ comprehensiveness, indicates 

if a tool has been validated, and comments on 

ease of adaptation and burden of use. 

Process Evaluation Measures

Process evaluation measures program reach (i.e., 

who the program has influenced or touched), 

activities and services, and documents program 

operations. The goal of process evaluation is to 

fully understand how a program is implemented. 

Process evaluation answers the questions:

• Did the program serve its intended audience 

(as defined and characterized by the 

level of environmental health risk and by 

demographics)? How many people were 

served? How many were not served and why?

• Were services delivered as planned, within 

target timeframes and budget, and in a way 

that left recipients satisfied?

Process evaluation provides information to 

make mid-course corrections, if needed, to 

enhance a program’s success (see Figure 6.5). 

It reviews information on the characteristics of 

families or residents receiving program services 

and analyzes a program’s performance against 

established benchmarks (deliverables) that 

reflect the intended goals. Benchmarks need 

to be established for each program phase and 

shared with project staff and partners.

Evaluation of Your Program’s Implementation
page 147

Figure 6.5 In Newport, Rhode 

Island, Process Evaluation Leads to 

Changes in Interventions, Referrals, 

and Policy

Healthy Residents, Healthy Homes Coalition 

was dedicated to using coordinated health, 

housing and social service responses to 

reduce the burden that asthma placed on 

families and individuals living at the Newport 

Housing Authority. The coalition included the 

Newport Housing Authority, local health and 

social service providers, Rhode Island state 

health officials, and regional organizations.

The evaluation team met weekly to review 

progress. The team included all staff who did 

home visits and the maintenance staff from 

public housing. One benefit of this approach 

is that different interpretations could be 

gleaned from the same data. For example, 

consistent mold and moisture problems in 

bathrooms led the maintenance staff member 

to suggest a change to the interventions: use 

porcelain pedestal sinks near bathtubs rather 

than cabinet–style sinks that are susceptible 

to moisture intrusion if water splashed out of 

the tub.

Regular reviews of the data also showed that at 

baseline 51 percent of the family housing and 

29 percent of senior or disabled housing units 

had at least one family member who smoked 

in the home. Smoking in homes was common. 

Participants requested referrals to smoking 

cessation services in large numbers, but the 

program had no health partners to provide 

those services. This led to identification of new 

partners for smoking cessation. On the housing 

intervention side, it also led to recognition that 

a policy change could reduce exposures to ETS 

throughout the properties and provided the 

basis for a new Housing Authority initiative—a 

smoke-free housing policy.

Any changes to program operations or 

interventions should be systematically 

documented. This can be done through 

program staff meeting summaries or coalition 

meeting minutes that communicate challenges, 

accomplishments, and changes in strategic 

direction. The documentation can then be used 

to compile a lessons learned document on 

conclusion of the program.

Process evaluations typically focus on the 

outputs of program operations. These measures 

typically come from data collection forms and 

protocols established to track activities within 

each program phase, such as intake forms, 

scopes of work, visual assessment forms, 

environmental questionnaires, environmental 

sampling documentation, health indicator 

forms and questionnaires, and program service 

delivery tracking tools. 

Outcome Evaluation Measures

Outcome evaluation focuses on the degree to 

which any change in health status or housing 

condition is attributable to a program’s services 

or interventions. The purpose of this type of 

evaluation is to measure the impact or the effect 

of the program and identify changes or benefits 

to clients due to program participation. 

Outcomes are directly tied to program goals. 

Changes in attitudes, values, knowledge, 

skills, behaviors, health status, and indoor 

environmental quality are examples of outcome 

measures. Outcome evaluation determines if your 

program is providing the right services to bring 

about the changes that you want to see in your 

target population’s health and their housing.

If your program wishes to attribute improvements 

in health and housing conditions to your 

interventions and not to chance, you will need to 

pay special attention to statistical methods and 

your evaluation design. This is addressed in detail 

in the Draft Evaluation Guide and in Appendix 

6.3. HUD’s 2007 evaluation of its healthy homes 

grantees found the majority of Demonstration 

grantees employed a pre-/post-intervention 

design using the results of the visual assessment 

and participant surveys to measure the effects 

of interventions and changes to attitudes and 

behaviors before and after intervention.13 In 

general, strengths of the pre/post design include 

simplicity and the ability to use data routinely 

collected in the field. However, the design 

is susceptible to biases that make it difficult 

to determine whether changes in observed 

outcomes are affected by confounding factors 

such as other outside events, individual growth and 

development, and the process of being observed.14

Evaluation of Your Program’s Implementation
page 148

Use of a control group provides a stronger basis 

for analyzing program effects. Control groups 

are groups of housing units or individuals that 

are comparable in terms of location, condition, 

residential characteristics or demographics 

but do not receive program services or 

interventions. Randomized control trials, in 

which participants are randomly selected and 

assigned to a treatment or control group, are 

the “gold standard” for experimental research 

designs.

Use of control groups and randomized designs 

is generally confined to research and is rarely 

used for evaluating programs. Ethical issues 

associated with the use of control groups in 

housing intervention research are reviewed in a 

publication by the NAS/IOM.

Health and Well-Being Outcomes

Health outcomes commonly tracked by healthy 

homes programs are listed in Figure 6.7.

Evaluation of Your Program’s Implementation

Figure 6.6 Common Program Outputs

• Program Outreach and Community Education

 Number and type of presentations 

− Audience (health care personnel, parents, 

contractors, educators, community, 

rental property owners, tenants, owneroccupants). Audiences can be divided into 

public and professional.

− Number of individuals reached

 Number of health fairs

− Number of interactions (participants, 

names recorded on sign in sheets, requests 

for follow up information)

− Pieces of literature distributed

 Number of housing units reached through 

door-to-door canvassing

 Number of media events

− Paid vs. unpaid

• Program Referrals 

 Number of referrals from medical providers

 Number of referrals from community-based 

organizations

 Number of requests for information and 

enrollment associated with different media 

placements

 Number of referrals by healthy housing 

programs to other housing programs

 Number of referrals by healthy housing 

programs to other health or social service 

programs

• Families/Individuals Recruited

 Demographics of participants and 

nonparticipants

 Level of housing risk in units of participants 

and non-participants

• Case Management/Care Coordination and 

Education

 Number of children tested for lead 

exposure

 Number of home visits

 Number of families receiving educational 

intervention

 Number of referrals provided to families for 

supportive health and social services

 Number of referrals to other services 

completed and not completed 

 Number of families receiving cleaning 

supplies

• Home Assessment

 Number of questionnaires administered

 Number of homes with assessments 

conducted

 Number of homes with environmental 

samples collected

 Number of environmental samples collected

• Housing Unit Remediation

 Number receiving enhanced ventilation and 

moisture control interventions

 Number receiving integrated pest 

management

 Number receiving lead hazard reduction

 Number of fire alarms and carbon 

monoxide detectors installed

• Work Force Development Activities

 Number of contractors recruited 

 Number of contractors recruited from the 

target population and area

 Number of individuals trained
page 149

Figure 6.7 Common Health and 

Well-Being Outcomes 

• Lead poisoning

 Reduction in post-intervention blood 

lead levels of resident children < 6 years 

of age 

• Asthma

 Decreased symptom severity

 Increased number of symptom-free days

 Improved child and caretaker quality of 

life

 Reduced number of missed school and 

work days

 Reduced number of emergency 

department visits and hospitalizations

 Decreased medical costs

 Reductions in use of rescue medication

 Reduced number of unscheduled doctor 

or clinic visits

• Unintentional injuries 

 Reduced number of emergency 

department visits and hospitalizations

 Decreased medical costs

• Changes in knowledge, attitude and 

behavior

 Increased evidence of smoking outdoors 

or participation in smoking cessation 

programs

 Increased storage of food in pest-proof 

containers 

 Increased use of lead-safe work 

practices

 Use of more effective cleaning practices

Figure 6.8 Common Housing 

Outcomes

• Change in level of pest infestation

• Change in concentrations of common 

allergens (e.g., dust mite, cockroach, 

mouse in floor dust)

• Change in number and type of injury 

hazards and other indicators of home 

safety 

• Changes in number and concentration of 

contaminants in air or other media

• Reduction in lead-based paint hazards

• Reduction in mold or moisture-damaged 

materials

• Presence of a working smoke or carbon 

monoxide detector after 12 months

• Reduction in radon levels

• Improved ventilation

• Increase in the number of homes where 

smoking is not permitted indoors

Evaluation of Your Program’s Implementation

Housing Outcomes 

Allergen, pest and moisture control are complex 

processes that often depend on structural changes 

in the home and behavioral changes by the 

residents. All housing interventions (treatments) 

are appropriate for outcome evaluation. Commonly 

tracked housing outcomes are listed in Figure 6.8.

Cost Measures

Fiscal accountability requires that programs 

use their funding in the most efficient and 

transparent way possible. Increasingly, healthy 

housing programs have been asked to produce 

per capita service delivery costs as a way to 

justify continued or new funding and as a basis 

for selecting service delivery methods. Mason 

and Brown’s (2010) publication on estimating 

costs for housing-related interventions to 

prevent specific illnesses highlights five types of 

cost studies employed in the public health and 

housing sectors. These include:

• Cost-of-illness (COI) studies that quantify the 

public health burden created by an illness 

by including all medical, non-medical, and 

productivity costs associated with an adverse 

health outcome.

• Cost analysis (CA) studies that focus on the 

costs of implementing an intervention, and 

may also document the costs saved as a result 

of the intervention (or the net costs after 

subtracting the total program costs from the 

cost of illnesses).

• Cost-effectiveness analyses (CEA) that 

calculate the ratio of net costs (as defined 

above) per improvement in health associated 

with the intervention (such as the costs 
page 150 Evaluation of Your Program’s Implementation

associated with symptom-free days). This is 

used to assess the relative efficiency of two or 

more interventions.

• Cost-utility analysis (CUA), a type of costeffectiveness analysis in which the health 

outcome measure includes quality- adjusted 

life years (prolonged life and quality of life).

• Cost-benefit analyses (CBA), the “gold 

standard” that compares the costs and 

consequences (positive and negative) of 

intervention strategies.17

Typically, healthy homes programs use cost analysis 

and cost-of-illness, and link these monetary values 

to program outputs. Most programs will not 

have the expertise to perform the more intensive 

economic analyses. (Refer to the Evaluation Guide 

for more details on these issues.) 

However, Mason and Brown note that all 

programs need to be thoughtful in determining 

the audience perspective for looking at the 

costs associated with different outcomes. Since 

low-income families typically bear more of the 

adverse health outcomes associated with poor 

housing, a societal perspective on costs and 

benefits, rather than one that focuses on who 

pays for specific services at a local or state level, 

may more equitable. Moreover, intangible costs, 

such as social justice, may be hard to monetize 

and include in the analysis, but are important.18

Healthy homes programs need to develop a 

cost-tracking system that includes the costs of 

implementation not only by activity, but ultimately 

on a per unit and intervention level. Quantifying 

the cost of service systems or program activities 

needs to include all direct labor costs, fringe 

benefit and indirect costs, educational, office and 

field supplies and materials, travel, and laboratory 

analyses. Examples of programmatic costs to track 

and quantify are illustrated in Figure 6.11.

Figure 6.9 In Seattle: Outcome Evaluations with Demonstrated Health 

and Housing Benefits

Seattle’s Breathe Easy healthy homes project 

(BEH) renovated 35 units in a public housing 

authority development to reduce asthma 

triggers. Interventions included improvements 

to building ventilation and energy efficiency, 

use of building materials with limited potential 

to outgas, treatments to assure smooth and 

cleanable floors, smoke-and pet-free policies, 

extensive client education on reducing asthma 

triggers, and asthma case management services 

provided by Community Health Workers. 

Evaluation data were collected at the initiation 

of the one-year CHW intervention in the old 

home and at one year after the move into the 

renovated housing units. A 2010 evaluation 

report compared pre- post data on 34 residents 

of BEH to a comparison group of 68 participants 

in an earlier year-long asthma-project who 

received the same CHW education model but 

whose units did not receive structural repairs. 

Clinical evaluation criteria included a detailed 

assessment of asthma severity, medication 

and health services use, administration of 

the pediatric quality of life tool, skin test 

sensitization, and pulmonary function tests. The 

pre-post comparison for BEH residents showed 

statistically significant changes (that is, results 

beyond what would be expected by chance) in: 

• Increases in the number of symptom-free days;

• Reductions in number of urgent care visits;

• Reductions in measured asthma triggers in 

house dust in the home; and

• Improvements in pulmonary function.

Comparison of the BEH residents to the control 

showed that the BEH group improved more on 

most measures, but there were few statistically 

significant differences apart from reduced 

nighttime symptoms. However, the mean 

number of environmental triggers in the BEH 

homes at one year were significantly reduced, 

with a modest construction cost of $5000–7000. 

The authors suggest that these costs can be 

recouped in a relatively short time through the 

potential cost savings in asthma care costs and 

missed work and school days.

Source: Takaro, TK, Krieger, J, Song, L, Sharify, D and Beaudet, N. 201116
page 151

Figure 6.10 In Phoenix: Lessons Learned from Outcome Evaluations Improve 

the Next Round Research

From 2003—2006, the Phoenix Healthy Homes 

Demonstration funded a study of respiratory and 

injury risk-reduction in 67 homes. Residents were 

low-income, primarily Hispanic, home-owners 

with at least one child in the home under age15. 

The program targeted for recruitment families 

with children with a diagnosis of asthma through 

the Phoenix Children’s Hospital, Phoenix Head 

Start, Arizona Department of Health Services’ 

Childhood Lead Poisoning Prevention Program, 

and the City of Phoenix Neighborhood Services 

Department’s Housing Rehabilitation Section. 

A multidisciplinary team of a health educator, 

bilingual home assessor, injury specialist, 

and a pediatric pulmonary nurse practitioner 

oversaw a pre-post intervention educational 

and assessment home visit. Interventions in the 

home included structural repairs, supplies and 

education, with a median cost of $1,139 and an 

average of $5,440 (excluding staff time). Thirtysix potential hazards were assessed including 7 

potential respiratory health hazards. The total 

number of hazards declined from an average of 

14 at baseline to 3.4 following intervention. The 

declines were statistically significant for 30 of the 

36 potential hazards.

Of the 62 caregivers who completed the 

questionnaire, 97 percent reported that their 

homes were safer after being part of the project. 

Nearly all respondents reported that the health 

of the child with asthma was better than before 

the project. The average number of respiratory 

health hazards per home dropped from 3.3 to 

0.9 from baseline to post-intervention (p<0.001). 

Observed dust in carpets and bedding, observed 

dust in the heating and cooling system, poor 

housekeeping, musty smell, and observed 

cockroach infestation were hazards in 52–69 

percent of the homes at baseline and were 

significantly reduced at post-intervention with 

the percent of homes with improvements ranging 

from 77 to 98 percent.

Injury-related hazards observed in at least 80 

percent of the study homes at baseline included: 

no fire escape route; no functional smoke alarms; 

improper storage of vitamins, medications, and 

household products; no emergency telephone 

numbers; sharp objects improperly stored; and 

no first aid kit. All individual structural injury 

hazards were significantly reduced from baseline 

to post-intervention with improvements in 

88–100 percent of the homes with hazards at 

baseline.

The project received a second round of funding 

in 2009. It revised its protocol to address lessons 

learned during the earlier outcome evaluation. 

These included:

• Assigning a staff member to track families 

more closely and to be sure that medical data 

and the post-intervention home visits were 

conducted in a more timely manner.

• Tracking asthma and injury outcomes through 

objective measures and over a longer period of 

time to supplement parent self-reports.

• Using more pictures and demonstrations 

during home visits to compensate for the fact 

that many of the families enrolled had low 

literacy levels in both English and Spanish.

• More effort to assess the effect of individual 

interventions.

• More evaluation of the educational 

components of the program.

Source: Dixon SL, Fowler C, Harris J, Moffat S, Martinez Y, Walton H, Ruiz B, Jacobs DE. 2009.16

Appendix 6.4 offers an example of a cost-benefit 

analysis that calculates health benefits, energy 

savings, and increased housing value as a result 

of window replacement and paint stabilization, 

and includes a formula that can be used by your 

program to calculate costs specific to your locality.

Disseminating Findings

Ultimately, evaluation results are only 

meaningful if they are used by decision-makers 

and stakeholders to improve the effectiveness 

of programs and develop or refine policies to 

protect the community from housing hazards. A 

Evaluation of Your Program’s Implementation
page 152

dissemination plan should include the provision 

of information to the following stakeholders:

• Program Participants. Specific information 

on the housing assessment and the outcome 

of the interventions should be provided 

to all housing occupants and owners in a 

timely manner. This is consistent with legal 

requirements that results of lead-based paint 

hazard testing, control treatments, and dust 

clearance are provided to owners (see Lead 

Disclosure Rule) and occupants are notified 

(see Lead Safe Housing Rule). Although there 

are few required notification standards for 

healthy homes treatments, it is recommended 

that similar information be provided to owners 

and occupants. 

Providing a summary of what was learned 

as a result of the program as a whole and 

how this information will be used to advance 

healthy homes initiatives is a meaningful 

way to acknowledge participation and raise 

awareness of the program’s next steps. It 

is important to provide the information in 

a format that is clear and understandable. 

Guidance can be found at http://www.

plainlanguage.gov/.

• Media. The media—print, radio, television, 

and social-media—can be used to raise 

awareness of the program at the onset and 

to lay the groundwork for dissemination of 

results. When planning to share information 

about the program and its impact with the 

media, it is advantageous to identify a family 

or families who benefited from the program 

who are able and willing to interact with the 

media. 

• Community. Community involvement in 

planning, implementing, and evaluating 

healthy homes programs can result in more 

effective and sustainable programs. Sharing 

information about program results, their 

meaning, and future activities can strengthen 

existing partnerships, assure the community’s 

continued support for the program, and 

demonstrates respect for their contributions. 

• Elected and Other Officials. Providing 

outcome information to elected officials, that 

highlights the impact on their constituents, 

is an effective way to assure future support 

and funding for your healthy homes program. 

Some programs have had good results when 

community members served by the program 

deliver this information. “Report cards” that 

compare an elected official’s district to other 

districts and the city and state as a whole 

provide meaningful contextual information.

• Funders. Most funders require regular 

program reports as a condition of financial 

support. When disseminating information 

about the program—at any stage—it is crucial 

to acknowledge the funding source(s).

• Health Plans. Insurance companies and HMOs 

may be interested in the costs of services 

and impact of interventions on use of health 

care (emergency room visits, hospitalization, 

and medication use). Some health plans are 

working with asthma programs to provide 

or reimburse for the costs of asthma case 

management.

Evaluation of Your Program’s Implementation

Figure 6.11 Programmatic Cost 

Data

• Outreach and Education Costs

 Forms and outreach materials 

(development and production) costs

 Free media and paid media costs (press 

releases, public service announcements, 

marketing campaigns; staff time needed 

for media interviews, health fairs)

 Public education and training offerings

• Health Intervention and Assessment 

Costs

 Home visits (education, case 

management) 

 Visual assessments

 Environmental sampling 

 Laboratory analysis 

• Housing Intervention Costs

 Treatment (specification) costs

 Average cost per housing unit

 Range of housing unit costs

 Specific intervention costs (IPM, 

moisture control, lead hazard reduction, 

safety kits)
Evaluation of Your Program’s Implementation page 153

• Partners. Sharing information about the 

program’s outcomes, what it means for future 

efforts, and brainstorming next steps provides 

the opportunity to celebrate a program’s 

partnerships and their importance in achieving 

positive change in health status and housing 

outcomes. This type of celebration can be 

used to conclude a program formally and 

solidify the partnership for the future.

• Peers/Colleagues. A robust evaluation can lay 

the groundwork for sharing your program’s 

outcomes and lessons learned on a national 

scale through conference presentations, 

poster sessions, and publications in peerreviewed literature. Academic partners who 

serve as external or third-party evaluators are 

great resources for this level of dissemination.

Recognize that evaluative information on the 

program also needs to be distributed in ways 

appropriate to the target audience. Press 

releases, press conferences, fact sheets, and 

maps are helpful for the media and elected 

officials. Community groups and program 

participants may prefer more graphics and 

pictures, as well as summaries in plain language. 

Websites provide a useful tool for dissemination 

to all audiences.
page 154

Chapter 6 References*

1

 U.S. Department of Housing and Urban 

Development. August 2010. Draft Evaluation 

Guide for Healthy Homes Grantees. Washington, 

DC: U.S. Department of Housing and Urban 

Development, Office of Healthy Homes and 

Lead Hazard Control.

2

 MMWR Recommendations and Reports. 1999. 

Framework for program evaluation in public 

health, MMWR—Morbidity and Mortality 

Weekly Reports. (48 RR11):1–40. Available: 

http://www.cdc.gov/mmwr/preview/mmwrhtml/

rr4811a1.htm.

3

 U.S. Department of Housing and Urban 

Development. August 2010. Draft Evaluation 

Guide for Healthy Homes Grantees. Washington, 

DC: U.S. Department of Housing and Urban 

Development, Office of Healthy Homes and 

Lead Hazard Control. pp. 1–2.

4

 MMWR Recommendations and Reports. 1999. 

Framework for program evaluation in public 

health. MMWR—Morbidity and Mortality 

Weekly Reports. (48 RR11):1–40, Box. 1.

5

 U.S. Department of Housing and Urban 

Development. August 2010. Draft Evaluation 

Guide for Healthy Homes Grantees. Washington, 

DC: U.S. Department of Housing and Urban 

Development, Office of Healthy Homes and 

Lead Hazard Control. pp.7–10. See also Tobari, 

J.R. and Hermann, J.A. 2001. Project Planning 

and Evaluation Guidebook: A Manual for 

Practitioners and Managers of Self-Sufficiency 

Demonstration Projects, pp. 11–20. Available: 

http://www.hud.gov/offices/lead/hhi/

hhiresources.cfm.

6

 Sundra, D.L., Scherer, J., and Anderson, L.A. 

2003. A Guide on Logic Model Development for 

CDC’s Prevention Research Centers. Available: 

http://www.ojp.usdoj.gov/BJA/evaluation/guide/

documents/cdc-logic-model-development.pdf.

7

 W.K. Kellogg Foundation. 2004. Logic Model 

Development Guide. Battle Creek, MI: W.K. 

Kellogg Foundation. Available: http://www.wkkf.

org/knowledge-center/resources/2010/LogicModel-Development-Guide.aspx.

Evaluation of Your Program’s Implementation

8

 Tobari, J.R. and Hermann, J.A. 2001. Project 

Planning and Evaluation Guidebook: A Manual 

for Practitioners and Managers of SelfSufficiency Demonstration Projects, pp. 39–50.

9

 U.S. Department of Housing and Urban 

Development. August 2010. Draft Evaluation 

Guide for Healthy Homes Grantees. Washington, 

DC: U.S. Department of Housing and Urban 

Development, Office of Healthy Homes and 

Lead Hazard Control. p. 23.

10 Sundra, D.L., Scherer, J., and Anderson, L.A. 

2003. A Guide on Logic Model Development for 

CDC’s Prevention Research Centers. pp. 8–12.

11 U.S. Department of Housing and Urban 

Development. August 2010. Draft Evaluation 

Guide for Healthy Homes Grantees. Washington, 

DC: U.S. Department of Housing and Urban 

Development, Office of Healthy Homes and 

Lead Hazard Control. p. 23.

12 MMWR Recommendations and Reports. 1999. 

Framework for program evaluation in public 

health. MMWR—Morbidity and Mortality Weekly 

Reports. (48 RR11):1–40, Box. 6.

13 U.S. Department of Housing and Urban 

Development. 2007. An Evaluation of HUD’s 

Healthy Homes Initiative: Current Findings and 

Outcomes. Figure 3–9, p. 28.

14 U.S. Department of Housing and Urban 

Development. August 2010. Draft Evaluation 

Guide for Healthy Homes Demonstration 

Grantees. Washington, DC: U.S. Department 

of Housing and Urban Development, Office of 

Healthy Homes and Lead Hazard Control. p 20.

15 Takaro, T.K., Krieger, J., Song, L., Sharify, 

D. and Beaudet, N. 2011. The BreatheEasy Home: The impact of asthma-friendly 

home construction on critical outcomes and 

trigger exposure. American Journal of Public 

Health.101(1), 55–62.

16 Dixon, S.L., Fowler, C., Harris, J., Moffat, S., 

Martinez, Y., Walton, H., Ruiz, B., and Jacobs, 

D.E. 2009. An examination of interventions to 

reduce respiratory health and injury hazards in 

homes of low-income families. Environmental 

Research. 109:123–130.

*Websites were verified during the drafting of this document but may have changed.
Evaluation of Your Program’s Implementation page 155

17 Mason, J. and Brown, M.J. 2010. Estimates 

of costs for housing-related interventions to 

prevent specific illnesses and deaths. Journal 

of Public Health Management Practice. 

(Supplement):S79–S89.

18 Mason, J. and Brown, M.J. 2010. Estimates 

of costs for housing-related interventions to 

prevent specific illnesses and deaths. Journal 

of Public Health Management Practice. 

(Supplement):S81.
page 156

Notes

Evaluation of Your Program’s Implementation