Technical Report

NREL/TP-6A2-43822

January 2009

Conceptual Soundness, Metric 

Development, Benchmarking, 

and Targeting for PATH 

Subprogram Evaluation 

Prepared for the U.S. Department of Housing 

and Urban Development (HUD)

Partnership for Advancing Technology in 

Housing

Gail Mosey, Elizabeth Doris, and 

Charles Coggeshall

National Renewable Energy Laboratory

Matt Antes, Jennifer Ruch, and John Mortensen 

Energetics
National Renewable Energy Laboratory

1617 Cole Boulevard, Golden, Colorado 80401-3393

303-275-3000 • www.nrel.gov

NREL is a national laboratory of the U.S. Department of Energy

Office of Energy Efficiency and Renewable Energy

Operated by the Alliance for Sustainable Energy, LLC

Contract No. DE-AC36-08-GO28308

Technical Report

NREL/TP-6A2-43822

January 2009

Conceptual Soundness, Metric 

Development, Benchmarking, 

and Targeting for PATH 

Subprogram Evaluation 

Prepared for the U.S. Department of Housing 

and Urban Development (HUD)

Partnership for Advancing Technology in 

Housing

Gail Mosey, Elizabeth Doris, and

Charles Coggeshall

National Renewable Energy Laboratory

Matt Antes, Jennifer Ruch, and John Mortensen

Energetics

Prepared under Task No. WF1K.1000 and Subcontract No. KLCX-4-44453-19
NOTICE

This report was prepared as an account of work sponsored by an agency of the United States government. 

Neither the United States government nor any agency thereof, nor any of their employees, makes any 

warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or 

usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not 

infringe privately owned rights. Reference herein to any specific commercial product, process, or service by 

trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, 

recommendation, or favoring by the United States government or any agency thereof. The views and 

opinions of authors expressed herein do not necessarily state or reflect those of the United States 

government or any agency thereof.

Available electronically at http://www.osti.gov/bridge

Available for a processing fee to U.S. Department of Energy

and its contractors, in paper, from:

U.S. Department of Energy

Office of Scientific and Technical Information

P.O. Box 62

Oak Ridge, TN 37831-0062

phone: 865.576.8401

fax: 865.576.5728

email: mailto:reports@adonis.osti.gov

Available for sale to the public, in paper, from:

U.S. Department of Commerce

National Technical Information Service

5285 Port Royal Road

Springfield, VA 22161

phone: 800.553.6847

fax: 703.605.6900

email: orders@ntis.fedworld.gov

online ordering: http://www.ntis.gov/ordering.htm

Printed on paper containing at least 50% wastepaper, including 20% postconsumer waste
i

Table of Contents

Table of Contents................................................................................................................. i

1. Background and Objectives......................................................................................... 1

2. Conceptual Soundness.................................................................................................. 4

 Goals............................................................................................................................... 4

 Activities......................................................................................................................... 5

 Metrics.......................................................................................................................... 10

 Framework for Metric Selection............................................................................... 13

 Summary....................................................................................................................... 16

3. Recommended Metrics for PATH’s Concept Home................................................ 17

 Program Background .................................................................................................... 17

 Which PATH Goal(s) Does This Program Address? ................................................... 17

 Major Activities............................................................................................................ 18

 Web Tools for Information Dissemination................................................................... 18

 Measuring the Impact ................................................................................................... 19

 Key Data Sources.......................................................................................................... 19

 Approach for Identifying and Recommending Performance Metrics........................... 20

 Recommended Metrics ................................................................................................. 21

 Additional Recommended Metrics............................................................................... 24

4. Recommended Metrics for PATH’s ToolBase Program........................................ 25

 Program Background .................................................................................................... 25

 Which PATH Goal(s) Does This Program Address? ................................................... 26

 Major Activities............................................................................................................ 26

 Measuring the Impact ................................................................................................... 28

 Key Data Sources.......................................................................................................... 28

 Approach for Identifying and Recommending Performance Metrics........................... 29

 Recommended Metrics ................................................................................................. 30

 Additional Recommended Metrics............................................................................... 35

5. Recommended Metrics for PATH’s Structural Insulated Panels Program.......... 36

 Program Background .................................................................................................... 36

 Which PATH Goal(s) Does This Program Address? ................................................... 36

 Major Activities............................................................................................................ 37

 Web Tools for Information Dissemination................................................................... 37

 Measuring the Impact ................................................................................................... 38

 Key Data Sources.......................................................................................................... 38

 Approach for Identifying and Recommending Performance Metrics........................... 39

 Recommended Metrics ................................................................................................. 40

 Additional Recommended Metrics............................................................................... 43

6. Recommended Metrics for PATH’s Steel Framing Program................................. 44

 Program Background .................................................................................................... 44

 Which PATH Goal(s) Does This Program Address? ................................................... 44

 Major Activities............................................................................................................ 45

 Web Tools for Information Dissemination................................................................... 46

 Measuring the Impact ................................................................................................... 47

 Key Data Sources.......................................................................................................... 48
ii

 Approach for Identifying and Recommending Performance Metrics........................... 48

 Recommended Metrics ................................................................................................. 50

 Additional Recommended Metrics............................................................................... 54

7. Recommended Metrics for PATH’s Insulating Concrete Forms Program........... 55

 Program Background .................................................................................................... 55

 Which PATH Goal(s) Does This Program Address? ................................................... 55

 Major Activities............................................................................................................ 56

 Web Tools for Information Dissemination................................................................... 57

 Measuring the Impact ................................................................................................... 57

 Key Data Sources.......................................................................................................... 58

 Approach for Identifying and Recommending Performance Metrics........................... 59

 Recommended Metrics ................................................................................................. 60

 Additional Recommended Metrics............................................................................... 63

8. Baselines and Targets................................................................................................. 65

 Baselines....................................................................................................................... 65

Selection of Baseline Years...................................................................................... 65

Targets........................................................................................................................... 66

 Key Challenges............................................................................................................. 67

 Subprogram Details ...................................................................................................... 67

 Concept Home Program............................................................................................ 68

 Key Events............................................................................................................ 68

 Concept Home Baseline and Target Values ......................................................... 68

 ToolBase Program .................................................................................................... 71

 Key Events............................................................................................................ 71

 ToolBase Baseline and Target Values.................................................................. 71

 Structural Insulated Panels Program......................................................................... 75

 Key Events............................................................................................................ 75

 SIPs Baseline and Target Values.......................................................................... 75

 Advanced Steel Framing Program............................................................................ 78

 Key Events............................................................................................................ 78

 Steel Framing Baseline and Target Values........................................................... 78

 Insulating Concrete Forms Program......................................................................... 81

 Key Events............................................................................................................ 81

 Baseline and Target Values .................................................................................. 81

9. Guidelines for Developing Metrics, Data Collection, Analysis and Reporting for 

PATH Subprograms....................................................................................................... 83

 Developing Metrics....................................................................................................... 83

 Data Collection ............................................................................................................. 84

 Analysis and Reporting................................................................................................. 86

10. Conclusions................................................................................................................ 88

11. References.................................................................................................................. 90

Appendix A. Metrics for Goals...................................................................................... 93

Appendix B. Metrics for Scoring................................................................................... 98
1

1. Background and Objectives

The U.S. Department of Housing and Urban Development (HUD) Office of Policy 

Development and Research has been extensively involved in supporting research and 

development (R&D) of U.S. building technologies. This R&D includes innovations, 

construction systems, products, standards, regulations, and code issues that affect the 

affordability, safety, and livability of the nation’s housing. As the interrelationships of 

these topics become more complex, the continued need to conduct research and 

demonstrations becomes even more critical. 

In addition to the research and demonstration efforts administered directly by the Office 

of Policy Development and Research, HUD administers the Partnership for Advancing 

Technology in Housing (PATH) program. PATH provides an unprecedented opportunity 

to the private and public sectors of the U.S. housing industry to advance state-of-the-art 

practices in the design and construction of housing. Many scholars argue that the meager 

investment in residential technology research necessitates a program like PATH. Despite 

the importance of the housing industry to the American economy, this innovative chasm 

is especially true for detached, single-family housing – a market that makes up most of 

America's housing stock. It can purportedly take 10 to 25 years for a new housing 

technology to achieve full market penetration. Often, it takes that much time to simply 

introduce an innovation to the market. As such, PATH spurs new R&D initiatives, 

enhances the access to and quality of information resources about them, and decreases the 

institutional barriers to their development, adoption, and diffusion in housing. 

Because accomplishing this mission is broad and the partnering approach is complex, the 

program has experimented with numerous strategies and activities since its inception. In 

1994, the White House convened representatives from all segments of the U.S. 

construction industry to consider a broad set of National Construction Goals, which were 

subsequently published and disseminated as potential goals for the industry and strategies 

for reconsidering governmental investments in research and diffusion efforts. During the 

next three years, the residential segment of the construction industry – represented by 

home builders, code officials, product manufacturers, federal researchers across the 

cabinet, and other interested parties – developed a research plan for implementing the 

goals for the housing sector. PATH was the outgrowth of those proposals, and was 

officially launched on May 4, 1998. HUD was selected as the appropriate agency to 

house PATH because of the compatibility of both entities’ missions; with a focus on the 

housing market and industrial context rather than on individual home performance issues 

like energy, HUD was well-suited for PATH’s innovation goals. Likewise, PATH’s 

emphasis on affordability and quality supported HUD’s overall mission and its specific 

goals for increased homeownership and expanded, decent housing options.

PATH is a public/private partnership program of HUD and is aimed at accelerating the 

development and use of advanced housing technology. To that end, PATH’s original 

goals are as follows: 

1) Reduce the monthly cost of new housing by 20% or more.
2

2) Cut the environmental impact and energy use of new homes by 50% or more, and 

reduce energy use in at least 15 million existing homes by 30% or more.

3) Improve durability and reduce maintenance costs by 50%.

4) Reduce by at least 10% the risk of life, injury, and property destruction from 

natural hazards, and decrease by at least 20% illnesses and injuries to residential 

construction workers. 

The National Research Council’s (NRC’s) PATH evaluation found no evidence or 

baseline data to indicate that the housing performance goals were measurable and 

achievable (NRC 2003). These and other challenges associated with the original numeric 

goals are well-documented and can be summarized in five main points (Martin 2005, 

RAND unreleased documents, Cohn (ed.) 2006):

• The program was broadly focused, but the goals involved measuring the end-user

market. 

• The metrics were not appropriate to measuring the impact of an innovationfocused program.

• Due to the public/private partnership structure of the program, metrics should 

measure the impact of the sector interaction through information sharing as well 

as research and development.

• The rate of impact of the program varies over time and should be reflected in the 

performance metrics. Variable impacts are due to the length of time from 

building conception to completion resulting in a slow rate of uptake for new 

technology.

• The housing market is large and complex with multiple participating stakeholders 

and national geographic scope. Separating the PATH programmatic impact from 

other impacts for the former metrics of overall housing costs and efficiency gains

is impossible given the complexity of the market.

All of these factors – the broad program focus, the research and development and 

public/private partnership program aspects, the slow rate of change, and complex 

industry structure – create challenges for directly attributing savings to the PATH 

program as the original goals require. 

Recognizing these challenges through an evaluation by the NRC (Cohn 2006) and 

realizing the need for program impact measurement changes to meet national government 

program reporting requirements (e.g., the Government Performance and Results Act or 

GPRA), PATH refocused and refined its goals in 2002. Prior to 2002, the goals were

overambitious, yet specific enough to enable development of precise quantifiable metrics 

tied to unambiguous goals that measure programmatic success. The revised goals focus

on addressing barriers to technology innovation, which consists of a broader emphasis on 

the application of creative new ideas and the processes by which new products and 

services enter the market. The goals refocused PATH and its associated objectives with 

the overall aim of increasing the impact of technology innovation in the home-building 

sector. The revised goals are listed in Table 1.
3

Table 1. Revised Goals of PATH Program

1) Remove barriers and facilitate technology development and adoption

a. Identify current and potential barriers and measure their impacts

b. Develop practical methods to overcome current barriers

c. Develop alternative future industrial processes that eliminate barriers

2) Improve technology transfer, development, and adoption through information 

dissemination

a. Establish and maintain noncommercial information sources on 

innovation

b. Understand behaviors, attitudes, and needs regarding innovation 

information

c. Have relevant information materials on innovation and innovators for 

different parties

3) Advance housing technologies’ research and foster development of new 

technology

a. Establish sustained public and private R&D investment resources

b. Clarify and assist innovators with processes from technology 

transfer to market penetration

c. Have comparative standards for innovation performance, cost, 

benefits, and agendas. 

Source: Martin 2005

The objective of this study is to evaluate the conceptual soundness of the PATH 

program’s revised goals and establish and apply a framework to identify and recommend

metrics that are the most useful for measuring PATH’s progress. This report provides an 

evaluative review of PATH’s revised goals, outlines a structured method for identifying 

and selecting metrics, proposes metrics and benchmarks for a sampling of individual 

PATH programs, and discusses other metrics that potentially could be developed that 

may add value to the evaluation process. The framework and individual program metrics 

can be used for ongoing management improvement efforts and to inform broader 

program-level metrics for government reporting requirements.

Chapter 2 addresses the conceptual soundness of the revised goals and the methodology 

for developing metrics, while the subsequent chapters describe the recommended metrics 

for five PATH subprograms: Concept Home (CH), ToolBase®, Structural Insulated 

Panels (SIPs), Advanced Steel Framing, and Insulated Concrete Forms (ICFs). These 

programs were selected for evaluation based partly on prominence in the PATH suite of 

program, longevity, and data availability. Baseline and target values for five of the 

subprograms are provided in Chapter 8. Chapter 9 suggests guidelines for developing 

metrics, data collection, analysis, and reporting; and conclusions are presented in 

Chapter 10.
4

Working Definitions for this Report

Framework: Methodology for identifying recommended 

metrics.

Candidate (or proposed or potential) metrics: Metrics 

that could potentially be used for measuring program 

performance but have not yet been scrutinized by 

applying the framework.

Recommended metrics: Metrics that are considered 

useful for measuring program performance based on 

application of the framework.

Additional recommended metrics: Metrics that would 

be considered useful for measuring program 

performance but lack a key attribute (e.g., data 

availability). Often additional recommended metrics are 

meant to fill in any gaps in the set of recommended 

metrics in order to provide a comprehensive evaluation.

2. Conceptual Soundness

This section evaluates the conceptual soundness of the PATH program’s revised goals 

and develops a framework for identifying the most useful metrics for measuring PATH’s 

progress. The revised PATH goals seem to be more in line with the program purpose of 

improving innovation in the residential sector and are in accordance with the program’s 

structure as a public/private partnership. 

This section presents the list of 

candidate metrics developed by 

the PATH program for 

measuring progress toward its 

goals. It also highlights the 

proposed framework to rank 

the candidate metrics on the 

basis of clarity of the metric (to 

be understood by 

stakeholders), the ability to be 

attributed to the program, the 

feasibility of the data, and the 

comparability of the metric 

over time. This methodology 

provides an initial step toward 

refining the proposed metrics, 

which, over the course of further evaluation and testing for usability, are refined to form a 

proposed set of metrics and recommendations for numeric goals for the PATH program. 

These recommended metrics will measure the progress and impact of the PATH program 

in the coming years, and help inform a long-term program assessment.

Goals

Based on NREL programmatic review experience, the new goals are evaluated for 

applicability and appropriateness by answering the following program management 

criteria: 

• Are the goals clear and unambiguous? We find these goals to be reflective of 

the general program vision and useful for an innovation-driven program with a 

public/private partnership structure. For a program of this structure, challenges 

arise from integrating the public and private sector and understanding and 

evaluating the needs of multiple market actors. These goals, even in their general 

nonquantified state, reflect an increased understanding and knowledge of both. 

• Do the goals align with PATH’s capabilities and actions? The goals do align 

PATH’s capabilities and actions because the goals were developed after the start

of the program and after its programmatic capabilities were already defined. A 

potential drawback to defining the goals after the activities were already in place

is that the goals are designed to fit the program actions rather than vice-versa. 

This possible disadvantage can be minimized with the careful selection of metrics 

that are directed toward measuring overall program market impact. 
5

• Are the goals focused on key market drivers? By way of PATH’s

public/private partnership structure, the goals do focus on market drivers. Because 

private industry is involved in the program, PATH administrators have unique 

access and opportunity to impact the market and the market drivers. These goals 

account for this partnership approach to understanding the needs of the market 

actors and how to engage them to influence key market drivers. 

• Do the goals incorporate leveraging with other market actors? Leveraging is 

directly considered through the second goal. Goal 2 acknowledges the need for 

understanding and increasing information dissemination to other market players. 

Despite the program’s limited resources to create market change, this will

leverage indirect program impacts by influencing external actors.

• Are the goals measurable? Measuring the program goals are a critical 

component of an overall assessment of PATH. Program metrics that are clear, 

attributable, feasible, and comparable over time are still in development. 

From a program planning and evaluation standpoint, these revised goals are reasonable 

and sound for a public/private partnership innovation program such as PATH. The new 

goals reflect the NRC recommendations to narrow the program’s focus, and they align 

with PATH’s decision to focus on technology innovation. These goals focus specifically 

on barriers to innovation such as regulatory barriers and lack of information 

dissemination in the residential-housing sector. The goals approach the home-building 

market from an innovation perspective. That is, the goals address market behavior of 

stakeholders in the innovation process and attempt to understand these behaviors in order 

to improve innovation in the market. They incorporate changes resulting from the 

public/private partnership by striving to impact the information regarding innovative 

technologies within and outside of PATH. All three goals strive to improve the 

environment for innovative thinking and focus on the process of innovation rather than 

end results of innovation (e.g., a 50% decrease in home energy use or energy costs). 

Activities

The PATH program has proposed a variety of associated inputs, activities, and outcomes

to meet the above outlined goals and objectives (Table 2). This section presents a 

summary of the activities suggested to meet each of these. It also briefly reviews these

activities for soundness based on the connection of the activities to the goals, the 

uniqueness of the activities related to other government programs, and the 

appropriateness of these activities within the program structure. The remainder of this 

section presents an overview of activities arranged according to the goals, which are 

restated below.

Goal 1 – Remove barriers and facilitate technology development and adoption with 

the following objectives:

a. Identify current and potential barriers and measure their impacts

b. Develop practical methods to overcome current barriers

c. Develop alternative future industrial processes that eliminate barriers
6

The first goal aims to remove barriers to technology development and deployment. The 

first two objectives for Goal 1, a and b, address identifying barriers and developing 

methods to overcome these barriers. The activities associated with the two initial 

objectives address identification and points of origination of barriers and effective ways 

to overcome them. 

The first objective, objective a, of Goal 1 addresses the need for identifying and studying 

the barriers and their impacts. The second objective, b, of Goal 1 considers the results of 

objective a and addresses developing options for overcoming these barriers. Activities

may include pilot programs and training sessions based on the specific barrier being 

addressed. This objective is difficult to apply directly to Goal 1. The recommendation for 

these barrier-reduction activities is to temporally space them behind barrier identification 

and public/private collaboration to determine methods to address the identified barriers. 

The activities associated with the third objective, objective c, of Goal 1, however, 

approach the goals from the perspective of a paradigm shift toward new views of

innovation. Therefore, they are able to be carried out simultaneously to the activities 

associated with the first objective. The impacts of these activities may be difficult to 

measure because, if successful, new industrial processes and methods will completely 

avoid existing barriers instead of having to identify the barriers in order to reduce their 

effects, as in the first two objectives. These activities do provide an opportunity for largescale change in the innovation patterns of the building industry, and therefore fit with the 

overarching aim of Goal 1 to remove barriers. 

Goal 2 – Improve technology transfer, development, and adoption through 

information dissemination with the following objectives:

a. Establish and maintain noncommercial information sources on innovation

b. Understand behaviors, attitudes, and needs regarding innovation information

c. Have relevant information materials on innovation and innovators for different parties

Goal 2, improving technology transfer and adoption, includes activities ranging from 

updating noncommercial information on products and services to outreach involving 

information distribution to relevant builders, owners, and remodelers. Like many 

government programs, the role of providing unbiased information on the innovation 

process as well as existing products and services is a critical part of the public side of the 

program. An Internet review indicates that the government-funded data repository for 

home-building information, ToolBase, is the only free, comprehensive, noncommercial 

source for a wide variety of information for the home-building industry. The ToolBase 

database maximizes the public/private partnership structure of the PATH program 

because it leverages relationships with private industry to gather a wealth of information,

which is then presented in a noncommercial way as a service to the greater homebuilding audience. It should be noted that the ToolBase database can be run 

simultaneously to and support the research and intent of Goal 1, the reduction/removal of 

barriers. 
7

Goal 3 – Advance housing technologies’ research and foster development of new 

technology with the following objectives:

a. Establish sustained public and private R&D investment resources

b. Clarify and assist innovators with processes from technology transfer to market 

penetration

c. Have comparative standards for innovation performance, cost, benefits, and agendas.

The activities associated with Goal 3 involve the final stages of innovation: increasing 

market penetration of innovative products and processes. These activities include 

understanding the market and, in order to turn technology awareness into technology 

transfer that results in market penetration, making that information more readily available 

to innovators through outreach. In addition, actions should be taken to leverage

intergovernmental agency and private company partnerships to encourage R&D funding

– a critical piece of the innovation pipeline. All of the activities listed could accomplish 

the third goal in some aspect, but further refinement will be based on how the activity can 

have measured success against the goal.

In summary, the activities suggested in the PATH strategic plan match the revised goals 

and objectives stated for the program. Given the large number and broad range of 

potential activities, a more refined list of selected activities will need to be made based on 

effectiveness of the activity on meeting a specific goal. To understand the potential 

impacts of each activity, implementers must select a set of effective metrics. The 

following section outlines a multiyear process to identify the most promising metrics for 

measuring PATH program impact on the established goals. This process will result in a 

refined list of activities to carry out the PATH program charter most effectively. 
8

Table 2. PATH Goals, Objectives, and Activities

Goal Objective Proposed Activities

Goal 1: Remove 

barriers and facilitate 

technology 

development and 

adoption

Identify current and potential 

barriers and measure their 

impacts

• Barrier-impact studies

• Barrier-identification forums

• Barrier-reduction collaboration forums

Develop practical methods to 

overcome current barriers

• Study Evaluation Service (ES) process, outreach to International Code 

Council (ICC) jurisdictions to enforce ES reports

• Prototype insurance initiative (same Insurer for manufacturer and builder

• Consumer utility and insurance programs, analysis of R&D tax incentives

• Innovators' training assistance on-site, professional curriculum review

• Prototype innovation marketing with retailers

• Additional valuation studies, development of comparable and installation 

listings' architect spec sheets

• Macroeconomic analysis of building activity and innovation rates

Develop alternative future 

industrial processes that 

eliminate barriers

• Performance-based codes prototypes and advocacy

• Prototype innovation insurance initiative

• Consumer programs advocacy, innovation adoption tax-credit studies

• Full labor-training programs and certifications, curriculum standards advocacy

• Prototype innovation marketing with suppliers

• Professional training requirements on innovation

Goal 2: Improve 

technology transfer, 

development, and 

adoption through 

information 

dissemination

Establish and maintain 

noncommercial information 

sources on innovation

• Tool Base and PATH generated info

• Manufacturer information and market training

• ToolBase and PATH collected info

Understand behaviors, 

attitudes, and needs 

regarding innovation 

information

• Market research agenda forum, research project scoping and funding

• Advanced research project scoping and funding, preliminary data

• Market research dissemination and monitoring

Have relevant information 

materials on innovation and 

innovators for different parties

• Home-buyer outreach

• Home-builder technical outreach

• Home-builder marketing and sales outreach

• Remodeler technical outreach

• Remodeler marketing and sales outreach

• Manufacturer outreach
9

Table 2. PATH Goals, Objectives, and Activities

Goal Objective Proposed Activities

Goal 3: Advance 

housing technologies’ 

research and foster 

development of new 

technology

Establish sustained public 

and private R&D investment 

resources

• National Science Foundation (NSF)-PATH program

• Joint research agendas, federal applied research funds

• Current research findings shared event, joint research agendas

• Collaborative federal/state research funds

• Identify foundations, convene events for agendas, develop preliminary 

projects

• Identify trade associations with research, model research programs; 

preliminary collaboration

• Identify and access venture capital (VC) funds for industrial collaborations for 

large and small manufacturers

• Preliminary research projects with individual firms or collaborations

• Roadmap research projects with individual firms or collaborations

Clarify and assist innovators 

with processes from 

technology transfer to market 

penetration

• PATH-related technology transfer efforts (basic to applied)

• State/local technology transfer outreach

• Network building of independent tech transfer efforts (basic to applied)

• PATH technology scanning

• PATH commercialization tools

• Commercialization center

• R&D capacity evaluation and assistance

• Testing facility clearinghouse and outreach to innovators, PATH field

evaluations

Have comparative standards 

for innovation performance, 

cost, and benefits and 

agendas

• PATH beyond-code performance testing protocols (e.g., durability)

• Testing protocol outreach

• Testing protocol expansion

• PATH roadmaps
10

Metrics 

After the revision of its goals, PATH identified about 50 potential metrics for measuring 

progress and quantifying the impacts of the program. These metrics were developed by 

the PATH program in 2005. This section maps the metrics in terms of measurability and 

applicability to the three goals of the program, and presents a framework for identifying 

the most useful, applicable, and feasible metrics for measuring progress toward those 

goals. Future research will result in further refinement and specific recommended 

numeric goals to gauge the progress of the PATH program.

Due to the slow-change nature of the housing market leading to a long-range nature of 

the goals, the metrics are divided into near term and long term. For summary purposes, 

only the long-term outcome potential metrics are listed in Table 3. The possible metrics 

listed here are preliminary; the following framework will help refine the list, and further 

research and understanding will provide a final list of proposed metrics. These metrics 

will be described in this paper for particular PATH programs – Concept Home, 

ToolBase, Structural Insulated Panels, Advanced Steel Framing, and Insulated Concrete 

Forms – and will include benchmark levels for future evaluation.
11

Table 3. Summary of Complete List of Potential Metrics for Measuring PATH Progress Toward Goals

Goal Objective Potential Metrics

Goal 1: Remove 

barriers and facilitate 

technology 

development and 

adoption

Identify current and potential 

barriers and measure their 

impacts

• Number of measures for determining policy effects on barriers

• Reduced or eliminated number of barriers identified

• Percent of event commitments completed annually

Develop practical methods to 

overcome current barriers

• Number of manufacturers using International Code Council Evaluation 

Service (ICC-ES)

• Number of Innovation Insurance Programs

• Consumer-use rate of incentives, tax-credit studies

• Percent of trained labor/innovation, number of university programs

• Number of innovations marketed by two largest retailers; number of spec 

sheets

• Percent of innovations included in various databases (if applicable)

Develop alternative future 

industrial processes that 

eliminate barriers

• Percent of code provisions drafted in performance basis

• Number of innovations insured under program

• Number of consumer programs, number of tax-credit programs

• Number of trained labor, number of accredited university programs

• Number of innovations marketed by 50 largest suppliers

• Percent of certified professionals in three fields (architecture, appraisal, 

inspection)

Goal 2: Improve 

technology transfer, 

development, and 

adoption through 

information 

dissemination

Establish and maintain 

noncommercial information 

sources on innovation

• ToolBase® use rates

• Number of noncommercial information sources

• Use rates of noncommercial information sources (non-PATH)

Understand behaviors, 

attitudes, and needs 

regarding innovation 

information

• Percentage completed of market research agenda

• Total research funding proportion to sales consistency

• Use of market research data

Have relevant information 

materials on innovation and 

innovators for different parties

• Proportion of home buyers purchasing homes with 5+ PATH technologies

• Proportion of homeowners remodeling with 5+ path technologies

• Proportion of home builders building homes with 5+ PATH technologies

• Proportion of remodelers remodeling homes w/ 5+ PATH technologies

• Proportion of remodelers marketing services with 5+ PATH technologies

• Market shares for all PATH-identified technologies of average share of firm 

sales by innovative products launched in the past 10 years
12

Goal Objective Potential Metrics

Goal 3: Advance 

housing technologies’ 

research and foster 

development of new 

technology

Establish sustained public 

and private R&D investment 

resources

• Total federal funding to academic research

• Total federal funding to applied research

• Annual international exchange events

• Total state/local applied research funding (proportional to sales)

• Total foundation applied research dollars (proportional to sales)

• Total association research projects and funding

• Total private-investment money, average money/technology (proportional to 

sales)

• Total roadmap research matching funding

Clarify and assist innovators 

with processes from 

technology transfer to market 

penetration

• Federal technology transfer funding and programs

• Number of state technology transfer housing programs

• Total independent technology transfer programs

• Number of housing product sales efforts by non-housing firms

• Number of manufacturers using tools

• Percent of manufacturers using R&D assistance services

• Percentage of firms ranked as mid- to high innovators

• Percent of fully tested innovations and market introduction

Have comparative standards 

for innovation performance, 

cost, and benefits and 

agendas

• Number of performance testing protocols

• Percentage of maintained commitments versus commitments made

• Number of non-PATH testing protocols

• Percentage of executed research projects; number of roadmaps
13

Framework for Metric Selection

This section describes a basic framework for narrowing a list of potential metrics (and 

any other suggested metrics) into a set of recommended metrics that measure PATH’s 

progress toward its revised goals. The set of recommended metrics should be considered 

for informing a broader assessment of PATH. As noted in the NRC report, there are no 

ideal assessment processes that can be applied to evaluating programs for advancing 

technologies. Programs need to be considered individually to determine the metrics that 

best respond to the specific program and technologies (NRC 2003). As such, the basic 

framework outlines a process for determining metrics that best represent the progress of 

the program and its specific activities in advancing housing technology and achieving its 

overall goals. The framework follows a four-step process as described below:

1. Develop a simple logic model of the program activities, outputs, and 

outcomes. The logic model translates the program activities and revised goals 

into a diagram that illustrates how these function collectively while being 

measured individually. A vital attribute for any metric is that it show a clear 

relationship to the desired goals and objectives, and a logic model provides the 

conceptual foundation for understanding this relationship. For the overall PATH 

program, potential metrics already have been identified and therefore this step 

may not be necessary. Developing metrics for individual programs within PATH, 

however, would significantly benefit from following this step. 

2. Identify candidate metrics based on the logic model. The logic model 

conceptualizes the activity-goal relationships. By identifying potential metrics 

based on a logic model, the analysis ensures that the metrics are appropriately 

matched with the goals and structure of each program. PATH goals focus on 

technology development, transfer, and adoption; and as a result, its metrics should 

measure these activities throughout the innovation cycle. Mapping potential 

metrics that are aligned with PATH goals and objectives means that the set of 

metrics are more likely to cover both the means (program activities) and the ends 

(program outcomes) and track the movement of technologies through the 

innovation cycle.

3. Apply a scoring system to the candidate metrics based on key 

characteristics. Several key characteristics for determining meaningful metrics 

apply to the PATH evaluation. These characteristics are used as criteria for 

scoring candidate metrics to refine the set into recommended metrics. The 

metrics or performance indicators should be as clear and precise as possible to 

remove ambiguities and ensure that the metric is understood by a wide audience. 

The metrics should, as possible, discern PATH’s influence versus influence from 

external factors. The metrics should be based on quantifiable, reliable data that is 

available at minimal collection cost or effort. The metrics should be trendable

(comparable over time) and enable fair comparisons to previous and/or future 

measurements. These characteristics and the scoring system are explained in 

greater detail later.
14

4. Rebalance and recommend set of metrics. This final step adds expert 

judgment to the recommendation process, which ensures that the objective 

scoring technique in the previous step generates a set of metrics that are 

representative of the breadth of PATH activities and includes measures of 

activities, outputs, and outcomes. If necessary, the set of recommended metrics 

are adjusted or rebalanced so that the various criteria required for such a 

multifaceted program evaluation as PATH are met.

The remainder of this section outlines the methodology and scoring system for refining 

the list of potential metrics based on four key characteristics: 

1. Understandable. Extent to which the metric is able to be understood by external 

stakeholders.

2. Attributable. Ability to accurately determine PATH contribution.

3. Feasible. Available and reliable data.

4. Comparable over time. Ability to make fair comparisons to previous and/or 

future measurements.

Characteristic 1: Understandable. The recommended metrics must clearly articulate 

the progress of PATH internally to decision makers and communicate externally to 

interested stakeholders. This characteristic of understandability to a wide range of 

stakeholders is particularly important for PATH metrics due to its public/private

partnership structure involving both an internal and external audience. This is a 

challenging criterion because external stakeholders are generally of broad backgrounds 

and difficult to classify into an easily identifiable audience. 

Each candidate metric is given a score between “0”and “2” based on the metric’s level of 

understandability as shown in Table 4.

Table 4. Scoring System for “Understandable”

Score Meaning

0 Too complex for internal or external stakeholders to understand.

1 Possibly understandable by stakeholders. Measure may have complicated 

units (e.g., Web hits per application per promotional period).

2 Straightforward metric. Simple units.

Characteristic 2: Attributable. As mentioned earlier, attribution of impact is a 

challenge associated with government programs. The structure of PATH as a 

public/private partnership makes it particularly difficult to distinguish the impact of the 

program from impacts due to external factors (e.g., partners’ activities, natural market 

forces, etc.). The ability to attribute PATH’s influence on the progress toward its goals is 

therefore a key criterion for determining the best set of recommended metrics for PATH. 

PATH has limited direct influence on ultimate outcomes or market penetration of a given 

technology; this will be determined primarily by the technology’s cost and performance 

advantages compared to its competition. Nonetheless, metrics that are able to attribute 

credit to PATH are beneficial in helping inform a long-term program assessment. 
15

These criteria for rating the proposed metrics require assessing whether the PATH impact 

can be separated from the market/private-sector impact. The scoring system is described 

in Table 5. 

Table 5. Scoring System for “Attributable”

Score Meaning

0 Largely unable to decipher whether progress/impacts can be attributed to 

PATH.

1 Partially able to determine PATH’s portion of the progress/impact.

2 Able to clearly distinguish PATH’s contribution.

Characteristic 3: Feasible. Feasibility consists of three characteristics of the data 

supporting the metric: available, reliable, and quantitative. A useful metric obtains

reliable measurements and data. The availability of quantitative data from reliable 

sources presents a major hurdle for the selection of PATH metrics. While the industry 

acknowledges the clear need to use quantitative measures, few sources of such data exist. 

Trade association data, census surveys, and Web site statistics offer the most promising 

leads. The importance of metrics data feasibility imparts a significant challenge and 

substantially reduces the number of metrics that can be used in a PATH assessment.

Table 6 defines the simple scoring system for estimating metric feasibility. The metrics 

with the highest score will be those with easily accessible, reliable, and quantitative data. 

Table 6. Scoring System for “Feasible”

Score Meaning

0 Quantitative data unavailable.

1 Quantitative data partially available. May need to contact multiple sources 

or extrapolate/interpolate gaps in data. Reliability may be questionable 

(e.g., bias risk).

2 Reliable quantitative data publicly available (e.g., U.S. census).

Characteristic 4: Comparable over time. PATH metrics are intended to communicate 

the progress and impacts of the program. To measure accomplishments, there must be 

consistent basis from which to measure. A metric that is comparable over time means 

that the methodology for reporting or manipulating the data does not change. It also 

requires that the metric is relevant from one year to the next. For instance, a one-time 

milestone of publishing a specific technical report is not comparable over time; however, 

the number of technical documents published per year may be comparable. Technologyspecific metrics also risk becoming outdated as new technologies evolve. This 

characteristic is critical for the metrics to be a useful contribution to the long-term 

assessment of PATH.
16

Table 7 outlines the scoring system for the ability of metrics to be compared over time. 

Table 7. Scoring System for “Comparable Over Time”

Score Meaning

0 One-time metric (e.g., meeting a milestone)

1 Comparable for a limited period of time, but environment may change that 

would make the metric obsolete (e.g., measuring specific component of 

technology)

2 Comparable year after year, indefinitely (e.g., measuring technology area)

This system of scoring metrics is developed through market and programmatic evaluation 

knowledge in the home-building and innovation sectors. It is designed to narrow down 

the list of candidate metrics to only those with a high potential for accurately measuring 

the impacts of the PATH program. Based on the combined scores for all four key 

characteristics, the metrics are ranked and considered for recommendation. The final step

involves using expert judgment to ensure that the set of metrics adequately represents the 

program activities, outputs, and outcomes. 

Chapter Summary

The PATH program goal is to advance innovation of technology development in the 

residential-housing sector. PATH’s initial goals, established at the program’s outset in

1998, focused on achieving percentage reductions in housing costs and energy. In 2002, 

an extensive evaluation found these goals to be poorly matched to the program’s role in 

expediting and facilitating the innovation process to impact the market. Consequently, the 

PATH program revised its goals to focus on the innovation process through technology 

development, transfer, and adoption. This report finds that the revised goals are more 

appropriately aligned with the original PATH charter because they more closely match 

the public/private partnership program structure and better target the innovation market, 

rather than the end-user market. These goals attempt to make real market impacts

through continued focus on market adoption of innovative technologies. 

The following chapters apply the framework described above and introduce 

recommended metrics for five individual PATH programs, which were selected partly

based on their prominence in PATH, their longevity, and data availability for these 

programs. 
17

3. Recommended Metrics for PATH’s Concept Home 

Program Background

The U.S. Department of Housing and Urban Development’s Partnership for Advancing 

Technology in Housing (PATH) administers the Concept Home program. This program

demonstrates a vision for the future of American residential home building by 

showcasing advanced technologies and innovative building practices, through the design, 

construction, and marketing of Concept Homes. The technologies and building practices 

offer opportunities for home builders to improve the durability, affordability, safety, and 

energy efficiency of residential housing in the United States.

The Concept Home program was conceived from PATH's “Technology Roadmap: Whole 

House and Building Process Redesign, and Technology Scanning.” The roadmap

concluded that modern homes are “currently built to be inflexible, with systems tangled 

behind interior walls and embedded in structural elements. But the home of the future 

will combine functions that make better use of labor, material, time, and money, 

consequently reducing cost and installation time” (PATH 2004). 

The Concept Home program has two overarching goals: 

• Demonstrate advanced technologies and building practices by creating the first 

whole-house prototype that can be easily altered with options, upgrades, and 

improvements. 

• Showcase Concept Home and promote its specific innovative technologies. 

The whole-house prototype represents an entirely new approach to home design that has 

the potential to reshape the U.S. residential-housing market. In the end, program 

implementers will construct several demonstration Concept Homes, which will each 

highlight a variety of technologies.

Which PATH Goal(s) Does This Program Address?

PATH goals focus on reducing the multiple constraints throughout the innovation 

pipeline in the residential-housing sector, and developing the infrastructure for increasing 

innovation in that pipeline. PATH’s three overall goals are:

• Identify and reduce barriers that impede innovation, including regulatory barriers 

(PATH Goal 1)

• Disseminate information to speed the development and adoption of advanced 

building technologies (PATH Goal 2)

• Advance housing technology research and foster development of new technology 

(PATH Goal 3)

The Concept Home program primarily addresses PATH Goal 2 (see Appendix A for a 

listing of metrics by program and the specific objectives addressed by the metrics.) 

Information on new technologies is disseminated to the housing market using a variety of 

methods, such as open house demonstrations, marketing events and conferences, Web 

pages, news releases, and case studies. 
18

Major Activities

Concept Home aligns its major activities with its overarching goals of technology 

demonstration and information sharing through the following:

Conceiving the Concept Home

PATH identifies core concepts and principles that will be the foundation for deciding 

which innovative technologies to incorporate into the Concept Home. For the first 

Concept Home – Omaha – the vision was to create a home that would be flexible (easy to 

update and expand for different life stages), efficient (easy to build and easy to live in), 

and sustainable (featuring green, energy-efficient products). 

Designing the Concept Home

PATH elicits a builder to contribute to the design of the home, select a site for the home, 

and construct the home. An advisory committee of industry professionals – including the 

selected builder and PATH – selects advanced technologies and building practices to 

include in the demonstration Concept Home. PATH identifies and coordinates with 

product manufacturers to provide the technologies and building materials.

Constructing the Concept Home

PATH provides technical support during the planning and construction phases through 

codes, standards, and certification development for the new technologies and practices. 

PATH does not provide direct financial support for constructing the home. 

Demonstrating and Promoting the Concept Home

A primary focus of the program centers on information dissemination of the home’s 

technologies. PATH produces marketing articles and Web pages about the Concept 

Home and the various technologies it showcases. It also stages promotional events such 

as open houses and walk-throughs of the demonstration home, and works with the media 

to publicize the program and its technologies. The target audience includes builders, 

manufacturers, home buyers, and the general public. 

Continually Improving the Concept Home

Following the construction and demonstration of a Concept Home, a performance 

evaluation is conducted to identify lessons learned that can be applied to the next home. 

For instance, PATH identifies ways to improve cost-effectiveness, durability, safety, 

and/or identifies any technologies that should be added or subtracted from the next 

Concept Home. 

Web Tools for Information Dissemination

PATHnet (http://PATHnet.org/) is PATH’s information Web portal that provides detailed 

information about the program’s ongoing and completed projects. The portal offers 

information on designs, technologies, and product manufacturers, and includes news 

releases and other publications related to Concept Home. 
19

ToolBase (http://www.toolbase.org/) is a PATH-sponsored technical information 

resource for the home-building industry that contains free, comprehensive, 

noncommercial, and un-biased “product descriptions, design & construction guides, best 

practices, performance reports, case studies, and other resources...that builders and 

remodelers will find useful.”1

Measuring the Impact

 Within the ToolBase site, a section called Technology 

Inventory (i.e., TechInventory), includes many or all of the technologies used by the 

Concept Home program. For example, the downloadable TechSpecs document, included 

in the TechInventory section of many of the technologies, provides in-depth information 

on costs, results from field evaluations, and tips for determining whether a specific 

technology is right for a certain kind of project. 

Metrics provide quantitative evidence of the progress and impacts of specific programs. 

Measuring impacts is often more difficult than measuring the progress of a program, but 

both are vital for gauging overall performance. Both types of metrics are discussed in the 

Recommended Performance Metrics section. Two approaches are considered for 

measuring the impacts: 

• Direct measurement of increased market penetration of the specific technologies 

that are included in each of the Concept Homes constructed

• Direct measurement of Concept Home information dissemination, including the 

extent and results of marketing campaigns and measured overall awareness of the 

Concept Home 

Although the first type of metric more directly measures the impacts on the market of the 

specific technologies of PATH Concept Homes, it would require extensive research on 

each of the included technologies. It may not be practical or feasible to collect this type 

of data and conduct the relevant analyses. Market-penetration metrics for each 

technology would be required to assess each technology in each demonstration Concept 

Home. Many of these technologies are in the development stages and the sales data are 

proprietary, which complicates measuring the program impacts. Even if market 

penetration data are available, the data are likely influenced by external factors that make 

attribution to PATH uncertain.

The second measurement type – evaluating the impact of Concept Home marketing 

campaigns to the broad community of builders, designers, manufacturers, home buyers, 

etc. – allows for practical measurement and a useful understanding of program effects 

with respect to PATH’s overall Goal 2: information dissemination. While this measure 

does not reveal direct market impacts of the program, it effectively shows the exposure of

information to the marketplace as a result of PATH efforts.

Key Data Sources

Data for Concept Home metrics are available primarily from two key sources:

 1 “About ToolBase.” <http://www.toolbase.org/about.aspx>.
20

D&R International 

D&R International, PATH’s marketing consultant, provides quantitative data on the 

attendance and frequency of promotional events as well as data on information 

distribution. D&R also supplies information on the number of publications and other 

articles created, and they track the estimated audience reached by such materials. 

NAHB Research Center

The National Association of Home Builders (NAHB) Research Center compiles statistics 

on Web site use of ToolBase, including page views and downloads. 

Potential additional data sources are presented in the Recommended Metrics section.

Approach for Identifying and Recommending 

Performance Metrics

The following approach, which is explained further in Chapter 2, is used for 

recommending performance metrics:

1. Develop a logic model of the Concept Home activities, outputs, and outcomes. 

The logic model translates the program activities and goals described earlier into a 

diagram that illustrates how these function collectively while being measured 

individually (see Figure 1 for logic model).

2. Identify candidate metrics based on the logic model that measure the progress of 

these activities, outputs, and outcomes toward achieving project and program 

goals.

3. Apply the scoring system described in the Conceptual Soundness report to the 

candidate metrics. The scoring system is based on general characteristics of 

useful metrics for the PATH program. Scores are based on four key criteria:

a. Understandable. Extent to which the metric is understood by external 

stakeholders

b. Attributable. Ability to determine PATH contribution

c. Feasible. Availability and reliability of data

d. Comparable over time. Ability to make fair comparisons to previous 

and/or future measurements

4. Tally scores and rebalance. Metrics are selected based on the scores and a 

rebalancing using expert judgment to ensure that the set of recommended metrics 

are representative of the breadth of PATH Concept Home activities and include 

measures of activities, outputs, and outcomes. 
21

Figure 1. Concept Home Logic Model

Recommended Metrics

Recommended metrics and potential data sources for Concept Home are presented in this 

section. Note that additional recommended metrics are proposed in the next section. That 

section represents measure(s) that would complement the current metrics set, but the data 

are not likely to be easily available. Together, these two sections represent a full set of 

recommended metrics that measure both the means and the ends of the program, and 

inform a process for long-term assessment and program improvement. See Appendix B 

for a listing of metrics and their respective scoring considering the attributes discussed in 

the Conceptual Soundness section—understandable, attributable, feasible, and 

comparable over time.

Note that Concept Home is unique from other PATH programs because of the multiyear 

cyclical nature of its activities. That is, one home is built every few years with activities 

purposefully intended to correspond with the timing of the home-building phase. 

Therefore, the metrics should account for the cyclical characteristics of this program. 

Typically, metrics are measured on an annual basis (e.g., average number of Web hits per 

month in a given year). Because the Concept Home cycle operates on a multiyear basis, 

several of the recommended metrics include “cumulative” statistics that cover the 

Idea conceived for 

Concept Home 

Activities

Concept Home is 

constructed

Walk-throughs, 

information dissemination, 

and promotional events

Core Concepts and 

Principles identified

Increased public and 

homebuyer awareness 

of new technologies

Ideas for improving costeffectiveness, quality, 

and adding/removing 

technologies

Increased awareness 

among builders and 

manufacturers regarding 

CH technologies

Specific advanced 

technologies selected to 

be incorporated on the 

Concept Home

Concept Home 

performance evaluation 

conducted and lessons 

applied to next home

Identify lessons learned

Builders and 

manufacturers solicited 

to become partners

Concept Home is 

demonstrated and 

promoted

Concept Home blueprints 

and specifications 

developed

Six principles selected to 

govern the design of the 

Concept Home

Outputs

Manufacturing partners, 

builders, and technologies 

selected

Increased residential 

homebuilding use of one 

or more of the principles 

and CH technologies of 

CH

“Design Guide” and other 

informative and 

marketing materials 

published

Concept Home 

construction completed 

Concept Home designed, 

sited, and land purchased

Accelerated and broader 

advancement of 

residential building 

technologies

“Technology scanning”

helps identify potential 

technologies

Outcomes

Concept Home is 

designed

Technical and marketing 

support

Testing and evaluation of 

the home during all periods 

of construction 

Testing and evaluation of 

the home and technologies 

during all periods of 

construction 
22

lifetime of the program. This technique helps avoid potentially misleading statistics if the 

period is limited to one year. 

Data Source: PATH personnel and pathnet.org. For example, a list of technology 

sponsors for Concept Home Omaha is available at: 

Number of partners and technologies

1) Cumulative number of industry partners (such as designers, builders, 

technology manufacturers) who have participated in Concept Home since the 

inception of the program

This metric measures the program’s ability to engage the home-building 

community and seek out relevant partners to contribute to the development and 

construction of Concept Homes. It measures a key Concept Home activity 

necessary to reach a wide audience about innovative technologies and practices in 

the industry. A possible addition to this metric would incorporate the percentage 

of the market that the industry partners represent, indicating the size of 

participants relative to the industry as a whole. Obtaining market data may be an 

obstacle for this addition and simply measuring the cumulative number of 

partners will be an adequate first-order measure.

http://www.pathnet.org/sp.asp?id=22428&pbg=3

2) Cumulative number of technologies that are promoted by Concept Home and 

included in completed homes since the inception of the Concept Home 

program 

This metric shows the program’s ability to incorporate advanced technologies into 

Concept Home design and construction. It provides an indicator of the extent that 

the program is generating opportunities to demonstrate and share information 

about advanced technologies to the housing market.

Data Source: pathnet.org. For example, the list of technologies for Concept 

Home Omaha is available at: http://www.pathnet.org/sp.asp?id=21935&pbg=3

3) Cumulative potential audience reached through mass marketing and news 

releases (e.g., television promotions, circulation of newspapers containing 

articles about Concept Home)

Information dissemination activities and outputs

This metric measures the extent that the program and its technologies and 

practices are exposed to a wide audience. Note that the potential audience for this 

metric is less likely to fully absorb the information about Concept Home than the 

audience for user-initiated information sources, such as PATHnet. For example, 

only a fraction of the recipients of a newspaper or magazine that covers a wide 

range of topics read a given article. Audience reached for user-initiated sources –

measured in Metrics 4 and 5 – are more likely to retain the information.

Data Source: D&R International
23

4) Cumulative audience reached via in-person information events, such as 

Concept Home promotional events and open houses 

This metric represents the captive in-person audience reached by the Concept 

Home program. While Metric 3 indicates the breadth of marketing coverage, this 

metric indicates the depth of coverage – the audience pursuing specific 

information about the technologies. 

Data Source: D&R International

5) Average monthly number of Web page hits that contain information about 

Concept Home and its technologies on PATH-sponsored Web sites (i.e., 

PATHnet, ToolBase, HUDUser)

Publishing information on PATHnet and ToolBase are key activities for 

disseminating information. Interested parties will frequently use these sites as 

long as these sites are updated with relevant, unbiased, accurate, clear, and 

concise information (NRC 2003). Complementing Metric 4, this metric represents 

the captive audience reached by the Concept Home program by measuring the 

audience pursuing specific information about the technologies. Metrics 3, 4, and 

5 collectively indicate the total audience reached by Concept Home marketing 

efforts. These metrics measure the extent that PATH’s information-dissemination 

activities are reaching various audiences: a wide audience (Metric 3) or a targeted 

audience (Metrics 4 and 5). These metrics also measure the trend over time of 

audience reached.

Data Source: D&R International for pathnet.org and NAHB Research Center for 

ToolBase.org

6) Annual percentage growth in number of Google hits when searching the 

following exact phrase: “Partnership for Advancing Technology in Housing” 

and “Concept Home”

This metric gauges overall public interest in Concept Home by measuring the 

availability of information and reference materials pertaining to the program on 

the Internet. This exact search phrase was considered meaningful because, when 

tested, each search result was relevant to the PATH Concept Home program. 

Google hits are a useful metric because they measure the number of pages 

referencing PATH and Concept Home not only from pathnet.org or toolbase.org, 

but also from external sites either in industry, academia, or government.

Data Source: Google.com
24

Additional Recommended Metrics

7) Annual administrative costs as a percentage of total program expenses

Program administration efficiency

This metric measures how efficiently PATH is operating the Concept Home 

program. 

Data Source: PATH may be able to provide the data for this metric.

Data Source: Measuring the number of new homes using one or more of the 

technologies requires estimating the U.S. new-housing market each year, and the 

technologies included in them. A survey of builders may provide a sufficient 

sample of the percentage of Concept Home technologies in new homes to 

extrapolate estimates for overall market share.

Market penetration outcomes

8) Cumulative revenues of products/technologies that were incorporated into 

Concept Home (real dollars)

Product revenues indicate the extent of an advanced technology being 

incorporated into the home-building market. Dollar values should be adjusted to 

real dollars to account for variances in the producer price index (i.e., inflation).

Data Source: Product revenues potentially can be collected from trade 

associations, or provided directly by the manufacturers.

9) Percentage of new homes in the United States using at least one of the 

technologies showcased in the Concept Home program

This metric gauges how deeply Concept Home technologies are penetrating the 

new residential home market, an ultimate outcome of the program. 
25

4. Recommended Metrics for PATH’s ToolBase Program 

Program Background

The U.S. Department of Housing and Urban Development’s Partnership for Advancing 

Technology in Housing (PATH) Program administers the ToolBase® program, and uses 

ToolBase.org as a portal to provide information to the building community. The 

objective of the ToolBase program is to give builders and remodelers access to the latest 

information on innovative building materials, processes, and systems (PATH 2003a).2

PATH’s involvement in ToolBase began in 1999 when it partnered with the National 

Association of Home Builders (NAHB) Research Center – a separately incorporated, notfor-profit subsidiary of the NAHB. The ToolBase.org Web site was launched in 2001, 

and relaunched with major updates in subsequent years (NAHB 2008). ToolBase.org is a 

Web-based portal that provides technical information to the home-building industry. It 

contains free, noncommercial, and unbiased product descriptions, design and construction 

guides, best practices, performance reports, case studies, and other resources useful for 

builders and remodelers. ToolBase provides stakeholders with quick and easy access to 

objective information, links to suppliers and products, and neutral reviews of specific 

housing techniques (Martin 2003).

 

The program offers a means for disseminating objective residential building technology 

information.

The ToolBase program currently focuses on maintaining and updating its comprehensive 

resource listing of emerging housing technologies on its Web site. Previously, the 

program has provided additional services and enhancements. In July 2007, these 

additional services were indefinitely discontinued due to the funding vehicle for these 

tasks expiring. Prior to July 2007, ToolBase services had included, for example, a tollfree telephone hotline staffed by technical experts to answer questions from builders and 

remodelers; a news service to report the latest building research information directly to 

trade, real estate, and local home-builder association editors; an expansion of its Web

site; and active involvement with program promotions (PATH 1999). 

The ToolBase site continues to offer and maintain technical details primarily in two 

sections: Technology Inventory (i.e., TechInventory) and Field Evaluations. 

TechInventory includes thorough technical information for many or all of the new 

technologies promoted by PATH. These technologies demonstrate great potential for 

improving housing performance, but are not yet widely accepted in the market. Field 

Evaluations provide detailed real-world results from builders who have tested advanced 

building technologies listed in the TechInventory.

 2 “PATH Technology Inventory.” <http://www.pathnet.org/sp.asp?id=1395>.
26

Which PATH Goal(s) Does This Program Address?

PATH’s goals focus on reducing the multiple constraints throughout the innovation 

pipeline in the residential-housing sector, and developing the infrastructure for increasing 

innovation in that pipeline. PATH’s three overall goals are:

 Identify and reduce barriers that impede innovation, including regulatory barriers 

(Goal 1)

 Disseminate information to speed the development and adoption of advanced 

building technologies (Goal 2)

 Advance housing technology research and foster development of new technology 

(Goal 3)

The ToolBase program primarily addresses PATH Goal 2 (see Appendix A for a listing 

of metrics by program and the specific objectives addressed by the metrics). The 

program also addresses Goal 3 through its demonstration activities. The TechInventory 

provides detailed technical information for a variety of emerging housing technologies 

and is available for any interested party to view (Goal 2). PATH also supports Field 

Evaluations, which are described by ToolBase.org as cooperative efforts between 

builders, remodelers, manufacturers, PATH, and the NAHB Research Center that place 

new technologies into homes so they can be evaluated in real-world conditions.3

Major Activities 

 These 

evaluations contribute to Goal 3, while publication of the findings of the nearly 40 Field 

Evaluations that have been completed applies to Goal 2.

When PATH first partnered with the NAHB Research Center, the program integrated the 

ToolBase Web pages with the Web site for the NAHB Research Center 

(www.nahbrc.org). In July 2001, after a major redesign effort, the toolbase.org site was 

launched as a separate Web site. The site again was revamped and relaunched in July 

2006, completely independent from the nahbrc.org Web site, with the ToolBase portal 

design the same that is seen today (McGee 2008). The scope of the program was altered 

in July 2007, although the Web site itself did not significantly change at that time. 

During these different phases, PATH’s involvement varied. 

The program’s major activities can be separated into two parts: pre-July 2007 activities 

and current activities. Prior to July 2007, PATH was able to provide funding to expand 

the Technology Inventory, develop and execute Field Evaluations, promote the ToolBase 

site and its technologies, and make other enhancements to the Web site. When one of the 

funding vehicles expired in July 2007, PATH and its partner NAHB Research Center 

were limited to maintaining the existing content in ToolBase, focusing on the Technology 

Inventory and Field Evaluation sections.

Major activities in PATH’s ToolBase program can be summarized as follows:

 3 “Field Evaluations.” <http://www.toolbase.org/FieldEvaluations/SearchHome.aspx>.
27

Technology Inventory

The TechInventory section of toolbase.org provides builders with a reliable 

source of information on new products or processes in the home-building 

industry. It lists details on each technology and contact information for the 

manufacturers. The TechInventory focuses on new housing industry 

technologies that are in use by 5% or less of the market. Adding information 

about technologies to the TechInventory aims to accelerate the awareness and 

acceptance process.

4,5

Field evaluations are real-world demonstrations to field test PATH 

technologies, provide the foundation for evaluating the new technologies, and 

convey their values to a variety of audiences including builders, remodelers, 

and manufacturers.

 Following the July 2007 change in scope, new 

technologies are no longer added to the inventory, but the TechInventory 

section remains an important section of the ToolBase site.

Field Evaluations

4 PATH uses the TechInventory list to select technologies 

for use in the field evaluations and demonstrations. Similar to the 

TechInventory, no significant new content has been added since July 2007. 

Despite this, the Field Evaluations section remains an important section for 

Web site visitors.

Promotions

Prior to July 2007, site managers used numerous news service promotions to 

increase traffic on the ToolBase site and increase awareness of its 

technologies. They also created a biweekly electronic news service called 

ToolBase E-News, which focused on residential construction industry news 

dealing with information technology and homebuilding.6

4

 Site managers 

established a ToolBase E-News RSS Feed to provide residential construction 

industry news directly to subscriber’s computers. They also produced a 

bimonthly technical print newsletter (ToolBase News) to provide practical 

information that building professionals could use immediately in their 

businesses. It was available free through local and state home-builder 

associations.

ToolBase also promoted its site through television and magazines. For 

example, PATH advertised on cable television stations HGTV and HGTV 

Pro. The program also reached builders and consumers by advertising in the 

magazine Fine Homebuilding. 

Other Enhancements

Other ToolBase enhancements pre-July 2007 included increased accessibility 

of the Web site, extending hours and adding Spanish language capability for 

its telephone hotline, creating a complementary Catalog of Building Products 

 4 “Technology Inventory.” <http://www.toolbase.org/TechInventory/About.aspx>.

5 “About ToolBase.” <http://www.toolbase.org/about.aspx>.

6 “Newsletters.” <http://www.toolbase.org/Newsletters/EnewsHome.aspx>.
28

and Services, and optimizing its ability to be found through Web searches 

(i.e., Web search optimization).

After July 2007, PATH-supported ToolBase activities included maintaining the 

TechInventory and Field Evaluations sections of the Web site. Maintenance activities 

include ensuring that the Web site is “live” and functioning on the Internet, that the links 

are active and not broken, and that technical material and contact information is updated 

and added to the site as it is received (no new materials for the TechInventory and Field 

Evaluations are being developed) (PATH and NAHB 2008).

Measuring the Impact 

Metrics provide quantitative evidence of the progress and impacts of specific programs. 

ToolBase is unique compared to other PATH programs in that its impacts can be 

approximated by home builders’ awareness and understanding of the technologies 

contained on the Web site, rather than by focusing on impact measures such as market 

penetration, which are common in other PATH programs. Direct measures for market 

penetration of the program’s technologies are not as applicable for ToolBase. By 

measuring ToolBase impacts based on awareness and understanding, the decision model 

borrows from technology market diffusion theory to assume the various stages that 

decision makers will go through to adopt a technology (Rogers 2003). The theory states 

that they will go through the following stages: becoming aware of the technology, 

understanding the technology, purchasing the technology, and confirming the decision to 

purchase the technology. Because ToolBase aims only to disseminate information and 

does not focus on attracting builders or consumers to purchase the technologies, the 

impacts can be measured by the first two stages of technology adoption: awareness and 

understanding. For this program, these stages can be approximated by the following 

methods:

1. Direct measurement of awareness of the technologies. Example metrics include 

how often TechInventory or Field Evaluations pages are accessed by unique 

visitors. 

2. Indirect measurement of technology understanding. This can be approximated by 

measuring the number and frequency of visitors returning to the site. Visitors that 

return to the site can indicate that they find the Web site useful and 

understandable.

These methods are intended to establish metrics that approximate the impact of the 

program’s information-dissemination activities. 

Key Data Sources 

Data for ToolBase metrics are available primarily from the NAHB Research Center. The 

center contributes content, hosts and maintains toolbase.org, and compiles statistics on 

use of the ToolBase Web site. Statistics include the number of unique visitors, page 

views, documents downloads, and several others. Potential additional data sources are 

presented in the Recommended Metrics section.
29

Prior to July 2001, ToolBase Web pages were integrated with the Web site for the NAHB 

Research Center, so there are no reliable statistics available for this period. Toolbase.org 

was launched as a separate Web site in July 2001; in 2002, the research center began to 

collect statistics separately for the ToolBase site. The current ToolBase site, launched in 

July 2006, is independent from the NAHBRC.org Web site and has its own set of more 

accurate statistics.

It should be noted that the research center reports that there are still some difficulties in 

gathering exact numbers. Due to the way the site is built, depending on how the user 

found a particular page, the Web statistics software (Urchin) has some difficulties

counting page hits. 

Approach for Identifying and Recommending 

Performance Metrics

The following approach, which is explained further in Chapter 2, is used for 

recommending performance metrics:

1. Develop a logic model of the ToolBase activities, outputs, and outcomes. The 

logic model translates the program activities and goals described earlier into a 

diagram that illustrates how these function collectively while being measured 

individually (see Figure 2 for logic model).

2. Identify candidate metrics based on the logic model that measure the progress of 

these activities, outputs, and outcomes toward achieving project and program 

goals.

3. Apply the scoring system described in the Conceptual Soundness report to the 

candidate metrics. The scoring system is based on general characteristics of 

useful metrics for the PATH program. Scores are based on four key criteria:

a. Understandable. Extent to which the metric is understood by external 

stakeholders

b. Attributable. Ability to determine PATH contribution

c. Feasible. Availability and reliability of data

d. Comparable over time. Ability to make fair comparisons to previous 

and/or future measurements

4. Tally scores and rebalance. Metrics are selected based on the scores and a 

rebalancing using expert judgment to ensure that the set of recommended metrics 

are representative of the breadth of ToolBase activities and include measures of 

activities, outputs, and outcomes. 
30

PATH partners with 

NAHB Research 

Center to develop 

webpages that 

provide information 

about PATH 

technologies

E-News electronic newsletter

Technology Inventory 

information gathered 

and shared

ToolBase News technical 

newsletter

Increased and 

confirmed use of 

PATH technologies

Website is 

strategically promoted 

to audiences

Press releases and public 

service announcements

ToolBase RSS news feeds

Design and Construction 

guides

Construction Methods, 

Homebuilding Topics, 

Buildings Systems, and Best 

Practices

Parties have access 

to and regularly 

receive up-to-date 

technology 

information

Decision-makers 

are aware of and 

understand PATH 

technologies

HGTV regular and pro 

televised advertisements

“Fine Homebuilding”

magazine advertisements

ToolBase optimized for web 

searches

PATH Field Evaluations 

conducted and published

Enhancements 

developed for 

ToolBase

Website designed 

and maintained

Toolbase TechInventory

pages and TechSpecs

created

Website provides 

helpful technical 

information

Intuitive web layout 

implemented

Activities Outputs Outcomes

Idea conceived to 

develop a website 

that provides detailed 

information about 

PATH technologies

Decision-makers 

decide to implement 

technologies in 

homes

Website is userfriendly

Information is 

distributed to 

range of audiences

Figure 2. ToolBase Logic Model

Recommended Metrics

Recommended metrics and potential data sources for ToolBase are presented in this 

section. Note that additional recommended metrics are proposed in the next section. 

That section includes measure(s) that would complement the current metrics set, but the 

data are not likely to be easily available. Together, these two sections represent a full set 

of recommended metrics that measure both the means (activities) and the ends (impacts) 

of the program, and inform a process for long-term assessment and program 

improvement. See Appendix B for a listing of metrics and their respective scoring 

considering the attributes – understandable, attributable, feasible, and comparable over 

time – discussed in Chapter 2.

As noted earlier, the scope of the ToolBase program changed significantly in July 2007 

when one of the two funding vehicles expired. The program previously had been 

involved with developing new content, promotions, and other enhancements. Following 

the expiration of one of the funding vehicles, the program switched to a maintenance

mode that focuses on ensuring the current site is functioning properly but is not actively 

promoting the site or adding significant new content. As a result of this change, the set of 

recommended metrics presented below cover both the historical period (pre-July 2007) 
31

and the current maintenance period. Analysis of the metrics data should consider the 

changing scope of the program.

Recommended metrics for the ToolBase program are organized into three groups: 

- Activities metrics measure the progress of key activities enabling the program to 

meet its mission of information dissemination of objective information on housing 

technologies. A subset of the Activities metrics measure the Design and 

Maintenance of the ToolBase site. 

- Awareness metrics are output/outcome measurements of the impact that the 

program is having on the first stage of technology diffusion (see Measuring the 

Impact of ToolBase section). 

- Understanding metrics provide indicators for the impact of ToolBase on the 

second stage of technology diffusion (see Measuring the Impact of ToolBase). 

These metrics generally use proxies to estimate users’ understanding of the 

ToolBase content.

1) Cumulative number of technologies featured in the TechInventory

This metric provides a historical and ongoing count of technologies featured in 

the TechInventory section of the ToolBase site. The metric measures the 

program’s aptness in adding technologies to the ToolBase Web site and enabling 

the broader mission of information dissemination about innovative technologies 

and practices in the housing industry. The TechInventory is a major focus of the 

ToolBase site, and the number of technologies in the inventory measures a key 

activity necessary to reach a wide audience. 

Data Source: NAHB Research Center

2) Cumulative number of Field Evaluations completed with results published 

on ToolBase

This metric provides a historical and ongoing count of the number of field 

evaluations whose results are featured on ToolBase. Field Evaluations and 

TechInventory are the top two areas where users can find technical information on 

the site, and the two areas the program would like to direct site visitors.

This metric provides an indicator for the program’s ability to include content on 

the site that is necessary for information dissemination about innovative building 

technologies and practices.

Data Source: NAHB Research Center

Activities

Web site design and maintenance

3) Total number of TechInventory and Field Evaluations page views per total 

number of user visits/sessions in each month
32

The metric indicates the ability of toolbase.org to direct users to the two areas of 

the site that provide the most information: the TechInventory and Field 

Evaluations sections. A large ratio for this metric would indicate that the design 

of the Web site is successfully bringing visitors to the TechInventory and Field 

Evaluations pages. Note that data are collected for only the top few (2 to 10, 

depending on the month of data collection) pages within each section, and these 

top pages can be used as a proxy for the total number of views of the sections. 

The total number of user visits includes the number of times the ToolBase site is 

accessed in a given month. If a visitor leaves the site and returns in the same 

month, it is counted as a separate visit. This metric measures output from the sitedesign activities.

Data Source: NAHB Research Center

4) Number of broken links in toolbase.org

This metric provides an indicator for the maintenance of the ToolBase site. In 

general, URLs of Web sites anywhere on the Internet can frequently change, and 

this affects the sites that link to those pages. If a URL for a Web page changes, 

then any Web site linking to the former URL will have a “broken link.” To fix 

the broken link, the new URL must be programmed into the Web site. Web sites 

that are actively maintained have few broken links. Broken links for any public 

Web site can be counted and monitored with free Web software, such as Xenu.

Data Source: Xenu or other similar software

6) Total number of toolbase.org page views per month

Awareness of ToolBase technologies 

5) Total number of unique visitors to ToolBase.org per month

This metric indicates the number of unique visitors to the toolbase.org site in a 

given month. A unique visitor means that if a person visits ToolBase more than 

once in a single month, then it is counted as only one unique visitor. Web 

statistics software from the NAHB Research Center is able to distinguish between 

a “unique” visitor and a repeat visitor by recording the IP address of the user and 

determining whether the same IP address had already logged into the site during 

that month. The total number of unique visitors should reflect any promotional 

and marketing efforts to attract new users to the site.

Data Source: NAHB Research Center 

This metric indicates the number of times visitors access a page on the 

toolbase.org site in a given month. A user often will access multiple pages during 

a single visit, and “page views” counts each page that the user sees. The total 

number of page views reflects the ability to attract and retain visitors to the site. 

Note that the number of page views is only comparable over time when the site 
33

format and structure is constant. For instance, if the Web site is reconfigured to 

include more material on fewer Web pages, then comparing the number of page 

views before and after the reconfiguration would not be meaningful.

Data Source: NAHB Research Center 

7) Total number of TechInventory page views per month

This metric provides an estimate of how many times a key section of the 

ToolBase Web site has been viewed. TechInventory and Field Evaluations 

(Metric 8) are the two sections of the site that provide important housing 

technology information. Compare Metrics 7 and 8 to Metric 3. Metric 3 indicates 

how well the site is directing users to the technical information sections, while 

Metrics 7 and 8 indicate the total number of page views of their respective

sections. Metrics 7 and 8 also should be considered in conjunction with Metric 9, 

which normalizes the number of Web hits based on the housing market – a key 

factor that is external to program influence. Note that data for TechInventory is 

available for only the top few (2 to 10, depending on the month of data collection) 

pages within in each section, and these top few pages will be used as a proxy for 

the total number in that section. The metric provides input into the target 

audience’s level of awareness of the technical information offered in ToolBase. 

Data Source: NAHB Research Center

8) Total number of Field Evaluations page views per month

Similar to Metric 7, this metric estimates the number of times a key section of 

ToolBase has been viewed. A Field Evaluation section that is relevant, up-todate, and provides useful information for builders should receive a sustained high 

number of page views. Data for Field Evaluations may be available only for the 

top (based on page views) few pages. If this is the case, these will be used as a 

proxy for the total number of Field Evaluation page views per month. In 

combination with Metrics 6, 7, and 9, this metric offers insight into the level of 

awareness of the ToolBase technologies.

Data Source: NAHB Research Center

9) Total number of visitors (“visitor sessions”) accessing toolbase.org per 

square feet of all new residential-housing units in the United States per 

month

This metric provides a count of the total number of user visits (or “visitor 

sessions”) to the ToolBase.org site per month – i.e., the number of times any 

person visits the ToolBase site – as a fraction of total square footage of new 

houses being built. This metric is useful in two ways: It represents the total 

number of times the site is accessed, including repeat visitors, and 2) it normalizes 

the traffic on the ToolBase site by accounting for the fluctuating housing market. 

It is important to account for key external factors, such as new housing units, so 

that attribution from program efforts can be correctly measured – rather than 
34

measuring factors outside the program’s control. Note that “user visits” are 

different than “page views,” which provide a count of every page within the site 

that is accessed. User visits count how many visits occur in a given month, i.e., 

how many visitors (not necessarily unique visitors) access the site. The metric 

offers insight into how well the site is attracting new and repeat visitors, 

accounting for changes in the residential-housing market – and provides input for

evaluating overall awareness of ToolBase technologies. 

Data Source: NAHB Research Center

10) Average number of page views per session

Understanding ToolBase technologies

This metric shows the number of page views per total number of visitor sessions. 

It is calculated as “total number of page views” (Metric 6) divided by “total 

number of visitor sessions.” The metric provides an indicator for the level of 

engagement that the user has with the site during any given visit to ToolBase. A 

larger number of page views per session can indicate that the user is finding 

multiple sources of useful information. Because ToolBase is a source for 

technical information and the Web site is structured such that multiple page views 

would typically be required to gather substantial details about a technology, it is 

expected that a user viewing many pages is likely to attain greater understanding 

of the content. Conversely, a user that views very few pages may not understand 

the material he/she is viewing or is not interested in further detail. Note that a 

reconfiguration of the Web site would require recalibrating the comparison of 

page views over time (see explanation for Metric 6).

Data Source: NAHB Research Center

11) Average number of repeat visits per month (“average sessions per visitor in a 

given month”)

This metric shows the number of times, on average, a visitor returns to the 

toolbase.org site in the same month. The metric is calculated as “total number of 

visitor sessions” divided by “total number of unique visitors” (Metric 5). For 

example, if the average sessions per visitor equals “2,” this indicates that the 

average ToolBase user visits twice per month; if average sessions per visitor 

equals “1,” this indicates that the average user visits just once (i.e., on average,

unique visitors do not return to the site during that month). Repeat visits provide 

an indicator that users understand the information, find the Web site useful, and 

are therefore returning back to the site. Together with Metric 10, this metric offers 

insight into understanding the stage of technology diffusion.

Data Source: NAHB Research Center
35

Additional Recommended Metrics

14) Percentage of home builders who are aware of the technologies in ToolBase, 

understand the technologies, decide use the technologies, and confirm their 

use of the technologies

Enhanced services activities

12) Total number of promotional events per year

This metric measures the number of PATH-sponsored promotional events related 

to ToolBase. The program currently is not able to conduct promotional events, 

but has done so historically. Measuring the total number of promotional events 

per year provides demonstrable evidence of efforts to increase awareness of the 

site and encourage interested parties to use the resources available on it. This 

metric could be used in conjunction with the Activities metrics (Metrics 1–4) to 

provide additional input for a program assessment. Promotional event data are

not currently available. 

Possible Data Source: NAHB Research Center

13) Total number of news subscribers – including subscribers to E-news, RSS 

feeds, and print newsletter mailings

Similar to Metric 12, this metric provides insight into the program’s promotional 

activities, which are no longer part of the ToolBase program. The metric offers 

an output indicator for the promotional activities that encourage builders to visit 

ToolBase.

Possible Data Source: NAHB Research Center

With ample resources, a survey of home builders could be conducted to provide 

data regarding the effectiveness of ToolBase information accelerating technology 

adoption. Program impacts could be measured from the survey data regarding the 

stages of the technology diffusion, as described earlier. Survey responses for 

“awareness,” “understanding,” “deciding to use the technology,” and “confirming 

its use” can be evaluated for builders who have accessed ToolBase and compared 

with those who do not visit the site. The difference between these data sets would 

offer useful input for measuring the impact of the program on the market. 

Possible Data Source: Survey of home builders
36

5. Recommended Metrics for PATH’s Structural Insulated 

Panels Program

Program Background

The U.S. Department of Housing and Urban Development’s Partnership for Advancing 

Technology in Housing (PATH) administers the Structural Insulated Panels (SIPs) 

program. This program seeks to accelerate the adoption of SIPs in the residentialhousing market. Compared to conventional wood framing materials, SIPs are thermally 

more efficient; they offer competitive cost, higher strength, improved fire resistance, and 

are made of more environmentally friendly materials. Despite these benefits, homes 

constructed using SIPs account for only a small fraction of the U.S. residential-housing 

market. 

Since PATH’s inception, the program has sought to accelerate adoption of technologies 

that provide an alternative to conventional wood framing. SIPs, Insulated Concrete 

Forms (ICFs), and steel framing were three technologies selected as the most likely 

alternatives to wood that would be able to penetrate the market. For all three of these 

technologies, PATH sought to identify the critical barriers impeding greater adoption into 

the marketplace. PATH identified regulatory barriers and lack of available information 

about the products as key obstacles to widespread deployment. The programs focused on 

removing the regulatory barriers and increasing dissemination of information regarding 

how the technologies work, their cost, and the value in their use. 

Which PATH Goal(s) Does This Program Address?

PATH goals focus on reducing the multiple constraints throughout the innovation 

pipeline in the residential-housing sector, and developing the infrastructure for increasing 

innovation in that pipeline. PATH’s three overall goals are:

 Identify and reduce barriers that impede innovation, including regulatory barriers 

(PATH Goal 1)

 Disseminate information to speed the development and adoption of advanced 

building technologies (PATH Goal 2)

 Advance housing technology research and foster development of new technology 

(PATH Goal 3)

The SIPs program addresses PATH Goals 1 and 2 (see Appendix A for a listing of 

metrics by program and the specific objectives addressed by the metrics). Program 

activities aligning with Goal 1 include identifying barriers during the development of the 

SIPs Prescriptive Methods document and working with organizations to address the 

regulatory hurdle of having SIPs adopted into building codes.

PATH posts specific technical information about SIPs on its toolbase.org Web site. 

Pathnet.org and huduser.org also offer information about SIPs to interested parties, 

helping the program accomplish PATH’s Goal 2.
37

Major Activities

At the outset of the SIPs program, PATH collaborated with the Structural Insulated Panel 

Association (SIPA), APA – The Engineered Wood Association, and SIP manufacturers 

across the United States to identify critical barriers impeding wider market uptake of 

structural insulated panels. Regulatory barriers and lack of available information about 

SIPs were determined to be the key barriers that PATH could address. Examples of 

regulatory barriers include the lack of prescriptive construction guidelines and the 

absence of SIPs in residential building codes. As guidance to help overcome these 

barriers, the program developed a “Prescriptive Method for Structural Panels” that 

establishes performance standards for SIPs and shares technical information about the 

technology, offering specific guidelines to facilitate the use of SIPs in wall systems for 

the construction of one- and two-family dwellings (PATH 2006a).

The Prescriptive Method and specifications were submitted to the International Code 

Council (ICC) for adoption into the International Residential Code (IRC) – a set of 

building codes commonly implemented by states and counties. The ICC develops codes 

used to construct residential and commercial buildings, including homes and schools. 

Most U.S. cities, counties, and states that adopt codes choose the International Codes 

developed by the ICC. In May 2007, the ICC voted to adopt SIPs into the IRC, and 

shortly thereafter they were included as a supplement to the 2006 IRC (SIPA 2007). The 

adoption of SIPs into the IRC addresses a major regulatory barrier to the widespread use 

of SIPs in residential housing (Goal 1). SIPs are a code-recognized option and “home 

builders using SIPs will no longer be required to conduct additional engineering to show 

equivalency to the IRC” (NAHB 2007). 

Marketing and information-dissemination activities of the SIPs program support efforts to 

increase development and adoption of the technology (Goal 2). The program broadly 

follows three points to achieve this goal.

 Communicate the technology’s technical soundness, establishing that 

performance of SIPs is equivalent, if not superior, to competing wood 

alternatives. PATH’s testing and evaluation activities support the dissemination of 

technical figures (PATH 2006b). 

 Convey costs of the technology in specific housing installations. The program 

encourages builders to select building materials and technologies based on value 

rather than defaulting to industry status quo. 

 Articulate benefits of using SIPs. PATH offers numerous resources that have 

been published or offered on its Web sites that seek to clearly express the value of 

using SIPs to the builder and customer.

Web Tools for Information Dissemination

PATHNET (http://PATHnet.org/) is PATH’s information Web portal that provides 

information on numerous housing technologies, including SIPs. The portal offers 

information on designs, technologies, and product manufacturers, and includes other 

publications related to SIPs. 
38

ToolBase (http://www.toolbase.org/) is a PATH-sponsored technical information 

resource for the home-building industry that contains free, comprehensive, 

noncommercial, and unbiased “product descriptions, design & construction guides, best 

practices, performance reports, case studies, and other resources...that builders and 

remodelers will find useful.”7

Measuring the Impact

 Within the ToolBase site, a section called Technology 

Inventory includes sections devoted to SIPs. This section of the site provides in-depth 

information on costs, results from field evaluations, and tips for determining whether 

SIPs are appropriate for certain kinds of projects. 

Metrics provide quantitative evidence of the progress and impacts of specific programs. 

Measuring impacts is often more difficult than measuring the progress of a program, but 

both are vital for gauging overall performance. Both types of metrics are discussed in the 

Recommended Performance Metrics section. Two approaches are considered for 

measuring the impacts: 

1. Measurement of increased market penetration of SIPs using a proxy, such as SIPs 

production data from manufacturers

2. Direct measurement of information dissemination, such as number of page views 

of the ToolBase Technology Inventory on SIPs

The first type of metric provides insight into the extent that SIPs have penetrated the 

market. Although it is difficult to measure exactly how many residential units are 

constructed using SIPs each year, production figures may provide a reasonable estimate. 

A complete profile of the technology’s market penetration would include the percentage 

of homes incorporating SIPs and details into the distribution of the technology, such as 

among large and small homes and geographic location. Production figures may be 

difficult to obtain if SIPs are produced by manufacturers who are not members of the 

Structural Insulated Panel Association (SIPA) (See Data Sources section). As SIPs 

become mainstream, it also would be expected that more companies will offer SIPs, thus 

making production estimates more challenging.

Before SIPs can attain significant market penetration, builders must be confident that the 

technology will perform adequately, be cost-effective, and be safe. Energy efficiency 

improvements provide an additional benefit. This program has taken on the role of 

disseminating this information to the residential-housing sector. The second type of 

metric allows for practical measurement and a useful understanding of program effects 

from information-dissemination efforts (PATH Goal 2). While this measure does not 

reveal direct market impacts of the program, it effectively shows the exposure of 

information to the marketplace as a result of PATH efforts.

Key Data Sources

Data for SIPs metrics are available primarily from two key sources:

 7 “About ToolBase” <http://www.toolbase.org/about.aspx>.
39

Structural Insulated Panel Association (SIPA)

SIPA provides SIPs production data and data about the number of housing units supplied 

with SIPs from member companies. 

NAHB Research Center

The National Association of Home Builders (NAHB) Research Center compiles statistics 

on Web site use of ToolBase, including page views and downloads.

Note that PATHnet pages for SIPs were developed as a resource for users interested in 

the SIPs Prescriptive Method rather than as a resource for increasing broad public 

knowledge or awareness of the technology. The Prescriptive Method and the relevant 

studies that contributed to the creation of this method were developed and conducted 

explicitly to be submitted to the International Code Council. Therefore, it is not expected 

that PATHnet page views or downloads would necessarily be an appropriate indicator of 

the program’s overall information-dissemination activities. On the other hand, ToolBase 

is specifically geared toward sharing technical information with the home-building 

community. Page views and downloads from ToolBase will more accurately reflect 

interest and awareness on the topic.

Approach for Identifying and Recommending 

Performance Metrics

The following approach, which is explained further in Chapter 2, is used for 

recommending performance metrics:

1. Develop a logic model of the SIPs program activities, outputs, and outcomes. The 

logic model translates the program activities and goals described earlier into a 

diagram that illustrates how these function collectively while being measured 

individually (see Figure 3 for logic model).

2. Identify candidate metrics based on the logic model that measure the progress of 

these activities, outputs, and outcomes toward achieving project and program 

goals.

3. Apply the scoring system described in the Conceptual Soundness report to the 

candidate metrics. The scoring system is based on general characteristics of 

useful metrics for the PATH program. Scores are based on four key criteria:

a. Understandable. Extent to which the metric is understood by external 

stakeholders

b. Attributable. Ability to determine PATH contribution

c. Feasible. Availability and reliability of data

d. Comparable over time. Ability to make fair comparisons to previous 

and/or future measurements

4. Tally scores and rebalance. Metrics are selected based on the scores and a 

rebalancing using expert judgment to ensure that the set of recommended metrics 

are representative of the breadth of PATH SIPs-related activities and include 

measures of process (i.e., activities and outputs) and end results (i.e., outcomes). 
40

Figure 3. SIPs Logic Model

Recommended Metrics

Recommended metrics and potential data sources for the SIPs program are presented in 

this section. Note that additional recommended metrics are proposed in the next section. 

That section includes measure(s) that would complement the current metrics set but 

where the data are not likely to be easily available. Together, these two sections 

represent a full set of recommended metrics that measure both the means and the ends of 

the program, and inform a process for long-term assessment and program improvement. 

See Appendix B for a listing of metrics and their respective scoring considering the 

attributes discussed in the Conceptual Soundness section – understandable, attributable, 

feasible, and comparable over time.

One of the challenges in evaluating this PATH program involves differentiating between 

increases in SIP adoption due to PATH activities and increases in SIP adoption due to 

external factors. It is also important that data be considered in the context of the U.S. 

home-building industry at the time of the measurement. Measured values may fluctuate 

with changes in demand in the housing market. These complicating factors have been 

considered when selecting the recommended metrics described in this section, and the 

metrics are tailored where possible to account for these factors.

Collaboration with the 

Structural Insulated 

Panel Association 

(SIPA), The Engineered 

Wood Association 

(APA), and SIP industry 

partners

Activities

Identification of critical 

barriers impeding 

greater market 

penetration of SIPs

Further work done to 

expand understanding of 

SIPs and increase 

information 

dissemination

“Prescriptive Method for 

Structural Insulated 

Panels (SIPs) Used In 

Wall Systems In 

Residential Construction”

published (Jun. 2006)

SIPs adopted into the 

2006 International 

Residential Code (IRC) 

(May 2007)

Building code changes, 

prescriptive specifications, 

and installation details 

make building with SIPs 

easier

Increased use of SIPs in 

residential housing

ToolBase TechSpecs

downloadable document 

TechInventory page created/updated for SIPs

created for SIPs and 

added to ToolBase.org

PATH Case Studies and Guides published, for example:

 “SIPs – The Homebuilding Industry’s Hybrid” (Feb. 2006)

 “SIPs Below Ground: Delivering Comfort Where It's Least 

Expected” (Mar. 2006) 

 “Steel SIPs: Hybrid Combines the Strength of Steel with 

the Speed of SIPs” (Aug. 2006)

“Panelized Wall 

Systems: Making the 

Connections” published 

(Aug. 2006)

PATH Field Evaluations

SIPs showcased in 

various PATHsponsored 

“Technologies in 

Practice”

Demonstrations

Outputs Outcomes
41

Note that SIPs, Insulating Concrete Forms (ICFs), and Advanced Steel Framing are all 

similar PATH programs. All three technologies represent alternatives to conventional 

wood options, and the PATH activities for these programs all focus on removing 

regulatory barriers through building-code changes and disseminating information about 

the value of the technology. Because of these similarities, the recommended metrics are 

intended to be consistent and comparable across all three similar programs.

Building code-related outputs 

1) Percentage of new homes in the United States that are built in a location that 

has adopted the International Residential Code (IRC) or other building 

codes that include SIPs

This metric measures the ability of the SIPs program and its partners to gain 

approval of SIPs in widely adopted residential building codes. Because the metric 

measures “percent of new homes,” it emphasizes building codes adopted in 

regions that are constructing a large number of new homes. This metric aligns 

with the program goal of removing regulatory barriers that inhibit greater market 

penetration of SIPs. 

Data Source: U.S. Census publishes the total number of new residential units built 

per year. International Code Council (http://www.iccsafe.org/) lists states and 

jurisdictions adopting certain building codes.

Information dissemination activities and outputs

2) Number of unique visitors to the ToolBase SIPs Technology Inventory home

page

Metric 2 indicates the number of interested parties referencing PATH’s ToolBase 

Web site for information specifically pertaining to SIPs. The information 

provided on this site is technical and tailored for an audience of home builders, 

designers, manufacturers, and others in the housing industry rather than the 

general public. The site offers industry professionals reliable information about 

performance, cost, and value of the technology. Measuring the “unique visitors”

estimates the number of different people accessing the information (as opposed to 

“page views,” which counts the number of times a site is visited, even if it is 

accessed multiple times by the same person). This measure provides a better 

indication of the breadth of audience that PATH is reaching. The URL for the 

ToolBase SIPs Technology Inventory home page is 

http://www.toolbase.org/Technology-Inventory/Whole-House-Systems/structuralinsulated-panels.

Data Source: NAHB Research Center
42

3) Average monthly downloads of the SIPs TechSpec (from ToolBase.org site)

This metric, which complements Metric 2, approximates the circulation of a key 

SIPs program document. The document articulates detailed information about the 

performance, cost, and value in PDF format that can be saved as a separate file. 

The number of downloads indicate the level of interest in the technology from 

building industry professionals, and indicate the ability of PATH to attract 

interested parties to download a key document. 

Data Source: NAHB Research Center

4) Annual percentage growth in the number of Google hits when searching the 

following exact phrase: "Partnership for Advancing Technology in Housing" 

and "Structural Insulated Panels" or "SIPS" 

This metric gauges overall public interest in the SIPs program by measuring the 

availability of information and reference materials pertaining to the program on 

the Internet. This exact search phrase was considered meaningful because, when 

tested, each search result was relevant to the PATH SIPs program. Google hits 

are a useful metric because they measure the number of pages referencing PATH 

and SIPs not only from pathnet.org or toolbase.org, but also from external sites 

either in industry, academia, or government.

Data Source: Google.com

Market penetration outputs and outcomes

This metric provides another estimate of market penetration and approximates the 

range of coverage of SIPs in new homes. This metric should be evaluated in 

combination with Metric 5 and Metric 7 to obtain a better profile of the market 

progress of the technology. While Metric 5 measures whether SIPs are an option 

5) Number of home builders offering SIPs 

This metric estimates the market penetration of SIPs in the building industry, and 

should be evaluated in conjunction with Metric 6 and Metric 7. It measures the 

ability of an interested home buyer to select the SIPs technology for use in their 

new house. An increased number of builders offering SIPs can be a leading 

indicator of increased adoption of the technology. A possible addition to this 

metric would incorporate the percentage of the market that these home builders 

represent, indicative of the market share of these builders. Obtaining this market 

data may be an obstacle; simply measuring the number of builders will be an 

adequate first-order measure.

Data Source: Structural Insulated Panel Association (SIPA). Although there may 

be other builders who offer SIPs, the number of SIPA home-builder members will 

be used as a proxy.

6) Percentage of new homes in the United States using SIPs 
43

offered by builders, Metric 6 measures whether builders are actually 

implementing SIPs in their homes. Compare this to Metric 7, which indicates the 

total production of SIPs but not the extent of broad market penetration.

Data Source: SIPA can provide the annual number of housing units supplied with 

SIPs. U.S. Census publishes the total number of new residential units built per 

year.

7) SIPs annual production for residential use (in square feet) per square feet of 

all new residential-housing units in the United States per year

Metric 7 estimates the total production of SIPs as a fraction of total square footage 

of new houses being built. Because the housing market fluctuates significantly 

from year to year, it is important to measure SIPs production compared to overall 

new home construction. This metric, together with Metric 5 and Metric 6, offer 

insights into the market penetration of the technology. Particularly useful will be 

comparisons of these three metrics over time – assessing the rate of improvement 

or acceleration of the technology into the marketplace.

Data Source: SIPA collects this data in an annual production survey submitted by 

all SIP manufacturers. U.S. Census publishes data necessary to compute the total 

square footage of new residential units built per year.

Metrics 5, 6, and 7 are also useful for evaluating the program retrospectively. 

Specifically, the impact of including SIPs in residential building codes can be 

quantified. The use of SIPs in residential-housing units before and after the

incorporation of SIPs into the International Residential Code will gauge the extent 

that this regulatory barrier has been removed. 

The market penetration attributable to PATH is difficult to separate from the 

market penetration from external factors. Metrics 1 through 4 are more directly 

correlated with specific PATH activities. Regardless, the ultimate objectives of 

the SIPs program align with the market-penetration metrics recommended here 

and are meaningful for evaluating the success of the program. 

Additional Recommended Metrics

8) Number of states that have greater than 0.5% of new residential-housing 

units built using SIPs

This metric captures the geographic distribution of the technology. Some states 

may have significant market penetration, while others are lagging. For example, 

SIPs use may be clustered in one or two states. This metric could highlight states 

with significant opportunity for SIPs market growth. A half percent was chosen 

as the cutoff because, according to an analysis of SIPA and U.S. Census data, the 

national average of SIPs in new homes for 2004, 2005, and 2006 was roughly 

0.5%.

Data Source: SIPA and U.S. Census data. State-level data for housing units with 

SIPs may be difficult to obtain.
44

6. Recommended Metrics for PATH’s Steel Framing 

Program

Program Background

The U.S. Department of Housing and Urban Development’s Partnership for Advancing 

Technology in Housing (PATH) administers the Steel Framing program. This project 

seeks to accelerate the adoption of steel framing in the residential-housing market. 

Compared to conventional framing materials, steel offers the highest strength-to-weight 

ratio of any building material. Steel frames are noncombustible, and they do not rot, 

warp, split, or change shape with moisture content. They offer price stability and 

consistent material quality (no regional variations).8

Which PATH Goal(s) Does This Program Address?

 Despite the advantages that steel 

framing offers to both home builders and consumers, homes constructed using steel 

framing account for only a small fraction of the U.S. residential framing market.

PATH seeks to accelerate adoption of technologies that provide an alternative to 

conventional wood framing. Steel Framing, Insulating Concrete Forms, and Structural 

Insulated Panels were three technologies identified at PATH’s inception as the most 

likely alternatives to wood that would be able to penetrate the residential-housing market. 

PATH identified critical barriers impeding greater adoption of these technologies into the 

marketplace. The three programs focus on addressing these critical barriers, including 

working with codes organizations to address regulatory barriers and increasing 

dissemination of information regarding how the technology works, its cost, and the value 

in its use.

PATH goals focus on reducing the multiple constraints throughout the innovation 

pipeline in the residential-housing sector, and developing the infrastructure for increasing 

innovation in that pipeline. PATH’s three overall goals are:

 Identify and reduce barriers that impede innovation, including regulatory barriers 

(Goal 1)

 Disseminate information to speed the development and adoption of advanced 

building technologies (Goal 2)

 Advance housing technology research and foster development of new technology 

(Goal 3)

The Steel Framing program primarily addresses PATH Goals 1 and 2 (see Appendix A 

for a listing of metrics by program and the specific objectives addressed by the metrics.) 

Program activities aligning with Goal 1 include identifying barriers during the 

development of steel framing prescriptive methods and other technical support 

documents, and working with organizations to address the regulatory hurdle of having 

steel framing adopted into building codes.

 8 “About Steel Framing.” < http://www.steelframing.org/sfa_aboutsteelframing.shtml>.
45

Helping the program accomplish Goal 2, PATH posts specific technical information 

about steel framing on its toolbase.org Web site. Pathnet.org and huduser.org also offer 

information about steel framing to interested parties. 

Prior to 2003, the Steel Framing program conducted research to demonstrate that steel is 

an acceptable alternative to wood in residential housing. This research early in the 

program aligns with PATH’s current Goal 3.

Major Activities

The Steel Framing program can be separated into two phases. Phase 1 involved research 

and demonstration on the acceptability of steel framing material as an alternative to 

wood. Phase 2 of the program works to develop a broader market for steel framing 

through published documents and information dissemination.

During Phase 1, PATH entered a cooperative agreement with the National Association of 

Home Builders (NAHB) and the American Iron and Steel Institute (AISI) to identify 

critical barriers impeding wider market uptake of steel framing. Regulatory barriers and 

lack of available information about steel framing were determined to be the key barriers 

that PATH could address. Examples of regulatory barriers include the lack of 

prescriptive construction guidelines and applications, and the absence of steel framing in 

residential building codes. As guidance to help overcome these barriers, the program 

developed prescriptive methods that established performance standards for steel framing 

and shared technical information about the technology.

The “Prescriptive Method for Residential Cold-Formed Steel Framing” was published in 

1997, and was based on three years of HUD-sponsored research that incorporated input 

from experts in the fields of construction, research, engineering, and building-code 

enforcement (HUD 1997a). This document and companion publications offered practical 

recommendations to builders and code officials on the application and performance of 

steel framing (HUD 1997b). In 2002, PATH partnered with the Steel Framing Alliance 

(SFA) (formerly, the North American Steel Framing Alliance) and funded the 

Manufactured Housing Research Alliance (MHRA) to produce the technical support 

document, “Design for a Cold-Formed Steel Framed Manufactured Home.” (PATH 

2002). The document was developed in conjunction with trade associations, builders, 

testing laboratories, and manufacturers and assessed the viability of substituting steel for 

wood as the structural skeleton of homes built under the HUD manufactured-home 

standards. 

These early documents represent an earlier phase of steel framing research (completed in 

2002) that demonstrated steel as an acceptable framing material under the performancebased HUD standards (HUD 2003). These methods were adopted by the International 

Code Council (ICC) into the International Residential Code (IRC 2000 and 2003), which 

is commonly incorporated into local and state building codes.9

 9 “ToolBase Technology Inventory Steel Framing.” <

 The adoption of steel 

http://www.toolbase.org/TechnologyInventory/Whole-House-Systems/steel-framing>.
46

framing into the IRC addresses a major regulatory barrier to the widespread use of steel 

framing in residential housing, allowing construction of site-built framed steel homes 

without the certification of a professional engineer (NAHB 2003).

The second phase of the program involved developing steel framing to the point of 

viability in a wider market. In 2003, PATH partnered with the SFA to examine and 

report on corrosion of galvanized fasteners used in cold-formed steel, and to produce 

“Hybrid Wood and Steel Details – Builders Guide,” a report that provides information for 

builders to construct hybrid cold-formed steel and wood homes (PATH 2006c and 

2003b). Also in 2003, the “Prescriptive Method for Connecting Cold-Formed Steel 

Framing to Insulating Concrete Form Walls in Residential Construction” was published 

to facilitate the construction of houses from insulating concrete forms and cold-formed 

steel, both of which are important PATH technologies (PATH 2003c). PATH also 

partnered with MHRA to explore the potential of steel framing for the construction of 

factory-built homes. This research into developing a wider market for steel frames 

critically assessed and refined the use of light-gauge steel design in the factory 

environment. 

The Steel Framing program marketing and information-dissemination activities support 

efforts to increase development and adoption of the technology. The program broadly 

follows three key messages for these activities:

• Technical soundness. Establish that performance of steel framing is equivalent, if 

not superior, to competing alternatives. 

• Costs. Provide objective estimates of the installation, operation, and maintenance 

costs in specific housing applications. 

• Benefits. Encourages builders and customers to consider the numerous benefits 

of using steel framing in their homes.

Web Tools for Information Dissemination

PATHNET (http://pathnet.org/) is PATH’s information Web portal that provides 

information on numerous housing technologies, including steel framing. The portal 

offers information on designs, technologies, and product manufacturers, and includes 

other publications related to steel framing. 

ToolBase (http://www.toolbase.org/) is a PATH-sponsored technical information 

resource for the home-building industry that contains free and unbiased product 

descriptions, design and construction guides, best practices, performance reports, case 

studies, and other resources for builders and remodelers.10

 10 “About ToolBase” <http://www.toolbase.org/about.aspx>.

 Within the ToolBase site, the

section called Technology Inventory includes sections devoted to steel framing. This 

section of the site provides in-depth information on costs, results from field evaluations, 

and tips for determining whether steel framing is appropriate for certain kinds of projects. 
47

Measuring the Impact 

Metrics provide quantitative evidence of the progress and impacts of specific programs. 

Measuring impacts is often more difficult than measuring the progress of a program, but 

both are vital for gauging overall performance. Recall that the metrics themselves do not 

determine how well a program is performing. Metrics provide insight into program 

performance and help inform a long-term program evaluation; comprehensive program 

evaluation should be accompanied with setting ambitious targets and time frames that are 

measured by the metrics. Both types of metrics – progress (activities, outputs) and 

impact (outcome) metrics – are discussed in the Recommended Performance Metrics 

section. Two approaches are considered for measurement: 

• Measurement of increased market penetration of steel framing technology using 

proxies, such as steel framing production data, steel frames supplied to housing 

units, market share of steel frames in residential building, and others.

• Direct measurement of information dissemination, such as number of page views 

of the ToolBase Technology Inventory on steel framing.

The first type of metric provides insight into the extent that steel frames have penetrated 

the market. Although it is difficult to measure exactly how many residential units are 

constructed using steel framing each year, production figures may provide a reasonable 

estimate. A complete profile of the technology’s market penetration would include the 

percentage of homes incorporating steel and details into the distribution of the 

technology, such as among large and small homes as well as geographic location. It 

would be expected that residential steel frames production correlate with use, such that 

the number of housing units supplied with steel frames would approximate the number of 

housing units constructed with steel. Production figures, however, may themselves be 

difficult to obtain if steel frames are produced by manufacturers who are not members of 

the Steel Framing Alliance (SFA) (see Data Sources section). In addition, as steel frames 

become mainstream, it would be expected that more manufacturers will produce steel 

frames, thus making production estimates more challenging.

The second type of metric allows for practical measurement and a useful understanding 

of program effects from information-dissemination efforts. Before steel framing can 

attain significant market penetration, builders must be confident that the technology will 

perform adequately, be cost-effective, and be safe. Energy efficiency improvements 

provide an additional benefit. This program has taken on the role of disseminating such 

information on steel framing to builders, consumers, and other interested parties in the 

residential-housing sector. While this measure does not reveal direct market impacts of 

the program, it effectively shows the exposure of information to the marketplace as a 

result of PATH efforts.
48

Key Data Sources

Data for Steel framing metrics are available primarily from two key sources:

Steel Framing Alliance (SFA)

The Steel Framing Alliance is a trade association that encourages the widespread use of 

cold-formed steel framing in residential construction. SFA may be able to provide 

relevant industry data, including data on steel framing manufacturing shipments to the 

residential industry. SFA may also provide market trends data for both single-family 

site-built and multifamily site-built homes including tons of light-gauge steel framing 

used and units built. SFA membership lists may provide useful data about the number of 

steel framing residential builders, suppliers/distributors, architects, and related 

manufacturers such as stud, truss, and component fabricators (SFA 2007 and SFA 

2003).

11

Approach for Identifying and Recommending 

Performance Metrics

NAHB Research Center

The National Association of Home Builders (NAHB) Research Center compiles statistics 

on use of the ToolBase Web site, including page views and downloads.

Note that PATHnet pages for steel framing were developed as a resource for users 

interested in the Steel Framing Prescriptive Methods and other reports rather than as a 

resource for increasing broad public knowledge or awareness of the technology. The 

Prescriptive Methods and the relevant studies that contributed to the creation of these 

methods were developed and conducted explicitly to be submitted to the International 

Code Council. Therefore, it is not expected that PATHnet page views or downloads 

would necessarily be an appropriate indicator of the program’s overall informationdissemination activities. On the other hand, ToolBase is specifically geared toward 

sharing technical information with the home-building community. Page views and 

downloads from ToolBase will more accurately reflect interest and awareness on the 

topic.

The following approach, which is explained further in Chapter 2, is used for 

recommending performance metrics:

1. Develop a logic model of the steel framing program activities, outputs, and 

outcomes. The logic model translates the program activities and goals described 

earlier into a diagram that illustrates how these function collectively while being 

measured individually (see Figure 4 for logic model).

2. Identify candidate metrics based on the logic model that measure the progress of 

these activities, outputs, and outcomes toward achieving project and program 

goals.

 11 “About Steel Framing Alliance.” <http://www.steelframing.org/sfa_aboutus.shtml>.
49

3. Apply the scoring system described in the Conceptual Soundness report to the 

candidate metrics. The scoring system is based on general characteristics of 

useful metrics for the PATH program. Scores are based on four key criteria:

a. Understandable. Extent to which the metric is understood by external 

stakeholders

b. Attributable. Ability to determine PATH contribution

c. Feasible. Availability and reliability of data

d. Comparable over time. Ability to make fair comparisons to previous 

and/or future measurements

4. Tally scores and rebalance. Metrics are selected based on the scores and a 

rebalancing using expert judgment to ensure that the set of recommended metrics 

are representative of the breadth of PATH steel framing-related activities and 

include measures of process (i.e., activities and outputs) and end results (i.e., 

outcomes). 
50

Figure 4. Steel Framing Logic Model

Recommended Metrics

Recommended metrics and potential data sources for the steel framing program are 

presented in this section. Note that additional recommended metrics are proposed in the 

next section. That section includes an additional measure that complements the current 

metrics set, but the data are not likely to be easily available. Together, these two sections 

represent a full set of recommended metrics that measure both the means and the ends of 

the program, and inform a process for long-term assessment and program improvement. 

See Appendix B for a listing of metrics and their respective scoring considering the 

attributes discussed in the Conceptual Soundness section – understandable, attributable, 

feasible, and comparable over time.

One of the challenges in evaluating this program involves differentiating between 

increases in steel framing adoption due to PATH activities and increases in steel framing 

use due to external factors. This is particularly challenging for public/private 

partnerships such as PATH where cooperative activities include both HUD-sponsored 

elements and private-sector elements. The partnership structure creates challenges but

does not preclude the use of meaningful outcome metrics that incorporate components of 

PATH and components of external activities from providing useful input toward a 

PATH collaborates with 

the Steel Framing 

Alliance (SFA)

Activities

Identification of critical 

barriers impeding 

greater market 

penetration of steel 

framing

Further work done to 

expand understanding of 

steel framing and 

increase information 

dissemination

“Prescriptive Method for 

Residential Cold-Formed 

Steel Framing” and 

“Commentary on the 

Prescriptive Method”

published (1997)

Steel framing adopted into 

the 2000 and 2003 

International Residential 

Codes (IRC)

The code change, 

prescriptive specifications, 

and installation details 

make building with steel 

framing easier

Increased use of steel 

framing in residential 

housing

TechSpecs 

downloadable document 

created for steel framing 

and added to 

ToolBase.org

TechInventory page created 

for steel framing and added to 

ToolBase.org

PATH Case Studies and Guides published, for example:

 “Screw Corrosion Study”

 “Hybrid Wood and Steel Details – Builder’s Guide”

 “Steel Framing Alternatives for Manufactured Housing”

“Design for a Cold-Formed 

Steel Framed Manufactured 

Home” published (2002)

PATH Field Evaluations

Steel framing showcased in 

various PATH-sponsored 

“Technologies in Practice”

Demonstrations

Cooperative agreements 

with the SFA and NAHB 

to investigate various 

aspects of steel framing 

as an alternative to wood

“Prescriptive Method for 

Connecting Cold-Formed 

Steel Framing to ICF Walls in 

Residential Construction”

published (2003)

“Steel Framing Prototype 

Development: Final 

Report” published (2003)

Outputs Outcomes
51

comprehensive program evaluation. It is also important that data be considered in the 

context of the U.S. home-building industry at the time of the measurement. Measured 

values may fluctuate with changes in demand in the housing market. These complicating 

factors have been considered when selecting the recommended metrics described in this 

section, and the metrics are tailored where possible to account for these factors.

Note that Steel Framing, Structural Insulated Panels, and Insulating Concrete Forms are 

all similar PATH programs. All three technologies represent alternatives to conventional 

wood options, and the PATH activities for these programs all focus on removing 

regulatory barriers through building-code changes and disseminating information about 

the value of the technology. Because of these similarities, the recommended metrics are 

intended to be consistent and comparable across all three similar programs.

Building code-related outputs 

1) Percentage of new homes in the United States that are built in a location that 

has adopted the International Residential Code (IRC) or other building 

codes that include steel framing

This metric measures the ability of the Steel Framing program and its partners to 

gain approval of the technology in widely adopted residential building codes. 

Because the metric measures “percent of new homes,” it emphasizes building 

codes adopted in regions that are constructing a large number of new homes. This 

metric aligns with the program goal of removing regulatory barriers that inhibit 

greater market penetration of steel framing. 

Data Sources: U.S. Census publishes the total number of new residential units 

built per year. International Code Council (http://www.iccsafe.org/) lists states 

and jurisdictions adopting certain building codes.

Metric 2 indicates the number of interested parties referencing PATH’s ToolBase 

Web site for information specifically pertaining to steel framing. The information 

provided on this site is technical and tailored for an audience of home builders, 

designers, manufacturers, and others in the housing industry rather than the 

general public. The site offers industry professionals reliable information about 

performance, cost, and value of the technology. Measuring the “unique visitors”

estimates the number of different people accessing the information (as opposed to 

“page views,” which counts the number of times a site is visited, even if it is 

accessed multiple times by the same person). This measure provides a better 

indication of the breadth of audience that PATH is reaching. The URL for the 

ToolBase steel framing Technology Inventory homepage is 

Information-dissemination activities and outputs

2) Number of unique visitors to the ToolBase Steel Framing Technology 

Inventory homepage
52

http://www.toolbase.org/Technology-Inventory/Whole-House-Systems/steelframing.

Data Source: NAHB Research Center

3) Average monthly downloads of the Cold-Formed Steel Framing TechSpec 

(from toolbase.org site)

This metric, which complements Metric 2, approximates the circulation of a key 

steel framing program document. This document is an important resource for 

builders who are considering making the switch to cold-formed steel. Document 

downloads estimate the level of interest among the building community and 

provide a leading indicator of market penetration. The metric also gauges the 

ability of PATH to attract interested parties to download a key document. The 

URL for the TechSpecs document for steel framing is: 

http://www.toolbase.org/pdf/techinv/steel_framing_techspec.pdf

Data Source: NAHB Research Center

4) Annual percentage growth in the number of Google hits when searching the 

following exact phrase: "Partnership for Advancing Technology in Housing" 

and "Steel Framing" 

This metric gauges overall public interest in the Steel Framing program by 

measuring the availability of information and reference materials pertaining to the 

program on the Internet. This exact search phrase was considered meaningful 

because, when tested, each search result was relevant to the PATH Steel Framing 

program. Measuring the absolute number of hits for this metric is not as 

important as measuring how the number of hits changes over time (i.e., the annual 

percentage growth). Google hits are a useful metric because they measure the 

number of pages referencing PATH and Steel Framing not only from pathnet.org 

or toolbase.org, but also from external sites either in industry, news media, 

academia, or government. Together with Metrics 2 and 3, they provide an 

objective method to measure this program’s information-dissemination activities.

Data Source: Google.com

Market penetration outputs and outcomes

Metric 5 estimates the total production of steel framing as a fraction of total 

square footage of new houses being built. Because the housing market fluctuates 

significantly from year to year, it is important to measure steel framing production 

compared to overall new-home construction. This metric, together with Metric 6 

and Metric 7, offer insights into the market penetration of the technology. 

Particularly useful will be comparisons of these three metrics over time –

5) Steel framing annual production for residential use (in square feet) per 

square feet of all new residential-housing units in the United States per year
53

assessing the rate of improvement or acceleration of the technology into the 

marketplace.

Data Sources: Steel Framing Alliance (SFA) collects data on annual shipments to 

the residential industry and can be used as a proxy for annual production. U.S. 

Census publishes data necessary to compute the total square footage of new 

residential units built per year.

6) Number of home builders offering steel framing 

This metric estimates the market penetration of steel framing in the building 

industry, and should be evaluated in conjunction with Metric 5 and Metric 7. It 

measures the ability of an interested home buyer to select steel framing for their 

new home. An increased number of builders offering steel framing can be a 

leading indicator of increased adoption of the technology. A possible addition to 

this metric would incorporate the percentage of the market that these home

builders represent, indicative of the market share of these builders. Obtaining this 

market data may be an obstacle; simply measuring the number of builders will be 

an adequate first-order measure.

Data Source: Steel Framing Alliance (SFA). Although there may be builders who 

offer steel framing that are not members of SFA, the number of SFA homebuilder members can be used as a proxy for the total number of builders offering 

the technology.

7) Percentage of new homes in the United States using steel framing 

This metric provides another estimate of market penetration and approximates the 

range of coverage of steel framing in new homes. This metric should be evaluated 

in combination with Metric 5 and Metric 6 to obtain a better profile of the market 

progress of the technology. While Metric 6 measures whether steel framing is an 

option offered by builders, Metric 7 measures whether builders are actually 

implementing steel framing in their homes. Compare this to Metric 5, which 

indicates the total production of steel framing but not the extent of broad market 

penetration. Recent data do not appear to be available online for a number of new 

homes with steel framing. If these data are not available, overall market share of 

steel frames in terms of mass (tons) compared to the entire framing market in tons 

could be considered as an alternate metric.12

 12 “Data and Statistics.” <http://www.steelframing.org/sfa_datastatistics.shtml>.

Data Sources: SFA may be able to provide data on the number of new homes 

built using steel frames. U.S. Census provides data on the number of new 

residential-housing units built each year.

Metrics 5, 6, and 7 are also useful for evaluating the program retrospectively. 

Specifically, the impact of including steel framing in residential building codes 

can be quantified. The use of steel framing in residential-housing units before and 

after the incorporation of steel framing into the International Residential Code 

will gauge the impact of addressing this regulatory barrier. 
54

The market penetration attributable to PATH is difficult to separate from the 

market penetration from external factors. Metrics 1 through 3 are more directly 

correlated with specific PATH activities. Regardless, the ultimate objectives of 

the Steel Framing program align with the market-penetration metrics 

recommended here and are meaningful for evaluating the successes of the 

program. 

Additional Recommended Metrics

8) Number of states that have greater than 1.5% of new residential-housing 

units built using steel framing

This metric captures the geographic distribution of the technology. Some states 

may have significant market penetration, while others are lagging. Steel framing 

use may be clustered in a few states, for example. Implementers chose 1.5% as 

the baseline because the technology had a 1.5 % market share of the residentialhousing sector in 2002.13

Data Source: SFA may be able to provide a breakdown by state of new homes 

built with steel framing. The total number of new homes is available from U.S. 

Census.

 A greater number of states exceeding the 2002 average 

market penetration for new homes indicates greater geographic distribution. This 

metric should be used in conjunction with Metrics 5, 6, and 7, which indicate 

overall market penetration. As overall market penetration increases, Metric 8 

should be expected to also increase. Discrepancies among the trends in Metrics 5, 

6, and 7 and Metric 8 can inform an analysis of geographic distribution as part of 

a broader evaluation of market penetration. 

 13 Steel Framing Alliance. “Data and Statistics.” 

<http://www.steelframing.org/sfa_datastatistics.shtml>.
55

7. Recommended Metrics for PATH’s Insulating Concrete 

Forms Program

Program Background

The U.S. Department of Housing and Urban Development’s Partnership for Advancing 

Technology in Housing (PATH) administers the Insulating Concrete Forms program. 

This program seeks to accelerate the adoption of Insulating Concrete Forms (ICFs) in the 

residential-housing market. Compared to conventional framing materials, ICFs have high 

R-values, low air infiltration, and high thermal mass; these attributes can reduce home 

energy use by 25% to 50% compared to wood- or steel-framed homes. In addition, ICFs 

reduce sound transmission and are structurally durable and cost-competitive.14

Which PATH Goal(s) Does This Program Address?

 Despite 

these benefits, homes constructed using ICFs account for only a small fraction of the U.S. 

residential-housing market. 

PATH seeks to accelerate adoption of technologies that provide an alternative to 

conventional wood framing. At PATH’s inception, three technologies were selected as 

the most likely alternatives to wood that would be able to penetrate the market: ICFs, 

Structural Insulated Panels (SIPs), and Steel Framing. PATH identified critical barriers 

impeding greater adoption of the technology into the marketplace. The ICFs program 

focused on addressing these critical barriers, including working with building-code 

organizations to address regulatory barriers and increasing dissemination of information 

regarding how the technology works, its cost, and the value in its use.

PATH goals focus on reducing multiple constraints throughout the innovation pipeline in 

the residential-housing sector, and developing the infrastructure for increasing innovation 

in that pipeline. PATH’s three overall goals are:

 Identify and reduce barriers that impede innovation, including regulatory barriers

(Goal 1)

 Disseminate information to speed the development and adoption of advanced 

building technologies (Goal 2)

 Advance housing technology research and foster development of new technology 

(Goal 3)

The ICFs program primarily addresses PATH Goals 1 and 2 (see Appendix A for a listing 

of metrics by program and the specific objectives addressed by the metrics.) Activities 

aligning with Goal 1 include developing an ICFs Prescriptive Methods document that 

identifies key barriers impeding innovation, and working with organizations to address 

the regulatory hurdle of incorporating ICFs into building codes. 

 14 “Benefits - Easy To Build With and Easy To Love: Advantages of ICFs.” 

<http://www.forms.org/?act=benefits>.
56

Supporting Goal 2, PATH posts specific technical information about ICFs on its 

toolbase.org Web site. Pathnet.org and huduser.org also offer links and information 

about ICFs to interested parties.

Major Activities

At the outset of the ICFs program, PATH collaborated with the NAHB Research Center 

and the Portland Cement Association (PCA) to identify critical barriers impeding greater 

market penetration of insulating concrete forms (ICFs). Regulatory barriers and lack of 

available information about ICFs were determined to be the key barriers that PATH could 

address. Examples of regulatory barriers include the lack of a consistent and 

comprehensive set of prescriptive requirements, such as prescriptive construction 

guidelines, and the absence of ICFs in residential building codes (PATH 1998). Lack of 

information includes technical guidance for home builders and general marketing of the 

technology for a wider audience.

To help overcome regulatory barriers, the program developed a prescriptive method titled 

“Prescriptive Method for Insulating Concrete Forms in Residential Construction,” which 

established performance standards for ICFs and shared technical information about the 

technology. This first report was published in May 1998 and was accepted in 2000 by 

the International Code Council (ICC) for adoption into the International Residential Code 

(IRC) – a set of building codes commonly adopted by states and counties. The ICC 

develops codes used to construct residential and commercial buildings, including homes 

and schools. Most U.S. cities, counties, and states that adopt codes choose the 

International Codes developed by the ICC. In January 2002, the Prescriptive Method was 

revised based on user feedback and expanded to include provisions for seismic design, 

wall construction requirements using Grade 60 reinforcing steel, and concrete mixes with 

selected compressive strengths (HUD 2002). 

ICFs program marketing and information-dissemination activities support efforts to 

increase adoption of the technology. The program broadly follows three key messages 

for its information-dissemination activities:

• Technical soundness. Establish that performance of ICFs is equivalent, if not 

superior, to competing alternatives. 

• Costs. Provide objective estimates of the installation, operation, and maintenance 

costs in specific housing applications. 

• Benefits. Encourage builders and customers to consider the numerous benefits of 

using ICFs in their homes.

HUD PATH has published several documents to convey these messages, including:

• “Prescriptive Method for Insulating Concrete Forms in Residential Construction”

• “Insulating Concrete Forms for Residential Construction”

• “In-Plane Shear Resistance of Insulating Concrete Form Walls”

• “Insulating Concrete Forms Lintel Tests”

• “Prescriptive Method for Connecting Cold-Formed Steel Framing to Insulating 

Concrete Form Walls in Residential Construction”
57

Web Tools for Information Dissemination

PATHnet (http://PATHnet.org/) is PATH’s information Web portal that provides 

information on numerous housing technologies, including ICFs. The portal offers 

information on designs, technologies, and product manufacturers, and includes other 

publications related to ICFs. 

ToolBase (http://www.ToolBase.org/) is a PATH-sponsored technical information 

resource for the home-building industry that contains free and unbiased product 

descriptions, design and construction guides, best practices, performance reports, case 

studies, and other resources for builders and remodelers.15

Measuring the Impact 

 Within the ToolBase site, the

section called Technology Inventory includes sections devoted to ICFs. This section of 

the site provides in-depth information on costs, results from field evaluations, and tips for 

determining whether ICFs are appropriate for certain kinds of projects. 

Metrics provide quantitative evidence of the progress and impacts of specific programs. 

Measuring impacts is often more difficult than measuring the progress of a program, but 

both are vital for gauging overall performance. The metrics themselves do not determine 

how well a program is performing, but rather provide insight into program performance 

and help inform a long-term program evaluation; comprehensive program evaluation 

should be accompanied with setting ambitious targets and time frames that are measured 

by the metrics. Both types of metrics – progress (activities, outputs) and impact 

(outcome) metrics – are discussed in the Recommended Performance Metrics section. 

Two approaches are considered for measurement: 

• Measurement of increased market penetration of ICFs using proxies, such as ICFs

production data, ICFs supplied to housing units, market share of ICFs in 

residential building, and others.

• Direct measurement of information dissemination, such as number of page views 

of the ToolBase Technology Inventory on ICFs

The first type of metric provides insight into the extent that ICFs have penetrated the 

market. Although it is difficult to measure exactly how many residential units are 

constructed using ICFs each year, production figures may provide a reasonable estimate. 

A complete profile of the technology’s market penetration would include the percentage 

of homes incorporating ICFs and details into the distribution of the technology, such as 

among large and small homes and geographic location. It would be expected that ICFs 

production correlate with use, such that the number of housing units supplied with ICFs 

would approximate the number of housing units constructed with ICFs. Production 

figures, however, may themselves be difficult to obtain if ICFs are produced by 

manufacturers who are not members of the Insulating Concrete Forms Association 

(ICFA) (see Data Sources section). In addition, as ICFs become mainstream, it would be 

 15 “About ToolBase” <http://www.toolbase.org/about.aspx>.
58

expected that more manufacturers will produce ICFs, thus making production estimates 

more challenging.

The second type of metric allows for practical measurement and a useful understanding 

of program effects from information-dissemination efforts. Before ICFs can attain 

significant market penetration, builders must be confident that the technology will 

perform adequately, be cost-effective, and be safe. Energy efficiency improvements 

provide an additional benefit. This program has taken on the role of disseminating such 

information on ICFs to builders, consumers, and other interested parties in the residentialhousing sector. While this measure does not reveal direct market impacts of the program, 

it effectively shows the exposure of information to the marketplace as a result of PATH 

efforts.

Key Data Sources

Data for ICFs metrics are available primarily from three key sources:

Insulating Concrete Form Association (ICFA)

The Insulating Concrete Form Association (ICFA) (http://www.forms.org/) is an 

organization “dedicated to expanding the use of insulating concrete forms through 

research, education, promotion, and public affairs work” (PCA 2008). ICFA may be able 

to provide industry data, such as annual shipment reports showing production trends. 

ICFA membership lists also may provide useful proxy data indicating number of system 

manufacturers, distributors, contractors, suppliers, etc.

Portland Cement Association (PCA) 

The Portland Cement Association (http://www.cement.org/) is an organization that 

represents cement companies in the United States and Canada and conducts market 

development, engineering, research, education, and public affairs programs. PCA offers 

relevant articles about ICFs and a list of known producers (PCA 2008).16

Note that PATHnet pages for ICFs were developed as a resource for users interested in 

the ICFs Prescriptive Methods (first and second editions) rather than as a resource for 

increasing broad public knowledge or awareness of the technology. The Prescriptive 

Methods and the relevant studies that contributed to the creation of these methods were 

developed and conducted explicitly to be submitted to the International Code Council. 

Therefore, it is not expected that PATHnet page views or downloads necessarily would 

be an appropriate indicator of the program’s overall information-dissemination activities. 

On the other hand, ToolBase is specifically geared toward sharing technical information 

 PCA may have 

data related to ICFs production or market data. 

National Association of Home Builders (NAHB) Research Center

The National Association of Home Builders (NAHB) Research Center compiles statistics 

on Web site use of ToolBase, including page views and downloads. 

 16 “Portland Cement Association” <http://www.cement.org/pca/>
59

with the home-building community. Page views and downloads from ToolBase will 

more accurately reflect interest and awareness on the topic.

Approach for Identifying and Recommending 

Performance Metrics

The following approach, which is explained further in Chapter 2, is used for 

recommending performance metrics:

1. Develop a logic model of the ICFs program activities, outputs, and outcomes. 

The logic model translates the program activities and goals described earlier into a 

diagram that illustrates how these function collectively while being measured 

individually (see Figure 5 for logic model).

2. Identify candidate metrics based on the logic model that measure the progress of 

these activities, outputs, and outcomes toward achieving project and program 

goals.

3. Apply the scoring system described in the Conceptual Soundness report to the 

candidate metrics. The scoring system is based on general characteristics of 

useful metrics for the PATH program. Scores are based on four key criteria:

a. Understandable. Extent to which the metric is understood by external 

stakeholders

b. Attributable. Ability to determine PATH contribution

c. Feasible. Availability and reliability of data

d. Comparable over time. Ability to make fair comparisons to previous 

and/or future measurements

4. Tally scores and rebalance. Metrics are selected based on the scores and 

rebalanced using expert judgment to ensure that the set of recommended metrics 

are representative of the breadth of PATH ICFs-related activities and include 

measures of process (i.e., activities and outputs) and end results (i.e., outcomes). 
60

Figure 5. ICFs Logic Model

Recommended Metrics

Recommended metrics and potential data sources for the ICFs program are presented in 

this section. Note that additional recommended metrics are proposed in the next section. 

That section includes an additional measure that complements the current metrics set, but 

the data are not likely to be easily available. Together, these two sections represent a full 

set of recommended metrics that measure both the means and the ends of the program, 

and inform a process for long-term assessment and program improvement. See Appendix 

B for a listing of metrics and their respective scoring considering the attributes discussed 

in the Conceptual Soundness section – understandable, attributable, feasible, and 

comparable over time.

One of the challenges in evaluating this program involves differentiating between 

increases in ICF adoption due to PATH activities and increases in ICF adoption due to 

external factors. This is particularly challenging for public/private partnerships such as 

PATH where cooperative activities include both HUD-sponsored elements and privatesector elements. The partnership structure creates challenges but does not preclude the 

use of meaningful outcome metrics that incorporate components of PATH and 

components of external activities from providing useful input toward a comprehensive 

program evaluation. It is also important that data be considered in the context of the U.S. 

home-building industry at the time of the measurement. Measured values may fluctuate 

with changes in demand in the housing market. These complicating factors have been 

Collaboration with NAHB 

Research Center and 

the Portland Cement 

Association (PCA)

Activities Outputs

PATH Field Evaluations

Identification of critical 

barriers impeding 

greater market 

penetration of ICFs

Further work done to 

expand understanding of 

ICFs and increase 

information 

dissemination

Several informative documents published, for example: 

 “ICFs for Residential Construction: Demonstration 

Homes”

 “Cost and Benefits of ICFs for Residential Construction”

 “ICFs: Installed Cost and Acoustic Performance“

 “ICFs: Comparative Thermal Performance”

 “In-Plane Shear Resistance of Insulating Concrete Form 

Walls”

“Prescriptive Method for 

Insulating Concrete Forms 

in Residential 

Construction” published 

(in 1998) and expanded 

(in 2002)

ICFs adopted into the 

2000 International 

Residential Code (IRC)

IRC code change, 

prescriptive specifications, 

and installation details 

make building with ICFs 

easier

Increased use of ICFs in 

residential housing

Outcomes

TechInventory page on 

ToolBase.org

created/updated for ICFs

ICFs showcased in 

various PATH-sponsored 

“Technologies in Practice”

Demonstrations
61

considered when selecting the recommended metrics described in this section, and the 

metrics are tailored where possible to account for these factors.

Note that ICFs, Structural Insulated Panels, and Steel Framing are all similar PATH 

programs. All three technologies represent alternatives to conventional wood options, 

and the PATH activities for these programs all focus on removing regulatory barriers 

through building code changes and disseminating information about the value of the 

technology. Because of these similarities, the recommended metrics are intended to be 

consistent and comparable across all three similar programs.

Building code-related outputs 

1) Percent of new homes in the United States that are built in a location that has 

adopted the International Residential Code (IRC) or other building codes 

that include ICFs

This metric measures the ability of the ICFs program and its partners to gain 

approval of ICFs in widely adopted residential building codes. Because the 

metric measures “percent of new homes,” it emphasizes building codes adopted in 

regions that are constructing greater numbers of new homes. This metric aligns 

with the program goal of removing regulatory barriers inhibiting greater market 

penetration of ICFs. 

Data Sources: U.S. Census publishes the total number of new residential units 

built per year. International Code Council (http://www.iccsafe.org/) lists states 

and jurisdictions adopting certain building codes.

Information dissemination activities and outputs

2) Number of unique visitors to the ToolBase ICFs Technology Inventory home

page

This metric indicates the number of interested parties referencing PATH’s 

ToolBase Web site for information specifically pertaining to ICFs. The 

information provided on this site is technical and tailored for an audience of home

builders, designers, manufacturers, and others in the housing industry rather than 

the general public. The site offers industry professionals reliable information 

about performance, cost, and value of the technology. Measuring the “unique 

visitors” estimates the number of different people accessing the information (as 

opposed to “page views,” which counts the number of times a site is visited, even 

if it is accessed multiple times by the same person). This measure provides a 

better indication of the breadth of audience that PATH is reaching. The URL for 

the ToolBase ICFs Technology Inventory home page is 

http://www.toolbase.org/Technology-Inventory/walls/Insulating-Concrete-Forms.

Data Source: NAHB Research Center
62

3) Annual percentage growth in the number of Google hits when searching the 

following exact phrase: "Partnership for Advancing Technology in Housing" 

and "Insulating Concrete Forms" or "ICFs"

This metric gauges overall public interest in the ICFs program by measuring the 

availability of information and reference materials pertaining to the program on 

the Internet. This exact search phrase was considered meaningful because, when 

tested, each search result was relevant to the PATH ICFs program. Measuring the 

absolute number of hits for this metric is not as important as measuring how the 

number of hits changes over time (i.e., the annual percentage growth). Google 

hits are a useful metric because they measure the number of pages referencing 

PATH and ICFs not only frompathnet.org or toolbase.org, but also from external 

sites either in industry, news media, academia, or government. Together with 

Metric 2, these metrics provide an objective method to measure this program’s 

information-dissemination activities.

Data Source: Google.com

Market penetration outputs and outcomes

4) ICFs annual production for residential use (in square feet) per square feet of 

all new residential-housing units in the United States per year

Metric 4 estimates the total production of ICFs as a fraction of total square 

footage of new houses being built. Because the housing market fluctuates 

significantly from year to year, it is important to measure ICFs production 

compared to overall new home construction. This metric, together with Metric 5 

and Metric 6, offers insights into the market penetration of the technology. 

Particularly useful will be comparisons of these three metrics over time, which 

will help assess the rate of improvement or acceleration of the technology into the 

marketplace.

Data Source: ICFA publishes ICF shipments of its members, and can be used to 

approximate ICF annual production. U.S. Census publishes data necessary to 

compute the total square footage of new residential units built per year.

5) Number of home builders offering ICFs 

This metric approximates the market penetration of ICFs in the building industry, 

and should be evaluated in conjunction with Metric 4 and Metric 6. It measures 

the ability of an interested home buyer to select the ICFs technology for use in 

his/her new house. An increased number of builders offering ICFs can be a 

leading indicator of increased adoption of the technology. A possible addition to 

this metric would incorporate the percentage of the market that these home

builders represent, indicative of the market share of these builders. Obtaining this 

market data may be an obstacle; simply measuring the number of builders will be 

an adequate first-order measure.
63

Data Source: Insulating Concrete Form Association (ICFA). Although there may 

be other builders who offer ICFs, the number of ICFA home-builder members 

will be used as a proxy.

6) Percent of new homes in the United States using ICFs 

This metric provides another estimate of market penetration and approximates the 

range of coverage of ICFs in new homes. This metric should be evaluated in 

combination with Metric 4 and Metric 5 to obtain a better profile of the market 

progress of the technology. While Metric 5 measures whether ICFs are an option 

offered by builders, Metric 6 measures whether builders are actually 

implementing ICFs in their homes. Compare this to Metric 4, which indicates the 

total production of ICFs but not the extent of broad market penetration.

Data Source: The Portland Cement Association (PCA) and/or the National 

Association of Home Builders Research Center (NAHBRC) may be able to 

provide data on the use of ICFs in the residential market. The number of houses 

built using ICFs can be compared to the U.S. Census total number of new 

residential units built in that year. 

Metrics 4, 5, and 6 are also useful for evaluating the program retrospectively. 

Specifically, the impact of including ICFs in residential building codes can be 

quantified. The use of ICFs in residential-housing units before and after the 

incorporation of ICFs into the International Residential Code will help gauge the 

impact of addressing this regulatory barrier. 

The market penetration attributable to PATH is difficult to separate from the 

market penetration due to external factors. Metrics 1 through 3 are more directly 

correlated with specific PATH activities. Regardless, the ultimate objectives of 

the ICFs program align with the market-penetration metrics recommended here 

and are meaningful for evaluating the successes of the program. 

Additional Recommended Metrics

7) Number of states that have greater than 2.7% of new residential-housing 

units built using ICFs

This metric captures the geographic distribution of the ICFs technology. Some 

states may have significant market penetration, while others are lagging. ICFs use 

may be clustered in a few states, for example. Implementers chose 2.7% as the 

baseline because, in 2001, ICFs were installed in 2.7% of all U.S. above-grade 

homes, according to the National Association of Home Builders and the Portland 

Cement Association (NAHB 2002). A greater number of states exceeding the 

2001 average market penetration for new homes indicates greater geographic 

distribution. This metric should be used in conjunction with Metrics 4, 5, and 6,

which indicate overall market penetration. As overall market penetration 

increases, Metric 7 should be expected to also increase. Discrepancies among the 
64

trends in Metrics 4, 5, and 6, and Metric 7 can inform an analysis of geographic 

distribution as part of a broader evaluation of market penetration. 

Data Source: ICFA collects annual shipment data by state. Number of housing 

units with ICFs may be more difficult to obtain.
65

8. Baselines and Targets

Baselines establish and record levels of activity by which future comparisons can be 

made. Baselines allow reliable, defendable, and straightforward tracking of later 

progress, establishing a standard to compare, measure, and eventually evaluate program 

improvements. Over time, comparing program performance against a baseline can help 

validate or invalidate whether a program is focusing on the most important issues or 

problems and thereby optimizing its effects and benefits. Establishing baselines is an 

important step of a comprehensive program assessment and provides critical information 

for management to improve performance, efficiency, and effectiveness.17

Baselines

Targets are used for tracking program progress toward predetermined goals, typically by 

comparing performance achieved against performance expected. Target setting is an 

important component in a comprehensive program assessment. Without targets, the 

usefulness of program assessment is significantly reduced. 

Baselines also are a necessary component of the President’s Management Agenda 

Program Assessment Rating Tool (PART). PART was developed by the Office of 

Management and Budget (OMB) to assess and improve federal program performance. 

PART assists in identification of programs’ strengths and weaknesses and can be used to 

inform funding and management decisions aimed at making programs more effective. 

Thorough assessment of government programs through PART, for example, requires 

analysis and tracking progress relative to a baseline.

Selection of Baseline Years

The U.S. Department of Housing and Urban Development’s Partnership for Advancing 

Technology in Housing (PATH) Program was founded in 1998 as a public/private 

partnership to improve innovation in the residential-housing sector. The program aimed 

to accelerate the development and use of advanced housing technology and was guided 

by a list of quantitative housing performance goals. 

The National Research Council’s (NRC) evaluation of PATH found that no evidence or 

baseline data could be identified to indicate that the housing performance goals were 

measurable and achievable (NRC 2003). These and other challenges associated with 

PATH’s original numeric goals were identified. Recognizing these issues, PATH 

refocused and refined its goals in 2002 to reflect the overall aim of increasing the impact 

of technology innovation in the home-building sector while aligning with PATH 

capabilities and actions.

The years of baseline recommendations for the PATH subprograms featured in this 

document reflect these two major periods in the PATH program. Baseline year 1998 

reflects the start of PATH and its original goals, which are presented in Chapter 2. The 

second baseline year, 2003, is the earliest year that impacts from the 2002 PATH 

 17 “Assessing Program Performance.” <http://www.whitehouse.gov/omb/part/>.
66

strategic revision can be measured. Baseline values also were identified for 2007 – the 

most recent full year that data are available. Using these earlier years, target values for 

2008 can be recommended (see Targets section). These recommendations are shown in 

this report following the baseline values.

It should be noted that, due to incomplete data sets (see Key Challenges), baseline years 

for some metrics varied from the default years of 1998, 2003, and 2007. For example, 

data provided by NAHBRC on ToolBase Web site activity (e.g., page views, downloads, 

and unique visitors), were only provided for 2005, 2006, and 2007. Additionally, for 

example, industry market research data from the Steel Framing Alliance (SFA) only 

reported percent of new housing starts using steel framing for a single year, 2002. 

For metrics where very few data points were available, baseline data for alternate years 

were estimated by extrapolation and/or proxy data. For example, for the missing SFA 

market-penetration data, the percent changes were tied to trends in steel framing 

production (tons) data from 1997 to 2005. Details and key assumptions for each baseline 

value are provided in the sections below.

Targets

The methods used to choose target values can include expert judgment, objective 

statistical analysis, complex formula, or other methods. Using a structured process 

enables a target setting methodology that is easy to understand and adjust later, if needed. 

A target-setting process should ensure that the targets are relevant to program goals and 

objectives. The target values presented in this study correspond to individual metrics 

developed specifically to align with program goals, and the targets already should be 

relevant to meaningfully gauging program progress. 

Even through the use of a structured process, as conducted in this study, target setting can 

be difficult. In its simplest form, target setting involves projecting future performance 

based strictly on historical performance. This can be done by using linear trends, or 

simply using the most recent data point as the target value for future years. In practice, 

programs are more complex and dynamic and multiple elements contribute, to varying 

degrees, to the program’s performance throughout the program’s active performance 

period. However, establishing targets based on multiple variants can complicate the 

process and lead to unnecessarily reducing the transparency of the targets. 

For this study, a straightforward structured approach based primarily on historical data 

trends was used for setting recommended targets. Linear regression was applied to the 

baseline values, and other relevant historical data and target values for 2008, 2013, and 

2018 were extrapolated. Where insufficient historical data were available, alternative 

data were used. For example, Insulating Concrete Form Association (ICFA) membership 

data were only available for 2008. Without the use of alternative data, both baseline and 

target membership values could not be easily determined. As such, ICF shipment data 

trends were used to extrapolate both the baseline and target values for ICFA membership.
67

PATH may want to examine each of the recommendations in detail and adjust the targets,

based on external factors likely to influence the target values. For instance, increased or 

deceased program funding may require adjustments to the aggressiveness of targets

achieved in subsequent years.

Key Challenges

During the baselining and target-setting processes, many challenges arose, primarily due 

to gaps in historical data. Some data that were thought to be publicly available are 

collected infrequently and incompletely. Data gaps exist from several sources, including 

directly from the program and its partners, as well as industry market-research 

organizations. 

Incomplete and infrequently collected data, in conjunction with limited information about 

historical subprogram activities, provided challenges in recommending appropriate 

baselines and targets. For example, comprehensive data sets often were not available 

from industry marketing groups such as SIPA, ICFA, and SFA. Historical membership 

data were either not collected, could not be obtained, or, due to various factors, turned out 

to be an inadequate measure of the metric. Alternative methods were used in such cases 

and are described below where applicable.

Incomplete data were particularly a challenge in analyzing the ToolBase use data. For 

example, one metric recommends measuring the “Total number of TechInventory page 

views per month”; however, only “Top 10” ToolBase individual Web pages (e.g.,

TechInventory page views for specific technologies), pages view were obtained. Further 

complicating the quantification of the metric for the baseline and target years is that these 

“Top 10” are not limited to only TechInventory page views – the Top 10 reported in any 

given month may include other ToolBase Web pages. Therefore, to report baselines and 

targets for this metric, only “Top 5” TechInventory technologies are considered (closer 

examination of the data set reveals that in each “Top 10” data set there are at least five

TechInventory pages included). 

In addition to the challenge of incomplete and infrequently collected data, methods of 

data collection varied over time. For example, toolbase.org Web site use statistics were 

collected with two different programs: WebTrends and Urchin. WebTrends and Urchin 

each used different formulas for data collection and, therefore, resulted in slightly 

different data categories, which introduced inconsistency and difficultly in interpretation.

Subprogram Details

Baseline and target recommendations have been developed for the five PATH 

subprograms that have been evaluated for this study. The following section contains two 

parts for each subprogram. The first part outlines key events in the programs’ history, 

which is presented so the reader can compare the historical activities with the 

corresponding baseline values. The second part presents the recommended baseline and 

target values. 
68

Accompanying spreadsheets provide further detail for the recommended baselines and 

targets. The spreadsheets contain full data sets, analysis, and projections that were used 

for each metric. There is a separate worksheet summarizing the baselines and targets of 

all relevant metrics for each of the five subprograms, and a separate tab within each 

worksheet for each metric.

Concept Home Program

Key Events

2004: The conception of the first Concept Home is completed.18

Summer 2007: Concept Home Omaha is completed and a ribbon-cutting event and open 

house is held.

 The Concept Home 

architectural model is unveiled during a reception in Washington, D.C. (PATH 2004).

2005: Concept Home technology and manufacturer partners are selected. Blueprints and 

specifications are detailed

2006: Builder and site for Concept Home Demonstration Project are selected (PATH 

2006d). HUD breaks ground in Omaha, Nebraska, on the first PATH Concept Home in 

the United States.

19

Fall 2007: PATH convenes experts and innovators to design the initial design of Concept 

Home in Charleston, South Carolina (PATH 2007).20

Concept Home Baseline and Target Values

HUD names Charleston builder for 

PATH Concept Home. Architects, builders, engineers, interior designers, and landscape 

architects participate in an integrated design review session.

Winter 2008: At the 2008 International Builders' Show, PATH reveals the design plans 

for the Charleston Concept Home. The home will serve as a model of sustainability, 

hurricane resistance, efficient building practices, and floor plan flexibility. 

Metric 1: Cumulative number of industry partners (such as designers, builders, technology 

manufacturers) who have participated in Concept Home since the inception of the program

Baseline Values:

1998 2003 2007

NA NA 51

The first Concept Home was unveiled in June 2004. The count of partners is updated each 

time that the PATHnet site adds to its Concept Home Sponsors page. 

Target Values:

2008 2013 2018

51 102 153

Because no new Concept Homes have been built since June 2007, that year and 2008 are 

 18 “History of the PATH Concept Home.” <http://www.pathnet.org/sp.asp?id=21947&pbg=1>. 19 “The Concept Home Experience!” <http://www.pathnet.org/sp.asp?id=24487#June%206>. 20 “Welcome to Concept Home Charleston!” <http://www.pathnet.org/sp.asp?id=11175>.
69

the same. Because it took from December 2004 to June 2007 to complete the first Concept 

Home, it would be reasonable to expect that a new Concept Home be completed every five

years. The number of sponsors (per Concept Home) should equal or exceed those of the first 

Concept Home.

Metric 2: Cumulative number of technologies that are promoted by Concept Home and 

included in completed homes since the inception of the Concept Home program

Baseline Values:

1998 2003 2007

NA NA 38

The first Concept Home was unveiled June 2004. The count of technologies showcased is 

updated each time that the PATHnet site adds to its technologies page. 

Target Values:

2008 2013 2018

38 76 114

Because no new Concept Homes have been built since June 2007, that year and 2008 are 

the same. Because it took from December 2004 to June 2007 to complete the first Concept 

Home, it would be reasonable to expect that a new Concept Home be completed every five 

years. The number of technologies (per Concept Home) should equal or exceed those of the 

first Concept Home.

Metric 3: Cumulative potential audience reached through mass marketing and news releases 

(e.g., television promotions, circulation of newspapers containing articles about Concept 

Home). For this metric, newspaper circulation only is reported

Baseline Values:

1998 2003 2007

NA NA 4,000,000

There were no Concept Home activities prior to June 2004. The 2007 baseline reflects the 

audience (circulation of newspapers) for the Concept Home articles in that year.

Target Values:

2008 2013 2018

4,000,000 8,000,000 12,000,000

Because no new Concept Homes have been built since June 2007, that year and 2008 are 

the same. Because it took from December 2004 to June 2007 to complete the first Concept 

Home, it would be reasonable to expect that a new Concept Home be completed every five 

years. The audience reached (per Concept Home) should equal or exceed that of the first 

Concept Home.

Metric 4: Cumulative audience reached via in-person information events, such as Concept 

Home promotional events and open houses

Baseline Values:

1998 2003 2007

NA NA 405

There were no Concept Home activities prior to June 2004. The 2007 baseline reflects 

attendance at Concept Home events in that year.

Target Values:

2008 2013 2018

405 810 1,215

Because no new Concept Homes have been built since June 2007, that year and 2008 are 

the same. Because it took from December 2004 to June 2007 to complete the first Concept 

Home, it would be reasonable to expect that a new Concept Home be completed every five 
70

years. The in-person attendance (per Concept Home) should equal or exceed those of the 

first Concept Home.

Metric 5: Average monthly number of Web page hits that contain information about Concept 

Home and its technologies on PATH-sponsored Web sites (i.e., PATHnet, ToolBase, 

HUDUser). For this metric only, page views of “About the Concept Home”21 were reported 

due to lack of data for other pages on the Concept Home (and related) Web sites.

Baseline Values:

2004 2006 2007

744 1,300 2,625

Target Values:

2008 2013 2018

2,900 5,900 8,900

Projections of trends based on linear regression of historical data were used to estimate 

targets, rounded to the nearest 100.

Metric 6: Annual percentage growth in the number Google hits when searching the following 

exact phrase: “Partnership for Advancing Technology in Housing” or “Concept Home”

Baseline Values:

1998 2003 2007

NA NA NA

Insufficient data are available to establish a baseline value. Data are available from spring 

2008 to present, but these are not yet enough to set a baseline.

Target Values:

2008 2013 2018

TBD TBD TBD

There are insufficient data to extrapolate a trend. Targets will be developed over time.

 21 “About the Concept Home.” < http://www.pathnet.org/sp.asp?id=11175>.
71

ToolBase Program

Key Events

Prior to July 2001: PATH partners with the NAHB Research Center, and the program 

integrated the ToolBase Web pages with the Web site for the NAHB Research Center 

(www.nahbrc.org). 

July 2001: After a major redesign effort, the toolbase.org site was launched as a separate 

Web site. 

July 2006: The Web site was again revamped and relaunched, completely independent 

from the nahbrc.org Web site. The ToolBase portal design is the same that is seen today

(McGee 2008). 

Prior to July 2007: PATH provides funding to expand the Technology Inventory, 

develop Field Evaluations, promote the ToolBase site and its technologies, and make 

other enhancements to the Web site

July 2007: The scope of the program is altered, although the Web site itself does not 

significantly change at this time. When one of the funding vehicles expires, PATH and 

its partner NAHB Research Center are limited to maintaining the existing content in 

ToolBase, focusing mainly on the Technology Inventory and Field Evaluation sections 

and ensuring that the Web site is “live” and functioning on the Internet. Currently, no 

new materials for the TechInventory and Field Evaluations are being developed

Note that baseline values for ToolBase are largely provided for 2003, 2005, and 2007, 

while other programs’ baseline years are typically 1998, 2001, and 2007. The years 2003 

and 2005 were used because of the availability of data. Prior to 2003, very limited data 

was obtained from the NAHB Research Center, which was the organization tracking 

ToolBase Web site use. 

ToolBase Baseline and Target Values

Metric 1: Cumulative number of technologies featured in the TechInventory

Baseline Values:

2003 2005 2007

171 197 246

The TechInventory is a major focus of the ToolBase site. Archive.org was used to view 

historical ToolBase TechInventory home pages and create a timeline of technologies and their 

dates of addition to the ToolBase site. 

Target Values:

2008 2013 2018

263 355 446

Projections based on linear regression of the historical data were used to estimate targets.
72

Metric 2: Cumulative number of Field Evaluations completed with results published on 

ToolBase

Baseline Values:

2003 2005 2007

21 34 41

Field Evaluations featured on the ToolBase site show information that is necessary for 

information dissemination about innovative building technologies and practices. Archive.org 

was used to view a historical list of Field Evaluations detailed in ToolBase.

Target Values:

2008 2013 2018

45 63 82

Projections based on linear regression of the historical data were used to estimate targets.

Metric 3: Total number of TechInventory page views (of top five technologies) per thousand 

user visits/sessions

Baseline Values:

2005 2006 2007

83 153 137

The total number of page views for the top five technologies in a given month for years 2004 

to 2008 was computed using several years of monthly “Top 10” ToolBase pages, of which at 

least five were TechInventory technologies. It should be noted that some data were missing

and estimates are based on the limited data set.

NAHBRC provided the number of visits/visitor sessions monthly from 2003 to 2008 for 

individual months and annual monthly averages. The annual monthly averages were used as 

the denominator to compute this metric. 

Target Values:

2008 2013 2018

150 172 182

Linear regression was used individually on the total number of page views of the top five 

technologies baseline data and on the visitor sessions baseline data. The projected target 

values of these two were computed into these target metric values (total page views/sessions) 

for 2008, 2013, and 2018.

Metric 4: Number of broken links in toolbase.org

Baseline Values:

2003 2005 2007

NA NA NA

The number of broken links provides an indicator for the maintenance of the ToolBase site. 

Current data can be collected through specialized software packages, but have not yet been 

applied to ToolBase. 

Target Values:

2008 2013 2018

TBD TBD TBD

Target values will be determined once sufficient broken link data for the current site has been 

collected. 
73

Metric 5: Total number of unique visitors to toolbase.org per month

Baseline Values:

2003 2005 2007

61,569 156,929 223,331

NAHBRC provided the number of unique visitors monthly from 2003 to 2008. Data were 

provided both for individual months and annual monthly averages. 

Target Values:

2008 2013 2018

260,000 440,000 620,000

Linear regression was used on the data set of the baseline values and additional years of 

“total number of unique visitors” data to estimate targets, rounded to the nearest 10,000.

Metric 6: Total number of toolbase.org page views per month

Baseline Values:

2003 2005 2007

365,166 473,738 659,014

NAHBRC provided the number of page views monthly from 2003 to 2008. Data were 

provided both for individual months and annual monthly averages. 

Target Values:

2008 2013 2018

700,000 1,000,000 1,300,000

Linear regression was used on the data set of the baseline values and additional years of 

“total number of toolbase.org page views per month” data to estimate targets, rounded to the 

nearest 100,000.

Metric 7: Total number of TechInventory page views per month (calculated using only the top 

five TechInventory technologies in each month).

Baseline Values:

2005 2006 2007

17,940 36,224 34,510

The total number of page views for the top five technologies in a given month for 2004 to 

2008 was computed using several years of monthly “Top 10” ToolBase pages, of which at 

least five were TechInventory technologies. 

Target Values:

2008 2013 2018

43,000 78,000 113,000

Linear regression was used on the data set of the baseline values and additional years 

of”‘total number of top five TechInventory page views per month” data to estimate targets, 

rounded to the nearest 1,000.

Metric 8: Total number of Field Evaluations page views per month

Baseline Values:

2003 2005 2007

Data requested Data requested Data requested

Baseline values will be determined pending data to be received from NAHBRC. 

Target Values:

2008 2013 2018

TBD TBD TBD

Target values will be determined pending data to be received from NAHBRC.
74

Metric 9: Total number of visitors (“visitor sessions”) accessing toolbase.org per million

square feet of all new residential-housing units in the United States per month

Baseline Values:

2003 2005 2007

331 648 981

This metric normalizes the traffic on the ToolBase site by accounting for the fluctuating 

housing market. NAHBRC provided the number of visitor sessions monthly from 2003 to 

2008 for individual months and annual monthly averages. Monthly visitor sessions were 

compared to total square feet (calculated) of new single-family housing units completed.

Target Values:

2008 2013 2018

1,000 1,800 2,500

Linear regression was used on the data set of the baseline values and additional years of 

“total number of visitors per month per sq. ft. housing” data to estimate targets, rounded to the 

nearest 100.

Metric 10: Average number of page views per session

Baseline Values:

2003 2005 2007

4.1 2.2 2.6

NAHBRC provided number of page views monthly and number of visitor sessions monthly 

from 2003 to 2008 for individual months and annual monthly averages. 

Target Values:

2008 2013 2018

2.6 4.1 4.1

A linear regression of the data for this metric results in downward trend for page views per 

session. Instead of projecting fewer page views per session, the target values selected for 

2008 are the same as the previous year (2007). In 2013 and 2018, page views per session 

are targeted to match the peak of 4.1 in 2003.

Metric 11: Average sessions per visitor in a given month (total number of visitor sessions / 

total number of unique visitors)

Baseline Values:

2003 2005 2007

1.4 1.4 1.1

Number of monthly visitor sessions and number of unique visitors were obtained from 2003 to 

2008. Data were provided both for individual months and annual monthly averages. 

Target Values:

2008 2013 2018

1.1 1.4 1.4

Due to reduced funding for 2008, the target values selected for this year are the same as the 

previous year (2007). Additional funding is expected and the return rate of visitors should 

improve and achieve that of previous years.
75

Structural Insulated Panels Program

Key Events

Early 2002: A TechInventory page on Structural Insulated Panels is added to the 

toolbase.org site.

Prior to June 2006: PATH collaborates with the Structural Insulated Panel Association 

(SIPA), APA–The Engineered Wood Association, and SIP manufacturers across the 

country to identify critical barriers impeding wider market uptake of structural insulated 

panels. Nearly a year of testing and evaluation goes into developing a prescriptive 

method for Structural Insulated Panels (PATH 2006b).

June 2006: Publication of the “Prescriptive Method for Structural Panels,” which 

establishes performance standards for SIPs and shares technical information about the 

technology. It also offers specific guidelines to facilitate the use of SIPs in wall systems 

for the construction of one- and two-family dwellings (PATH 2006a).

After June 2006: The Prescriptive Method and specifications are submitted to the 

International Code Council (ICC) for adoption into the International Residential Code 

(IRC).

August 2006: A TechSpecs downloadable document (PDF format) on Structural 

Insulated Panels is added to the SIPs TechInventory page of toolbase.org site.

May 2007: The ICC votes to adopt SIPs into the IRC, and shortly thereafter they are

included as a supplement to the 2006 International Residential Code (SIPA 2007).

SIPs Baseline and Target Values

Metric 1: Percent of new homes in the United States that are built in a location that has 

adopted the International Residential Code (IRC) or other building codes that include SIPs

Baseline Values:

1998 2003 2007

0% 0% 25%

SIPs were included in the 2007 supplement to the 2006 International Residential Code, 

therefore baseline values prior to 2006 equal zero.

Target Values:

2008 2013 2018

31% 62% 93%

Projections of historical trends were used to estimate target values.

Metric 2: Average monthly page views of the ToolBase SIPs Technology Inventory home

page

Baseline Values:

2005 2006 2007

3,704 4,370 5,731

SIPs were added to the ToolBase TechInventory in January 2002. Data for page views are 

available for 2005 to present. 
76

Target Values:

2008 2013 2018

6,600 11,700 16,800

Projections of historical trends using linear regression were used to estimate targets and

rounded to the nearest 100.

Metric 3: Average monthly downloads of the SIPs TechSpec (from toolbase.org site)

Baseline Values:

1998 2003 2007

NA NA Data requested

The TechSpec for SIPs was added to ToolBase in 2006. Baseline values prior to 2006 are 

not applicable. A baseline value for 2007 will be determined based on data from NAHBRC. 

Target Values:

2008 2013 2018

TBD TBD TBD

Target values are to be determined pending data to be received from NAHBRC.

Metric 4: Annual percentage growth in the number of Google hits when searching the 

following exact phrase: "Partnership for Advancing Technology in Housing" and "Structural 

Insulated Panels" or "SIPS"

Baseline Values:

1998 2003 2007

NA NA NA

Insufficient data are available to establish a baseline value. Data are available from spring 

2008 to present, but these are not yet enough to set a baseline. 

Target Values:

2008 2013 2018

TBD TBD TBD

Currently there are insufficient data to extrapolate a trend. Targets will be developed over 

time.

Metric 5: Number of home builders offering SIPs (trend approximated by using SIPA homebuilder members)

Baseline Values:

1998 2003 2007

NA TBD TBD

The number of home builders offering SIPs was to be approximated by using trade 

association (SIPA) membership numbers. However, after personal correspondence with a 

SIPA representative, baseline values should be determined using an alternative method or 

determined later. SIPA did not have a builder program in 1998, and from 2003-2007, SIPA’s 

Board of Directors voted for a significant dues increase across all membership categories. 

This drastically affected membership numbers. Considering these external factors influencing 

SIPA membership numbers, the method for determining the approximate number of home

builders offering SIPs will be reevaluated.

Target Values:

2008 2013 2018

TBD TBD TBD

Targets will be developed over time once SIPA membership stabilizes.
77

Metric 6: Percent of new homes in the United States using SIPs

Baseline Values:

2002 2003 2007

0.63% 0.56% 0.69%

This metric provides an estimate of market penetration, and approximates the range of 

coverage of SIPs in new single-family homes. SIP home construction was compared to new 

housing units completed (1 unit single-family).

Target Values:

2008 2013 2018

0.63% 0.67% 0.72%

Projections of historical trends were used to estimate target values.

Metric 7: SIPs annual production for residential use (in square feet) per million square feet of 

all new residential (single-family) housing units in the United States per year

Baseline Values:

2001 2003 2007

6,585 7,095 9,188

This metric gauges SIPs production compared to overall new home construction. SIP 

production was compared to total square feet (calculated) of new single-family housing units 

completed.

Target Values:

2008 2013 2018

8,000 10,000 11,000

Projections of historical trends using linear regression were used to set the baseline values, 

and additional years of “SIPs production (sq. ft.) per sq. ft. housing” data were used to 

estimate targets. Targets were rounded to the nearest 1,000.
78

Advanced Steel Framing Program

Phase 1 involves research and demonstration on the acceptability of steel framing 

material as an alternative to wood. 

Phase 2 of the program works to develop a broader market for steel framing through 

published documents and information dissemination.

Phase 1 Key Events

Prior to 1997: PATH enters a cooperative agreement with the National Association of 

Home Builders (NAHB) and the American Iron and Steel Institute (AISI) to identify 

critical barriers impeding wider market uptake of steel framing. 

1997: The “Prescriptive Method for Residential Cold-Formed Steel Framing” is

published, and is based on three years of HUD-sponsored research that incorporates input 

from experts in the fields of construction, research, engineering, and building-code 

enforcement (HUD 1997a). 

2002: PATH partners with the Steel Framing Alliance (SFA) (formerly, the North 

American Steel Framing Alliance) and funds the Manufactured Housing Research 

Alliance (MHRA) to produce the technical support document, “Design for a ColdFormed Steel Framed Manufactured Home” (PATH 2002). The document – developed in 

conjunction with trade associations, builders, testing laboratories, and manufacturers –

assesses the viability of substituting steel for wood as the structural skeleton of homes 

built under the HUD manufactured-home standards. 

2002: Steel framing manufacturing methods are adopted by the International Code 

Council (ICC) into the International Residential Code of 2000 and 2003. 

Phase 2 Key Events

2003: PATH partners with the SFA to examine and report on corrosion of galvanized 

fasteners used in cold-formed steel, and to produce “Hybrid Wood and Steel Details –

Builders Guide,” a report that provides information for builders to construct hybrid coldformed steel and wood homes (PATH 2006c and PATH 2003b). 

2003: The “Prescriptive Method for Connecting Cold-Formed Steel Framing to 

Insulating Concrete Form Walls in Residential Construction” is published to facilitate the 

construction of houses from insulating concrete forms and cold-formed steel, both of 

which are important PATH technologies (PATH 2003c). 

Steel Framing Baseline and Target Values

Metric 1: Percent of new homes in the United States that are built in a location that has 

adopted the International Residential Code (IRC) or other building codes that include Steel 

Framing

Baseline Values:

1998 2003 2007

0% 50% 77%
79

Steel Framing was adopted into IRC 2000 and IRC 2003 codes in 2002. The baseline value 

for 1998 is assumed to be zero. 

Target Values:

2008 2013 2018

83% 100% 100%

Projections of historical data were used to estimate targets.

Metric 2: Average monthly page views of the ToolBase Steel Framing Technology Inventory 

home page

Baseline Values:

1998 2005 2006

NA 3,879 5,223

NAHB provided data for Steel Framing TechInventory dating back to 2005. In 1998, it is 

assumed that a Steel Framing page did not exist on ToolBase.

Target Values:

2008 2013 2018

7,900 14,600 21,400

Projections of historical data were used to estimate targets, rounded to the nearest 100.

Metric 3: Average monthly downloads of the Cold-Formed Steel Framing TechSpec (from 

ToolBase site)

Baseline Values:

1998 2003 2007

NA NA NA

The TechSpec for Cold-Formed Steel Framing was added to ToolBase in February 2008. 

Prior to 2008, the number of downloads are not applicable.

Target Values:

2008 2013 2018

TBD TBD TBD

Target values are to be determined over time and pending data to be received from NAHBRC.

Metric 4: Annual percentage growth in the number of Google hits when searching the 

following exact phrase: "Partnership for Advancing Technology in Housing" and "Steel 

Framing"

Baseline Values:

1998 2003 2007

NA NA NA

Insufficient data are available to establish a baseline value. Data are available from spring 

2008 to present, but these are not yet enough to set a baseline.

Target Values:

2008 2013 2018

TBD TBD TBD

There are insufficient data to extrapolate a trend. Targets will be developed over time.

Metric 5: Steel framing annual production for residential use (in tons) per million square feet 

of all new residential-housing units in the United States per year

Baseline Values:

1998 2001 2005

40 62 54

This metric measures steel framing production compared to overall new home construction. 
80

Steel Framing production was compared to total square feet (calculated) of new single-family 

housing units completed.

Target Values:

2008 2013 2018

71 85 99

Projections of historical data were used to estimate targets.

Metric 6: Number of home builders offering steel framing 

Baseline Values:

1998 2003 2007

37 62 81

Estimates of SFA membership values in conjunction with trends in historical steel framing 

production were used as a proxy for the total number of home builders offering steel framing. 

Target Values:

2008 2013 2018

86 110 135

Projections of historical data were used to estimate targets for the number of home builders 

offering steel framing.

Metric 7: Percent of new homes in the United States using steel framing

Baseline Values:

1998 2001 2007

1.0% 1.5% 2.1%

This metric provides an estimate of market penetration of steel framing in new homes. Steel 

framing home construction was compared to total new housing units completed (1 unit singlefamily). Limited data were obtained to extrapolate a trend in “percent of new homes in the 

United States using steel framing.” These data were augmented with trends in historical steel 

framing production.

Target Values:

2008 2013 2018

2.3% 2.9% 3.5%

Target “percent of new homes in the United States using steel framing” values were estimated 

by extrapolating trends in historical steel framing production.
81

Insulating Concrete Forms Program

Key Events

1998: PATH collaborates with the NAHB Research Center and the Portland Cement 

Association (PCA) to identify critical barriers impeding greater market penetration of 

insulating concrete forms.

May 1998: The “Prescriptive Method for Insulating Concrete Forms in Residential 

Construction” is first published. It establishes performance standards for ICFs and shares

technical information about the technology (PATH 1998).

2000: The Prescriptive Method is accepted in 2000 by the International Code Council 

(ICC) for adoption into the International Residential Code (IRC).

Early 2002: A TechInventory page on Insulated Concrete Forms is added to the 

ToolBase site.

Baseline and Target Values

Metric 1: Percent of new homes in the United States that are built in a location that has 

adopted the International Residential Code (IRC) or other building codes that include ICFs

Baseline Values:

1998 2003 2007

0% 50% 77%

ICFs were adopted into the IRC in 2000. Baseline values prior to 2000 are assumed to be

zero. 

Target Values:

2008 2013 2018

83% 100% 100%

Projections of historical trends using linear regression were used to estimate targets.

Metric 2: Average monthly page views of the ToolBase ICFs Technology Inventory home

page

Baseline Values:

2005 2006 2007

3,425 3,700 4,812

ICFs were added to the ToolBase TechInventory in January 2002; page views data provided 

by NAHB Research Center. 

Target Values:

2008 2013 2018

4,400 5,800 7,100

Projections of historical trends using linear regression were used to estimate targets and 

rounded to the nearest 100.
82

Metric 3: Annual percentage growth in the number of Google hits when searching the 

following exact phrase: "Partnership for Advancing Technology in Housing" and "Insulating 

Concrete Forms" or "ICFs"

Baseline Values:

1998 2003 2007

NA NA NA

Insufficient data are available to establish a baseline value. Data are available from spring 

2008 to present, but these are not yet enough to set a baseline.

Target Values:

2008 2013 2018

TBD TBD TBD

There are insufficient data to extrapolate a trend. Targets will be developed over time.

Metric 4: ICFs annual production for residential use (in square feet) per million square feet of 

all new residential-housing units in the United States per year

Baseline Values:

1998 2003 2005

6,908 10,890 10,448

This metric measures ICFs production compared to overall new home construction. ICFs 

production was compared to total square feet (calculated) of new single-family housing units 

completed.

Target Values:

2008 2013 2018

14,000 18,000 22,000

Projections of historical data using linear regression were used to estimate targets, rounded to 

the nearest 1,000.

Metric 5: Number of home builders offering ICFs 

Baseline Values:

1998 2003 2007

82 147 199

Current estimates of ICFA membership in conjunction with trends in the number of square 

feet of ICF wall forms shipped were used as a proxy for the number of home builders offering 

ICFs.

Target Values:

2008 2013 2018

212 277 342

Projections of historical data using linear regression were used to estimate targets.

Metric 6: Percent of new homes in the United States using ICFs

Baseline Values:

1998 2003 2007

1.0% 4.0% 7.4%

This metric provides an estimate of market penetration and it approximates the range of 

coverage of ICFs in new single-family homes. ICF home construction was compared to new 

housing units completed (1 unit single-family).

Target Values:

2008 2013 2018

7.8% 10.9% 14.0%

Projections of historical data using linear regression were used to estimate targets.
83

9. Guidelines for Developing Metrics, Data Collection, 

Analysis and Reporting for PATH Subprograms

The PATH program is undergoing a process of developing and implementing a new set 

of program metrics. This report discusses the development of high-level metrics that 

respond to PATH’s new mission, goals, and activities, and metrics for individual 

subprograms. Recommended metrics for five subprograms have been developed, along 

with data-collection methodologies, baselines, and targets recommendations. PATH may 

want to conduct similar evaluations for its remaining subprograms. This chapter outlines 

a general approach and guidelines for developing metrics, collecting data, and analyzing 

the data to produce baselines and targets.

Developing Metrics

PATH’s development of recommended metrics for its subprograms is based on a welldocumented and widely used approach. This approach follows three basic steps:

1. Develop a logic model for the subprogram

2. Identify potential metrics for the outputs and outcomes in the logic model

3. Recommend metrics for program evaluation

Step 1: Develop a simple logic model of the program activities, outputs, and 

outcomes. A logic model is a graphical depiction showing how program activities 

lead to intended outcomes (Figure 6). Developing a logic model is an essential first 

step because it describes the subprogram’s design. In-depth knowledge of the 

subprogram activities and goals is critical for developing the logic models. 

Figure 6. Logic Model Diagram

The logic model translates the program activities and revised goals into a diagram 

that illustrates how these function collectively while being measurable individually. 

A vital attribute for any metric is that it shows a clear relationship to the desired goals 

and objectives, and a logic model provides the conceptual foundation for 

understanding this relationship. Potential metrics for the PATH program overall have 

Assess ongoing operations

Make minor adjustments 

and address any 

management problems

Appraise individual 

projects and programs

Adjust allocation of 

incremental resources

Draw conclusions 

about efficacy of 

different policy 

approaches

Evaluate effects on 

society and economy

Develop metrics

Ensure meaningful, 

verifiable, unbiased 

indicators

Inputs Activities Immediate 

Outputs

Resulting 

Outputs

Intermediate

Outcomes

End 

Outcomes

Planning Short term Medium term Long term
84

already been identified, but the process of developing metrics for individual 

subprograms within PATH would significantly benefit from such a model.

Step 2: Identify potential metrics based on the logic model. The components of the logic 

model (outputs and outcomes for each activity) are then used as a framework to identify 

potential metrics. Potential metrics should cover the breadth of activities, ensuring that 

every major element of the subprogram is represented. Identifying potential metrics 

based on the model ensures that the metrics are appropriately matched with the goals and 

structure of the program. Mapping potential metrics that are aligned with PATH 

subprogram goals and objectives means that the set of metrics are more likely to cover 

both the means (activities) and the ends (outcomes), and track the movement of 

technologies through the innovation cycle.

Step 3: Decide on recommended metrics. This step involves narrowing the list of 

potential metrics into a set of recommended metrics that measure the subprogram’s 

progress. The set of recommended metrics should be considered for informing a 

broader assessment of PATH. The process for determining metrics that best represent 

the progress of the program as well as its specific activities aimed at advancing 

housing technology and achieving its overall goals is presented in Chapter 2. The 

framework involves applying a scoring system to the potential metrics based on 

several key characteristics. These characteristics are used as criteria for scoring 

potential metrics to refine the set into recommended metrics, and include: 

understandable, attributable, feasible, and comparable over time. These 

characteristics and the scoring system are explained in greater detail in Chapter 2.

Determining appropriate metrics also involves applying expert judgment to check that 

the objective scoring technique generates a set of metrics that are representative of the 

breadth of PATH activities and includes measures of activities, outputs, and 

outcomes. If necessary, the set of recommended metrics should be adjusted or 

rebalanced so that the various criteria required for the evaluation of a multifaceted 

program such as PATH are met.

Data Collection

Depending on the individual characteristics of each metric, a data collection plan may 

need to be developed separately for each metric. For instance, a comprehensive set of 

metrics will include measurements on a wide range of activities such as Web site hits, 

promotional activities, market penetration, and others. As such, it is difficult to develop a 

generalized guidance procedure for data collection applicable to all of PATH’s 

subprograms. Nonetheless, there is at least minimal overlap between data sources and 

collection methodologies among subprograms. The types of sources and specific 

examples are provided as follows:
85

D&R International 

D&R International, PATH’s marketing consultant, provides quantitative data on the 

attendance and frequency of promotional events as well as data on information 

distribution. D&R also supplies information on the number of publications and other 

articles created, and they track the estimated audience reached by such materials. 

NAHB Research Center

The National Association of Home Builders (NAHB) Research Center compiles statistics 

on Web site use of ToolBase, including number of unique visitors, page views, document 

downloads, and several others.

The Portland Cement Association (http://www.cement.org/) is an organization that 

represents cement companies in the United States and Canada and conducts market 

Trade Associations

Trade associations and advocacy organizations can be an excellent source of relevant and 

timely data. Obtaining data from these organizations can involve time-consuming efforts 

to connect with the appropriate personnel, but can often result in a wealth of directly 

relevant and detailed information. Specific examples of trade associations that have been 

useful for data collection efforts to date include:

Insulating Concrete Form Association (ICFA)

Insulating Concrete Form Association (ICFA) (http://www.forms.org/) is an 

organization “dedicated to expanding the use of insulating concrete forms through 

research, education, promotion, and public affairs work”. ICFA provides industry 

data, such as annual shipment reports showing production trends. ICFA membership 

lists provide useful proxy data indicating the number of system manufacturers, 

distributors, contractors, suppliers, etc.

Steel Framing Alliance (SFA)

The Steel Framing Alliance is a trade association that encourages the widespread use 

of cold-formed steel framing in residential construction. SFA provides data on steel 

framing manufacturing, such as number of shipments to the residential industry and 

market trends data for both single-family site-built and multi-family site-built homes 

including tons of light-gauge steel framing used and units built. SFA membership 

lists offer useful data about the number of steel framing residential builders, 

suppliers/distributors, architects, and related manufacturers such as stud, truss, and 

component fabricators.

Structural Insulated Panel Association (SIPA)

The mission of the Structural Insulated Panel Association is to “increase the use and 

acceptance of SIPs through a strong trade association that provides an industry forum 

for promotion, communication, education, research, and quality assurance.” SIPA 

provides SIPs production data and data about the number of housing units supplied 

with SIPs from member companies. 

Portland Cement Association (PCA) 
86

development, engineering, research, education, and public affairs programs. PCA 

offers relevant articles about ICFs and a list of known producers. 

Census Bureau

The U.S. Census Bureau publishes a wide range of reliable information about housing in 

the United States. For example, the Census Bureau reports total square footage of new 

residential units built per year, a useful data point to normalize fluctuations in 

implementation of housing technologies due to changes in the U.S. housing market.

Analysis and Reporting

Building Code Organizations

The International Code Council (http://www.iccsafe.org/) lists states and jurisdictions 

adopting the most widely accepted building codes.

Detailed data-collection methodologies tailored for each metric and subprogram can be 

found in the Excel worksheets accompanying the “Baseline and Target Memo, Sept 

2008.”

The full set of recommended metrics measure both the means (activities and outputs) and 

the ends (outcomes) of the program, provide meaningful baselines and target values to 

gauge future progress, and inform a process for long-term assessment and program 

improvement.

The difficultly in measuring and analyzing outcomes of research programs, particularly 

those with a public/private component, has been well-documented in previous reports 

(e.g., NRC 2003). In the analysis of PATH subprograms, one technique to overcome 

these difficulties includes emphasizing “intermediate outcomes,” as suggested by the 

section of the 2007 OMB PART guidance titled Categories of Performance Measures. 

The guidance proposes the use of proxies when such difficulties arise. Proxies were used 

for several metrics in the evaluation of PATH’s five subprograms. The PART guidance 

offers the following description:

Programs that cannot define a quantifiable outcome measure – such as 

programs that focus on process-oriented activities (e.g., data collection, 

administrative duties, or survey work) – may adopt a “proxy” outcome 

measure. Identification of intermediate steps brings into the PART process 

an important family of existing results that may lend themselves to 

qualitative and sometimes quantitative assessment, which can provide useful 

new data points for reviewers. The terms in which those steps are described 

depend on the agency, its mission, and the nature and scope of its work.

An important consideration when analyzing metrics involves differentiating between 

increases in technology adoption as a result of PATH program activities and increases in 

technology implementation due to external factors. It is particularly challenging for 

public/private partnerships such as PATH where cooperative activities include both 

HUD-sponsored elements and private-sector elements. The partnership structure creates 
87

challenges but does not preclude the use of meaningful outcome metrics that incorporate 

components of PATH as well as components of external activities from providing useful 

input toward a comprehensive program evaluation. It is also important that data be 

considered in the context of the U.S. home-building industry at the time of the 

measurement. Measured values may fluctuate with changes in demand in the housing 

market. Such complicating factors should be considered when selecting, analyzing, and 

reporting metrics.

Another key issue to consider during metrics analysis is the program funding level over 

time. This is particularly relevant when setting targets for future performance. Budget 

decisions can have a profound impact on short-, mid-, and long-term performance targets. 

For example, the scope of the ToolBase program changed significantly in July 2007 when 

one of the two funding vehicles expired. The program previously had been actively 

involved with developing new content, promotions, and other enhancements. Following 

the expiration of one of the funding vehicles, the program switched to a maintenance 

mode that focuses on ensuring the current site is functioning properly but is not actively 

promoting the site or adding significant new content. As a result of this change, the 

recommended metrics covered both the historical period (pre-July 2007) and the current 

maintenance period. Analysis of the metrics data should consider the changing scope of 

the program.

The task of analyzing research and public/private partnership programs such as PATH 

consists of assessing multiple factors and can be complex. The guidance outlined in this 

memo and detailed in accompanying reports, spreadsheets, and metrics memos offers 

structured processes that require in-depth knowledge of the program activities, logic 

models covering outputs and outcomes, an objective metrics-scoring system tailored to 

program goals, and expert judgment in final selections of metrics and analysis of the 

metrics. It is important to keep in mind that each PATH subprogram may require a 

unique approach that may differ from the general guidance outlined here. The individual 

subprograms should ultimately feed into an overall evaluation of PATH’s progress 

toward achieving its goals.
88

10. Conclusions

The development of metrics, baselines, and targets for individual PATH programs 

provides an important step toward generating a long-term comprehensive evaluation of 

the overall PATH program. Further evaluation and testing will enable the development 

of an agreed-upon set of metrics and specific numeric targets for PATH as a whole. The 

recommended metrics presented in this report can be implemented to help measure the 

progress and impact of the PATH program in the coming years, and contribute to a 

continuing program assessment.

Recommended metrics for selected programs provide PATH the ability to evaluate 

program activities and address progress toward achieving its overall goals. These metrics 

from individual programs can be combined and used to inform higher-level metrics –

such as those presented in Section 2 of this report – for external program review. 

Individual program performance measures also can be useful for internal and ongoing 

management-improvement efforts. 

PATH’s efforts to refocus its original goals regarding the promotion and development of 

new technologies and the diffusion of such technologies are evidenced by the intended 

outcomes of the individual program activities examined in this study. The new program 

goals align with its overarching mission to advance technology that improves housing 

performance regarding affordability, durability, sustainability, and safety. This involves 

encouraging innovation, information dissemination for diffusion of technology, and 

administering public-private partnerships – all of which are conducted through numerous 

individual program activities.

The measurement techniques for individual programs presented in this study are intended 

to allow an evaluation of how well the activities each program undertakes are 

contributing to outputs and outcomes that help achieve its goals. The recommended 

metrics (and corresponding baseline and targets) focus on quantity and quality of the 

results of these activities, and, as much as possible, measure improvements in housing 

industry innovation that are attributable to PATH. 

The limited availability of data that measures the influence of program activities on 

decision making in the housing industry presents obstacles for program evaluation. 

Many of the recommended metrics focus on program outputs (e.g., the number of 

products or Web page hits) rather than directly measuring outcomes (e.g., market 

penetration of specific PATH technologies) because data for outputs are more readily 

available. Outcomes metrics are particularly important for gauging how well a program’s

activities align with high-level PATH goals, but are difficult to obtain. In such instances, 

proxies are often the best or only option for identifying particular baselines or targets and 

have been widely used in the recommended metrics.

Attribution also remains a key challenge in measuring program performance. While 

metrics can demonstrate improvements in the number of participating partners, the 

frequency of visitors to specific Web pages, or the number of builders in affiliated trade 
89

groups, determining the influence from PATH involves estimation and professional 

judgment. Identifying market impacts that are the result of PATH activities (as opposed 

to impacts that are due to external factors) is particularly difficult for programs with 

significant public/private partnership involvement. Comprehensive data collection 

efforts would help improve program attribution. Specifically, data on the rate of 

diffusion of new housing technologies would enable PATH to better understand the 

conditions in the marketplace and observe the impacts of the program activities. Such 

data could be collected through surveys or other sources as part of a continuous, 

comprehensive program assessment.

Successful long-term PATH program evaluation will benefit from refining and selecting 

metrics and appropriate targets, expanding data-collection capabilities, and improving 

attribution of PATH efforts to advancing innovation in housing technology.
90

11. References

Cohn, M. (ed.) (2006). “Proceedings of a Workshop to Review PATH Strategy, 

Operating Plan and Performance Measures.” Washington, D.C., National 

Academies Press. Available: http://www.nap.edu/catalog/11661.html

Hassell, S., A. Wong, A. Houser, D. Knopman, and M. Bernstein. (2003). “Building 

Better Homes: Government Strategies for Promoting Innovation in Housing.” 

Prepared for HUD-PATH by the RAND Science and Technology Policy Institute. 

Washington, D.C., RAND Corporation.

Housing and Urban Development (HUD). (2003). “Steel Framing Prototype 

Development: Final Report,” December 2003. Available at 

http://www.huduser.org/Publications/pdf/SteelReport.pdf. 

HUD. (2002). Prescriptive Method for Insulating Concrete Forms in Residential 

Construction, Second Edition,” January 2002. Available at 

http://www.huduser.org/Publications/PDF/icf_2ed.pdf. 

HUD. (1997a) “Prescriptive Method for Residential Cold-Formed Steel Framing,” 

November 1997. Accessed 2008 at 

http://www.huduser.org/publications/destech/pm2.html

HUD. (1997b). “Commentary on the Prescriptive Method for Residential Cold-Formed 

Steel Framing, Second Edition,” November 1997. Accessed 2008 at 

http://www.huduser.org/publications/destech/commento.html

Martin, C. (2005). “PATH: Program Review & Strategy, Performance Metrics, & 

Operating Plan.” Available in appendices in Cohn 2006.

Martin, C. (2003). “ToolBase Summary.” 

McGee, S. (2008). “A History of the ToolBase Portal.” May 2008.

National Renewable Energy Laboratory (NREL). (2008). “Conceptual Soundness and 

Metric Selection Indicators for the PATH Program,” rev April 2008.

National Research Council (NRC). (2008). “Evaluating Research Efficiency.” 

Washington, D.C. The National Academies Press, 2008.

NRC. (2003). “Promoting Innovation: 2002 Assessment of the Partnership for Advancing 

Technology in Housing. Washington, DC: The National Academies Press. 2003.

National Association of Home Builders (NAHB). (2008) “ToolBase.org: Overview of 

Past Performance and Future Enhancements.” NAHB Research Center, April 22, 

2008.
91

NAHB. (2007). “SIPs Code Change Lowers Hurdles on the Job Site.” Oct. 8, 2007.

NAHB. (2003). “Steel Framing Prototype Development: Final Report,” December 2003. 

Available at http://www.toolbase.org/Construction-Methods/ManufacturedHousing/steel-framing-prototype.

NAHB. (2002). “ICFs experience growth in residential market,” Walls and Ceilings

magazine, December 2002. Accessed 2008 at 

http://findarticles.com/p/articles/mi_hb5593/is_/ai_n23570727.

Office of Management and Budget (OMB). (2007). “Guide to the Program Assessment 

Rating Tool (PART).” January 2007

Partnership for Advancing Technology in Housing (PATH). (2007). “PATH Convenes 

Experts and Innovators to Design Concept Home Charleston,” October 26, 2007. 

Accessed 2008 at http://www.pathnet.org/sp.asp?id=24490.

PATH. (2006a). “Prescriptive Method for Structural Insulated Panels (SIPs) Used In 

Wall Systems In Residential Construction,” June 2006. Accessed 2008 at 

http://www.pathnet.org/sp.asp?id=19134. 

PATH. (2006b). “SIPA and PATH Release Prescriptive Method for Building with 

Structural Insulated Panels,” July 2006. Accessed 2008 at 

http://www.pathnet.org/sp.asp?id=19133.

PATH. (2006c). “Steel Framing Technology Development and the Steel Framing 

Alliance,” November 2006. Accessed 2008 at 

http://www.pathnet.org/sp.asp?id=1381.

PATH. (2006d). “HUD Selects Builder and Site for Concept Home Demonstration 

Project.” January 2006. Accessed 2008 at 

<http://www.pathnet.org/sp.asp?id=17403

PATH. (2004). “PATH Unveils Home of the Future,” June 25, 2004. Accessed 2008 at

http://www.pathnet.org/sp.asp?id=12066

PATH.. (2003a). “PATH/ToolBase Expands Services for Builders/Remodelers.” 

Accessed 2008 at http://www.pathnet.org/sp.asp?id=1606. 

PATH. (2003b). “Hybrid Wood and Steel Details – Builder's Guide.” July 2003. 

Available at http://www.pathnet.org/sp.asp?id=10230.

PATH. (2003c). “Prescriptive Method for Connecting Cold-Formed Steel Framing to 

Insulating Concrete Form Walls in Residential Construction.” February 2003. 

Available at http://www.pathnet.org/sp.asp?id=9092.
92

PATH. (2002). “Design for a Cold-Formed Steel Framed Manufactured Home: Technical 

Support Document,” March 2002. Accessed 2008 at 

http://www.pathnet.org/sp.asp?id=9089.

PATH. (2000). “Strategy and Operating Plan.” Washington, D.C., PATH Program.

PATH (1999). “PATH Teams Up With NAHB Research Center's ToolBase SERVICES . 

. . Awards $500,000 to Expand Information Services for Builders, Remodelers.” 

Accessed 2008 at http://www.pathnet.org/sp.asp?id=1626. 

PATH. (1998). “Prescriptive Method for Insulating Concrete Forms in Residential 

Construction,” May 1998. Available at <http://www.pathnet.org/si.asp?id=440>.

PATH and NAHB. (2008). Personal correspondence with PATH and NAHB Research 

Center, May 2008.

Portland Cement Association (PCA). (2008). “Insulating Concrete Forms.” Accessed 

2008 at http://www.cement.org/homes/ch_bs_icf.asp. 

Rogers, E.M. (2003). “Diffusion of Innovations.” New York: Free Press. 

Steel Framing Alliance (SFA). (2007). “Framework: Your Gateway to the Steel Framing 

Industry.” January-Febreuary 2007. Available at 

http://www.steelframing.org/PDF/framework/0107framework.pdf 

SFA. (2003). “Framework: Your Gateway to the Steel Framing Industry.” January-March 

2003. Available at http://www.steelframing.org/PDF/framework/FameworkJanuary-March-03.pdf.

Structural Insulated Panels Association (SIPA). (2007). “Structural Insulated Panels 

Adopted into the International Residential Code,” May 23, 2007. Accessed 2008 

at http://www.sips.org/content/news/index.cfm?PageId=191
93

Appendix A. Metrics for Goals

Individual program metrics align with PATH’s revised goals and objectives, as shown in 

Table A1. Note, however, that the individual program metrics were developed to provide 

insight into how well an individual program was performing and progressing toward 

meeting its goals. While individual program goals contribute to addressing PATH’s 

overall goals shown in this table, not every program is intended to directly address a 

broad range of PATH goals and objectives. Checks were placed in boxes for metrics that 

most closely align with the overall PATH goals and objectives. Several of the overall 

goals and objectives are interrelated and metrics are likely to address some of those that 

are not checked, albeit to a lesser extent. Individual program metrics for these five 

subprograms and additional programs evaluated in the future may be combined and used 

to develop higher-level PATH metrics, which may address a wider range of goals and 

objectives than are checked here.
94

Table A1. Objectives Addressed by Metrics

Goal 1: Remove barriers and 

facilitate technology development 

and adoption

Goal 2: Improve technology transfer, 

development, and adoption through 

information dissemination

Goal 3: Advance housing 

technologies’ research and foster 

development of new technologies

Objectives:

Identify current and potential barriers and measure their impactsDevelop practical methods to overcome current barriersDevelop alternative future industrial processes that eliminate barriersEstablish and maintain noncommercial information sources on innovationUnderstand behaviors, attitudes, and needs regarding innovation informationHave relevant information materials on innovation and innovators for different partiesEstablish sustained public and private R&D investment resourcesClarify and assist innovators with processes from technology transfer to market penetrationHave comparative standards for innovation performance, cost, and benefits & agendas

Concept Home

Cumulative number of industry partners (such as 

designers, builders, technology manufacturers) who 

have participated in Concept Home since the inception 

of the program

x x

Cumulative number of technologies that are promoted 

by Concept Home and included in completed homes 

since the inception of the Concept Home program

x x x

Cumulative potential audience reached through mass 

marketing and press releases (e.g., television 

promotions, circulation of newspapers containing

articles about Concept Home)

x x

Cumulative audience reached via in-person information 

events, such as Concept Home promotional events and 

open houses

x x

Average monthly number of Web page hits that contain 

information about Concept Home and its technologies 

on PATH-sponsored Web sites (i.e., PATHnet, 

ToolBase, HUDUser)

x x
95

Annual percentage growth in the number Google hits 

when searching the following exact phrase: “Partnership 

for Advancing Technology in Housing” “Concept Home”

xx

Annual administrative costs as a percentage of total 

program expenses

Cumulative revenues of products/technologies that 

were incorporated into Concept Home (real $)

xx

Percent of new homes in the U.S. using at least one of 

the technologies showcased in the Concept Home 

program

xxx

ToolBase

Cumulative number of technologies featured in the 

TechInventory.

xx

Cumulative number of Field Evaluations completed with 

results published on ToolBase.

xxx

Total number of TechInventory and Field Evaluations 

page views per total number of user visits/sessions in 

each month. 

xx

Number of broken links in ToolBase.org

x

Total number of unique visitors to ToolBase.org per 

month

xx

Total number of ToolBase.org page views per month

xx

Total number of TechInventory page views per month

xxx

Total number of Field Evaluations page views per 

month

xx

Total number of visitors ('visitor sessions') accessing 

ToolBase.org per square feet of all new residential-

housing units in the U.S. per month

xx

Average number of page views per session

x

Average number of repeat visits per month

x

SIPs

Percent of new homes in the U.S. that are built in a 

location that has adopted the International Residential 

Code (IRC) or other building codes that include SIPs

xxx
96

Average monthly page views of the ToolBase SIPs 

Technology Inventory homepage

xx

Average monthly downloads of the Structural Insulated 

Panels TechSpec (from ToolBase.org site)

xx

Annual percentage growth in the number of Google hits 

when searching the following exact phrase: "Partnership 

for Advancing Technology in Housing" & "Structural 

Insulated Panels" OR "SIPS". 

xx

Number of home builders offering SIPs 

xxxx

Percent of new homes in the U.S. using SIPs

xxx

SIPs annual production for residential use (in square 

feet) per million square feet of all new residential-

housing units in the U.S. per year

xxx

Number of states that have greater than 0.5% of new 

residential-housing units built using SIPs

xx

Steel Framing

Percent of new homes in the U.S. that are built in a 

location that has adopted the International Residential 

Code (IRC) or other building codes that include Steel 

Framing

xxx

Average monthly page views of the ToolBase Steel 

Framing Technology Inventory homepage

xx

Average monthly downloads of the Cold-Formed Steel 

Framing TechSpec (from ToolBase.org site)

xx

Annual percentage growth in the number of Google hits 

when searching the following exact phrase: "Partnership 

for Advancing Technology in Housing" & "Steel 

Framing"

xx

Steel framing annual production for residential use (in 

tons) per million square feet of all new residential-

housing units in the U.S. per year

xxx

Number of home builders offering steel framing 

xxxx

Percent of new homes in the U.S. using steel framing 

xxxx

Number of states that have greater than 1.5% of new 

residential-housing units built using steel framing

xx
97

ICFs

Percent of new homes in the U.S. that are built in a 

location that has adopted the International Residential 

Code (IRC) or other building codes that include ICFs

x x x

Average monthly page views of the ToolBase ICFs 

Technology Inventory homepage x x

Annual percentage growth in the number of Google hits 

when searching the following exact phrase: "Partnership 

for Advancing Technology in Housing" & "Insulating 

Concrete Forms" OR "ICFs"

x x

ICFs annual production for residential use (in square 

feet) per million square feet of all new residentialhousing units in the U.S. per year

x x x

Number of home builders offering ICFs x x x x

Percent of new homes in the U.S. using ICFs x x x

Number of states that have greater than 2.7% of new 

residential-housing units built using ICFs x x
98

Appendix B. Metrics Scoring for Key Characteristics

The following tables present the results from applying the “key characteristics” scoring system described in Chapter 2 to the candidate 

metrics. The scores are used simply to guide the process of refining the candidate set of metrics into a recommended set. The scores 

provide important input for determining the set of recommended metrics but are not the sole determining factor. Balance across 

program areas and stages (e.g., activities, outputs, and outcomes) and judgment about measuring the program impacts that are 

considered essential also represent important criteria for metrics selection.

The scoring system considered four key characteristics (a detailed discussion is provided in Chapter 2), including:

Understandable. The metric should be easily understood by external stakeholders. 

Attributable. It should be easy to determine PATH’s contribution to the metric’s value or result. 

Feasible. Reliable data should be available in order to obtain the metric’s value. 

Comparable over time. The current value of the metric should be able to be directly compared to past and future metric values.

For each metric, and for each of the four characteristics, a score of 0, 1, or 2 is applied. A score of 2 indicates that the metric is highly

desirable for a particular characteristic; 0 is undesirable. The four characteristic scores are summed for each metric. It is important to 

note that the scores are based on current knowledge of the PATH program by the report authors and reviewers. Scores for the 

candidate metrics should be interpreted as a filter that provides insight into the analysis of possible metrics, helping to create a set of 

recommended metrics for the program. Scores should not be viewed as a rigid prioritization, and the scores are not the sole indicators 

for selecting among the candidate metrics – they provide guidance for metrics selection. 

After the candidate metrics are scored, the collective set of metrics is considered regarding its representation of the breadth of PATH

activities, and intended outputs and outcomes. This final step adds expert judgment to the recommendation process and, if necessary, 

allows for the set of recommended metrics to be adjusted or rebalanced. It assures that the set of metrics cover the full range of PATH 

activities, outputs, and outcomes; and that the metrics address the most important intended impacts of the program. Due to this final 

step, the candidate metrics with the highest “key characteristics” scores do not necessarily become part of the set of recommended 

metrics. 
99

Table B1. Concept Home

Bin

Activity Areas for Concept 

Home:

Metric

(Recommended metric # in parentheses)

Measures 

Input, 

Activity, 

Output, or 

Outcome

Understandable Attributable Feasible Comparable 

over time Sum

Conceive

Design

Construct

Demonstrate

Improve

Development

x Number of Core Concepts identified Activity 1 2 2 1 6

x

Number of "principles" created to form the 

foundation for incorporating innovative 

technologies and systems

Output 0 2 1 1 4

Construction

x Number of man hours spent constructing the 

Concept Home Output 2 0 1 2 5

x Frequency of testing and evaluation of the home 

during all periods of construction Output 1 1 1 1 4

Partners and technologies

x x Number of industry partners (#1) Activity 2 1 2 2 7

x x x Number of technologies included in and 

promoted by the Concept Home (#2) Output 2 1 2 2 7

x x Number of technology sponsors Activity 2 1 2 2 7

x

Number of advanced construction technologies 

selected to be included in Concept Home Output 1 2 1 2 6

Information dissemination

x

Number of Web page hits about Concept Home 

on PATHnet, ToolBase, or HUDUser (#5) Output 2 1 2 2 7

x Information availability on Concept Home on 

the internet (via Web search) (#6) Output 1 1 2 2 6

x Number of downloads of "Design Guide " Output 2 1 2 1 6

x Number of open houses per Concept Home Activity 2 1 2 1 6
100

x Number of materials published Output

12126

x Potential audience reached through mass 

marketing (#3) Output

21115

x

Audience reached via in-person information 

events (#4) Output

21115

x Frequency of Concept Home 

Web page updates Activity12115

x Number of promotional events Activity

21115

x Number in attendance 

– for both open houses and 

other events Output

21115

x Audience reached by all publications and materials Output

21115

Administration

efficiency

Administrative costs as a % of total program 

expenses (#7) Input

22116

Market penetration

x Revenues of Concept Home technologies in the 

market (#8) Outcome

21115

x % of new homes using one or more Concept 

Home technologies (#9) Outcome

21115

x Number of green homes built and LEED certified Outcome

20215

x % of new homes using one or more Concept Home 

"principles" Outcome

11013

Improvement

x % of technologies evaluated further by PATH

Activity11114

x Number of ideas for improving cost

-effectiveness, 

quality, etc. in the Concept Home Output

11013
101

Table B2. ToolBase

Bin Metric

(Recommended metric # in parentheses)

Measures 

Input, 

Activity, 

Output, or 

Outcome

Understandable Attributable Feasible Comparable 

over time Sum

Activities

Cumulative number of technologies featured in the TechInventory (#1) Activity 2 2 2 2 8

Cumulative number of Field Evaluations (#2) Activity 1 1 2 2 6

Number of PATH-sponsored promotional events Activity 1 2 1 1 5

PATH's cost share of field evaluations Activity 1 2 1 1 5

Website

Number of broken links in ToolBase.org (#4) Output 1 2 2 1 6

TechInventory and Field Evaluations page views per user visit/session 

in each month (#3) Output 1 1 2 2 6

Frequency of search optimization (for ToolBase.org) conducted Activity 1 2 1 1 5

Number of external links to ToolBase.org Output 1 1 1 1 4

Awareness

Number of ToolBase.org page views per month (#6) Output 2 1 2 2 7

Number of TechInventory page views per month (#7) Output 2 1 2 2 7

Number of Field Evaluations page views per month (#8) Output 2 1 2 2 7

Number of unique visitors to ToolBase.org per month (#5) Output 2 1 2 1 6

Number of visitors ('visitor sessions') accessing ToolBase.org per 

square feet of new residential-housing units in the U.S. per month (#9) Output 1 1 1 1 4

Average number of U.S. states represented by site visitors per month Output 1 0 2 1 4

Understanding

Average number of page views per session (#10) Output 1 1 2 2 6

Average number of repeat visits per month (i.e. Number of visitor 

sessions/Number of unique visitors) (#11) Output 1 1 2 2 6

Average number of page views per day Output 1 1 2 2 6

Average number of new visitors per month Output 1 1 2 2 6

Average number of sessions per day Output 1 1 2 2 6

Enhanced 

services

Number of promotional events per year (#12) Output 2 2 1 1 6

Number of news subscribers – including subscribers to E-news, RSS 

feeds, and print newsletter mailings (#13) Output 2 1 1 1 5

Percentage of E-news subscribers who use a link to access ToolBase Output 2 1 1 1 5

Number of press releases about ToolBase Output 2 1 1 1 5
102

Number of 'email this article' usage per month Output 2 1 1 1 5

Audience reached by all promotional events Output 2 1 0 1 4

Confirmation

Percentage of home builders who are aware of, understand, decide to 

use, and finally, confirm their use of a ToolBase technology (#14) Outcome 0 1 0 1 2

Table B3. Structural Insulated Panels (SIPs)

Bin Metric

(Recommended metric # in parentheses)

Measures 

Input, 

Activity, 

Output, or 

Outcome

Understandable Attributable Feasible Comparable 

over time Sum

Building 

codes

% of new homes built in locations that use the IRC or other building codes 

that include SIPs (#1) Outcome 2 1 1 2 6

Partners and 

technologies

Number of PATH partners (including organizations, builders, or manufacturers) 

for SIP project(s) Activity 2 1 1 2 6

% of market share that “manufacturer” or “builder” partners represent Output 1 1 1 2 5

Information 

dissemination

Average monthly page views of the ToolBase SIPs Technology Inventory 

homepage (#2) Output 2 1 2 2 7

Number of downloads of SIPs TechSpec from ToolBase (#3) Output 2 1 2 2 7

Information availability on PATH SIPs program on the internet (via Web 

search) (#4) Output 1 1 2 2 6

Number of downloads of “Prescriptive Method for SIPs Used In Wall Systems In 

Residential Construction” or “Panelized Wall Systems: Making the Connections“ 

(from PATHnet)

Output 2 1 2 1 6

Market Penetration

Number of home builders offering SIPs (#5) Outcome 2 1 1 2 6

Percent of new homes in the U.S. built using SIPs (#6) Outcome 2 1 1 2 6

SIPs annual production (sq. ft.) for residential use as a fraction of total 

annual residential construction (sq. ft.) (#7) Outcome 1 1 2 2 6

No. of states that have greater than 0.5% of new residential-housing units 

built using SIPs (#8) Outcome 1 1 1 2 5

Market share of construction companies (e.g. builders) offering SIPs Outcome 1 0 1 2 4
103

Table B4. Steel Framing

Bin Metric

(Recommended metric # in parentheses)

Measures 

Input, 

Activity, 

Output, or 

Outcome

Understandable Attributable Feasible Comparable 

over time Sum

Building 

codes

% of new homes built in locations that use the IRC or other building codes 

that include steel framing (#1) Outcome 2 1 1 2 6

Partners and 

technologies

Number of PATH partners (including organizations, builders, or manufacturers) 

for steel framing project(s) Activity 2 1 1 2 6

Number of cooperative agreements with SFA Output 1 1 1 1 4

% of market share that “manufacturer” or “builder” partners represent Output 1 1 1 2 5

Information 

dissemination

Average monthly page views of the ToolBase Steel Framing Technology 

Inventory homepage (#2) Output 2 1 2 2 7

Number of downloads of Cold-formed Steel Framing TechSpec from 

ToolBase (#3) Output 2 1 2 2 7

Information availability on PATH steel framing program on the internet (via 

Web search) (#4) Output 1 1 2 2 6

Number of published technical documents on steel framing and its applications Output 2 1 2 1 6

Market Penetration

Steel framing annual production (tons) for residential use as a fraction of 

total annual residential construction (sq. ft.) (#5) Outcome 1 1 2 2 6

Number of home builders offering steel framing (#6) Outcome 2 1 1 2 6

Percent of new homes in the U.S. built using steel framing (#7) Outcome 2 1 1 2 6

No. of states that have greater than 1.5% of new residential-housing units 

built using steel framing (#8) Outcome 1 1 1 2 5

Number of SFA members (indicating interest in steel framing from 

supplier/builder community) Output 1 0 2 2 5

Market share of construction companies (e.g. builders) offering steel framing Outcome 1 0 1 2 4
104

Table B5. Insulated Concrete Forms (ICFs)

Bin Metric

(Recommended metric # in parentheses)

Measures 

Input, 

Activity, 

Output, or 

Outcome

Understandable Attributable Feasible Comparable 

over time Sum

Building 

codes

% of new homes built in locations that use the IRC or other building codes 

that include ICFs (#1) Outcome 2 1 1 2 6

Partners and 

technologies

Number of PATH partners (including organizations, builders, or manufacturers) 

for ICF project(s) Activity 2 1 1 2 6

% of market share that “manufacturer” or “builder” partners represent Output 1 1 1 2 5

Information 

dissemination

Average monthly page views of the ToolBase ICFs Technology Inventory 

homepage (#2) Output 2 1 2 2 7

Information availability on PATH ICFs program on the internet (via Web 

search) (#3) Output 1 1 2 2 6

Number of downloads or page views of ICF-related articles and reports on 

PATHnet, ToolBase, or HUDUser Output 2 1 2 1 6

Number of downloads of ICFs TechSpec from ToolBase (note: this key 

document does not appear to be available to download) Output 2 1 0 2 5

Market Penetration

ICFs annual production (sq. ft.) for residential use as a fraction of total 

annual residential construction (sq. ft.) (#4) Outcome 1 1 2 2 6

Number of home builders offering ICFs (#5) Outcome 2 1 1 2 6

Percent of new homes in the U.S. built using ICFs (#6) Outcome 2 1 1 2 6

No. of states that have greater than 2.7% of new residential-housing units 

built using ICFs (#7) Outcome 1 1 1 2 5

Number of ICFA members (indicating interest in ICFs from supplier/builder 

community) Output 1 0 2 2 5

Market share of construction companies (e.g. builders) offering ICFs Outcome 1 0 1 2 4
F1147-E(10/2008)

REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing the burden, to Department of Defense, Executive Services and Communications Directorate (0704-0188). Respondents 

should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a 

currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ORGANIZATION.

1. REPORT DATE (DD-MM-YYYY)

January 2009

2. REPORT TYPE

Technical Report

3. DATES COVERED (From - To)

4. TITLE AND SUBTITLE

Conceptual Soundness, Metric Development, Benchmarking, and 

Targeting for PATH Subprogram Evaluation 

5a. CONTRACT NUMBER

DE-AC36-08-GO28308

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

G. Mosey, E. Doris, and C. Coggeshall (NREL); 

M. Antes, J. Ruch, and J. Mortenson (Energetics)

5d. PROJECT NUMBER

NREL/TP-6A2-43822

5e. TASK NUMBER

WF1K.1000

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

National Renewable Energy Laboratory

1617 Cole Blvd.

Golden, CO 80401-3393

8. PERFORMING ORGANIZATION

REPORT NUMBER

NREL/TP-6A2-43822 and

KLCX-4-44453-19

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)

NREL

11. SPONSORING/MONITORING

AGENCY REPORT NUMBER

12. DISTRIBUTION AVAILABILITY STATEMENT

National Technical Information Service

U.S. Department of Commerce

5285 Port Royal Road

Springfield, VA 22161

13. SUPPLEMENTARY NOTES

14. ABSTRACT (Maximum 200 Words)

The objective of this study is to evaluate the conceptual soundness of the U.S. Department of Housing and Urban 

Development (HUD) Partnership for Advancing Technology in Housing (PATH) program’s revised goals and 

establish and apply a framework to identify and recommend metrics that are the most useful for measuring PATH’s 

progress. This report provides an evaluative review of PATH’s revised goals, outlines a structured method for 

identifying and selecting metrics, proposes metrics and benchmarks for a sampling of individual PATH programs, and 

discusses other metrics that potentially could be developed that may add value to the evaluation process. The 

framework and individual program metrics can be used for ongoing management improvement efforts and to inform 

broader program-level metrics for government reporting requirements.

15. SUBJECT TERMS

NREL; U.S. Department of Housing and Urban Development; HUD; Partnership for Advancing Technology in 

Housing; PATH; Concept Home; ToolBase Program; Structural Insulated Panels Program; Steel Framing Program; 

Insulating Concrete Forms Program; Energetics

16. SECURITY CLASSIFICATION OF: 17. LIMITATION 

OF ABSTRACT

UL

18. NUMBER 

OF PAGES

19a. NAME OF RESPONSIBLE PERSON

a. REPORT

Unclassified

b. ABSTRACT

Unclassified

c. THIS PAGE

Unclassified 19b. TELEPHONE NUMBER (Include area code)

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18