# Presentation: 252573

## CPD Outcome  Measurement System

**U. S. Department of **
**Housing and Urban Development**
**Office of Community Planning and Development**

## Overview

**Why outcome measurement now?**
**Approach to developing the CPD Outcome Measurement System**
**Key elements of the System**
**Federal Register Notice **
**Process for implementation**

## Section 1: Why Outcome Measurement Now?

## What is Outcome Measurement?

**Process of identifying goals and measuring whether goals were achieved**
**Focused on answering the question:**
*How did this program make a difference?*
**Not:**
- Individual grantee productivity review
- Monitoring
- Benchmarking

## Why is Outcome Measurement Important?

**Facilitates local grantee decisions about:**
- Program design
- Personnel
- Resource investment
**Helps build a high performing team**
**Required by Federal law**

## Federal Requirements

**Government Performance Results Act**
**President’s Management Agenda**
- OMB/PART
**Common purposes **
- Management reforms to improve Federal program results
- Strive to link budget decision-making with performance results

## What is the PART?	

**OMB management tool **
**Assesses program effectiveness **
**Informs management and funding decisions**
**Based on standard set of factors **
- Rates and compares performance across all Federal programs

## Components of the PART

**Four key elements:**
- Program Purpose and Design
- Strategic Planning
- Program Management
- Program Results
**Rates programs on a scale 1-100 and determines whether programs are “effective”**

## How’d We Do So Far?

**Mixed results:**
- CDBG: Ineffective
- HOME: Moderately Effective
- HOPWA: Results Not Demonstrated
- ESG: Program has not been PARTed
**Formula grant programs with broad goals are particularly difficult to review**
- Many activities
- Numerous entities involved
- Needs are long term and varied across communities

## Why Address Outcome Measurement Now?

**All CPD programs will be PARTed again in next few years**
- PART scores were discussed in the current budget negotiations
**HUD and grantees need to explain how these programs help families and communities**
**To do this, need a common outcome system across all grantees**
- Data that can be nationally aggregated
- Need consistency of reporting type, frequency

## Status of CPD Outcome Measurement System

**Developed/approved by Working Group**
**Briefings About System - Conferences and  Congress**
**HUD issued Federal Register Notice (June 10, 2005)**
**Regional Feedback Sessions in July & early August**
**Final Notice issued by year end**
**HUD is working on how to implement the system**

## Affected HUD Programs

**Community Development Block Grant Program (CDBG)**
**HOME Investment Partnerships Program (HOME)**
**Housing Opportunities for Persons With AIDS Program (HOPWA)**
**Emergency Shelter Grants Program (ESG)**

## Section 2: Approach to Developing the CPD System

## Why is Performance Measurement Important?

**Now required by Federal law **
- Government Performance Results Act (GPRA)
- President’s Management Agenda (OMB/PART)
**Facilitates management decisions about:**
- Program design
- Personnel
- Resource investment
**Important to “telling our story” about our accomplishments to Congress**

**Notes:**
Telling our Story - As you heard, performance measurement is now required. In today’s tight budget environment, all domestic program must be defendable and demonstrate results. When CDBG was proposed to be transferred and consolidated with other programs at Commerce, part of the Administration’s rationale was that the program lacked proper outcomes. But because we had developed this outcome framework, this made all the difference on the Hill. Members of Congress on both the House and Senate side, pointed to our outcome framework and list of indicators as justification not to move the program. 

## Implications for HUD CPD Grantees 

**We must report on our accomplishments in a new way**
- Describe not only outputs (such as units or households) but also the outcomes, results of our programs
**The proposed Outcome Framework will help HUD to better demonstrate CPD program effectiveness  **

**Notes:**
After Second Bullet - The proposed outcome framework meets the GRPA  and PART requirements and covers most of programs and activities of CPD’s four core formula programs. It does not change any of the programs ( CDBG, HOME, HOPWA or ESG), it just provides a different way to report on and examine what we are already doing at the state, city and county levels in our programs. 

## Grantee Organizations Responded

**Over the past 2 years:**
**COSCDA group developed draft Outcome Measures Framework to provide starting point for bigger group**
**Working group formed to further refine the framework – over 25 people**

## Working Group 

- Council of State Community Development Agencies (COSCDA)
- National Community Development Association (NCDA)
- National Association for County Community Economic Development (NACCED)
- National Association of Housing and Redevelopment Officials (NAHRO)
- National Council of State Housing Agencies (NCHSA)
- State Grantees
- Entitlement Grantees and PJs, including cities and counties
- Key HUD Staff from HUD’s Office of Community Planning and Development
- Key OMB Staff in charge of evaluating our programs

## The Reason 

**We needed a way to tell our story**
**To support and strengthen CDBG, HOME, HOPWA and ESG programs by depicting value through outcomes**
**To respond to the assessment of CDBG and HOME by OMB “performance assessment rating tool” (PART)**
**To comply with HUD CPD Notice 03-09**

## The Reason (cont.)

**To create a way for HUD grantees to report on their individual program outcomes while simultaneously participating in a nationwide reporting process**
**To provide framework for positive change of Con Plan process and current reporting means (PERS, CAPER)**

**Notes:**
After First Bullet - Our idea was to have HUD aggregate and study the outcomes of our activities so that they might be able to better report on the effectiveness of these programs.

## The Task

**Performance Measures for HUD Con Plan Programs- All Four CPD Formula Programs!**
- HOME Investment Partnership
- Community Development Block Grant
- Emergency Shelter Grant
- Housing Opportunities for Persons With AIDS

**Notes:**
In choosing to create a system for all four programs together, we recognized that there are specific outcomes and purposes for each of these programs, but we felt is was important to look to the statute to guide us in deciding on what to report to Congress. We found that goals and objectives in each of the program statutes were very similar or overlapped, and that is how we arrived at the overarching goals or objectives as they are called in the HUD notice. And, as  we delved into it we also realized that there are several activities in each of these programs that are the same or extremely similar, which result in the same outcomes. 
For example, there are rehab programs in both HOME and CDBG, there are new homeowner programs in both HOME and CDBG, there are shelter assistance programs in both CDBG and ESG, there are new units make affordable by HOME, ESG and HOPWA, and so on. And, as stated earlier, we were also looking to find a uniform way to plan and report on the programs covered by the Consolidated Plan in our CAPERs, PERs and IDIS. 

## What Happened in the Last Year? 

**Working Group met several times to develop outcomes and indicators**
**Innovative consensus building process**
**Extensive participation by Grantees:**
- Selected outcomes and indicators, using stringent screens and criteria to assure system will enable us to better tell our story to Congress without overburdening grantees
- Selected data sources
- Tested Framework on real activities and projects

**Notes:**
After bullet on Innovative consensus process – grantees, HUD and OMB had equal votes, the process was respectful and no one point of view dominated

After last bullet - We looked at a number of systems and several indicators before we reached consensus on the ones on the list in the notice. We had several criteria, such as – 

Was the indicator and outcome an important result of the programs  that we wanted to make sure  Congress and OMB were aware of ? 
Was the data already reported to HUD?
if not, was already available at the grantee level or the subrecipient level? We had to consider the state, cities and counties, as data availability varies across these 3 different levels of government.
Was it going to burden a lot of grantees and subrecipients?
 Was it  new data that was going to have to be collected, and if so, was it really worth it?  

And yes, there are some new data that you will have to collect and report, you will hear about that later in this presentation, but, considering the wide array of indicators proposed by OMB and HUD, we think we did a good job in minimizing the data burden to grantees. After we go through the list we will fill you in on some of reasons why certain indicators were selected in order to be responsive to OMB and HUD’s view of what outcomes they wanted us to report on.

## How did we approach the problem?

**Ground rules used to create Outcome Framework and Performance Outcome Reporting System:**
- System had to maintain program flexibility and reflect bottom-up approach to project development by grantees (local choice)
- System had to differentiate our programs from other federal programs that fund similar activities
- System had to work for all Con Plan programs
- System had to reflect true program(s) purpose

**Notes:**
After first bullet 
Having these outcomes do not overrule local decision making on projects, programs and activities. If you don’t do a program or project or activity that pertains to a particular outcome or indicator then you don’t have to change your programming just so you have something to report on in that section, no, you don’t have to report at all on it. There are only a few outcomes and indicators which are mandatory under this system, and they are designed to be applicable to all programs and activities.

After 2nd bullet
We had to show that the results of these programs cannot be achieved by other federal programs so we chose outcomes and indicators with that in mind.

## How? - continued

- System had to recognize opportunity for multiple outcomes
- System had to recognize similar projects funded for different purposes
- System had to utilize readily available data collected at application stage or close-out stage
- System had to define outcomes, not impacts

**Notes:**
It is not uncommon for outcome measurement systems to  allow for multiple outcomes from a particular effort.  
As we developed the system it became clear that there are going to be some projects that have multiple outcomes and indicators that apply to it, that is ok, we won’t be double counting the units or people, we will just be describing the results of the project in more than one way. 

Similar projects funded for different purposes – the beauty of this system is its flexibility. The key to understanding and choosing which outcome to report on is to focus on your intent when you funded the project or activity. Why I funded a project in my city or state can be different than why you did, that is why the same project may result in a different outcome and why you and I will use a different indicator or outcome category for the same activity.

The outcomes we chose are qualitative descriptions of the results of the projects and activities we fund, they are not measurements of the impacts of the activities on people or places. This was a big compromise for OMB.  OMB  wanted us, the grantees, to measure what they thought the impacts of CDBG should be, such us, the impact of CDBG on the crime rate in a neighborhood. We tired and rried to come up with ways to do so, and finally we successfully convinced thme that It was impossible to develop a standardized set of data and impacts across cities, counties and states to use in assessing impact. That does not mean you should not try to do so at the state or local level if that is something that is important to your community.  And HUD may be able to study the impact of our programs with some of the outcome data we will now five them. 

## How? - continued

- System had to allow for “roll-up” of data to national level to show accomplishments and value
- System had to minimize change in data collection
- System could not create additional work that has no value

**Notes:**
In other words, we didn’t want to just be sending more info to Washington that you wonder what the heck they need or want it for or wonder if anyone looks at it!

## Ties to Con Plan, Annual Action Plan, IDIS and CAPER 

**Group also made several recommendations on how to Incorporate into IDIS And Con Plan**
**A proper planning system: **
**	****Plan > Measure > Report >Plan, again**
**Reduced burden – redesigned annual Action Plan and redesigned CAPER/PERS**
**Focus reporting on outcomes instead of outputs**
**Symmetry between project timelines and Consolidated Plan versus project timelines and annual year**

**Notes:**
We recommended that the Consolidated Plan, Annual Plan, CAPER, PER, and IDIS be revised to integrate this new outcome measurement system. Reporting on outcomes should be based on the Con Plan term, not the program year.  

## Outcome Indicators 

**17 Indicators to chose from**
**Some indicators are required given the activity**
- Other indicators will be chosen by the Grantee given the specific activity and the Grantee’s intended outcome from the project

**Notes:**
In the next presentation you will hear about these indicators

## Section 3: Key Elements of the System

## Outcome Measurement System:  Key Elements

**Three Main Components**
- Objectives
- Outcomes
- Indicators

## Outcome Measurement Framework Steps

## Outcome Measurement System:  Selecting a Objective

**Step 2: ****Select **_**Objectives**_** Related to Goals**

**Suitable Living Environment **

**Decent Housing**
** **

**Economic Opportunity **

**Choose a objective based on:**
- Type of activity
- Funding source
- Local program intent

## Outcome Measurement System:  Selecting an Outcome

**Choose an outcome based on:**
- Purpose for the activity
- Answer the question: *Why did the grantee fund this activity?*
**Can have more than one outcome per activity**
- Example: housing project in target neighborhood that both improves affordability and sustainability

## Defining the Outcomes

**Availability/Accessibility:**
- Make basics available to LMI persons
- Example: providing clean water where none existed before
**Affordability:**
- Makes an activity more affordable for LMI persons
- Example: providing low interest loans to LMI homebuyers
**Sustainability:**
- Using resources in a targeted area to help make that area more viable
- Example: a downtown improvement project in a LMI community

## Determining Results – Outcomes & Outcome Statements

**Together objectives and outcomes result in outcome statements**
- Combine the objective and outcome to identify the outcome statement
**Example:**
- Objective: Decent housing
- Outcome: Affordability
- Resulting Outcome Statement:  Create decent housing through improved affordability

## Link Between Objectives, Outcome, and Outcome Statements

## Measuring Outcomes – Using Indicators

**What are indicators?**
**Indicators tell whether an outcome is occurring**
**Can be direct or indirect measures (proxy)**
**A set of indicators is often used to measure an outcome**
**Individual indicators can be used for more than one outcome**

## Measuring Outcomes – Using Indicators

**Indicators used in this framework:**
**Five common indicators are relevant to most activities**
**Remaining 17 specific indicators used when apply to the:**
- Activity; and
- Intent (i.e., outcome)
**Some indicators have several elements**

## Common Outcome Indicators

**For most activities, grantees report:**
- Funds leveraged
- Number of persons, households, units
- Income levels of persons or households by 30%, 50%, 60% or 80% of an area median income
- Number of communities/neighborhoods assisted
- Current Racial/Ethnic and disability categories
**For some CDBG activities, not all will be required depending on national objective**

## What are the Specific Indicators?

**Vary depending on activity**
**Some indicators are similar across activities**
- For example: Number of 504 accessible units is reported for both rental rehab and rental new construction
**Some activities will have multiple outcomes and multiple types of indicators**

## Types of Specific Indicators

**Infrastructure or public service (indicator 1):**
- Persons with new or improved access or increase in standard service
**Targeted revitalization (indicator 2):**
- Report on range of outcomes such as jobs, businesses, households etc. in target area
**Other physical improvements (indicators 3, 4, 5):**
- Addressing slum/blight, commercial facades, acres of brownfields

## Types of Specific Indicators

**Rental Housing (indicators 6, 7)**
- Report on units and accessibility, as well as topics such as years of affordability, units for chronically homeless persons
**Homeowner rehab (indicator 8)**
- Indicators such as standard units, units meeting IBC and Energy Star energy standards, units in LBP compliance
**Homeownership (indicator 9)**
- Answer series of yes/no questions about homebuyers and assistance

## Types of Specific Indicators

**Job creation/retention (indicators 10, 11)**
- Report on health benefits, type of job, employment status
**Business assistance (indicators 12, 13, 14)**
- Indicators such as number of businesses, DUNS number, NAIC code, whether business serves neighborhood
- Business operations (13) will be determined by HUD

## Types of Specific Indicators

**Homebuyer units developed (indicator 15)**
- Report on indicators such as number of units, years of affordability, number 504 accessible, number subsidized by program
**TBRA (indicator 16)**
- Indicators such as number of units, number of units for persons with AIDS
**Homeless shelter (indicator 17)**
- Number of persons stabilized

## Selecting Specific Outcome Indicators

**Key questions**
- What is the intent of this activity?
- What types of entities will benefit from this activity?
- Does this activity benefit a specific, targeted geographic area?
- Does the activity have a physical impact on a site or area?
- Does this activity create jobs or create/sustain businesses?
- Is this activity targeted at a particular population (e.g., persons with AIDS, homeless persons, disabled persons)?

## Implementation of the Outcome Indicators

**Depending on the activity, IDIS will offer list of available indicators**
**Indicators will be required if they apply to the activity**
- For example: if doing job creation activity will not only report on number of jobs but also on:
- Employer health benefits
- Type of job
- Employment status before job

## Selecting Outcomes & Indicators – CDBG Example

**Example #1: Road to industrial park**
- Objective – Economic Opportunity
- Outcome – Improved Accessibility
- Outcome Statement – Provide economic opportunity through improved accessibility
- Examples of Indicators:
- Number of new businesses assisted
- Good or service that business provides to meet community need
- Number of jobs
- Of these, jobs with health benefits
- Employment status before before taking job

## Selecting Outcomes & Indicators – CDBG Example

**Example #2: Housing rehabilitation in target area**
- Objective – Suitable living environment & decent housing
- Outcomes – Improved sustainability & affordability
- Outcome Statements –
- Create decent housing with improved affordability
- Enhance suitable living environ. with improved sustainability
- Examples of Indicators:
- Number of units brought from substandard to standard condition
- Number of units brought into compliance with lead safe housing rule
- Number of units meeting IBC or Energy Star standards

## Selecting Outcomes & Indicators – CDBG Example

**Example #3: An after school gang prevention program (public service)**
- Objective – Suitable living environment
- Outcome – Improved accessibility
- Outcome Statement –
- Enhance suitable living environ. through improved accessibility
- Examples of Indicators:
- Amount of money leveraged
- Persons assisted
- Number of communities assisted
- Number of households assisted with new access to a service

## Selecting Outcomes & Indicators – HOME Example

**Example: Rental rehab project**
- Objective – Decent housing
- Outcomes – Improved affordability & availability
- Outcome Statements:
- Create decent housing with improved affordability
- Create decent housing with improved availability
- Examples of Indicators:
- Total number of units assisted with HOME Program funds
- Number of years of affordability guaranteed
- Number of existing units rehabbed
- Amount of money leveraged
- Number of units made 504 accessible

## Selecting Outcomes & Indicators –HOPWA Example

**Example: Services for persons with HIV/AIDS**
- Objective – Suitable living environment
- Outcome – Improved accessibility
- Outcome Statement:
- Enhance suitable living environment with improved access
- Examples of Indicators:
- Amount of money leveraged
- Number of persons or households (HH) assisted
- Number of HH with improved access to service or benefits
- Income level by HH (30%, 50%, 60%, 80%)

## Selecting Outcomes & Indicators – ESG Example

**Example: Transitional housing for the homeless**
- Objective – Decent housing
- Outcomes – Improved availability & affordability
- Outcome Statements:
- Create decent housing with improved availability
- Create decent housing with improved affordability
- Examples of Indicators:
- Number of affordable units rehabbed for the homeless
- Number of affordable units rehabbed for the chronically homeless
- Total number of units assisted with ESG Program funds
- Number of years affordability guaranteed

## Outcome Measurement System – Keep In Mind

**System will evolve and be refined over time**
- These outcomes and indicators were most common and least burdensome
- Working Group recognizes that not all activities are covered
**Grantees encouraged to develop additional outcomes and indicators for use locally**
- Helps to tell the story locally
- Local decision – do not have to report to HUD

## Using the Outcome Measurement System

**Grantees will need to:**
- Determine _goals_ of activities/projects
- Select _objectives_ and _outcomes_
- Indicate anticipated outcomes in Con Plan (annual)
- Indicate outcomes and report on applicable indicators in CAPER/PER/IDIS
**HUD will aggregate data and report on outcomes at a national level**
- Report outcomes to public, Congress, & OMB
- Grantees present results locally

## Why These Indicators?

**New data and reporting will be required**
**Carefully selected**
**Recommended indicators – compromise and consensus**

**Notes:**
Grantees, HUD and OMB all had to compromise. As stated earlier, there were several indicators discussed and debated in our process. All in all, over 35 indicators were initially considered by the Group, many with subsets like the ones in the consensus list. Uniformity and availability of data were key obstacles to many of the indicators OMB and HUD originally wanted, such as improved wealth of beneficiaries after homeownership, impact of HOME or CDBG activities on the crime rate in a neighborhood, wage rates of new hires in jobs created, or whether the jobs provided a livable wage.  While certainly some of these impacts  would be worth while to know, they were too labor intensive for subrecipients or grantees, and because of the differences in the way state, counties and cities operate their programs it was impossible to get standardized data at this time. 
There were also a number of outcomes and indicators proposed by HUD and OMB that grantees argued against because we did not agree that the indicators reflected our reasons for funding certain activities. For example, OMB wanted to the outcome of rehab programs to result in increased property values and asset accumulation. Well, in our view, that could mean gentrification, something we try to avoid, or many times rehab programs do emergency repairs or weatherization to help keep an elderly homeowners in their life long homes, not to increase their property assessment so they have to pay increased taxes. We were able to explain these issues to OMB and HUD and successfully pared down the list of results to common outcomes that we believe will help us tell a compelling story to Congress.  

## Summary

**Performance measurement is now mandatory**
- Collecting and reporting performance data will not be optional but individual outcomes indicators will vary depending on activity
**Proposed Outcome framework will not change the activities chosen by grantees but may require new ways of reporting data**
- Added data collection burden has been minimized
**Program flexibility is maintained - Objectives and outcomes are determined by grantees based on the intent of the activity**
**Grantees can add other objectives and outcomes specific to their state or local initiatives or priorities**

## Timing is Crucial

**The Working Group encourages grantees to offer refinements to these proposed indicators during this process**
**Constructive input encouraged **
**Recommend implementation in FY2007 **

**Notes:**
Here we need to say that while we welcome and encourage comments, we should not attempt any wholesale changes to the proposed system unless it is really unworkable. Why? We need to continue implementation so that we can continue to use the system as an important justification for continued funding for the programs. Also, the system may evolve over time but if we present a united or collected opinion on these indicators, we will be better able to “ward off” some the more unreasonable indicators OMB and other s would like to see put into place.

## Section 4: Federal Register Notice & Process for Implementation

## Federal Register Notice (June 10, 2005)

**Outlines purpose & key features of proposed CPD outcome measurement system**
**Solicits comments**
**Key sections:**
- Part I –     Background
- Part II –    Performance Measurement Objectives
- Part III –   Opportunities for Public Participation
- Appendix – Proposed Outcome Measurement System

## Process for Implementation

**Federal Register Notice **
**Regional Sessions July & August 2005**
- Education
- Facilitated feedback
**Issue final notice December 2005**
**Offer training on final system & its use**
**Trial run of updated IDIS - Spring 2006**
**Revised IDIS system available Fall 2006**

## Process for Implementation

**Grantee transition**
- Provide feedback
- Final HUD Notice issued
- Grantees attend training, receive additional guidance from HUD, use trial version of IDIS
- Identify the selected outcomes and indicators in their next Con Plans with a transition period (next Annual Plan or 5-year Plan, whichever comes first)
- Transition to reporting on applicable indicators during 2007 Program Year through CAPER, PER, and IDIS

## Working Group Implementation Recommendations

**Reduce grantee burden – design changes for:**
- Consolidated Plan
- Annual Action Plan
- CAPER/PER

## Suggested Changes to Con Plan
Create a template to include performance measures
Suggested Changes to Annual Plan
Annual plan to only be updates to 5 yr
Move some 5 yr items
Delete listing of individual projects

- Create a template to include performance measures
**Suggested Changes to Annual Plan**
- Annual plan to only be updates to 5 yr
- Move some 5 yr items
- Delete listing of individual projects

**Working Group Implementation Recommendations**

## CAPER/PER & IDIS
Numbers reported throughout year
Narrative still submitted annually
Summary report on performance at end of Con Plan period (3 to 5 yrs)
IDIS will take the place of paper reports
IDIS will be updated with a trial period and “test users”

- Numbers reported throughout year
- Narrative still submitted annually
- Summary report on performance at end of Con Plan period (3 to 5 yrs)
- IDIS will take the place of paper reports
- IDIS will be updated with a trial period and “test users”

**Working Group Implementation Recommendations**

## National Training
Performance Measures
IDIS
Con Plan
Anticipate offering training in Winter 2006

- Performance Measures
- IDIS
- Con Plan
**Anticipate offering training in Winter 2006**

**Working Group Implementation Recommendations**

## Section 5: Today’s Session – Gathering Your Feedback

## This Feedback Session

**Join any break out session**
- All sessions same
- States encouraged to go to identified room
- Urban counties and HOME consortia may wish to join states, others also welcome
**In the break-out session**
- Ask questions about the outcome system
- Share your feedback on the system and indicators
- Ask questions about HUD plans for implementation
- Provide feedback on the implementation process

## This Feedback Session

**As you provide comments, consider:**
- Does the system (outcomes and indicators) cover most of the key activities you fund?
- What about the proposed system needs to be changed or further clarified?
- What barriers do you see to the implementation of the system and how can these be overcome?

## Also consider these questions:
What information regarding Consolidated Plan changes do you need most? 
What information regarding changes to IDIS, CAPER, and PER do you need most? 
What types of outreach materials or training would most help you to implement this system?

- What information regarding Consolidated Plan changes do you need most?
- What information regarding changes to IDIS, CAPER, and PER do you need most?
- What types of outreach materials or training would most help you to implement this system?

**This Feedback Session**

## Next Steps

**Select a break-out group**
**Break-outs will last from 1:15 until 5:00**
- Includes 30 minutes at end for break-out wrap-up
**After all five regional sessions:**
- HUD will summarize comments and make available
- Working group and HUD will review comments and make changes
- Final notice will be published
- Training and implementation will begin

## Questions?

**Come to microphone for any questions on:**
- Reasons for system
- Working group approach
- Structure of the framework
- Implementation
**Lunch from 11:45 to 1:15**
- See restaurant list in packet
**Thank you for attending and sharing your insight!**