of CPD Formula Grants

Risk-Based

 Monitoring

Risk-Based

 Monitoring

U.S. Department of Housing and Urban Development

Office of Policy Development and Research
Risk-Based

 Monitoring

of CPD Formula Grants

U.S. Department of Housing and Urban Development

Office of Policy Development and Research

Paul Joice

U.S. Department of Housing and Urban Development

Office of Policy Development and Research

Program Evaluation Division

Brent Mast

U.S. Department of Housing and Urban Development

Office of Policy Development and Research

Monitoring Division

December 2009
Risk-Based Monitoring of CPD Formula Grants iii

Contents

I. Executive Summary ................................................................ 1

II. Introduction ........................................................................... 3

A. Background of CPD Risk-Based Monitoring................................ 3

B. Extant Research.................................................................... 4

C. Overview of Study Methodology .............................................. 5

III. Process................................................................................... 7

A. Annual Risk Analysis Procedure............................................... 7

B. Year-to-Year Changes .......................................................... 10

IV. General Analysis................................................................... 12

A. Overview of Available Data................................................... 12

B. Comparison of Means .......................................................... 14

C. General Conclusions ............................................................ 15

V. Subfactor Analysis................................................................ 17

A. Correlation Analysis............................................................. 17

B. Multivariate Analysis............................................................ 18

CDBG................................................................................ 19

HOME................................................................................ 20

C. Subfactor Conclusions.......................................................... 21

VI. Potential Improvements ....................................................... 25

Appendixes

A. Data Management and Manipulation........................................... 29

B. Bivariate Correlation Analysis .................................................... 31

C. Full Correlation Matrix .............................................................. 35

D. CDBG Ordinal Logit.................................................................. 40

E. HOME Ordinal Logit.................................................................. 43

F.	 Field Office Random Effects....................................................... 46

G.	 Subfactor Definitions (adapted from the Notice CPD-06-10) .......... 48
Risk-Based Monitoring of CPD Formula Grants 1

I. Executive Summary

This report presents analysis conducted by the U.S. Department of Housing and 

Urban Development’s (HUD’s) Office of Policy Development and Research (PD&R)

to determine the effectiveness of the risk analysis process used by HUD’s Office

of Community Planning and Development (CPD). Risk analysis is conducted on

recipients of grants from CPD in order to identify those programs most susceptible 

to waste, fraud, abuse, or mismanagement. CPD staff then monitor the programs 

identified as high risk as well as a sample of other lower risk programs. This

report focuses on risk analysis and monitoring of CPD’s four formula grants: the 

Community Development Block Grant (CDBG), HOME Investment Partnerships

Program (HOME), Emergency Shelter Grants (ESG), and Housing Opportunities for

Persons with AIDS (HOPWA).

We find that, in the CDBG and HOME programs, grantees that score high in the

risk analysis process are significantly more likely to have “findings” (statutory or

regulatory violations) than those with low risk scores. In the ESG and HOPWA

programs, the evidence is inconclusive. This suggests that, at least for CDBG 

and HOME, the risk analysis process is accurately identifying “risk.” Yet there is

room for improvement, particularly in two respects. First, with the abundance 

of data available to HUD, it would be reasonable to expect a higher degree of 

success. Half of low- and medium-risk grantees still have findings, and a third of

“high risk” grantees have none. Second, the risk analysis process is a drain on

the scarce time and resources of CPD field staff. Revisions to the process could

provide savings to the Department in terms of staff time. As CPD receives new 

responsibilities, including the Neighborhood Stabilization Program (NSP), supplemental funding through CDBG and HOME, and the new Homelessness Prevention 

and Rapid Re-Housing Program (HPRP), it is critical that overarching tasks like risk

analysis and monitoring are effective and efficient.

To identify potential improvements, this report examines the components of the 

risk analysis process. Currently, many of the “subfactors” that make up the risk

analysis process have complicated, multifaceted definitions; furthermore, they

often have no statistical relationship with the number of findings a program will

have. In the HOME program, CPD staff rate grantees on 21 subfactors to determine the risk score; yet only four of these subfactors actually have a statistically

significant relationship with findings. The CDBG risk analysis process includes 17

subfactors, only five of which are statistically significant predictors of findings.

We also find that there is significant variation across field offices in the number of

findings issued, after accounting for all other variables in the model.

To save staff time and maintain a standardized system for assessing risk, CPD

could make the following five adjustments to the risk analysis process:
Risk-Based Monitoring of CPD Formula Grants 2

•	Use fewer subfactors, which simply and directly estimate staff capacity, 

program complexity, and past performance.

•	Develop a subfactor to explicitly incorporate the judgment of the evaluator and/or CPD Management Representative.

•	Ensure strict adherence to limited exception criteria.

•	Randomly sample low- and medium-risk grantees for monitoring. 

•	Increase reliance on remote monitoring for low- and medium-risk

grantees.

The greatest benefit of these changes would be a reduction in the time and

resources required for risk analysis and monitoring, but there are two very valuable additional benefits. First, including an element of randomness in the process,

specifically for lower risk grantees, should improve performance by maintaining a

constant threat of monitoring. Grantees must be prepared to be monitored each 

year. Second, these changes will be highly conducive to further testing. Random 

selection of a subset of the grantees to be monitored will allow for extremely 

rigorous statistical analysis. In addition, the recommended adjustments to the

subfactors will make it easier to tweak the subfactor weightings in the future. 

Many of these recommendations represent substantial changes to the way CPD 

currently does risk analysis and monitoring. For newly created CPD programs 

such as NSP and HPRP, some of these characteristics could immediately form the

foundation of the risk analysis and monitoring guidelines that must be developed. 

For CDBG, ESG, HOME, and HOPWA, however, it may be necessary to gradually

adjust the process rather than overhauling it all at once. It may even be possible

and desirable to test the new model; for example, the many small competitive

grant programs run by CPD, which require monitoring resources disproportionate 

to their size, could be used in a demonstration. PD&R looks forward to working

with CPD staff to continue to improve this important administrative practice. 
Risk-Based Monitoring of CPD Formula Grants 3

II. Introduction

A. Background of CPD Risk-Based Monitoring

To meet the requirements of the Federal Managers’ Financial Integrity Act

of 1982, Office of Management and Budget (OMB) Circular A-123, and OMB

Circular A-127, the U.S. Department of Housing and Urban Development 

(HUD) has established a system of management controls and set forth

these policies in Handbook 1840.1, Departmental Management Control 

Program. The purpose of the Management Control Program is to protect 

against fraud, waste, abuse and mismanagement in HUD’s programs. 

Chapter 2 of the Departmental Management Control Program handbook 

provides guidance for defining risk and determining how susceptible HUD

programs are to fraud, waste, abuse, and mismanagement. For HUD’s 

Office of Community Planning and Development (CPD) grant programs, this

risk is directly dependent on the risk posed by grantees, so CPD’s Management Control Program consists primarily of risk analysis and monitoring 

of grantees. Chapter 7 of the Departmental Management Control Program

handbook provides guidance on risk-based monitoring and states that all 

monitoring should incorporate risk-based concepts and strategies.

CPD’s risk analysis process is a highly structured and quantitative system 

for rating and ranking grantees and their programs, in order to identify 

those that pose the greatest risk to the integrity of CPD’s programs. The 

risk analysis process has two important benefits. First, it ensures that CPD

grantees are treated consistently across HUD’s 43 CPD field offices. Second,

it enables CPD to allocate scarce monitoring and technical assistance 

resources efficiently and effectively. This second point is critical. According

to a report published by the Government Accountability Office (GAO), from

1993 to 2006, CPD staff decreased by 20 percent while CPD responsibilities 

increased substantially.1 Over the same time period, the Community Development Block Grant (CDBG) program experienced a 27 percent increase in

the number of entitlement communities, and funding tripled for the Housing Opportunities for Persons With AIDS (HOPWA) and Emergency Shelter

Grants (ESG) programs.2 In some ways, these trends have continued since

2006, leading to an even larger gap between resources and responsibilities. 

In the same report, GAO also noted that 39 percent of CPD field office staff

would be eligible to retire by February 2009. That very month—February 

2009—saw the passage of the American Reinvestment and Recovery Act, 

1 “Community Development Block Grants; Program Offers Recipients Flexibility but Oversight Can Be

Improved.” July 2006. (http://www.gao.gov/new.items/d06732.pdf).

2 http://www.hud.gov/offices/cpd/homeless/budget/esghistory.pdf.
Risk-Based Monitoring of CPD Formula Grants 4

which channels $6.75 billion through CPD programs. As CPD’s responsibilities swell, its resources are increasingly strained.

It is more and more important that CPD’s risk analysis and monitoring system

is streamlined, efficient, and effective. A streamlined process reduces redundancy and avoids unnecessary complexity. An efficient process reduces

the amount of time required by CPD staff to accomplish a set amount of 

work. An effective process ensures that the work being done is accomplishing 

its ultimate goals. Applied to risk analysis, this means a process that is simple 

and logical, can be done quickly and easily by CPD staff, and successfully 

identifies and targets resources to grantees that pose the greatest risk to

HUD. The Management Control Program handbook stipulates that the riskbased monitoring strategy should be periodically evaluated to determine its 

effectiveness and whether modifications should be made to better achieve

objectives. That directive is the impetus for this study.

The purpose of the CPD risk analysis process is to identify programs that 

pose the greatest risk to the integrity of CPD’s programs, so that a greater 

share of monitoring resources can be used to mitigate that risk. The 

primary objective of the study is to use statistical analysis to determine

how effectively CPD’s risk analysis process targets monitoring resources to 

grantees that pose the greatest risk to HUD. The report will also identify 

opportunities to make the process more streamlined and efficient.

B. Extant Research

The concept of risk has been thoroughly studied by both academics and 

practitioners, but published research tends to be primarily in the fields of

finance and medicine—the risk of a financial instrument defaulting, or the

risk of a negative health outcome. These studies are relevant to the choice 

of statistical models; however, the types of risks differ so extensively from

risk in a federal government grant program that this published research 

offers little guidance. 

In the federal government, GAO is the primary watchdog for financial

management. As noted previously, GAO has specifically studied CPD’s riskbased monitoring process. Given the federal government’s widespread use 

of grants—to states and local governments as well as non-governmental 

entities—the GAO has published an abundance of material on grants management and mitigating risk. However, it appears that the GAO has never 

published a rigorous statistical analysis of a federal agency’s risk-based 

grants monitoring strategy, such as that which is presented in this report. 

The most relevant published report is a review of the Federal Motor Carrier 
Risk-Based Monitoring of CPD Formula Grants 5

Safety Administration’s (FMCSA’s) program for reducing the number of

crashes involving large trucks and buses that operate in interstate commerce.3 FMCSA uses a data-driven analytical model called SafeStat to 

identify carriers that pose high crash risks. SafeStat creates a risk score 

using a formula that weights certain vehicle safety related data elements. 

SafeStat is similar in many respects to CPD’s risk analysis process but 

relies more on data and less on subjective judgment. The GAO report used

a negative binomial regression to conduct a parallel risk analysis and found 

that, over an 18-month followup period, the GAO model was more successful 

than SafeStat in identifying carriers that pose the greatest crash risk. The 

report recommended that FMCSA adopt a similar regression-based risk model.

C. Overview of Study Methodology

Several aspects of the CPD risk analysis and monitoring process make it 

highly conducive to statistical evaluation. 

•	First, the process includes a rating system that results in scores for each 

program and each grantee on a series of “subfactors.” These subfactors

work fairly well as independent variables that one would expect to contribute to a program’s riskiness. 

•	Second, all instances of monitoring have a documented outcome: a finding, a concern, or neither. A finding can only be issued for a violation of

a “statutory, regulatory or program requirement,” which makes it more

significant and more clearly defined than a concern. Although they are

not perfectly indicative of “fraud, waste, abuse and mismanagement,”

findings work fairly well as a dependent variable that proxies risk.

•	Third, both “high-risk” and “low-risk” programs (as determined by the

risk analysis process) are monitored, allowing for comparison of the two

groups. The extent to which “high-risk” programs have more findings

than “low-risk” programs will indicate how effectively the risk analysis

process identifies programs that actually are high risk.

•	Finally, since 2003, all of this information has been systematically collected and stored in CPD’s Grants Management Process (GMP) database.

Each year CPD issues a Notice that establishes the rating system to be

used in that year’s risk analysis process, and any other changes to the 

process. From 2005 to 2007, the process remained largely unchanged, 

3 “Motor Carrier Safety; A Statistical Approach Will Better Identify Commercial Carriers That Pose High

Crash Risks Than Does the Current Federal Approach.” June 2007. (http://www.gao.gov/new.items/

d07585.pdf).
Risk-Based Monitoring of CPD Formula Grants 6

thereby allowing those years to be combined for this study.4 Although risk 

analysis and monitoring is conducted for almost all CPD grant programs 

(competitive and formula), this study focuses solely on CPD’s four formula

grant programs—CDBG, HOME Investment Partnerships Program (HOME),

HOPWA, and ESG.5 In theory, risk should manifest itself differently in each

of these programs, so it would not be valid to use the same independent 

variables to predict findings for different programs. It is therefore highly

appropriate that the existing risk analysis criteria are slightly different for 

each program. Because of this, each program will be modeled, analyzed,

and presented separately. However, the basic methodology will be consistent.

As noted previously, the risk score subfactors serve as independent variables and findings serve as the dependent variable. The derivation and

manipulation of these variables is described more thoroughly in Sections 

III and V of this report. The study uses two statistical techniques for to

estimate the relevance of each subfactor. First is a simple bivariate correlation analysis, which examines the relationship between the dependent 

variable and each independent variable one at a time. Then a more 

thorough multivariate model is developed to account for interrelationships 

between independent variables. These techniques are used to determine 

the overall effectiveness of the risk analysis process and the effectiveness 

of each subfactor used in the process.

4 Changes primarily consisted of increasing or decreasing the score assigned to a particular subfactor.

5 CPD’s competitive grants go to non-governmental entities, which makes them much more difficult to

match across the various GMP data elements. Nonetheless, because of the nature of the competitive

grants (small dollar amounts but a very large number of grantees), risk analysis is critical. Further

study of the CPD competitive grants risk analysis system is warranted.
Risk-Based Monitoring of CPD Formula Grants 7

III. Process

A. Annual Risk Analysis Procedure 

This section presents the risk analysis process in a step-by-step manner to 

show how risk scores are calculated, and how that process leads to monitoring and findings.

Each year, CPD issues a Notice on the subject of Risk Analyses for Monitoring CPD Grant Programs.6 This Notice provides a consistent methodology

for conducting risk analyses, and sets priorities that will guide the process 

of determining which grant recipients should be monitored. All grantees 

with open grants must be reviewed, with a separate risk score assigned 

for each program they participate in. For example, the city of Boston will 

receive a separate score for CDBG, HOME, HOPWA, and ESG because it

participates in all four formula grant programs. The review is conducted by 

a CPD staff member (CPD Rep) at the field office for the region in which the

grantee operates. The CPD Rep is advised to use a variety of information 

sources to conduct his or her analysis, including:

•	Integrated Disbursement and Information System (IDIS),

•	Consolidated Annual Performance and Evaluation Reports (CAPERS),

•	Annual Performance Reports (APRs),

•	Prior monitoring visits,

•	Audits, and 

•	Citizen complaints.

The annual Risk Analysis Notice also includes a series of attachments that

constitute the risk analysis scoring rubric. There are five of these worksheets: one each for CDBG, HOME, HOPWA, ESG, and Competitive Grants.

The worksheets are broken down into five factors (except for CDBG, which

uses four factors). These factors are general themes, such as Financial

and Management. Each factor is broken into the subfactors against which 

the grantee is actually scored, and which serve as the “red flags” that the

Departmental Management Control Program handbook recommends. These 

red flags are the criteria for determining risk exposure to the Department,

the likelihood that a program participant is out of compliance with program 

requirements, or that the participant has performed unacceptably. 

6 For fiscal year (FY) 2008, Notice CPD-07-07; for FY 2007, Notice CPD-06-10; for FY 2006, Notice

CPD-05-08; for FY 2005, Notice CPD 04-12.
Risk-Based Monitoring of CPD Formula Grants 8

A. Grant Amount

i.	 The Participating Jurisdiction’s (PJ) grant amount

for the most recently completed program year 

falls within the top quartile of all HOME funded 

communities within the Office’s jurisdiction for the

same program year. 

High 5

ii.	 The PJ’s grant amount for the most recently

completed program year falls within the second 

quartile of all HOME grants awarded within the 

Office’s jurisdiction within the same program year.

Medium 3

iii.	 The PJ’s grant amount for the most recently

completed program year falls within the third or 

fourth quartile of all HOME grants awarded within 

the Office’s jurisdiction for the same program year.

Low 1

For each subfactor, the CPD Rep chooses the level of risk (referred to as “risk

definition” in the worksheet) that most accurately matches the grantee.

The worksheet includes an explanation accompanying each risk level that 

is intended to guide the CPD Rep in choosing the appropriate category. 

Usually there are three options, corresponding to low, medium, and high 

levels of risk.7 The CPD Rep must choose one risk level for each subfactor, 

using only the options provided. The excerpt in Figure 1, from the HOME 

risk analysis worksheet, illustrates the subfactor Grant Amount within the 

Financial factor, showing the criteria provided for high, medium, and low risk. 

Although the subfactor in Figure 1 is relatively simple and straightforward, 

others are more complex. Figure 2 shows the Meeting National Objectives

subfactor, within the Services factor for CDBG. Appendix G includes a 

complete list of subfactors and their definitions.

For each subfactor, the CPD Rep must determine which category the 

grantee falls into and assign the corresponding score. The risk levels and 

scores are not continuous, and only the listed options may be chosen. On 

the excerpt pasted as Figure 1, the CPD Rep cannot rate the grantee as 

“medium-high risk” and give a score of 4. The entire worksheet includes

a total of 100 points, meaning that a grantee that is high risk on every 

subfactor for a particular program would get a risk score of 100 for that 

program. The lowest possible score is 16 for HOME, 17 for CDBG and 

HOPWA, and 18 for ESG.

7 However, for some subfactors the choices are limited to high or low, or to high or none.

Figure 1. Excerpt From HOME Risk Analysis Worksheet

Factor 1—Financial Risk

Definition

Risk

Score

Evaluator’s

Rating

Evaluator’s 

Comments
Risk-Based Monitoring of CPD Formula Grants 9

After the risk analysis is complete, and the grantee is assigned a score for 

each open grant it administers, the field office must translate this into a

monitoring work plan. First, all grantees are ranked by their overall risk 

score, from highest risk to lowest risk. Each grantee’s overall risk score is 

simply the average of its individual program risk scores. All grantees with 

an overall risk score of 51 or higher are considered high risk. A grantee with 

a risk score between 30 and 51 is considered medium risk, and a grantee 

with a score less than or equal to 30 is considered low risk.8 All high-risk 

grantees must be monitored, unless they are granted an exception. Low-risk 

grantees may also be monitored, depending on the available resources. 

The process of granting exceptions is a key aspect of the overall risk 

assessment and monitoring process. This one step can substantially change 

the population being monitored; depending on how and when exceptions

are granted, the subset of the grantee population that is actually monitored 

may suffer from selection bias that is not captured by the subfactor scores. 

The exceptions step in the process has changed over the years, and will be 

discussed below. After the scores have been tallied and exceptions granted, 

A. Meeting National Objectives

i. Sanctions have been placed on the grantee 

for noncompliance with national objectives or

eligibility requirements during the most recently 

completed program year OR the grantee has 

not taken recommended corrective actions to 

address outstanding sanctions OR there are known 

problems identified through review of reports or

information received that indicates that the grantee 

is currently not meeting the national objectives OR

is implementing ineligible activities.

High 12

ii. The grantee has been in noncompliance of program 

national objectives or eligibility requirements one

or more times in the past three years AND the 

grantee has taken appropriate corrective actions 

to address any outstanding sanctions and/or 

monitoring audit findings related to eligibility and

national objectives.

Medium 6

iii. Activities carried out by grantee during the three 

most recent program years are in compliance 

with national objectives and eligible activity

requirements AND there are no known problems.

Low 1

Figure 2. Excerpt From CDBG Risk Analysis Worksheet

Factor 4—Services Factor 

Definition

Factor

Score

Evaluator’s

Rating

Evaluator’s 

Comments

8 In the remainder of this study, unless otherwise specified, we will use “low risk” to mean “not high

risk.” In other words, low risk (a score of 30 or less) and medium risk (a score of 31 to 50) have been

grouped together.
Risk-Based Monitoring of CPD Formula Grants 10

the field office develops a work plan that sets forth which grantees will

be monitored, when, and by whom. The work plan also establishes the 

method of monitoring—that is, whether the grantee will be monitored on 

site or remotely.9

In the case of on-site monitoring, a CPD Rep goes on site to meet with the 

grantee and review files. Depending on the program, the CPD Rep may also 

visit project sites and meet with organizations partnering with the grantee.

Throughout this process, the Rep is guided by CPD’s highly standardized,

comprehensive Monitoring Handbook. The nearly 200 exhibits in the handbook address every program or activity a CPD Rep is likely to encounter, 

and ensure that monitoring is complete and consistent. These exhibits 

guide the CPD Rep through a series of questions that document the grantee’s performance and whether the grantee is complying with all program 

requirements. Based on the answers to these questions, the handbook 

clearly prescribes when a finding should be issued. CPD Reps are required

to document their efforts extensively and to enter that documentation into 

GMP when the monitoring is complete.

B. Year-to-Year Changes

In the annual Risk Analysis Notice, CPD has an opportunity to modify

the risk analysis process for that year. These year-to-year changes may 

be problematic for statistical analysis; data from multiple years cannot

be combined into a single sample if those data have different meanings. 

There are three primary types of changes to the process: changing the 

exceptions, changing or eliminating factors or subfactors, and changing 

the subfactor scores. Changing the exceptions is a problem if it means that 

the population of grantees monitored in different years was systematically different. Changing or eliminating subfactors is a problem because 

the subfactors are our independent variables, and must be available for 

the whole sample. Changing the subfactor scores is not as significant of a

problem, especially if independent variables are normalized to adjust for

their different ranges.

From 2005 to 2006, the risk analysis process was coordinated, and the 

Notice remained entirely unchanged.10 In 2007, CPD made moderate

changes to the process. While the subfactors and their scores remained

nearly identical, there were major changes to the exceptions. In 2005

9 According to Notice CPD-04-12 and Notice CPD-06-10, remote monitoring is encouraged for low-risk

grantees; however, GMP data documenting monitoring type is inadequate. A total of 83 percent of

monitoring observations did not document whether the monitoring occurred on site or remotely. Of 

the remaining 17 percent of observations, almost all were labeled as on-site monitoring.

10 The FY 2006 notice, CPD-05-08 simply extends the provisions of Notice CPD-04-12.
Risk-Based Monitoring of CPD Formula Grants 11

and 2006, there were three exceptions: A, B, and X. Exception A could be 

used if the Office of Inspector General (OIG) was in the midst of an audit

of the grant program or site. Exception B could be used if the CPD Director 

deemed monitoring of a particular program to be “administratively infeasible.” Exception X was simply “Other (explain),” with no stated restrictions.11

During 2005 and 2006, Exception A was used three times; Exception B

was used 382 times, and Exception X was used 294 times.12 In 2007, the

available exceptions were changed to make them more consistent and less 

discretionary. Four exceptions were available but only two could be used 

to excuse a grantee from monitoring: if the grantee was being audited by 

OIG, or if the grantee had been monitored within the past two years. Thus,

in 2006 a resource-constrained CPD office could use Exception B to avoid

monitoring that would be “administratively infeasible”—such as a far-away

grantee that would require travel—but in 2007 that grantee would have 

to be monitored. For these reasons, there may be systematic differences 

between the population monitored in 2007 and the population monitored 

in 2005 and 2006. However, we control for these differences by including 

program year as a variable in our model. 

In 2008, the Notice was overhauled; several subfactors were deleted,

others were added, and others were substantially modified. These changes

prevent 2008 data from being combined with data from previous years, so 

this analysis considers only data from program years 2005 through 2007.

11 2005 to 2006 Exceptions:

A.	 The Office of Inspector General is currently auditing the grant program or project site.

B.	 The CPD Director determines that monitoring of this program or project site is administratively

infeasible in the current year, given other monitoring actions. HUD will make use of technical 

assistance support, as needed, to mitigate potential problems or already has the grantee 

working on a schedule for needed corrective actions. Future monitoring will give priority 

considerations if issues continue or extenuating circumstances or new information increases risk 

of this grantee. 

X.	 Other (explain).

12 Exceptions were made for grantees in lower risk categories as well as high-risk grantees.
Risk-Based Monitoring of CPD Formula Grants 12

A. Overview of Available Data

The comprehensive source of data documenting the risk assessment and 

monitoring process is the GMP system. The GMP system is the interface 

through which field offices enter information about grant recipients. It

contains several “modules” that focus on various aspects of a grantee. Data

analyzed in this report were harvested primarily from three modules—Risk

Analysis, Monitoring, and Findings. See Appendix A for an explanation of 

how data from these modules were matched into analytic files.

The unit of analysis that we use is the program; each grant program, in

each year, for each grantee, is a separate observation. For example, in 

2005 the city of Boston administered all four formula grant programs, 

and each of these programs is a unique observation. A brief note about 

the calendar is in order. Every recipient of a grant through the four CPD 

formula programs goes through a Consolidated Planning process each year, 

in which it states, for each grant, what activities it plans to pursue. The 

grantee chooses the dates that start and end the year, establishing what is 

known as their “program year.” Thus, the variable “program year” is only a

rough approximation of calendar (or fiscal) year and depends on the particular grantee and on the CPD Rep who entered the information into GMP. 

In program years 2005 through 2007, 6,773 programs went through the

risk analysis process. Over this period, there were 1,192 individual cities 

and states administering the 6,773 grant programs. Forty-six percent of 

these grantees administered three or fewer programs over the three year 

period; almost all of these are grantees that participated only in CDBG

each year (n=471). A total of 104 grantees used all four formula grants in

all three years. 

Because the programs have different subfactors and different scoring 

weights, the risk analysis process affects them differently. In addition, the

process of granting exceptions and identifying low-risk grantees for monitoring may be biased by program type. Table 1 and the following series of 

figures illustrate the effects of the risk analysis and monitoring process on

each of the four formula programs. 

Table 1 and Figure 3 show the entire universe of 6,773 programs in this 

analysis, broken down by program type. The progression from Figure 3 to 

Figure 4 shows how the risk analysis process alters this universe; Figure 4

includes only high-risk programs. CDBG makes up more than 50 percent 

of all programs but only 41 percent of high-risk programs. Similarly, ESG 

IV. General Analysis
Risk-Based Monitoring of CPD Formula Grants 13

accounts for 16 percent of all programs but only 10 percent of high-risk 

programs. Balancing this out is the overrepresentation of HOME programs 

in the high-risk pool; the HOME share increases from 28 to 43 percent.

From 2005 through 2007 the risk analysis system was more likely to identify HOME programs as high risk. Whether that identification is appropriate

depends on whether HOME programs actually were more risky than the 

other programs.

Although CDBG programs make up only 41 percent of high-risk programs, 

Figure 5 shows that they account for 56 percent of all programs monitored. 

Conversely, ESG, HOME, and HOPWA are all much less likely to be monitored than their risk scores would suggest. In particular, HOME makes up

43 percent of all high-risk programs but only 31 percent of all programs 

monitored.

Number of programs 3,417 1,102 1,895 359 6,773

Average risk score 38 37 42 39 39

Number of programs monitored 983 167 549 69 1,768

Percent of programs monitored (within column) 29% 15% 29% 19% 26%

Table 1. Summary of Monitoring by Program Type for All Programs That Underwent Risk 

Analysis

Figure 3. All Programs 

Undergoing Risk Analysis, by 

Program Type, 2005–2007

Figure 4. All High-Risk 

Programs, by Program Type, 

2005–2007

Figure 5. All Programs That 

Were Monitored, by Program 

Type, 2005–2007

CDBG ESG HOME HOPWA Total

359

5.3%

N = 6,773 N = 1,147 N = 1,768

73

6.36%

69

3.9%

1,895

27.98%

493

42.98%

549

31.05%

1,102

16.27% 111

9.68%

167

9.45%

3,417

50.45%

470

40.98%

983

55.6%

CDBG ESG HOME HOPWA
Risk-Based Monitoring of CPD Formula Grants 14

Two factors cause the differences between Figures 4 and 5, and both 

introduce potential bias. First, many of the high-risk programs shown in 

Figure 3 were granted exceptions; of the 1,147 high-risk programs, 458

(40 percent) were not monitored.13 Second, field offices have the discretion

to monitor other low-risk programs if resources are available. Figure 4 

includes 1,079 of these low-risk programs; 1,079 of the 1,768 programs 

(61 percent) that were monitored were low risk. 

B. Comparison of Means

Because of the abundance of monitoring done on lower risk programs, 

the population of monitored programs can be split into two groups for 

an independent samples t-test. Comparing the high-risk group and the 

low-risk group on a measure of actual risk shows whether the risk analysis 

process effectively identifies programs in violation of statutory, regulatory,

or program requirements. 

Table 2 shows the proportion of monitored programs that had a finding,

by program type. A p-value of less than .05 means that the difference 

between the low-risk programs and the high-risk programs is statistically 

significant. For each of the four program types, those programs identified

as high risk appear more likely to have a finding than the programs identified as low risk; however, for ESG and HOPWA this was not a statistically

significant difference. While there is strong evidence that “high-risk” CDBG

and HOME grantees really are a higher risk group than “low-risk” CDBG

and HOME grantees, the evidence is inconclusive for ESG and HOPWA. This

is heavily influenced by the small sample size; only 69 HOPWA programs

and 167 ESG programs were monitored over the three years under analysis.

13 Presumably, these high-risk programs were not monitored because they were granted an exception, 

although GMP data cannot verify this with certainty. See Appendix A.

CDBG 51% 76% 0.00

ESG 35% 46% 0.15

HOME 42% 67% 0.00

HOPWA 53% 69% 0.19

Total 48% 69% 0.00

Table 2. Proportion of Monitored Programs Having at Least One Finding

Low-Risk 

Programs

High-Risk 

Programs P-Value
Risk-Based Monitoring of CPD Formula Grants 15

A slightly different picture emerges when looking at the number of findings

as the dependent variable. Overall, programs identified as high risk had a

mean of 2.58 findings, while low-risk programs averaged 1.26. However,

the average number of findings is largely a function of the proportion of

grantees with any findings; the large number of zero values skews the

mean. Excluding those programs that had no findings illustrates the extent 

of the problems identified in certain programs. Table 3 shows that even

with zero-finding observations excluded, high-risk programs had more

findings than low-risk programs did. This observation was also true within

each of the four program types, although for HOPWA the difference was

not statistically significant. The difference was largest for HOME, perhaps

justifying the higher risk scores for HOME programs.

CDBG 2.8 3.5 0.00 25%

ESG 1.7 2.6 0.02 49%

HOME 2.7 4.4 0.00 63%

HOPWA 2.2 3.0 0.25 33%

Total 2.7 3.7 0.00 42%

Table 3. Mean Number of Findings per Monitoring, Excluding Programs With 

Zero Findings

Low-Risk 

Programs

High-Risk 

Programs P-Value Increase

C. General Conclusions

This simple statistical test indicates that the risk analysis process is having 

some success. Overall, it is identifying programs that are more likely to 

have findings and tend to have more of them. For the CDBG and HOME

programs, an especially stark contrast exists. For CDBG, ESG, and HOME, 

among programs that have at least one finding, high-risk programs have a

larger total number of findings than low-risk programs do.

However, there are still several reasons for concern. First of all, it is somewhat alarming that programs not identified as high risk still have findings

at a rate of 48 percent. The programs in that 48 percent also average 2.7 

findings. These two statistics combined indicate that programs for which

monitoring is not required are still frequently found to be in violation of 

multiple program requirements. This may be partially due to our decision 

to group low-risk programs (those with risk scores of 30 or lower) and

medium-risk programs (those with risk scores of 31 to 50) for the purposes

of this analysis. Although low- and medium-risk programs are purportedly 

treated the same (in that monitoring is not required of either), the risk

analysis process does draw a distinction between them and they could be 
Risk-Based Monitoring of CPD Formula Grants 16

treated differently in the future. In fact, programs with risk scores of 30 or

lower had findings only 36 percent of the time, while programs with risk

scores of 31 to 50 had findings 49 percent of the time.

Finally, given the abundance of information available to HUD, this sort of 

predictive ability should be expected; the question is not whether the risk

analysis process is successful, but whether it is successful enough. CPD’s 

risk analysis process is already a valuable and effective tool in the effort to 

balance resources against risk, but it could be improved. The next section 

considers how the process could be improved, by analyzing the risk analysis subfactors.
Risk-Based Monitoring of CPD Formula Grants 17

V. Subfactor Analysis

A. Correlation Analysis

Simple bivariate correlation analysis provides a suitable starting point for 

exploring the relationship between the dependent variable—number of 

findings—and each of the independent variables. In addition to the subfactors, the following independent variables are analyzed: total risk score,

total score on each of the factors (a simple sum of the subsidiary subfactors), and a dummy variable indicating high-risk status. These results are

fully documented in Appendix B, with highlights provided below. 

For bivariate analysis of CDBG, the notable finding is that almost every

independent variable tested was correlated with findings. Because the

CDBG risk analysis process was shown above to be working fairly well, this 

extensive correlation is not surprising. It is more noteworthy to observe

the several variables that were not correlated with findings. Uncorrelated

variables include three of the five financial subfactors—Grant Amount,

Timeliness of Expenditures, and Program Income—as well as the Environmental/Relocation subfactor. 

For ESG, only a few of the independent variables we tested were correlated 

with findings. The Financial and Management factors were correlated with

findings; however, within those categories, the only individual subfactors

with a statistically significant correlation coefficient were Expenditure Rate

and Timely and Accurate Submissions. 

HOME data, like CDBG data, showed that most independent variables we 

tested were correlated with findings. Notable exceptions were the Financial

factor and three of the five subfactors that it comprises. Grant Amount,

CHDO (Community Housing Development Organization) Reservations,

and Program Income all were shown to be uncorrelated with findings. In

addition to the Financial subfactors, six other subfactors did not have a 

statistically significant correlation with findings: Multiple Funding Sources,

Program Delegations, Affordability Requirements, OIG Audit, Environmental/Relocation, and Responsiveness.

Bivariate correlation analysis of HOPWA data revealed few independent

variables with a statistically significant correlation with findings. The Satisfaction factor and both of its component subfactors—Citizen Complaints

and Responsiveness—were very highly correlated with findings. Multiple

Sponsors and Size of Funding were also significant subfactors.
Risk-Based Monitoring of CPD Formula Grants 18

This simple analysis is a good starting point for examining whether particular 

subfactors are useful, but it must be supplemented. The primary weakness 

is that this analysis does not account for how the subfactors (and other

independent variables) are correlated with each other. To illustrate this 

correlation between subfactors, we created a full correlation matrix for 

CDBG and HOME, included in Appendix C. Although it is difficult to distill

this large amount of information, it is clear that there is a very high degree 

of correlation between the subfactors. For example, the Audit and Financial 

Compliance subfactors, under the Financial factor, are both correlated with 

findings. However, they are also both correlated with every other subfactor 

used in the risk analysis process.

B. Multivariate Analysis

For CDBG and HOME, the relatively large sample size allows for a more

rigorous analysis to address the weaknesses of the previous section—

namely the correlation between subfactors. To do this, the analysis must 

estimate the marginal effect of each subfactor on findings, while controlling for mean effects of the remaining independent variables. The most 

common way of doing this is via ordinary least squares (OLS) regression;

however, OLS would not be appropriate for this analysis for two reasons. 

First, the dependent variable (findings) is discrete, rather than continuous.

Second, the distribution of values is highly skewed to the right, with a large 

number of zeroes. The mean number of findings is 1.77, but 63 percent of

monitored programs have either zero or one finding.

Appropriate regressions include models for count data (negative binomial

or Poisson distributions) and ordinal data (cumulative logit or probit

distributions). Models for count data were fairly accurate in predicting the

number of findings, but did a very poor job of predicting the high number

of cases with zero findings. This report presents findings from a cumulative

logit model, which was the best among the models tested at predicting 

whether a program would have any findings at all and also at predicting

the total number of findings.

In this model the dependent variable is not a number, but the likelihood of

the outcome being in a certain category. Tests indicated eight categories 

of findings were sufficient to adequately model the data. Figure 6 depicts

the distribution of findings, with all programs having 7 or more findings

grouped together.
Risk-Based Monitoring of CPD Formula Grants 19

The independent variables in the model include the following: all the risk 

analysis subfactors, a dummy variable to distinguish program years 2005 

to 2006 from 2007, and dummy variables for the different field offices.

Separate models were estimated for CDBG and HOME, but the only differences are the subfactor variables. Appendixes D and E include full documentation of these models and their results. The ensuing sections briefly

present the statistically significant findings.

CDBG

The CDBG logit model contains 17 subfactors, of which five are statistically significant determinants of findings: Grant Amount, Financial

Compliance, Program Complexity, Timely and Accurate Submissions, and 

Responsiveness. The 2007 dummy variable is not statistically significant,

indicating that there were no differences between the years in the 

underlying process.

Several subfactors that were correlated with findings in the simple bivariate

analysis above do not appear statistically significant in multivariate analysis.

For example, two subfactors are related to audits—one under the Financial 

factor (A-133 Audits) and one under the Management factor (OIG Audit).

Bivariate analysis shows both of these subfactors to be correlated with 

findings, but when we control for the other variables in a regression, that

correlation dissipates. The same happens with the Staff Capacity, Financial 

Compliance, and Meeting National Objectives subfactors, perhaps due to

the fact that they are all correlated with each of the other 16 subfactors. 

The opposite effect occurs with the Grant Amount subfactor; it was not

Figure 6. Distribution of Number of Findings for All Programs That Were 

Monitored Number of programs

900

800

700

600

500

400

300

200

100

0

0

781

324

204

163

88 69 5089

1 2 3 4 5 6 7 or more

Number of findings
Risk-Based Monitoring of CPD Formula Grants 20

statistically significant in the correlation analysis, but it is in the regression

analysis. More interesting is the fact that the impact is negative. If two

grantees have the exact same subfactor scores on everything except Grant 

Amount, the one with a larger grant is likely to have fewer findings. This may

be due to the fact that larger grantees are more sophisticated and less prone 

to findings; however, grantee sophistication should be accounted for by the

Staff Capacity subfactor. Either there is some other reason that large grantees get fewer findings, or the Staff Capacity subfactor is poorly specified.

A second interesting result is the very strong, and statistically significant,

relationship between the subfactor Responsiveness (under the Satisfaction factor) and findings. The magnitude of this effect is more than three

times as large as the next most influential subfactor (Timely and Accurate

Submissions). This suggests that a grantee’s failure to respond to citizen

complaints is a very strong predictor of findings.

Finally, the field office random effects show considerable variation between

field offices. Field offices fall into three categories. Most (27 out of 43)

do not have any statistically significant difference from the mean; they

identified findings at the rate we would expect, given the risk scores of

the programs they were monitoring. Nine were more likely to give findings

than their grantees’ risk scores would suggest. Seven were less likely to 

give findings than their grantees’ risk scores would suggest. For example,

the Portland field office had an above-average likelihood of giving findings, but the Seattle field office was not different from the mean. The field

offices in both Minneapolis and Atlanta also had an above-average likelihood of giving findings, while the field offices in Denver and San Francisco

had a below-average likelihood of giving findings. Appendix F contains the

full list of field office effects.

HOME

The HOME logit model contains 21 subfactors, of which only four are statistically significant determinants of findings: Commitments and Expenditures,14 A-133 Audits, Staff Capacity, and Program Progress. As with CDBG, 

many HOME subfactors that appeared important in the simple analysis 

above drop out in the multivariate analysis. Subfactors such as Financial 

Compliance, Physical Condition of Projects, Affordability Requirements,

and On-Site Monitoring no longer appear to be good predictors of findings.

The full correlation matrix in Appendix C shows that these subfactors were 

frequently correlated with all, or nearly all, of the other subfactors. The 

HOME risk assessment has so many subfactors that the marginal effect 

of each one is often zero.

14 The full name of this subfactor is “Commitments, CHDO Reservations, and Expenditures.”
Risk-Based Monitoring of CPD Formula Grants 21

As with CDBG, the field office random effects show variation between

field offices. However, there is less variation; 33 out of 43 field offices did

not have a statistically significant effect on findings (compared with 27

for CDBG). Five field offices were more likely to give findings than their

grantees’ risk scores would predict, and five offices were less likely to give 

findings than their grantees’ risk scores would predict. For both CDBG and

HOME, field offices in Minneapolis, Portland, Ft. Worth, and Atlanta issued

more findings that our model would have predicted, and field offices in

Denver, Columbus, San Francisco, and Birmingham all issued fewer findings than our model would have predicted. The fact that these field offices

exhibited the same tendency in both the HOME and CDBG models is a 

clear indicator that something is different in those field offices.

It is critical to fully understand the meaning, and potential causes, of these

field office random effects. Essentially, it means that some field offices gave

more or fewer findings than our risk analysis model would predict based on

the risk scores. One explanation is that the risk analysis process is missing 

something—for example, it could be that programs in Minneapolis and Portland are riskier in some way that is not captured by the existing risk analysis subfactors. Similarly, the Denver and San Francisco field offices’ lowerthan-expected rate of findings could mean that programs in those cities are

very well run in some way that the existing risk analysis subfactors do not 

capture. A second possible explanation is that some HUD staff are more (or

less) zealous or demanding. Perhaps staff in the Minneapolis and Portland

offices are better at identifying findings, or more willing to issue them, than

the average CPD Rep. Finally, this pattern could be caused by the choices 

each field office makes about whom to monitor—which high-risk programs

get exceptions and which low- or medium-risk programs are chosen to 

be monitored. Some field offices may choose to monitor the most difficult

grantees, while others may choose to monitor more low-risk grantees.

It is impossible for our analysis to determine which of these scenarios is

occurring. What is clear is that the risk analysis process does not perfectly

capture a program’s risk level, and field office staff can contribute valuable

insight.

C. Subfactor Conclusions

It appears that quite a few subfactors are not contributing to the accurate

identification of risk; should they simply be removed? That action alone

would not necessarily improve the statistical model or the risk analysis 

process. As demonstrated in Appendix C, a very high degree of correlation 

exists between the subfactors; this collinearity undermines the strength of

the ordinal logit model, especially when one considers adding or dropping 
Risk-Based Monitoring of CPD Formula Grants 22

variables in the model. As a result, it would be unwise to rely exclusively 

on the results of statistical analysis to revise the risk analysis process. 

Statistical models should always be guided by theory, and the theory of 

risk must form the context of this analysis.

The HOME regression results demonstrate the limits of the statistical 

model. The four subfactors that show up as significant are Commitments

and Expenditures, A-133 Audits, Staff Capacity, and Program Progress. 

Of those subfactors, a strong a priori case exists for Staff Capacity being 

an indicator of likely findings; it flags grantees that have “demonstrated

an inability to administer the HOME program,” or have open vacancies for

key staff. Program Progress and Commitments and Expenditures both flag

grantees that are missing deadlines or submitting incomplete reports. The 

A-133 Audits subfactor would seem to fall into this category as well (failing

to submit a report).15 Missing reporting deadlines probably does indicate 

risk, but these subfactors do not intuitively seem to be among the four 

most important subfactors.

CPD management is ideally suited to understand the theoretical basis 

of risk in its grant programs and to select the most important causes of 

grantee risk, based on that theory. Discussions with CPD staff indicate that 

two major causes of risk are insufficient staff capacity and highly complicated programs. Unqualified, inexperienced, or short-handed staff are at

risk for waste and mismanagement, no matter what other characteristics 

their programs have, and even competent staff can botch a very challenging or complex program. Program complexity may also be associated with 

outright fraud, because complex programs may be easier to manipulate 

without being noticed. These complementary factors—staff capacity and 

program complexity—must both be accounted for in any theoretically valid 

model of risk. However, staff capacity and program complexity alone may 

not sufficiently explain the variation in grantee risk. Including an indicator of past performance can serve as a proxy for some of the other risky 

aspects of a grantee or program that are difficult to measure.

In light of these three themes of grantee risk—staff capacity, program complexity, and past performance—the subfactors identified by our ordinal logit

model as statistically significant can be more properly assessed. Tables 4,

5, and 6 highlight key language from the definitions of several statistically

significant subfactors, showing how they relate to these three themes.

15 The A-133 Audits subfactor flags grantees that failed to submit their A-133 Audit on time, OR have

open findings and are overdue in carrying out corrective actions. Currently, there is no way to know

which of those two clauses make this subfactor significant.
Risk-Based Monitoring of CPD Formula Grants 23

CDBG—Financial Compliance “...staff has demonstrated an inability to administer the financial management

responsibilities for the CDBG program....”

“...vacancies for key financial management staff (have existed for X months)....”

“...key financial staff have been hired in the past program year and have not received

CDBG financial management training.”

CDBG—Program Complexity “...(grantee uses) CBDOs and/or subrecipients to assist in carrying out such activities.”

CDBG—Responsiveness “Grantee has failed to respond to complaints and/or citizen inquiries....”

HOME—Staff Capacity “...staff has demonstrated an inability to administer the HOME program....”

“...vacancies for HOME key staff (have existed for X months)....”

“...key HOME staff have been hired in the past program year and have not received

HOME program training.”

CDBG—Timely and Accurate 

Submissions

"...grantee’s required submissions are incomplete OR are received 30 days or more 

after prescribed timeframes." (or were late/incomplete in the prior 3 years)

CDBG—Responsiveness "Grantee has failed to respond to complaints and/or citizen inquiries... during the most

recent program year."

HOME—Commitments, 

CHDO Reservations, and 

Expenditures

"...the most recent commitment and expenditure deadlines were not met."

“... the PJ missed a deadline one or more times within three most recent program

years....”

HOME—Audits "An A-133 audit ... has not been submitted to the Federal Audit Clearinghouse."

"...PJ has open findings and is overdue in carrying out any agreed upon corrective

action...."

CDBG—Program Complexity "Grantee or State recipients undertakes activities beyond those described in 24 CFR 

570.201 and 570.202."

"...(grantee uses) CBDOs and/or subrecipients to assist in carrying out such activities."

Table 4. Statistically Significant Subfactors Related to Staff Capacity

Table 6. Statistically Significant Subfactors Related to Past Performance

Table 5. Statistically Significant Subfactors Related to Program Complexity

Subfactor Key clauses related to staff capacity

Subfactor Key clauses related to past performance

Subfactor Key clauses related to program complexity
Risk-Based Monitoring of CPD Formula Grants 24

The subfactors have complex definitions with multiple clauses; most of

the subfactors in the tables above have other clauses that were left out 

because they were not relevant to staff capacity, program complexity, or 

past performance. These multiple clauses obscure the real “meaning” of a

subfactor, and blur the lines between the various subfactors; the Financial

Compliance and Staff Capacity subfactors contain clauses that are nearly 

identical.16 Still, it is important that these subfactors are not only statistically significant but also have some clauses connected to theoretical causes

of risk. Because the current form of these subfactors is far from ideal, the 

risk analysis process could be improved and simplified by revising and

consolidating these subfactors under the themes of staff capacity, program 

complexity, and past performance.

16 “staff has demonstrated an inability to administer the financial management responsibilities for the

CDBG program” and “staff has demonstrated an inability to administer the CDBG program”.
Risk-Based Monitoring of CPD Formula Grants 25

VI. Potential Improvements

Statistical analysis shows that the CPD risk analysis process is identifying 

higher risk grantees, but the process could be more effective and efficient.

Many of the subfactors show no statistical relationship with the number of 

findings a program has. In such cases, it is difficult to determine why no 

relationship exists, due to the fact that these subfactors often have complicated, multifaceted definitions.

Several possible changes to the risk analysis process could have a variety 

of benefits, including saving staff time, making better use of monitoring

resources, and increasing the potential for future research. CPD could 

consider the following five changes:

1. Use fewer subfactors, which simply and directly estimate the critical 

themes of staff capacity, program complexity, and past performance.

2. Develop a subfactor to explicitly incorporate the judgment of the CPD

Rep and/or CPD Management Representative.

3. Ensure strict adherence to limited exception criteria.

4. Randomly sample low- and medium-risk grantees for monitoring. 

5. Increase reliance on remote monitoring for low- and medium-risk

grantees.

1. Fewer Subfactors

This first recommendation is the one most clearly supported statistical

analysis. The fact that so many subfactors are not functioning as “red

flags” for risk indicates that a more parsimonious model could be just as

effective. The subfactors that remain should be simple (that is, they should

incorporate fewer “AND”s and “OR”s) and focus on a few key determinants

of risk. 

Because of the complex and overlapping definitions of the current subfactors, simply keeping some subfactors and dropping others would be ill 

advised. Instead, we recommend that CPD develop one or two simple

indicators for each of the following three categories: program complexity, 

staff capacity, and past performance. These indicators could effectively 

identify those programs most susceptible to waste, fraud, abuse, or mismanagement, using only a small amount of information that CPD Reps can 

easily supply. 
Risk-Based Monitoring of CPD Formula Grants 26

2. Field Office Adjustment

This study also shows variation across field offices that is not captured by

the risk analysis process. This variation could become more prevalent if 

some subfactors are eliminated. Currently, the only outlets for the subjective

judgment of field office staff are the use of exceptions, deliberate selections

for low or moderate risk monitoring, and outright manipulation of subfactor 

scores. The risk analysis process should explicitly acknowledge and incorporate the subjective judgment of HUD field office staff. For example, the

evaluator could be given the opportunity to adjust a program’s risk score

up or down within a certain range if he or she thinks that the score does 

not otherwise accurately reflect the risk posed by that program. This will

enable field office staff to contribute their valuable perspective, giving them

more ownership over the risk analysis selections and correcting any weaknesses in the rest of the risk analysis process. Furthermore, by quantifying 

the field office judgment and explicitly including it in the process, future

analysis will be much more rigorous and useful (for instance, management

will be better able to identify trends across field offices).

3. Limited Use of Exceptions

From 2005 through 2007, 40 percent of high-risk programs were not 

monitored; that is an alarmingly high exception rate. Giving CPD field

office staff an explicit outlet for their unique local knowledge and opinions

makes it even more important to strictly adhere to the extremely limited 

exception criteria. If the field office can directly influence the score and a

grantee is still high risk, it should be monitored. Of course, exceptions do 

make sense in some situations, and exceptions used in FY 2008 and FY 

2009 are reasonable. Being monitored and audited by the OIG at the same

time would be somewhat redundant and a huge burden on the grantee. 

Grantees that have recently been monitored or are set to receive technical 

assistance should already be “on the radar” of CPD staff, and monitoring

them may not reveal anything interesting. Nonetheless, CPD management

should ensure that these exceptions are not being abused. 

4. Random Sampling 

The greatest benefit of these changes would be a reduction in the time and

resources required for risk analysis and monitoring, but these changes offer 

two very valuable additional benefits, particularly from random sampling.

First, including an element of randomness in the process should improve 

performance by maintaining a constant possibility of monitoring—grantees 

must be prepared to be monitored each year and will be deterred from 
Risk-Based Monitoring of CPD Formula Grants 27

malfeasance or negligence. Second, randomly selecting a subset of the grantees to be monitored would allow for extremely rigorous statistical analysis, 

overcoming several methodological limitations to this particular study.

A strategy for implementing a random sample could start with the fact 

that 61 percent of all monitoring in program years 2005 through 2007 

was done on low- and medium-risk programs. This directly contradicts the 

guidance in the Departmental Management Control Program handbook 

that a greater share of monitoring resources should be used to address 

high-risk programs. These low- and medium-risk programs do not have 

to be monitored, and CPD has flexibility in deciding whether to do so. CPD

could continue to monitor all high-risk programs but implement a random 

sampling regimen for low- and medium-risk programs. One way to do this 

would be: 

•	Have each field office rank its programs in order of risk.

•	Label the top 20 percent of programs as “high risk,” and monitor all of

them.

•	Label the next 30 percent of programs as “medium risk,” and use simple

random sampling to monitor 50 percent of them.

•	Label the bottom 50 percent of programs as “low risk” and use simple

random sampling to monitor 10 percent of them.

•	 If exceptions must be allowed, they should be available to low-, medium-,

and high-risk programs and used in only very limited situations.

One drawback to this system is that randomly sampling grantees could 

prove administratively difficult. A random sample could identify a group of

programs that the field office could not possibly monitor in the same year

due to insufficient travel funds. There are three possible solutions to this

dilemma. The first is that CPD devote more resources to travel and find a

way to share those resources across field offices; while a few field offices

may randomly be required to monitor many faraway grantees all at once, 

it is highly unlikely that all field offices will find themselves in that predicament at the same time. The second possible solution is that the sampling 

system be geographically weighted, to make faraway grantees less likely to 

be selected; however, this could cause a moral hazard among those faraway grantees, since they may know they are less likely to be monitored. 

The third possible solution, discussed in more detail in the following section, is to increase reliance on remote monitoring for the low- and mediumrisk grantees that would make up the random sample.
Risk-Based Monitoring of CPD Formula Grants 28

5. Remote Monitoring

The subfactors that are not statistically valid predictors of risk still identify 

important pieces of information. Although Environmental/Relocation is, for 

all four formula programs, one of the weakest subfactors for predicting 

findings, it is part of the rules for each program, and compliance must be

ensured. But compliance is the purpose of monitoring; it is not the purpose

of risk analysis. Fewer subfactors will result in less compliance information 

being collected through risk analysis, but that could be balanced out by 

the collection of more compliance information through monitoring. While

on-site monitoring requires significant resources, many elements of compliance can be verified off site with a high level of accuracy. Obviously, any

compliance issue contained in the risk analysis process has been verified

off site. Relying more heavily on remote monitoring—perhaps relying 

exclusively on remote monitoring for low-risk programs—will enable CPD to 

continue collecting compliance information but to do so within the proper 

framework: monitoring.

It may not be possible to conduct remote monitoring as thoroughly as

on-site monitoring. Program compliance issues can be observed from documents and phone calls, but fraud and waste may require an on-site visit. 

As a result, high-risk grantees should continue to be monitored on site. 

These five changes could cause significant upheaval if instituted immediately. As a result, PD&R staff has been working with CPD staff, particularly

from the Office of Policy Development and Coordination, to plan a phased

implementation. In addition, many decisions still need to be made by the

program experts in CPD before a new risk analysis model can be developed. As CPD considers the possibility of modifying the risk analysis and 

monitoring process, PD&R staff will remain available for support.
Risk-Based Monitoring of CPD Formula Grants 29

Appendix A. Data Management and Manipulation

The final analysis was conducted on data downloaded on March 9, 2009 via

ODBC connection to CPD’s Grants Management Process system. The Tables 

from which data were pulled are:

•	Table 2-39: Findings 

•	Table 2-42: Grantee

•	Table 2-49: Monitoring

•	Table 2-72: Risk Formula Exception

•	Table 2-73: Risk Formula Program CDBG

•	Table 2-74: Risk Formula Program ESG

•	Table 2-75: Risk Formula Program HOME

•	Table 2-76: Risk Formula Program HOPWA

From the Findings table, the critical variables downloaded were findings_id,

monitor_id and finding_number. Monitor_id uniquely identifies each monitoring case (monitoring of one program type at one grantee). A single

monitoring can have more than one finding, and each finding is recorded

as a separate observation in this table. Thus the unique identifier (or primary key) within this table is findings_id. Finding_number enumerates the

findings for a particular monitoring case.

From the Monitoring table, the critical variables downloaded were monitor_

id, full_grantee_id, prg_yr, program_type_id, and monitor_type. Monitor_

id is the same as in the Findings table. Full_grantee_id uniquely identifies

each grantee, and program_type_id identifies the type of program being

monitored. Prg_yr identifies the program year of the program at the time it

was monitored. Monitor_type indicates whether the monitoring was remote

or on-site (though it was frequently blank).

From each of the four “Risk Formula Program” tables, a similar set of variables was downloaded. Each table includes the variables full_grantee_id,

prg_yr, and grant_amount. Each table then includes a variable for every

subfactor used for that particular program type, with a prefix indicating the

program type and the factor grouping (i.e. financial, management, physical,

services, satisfaction). For instance, the CDBG table (2-73) includes variables cdbg_f_audits and cdbg_mgt_staff_cap, and the HOME table (2-75)

includes variables home_f_audits and home_mgt_staff_cap. These files do

not contain the variable program_type_id, but we created it easily, since

data for each program type was in separate tables.
Risk-Based Monitoring of CPD Formula Grants 30

The Risk Formula Exception table includes documentation of all exceptions 

granted. This table includes full_grantee_id, pgr_yr, f_excep_code, and

f_excep_comment. However, it does not include monitor_id. Because the

monitoring table has programs as its observations, but the exception table 

has grantees as its observations, we cannot explain perfectly which programs got exceptions.

From the Grantee table, a variety of grantee details were downloaded, 

including full_grantee_id, fo_code and fo_name (Field Office identifiers),

and prg_start_date (month and day on which the grantee’s program year

begins).

Merging to analytic files and processing

The tables above had to be processed, formatted, and merged together to 

analyze the relationships between variables contained in different tables.

First, we aggregated the Findings table by monitor_id and created a new

variable, finding_sum. This showed the total number of findings issued for

each instance of monitoring. We then merged this data to the Monitoring

table, linked on the monitor_id variable. This new file contained observations for all monitoring cases, including the full_grantee_id, monitor_id,

program_type_id, and finding_sum. Using the full_grantee_id and program_type_id variables as links, we could then merge this file with the

data from the risk analysis tables.

To create our analytic files, we focused only on cases where the prg_yr

was 2005, 2006, or 2007, and where risk analysis was conducted. There 

were 25 cases where a program was monitored but had no risk analysis;

these cases were deleted. We created new binary variables to identify state

programs, programs that were monitored, programs that had findings,

and programs that were high risk. We also created a file with grantees

as observations to indicate which grantees oversaw all program types or 

only 1, 2, or 3 of them; while variables like these could reveal interesting

trends, they were not featured in this analysis.
Risk-Based Monitoring of CPD Formula Grants 31

Appendix B. Bivariate Correlation Analysis

The following four tables present Pearson’s coefficients of the correlation

between each of the independent variables and the dependent variable, 

number of findings. Variables in bold are analytical variables derived from

the risk analysis subfactors. For instance, Risk Score is the sum of all 

subfactors. High Risk is a dummy variable where 1 indicates a risk score 

over 50. Financial Factor is the sum of all subfactors grouped under the 

financial factor. Highlighted P-values exceed the .05 threshold for statistical significance.

Risk Score 0.2698 0.0000

High Risk 0.2156 0.0000

Financial Factor 0.1569 0.0000

CDBG_F_GRANT_AMT 0.0204 0.5226

CDBG_F_TIMELINESS 0.0316 0.3230

CDBG_F_PROG_INCOME 0.0477 0.1352

CDBG_F_AUDITS 0.1628 0.0000

CDBG_F_FINANCIAL_COMP 0.2207 0.0000

Management Factor 0.2340 0.0000

CDBG_MGT_PRG_COMPLEX 0.0941 0.0031

CDBG_MGT_COMPLIANCE 0.2038 0.0000

CDBG_MGT_PRG_ADM_ALLOW 0.0658 0.0390

CDBG_MGT_STAFF_CAP 0.1893 0.0000

CDBG_MGT_OIG 0.0733 0.0215

CDBG_MGT_MONITOR 0.0642 0.0443

CDBG_MGT_SEC108 0.1281 0.0001

CDBG_MGT_ENV_RELOC 0.0575 0.0716

Satisfaction Factor 0.0966 0.0024

CDBG_CS_CITZ_COMP 0.0865 0.0067

CDBG_CS_RESPONSE 0.1050 0.0010

Services Factor 0.2034 0.0000

CDBG_PSD_NATL_OBJ 0.2014 0.0000

CDBG_PSD_PUB_ALLOW 0.0643 0.0439

CDBG Pearson’s Correlation Coefficients

Variable Correlation P-Value
Risk-Based Monitoring of CPD Formula Grants 32

Risk Score 0.1547 0.0459

High Risk 0.2339 0.0024

Financial Factor 0.1576 0.0420

ESG_F_GRANT_AMT – 0.0574 0.4616

ESG_F_AUDITS – 0.0131 0.8667

ESG_F_EXP_RATE 0.3817 0.0000

ESG_F_FINANCIAL_COMP 0.1147 0.1400

Physical Factor 0.0761 0.3285

ESG_PHY_REHAB 0.0761 0.3285

Management Factor 0.1799 0.0200

ESG_MGT_PRG_COMPLEX 0.0619 0.4265

ESG_MGT_COMPLIANCE 0.1695 0.0286

ESG_MGT_PRG_ADM_ALLOW – 0.0595 0.4446

ESG_MGT_STAFF_CAP 0.1323 0.0883

ESG_MGT_OIG – 0.0243 0.7555

ESG_MGT_REVIEW 0.1301 0.0938

ESG_MGT_STAFF_COSTS 0.0214 0.7839

ESG_MGT_ENV_RELOC – 0.0522 0.5030

Satisfaction Factor – 0.0241 0.7573

ESG_CS_CITZ_COMP 0.0002 0.9979

ESG_CS_RESPONSE – 0.0469 0.5476

Services Factor – 0.0374 0.6310

ESG_PSD_NATL_OBJ 0.0655 0.4006

ESG_PSD_HOMELESS_PREV – 0.0722 0.3539

ESG_PSD_ESS_SERVICE – 0.0878 0.2590

ESG Pearson’s Correlation Coefficients 

Variable Correlation P-Value
Risk-Based Monitoring of CPD Formula Grants 33

Risk Score 0.2858 0.0000

High Risk 0.2735 0.0000

Financial Factor 0.0778 0.0684

HOME_F_GRANT_AMT – 0.0277 0.5169

HOME_F_EXP_CHDO – 0.0401 0.3480

HOME_F_PROG_INCOME 0.0144 0.7355

HOME_F_AUDITS 0.1021 0.0168

HOME_F_FINANCIAL_COMP 0.1533 0.0003

Physical Factor 0.1750 0.0000

HOME_PHY_COND_PROJ 0.1750 0.0000

Management Factor 0.2621 0.0000

HOME_MGT_FUNDING 0.0194 0.6504

HOME_MGT_PRG_DESIGN 0.1104 0.0096

HOME_MGT_CHDO_ACTV 0.1111 0.0092

HOME_MGT_PRG_DELEG 0.0562 0.1887

HOME_MGT_AFFORDABILITY 0.0398 0.3523

HOME_MGT_STAFF_CAP 0.1550 0.0003

HOME_MGT_MONITOR_BY_PJ 0.1488 0.0005

HOME_MGT_CAP_OVERSIGHT 0.1313 0.0021

HOME_MGT_AUDIT 0.0230 0.5906

HOME_MGT_MONITOR 0.1812 0.0000

HOME_MGT_ENV_RELOC 0.0234 0.5849

Satisfaction Factor 0.1078 0.0115

HOME_CS_CITZ_COMP 0.1048 0.0141

HOME_CS_RESPONSE 0.0777 0.0690

Services Factor 0.2399 0.0000

HOME_PSD_INC_TARGET 0.1465 0.0006

HOME_PSD_PRG_PROGRESS 0.2156 0.0000

HOME Pearson’s Correlation Coefficients 

Variable Correlation P-Value
Risk-Based Monitoring of CPD Formula Grants 34

Risk Score 0.0949 0.4380

High Risk 0.2006 0.0984

Financial Factor 0.1146 0.3486

HOPWA_F_AUDITS 0.0752 0.5390

HOPWA_F_TIMELY_EXP – 0.0006 0.9962

HOPWA_F_FUNDING 0.2505 0.0379

HOPWA_F_MONITOR_MGT 0.0231 0.8505

Physical Factor 0.0475 0.6984

HOPWA_PHY_ASSET_PROB 0.0535 0.6623

HOPWA_PHY_ACQ 0.0247 0.8404

HOPWA_PHY_MULTIPLE_SITES 0.0393 0.7486

Management Factor – 0.0498 0.6846

HOPWA_MGT_OIG 0.0996 0.4153

HOPWA_MGT_STAFF_CAP 0.0547 0.6554

HOPWA_MGT_ONSITE_MONITOR – 0.1223 0.3167

HOPWA_MGT_ACCURATE_SUBM 0.0562 0.6467

HOPWA_MGT_PROG_ADM_CAP – 0.1098 0.3692

HOPWA_MGT_ENV_RELOC – 0.1679 0.1678

Satisfaction Factor 0.3095 0.0097

HOPWA_CS_CITZ_COMP 0.2772 0.0211

HOPWA_CS_RESPONSE 0.2851 0.0176

Services Factor 0.1507 0.2165

HOPWA_PSD_MEET_PROG_OBJ – 0.0824 0.5010

HOPWA_PSD_MUTIPLE_PRG 0.2502 0.0381

HOPWA Pearson’s Correlation Coefficients 

Variable Correlation P-Value
Risk-Based Monitoring of CPD Formula Grants 35

F_GRANT_AMT 1.00 0.37 0.05 0.09 0.42

F_TIMELINESS 1.00 0.15 0.19

F_PROG_INCOME 0.37 1.00 0.06 0.07 0.30

F_AUDITS 0.05 0.15 0.06 1.00 0.27 0.08

F_FINANCIAL_COMP 0.09 0.19 0.07 0.27 1.00 0.15

MGT_PRG_COMPLEX 0.42 0.30 0.08 0.15 1.00

MGT_COMPLIANCE 0.05 0.15 0.23 0.36 0.09

MGT_PRG_ADM_ALLOW 0.09 0.07 0.23 0.06

MGT_STAFF_CAP 0.08 0.17 0.03 0.22 0.52 0.18

MGT_OIG 0.08 0.07 0.14 0.18 0.11

MGT_MONITOR 0.10 0.13 0.04

MGT_SEC108 0.30 0.26 0.11 0.12 0.28

MGT_ENV_RELOC 0.14 0.04 0.07 0.15 0.13

CS_CITZ_COMP 0.20 0.10 0.09 0.11 0.19 0.19

CS_RESPONSE 0.10 0.08 0.06 0.11 0.13 0.08

PSD_NATL_OBJ 0.15 0.13 0.11 0.22 0.37 0.23

PSD_PUB_ALLOW 0.03 0.11 0.17 0.06

Appendix C. Full Correlation Matrix

CDBG Subfactor Correlation Matrix

Subfactors that were not correlated to each other are marked by highlighted 

cells. Cells with no background color indicate a statistically significant correlation between two subfactors; the number in the cell is the correlation

coefficient.

Variable F_Grant_

Amt

F_

Timeliness

F_Prog_

Income F_Audits F_Financial_Comp

Mgt_Prg_ 

Complex
Risk-Based Monitoring of CPD Formula Grants 36

F_GRANT_AMT 0.05 0.08 0.08 0.30

F_TIMELINESS 0.15 0.09 0.17

F_PROG_INCOME 0.03 0.07 0.26

F_AUDITS 0.23 0.07 0.22 0.14 0.10 0.11

F_FINANCIAL_COMP 0.36 0.23 0.52 0.18 0.13 0.12

MGT_PRG_COMPLEX 0.09 0.06 0.18 0.11 0.04 0.28

MGT_COMPLIANCE 1.00 0.16 0.41 0.12 0.14 0.10

MGT_PRG_ADM_ALLOW 0.16 1.00 0.19 0.06 0.07 0.06

MGT_STAFF_CAP 0.41 0.19 1.00 0.14 0.16 0.12

MGT_OIG 0.12 0.06 0.14 1.00 0.14

MGT_MONITOR 0.14 0.07 0.16 1.00

MGT_SEC108 0.10 0.06 0.12 0.14 1.00

MGT_ENV_RELOC 0.16 0.06 0.20 0.07 0.08 0.09

CS_CITZ_COMP 0.14 0.08 0.21 0.13 0.04 0.13

CS_RESPONSE 0.13 0.14 0.09

PSD_NATL_OBJ 0.32 0.15 0.40 0.18 0.13 0.17

PSD_PUB_ALLOW 0.19 0.29 0.19 0.10 0.05 0.05

F_GRANT_AMT 0.14 0.20 0.10 0.15 0.03

F_TIMELINESS 0.10 0.08 0.13

F_PROG_INCOME 0.04 0.09 0.06 0.11

F_AUDITS 0.07 0.11 0.11 0.22 0.11

F_FINANCIAL_COMP 0.15 0.19 0.13 0.37 0.17

MGT_PRG_COMPLEX 0.13 0.19 0.08 0.23 0.06

MGT_COMPLIANCE 0.16 0.14 0.13 0.32 0.19

MGT_PRG_ADM_ALLOW 0.06 0.08 0.15 0.29

MGT_STAFF_CAP 0.20 0.21 0.14 0.40 0.19

MGT_OIG 0.07 0.13 0.18 0.10

MGT_MONITOR 0.08 0.04 0.13 0.05

MGT_SEC108 0.09 0.13 0.09 0.17 0.05

MGT_ENV_RELOC 1.00 0.15 0.12 0.17 0.04

CS_CITZ_COMP 0.15 1.00 0.39 0.19 0.04

CS_RESPONSE 0.12 0.39 1.00 0.13

PSD_NATL_OBJ 0.17 0.19 0.13 1.00 0.15

PSD_PUB_ALLOW 0.04 0.04 0.15 1.00

Variable Mgt_ 

Compliance

Mgt_Prg_ 

Adm_Allow

Mgt_Staff_ 

Cap Mgt_Oig Mgt_ Monitor

Mgt_

Sec108

Variable Mgt_Env_ 

Reloc

Cs_Citz_ 

Comp

Cs_ 

Response

Psd_Natl_ 

Obj

Psd_Pub_ 

Allow

CDBG Subfactor Correlation Matrix, continued
Risk-Based Monitoring of CPD Formula Grants 37

HOME_F_GRANT_AMT 1.00 0.06 0.08 0.12 0.05 0.35

HOME_F_EXP_CHDO 1.00 0.21 0.08 0.27 0.12

HOME_F_PROG_INCOME 0.06 0.21 1.00 0.16 0.27 0.10

HOME_F_AUDITS 0.08 0.08 0.16 1.00 0.26 0.07 0.07

HOME_F_FINANCIAL_COMP 0.12 0.27 0.27 0.26 1.00 0.16 0.07

HOME_PHY_COND_PROJ 0.05 0.12 0.10 0.07 0.16 1.00 0.07

HOME_MGT_FUNDING 0.35 0.07 0.07 0.07 1.00

HOME_MGT_PRG_DESIGN 0.32 0.05 0.10 0.12 0.40

HOME_MGT_CHDO_ACTV 0.17 0.24 0.11 0.10 0.21 0.11 0.24

HOME_MGT_PRG_DELEG 0.17 0.07 0.11 0.12 0.16

HOME_MGT_AFFORDABILITY 0.10 0.06 0.09 0.22 0.11

HOME_MGT_STAFF_CAP 0.23 0.17 0.16 0.50 0.12 0.08

HOME_MGT_MONITOR_BY_PJ 0.18 0.12 0.11 0.32 0.33 0.09

HOME_MGT_CAP_OVERSIGHT 0.05 0.25 0.18 0.11 0.38 0.19 0.05

HOME_MGT_AUDIT 0.10 0.05 0.16 0.20 0.05 0.10

HOME_MGT_MONITOR 0.08 0.05 0.09 0.16 0.45 0.05

HOME_MGT_ENV_RELOC 0.12 0.10 0.19 0.12 0.09

HOME_CS_CITZ_COMP 0.12 0.13 0.12 0.12 0.22 0.10 0.07

HOME_CS_RESPONSE 0.08 0.09 0.10 0.09 0.12 0.12 0.05

HOME_PSD_INC_TARGET 0.11 0.12 0.08 0.13 0.36 0.08

HOME_PSD_PRG_PROGRESS 0.11 0.28 0.26 0.12 0.37 0.19 0.13

Variable

F_

Grant_ 

Amt

F_

Exp_ 

Chdo

F_

Prog_ 

Income

F_

Audits

F_

Financial_ 

Comp

Phy_

Cond_ 

Proj

Mgt_ 

Funding

HOME Subfactor Correlation Matrix

Subfactors that were not correlated to each other are marked by highlighted cells. Cells with no background color indicate a statistically significant correlation between two subfactors; the number in the cell is the

correlation coefficient.
Risk-Based Monitoring of CPD Formula Grants 38

HOME_F_GRANT_AMT 0.32 0.17 0.17 0.05

HOME_F_EXP_CHDO 0.24 0.07 0.10 0.23 0.18 0.25

HOME_F_PROG_INCOME 0.11 0.06 0.17 0.12 0.18

HOME_F_AUDITS 0.05 0.10 0.09 0.16 0.11 0.11

HOME_F_FINANCIAL_COMP 0.10 0.21 0.11 0.22 0.50 0.32 0.38

HOME_PHY_COND_PROJ 0.12 0.11 0.12 0.11 0.12 0.33 0.19

HOME_MGT_FUNDING 0.40 0.24 0.16 0.08 0.09 0.05

HOME_MGT_PRG_DESIGN 1.00 0.25 0.26 0.07 0.10 0.14 0.15

HOME_MGT_CHDO_ACTV 0.25 1.00 0.17 0.16 0.28 0.19 0.25

HOME_MGT_PRG_DELEG 0.26 0.17 1.00 0.09 0.10 0.12 0.20

HOME_MGT_AFFORDABILITY 0.07 0.16 0.09 1.00 0.25 0.26 0.25

HOME_MGT_STAFF_CAP 0.10 0.28 0.10 0.25 1.00 0.36 0.42

HOME_MGT_MONITOR_BY_PJ 0.14 0.19 0.12 0.26 0.36 1.00 0.51

HOME_MGT_CAP_OVERSIGHT 0.15 0.25 0.20 0.25 0.42 0.51 1.00

HOME_MGT_AUDIT 0.09 0.14 0.07 0.09 0.18 0.13 0.13

HOME_MGT_MONITOR 0.09 0.11 0.09 0.09 0.16 0.31 0.19

HOME_MGT_ENV_RELOC 0.10 0.18 0.19 0.12 0.20 0.15 0.21

HOME_CS_CITZ_COMP 0.11 0.14 0.13 0.09 0.19 0.20 0.20

HOME_CS_RESPONSE 0.07 0.09 0.07 0.08 0.11 0.13 0.11

HOME_PSD_INC_TARGET 0.08 0.10 0.17 0.14 0.29 0.17

HOME_PSD_PRG_PROGRESS 0.15 0.38 0.12 0.22 0.38 0.29 0.36

Variable

Mgt_

Prg_ 

Design

Mgt_

Chdo 

_Actv

Mgt_

Prg _

Deleg

Mgt_ 

Afford

Mgt_

Staff _

Cap

Mgt_

Monitor 

_By_Pj

Mgt_

Cap_ 

Oversight

HOME Subfactor Correlation Matrix, continued
Risk-Based Monitoring of CPD Formula Grants 39

HOME_F_GRANT_AMT 0.10 0.12 0.12 0.08 0.11

HOME_F_EXP_CHDO 0.08 0.10 0.13 0.09 0.11 0.28

HOME_F_PROG_INCOME 0.05 0.05 0.12 0.10 0.12 0.26

HOME_F_AUDITS 0.16 0.09 0.12 0.09 0.08 0.12

HOME_F_FINANCIAL_COMP 0.20 0.16 0.19 0.22 0.12 0.13 0.37

HOME_PHY_COND_PROJ 0.05 0.45 0.12 0.10 0.12 0.36 0.19

HOME_MGT_FUNDING 0.10 0.05 0.09 0.07 0.05 0.08 0.13

HOME_MGT_PRG_DESIGN 0.09 0.09 0.10 0.11 0.07 0.08 0.15

HOME_MGT_CHDO_ACTV 0.14 0.11 0.18 0.14 0.09 0.10 0.38

HOME_MGT_PRG_DELEG 0.07 0.09 0.19 0.13 0.07 0.12

HOME_MGT_AFFORDABILITY 0.09 0.09 0.12 0.09 0.08 0.17 0.22

HOME_MGT_STAFF_CAP 0.18 0.16 0.20 0.19 0.11 0.14 0.38

HOME_MGT_MONITOR_BY_PJ 0.13 0.31 0.15 0.20 0.13 0.29 0.29

HOME_MGT_CAP_OVERSIGHT 0.13 0.19 0.21 0.20 0.11 0.17 0.36

HOME_MGT_AUDIT 1.00 0.06 0.10 0.11 0.06 0.15

HOME_MGT_MONITOR 0.06 1.00 0.05 0.07 0.30 0.18

HOME_MGT_ENV_RELOC 0.10 0.05 1.00 0.18 0.11 0.10 0.14

HOME_CS_CITZ_COMP 0.11 0.07 0.18 1.00 0.50 0.05 0.13

HOME_CS_RESPONSE 0.11 0.50 1.00 0.08 0.11

HOME_PSD_INC_TARGET 0.06 0.30 0.10 0.05 0.08 1.00 0.22

HOME_PSD_PRG_PROGRESS 0.15 0.18 0.14 0.13 0.11 0.22 1.00

Variable Mgt_

Audit

Mgt_ 

Monitor

Mgt_

Env _

Reloc

Cs_

Citz _

Comp

Cs_ 

Response

Psd_ 

Inc_

Target

Psd_

Prg _

Progress

HOME Subfactor Correlation Matrix, continued
Risk-Based Monitoring of CPD Formula Grants 40

Appendix D. CDBG Ordinal Logit

Number of Observations Read 983

Number of Observations Used 983

Model Information

Parameter Estimates: Highlighted rows indicate parameters that are 

statistically significant at the .05 level.

Intercept 7 – 3.762 0.000

Intercept 6 – 3.115 0.000

Intercept 5 – 2.595 0.000

Intercept 4 – 2.179 0.000

Intercept 3 – 1.454 0.000

Intercept 2 – 0.773 0.000

Intercept 1 0.201 0.286

CDBG_F_GRANT_AMT – 0.089109 0.048

CDBG_F_TIMELINESS 0.0036072 0.863

CDBG_F_PROG_INCOME 0.0099297 0.809

CDBG_F_AUDITS 0.0452403 0.293

CDBG_F_FINANCIAL_COM 0.121874 0.017

CDBG_MGT_PRG_COMPLEX 0.0924893 0.024

CDBG_MGT_COMPLIANCE 0.1857741 0.000

CDBG_MGT_PRG_ADM_ALL 0.0146532 0.836

CDBG_MGT_STAFF_CAP 0.0875281 0.058

CDBG_MGT_OIG 0.1197305 0.547

CDBG_MGT_MONITOR 0.0112862 0.481

CDBG_MGT_SEC108 0.068088 0.098

CDBG_MGT_ENV_RELOC 0.0239441 0.894

CDBG_CS_CITZ_COMP – 0.04677 0.466

CDBG_CS_RESPONSE 0.7036885 0.040

CDBG_PSD_NATL_OBJ 0.0244642 0.224

CDBG_PSD_PUB_ALLOW 0.0150065 0.791

fy2 FY 2005,06 0.0913601 0.490

Intercept ALBUQUERQUE – 0.821196 0.084

Intercept ANCHORAGE – 0.479994 0.361

Variable Variable label Parameter Pvalue

Data Set GMP.HOME4

Response Variable findtrunc

Response Distribution Multinomial (ordered)

Link Function Cumulative Logit

Variance Function Default

Variance Matrix Blocked By FO_NAME

Estimation Technique Residual PL

Degrees of Freedom Method Containment
Risk-Based Monitoring of CPD Formula Grants 41

Intercept ATLANTA 0.8095092 0.048

Intercept BALTIMORE 0.3225549 0.530

Intercept BIRMINGHAM – 1.271152 0.002

Intercept BOSTON – 0.039254 0.890

Intercept BUFFALO – 0.641203 0.051

Intercept CARIBBEAN 0.713128 0.091

Intercept CHICAGO – 0.611947 0.046

Intercept COLUMBIA – 0.82816 0.048

Intercept COLUMBUS – 0.874876 0.010

Intercept DENVER – 1.150944 0.000

Intercept DETROIT 0.7110142 0.010

Intercept FT WORTH 0.82225 0.002

Intercept GREENSBORO – 0.361672 0.338

Intercept HARTFORD – 0.008295 0.985

Intercept HONOLULU 0.2654118 0.634

Intercept HOUSTON 0.5306184 0.444

Intercept INDIANAPOLIS 0.802613 0.096

Intercept JACKSON – 0.129098 0.830

Intercept JACKSONVILLE – 0.488812 0.202

Intercept KANSAS CITY – 0.081212 0.853

Intercept KNOXVILLE 0.1675342 0.704

Intercept LITTLE ROCK 0.7279195 0.045

Intercept LOS ANGELES 0.6093267 0.016

Intercept LOUISVILLE 0.5315575 0.297

Intercept MILWAUKEE – 0.368817 0.361

Intercept MINNEAPOLIS 2.030676 0.000

Intercept NEW ORLEANS – 0.151728 0.763

Intercept NEW YORK – 0.082174 0.830

Intercept NEWARK – 0.061503 0.851

Intercept OKLAHOMA CITY – 0.966685 0.078

Intercept OMAHA 0.8772831 0.022

Intercept PHILADELPHIA 0.1459947 0.694

Intercept PITTSBURGH – 0.384321 0.411

Intercept PORTLAND 1.0379489 0.021

Intercept RICHMOND – 1.104926 0.004

Intercept SAN ANTONIO – 0.217111 0.649

Intercept SAN FRANCISCO – 0.822653 0.002

Intercept SEATTLE 0.3465583 0.327

Intercept SOUTH FLORIDA 0.7781131 0.043

Intercept ST LOUIS – 0.449168 0.479

Intercept WASHINGTON DC 0.1668885 0.785

Variable Variable label Parameter Pvalue

Parameter Estimates, continued
Risk-Based Monitoring of CPD Formula Grants 42

CDBG Predicted Findings: The graphic below depicts the relationship 

between the number of findings predicted by our logit model and the actual 

number of findings (truncated at 7). There is a tendency for higher actual 

findings to have higher predicted findings. For instance, of the 192 programs 

with 1 finding, our model predicted more than 2 findings only 51 times (27%). 

But for the 34 programs with 6 findings, our model predicted more than 

2 findings 26 times (77%). This relationship is captured by our statistical 

model. However, a casual glance at this graphic shows how imprecise the 

model is. In several instances, the model predicted more than 3 findings for 

programs that actually had zero. Similarly, the model predicted less than 3 

findings for 48 of the 80 programs that actually had 6 or more findings. 

CDBG Logit Predictions Predicted Findings

6

5

4

3

2

1

0

0 2 4 6 8

Actual Findings (truncated at 7)

Observation
Risk-Based Monitoring of CPD Formula Grants 43

Appendix E. HOME Ordinal Logit

Data Set GMP.HOME4

Response Variable findtrunc

Response Distribution Multinomial (ordered)

Link Function Cumulative Logit

Variance Function Default

Variance Matrix Blocked By FO_NAME

Estimation Technique Residual PL

Degrees of Freedom Method Containment

Number of Observations Read 549

Number of Observations Used 549

Model Information

Parameter Estimates: Highlighted rows indicate parameters that are 

statistically significant at the .05 level.

Intercept 7 – 3.1396 0.0000

Intercept 6 – 2.7671 0.0000

Intercept 5 – 2.2419 0.0000

Intercept 4 – 1.7354 0.0000

Intercept 3 – 1.0788 0.0001

Intercept 2 – 0.4469 0.0705

Intercept 1 0.3705 0.1295

HOME_F_GRANT_AMT – 0.0546 0.3532

HOME_F_EXP_CHDO – 0.1698 0.0272

HOME_F_PROG_INCOME – 0.1562 0.2708

HOME_F_AUDITS 0.1732 0.0076

HOME_F_FINANCIAL_COM 0.0312 0.6851

HOME_PHY_COND_PROJ 0.0090 0.6934

HOME_MGT_FUNDING – 0.1335 0.2200

HOME_MGT_PRG_DESIGN 0.1847 0.0857

HOME_MGT_CHDO_ACTV 0.0752 0.4521

HOME_MGT_PRG_DELEG 0.0035 0.9717

HOME_MGT_AFFORDABILI – 0.1911 0.2722

HOME_MGT_STAFF_CAP 0.0708 0.0322

HOME_MGT_MONITOR_BY_ 0.0152 0.8331

HOME_MGT_CAP_OVERSIG 0.0551 0.2391

HOME_MGT_AUDIT – 0.6050 0.1588

HOME_MGT_MONITOR 0.0435 0.0780

HOME_MGT_ENV_RELOC – 0.1261 0.5984

HOME_CS_CITZ_COMP 0.0986 0.3372

HOME_CS_RESPONSE 0.1417 0.8000

Variable Variable label Parameter Pvalue
Risk-Based Monitoring of CPD Formula Grants 44

Variable Variable label Parameter Pvalue

HOME_PSD_INC_TARGET 0.0433 0.6846

HOME_PSD_PRG_PROGRES 0.1860 0.0040

fy2 FY 2005,06 – 0.3450 0.0539

Intercept ALBUQUERQUE – 0.3237 0.5905

Intercept ANCHORAGE 0.0144 0.9815

Intercept ATLANTA 1.8503 0.0002

Intercept BALTIMORE 1.1717 0.0674

Intercept BIRMINGHAM – 1.0648 0.0435

Intercept BOSTON – 0.4145 0.3340

Intercept BUFFALO – 0.3383 0.4473

Intercept CARIBBEAN 0.5526 0.2873

Intercept CHICAGO – 0.2155 0.6320

Intercept COLUMBIA – 0.6676 0.2471

Intercept COLUMBUS – 1.3107 0.0081

Intercept DENVER – 1.5769 0.0018

Intercept DETROIT – 0.1784 0.6586

Intercept FT WORTH 0.9994 0.0228

Intercept GREENSBORO – 0.3507 0.4052

Intercept HARTFORD 0.1299 0.8444

Intercept HONOLULU 0.3323 0.6278

Intercept HOUSTON 0.2603 0.7081

Intercept INDIANAPOLIS 0.0986 0.8506

Intercept JACKSON 0.1328 0.8459

Intercept JACKSONVILLE 0.5388 0.2492

Intercept KANSAS CITY – 0.4973 0.3543

Intercept KNOXVILLE – 0.2965 0.5860

Intercept LITTLE ROCK – 1.2983 0.0316

Intercept LOS ANGELES – 0.2484 0.4790

Intercept LOUISVILLE 0.4108 0.5443

Intercept MILWAUKEE – 0.9295 0.1123

Intercept MINNEAPOLIS 2.0643 0.0002

Intercept NEW ORLEANS 1.0978 0.0291

Intercept NEW YORK – 0.6333 0.2050

Intercept NEWARK – 0.1153 0.7677

Intercept OKLAHOMA CITY – 0.5968 0.3934

Intercept OMAHA 0.9257 0.0898

Intercept PHILADELPHIA 0.2294 0.5765

Intercept PITTSBURGH – 0.5058 0.3683

Intercept PORTLAND 1.7320 0.0032

Intercept RICHMOND – 0.3702 0.4467

Intercept SAN ANTONIO 0.3994 0.5342

Intercept SAN FRANCISCO – 1.2106 0.0025

Intercept SEATTLE – 0.0514 0.9116

Intercept SOUTH FLORIDA 0.3756 0.4829

Intercept ST LOUIS – 0.8573 0.2938

Intercept WASHINGTON DC 0.7354 0.2299

Parameter Estimates, continued
Risk-Based Monitoring of CPD Formula Grants 45

HOME Predicted Findings: The graphic below depicts the relationship 

between the number of findings predicted by our logit model and the actual 

number of findings (truncated at 7). There is a tendency for higher actual 

findings to have higher predicted findings. This is more visible graphically 

than it was for the CDBG Logit Predictions plot. There is still a large range 

for predicted values; programs that actually had 7 or more findings were 

predicted by the model to have anywhere from 1 to 6.

HOME Logit Predictions Predicted Findings

7

6

5

4

3

2

1

0

0 2 4 6 8

Actual Findings (truncated at 7)

Predf
Risk-Based Monitoring of CPD Formula Grants 46

Appendix F. Field Office Random Effects

CDBG: Field offices with no statistically significant effect on findings are

listed alphabetically in the first column. Field offices with a statistically significant positive effect on findings (issuing more findings, on average, than

risk scores would suggest) are listed are listed in Group 2, in order of the

magnitude of that effect. Field office with a statistically significant negative effect on findings (issuing fewer findings, on average, than risk scores

would suggest) are listed in Group 3, in order of magnitude of that effect.

ALBUQUERQUE MINNEAPOLIS BIRMINGHAM

ANCHORAGE PORTLAND DENVER

BALTIMORE OMAHA RICHMOND

BOSTON FT WORTH COLUMBUS

BUFFALO ATLANTA COLUMBIA

CARIBBEAN SOUTH FLORIDA SAN FRANCISCO

GREENSBORO LITTLE ROCK CHICAGO

HARTFORD DETROIT

HONOLULU LOS ANGELES

HOUSTON

INDIANAPOLIS

JACKSON

JACKSONVILLE

KANSAS CITY

KNOXVILLE

LOUISVILLE

MILWAUKEE

NEW ORLEANS

NEW YORK

NEWARK

OKLAHOMA CITY

PHILADELPHIA

PITTSBURGH

SAN ANTONIO

SEATTLE

ST LOUIS

WASHINGTON DC

Group 1: No Effect Group 2: More Findings Group 3: Fewer Findings
Risk-Based Monitoring of CPD Formula Grants 47

HOME: Field offices with no statistically significant effect on findings are

listed alphabetically in the first column. Field offices with a statistically significant positive effect on findings (issuing more findings, on average, than

risk scores would suggest) are listed are listed in Group 2, in order of the

magnitude of that effect. Field office with a statistically significant negative effect on findings (issuing fewer findings, on average, than risk scores

would suggest) are listed in Group 3, in order of magnitude of that effect.

ALBUQUERQUE MINNEAPOLIS DENVER

ANCHORAGE ATLANTA COLUMBUS

BALTIMORE PORTLAND LITTLE ROCK

BOSTON NEW ORLEANS SAN FRANCISCO

BUFFALO FT WORTH BIRMINGHAM

CARIBBEAN

CHICAGO

COLUMBIA

DETROIT

GREENSBORO

HARTFORD

HONOLULU

HOUSTON

INDIANAPOLIS

JACKSON

JACKSONVILLE

KANSAS CITY

KNOXVILLE

LOS ANGELES

LOUISVILLE

MILWAUKEE

NEW YORK

NEWARK

OKLAHOMA CITY

OMAHA

PHILADELPHIA

PITTSBURGH

RICHMOND

SAN ANTONIO

SEATTLE

SOUTH FLORIDA

ST LOUIS

WASHINGTON DC

Group 1: No Effect Group 2: More Findings Group 3: Fewer Findings
Risk-Based Monitoring of CPD Formula Grants 48

Appendix G. Subfactor Definitions (adapted 

from the Notice CPD-06-10)

These are the high risk definitions for all subfactors in the 2007 risk analysis process. To view the full definitions, consult the appendixes of Notice

CPD-06-10, at: http://www.hud.gov/offices/cpd/lawsregs/notices/2006/.

Factor 1—Financial

A. Grant Amount

Maximum of 5 points

The community’s grant amount for the most recently completed program year falls within 

the top quartile of all CDBG-funded communities within the Office’s jurisdiction for the same

program year. 

B. Timeliness

Maximum of 10 points

If evaluating an entitlement grantee—The grantee is currently untimely as the amount 

of CDBG funds available to the grantee 60 days prior to the end of the most recent program 

year is more than 1.5 times the grant amount for the current program year.

If evaluating a State grantee—The current rate of expenditure for the past 12 months is 

less than 1.0 and the State has a ratio of greater than 2.5, 60 days prior to the start of the 

program year or the State has not distributed and announced 100% of its State CDBG grant 

excluding State Administration and TA within 15 months of the date of its last grant award.

C.	 Program Income

Maximum of 5 points

The grantee, State recipient(s) or its sub-recipient(s) received program income over

$100,000 per year directly generated from the use of CDBG funds during the most recently 

completed program year. 

D. Audits

Maximum of 5 points

An A-133 audit due for the most recently completed reporting period or any previous 

reporting period within the three most recent program years has not been submitted to the 

Federal Audit Clearinghouse within prescribed timeframe OR a grantee has open findings and

is overdue in carrying in carrying out any agreed upon corrective action.

E. Financial Compliance

Maximum of 5 points

During the most recent program year, staff has demonstrated an inability to administer 

the financial management responsibilities for the CDBG program as evidenced thru one or

more violation(s) of regulations or deficiencies of Part 85, Part 84, A-87 or A-110 or such

equivalent requirements as set forth by the State CDBG program OR one or more vacancies 

for key financial management staff of CDBG programs have existed for more than six months

(key financial management staff is defined as staff with direct oversight of financial records

and or distribution of program funds).

Factor 2—Management

A. Program Complexity

Maximum of 5 points

Grantee or State recipients undertakes activities beyond those described in 24 CFR 570.201 

and 570.202 AND utilizes Community Based Development Organizations (CBDOs) and/or

subrecipients to assist in carrying out such activities. 

B. Timely and Accurate 

Submissions

Maximum of 5 points

One and/or more of grantee’s required submissions are incomplete OR are received 30 days 

or more after prescribed timeframes. This includes: Consolidated Plans, Annual Action Plans, 

PERS and CAPERS during the most recent program year. 

C. Program Administration 

CAP

Maximum of 5 points

The grantee has exceeded the administration CAP for the CDBG program for the most 

recently completed program year.

D. Staff Capacity

Maximum of 5 points

During the most recent program year, staff has demonstrated an inability to administer the 

CDBG program as evidenced through one or more violations of regulations, or monitoring 

findings related to the CDBG program that the grantee has failed to resolve within the

last six months OR there are one or more vacancies for CDBG key staff that have existed 

for more than six months (key staff is defined as staff with assigned management and

administrative responsibilities for program compliance with rules and regulations).

CDBG subfactors
Risk-Based Monitoring of CPD Formula Grants 49

E.	 OIG Audit

Maximum of 5 points

An OIG Audit is scheduled or currently underway and a final report has not been issued OR a

previous OIG Audit identified one or more recommendations that have not been cleared AND the

grantee is not on schedule for carrying out such recommendations as of the date of this review.

F. On Site Monitoring

Maximum of 15 points

HUD has not conducted an on-site monitoring of the CDBG program for this grantee within 

the last two program years OR there are one or more overdue open findings.

G. Section 108 Activity

Maximum of 5 points

The grantee manages a Section 108 Loan portfolio of more than $2,000,000 or funds that 

are the equivalent of 25% or more of the grantee’s most recent CDBG grant amount.

H. Environmental/

Relocation

Maximum of 2 points

The grantee has not demonstrated a record of program compliance or currently has known 

compliance problems with either Environmental (Part 50 or 58) or Uniform Relocation

Acquisition Act requirements.

Factor 3—Satisfaction

A. Citizen Complaints

Maximum of 5 points

Citizen complaints have been received during the most recently completed program

year through such sources as; citizen letters, phone calls, hot line complaints,

newspapers articles, etc., and when considering the grantee’s response resulted in 

violations of CDBG regulations or findings.

B. Responsiveness

Maximum of 2 points

Grantee has failed to respond to complaints and/or citizen inquiries forwarded

through HUD within prescribed timeframes during the most recent program year. 

Factor 4—Services

A. Meeting National

Objectives

Maximum of 12 points

Sanctions have been placed on the grantee for noncompliance with national 

objectives or eligibility requirements during the most recently completed program

year OR the grantee has not taken recommended corrective actions to address 

outstanding sanctions OR there are known problems identified through review

of reports or information received that indicates that the grantee is currently not 

meeting the national objectives OR is implementing ineligible activities.

B. Public Service CAP

Maximum of 6 points

Grantee exceeded the public service CAP for the most recently completed program 

year.

CDBG subfactors, continued

Factor 1—Financial

A. Grant Amount

Maximum of 5 points

The Participating Jurisdiction’s (PJ) grant amount for the most recently completed program

year falls within the top quartile of all HOME funded communities within the Office’s

jurisdiction for the same program year. 

B. Commitments, CHDO 

Reservations, and 

Expenditures

Maximum of 5 points

HOME Production Reports indicate that the most recent commitment and expenditure 

deadlines were not met.

C.	 Program Income

Maximum of 3 points

The PR 27 indicates that the PJ is not receipting program income. 

D. Audits

Maximum of 5 points

An A-133 audit due for the most recently completed reporting period or any previous 

reporting period within the three most recent program years has not been submitted to the 

Federal Audit Clearinghouse within prescribed timeframe OR a PJ has open findings and is

overdue in carrying out any agreed upon corrective action.

E. Financial Compliance 

Maximum of 5 points

During the most recent program year, staff has demonstrated an inability to administer the 

financial management responsibilities for the HOME program as evidenced through one or

more violations of regulations or deficiencies of Part 85, Part 84, A-87 or A-110 OR one or 

more vacancies for key financial management staff of HOME programs have existed for more

than six months (key financial management staff is defined as staff with direct oversight of

financial records and or distribution of program funds).

HOME subfactors
Risk-Based Monitoring of CPD Formula Grants 50

HOME subfactors, continued

Factor 2—Physical

A. Physical Condition 

of Projects

Maximum of 12 points

HUD has not conducted an onsite review of the physical conditions of any HOME units in 

more than 3 years OR Previous monitoring (on-site or remote) identified findings concerning

the physical condition of HOME properties which have not been resolved as of this date OR

HOME projects did not meet applicable standards at completion or are not maintained in

standard and habitable conditions for the last two most recently completed program years 

which was determined by such means as the CAPER review or citizen correspondence.

Factor 3—Management

A. Multiple Funding 

Sources

Maximum of 2 points

There are large (25 or more units) rental projects OR other projects with three or more

funding sources.

B. Program Design

Maximum of 2 points

PJ is administering more than three HOME-funded programs/activities OR since the HOME 

program was last monitored on-site, the PJ has undertaken new activities or made changes

to an existing program.

C. CHDO activities

Maximum of 2 points

Based on the PR 25 or SNAPSHOT reports, the PJ’s CHDO activities are not progressing from

reservations to commitment, from commitments to disbursement or CHDOs are responsible 

for carrying out activities that are complex (i.e., funding from more than one source, more

than 25 units, or new project types) in nature.

D. Program Delegations

Maximum of 2 points

Program functions are being delegated to, and carried out by other entities such as: state 

recipients, contractors, lenders, and/or real estate professionals.

E. Affordability 

Requirements

Maximum of 2 points

More than one project in the most recently completed program year has not complied with

affordability requirements.

F. Staff Capacity

Maximum of 9 points

During the most recent program year, staff has demonstrated an inability to administer the 

HOME program as evidenced through one or more violations of regulations or monitoring 

findings related to the HOME program, that the grantee has failed to resolve within the last

six months OR one or more vacancies for HOME key staff have existed for more than six 

months (Key staff is defined as staff with assigned management and administrative 

responsibilities for program compliance with rules and regulations). 

G.	 Ongoing Project

Monitoring by PJ’s

Maximum of 5 points

In the most recent program year, monitoring or other information available to the field office

(e.g., through sampling, inquiries or complaints) indicated that the PJ was not monitoring

or might not be adequately monitoring HOME rents, occupancy requirements, and physical 

conditions of projects under a period of affordability.

H. Subrecipient/Consortia 

Members/CHDOs/State 

Recipient Capacity and 

Oversight

Maximum of 7 points

Available information (e.g., internal PJ monitoring reports, monitoring plans, audits, citizen

correspondence, previous HUD monitoring audits, etc.) indicate that PJ has not carried out

oversight responsibilities in regards to subrecipients/state recipients or has not reviewed 

performance of subrecipients/state recipients within the last two program years.

I.	 OIG Audit

Maximum of 2 points

An OIG Audit is scheduled or currently underway and a final report has not been issued OR a 

previous OIG Audit identified one or more recommendations that have not been cleared and

the grantee is not on schedule for carrying out such recommendations as of the date of this 

review.

J.	 On-Site Monitoring

Maximum of 15 points

HUD has not conducted an on-site monitoring of the HOME program for this grantee within 

the last two program years OR there are one or more overdue open findings.

K.	 Environmental/

Relocation

Maximum of 2 points

The grantee has not demonstrated a record of program compliance or currently has known 

compliance problems with either Environmental (Part 50 or 58) or Uniform Relocation

Acquisition Act requirements.

Factor 4—Satisfaction

A.	 Citizen Complaints

Maximum of 5 points

Citizen complaints have been received during the most recently completed program year

through such sources as: citizen letters, phone calls, hot line complaints, newspapers

articles, etc., and when considering the PJ’s response resulted in violations of HOME

regulations or findings.
Risk-Based Monitoring of CPD Formula Grants 51

ESG subfactors

Factor 1—Financial

A. Grant Amount

Maximum of 5 points

The community’s grant amount for the most recently completed program year falls within 

the top quartile of all ESG funded communities within the Office’s jurisdiction for the same

program year. 

B. Audits

Maximum of 5 points

An A-133 audit due for the most recently completed reporting period or any previous 

reporting period within the three most recent program years has not been submitted to the 

Federal Audit Clearinghouse within prescribed timeframe OR a grantee has open findings and

is overdue in carrying out any agreed upon corrective action.

C. 24 Month Expenditure 

Provisions

Maximum of 5 points

The grantee has failed to carry out activities that would provide for all funds to be expended 

within the most recent 24-month timeframe as evidenced by the most currently submitted 

CAPER and other reports as of the date of this review.

D. Financial Compliance

Maximum of 5 points

During the most recent program year, staff has demonstrated an inability to administer the 

financial management responsibilities for the ESG program as evidenced through one or

more violation of regulations or deficiencies of Part 85, Part 84, A-87 or A-110 OR one or 

more vacancies for key financial management staff of ESG programs have existed for more

than six months (Key financial management staff is defined as staff with direct oversight of

financial records and or distribution of program funds).

Factor 2—Physical

A. Rehabilitation

Maximum of 10 points

HUD has not conducted an onsite review of the physical conditions of any ESG rehabilitation 

project in more than three program years OR previous monitoring (on-site or remote)

identifies findings concerning the physical condition of ESG rehabilitated properties OR the 

grantee has not met its services obligation (either ten or three years as applicable).

Factor 3—Management

A. Program Complexity

Maximum of 5 points

Grantee undertakes three or more activities provided for at 24 CFR 576.21 OR uses a subrecipient to assist in carrying out such activities; OR activities are being currently undertaken 

that have not been carried out since the grantee was last monitored on site for the ESG program.

B. Timely and Accurate 

Submissions

Maximum of 5 points

One and/or more of grantee’s required submissions are incomplete OR are received 30 days 

or more after prescribed timeframes. This includes: Consolidated Plans, Annual Actions 

Plans and CAPERS during the most recent program year. 

C. Program 

Administration CAP

Maximum of 5 points

The grantee has exceeded the administration CAP for the ESG program for the most recently 

completed program year.

D. Staff Capacity

Maximum of 5 points

During the most recent program year, staff has demonstrated an inability to administer 

the ESG program as evidenced through one or more violations of regulations or monitoring 

findings related to the ESG program that the grantee has failed to resolve within the last six

months OR there are one or more vacancies for ESG key staff have existed for more than six 

months (Key staff is defined as staff with assigned management and administrative 

responsibilities for program compliance with rules and regulations). 

E.	 OIG Audit

Maximum of 3 points

An OIG Audit is scheduled or currently underway and a final report has not been issued OR a 

previous OIG Audit identified one or more recommendations that have not been cleared AND the 

grantee is not on schedule for carrying out such recommendations as of the date of this review.

HOME subfactors, continued

B. Responsiveness

Maximum of 2 points

Grantee has failed to respond to complaints and/or citizen inquiries forwarded through HUD

within prescribed timeframes during the most recent program year.

Factor 5—Services

A.	 Income Targeting

Maximum of 3 points

Income determinations procedures have not been monitored within the most recent three

program years, OR the previous monitoring has found that the PJ did not meet incometargeting requirements or was incorrectly determining income. 

B. Program Progress

Maximum of 5 points

PR 22 report (IDIS Report), HOME SNAPSHOT or other information show that more than

20% of PJ projects are not moving from commitment to construction and from construction

to completion within prescribed timeframe OR PR 16 and 22 indicate that occupancy data is 

not being entered for completed projects.
Risk-Based Monitoring of CPD Formula Grants 52

ESG subfactors, continued

F. On-Site Monitoring

Maximum of 15 points

HUD has not conducted an on-site monitoring of the ESG program for this grantee within the 

last two program years OR there are one or more overdue open findings.

G. Staff Costs

Maximum of 5 points

Staff operating costs are classified improperly OR have exceeded ten percent of annual 

allocation during the most recently completed program year.

H. Environmental/

Relocation

Maximum of 2 points

The grantee has not demonstrated a record of program compliance or currently has known 

compliance problems with either Environmental (Part 50 or 58) or Uniform Relocation

Acquisition Act requirements.

Factor 4—Satisfaction

A.	 Citizen Complaints

Maximum of 5 points

Citizen complaints have been received during the most recently completed program year

through such sources as: citizen letters, phone calls, hot line complaints, newspapers

articles, etc., and when considering the grantee’s response resulted in violations of ESG 

regulations or findings.

B. Responsiveness

Maximum of 5 points

Grantee has failed to respond to complaints and/or citizen inquiries forwarded through HUD

within prescribed timeframes during the preceding program year. 

Factor 5—Services

A. Meeting Program 

Objectives

Maximum of 5 points

Sanctions have been placed on grantee for failing to meet program requirements during the 

most recently completed program year OR the grantee is not complying with sanctions that 

were previously placed on them within the three most recent program years OR there are 

known problems identified through review of reports or information received that indicate

grantee is currently not in compliance or is carrying out ineligible activities. 

B. Homeless Prevention

Maximum of 5 points

Homeless prevention activities are classified improperly OR exceed more than 30 percent of 

the annual allocation during the most recently completed program year. 

C. Essential Services

Maximum of 5 points

Essential services activities are classified improperly OR exceed more than 30 percent during 

the most recently completed program year and no waiver was granted. 

HOPWA subfactors

Factor 1—Financial

A. Audits

Maximum of 5 points

An A-133 audit due for the most recently completed reporting period or any previous 

reporting period within the three most recent program years has not been submitted to the 

Federal Audit Clearinghouse within prescribed timeframe OR a grantee has open findings and

is overdue in carrying out any agreed upon corrective action.

B. Timely Expenditures

Maximum of 10 points

A grantee’s performance has been untimely in the expenditure of funds in accordance with 

program requirements OR a prior problem of this nature was not resolved as of the date of 

this assessment.

C.	 Size of funding

Maximum of 5 points

The total amount of unexpended balances under the program as 

of the date of this review is $2,000,000 or more.

D. Financial Compliance

Maximum of 5 points

During the most recent program year, staff has demonstrated an inability to administer the 

financial management responsibilities for the HOPWA program as evidenced through one

or more violation of regulations or deficiencies of Part 85, Part 84, A-87 or A-110 OR one 

or more vacancies for key financial management staff of HOPWA programs have existed

for more than six months (Key financial management staff is defined as staff with direct

oversight of financial records and or distribution of program funds).

Factor 2—Physical

A. Existing or Previous 

Physical Asset Problems

Maximum of 5 points

A problem or finding has been identified in the development, maintenance or operation of

a HOPWA-funded physical asset or other physical site-related activity; and has not been

resolved as of the date of this review OR the physical asset has not been monitored within 

the most recent three program years.
Risk-Based Monitoring of CPD Formula Grants 53

B. Acquisition, 

Construction and 

Rehabilitation of 

Physical Assets

Maximum of 5 points

HOPWA funds were used for the acquisition or construction or rehabilitation of twenty-four or

more units of a physical asset within the most recent three program years.

C. Multiple Sites for 

Physical Assets

Maximum of 5 points

HOPWA funds are used for the development, or maintenance or operation of physical assets

at more than 7 current facility sites within the most recent three program years.

Factor 3—Management

A.	 OIG Audit

Maximum of 3 points

An OIG Audit is scheduled, or is currently underway and a final report has not been issued,

OR a previous OIG Audit identified one or more recommendations that have not been

cleared, and the grantee is not on schedule for carrying out such recommendations as of the 

date of this review.

B. Staff Capacity

Maximum of 10 points

During the most recent program year, staff has demonstrated an inability to administer the 

HOPWA program as evidenced through one or more violations of regulations, or monitoring

findings related to the HOPWA program that the grantee has failed to resolve within the last

six months OR there are one or more vacancies for HOPWA key staff have existed for more

than six months (Key staff is defined as staff with assigned management and administrative

responsibilities for program compliance with rules and regulations).

C. On-Site Monitoring

Maximum of 15 points

HUD has not conducted an on-site monitoring of the HOPWA program for this grantee within

the last three program years OR there are one or more overdue open findings.

D. Timely and Accurate 

Submissions

Maximum of 5 points

One and/or more of grantee’s required submissions are incomplete OR are received 30 days 

or more after prescribed timeframes. This includes: Consolidated Plans, Annual Action Plans, 

PERS and CAPERS during the most recent program year. 

E. Program Administration 

CAP

Maximum of 2 points

The grantee has exceeded the administration CAP for the HOPWA program for the most

recently completed program year.

F. Environmental/

Relocation

Maximum of 5 points

The grantee has not demonstrated a record of program compliance, or currently has known 

compliance problems with either Environmental (Part 50 or 58) or Uniform Relocation

Acquisition Act requirements.

Factor 4—Satisfaction

A.	 Citizen Complaints

Maximum of 5 points

Citizen complaints have been received during the most recently completed program year

through such sources as: citizen letters, phone calls, hot line complaints, newspapers

articles, etc., and when considering the grantee’s response resulted in violations of HOPWA

regulations or findings.

B. Responsiveness

Maximum of 5 points

Grantee has failed to respond to complaints and/or citizen inquiries forwarded through HUD

within prescribed timeframes during the most recent program year.

Factor 5—Services

A. Meeting Program 

Objectives

Maximum of 5 points

Sanctions have been placed on grantee for failing to meet program requirements during the 

most recently completed program year OR the grantee has not taken corrective actions to 

address outstanding sanctions that were previously placed on them within the three most 

recent program years OR there are known problems identified through review of reports or

information received that indicate grantee is currently not in compliance, or is carrying out 

ineligible activities. 

B. Multiple Sponsors

Maximum of 5 points

A grantee carries out a program with five or more sponsors AND/OR the grantee or sponsor 

receives funding from more than two additional entities (e.g., HHS, State, City, Foundation,

etc.) within the most recent three program years

HOPWA subfactors, continued