National Institute of 

Standards and Technology 

and 

National Oceanic and Atmospheric 

Administration 

NIST and NOAA Monitor 

Their Recovery Act Programs, 

but Performance Metrics Need to 

Measure Outcomes 

Final Report No. ARR-19881 

May 21, 2010 

For Public Release 

Office of Audit and Evaluation 

U.S. DEPARTMENT OF COMMERCE 

Office of Inspector General 
c«TOF 

UNITED STATES DEPARTMENT OF COMMERCE 

Office of Inspector General 

Washington. D.C. 20230 

May 21, 2010 

MEMORANDUM FOR:	 Dr. Patrick D. Gallagher. Director 

National Institute of Standards and Technology 

Dr. Jane Lubchenco. Administrator 

National Oceanic and Atmospheric Administration 

FROM:	 Ann C. Filers \A^ 

Sr. Advisor and Project Lead, OIG Recovery Act Task Force 

SUBJECT:	 Programs ' Progress Monitored to Mitigate Risk, but Metrics Need 

to Measure Outcomes, Final Report No. ARR-19881 

We are providing you with our final report on NIST"s and NOAA's implementation of American 

Recovery and Reinvestment Act of 2009 performance measurement requirements. 

We found that both NOAA and NIST have established mechanisms to monitor the progress of 

Recovery Act programs to more actively manage risk. The Recovery Act's emphasis on 

accountability and performance has resulted in improved management capabilities and systems, 

especially with respect to nonfmancial risk management. However, while the bureaus created or 

improved existing systems to keep track of indicators of progress—e.g.. dollars spent, milestones 

met, jobs created—they had not developed measures of substantive outcomes in order to assess 

specific program goals or the broader public benefits of significant programs funded by the 

Recovery Act. We discussed our findings with NOAA's chief financial officer (CFO) on 

February 22, 2010, and with NIST'S deputy CFO and other NIST officials on February 25; we 

then provided a draft report to the bureaus on April 16. 

NOAA's April 22 response to our draft report says that NOAA agrees with the report and has 

already begun to modify its Recovery Act performance metrics to address the report's 

recommendation. NIST's May 14 response reemphasized the challenges we discussed in the 

report regarding measuring core science programs. However, NIST agreed that its performance 

metrics for the Scientific and Technical Research Services equipment program should be 

strengthened and that it would develop an appropriate set ofintermediate measures. We 

summarize both bureaus' responses in our report. 

Within 60 days from the date of this memorandum, please provide us with an action plan that 

responds to our recommendation. Please also accept our thanks to NOAA and NIST officials for 

the courtesies shown to us during the inspection. If you have any questions, please contact me at 

(202) 482-4328 or John Bunting at (202) 527-0635. 
  

Attachment 

cc: David Robinson, Chief Financial Officer, NIST 

Stephen Kunze, Deputy Chief Financial Officer, NIST 

Rachel Kinney, Recovery Act and Audit Liaison, NIST 

Maureen Wylie, Chief Financial Officer, NOAA 

Mack Cato, Audit Liaison, NOAA 

Ronald Rhodes, Recovery Act Liaison, NOAA 

Lisa Casias, Deputy Chief Financial Officer/Director for Financial Management, Office

 of Financial Management 

Ellen Herbst, Senior Adviser for Recovery Implementation, Office of the Vice President 
Report In Brief Report In Brief 

U.S. Department of Commerce Office of Inspector General 

May 21, 2010 

Why We Did this Review 

Background 

On February 17, 2009, the President 

signed the Recovery Act into law. 

The act requires an unprecedented 

degree of transparency and accountability from agencies that receive 

funding, and sets forth specific 

responsibilities for the agencies to 

manage funds and program operations. Five operating units in the 

Department of Commerce, plus the 

Office of Inspector General (OIG), 

received Recovery Act funds. 

OIG has been proactive in our 

oversight of Commerce’s Recovery 

Act programs and activities, including the bureaus’ implementation of 

the act’s performance measurement 

requirements. 

The American Recovery and Reinvestment Act of 2009 and the Office of Management and Budget’s 

(OMB) implementation guidance 

require agencies to monitor the 

progress of Recovery Act programs 

to identify areas of, and address 

concerns about, high risk or low 

performance. OMB also requires 

agencies to include accountability 

objectives as part of the risk-mitigation process and to develop 

quantifiable performance measures 

that address the use of Recovery 

Act funds to meet the act’s goals. 

We examined NIST’s $200 million 

Scientific and Technical Research 

Services (STRS) account and 

NOAA’s $600 million Procurement, Acquisition, and Construction account to determine 

whether (1) the bureaus are 

monitoring the progress of certain 

programs funded by the Recovery Act; (2) NIST and NOAA 

have mitigated risks or addressed 

schedule- or cost-related concerns; 

and (3) the bureaus’ performance 

metrics comply with OMB requirements. 

American Recovery and Reinvestment Act 

NIST and NOAA Monitor Their Recovery Act Programs, 

but Performance Metrics Need to Measure Outcomes

 (ARR-19881) 

What We Found 

In order to more actively manage risk, the National Institute of Standards and 

Technology (NIST) and the National Oceanic and Atmospheric Administration 

(NOAA) have established mechanisms to monitor the progress of Recovery 

Act programs. The Recovery Act’s emphasis on accountability and performance has resulted in improved management capabilities and systems, especially with respect to nonfinancial risk management. These improvements have 

the potential to extend past the Recovery Act into other programs. 

However, while the bureaus have created new systems or improved existing 

systems to keep track of indicators of progress—e.g., dollars spent, milestones 

met, jobs created—they have not developed measures of substantive outcomes 

that assess specific program goals or the broader public benefits of significant 

programs funded by the Recovery Act. 

What We Recommended 

NOAA and NIST should improve their performance metrics for the more 

significant Recovery Act programs, focusing on intermediate outcomes that 

assess the programs’ benefits. For example, performance metrics should track 

whether an investment has improved the body of knowledge in a particular 

field, disseminated newly developed tools and models, supported a research or 

technological innovation, or made other advancements in science and technology for the public’s benefit. 

We issued a draft report on our findings in April 2010, and gave the bureaus 

the opportunity to respond. NOAA agreed with the report and has already 

begun to modify its Recovery Act performance metrics to address our recommendation. NIST’s response reemphasized the challenges we discussed in the 

report regarding measuring core science programs. However, NIST agreed 

that some of its performance metrics for the STRS equipment program should 

be strengthened, and that it would develop an appropriate set of intermediate 

measures. 
U.S. Department of Commerce Final Report ARR-19881 

Office of Inspector General May 21, 2010 

Introduction 

The American Recovery and Reinvestment Act of 2009 and the Office of Management and 

Budget’s (OMB) implementation guidance require agencies to monitor the progress of Recovery 

Act programs to identify areas of, and address concerns about, high risk or low performance. 

OMB also requires agencies to include accountability objectives as part of the risk-mitigation 

process (see box) and to develop quantifiable performance measures that address the use of 

Recovery Act funds to meet the act’s goals. 

The U.S. Government Accountability Office (GAO) defines performance measurement as “the 

ongoing monitoring and reporting of program accomplishments, particularly progress toward 

pre-established goals . . . typically conducted by program or agency management.”1 Performance 

measurement during a project involves the timely, cost-effective oversight and management of a 

program’s schedule, as well as its milestone progress and possible risks to its completion. 

Performance measurement often addresses the type or level of program activities conducted 

(process), the direct products and services 

Recovery Act Accountability Objectives   delivered by a program (outputs), and the results 

of those products and services (outcomes). 

•	 Funds are awarded and distributed in Performance measurement also focuses on 

a prompt, fair, and reasonable whether a program has achieved its objectives, 

manner.   often expressed as measurable performance 

•	 Projects funded under this act avoid standards. An agency’s inattention to a program’s 

unnecessary delays and cost schedule, milestone progress, and risks inhibits its 

overruns.   ability to mitigate those risks and take timely 

•	 Program goals are achieved, including action to support complicated or struggling 

specific program outcomes and programs. 

improved results on broader

economic indicators. Performance-based data demonstrate whether, in 

•	 The recipients and uses of all funds what ways, and why a program is working. Such 

are transparent to the public, and the data inform decisions by senior leadership, 

public benefits of these funds are program managers, the executive branch, and 

reported clearly, accurately, and in a Congress. Citizens also want to know that 

timely manner. Recovery Act funds are awarded fairly and 

•	 Funds are used for authorized
 promptly, along with the public benefits of these 

purposes and instances of fraud,
 investments. 

waste, error, and abuse are
 

mitigated.
 The objectives of this review were to 

(1) determine whether NIST and NOAA have 

taken action to monitor the progress of certain Source: OMB Memorandum M‐09‐15   programs funded by the Recovery Act in order to 

mitigate risk; (2) determine whether NIST and 

NOAA have mitigated risks or addressed schedule- or cost-related concerns; and (3) assess 

whether the bureaus’ performance metrics comply with OMB requirements. The Office of 

Inspector General (OIG) assessed NIST’s $220-million Scientific and Technical Research 

                                                            

1

U. S. Government Accountability Office, Performance Measurement and Evaluation—Definitions and 

Relationships (GAO-05-739SP, May 2005). 

1
 
 

U.S. Department of Commerce Final Report ARR-19881 

Office of Inspector General May 21, 2010 

Services (STRS) account and NOAA’s $600-million Procurement, Acquisition, and Construction 

(PAC) account (see tables 1, 2, and 3 on pages 4, 7, and 8, respectively). 

We conducted our work at NOAA in Silver Spring, Maryland, and at NIST in Gaithersburg, 

Maryland. We met with NOAA and NIST program officials to discuss their development of 

performance measures to meet the compliance requirements of the Recovery Act as described in 

OMB Memorandum M-09-15, Updated Implementing Guidance for the American Recovery and 

Reinvestment Act of 2009. We reviewed policies, procedures, plans, and metrics for monitoring 

and reporting the progress of certain Recovery Act programs. Also, we assessed relevant 

management information systems that support Recovery Act performance measurement tracking 

and reporting. 

We performed this review from January through April 2010, under authority of the Inspector 

General Act of 1978, as amended; Department Organization Order 10-13, dated August 31, 

2006, as amended; and in accordance with the Quality Standards for Inspections (revised 

January 2005) issued by the President’s Council on Integrity and Efficiency. 

Summary of Findings 

In order to more actively manage risk, NOAA and NIST have established mechanisms to 

monitor the progress of Recovery Act programs. The Recovery Act’s emphasis on accountability 

and performance has resulted in improved management capabilities and systems at these 

bureaus, especially with respect to nonfinancial risk management. These improvements have the 

potential to extend past the Recovery Act into other programs. However, while NOAA and NIST 

have created new systems or improved existing systems to keep track of indicators of progress— 

e.g., dollars spent, milestones met, jobs created—they have not developed measures of 

substantive outcomes in order to assess whether specific program goals are being achieved or the 

broader public benefits of significant programs funded by the Recovery Act. 

2
 
‐ ‐

U.S. Department of Commerce Final Report ARR-19881 

Office of Inspector General May 21, 2010 

Findings and Recommendation 

I.	 NIST and NOAA Have Established Mechanisms to Monitor the Progress of Programs 

to More Actively Manage Risk 

For Recovery Act investments, OMB required each agency to include in its program specific 

plans a delivery schedule with “milestones for major phases of the program’s activities (e.g., the 

procurement phase, planning phase, project execution phase . . .) with planned deliver

‐

y date(s).”2 

As part of the risk management process, agencies are required to identify and mitigate the 

potential risk of “unnecessary delays [in the delivery schedule] and cost overruns.”3 To assist in 

determining how well agencies are dealing with this risk, they are required to track program 

progress to determine whether funds are being obligated and expended in a timely manner.4 

 

OMB provides guidance on the intersection between performance measurement and risk 

management, stating that agencies must develop measures of “expected quantifiable outcomes 

consistent with the intent and requirements of [the Recovery Act] and the risk management 

requirements of Section 3 [of the OMB guidance].”5 Specifically, “[a]gencies should prepare risk 

mitigation plans for those risks with the highest probability of occurrence and the greatest impact 

[on program performance] if not mitigated [and] whenever possible; agencies should identify 

quantifiable measures of performance, including ranges of acceptable and unacceptable 

performance.”6 

NIST and NOAA have taken positive steps to comply with these OMB requirements for their 

Recovery Act investments, as detailed in table 1. For example, nonfinancial risk management has 

become a key factor for both NIST and NOAA officials working on Recovery Act programs, 

who indicated that increased attention to risk and more regular discussions about how to mitigate 

a risk or resolve a problem have helped its bureaus set priorities and balance competing 

concerns. Many officials also pointed to the consistent use of project management tools, either 

the software used by NOAA or the database developed by NIST, which have informed and 

complemented the risk assessment. 

 

                                                            

2

OMB Memorandum M‐09‐10 at 2.8(e); OMB Memorandum M‐09‐15 at 2.8(e). 3

OMB Memorandum M‐09‐15 at 3.1. This risk must be monitored and reassessed by the agency throughout the 

period of availability of Recovery Act funds. OMB Memorandum M‐09‐15 at 3.11. In addition, Section 1512 of the 

Recovery Act requires recipients of funds to report to the agency on the completion status of projects and activities. 4

See OMB Memorandum M‐09‐15 at 3.7. 5

OMB Memorandum M‐09‐15 at 2.8(g). With respect to specific programs, the risk management requirements of 

Section 3 require agencies to identify, prioritize, and plan to mitigate programmatic risks. See OMB Memorandum 

M 09 15 at 3.9. 6

OMB Memorandum M-09-15 at 3.12.
 

3
 
U.S. Department of Commerce Final Report ARR-19881 

Office of Inspector General May 21, 2010 

 a As discussed in finding II, these performance measures do not generally support measurement of 

performance outcomes. 

One positive consequence of the Recovery Act’s increased emphasis on accountability and 

performance is that NIST’s and NOAA’s risk management capabilities (especially nonfinancial 

 

 

 

 

OMB and Recovery Act Requirements 

Mechanism 

Track Progress Against Timely 

Expenditure of Recovery Act 

Funds 

Risk 

Assessment 

Quantifiable 

Performance 

Measures 

Other 

Benefits 

Recovery Act 

Account 

Milestones for 

Program 

Activity 

Identify and 

Mitigate Risk 

of Delay and 

Cost 

Overruns 

NIST 

Scientific and 

Technical 

Research 

Services 

(STRS) 

Account: $220 

million plus 

$30 million in 

transfers 

• Recovery 

Act 

Steering 

Committee 

• Program 

Management Office 

• Database 

for tracking 

obligations 

and 

program 

progress 

Monthly action 

plans address 

funds’ obligation 

status, 

milestones, risk 

identification, 

and risk 

assessment 

Steering 

Committee 

meets as 

needed to 

identify, 

prioritize, and 

mitigate 

emerging risks, 

focusing on 

obligation 

• Contractor 

develops 

and 

implements 

risk 

management 

system 

• All 

Recovery 

Act projects 

are 

assessed 

for risk 

(low/ 

medium/ 

high) 

• Counts jobs 

created 

• Tracks acquisition 

milestones, 

including 

obligations 

and expendituresa 

Permanent 

governance 

improvement: 

• First time 

nonfinancial 

risk systematically 

measured 

and 

managed 

• Potential to 

extend past 

Recovery 

Act 

NOAA 

Procurement, 

Acquisition, 

and 

Construction 

(PAC) 

Account: $600 

million 

• Accountability 

Oversight 

Board 

• Web- based 

executive 

dashboard 

• Tracking of 

selected 

higher risk 

programs 

Program 

managers 

present status 

updates on 

budget, risk 

management, 

project 

progress, and 

performance 

measurement to 

Oversight Board 

• Oversight 

Board meets 

weekly to 

review 

projects and 

intervenes 

as 

necessary 

• Action items 

follow each 

meeting 

• Recovery 

Act program 

manager 

monitors 

follow-up 

• Board has 

elevated 

higher risk 

projects for 

heightened 

scrutiny 

• Board has 

actively 

coordinated 

with 

external 

stakeholders and 

mitigated 

cascading 

effects of 

Close 

tracking to be 

extended to 

all Recovery 

Act projects 

Table 1: Mechanisms Established by NIST and NOAA to Monitor 

Program Progress and Manage Risk 

Source: OIG analysis based on bureau data 

early delay 

4
 
U.S. Department of Commerce Final Report ARR-19881 

Office of Inspector General May 21, 2010 

risk management) and project management capacities are improving. Before the Recovery Act, 

NOAA managed construction and repair of facilities and vessels in accordance with existing 

OMB requirements and current business practices in those industries, and used project delivery 

schedules and risk assessments. Nonfinancial risk management, however, was less mature at 

NOAA than its financial risk management capacity. While NIST’s financial risk management 

expertise was reportedly well developed, several NIST officials indicated that its nonfinancial 

risk management capabilities were less developed when the Recovery Act became law. 

Likewise, NIST officials acknowledged that it traditionally did not have strong project 

management experience, except for some recent improvements in the facilities construction and 

repair areas. 

NIST and NOAA officials offered a few positive examples of schedule-delay risks mitigated or 

cost overruns averted due to early detection and subsequent follow-up. While the examples 

provided have had minor impact so far, they do demonstrate that increased attention to schedule 

and risk can help to avoid unnecessary delays and cost overruns. At this point, however, with 

such a small percentage of Recovery Act funds expended from the NIST STRS and NOAA PAC 

accounts, it may be too soon to see more tangible benefits of this enhanced oversight until more 

funds are in the hands of contractors and grantees. 

II. Most NOAA PAC and NIST STRS Performance Metrics Do Not Measure Outcomes 

As part of its Recovery Act performance plans, OMB requires each agency to measure “expected 

quantifiable outcomes consistent with the intent and requirements of the legislation and risk 

management. . .”7

 OMB further states, “[E]ach outcome [must be] supported by a corresponding 

quantifiable output(s)—agencies must specify the length of the period between measurements 

(e.g., monthly, quarterly), measurement methodology, and how results will be made readily 

assessable to the public.”8 Agencies are required to make this information publicly available. The 

Recovery Act also requires recipients to report 

on the creation and retention of jobs as well as “Government operates more effectively   the completion status of projects and 

9 when it focuses on outcomes....” activities.

OMB 2011 Budget Submission Guidance  

In April 2009, NIST and NOAA submitted “Performance and Management,”  

their Recovery Act plans to Congress, which January 8, 2010

approved the plans—including the 

performance measures—the following month.10 A key metric required by the Recovery Act is to 

track job creation, which is considered an outcome measure. All NIST and NOAA Recovery Act 

programs that were assessed in this review comply with this Recovery Act requirement. 

                                                            

7

OMB Memorandum M‐09‐15 at 2.8(g). 8

OMB Memorandum M‐09‐15 at 2.8(g). 

9

Recovery Act, 1512(c)(3)(B)‐(D). This review does not address reporting or job creation or retention.
 10Some Recovery Act performance measures are the same as the Government Performance and Results Act of 1993
 

(GPRA) measures. In 2005, GAO wrote, “Seeking to promote improved federal management and the increased
 

efficiency and effectiveness of federal programs, GPRA instituted a government-wide requirement for agencies to
 

set goals and report annually on program performance,” and in enacting GPRA, “Congress expressed frustration 

that executive branch and congressional decision making was often hampered by the lack of good information on
 

the results of federal program efforts.” [See footnote 1.]
 

5
 
                                                           

       

       

   

     

         

         

   

   

      

       

     

     

       

       

       

   

     

   

   

     

U.S. Department of Commerce Final Report ARR-19881 

Office of Inspector General May 21, 2010 

Other than counting jobs, however, which is ultimately the main gauge of the Recovery Act’s 

success, most of the NIST and NOAA programs reviewed do not have outcome-oriented 

performance measures. Therefore, NIST and NOAA are not assessing whether program goals, 

specific program outcomes, and improved results are achieved. In fact, many of the Recovery 

Act measurements that were submitted by NIST and NOAA (and approved by OMB) are not 

performance measures as envisioned in the Recovery Act guidance or as defined by GAO. 

Rather, they are measures of the percentage of obligations (such as NIST STRS projects) or the 

completion of a schedule’s milestones (such as NOAA facility and vessel construction and repair 

projects). While both are crucial aspects to track in project and financial management, these 

metrics do not measure expected outcomes and do not 

satisfy the Recovery Act’s intent to achieve program goals Agencies will take actions

by meeting these outcomes. beyond standard practices to

adapt current performance

evaluation and review NOAA uses measures of a program’s progress, such as a 

processes to include the ability schedule’s milestones that have been achieved, as its 

to report on program and performance measures in five out of eight PAC programs. 

economic outcomes. For example, in the Facility Maintenance and Repair 

program, the performance measures focus on specific 

—OMB Memorandum M‐09‐15, outputs, such as “Percentage of Planned Milestones Met,” Grants, section 5.4 and these metrics are monitored by the program manager 

and NOAA’s Accountability Oversight Board as Agencies will give heightened

management performance measures. While measuring the progress of attention to

Recovery Act program schedule is vital, these milestone-related acquisition

planning performance metrics do not measure expected quantifiable to achieve mitigation

of schedule cost and outcomes. Next Generation Radar (NEXRAD) is the 

performance risk, and define notable exception among the NOAA PAC programs, with 

contract requirements that seven outcome-oriented performance measures focused on 

deliver meaningful and improving severe weather forecasting.11 For example, 

measurable outcomes. NEXRAD measures the lead time and accuracy of severe 

weather warnings for flash floods, tornados, and winter 

—OMB Memorandum M‐09‐15, storms. 

Contracts, section 6.1

Tables 2 and 3 show several ways that NIST and NOAA 

are tracking the progress of Recovery Act programs, monitoring Recovery Act spending and 

project milestones, and counting jobs. The bureaus’ Recovery Act performance metrics, 

however, are still not outcome oriented. For example, NOAA could develop a performance 

metric for its vessel construction or repair activities that would measure its ability to more 

effectively manage fisheries’ stocks. NIST could develop an outcome-oriented performance 

metric that would measure the advances made in a particular scientific field due to NIST’s 

purchases and use of highly technical scientific equipment. In such ways, NIST and NOAA 

would develop performance measures that are aligned with the purposes of the investments, as 

outlined in their spending plans. 

11These are the same as the program’s GPRA measures. As another example, NOAA’s GPRA Goal 3 measures 

climate outcomes, relevant to the Recovery Act’s High Performing Computers investment. 

6
 
 

U.S. Department of Commerce Final Report ARR-19881 

Office of Inspector General May 21, 2010 

Table 2: NIST Tracks Progress of Recovery Act Programs, 

but Most Performance Metrics Do Not Measure Outcomes 

Intended 

NIST Scientific and Technical 

Research Services (STRS) 

Account 

($ millions) 

Purpose

(Recovery

Act 

Spending

Plan) 

Recovery

Act Outcome 

Measures 

Established? 

Measures 

Jobs 

Created 

or 

Saved? 

Spending

Progress

and 

Milestones 

Tracked? 

Advanced Scientific 

Equipment 

$119 Strengthen 

programs that 

enable longNo Yes Yes 

term 

economic 

growth 

through 

innovation 

Measurement Science and 77 Advance No Yes Yes 

Engineering Grants and 

Research Fellowships 

NIST 

research 

through 

innovation; 

fellowship 

awardees will 

advance NIST 

research 

Other Remaining Programs, 

including Research and IT 

Infrastructure Contracts, 

54 Create tools 

to help 

vendors test 

No Yes Yes 

Standards for Health their health IT 

Information Technologya

, and 

SmartGrid Standardsb products; support 

development 

of 

interoperable 

Smart Grid; 

improve NIST 

IT 

infrastructure, 

etc. 

Total $250 

Source: OIG analysis of NIST data 

a

Transfer from Department of Health and Human Services 

b

Transfer from Department of Energy 

7
 
U.S. Department of Commerce Final Report ARR-19881 

Office of Inspector General May 21, 2010 

Table 3: NOAA Tracks Progress of Recovery Act Programs,

but Most Performance Metrics Do Not Measure Outcomes 

NOAA Procurement, Acquisition

and Construction (PAC) Account 

($ millions) 

Intended 

Purpose

(Recovery Act 

Spending

Plan) 

Recovery

Act Outcome 

Measures 

Established? 

Measures 

Jobs 

Created 

or 

Saved? 

Spending

Progress

and 

Milestones 

Tracked? 

Climate Computing and Modeling $170 Improve the 

accuracy of 

seasonal climate 

No Yes Yes 

and global 

climate change 

assessments 

Pacific Regional Center 142 Improve 

operations and 

mission 

No Yes Yes 

performance; 

operational 

savings; greater 

program 

collaboration 

within NOAA and 

externally 

Southwest Fisheries Science 102 Enable NOAA No Yes Yes 

Center Laboratory Replacement and partners to 

benefit from 

strategic and 

functional 

relationships with 

local research 

and educational 

organizations 

Vessel Construction 78 Improve the 

ability to more 

accurately 

manage fisheries 

stocks 

No Yes Yes 

Accelerate Satellite Development 74 Mitigate cost and 

schedule risk for 

No Yes Yes 

existing satellite 

program 

Other Remaining Facility 

Construction, Maintenance, and 

Repair 

27 Support repairs to 

specific facilities; 

comply with 

building codes; 

support satellite 

missions 

No Yes Yes 

NEXRAD Radar Systems & Dual 

Polarization 

7 Significant 

improvement in 

precipitation 

estimation and 

Yes Yes Yes 

distinction; 

general 

improvement in 

data quality 

 

Total $600 

Source: OIG analysis of NOAA data 

8 
     

           

           

           

         

             

               

             

           

           

             

               

             

               

               

         

     

         

             

       

         

   

  

                                                           

 

U.S. Department of Commerce Final Report ARR-19881 

Office of Inspector General May 21, 2010 

Research Investments:

Outcomes and Efficiency

In 2008, the Environmental Protection Agency

sponsored a National Research Council study

due to the Environmental Protection Agency’s

“difficulties in answering [OMB's] questions

about efficiency and OMB's insistence that the

agency find ways to measure the efficiency of

its research based on outcomes, rather than

outputs . . .” [emphasis added].

NRC found that measuring research efficiency

“based on outcomes ... is neither achievable

nor valid, because such outcomes occur far in

the future and are highly dependent upon

actions taken by many other people who may

or may not use the research findings.” The

report does, however, recommend evaluating

efficiency based on "intermediate

outcomes"— assessing, for example, whether

a program has improved the body of

knowledge for decision‐making, or

disseminated newly developed tools and

models.

Source:

www8.nationalacademies.org/onpinews/newsi

tem.aspx?RecordID=12150

NIST and NOAA officials point out that in general, outcome-oriented performance measurement 

is challenging, especially with research activities and projects such as office construction. A 

senior NOAA official noted that the bureau 

concentrates on monitoring construction 

schedules, stating that “[NOAA] focuses 

appropriately on schedule and risk,” given the 

short time frame in which to spend the 

Recovery Act funds. One official contended 

that the OMB guidance related to measuring 

outcomes for this type of activity in 2 years 

was “un-executable” and that delivering 

Recovery Act projects on time and under cost 

should be the priority in the short term. 

NIST officials explained the difficulty in 

developing outcome-oriented metrics for 

research organizations, and cited a 2008 

National Research Council (NRC) report that 

found that measuring research efficiency based 

on “ultimate outcomes” is neither achievable 

nor valid. The report, however, endorsed 

evaluating efficiency based on “intermediate 

outcomes” (see box). 

NIST worked to improve its GPRA 

performance measures related to several 

Recovery Act programs for NIST’s 2011 

Congressional Budget Submission, which is a 

positive step.12 Considerable additional work 

remains, however, for NIST (and NOAA) to 

develop outcome-oriented measures for 

Recovery Act investments so that they are 

more useful to all stakeholders, not only to 

internal ones such as project managers and senior accountability officials. 

We agree that program efficiency is an important outcome measure, but note the bureaus must 

equally consider and measure program effectiveness.

13 Other than measuring jobs, NIST and 

NOAA do not evaluate the public benefits of these funds. For example, if NOAA contracts to 

12NIST updated its performance metrics related to the following programs: Measurement Science and Engineering 

Grants, Post-doctoral Research Fellowships, and Measurement Science and Engineering Fellowship Program. For 

example, NIST fellowship programs measure outputs of “knowledge transfer” such as the number of reports or 

the number of times a report is referenced by another researcher. Compared with measuring obligation percentage 

rates, measuring such outputs is relevant, and is important progress toward measuring outcomes. 13 NRC conducts a yearly assessment focusing on quality, relevance, and technical merit of the NIST Laboratories 

Program. As a result of its efficiency evaluation, this year NRC will conduct an assessment of four NIST 

laboratories, last assessed in 2008. NRC assessment reports are available at www.nist.gov/director/nrc. 

9
 
U.S. Department of Commerce Final Report ARR-19881 

Office of Inspector General May 21, 2010 

build a research facility or to construct a research vessel or if NIST provides for research 

fellowships or purchases highly technical scientific equipment with Recovery Act funds, the 

scientific improvements or public benefits of these tax dollar investments should be identified. If 

NOAA and NIST do not do so, the bureaus cannot accurately measure the effectiveness of 

Recovery Act programs. 

Recommendation 

NOAA and NIST should improve their performance metrics for the more significant Recovery 

Act programs, focusing on intermediate outcomes that assess the programs’ benefits. For 

example, performance metrics should track whether an investment has improved the body of 

knowledge in a particular field, disseminated newly developed tools and models, supported a 

research or technological innovation, or made other advancements in science and technology for 

the public’s benefit. 

Summary of Bureaus’ Comments 

NOAA’s April 22 response to our draft report says that NOAA agrees with the report and has 

already begun to modify its Recovery Act performance metrics to address the report’s 

recommendation. NIST’s May 14 response reemphasized the challenges we discussed in the 

report regarding measuring core science programs. However, NIST agreed that its performance 

metrics for the STRS equipment program should be strengthened, and stated that it would 

develop an appropriate set of intermediate measures. 

10