HEALTH CARE 

QUALITY 

MEASUREMENT

HHS Should Address 

Contractor 

Performance and Plan 

for Needed Measures

Report to Congressional Committees

January 2012

GAO-12-136

United States Government Accountability Office

GAO
United States Government Accountability Office

Highlights of GAO-12-136, a report to 

congressional committees

January 2012

HEALTH CARE QUALITY MEASUREMENT

HHS Should Address Contractor Performance and 

Plan for Needed Measures

Why GAO Did This Study

The Medicare Improvements for 

Patients and Providers Act of 2008 

(MIPPA) directed the Department of 

Health and Human Services (HHS) to 

enter into a 4-year contract with an 

entity to perform various activities 

related to health care quality 

measurement. In January 2009, HHS 

awarded a contract to the National 

Quality Forum (NQF), a nonprofit 

organization that endorses health care 

quality measures—that is, recognizes 

certain ones as national standards. In 

2010, the Patient Protection and 

Affordable Care Act (PPACA) 

established additional duties for NQF. 

This is the second of two reports 

MIPPA required GAO to submit on 

NQF’s contract with HHS. In this 

report—which covers NQF’s 

performance under the contract from 

January 14, 2010, through August 31, 

2011—GAO examines (1) the status of 

projects under NQF’s required contract 

activities and (2) the extent to which 

HHS used or planned to use the 

measures it has received from NQF 

under the contract to meet its quality 

measurement needs, as of August 

2011. GAO interviewed NQF and HHS 

officials, reviewed relevant laws, and 

reviewed HHS and NQF documents.

What GAO Recommends

GAO recommends HHS: (1) use all 

monitoring tools required under the 

contract to help address NQF’s 

performance, (2) complete testing of 

retooled measures, and 

(3) comprehensively plan for its quality 

measurement needs. HHS neither 

agreed nor disagreed with these 

recommendations. NQF concurred with 

many of the findings in the report and 

provided additional context.

What GAO Found

NQF has made progress on projects under its contract activities, as of August 

2011. Specifically, NQF has completed or made progress on 60 of 63 projects. 

For example, NQF has completed projects to endorse measures related to 

various topics, including nursing homes. However, for more than half of the 

projects, NQF did not meet or did not expect to meet the initial time frames 

approved by HHS. For example, NQF completed one project to retool 

measures—that is, convert previously endorsed quality measures to an 

electronic format. While the retooling project was expected to be completed by 

September 2010, its completion was delayed by 3 months. NQF and HHS 

officials identified various reasons that contributed to this delay, including an 

expansion of the project’s scope and complexity. As a result of the delay, HHS 

did not have all the retooled measures it expected to include in its Electronic 

Health Records (EHR) Incentive Program. The delay of this project was also a 

contributing factor to NQF exceeding its estimated cost for its entire contract 

activity related to EHR by about $560,000 in the second contract year— 

January 14, 2010, through January 13, 2011. While HHS monitored NQF’s 

progress through monthly progress reports and approved changes to time frames 

and costs, HHS did not use all of the tools for monitoring that are required under 

the contract. Specifically, HHS did not conduct an annual performance evaluation 

to assess timeliness and cost issues that could have helped to inform NQF’s 

future scope of work. Until August 2011, HHS did not enforce the provision for 

NQF to submit a financial graph to compare monthly costs for each contract 

activity with cost estimates, which is information not included in monthly progress 

reports. These tools could have provided additional, more detailed information to 

help identify instances in which NQF might have been at risk of not meeting time 

frames or exceeding cost estimates, which could have provided HHS an 

opportunity to make any appropriate changes to NQF’s activities.

HHS had used or planned to use about half of the measures—164 of 344—that it 

received from NQF under the contract, as of August 2011. For example, HHS 

used 44 measures that NQF retooled under the contract in its EHR Incentive 

Program. HHS officials stated that the 44 measures used in the program 

contained errors, which required corrections. HHS officials also have not yet 

tested the retooled measures to assess the feasibility of implementing them in 

the electronic format; therefore, HHS runs the risk that some of these measures 

may not work as intended when implemented. HHS officials told GAO they 

expect to evaluate if and how they could use all of the remaining measures HHS 

received under the contract. However, HHS has not determined how PPACA 

requirements for quality measurement may have changed its needs for endorsed 

quality measures. As a result, HHS has not established a comprehensive plan 

that identifies its measurement needs and time frames for obtaining endorsed 

measures and that accounts for relevant PPACA requirements. Without such a 

plan, HHS may be limited in its efforts to prioritize which specific measures it 

needs to develop and to have endorsed by NQF during the remainder of the NQF 

contract. As a result, HHS may be unable to ensure that the agency receives the 

quality measures needed to meet PPACA requirements, including time frames 

for implementing quality measurement programs.

View GAO-12-136. For more information, 

contact Linda T. Kohn at (202) 512-7114 or

kohnl@gao.gov.
Page i GAO-12-136 NQF Contract Performance

Letter 1 

Background 7

NQF Made Progress on Projects under Each of Its Contract 

Activities but Has Not Met Expected Time Frames and Has 

Exceeded Cost Estimates 13

NQF Reported Costs and Fixed Fees of over $20 Million from 

January 2010 through August 2011 27

HHS Used or Planned to Use Some NQF Measures Received under 

the Contract but Has Not Comprehensively Planned for Other 

Measurement Needs to Implement PPACA 30

Conclusions 37

Recommendations for Executive Action 39

Agency and Other External Comments and Our Evaluation 40

Appendix I Health Care Quality Measurement Framework 45

HHS’s Role in Planning and Identifying Gap Areas 47

Appendix II How GAO Categorized NQF’s 16 Tasks under the Contract into 

Nine Contract Activities 49

Appendix III Status of NQF’s Nine Contract Activities and the Projects under 

Each Activity GAO Identified 51

Appendix IV Summary of Projects to Review the National Quality Forum’s 

Endorsement Process 62

Appendix V Comments from the Department of Health and Human Services 64

Appendix VI Comments from the National Quality Forum 69

Contents
Page ii GAO-12-136 NQF Contract Performance

Appendix VII GAO Contact and Staff Acknowledgments 74

Tables

Table 1: Description of the Nine NQF Contract Activities 

Categorized by GAO 9

Table 2: Status of Measures Used or Planned for Use under the 

NQF Contract, as of August 31, 2011 33

Table 3: GAO’s Categorization of the NQF Contract’s 16 Tasks into 

Nine Activities 50

Table 4: Status of Projects under NQF’s Contract Activity to 

Provide Recommendations on a National Strategy and 

Priorities for Quality Measurement, as of August 31, 2011 52

Table 5: Status of Projects under NQF’s Contract Activity to 

Endorse Health Care Quality Measures, as of August 31, 

2011 53

Table 6: Status of Projects under NQF’s Contract Activity to 

Maintain Previously Endorsed Health Care Quality 

Measures, as of August 31, 2011 54

Table 7: Status of Projects under NQF’s Contract Activity to 

Promote the Development and Use of Electronic Health 

Records, as of August 31, 2011 56

Table 8: Status of Projects under NQF’s Contract Activity to Submit 

an Annual Report to Congress and the Secretary of HHS, 

as of August 31, 2011 57

Table 9: Status of Projects under NQF’s Contract Activity to 

Provide Multistakeholder Input into HHS’s National 

Strategy for Quality Improvement in Healthcare, as of 

August 31, 2011 58

Table 10: Status of Projects under NQF’s Contract Activity to 

Provide Multistakeholder Input on Selection of Quality 

Measures, as of August 31, 2011 59

Table 11: Status of Projects under NQF’s Other Health Care Quality 

Measurement Activity Not Identified in MIPPA or PPACA, 

as of August 31, 2011 60

Table 12: Status of Projects under NQF’s Administrative Contract 

Activity 61
Page iii GAO-12-136 NQF Contract Performance

Figures

Figure 1: Estimated Time Frames and Completion Dates for All 

Five Projects under NQF’s Endorsement Contract 

Activity, as of August 31, 2011 20

Figure 2: Estimated Time Frames for Eight Projects Related to 

NQF’s Contract Activity on the Promotion and Use of 

EHRs, as of August 31, 2011 22

Figure 3: NQF’s Reported Contract Activity Costs and Fixed Fees 

Totaling Approximately $12.8 Million during the Second 

Contract Year (Jan. 14, 2010—Jan. 13, 2011) 28

Figure 4: NQF’s Reported Contract Activity Costs and Fixed Fees 

Totaling Approximately $9.6 Million during the Period of 

the Third Contract Year Covered in Our Review (Jan. 14, 

2011—Aug. 31, 2011) 29

Figure 5: Health Care Quality Measurement Framework 46
Page iv GAO-12-136 NQF Contract Performance

Abbreviations

ASPE Assistant Secretary for Planning and

 Evaluation

CHIP Children’s Health Insurance Program

CHIPRA Children’s Health Insurance Program 

 Reauthorization Act of 2009

CDP consensus development process

CMS Centers for Medicare & Medicaid Services

EHR Electronic Health Record

ESRD end-stage renal disease

FAR Federal Acquisition Regulation

HAC Healthcare Acquired Condition

HHS Department of Health and Human Services

IT Information Technology

IEP Internal Evaluation Plan

LTCH Long-Term Care Hospitals

MAP Measure Applications Partnership

MAT Measure Authoring Tool

Mathematica Mathematica Policy Research, Inc.

MIPPA Medicare Improvements for Patients and

 Providers Act of 2008 

NPP National Priorities Partnership

NQF National Quality Forum

National Quality Strategy National Strategy for Quality Improvement

 in Healthcare 

NTTAA National Technology Transfer and 

 Advancement Act of 1995

OMB Office of Management and Budget

PPACA Patient Protection and Affordable Care Act

PPS Prospective Payment System

QDM Quality Data Model

SRE Serious Reportable Event

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-12-136 NQF Contract Performance

United States Government Accountability Office

Washington, DC 20548

January 13, 2012

Congressional Committees

The nation’s health care spending was about $2.5 trillion in 2009. Health 

care spending has been increasing in recent years by an average of 

nearly 7 percent per year since 2000, and is projected to reach over 

$4.5 trillion by 2020.1 However, higher levels of spending, such as 

spending more to increase the number or technical complexity of 

treatments, do not necessarily lead to a corresponding increase in the 

quality of care.2 To better align quality and health care spending, the 

Department of Health and Human Services (HHS) has developed various 

programs and initiatives to measure health care quality and to provide 

financial incentives to health care providers to improve quality and to 

reduce the use of unnecessary and costly services. These programs and 

initiatives include, for example, pay-for-performance programs that rely on 

information collected on various health care quality measures.3

 

1

Office of the Actuary, Centers for Medicare & Medicaid Services (CMS), National Health 

Expenditures Tables, table 1, downloaded 09/2/11 from 

https://www.cms.gov/NationalHealthExpendData/downloads/tables.pdf. In 2009, the 

national spending on health care accounted for 17.6 percent of gross domestic product, 

which totaled $14.1 trillion. The federal government share of health care spending for that 

same year was $678 billion, or 27 percent.

Health 

care quality measures are used to evaluate how health care is delivered, 

and information obtained from such measures can promote accountability 

among health care providers. The Patient Protection and Affordable Care 

Act (PPACA), enacted in March 2010, directed HHS to expand its current 

programs and initiatives, such as pay-for-performance programs, and to 

implement new ones that will use health care quality measures. PPACA 

establishes specific time frames for HHS to implement these programs 

and initiatives that rely on various health care quality measures over the 

next several years. For example, PPACA requires HHS to establish a 

2

GAO, Value in Health Care: Key Information for Policymakers to Assess Efforts to 

Improve Quality While Reducing Costs, GAO-11-445 (Washington, D.C.: July 26, 2011). 

3

The Physician Quality Reporting System, for example, is an HHS quality reporting 

program that provides an incentive payment to eligible professionals, such as physicians, 

who satisfactorily report data on various quality measures for services that are covered by 

Medicare. 

 
 

Page 2 GAO-12-136 NQF Contract Performance

hospital value-based purchasing program.4 HHS estimates that in fiscal 

year 2013 its Hospital Value-Based Purchasing Program5 will provide 

$850 million in Medicare incentive payments to hospitals that meet certain 

performance measures.6

For many of HHS’s health care quality measurement programs and 

initiatives, HHS is required to give consideration to the quality measures 

endorsed by a consensus-based entity, such as the National Quality 

Forum (NQF). NQF is the entity in the United States with the lead 

responsibility for endorsing health care quality measures—that is, 

determining which measures should be recognized as national standards. 

The Medicare Improvements for Patients and Providers Act of 2008 

(MIPPA) directed the agency to enter into a 4-year contract with an entity 

to endorse health care quality measures and conduct other activities, and 

HHS contracted with NQF in January 2009.

Additionally, HHS is required under PPACA to 

establish value-based payments for physicians, develop a plan for valuebased purchasing programs for skilled nursing home facilities and home 

health agencies, and create pilot programs that test the implementation of 

a value-based purchasing program for other providers—psychiatric, longterm care, inpatient rehabilitation, and cancer hospitals as well as hospice 

programs.

7

 

4

Value-based purchasing programs are pay-for-performance programs that require 

providers to collect and report information on health care quality measures and adjust 

payment levels based on providers’ performance against the measures. 

NQF is a nonprofit 

organization established in 1999 in order to foster agreement on national 

standards for measurement and public reporting of health care 

performance data. MIPPA provided $10 million per year for each of fiscal 

years 2009 through 2012—$40 million in total—from the Medicare Trust 

Funds for this 4-year contract, which covers the period from January 14, 

2009, through January 13, 2013. PPACA provided an additional 

5

For payment purposes, CMS divides hospital providers into various categories, including 

acute care inpatient hospitals, long-term care hospitals, and psychiatric hospitals, among 

others. The Hospital Value-Based Purchasing Program applies to acute care inpatient 

hospitals. 

6

See 76 Fed. Reg. 2454 (Jan. 13, 2011). PPACA specified that the total amount of valuebased payments must be equal to the total amount of reductions to payments for all 

hospitals, so the program has a net budget-neutral impact. Also see Pub. L. No. 111-148, 

§ 3001, 124 Stat. 119, 353-62 (2010).

7

Pub. L. No. 110-275, § 183, 122 Stat. 2494, 2583-86. The contract may be renewed at 

the end of the 4-year period after a subsequent bidding process. 
 

Page 3 GAO-12-136 NQF Contract Performance

$100 million—$20 million per year for each of fiscal years 2010 through 

2014—for additional health care quality measurement work to be 

conducted by NQF and HHS through fiscal year 2014.8

MIPPA also directed us to submit two reports on the performance of and 

costs incurred by NQF under its 4-year contract with HHS.

Because PPACA 

does not specify the proportion of this amount for NQF activities, HHS 

officials told us that they plan to obligate approximately $10 million per 

year for the NQF contract, beginning in 2011, and that they may extend 

the contract through fiscal year 2014.

9 On July 14, 

2010, we issued the first report, which contained information about NQF’s 

performance and costs during the first contract year, which began 

January 14, 2009, and ended January 13, 2010.10

To describe the status of projects under NQF’s required contract activities 

from January 2010 through August 2011, we reviewed relevant provisions 

in MIPPA and PPACA and documents including the HHS contract with 

NQF, NQF’s 2009, 2010, and 2011 final annual work plans that cover the 

This second report 

covers NQF’s performance and costs for the second contract year—

January 14, 2010, through January 13, 2011—and part of the third 

contract year—January 14, 2011, through August 31, 2011. Specifically, 

this report examines (1) the status of projects under the activities NQF is 

required to complete under its contract with HHS, (2) the costs and fixed 

fees NQF reported under the contract from January 2010 through August 

2011, and (3) the extent to which HHS used or planned to use the 

measures it receives from NQF under the contract to meet its quality 

measurement needs, as of August 31, 2011. In addition, appendix I of this 

report provides information about a framework for the various stages of a 

health care quality measure, including measure endorsement, as 

described by HHS and NQF officials and others.

 

8

Pub. L. No. 111-148, § 3014(c), 124 Stat. 119, 387 (2010).

9

MIPPA states that GAO’s reports shall be submitted no later than 18 months and 

36 months, respectively, after the effective date of the contract.

10GAO, Health Care Quality Measurement: The National Quality Forum Has Begun a 

4-Year Contract with HHS, GAO-10-737 (Washington, D.C.: July 14, 2010). In this report, 

we found that (1) NQF had begun work for each of the five duties required by MIPPA 

related to quality measures, (2) NQF reported costs and fixed fees totaling approximately 

$6.5 million for the first contract year, and (3) NQF and HHS rely on reviews of NQF 

invoices in order to help ensure that NQF’s reported costs are proper.
 

Page 4 GAO-12-136 NQF Contract Performance

contract activities in response to MIPPA,11 NQF’s 2011 technical proposal 

for the contract activities in response to PPACA,12 and NQF’s monthly 

progress and monthly internal evaluation plan reports that NQF is 

required to submit to HHS.13

To assess the costs and fixed fees NQF reported under the contract from 

January 2010 through August 2011, we reviewed NQF documents that 

For purposes of our work, we categorized 

the 16 tasks that NQF is required to perform under the contract from 

January 2010 through August 2011 into 9 contract activities. (See app. II 

for more detailed information on how we categorized these contract 

activities.) In addition, we identified the various projects NQF is required 

to perform under the 9 contract activities we identified. Specifically, for the 

purposes of our work, we identified 63 projects NQF is required to 

perform under the 9 contract activities, as shown in appendix III. We then 

compared the information we obtained from NQF on its progress on the 

63 projects under the 9 activities to the requirements for NQF’s 

performance under the contract with HHS and the expected time frames 

and cost estimates initially established by HHS and NQF in the final 

annual MIPPA work plans and other NQF documents for completing the 

work. We also interviewed NQF officials responsible for implementing the 

contract and HHS officials responsible for managing the contract and 

overseeing NQF’s performance. We also compared evidence from HHS 

and NQF documents with evidence from discussions with HHS and NQF 

officials to corroborate any issues or challenges with NQF’s performance. 

Based on our comparison and interviews with officials, we identified any 

changes to the work NQF is required to perform and changes to the time 

frames or cost estimates established in the final work plans developed 

annually to respond to MIPPA and a budget developed as part of NQF’s 

technical proposal to respond to PPACA.

 

11Under the contract, NQF is required to work with HHS to develop an annual work plan 

for each contract year that outlines the contract activities in response to MIPPA. The work 

plan also delineates scheduled time frames for the work NQF is expected to perform that 

contract year and cost estimates for the activities. NQF officials told us HHS obligates 

funds on a yearly basis following HHS’s approval of the work plan. 

12The technical proposal is submitted to HHS and incorporated into the scope of work. 

13The contract requires that monthly progress reports include NQF’s accomplishments for 

the month under each of the contract activities, any problems encountered or anticipated 

and the impact on project time frames, and a financial graph that compares NQF’s 

monthly reported costs for each of the contract activities with initial cost estimates, among 

other things.
 

Page 5 GAO-12-136 NQF Contract Performance

include financial information related to the contract, such as NQF’s final 

work plans developed in 2010 and 2011 to respond to MIPPA and a 

budget developed as part of NQF’s technical proposal to respond to 

PPACA that describe NQF’s estimated costs for each of the nine activities 

we identified under the contract. We obtained and summarized 

information from NQF monthly invoices submitted to HHS from January 

14, 2010, through August 31, 2011, for direct14 and indirect15 costs 

incurred by NQF under the contract and fixed fees.16 We compared 

NQF’s reported costs with its cost estimates for the activities in NQF’s 

2010 final annual work plan and determined whether or to what extent 

NQF’s reported costs exceeded its cost estimates. We did not assess the 

accuracy of NQF’s reported costs.17

To examine the extent to which HHS used or planned to use measures it 

receives from NQF under the contract to meet its quality measurement 

needs, as of August 2011, we reviewed statutory requirements and 

documents related to HHS health care quality measurement efforts. We 

relied on HHS to identify programs and initiatives that use or will use 

health care quality measures, such as programs or initiatives established 

We also reviewed the HHS contract 

with NQF and any contract modification since January 14, 2010. We also 

interviewed NQF and HHS officials responsible for reviewing and 

approving the costs and fixed fees submitted under the contract. Based 

on our review of documents and interviews with NQF and HHS officials, 

we determined that the reported costs and fixed-fee data were sufficiently 

reliable for the purposes of this report.

 

14Direct costs are costs incurred specifically for the contract, such as labor costs and 

payments to subcontractors and consultants.

15Indirect costs are costs that are not directly attributable to a specific project or function. 

These costs cover additional items such as employee benefits, overhead, and 

administrative costs.

16NQF’s contract requires that, in accordance with 48 C.F.R. § 52.216-8, the fixed fee be 

paid monthly until fee payments reach 85 percent of the total amount of the fixed fee 

authorized, and that after they reach 85 percent HHS may withhold a reserve up to 

15 percent or $100,000, whichever is less, to protect the government’s interest. 

17NQF and HHS rely on reviews of NQF invoices in order to help ensure that NQF’s 

reported costs are proper. NQF officials told us that they review the invoices prior to 

submitting them to HHS and carry out other activities, such as using an electronic system 

to track labor hours, in order to help ensure that the costs they report in the invoices are 

proper. HHS also conducts reviews of NQF invoices, which are governed by HHS policies 

and procedures and by requirements applicable to federal contracts generally. For more 

information, see GAO-10-737. 
 

Page 6 GAO-12-136 NQF Contract Performance

by MIPPA, PPACA, and the Health Information Technology for Economic 

and Clinical Health Act.18 As a result, the health care quality programs 

and initiatives mentioned in our report may not represent a 

comprehensive list of all those that use or plan to use measures received 

from NQF under the contract. We interviewed officials from various 

divisions within HHS that may use measures that were nationally 

recognized—or NQF-endorsed—in their various programs and initiatives 

related to health care quality measurement in order to corroborate the 

information we obtained from documentation. We obtained information 

about how HHS selects measures for its programs and initiatives from our 

interviews with HHS officials and reviews of HHS documents. We 

determined how HHS has used and planned to use measures that were 

received under the contract with NQF by identifying all the measures that 

HHS received under the contract and matching each measure to the 

quality measures being used or considered for use in HHS programs and 

initiatives.19

 

18Pub. L. No. 111-5, div. A, title XIII, div. B, title IV, 123 Stat. 115, 226, 467 (2009). 

For the purpose of this report, we defined a measure as used 

if the measure was identified by HHS and included in an HHS program or 

initiative as of August 31, 2011, and indicated as such by a final rule or 

other HHS documentation. We defined a measure as planned for use if 

the measure was identified by HHS for potential use in an HHS program 

or initiative as of August 31, 2011, and indicated as such by a proposed 

rule or other HHS documentation. In addition, we reviewed HHS 

documents and interviewed HHS officials to determine whether or to what 

extent HHS has plans for using work conducted under the NQF contract 

to meet PPACA requirements related to quality measures for HHS 

programs and initiatives. For example, we reviewed HHS plans or 

strategies such as the 2011 National Strategy for Quality Improvement in 

19HHS officials explained that any measure developer can submit a measure to be 

considered for NQF endorsement. Each measure developer has its own purpose or plans 

for implementing a measure, which may or may not involve HHS quality and public 

reporting programs. Therefore, all measures submitted may not be applicable to a 

particular HHS health care quality program or initiative. HHS officials told us that they will 

review all the measures received under the contract to determine if they are applicable to 

their health care quality programs or initiatives. 
 

Page 7 GAO-12-136 NQF Contract Performance

Healthcare (National Quality Strategy),20

We conducted this performance audit from April 2011 through January 

2012 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objective. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives.

that describe the agency’s plans

to help meet its quality measurement needs.

NQF is a nonprofit organization established in 1999 in order to foster 

agreement, or consensus, on national standards for measuring and public 

reporting of health care performance data.21

 

20The 2011 National Strategy for Quality Improvement in Healthcare, also referred to as 

the National Quality Strategy, focused on six priorities: (1) making care safer; (2) ensuring 

person- and family-centered care; (3) promoting effective communication and coordination 

of care; (4) promoting the most effective prevention and treatment of the leading causes of 

mortality, starting with cardiovascular disease; (5) working with communities to promote 

wide use of best practices to enable healthy living; and (6) making quality care more 

affordable. The National Quality Strategy was due on January 1, 2011, as required by 

PPACA, and was issued on March 21, 2011. HHS is expected to provide annual updates 

to the report.

Its membership includes 

more than 400 organizations that represent multiple sectors of the health 

care system, including providers, consumers, and researchers. NQF’s 

mission focuses on three core areas: (1) building consensus on national 

priorities and goals for performance improvement and working in 

partnership to achieve them, (2) endorsing national consensus standards 

for measuring and publicly reporting on performance, and (3) promoting 

the attainment of national goals through education and outreach 

programs.

21NQF is recognized as a consensus body under the National Technology Transfer and 

Advancement Act of 1995 (NTTAA) and the Office of Management and Budget (OMB) 

Circular A-119, which provides NQF the authority needed to facilitate its public/private 

partnership. OMB Circular A-119 defines “consensus” as a “general agreement, not 

necessarily unanimity” so that attempts may be made to resolve objections and members 

may have an opportunity to amend their votes after reviewing comments. In addition, the 

circular states that the voluntary consensus process must adhere to the following five 

elements: openness, balance, due process, consensus, and an appeal mechanism. See 

Pub. L. No. 104-113, 110 Stat. 775 (1996).

Background 
 

Page 8 GAO-12-136 NQF Contract Performance

Prior to its contract with HHS, NQF established a consensus development 

process (CDP) to evaluate available health care quality measures to 

determine which ones are qualified to be endorsed—that is, recognized—

as national standards. Under this process, organizations that develop 

quality measures submit them to NQF for consideration, in response to 

specific solicitations by NQF.22 NQF forms a committee of experts from its 

member organizations as well as other organizations and agencies to 

conduct an objective and transparent review of these quality measures

against four standardized criteria established by NQF, such as whether 

the measures are scientifically acceptable.23

As of October 2011, NQF has endorsed over 600 health care quality 

measures in 27 areas, such as cancer and diabetes. HHS uses NQFendorsed measures in its programs and initiatives to promote quality 

measurement, and NQF continues to endorse quality measures separate 

from its contract with HHS.

After this committee 

evaluates the measures against these criteria, NQF’s process allows for a 

period during which its member organizations and the public may 

comment on the committee’s recommendation for each measure. The 

process also provides for a period for its member organizations to vote on 

whether the measures should be endorsed by NQF as a national 

standard. Ultimately, NQF’s board of directors makes a final decision on 

whether NQF should formally endorse the measures.

NQF’s work under the contract includes endorsement of quality measures 

and other activities that are expected to support HHS’s quality 

measurement efforts, such as through value-based purchasing programs. 

Specifically, NQF’s work under the contract consists of various projects 

under the nine contract activities related to health care quality 

measurement. The work plans developed annually to respond to MIPPA 

and NQF’s technical proposal to respond to PPACA delineate the projects 

 

22NQF does not develop quality measures itself; rather, NQF endorses quality measures 

developed by other organizations, such as the Joint Commission, the National Committee 

for Quality Assurance, and the American Medical Association.

23A review for scientific acceptability includes an assessment of data from measure testing 

conducted by measure developers to help ensure that the measure produces consistent 

and credible results about the quality of care when implemented. The remaining three 

criteria NQF established for its CDP process are (1) importance to measure and report, 

(2) usability, and (3) feasibility.

NQF’s Contract with HHS
 

Page 9 GAO-12-136 NQF Contract Performance

NQF is required to conduct under the nine contract activities, as well as 

expected time frames and cost estimates for the projects for each year. 

Table 1 provides more detailed information on the nine contract activities. 

Some of these activities are required by either MIPPA or PPACA, while 

others are quality measurement activities established by HHS or 

administrative activities.

Table 1: Description of the Nine NQF Contract Activities Categorized by GAO 

NQF contract activity categorized by GAO Description of activity 

Contract activities required by MIPPA or PPACA

Recommendations on a National Strategy 

and Priorities for Quality Measurement 

(as required by MIPPA)

NQF will synthesize evidence and convene key stakeholders to make 

recommendations on a national strategy and priorities for health care performance 

measurement.

Endorsement of Health Care Quality Measures 

(as required by MIPPA) 

NQF will conduct endorsement of quality measures using its consensus 

development process to determine which ones should be endorsed as national 

standards. Specifically, it will provide for the endorsement of quality measures 

through projects that focus on priority areas identified by HHS, such as patient 

outcomes and patient safety. In addition to endorsing measures, this activity 

includes an evaluation of NQF’s endorsement process to identify ways to improve 

its efficiency and effectiveness.

Maintenance of Endorsed Quality Measures 

(as required by MIPPA)

NQF will use its process to ensure that endorsed measures are maintained—that is, 

updated or retired. This process is similar to NQF’s endorsement process, in that it 

involves a review of measures against established criteria, a period for public 

comment, and a final decision by NQF’s board of directors.a Under this activity, NQF 

will conduct maintenance projects using its 3-year review cycle that compares 

previously endorsed measures against new measures submitted for endorsement. 

In addition to the 3-year review cycles, NQF can also maintain measures through: 

(1) annual updates, which is a process where NQF annually reviews and receives 

information on a measure to determine whether the measure has undergone any 

changes; (2) ad hoc reviews where NQF reviews an endorsed measure at any time 

based on justifiable evidence to substantiate the review; and (3) time-limited 

endorsement reviews, which is a process where NQF will review time-limited 

endorsed measures—measures that have been endorsed for a limited period of 

time to allow for testing by measure developers.

Promotion of the Development and Use of

Electronic Health Records (as required by

MIPPA) 

NQF will undertake projects to promote the development and use of electronic 

health records for use in quality measurement, including to “retool” measures—that 

is, convert previously endorsed quality measures to an electronic format that is 

compatible with electronic health records. In addition, NQF will undertake projects 

related to developing tools to capture data for performance measurement, creating 

a framework that defines a standardized set of data that should be captured in 

patients’ electronic health records, and providing data to providers to support clinical 

decisions. These efforts had not been performed by NQF prior to the contract. 
 

Page 10 GAO-12-136 NQF Contract Performance

NQF contract activity categorized by GAO Description of activity 

Annual Report to Congress and the Secretary 

of Health and Human Services (as required 

by MIPPA and amended by PPACA)

NQF is required to submit an annual report to Congress and the Secretary of HHS 

that provides a summary of NQF’s progress on activities under the contract during 

the year.

PPACA established additional requirements that NQF must perform for this activity 

that include providing additional information in its annual report to Congress and the 

Secretary of HHS, beginning with its report due March 1, 2012. Specifically, NQF 

will include additional information in the report on gaps in endorsed and 

nonendorsed health care quality measures and a summary of activities conducted 

by multistakeholder groups during the third contract year (January 14, 2011 – 

January 13, 2012).

Multistakeholder Input into HHS’s National

Strategy for Quality Improvement in Health Care 

(as required by PPACA)

In 2010, NQF convened a multistakeholder group to provide input on national 

priorities for improvement in population health and in the delivery of health care 

services for consideration under HHS’s National Strategy for Quality Improvement in 

Healthcare, or the National Quality Strategy.b

In 2011, NQF will continue to engage a multistakeholder group to provide input on a 

plan for measuring and improving health and health care focusing on the six 

priorities identified in HHS’s 2011 National Quality Strategy.c In addition, NQF will 

conduct endorsement projects for cross-cutting measuresd that can support national 

implementation of these priorities.

Multistakeholder Input on the Selection of

Quality Measures for use in Payment Programs 

and Value-Based Purchasing Programs Under 

PPACA, Other Private/Public Payers, and Other 

Programs (as required by PPACA)

NQF will convene a multistakeholder group to provide input on the selection of 

measures for use in various public programs, including measures for use in quality 

reporting programs for physicians, postacute care programs under Medicare, 

Prospective Payment System (PPS) exempt cancer hospitals,e and hospice care. In 

addition, the group will provide input on measures that address quality issues 

identified for vulnerable populations, such as dual-eligible beneficiaries.

Other health care quality measurement activities not identified in MIPPA or PPACA

Other health care quality measurement activity NQF will work on additional health care quality measurement projects established 

by HHS to fill immediate areas of need, including two white papers related to 

resource use and efficiency; efficiency and resource use endorsement projects; and 

projects related to measure harmonization.f 

Administrative activities 

Administrative activity NQF will perform a number of administrative activities that focus on project planning 

and contract management: (1) conduct an opening meeting between HHS and 

NQF, (2) develop an annual work plan, (3) develop a monthly internal evaluation 

plan for quality assurance, (4) develop a monthly progress report, (5) conduct 

weekly conference calls, and (6) develop a public website.g

 

Source: GAO analysis of MIPPA, PPACA, and the NQF contract as well as the 2010 and 2011 NQF final annual MIPPA work plans and

2011 PPACA technical proposal.

a

Prior to its contract with HHS, NQF established a process for maintenance of measures. In May 

2010, NQF updated this process by requiring all previously endorsed measures to go through a 

comprehensive review every 3 years that will include a comparison against newly submitted 

measures submitted for endorsement. For example, NQF will review all previously endorsed and 

newly submitted measures related to cardiovascular health simultaneously.

b

NQF convened the National Priorities Partnership to complete this work.

c

The six priorities identified in the National Quality Strategy are: (1) making care safer; (2) ensuring 

person- and family-centered care; (3) promoting effective communication and coordination of care; 

(4) promoting the most effective prevention and treatment of the leading causes of mortality, starting 

with cardiovascular disease; (5) working with communities to promote wide use of best practices to 

enable healthy living; and (6) making quality care more affordable.

d

Cross-cutting measures are those that affect all or most patients, such as patient safety.
 

Page 11 GAO-12-136 NQF Contract Performance

e

PPS-exempt cancer hospitals are those exempt from the Prospective Payment System (PPS), which 

is a system of Medicare reimbursement in which providers receive a predetermined, fixed payment 

based on the service provided and its classification, such as the patient’s diagnosis at the time of 

hospital admission.

f

Harmonization is a process where related measures are standardized in a manner by which they can 

be used in multiple settings when relevant. NQF defines “related measures” as those that either 

(1) address the same concepts for measure focus (target process, condition, event, outcome) but a 

different target population; or (2) address different concepts for measure focus (target process, 

condition, event, outcome) but the same target population.

g

Development of a public website was part of the 2010 final annual MIPPA work plan but this work 

was canceled in 2011.

To help determine the activities and the projects under the nine contract 

activities that NQF is expected to perform during each contract year, HHS 

has established an interagency workgroup that comprises officials from 

multiple divisions within HHS, including the Agency for Healthcare 

Research and Quality, the Centers for Medicare & Medicaid Services 

(CMS), and the Office of the National Coordinator for Health Information 

Technology. The workgroup is responsible for prioritizing and selecting 

the activities and projects under each activity that NQF is expected to 

perform during each contract year.24 HHS officials told us that the 

representatives from these various HHS agencies provide input on the 

work NQF is expected to perform, including determining quality measures 

requested from NQF for their respective programs. The activities and 

projects selected by the interagency workgroup become part of NQF’s 

scope of work under the contract.25

For the NQF contract, HHS selected a cost-plus-fixed-fee contract, under 

which HHS reimburses NQF for actual costs incurred under the contract 

in addition to a fixed fee that is unrelated to costs. Cost-plus-fixed-fee 

contracts are used for efforts such as research, design, or study efforts 

where costs and technical uncertainties exist and it is desirable to retain 

Some of the projects under the 

contract activities that NQF is expected to perform during the year will be 

ongoing from the previous contract year while new work will be 

incorporated into the work plan as necessary.

 

24The office of the Assistant Secretary for Planning and Evaluation (ASPE)—a component 

of HHS—is responsible for facilitating the workgroup discussions, which occur at least 

biweekly. 

25The contract requires NQF to develop a work plan for each contract year that describes 

the MIPPA projects under the activities that NQF is required to perform. The MIPPA 

workplan includes expected time frames and cost estimates for projects under the contract 

activities. The work NQF is required to perform in response to PPACA is outlined in a 

technical proposal submitted to HHS and incorporated into the scope of work. 
 

Page 12 GAO-12-136 NQF Contract Performance

as much flexibility as possible in order to accommodate change.26

However, this type of contract provides only a minimum incentive to the 

contractor to control costs. As we reported in 2009, these contracts are 

suitable when the cost of work to be done is difficult to estimate and the 

level of effort required is unknown.27 This cost-plus-fixed-fee contract is 

NQF’s first cost-reimbursement contract. For cost-reimbursement 

contracts, the Federal Acquisition Regulation (FAR)28 requires 

appropriate government surveillance29 during performance to provide 

reasonable assurance that efficient methods and effective cost controls 

are used. Under the FAR, contracts are to contain a provision for agency 

approval of a contractor’s subcontracts.30

Two HHS components are principally responsible for administering the 

NQF contract: the office of the Assistant Secretary for Planning and 

Evaluation (ASPE) and CMS—an agency within HHS.

HHS’s contract with NQF 

contains this provision and also requires the approval of consultants 

engaged under the contract. The review and approval of NQF’s use of 

subcontractors and consultants require appropriate support 

documentation provided by NQF to HHS, including a description of the 

services, the proposed price, and a negotiation memo that reflects the 

principal elements of the price negotiations between NQF and the 

subcontractor or consultant. Under its contract with HHS, NQF has 

utilized 31 subcontractors and 16 consultants since January 14, 2010, to 

provide support to NQF on many of the contract activities and associated 

projects.

31

 

26See 48 C.F.R. §§ 16.301-2, 16.306 (2010).

Specifically, the 

27For more information on cost-plus-fixed-fee contracts, see GAO, Contract Management: 

Extent of Federal Spending under Cost-Reimbursement Contracts Unclear and Key 

Controls Not Always Used, GAO-09-921 (Washington, D.C.: Sept. 30, 2009).

2848 C.F.R. ch.1. The FAR establishes uniform policies for acquisition of supplies and 

services by executive agencies. Agency acquisition regulations may implement or 

supplement the FAR.

29Surveillance includes oversight of a contractor’s work to provide assurance that the 

contractor is providing timely and quality goods or services and to help mitigate any 

contract performance problems. 

3048 C.F.R. §§ 44.204(a)(1), 52.244-2 (2010). 

31Within CMS, the Office of Acquisition and Grants Management is responsible for 

administering the NQF contract.
 

Page 13 GAO-12-136 NQF Contract Performance

project officer for the NQF contract is a representative of ASPE.32 This 

individual is responsible for program management and works with the 

contracting officer to oversee the contract. The contracting officer for the 

NQF contract, responsible for administering the contract, is a 

representative of CMS.33 The contract outlines various activities that the 

contracting officer is required to perform, such as conducting an annual 

evaluation of the contractor’s performance.

From January 14, 2010, through August 31, 2011, NQF made progress 

on projects under its contract activities. However, our review of NQF 

documents found that NQF had not met or did not expect to meet time 

frames on more than half of the projects, and it exceeded its cost 

estimates for projects under three of the contract activities. HHS did not 

use all tools for monitoring that are required under the contract.

 

32The project officer serves as the technical representative of the contracting officer, and 

provides technical direction to NQF for all tasks described in the NQF contract. In addition, 

the project officer monitors NQF’s performance and reviews invoices for payment. Another 

HHS official within ASPE, the project manager, also provides support to the project officer 

for the NQF contract.

33The contracting officer enters into, administers, and terminates government contracts. 

The contracting officer negotiates and prepares contract documents, modifies terms or 

conditions of the contract, and approves payment of invoices, among other tasks.

NQF Made Progress 

on Projects under 

Each of Its Contract 

Activities but Has Not 

Met Expected Time 

Frames and Has 

Exceeded Cost 

Estimates
 

Page 14 GAO-12-136 NQF Contract Performance

From January 14, 2010, through August 31, 2011, NQF has made 

progress on 60 of the 63 projects under the activities required under its 

contract with HHS. Specifically, NQF had completed 26 projects and was 

continuing to work on the remaining 34 projects.34

• Endorsement of Measures Activity. NQF endorsed 101 measures 

since the beginning of the second contract year by conducting work 

on endorsement projects on different topic areas.

(App. III provides the 

status of all contract activities and the projects under each activity NQF 

was expected to perform during our reporting period.) Examples of 

projects under the contract activities include both completed and 

continuing projects:

35 Specifically, NQF 

completed two projects to endorse 38 outcome measures related to 

20 high-priority conditions identified by CMS that account for the 

majority of Medicare’s costs,36

 

34The contract activities and related projects that NQF is expected to perform during each 

year are determined by HHS and NQF, as they develop NQF’s scope of work. As a result, 

the number of projects varies by year, and some project time frames may extend beyond 

1 contract year. For our reporting period, we determined that NQF is conducting work on 

63 projects. As of August 31, 2011, of the 63 projects we identified, 26 have been 

completed, 34 are in progress, 2 have been canceled, and 1 has not yet started. See 

app. III for more details on each of these projects.

and mental health and child health 

conditions; and 21 performance measures for chronic and postacute 

care nursing facilities. NQF also worked on two projects related to 

child health quality and patient safety. As of August 2011, NQF 

endorsed 41 child health quality measures and 1 patient safety 

measure under these projects. NQF expected to complete the child 

health quality project in September 2011 and the patient safety project 

35HHS told us that they consider an NQF measure to be endorsed when the NQF Board 

ratifies the measures. However, NQF is required to submit a final report to HHS following 

ratification to complete an endorsement project. For purposes of our report, we considered 

an endorsement project to be completed when the board ratifies the measures that are the 

subject of the project. 

36The 20 conditions are acute myocardial infarction, Alzheimer’s disease and related 

disorders, atrial fibrillation, breast cancer, cataract, chronic kidney disease, chronic 

obstructive pulmonary disorder, colorectal cancer, congestive heart failure, diabetes, 

endometrial cancer, glaucoma, hip/pelvic fracture, ischemic heart disease, lung cancer, 

major depression, osteoporosis, prostate cancer, rheumatoid arthritis and osteoarthritis, 

and stroke/transient ischemic attack. 

NQF Made Progress on 

Projects under Each of the 

Contract Activities as of 

August 31, 2011
 

Page 15 GAO-12-136 NQF Contract Performance

in December 2011.37 In addition, NQF completed a contractually 

required review of its endorsement process, subcontracting with 

Mathematica Policy Research, Inc. (Mathematica). The review 

focused on the timeliness and effectiveness of the endorsement 

process;38 identified inefficiencies, including those that may contribute 

to delays; and recommended, among other steps, that NQF create a 

schedule for its endorsement process for measure developers and 

develop feasible time lines that include clear goals for each 

endorsement project. HHS officials stated that Mathematica’s 

recommendations were valuable because much of the work under the 

NQF contract needs to be completed in an accelerated timeline to 

help fill critical measurement gaps associated with HHS’s health care 

quality programs and initiatives. For more information about this 

review, see appendix IV.

• Maintenance of Endorsed Quality Measures Activity. NQF 

maintained—that is, updated or retired—124 measures under the 

contract since the beginning of the second contract year.39 These 

included 41 measures reviewed under NQF’s 3-year review cycle 

related to diabetes, mental health, and musculoskeletal conditions.40

In addition, 83 measures were maintained under NQF’s other 

maintenance review processes.41

 

37NQF completed the child health quality project by endorsing three additional child health 

quality measures in September 2011, as scheduled. As of November 2011, NQF was 

continuing to conduct work on the patient safety project, such as through the endorsement 

of an additional patient safety measure in September 2011. NQF expected to complete 

additional review of measures under this project by December 2011. 

NQF was also continuing to work on 

maintenance projects it initiated in 2010 for measures related to 

cardiovascular and surgery measures. As of August 2011, the two 

38Mathematica reviewed 23 projects in its analysis, including 7 endorsement projects 

conducted under the HHS contract.

39Under NQF’s maintenance process, NQF can retire a measure, or remove its 

endorsement status for various reasons, such as if the board determines a measure is 

obsolete. 

40Although these measures were reviewed under the contract, they were not reviewed 

under NQF’s updated Endorsement Maintenance Process because the reviews began 

prior to its implementation. 

41During our reporting period, NQF maintained 80 measures using its time-limited 

endorsement review process in areas such as infectious diseases and neurology. NQF 

also maintained 3 measures under its ad hoc review process related to various conditions, 

including patient safety.
 

Page 16 GAO-12-136 NQF Contract Performance

projects were expected to be completed by December 2011 and 

January 2012, respectively.42

• Promotion of the Development and Use of Electronic Health 

Records Activity. NQF has made progress on three projects related 

to retooling—that is, converting previously endorsed quality measures 

to an electronic format that is compatible with electronic health 

records (EHR). First, NQF completed initial retooling of 113 

measures. This work is intended to allow data from EHRs to be used 

for quality measurement, which is a part of HHS’s long-term goal to 

use health information technology to exchange information and 

improve quality of care. Second, as of August 2011, NQF convened 

an expert review panel to review the retooled measures to ensure that 

each retooled measure is properly formatted, the logic is correctly 

stated, and the intent of the measures is maintained in the electronic 

format that will use data obtained from EHRs, instead of from claims 

as originally formatted. Third, as of August 2011, NQF was expected 

to complete another project to provide an updated list of the 113 

retooled measures to HHS by December 2011, which would 

incorporate any revisions identified by the expert review panel and 

others involved in the retooling process. After these updated 

measures are completed, HHS officials told us that they will contract 

with other entities to conduct testing of some of the 113 retooled 

measures to assess the feasibility of implementing the measures in 

the electronic format. Although NQF’s endorsement process requires 

that measure developers submit data on validity and reliability testing 

of measures they submit for endorsement, this testing does not 

include feasibility testing for implementing the measures in an 

electronic format for performance measurement. As of December 

2011, HHS officials did not provide an expected date of completion for 

this feasibility testing but told us that they have awarded two contracts 

that include this in their scope of work.43

 

42As of November 2011, NQF was still completing the cardiovascular and surgery projects 

and expected to complete the work by December 2011 and January 2012, respectively, as 

scheduled. 

In addition to the retooling 

43HHS officials told us that they awarded contracts to Mathematica Policy Research, Inc., 

and Abt Associates to conduct feasibility testing on 69 of 113 retooled measures. These 

contracts were awarded on June 8, 2011, and July 8, 2011, respectively. For the 

remaining 44 retooled measures, HHS officials told us that HHS issued a solicitation for a 

contract to conduct feasibility testing of the measures; however, they did not receive any 

bids. As of December 2011, they said that they will not be issuing a solicitation for a new 

contract to test the 44 retooled measures. 
 

Page 17 GAO-12-136 NQF Contract Performance

projects, NQF is developing a software tool—the Measure Authoring 

Tool—to allow measure developers to create standardized electronic 

measures that help capture information in EHRs so that less retooling 

would be needed in the future. As of August 2011, NQF was 

completing final testing of the beta, or initial, version of this tool. NQF 

expected to complete testing and publish an updated version for 

public use by January 2012.44

• Multistakeholder Input into HHS’s National Quality Strategy 

Activity. NQF convened the National Priorities Partnership (NPP), a 

multistakeholder group expected to provide annual input on national 

priorities, among other things, to be considered in the National Quality 

Strategy.45 As of August 2011, the NPP was completing a report on 

this input, which was then published in September 2011. The report 

noted the need for a national comprehensive strategy that identifies 

core sets of standardized measures to meet each of the national 

priorities HHS identified in the 2011 National Quality Strategy, among 

other things. The NPP noted in the report that a common data 

platform, core measure set, and public reporting mechanism are key 

components of the infrastructure for performance measurement. It 

also highlighted that a strategic plan, road map, and timeline for 

establishing an infrastructure should be accelerated to allow for rapid 

implementation over the next 5 years. Additionally, the NPP reported 

that it was critical that all federal programs drive toward the 

establishment of a common platform for measurement and reporting.

• Multistakeholder Input on the Selection of Quality Measures 

Activity. NQF has convened the Measure Applications Partnership 

(MAP). The MAP is a multistakeholder group that is expected to 

conduct work in two areas. First, the MAP is expected to provide input 

to the Secretary of HHS on the selection of quality measures for use 

in payment programs and value-based purchasing programs required 

by PPACA, among others. The MAP will review a list of measures 

published by the Secretary of HHS on December 1 of each year, and 

 

44As of October 2011, NQF completed testing of the beta version that was made available 

for public use in September 2011 and expected to publish an updated version in January 

2012, as scheduled.

45The National Priorities Partnership (NPP) represents over 48 major national 

organizations from the public and private sectors, including consumers, purchasers, 

clinicians, and communities. Under PPACA, HHS is required to solicit annual input from 

such a group on the National Quality Strategy. 
 

Page 18 GAO-12-136 NQF Contract Performance

develop a report that contains a framework to help guide measure 

selection. The MAP will provide its annual input beginning February 1, 

2012, for measures used in the following 11 programs: hospice, 

hospital inpatient, hospital outpatient, physician offices, cancer 

hospitals, end-stage renal disease (ESRD) facilities, inpatient 

rehabilitation facilities, long-term care hospitals, hospital value-based 

purchasing, psychiatric hospitals, and home health care. Second, the 

MAP is expected to publish reports that provide input on the selection 

of measures for use in various quality reporting programs, including 

those for physicians. As of August 2011, the MAP had held meetings 

and initiated its work for reports due October 1, 2011.46

• Other Health Care Quality Measurement Activity. NQF completed 

a project to endorse six imaging efficiency measures. NQF was also 

continuing to work on a project to identify existing quality measures 

and gap areas related to measurement of regionalized emergency 

care services.

47

Our review of NQF documents found that NQF had not met or did not 

expect to meet time frames on more than half of the projects under the 

contract activities that were completed or ongoing, as of August 2011. 

Specifically, our review of documents found that NQF had not met 

expected time frames on 18 of the 26 projects it completed under the nine 

contract activities. Further, NQF did not expect to meet time frames on 14 

of the 34 projects on which it was continuing to work. The delays of these 

projects under the contract activities varied in time from about 1 to 

12 months. HHS officials told us they approved all changes to the time 

frames, which were established by HHS and NQF in NQF’s 2010 and 

2011 final annual MIPPA work plans and the PPACA technical proposal. 

Appendix III provides the status for all projects related to each of the nine 

contract activities, including information on their expected and actual time 

frames for completion during our reporting period. Examples of projects 

under the contract activities for which NQF did not meet or did not expect 

to meet expected time frames include the following:

 

46On October 4, 2011, NQF reported that it submitted these reports to HHS on October 1, 

2011, as scheduled. 

47Regionalized emergency medical care refers to directing patients to emergency facilities 

with optimal capabilities for a given type of illness or injury to help coordinate emergency 

care across a region.

NQF Did Not Meet 

Expected Time Frames 

and Exceeded Cost 

Estimates
 

Page 19 GAO-12-136 NQF Contract Performance

• Endorsement of Measures Activity. NQF did not meet or did not 

expect to meet time frames for all five endorsement projects under the 

endorsement contract activity.48 (See app. III for details on the five 

projects.) For example, NQF was expected to complete an 

endorsement project for nursing home quality measures in July 2010; 

however, the measures were not endorsed until February 2011. (See 

fig. 1 for estimated time frames and actual completion dates for all 

projects related to the endorsement contract activity.)

 

48As of August 2011, three of these five projects had been completed. However, NQF was 

still completing the patient safety project, which was delayed approximately 10 months to 

allow for harmonization of measures, and the child health quality project. 
 

Page 20 GAO-12-136 NQF Contract Performance

Figure 1: Estimated Time Frames and Completion Dates for All Five Projects under NQF’s Endorsement Contract Activity, as 

of August 31, 2011

Note: For purposes of our report, we consider an endorsement project completed when the board 

ratifies its measures.

a

NQF had three separate reviews of measures during Phases I and II, resulting in three separate 

board endorsements. The circles reflect those three reviews.

b

NQF endorsed one out of two patient safety measures under phase II of the patient safety project. 

The remaining measure was still under consideration by the board and is scheduled to be reviewed 

for endorsement on September 15, 2011.

c

Mathematica’s report on its evaluation of NQF’s CDP process, or the endorsement process, and 

NQF’s response to Mathematica’s recommendations in the report were published in January 2011. 

NQF solicited comments on its proposed revisions and additions to its endorsement process, as a 

result of the Mathematica evaluation, through April 2011.

d

NQF endorsed 41 out of 44 child health quality measures on August 15, 2011. As of August 2011, 

the remaining 3 measures were still under consideration by the board and were scheduled to be 

reviewed for endorsement on September 15, 2011.
 

Page 21 GAO-12-136 NQF Contract Performance

NQF officials stated that several factors contributed to NQF exceeding 

the expected time frames for the five endorsement projects, including 

the high volume of measures submitted for review, the amount of time 

it took to harmonize measures49

• Promotion of the Development and Use of Electronic Health 

Records Activity. NQF did not meet or did not expect to meet 

expected time frames for five out of eight projects related to the EHR 

contract activity.

between measure developers, and a 

need for additional technical expertise on review panels.

50 For example, NQF was expected to complete its 

initial retooling of 113 endorsed quality measures into electronic 

formats by September 2010, but this effort was not completed until 

December 2010. (See fig. 2 for estimated time frames and actual 

completion dates for all projects related to the EHR contract activity.) 

In addition, NQF was expected to complete the project to convene an 

expert panel to review the 113 retooled measures by January 2011. 

However, the panel did not complete its review of the 113 measures 

until June 2011.

 

49Harmonization is a process where related measures are standardized in a manner by 

which they can be used in multiple settings when relevant. NQF defines “related 

measures” as those that either (1) address the same concepts for measure focus (target 

process, condition, event, outcome) but a different target population; or (2) address 

different concepts for measure focus (target process, condition, event, outcome) but the 

same target population. NQF officials stated in some instances NQF needed to facilitate 

harmonization discussions between measure developers to help ensure that the most 

appropriate measure was being used, and that the measure portfolio did not include 

duplicative measures, and to avoid confusion within the health care community. 

50Two of the five projects that have not met expected time frames are still in progress and 

are expected to be completed by the end of the third contract year. The remaining three 

have been completed.
 

Page 22 GAO-12-136 NQF Contract Performance

Figure 2: Estimated Time Frames for Eight Projects Related to NQF’s Contract Activity on the Promotion and Use of EHRs, as 

of August 31, 2011

a

In January 2011, NQF completed a version of the tool that was released for beta testing (reflected by 

the circle). NQF is now updating the version, which it expects to release in January 2012.

b

NQF officials told us that the final report was completed in April 2010; however, additional work was 

required to complete the project through the beginning of July 2010.

c

NQF is expected to publish several versions of the QDM throughout the contract term to allow for 

ongoing review and updates. The circles indicate the versions that have been published or are 

expected to be published, as of August 2011.

d

NQF was unable to complete all 113 retooled measures by September 2010, as initially scheduled. 

NQF submitted versions of 44 measures in July 2010, as requested by HHS. The final versions of all 

113 measures were not completed until December 2010.
 

Page 23 GAO-12-136 NQF Contract Performance

e

NQF officials told us the eMeasure Review Panel was not able to complete its review of the retooled 

measures by its initial due date of January 13, 2011, because all 113 measures were not retooled 

until December 2010. As a result, the review was not completed until June 2011. As of August 2011, 

the review panel was expected to submit its final report in October 2011. NQF officials told us in 

October 2011 this report was completed, as scheduled.

According to HHS and NQF officials, several factors contributed to 

NQF exceeding expected time frames for the retooling project under 

the EHR contract activity. HHS officials stated that the first set of 44 

retooled measures submitted had errors that required correction. For 

example, HHS officials stated that they found errors in the electronic 

coding of these 44 retooled measures requiring NQF and its 

subcontractors who retooled the measures to make corrections. In 

addition, HHS and NQF officials stated that after starting the retooling 

project, they quickly learned that the estimated time frames for the 

retooling project, as well as other projects related to the EHR activity, 

were overly ambitious, given the scope and complexity of the work. 

For example, HHS officials noted that retooling of quality measures 

into electronic format had never been attempted before and the 

technical complexity and labor required to complete the project were 

greater than anticipated. NQF officials also told us that HHS’s 

requests to modify the scope of work for this project often required 

changing the time frame for completing the retooled measures. These 

factors resulted in an extension of the project that delayed the final 

delivery of the 113 retooled measures as well as contributed to the 

need for additional staff at NQF.

• Other Health Care Quality Measurement Activity. NQF was 

expected to complete two projects under the other health care quality 

measurement activity related to efficiency and resource use—one 

white paper on resource use51 and another on geographic-level 

efficiency52

 

51Resource-use measures compare resources used (i.e., cost of care) with either (1) a 

specific population, (2) a specific service, or (3) a series of related services over time, to 

help identify the resources necessary for delivering high-quality care and to promote 

greater efficiency in delivery of health care services. 

by July 2010. These white papers were intended to 

provide information for an endorsement project on resource-use 

measures that began in January 2011. However, as of August 2011, 

the resource-use paper was still under review by HHS, and NQF 

52Geographic-level efficiency measures would be used for quality measure reporting at 

the geographic and population level. 
 

Page 24 GAO-12-136 NQF Contract Performance

officials stated they expected to receive comments in September 

2011.53 The geographic-level efficiency paper was canceled in June 

2010 at the request of HHS. NQF initially intended to subcontract the 

work on these two projects, but officials told us that they were unable 

to identify a subcontractor at the level of funding approved for this 

project. As a result, HHS approved NQF’s proposal to complete this 

work internally. HHS officials stated that the drafts NQF submitted on 

both topics were poor in quality and did not meet its needs, resulting 

in HHS requesting additional revisions for the resource-use white 

paper that delayed its completion, and requesting the cancellation of 

the geographic-level efficiency white paper.

• Administrative Activity. NQF did not meet the expected time frames 

for completing one of the required projects under the administrative 

activity—finalizing its annual work plan. Specifically, the NQF contract 

requires NQF to develop an annual work plan and to receive final 

approval from HHS within the first 4 weeks of each contract year; 

however, NQF did not meet this requirement in 2010 or 2011. For 

example, the final 2011 MIPPA annual work plan was not developed 

by NQF and approved by HHS until April 1, 2011. According to NQF 

and HHS officials, the 2011 MIPPA work plan was not developed and 

approved on time due to extended discussions on the scope and cost 

estimates of NQF’s EHR activities. HHS officials told us that the 

primary reason for the extended discussions was that they expected 

the costs to reflect all the work needed to complete the Measure 

Authoring Tool (MAT) by the end of the second contract year. 

However, they said that NQF only submitted a beta version of the tool 

by the end of the second contract year, which was not the version 

expected by HHS. NQF officials told us that the version was never 

intended to be final but rather a beta version, consistent with their 

understanding of HHS’s expectations. As a result, HHS and NQF 

officials needed to evaluate the scope of work and cost estimates for 

this and other projects. Further, NQF officials told us the delay in 

completing the 2011 MIPPA annual work plan resulted in the 

interruption of NQF’s ongoing work related to the MAT under the EHR 

contract activity. The delay also delayed its receipt of funding for 

some new or ongoing work under the contract. In some instances, 

NQF chose to start new or continue ongoing work with its own 

funding. For example, NQF officials stated that NQF began work 

 

53HHS officials told us that they accepted the final version of this paper in October 2011. 
 

Page 25 GAO-12-136 NQF Contract Performance

related to the MAP using its own funds until HHS authorized the work. 

In addition, the delay in completing the 2011 MIPPA work plan 

resulted in the need to set the start date for fall 2011 rather than 

earlier in the contract year for some of the projects under the 

maintenance activity.

NQF also exceeded its cost estimates for projects under three of the 

contract activities. HHS officials told us they approved the changes to the 

cost estimates and in some cases modified NQF’s scope of work to help 

ensure that NQF’s costs did not exceed the amount HHS had obligated 

for the contract activities. NQF officials stated that in certain cases, not 

meeting expected time frames contributed to NQF exceeding these cost 

estimates. For example, the delays in projects related to the EHR contract 

activity, including expanding the scope of the retooling project, 

contributed to NQF exceeding its cost estimate of about $3.8 million for 

the entire EHR contract activity by about $560,000 in the second contract 

year.54 In another example, the delays in finalizing the 2010 and 2011 

MIPPA work plans contributed in part to NQF exceeding its cost estimate 

for developing and finalizing these plans, which is a project under the 

administrative contract activity. Specifically, while NQF estimated that 

completion of the annual work plan would cost approximately $77,000, 

NQF reported an actual cost of $176,590. In addition, NQF also exceeded 

its cost estimate for the endorsement contract activity during the second 

contract year for various reasons, including a need for additional technical 

experts for review panels. Specifically, NQF exceeded estimated costs of 

about $3.1 million for the entire endorsement activity by about $146,000 

in the second contract year.55

 

54NQF officials told us that it was originally required to retool 44 measures under the 

contract; however, HHS expanded the scope of work in 2010 to 113 measures. This 

required additional labor and other resources that increased the costs associated with the 

retooling project. 

While HHS officials told us they approved 

all changes to the cost estimates, in certain cases they reduced the scope 

of NQF’s work in 2011 to ensure that total available funding for the 

contract year was not exceeded and that sufficient funding was available 

for ongoing projects. For example, HHS officials told us that they had 

hoped to start several new endorsement projects beginning in 2011; 

55Our previous work shows that without the ability to generate reliable cost estimates, 

programs are at risk of experiencing cost overruns, missed deadlines, and performance 

shortfalls. GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 

Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: 

March 2009). 
 

Page 26 GAO-12-136 NQF Contract Performance

however, these projects were not included in the 2011 final annual MIPPA 

work plan so that funding would be available for NQF to complete its 

ongoing projects, including work that was delayed under the EHR 

contract activity. In addition, HHS requested that NQF discontinue its 

work on one project related to the development of a public website for 

2011, which is associated with the administrative contract activity.

HHS officials told us that to help monitor NQF’s performance on the 

projects under the contract activities, they rely on NQF to report any 

issues, including those related to time frames or cost estimates, in the 

monthly progress reports that NQF is required to submit to HHS or during 

phone calls held at least monthly. While HHS monitored NQF’s progress 

and approved changes to the time frames and cost estimates for the 

projects under the contract activities, HHS did not use available tools for 

monitoring that are required under NQF’s contract. These tools could 

have helped to provide an opportunity for HHS to make any appropriate 

changes to NQF’s projects. For example, HHS did not conduct an annual 

performance evaluation required by the contract that would assess 

timeliness and cost control issues, among other things, for the previous 

contract year.56

 

56The requirement in the contract to conduct annual performance evaluations is consistent 

with a FAR provision that provides that agencies may specify that interim performance 

evaluations be performed for multiyear contracts. See 48 C.F.R. § 42.1502(a) (2010).

The results of such an evaluation could help HHS officials 

to consider potential timeliness and cost issues when determining NQF’s 

scope of work for the next year. Further, while monthly progress reports 

and invoices include information on NQF’s costs, these documents do not 

compare reported costs to initial cost estimates. HHS officials told us that, 

prior to August 2011, they had not enforced a contractual requirement for 

NQF to submit—nor had it received from NQF—a financial graph in its 

monthly progress reports that provides information comparing NQF’s 

monthly incurred costs for each of the contract activities with initial cost 

estimates. Instead, HHS officials informally requested that NQF provide 

them with the financial status of the contract activities in midyear 2010, 

which helped them to plan for NQF’s work under the contract for 2011. 

Having a financial graph in the monthly progress report could have helped 

HHS officials to identify instances where any contract activity was 

approaching or exceeding NQF’s initial cost estimates prior to HHS’s 

midyear review. This, in turn, could have provided HHS and NQF an 

opportunity to adjust estimates of future costs for these or related 

HHS Did Not Use All 

Monitoring Tools Required 

under the NQF Contract
 

Page 27 GAO-12-136 NQF Contract Performance

activities earlier in the contract year. HHS officials had asked NQF to 

begin to include such a financial graph in its monthly progress reports 

beginning in August 2011.

From January 14, 2010, through August 31, 2011, NQF reported a total of 

approximately $22.4 million in costs and fixed fees on monthly invoices 

submitted to HHS for projects under activities conducted in response to 

MIPPA and PPACA. Specifically, NQF reported about $12.8 million in 

total costs and fixed fees for the contract activities it performed during the 

second contract year—January 14, 2010, through January 13, 2011.57

From January 14, 2011, through August 31, 2011, part of the third 

contract year, NQF reported an additional $9.6 million in total costs and 

fixed fees.58

During the second contract year, the majority of NQF’s reported costs 

were related to the promotion of the development and use of EHRs 

(36 percent, or $4.6 million) and endorsement of health care quality 

measures (26 percent, or $3.3 million). Figure 3 illustrates the costs and 

fixed fees NQF reported for eight of the nine contract activities we 

reviewed that occurred during the second contract year. The ninth 

contract activity relates to multistakeholder input on the selection of 

quality and efficiency measures, as directed by PPACA. This contract 

activity did not begin until after January 14, 2011, which is the start of the 

third contract year.

 

57According to HHS, total funding obligated was approximately $13.6 million for the 

second contract year, which covered January 14, 2010, through January 13, 2011.

58According to HHS, total funding obligated was approximately $25 million for the third 

contract year, which covers January 14, 2011, through January 13, 2012. 

NQF Reported Costs 

and Fixed Fees of 

over $20 Million from 

January 2010 through 

August 2011
 

Page 28 GAO-12-136 NQF Contract Performance

Figure 3: NQF’s Reported Contract Activity Costs and Fixed Fees Totaling 

Approximately $12.8 Million during the Second Contract Year (Jan. 14, 2010— 

Jan. 13, 2011)

Note: The figure describes eight of the nine contract activities we reviewed.

For the part of the third contract year covered in our review—January 14, 

2011, through August 31, 2011—almost one-half of NQF’s reported costs 

were for the activity to promote the development and use of EHRs and for 

the activity to provide multistakeholder input on the selection of quality 

and efficiency measures. Each of these activities accounted for 

22 percent, or about $2.1 million of NQF’s reported costs. Other costs 

reported by NQF include those for the activity related to providing 

multistakeholder input into HHS’s annual National Quality Strategy 

($1.55 million, or 16 percent) and those for the activity related to the 

maintenance of endorsed quality measures activity (13 percent, or 
 

Page 29 GAO-12-136 NQF Contract Performance

$1.29 million). Figure 4 illustrates the costs and fixed fees NQF reported 

for the part of the third contract year covered in our review for each of the 

nine contract activities we reviewed.

Figure 4: NQF’s Reported Contract Activity Costs and Fixed Fees Totaling 

Approximately $9.6 Million during the Period of the Third Contract Year Covered in 

Our Review (Jan. 14, 2011—Aug. 31, 2011)
 

Page 30 GAO-12-136 NQF Contract Performance

According to HHS, as of August 2011, about $55.2 million remains 

available for the NQF contract.59 About $15.1 million in MIPPA funding 

remains available for work to be conducted through January 13, 2013. In 

addition, HHS plans to obligate approximately $40.1 million of its PPACA 

funding through 2014 for NQF’s activities related to health care quality 

measurement in response to PPACA.60

For its various programs or initiatives, HHS has used or planned to use 

about one-half of the quality measures that NQF has endorsed, 

maintained, or retooled under the contract, as of August 31, 2011, and 

HHS officials expect to evaluate if and how the remaining measures will 

be used. However, HHS has not comprehensively determined how it will 

use NQF’s work under the contract to implement PPACA requirements 

related to quality measures.

 

59As of August 2011, NQF had reported costs and fixed fees totaling $28.9 million since 

the contract was awarded in January 2009. Specifically in our July 2010 report, we 

highlighted that NQF reported costs and fixed fees totaling approximately $6.5 million for 

the first contract year—January 14, 2009, to January 13, 2010. In addition, NQF had 

reported about $22.4 million in costs and fixed fees from January 14, 2010, through 

August 31, 2011—our reporting period that covers the second and part of the third 

contract years. See GAO-10-737. 

60From January 14, 2011, through August 31, 2011, NQF reported about $4 million in 

costs and fixed fees for PPACA-required activities. HHS officials told us that they plan to 

obligate $10 million of the PPACA funding for health care quality measurement each fiscal 

year from 2011 through 2014. Therefore, at least $40 million in PPACA funding is 

expected to be available to NQF under the contract. HHS officials said that because 

PPACA does not specify the proportion of the $100 million for NQF’s and HHS’s health 

care quality measurement work for NQF, they determined that $10 million per year was a 

reasonable amount. 

HHS Used or Planned 

to Use Some NQF 

Measures Received 

under the Contract 

but Has Not 

Comprehensively 

Planned for Other 

Measurement Needs 

to Implement PPACA
 

Page 31 GAO-12-136 NQF Contract Performance

According to HHS officials, HHS has used or planned to use61 about onehalf (164) of the 344 health care quality measures it has received from 

NQF through various endorsement, maintenance, and retooling projects 

under the contract, as of August 31, 2011.62 For example, of the 164 

measures used or planned for use, 44 were used in CMS’s Medicare and 

Medicaid EHR Incentive Program after being retooled—that is, converted 

to an electronic format that is compatible with EHRs—under the NQF 

contract.63

 

61For the purpose of this report, we defined a measure as used if the measure was 

identified by HHS and included in an HHS program or initiative as of August 31, 2011, and 

indicated as such by a final rule or other HHS documentation. We defined a measure as 

planned for use if the measure was identified by HHS for potential use in an HHS program 

or initiative as of August 31, 2011, and indicated as such by a proposed rule or other HHS 

documentation. 

Although these 44 retooled measures were used in the EHR 

Incentive Program, HHS officials stated that NQF and HHS detected 

coding and other errors in the versions of the 44 retooled measures that 

were published in the program’s final rule in July 2010 that required NQF 

to make corrections to them after publication of the final rule. NQF did not 

submit the revised versions of the 44 retooled measures published in the 

final rule to HHS until December 2010. HHS officials stated that because 

the final rule had already been published prior to receiving the final 

62Measures that are endorsed, maintained, and retooled under the NQF contract are 

counted separately, even if they are the same measure. Therefore, the 344 measures 

HHS has received from NQF do not represent 344 distinct measures. For example, a 

measure that had been maintained under the contract and later retooled would be counted

as two separate measures. Therefore, the measure would appear twice in our count of the 

total number of measures received from NQF—once as a maintained measure and then 

separately as a measure that was retooled. Similarly, the measure would also be counted 

as two separate measures for purposes of determining the number used or planned for 

use. If both versions of the measure were used or planned for use in HHS programs or 

initiatives, then it would also appear twice in our count of measures used or planned for 

use.

63See 75 Fed. Reg. 44,314 (July 28, 2010). The Medicare and Medicaid EHR Incentive 

programs will provide incentive payments to eligible Medicare and Medicaid professionals 

and hospitals that adopt and use health information technology, and reduce payments for 

those Medicare professionals or hospitals that do not. The Medicare and Medicaid EHR 

programs are divided into three stages. Stage I, which is underway, focuses on 

establishing the functionalities in certified EHR technology that will allow for continuous 

quality improvement and ease of information exchange. Stages II and III will potentially 

become more stringent as the capabilities of health information technology infrastructure 

increase. In 2011, part of the requirement for Stage I of the Medicare EHR Program is that 

eligible professionals attest that they have met the program’s reporting requirements, 

which includes that they report on a certain number of measures that were retooled by 

NQF.

HHS Has Used or Planned 

to Use 164 of the 344 

Measures That NQF Has 

Endorsed, Maintained, or 

Retooled under the 

Contract 
 

Page 32 GAO-12-136 NQF Contract Performance

formatted measures, CMS listed general guidance on its website to 

address the errors. HHS officials told us that these 44 measures are 

being used but have not yet been tested to assess the feasibility of 

implementing them in the electronic format.64 Until the testing is complete, 

HHS runs the risk that some of these measures may not work as intended 

when implemented in electronic format for performance measurement. As 

a result, the agency does not have reasonable assurance that the 

retooled versions of the measures will correctly capture information from 

EHRs. In addition to the 44 retooled measures used in the EHR Incentive 

Program, HHS also has used or planned to use 120 measures that it 

received from endorsement and maintenance projects under the NQF 

contract for various HHS programs and initiatives. (See table 2 for details 

on specific programs in which HHS has used or planned to use health 

care quality measures received from NQF under the contract.)

 

64HHS officials said that the 44 measures are not included as part of the scope of work 

under the two contracts under which feasibility testing of the retooled measures will be 

performed.
 

Page 33 GAO-12-136 NQF Contract Performance

Table 2: Status of Measures Used or Planned for Use under the NQF Contract, as of August 31, 2011

Projects 

Total measures 

received from NQF 

under the contract

Measures used 

or planned for 

use by HHS

Description of HHS programs and initiatives 

for which NQF measures are being used or 

planned for use

Promote the development and use of electronic health records

Measure retooling 113 44 Forty-four measures have been used in Stage I of 

the Electronic Health Records (EHR) Incentive 

Program.a

Total retooled measures 113 44

Endorse health care quality measures

Patient outcomes 38 6 Six measures are planned for use in various 

programs including the Hospital Inpatient Quality 

Reporting Program.b

Patient safety 1 0 No measures used or planned for use.

Nursing homes 21 17 Seventeen measures are planned for use in 

various programs including Nursing Home 

Compare.c

Child health quality 41 8 Eight measures have been used in CMS Medicaid 

Programs as part of CMS’s Children’s Health 

Insurance Program Reauthorization Act of 2009 

(CHIPRA) Initial Core Set of Children’s Health Care 

Quality Measures.

d

 

Total endorsed measures 101 31

Maintain previously endorsed health care quality measurese

3-Year maintenance review cyclef

Diabetes 13 10 Twenty-three measures have been used or are 

planned for use in various programs and initiatives 

including the Physician Quality Reporting Systemg

and the Hospital Outpatient Quality Reporting 

Program.h

Mental health 11 3 

Musculoskeletal conditions 17 10

Other maintenance review processes

Ad hoc review 3 2 Two measures have been used or are planned for 

use in various programs, including the EHR 

Incentive Program and the Hospital Inpatient 

Quality Reporting Program.

Time-limited review 80 63 Sixty-three measures have been used or are 

planned for use in various programs and initiatives, 

including the EHR Incentive Program and the 

Physician Quality Reporting System.

Total maintained measures 124 88
 

Page 34 GAO-12-136 NQF Contract Performance

Projects 

Total measures 

received from NQF 

under the contract

Measures used 

or planned for 

use by HHS

Description of HHS programs and initiatives 

for which NQF measures are being used or 

planned for use

Other health care quality measurement activity

Imaging efficiency measures 6 1 One measure is planned for use in the Hospital 

Outpatient Quality Reporting Program.

Total measures from other 

quality measurement activity

6 1 

Total measures 344 164

Source: GAO analysis of final rules, proposed rules, and other HHS documentation.

Note: Measures that are endorsed, maintained, and retooled under the NQF contract are counted 

separately, even if they are the same measure. Therefore, the 344 measures HHS has received from 

NQF do not represent 344 distinct measures. For example, a measure that had been maintained 

under the contract and later retooled would be counted as two separate measures. Therefore, the 

measure would appear twice in our count of the total number of measures received from NQF—once 

as a maintained measure and then separately as a measure that was retooled. Similarly, the measure 

would also be counted as two separate measures for purposes of determining the number used or 

planned for use. If both versions of the measure were used or planned for use in HHS programs or 

initiatives, then it would also appear twice in our count of measures used or planned for use.

a

The EHR Incentive Program provides incentive payments to eligible Medicare and Medicaid 

providers that adopt and use health information technology (IT) and reduces payments for those 

Medicare professionals or hospitals that do not.

b

The Hospital Inpatient Quality Reporting program provides hospitals a financial incentive to report 

the quality of their services as well as provides CMS data to help consumers make more informed 

decisions about their health care.

c

Nursing Home Compare is a tool of the Nursing Home Quality Initiative that allows consumers, 

providers, states, and researchers to compare information on health quality measures among various 

nursing homes.

d

CMS’s CHIPRA Initial Core Set of Children’s Health Care Quality Measures identifies an initial core 

set of child health quality measures for voluntary use by state Medicaid and Children’s Health 

Insurance Program (CHIP) programs to evaluate the quality of care and health outcomes in children 

enrolled in Medicaid and CHIP programs.

e

NQF can also maintain measures through its (1) annual updates, (2) ad hoc requests, and (3) timelimited endorsement review processes. NQF officials told us that as of August 2011, 83 measures 

have been maintained using these processes, including 80 under time-limited review and 3 under ad 

hoc request.

f

NQF officials told us the review of the diabetes, mental health, and musculoskeletal measures were 

maintained under its previous 3-year maintenance review process because the project began in 

September 2009, which is prior to the updated process that took effect in August 2010.

g

The Physician Quality Reporting System provides an incentive payment to eligible professionals who 

select among 240 measures to report.

h

CMS’s Hospital Outpatient Quality Reporting is a pay-for-performance program for outpatient 

hospital services.

HHS officials told us that they expect to evaluate if and how they could 

use all of the remaining 180 of the 344 quality measures that were 

endorsed, maintained, or retooled under the NQF contract that are not 

currently in use or planned for use in HHS programs or initiatives. 

According to HHS officials, any measure developer can submit a measure 

to be considered for NQF endorsement. Therefore, all the measures 
 

Page 35 GAO-12-136 NQF Contract Performance

received under the contract may not be applicable to a particular HHS 

health care quality program or initiative. HHS officials told us that they will 

review the remaining 180 measures to determine if they are applicable to 

their health care quality programs or initiatives. The officials expect that 

many of these measures will be used in HHS programs or initiatives 

required by PPACA. For example, HHS officials told us that they will 

consider implementation of most of the retooled measures in future 

stages of the EHR Incentive Program. In addition, PPACA directed HHS 

to establish a hospital value-based purchasing program, as well as to 

make plans or begin pilot programs for value-based purchasing in other 

settings of care. The hospital value-based purchasing program will use 

various quality measures and depend on the information collected on 

them to determine payments to providers. PPACA also required the 

development of no less than 10 provider-level outcome measures for 

hospitals and physicians by March 2012. Further, PPACA directed HHS 

to identify quality measures that could be used to evaluate hospice 

programs and publish these measures by October 1, 2012. HHS officials 

told us that they are in the process of determining whether or to what 

extent the remaining 180 measures HHS has received under the NQF 

contract can be used to address the new measurement needs and 

priorities established by PPACA. HHS officials told us that they prefer to 

use NQF-endorsed measures to meet HHS’s measurement needs 

because these quality measures are nationally recognized standards and 

in some cases HHS is required to use them.65

 

65See, e.g., Pub. L. No. 111-148, § 3004(a), 124 Stat. 119, 368 (2010) (codified as 

amended at 42 U.S.C. § 1395ww(m)(5)) (directing HHS to publish measures for long-term 

care hospitals and requiring the department to consider measures endorsed by NQF). For 

specific areas for which NQF has not endorsed feasible and practical measures, HHS may 

specify measures that have not been endorsed by NQF as long as due consideration is 

given to measures that have been endorsed or adopted by a consensus organization.
 

Page 36 GAO-12-136 NQF Contract Performance

Although HHS has taken steps to determine how it can use the measures 

received under the contract with NQF, the agency does not have a 

comprehensive plan for determining how it will use the remainder of the 

work conducted under NQF’s contract to implement PPACA 

requirements, including plans for additional quality measures that need to 

be endorsed during the remaining contract years. HHS officials told us 

that HHS determines on an annual basis which activities—including work 

on quality measures—NQF is to perform under the contract through the 

interagency workgroup. The workgroup is comprised of representatives 

from various HHS agencies and allows them to provide input on their 

needs, including quality measures that need endorsement from NQF, for 

their respective programs. However, HHS officials told us that each HHS 

program assesses its quality measurement needs separately and 

provides varying levels of detail about its needs. Therefore, it is unclear 

the extent to which all programs consistently incorporate PPACA’s quality 

measurement requirements and deadlines into these assessments. The 

NPP’s September 2011 report noted the importance of greater alignment 

of national quality measurement efforts, including the establishment of a 

comprehensive measurement strategy that identifies core measure sets, 

among other things. In addition, the report noted that all federal programs 

should work toward the establishment of a common platform for 

measurement and reporting. Without a comprehensive plan that 

delineates HHS’s quality measurement needs, and given that each 

program assesses its quality measurement needs separately, the 

interagency workgroup may not be able to systematically ensure that all 

of HHS’s quality measurement needs that implement PPACA 

requirements align with the selection and prioritization of activities for 

NQF to complete under the contract.

While HHS has begun various efforts to assess its quality measurement 

needs, the lack of a plan that comprehensively determines the impact of 

PPACA on its needs could affect the agency’s progress on its quality 

measurement efforts as well as how it selects and prioritizes NQF’s 

contract activities. Officials told us that prior to PPACA’s enactment, CMS 

maintained a 5-year plan that listed its measurement needs based on 

agency priorities and the priorities established by the NPP for some of its 

HHS Does Not Have a 

Comprehensive Plan for 

Determining How It Will 

Use NQF’s Work under the 

Contract to Implement 

PPACA Requirements 

Related to Quality 

Measurement
 

Page 37 GAO-12-136 NQF Contract Performance

programs.66 However, officials told us that they have not updated this list 

to reflect the requirements related to quality measurement and time 

frames established by PPACA. In March 2011, HHS published the 

National Quality Strategy as required by PPACA, which included six 

priority areas of focus. The report was required by PPACA to include 

agency-specific plans, goals, benchmarks, and standardized quality 

metrics for each priority area, but did not do so. HHS officials stated that 

this document describes HHS’s initial plan for these elements and that 

they may be included in future versions of the strategy. In June 2011, 

HHS officials told us that they plan to convene a Quality Measurement 

Task Force within CMS with a goal to comprehensively align, coordinate, 

and approve the development, maintenance, and implementation of 

health care quality measures for use in various CMS programs. As of 

August 2011, the task force was in an early stage of development, and 

therefore it is too early to determine whether it will accomplish its goal. 

Although these various HHS efforts are key steps toward helping the 

agency meet its quality measurement needs, they are not guided by a 

comprehensive plan that synthesizes key priority areas identified in 

various sources, such as those reported by the NPP or in the National 

Quality Strategy, for which measures may be needed. Without such a 

plan, HHS may be limited in its efforts to prioritize which specific 

measures it needs to develop and have endorsed by NQF for its health 

care quality programs and initiatives established by PPACA. As a result, 

HHS may be unable to ensure that the agency receives the quality 

measures needed to meet PPACA requirements and specified time 

frames related to quality measurement.

Health care quality measures are increasingly important to HHS as it uses 

and will continue to use them in its existing and forthcoming programs 

and initiatives to evaluate health care delivery. For example, HHS’s valuebased purchasing programs are pay-for-performance programs that will 

 

66The NPP identified six priorities in a November 2008 report: (1) patient and family 

engagement, (2) population health, (3) patient safety, (4) care coordination, (5) palliative 

and end-of-life care, and (6) elimination of overuse. In its October 2010 report to HHS, the 

NPP identified two additional priority areas: equitable access to ensure that all patients 

have access to affordable, timely, and high-quality care; and infrastructure supports, such 

as health information technology (IT), to address underlying system changes necessary to 

attain the goals of the other priority areas. See National Priorities Partnership, Input To 

The Secretary Of Health And Human Services On Priorities For The 2011 National Quality 

Strategy (Washington, D.C.: 2010).

Conclusions 
 

Page 38 GAO-12-136 NQF Contract Performance

require providers to collect and report information on health care quality 

measures and adjust payment levels based on providers’ performance 

against the measures. PPACA has increased HHS’s quality measurement 

needs, and the time frames specified in the law have also increased the 

urgency of obtaining endorsed quality measures—which are nationally 

recognized standards and in some cases are required by statute—to 

meet these needs. Given that NQF is the entity in the United States with 

lead responsibility for endorsing health care quality measures, NQF’s 

endorsement activities under the contract are of key importance to help 

meet HHS’s quality measurement needs.

However, NQF’s endorsement process takes time. For more than half of 

the projects, including all five projects in the endorsement activity, NQF 

did not meet or did not expect to meet the initial time frames approved by 

HHS. In addition, projects under three of the contract activities have 

exceeded initial cost estimates, which resulted in HHS’s modification of 

NQF’s scope of work in some instances to help ensure that NQF’s costs 

did not exceed the funding allocated for the contract activities. While HHS 

received information in monthly progress reports to help monitor NQF’s 

performance under the contract, the agency did not use all of the 

monitoring tools required under the contract to help address issues 

related to time lines and cost estimates. These monitoring tools included 

an annual performance evaluation that could help HHS officials consider 

potential issues related to NQF’s time frames and cost estimates when 

planning work for the next year and a financial graph to be included in 

NQF’s monthly progress reports. The graph would have compared 

reported costs to initial cost estimates, which is something that monthly 

progress reports do not do. Although HHS officials reported that they 

recently began in August 2011 to enforce the contractual requirement for 

NQF to submit the graph, they have not implemented the required annual 

performance evaluation. By not taking advantage of these tools, HHS 

runs the risk of not having detailed and timely information that could help 

identify instances in which NQF might be at risk of not meeting time 

frames or exceeding estimated costs. Identifying such instances could 

provide an opportunity for HHS to make any appropriate changes to 

NQF’s scope of work, including setting priorities to ensure that HHS 

receives the quality measures it needs in a timely manner.

With the time remaining under the contract, HHS has an opportunity to 

ensure that the work performed under NQF’s contract better meets the 

agency’s needs for its programs and initiatives. However, HHS has not 

developed a plan that comprehensively identifies its quality measurement 

needs for its programs and initiatives in light of PPACA’s requirements or 
 

Page 39 GAO-12-136 NQF Contract Performance

determines how it will use the work conducted during the remaining years 

of the NQF contract to help it meet these needs. In addition, critical tasks 

may need to be completed outside of the NQF contract. For example, 

HHS requested that NQF retool 113 measures under the contract and 

used 44 of the 113 measures that included errors in its EHR Incentive 

Program. As of November 2011, feasibility testing related to 

implementation of the retooled measures had not been completed, and 

HHS expected to perform this work outside of the NQF contract. Until the 

testing is completed, HHS runs the risk that some of the retooled 

measures may not work as intended when implemented in electronic 

format for performance measurement, which is a concern because use of 

these measures is an important component of HHS’s long-term goal for 

providers to use health information technology (IT) to exchange 

information and improve the quality of care.

Without a comprehensive plan, HHS lacks assurance that its selection of 

the work to be performed by NQF—and the approximately $55.2 million 

that the agency expects to spend for remaining work under the NQF 

contract—will be prioritized in the most effective way possible. Given that 

PPACA includes time frames for the implementation of quality 

measurement programs, NQF’s pace in completing some of the work 

under the contract—particularly the endorsement activity—raises 

concerns. If the endorsement projects continue to require extended 

completion times, HHS runs the risk of not having all the endorsed 

measures it needs for implementing its programs and initiatives. Should 

this occur, HHS may need to select nonendorsed measures for its 

programs and initiatives that have not undergone an objective and 

transparent review by NQF.

To help ensure that HHS receives the quality measures it needs to 

effectively implement its quality measurement programs and initiatives 

within required time frames, we recommend that the Secretary of HHS 

take the following three actions:

• use monitoring tools required under the NQF contract to obtain 

detailed and timely information on NQF’s performance and use that 

information to inform any appropriate changes to time frames, 

projects, and cost estimates for the remaining contract years;

• ensure that testing of the electronic versions of the measures retooled 

by NQF that are being used or are planned for use in the Medicare 

and Medicaid EHR Incentive programs is completed in a timely 

Recommendations for 

Executive Action
 

Page 40 GAO-12-136 NQF Contract Performance

manner to help identify potential errors and address issues of 

implementation; and

• develop a comprehensive plan that identifies the quality measurement 

needs of HHS programs and initiatives, including PPACA 

requirements, and provides a strategy for using the work NQF 

performs under the contract to help meet these needs.

We provided a draft of this report to HHS and NQF for review and 

comment. HHS neither agreed nor disagreed with our recommendations 

and provided general comments. NQF concurred with many of the 

findings in the report and provided clarification and additional context on 

the findings and recommendations. HHS and NQF’s letters conveying 

their comments are reproduced in appendixes V and VI, respectively. In 

addition to the overall comments discussed below, we received technical 

comments from HHS and NQF, which we incorporated into our report as 

appropriate.

HHS’s comments included separate general comments from CMS and 

ASPE that provided context on aspects of our findings and 

recommendations. CMS’s comments stated that the draft report suggests 

that CMS must use all of the measures endorsed by NQF, and noted that 

not all NQF-endorsed measures are suitable for HHS quality reporting 

and public reporting programs. Although our draft report did not state that 

CMS must use all of the measures endorsed by NQF, we modified it to 

note specifically, among other things, that all measures received under 

the contract may not be applicable to a particular HHS health care quality 

program or initiative. CMS also stated that the report suggests that CMS 

has not developed measurement plans for various provisions of PPACA 

related to quality reporting, public reporting, and value-based purchasing 

programs. CMS provided additional context for current planning efforts to 

address these requirements, including its Quality Measurement Task 

Force. The draft report acknowledged this and other CMS planning efforts 

to address the health care quality requirements contained in PPACA and 

noted that, as of August 2011, this initiative was just beginning. Further, 

while various efforts are underway and CMS’s comments state that it has 

documented how quality measures will be used to address all relevant 

provisions of PPACA, CMS has not provided documentation of 

comprehensive plans to address PPACA requirements that include 

alignment across programs, detailed time frames to meet PPACA 

deadlines, or how it will use the NQF contract to help ensure that it 

receives the endorsed measures it needs to meet these requirements.

Agency and Other 

External Comments 

and Our Evaluation

HHS Comments
 

Page 41 GAO-12-136 NQF Contract Performance

ASPE’s comments noted, with respect to our first recommendation, that 

HHS used all except two of the monitoring tools called for in the contract. 

As noted in the draft report, HHS began receiving the monthly financial 

graph—one of the two monitoring tools—from NQF in August 2011. Also, 

ASPE noted its plans to update its performance evaluation system with 

NQF performance information for the first 2 contract years—the period 

January 14, 2009, through January 13, 2011—and to complete a final 

performance evaluation at the end of the contract in January 2013, which 

is the end of the fourth contract year. It did not indicate any plans to 

conduct the annual performance evaluation for the third contract year—

January 14, 2011, through January 13, 2012—which would be consistent 

with the contract’s requirements. With respect to our second 

recommendation, ASPE provided technical comments and also told us 

that CMS issued a contract solicitation to test the retooled measures, but 

CMS did not receive any bids. Instead, ASPE noted in its comments that 

two of CMS’s current contractors will conduct feasibility testing on 69 of 

the 113 retooled measures that are planned for use in HHS’s EHR 

Incentive programs. CMS does not plan to issue a solicitation for a new 

contract to test the feasibility of the remaining 44 retooled measures, 

which are currently being used in HHS’s EHR Incentive Program. We 

noted these comments in the report. Regarding our third 

recommendation, ASPE stated that the measures that are not currently in 

“use” are being evaluated by HHS and that any conclusions that they will 

not be used are not accurate. Our draft report provided information on 

which measures were used or planned for use as of August 2011, and 

indicated that the remaining measures may be used in the future. 

Specifically, the report noted that HHS officials expect that many of these 

measures will be used in HHS programs or initiatives, and that HHS 

officials told us that they will review all the measures received under the 

contract to determine if they are applicable to their health care quality 

programs or initiatives. ASPE’s comments also noted that our draft report 

did not include information on all NQF-endorsed measures used by the 

various agencies within HHS. As noted in the draft report, we relied on 

HHS to identify programs and initiatives across HHS that use or plan to 

use these health care quality measures and recognize that those included 

in our report may not represent a comprehensive list of all health care 

quality programs and initiatives. As we recommended in our report, 

having a comprehensive plan could help HHS identify programs or 

initiatives that use or plan to use health care quality measures, including 

those endorsed by NQF.
 

Page 42 GAO-12-136 NQF Contract Performance

NQF’s comments state that it is providing its services to HHS under a 

cost reimbursement contract, which is used in circumstances where 

aspects of performance, such as time frames, cost estimates, and scope 

of work, cannot be reasonably estimated, and therefore, should not be 

expected. As noted in the draft report, the contract type used for this work 

is used for efforts such as research, design, or study efforts where costs 

and technical uncertainties exist and it is desirable to retain as much 

flexibility as possible in order to accommodate change. However, the draft 

report also noted that this type of contract provides only a minimum 

incentive to the contractor to control costs. Given the risk associated with 

this type of contract, the fact that NQF has not met expected time frames 

on about half of its projects as of August 2011, and that NQF exceeded 

its initial cost estimates for some of its projects under its contract 

activities, it is especially important that HHS obtain detailed and timely 

information on NQF’s performance and use that information to inform any 

appropriate changes to time frames, projects, and cost estimates for the 

remaining contract years, as noted in our recommendations. NQF’s 

comments also state that time frames and costs for the work performed 

under the contract were initial estimates based on an early understanding 

of the work, that HHS and NQF understood that there would likely be 

changes to them as a result of the complexity and novelty of the work, 

and that they have worked collaboratively throughout the contract period 

to address these and other factors. As noted in the draft report, the final 

work plans, the technical proposal, and other documents that we 

reviewed included initial time frames for all projects and costs for the work 

performed during the contract year that were approved by HHS in 

collaboration with NQF. The draft report also notes several examples of 

reasons why the time frames and costs were modified over time. 

Contributing factors include the high volume of measures submitted, 

changes to the scope of work, and the novelty and complexity of the 

work.

We are sending copies of this report to the Secretary of Health and 

Human Services and other interested parties. In addition, the report will 

be available at no charge on GAO’s website at http://www.gao.gov. 

NQF Comments
 

Page 43 GAO-12-136 NQF Contract Performance

If you or your staff have any questions about this report, please contact 

me at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Office 

of Congressional Relations and Office of Public Affairs can be found on 

the last page of this report. Other major contributors to this report are 

listed in appendix VII.

Linda T. Kohn

Director, Health Care
Page 44 GAO-12-136 NQF Contract Performance

List of Committees

The Honorable Max Baucus

Chairman

The Honorable Orrin G. Hatch

Ranking Member

Committee on Finance

United States Senate

The Honorable Tom Harkin

Chairman

The Honorable Michael B. Enzi

Ranking Member

Committee on Health, Education, Labor, and Pensions

United States Senate

The Honorable Fred Upton

Chairman

The Honorable Henry A. Waxman

Ranking Member

Committee on Energy and Commerce

House of Representatives

The Honorable David Camp

Chairman

The Honorable Sander M. Levin

Ranking Member

Committee on Ways and Means

House of Representatives
Appendix I: Health Care Quality Measurement 

Framework

Page 45 GAO-12-136 NQF Contract Performance

Figure 5 illustrates a health care quality measurement framework of the 

various stages that a quality measure will go through, as described by the 

Department of Health and Human Services (HHS) and National Quality 

Forum (NQF) officials and others. These stages include measure 

development, endorsement, selection, and use, among others. This 

framework also shows examples of which entities, including HHS and 

NQF, are involved in each of the stages.

Appendix I: Health Care Quality 

Measurement Framework
Appendix I: Health Care Quality Measurement 

Framework

Page 46 GAO-12-136 NQF Contract Performance

Figure 5: Health Care Quality Measurement Framework

a

NQF officials told us that the development of data platforms allows measure developers to create 

quality measures directly in an electronic format as well as retool—that is, convert measures from a 

paper-based format to an electronic format. For example, NQF’s work to create a measure authoring 

tool provides measure developers with a tool to create standardized electronic measures that help 

capture information in electronic health records. NQF officials told us development of data platforms, 

such as through the use of the measure authoring tool, is increasingly important as NQF will begin to 

require that measure developers include electronic specifications in quality measures beginning in 

May 2012.
Appendix I: Health Care Quality Measurement 

Framework

Page 47 GAO-12-136 NQF Contract Performance

As an example of actions taken during the second stage of this quality 

measurement framework, HHS officials described two different processes 

used for planning and identifying gap areas.

The Centers for Medicare & Medicaid Services (CMS) Office of Clinical 

Standards and Quality1 has developed a standardized approach to 

identify quality measures that it uses in its health quality initiatives and 

programs using CMS’s Measures Management System.2 The Measures 

Management System requires the convening of a technical expert panel 

in the initial planning stage.3

In contrast, CMS’s Center for Medicaid, Children’s Health Insurance 

Program (CHIP), and Survey & Certification Office—the CMS center 

which implements CHIP—uses a measure identification process that 

relies on existing measures rather than development of new measures, 

according to officials. This office worked with a technical advisory group, 

Once convened, the technical expert panel is 

expected to work with measure developers who will gather information 

that will help the panel determine whether measures need to be 

developed for a program or initiative. During this stage, measure 

developers may conduct environmental scans or literature reviews, to 

determine the existence of measures that could be used for a program or 

initiative. If a measure does not exist, then the developer will work with 

CMS to develop the needed measures for the program or initiative, 

including measure testing. Upon development of the measures, the 

technical expert panel will evaluate them based on (1) importance to 

making significant gains in health care quality and improving health 

outcomes, (2) scientific acceptability of the measure properties including 

tests of reliability and validity, (3) usability, and (4) feasibility. Measures 

recommended by the panel are generally submitted for NQF 

endorsement.

 

1

CMS is an agency within HHS. 

2

The Measures Management System is used by divisions within CMS and provides 

guidance to CMS-contracted measure developers on the development and maintenance 

of measures. It was developed to help CMS manage an ever-increasing demand for 

quality measures to use in its various public reporting and quality programs as well as in 

value-based purchasing initiatives.

3

The technical expert panel consists of a group of recognized experts in relevant fields 

including clinicians, statisticians, quality improvement experts, methodologists, and 

pertinent measure developers who are recruited to provide input on the measure under 

development by the contract.

HHS’s Role in 

Planning and 

Identifying Gap Areas
Appendix I: Health Care Quality Measurement 

Framework

Page 48 GAO-12-136 NQF Contract Performance

the Subcommittee on Children’s Healthcare Quality Measures for 

Medicaid and CHIP, to recommend an initial core set of measures for the 

CHIP. With assistance from CMS, the subcommittee evaluated measures 

based on importance, validity, and feasibility. CMS officials told us that 

they considered existing NQF-endorsed and non-NQF-endorsed 

measures based on the measurement needs of the program, and relied 

on measure testing conducted by the measure developers. Officials 

stated that they have also relied on the subcommittee to evaluate 

candidate measures for Medicaid child health programs. Officials said 

that they are not required to submit measures that will be used for 

Medicaid programs for NQF endorsement.
Appendix II: How GAO Categorized NQF’s 16 

Tasks under the Contract into Nine Contract 

Activities

Page 49 GAO-12-136 NQF Contract Performance

From January 14, 2010, through August 31, 2011, the National Quality 

Forum’s (NQF) contract with the Department of Health and Human 

Services (HHS) included 16 tasks that NQF is required to perform. For 

purposes of our work, we categorized these tasks into nine contract 

activities. Specifically, in certain cases, we grouped activities that covered 

related areas of work into a single contract activity. For example, we 

consolidated the six administrative activities NQF is required to perform 

into a single contract activity. (See table 3 that shows how we 

consolidated these contract activities.)

NQF was required to perform specific projects under the nine contract 

activities we identified. For example, under the endorsement contract 

activity, NQF was required to complete an endorsement project related to 

patient outcome measures. For purposes of our work, we identified and 

reviewed 63 projects NQF is required to perform under the nine contract 

activities, as shown in appendix III.

Appendix II: How GAO Categorized NQF’s 16 

Tasks under the Contract into Nine Contract 

Activities
Appendix II: How GAO Categorized NQF’s 16 

Tasks under the Contract into Nine Contract 

Activities

Page 50 GAO-12-136 NQF Contract Performance

Table 3: GAO’s Categorization of the NQF Contract’s 16 Tasks into Nine Activities

NQF contract’s 16 tasksa Nine contract activities categorized by GAO 

1. Opening meeting between HHS and NQF 1. Administrative activityb

2. Development of the annual work plan 

3. Quality assurance (development of an internal evaluation 

plan and monthly call to discuss the plan)

4. Weekly conference calls 

5. Monthly progress reports 

6. Develop public website for project documentsc

7. Formulation of national strategy and priorities for health care 

quality measures, as specified by the Medicare 

Improvements for Patients and Providers Act of 2008 

(MIPPA) 

2. Recommendations on a National Strategy and Priorities for

 Quality Measurement (MIPPA)

8. Implementation of a consensus process for endorsement of 

health care quality measures (MIPPA)

3. Endorsement of Health Care Quality Measures (MIPPA)

9. Implementation of a consensus process for maintenance of 

previously endorsed health care quality measures (MIPPA)

4. Maintenance of Endorsed Quality Measures (MIPPA)

10. Promotion of electronic health records (MIPPA) 5. Promotion of the Development and Use of Electronic Health 

 Records (MIPPA) 

11. Submission of annual report to Congress and the Secretary 

of Health and Human Services (MIPPA)

12. Additional Requirements for Annual Report—Measure Gaps 

and Inadequacies and Input of the Multistakeholder Groups, 

as specified by the Patient Protection and Affordable Care 

Act (PPACA)

6. Annual Report to Congress and the Secretary of HHS (MIPPA 

 and PPACA)

13. Focused Measure Development, Harmonization, and 

Endorsement Efforts to Fill Critical Gaps in Performance 

Measurement (not specified by MIPPA or PPACA) 

7. Other Health Care Quality Measurement Activity Not Identified 

 in MIPPA or PPACA

14. Provision of Input on Priorities for the National Strategy for 

Quality Improvement in Healthcare (PPACA)

15. Input to the Annual National Quality Strategy (PPACA) 

8. Multistakeholder Input into HHS’s National Strategy for Quality

 Improvement in Health Care (as required by PPACA)

16. Selection of Quality Measures for Use in Payment Programs 

and Value-Based Purchasing Programs under PPACA, 

Other Private/Public Payers, and Other Programs (PPACA)

9. Multistakeholder Input on the Selection of Quality Measures for 

 use in Payment Programs and Value-Based Purchasing 

 Programs Under PPACA, Other Private/Public Payers, and 

 Other Programs (as required by PPACA)

Source: GAO analysis of MIPPA, PPACA, the NQF contract, the 2010 and 2011 final MIPPA work plans, and the 2011 PPACA 

technical proposal.

a

The numbering of the 16 tasks, or contract activities, does not reflect the numbering used under the 

NQF contract with HHS.

b

A number of activities NQF performs under the HHS contract are administrative and are not directly 

related to NQF’s work on health care quality measurement. Rather, these activities focus on project 

planning and contract management efforts.

c

Development of a public website was part of the 2010 final annual work plan but this work was 

canceled in 2011.
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 51 GAO-12-136 NQF Contract Performance

The tables below provide a status update on the projects that the National 

Quality Forum (NQF) is required to complete under the nine contract 

activities we identified (see app. II). The contract activities and the 

projects under the activities NQF is expected to perform are determined 

on an annual basis by the Department of Health and Human Services 

(HHS) and NQF. As a result, the number of projects under the contract 

activities varies by contract year. For our reporting period—January 2010 

through August 2011—we determined that NQF was required to conduct 

work on 63 projects under the contract activities we reviewed. To 

determine initial time frames for each project, we calculated the 

approximate time between expected start and end dates established in 

NQF’s 2009, 2010, and 2011 final annual Medicare Improvements for 

Patients and Providers Act of 2008 (MIPPA) work plans, the 2011 Patient 

Protection and Affordable Care Act (PPACA) technical proposal, and 

other NQF documents. Actual time frames were determined by 

calculating the approximate time between the actual start date and the 

actual date of completion. For projects that were not yet complete as of 

August 2011, we included an expected time frame based on the 

approximate difference between the actual start date and the expected 

date of completion. NQF and HHS officials stated that any changes to the 

initial time frames were approved by HHS.

Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each 

Activity GAO Identified
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 52 GAO-12-136 NQF Contract Performance

Table 4: Status of Projects under NQF’s Contract Activity to Provide Recommendations on a National Strategy and Priorities 

for Quality Measurement, as of August 31, 2011

Projects under contract activity to 

provide recommendations on a 

national strategy and priorities for 

quality measurement (MIPPA) Description of project Status 

Expected 

time frame 

(estimated 

months) 

Actual time 

frame 

(estimated 

months)

Formulate national strategies and 

priorities

Report that prioritizes 20 high-impact Medicare 

conditions and associated measure gaps.

 14 15

Evaluation of the uses of NQFendorsed measuresReport that investigates the use of NQF-endorsed 

measures.

 9 exp. 18

Measurement development and 

endorsement agenda

Report that sets a national measure development and 

endorsement agenda.

 10 10

Measurement framework for multiple 

chronic conditions 

Report that presents an analysis of measures being 

used to assess quality of care for people with multiple 

chronic conditions.

 14 exp. 13a

Advice to the Office of the National 

Coordinator on meaningful use of 

measures

Report that analyzes measures targeted for use in the 

Medicare Electronic Health Record Incentive Program, 

specifically examining how health information technology 

(IT) tools can improve the efficiency, quality, and safety 

of health care delivery. 

 2 2 

Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress 

and Internal Evaluation Plan (IEP) reports from January 2010 to August 2011.

Note: Status is defined by the following symbols. 

 = Completed  = In progress exp. = expected time frame

a

The initial time line for this work was June 2010 through August 2011, which is approximately 

14 months. However, due to delays in approving the subcontractor NQF engaged to perform the 

work, the project was not started on time. As a result, the modified time line for the project is April 

2011 through May 2012, which represents a delay of approximately 9 months.
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 53 GAO-12-136 NQF Contract Performance

Table 5: Status of Projects under NQF’s Contract Activity to Endorse Health Care Quality Measures, as of August 31, 2011

Projects under NQF’s contract 

activity to endorse health care 

quality measures (MIPPA)a Description of project Status 

Expected 

time frame 

(estimated 

months)

Actual time 

frame 

(estimated 

months)

Patient outcomes Endorsement of health care outcome measures related 

to 20 high-impact Medicare conditions identified by the 

Centers for Medicare & Medicaid Services (CMS), and 

mental health, and child health conditions. 

 14 13-19b

Patient safety Endorsement of patient safety measures and other 

projects related to patient safety, including a framework 

for public reporting of patient safety information, 

endorsement of serious reportable events (SRE), and 

collaboration with state-based reporting agencies.

 17 23, exp. 27c

Nursing homes Endorsement of performance measures for chronic and 

postacute care nursing facilities.

 10 17

Evaluation of Consensus 

Development Process

A comprehensive analysis of NQF’s endorsement 

process by Mathematica Policy Research, Inc., to 

identify areas of improvement related to timeliness and 

effectiveness.

 9d 18

Child health quality Endorsement of measures that could be used in public 

reporting for certain conditions or cross-cutting areas 

applicable to the Medicaid population.

 11 exp. 16e

Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress 

and IEP reports from January 2010 to August 2011.

Note: Status is defined by the following symbols.

 = Completed  = In progress exp. = expected time frame

a

HHS officials told us that they consider an NQF measure endorsed when the board ratifies the 

measures. However, NQF is required to submit a final report to HHS following ratification to complete 

the project. For purposes of our report, we consider the project completed when the board endorses 

the measures.

b

The patient outcomes project was conducted in three phases. Phases I and II were initially expected 

to be completed in 13 months; however, the project was completed in 19 months. NQF officials told 

us a high volume of measures contributed to the delays. Phase II was initially expected to take 

15 months but it took 17 months.

c

As of August 2011, NQF endorsed one patient safety measure under phase II of the endorsement 

project and expects to complete this phase in September 2011 following the review of an additional 

patient safety measure. Phase I is expected to be completed by December 2011.

d

In the 2009 final work plan, this project was expected to be completed between October 2009 and 

July 2010, or approximately 9 months. NQF officials stated the delays in completing this work were a 

result of additional time needed for data collection efforts, among other things.

e

In August 2011, NQF announced endorsement of 41 out of 44 child health quality measures under 

the child health quality endorsement project. The remaining 3 measures are still under consideration 

and are scheduled to be reviewed for endorsement in September 2011.
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 54 GAO-12-136 NQF Contract Performance

Table 6: Status of Projects under NQF’s Contract Activity to Maintain Previously Endorsed Health Care Quality Measures, as 

of August 31, 2011

Projects under NQF’s contract 

activity to maintain health care 

quality measures (MIPPA)a Description of project Status 

Expected 

time frame 

(estimated 

months)

Actual time 

frame 

(estimated 

months)

Updated 3-year endorsement 

maintenance cycle review process 

projects (6 projects)

Six maintenance projects using NQF’s process that 

reviews previously endorsed measures across 22 topic 

areas every 3 years using three measure maintenance 

cycles.b The projects relate to the following conditions: 

• Cardiovascular

• Surgical

• Cancer

• Pulmonary/critical care

• Perinatal

• Renal care 

 Variousc All ongoing

3-Year maintenance cycle review 

prior to updated process 

(3 projects)

Three maintenance projects conducted under the old NQF 

maintenance review process related to the following 

conditions:

• Diabetes

• Mental health

• Musculoskeletal conditions 

 No specific 

time framed 20

Time-limited endorsement review Maintenance review of measures that have been granted a 

time-limited endorsement—measures that require 

additional testing to be conducted by measure developers 

within 12 months of endorsement.

 No specific 

time framee No specific 

time frame 

Annual measure maintenance 

update

Review of measures submitted by measure developers to 

determine whether the measure has undergone any 

change.

 Variousf Ongoing

Ad hoc review Review of measures that were initiated during the contract 

year by request and based on justifiable evidence to 

substantiate the review.

 n/a n/ag

Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress 

and IEP reports from January 2010 to August 2011.

Note: Status is defined by the following symbols.

 = Completed  = In progress n/a = no time frame was established

a

In addition to the 3-Year Maintenance Review cycles, NQF can also maintain measures through 

(1) annual updates, (2) ad hoc requests, and (3) time-limited endorsement reviews. As of August 

2011, NQF has maintained 3 measures under ad hoc requests and 80 measures under time-limited 

endorsement reviews. NQF officials told us they also began annual updates for 59 measures during 

our reporting period.
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 55 GAO-12-136 NQF Contract Performance

b

NQF has developed three cycles of review—cycles A, B, and C. Cycle A focuses on measures 

related to cardiovascular; surgery; prevention; endocrine; GU/GYN; mental health; and 

musculoskeletal disorders. Cycle B focuses on measures related to cancer; pulmonary/critical care; 

palliative and end-of-life care; Perinatal; renal; safety; disparities; and care coordination. Cycle C will 

focus on health care infrastructure; head/eye/ear/nose/throat conditions; infectious disease; 

neurology; patient experience and engagement; functional status; and gastrointestinal. NQF is 

expected to complete Cycle A by the end of 2010, Cycle B by the end of 2011, and Cycle C by the 

end of 2012. As of August 2011, the cardiovascular and surgery measures from Cycle A are still 

under review.

c

As of August 2011, NQF was conducting six maintenance projects under its updated 3-year review 

cycle. The initial and estimated time frames for these projects vary by project. For example, review for 

phases I and II of the cardiovascular measures is expected to take 15 months and review for phase I 

of the surgical measures is expected to take 13 months and phase 2 is expected to take 16 months. 

NQF also initiated the review of renal measures in May 2011 and was initially expected to take 

7 months; however, the time line has been extended to 8 months. As of November 2011, NQF has 

issued a call for measures for Perinatal measures and expects to issue a call for cancer and 

pulmonary/critical care measures in November 2011 and January 2012, respectively.

d

NQF officials stated that no estimated time frame for completion was established for these three 

projects. Although officials stated they anticipated the length of the review to be similar to other NQF 

projects, they encountered delays that extended the time frame, such as a high volume of measures 

under review.

e

NQF officials provided us with a time line for completing time-limited reviews that included an initial 

start date of January 2011 but did not provide specific expected dates of completion.

f

NQF officials told us that the annual reviews did not begin until 2011 and are conducted in cycles.

g

NQF officials told us that there is no time line established for ad hoc reviews because requests for 

ad hoc reviews will be considered by NQF on a case-by-case basis and can be made at any time.
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 56 GAO-12-136 NQF Contract Performance

Table 7: Status of Projects under NQF’s Contract Activity to Promote the Development and Use of Electronic Health Records, 

as of August 31, 2011

Project under NQF’s 

contract activity to 

promote the development 

and use of electronic health 

records (MIPPA) Description of project Status 

Expected 

time frame 

(estimated 

months)

Actual time 

frame 

(estimated 

months)

Measure authoring tool Development of a software tool that measure developers 

will use to create standardized eMeasures.

 15 exp. 28

Clinical decision support Report that provides a framework for Clinical Decision 

Support rules and key elements for health IT systems to 

help ensure compatibility and functionality with the 

Quality Data Model.

 14 17

eMeasure creation and testing Establish a uniform standard for electronic measures.  13 13

Quality data model Maintenance and revision of a health IT model that 

allows data from electronic health records to be collected 

and used in quality measurement.

 28 exp. 28a

Health information technology (IT) 

utilization expert panel

Expert panel convened to examine the information 

needed to measure effective health IT use. 

 12 14

Initial measure retooling “Retooling”—that is, converting 113 NQF-endorsed 

health care quality measures from paper-based format to 

an eMeasure format.

 19 22

Update of retooled measures Publish a list of updated versions of the initial 113 

measures under the contract that describes any changes 

made to the measures, including those identified during 

public comment and by a review panel. 

 10 exp. 10

eMeasure syntactical review panel Expert panel that conducts a thorough review of the 

retooled measures to help ensure that a measure’s 

intent remains intact for continued NQF endorsement.

 9 15, exp. 19b

Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress 

and IEP reports from January 2010 to August 2011.

Note: Status is defined by the following symbols.

 = Completed  = In progress exp. = expected time frame

a

NQF officials told us they anticipate a number of versions will be published annually under the 

contract to allow for ongoing review and updates. NQF expects to publish an updated version for 

public use in January 2012.

b

In June 2011, the review panel completed its review of the 113 retooled measures and expects to 

publish a report that synthesizes comments on and updates to the measures by October 2011. NQF 

officials told us that a request by HHS to expand the scope of this work extended the time line by 

3 months.
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 57 GAO-12-136 NQF Contract Performance

Table 8: Status of Projects under NQF’s Contract Activity to Submit an Annual Report to Congress and the Secretary of HHS, 

as of August 31, 2011

Project under NQF’s 

contract activity to submit 

an annual report to Congress 

and the Secretary of HHS 

(MIPPA and PPACA) Description of project Status 

Expected 

time frame 

(estimated 

months)

Actual

time frame 

(estimated 

months)

Report due March 1, 2010 Summary of NQF’s progress on activities under the contract 

during the first contract year (2009).

 12 12

Report due March 1, 2011 Summary of NQF’s progress on activities under the contract 

during the second contract year (2010).

 13 12

Report due March 1, 2012 Summary of NQF’s progress on activities under the contract, 

including a report on gaps in endorsed and nonendorsed 

health care quality measures and a summary of activities 

conducted by multistakeholder groups during the third 

contract year (2011).

 13 exp. 12

Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress 

and IEP reports from January 2010 to August 2011.

Note: Status is defined by the following symbols.

 = Completed  = In progress exp. = expected time frame
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 58 GAO-12-136 NQF Contract Performance

Table 9: Status of Projects under NQF’s Contract Activity to Provide Multistakeholder Input into HHS’s National Strategy for 

Quality Improvement in Healthcare, as of August 31, 2011

Project under NQF’s 

contract activity to provide 

multistakeholder input to 

HHS’s National Quality 

Strategy (PPACA) Description of project Status 

Expected 

time frame 

(estimated 

months)

Actual time 

frame 

(estimated 

months)

Recommendations on HHS’s 

National Strategy for Quality 

Improvement in Healthcare 

(2010)

Multistakeholder input on national priorities related to 

improvement in population health and in delivery of health care 

services that HHS will consider when developing the National 

Quality Strategy issued in 2011.

 1 1 

Provision of input on priorities 

for HHS’s annual National 

Quality Strategy

Multistakeholder input on a plan for measuring and improving 

health and health care with a focus on the six priorities 

identified in HHS’s National Quality Strategy.

 5 exp. 6

Provision of input on 

Partnership for Patientsa Quarterly webinars, facilitated by the National Priorities 

Partnership (NPP), for purposes of gathering input issues 

related to HHS’s Partnership for Patients initiative.

 7 exp. 5

Care coordination A two-phase project that will include an endorsement project 

related to prioritized gap areas related to care coordination, 

such as transitions across settings and providers; and (2) two 

reports that will address issues related to care coordination 

measures, such as readiness of health information technology 

systems.

 Phase 1: 6

Phase 2: 13 

Phase 1: 

exp. 10

Phase 2: 

exp. 15

Palliative and end-of-life care Endorsement project for individual and composite measures or 

cross-cutting measures related to palliative and end-of life care.

 9 exp.. 9

Patient safety Initiative that includes (a) an endorsement project for patient 

safety complications measures; (b) work related to SRE 

codification; and (c) development of an integrated strategy for 

high-profile areas in patient safety.

 (a) 9 or 10, 

to be 

determinedb

(b) 7

(c) 6

(a) exp.9

(b) exp. 8

(c) exp. 6

Population health Endorsement project for (a) preventive services and population 

health measures and (b) a report that provides information 

related to key topic areas related to population health, including 

applicable NQF evaluation criteria.

 (a) 9

(b) 5

(a) exp. 9

(b) exp. 6

Patient reported outcomes Two reports—one that addresses issues related to use of 

patient reported outcomes measures, including methodological 

issues, and another that discusses information needed to 

prepare measures for the endorsement process. 

 9 exp. 9

Source: GAO analysis of NQF’s contract with HHS, the 2011 PPACA technical proposal, and NQF’s monthly progress and IEP reports 

from January 2010 to August 2011.

Note: Status is defined by the following symbols.

 = Completed  = In progress exp. = expected time frame

a

In April 2011, HHS launched the Partnership for Patients, which is a national partnership with the 

goal to save 60,000 lives by improving patient safety through reduction in preventable injuries, 

complications, and hospital readmissions over the next 3 years.
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 59 GAO-12-136 NQF Contract Performance

b

NQF officials told us that they expect the time lines for the endorsement project to be extended 

because the project will be conducted in phases to accommodate additional measures. However, as 

of August 2011, NQF officials told us they are still finalizing the scope of work for this project, 

including determining if there will be any impact on the overall time line.

Table 10: Status of Projects under NQF’s Contract Activity to Provide Multistakeholder Input on Selection of Quality 

Measures, as of August 31, 2011

Project under NQF’s 

contract activity to provide 

multistakeholder input on 

selection of quality measures 

(PPACA) Description of project Status 

Expected 

time frame 

(estimated 

months)

Actual time 

frame 

(estimated 

months)

Multistakeholder prerulemaking

input

Multistakeholder input provided annually by the Measure 

Applications Partnership (MAP) to HHS on the selection 

of measures to be implemented through the federal 

rulemaking process for various programs.a

 11 exp. 11

Measures for use in improvement 

of quality reporting programs

Multistakeholder input provided to HHS by the MAP to 

help develop a coordination strategy for performance 

measurement and identify health care quality measures 

for use in four quality reporting programs: (a) physicians, 

(b) postacute care programs under Medicare, (c) PPSexempt cancer hospitals, and (d) hospice care.

 (a) 7

(b) 11

(c) 15

(d) 15

(a) exp. 7

(b) exp. 11

(c) exp. 15

(d) exp. 15

Measures for dual eligible 

beneficiaries

Multistakeholder input provided to HHS by the MAP on a 

measurement strategy and potential new health care 

quality measures that address quality issues identified 

for dual eligible beneficiaries, such as multiple chronic 

conditions.

 15 exp. 15

Measurement strategy for 

readmissions and healthcareacquired conditions (HAC)

Multistakeholder input provided to HHS by the MAP on a 

coordination strategy for readmission and HAC 

measurement across public and private payers.

 18 exp. 17

Source: GAO analysis of NQF’s contract with HHS, the 2011 PPACA technical proposal, and NQF’s monthly progress and IEP reports 

from January 2010 to August 2011.

Note: Status is defined by the following symbols.

 = In progress exp. = expected time frame

a

The MAP will provide input on measures beginning February 1, 2012, based on a list of measures 

published on December 1 of each year by the Secretary of HHS. The 11 programs included in this 

prerulemaking input are: hospice, hospital inpatient, hospital outpatient, physicians’ offices, cancer 

hospitals, end-stage renal disease (ESRD) facilities, inpatient rehabilitation facilities, long-term care 

hospitals, CMS hospital value-based purchasing, psychiatric hospitals, and home health care.
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 60 GAO-12-136 NQF Contract Performance

Table 11: Status of Projects under NQF’s Other Health Care Quality Measurement Activity Not Identified in MIPPA or PPACA, 

as of August 31, 2011

Project under NQF’s 

other health care 

quality measurement 

contract activity Description of project Status 

Expected time 

frame 

(estimated 

months)

Actual time 

frame 

(estimated 

months)

Imaging efficiency measures Endorsement project for imaging efficiency measures 

because Medicare spends approximately $14 billion 

annually on outpatient imaging studies.

 8 15-18a

Resource-use measures Endorsement project for measures to help assess the 

cost of health care services provided.

 12-14b exp.11-14

Resource-use white paper White paper on resource-use measures that will inform a 

2011 endorsement project on this topic. 

 9 exp. 22

Geographic-level efficiency 

measurement white paper

White paper on geographic-level efficiency measures that 

are used for quality measure reporting at the geographic 

and population level.

- - canceled (June 2010)

Harmonization Report that provides operational guidance related to 

harmonization to be used in future NQF consensus 

development projects.

 9 9c

ICD-10 conversiond Report that examines implications of the transition from 

an HHS code set that is used to classify health care on 

NQF’s measure maintenance process and provides 

recommendations to assist measure developers and 

NQF during the transition.

 11 15

Measure developmente Maintenance of subcontracts with eight measure 

developers to allow for development of critical measures 

as needed.

 n/a

Regionalized emergency medical 

care services

Identification of existing quality measures and gap areas 

related to measurement of regionalized emergency care 

services to inform creation of a framework to guide 

measure development in this area.

 14 exp. 18

Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress 

and IEP reports from January 2010 to August 2011.

Note: Status is defined by the following symbols.

 = Completed  = In progress  = Not started - - = Canceled exp. = expected time frame

n/a = no time frame was established

a

The imaging efficiency measures endorsement project was initially expected to endorse measures 

by July 2010; however, due to measure harmonization efforts, NQF divided the endorsement review 

in two phases. The first six measures were endorsed in February 2011. In May 2011, NQF issued a 

second report announcing endorsement of another measure.

b

The resource use endorsement project will be conducted in two cycles—cycle 1 is expected to be 

completed in 11 months and cycle 2 is expected to be completed in 14 months.

c

The initial time line for completing this project was February 2010 through November 2010; however, 

the actual time line for completion was from March 2010 through December 2010, which delayed 

delivery of the report by 1 month.
Appendix III: Status of NQF’s Nine Contract 

Activities and the Projects under Each Activity 

GAO Identified

Page 61 GAO-12-136 NQF Contract Performance

d

HHS uses the International Classification of Diseases, Ninth Revision, Clinical Modification 

(ICD-9-CM) code sets to classify health care. In 2013, HHS will transition to the Tenth Revision, or 

ICD-10-CM/PCS. The ICD-10 update activity aims to help HHS prepare for this transition by 

examining the impact of the change on measure development and submitting measures for NQF 

endorsement.

e

HHS officials stated that HHS has never implemented this project and likely will not do so through 

the remainder of the contract period.

Table 12: Status of Projects under NQF’s Administrative Contract Activity

Project under NQF’s 

administrative 

contract activity Description of project Status 

Expected time 

frame 

(estimated 

months)

Actual time 

frame 

(estimated 

months)

Opening meeting with HHS Meeting between NQF and HHS officials at the 

beginning of the contract year to review activities to be 

performed during for the year.

 This is a single 

event held 

annually. 

2010 annual work plan Development of a draft and final work plan that outlines 

specific activities to be performed during the 2010 

contract year.

 3 4 

2011 annual work plan Development of a draft and final work plan that outlines 

specific activities to be performed during the 2011 

contract year related to NQF’s activities funded by 

MIPPA. 

 5 6 

Quality assurance and monthly 

Internal Evaluation Plan (IEP) 

reports

Monthly report compiled by NQF that includes process 

and outcome measures to help monitor NQF’s 

progress under the contract.

 All completed in reporting 

period, ongoing annually 

Weekly conference call with 

HHS

Weekly conference call with HHS and NQF to go over 

projects under each activity.

 Not all held in reporting 

period,a ongoing annually

Monthly progress report Monthly report compiled by NQF that includes a 

summary for each activity and the status of project 

planning, implementation, management, quality 

assurance, and evaluation that has occurred during the 

previous month’s contract period.

 All completed in reporting 

period, ongoing annuallyb

Public website Overhaul and maintenance of NQF’s website to allow 

measure developers, members, and the public easier 

access to relevant documents.

- - canceled (February 2011)

Source: GAO analysis of NQF’s contract with HHS, the 2010 and 2011 final MIPPA annual work plans, and NQF’s monthly progress 

and IEP reports from January 2010 to August 2011.

Note: Status is defined by the following symbols.

 = Completed - - = Canceled

a

NQF officials told us that this meeting did not occur each week, as scheduled, because in some 

instances those involved determined it was unnecessary.

b

From January 2010 to June 2010, NQF was required to submit its monthly progress reports by the 

1st of every month; however, it did not submit these reports until the 15th of the month. In June 2010, 

HHS modified the due date to the 15th, beginning in July 2010. Since that time, all reports have been 

submitted on time.
Appendix IV: Summary of Projects to Review 

the National Quality Forum’s Endorsement 

Process

Page 62 GAO-12-136 NQF Contract Performance

As part of a project under its contract with the Department of Health and 

Human Services (HHS), NQF was required to review its endorsement 

process. To complete this project, the National Quality Forum (NQF) 

subcontracted with Mathematica Policy Research, Inc. (Mathematica), to 

conduct a review of NQF’s endorsement process, as requested by HHS. 

HHS officials stated that, given the importance of the endorsement 

process as part of the health care quality measurement framework, they 

requested that an objective and thorough review of NQF’s endorsement 

process that focused on timeliness, efficiency, and effectiveness should 

be conducted. For example, they stated that they were interested in 

whether there were any efficiencies that could be implemented to shorten 

the process while maintaining an objective review of the health care 

quality measures that were evaluated under the process. Mathematica 

initiated its review of NQF’s endorsement process in October 2009 and 

completed the work in December 2010.

In December 2010, Mathematica submitted a final report to NQF and 

recommended eight areas where improvements could be made and 

inefficiencies could be addressed in the endorsement process. In the final 

report, Mathematica noted that the current process is lengthy and the 

timeliness of the endorsement projects varies substantially. The report 

further noted that the length of the endorsement process affects the 

availability of endorsed measures for end users, such as HHS. To help 

reduce the time required to complete projects, Mathematica 

recommended that NQF create a schedule for its endorsement process 

for measure developers and develop feasible time lines that include clear 

goals for each endorsement project.

As of May 2011, NQF officials stated that NQF has taken steps or plans 

to take steps in its future projects to address the eight areas for 

improvement Mathematica identified. For example, as of May 2011, NQF 

has solicited measures earlier based on a tentative annual project 

schedule to reduce the time lines of its endorsement process and 

reduced the period for voting by NQF member organizations from 30 to 

15 days. NQF officials stated that they believe their efforts to implement 

the recommendations will shorten the time lines for the endorsement 

projects by 3 to 4 months without compromising the integrity of the 

endorsement process and measures to be evaluated under the process. 

HHS officials stated Mathematica’s recommendations were valuable 

because much of the work under the NQF contract needs to be 

completed in an accelerated time line to help fill critical measurement 

gaps associated with HHS’s health care quality programs and initiatives. 

They noted that it is too soon to tell the effects of these changes on the 

Appendix IV: Summary of Projects to Review 

the National Quality Forum’s Endorsement 

Process
Appendix IV: Summary of Projects to Review 

the National Quality Forum’s Endorsement 

Process

Page 63 GAO-12-136 NQF Contract Performance

endorsement process, but they plan to monitor implementation of the 

changes in NQF’s 2011 endorsement projects under the contract. In 

addition, as of September 2011, HHS approved a new project under the 

contract to identify how the endorsement process can best align with 

HHS’s time frame for needed measures. As part of this project, NQF is 

expected to work with a consulting group to identify key performance 

metrics and define milestones and time lines to help streamline its 

endorsement process.
Appendix V: Comments from the Department 

of Health and Human Services

Page 64 GAO-12-136 NQF Contract Performance

Appendix V: Comments from the Department 

of Health and Human Services
Appendix V: Comments from the Department 

of Health and Human Services

Page 65 GAO-12-136 NQF Contract Performance
Appendix V: Comments from the Department 

of Health and Human Services

Page 66 GAO-12-136 NQF Contract Performance
Appendix V: Comments from the Department 

of Health and Human Services

Page 67 GAO-12-136 NQF Contract Performance
Appendix V: Comments from the Department 

of Health and Human Services

Page 68 GAO-12-136 NQF Contract Performance
Appendix VI: Comments from the National 

Quality Forum

Page 69 GAO-12-136 NQF Contract Performance

Appendix VI: Comments from the National 

Quality Forum
Appendix VI: Comments from the National 

Quality Forum

Page 70 GAO-12-136 NQF Contract Performance
Appendix VI: Comments from the National 

Quality Forum

Page 71 GAO-12-136 NQF Contract Performance
Appendix VI: Comments from the National 

Quality Forum

Page 72 GAO-12-136 NQF Contract Performance
Appendix VI: Comments from the National 

Quality Forum

Page 73 GAO-12-136 NQF Contract Performance
Appendix VII: GAO Contact and Staff 

Acknowledgments

Page 74 GAO-12-136 NQF Contract Performance

Linda T. Kohn, (202) 512-7114 or kohnl@gao.gov

In addition to the contact named above, Will Simerl, Assistant Director; 

La Sherri Bush; Krister Friday; Amy Leone; Carla Lewis; John Lopez; 

Elizabeth Martinez; Lisa Motley; Teresa Tucker; Carla Willis; and William 

T. Woods made key contributions to this report.

Appendix VII: GAO Contact and Staff 

Acknowledgments 

GAO Contact

Staff 

Acknowledgments

(290911)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (www.gao.gov). Each weekday afternoon, 

GAO posts on its website newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube. 

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. 

Visit GAO on the web at www.gao.gov. 

Contact:

Website: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations 

Public Affairs

Please Print on Recycled Paper.