Health Care Quality Measurement: The National Quality Forum Has Begun a 4-Year Contract with HHS
Released: 2010-07-14T00:00:00

Letter

Background

NQF is a nonprofit organization established in 1999 that fosters agreement on national standards for measurement and public reporting of health care performance data. Its membership includes more than 400 organizations that represent multiple sectors of the health care system, including providers, consumers, and researchers. NQF uses a consensus development process to evaluate and endorse consensus standards, including quality measures, best practices, frameworks, and reporting guidelines. NQF has endorsed over 600 quality measures in 27 areas, such as cancer and diabetes. NQF endorses quality measures developed by other organizations, such as the Joint Commission, the National Committee for Quality Assurance, and the American Medical Association, rather than developing quality measures itself. HHS has used a number of NQF-endorsed measures in initiatives to promote quality measurement, and NQF continues to endorse quality measures separate from this contract.

Duties Established in MIPPA

MIPPA established five duties related to the use of quality measures. See table 1 for a description of the duties.

NQF Contract

For the NQF contract, HHS selected a cost-plus-fixed-fee contract—NQF’s first cost-reimbursement contract. Under the cost-plus-fixed-fee contract, HHS will reimburse NQF for costs incurred under the contract in addition to a fixed fee that is paid regardless of other costs. Cost-plus-fixed-fee contracts are used for efforts such as research, design, or study efforts where cost and technical uncertainties exist and it is desirable to retain as much flexibility as possible in order to accommodate change. However, this type of contract provides only a minimum incentive to the contractor to control costs. As we reported in 2009, these contracts are suitable when the cost of work to be done is difficult to estimate and the level of effort required is unknown.

Under the FAR, cost-reimbursement contracts may only be used when the contractor’s accounting system is adequate for determining costs under the contract to help prevent situations where contractors bill the government for unallowable costs. One method an agency can use to determine if an accounting system is adequate is to perform a preaward survey of a potential contractor’s accounting system prior to awarding a contract. This review serves as a key control to determine whether the potential contractor has an adequate accounting system in place to accurately and consistently record costs and submit invoices for costs. HHS conducted two preaward surveys of NQF’s accounting system. HHS’s initial review, in November 2007, found that NQF’s accounting system was inadequate because the system could not identify and separate unallowable costs, among other issues. NQF subsequently replaced its accounting system, and a second HHS review in November 2008 found that the system was adequate.

Under the FAR, contracts are to contain provisions for agency approval of a contractor’s subcontracts. HHS’s contract with NQF contains this provision and also requires the approval of consultants. This review requires appropriate support documentation provided by the contractor to the agency, including a description of the services to be subcontracted, the proposed subcontract price, and a negotiation memo that reflects the principal elements of the subcontract price negotiations between the contractor and subcontractor.

Two HHS components are principally responsible for administering the NQF contract: the office of the Assistant Secretary for Planning and Evaluation (ASPE) and the Centers for Medicare & Medicaid Services (CMS)—an operational division within HHS. To conduct oversight of the NQF contract, HHS assembled staff in these two units with experience in acquisitions, contract management, and program management. Specifically, the project officer for the NQF contract, responsible for program management and performance assessment, is a representative of ASPE. The contracting officer for the NQF contract, responsible for administering the contract, is a representative of CMS. The contracting officer and project officer should perform a comprehensive review of contractor invoices to determine if the contractor is billing costs in accordance with the contract terms and applicable government regulations.

NQF Has Begun Work for Each of the Five Duties Required by MIPPA Related to Quality Measures

As of January 13, 2010—the end of the first year of HHS’s 4-year contract with NQF to implement the MIPPA duties—NQF had begun work for each of the five duties required by MIPPA related to health care quality measures: (1) make recommendations on a national strategy and priorities; (2) endorse quality measures; (3) maintain endorsed quality measures; (4) promote electronic health records; and (5) report annually to Congress and the Secretary of HHS. While NQF began work for each of the duties in the first contract year, HHS determines on an annual basis the specific work NQF will be expected to perform under the five MIPPA duties in each contract year.

Recommendations on a National Strategy and Priorities for Quality Measurement. NQF has taken steps to begin the duty of making recommendations on a national strategy and priorities for quality measurement. In October 2009, NQF established a committee of stakeholders that is expected to develop recommendations about a national strategy and priorities for quality measurement. NQF published the recommended priorities in May 2010. The committee’s recommendations are expected to be based on a synthesis of evidence that NQF has collected, using a subcontractor, on 20 conditions that account for the majority of Medicare’s costs. The subcontractor collected evidence on existing quality measures for these conditions and identified gaps where quality measures did not exist. The subcontractor also collected evidence related to each condition, such as information on each condition’s prevalence, treatment costs, variability in providers’ treatment of the condition, disparities in treatment for patients with the condition, and potential to improve quality of care for the condition. The committee is expected to consider this evidence when developing recommendations on a national strategy and priorities for quality measurement. Under PPACA, NQF’s recommendations on a national strategy and priorities must be considered by HHS when it develops a national strategy for quality improvement, which HHS is required to submit to Congress by January 1, 2011.

Endorsement of Measures. NQF has taken steps to provide for the endorsement of quality measures. Prior to its contract with HHS, NQF established a process for endorsing quality measures. Under this process, organizations that develop quality measures submit them to NQF for consideration, in response to specific solicitations by NQF. NQF forms a committee of experts from its member organizations as well as other organizations and agencies to review these quality measures against NQF- established criteria, such as the usability and feasibility of the measure. After this committee evaluates the measures against these criteria, NQF’s process allows for a period during which its member organizations and the public may comment on the committee’s recommendation for each measure. The process also provides for a period for its member organizations to vote on whether the measures should be endorsed by NQF as a national standard. Ultimately, NQF’s board of directors makes a final decision on whether NQF should formally endorse the measures. (See app. I for a detailed description of NQF’s endorsement process.)

In order to provide for the endorsement of quality measures under this duty, NQF has taken several steps. Specifically, NQF initiated projects and solicited measures to be endorsed using its process for each of these projects. These projects relate to quality measurement in nursing homes, patient safety, and patient outcomes, and are scheduled to be completed between December 2010 and May 2011. In addition to endorsing measures, NQF also hired a subcontractor to evaluate its endorsement process and recommend ways to improve its efficiency and effectiveness. The subcontractor’s report and NQF’s approval of proposed enhancements to the process are due January 2011.

Maintenance of Endorsed Quality Measures. NQF has taken steps to ensure that endorsed measures are maintained—that is, updated or retired. Prior to its contract with HHS, NQF established a process for maintenance of measures. According to NQF, once a quality measure has been endorsed, updated information on the measure’s specifications should be submitted to NQF annually and the measure should be comprehensively reviewed under the maintenance process every 3 years. NQF’s maintenance process is similar to NQF’s endorsement process, in that it involves a review of measures against NQF-established criteria, a period for public comment, and a final decision by NQF’s board of directors. In order to implement this process under its contract with HHS, NQF began maintenance reviews for 191 measures in 14 areas such as diabetes and cardiovascular care. The measures were identified by HHS as being of interest to, or actually used by, HHS programs. By the end of the first contract year, NQF had not determined completion dates for maintenance of the 191 measures. As of May 2010, maintenance of the 191 measures identified by HHS is scheduled to be completed by the end of 2012.

Promotion of the Development and Use of Electronic Health Records. NQF has taken steps towards completing the duty of promoting the development and use of electronic health records for use in quality measurement. As of January 13, 2010, NQF had begun to implement a framework that defines a standardized set of data that should be captured in patients’ electronic health records. The framework, known as the Quality Data Set (QDS), is intended to allow data from electronic health records to be collected and used in quality measurement. Implementation and maintenance of the QDS is scheduled to continue through the end of the 4-year contract, which ends January 13, 2013. To further promote the development and use of electronic health records in quality measurement, NQF began additional activities. For example, NQF established a panel of experts to recommend additional capabilities to measure utilization. According to NQF officials, efforts under this duty are scheduled for completion between March 2010 and January 2013.

Annual Report to Congress and the Secretary of HHS. NQF submitted its first annual report to Congress and the Secretary of HHS on March 1, 2009. HHS published this report, with its comments, in the Federal Register on September 10, 2009. NQF submitted its second annual report, which also covers activities it performed during the first contract year, to Congress and the Secretary on March 1, 2010.

While NQF has begun work for each of the duties in the first contract year, HHS determines on an annual basis the specific work NQF will be expected to perform under the five MIPPA duties each contract year. Specifically, HHS gives direction for and then approves annual plans that NQF develops. These plans can include work begun in prior contract years that has not been completed. HHS can adjust work in the annual plans in support of each of the five duties. For example, HHS officials told us that in future contract years, they may select additional projects for the endorsement of quality measures, and additional measures for maintenance reviews.

NQF Reported about $6.5 Million in Costs and Fixed Fees for the First Contract Year

NQF reported costs and fixed fees totaling approximately $6.5 million for the first year of its contract with HHS, which ended January 13, 2010. The amount NQF reported included direct and indirect costs, as well as fixed fees. Direct costs, which are costs incurred specifically for this contract, represented the largest percentage—about $3.2 million, or 49 percent—of the amount NQF reported (see fig. 1). NQF’s reported direct costs were largely labor costs for NQF employees and payments to subcontractors and consultants. In addition to direct costs, NQF reported about $2.9 million in indirect costs for the first contract year. Indirect costs cover additional items, such as employee benefits, overhead, and administrative costs. NQF calculates its indirect costs based on a formula that takes into account an indirect-cost rate approved by HHS and the amounts of certain direct costs. For example, the formula estimates indirect costs such as employee benefits by multiplying an indirect-cost rate by the amount of direct costs for labor. Finally, in addition to its direct and indirect costs, NQF reported fixed fees of approximately $360,000 during the first contract year. HHS pays these fixed fees to NQF in addition to reimbursing the organization for its costs.

Of the approximately $6.5 million in costs and fixed fees NQF reported for the first contract year, most were incurred in the second half of the contract year. Costs and fixed fees in the second half of the contract year, from July 1, 2009, to January 13, 2010, totaled over $5 million. NQF staff told us that costs in the first half of the contract year were primarily for activities such as development of solicitations for subcontractors. Costs in the second half of the contract year were primarily for activities related to quality measurement, such as endorsement of quality measures and promotion of electronic health records for use in quality measurement.

NQF and HHS Rely on Reviews of NQF Invoices in Order to Help Ensure That NQF’s Reported Costs Are Proper

NQF reviews invoices and carries out other activities prior to submitting them to HHS in order to help ensure that reported costs are proper. HHS requires its officials to follow certain procedures when reviewing these invoices.

NQF Reviews Invoices and Carries out Other Activities in Order to Help Ensure That Its Reported Costs Are Proper

NQF officials told us their organization has several ways to help ensure that the contract costs it reports to HHS are proper. According to NQF officials, invoices are electronically generated using NQF’s accounting system and then reviewed before submitting the invoices to HHS for payment. These reviews are conducted by two senior staff—the NQF Project Director, who manages the contract, and the Chief Financial Officer. These officials meet to review costs reported in each month’s invoice. NQF officials told us that as part of their reviews, the two officials compare the current month’s invoice to the previous month’s invoice to identify discrepancies or cost trends that seem unusual and that the officials investigate such discrepancies or trends when necessary. After this review, the Chief Financial Officer signs the invoice. During our review of NQF’s invoices for the first contract year, we found that the Chief Financial Officer signed the invoices as the officials described to us.

In addition to the review of invoices, NQF officials described other ways the organization helps to ensure that the costs it reports to HHS are proper. In particular, NQF officials told us NQF uses an electronic timesheet system in order to track employee labor hours. NQF officials told us that the timesheet system allows NQF employees to track their labor hours by project and have their labor hours reviewed and approved by the appropriate NQF officials. In addition to the timesheet system, NQF officials told us that their organization established a written procurement policy in August 2009 and revised it in January 2010 to guide how they track other direct costs—specifically, payments to subcontractors and consultants—that are reported in NQF’s invoices. NQF officials told us that under its procurement policy, NQF officials are to obtain the appropriate approval signatures for payments on invoices as well as other payments for subcontractors and consultants once the services have been received. Furthermore, according to the policy, NQF officials are to document how key procurement decisions are made, such as the basis for setting an award cost or price for a subcontractor or consultant. Having a well-designed procurement policy can help reduce the risk of inappropriate payments or pricing related to subcontractors and consultants. During our review of NQF subcontractor and consultant files for the period prior to January 2010—before NQF revised its procurement policy—we found that NQF did not always document approvals for subcontractor payment and did not document that it had determined that its consultant pricing was reasonable.

HHS Requires Its Officials to Review NQF Invoices following Certain Procedures in Order to Help Ensure That Reported Costs Are Proper

Like NQF, HHS relies on reviews of NQF’s invoices in order to help ensure that reported costs are proper. Two HHS officials assigned to oversee the NQF contract, the project officer and the contracting officer, are responsible for these reviews. When conducting their reviews, the two officials are required to follow certain procedures established in HHS policies. For example, under these policies, the project officer is required to review NQF’s invoices to determine whether billed services were actually provided and are supported with adequate documentation. Similarly, the contracting officer is required to review the invoices to determine whether NQF’s reported costs are consistent with its contract, accurately calculated, and have adequate documentation. Both officials are required to document when they approve invoices for payment to NQF. When we reviewed HHS documentation and interviewed HHS officials during the course of our work, we found that the contracting officer and project officer had generally followed the review procedures required by HHS policy.

Table 2 provides more detailed information on the procedures that the project and contracting officers are required to follow when reviewing NQF invoices. Table 2 also provides information we obtained from HHS officials on how they implemented these requirements.

Concluding Observations

While NQF has begun work in the first year of its contract for the five duties related to quality measurement established by MIPPA, it is too early for us to assess whether, or to what extent, NQF will be successful in carrying out these duties. This report describes NQF’s work for the first of 4 contract years, and HHS has flexibility to determine on an annual basis the specific work it expects NQF to perform for each of the MIPPA duties. Therefore, it is not yet known exactly what work NQF will be expected to complete during the remainder of the contract period. In addition, other events related to quality measurement, such as the completion of HHS’s national strategy for quality improvement, are expected to occur before the end of the 4-year contract period and may have some influence on NQF’s specific work for the five MIPPA duties. Our second report will provide another opportunity to review NQF’s performance and costs.

Agency and Other External Comments

We provided drafts of this report to HHS and NQF for comment. Both HHS and NQF provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the Secretary of Health and Human Services and other interested parties. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix II.

Appendix I: National Quality Forum’s Endorsement Process and Example Project

The National Quality Forum (NQF) established its endorsement process in 2000. NQF’s process includes the nine steps described in table 3 below. The table also provides information on the endorsement process as applied to a project to endorse a number of measures related to home health care, such as measures on education provided to patients and caregivers on medications for care and increases in the number of pressure ulcers. This project was initiated prior to the NQF contract with the Department of Health and Human Services (HHS) that was required by the Medicare Improvements for Patients and Providers Act of 2008. NQF announced a call for nominations for steering committee members for this project in August 2008 and the final set of 20 endorsed measures was announced on March 31, 2009.

Appendix II: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contact named above, Will Simerl, Assistant Director; La Sherri Bush; Helen Desaulniers; Krister Friday; Natalie Herzog; Carla Lewis; Lisa Motley; Ruth S. Walk; Rasanjali Wickrema; and William T. Woods made key contributions to this report.