Health Care Quality Measurement: HHS Should Address Contractor Performance and Plan for Needed Measures
Released: 2012-01-13T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: The Medicare Improvements for Patients and Providers Act of 2008 (MIPPA) directed the Department of Health and Human Services (HHS) to enter into a 4-year contract with an entity to perform various activities related to health care quality measurement. In January 2009, HHS awarded a contract to the National Quality Forum (NQF), a nonprofit organization that endorses health care quality measures—that is, recognizes certain ones as national standards. In 2010, the Patient Protection and Affordable Care Act (PPACA) established additional duties for NQF. This is the second of two reports MIPPA required GAO to submit on NQF’s contract with HHS. In this report—which covers NQF’s performance under the contract from January 14, 2010, through August 31, 2011—GAO examines (1) the status of projects under NQF’s required contract activities and (2) the extent to which HHS used or planned to use the measures it has received from NQF under the contract to meet its quality measurement needs, as of August 2011. GAO interviewed NQF and HHS officials, reviewed relevant laws, and reviewed HHS and NQF documents.

section_title: What GAO Found
paragraphs: NQF has made progress on projects under its contract activities, as of August 2011. Specifically, NQF has completed or made progress on 60 of 63 projects. For example, NQF has completed projects to endorse measures related to various topics, including nursing homes. However, for more than half of the projects, NQF did not meet or did not expect to meet the initial time frames approved by HHS. For example, NQF completed one project to retool measures—that is, convert previously endorsed quality measures to an electronic format. While the retooling project was expected to be completed by September 2010, its completion was delayed by 3 months. NQF and HHS officials identified various reasons that contributed to this delay, including an expansion of the project’s scope and complexity. As a result of the delay, HHS did not have all the retooled measures it expected to include in its Electronic Health Records (EHR) Incentive Program. The delay of this project was also a contributing factor to NQF exceeding its estimated cost for its entire contract activity related to EHR by about $560,000 in the second contract year— January 14, 2010, through January 13, 2011. While HHS monitored NQF’s progress through monthly progress reports and approved changes to time frames and costs, HHS did not use all of the tools for monitoring that are required under the contract. Specifically, HHS did not conduct an annual performance evaluation to assess timeliness and cost issues that could have helped to inform NQF’s future scope of work. Until August 2011, HHS did not enforce the provision for NQF to submit a financial graph to compare monthly costs for each contract activity with cost estimates, which is information not included in monthly progress reports. These tools could have provided additional, more detailed information to help identify instances in which NQF might have been at risk of not meeting time frames or exceeding cost estimates, which could have provided HHS an opportunity to make any appropriate changes to NQF’s activities.

HHS had used or planned to use about half of the measures—164 of 344—that it received from NQF under the contract, as of August 2011. For example, HHS used 44 measures that NQF retooled under the contract in its EHR Incentive Program. HHS officials stated that the 44 measures used in the program contained errors, which required corrections. HHS officials also have not yet tested the retooled measures to assess the feasibility of implementing them in the electronic format; therefore, HHS runs the risk that some of these measures may not work as intended when implemented. HHS officials told GAO they expect to evaluate if and how they could use all of the remaining measures HHS received under the contract. However, HHS has not determined how PPACA requirements for quality measurement may have changed its needs for endorsed quality measures. As a result, HHS has not established a comprehensive plan that identifies its measurement needs and time frames for obtaining endorsed measures and that accounts for relevant PPACA requirements. Without such a plan, HHS may be limited in its efforts to prioritize which specific measures it needs to develop and to have endorsed by NQF during the remainder of the NQF contract. As a result, HHS may be unable to ensure that the agency receives the quality measures needed to meet PPACA requirements, including time frames for implementing quality measurement programs.

section_title: What GAO Recommends
paragraphs: GAO recommends HHS: (1) use all monitoring tools required under the contract to help address NQF’s performance, (2) complete testing of retooled measures, and (3) comprehensively plan for its quality measurement needs. HHS neither agreed nor disagreed with these recommendations. NQF concurred with many of the findings in the report and provided additional context.

section_title: Letter
paragraphs: The nation’s health care spending was about $2.5 trillion in 2009. Health  care spending has been increasing in recent years by an average of  nearly 7 percent per year since 2000, and is projected to reach over   $4.5 trillion by 2020. However, higher levels of spending, such as  spending more to increase the number or technical complexity of  treatments, do not necessarily lead to a corresponding increase in the  quality of care. To better align quality and health care spending, the  Department of Health and Human Services (HHS) has developed various  programs and initiatives to measure health care quality and to provide  financial incentives to health care providers to improve quality and to  reduce the use of unnecessary and costly services. These programs and  initiatives include, for example, pay-for-performance programs that rely on  information collected on various health care quality measures. Health  care quality measures are used to evaluate how health care is delivered,  and information obtained from such measures can promote accountability  among health care providers. The Patient Protection and Affordable Care  Act (PPACA), enacted in March 2010, directed HHS to expand its current  programs and initiatives, such as pay-for-performance programs, and to  implement new ones that will use health care quality measures. PPACA  establishes specific time frames for HHS to implement these programs  and initiatives that rely on various health care quality measures over the  next several years. For example, PPACA requires HHS to establish a  hospital value-based purchasing program. HHS estimates that in fiscal  year 2013 its Hospital Value-Based Purchasing Program will provide  $850 million in Medicare incentive payments to hospitals that meet certain  performance measures. Additionally, HHS is required under PPACA to  establish value-based payments for physicians, develop a plan for value- based purchasing programs for skilled nursing home facilities and home  health agencies, and create pilot programs that test the implementation of  a value-based purchasing program for other providers—psychiatric, long- term care, inpatient rehabilitation, and cancer hospitals as well as hospice  programs.

For many of HHS’s health care quality measurement programs and  initiatives, HHS is required to give consideration to the quality measures  endorsed by a consensus-based entity, such as the National Quality  Forum (NQF). NQF is the entity in the United States with the lead  responsibility for endorsing health care quality measures—that is,  determining which measures should be recognized as national standards.  The Medicare Improvements for Patients and Providers Act of 2008  (MIPPA) directed the agency to enter into a 4-year contract with an entity  to endorse health care quality measures and conduct other activities, and  HHS contracted with NQF in January 2009. NQF is a nonprofit organization established in 1999 in order to foster agreement on national  standards for measurement and public reporting of health care  performance data. MIPPA provided $10 million per year for each of fiscal  years 2009 through 2012—$40 million in total—from the Medicare Trust  Funds for this 4-year contract, which covers the period from January 14,  2009, through January 13, 2013. PPACA provided an additional   $100 million—$20 million per year for each of fiscal years 2010 through  2014—for additional health care quality measurement work to be  conducted by NQF and HHS through fiscal year 2014. Because PPACA  does not specify the proportion of this amount for NQF activities, HHS  officials told us that they plan to obligate approximately $10 million per  year for the NQF contract, beginning in 2011, and that they may extend  the contract through fiscal year 2014.

MIPPA also directed us to submit two reports on the performance of and  costs incurred by NQF under its 4-year contract with HHS. 2010, we issued the first report, which contained information about NQF’s  performance and costs during the first contract year, which began  January 14, 2009, and ended January 13, 2010. This second report  covers NQF’s performance and costs for the second contract year— January 14, 2010, through January 13, 2011—and part of the third  contract year—January 14, 2011, through August 31, 2011. Specifically,  this report examines (1) the status of projects under the activities NQF is  required to complete under its contract with HHS, (2) the costs and fixed  fees NQF reported under the contract from January 2010 through August  2011, and (3) the extent to which HHS used or planned to use the  measures it receives from NQF under the contract to meet its quality  measurement needs, as of August 31, 2011. In addition, appendix I of this  report provides information about a framework for the various stages of a  health care quality measure, including measure endorsement, as  described by HHS and NQF officials and others.

MIPPA states that GAO’s reports shall be submitted no later than 18 months and   36 months, respectively, after the effective date of the contract. contract activities in response to MIPPA, NQF’s 2011 technical proposal  for the contract activities in response to PPACA, and NQF’s monthly  progress and monthly internal evaluation plan reports that NQF is  required to submit to HHS. For purposes of our work, we categorized  the 16 tasks that NQF is required to perform under the contract from  January 2010 through August 2011 into 9 contract activities. (See app. II  for more detailed information on how we categorized these contract  activities.) In addition, we identified the various projects NQF is required  to perform under the 9 contract activities we identified. Specifically, for the  purposes of our work, we identified 63 projects NQF is required to  perform under the 9 contract activities, as shown in appendix III. We then  compared the information we obtained from NQF on its progress on the  63 projects under the 9 activities to the requirements for NQF’s  performance under the contract with HHS and the expected time frames  and cost estimates initially established by HHS and NQF in the final  annual MIPPA work plans and other NQF documents for completing the  work. We also interviewed NQF officials responsible for implementing the  contract and HHS officials responsible for managing the contract and  overseeing NQF’s performance. We also compared evidence from HHS  and NQF documents with evidence from discussions with HHS and NQF  officials to corroborate any issues or challenges with NQF’s performance.  Based on our comparison and interviews with officials, we identified any  changes to the work NQF is required to perform and changes to the time  frames or cost estimates established in the final work plans developed  annually to respond to MIPPA and a budget developed as part of NQF’s  technical proposal to respond to PPACA.

To assess the costs and fixed fees NQF reported under the contract from  January 2010 through August 2011, we reviewed NQF documents that  include financial information related to the contract, such as NQF’s final  work plans developed in 2010 and 2011 to respond to MIPPA and a  budget developed as part of NQF’s technical proposal to respond to  PPACA that describe NQF’s estimated costs for each of the nine activities  we identified under the contract. We obtained and summarized  information from NQF monthly invoices submitted to HHS from January  14, 2010, through August 31, 2011, for direct and indirect costs  incurred by NQF under the contract and fixed fees. We compared  NQF’s reported costs with its cost estimates for the activities in NQF’s  2010 final annual work plan and determined whether or to what extent  NQF’s reported costs exceeded its cost estimates. We did not assess the  accuracy of NQF’s reported costs. We also reviewed the HHS contract  with NQF and any contract modification since January 14, 2010. We also  interviewed NQF and HHS officials responsible for reviewing and  approving the costs and fixed fees submitted under the contract. Based  on our review of documents and interviews with NQF and HHS officials,  we determined that the reported costs and fixed-fee data were sufficiently  reliable for the purposes of this report.

To examine the extent to which HHS used or planned to use measures it  receives from NQF under the contract to meet its quality measurement  needs, as of August 2011, we reviewed statutory requirements and  documents related to HHS health care quality measurement efforts. We  relied on HHS to identify programs and initiatives that use or will use  health care quality measures, such as programs or initiatives established  by MIPPA, PPACA, and the Health Information Technology for Economic  and Clinical Health Act. As a result, the health care quality programs  and initiatives mentioned in our report may not represent a  comprehensive list of all those that use or plan to use measures received  from NQF under the contract. We interviewed officials from various  divisions within HHS that may use measures that were nationally  recognized—or NQF-endorsed—in their various programs and initiatives  related to health care quality measurement in order to corroborate the  information we obtained from documentation. We obtained information  about how HHS selects measures for its programs and initiatives from our  interviews with HHS officials and reviews of HHS documents. We  determined how HHS has used and planned to use measures that were  received under the contract with NQF by identifying all the measures that  HHS received under the contract and matching each measure to the  quality measures being used or considered for use in HHS programs and   For the purpose of this report, we defined a measure as used  initiatives.if the measure was identified by HHS and included in an HHS program or  initiative as of August 31, 2011, and indicated as such by a final rule or  other HHS documentation. We defined a measure as planned for use if  the measure was identified by HHS for potential use in an HHS program  or initiative as of August 31, 2011, and indicated as such by a proposed  rule or other HHS documentation. In addition, we reviewed HHS  documents and interviewed HHS officials to determine whether or to what  extent HHS has plans for using work conducted under the NQF contract  to meet PPACA requirements related to quality measures for HHS  programs and initiatives. For example, we reviewed HHS plans or  strategies such as the 2011 National Strategy for Quality Improvement in  Healthcare (National Quality Strategy),to help meet its quality measurement needs. that describe the agency’s plans  We conducted this performance audit from April 2011 through January  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objective. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: NQF is a nonprofit organization established in 1999 in order to foster  agreement, or consensus, on national standards for measuring and public  reporting of health care performance data.more than 400 organizations that represent multiple sectors of the health  care system, including providers, consumers, and researchers. NQF’s  mission focuses on three core areas: (1) building consensus on national  priorities and goals for performance improvement and working in  partnership to achieve them, (2) endorsing national consensus standards  for measuring and publicly reporting on performance, and (3) promoting  the attainment of national goals through education and outreach  programs.

Its membership includes  Prior to its contract with HHS, NQF established a consensus development  process (CDP) to evaluate available health care quality measures to  determine which ones are qualified to be endorsed—that is, recognized— as national standards. Under this process, organizations that develop  quality measures submit them to NQF for consideration, in response to  specific solicitations by NQF. NQF forms a committee of experts from its  member organizations as well as other organizations and agencies to  conduct an objective and transparent review of these quality measures  against four standardized criteria established by NQF, such as whether  the measures are scientifically acceptable. After this committee  evaluates the measures against these criteria, NQF’s process allows for a  period during which its member organizations and the public may  comment on the committee’s recommendation for each measure. The  process also provides for a period for its member organizations to vote on  whether the measures should be endorsed by NQF as a national  standard. Ultimately, NQF’s board of directors makes a final decision on  whether NQF should formally endorse the measures.

As of October 2011, NQF has endorsed over 600 health care quality  measures in 27 areas, such as cancer and diabetes. HHS uses NQF- endorsed measures in its programs and initiatives to promote quality  measurement, and NQF continues to endorse quality measures separate  from its contract with HHS.
subsections: section_title: NQF’s Contract with HHS
paragraphs: NQF’s work under the contract includes endorsement of quality measures  and other activities that are expected to support HHS’s quality  measurement efforts, such as through value-based purchasing programs.  Specifically, NQF’s work under the contract consists of various projects  under the nine contract activities related to health care quality  measurement. The work plans developed annually to respond to MIPPA  and NQF’s technical proposal to respond to PPACA delineate the projects  NQF is required to conduct under the nine contract activities, as well as  expected time frames and cost estimates for the projects for each year.  Table 1 provides more detailed information on the nine contract activities.  Some of these activities are required by either MIPPA or PPACA, while  others are quality measurement activities established by HHS or  administrative activities.

To help determine the activities and the projects under the nine contract  activities that NQF is expected to perform during each contract year, HHS  has established an interagency workgroup that comprises officials from  multiple divisions within HHS, including the Agency for Healthcare  Research and Quality, the Centers for Medicare & Medicaid Services  (CMS), and the Office of the National Coordinator for Health Information  Technology. The workgroup is responsible for prioritizing and selecting  the activities and projects under each activity that NQF is expected to  perform during each contract year. HHS officials told us that the  representatives from these various HHS agencies provide input on the  work NQF is expected to perform, including determining quality measures  requested from NQF for their respective programs. The activities and  projects selected by the interagency workgroup become part of NQF’s  scope of work under the contract. Some of the projects under the  contract activities that NQF is expected to perform during the year will be  ongoing from the previous contract year while new work will be  incorporated into the work plan as necessary.

For the NQF contract, HHS selected a cost-plus-fixed-fee contract, under  which HHS reimburses NQF for actual costs incurred under the contract  in addition to a fixed fee that is unrelated to costs. Cost-plus-fixed-fee  contracts are used for efforts such as research, design, or study efforts  where costs and technical uncertainties exist and it is desirable to retain  as much flexibility as possible in order to accommodate change.  However, this type of contract provides only a minimum incentive to the  contractor to control costs. As we reported in 2009, these contracts are  suitable when the cost of work to be done is difficult to estimate and the  level of effort required is unknown. This cost-plus-fixed-fee contract is  NQF’s first cost-reimbursement contract. For cost-reimbursement  contracts, the Federal Acquisition Regulation (FAR) requires  appropriate government surveillance during performance to provide  reasonable assurance that efficient methods and effective cost controls  are used. Under the FAR, contracts are to contain a provision for agency  approval of a contractor’s subcontracts. HHS’s contract with NQF  contains this provision and also requires the approval of consultants  engaged under the contract. The review and approval of NQF’s use of  subcontractors and consultants require appropriate support  documentation provided by NQF to HHS, including a description of the  services, the proposed price, and a negotiation memo that reflects the  principal elements of the price negotiations between NQF and the  subcontractor or consultant. Under its contract with HHS, NQF has  utilized 31 subcontractors and 16 consultants since January 14, 2010, to  provide support to NQF on many of the contract activities and associated  projects.

Two HHS components are principally responsible for administering the  NQF contract: the office of the Assistant Secretary for Planning and  Evaluation (ASPE) and CMS—an agency within HHS. Specifically, the project officer for the NQF contract is a representative of ASPE. This  individual is responsible for program management and works with the  contracting officer to oversee the contract. The contracting officer for the  NQF contract, responsible for administering the contract, is a  representative of CMS.contracting officer is required to perform, such as conducting an annual  evaluation of the contractor’s performance.

section_title: NQF Made Progress on Projects under Each of Its Contract Activities but Has Not Met Expected Time Frames and Has Exceeded Cost Estimates
paragraphs: From January 14, 2010, through August 31, 2011, NQF made progress  on projects under its contract activities. However, our review of NQF  documents found that NQF had not met or did not expect to meet time  frames on more than half of the projects, and it exceeded its cost  estimates for projects under three of the contract activities. HHS did not  use all tools for monitoring that are required under the contract.
subsections: section_title: NQF Made Progress on Projects under Each of the Contract Activities as of August 31, 2011
paragraphs: From January 14, 2010, through August 31, 2011, NQF has made  progress on 60 of the 63 projects under the activities required under its  contract with HHS. Specifically, NQF had completed 26 projects and was   (App. III provides the  continuing to work on the remaining 34 projects.status of all contract activities and the projects under each activity NQF  was expected to perform during our reporting period.) Examples of  projects under the contract activities include both completed and  continuing projects:

Endorsement of Measures Activity. NQF endorsed 101 measures  since the beginning of the second contract year by conducting work  on endorsement projects on different topic areas. Specifically, NQF  completed two projects to endorse 38 outcome measures related to  20 high-priority conditions identified by CMS that account for the  majority of Medicare’s costs, and mental health and child health  conditions; and 21 performance measures for chronic and postacute  care nursing facilities. NQF also worked on two projects related to  child health quality and patient safety. As of August 2011, NQF  endorsed 41 child health quality measures and 1 patient safety  measure under these projects. NQF expected to complete the child  health quality project in September 2011 and the patient safety project  in December 2011. In addition, NQF completed a contractually  required review of its endorsement process, subcontracting with  Mathematica Policy Research, Inc. (Mathematica). The review  focused on the timeliness and effectiveness of the endorsement  process; identified inefficiencies, including those that may contribute  to delays; and recommended, among other steps, that NQF create a  schedule for its endorsement process for measure developers and  develop feasible time lines that include clear goals for each  endorsement project. HHS officials stated that Mathematica’s  recommendations were valuable because much of the work under the  NQF contract needs to be completed in an accelerated timeline to  help fill critical measurement gaps associated with HHS’s health care  quality programs and initiatives. For more information about this  review, see appendix IV.

Maintenance of Endorsed Quality Measures Activity. NQF  maintained—that is, updated or retired—124 measures under the  contract since the beginning of the second contract year. These  included 41 measures reviewed under NQF’s 3-year review cycle  related to diabetes, mental health, and musculoskeletal conditions.  In addition, 83 measures were maintained under NQF’s other  maintenance review processes. NQF was also continuing to work on  maintenance projects it initiated in 2010 for measures related to  cardiovascular and surgery measures. As of August 2011, the two  projects were expected to be completed by December 2011 and  January 2012, respectively.

Promotion of the Development and Use of Electronic Health  Records Activity. NQF has made progress on three projects related  to retooling—that is, converting previously endorsed quality measures  to an electronic format that is compatible with electronic health  records (EHR). First, NQF completed initial retooling of 113  measures. This work is intended to allow data from EHRs to be used  for quality measurement, which is a part of HHS’s long-term goal to  use health information technology to exchange information and  improve quality of care. Second, as of August 2011, NQF convened  an expert review panel to review the retooled measures to ensure that  each retooled measure is properly formatted, the logic is correctly  stated, and the intent of the measures is maintained in the electronic  format that will use data obtained from EHRs, instead of from claims  as originally formatted. Third, as of August 2011, NQF was expected  to complete another project to provide an updated list of the 113  retooled measures to HHS by December 2011, which would  incorporate any revisions identified by the expert review panel and  others involved in the retooling process. After these updated  measures are completed, HHS officials told us that they will contract  with other entities to conduct testing of some of the 113 retooled  measures to assess the feasibility of implementing the measures in  the electronic format. Although NQF’s endorsement process requires  that measure developers submit data on validity and reliability testing  of measures they submit for endorsement, this testing does not  include feasibility testing for implementing the measures in an  electronic format for performance measurement. As of December  2011, HHS officials did not provide an expected date of completion for  this feasibility testing but told us that they have awarded two contracts  that include this in their scope of work. In addition to the retooling  projects, NQF is developing a software tool—the Measure Authoring  Tool—to allow measure developers to create standardized electronic  measures that help capture information in EHRs so that less retooling  would be needed in the future. As of August 2011, NQF was  completing final testing of the beta, or initial, version of this tool. NQF  expected to complete testing and publish an updated version for  public use by January 2012.

Multistakeholder Input into HHS’s National Quality Strategy  Activity. NQF convened the National Priorities Partnership (NPP), a  multistakeholder group expected to provide annual input on national  priorities, among other things, to be considered in the National Quality  Strategy. As of August 2011, the NPP was completing a report on  this input, which was then published in September 2011. The report  noted the need for a national comprehensive strategy that identifies  core sets of standardized measures to meet each of the national  priorities HHS identified in the 2011 National Quality Strategy, among  other things. The NPP noted in the report that a common data  platform, core measure set, and public reporting mechanism are key  components of the infrastructure for performance measurement. It  also highlighted that a strategic plan, road map, and timeline for  establishing an infrastructure should be accelerated to allow for rapid  implementation over the next 5 years. Additionally, the NPP reported  that it was critical that all federal programs drive toward the  establishment of a common platform for measurement and reporting.

Multistakeholder Input on the Selection of Quality Measures  Activity. NQF has convened the Measure Applications Partnership  (MAP). The MAP is a multistakeholder group that is expected to  conduct work in two areas. First, the MAP is expected to provide input  to the Secretary of HHS on the selection of quality measures for use  in payment programs and value-based purchasing programs required  by PPACA, among others. The MAP will review a list of measures  published by the Secretary of HHS on December 1 of each year, and  develop a report that contains a framework to help guide measure  selection. The MAP will provide its annual input beginning February 1,  2012, for measures used in the following 11 programs: hospice,  hospital inpatient, hospital outpatient, physician offices, cancer  hospitals, end-stage renal disease (ESRD) facilities, inpatient  rehabilitation facilities, long-term care hospitals, hospital value-based  purchasing, psychiatric hospitals, and home health care. Second, the  MAP is expected to publish reports that provide input on the selection  of measures for use in various quality reporting programs, including  those for physicians. As of August 2011, the MAP had held meetings  and initiated its work for reports due October 1, 2011.

Other Health Care Quality Measurement Activity. NQF completed  a project to endorse six imaging efficiency measures. NQF was also  continuing to work on a project to identify existing quality measures  and gap areas related to measurement of regionalized emergency  care services.

section_title: NQF Did Not Meet Expected Time Frames and Exceeded Cost Estimates
paragraphs: Our review of NQF documents found that NQF had not met or did not  expect to meet time frames on more than half of the projects under the  contract activities that were completed or ongoing, as of August 2011.  Specifically, our review of documents found that NQF had not met  expected time frames on 18 of the 26 projects it completed under the nine  contract activities. Further, NQF did not expect to meet time frames on 14  of the 34 projects on which it was continuing to work. The delays of these  projects under the contract activities varied in time from about 1 to   12 months. HHS officials told us they approved all changes to the time  frames, which were established by HHS and NQF in NQF’s 2010 and  2011 final annual MIPPA work plans and the PPACA technical proposal.  Appendix III provides the status for all projects related to each of the nine  contract activities, including information on their expected and actual time  frames for completion during our reporting period. Examples of projects  under the contract activities for which NQF did not meet or did not expect  to meet expected time frames include the following:

Endorsement of Measures Activity. NQF did not meet or did not  expect to meet time frames for all five endorsement projects under the  endorsement contract activity. (See app. III for details on the five  projects.) For example, NQF was expected to complete an  endorsement project for nursing home quality measures in July 2010;  however, the measures were not endorsed until February 2011. (See  fig. 1 for estimated time frames and actual completion dates for all  projects related to the endorsement contract activity.)

NQF officials stated that several factors contributed to NQF exceeding  the expected time frames for the five endorsement projects, including  the high volume of measures submitted for review, the amount of time  it took to harmonize measures between measure developers, and a  need for additional technical expertise on review panels.

Promotion of the Development and Use of Electronic Health  Records Activity. NQF did not meet or did not expect to meet  expected time frames for five out of eight projects related to the EHR  contract activity. For example, NQF was expected to complete its  initial retooling of 113 endorsed quality measures into electronic  formats by September 2010, but this effort was not completed until  December 2010. (See fig. 2 for estimated time frames and actual  completion dates for all projects related to the EHR contract activity.)  In addition, NQF was expected to complete the project to convene an  expert panel to review the 113 retooled measures by January 2011.  However, the panel did not complete its review of the 113 measures  until June 2011.

According to HHS and NQF officials, several factors contributed to  NQF exceeding expected time frames for the retooling project under  the EHR contract activity. HHS officials stated that the first set of 44  retooled measures submitted had errors that required correction. For  example, HHS officials stated that they found errors in the electronic  coding of these 44 retooled measures requiring NQF and its  subcontractors who retooled the measures to make corrections. In  addition, HHS and NQF officials stated that after starting the retooling  project, they quickly learned that the estimated time frames for the  retooling project, as well as other projects related to the EHR activity,  were overly ambitious, given the scope and complexity of the work.  For example, HHS officials noted that retooling of quality measures  into electronic format had never been attempted before and the  technical complexity and labor required to complete the project were  greater than anticipated. NQF officials also told us that HHS’s  requests to modify the scope of work for this project often required  changing the time frame for completing the retooled measures. These  factors resulted in an extension of the project that delayed the final  delivery of the 113 retooled measures as well as contributed to the  need for additional staff at NQF.

Other Health Care Quality Measurement Activity. NQF was  expected to complete two projects under the other health care quality  measurement activity related to efficiency and resource use—one  white paper on resource use and another on geographic-level  efficiency by July 2010. These white papers were intended to  provide information for an endorsement project on resource-use  measures that began in January 2011. However, as of August 2011,  the resource-use paper was still under review by HHS, and NQF  officials stated they expected to receive comments in September  2011. The geographic-level efficiency paper was canceled in June  2010 at the request of HHS. NQF initially intended to subcontract the  work on these two projects, but officials told us that they were unable  to identify a subcontractor at the level of funding approved for this  project. As a result, HHS approved NQF’s proposal to complete this  work internally. HHS officials stated that the drafts NQF submitted on  both topics were poor in quality and did not meet its needs, resulting  in HHS requesting additional revisions for the resource-use white  paper that delayed its completion, and requesting the cancellation of  the geographic-level efficiency white paper.

Administrative Activity. NQF did not meet the expected time frames  for completing one of the required projects under the administrative  activity—finalizing its annual work plan. Specifically, the NQF contract  requires NQF to develop an annual work plan and to receive final  approval from HHS within the first 4 weeks of each contract year;  however, NQF did not meet this requirement in 2010 or 2011. For  example, the final 2011 MIPPA annual work plan was not developed  by NQF and approved by HHS until April 1, 2011. According to NQF  and HHS officials, the 2011 MIPPA work plan was not developed and  approved on time due to extended discussions on the scope and cost  estimates of NQF’s EHR activities. HHS officials told us that the  primary reason for the extended discussions was that they expected  the costs to reflect all the work needed to complete the Measure  Authoring Tool (MAT) by the end of the second contract year.  However, they said that NQF only submitted a beta version of the tool  by the end of the second contract year, which was not the version  expected by HHS. NQF officials told us that the version was never  intended to be final but rather a beta version, consistent with their  understanding of HHS’s expectations. As a result, HHS and NQF  officials needed to evaluate the scope of work and cost estimates for  this and other projects. Further, NQF officials told us the delay in  completing the 2011 MIPPA annual work plan resulted in the  interruption of NQF’s ongoing work related to the MAT under the EHR  contract activity. The delay also delayed its receipt of funding for  some new or ongoing work under the contract. In some instances,  NQF chose to start new or continue ongoing work with its own  funding. For example, NQF officials stated that NQF began work  related to the MAP using its own funds until HHS authorized the work.  In addition, the delay in completing the 2011 MIPPA work plan  resulted in the need to set the start date for fall 2011 rather than  earlier in the contract year for some of the projects under the  maintenance activity.

NQF also exceeded its cost estimates for projects under three of the  contract activities. HHS officials told us they approved the changes to the  cost estimates and in some cases modified NQF’s scope of work to help  ensure that NQF’s costs did not exceed the amount HHS had obligated  for the contract activities. NQF officials stated that in certain cases, not  meeting expected time frames contributed to NQF exceeding these cost  estimates. For example, the delays in projects related to the EHR contract  activity, including expanding the scope of the retooling project,  contributed to NQF exceeding its cost estimate of about $3.8 million for  the entire EHR contract activity by about $560,000 in the second contract  year. In another example, the delays in finalizing the 2010 and 2011  MIPPA work plans contributed in part to NQF exceeding its cost estimate  for developing and finalizing these plans, which is a project under the  administrative contract activity. Specifically, while NQF estimated that  completion of the annual work plan would cost approximately $77,000,  NQF reported an actual cost of $176,590. In addition, NQF also exceeded  its cost estimate for the endorsement contract activity during the second  contract year for various reasons, including a need for additional technical  experts for review panels. Specifically, NQF exceeded estimated costs of  about $3.1 million for the entire endorsement activity by about $146,000   While HHS officials told us they approved  in the second contract year.all changes to the cost estimates, in certain cases they reduced the scope  of NQF’s work in 2011 to ensure that total available funding for the  contract year was not exceeded and that sufficient funding was available  for ongoing projects. For example, HHS officials told us that they had  hoped to start several new endorsement projects beginning in 2011;  however, these projects were not included in the 2011 final annual MIPPA  work plan so that funding would be available for NQF to complete its  ongoing projects, including work that was delayed under the EHR  contract activity. In addition, HHS requested that NQF discontinue its  work on one project related to the development of a public website for  2011, which is associated with the administrative contract activity.

section_title: HHS Did Not Use All Monitoring Tools Required under the NQF Contract
paragraphs: HHS officials told us that to help monitor NQF’s performance on the  projects under the contract activities, they rely on NQF to report any  issues, including those related to time frames or cost estimates, in the  monthly progress reports that NQF is required to submit to HHS or during  phone calls held at least monthly. While HHS monitored NQF’s progress  and approved changes to the time frames and cost estimates for the  projects under the contract activities, HHS did not use available tools for  monitoring that are required under NQF’s contract. These tools could  have helped to provide an opportunity for HHS to make any appropriate  changes to NQF’s projects. For example, HHS did not conduct an annual  performance evaluation required by the contract that would assess  timeliness and cost control issues, among other things, for the previous  contract year. The results of such an evaluation could help HHS officials  to consider potential timeliness and cost issues when determining NQF’s  scope of work for the next year. Further, while monthly progress reports  and invoices include information on NQF’s costs, these documents do not  compare reported costs to initial cost estimates. HHS officials told us that,  prior to August 2011, they had not enforced a contractual requirement for  NQF to submit—nor had it received from NQF—a financial graph in its  monthly progress reports that provides information comparing NQF’s  monthly incurred costs for each of the contract activities with initial cost  estimates. Instead, HHS officials informally requested that NQF provide  them with the financial status of the contract activities in midyear 2010,  which helped them to plan for NQF’s work under the contract for 2011.  Having a financial graph in the monthly progress report could have helped  HHS officials to identify instances where any contract activity was  approaching or exceeding NQF’s initial cost estimates prior to HHS’s  midyear review. This, in turn, could have provided HHS and NQF an  opportunity to adjust estimates of future costs for these or related  activities earlier in the contract year. HHS officials had asked NQF to  begin to include such a financial graph in its monthly progress reports  beginning in August 2011.

section_title: NQF Reported Costs and Fixed Fees of over $20 Million from January 2010 through August 2011
paragraphs: From January 14, 2010, through August 31, 2011, NQF reported a total of  approximately $22.4 million in costs and fixed fees on monthly invoices  submitted to HHS for projects under activities conducted in response to  MIPPA and PPACA. Specifically, NQF reported about $12.8 million in  total costs and fixed fees for the contract activities it performed during the  second contract year—January 14, 2010, through January 13, 2011.  From January 14, 2011, through August 31, 2011, part of the third  contract year, NQF reported an additional $9.6 million in total costs and  fixed fees.

During the second contract year, the majority of NQF’s reported costs  were related to the promotion of the development and use of EHRs   (36 percent, or $4.6 million) and endorsement of health care quality  measures (26 percent, or $3.3 million). Figure 3 illustrates the costs and  fixed fees NQF reported for eight of the nine contract activities we  reviewed that occurred during the second contract year. The ninth  contract activity relates to multistakeholder input on the selection of  quality and efficiency measures, as directed by PPACA. This contract  activity did not begin until after January 14, 2011, which is the start of the  third contract year.

For the part of the third contract year covered in our review—January 14,  2011, through August 31, 2011—almost one-half of NQF’s reported costs  were for the activity to promote the development and use of EHRs and for  the activity to provide multistakeholder input on the selection of quality  and efficiency measures. Each of these activities accounted for   22 percent, or about $2.1 million of NQF’s reported costs. Other costs  reported by NQF include those for the activity related to providing  multistakeholder input into HHS’s annual National Quality Strategy   ($1.55 million, or 16 percent) and those for the activity related to the  maintenance of endorsed quality measures activity (13 percent, or   $1.29 million). Figure 4 illustrates the costs and fixed fees NQF reported  for the part of the third contract year covered in our review for each of the  nine contract activities we reviewed.

According to HHS, as of August 2011, about $55.2 million remains  available for the NQF contract. About $15.1 million in MIPPA funding  remains available for work to be conducted through January 13, 2013. In  addition, HHS plans to obligate approximately $40.1 million of its PPACA  funding through 2014 for NQF’s activities related to health care quality  measurement in response to PPACA.

section_title: HHS Used or Planned to Use Some NQF Measures Received under the Contract but Has Not Comprehensively Planned for Other Measurement Needs to Implement PPACA
paragraphs: For its various programs or initiatives, HHS has used or planned to use  about one-half of the quality measures that NQF has endorsed,  maintained, or retooled under the contract, as of August 31, 2011, and  HHS officials expect to evaluate if and how the remaining measures will  be used. However, HHS has not comprehensively determined how it will  use NQF’s work under the contract to implement PPACA requirements  related to quality measures.
subsections: section_title: HHS Has Used or Planned to Use 164 of the 344 Measures That NQF Has Endorsed, Maintained, or Retooled under the Contract
paragraphs: According to HHS officials, HHS has used or planned to use about one- half (164) of the 344 health care quality measures it has received from  NQF through various endorsement, maintenance, and retooling projects  under the contract, as of August 31, 2011. For example, of the 164  measures used or planned for use, 44 were used in CMS’s Medicare and  Medicaid EHR Incentive Program after being retooled—that is, converted  to an electronic format that is compatible with EHRs—under the NQF  contract. Although these 44 retooled measures were used in the EHR  Incentive Program, HHS officials stated that NQF and HHS detected  coding and other errors in the versions of the 44 retooled measures that  were published in the program’s final rule in July 2010 that required NQF  to make corrections to them after publication of the final rule. NQF did not  submit the revised versions of the 44 retooled measures published in the  final rule to HHS until December 2010. HHS officials stated that because  the final rule had already been published prior to receiving the final  formatted measures, CMS listed general guidance on its website to  address the errors. HHS officials told us that these 44 measures are  being used but have not yet been tested to assess the feasibility of  implementing them in the electronic format. Until the testing is complete,  HHS runs the risk that some of these measures may not work as intended  when implemented in electronic format for performance measurement. As  a result, the agency does not have reasonable assurance that the  retooled versions of the measures will correctly capture information from  EHRs. In addition to the 44 retooled measures used in the EHR Incentive  Program, HHS also has used or planned to use 120 measures that it  received from endorsement and maintenance projects under the NQF  contract for various HHS programs and initiatives. (See table 2 for details  on specific programs in which HHS has used or planned to use health  care quality measures received from NQF under the contract.)

HHS officials told us that they expect to evaluate if and how they could  use all of the remaining 180 of the 344 quality measures that were  endorsed, maintained, or retooled under the NQF contract that are not  currently in use or planned for use in HHS programs or initiatives.  According to HHS officials, any measure developer can submit a measure  to be considered for NQF endorsement. Therefore, all the measures  received under the contract may not be applicable to a particular HHS  health care quality program or initiative. HHS officials told us that they will  review the remaining 180 measures to determine if they are applicable to  their health care quality programs or initiatives. The officials expect that  many of these measures will be used in HHS programs or initiatives  required by PPACA. For example, HHS officials told us that they will  consider implementation of most of the retooled measures in future  stages of the EHR Incentive Program. In addition, PPACA directed HHS  to establish a hospital value-based purchasing program, as well as to  make plans or begin pilot programs for value-based purchasing in other  settings of care. The hospital value-based purchasing program will use  various quality measures and depend on the information collected on  them to determine payments to providers. PPACA also required the  development of no less than 10 provider-level outcome measures for  hospitals and physicians by March 2012. Further, PPACA directed HHS  to identify quality measures that could be used to evaluate hospice  programs and publish these measures by October 1, 2012. HHS officials  told us that they are in the process of determining whether or to what  extent the remaining 180 measures HHS has received under the NQF  contract can be used to address the new measurement needs and  priorities established by PPACA. HHS officials told us that they prefer to  use NQF-endorsed measures to meet HHS’s measurement needs  because these quality measures are nationally recognized standards and  in some cases HHS is required to use them.

section_title: HHS Does Not Have a Comprehensive Plan for Determining How It Will Use NQF’s Work under the Contract to Implement PPACA Requirements Related to Quality Measurement
paragraphs: Although HHS has taken steps to determine how it can use the measures  received under the contract with NQF, the agency does not have a  comprehensive plan for determining how it will use the remainder of the  work conducted under NQF’s contract to implement PPACA  requirements, including plans for additional quality measures that need to  be endorsed during the remaining contract years. HHS officials told us  that HHS determines on an annual basis which activities—including work  on quality measures—NQF is to perform under the contract through the  interagency workgroup. The workgroup is comprised of representatives  from various HHS agencies and allows them to provide input on their  needs, including quality measures that need endorsement from NQF, for  their respective programs. However, HHS officials told us that each HHS  program assesses its quality measurement needs separately and  provides varying levels of detail about its needs. Therefore, it is unclear  the extent to which all programs consistently incorporate PPACA’s quality  measurement requirements and deadlines into these assessments. The  NPP’s September 2011 report noted the importance of greater alignment  of national quality measurement efforts, including the establishment of a  comprehensive measurement strategy that identifies core measure sets,  among other things. In addition, the report noted that all federal programs  should work toward the establishment of a common platform for  measurement and reporting. Without a comprehensive plan that  delineates HHS’s quality measurement needs, and given that each  program assesses its quality measurement needs separately, the  interagency workgroup may not be able to systematically ensure that all  of HHS’s quality measurement needs that implement PPACA  requirements align with the selection and prioritization of activities for  NQF to complete under the contract.

While HHS has begun various efforts to assess its quality measurement  needs, the lack of a plan that comprehensively determines the impact of  PPACA on its needs could affect the agency’s progress on its quality  measurement efforts as well as how it selects and prioritizes NQF’s  contract activities. Officials told us that prior to PPACA’s enactment, CMS  maintained a 5-year plan that listed its measurement needs based on  agency priorities and the priorities established by the NPP for some of its  programs.to reflect the requirements related to quality measurement and time  frames established by PPACA. In March 2011, HHS published the  National Quality Strategy as required by PPACA, which included six  priority areas of focus. The report was required by PPACA to include  agency-specific plans, goals, benchmarks, and standardized quality  metrics for each priority area, but did not do so. HHS officials stated that  this document describes HHS’s initial plan for these elements and that  they may be included in future versions of the strategy. In June 2011,  HHS officials told us that they plan to convene a Quality Measurement  Task Force within CMS with a goal to comprehensively align, coordinate,  and approve the development, maintenance, and implementation of  health care quality measures for use in various CMS programs. As of  August 2011, the task force was in an early stage of development, and  therefore it is too early to determine whether it will accomplish its goal.  Although these various HHS efforts are key steps toward helping the  agency meet its quality measurement needs, they are not guided by a  comprehensive plan that synthesizes key priority areas identified in  various sources, such as those reported by the NPP or in the National  Quality Strategy, for which measures may be needed. Without such a  plan, HHS may be limited in its efforts to prioritize which specific  measures it needs to develop and have endorsed by NQF for its health  care quality programs and initiatives established by PPACA. As a result,  HHS may be unable to ensure that the agency receives the quality  measures needed to meet PPACA requirements and specified time  frames related to quality measurement.

section_title: Conclusions
paragraphs: Health care quality measures are increasingly important to HHS as it uses  and will continue to use them in its existing and forthcoming programs  and initiatives to evaluate health care delivery. For example, HHS’s value- based purchasing programs are pay-for-performance programs that will  require providers to collect and report information on health care quality  measures and adjust payment levels based on providers’ performance  against the measures. PPACA has increased HHS’s quality measurement  needs, and the time frames specified in the law have also increased the  urgency of obtaining endorsed quality measures—which are nationally  recognized standards and in some cases are required by statute—to  meet these needs. Given that NQF is the entity in the United States with  lead responsibility for endorsing health care quality measures, NQF’s  endorsement activities under the contract are of key importance to help  meet HHS’s quality measurement needs.

However, NQF’s endorsement process takes time. For more than half of  the projects, including all five projects in the endorsement activity, NQF  did not meet or did not expect to meet the initial time frames approved by  HHS. In addition, projects under three of the contract activities have  exceeded initial cost estimates, which resulted in HHS’s modification of  NQF’s scope of work in some instances to help ensure that NQF’s costs  did not exceed the funding allocated for the contract activities. While HHS  received information in monthly progress reports to help monitor NQF’s  performance under the contract, the agency did not use all of the  monitoring tools required under the contract to help address issues  related to time lines and cost estimates. These monitoring tools included  an annual performance evaluation that could help HHS officials consider  potential issues related to NQF’s time frames and cost estimates when  planning work for the next year and a financial graph to be included in  NQF’s monthly progress reports. The graph would have compared  reported costs to initial cost estimates, which is something that monthly  progress reports do not do. Although HHS officials reported that they  recently began in August 2011 to enforce the contractual requirement for  NQF to submit the graph, they have not implemented the required annual  performance evaluation. By not taking advantage of these tools, HHS  runs the risk of not having detailed and timely information that could help  identify instances in which NQF might be at risk of not meeting time  frames or exceeding estimated costs. Identifying such instances could  provide an opportunity for HHS to make any appropriate changes to  NQF’s scope of work, including setting priorities to ensure that HHS  receives the quality measures it needs in a timely manner.

With the time remaining under the contract, HHS has an opportunity to  ensure that the work performed under NQF’s contract better meets the  agency’s needs for its programs and initiatives. However, HHS has not  developed a plan that comprehensively identifies its quality measurement  needs for its programs and initiatives in light of PPACA’s requirements or  determines how it will use the work conducted during the remaining years  of the NQF contract to help it meet these needs. In addition, critical tasks  may need to be completed outside of the NQF contract. For example,  HHS requested that NQF retool 113 measures under the contract and  used 44 of the 113 measures that included errors in its EHR Incentive  Program. As of November 2011, feasibility testing related to  implementation of the retooled measures had not been completed, and  HHS expected to perform this work outside of the NQF contract. Until the  testing is completed, HHS runs the risk that some of the retooled  measures may not work as intended when implemented in electronic  format for performance measurement, which is a concern because use of  these measures is an important component of HHS’s long-term goal for  providers to use health information technology (IT) to exchange  information and improve the quality of care.

Without a comprehensive plan, HHS lacks assurance that its selection of  the work to be performed by NQF—and the approximately $55.2 million  that the agency expects to spend for remaining work under the NQF  contract—will be prioritized in the most effective way possible. Given that  PPACA includes time frames for the implementation of quality  measurement programs, NQF’s pace in completing some of the work  under the contract—particularly the endorsement activity—raises  concerns. If the endorsement projects continue to require extended  completion times, HHS runs the risk of not having all the endorsed  measures it needs for implementing its programs and initiatives. Should  this occur, HHS may need to select nonendorsed measures for its  programs and initiatives that have not undergone an objective and  transparent review by NQF.

section_title: Recommendations for Executive Action
paragraphs: To help ensure that HHS receives the quality measures it needs to  effectively implement its quality measurement programs and initiatives  within required time frames, we recommend that the Secretary of HHS  take the following three actions: use monitoring tools required under the NQF contract to obtain  detailed and timely information on NQF’s performance and use that  information to inform any appropriate changes to time frames,  projects, and cost estimates for the remaining contract years; ensure that testing of the electronic versions of the measures retooled  by NQF that are being used or are planned for use in the Medicare  and Medicaid EHR Incentive programs is completed in a timely  manner to help identify potential errors and address issues of  implementation; and develop a comprehensive plan that identifies the quality measurement  needs of HHS programs and initiatives, including PPACA  requirements, and provides a strategy for using the work NQF  performs under the contract to help meet these needs.

section_title: Agency and Other External Comments and Our Evaluation
paragraphs: We provided a draft of this report to HHS and NQF for review and  comment. HHS neither agreed nor disagreed with our recommendations  and provided general comments. NQF concurred with many of the  findings in the report and provided clarification and additional context on  the findings and recommendations. HHS and NQF’s letters conveying  their comments are reproduced in appendixes V and VI, respectively. In  addition to the overall comments discussed below, we received technical  comments from HHS and NQF, which we incorporated into our report as  appropriate.
subsections: section_title: HHS Comments
paragraphs: HHS’s comments included separate general comments from CMS and  ASPE that provided context on aspects of our findings and  recommendations. CMS’s comments stated that the draft report suggests  that CMS must use all of the measures endorsed by NQF, and noted that  not all NQF-endorsed measures are suitable for HHS quality reporting  and public reporting programs. Although our draft report did not state that  CMS must use all of the measures endorsed by NQF, we modified it to  note specifically, among other things, that all measures received under  the contract may not be applicable to a particular HHS health care quality  program or initiative. CMS also stated that the report suggests that CMS  has not developed measurement plans for various provisions of PPACA  related to quality reporting, public reporting, and value-based purchasing  programs. CMS provided additional context for current planning efforts to  address these requirements, including its Quality Measurement Task  Force. The draft report acknowledged this and other CMS planning efforts  to address the health care quality requirements contained in PPACA and  noted that, as of August 2011, this initiative was just beginning. Further,  while various efforts are underway and CMS’s comments state that it has  documented how quality measures will be used to address all relevant  provisions of PPACA, CMS has not provided documentation of  comprehensive plans to address PPACA requirements that include  alignment across programs, detailed time frames to meet PPACA  deadlines, or how it will use the NQF contract to help ensure that it  receives the endorsed measures it needs to meet these requirements.

ASPE’s comments noted, with respect to our first recommendation, that  HHS used all except two of the monitoring tools called for in the contract.  As noted in the draft report, HHS began receiving the monthly financial  graph—one of the two monitoring tools—from NQF in August 2011. Also,  ASPE noted its plans to update its performance evaluation system with  NQF performance information for the first 2 contract years—the period  January 14, 2009, through January 13, 2011—and to complete a final  performance evaluation at the end of the contract in January 2013, which  is the end of the fourth contract year. It did not indicate any plans to  conduct the annual performance evaluation for the third contract year— January 14, 2011, through January 13, 2012—which would be consistent  with the contract’s requirements. With respect to our second  recommendation, ASPE provided technical comments and also told us  that CMS issued a contract solicitation to test the retooled measures, but  CMS did not receive any bids. Instead, ASPE noted in its comments that  two of CMS’s current contractors will conduct feasibility testing on 69 of  the 113 retooled measures that are planned for use in HHS’s EHR  Incentive programs. CMS does not plan to issue a solicitation for a new  contract to test the feasibility of the remaining 44 retooled measures,  which are currently being used in HHS’s EHR Incentive Program. We  noted these comments in the report. Regarding our third  recommendation, ASPE stated that the measures that are not currently in  “use” are being evaluated by HHS and that any conclusions that they will  not be used are not accurate. Our draft report provided information on  which measures were used or planned for use as of August 2011, and  indicated that the remaining measures may be used in the future.  Specifically, the report noted that HHS officials expect that many of these  measures will be used in HHS programs or initiatives, and that HHS  officials told us that they will review all the measures received under the  contract to determine if they are applicable to their health care quality  programs or initiatives. ASPE’s comments also noted that our draft report  did not include information on all NQF-endorsed measures used by the  various agencies within HHS. As noted in the draft report, we relied on  HHS to identify programs and initiatives across HHS that use or plan to  use these health care quality measures and recognize that those included  in our report may not represent a comprehensive list of all health care  quality programs and initiatives. As we recommended in our report,  having a comprehensive plan could help HHS identify programs or  initiatives that use or plan to use health care quality measures, including  those endorsed by NQF.

section_title: NQF Comments
paragraphs: NQF’s comments state that it is providing its services to HHS under a  cost reimbursement contract, which is used in circumstances where  aspects of performance, such as time frames, cost estimates, and scope  of work, cannot be reasonably estimated, and therefore, should not be  expected. As noted in the draft report, the contract type used for this work  is used for efforts such as research, design, or study efforts where costs  and technical uncertainties exist and it is desirable to retain as much  flexibility as possible in order to accommodate change. However, the draft  report also noted that this type of contract provides only a minimum  incentive to the contractor to control costs. Given the risk associated with  this type of contract, the fact that NQF has not met expected time frames  on about half of its projects as of August 2011, and that NQF exceeded  its initial cost estimates for some of its projects under its contract  activities, it is especially important that HHS obtain detailed and timely  information on NQF’s performance and use that information to inform any  appropriate changes to time frames, projects, and cost estimates for the  remaining contract years, as noted in our recommendations. NQF’s  comments also state that time frames and costs for the work performed  under the contract were initial estimates based on an early understanding  of the work, that HHS and NQF understood that there would likely be  changes to them as a result of the complexity and novelty of the work,  and that they have worked collaboratively throughout the contract period  to address these and other factors. As noted in the draft report, the final  work plans, the technical proposal, and other documents that we  reviewed included initial time frames for all projects and costs for the work  performed during the contract year that were approved by HHS in  collaboration with NQF. The draft report also notes several examples of  reasons why the time frames and costs were modified over time.  Contributing factors include the high volume of measures submitted,  changes to the scope of work, and the novelty and complexity of the  work.

We are sending copies of this report to the Secretary of Health and  Human Services and other interested parties. In addition, the report will  be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Office  of Congressional Relations and Office of Public Affairs can be found on  the last page of this report. Other major contributors to this report are  listed in appendix VII.

section_title: Appendix I: Health Care Quality Measurement Framework
paragraphs: Figure 5 illustrates a health care quality measurement framework of the  various stages that a quality measure will go through, as described by the  Department of Health and Human Services (HHS) and National Quality  Forum (NQF) officials and others. These stages include measure  development, endorsement, selection, and use, among others. This  framework also shows examples of which entities, including HHS and  NQF, are involved in each of the stages.
subsections: section_title: HHS’s Role in Planning and Identifying Gap Areas
paragraphs: As an example of actions taken during the second stage of this quality  measurement framework, HHS officials described two different processes  used for planning and identifying gap areas.

The Centers for Medicare & Medicaid Services (CMS) Office of Clinical  Standards and Quality has developed a standardized approach to  identify quality measures that it uses in its health quality initiatives and  programs using CMS’s Measures Management System. The Measures  Management System requires the convening of a technical expert panel  in the initial planning stage. Once convened, the technical expert panel is  expected to work with measure developers who will gather information  that will help the panel determine whether measures need to be  developed for a program or initiative. During this stage, measure  developers may conduct environmental scans or literature reviews, to  determine the existence of measures that could be used for a program or  initiative. If a measure does not exist, then the developer will work with  CMS to develop the needed measures for the program or initiative,  including measure testing. Upon development of the measures, the  technical expert panel will evaluate them based on (1) importance to  making significant gains in health care quality and improving health  outcomes, (2) scientific acceptability of the measure properties including  tests of reliability and validity, (3) usability, and (4) feasibility. Measures  recommended by the panel are generally submitted for NQF  endorsement.

In contrast, CMS’s Center for Medicaid, Children’s Health Insurance  Program (CHIP), and Survey & Certification Office—the CMS center  which implements CHIP—uses a measure identification process that  relies on existing measures rather than development of new measures,  according to officials. This office worked with a technical advisory group,  the Subcommittee on Children’s Healthcare Quality Measures for  Medicaid and CHIP, to recommend an initial core set of measures for the  CHIP. With assistance from CMS, the subcommittee evaluated measures  based on importance, validity, and feasibility. CMS officials told us that  they considered existing NQF-endorsed and non-NQF-endorsed  measures based on the measurement needs of the program, and relied  on measure testing conducted by the measure developers. Officials  stated that they have also relied on the subcommittee to evaluate  candidate measures for Medicaid child health programs. Officials said  that they are not required to submit measures that will be used for  Medicaid programs for NQF endorsement.

section_title: Appendix II: How GAO Categorized NQF’s 16 Tasks under the Contract into Nine Contract Activities
paragraphs: From January 14, 2010, through August 31, 2011, the National Quality  Forum’s (NQF) contract with the Department of Health and Human  Services (HHS) included 16 tasks that NQF is required to perform. For  purposes of our work, we categorized these tasks into nine contract  activities. Specifically, in certain cases, we grouped activities that covered  related areas of work into a single contract activity. For example, we  consolidated the six administrative activities NQF is required to perform  into a single contract activity. (See table 3 that shows how we  consolidated these contract activities.)

NQF was required to perform specific projects under the nine contract  activities we identified. For example, under the endorsement contract  activity, NQF was required to complete an endorsement project related to  patient outcome measures. For purposes of our work, we identified and  reviewed 63 projects NQF is required to perform under the nine contract  activities, as shown in appendix III.

section_title: Appendix III: Status of NQF’s Nine Contract Activities and the Projects under Each Activity GAO Identified
paragraphs: The tables below provide a status update on the projects that the National  Quality Forum (NQF) is required to complete under the nine contract  activities we identified (see app. II). The contract activities and the  projects under the activities NQF is expected to perform are determined  on an annual basis by the Department of Health and Human Services  (HHS) and NQF. As a result, the number of projects under the contract  activities varies by contract year. For our reporting period—January 2010  through August 2011—we determined that NQF was required to conduct  work on 63 projects under the contract activities we reviewed. To  determine initial time frames for each project, we calculated the  approximate time between expected start and end dates established in  NQF’s 2009, 2010, and 2011 final annual Medicare Improvements for  Patients and Providers Act of 2008 (MIPPA) work plans, the 2011 Patient  Protection and Affordable Care Act (PPACA) technical proposal, and  other NQF documents. Actual time frames were determined by  calculating the approximate time between the actual start date and the  actual date of completion. For projects that were not yet complete as of  August 2011, we included an expected time frame based on the  approximate difference between the actual start date and the expected  date of completion. NQF and HHS officials stated that any changes to the  initial time frames were approved by HHS.

section_title: Appendix IV: Summary of Projects to Review the National Quality Forum’s Endorsement Process
paragraphs: As part of a project under its contract with the Department of Health and  Human Services (HHS), NQF was required to review its endorsement  process. To complete this project, the National Quality Forum (NQF)  subcontracted with Mathematica Policy Research, Inc. (Mathematica), to  conduct a review of NQF’s endorsement process, as requested by HHS.  HHS officials stated that, given the importance of the endorsement  process as part of the health care quality measurement framework, they  requested that an objective and thorough review of NQF’s endorsement  process that focused on timeliness, efficiency, and effectiveness should  be conducted. For example, they stated that they were interested in  whether there were any efficiencies that could be implemented to shorten  the process while maintaining an objective review of the health care  quality measures that were evaluated under the process. Mathematica  initiated its review of NQF’s endorsement process in October 2009 and  completed the work in December 2010.

In December 2010, Mathematica submitted a final report to NQF and  recommended eight areas where improvements could be made and  inefficiencies could be addressed in the endorsement process. In the final  report, Mathematica noted that the current process is lengthy and the  timeliness of the endorsement projects varies substantially. The report  further noted that the length of the endorsement process affects the  availability of endorsed measures for end users, such as HHS. To help  reduce the time required to complete projects, Mathematica  recommended that NQF create a schedule for its endorsement process  for measure developers and develop feasible time lines that include clear  goals for each endorsement project.

As of May 2011, NQF officials stated that NQF has taken steps or plans  to take steps in its future projects to address the eight areas for  improvement Mathematica identified. For example, as of May 2011, NQF  has solicited measures earlier based on a tentative annual project  schedule to reduce the time lines of its endorsement process and  reduced the period for voting by NQF member organizations from 30 to  15 days. NQF officials stated that they believe their efforts to implement  the recommendations will shorten the time lines for the endorsement  projects by 3 to 4 months without compromising the integrity of the  endorsement process and measures to be evaluated under the process.  HHS officials stated Mathematica’s recommendations were valuable  because much of the work under the NQF contract needs to be  completed in an accelerated time line to help fill critical measurement  gaps associated with HHS’s health care quality programs and initiatives.  They noted that it is too soon to tell the effects of these changes on the  endorsement process, but they plan to monitor implementation of the  changes in NQF’s 2011 endorsement projects under the contract. In  addition, as of September 2011, HHS approved a new project under the  contract to identify how the endorsement process can best align with  HHS’s time frame for needed measures. As part of this project, NQF is  expected to work with a consulting group to identify key performance  metrics and define milestones and time lines to help streamline its  endorsement process.

section_title: Appendix V: Comments from the Department of Health and Human Services

section_title: Appendix VI: Comments from the National Quality Forum

section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Will Simerl, Assistant Director;   La Sherri Bush; Krister Friday; Amy Leone; Carla Lewis; John Lopez;  Elizabeth Martinez; Lisa Motley; Teresa Tucker; Carla Willis; and William  T. Woods made key contributions to this report.


Question-summary annotations: questions: question: What issues are there with NQF's retooled measures?
answer: For example, HHS used 44 measures that NQF retooled under the contract in its EHR Incentive Program. HHS officials stated that the 44 measures used in the program contained errors, which required corrections.
child_questions: question: Why might it be risky for HHS to use these retooled measures?
answer: HHS officials also have not yet tested the retooled measures to assess the feasibility of implementing them in the electronic format; therefore, HHS runs the risk that some of these measures may not work as intended when implemented.

question: To what extent had HHS planned to use NQF’s retooled measures?
answer: HHS had used or planned to use about half of the measures—164 of 344—that it received from NQF under the contract, as of August 2011.

question: What have HHS officials told GAO regarding the remaining HHS measures?
answer: HHS officials told GAO they expect to evaluate if and how they could use all of the remaining measures HHS received under the contract.
child_questions: question: What has HHS still not determined or established?
answer: However, HHS has not determined how PPACA requirements for quality measurement may have changed its needs for endorsed quality measures. As a result, HHS has not established a comprehensive plan that identifies its measurement needs and time frames for obtaining endorsed measures and that accounts for relevant PPACA requirements.
child_questions: question: How does a lack of a comprehensive measurement plan affect HHS?
answer: Without such a plan, HHS may be limited in its efforts to prioritize which specific measures it needs to develop and to have endorsed by NQF during the remainder of the NQF contract. As a result, HHS may be unable to ensure that the agency receives the quality measures needed to meet PPACA requirements, including time frames for implementing quality measurement programs.

questions: question: What did MIPPA ask HHS to do?
answer: The Medicare Improvements for Patients and Providers Act of 2008 (MIPPA) directed the Department of Health and Human Services (HHS) to enter into a 4-year contract with an entity to perform various activities related to health care quality measurement.
child_questions: question: With what entity did the HHS decide to work?
answer: In January 2009, HHS awarded a contract to the National Quality Forum (NQF), a nonprofit organization that endorses health care quality measures—that is, recognizes certain ones as national standards.

question: What did PPACA do?
answer: In 2010, the Patient Protection and Affordable Care Act (PPACA) established additional duties for NQF.

question: What does GAO examine in its report?
answer: This is the second of two reports MIPPA required GAO to submit on NQF’s contract with HHS. In this report—which covers NQF’s performance under the contract from January 14, 2010, through August 31, 2011—GAO examines (1) the status of projects under NQF’s required contract activities and (2) the extent to which HHS used or planned to use the measures it has received from NQF under the contract to meet its quality measurement needs, as of August 2011.
child_questions: question: What resources does GAO use in making its report?
answer: GAO interviewed NQF and HHS officials, reviewed relevant laws, and reviewed HHS and NQF documents.