Medicare Physician Feedback Program: CMS Faces Challenges with Methodology and Distribution of Physician Reports
Published: 2011-08-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Medicare Improvements for Patients and Providers Act of 2008 (MIPPA) directed the Department of Health and Human Services (HHS) to develop a program to give physicians confidential feedback on the resources used to provide care to Medicare beneficiaries. In response, HHS's Centers for Medicare & Medicaid Services (CMS) has established and implemented the Physician Feedback Program by distributing feedback reports to an increasing number of physicians that provided data on resources used and the quality of care. MIPPA mandated that GAO conduct a study of this program. To address this mandate, GAO identified (1) methodological challenges CMS faces in developing feedback reports and approaches CMS has tested to address them and (2) challenges CMS faces in distributing feedback reports and CMS's plans to address them. GAO interviewed CMS officials and representatives from the program contractor and reviewed relevant documentation.
section_title: What GAO Found
paragraphs: CMS faces challenges incorporating resource use and quality measures for physician feedback reports that are meaningful, actionable, and reliable. CMS had difficulty measuring the resources used by physicians to treat specific episodes of an illness, such as a stroke or a hip fracture, and the quality measures it used in the program's most recent phase applied to a limited number of physicians. CMS must also make decisions to address several other methodological challenges with developing feedback reports: how to account for differences in beneficiary health status, how to attribute beneficiaries to physicians, how to determine the minimum number of beneficiaries a physician needs to treat to receive a report, and how to select physicians' peer groups for comparison. These decisions involve trade-offs; for example, a higher minimum case size requirement increases the reliability of the information in the reports, but it decreases the number of physicians eligible to receive one. While CMS has tested different approaches to measuring and comparing physician performance, methodological difficulties remain in developing feedback reports. CMS also faced challenges distributing feedback reports to physicians that its plans for improvement may not entirely address. In the most recent phase of the program, about 82 percent of physicians in CMS's sample were not eligible to receive a report after CMS's methodological decisions were applied. CMS plans to make a number of methodological changes in the next phase, but significantly increasing eligibility will continue to be challenging. The electronic distribution of feedback reports also presented multiple challenges that resulted in few physicians accessing their electronic reports in the most recent phase. Factors that may have contributed to this low access rate include CMS's difficulty in obtaining accurate contact information, burdensome methods for electronic distribution, and lack of a strong incentive for physicians to review their reports. CMS conducted limited follow-up with physicians for whom feedback reports were produced. CMS plans to use a new distribution method in a four-state region in the next reporting phase.
section_title: What GAO Recommends
paragraphs: GAO is recommending that CMS use methodological approaches that increase physician eligibility for reports, statistically analyze the impact of its methodological decisions on report reliability, identify and address factors that may have prevented physicians from reading reports, and obtain input from a sample of physicians on the usefulness and credibility of reports. CMS concurred with these recommendations.

section_title: Letter
paragraphs: In recent years, we and other federal fiscal experts—including the  Congressional Budget Office (CBO) and the Medicare Trustees—have  noted the rise in Medicare spending and the serious long-term financial  challenges the program faces. Physicians play a central role in the  generation of Medicare expenditures both through the services they  provide and the services they order, including hospital admissions,  diagnostic tests, and referrals to other physicians. There is evidence that  not all of these services may be necessary or appropriate, and that  greater spending does not necessarily result in better health outcomes.  As a result, policymakers have been exploring methods to reduce costs  and encourage physicians to practice efficiently—that is, to provide and  order only those services that are necessary, sufficient, and appropriate  to meet a beneficiary’s health care needs.
Efficiency may be encouraged by physician profiling, which measures and  compares a physician’s performance to a benchmark, such as the  performance of his or her peers. Certain public and private health care  purchasers routinely profile physicians in their networks and use the  results for a number of purposes, including developing physician “report  cards” or feedback reports and placing physicians in tiered networks that  can be used to steer patients toward the most efficient providers. We and  others have recommended that the Centers for Medicare & Medicaid  Services (CMS), the agency within the Department of Health and Human  Services (HHS) that administers the Medicare program, profile physicians  and provide them with feedback on their use of health care resources to  help identify and reduce overuse of Medicare services. In addition to  profiling physicians on the resources used to provide care to Medicare  beneficiaries, they can also be profiled on the quality of that care. Some  specialty societies have called for the inclusion of quality measures in  feedback reports and cautioned that focusing solely on costs could create  a disincentive to providing appropriate, high-quality care.
The Medicare Improvements for Patients and Providers Act of 2008  (MIPPA) required HHS to establish and begin implementing by January 1,  2009, a Physician Feedback Program that would include distribution of  confidential feedback reports to physicians on the resources used to  provide care to Medicare beneficiaries. MIPPA gave HHS the flexibility to  apply the program to certain types of physicians, such as those who treat  conditions that have high costs, and also provided flexibility on whether to  provide reports to physician groups and whether to include information on  quality. Because developing feedback reports requires a number of  methodological decisions, such as selecting performance measures that  accurately reflect physicians’ resource use and quality of care, CMS has  implemented the program in phases by testing different approaches for  developing feedback reports and distributing reports to a small number of  physicians and physician groups.
The Patient Protection and Affordable Care Act (PPACA), which was  enacted in 2010, directed HHS to adjust Medicare payments to  physicians based on the quality of care provided compared to the cost  using a “value-based payment modifier.” HHS is directed to begin paying  a limited group of physicians and physician groups differentially using the  payment modifier on January 1, 2015, and all physicians and physician  groups by January 1, 2017. The law also states that HHS is to  coordinate the Physician Feedback Program with the value-based  payment modifier. CMS has said that it intends to use the quality and  cost measures from the Physician Feedback Program to develop the  payment modifier and plans to distribute at least one feedback report to  physicians before paying them differentially based on their performance.
MIPPA mandated that GAO conduct a study of the Physician Feedback  Program and report on our findings no later than March 1, 2011. To  respond to this requirement, we conducted a series of briefings for  congressional staff on our preliminary findings beginning in February  2011. This report contains information we provided during those briefings  as well as additional information. Specifically, we (1) identified  methodological challenges CMS faces in developing physician feedback  reports and the approaches CMS has tested to address them and   (2) identified challenges CMS faces in distributing physician feedback  reports and CMS’s plans to address them.
To address these objectives, we interviewed relevant CMS officials and  representatives from Mathematica Policy Research, Inc. (Mathematica),  the contractor that assisted with the development and testing of different  methodologies and distribution methods for the Physician Feedback  Program. We reviewed internal agency reports and relevant studies,  including reports by CMS contractors and the Medicare Payment  Advisory Commission (MedPAC), summaries of comments provided by  physicians who received feedback reports from CMS, and public  comments submitted by medical specialty societies and other  stakeholders in response to the portion of CMS’s 2011 proposed  physician fee schedule rule related to the Physician Feedback Program.  In addition, we attended a CMS listening session on the Physician  Feedback Program, at which representatives of medical specialty  societies and other stakeholders commented on the methodological  approaches CMS is considering in developing feedback reports. We  limited our study to challenges with feedback report methodology and  distribution as our initial audit work indicated that these were the primary  challenges faced by the agency in its implementation of the Physician  Feedback Program. Our work is based on the most current information  available as of June 7, 2011.
We conducted this performance audit from June 2010 through August  2011 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Physicians can be profiled on the health care they provide to Medicare  beneficiaries using measures in two performance dimensions: the  resources used to provide care to beneficiaries and the quality of that  care. CMS has established goals and made progress in developing its  Physician Feedback Program.
subsections: section_title: Resource Use Measures
paragraphs: Resource use can be measured using two methods: the per capita  method and the per episode method. The per capita method measures  the resources used by a physician to treat his or her Medicare  beneficiaries over a fixed period of time. By definition, it is a  comprehensive measure of a physician’s practice patterns because it  includes all health care resources used and is generally considered more  straightforward than the per episode method to measure and understand.
The per episode method measures the resource use associated with  treating a specific episode of an illness in a beneficiary—for example, a  stroke or a hip fracture. An episode of care may refer to all services  related to a health condition with a given diagnosis from a patient’s first  encounter with a health care provider through the completion of the last  encounter related to that condition, including postacute services such as  home health, skilled nursing, and rehabilitation. Since this method  provides condition-specific results, it may provide more useful, or  “actionable,” feedback to physicians. Per episode costs are generally  considered more difficult to measure than per capita costs since it can be  challenging to determine whether a particular health care service should  be grouped to one episode of care or another. Per episode costs may be  determined using “episode groupers,” which are software programs that  use diagnosis codes to assign claims to clinically distinct episodes of  care.
Using both the per capita and per episode methods may more fully  capture differences in resource use among physicians. For example, in a  2006 report, MedPAC found that beneficiaries in Miami had significantly  lower per episode costs for coronary artery disease than beneficiaries in  Minneapolis, suggesting that Miami physicians were providing more  efficient care for coronary artery disease. However, MedPAC noted that  because the beneficiaries in Miami had more episodes of care for this  disease, physicians in Miami actually used more health care resources in  total to treat their coronary artery disease beneficiaries than physicians  treating similar patients in Minneapolis. In this case, the per capita  method and the per episode method together would provide a more  complete picture of physicians’ resource use than either method by itself.
section_title: Quality Measures
paragraphs: Health care quality measures can be used to evaluate how well health  care is delivered, and information obtained from such measures can  promote accountability among physicians. Quality measures can be  classified as process or outcome measures. Process measures assess  whether appropriate clinical practices, such as screening and diagnosis,  were followed. An example of a process measure is whether a patient  with high blood pressure received appropriate medication. Outcome  measures assess a patient’s health status after receiving health care  services. An example of an outcome measure is tracking the percentage  of patients who were diagnosed with high blood pressure and whose  blood pressure was adequately controlled during the measurement year.
Efforts are under way by a range of organizations, including CMS and the  National Committee for Quality Assurance (NCQA), to develop measures  of physician quality, and by the National Quality Forum (NQF) to endorse  the quality measures developed by others. For example, NCQA created  the Healthcare Effectiveness Data and Information Set (HEDIS)®, which  is a tool used by over 90 percent of health plans in the nation and  includes measures of both health plan and physician performance. CMS  has developed the Physician Quality Reporting System (PQRS), which is  a quality reporting program that provides an incentive payment to  professionals who satisfactorily report data on quality measures for  covered professional services furnished during a specified reporting  period. CMS also contracted with Masspro, a quality improvement  organization for Massachusetts, to calculate performance rates for the  Generating Medicare Physician Quality Performance Measurement  Results (GEM) project. The GEM project used 2006 and 2007 Medicare  administrative claims data to generate performance rates for 12 process  measures that were drawn from HEDIS®, such as persistence of beta  blocker treatment after a heart attack.
section_title: CMS Physician Feedback Program Goals and Development
paragraphs: CMS established the Physician Feedback Program in 2008 with the goal  of encouraging higher-quality and more efficient medical practice and  creating a transparent process for developing meaningful, actionable, and  fair physician performance indicators that could later be used in CMS’s  value-based purchasing initiative. Feedback reports can help ensure  quality health care and control costs in three ways. First, the feedback  reports are intended to be educational by providing useful information to  physicians on how their resource use and quality of care compare to their  peers’. Second, the reports are intended to be actionable by helping  physicians identify and develop strategies for improving quality and  reducing costs in their practices. Third, the reports are intended to help  physicians become familiar with the resource use and quality measures  that the agency plans to use to adjust their Medicare reimbursement  under the value-based payment program. CMS intends to distribute at  least one feedback report to physicians before paying them differentially  under the value-based modifier.
CMS has implemented the program in phases by distributing feedback  reports to an increasing number of physicians in selected metropolitan  areas. In each phase, CMS conducted pretesting to obtain physicians’  reactions to the methodology and format of mock feedback reports,  distributed feedback reports populated with actual performance data, and  followed up with a sample of the profiled physicians to obtain their input  on the reports. In Phase I, CMS distributed feedback reports to   239 physicians who practiced in one of 12 metropolitan areas. These  reports were distributed in April and August 2009, and included  information about physicians’ resource use but not their quality of care. In  Phase II, CMS expanded the program to distribute feedback reports to 36  physician groups and to 1,641 individual physicians who practiced within  these groups from the same 12 metropolitan areas used in Phase I.  Phase II reports were produced in November 2010, and included  resource use measures and selected quality measures as well as  information on beneficiaries’ hospital admissions. In addition, the reports  to physician groups included hospitalization rates for ambulatory care  sensitive conditions—acute conditions for which effective outpatient care  could have prevented complications or more severe disease. The reports  also contained the average per capita costs of treating Medicare  beneficiaries, as well as per capita costs by specific categories of service,  such as laboratory tests and imaging services. In addition, the reports  provided summary information about the average annual cost of treating  a subset of Medicare beneficiaries with selected common chronic  conditions: congestive heart failure, chronic obstructive pulmonary  disease, coronary artery disease, diabetes, and prostate cancer.
CMS plans to continue to develop feedback reports and distribute them to  an increasing number of physicians and physician groups. It plans to  distribute Phase III reports to about 20,000 physicians in late 2011, and  intends to provide feedback reports to all physicians and physician groups  by 2017.
section_title: CMS Tested Various Approaches but Still Faces Several Methodological Challenges in Developing Physician Feedback Reports
paragraphs: CMS faces challenges in selecting resource use and quality performance  measures that make feedback reports meaningful, reliable, and  actionable. In addition, the agency faces trade-offs in making other key  methodological decisions concerning risk adjustment, attribution of  beneficiaries to physicians, minimum case size, and peer group selection.  While CMS has tested different approaches to developing feedback  reports, challenges remain in making methodological decisions that will  enable CMS to accomplish its program goals.
subsections: section_title: CMS Tested Approaches to Measuring Physician Resource Use and Quality but Still Faces Challenges
paragraphs: Measuring resource use. CMS intends to use both per capita and per  episode methods to measure physicians’ resource use, but it faces  particular challenges in determining per episode costs for the Medicare  population. In Phase I, CMS tested two commercially available episode  groupers, but found that these groupers had the following shortcomings  when used with Medicare claims data:    Because of the prevalence of comorbidities in the Medicare  population, a beneficiary can be treated for several different  conditions concurrently, and it was difficult for the groupers to  determine which services belonged with a given episode.    Because diagnosis coding used for different Medicare claim types  was inconsistent, claims from different sources were not always linked  to the same episode of care, even when they appeared to be clinically  related. For example, hospital, physician, and skilled nursing facility  claims have slightly different diagnostic information.    Because it was difficult to identify the appropriate beginning and end  of an episode involving a chronic condition, the commercial groupers  did not work well to create episodes of care for the Medicare  population, since a significant portion of Medicare beneficiaries have  chronic conditions.
CMS concluded that per episode measurements included in Phase I  reports were inaccurate, and discontinued use of the commercial  groupers. Some medical specialty societies and other stakeholders  commended this decision.
CMS intends for shortcomings to be addressed by the Medicare-specific  episode grouper under development. In September 2010, CMS awarded  four contracts to develop a Medicare-specific episode grouper. CMS  plans to select a grouper developed under one of these four contracts for  future feedback reports. However, it is not clear that all the problems  identified with the commercial groupers can be solved by a Medicare- specific grouper and the timeline for its development is challenging.
In Phase II feedback reports, CMS elected to provide information on  resource use for beneficiaries with five high-cost, high-volume chronic  conditions. Because episode measures were not available, it used per  capita measures as proxies for per episode costs for patients with  diabetes, congestive heart failure, coronary artery disease, chronic  obstructive pulmonary disease, and prostate cancer. These proxies  included all the resources used to treat beneficiaries with these select  chronic conditions, regardless of whether the resource use was related to  that specific condition. CMS officials stated that these proxies were  adequate substitutes for episode-based cost measures for these chronic  conditions. In Phase III reports, CMS plans to provide per capita  information on subgroups with the same chronic conditions as in Phase II  with the exception of prostate cancer.
Measuring quality. CMS faces the challenge of incorporating into its  feedback reports quality measures that are available, apply to specialists,  and provide information on patient outcomes. Phase I reports did not  contain quality measures. In Phase II, CMS included 12 GEM measures  in feedback reports. These measures have the advantage of being  readily available because they are based on claims data. For Phase III,  CMS is considering 28 claims-based quality measures, which are  endorsed by NQF. These 28 measures, most of which are HEDIS®  measures, were vetted by an interagency committee composed of  medical officers and other internal experts who reviewed the  specifications of each measure, including whether the measure was an  appropriate reflection of physician care and whether it was evidence  based.
While a number of quality measures available to CMS for use in feedback  reports are applicable to primary care physicians, there are fewer  measures for specialists. For example, the GEM measures used in   Phase II reports are only applicable to primary care physicians and a  limited number of specialists, such as cardiologists. In addition, the   28 measures CMS is considering for Phase III reports are, as a whole,  mostly applicable to primary care physicians, although individual  measures apply to certain specialists. Some stakeholders have  encouraged CMS to work with specialty societies to develop adequate  quality measures. CMS officials stated that while the agency is willing to  work with these specialty societies to ensure that selected measures  accurately reflect physicians’ practices, CMS prefers to use NQF- endorsed quality measures and many of the measures that specialty  societies have created have not yet achieved NQF endorsement. In  addition, CMS anticipates using PQRS measures that are applicable to  specialists, but according to CMS officials, it has not done so yet because  of limitations with the PQRS program, such as low physician participation  rates. CMS officials said that PQRS has measures that are applicable to  every type of physician, and the agency is working to increase physician  participation in PQRS, which is currently voluntary. They expect program  participation rates to increase when, in 2013, CMS plans to begin  penalizing physicians who fail to report PQRS measures.
In addition, the GEM measures CMS used for Phase II reports are  process measures, which show whether a physician followed generally  accepted recommendations for clinical practice but may not reflect the  impact of the health care services on the health status of a beneficiary.  CMS officials have stated that although there is a need to evaluate  physician quality of care based on outcome measures, there are currently  few suitable measures. NQF has also stated that there is a need to  develop additional outcome quality measures, and funding of $75 million  is authorized for this in each of fiscal years 2010 through 2014. In  addition, CMS officials stated that PQRS contains a number of clinical  outcome measures, and it is likely that moving forward physician  feedback reports will include these PQRS outcome measures.
section_title: Other Key Methodological Decisions Involve Trade- offs, and CMS Has Tested Different Approaches to Inform These Decisions
paragraphs: Determining risk adjustment factors. CMS faces trade-offs in deciding  which factors to use for risk adjustment, which accounts for differences  outside the physician’s control, such as beneficiary health status.  Because sicker beneficiaries are expected to use more health care  resources than healthier beneficiaries, the health status of physicians’  beneficiaries must be taken into account to make meaningful  comparisons among physicians. Without risk adjusting resource use,  physicians who treat sicker beneficiaries could appear to use resources  less efficiently than their peers in their feedback reports. CMS used the  Hierarchical Condition Categories (HCC) model to risk adjust per capita  resource use in the Phase I and Phase II feedback reports. This model  was originally developed for risk adjustment in Medicare managed care.  The HCC model used in Phase II feedback reports is a method of  adjusting for the expected resource use of Medicare beneficiaries based  on the health conditions they experienced during the previous year and  other factors, such as gender and age.
There are trade-offs involved in determining whether to use a prospective  or concurrent risk adjustment model. A prospective model uses risk  factors from a previous period to predict physicians’ spending for a future  period. A prospective model works well for some health conditions, such  as chronic conditions, which are accurate predictors of health spending  not only in the current year, but also in future years. Conversely, a  concurrent model uses factors from the current period to adjust health  spending for that period. The concurrent model may risk adjust health  care costs incurred in the current year more fully by including acute  conditions, such as a broken leg, as well as acute exacerbations of  chronic illnesses, such as hospitalizations resulting from uncontrolled  diabetes. However, it may be appropriate to categorize the expenditures  associated with some complications as part of the physician’s  performance, as opposed to factors outside of the physician’s control that  require risk adjustment. For example, if a beneficiary needed to be  hospitalized because of poorly managed diabetes, it could be appropriate  to hold the physician accountable for those costs. CMS used a  prospective model in Phase II feedback reports.
CMS must also decide which factors, if any, should be added to the HCC  model. Although CMS officials believe the HCC risk adjustment model  adequately risk adjusted per capita costs, some stakeholders have  questioned CMS’s use of the HCC model and have urged CMS to adjust  for additional factors that affect costs that CMS did not include. These  factors include some socioeconomic indicators, patient noncompliance,  and care setting. Some medical specialty societies and other  stakeholders have stated that if CMS does not risk adjust physician  resource use adequately, physicians could be discouraged from treating  atypical or disadvantaged populations that may be more costly to treat.  Although risk adjusting for additional factors could help address these  concerns, there may be a case for not including them. For example,  noncompliance with physicians’ instructions may suggest that physicians  have not adequately educated their patients on the importance of  compliance. CMS officials also explained that they do not want to adjust  for factors that can provide meaningful information about differences in  practice patterns. CMS officials said that they plan to continue using the  HCC model to risk adjust per capita costs in Phase III.
Selecting an attribution method. CMS faces trade-offs in determining  how to assign responsibility, or “attribute” beneficiaries’ care to  physicians, in a way that promotes program goals. Program goals include  maximizing the number of physicians eligible for feedback reports and  encouraging care coordination, while also ensuring that physicians are  not held accountable for care they did not provide or influence. Medicare  fee-for-service beneficiaries may seek care from any Medicare provider  and often receive care from several physicians and other providers. This  makes it difficult to attribute responsibility for all of the health care  provided. Attributing care to the physician who directly provided it may  appear to be straightforward, but it may not adequately reflect relative  responsibility for that care. For example, individual physicians may have  control over some costs directly incurred by another physician by referring  beneficiaries to specialists. Physicians may also indirectly affect other  health care costs by exercising their judgment regarding hospital and  postacute care decisions. As a result, determining to whom a  beneficiary’s care should be attributed is an important methodological  decision.
In Phase I, CMS tested two attribution methods—a single and a multiple  provider attribution method.    A single provider attribution method holds one physician responsible  for all of a beneficiary’s care. This method is designed to identify the  principal “decision maker,” such as the beneficiary’s primary care  physician, and holds this physician responsible for all care provided,  including referrals and services provided by other physicians. The  single provider method CMS tested attributed a beneficiary’s entire  cost of care to the single physician who provided the most evaluation  and management (E&M) services that the beneficiary received.    A multiple provider attribution method holds more than one physician  responsible for the care provided to a beneficiary. This method  assumes that any one physician is unlikely to have complete  responsibility for all of that care. The multiple provider method CMS  tested held all physicians who billed for at least 10 percent of a  beneficiary’s E&M costs partially responsible for that beneficiary’s  care by attributing resource use in proportion to the amount of care  provided by a given physician.
In Phase II, CMS officials used a single provider attribution method. The  agency generally prefers single provider attribution, believing that it  encourages physicians to coordinate care. However, CMS has not  provided evidence that using a single provider attribution method would  lead to increased coordination, and physicians may not accept this  method as a credible way to attribute costs. According to Mathematica  officials, physicians profiled in Phase I generally preferred the multiple  provider attribution method. These physicians’ comments reflected  concerns that it was unfair to attribute other providers’ resource use to  them. Furthermore, most of the physicians and other stakeholders who  provided comments to Mathematica during pretesting in Phase II thought  it inappropriate to be held accountable, even partially, for care provided  by other physicians. Both specialists and primary care physicians told  Mathematica that they did not have control over how another provider  treated a beneficiary. Specialists noted that they treated beneficiaries for  certain conditions and would not have knowledge of or be responsible for  care unrelated to those conditions. Similarly, Mathematica reported that  primary care physicians felt they had little control over the care provided  by the specialists to whom they referred beneficiaries.
Despite physicians’ concerns about being held responsible for care they  did not directly provide, they do have indirect control over some costs  incurred by other providers, such as referrals to specialists and decisions  about hospitalizations. Given that there is no definitive way to determine  which costs a physician was indirectly or directly responsible for, a  multiple provider attribution method may be the more reasonable way to  attribute costs. For example, the multiple provider method CMS tested in  Phase I held physicians accountable for a proportion of the total care  provided to a beneficiary. Under this method, a physician who billed for  70 percent of a beneficiary’s total E&M services was assigned 70 percent  of the total Medicare resources used by that beneficiary—including office  visits, hospitalizations, skilled nursing facility stays, and diagnostic tests  and procedures.
A multiple provider attribution approach also increases the number of  physicians potentially eligible to receive feedback reports. Because  multiple provider attribution holds more than one physician accountable  for a beneficiary’s care, more physicians will have patients attributed to  them, thus increasing the number of physicians eligible for feedback  reports. CMS officials recognize that using a single attribution method will  not allow all physicians to be eligible to receive a report, and noted that it  is likely that some Phase III reports will use a multiple provider attribution  method to assign resource use to physicians.
CMS set a threshold for the minimum amount of care that a physician or  physician group needed to provide in order to be assigned responsibility  for all or part of that beneficiary’s care. For example, in Phase II,  individual physicians needed to bill for at least 20 percent of a  beneficiary’s total E&M costs, and physician groups needed to bill for at  least 30 percent of the total E&M costs in order to be assigned  responsibility for that beneficiary’s care. The minimum threshold was  intended to reduce the likelihood that physicians and groups would be  assigned responsibility for beneficiaries for whom they provided only  minimal care. CMS is considering setting a lower threshold in Phase III to  increase the number of physicians eligible to receive reports.
Determining minimum case size. CMS faces a challenging trade-off in  determining the minimum number of Medicare beneficiaries or episodes  of care a physician must have to produce reliable information without  excluding a large number of physicians—those without enough  beneficiaries or episodes—from receiving a report. A higher minimum  increases the reliability of the information, but decreases the number of  physicians eligible to receive a report. In contrast, decreasing the  minimum case size increases the number of physicians receiving reports  but reduces reliability.
In Phase I, CMS conducted a statistical reliability test to determine the  minimum number of episodes a physician needed to be eligible for a  feedback report. Reliability indicates how confidently one can classify a  physician’s performance relative to that of his or her peers. Estimates for  this test range from zero to one, with an estimate above 0.8 generally  considered a strong indicator of reliability. CMS used an estimate of 0.5— which is considered a moderate level of reliability—to help ensure that  enough physicians would be eligible for Phase I reports. However, few  physicians met the minimum case size requirements for certain episodes,  such as acute myocardial infarctions, even when using this moderate  level of reliability. CMS did not conduct a reliability test to determine the  minimum number of beneficiaries a physician must treat for per capita  cost measurement.
In Phase II, CMS provided feedback reports to physicians with at least 30  Medicare beneficiaries attributed to them. CMS did not conduct reliability  tests for this estimate, stating that a minimum case size of 30 is generally  accepted in the research community. However, as some stakeholders  have noted, the appropriate minimum case size may vary by condition,  suggesting that CMS should instead use a measure of reliability or  precision to establish the appropriate case size. For example, Phase II  reports contained resource use information for five high-cost, high-volume  chronic conditions, and it is likely that different minimum case sizes were  needed to generate reliable information for different conditions, such as  diabetes and coronary artery disease. CMS officials noted that minimum  case size is a major factor in excluding physicians from receiving  feedback reports. CMS officials have considered reducing the minimum  case size from 30 to 20 beneficiaries for Phase III reports. Officials  analyzed the potential effect of this change on individual physicians’ per  capita resource use rankings, and found that nearly all physicians were  ranked in the same quartile when the case size was lowered from 30 to  20. According to CMS officials, this change would increase the number of  physicians eligible to receive Phase III reports by about 10 percent.
Selecting peer groups for comparisons. CMS faces trade-offs in  balancing stakeholders’ preferences that feedback reports compare  physicians only to those most like themselves—that is, peer groups  representing narrow subspecialties or limited geographic areas—with the  need to establish a minimum peer group size that is large enough to  make statistically significant comparisons.
Individual-level feedback reports distributed in Phases I and II contained  two peer group comparisons: (1) physicians in the same specialty in the  same metropolitan area and (2) physicians in the same specialty across  all 12 metropolitan areas, which was meant to serve as a proxy for a  nationwide comparison. Some stakeholders have encouraged CMS to  compare physicians within a limited geographic area. However, if a large  number of physicians in a limited geographic area were practicing  inefficiently, a nationwide sample might be needed to identify the  inefficiencies.
In addition, some medical specialty societies and other stakeholders  urged CMS to compare physicians only within narrow subspecialties. For  example, the American Urological Association noted that surgeons with  active surgery practices are substantially different from those who engage  primarily in medical management of urological conditions, and  comparisons that do not differentiate between these distinct types of  physicians are not meaningful to physicians and do not promote learning  and improvement.
However, if CMS were to identify and compare physicians in smaller  subspecialties, it would face the challenge of ensuring that the peer group  size was large enough to make meaningful comparisons across  physicians. In Phase I, CMS did not impose a minimum peer group size,  but in Phase II it imposed a minimum peer group size of 30 physicians.  However, because not all individual physicians had peer groups  consisting of 30 physicians practicing in the same geographic area and in  the same specialty, some physicians received a report that did not  contain information on all performance measures. CMS officials said they  may use a minimum peer group size of 15 for Phase III feedback reports.
section_title: CMS’s Plans for Improvement May Not Fully Address Challenges in Distributing Reports to Physicians
paragraphs: The majority of sampled physicians were not eligible to receive a Phase II  report after CMS’s methodological decisions were applied. CMS officials  plan to revise their methodology to increase eligibility for Phase III  reports, but significantly increasing the number of physicians who are  eligible will be challenging. Further, CMS faced multiple challenges with  the electronic distribution of feedback reports to eligible physicians, and  as a result, few physicians accessed their reports. CMS officials plan to  use a new distribution method for Phase III reports.
subsections: section_title: Few Sampled Physicians Were Eligible to Receive a Feedback Report; Significantly Increasing Eligibility Will Continue to Be Challenging
paragraphs: Over 80 percent of CMS’s initial sample of 9,189 physicians were  ineligible to receive a Phase II feedback report after CMS’s  methodological decisions, such as minimum case size requirements,  were applied. To identify physicians for the Phase II reports, CMS began  with a sample of 9,189 individual physicians affiliated with 36 physician  groups. To be eligible for a Phase II report, individual physicians needed  to meet CMS’s criteria by having the following:    At least 30 Medicare beneficiaries attributed to them to meet the  minimum case size requirement for per capita resource use  measures. Of the 9,189 physicians in the original sample, 2,205   (24 percent) had at least 30 beneficiaries attributed to them.    At least 11 Medicare beneficiaries attributed to them who were  eligible for 1 or more of the 12 GEM quality measures. Of the  9,189 physicians in the original sample, 2,661 physicians (29 percent)  had at least 11 beneficiaries attributed to them who were eligible for at  least 1 of the 12 GEM quality measures.    A sufficient number of attributed beneficiaries for both the per  capita resource use and GEM quality measures. Of the 9,189  physicians in the original sample, 1,733 physicians (19 percent) had a  sufficient number of beneficiaries attributed to them for the per capita  resource use and GEM quality measures.    At least 30 individual physicians in the same medical specialty  and geographic area for a peer group. Of the remaining 1,733  individual physicians, 1,645 physicians had a peer group of at least 30  individual physicians.
Figure 1 shows the number of physicians excluded by each criterion.
CMS’s methodological criteria also excluded many specialists from  receiving feedback reports. Over 90 percent of Phase II reports were  created for generalists, such as internal medicine or family practice  physicians. The single provider attribution method used by CMS—which  assigned a beneficiary to the single physician who billed for the greatest  number of E&M services for the beneficiary—limited the number of  specialists eligible for a report, since specialists often provide fewer but  more expensive E&M services to beneficiaries than generalists.  Physicians also needed to have at least one GEM quality measure to  receive a Phase II report, but the GEM measures were only applicable to  a limited number of specialists, such as cardiologists and nephrologists.
In addition, many of the 1,641 physicians eligible to receive a Phase II  feedback report did not meet the methodological criteria needed to  receive information on all performance measures, such as resource use  for the five chronic condition subgroups or the 12 GEM quality measures.  For example, only 5 percent of the 1,641 physicians eligible for Phase II  reports were eligible to receive resource use information for their  beneficiaries with chronic obstructive pulmonary disease, and none were  eligible to receive this information for their beneficiaries with prostate  cancer. Similarly, none of the 1,641 physicians eligible for Phase II  reports were eligible to receive information for 3 of the 12 GEM quality  measures. By contrast, the majority of the 36 physician groups profiled  received information on all performance measures (see table 1).
As we stated earlier in this report, CMS is considering a number of  methodological changes in Phase III, such as using a multiple provider  attribution rule and lowering the minimum case size and peer group  requirements. While such changes could lead to a modest increase in  physician eligibility for Phase III reports, significantly increasing  eligibility—particularly for individual physicians with small case sizes—will  continue to be challenging.
section_title: Multiple Challenges with Distribution Resulted in Few Physicians Accessing Their Electronic Feedback Reports
paragraphs: CMS faced multiple challenges distributing Phase II feedback reports,  and as a result of these challenges, few physicians accessed their  reports. In November 2010, CMS mailed letters to 36 physician groups  and 1,641 individual physicians affiliated with those groups to notify them  that electronic feedback reports were available for their review. However,  as of March 2011—approximately 4 months later—less than 60 percent of  physician groups and less than 10 percent of individual physicians had  accessed their reports electronically.
Major challenges with Phase II distribution were CMS’s difficulty obtaining  physicians’ contact information, methods of electronic distribution that  were burdensome for physicians, and lack of a strong incentive for  physicians to review the reports.
Contact information. The lack of a comprehensive database with  accurate names and addresses for physicians and physician groups  made it difficult for CMS to notify physicians and physician groups about  the availability of their feedback reports. Although reports in Phase II were  produced in electronic form, CMS mailed hard copy notification letters to  tell individual physicians and physician groups that an electronic feedback  report was available and to provide instructions for accessing it. Because  available databases contained incomplete or conflicting contact  information, CMS had to use multiple sources, including Internet  searches, to compile names and addresses—a process that took  approximately 5 months.
Despite CMS’s efforts to obtain accurate contact information, some  individual physicians and physician groups did not receive a notification  letter and therefore did not know that a feedback report was available to  them. In follow-up phone calls, CMS found that 27 of the 32 physician  groups reached reported that they had not seen the notification letter and  could not verify whether it had been received. Many of these physician  groups reported that the notification letter was not addressed to the most  appropriate person within the group practice, such as the director of  quality assurance. CMS also called a sample of 10 individual physicians  to ask whether they had received the notification letter. Of these  physicians, 1 was retired, 1 reported not receiving the letter, and the  remaining 8 had no memory of receiving the letter. In addition, nearly   10 percent of the notification letters mailed to individual physicians were  marked undeliverable and returned to CMS.
Distribution method. CMS’s electronic distribution method for Phase II  reports was burdensome for some profiled physicians and physician  groups. CMS transitioned from hard copy distribution of feedback reports  in Phase I to electronic distribution in Phase II based on physicians’  complaints that the reports distributed in Phase I were too long and  cumbersome to manage in hard copy. According to CMS, electronic  distribution was meant to help physicians navigate the reports. CMS used  two methods to electronically distribute feedback reports in Phase II—one  for individual physicians and one for physician groups.
Individual physicians were instructed in the notification letter to contact  their Medicare Administrative Contractor (MAC) to request a copy of their  feedback report. In a report to CMS, Mathematica reported that finding  contact information for the correct MAC may not have been a  straightforward process for physicians. For example, the notification letter  directed physicians to a directory with toll-free phone numbers listed by  state for all MAC contact centers, requiring physicians to choose from  several possible numbers. Mathematica also reported that MAC  customer service representatives were not always aware of the feedback  reports or the process for distributing them to physicians. According to  CMS’s estimate, the majority of individual physicians did not contact their  MACs to request their reports. In February 2011, CMS mailed hard copies  of the 1,596 feedback reports that had not yet been electronically  accessed by individual physicians.
In theory, the electronic distribution method for physician groups should  have been more straightforward since groups were instructed to  download their feedback reports from the Individuals Authorized Access  to CMS Computer Services (IACS) system, which is the same system  used to distribute PQRS reports. However, 8 of the 32 physician groups  CMS reached in its follow-up calls reported difficulty downloading their  reports from the IACS system. For example, some groups did not know  that they needed to register for an IACS account—a process that takes  approximately 10 business days to complete—while others reported not  being able to download their feedback reports even after logging onto the  IACS system. CMS subsequently e-mailed feedback reports directly to  those physician groups that had trouble downloading their reports through  IACS.
CMS officials recognized the limitations with the distribution method for  Phase II reports, and they plan to use a new distribution method for  Phase III reports. CMS currently plans to distribute reports to 20,000  individual physicians in one four-state region—Nebraska, Missouri, Iowa,  and Kansas. According to CMS officials, the MAC serving this region has  e-mail addresses for most physicians in the area. CMS plans to e-mail  Phase III reports directly to physicians in this region, thereby avoiding the  need to mail hard copy notification letters. In addition to distributing  reports to individual physicians in the four-state region, CMS also plans to  distribute Phase III reports to 35 physician groups that have participated  in the PQRS group practice reporting option. CMS intends to e-mail  feedback reports to these 35 physician groups.
Incentive to access reports. Physicians did not have a strong incentive  to access their Phase II feedback reports. The notification letter sent by  CMS said that these reports were “for informational purposes only” and  that they would not affect physicians’ participation in the Medicare  program or their Medicare payments. In pretesting for Phase II, many  physicians noted that they would be unlikely to review a feedback report  closely unless they had an incentive to do so. CMS officials said that they  did not want to emphasize that the types of cost and quality measures  contained in the feedback reports could affect physicians’ payments in the  future because they did not want the reports to sound threatening.  Several physician groups suggested that CMS send feedback reports for  those physicians affiliated with a group practice to the group’s  administrator, noting that individual physicians generally contact their  administrators for guidance on such reports. In a report to CMS,  Mathematica also noted that medical directors or others with quality  oversight responsibilities in larger group practices would be more  receptive to feedback reports than individual physicians. They added that  these individuals are more familiar with the data used to create feedback  reports, and have more experience analyzing quality and cost information  for practice improvement.
CMS made follow-up calls to representatives of 15 of the 36 profiled  physician groups to obtain their input on the feedback reports, but it  conducted minimal follow-up with individual physicians. At the time CMS  attempted to follow up with individual physicians, only 4 had contacted  their MACs to request a feedback report. Three of these physicians were  unwilling to participate in a follow-up call about the report, and 1 physician  was unable to download the feedback report that had been sent via   e-mail. Similarly, CMS called a sample of 10 physicians who had not  requested their feedback reports to ask why they had not done so, and 8  of these physicians expressed no interest in their reports.
section_title: Conclusions
paragraphs: In light of concerns about the long-term fiscal challenges facing the  Medicare program, the Physician Feedback Program is an important  effort that could encourage more efficient medical practice as well as  higher-quality care. CMS has worked under challenging timelines to test  different approaches to feedback report methodology and distribution.  Initial phases of the program indicate that significant changes will need to  be made for the program to meet its goal of producing reports with  meaningful, actionable, and fair performance measures that apply to the  majority of Medicare physicians. CMS will need to do more to solicit input  and reactions from physicians and physician groups on the methodology  and distribution of reports while the stakes are still relatively low—that is,  before CMS begins paying physicians based on their performance on the  resource use and quality measures included in the feedback reports  beginning January 1, 2015.
In the first two phases, CMS tested different methodological approaches  to developing feedback reports; however, the majority of physicians in the  most recent phase were ineligible for a feedback report once CMS’s  methodological criteria were applied. For example, CMS used a single  provider attribution method in the most recent phase, believing that it may  improve care coordination—but this method limited physician eligibility,  and there is limited evidence to suggest that using this method would  increase coordination. And while we also agree with CMS’s decision to  include quality measures in feedback reports, some physicians who  would have been eligible to receive information on their resource use  were disqualified from receiving a Phase II report because they were not  eligible for at least 1 of the 12 GEM quality measures. Further, none of  the individual physicians who were eligible for a Phase II report had  enough beneficiaries attributed to them to receive performance data on all  12 quality measures. CMS did not face such sample size issues in the  feedback reports it developed for physician groups.
CMS has not conducted the rigorous statistical analysis it needs to fully  understand the impact of its methodological decisions on reliability. For  example, CMS used a minimum case size of 30 beneficiaries for Phase II  reports, but did not conduct reliability testing to determine this number.  The results of such testing can and should influence how CMS ultimately  uses the information. Lower levels of reliability may be acceptable if  feedback reports remain confidential and are used solely for educational  purposes. However, since CMS ultimately intends to pay physicians  based on their performance as measured in the feedback reports, it must  be reasonably confident that these measures reflect real differences in  medical practice. It will also be difficult for CMS to obtain physician and  stakeholder buy-in if it does not clearly demonstrate that its performance  measures are reliable and robust.
Furthermore, CMS faces challenges distributing feedback reports to  physicians and physician groups that are eligible to receive them. CMS  transitioned to electronic distribution based on physicians’ complaints that  hard copy reports were too long and cumbersome, yet few physicians  accessed their Phase II reports electronically. Moreover, CMS conducted  limited follow-up with profiled physicians to obtain their input on the  feedback reports. As a result, the agency missed an important opportunity  to increase physician engagement in the program and to ensure that their  concerns are addressed while the program is still in its infancy.
section_title: Recommendations for Executive Action
paragraphs: In order to develop feedback reports that are more reliable, credible,  accessible, and applicable to a greater number of Medicare physicians,  we recommend that the Administrator of CMS take the following four  actions:    Use methodological approaches that increase the number of  physicians eligible to receive a report, such as    multiple provider attribution methods, which could also enhance  credibility of the reports with physicians, and    distributing feedback reports that include only resource use  information, if quality information is unavailable.    Conduct statistical analyses of the impact of key methodological  decisions on reliability.
Identify factors that may have prevented physicians from accessing  their reports and, as applicable, develop strategies to improve the  process for distributing reports and facilitating physicians’ access to  them.    Obtain input from a sample of physicians who received feedback  reports on the usefulness and credibility of the performance measures  contained in the reports and consider using this information to revise  future reports.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from CMS, which  are reprinted in appendix I. CMS concurred with our recommendations  and identified actions agency officials are taking to implement them.  These actions include refining the attribution methodology to increase the  number of physicians receiving feedback reports in Phase III, analyzing  the number of cases required to reliably measure quality and make  credible comparisons, developing new strategies for distributing feedback  reports, and obtaining input from individual physicians and physician  groups about the information contained in the feedback reports. If these  actions are implemented in accordance with our recommendations, CMS  will be better positioned to meet its goals and objectives for the Physician  Feedback Program. CMS also provided technical comments, which we  incorporated as appropriate.
We are sending copies of this report to the Administrator of CMS and  relevant congressional committees. The report also will be available at no  charge on the GAO website at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff members who made major  contributions to this report are listed in appendix II.
section_title: Appendix I: Comments from the Department of Health and Human Services
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Phyllis Thorburn, Assistant  Director; William A. Crafton; Cathleen Hamann; Julian Klazkin; Amanda  Pusey; Jessica C. Smith; and Rachael Wojnowicz made key contributions  to this report.