Medicare Physician Feedback Program: CMS Faces Challenges with Methodology and Distribution of Physician Reports
Released: 2011-08-12T00:00:00

Letter

Background

Physicians can be profiled on the health care they provide to Medicare beneficiaries using measures in two performance dimensions: the resources used to provide care to beneficiaries and the quality of that care. CMS has established goals and made progress in developing its Physician Feedback Program.

Resource Use Measures

Resource use can be measured using two methods: the per capita method and the per episode method. The per capita method measures the resources used by a physician to treat his or her Medicare beneficiaries over a fixed period of time. By definition, it is a comprehensive measure of a physician’s practice patterns because it includes all health care resources used and is generally considered more straightforward than the per episode method to measure and understand.

The per episode method measures the resource use associated with treating a specific episode of an illness in a beneficiary—for example, a stroke or a hip fracture. An episode of care may refer to all services related to a health condition with a given diagnosis from a patient’s first encounter with a health care provider through the completion of the last encounter related to that condition, including postacute services such as home health, skilled nursing, and rehabilitation. Since this method provides condition-specific results, it may provide more useful, or “actionable,” feedback to physicians. Per episode costs are generally considered more difficult to measure than per capita costs since it can be challenging to determine whether a particular health care service should be grouped to one episode of care or another. Per episode costs may be determined using “episode groupers,” which are software programs that use diagnosis codes to assign claims to clinically distinct episodes of care.

Using both the per capita and per episode methods may more fully capture differences in resource use among physicians. For example, in a 2006 report, MedPAC found that beneficiaries in Miami had significantly lower per episode costs for coronary artery disease than beneficiaries in Minneapolis, suggesting that Miami physicians were providing more efficient care for coronary artery disease. However, MedPAC noted that because the beneficiaries in Miami had more episodes of care for this disease, physicians in Miami actually used more health care resources in total to treat their coronary artery disease beneficiaries than physicians treating similar patients in Minneapolis. In this case, the per capita method and the per episode method together would provide a more complete picture of physicians’ resource use than either method by itself.

Quality Measures

Health care quality measures can be used to evaluate how well health care is delivered, and information obtained from such measures can promote accountability among physicians. Quality measures can be classified as process or outcome measures. Process measures assess whether appropriate clinical practices, such as screening and diagnosis, were followed. An example of a process measure is whether a patient with high blood pressure received appropriate medication. Outcome measures assess a patient’s health status after receiving health care services. An example of an outcome measure is tracking the percentage of patients who were diagnosed with high blood pressure and whose blood pressure was adequately controlled during the measurement year.

Efforts are under way by a range of organizations, including CMS and the National Committee for Quality Assurance (NCQA), to develop measures of physician quality, and by the National Quality Forum (NQF) to endorse the quality measures developed by others. For example, NCQA created the Healthcare Effectiveness Data and Information Set (HEDIS)®, which is a tool used by over 90 percent of health plans in the nation and includes measures of both health plan and physician performance. CMS has developed the Physician Quality Reporting System (PQRS), which is a quality reporting program that provides an incentive payment to professionals who satisfactorily report data on quality measures for covered professional services furnished during a specified reporting period. CMS also contracted with Masspro, a quality improvement organization for Massachusetts, to calculate performance rates for the Generating Medicare Physician Quality Performance Measurement Results (GEM) project. The GEM project used 2006 and 2007 Medicare administrative claims data to generate performance rates for 12 process measures that were drawn from HEDIS®, such as persistence of beta blocker treatment after a heart attack.

CMS Physician Feedback Program Goals and Development

CMS established the Physician Feedback Program in 2008 with the goal of encouraging higher-quality and more efficient medical practice and creating a transparent process for developing meaningful, actionable, and fair physician performance indicators that could later be used in CMS’s value-based purchasing initiative. Feedback reports can help ensure quality health care and control costs in three ways. First, the feedback reports are intended to be educational by providing useful information to physicians on how their resource use and quality of care compare to their peers’. Second, the reports are intended to be actionable by helping physicians identify and develop strategies for improving quality and reducing costs in their practices. Third, the reports are intended to help physicians become familiar with the resource use and quality measures that the agency plans to use to adjust their Medicare reimbursement under the value-based payment program. CMS intends to distribute at least one feedback report to physicians before paying them differentially under the value-based modifier.

CMS has implemented the program in phases by distributing feedback reports to an increasing number of physicians in selected metropolitan areas. In each phase, CMS conducted pretesting to obtain physicians’ reactions to the methodology and format of mock feedback reports, distributed feedback reports populated with actual performance data, and followed up with a sample of the profiled physicians to obtain their input on the reports. In Phase I, CMS distributed feedback reports to 239 physicians who practiced in one of 12 metropolitan areas. These reports were distributed in April and August 2009, and included information about physicians’ resource use but not their quality of care. In Phase II, CMS expanded the program to distribute feedback reports to 36 physician groups and to 1,641 individual physicians who practiced within these groups from the same 12 metropolitan areas used in Phase I. Phase II reports were produced in November 2010, and included resource use measures and selected quality measures as well as information on beneficiaries’ hospital admissions. In addition, the reports to physician groups included hospitalization rates for ambulatory care sensitive conditions—acute conditions for which effective outpatient care could have prevented complications or more severe disease. The reports also contained the average per capita costs of treating Medicare beneficiaries, as well as per capita costs by specific categories of service, such as laboratory tests and imaging services. In addition, the reports provided summary information about the average annual cost of treating a subset of Medicare beneficiaries with selected common chronic conditions: congestive heart failure, chronic obstructive pulmonary disease, coronary artery disease, diabetes, and prostate cancer.

CMS plans to continue to develop feedback reports and distribute them to an increasing number of physicians and physician groups. It plans to distribute Phase III reports to about 20,000 physicians in late 2011, and intends to provide feedback reports to all physicians and physician groups by 2017.

CMS Tested Various Approaches but Still Faces Several Methodological Challenges in Developing Physician Feedback Reports

CMS faces challenges in selecting resource use and quality performance measures that make feedback reports meaningful, reliable, and actionable. In addition, the agency faces trade-offs in making other key methodological decisions concerning risk adjustment, attribution of beneficiaries to physicians, minimum case size, and peer group selection. While CMS has tested different approaches to developing feedback reports, challenges remain in making methodological decisions that will enable CMS to accomplish its program goals.

CMS Tested Approaches to Measuring Physician Resource Use and Quality but Still Faces Challenges

Measuring resource use. CMS intends to use both per capita and per episode methods to measure physicians’ resource use, but it faces particular challenges in determining per episode costs for the Medicare population. In Phase I, CMS tested two commercially available episode groupers, but found that these groupers had the following shortcomings when used with Medicare claims data:  Because of the prevalence of comorbidities in the Medicare population, a beneficiary can be treated for several different conditions concurrently, and it was difficult for the groupers to determine which services belonged with a given episode.  Because diagnosis coding used for different Medicare claim types was inconsistent, claims from different sources were not always linked to the same episode of care, even when they appeared to be clinically related. For example, hospital, physician, and skilled nursing facility claims have slightly different diagnostic information.  Because it was difficult to identify the appropriate beginning and end of an episode involving a chronic condition, the commercial groupers did not work well to create episodes of care for the Medicare population, since a significant portion of Medicare beneficiaries have chronic conditions.

CMS concluded that per episode measurements included in Phase I reports were inaccurate, and discontinued use of the commercial groupers. Some medical specialty societies and other stakeholders commended this decision.

CMS intends for shortcomings to be addressed by the Medicare-specific episode grouper under development. In September 2010, CMS awarded four contracts to develop a Medicare-specific episode grouper. CMS plans to select a grouper developed under one of these four contracts for future feedback reports. However, it is not clear that all the problems identified with the commercial groupers can be solved by a Medicare- specific grouper and the timeline for its development is challenging.

In Phase II feedback reports, CMS elected to provide information on resource use for beneficiaries with five high-cost, high-volume chronic conditions. Because episode measures were not available, it used per capita measures as proxies for per episode costs for patients with diabetes, congestive heart failure, coronary artery disease, chronic obstructive pulmonary disease, and prostate cancer. These proxies included all the resources used to treat beneficiaries with these select chronic conditions, regardless of whether the resource use was related to that specific condition. CMS officials stated that these proxies were adequate substitutes for episode-based cost measures for these chronic conditions. In Phase III reports, CMS plans to provide per capita information on subgroups with the same chronic conditions as in Phase II with the exception of prostate cancer.

Measuring quality. CMS faces the challenge of incorporating into its feedback reports quality measures that are available, apply to specialists, and provide information on patient outcomes. Phase I reports did not contain quality measures. In Phase II, CMS included 12 GEM measures in feedback reports. These measures have the advantage of being readily available because they are based on claims data. For Phase III, CMS is considering 28 claims-based quality measures, which are endorsed by NQF. These 28 measures, most of which are HEDIS® measures, were vetted by an interagency committee composed of medical officers and other internal experts who reviewed the specifications of each measure, including whether the measure was an appropriate reflection of physician care and whether it was evidence based.

While a number of quality measures available to CMS for use in feedback reports are applicable to primary care physicians, there are fewer measures for specialists. For example, the GEM measures used in Phase II reports are only applicable to primary care physicians and a limited number of specialists, such as cardiologists. In addition, the 28 measures CMS is considering for Phase III reports are, as a whole, mostly applicable to primary care physicians, although individual measures apply to certain specialists. Some stakeholders have encouraged CMS to work with specialty societies to develop adequate quality measures. CMS officials stated that while the agency is willing to work with these specialty societies to ensure that selected measures accurately reflect physicians’ practices, CMS prefers to use NQF- endorsed quality measures and many of the measures that specialty societies have created have not yet achieved NQF endorsement. In addition, CMS anticipates using PQRS measures that are applicable to specialists, but according to CMS officials, it has not done so yet because of limitations with the PQRS program, such as low physician participation rates. CMS officials said that PQRS has measures that are applicable to every type of physician, and the agency is working to increase physician participation in PQRS, which is currently voluntary. They expect program participation rates to increase when, in 2013, CMS plans to begin penalizing physicians who fail to report PQRS measures.

In addition, the GEM measures CMS used for Phase II reports are process measures, which show whether a physician followed generally accepted recommendations for clinical practice but may not reflect the impact of the health care services on the health status of a beneficiary. CMS officials have stated that although there is a need to evaluate physician quality of care based on outcome measures, there are currently few suitable measures. NQF has also stated that there is a need to develop additional outcome quality measures, and funding of $75 million is authorized for this in each of fiscal years 2010 through 2014. In addition, CMS officials stated that PQRS contains a number of clinical outcome measures, and it is likely that moving forward physician feedback reports will include these PQRS outcome measures.

Other Key Methodological Decisions Involve Trade- offs, and CMS Has Tested Different Approaches to Inform These Decisions

Determining risk adjustment factors. CMS faces trade-offs in deciding which factors to use for risk adjustment, which accounts for differences outside the physician’s control, such as beneficiary health status. Because sicker beneficiaries are expected to use more health care resources than healthier beneficiaries, the health status of physicians’ beneficiaries must be taken into account to make meaningful comparisons among physicians. Without risk adjusting resource use, physicians who treat sicker beneficiaries could appear to use resources less efficiently than their peers in their feedback reports. CMS used the Hierarchical Condition Categories (HCC) model to risk adjust per capita resource use in the Phase I and Phase II feedback reports. This model was originally developed for risk adjustment in Medicare managed care. The HCC model used in Phase II feedback reports is a method of adjusting for the expected resource use of Medicare beneficiaries based on the health conditions they experienced during the previous year and other factors, such as gender and age.

There are trade-offs involved in determining whether to use a prospective or concurrent risk adjustment model. A prospective model uses risk factors from a previous period to predict physicians’ spending for a future period. A prospective model works well for some health conditions, such as chronic conditions, which are accurate predictors of health spending not only in the current year, but also in future years. Conversely, a concurrent model uses factors from the current period to adjust health spending for that period. The concurrent model may risk adjust health care costs incurred in the current year more fully by including acute conditions, such as a broken leg, as well as acute exacerbations of chronic illnesses, such as hospitalizations resulting from uncontrolled diabetes. However, it may be appropriate to categorize the expenditures associated with some complications as part of the physician’s performance, as opposed to factors outside of the physician’s control that require risk adjustment. For example, if a beneficiary needed to be hospitalized because of poorly managed diabetes, it could be appropriate to hold the physician accountable for those costs. CMS used a prospective model in Phase II feedback reports.

CMS must also decide which factors, if any, should be added to the HCC model. Although CMS officials believe the HCC risk adjustment model adequately risk adjusted per capita costs, some stakeholders have questioned CMS’s use of the HCC model and have urged CMS to adjust for additional factors that affect costs that CMS did not include. These factors include some socioeconomic indicators, patient noncompliance, and care setting. Some medical specialty societies and other stakeholders have stated that if CMS does not risk adjust physician resource use adequately, physicians could be discouraged from treating atypical or disadvantaged populations that may be more costly to treat. Although risk adjusting for additional factors could help address these concerns, there may be a case for not including them. For example, noncompliance with physicians’ instructions may suggest that physicians have not adequately educated their patients on the importance of compliance. CMS officials also explained that they do not want to adjust for factors that can provide meaningful information about differences in practice patterns. CMS officials said that they plan to continue using the HCC model to risk adjust per capita costs in Phase III.

Selecting an attribution method. CMS faces trade-offs in determining how to assign responsibility, or “attribute” beneficiaries’ care to physicians, in a way that promotes program goals. Program goals include maximizing the number of physicians eligible for feedback reports and encouraging care coordination, while also ensuring that physicians are not held accountable for care they did not provide or influence. Medicare fee-for-service beneficiaries may seek care from any Medicare provider and often receive care from several physicians and other providers. This makes it difficult to attribute responsibility for all of the health care provided. Attributing care to the physician who directly provided it may appear to be straightforward, but it may not adequately reflect relative responsibility for that care. For example, individual physicians may have control over some costs directly incurred by another physician by referring beneficiaries to specialists. Physicians may also indirectly affect other health care costs by exercising their judgment regarding hospital and postacute care decisions. As a result, determining to whom a beneficiary’s care should be attributed is an important methodological decision.

In Phase I, CMS tested two attribution methods—a single and a multiple provider attribution method.  A single provider attribution method holds one physician responsible for all of a beneficiary’s care. This method is designed to identify the principal “decision maker,” such as the beneficiary’s primary care physician, and holds this physician responsible for all care provided, including referrals and services provided by other physicians. The single provider method CMS tested attributed a beneficiary’s entire cost of care to the single physician who provided the most evaluation and management (E&M) services that the beneficiary received.  A multiple provider attribution method holds more than one physician responsible for the care provided to a beneficiary. This method assumes that any one physician is unlikely to have complete responsibility for all of that care. The multiple provider method CMS tested held all physicians who billed for at least 10 percent of a beneficiary’s E&M costs partially responsible for that beneficiary’s care by attributing resource use in proportion to the amount of care provided by a given physician.

In Phase II, CMS officials used a single provider attribution method. The agency generally prefers single provider attribution, believing that it encourages physicians to coordinate care. However, CMS has not provided evidence that using a single provider attribution method would lead to increased coordination, and physicians may not accept this method as a credible way to attribute costs. According to Mathematica officials, physicians profiled in Phase I generally preferred the multiple provider attribution method. These physicians’ comments reflected concerns that it was unfair to attribute other providers’ resource use to them. Furthermore, most of the physicians and other stakeholders who provided comments to Mathematica during pretesting in Phase II thought it inappropriate to be held accountable, even partially, for care provided by other physicians. Both specialists and primary care physicians told Mathematica that they did not have control over how another provider treated a beneficiary. Specialists noted that they treated beneficiaries for certain conditions and would not have knowledge of or be responsible for care unrelated to those conditions. Similarly, Mathematica reported that primary care physicians felt they had little control over the care provided by the specialists to whom they referred beneficiaries.

Despite physicians’ concerns about being held responsible for care they did not directly provide, they do have indirect control over some costs incurred by other providers, such as referrals to specialists and decisions about hospitalizations. Given that there is no definitive way to determine which costs a physician was indirectly or directly responsible for, a multiple provider attribution method may be the more reasonable way to attribute costs. For example, the multiple provider method CMS tested in Phase I held physicians accountable for a proportion of the total care provided to a beneficiary. Under this method, a physician who billed for 70 percent of a beneficiary’s total E&M services was assigned 70 percent of the total Medicare resources used by that beneficiary—including office visits, hospitalizations, skilled nursing facility stays, and diagnostic tests and procedures.

A multiple provider attribution approach also increases the number of physicians potentially eligible to receive feedback reports. Because multiple provider attribution holds more than one physician accountable for a beneficiary’s care, more physicians will have patients attributed to them, thus increasing the number of physicians eligible for feedback reports. CMS officials recognize that using a single attribution method will not allow all physicians to be eligible to receive a report, and noted that it is likely that some Phase III reports will use a multiple provider attribution method to assign resource use to physicians.

CMS set a threshold for the minimum amount of care that a physician or physician group needed to provide in order to be assigned responsibility for all or part of that beneficiary’s care. For example, in Phase II, individual physicians needed to bill for at least 20 percent of a beneficiary’s total E&M costs, and physician groups needed to bill for at least 30 percent of the total E&M costs in order to be assigned responsibility for that beneficiary’s care. The minimum threshold was intended to reduce the likelihood that physicians and groups would be assigned responsibility for beneficiaries for whom they provided only minimal care. CMS is considering setting a lower threshold in Phase III to increase the number of physicians eligible to receive reports.

Determining minimum case size. CMS faces a challenging trade-off in determining the minimum number of Medicare beneficiaries or episodes of care a physician must have to produce reliable information without excluding a large number of physicians—those without enough beneficiaries or episodes—from receiving a report. A higher minimum increases the reliability of the information, but decreases the number of physicians eligible to receive a report. In contrast, decreasing the minimum case size increases the number of physicians receiving reports but reduces reliability.

In Phase I, CMS conducted a statistical reliability test to determine the minimum number of episodes a physician needed to be eligible for a feedback report. Reliability indicates how confidently one can classify a physician’s performance relative to that of his or her peers. Estimates for this test range from zero to one, with an estimate above 0.8 generally considered a strong indicator of reliability. CMS used an estimate of 0.5— which is considered a moderate level of reliability—to help ensure that enough physicians would be eligible for Phase I reports. However, few physicians met the minimum case size requirements for certain episodes, such as acute myocardial infarctions, even when using this moderate level of reliability. CMS did not conduct a reliability test to determine the minimum number of beneficiaries a physician must treat for per capita cost measurement.

In Phase II, CMS provided feedback reports to physicians with at least 30 Medicare beneficiaries attributed to them. CMS did not conduct reliability tests for this estimate, stating that a minimum case size of 30 is generally accepted in the research community. However, as some stakeholders have noted, the appropriate minimum case size may vary by condition, suggesting that CMS should instead use a measure of reliability or precision to establish the appropriate case size. For example, Phase II reports contained resource use information for five high-cost, high-volume chronic conditions, and it is likely that different minimum case sizes were needed to generate reliable information for different conditions, such as diabetes and coronary artery disease. CMS officials noted that minimum case size is a major factor in excluding physicians from receiving feedback reports. CMS officials have considered reducing the minimum case size from 30 to 20 beneficiaries for Phase III reports. Officials analyzed the potential effect of this change on individual physicians’ per capita resource use rankings, and found that nearly all physicians were ranked in the same quartile when the case size was lowered from 30 to 20. According to CMS officials, this change would increase the number of physicians eligible to receive Phase III reports by about 10 percent.

Selecting peer groups for comparisons. CMS faces trade-offs in balancing stakeholders’ preferences that feedback reports compare physicians only to those most like themselves—that is, peer groups representing narrow subspecialties or limited geographic areas—with the need to establish a minimum peer group size that is large enough to make statistically significant comparisons.

Individual-level feedback reports distributed in Phases I and II contained two peer group comparisons: (1) physicians in the same specialty in the same metropolitan area and (2) physicians in the same specialty across all 12 metropolitan areas, which was meant to serve as a proxy for a nationwide comparison. Some stakeholders have encouraged CMS to compare physicians within a limited geographic area. However, if a large number of physicians in a limited geographic area were practicing inefficiently, a nationwide sample might be needed to identify the inefficiencies.

In addition, some medical specialty societies and other stakeholders urged CMS to compare physicians only within narrow subspecialties. For example, the American Urological Association noted that surgeons with active surgery practices are substantially different from those who engage primarily in medical management of urological conditions, and comparisons that do not differentiate between these distinct types of physicians are not meaningful to physicians and do not promote learning and improvement.

However, if CMS were to identify and compare physicians in smaller subspecialties, it would face the challenge of ensuring that the peer group size was large enough to make meaningful comparisons across physicians. In Phase I, CMS did not impose a minimum peer group size, but in Phase II it imposed a minimum peer group size of 30 physicians. However, because not all individual physicians had peer groups consisting of 30 physicians practicing in the same geographic area and in the same specialty, some physicians received a report that did not contain information on all performance measures. CMS officials said they may use a minimum peer group size of 15 for Phase III feedback reports.

CMS’s Plans for Improvement May Not Fully Address Challenges in Distributing Reports to Physicians

The majority of sampled physicians were not eligible to receive a Phase II report after CMS’s methodological decisions were applied. CMS officials plan to revise their methodology to increase eligibility for Phase III reports, but significantly increasing the number of physicians who are eligible will be challenging. Further, CMS faced multiple challenges with the electronic distribution of feedback reports to eligible physicians, and as a result, few physicians accessed their reports. CMS officials plan to use a new distribution method for Phase III reports.

Few Sampled Physicians Were Eligible to Receive a Feedback Report; Significantly Increasing Eligibility Will Continue to Be Challenging

Over 80 percent of CMS’s initial sample of 9,189 physicians were ineligible to receive a Phase II feedback report after CMS’s methodological decisions, such as minimum case size requirements, were applied. To identify physicians for the Phase II reports, CMS began with a sample of 9,189 individual physicians affiliated with 36 physician groups. To be eligible for a Phase II report, individual physicians needed to meet CMS’s criteria by having the following:  At least 30 Medicare beneficiaries attributed to them to meet the minimum case size requirement for per capita resource use measures. Of the 9,189 physicians in the original sample, 2,205 (24 percent) had at least 30 beneficiaries attributed to them.  At least 11 Medicare beneficiaries attributed to them who were eligible for 1 or more of the 12 GEM quality measures. Of the 9,189 physicians in the original sample, 2,661 physicians (29 percent) had at least 11 beneficiaries attributed to them who were eligible for at least 1 of the 12 GEM quality measures.  A sufficient number of attributed beneficiaries for both the per capita resource use and GEM quality measures. Of the 9,189 physicians in the original sample, 1,733 physicians (19 percent) had a sufficient number of beneficiaries attributed to them for the per capita resource use and GEM quality measures.  At least 30 individual physicians in the same medical specialty and geographic area for a peer group. Of the remaining 1,733 individual physicians, 1,645 physicians had a peer group of at least 30 individual physicians.

Figure 1 shows the number of physicians excluded by each criterion.

CMS’s methodological criteria also excluded many specialists from receiving feedback reports. Over 90 percent of Phase II reports were created for generalists, such as internal medicine or family practice physicians. The single provider attribution method used by CMS—which assigned a beneficiary to the single physician who billed for the greatest number of E&M services for the beneficiary—limited the number of specialists eligible for a report, since specialists often provide fewer but more expensive E&M services to beneficiaries than generalists. Physicians also needed to have at least one GEM quality measure to receive a Phase II report, but the GEM measures were only applicable to a limited number of specialists, such as cardiologists and nephrologists.

In addition, many of the 1,641 physicians eligible to receive a Phase II feedback report did not meet the methodological criteria needed to receive information on all performance measures, such as resource use for the five chronic condition subgroups or the 12 GEM quality measures. For example, only 5 percent of the 1,641 physicians eligible for Phase II reports were eligible to receive resource use information for their beneficiaries with chronic obstructive pulmonary disease, and none were eligible to receive this information for their beneficiaries with prostate cancer. Similarly, none of the 1,641 physicians eligible for Phase II reports were eligible to receive information for 3 of the 12 GEM quality measures. By contrast, the majority of the 36 physician groups profiled received information on all performance measures (see table 1).

As we stated earlier in this report, CMS is considering a number of methodological changes in Phase III, such as using a multiple provider attribution rule and lowering the minimum case size and peer group requirements. While such changes could lead to a modest increase in physician eligibility for Phase III reports, significantly increasing eligibility—particularly for individual physicians with small case sizes—will continue to be challenging.

Multiple Challenges with Distribution Resulted in Few Physicians Accessing Their Electronic Feedback Reports

CMS faced multiple challenges distributing Phase II feedback reports, and as a result of these challenges, few physicians accessed their reports. In November 2010, CMS mailed letters to 36 physician groups and 1,641 individual physicians affiliated with those groups to notify them that electronic feedback reports were available for their review. However, as of March 2011—approximately 4 months later—less than 60 percent of physician groups and less than 10 percent of individual physicians had accessed their reports electronically.

Major challenges with Phase II distribution were CMS’s difficulty obtaining physicians’ contact information, methods of electronic distribution that were burdensome for physicians, and lack of a strong incentive for physicians to review the reports.

Contact information. The lack of a comprehensive database with accurate names and addresses for physicians and physician groups made it difficult for CMS to notify physicians and physician groups about the availability of their feedback reports. Although reports in Phase II were produced in electronic form, CMS mailed hard copy notification letters to tell individual physicians and physician groups that an electronic feedback report was available and to provide instructions for accessing it. Because available databases contained incomplete or conflicting contact information, CMS had to use multiple sources, including Internet searches, to compile names and addresses—a process that took approximately 5 months.

Despite CMS’s efforts to obtain accurate contact information, some individual physicians and physician groups did not receive a notification letter and therefore did not know that a feedback report was available to them. In follow-up phone calls, CMS found that 27 of the 32 physician groups reached reported that they had not seen the notification letter and could not verify whether it had been received. Many of these physician groups reported that the notification letter was not addressed to the most appropriate person within the group practice, such as the director of quality assurance. CMS also called a sample of 10 individual physicians to ask whether they had received the notification letter. Of these physicians, 1 was retired, 1 reported not receiving the letter, and the remaining 8 had no memory of receiving the letter. In addition, nearly 10 percent of the notification letters mailed to individual physicians were marked undeliverable and returned to CMS.

Distribution method. CMS’s electronic distribution method for Phase II reports was burdensome for some profiled physicians and physician groups. CMS transitioned from hard copy distribution of feedback reports in Phase I to electronic distribution in Phase II based on physicians’ complaints that the reports distributed in Phase I were too long and cumbersome to manage in hard copy. According to CMS, electronic distribution was meant to help physicians navigate the reports. CMS used two methods to electronically distribute feedback reports in Phase II—one for individual physicians and one for physician groups.

Individual physicians were instructed in the notification letter to contact their Medicare Administrative Contractor (MAC) to request a copy of their feedback report. In a report to CMS, Mathematica reported that finding contact information for the correct MAC may not have been a straightforward process for physicians. For example, the notification letter directed physicians to a directory with toll-free phone numbers listed by state for all MAC contact centers, requiring physicians to choose from several possible numbers. Mathematica also reported that MAC customer service representatives were not always aware of the feedback reports or the process for distributing them to physicians. According to CMS’s estimate, the majority of individual physicians did not contact their MACs to request their reports. In February 2011, CMS mailed hard copies of the 1,596 feedback reports that had not yet been electronically accessed by individual physicians.

In theory, the electronic distribution method for physician groups should have been more straightforward since groups were instructed to download their feedback reports from the Individuals Authorized Access to CMS Computer Services (IACS) system, which is the same system used to distribute PQRS reports. However, 8 of the 32 physician groups CMS reached in its follow-up calls reported difficulty downloading their reports from the IACS system. For example, some groups did not know that they needed to register for an IACS account—a process that takes approximately 10 business days to complete—while others reported not being able to download their feedback reports even after logging onto the IACS system. CMS subsequently e-mailed feedback reports directly to those physician groups that had trouble downloading their reports through IACS.

CMS officials recognized the limitations with the distribution method for Phase II reports, and they plan to use a new distribution method for Phase III reports. CMS currently plans to distribute reports to 20,000 individual physicians in one four-state region—Nebraska, Missouri, Iowa, and Kansas. According to CMS officials, the MAC serving this region has e-mail addresses for most physicians in the area. CMS plans to e-mail Phase III reports directly to physicians in this region, thereby avoiding the need to mail hard copy notification letters. In addition to distributing reports to individual physicians in the four-state region, CMS also plans to distribute Phase III reports to 35 physician groups that have participated in the PQRS group practice reporting option. CMS intends to e-mail feedback reports to these 35 physician groups.

Incentive to access reports. Physicians did not have a strong incentive to access their Phase II feedback reports. The notification letter sent by CMS said that these reports were “for informational purposes only” and that they would not affect physicians’ participation in the Medicare program or their Medicare payments. In pretesting for Phase II, many physicians noted that they would be unlikely to review a feedback report closely unless they had an incentive to do so. CMS officials said that they did not want to emphasize that the types of cost and quality measures contained in the feedback reports could affect physicians’ payments in the future because they did not want the reports to sound threatening. Several physician groups suggested that CMS send feedback reports for those physicians affiliated with a group practice to the group’s administrator, noting that individual physicians generally contact their administrators for guidance on such reports. In a report to CMS, Mathematica also noted that medical directors or others with quality oversight responsibilities in larger group practices would be more receptive to feedback reports than individual physicians. They added that these individuals are more familiar with the data used to create feedback reports, and have more experience analyzing quality and cost information for practice improvement.

CMS made follow-up calls to representatives of 15 of the 36 profiled physician groups to obtain their input on the feedback reports, but it conducted minimal follow-up with individual physicians. At the time CMS attempted to follow up with individual physicians, only 4 had contacted their MACs to request a feedback report. Three of these physicians were unwilling to participate in a follow-up call about the report, and 1 physician was unable to download the feedback report that had been sent via e-mail. Similarly, CMS called a sample of 10 physicians who had not requested their feedback reports to ask why they had not done so, and 8 of these physicians expressed no interest in their reports.

Conclusions

In light of concerns about the long-term fiscal challenges facing the Medicare program, the Physician Feedback Program is an important effort that could encourage more efficient medical practice as well as higher-quality care. CMS has worked under challenging timelines to test different approaches to feedback report methodology and distribution. Initial phases of the program indicate that significant changes will need to be made for the program to meet its goal of producing reports with meaningful, actionable, and fair performance measures that apply to the majority of Medicare physicians. CMS will need to do more to solicit input and reactions from physicians and physician groups on the methodology and distribution of reports while the stakes are still relatively low—that is, before CMS begins paying physicians based on their performance on the resource use and quality measures included in the feedback reports beginning January 1, 2015.

In the first two phases, CMS tested different methodological approaches to developing feedback reports; however, the majority of physicians in the most recent phase were ineligible for a feedback report once CMS’s methodological criteria were applied. For example, CMS used a single provider attribution method in the most recent phase, believing that it may improve care coordination—but this method limited physician eligibility, and there is limited evidence to suggest that using this method would increase coordination. And while we also agree with CMS’s decision to include quality measures in feedback reports, some physicians who would have been eligible to receive information on their resource use were disqualified from receiving a Phase II report because they were not eligible for at least 1 of the 12 GEM quality measures. Further, none of the individual physicians who were eligible for a Phase II report had enough beneficiaries attributed to them to receive performance data on all 12 quality measures. CMS did not face such sample size issues in the feedback reports it developed for physician groups.

CMS has not conducted the rigorous statistical analysis it needs to fully understand the impact of its methodological decisions on reliability. For example, CMS used a minimum case size of 30 beneficiaries for Phase II reports, but did not conduct reliability testing to determine this number. The results of such testing can and should influence how CMS ultimately uses the information. Lower levels of reliability may be acceptable if feedback reports remain confidential and are used solely for educational purposes. However, since CMS ultimately intends to pay physicians based on their performance as measured in the feedback reports, it must be reasonably confident that these measures reflect real differences in medical practice. It will also be difficult for CMS to obtain physician and stakeholder buy-in if it does not clearly demonstrate that its performance measures are reliable and robust.

Furthermore, CMS faces challenges distributing feedback reports to physicians and physician groups that are eligible to receive them. CMS transitioned to electronic distribution based on physicians’ complaints that hard copy reports were too long and cumbersome, yet few physicians accessed their Phase II reports electronically. Moreover, CMS conducted limited follow-up with profiled physicians to obtain their input on the feedback reports. As a result, the agency missed an important opportunity to increase physician engagement in the program and to ensure that their concerns are addressed while the program is still in its infancy.

Recommendations for Executive Action

In order to develop feedback reports that are more reliable, credible, accessible, and applicable to a greater number of Medicare physicians, we recommend that the Administrator of CMS take the following four actions:  Use methodological approaches that increase the number of physicians eligible to receive a report, such as  multiple provider attribution methods, which could also enhance credibility of the reports with physicians, and  distributing feedback reports that include only resource use information, if quality information is unavailable.  Conduct statistical analyses of the impact of key methodological decisions on reliability.

Identify factors that may have prevented physicians from accessing their reports and, as applicable, develop strategies to improve the process for distributing reports and facilitating physicians’ access to them.  Obtain input from a sample of physicians who received feedback reports on the usefulness and credibility of the performance measures contained in the reports and consider using this information to revise future reports.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from CMS, which are reprinted in appendix I. CMS concurred with our recommendations and identified actions agency officials are taking to implement them. These actions include refining the attribution methodology to increase the number of physicians receiving feedback reports in Phase III, analyzing the number of cases required to reliably measure quality and make credible comparisons, developing new strategies for distributing feedback reports, and obtaining input from individual physicians and physician groups about the information contained in the feedback reports. If these actions are implemented in accordance with our recommendations, CMS will be better positioned to meet its goals and objectives for the Physician Feedback Program. CMS also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the Administrator of CMS and relevant congressional committees. The report also will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made major contributions to this report are listed in appendix II.

Appendix I: Comments from the Department of Health and Human Services

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Phyllis Thorburn, Assistant Director; William A. Crafton; Cathleen Hamann; Julian Klazkin; Amanda Pusey; Jessica C. Smith; and Rachael Wojnowicz made key contributions to this report.