Medicare: Per Capita Method Can Be Used to Profile Physicians and Provide Feedback on Resource Use
Published: 2009-09-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Medicare Improvements for Patients and Providers Act of 2008 directed the Secretary of Health and Human Services to develop a program to give physicians confidential feedback on the Medicare resources used to provide care to Medicare beneficiaries. GAO was asked to evaluate the per capita methodology for profiling physicians--a method which measures a patient's resource use over a fixed period of time and attributes that resource use to physicians--in order to assist the Centers for Medicare & Medicaid Services (CMS) with the development of a physician feedback approach. In response, this report examines (1) the extent to which physicians in selected specialties show stable practice patterns and how beneficiary utilization of services varies by physician resource use level; (2) factors to consider in developing feedback reports on physicians' performance, including per capita resource use; and (3) the extent to which feedback reports may influence physician behavior. GAO focused on four medical specialties and four metropolitan areas chosen for their geographic diversity and range in average Medicare spending per beneficiary. To identify considerations for developing a physician feedback system, GAO reviewed the literature and interviewed officials from health plans and specialty societies. Further, GAO drew upon literature and interviews to develop an illustration of how per capita measures could be included in a physician feedback report.
section_title: What GAO Found
paragraphs: Using 2005 and 2006 Medicare claims data and a per capita methodology, GAO found that specialist physicians showed considerable stability in resource use despite high patient turnover. This stability suggests that per capita resource use is a reasonable approach for profiling specialist physicians because it reflects distinct patterns of a physician's resource use, not the particular population of beneficiaries seen by a physician in a given year. GAO also found that our per capita method can differentiate specialists' patterns of resource use with respect to different types of services, such as institutional services, which were a major factor in beneficiaries' resource use. In particular, patients of high resource use physicians used more institutional services than patients of low resource use physicians. GAO identified four key considerations in developing feedback reports on physician performance. To illustrate how per capita measures could be included in a physician feedback report, we developed a mock report containing three types of per capita measures. Although the literature suggested that feedback alone has no more than a moderate influence on physicians' behavior, the potential influence of feedback from CMS on Medicare costs may be greater, in part because of the relatively large share of physicians' practice revenues that Medicare typically represents. CMS reviewed a draft of this report and broadly agreed with our findings.

section_title: Letter
paragraphs: In recent years, evidence has mounted that the Medicare program is  unsustainable in its present form. Because of rising health care costs and  the aging of baby boomers into eligibility for Medicare, future program  spending is projected to consume an increasing share of the government’s  resources. In their 2009 annual report, the Medicare Trustees projected  that Medicare expenditures, which reached $468 billion in 2008, will  increase in future years at a faster pace than the overall economy, rising  from 3.2 percent of gross domestic product in 2008 to 11.4 percent by 2083.
Physicians play a central role in the generation of health care  expenditures, through both the services they provide and the services they  order, including hospital admissions, diagnostic tests, and referrals to  other physicians. The evidence suggests that some of the spending for  services provided and ordered by physicians may not be warranted. For  example, the wide variation in Medicare spending for physician services— unrelated to beneficiary health status or outcomes—indicates that health  needs alone do not determine spending.
Consistent with physicians’ central role in providing and ordering services  and their influence on the amount of spending for patient services,  physician groups, insurers, and Medicare officials have turned to profiling  as a possible tool to help identify and contain overuse of services and the  resulting high expenditures. In profiling, the resource use of a physician’s  patients is compared to a benchmark. In our previous report on profiling,  Medicare: Focus on Physician Practice Patterns Can Lead to Greater  Program Efficiency, we profiled generalist physicians and found that in  each of the 12 metropolitan areas we studied there were physicians who,  relative to their peers in the same area, treated a disproportionate share of  overly expensive patients. In that report we used a profiling methodology  known as per capita, which measures per patient resource use for a  defined population over a fixed period of time and attributes that resource  use to physicians. We recommended that the Administrator of the Centers  for Medicare & Medicaid Services (CMS) develop a profiling system to  identify individual physicians with inefficient practice patterns and  provide incentives for physicians to improve the efficiency of care they  provide. In our subsequent testimony on physician feedback to the  Subcommittee on Health of the House Ways and Means Committee we  stated that providing feedback to physicians on their practice patterns  could be a promising step toward encouraging efficiency in Medicare. The  Medicare Payment Advisory Commission (MedPAC) has also  recommended providing feedback to physicians on their resource use. In  its reports, MedPAC has explored an episode-based profiling methodology,  which measures resource use for treating a particular episode of illness— for example, a stroke or heart attack—and attributes that resource use to  physicians.
Following the issuance of our report and subsequent testimony, Congress  passed the Medicare Improvements for Patients and Providers Act of 2008  (MIPPA), which directed the Secretary of Health and Human Services  (HHS) to develop a program to provide physicians confidential feedback  on the Medicare resources used to provide care to Medicare beneficiaries.  MIPPA gave HHS the flexibility to measure resource use on a per capita  basis, an episode basis, or both. In response to this mandate, CMS is  currently testing both profiling methodologies in its Physician Resource  Use Measurement and Reporting Program. MIPPA also directed us to  submit a report to Congress on CMS’s physician feedback program by  2011.
In your letter of August 22, 2007, you pointed out that both the per capita  and episode-based methods could be used to identify inefficient  physicians, but noted that less is known about the per capita method. At  that time, you asked us to evaluate the per capita method for profiling  physicians in order to assist CMS with the development of a physician  feedback approach for Medicare. This report explores the use of a per  capita method to profile physicians based on their patients’ level of  resource use, and discusses the development and influence of feedback  reports. Specifically, this report examines (1) the extent to which  physicians in selected specialties show stable practice patterns and how  beneficiary utilization of services varies by physician resource use level;  (2) factors to consider in developing feedback reports on physicians’  performance, including per capita resource use; and (3) the extent to  which feedback reports may influence physician behavior.
We focused our analysis on four diverse specialties—a medical specialty  (cardiology), a diagnostic specialty (diagnostic radiology), a primary care  specialty (internal medicine), and a surgical specialty (orthopedic  surgery); and four metropolitan areas—Miami, Fla.; Phoenix, Ariz.;  Pittsburgh, Pa.; and Sacramento, Calif. We chose these areas for their  geographic diversity, range in average Medicare spending per beneficiary,  and number of physicians in each of the four specialties. We limited our  study to physicians who participate in Medicare fee-for-service (FFS). Our  results are not generalizable to other geographic areas and specialties.
To measure beneficiaries’ resource use, we first adjusted for beneficiaries’  health conditions, because sick beneficiaries are expected to use more  resources than healthy beneficiaries. Using Hierarchical Condition  Category (HCC) and expenditure data obtained from CMS, we estimated a  risk adjustment model that uses the same 70 HCCs as the model CMS uses  to set managed care capitation rates. HCCs are a way of summarizing an  individual’s diagnoses into major medical conditions, such as vascular  disease or severe head injury. Given beneficiaries’ HCCs during the year,  we used our model to estimate Medicare’s expected annual expenditures  for services provided to the beneficiaries in our study. Based on these  expected expenditures, we placed beneficiaries into 25 discrete risk  categories. Within each risk category and metropolitan area, we ranked  beneficiaries from 1 to 100 by their total annual Medicare FFS  expenditures such that the average beneficiary in a given risk category and  metropolitan area had a rank of 50. We used this rank as our risk-adjusted  measure of beneficiaries’ resource use.
Our measure of physicians’ resource use is derived from the resource use  of their patients. For all physicians in our study, we calculated the  average rank of their patients. We then used this average to rank  physicians within the same metropolitan area and specialty on a scale of   1 to 100. This measure reflects how expensive a physician’s patients are  compared to the patients of other physicians in the same specialty and  area after adjusting for differences in patient health status.
To examine the stability of physicians’ resource use from a year-to-year  perspective, we analyzed data for 2005 and 2006. We divided physicians’  and beneficiaries’ resource use into quintiles and examined which  physicians and beneficiaries stayed in the same resource use quintile from  2005 to 2006 and which ones did not. We also examined the degree of  turnover in the patients seen by physicians between 2005 and 2006. In  addition, we used the physician quintiles to examine how beneficiary  utilization of selected services in 2006 varied by physician resource use  quintile.
We concluded that the information on Medicare claims that we used in  this report was sufficiently reliable for the purpose of our analysis,  because it is a record of Medicare’s payments to health care providers. We  obtained beneficiaries’ FFS expenditures from claims information, and we  used data from CMS files containing enrollment and institutional status in  order to determine whether beneficiaries were eligible for our study. CMS  provided us with a file containing beneficiaries’ HCCs, which we used to  estimate their expected expenditures. We obtained physicians’ specialties  from Medicare physician files that CMS uses to administer the program  and set payment rates. CMS and its contractors closely monitor these files,  so they are generally considered reliable. In addition, we interviewed  relevant CMS officials concerning the data and consulted data  documentation maintained by CMS. We consider the data sufficiently  reliable for our purposes.
To determine factors to consider in developing reports to provide  feedback to physicians on their performance, including their per capita  resource use, and the extent to which feedback reports may influence  physician behavior, we reviewed selected literature and interviewed  experts. To identify relevant literature, we searched 31 databases,  including MEDLINE and Science Citation Index, using terms such as  “physician performance feedback,” for journal articles and other  documents published between January 1, 2000, and February 13, 2009.  From reference lists in documents identified during that search, we  identified additional documents that met our criteria. We selected for  review three types of documents: (1) meta-analyses, reviews, or scans of  the literature on the effectiveness of providing performance feedback to  physicians; (2) evaluations of various efforts to provide performance  feedback to physicians; and (3) documents that provided guidance from  experts on methods for providing performance feedback to physicians. In  addition to reviewing selected literature, we conducted interviews with  officials of four specialty societies to identify specialty-specific  perspectives and concerns, and to solicit officials’ comments on a mock  feedback report we designed. We also conducted interviews with officials  of the five health insurers with the highest revenues in 2007 about their  experiences with feedback reports.
There are several limitations to our findings. Our findings cannot be  generalized to other areas or specialties. We also restricted our scope to  individual physicians and did not analyze group practices. Most  importantly, we did not pilot our mock report, which illustrates how per  capita measures could be included in a physician feedback report, or test  it by giving physicians feedback based on actual resource use.  Consequently, we are unable to evaluate how helpful it would be to  physicians and, particularly, whether it has potential for increasing  physicians’ efficiency.
We conducted our work from February 2008 to September 2009 in  accordance with all sections of GAO’s Quality Assurance Framework that  are relevant to our objectives. The framework requires that we plan and  perform the engagement to obtain sufficient and appropriate evidence to  meet our stated objectives and to discuss any limitations in our work. We  believe that the information and data obtained, and the analysis  conducted, provide a reasonable basis for any findings and conclusions.
subsections: section_title: Background
paragraphs: We, MedPAC, and the Congressional Budget Office (CBO) have all  suggested that CMS profile physician resource use and provide feedback  to physicians as a step toward improving the efficiency of care financed by  Medicare. In July 2008, Congress passed MIPPA, which directed the  Secretary of HHS to establish a program by January 1, 2009, to provide  physicians confidential feedback on the Medicare resources used to  provide care to beneficiaries. MIPPA gave HHS the flexibility to measure  resource use on a per capita basis, an episode basis, or both. In response  to the MIPPA mandate, CMS is pursuing its Physician Resource Use  Measurement and Reporting Program. (See table 1.)
subsections: section_title: Key Decisions in Physician Profiling
paragraphs: When profiling physicians on their resource use, five key decisions must  be made:    Which resource use measurement methodology to use. There are two  main profiling methodologies: per capita and episode-based. Using both  types of measures of resource use may provide more meaningful results by  more fully capturing the relevant characteristics of a physician’s practice  patterns.
How to account for differences in patient health status. Accounting for  differences in patient health status, a process sometimes referred to as  risk-adjustment, is an important and challenging aspect of physician  profiling. Because sicker patients are expected to use more health care  resources than healthier patients, we believe the health status of patients  must be taken into account to make meaningful comparisons among  physicians. There are various risk-adjustment methods and the suitability  of a given method will depend on characteristics of the physicians to be  profiled and their patients.
How to attribute resource use to physicians. Important attribution  decisions include whether to assign a patient’s resource use to the single  physician who bears the greatest responsibility for the resource use, to all  physicians who bore any responsibility, or to all physicians who met a  given threshold of responsibility, such as providing a certain percentage of  the expenditures or volume of services. A single attribution approach may  not be applicable for all types of measures or for all types of physician  specialties.
What benchmark(s) to use. Physician profiling involves comparing  physicians’ resource use to a benchmark. There are differing opinions on  what are the most appropriate and meaningful comparative benchmarks.
How to determine what is a sufficient sample size to ensure meaningful  comparisons. The feasibility of using resource use measures to compare  physicians’ performance depends, in part, on two factors: the availability  of enough data on each physician to compute a resource use measure and  a sufficient number of physicians to provide meaningful comparisons. It is  important to calculate resource use measures only for physicians with  sufficient sample sizes in order to address concerns that a physician’s  profile may be distorted by a few aberrant cases. There is no consensus on  what sample size is adequate to ensure meaningful measures.
section_title: CMS’s Resource Use Measurement and Reporting Program
paragraphs: Responding to the MIPPA mandate to establish a physician feedback  program by January 1, 2009, CMS began in April 2008 to develop its  program for reporting to physicians on their resource use. In the first  phase of the program, CMS identified eight priority conditions and  disseminated approximately 310 Resource Use Reports to physicians in  selected specialties who practiced in one of 13 geographic areas. The  reports generally included both per capita and episode-based resource use  measures that were calculated according to five different attribution rules.  The reports also contained multiple cost benchmarks relative to  physicians in the same specialty and geographic area. In Phase II, CMS is  proposing to expand the program by adding quality measures and  reporting on groups of physicians as a mechanism for addressing small  sample size issues.
section_title: Per Capita Profiling Method Shows Specialist Physicians’ Practice Patterns Relatively Stable Over 2 Years; Patients of High Resource Use Physicians Used More Institutional Services Than Other Patients
paragraphs: Using a per capita profiling method, we found that from 2005 to 2006,  specialist physicians showed considerable stability in their practice  patterns, as measured by resource use—greater stability than their  patients, despite high patient turnover. We also found that our per capita  method can differentiate specialists’ patterns of resource use with respect  to different types of services, such as institutional services, which were a  major factor in beneficiaries’ resource use. In particular, patients of high  resource use physicians used more institutional services than patients of  low resource use physicians.
subsections: section_title: Specialist Physicians’ Resource Use More Stable Than Beneficiaries’ Resource Use
paragraphs: Using a per capita method to profile specialist physicians, we found that  their practice patterns, as measured by the level of their resource use, was  relatively stable over 2005 and 2006 by comparison with individual  beneficiaries’ resource use (see figure 1). This is true despite the fact that  our measure of physicians’ resource use is derived from their patients’  resource use and that the specific patients whom physicians see are not  always the same from year to year. Among the physicians we studied, less  than one-third of patients seen by study physicians in 2005 were also seen  by the same physician in 2006. This stability suggests that per capita  resource use is a reasonable approach for profiling physicians, because it  reflects distinct patterns of a physician’s resource use, not the particular  population of beneficiaries seen by a physician in a given year.
We divided both physician and beneficiary resource use into five groups of  approximately equal size (quintiles) and found that, on average across the  four metropolitan areas and four specialties, 58 percent of physicians and  30 percent of beneficiaries were in the same quintile of resource use in  2005 and 2006. The pattern was even more pronounced for the top  resource use quintile: 72 percent of physicians and 35 percent of  beneficiaries remained in that quintile. If the level of physicians’ and  beneficiaries’ resource use was purely random, only 20 percent would be  expected to have remained in the same quintile.
We also examined the stability of physicians’ resource use by specialty and  found a similar pattern, although not to the same extent in all specialties.  The average percentage of physicians who were in the same resource use  quintile in 2005 and 2006 ranged from 48 percent for orthopedic surgeons  to 60 percent for internists. Resource use in the top quintile was more  stable and ranged from 69 percent for diagnostic radiologists to 74 percent  for internists. (See table 2.)
In each of the four metropolitan areas, physicians showed greater stability  in their resource use than individual beneficiaries, although the  percentages varied. For example, the percentage of physicians remaining  in the top quintile ranged from 68 percent in Phoenix to 76 percent in  Miami. For beneficiaries, the percentage in the top quintile ranged from   31 percent in Phoenix to 39 percent in Miami. (See table 3.)
The greater stability of physicians’ resource use compared to beneficiaries’  resource use could be due to their individual practice styles, as well as to a  range of other factors, such as participation in formal or informal referral  networks. These networks have a range of providers, including other  physicians, who treat their patients and refer them for treatment, testing,  and admissions to hospitals.
section_title: Beneficiary Use of Institutional Services Varies by Physician Resource Level
paragraphs: Beneficiaries seen by high resource use physicians generally were heavier  users of institutional services than those seen by lower resource use  physicians, and institutional services accounted for more than one-half of  total patient expenditures. This pattern was consistent across three of the  four specialties we studied, with orthopedic surgery being the exception.
Institutional services were the major driver of Medicare expenditures for  beneficiaries in physicians’ practices, accounting on average for 54  percent of expenditures. Services provided by a particular physician in our  study directly to that physician’s patients accounted for only 2 percent of  total expenditures or about $350 for each beneficiary in a physician’s  practice. All other services—those provided by other physicians, home  health care, hospice care, outpatient services, and durable medical  equipment—accounted for the remaining 44 percent of expenditures. (See  fig. 2.)
Expenditures for institutional services for a physician’s patients grew as  the level of physician resource use increased. Dividing the level of  physician resource use into quintiles, we examined the relationship of  physicians’ resource use and expenditures for services provided to their  patients. Average expenditures for institutional services increased more  steeply by physician resource quintile than expenditures for all other  services.
The four specialties all exhibited this pattern of increasing beneficiary  expenditures for institutional services accompanying increasing physician  resource use, although for orthopedic surgery the increase was small. The  increase in average beneficiary expenditures for all other services that  accompanied increasing physician resource use was similar for three of  the four specialties and was steeper for internal medicine.
We also examined the average number of physicians seen by the Medicare  beneficiaries we studied and found that it was positively associated with  increasing physician resource use. Overall, the number of physicians seen  increased from an average of about 13 physicians per beneficiary in the  lowest quintile of resource use to more than 23 in the highest. The increase  in the number of physicians seen was accompanied by an increase in  average beneficiary expenditures for institutional services that was  steeper than the rise in other services.
section_title: Research Literature, Health Insurers, and Specialists Identified Considerations in Developing Physician Feedback Reports on Resource Use
paragraphs: Through our review of selected literature and interviews with officials of  health insurance companies, specialty societies, and profiling experts, we  identified several key considerations in developing reports to provide  feedback to physicians on their performance, including their per capita  resource use. We also drew on information from these sources to develop  an example of how per capita measures could be presented in a physician  feedback report.
subsections: section_title: Key Considerations in Developing Physician Feedback Reports Include Content, Design, Dissemination Strategy, and Transparency
paragraphs: We identified four key considerations in developing reports to provide  feedback to physicians (see table 4).
Our review of selected literature suggested that a physician feedback  report should contain three basic elements: an explanation of the  information contained in the report (which we will discuss in the context  of transparency), measures describing the performance of the physician or  physicians to whom the report is directed, and comparative benchmarks.
Measures. Both the selected literature we reviewed and the officials we  interviewed supported including measures of quality along with measures  of cost, and ensuring that measures are actionable by providing  information that can help physicians improve their performance. The  officials we interviewed were divided as to whether these measures should  reflect physicians’ performance at the individual level or the group level.
Quality measures. All five of the insurers we contacted were profiling  physicians in terms of quality and cost, and four of the five had adopted a  model code for physician ranking programs that called for rankings to be  based on quality as well as cost. Most of the specialty society officials we  interviewed also called for the inclusion of quality measures in physician  feedback reports, and some cautioned that focusing solely on costs could  create perverse incentives—for example, encouraging physicians to  reduce inappropriately the level of care provided to patients. The lack of  widely accepted, claims-based quality measures for some specialties has  limited the number of specialties some insurers profile. For example, at  the time of our interview, one insurer was profiling physicians in only one  specialty (cardiology) while planning to begin profiling other specialties  within a year.
Actionable measures. According to one research report we reviewed,  little research has been done to determine how the reporting of global  scores—such as an overall per capita cost rank—influences physician  behavior, but experts on physician profiling and a broad array of  stakeholders, including physicians and insurance company officials,  agreed that performance data should be disaggregated into enough  categories to enable physicians to identify practice patterns to change.  According to some profiling experts, resource use reports must pinpoint  physicians’ overuse and misuse of resources, and identify practices that  add costs but do not improve desired outcomes. Similarly, specialty  society officials we interviewed emphasized the importance of including  measures that focus on areas in which the physician has control.
Individual versus group measures. Another measurement  consideration is whether physicians in group practices should be profiled  as individuals or as a group. The insurers we contacted took varying  approaches. In some cases, the approach was driven by contracting  arrangements, with insurers constructing group profiles for physicians  with whom they had group contracts. One insurance company official  pointed out that profiling at the group level allows more physicians to be  profiled, as it increases the data available to construct a profile. Another  official advocated profiling at the individual level because he believes  physicians are more interested in assessments of individual performance.  Officials of the four specialty societies generally saw some merit to both  approaches, but some underscored the difficulty of identifying group  affiliations or noted that groups are not necessarily homogeneous enough  for a group assessment to be appropriate.
Comparative benchmarks. One consideration addressed by multiple  publications we reviewed was the kind of benchmark to which physicians’  performance should be compared. For example, a physician’s  performance may be compared to (1) an evidence-based standard, (2) a  standard based on professional judgment, such as the consensus standards  endorsed by the National Quality Forum, or (3) to a statistical norm, such  as the average for a physician’s peers locally or nationally.
Although studies we reviewed offered conflicting evidence as to whether  including peer comparisons in physician feedback reports increases their  effectiveness, some profiling experts and specialty society officials believe  comparative information is useful and of interest to physicians. In the  literature we reviewed, for example, one profiling expert suggested that  such comparisons can motivate behavior change by taking advantage of  physicians’ desire to perform at least as well as their peers; another stated  that performance statistics are not meaningful to physicians without peer  comparisons.
A physician’s peer group can be defined in various ways. According to one  study, some organizations that provide performance feedback to  physicians have found comparisons within specialty and locality most  useful to and most frequently requested by physicians. Representatives of  some of these organizations said physicians find local information more  relevant because it reflects the practice patterns of their geographic area.  All five insurers we contacted compare physicians to others in the same  market and specialty; one of the five also compares physicians to peers  nationwide on some measures. In contrast, officials of all four specialty  societies recommended comparisons at the national level, with officials of  one society stating that there is no scientific basis for regional variations in  practice patterns. There was less agreement about whether physicians  should be compared to others in their specialty or to a more narrowly  defined group. Officials of one specialty society advocated comparisons at  the subspecialty level in recognition of the variation in resource use  patterns among subspecialists. Another official pointed out that such  comparison groups could be difficult to define because physicians in some  specialties tend to have multiple subspecialties. Because views differ on  appropriate comparison groups, one hospital-owned healthcare alliance  plans to incorporate in its physician reports a customizable feature that  will allow users to select the peer comparison they wish to see.
Comparisons to physicians’ own past performance (trend data) are  commonly presented in feedback reports, and the majority of physicians  surveyed in one study found these comparisons useful.
subsections: section_title: Report Design
paragraphs: The selected literature we reviewed offered little hard evidence on how  feedback reports should be designed to engage physicians’ interest or to  prove their comprehension of the material. However, researchers and  profiling experts offered some comments and suggestions based either on  their experience with clinical performance measurement or on an analysis  of the literature on consumer behavior and its possible implications for  physician reporting (see table 5).
The amount and combination of material that should be included in a  single report is an important consideration. According to one publication  that summarized a review of multiple feedback reports, some  organizations issue separate reports on efficiency/cost and  effectiveness/clinical quality, in part to avoid diluting the impact of either  set of measures. Others believe a single report gives physicians a more  complete picture of their performance.
Officials of the three insurers we contacted that routinely issued feedback  reports to physicians said that their companies produced summary  reports, typically one to two pages in length, containing high-level  information, but also made more detailed information, such as patient- level data, available to physicians. One insurer’s summary report consisted  of one page of cost efficiency measures and one page of effectiveness  measures. The cost efficiency page presented average cost per episode of  care by service category for the physician and the physician’s peer group,  as well as the ratio of the two, in both tabular and graphic form. The  effectiveness page presented process-of-care measures for selected  conditions, including cardiovascular disease and asthma. Company  officials said summary reports were limited to two pages to accommodate  physicians’ attention spans and that the two sets of measures were  presented separately to discourage attempts to link the two. Specialty  society officials agreed reports should be short—most proposed one to  two pages—and strongly recommended that information be presented  graphically to the extent possible. One official, noting that physicians are  very visually oriented, recommended feedback reports consisting mainly  of easily understood graphics.
The selected literature we reviewed, our interviews with specialty society  officials, and existing physician feedback reports suggested reports can be  kept short by segmenting some information into separate documents—for  example, a cover letter that explains the report’s purpose, a description of  the profiling methodology, a set of frequently asked questions, and a list of  definitions.
section_title: Report Dissemination
paragraphs: Some key considerations with respect to report dissemination are which  physicians should receive reports, how frequently to issue reports, and  whether to issue reports in hardcopy or electronically.
Which physicians should receive feedback reports. One major  decision is whether to issue reports to all physicians for whom  performance measures can be calculated or only to a subset who fail to  meet certain performance standards—a decision that may involve  weighing reporting costs against potential impacts. None of the studies we  reviewed directly addressed this issue, but all of the specialty society  officials we interviewed advised sending reports to all or nearly all  physicians, rather than just to poor performers. They gave several reasons:  to provide positive recognition to physicians who are performing well; to  avoid singling out certain physicians as poor performers, especially on the  basis of excess costs over which they have little control; and to create  opportunities for voluntary peer-to-peer learning among physicians who  are at different points along the performance spectrum. Similarly, all three  of the insurers that routinely issued feedback reports sent them to all  physicians for whom they had performance measures.
Frequency of reporting. According to one book we reviewed,  organizations that provide feedback to physicians should do so more than  once a year to give physicians an opportunity to improve their  performance in a timely manner. However, because of the time needed to  gather sufficient data to identify trends and patterns of performance, many  organizations provide feedback no more than twice a year. Of the two  insurers that told us how frequently they issued feedback reports, one did  so annually and the other at least every 6 months. Officials of the latter  company said the frequency of their reporting was limited by the number  of claims in their dataset and suggested that CMS would not face the same  limitations.
Hardcopy versus electronic dissemination. Reports can be  disseminated in hardcopy through various channels, such as the mail, or  electronically, through e-mail or a Web site. One literature scan we  reviewed cited certain advantages of electronic formats such as Web- based applications. Specifically, they allow users to organize information  as they choose and are well suited to presenting data from the general to  the specific, which facilitates information processing. Although this report  noted some concerns about physicians’ access to the Internet, according  to a report based on a national survey of physicians in December 2002 and  January 2003, almost all respondents said they had Internet access, and  most said they considered it important for patient care.
Of the three insurers that routinely issued feedback reports, two issued  them electronically and one issued them in hardcopy. Officials of the latter  company said that staff typically hand-delivered the reports to physicians  during on-site visits in order to discuss the results. Officials of most of  the specialty societies we contacted did not advocate one dissemination  mode over the other, but some noted that organizations that issue reports  electronically must confront certain challenges, such as ensuring that  security features do not make access difficult, addressing the lack of high- speed Internet service in some areas, and determining whether to send  reports by e-mail or to instruct physicians to access them on the Internet.  One specialty society official recommended using both modes of  dissemination to accommodate different preferences.
section_title: Transparency
paragraphs: Both the selected literature we reviewed and our interviews with officials  from insurance companies and specialty societies underscored the  importance of ensuring transparency regarding the purpose of the report  and the methodology and data used to construct performance measures.
Purpose. According to one literature scan, feedback reports should  explicitly state their purpose—for example, to reduce costs, improve  quality, or simply to provide information—and should highlight any items  for which the physician will be held accountable.
Methodology. Two important considerations are where to provide  information about methodology—whether in the report itself or through  some other mechanism, such as a Web page—and how much technical  detail to provide. Some of the insurers we contacted provide information  on-line about their profiling methodologies, including details about  measures, attribution of care to physicians, risk adjustment, and statistical  issues. In addition, some of the officials we interviewed said that company  staff will meet with physicians to explain the profiling methodology, if  requested. For example, officials of one company said that it has on staff  four profiling experts, mostly nurses, in addition to about 20 medical  directors who can answer physicians’ questions.
Specialty society officials we interviewed highlighted a potential trade-off  between providing enough information in the report to persuade  physicians of the validity of the measures and keeping the report concise  enough to maintain physicians’ interest. All of the officials we interviewed  agreed that physicians should have access to details about the  methodology; some suggested this information might best be disseminated  through a Web site. Explaining how the data are risk-adjusted to account  for differences in physicians’ patient populations was cited by specialty  society officials as particularly important.
Data. Another consideration is ensuring transparency with regard to the  data used in profiling—making patient-level detail available so physicians  can reconcile performance measures with their own information about  their practices. All five of the health insurers we contacted provided  opportunities for physicians to examine patient-level data and file appeals  before results are made public, although their processes or policies for  doing so varied (see table 6).
Officials of one of the two insurers that made detailed data available on- line said their company previously sent hardcopy reports to physicians,  but learned from medical office managers that they would prefer an on- line format that could be manipulated to facilitate physician comparisons.  Officials of the other insurer said that their company planned to make the  data available in a manipulatable format soon. Most of the specialty  society officials we interviewed agreed that patient-level data should be  made available to physicians, but some predicted that few physicians  would access them. Two interviewees suggested practice size would  probably be a factor; one added that physicians in smaller groups would  likely lack the resources and skills to analyze the data.
section_title: Per Capita Measures Can Be Presented in a Physician Feedback Report
paragraphs: Drawing upon lessons culled from the literature and our interviews, we  developed a mock report that illustrates how per capita measures could be  included in a physician feedback report. Such a report could also include  other measures such as quality measures and episode-based resource use  measures. We included two types of per capita measures—risk-adjusted  cost ranks and risk-adjusted utilization rates—each presented with local  and national comparative benchmarks. To provide further context, we also  included per capita measures showing how the average Medicare costs of  patients the physician treated at least once were distributed among service  categories, and the percentage of those costs that were for services  directly provided by the physician to whom the report is directed. We kept  the mock report under two pages and included minimal text, while  ensuring transparency by indicating the availability of methodology details  and supporting data. To accommodate physicians’ differing dissemination  preferences, we designed the mock report to be available in both  electronic and hardcopy formats. (See fig. 3.)
Specialty society officials who vetted a draft of the mock report made  several recommendations. Some recommendations centered on taking  advantage of electronic capabilities, such as adding hovers to define key  terms (see fig. 4), creating interactive features to let physicians explore  “what if” scenarios, and including links to educational materials and  specialty guidelines. Officials also recommended adding information on  pharmaceutical costs, a category we did not include because not all  beneficiaries are enrolled in a Medicare Part D prescription drug plan.
A patient’s risk adjusted cost rank is calculated  by comparing the patient’s Medicare costs to all  other Cityville patients with similar risk scores  and represents how unexpectedly expensive or  inexpensive the patient’s Medicare-covered care was.  Your rank is the average rank of all patients you  treated at least once. See Glossary for more details.
All providers: All providers: 100% of total 100% of total ($13,422) ($13,422)
You: You: 11% of category 11% of category ($1,449) ($1,449)
More generally, specialty society officials said that they particularly liked  the graphs and charts in our mock report. One official added that our  report was easier to understand than other reports he had seen and that he  thought it would get physicians’ attention. Another official commented  how the presented per capita measures could give physicians insight on  the care their patients are receiving that they were not previously aware  of—a perspective other cost measures could not provide. However,  multiple officials said the measures as presented were too broad to be  actionable and might not seem relevant to physicians, as most physicians  feel responsible only for the costs of services they directly order or  provide, not for the total cost of patients’ care. Two officials suggested  that these per capita measures would have more value in health care  systems that emphasized coordination of care.
section_title: Potential Influence of Feedback Regarding Medicare Costs on Physician Behavior Is Uncertain
paragraphs: Our review of available literature on the effectiveness of physician  feedback suggests that feedback alone generally has no more than a  moderate influence on physician behavior. However, the potential  influence of feedback from CMS regarding Medicare costs is uncertain,  and may be greater than that of feedback from other sources, because  Medicare reimbursement typically represents a larger share of physicians’  practice revenues than that from other insurers.
In general, studies examining the effect of feedback on physicians’  behavior have found it to have a small to moderate effect. Factors that  appear to influence the effectiveness of feedback include its source,  frequency, and intensity. For example, one review of the literature  concluded that physicians were more likely to be influenced by reports  from a source they expected to continue monitoring their performance.  This review also found that repeated feedback over a period of several  years may be more likely to get physicians’ attention. Another review  reported that the intensity of the feedback appeared to influence its  effectiveness. The review cited individual, written feedback containing  information about costs or numbers of tests, but no personal incentives, as  among the least intensive, and therefore likely to be among the least  effective approaches.
Consistent with the literature we reviewed, most of the insurance  company officials we interviewed questioned whether providing  performance feedback to physicians would have a significant impact on  the physicians’ behavior in the absence of other incentives. While all five  insurers profiled physicians, none used the results solely to provide  feedback. Officials of four of the five insurance companies said that to  affect physicians’ behavior, profiling results must be made public, thus  influencing patients’ choice of physicians, or linked to monetary  incentives, as in pay-for-performance arrangements. However, officials of  one company disagreed, stating that feedback alone can affect physicians’  behavior if the reports show how they rank against their peers and make  clear what behavior they need to change to improve their efficiency. These  officials also said that the impact of feedback could depend on the size of  physicians’ practices and whether they have the resources to review the  reports and the management structure to affect changes.
Whether the experiences of private insurers or the lessons from the  literature on the influence of feedback will hold in the case of the  Medicare program is uncertain. A survey conducted in 2004-2005 found  that, for most physicians, Medicare represented more than one-quarter of  practice revenue, and for 17 percent of physicians, the proportion was  more than one-half. Because physicians typically contract with a dozen  or more health insurance plans, few, if any, of these plans are likely t o  represent as large a share of physicians’ practice revenue as Medicare.  Hence, the impact of feedback from CMS might be greater than that from  other sources. In addition, one profiling expert suggested that physicians  might expect feedback from CMS to be only the first step in efforts to  influence physicians’ behavior—to be followed, for example, by public  reporting of profiling results. This perspective comports with  recommendations in our earlier report. providing feedback on a confidential basis would be an appropriate first  step. One said it would allow time to test the profiling methodology and  gauge physicians’ reactions; the other said it would provide an opportunity  for physicians to vet the measures and identify any errors.
See GAO-07-307. suggestions for enhancing its effectiveness. Other suggestions can be  drawn from the literature we reviewed. These suggestions included:    providing advance notice of feedback reports (through presentations,  letters, or other communications) to help ensure that physicians open and  read the reports;   working through credible intermediaries, such as medical societies or  locally prominent physicians, to assure physicians that the feedback  process is reasonable and legitimate;    providing opportunities for physicians to discuss the reports through  videoconferences, teleconferences, or on-line discussion groups; and   offering in-person follow up, possibly drawing on the resources of the  Medicare Quality Improvement Organizations.
Involving physicians in the development of a feedback system may also  enhance its effectiveness. One literature scan concluded that physician  involvement in system design was vital for obtaining physician buy-in.  Information from insurers suggested that, although physicians may not  always be involved in initial development of feedback systems, their  feedback can prompt modifications. Some insurance officials we  interviewed described an iterative process involving ongoing  communication with physicians and continuous modification of reports  and systems. For example, officials of one insurance company said that  the company did not seek initial input from physicians—in the belief that  they would not have been able to provide much input without a complete  understanding of the data and methodology—but took into account  physicians’ responses to earlier, less formal systems. Officials of other  companies described various mechanisms for obtaining physicians’  perspectives, including formal physician advisory councils, regular  meetings with officials of national medical societies, and town hall  meetings with physicians at the local level.
section_title: Concluding Observations
paragraphs: Profiling physicians to improve efficiency is used by some private  insurance companies and, at the direction of Congress, is being adopted by  the Medicare program. We believe that a per capita methodology is a  useful approach to profiling physicians on their practice efficiency and  could be part of a feedback program that could also include quality  measures and episode-based resource use measures.
Our findings are consistent with those of our previous report on physician  profiling in which, through analysis of physician practice patterns, we  determined that CMS could use profiling to improve the efficiency of  Medicare. Despite a more diverse mix of physician specialties in our  present analysis, and with certain exceptions noted in our findings, we  found substantial consistency in certain patterns we observed across  metropolitan areas and specialties. We also found consistency across time  in that physicians who showed high resource use in one year tended to  stay high in the subsequent year.
section_title: Agency and Professional Association Comments and Our Evaluation
paragraphs: We provided a draft of this report to the HHS for comment and received  written comments from CMS, which are reprinted in appendix II. We also  solicited comments on the draft report from representatives of the  American Academy of Orthopaedic Surgeons (AAOS), the American  College of Cardiology (ACC), the American College of Physicians, and the  American College of Radiology. We received oral comments from the first  two.
section_title: CMS Comments
paragraphs: Our draft report did not include any recommendations for CMS to respond  to. CMS broadly agreed with each of our three findings:    CMS agreed that the per capita methodology is a useful approach to  measuring physicians’ resource use and noted that per capita  measurement is one of the cost of care measures included in CMS’s  Physician Resource Use Management and Reporting Program. CMS also  agreed that the consistency of our per capita measure across years is an  important finding and stated that the agency intends to examine measure  consistency in the ongoing administration of its program.
CMS found the attention in our report to considerations for developing a  physician feedback system to be particularly helpful. CMS listed several  examples of how its program already addresses many of these  considerations and is in the process of addressing others. We agree with  CMS that some of the approaches described in our report would require  significant resources and recognize that CMS will need to investigate how  to balance the trade-offs between different approaches in order to best  leverage its resources.
CMS agreed that physician feedback may have a moderate influence on  physician behavior. CMS further stated its commitment to developing  meaningful, actionable, and fair measurement tools for physician resource  use that, along with quality measures, will provide a comprehensive  assessment of performance. We continue to believe that providing  physicians feedback on their performance could be a promising step  toward encouraging greater efficiency in Medicare; however, we are still  concerned that efforts to achieve greater efficiency that rely solely on  physician feedback without financial or other incentives will be  suboptimal.
CMS also provided technical comments, which we incorporated as  appropriate.
The representatives of AAOS and ACC raised no major issues with regard  to the substance of the report. The AAOS representative said that the  report captured well the key aspects of physician profiling and the key  considerations in developing physician feedback reports. The ACC  representatives endorsed the overall approach of a feedback report  consisting of a high-level summary accompanied by additional sections  with greater detail and a separate document that explains the  methodology in detail. The representatives of both groups said that  physicians should be provided feedback on both quality and resource use,  but differed on whether they should be presented in the same report. Both  groups also stressed that physicians should only be compared to  physicians within their specialty or subspecialty.
Both the AAOS and the ACC representatives commented on the design of  our mock report. Both said that the measures of physician resource use by  type of service and the benchmark comparisons were easy to understand.  They had difficulty, however, in understanding a related measure that  shows the physician’s share of payments by service category. We did not  alter our mock report in response to these comments, but believe that the  concerns they expressed should be taken into account by organizations  designing physician feedback reports.
The representatives of both groups stressed the importance of risk  adjustment in the measurement of physician resource use and suggested  that we include a fuller explanation of risk adjustment techniques in our  report. We did not expand our explanation of such techniques because  they are not the focus of this report; however, we acknowledge the  important role played by risk adjustment techniques in constructing  physician feedback reports on resource use.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the Acting  Administrator of CMS, committees, and others. The report will also be  available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions, please contact me at (202) 512-7114  or steinwalda@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of the report.  GAO staff who made major contributions to this report are listed in  appendix III.
section_title: Appendix I: Methodology
paragraphs: This appendix describes the per capita methodology that we used to  measure beneficiaries’ and physicians’ Medicare fee-for-service (FFS)  resource use. We focused our analysis on four diverse specialties: a  medical specialty (cardiology), a diagnostic specialty (diagnostic  radiology), a primary care specialty (internal medicine), and a surgical  specialty (orthopedic surgery). We included diagnostic radiologists in our  study because they are less amenable to episode grouping, the major  alternative to per capita profiling of physicians. We limited our analysis to  physicians in these specialties who practiced in one of four areas: Miami,  Fla.; Phoenix, Ariz.; Pittsburgh, Pa.; and Sacramento, Calif. We chose  these areas for their geographic diversity, range in average Medicare  spending per beneficiary, and number of physicians in each of the four  specialties. Our results apply only to the four specialties in the four  metropolitan areas we studied.
To conduct our analysis, we obtained 2005 and 2006 Centers for Medicare  & Medicaid Services (CMS) data from the following sources: (1) Medicare  claims files that include data on physician, durable medical equipment,  skilled nursing, home health, hospice, and hospital inpatient and  outpatient services; (2) Denominator File, a database that contains  enrollment and entitlement status information for all Medicare  beneficiaries in a given year; (3) Hierarchical Condition Category (HCC)  files that summarize Medicare beneficiaries’ diagnoses; (4) files  summarizing the institutional status of beneficiaries; and (5) Unique  Physician Identification Number Directory, which contains information on  physicians’ specialties.
subsections: section_title: Adjustment for Differences in Patient Health Status
paragraphs: In order to develop a resource use measure that accounts for differences  in health status between beneficiaries, we developed a risk adjustment  model that uses an individual’s diagnoses during the year to estimate the  total Medicare FFS expenditures expected for the individual in that year.  As our inputs to the model, we used the same 70 HCCs as those in the  model CMS uses to set managed care capitation rates. HCCs are a way of  summarizing an individual’s diagnoses into major medical conditions, such  as vascular disease or severe head injury. To estimate our model, we used  HCC and expenditure data for 2005 and 2006 five percent national samples  of Medicare FFS beneficiaries.
section_title: Methodology Used to Determine Beneficiaries’ Resource Use
paragraphs: For all Medicare FFS beneficiaries who received at least one service in  2005 or 2006 from a physician located in any of our four metropolitan  areas and who also did not meet our exclusion criteria (see footnote 5),  we used our risk adjustment model to estimate their total expected  Medicare FFS expenditures. Based on their expected expenditures, we  placed beneficiaries into 1 of 25 discrete risk categories. The categories  were ordered in terms of health status from healthiest (category 1) to  sickest (category 25). Next, within each risk category and metropolitan  area, we ranked beneficiaries from 1 to 100 by their total actual annual  Medicare expenditures, such that the average beneficiary in a given risk  category and metropolitan area had a rank of 50. We used this rank as our  risk-adjusted measure of beneficiary resource use.
To examine the stability of beneficiaries’ resource use, we divided the 2005  and 2006 beneficiary populations into five ascending groups of nearly  equal size (quintiles) based on the level of their resource use. We then  identified beneficiaries in each of the four metropolitan areas who saw a  physician in their area in 2005 and again in 2006. We measured the stability  of beneficiaries’ resource use as the percentage of beneficiaries who  remained in the same quintile in 2006 that they were in during 2005. In  addition, we determined the percentage of beneficiaries who remained in  the highest resource quintile.
section_title: Methodology Used to Determine and Compare Physicians’ Resource Use
paragraphs: For the purposes of this study, we defined a physician’s practice as all  Medicare FFS beneficiaries who did not meet our exclusion criteria and  who had at least one evaluation and management visit with the physician  during the calendar year for cardiologists, internists, and orthopedic  surgeons, or who received any service from the physician for diagnostic  radiologists. To ensure that a physician’s resource use measure would  not be overly influenced by a few patients with unusually high or low  Medicare expenditures, we excluded physicians with small practices— those who treated fewer than 100 of the Medicare patients in our study  during the year. For all physicians, we calculated the average beneficiary  resource use rank of the patients in their practices, which ranged from a  low of 26.0 to a high of 91.8 in 2006. Next, within each metropolitan area  and specialty, we ranked physicians on the basis of this average from 1 to  100 such that the average measure of physician resource use was 50. We  used this rank as our measure of physician resource use. This measure  reflects how expensive a physician’s patients are compared to the patients  of other physicians in the same specialty and area after adjusting for  differences in patient health status. For example, a cardiologist in Miami is  only compared to other cardiologists in Miami.
To examine physicians’ resource use, we divided the physicians into five  ascending groups (quintiles) of nearly equal size based on the measure of  their resource use described above. In the same manner as we measured  the stability of beneficiaries’ resource use, we measured the stability of  physicians’ resource use by determining the percentage of them who  remained in the same physician resource use quintile from 2005 to 2006.  We also measured the degree of turnover in the patients seen by  physicians by computing the percentage of patients seen in 2005 by each  physician that were also seen by the same physician in 2006.
We examined utilization patterns by physician resource use quintile by  decomposing the 2006 Medicare expenditures of physicians’ patients into  those for institutional services (inpatient hospital and skilled nursing  care), those for services provided directly by the physician to his or her  patients, and those for all other services—outpatient hospital, home health  care, hospice care, durable medical equipment, and all other Part B  services of Part B providers and suppliers. We also measured the number  of physicians seen by a physicians’ patients by physician resource use  quintile.
Although our measure of a beneficiary’s resource use is independent of the  beneficiary’s health status, there was an association between physician  resource use and the mix of healthy and sick patients in physicians’  practices—physicians who ranked high in terms of resource use also  treated a larger proportion of beneficiaries who were in poor health than  did physicians who ranked low in resource use. However, the resource use  of all their patients was also consistently higher than that of low resource  use physicians’ patients regardless of patient health status. Figure 5 shows  the average resource use of beneficiaries in five health status categories  across the five physician resource use quintiles. For example, patients in  the healthiest category who were treated by physicians in the highest  resource use quintile had an average resource use rank of 74, whereas  similarly healthy patients treated by physicians in the lowest quintile had  average resource use rank of 53. This ordering of the differences in patient  resource use by the level of physician resource use is repeated across all  health categories. It indicates that physicians have consistent patterns of  resource use with respect to all of their patients, regardless of their  patients’ health status.
The mix of healthy and sick patients in physicians’ practices did not affect  the positive relationship we found between average institutional  expenditures per beneficiary and physician resource use level. Within each  beneficiary health category, the patients of high resource use physicians  had average institutional expenditures that exceeded those of the patients  of physicians with lower resource use. Similar analyses showed that  patient mix did not affect (1) the positive relationship between physicians’  resource use and the average number of physicians seen by their patients,  (2) the positive relationship between physicians’ resource use and  expenditures for all other services provided their patients, and (3) the  steeper rise in the use of institutional services by physicians’ patients with  increasing physician resource use as compared to the rise in the use of all  other services.
section_title: Appendix II: Comments from the Centers for Medicare & Medicaid Services
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: A. Bruce Steinwald, (202) 512-7114, or steinwalda@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Phyllis Thorburn, Assistant  Director; Alison Binkowski; Nancy Fasciano; Richard Lipinski; Drew Long;  Jessica Smith; Maya Tholandi; and Eric Wedum made key contributions to  this report.
section_title: Bibliography
paragraphs: Balas, E. Andrew, Suzanne Austin Boren, Gordon D. Brown, Bernard G.  Ewigman, Joyce A. Mitchell, and Gerald T. Perkoff. “Effect of Physician  Profiling on Utilization: Meta-analysis of Randomized Clinical Trials.”  Journal of General Internal Medicine, vol. 11, no. 10 (1996): 584-590.
Beckman, Howard B., Anthony L. Suchman, Kathleen Curtin, and Robert  A. Greene. “Physician Reactions to Quantitative Individual Performance  Reports.” American Journal of Medical Quality, vol. 21, no. 3 (2006): 192- 199.
Beckman, Howard B., Thomas Mahoney, and Robert A. Greene. Current  Approaches to Improving the Value of Care: A Physician’s Perspective.  The Commonwealth Fund. November 2007.
Hartig, J.R., and Jeroan J. Allison. “Physician Performance Improvement:  An Overview of Methodologies.” Clinical and Experimental  Rheumatology, vol. 25, supplement 47 (2007): S50-S54.
Jamtvedt, Gro, Jane M. Young, Doris Tove Kristoffersen, Mary Ann  O’Brien, and Andrew D. Oxman. “Audit and Feedback: Effects on  Professional Practice and Health Care Outcomes.” Cochrane Database of  Systematic Reviews, no. 2 (2006).
Jamtvedt, Gro, Jane M. Young, Doris Tove Kristoffersen, Mary Ann  O’Brien, and Andrew D. Oxman. “Does Telling People What They Have  Been Doing Change What They Do? A Systematic Review of the Effects of  Audit and Feedback.” Quality & Safety in Health Care, vol. 15, no. 6  (2006): 433-436.
Kiefe, Catarina I., Jeroan J. Allison, O. Dale Williams, Sharina D. Person,  Michael T. Weaver, and Norman W. Weissman. “Improving Quality  Improvement Using Achievable Benchmarks for Physician Feedback: A  Randomized Controlled Trial.” JAMA, vol. 285, no. 22 (2001): 2871-2879.
Marder, Robert J., Mark A. Smith, and Richard A. Sheff. “Changing  Physician Practice: Providing Physicians Useful Feedback.” In Effective  Peer Review: A Practical Guide to Contemporary Design, 2nd ed., 153- 164. Marblehead, Mass.: HCPro, Inc., 2007.
Micklitsch, Christine N. and Theresa A. Ryan-Mitlyng. Physician  Performance Management: Tool for Survival and Success. Englewood,  Colo.: Medical Group Management Association, 1996.
Mold, James W., Cheryl A. Aspy, and Zsolt Nagykaldi. “Implementation of  Evidence-Based Preventive Services Delivery Processes in Primary Care:  An Oklahoma Physicians Resource/Research Network (OKPRN) Study.”  Journal of the American Board of Family Medicine, vol. 21, no. 4 (2008):  334-344.
Nathanson, Philip. “Influencing Physician Practice Patterns,” Topics in  Health Care Financing, vol. 20, no. 4 (1994): 16-25.
Pacific Business Group on Health. Advancing Physician Performance  Measurement: Using Administrative Data to Assess Physician Quality  and Efficiency. September 2005.
Paxton, E. Scott, Barton H. Hamilton, Vivian R. Boyd, and Bruce L. Hall.  “Impact of Isolated Clinical Performance Feedback on Clinical  Productivity of an Academic Surgical Faculty.” Journal of American  College of Surgeons, vol. 202, no. 5 (2006): 737-745.
Teleki, Stephanie S., Rebecca Shaw, Cheryl L. Damberg, and Elizabeth A.  McGlynn. Providing Performance Feedback to Individual Physicians:  Current Practice and Emerging Lessons. RAND Health Working Paper  Series. July 2006.
Van Hoof, Thomas J., David A. Pearson, Tierney E. Giannotti, Janet P.  Tate, Anne Elwell, Judith K. Barr, and Thomas P. Meehan. “Lessons  Learned from Performance Feedback by a Quality Improvement  Organization.” Journal for Healthcare Quality, vol. 28, no. 3 (2006): 20-31.
Veloski, Jon, James R. Boex, Margaret J. Grasberger, Adam Evans, and  Daniel B. Wolfson. “Systematic Review of the Literature on Assessment,  Feedback and Physicians’ Clinical Performance: BEME Guide No. 7.”  Medical Teacher, vol. 28, no. 2 (2006): 117-128.