MEMORANDUM

TO: Michael Wroblewski, Craig Caplan, Sabrina Ahmed

FROM: Wil Lim, Gene Rich, Jeff Ballou, Bob Schmitz DATE: 3/27/2013

SUBJECT: Comparability Peer Grouping: Refinements and Assessments

The purpose of this memo is to describe results from refinements to the comparability peer 

grouping (CPG) benchmarking methodology. These refinements include different (1) specialtymix and (2) practice size categorizations. Applying these refinements, we assessed the following 

aspects of CPG benchmarking: peer groups, benchmarks, and cost tiers. We assessed peer groups 

constructed from the national sample of TINs (that is, those with at least 1 eligible professional, 

or EP). We assessed peer groups, benchmarks, and cost tiers based on the sample of 870 TINs in 

a nine-state region with at least 25 EPs.

Key Findings

1. When practices are characterized by the share of EPs from each CMS specialty 

represented in the practice, peer groups of at least 30 peers with identical specialty 

configurations can be formed for 93% percent of practices nationwide.

2. Constructing peer groups based on practice size in addition to specialty mix has a 

relatively small impact on peer groups and benchmarks.

3. Constructing CPG benchmarks based on peer specialty mix reduces differences between 

practices’ per capita costs and their benchmark means relative to using a national 

benchmark, on average. This reflects an increase in peer comparability moving from a 

national peer group to CPG peer groups.

4. Medical group practices dominated by specialties that are thought to be 

disproportionately represented among outliers when using a national benchmark appear 

much less likely to be categorized as outliers under CPG.

5. Although results relating to benchmarks and cost tiers are inferred from the 870-TIN 

sample, these are potentially conservative estimates of the ability of CPG to compute 

benchmarks from more comparable peers. This is because larger samples will generally 

increase the number of peers with any given specialty configuration.
MEMO TO: Michael Wroblewski, Craig Caplan, Sabrina Ahmed 

FROM: Wil Lim, Gene Rich, Jeff Ballou, Bob Schmitz

DATE: 3/27/2013

PAGE: 2

Description of Refinements

To determine peers, for each practice we computed the percentage of EPs in each specialty, 

which we then classified into one of six categorical percentage groupings: 0%, 1-20%, 21-40%, 

41-60%, 61-80%, and 81-100%. Each practice’s profile is then determined by the percentage 

grouping for each specialty category. Note that in two of the categorizations we tested, we 

restricted profiling to be based on physicians only. We tested the following three specialty 

categorizations:

1. “Part B Drug Specialties”

a. Allergy-immunology; infectious disease; rheumatology; hematology; medical 

oncology; hematology/oncology

b. Other physician specialties

c. Non-physicians

2. “All CMS Specialties”

a. Each of the CMS physician specialties

3. “15 Specialties”

a. Primary care physicians (consisting of general practice, family practice, internal 

medicine, and geriatric medicine)

b. Emergency medicine

c. Cardiology

d. General surgery

e. Psychiatry

f. Anesthesiology

g. Orthopedic surgery

h. Neurology

i. Diagnostic radiology

j. Gastroenterology

k. Hematology/oncology (includes hematology/oncology, hematology, oncology)

l. Urology

m. Obstetrics/gynecology

n. Pediatrics

o. Other (includes all other physician specialties)

Similarly, we tested stratifying practices based on practice size. We tested the following 

practice size categorizations:

1. National sample

a. 1+ EPs

b. 1-24 EPs; 25-99 EPs; 100+ EPs
MEMO TO: Michael Wroblewski, Craig Caplan, Sabrina Ahmed 

FROM: Wil Lim, Gene Rich, Jeff Ballou, Bob Schmitz

DATE: 3/27/2013

PAGE: 3

c. 1-24 EPs; 25-49 EPs; 50-99 EPs; 100+ EPs

2. 870-TIN sample

a. 25+ EPs

b. 25-99 EPs; 100+ EPs

c. 25-49 EPs; 50-99 EPs; 100+ EPs

CPG Peer Groups

Table 1 shows, for each specialty and practice categorization, the distribution of 

comparability levels required to attain a peer group of 30 practices (inclusive of the practice 

itself). In general, the finer the categorization, the more levels necessary to complete a peer 

group of at least 30 practices. Of the 870 TINs that were categorized based on Part B Drug 

Specialties and regardless of practice size, 88 percent (766) had at least 30 peers in their same 

“bucket”, where a bucket is defined as having the same specialty percentage categorizations. No 

practices categorized based on 15 Specialties categories or All CMS Specialties categories had at 

least 30 peers in their same bucket. It is important to note that, by definition, coarser specialty 

categorizations will always have more peers in the same bucket. Since the Part B Drug 

Specialties categorization is coarser then the 15 Specialties and All CMS Specialties

categorizations, it inherently has more practices in each bucket than the others. Thus, this metric 

is of limited use for assessing the relative superiority of one specialty categorization over 

another.

Table 1 also shows the corresponding peer groups constructed from the national sample of 

TINs. Nearly all practices that were categorized based on Part B Drug Specialties and regardless 

of practice size had at least 30 peers in their same bucket, and 96 percent of practices that were 

categorized based on the 15 Specialties categories and regardless of practice size had at least 30 

peers in their same bucket. For the All CMS Specialties categories, this percentage was 93 

percent.

Table 1 also shows that these “completion percentages” vary by practice size. Using the All 

CMS Specialties categories to peer group, 94 percent of practices with 1-24 EPs, 34 percent of 

practices with 25-99 EPs, and 7 percent of practices with 100+ EPs had at least 30 peers in their 

same bucket. There are two main reasons for this. First, there are many more small practices than 

large practices—97 percent of practices have fewer than 25 EPs. Second, smaller practices have 

inherently less varied specialty mixes than larger, often multi-specialty, practices. The latter has 

two implications: small and large practices tend not to be peer-grouped together, and large 

practices have fewer peers that are very similar in specialty mix.

In summary, we would be able to construct peer groups consisting of other practices in the 

same bucket for over 90 percent of practices, even when using the All CMS Specialties 
MEMO TO: Michael Wroblewski, Craig Caplan, Sabrina Ahmed 

FROM: Wil Lim, Gene Rich, Jeff Ballou, Bob Schmitz

DATE: 3/27/2013

PAGE: 4

categories. Moreover, stratifying also by practice size when creating peer groups has a relatively 

small effect.

Benchmarks: CPG vs. National Average

One useful metric for assessing the comparability of practices to their peer groups is the 

average per capita cost difference between practices and their benchmark (peer group 

performance). For example, dermatologists have substantially lower per capita costs than other 

specialties on average because the patients they see tend to be healthier in a way that might not 

be fully captured by HCC scores. However, if we were instead to compare a practice having a 

large proportion of dermatologists to other practices also having a large proportion of 

dermatologists, we expect the difference in per capita costs to be lower. Furthermore, we expect 

that these differences would be driven more by actual variation in performance, since the types 

of patients seen are likely to be more similar. Thus, as we improve comparability of peer groups, 

we expect deviations from benchmarks to diminish as we reduce the component due to treating 

different types of patients and retain differences due to actual performance.

Table 2 shows that per capita cost differences between practices and their peer groups is 

smaller, and in many cases substantially so, when benchmarking against a comparable peer 

group versus benchmarking against the national average. On average, the difference is relatively 

unchanged when creating peer groups based on the Part B Drug Specialties categories versus 

using the national average. However, when creating peer groups based on the 15 Specialties and 

All CMS Specialties categories, these differences are reduced by 39 percent and 59 percent, 

respectively. These differences are consistent regardless of whether we additionally use practice 

size categories to construct peer groups. Note that while creating peer groups based on Part B 

Drug Specialties categories has little impact overall, it does improve comparisons for the 

specialties for whom Part B drugs represent a significant part of their revenues, which we 

address in the next section.

These findings suggest that constructing more comparable peer groups based on specialty 

mix substantially reduces the noise in measuring relative cost performance.

Benchmarks for Groups Dominated by a Single Physician Specialty

While the previous table shows that CPG creates more comparable peer groups overall and 

reduces the noise in assessing relative performance, it is useful to assess its impact for certain 

specialties, particularly those disproportionately represented among outliers when using a 

national benchmark. To do this, we measure per capita cost differences between practices and 

their benchmarks for practices where a single specialty accounts for at least 60 percent of the 

physicians in a practice. These results are displayed in Table 3. Consider, for example, the 

practices composed of at least 60 percent of dermatologists. As expected, their per capita costs 

are more than $4,000 lower than the national average. However, by benchmarking against more 
MEMO TO: Michael Wroblewski, Craig Caplan, Sabrina Ahmed 

FROM: Wil Lim, Gene Rich, Jeff Ballou, Bob Schmitz

DATE: 3/27/2013

PAGE: 5

comparable practices, particularly based on 15 Specialties categories or All CMS Specialties 

categories, this difference is reduced by more than one third. Conversely, for practices with a 

large percentage of physicians specializing in hematology/oncology, their per capita costs are 

nearly $5,000 higher than the national average. In this case, CPG reduces the per capita cost 

differences by more than 75 percent for all specialty categorization methods. Note that the Part B 

Drug Specialties categorization also performs well in this case because hematology/oncology 

physicians are included among specialties with a high proportion of Part B Drug revenue.

While the number of practices with a high percentage of any given specialty is low, the 

findings in this section are consistent across several specialties, suggesting that these are not 

random aberrations. In addition, in Table 3b, we display results from repeating the exercise for 

practices with various percentages of hematology/oncology physicians. We find that, relative to 

using a national benchmark, CPG reduces the average absolute difference between per capita 

costs for practices with hematology/oncology physicians and their peer group means, regardless 

of the hematology/oncology percentage.

These findings highlight that in creating more comparable peer groups, CPG can reduce the 

average absolute deviation from the peer group benchmark, relative to the national benchmark, 

for practices that consist of physicians with particular specialties.

Cost Tiering

Ultimately, benchmarks are used to compute cost scores for assigning practices to low, 

average, or high cost tiers. Specifically, practices are assigned to tiers based on whether their cost 

scores fall at least one standard deviation from the mean cost score and are also statistically 

significantly different from the mean cost score. We address this here, with two caveats. First, 

we consider only overall per capita costs and not a composite of overall and condition specific 

per capita costs. Because overall per capita costs are highly correlated with condition specific per 

capita costs, we expect this to have a minimal impact on these findings. Second, we construct 

tiers based on only the one standard deviation from the mean criterion. That is, we do not 

consider statistical significance. While doing so would lead to more parsimonious tier 

assignments, we expect this to have a minimal impact as previous analyses on the 870 TIN 

sample suggest that less than 10 percent of one standard deviation outliers are not statistically 

different from the mean. 

Table 4 shows the distribution of the practices in which a single physician specialty accounts 

for at least 60 percent of physicians into low, average, and high cost tiers for various 

benchmarks. Although the samples are small, they show CPG moving some dermatologist 

dominated practices that would be considered low by national benchmarking into the average 

tier. Conversely, CPG moved some hematology/oncology practices from high cost tiers into 

average tiers (the TIN tiered as high using the All CMS Specialties categorization is near the 

threshold separating high cost from average cost, at 1.06 standard deviations above the mean).
MEMO TO: Michael Wroblewski, Craig Caplan, Sabrina Ahmed 

FROM: Wil Lim, Gene Rich, Jeff Ballou, Bob Schmitz

DATE: 3/27/2013

PAGE: 6

Again, this is consistent with CPG reducing the average absolute deviation from the peer group 

benchmark, relative to using a national benchmark, for these specialties.

It is also useful to know overall the distribution of practices into tiers using CPG versus the 

national benchmark. Tables 5 compare the assignment of practices to tiers using CPG versus the

national benchmark. In general, while CPG brings in TINs previously classified as high or low 

(due at least in part to the 60 plus percent of a specialty cases discussed previously), the 

methodology tends to produce more outliers. However, this likely reflects CPG improving 

benchmarking. Because there are large outliers when using the national average as a benchmark, 

this causes the standard deviation to be skewed upward, resulting in fewer high and low 

categorizations. However, as CPG reduces deviations from benchmarks, particularly for extreme 

outliers, the standard deviation shrinks, resulting in more outliers. This explains why CPG using 

the All CMS Specialties categorization regardless of size produces 33 percent outliers and the 

national benchmark only produces 14 percent. Also, keep in mind that additionally applying the 

statistical significance criterion will lead to slightly more parsimonious tiering.

Thus, consistent with finding that CPG improves benchmarking by shrinking the average 

absolute deviation from the peer group benchmark, CPG appears to improve cost tiering relative 

to using a national benchmark. Practices which tend to be disproportionately represented as 

outliers when using the national benchmark are less likely to be classified as such under CPG.

Discussion

These refinements addressed two important concerns about CPG: whether most peer groups 

would be constructed from peer groups with identical specialty categorical-percentage 

distributions and whether it would reduce the bias faced by certain specialties when using the 

national benchmark. In both cases, the response is “yes”. It should also be emphasized that CPG 

creates these benchmarks by forming peer groups at the medical group practice level, and not 

relying on the performance of individual physicians, addressing another concern by stakeholders.

One concern about the Part B Drug Specialties and 15 Specialties categorizations is the 

somewhat arbitrary profiling of practices. While the purpose of the Part B Drug Specialties 

categorization in particular was clear in addressing the two concerns above, it is useful to know 

that these were addressed by using the national sample and all CMS physician specialties, 

without the need to target specific ones.

It should also be noted that the 870 TIN sample produces conservative estimates for the 

performance of CPG. Using the national sample, versus a smaller sample, will likely result in 

more comparable peer groups from having more peers for any given specialty distribution. The 

results here suggest that these more comparable peer groups will result in further improvement to 

benchmarking.
MEMO TO: Michael Wroblewski, Craig Caplan, Sabrina Ahmed 

FROM: Wil Lim, Gene Rich, Jeff Ballou, Bob Schmitz

DATE: 3/27/2013

PAGE: 7

A useful next step for testing any benchmarking methodology is to conduct a split sample 

test, which we are conducting for CPG and national benchmarking. In this test, beneficiaries for 

each group are randomly divided into two samples, A and B. Using each sample, groups are 

tiered into low, average, and high. If a method were to accurately assess groups’ true 

performance, the tiers should vary little between the two samples.

Lastly, the potential for improved benchmarking to produce more outliers is an important 

consideration to keep in mind when testing different ways of tiering. Appropriately adjusting 

tiering approaches in light of improved benchmarks will be important for assessing relative 

performance of practices and achieving other policy objectives, such as determining the suitable 

percentage of outliers. 

cc: Mai Hubbard; Mary Laschober; Jessica Heeringa