Profiling Care Provided by Different Groups of Physicians: Effects of Patient Case-Mix (Bias) and Physician-Level Clustering on Quality Assessment Results

Identification of physician characteristics associated with optimal management of chronic disease could serve as the basis for designing quality improvement initiatives. However, assessment of the quality of care provided by physicians with different characteristics (for example, generalists vs. specialists, male vs. female, or those in health maintenance organizations vs. those with fee-for-service arrangements) presents a unique set of methodologic problems that stem from the nature of outpatient medicine. First, if older or sicker patients with multiple diseases have different needs for health care services and different health outcomes independent of the quality of care they receive, physicians who see such patients may appear to provide lower quality of care than do those who see younger patients with less comorbid disease. Accounting for these patient characteristics (that is, avoiding case-mix bias) is therefore an essential feature of fair and accurate comparisons of physicians' quality of care. Second, patients with specific characteristics (such as age, sex, or health problems) choose and remain with physicians who have specific characteristics (such as age, sex, specialty training, proximity, or practice style). Patients in a physician's practice (or even those seen by a small group of physicians practicing together) might therefore cluster, that is, be more like each other and differ from patients who are drawn to another physician's practice. Thus, patients are not independent observations, as assumed by standard statistical techniques. While also inherent in any unit or cluster being compared (such as a health plan, a hospital, or a state), the smaller and more homogenous the unit (for example, the individual physician's office practice as opposed to a hospital), the greater the potential for a clustering problem (1). Accounting for physician-level clustering is therefore another key feature of scientifically sound profiling of physicians' quality of care. Finally, to be reliable for profiling at the physician level, quality-of-care measures must be chosen that register the physician's style, or thumbprint. For example, measures of the process of medical care (such as test ordering or interpersonal behavior) are more directly under the physician's control and are therefore more likely than some outcome and utilization measures to register the physician's thumbprint (1, 2). To detect physician style reliably, a sufficient number of patients per physician must be sampled. Although these methodologic issues have been recognized in the design and analysis of randomized trials (1, 3, 4), attempts to address both patient case-mix and physician-level clustering in observational studies of quality-of-care assessment have been limited (2, 5-12). As part of a continuing effort to draw attention to quality of care, the American Diabetes Association (ADA) developed the Provider Recognition Program to acknowledge physicians who provide high-quality diabetes care. To illustrate the effects of patient characteristics and physician-level clustering on quality assessment results among patients with diabetes, we used ADA Provider Recognition Program data to compare two groups of physicians, in this case generalists and specialists, whose results were expected to differ (7, 13). Methods Definitions Groups of physicians refers to physicians who share a characteristic, such as specialty; it does not designate a collection of physicians practicing together. Site or practice refers to the office setting, as in solo or group practice or endocrinologist practices, not to the behavior of physicians (as in practice patterns). We use a broad definition of case-mix that includes such patient characteristics as age, sex, level of education, and health status. Physician-level clustering refers to the variation in any measurements (patient characteristics or quality-of-care measures) associated with the physician. In this study, it reflects the extent to which rates of performance of quality measures differ between physicians within a specialty. The effect of this clustering is summarized by the inflation factor (also called design effect [1]), which is the ratio of estimated variances of differences between specialty groups, with and without adjustment for clustering (see Glossary). Site Selection and Patient Sampling A convenience sample of 15 managed care plans, 10 clinics, and 6 endocrinology practices responded to a nationwide ADA solicitation to participate in a diabetes-related quality-of-care study. Seven of the 15 managed care plans agreed to recruit physicians. All 10 clinics and all 6 of the endocrinology practices agreed to have some or all of their physicians participate. Thirty-one practice sites that included a total of 63 physicians agreed to participate in the study. Of these 31 sites, 2 dropped out because of the burden of data collection. Each participating practice self-reported specialty of the practice. To avoid misclassification of practices, the investigators verified specialty. Physicians at each participating practice were asked to submit data only for those patients for whom they provided the principal diabetes care. To maximize the reliability of physician-level quality measures, approximately 35 patients per site were to be sampled. All but 3 practice sites recruited 27 or more patients; the average number of patients sampled per site was 67. All patients received most of their diabetes care at the practice site. Practice sites recruited consecutive patients starting at an index date. Data Collection Clinical data were abstracted from medical records for all visits to participating physicians in the 12 months immediately preceding the index date. The care assessed was delivered between October 1994 and July 1996. Following detailed ADA specifications, personnel at each practice site abstracted study measures from patients' medical records. Abstraction by sites was determined to agree with expert ADA personnel by performing duplicate abstraction at two participating practices. A survey to collect patient case-mix variables and ratings of interpersonal care was also administered at the time of recruitment. A majority of patients (69%) returned the survey by mail; the remainder (31%) completed the survey in the physician's office. The method of survey administration did not affect the variables analyzed. Completed patient surveys were obtained from 1258 (73.5%) patients. Compared with responders, nonresponders had a higher mean hemoglobin A1c level (0.091 [9.1%] vs. 0.087 [8.7%]), had had diabetes for less time (10.8 vs. 14.3 years), and fewer received most of their care from an endocrinologist (68% vs. 73%). Since clinical measures not found in the medical record were scored as not performed rather than as missing, none of these measures was imputed. Missing data from the survey exceeded 5% for only four variables. Values were imputed for single-item measures by using Hot-Deck (14); multi-item scales were imputed by using each respondent's average of completed items for those completing at least 50% of scale items. Although multiple imputation methods are available, a single imputation procedure for missing data was chosen for this project in order to focus on the issues of clustering and case-mix, apart from additional sources of variability. Measures A panel of experts recruited by the ADA identified process and outcome measures that together defined a comprehensive evaluation of quality of care for diabetes. These measures were lipid profile (frequency of measurement and levels of total cholesterol, high-density lipoprotein cholesterol, low-density lipoprotein cholesterol, and triglycerides), dilated eye examination (a note, report, or letter from a physician or optometrist documenting the date of the examination, or a dated photograph of the retina), foot examination (reference to a visual inspection or sensory examination of the feet), blood pressure, urinalysis or any test for urinary protein (such as 24-hour collection or dipstick protein testing), and self-monitoring of blood glucose (any mention of patient-collected blood glucose values and any mention of patient self-adjustment of diet or treatment on the basis of blood glucose results). Glycated hemoglobin was recorded as hemoglobin A1c or total glycated hemoglobin. Total glycated hemoglobin values were converted to hemoglobin A1c values. For items that appeared more than once during the study year, the most recent value recorded was used. Satisfaction with care was measured by using four well-tested survey items, rated on a five-point scale from excellent to poor (15): personal manner of the health care provider, overall diabetes care, explanation of laboratory results, and how well questions were answered. The proportion of excellent scores was used in the analysis. Self-reported health status was measured by using a single five-point rating scale from excellent to poor. This item has been shown to reflect the summary of diabetes complications and comorbid conditions (16). Statistical Analysis All data were analyzed by using SAS software (17). We used univariate analysis to examine frequency distributions and variability in study measures. Derived variables from the patient survey were tested for internal consistency reliability by using the Cronbach value; reliability coefficients exceeded 0.70 for all study measures. Characteristics of respondents and nonrespondents were compared by using t-tests for continuous variables and chi-square tests for proportional variables. For each process and outcome measure, regression models were used to calculate estimated odds ratios and 95% CIs comparing the two specialties. Odds ratios were estimated from regression models with no adjustments, inclusion of patient case-mix only, inclusion of physician-level clustering only, and inclusion of both case-mix and physician-level clustering. Although the