The role of reputation in U.S. News & World Report's rankings of the top 50 American hospitals.

For the past 20 years, the newsmagazine U.S. News & World Report has published annual rankings of the top 50 American hospitals in several specialties. These rankings generate widespread attention among the general public, health care providers, and policymakers. Other media outlets, such as newspapers, amplify the visibility of these rankings through their own stories about top-ranked hospitals. Hospital leaders generally believe that the U.S. News & World Report rankings are accurate and use them as marketing tools to attract patients (1, 2). In addition, political leaders and health policy researchers searching for models for health care reform often focus on U.S. News & World Report's top-ranked hospitals (3). U.S. News & World Report's rankings combine 3 quality domains with approximately equal weighting: structure, process, and outcomes. Structure is the people and resources involved in providing health care, process refers to how care is delivered, and outcomes are the results of such care. Structure and outcomes were assessed by objective measures, including nurse staffing, patient volume, patient safety adverse events, and adjusted mortality. Process was assessed primarily by the single subjective measure of hospital reputation. The rankings have been criticized for emphasizing too strongly the subjective reputation of a few nationally prominent hospitals, but the role of reputation has not been quantified (1, 4). The goal of this study is to quantify the role of reputation in determining the relative standings of the top 50 hospitals in each of the 12 specialties evaluated (cancer; diabetes and endocrine disorders; digestive disorders; ear, nose, and throat; geriatric care; gynecology; heart and heart surgery; kidney disorders; neurology and neurosurgery; orthopedics; respiratory disorders; and urology). This study particularly attempts to determine whether a high degree of variation in the subjective reputation of hospitals results in a composite ranking that is dominated by subjective reputation. Understanding the role of reputation may help users interpret the rankings more knowledgeably and may inform efforts to improve the assessment of hospital quality. Methods Hospital Rankings Data For its 2009 rankings, U.S. News & World Report selected 1859 American hospitals on the basis of their teaching status, number of beds, number of Medicare inpatient discharges, and availability of specific medical technologies. Subjective and objective components were then used to calculate a specialty-specific index of hospital quality, defined as the total U.S. News score, for each hospital. Subjective reputation was assessed from survey responses from approximately 250 board-certified physicians from across the country who were randomly selected from the American Medical Association's Physician Masterfile in each specialty. These physicians were asked to identify 5 hospitals in their specialty that provide the best care to patients. Response rates were typically 40% to 50% across specialties and generally were similar across the country. Objective components obtained from Medicare reimbursement claims included a risk-adjusted mortality index, a patient safety index (indicators of potential adverse events), and the number of discharges. Objective components obtained from the American Hospital Association or the American Nursing Association included nurse-to-patient ratio; Nurse Magnet hospital designation, defined as meeting standards established by the American Nurses Credentialing Center; and availability of key medical technologies, specific patient services, a trauma center, and intensivists. For specific specialties, additional information was obtained about whether hospitals were designated or accredited as specialized centers for cancer, bone marrow transplantation, epilepsy, and Alzheimer disease (5, 6). In addition, reputation alone was used to rank 4 other specialties (ophthalmology, psychiatry, rehabilitation, and rheumatology), although these specialties were not included in this study. These subjective and objective components were combined as a weighted sum and then transformed into a 100-point scale to create the total U.S. News score. These components and the total U.S. News scores for the 2009 rankings were obtained from a publicly available report (5). For each specialty, the 50 hospitals with the highest total score were ranked numerically and published in U.S. News & World Report. The remaining hospitals were not numerically ranked, but their individual total scores are available on the rankings Web site (7). To evaluate the distribution of reputation scores among unranked hospitals, a random-number generator was used to identify 100 hospitals that were unranked in any specialty. Reputation scores were obtained from the rankings Web site for these hospitals. A methods report (5) provides further details on all data elements and related calculations of the hospital rankings. Statistical Analysis Four complementary approaches were used to understand the role of reputation score. First, descriptive statistics (mean, SD, and percentage) were used to examine the distribution of reputation score and other components of the total U.S. News score for the 50 top-ranked hospitals in each specialty. In addition, the extent of variation in each component was quantified by calculating a coefficient of variation (defined as 100 the SD-to-mean ratio). Second, the distribution of reputation scores was examined among the 100 randomly selected unranked hospitals. Third, the 50 top-ranked hospitals in each specialty were ranked on the basis of reputation score alone. Hospitals with identical reputation scores received the same rank equal to the average of the ranks they would have received without having identical scores. These rankings based on reputation alone were compared with those based on the total U.S. News score. For example, the percentage of agreement was determined by the number of top 20 hospitals identified by the total U.S. News score that also were categorized as such on the basis of reputation score alone. A similar method was used to examine agreement between the 2 methods for the top hospital, the top 5 hospitals, and the top 10 hospitals in each specialty. Fourth, the association between reputation score and the total score from all objective measures was examined among the 50 top-ranked hospitals in each specialty. The exact formula for calculating the total U.S. News score from individual components is not published, but it is a linear function of these components (5). As a result, multiple linear regression was used to determine the relationship between the total U.S. News score and its components. For each specialty, the outcome variable was the total U.S. News score and the predictor variables were reputation score as well as all objective measures, including the risk-adjusted mortality index, patient safety index, number of discharges, nurse-to-patient ratio, Nurse Magnet hospital designation, number of key technologies, number of patient services, trauma center designation, and presence of intensivists. The Pearson correlation coefficient squared (r2 ) for these specialty-specific regression models ranged from 0.99 to 1.00, implying a nearly perfect fit. The regression models were then used to estimate the portion of the total U.S. News score derived from all the objective components, referred to as the total objective measures score. The nonparametric Spearman rank correlation was calculated to assess the univariate associations among reputation score, total objective measures score, and total U.S. News score. The 2 values were used to indicate the proportion of variation in ranks of 1 score that were accounted for by the other score (8). Data about specialized centers for cancer, bone marrow transplantation, epilepsy, and Alzheimer disease are relevant for only 2 of the 12 specialties studied but were included in the appropriate regression models. All analyses were performed using JMP software, version 6.0.2 (SAS Institute, Cary, North Carolina). Role of the Funding Source No funding was received for this study. Results Table 1 summarizes the total U.S. News score, reputation score, and other components for the top-ranked 50 hospitals in each specialty. Only a few hospitals had sizeable reputation scores. On average, 64% of reputation scores among top-ranked hospitals across all specialties were less than 5.0, 31% were 5.0 to 30.0, and 4% were greater than 30.0. By contrast, all of the randomly selected unranked hospitals had reputation scores less than 5.0. Table 1. Total U.S. News Score and Its Components for the 50 Top-Ranked Hospitals Table 2 quantifies the extent of variation in reputation score and other components of total U.S. News score for all 12 specialties. The coefficients of variation for reputation score were generally greater than 100%, indicating a high degree of variation across the 50 hospitals in each specialty. The coefficients of variation for all other components were less than 100%, indicating a smaller degree of variation. For example, among 50 top-ranked cancer care hospitals, the coefficient of variation was 177% for reputation score and ranged from 6% to 63% for other components. The Figure illustrates the high correlation between total and reputation scores for cancer care. Extent of variation was not calculated for items that can only take on 1 of 2 values (for example, a facility is or is not a Nurse Magnet hospital), because the variation possible in such dichotomous items is inherently limited and is evident from the percentages for these items listed in Table 1. Table 2. Variation in Components of Total U.S. News Score for the 50 Top-Ranked Hospitals Figure. Total U.S. News score vs. reputation score among the 50 top-ranked cancer hospitals. The vast majority of rankings based on reputation score alone agreed with U.S. News overall rankings (Table 3