first assessed the prevalence of the missing values. We didn’t include in the model any variable with a high missing value rate (greater than 10%). We then created a dummy variable to represent the missing value corresponding to the abstracted variable, and used the dummy variable together with the abstracted variable in the model. This method of modeling missing data assumes that the data are missing at random and permits inclusion of all available cases, although it is not as efficient as multiple imputation procedures. We computed measures of model fit and discrimination for the medical record model similar to those computed for the claims-based models. RSMRs We calculated the RSMR for each hospital using the estimated hospital-specific parameters from the respective hierarchical models. These rates are obtained as the ratio of predicted to expected mortality (similar to the observed/expected ratio), multiplied by the national unadjusted rate [21]. The ratio is predicted mortality in each hospital, given its patient mix and hospital-specific effect, divided by the expected mortality in that hospital given the same patient mix and the national average [21]. The predicted hospital outcome is the number of expected mortalities in the ‘‘specific’’ hospital. Operationally, we estimated all the model parameters by regressing the outcomes on the predictors for the entire sample. We then multiplied the estimated regression coefficients by the patient characteristics observed in each hospital, added the hospital-specific estimated intercept, transformed to the probability scale, and then summed over all patients in the hospital. The expected outcome for each hospital is the number of 30-day deaths expected in the hospital given the observed case-mix. We then multiplied the, estimated regression coefficients by the patient characteristics observed in each hospital, added the national intercept, transformed to the probability scale, and then summed overall all patients in the hospital. This strategy is a form of indirect standardization. The higher a hospital’s predicted 30-day mortality rate, relative to expected mortality for the hospital’s particular case mix of patients, the higher its adjusted mortality rate will be. To assess the validity of the administrative data model, we calculated state-specific RSMR using the procedures described above and compared these rates with state-specific RSMRs obtained from the medical record model. Because the selection of cases for chart abstraction as a part of the National Pneumonia Project [14] was done at the state level, the number of patients with available medical record data from each hospital was relatively small. We thus used 2 approaches to examine the relationship between the RSMRs obtained from administrative data and medical record data. First, we estimated a linear regression equation describing the linear association between the 2 rates, weighting each state by the number of hospitalizations, and calculated the intercept and the slope of this equation. A slope close to 1 and an intercept close to 0 would indicate a strong linear relationship. Second, for each state we calculated the difference between the RSMR based on the claims data and the medical record data, and then summarized the distribution of these differences among the hospitals using the average, median, and maximum difference. Differences close to 0 would support agreement between the claims-based and medical record-based approaches. Stability of the Model over Time We compared the performance of the claims model over time in various validation cohorts, calculating similar model fit indices as those used in the derivation and validation samples. To assess whether we included too many risk factors in our final model, we Claims Model for Mortality calculated indices that quantify overfitting. Specifically, we used the coefficients estimated from the derivation model to predict the log-odds of mortality in the validation cohorts. This was accomplished by multiplying the observed risk factors in each validation cohort and summing over the covariates for a subject to obtain a mortality score. Using these scores for each subject, we then estimated a logistic regression model in which the outcome was observed mortality and the single covariate was the risk score. The intercept and slope obtained from this model are referred to as overfitting indices. If there is overfitting, we would expect the slopes to be different from 1 and the intercepts to be different from 0. We repeated this process for each validation dataset, each time calculating a risk score using the regression estimates from our derivation model. After computing the overfitting statistics, in each validation dataset, we recalibrated the model so that we used the same variables but fit the model to the data for each specific cohort. For each model, we calculated the same indices for assessing model performance [19] as in the derivation model. All analyses were conducted using SAS version 8.02 (SAS Institute Inc., Cary, NC). Models were fitted separately to each year of data. The hierarchical models were estimated with the use of the GLIMMIX macro in SAS. Ethics committee approval and written consent from patients for the analysis were not required. This analysis was performed under a CMS-directed contract with the Colorado Foundation for Medical Care (CFMC), the Colorado Quality Improvement Organization. As such, CFMC functions as a health oversight agency; institutional ethics committee approval and written consent for release of information is therefore not required. Results Patient Characteristics and Administrative Model The patient study samples used to create the derivation and validation cohorts for the administrative claims models are summarized in Table S1 and Figure 1. The 2000 MEDPAR dataset included 683,280 pneumonia discharges from 4,684 hospitals. The model derivation cohort consisted of approximately half of the eligible patients discharged in 2000 resulting in 224,608 cases randomly selected from 4,644 hospitals. The mean age of the cohort was 80.268 years. The cohort included 55.8% women and 12.0% nonwhite patients. The overall unadjusted 30-day mortality rate for the derivation cohort was 15.1%. The median annual number of Medicare pneumonia hospitalizations from the deriva- tion sample hospitals was 37, ranging from 1 to 396, and across these hospitals (25th and 75th percentiles, 18 and 67, respectively). On the basis of clinical review of candidate variables, bivariate analysis, and the stepwise GLM procedure, we identified 31 variables (2 demographic and 29 clinical) for the final model. The estimated parameters from the HGLM are summarized in Table S2. No variable has a large standard error (SE) (all ,0.06), and no variable has a variance inflation factor (VIF) .1.30 (VIF customarily should not exceed 10), indicating that the model does not have a collinearity problem. Model discrimination, calibration, and fit are summarized in Table S3. The p value of the goodness- of-fit for the GLM was ,0.001 but the overfitting indices suggest that there is not an issue with overfitting. The area under the ROC curve was 0.72. The patient-level observed mortality rate ranges from 2.7% in the lowest predicted decile to 35.6% in the highest predicted decile, a range of 32.9%. The adjusted R2 was 0.13. Distribution of the standardized 30-day mortality rate is shown in Figure 2. Using the complete 2000 dataset, the observed mortality rate ranged from 0% to 100% across the 4,684 hospitals. The 25th, PLoS ONE | www.plosone.org 4 April 2011 | Volume 6 | Issue 4 | e17401