Claims Model for Mortality Figure 1. Development of the derivation and validation cohorts for the pneumonia mortality models. The administrative derivation cohort included a stratified random sample of half of the eligible pneumonia discharges for calendar year 2000. Subsequent administrative validation cohorts included the other half of eligible discharges from 2000, and all eligible discharges for calendar years 1998, 1999, and 2001–2003. The chart model derivation cohort included 50,858 eligible patients with abstracted medical record data. An additional administrative validation cohort using the claims of patients in the chart model derivation cohort was created to compare state-specific standardized mortality rates generated by the 2 models. doi:10.1371/journal.pone.0017401.g001 assemble clinically coherent codes into candidate variables [15]. This system, which includes 189 categories, was developed by physician and statistical consultants under a contract to CMS and is publicly available. The HCC candidate variables considered for this model were derived from the secondary diagnosis and procedure codes from the index hospitalization and from the principal and secondary diagnosis codes from hospitalizations, institutional outpatient visits, and physician encounters within the 12 months before the index hospitalization. We combined categories of HCC variables based on clinical judgment and bivariate associations, and eliminated candidate variables with a ,1% frequency. Additional candidate variables included demo- graphic (age, sex) and procedural factors (history of bypass surgery or percutaneous coronary intervention in the past year). A total of 138 HCC variables met the above criteria and were included in the initial model. The final number of variables in the model included 2 demographic variables (age and sex), 21 comorbidity variables, and 8 pneumonia-specific variables. Model Development. We estimated hierarchical generalized linear models (HGLMs) [16–18]. We modeled the log-odds of mortality within 30-days of admission as a function of patient demographic and clinical characteristics, and a random hospital- specific effect. This strategy accounts for within-hospital correlation of the observed outcomes, and reflects the assumption that underlying differences in quality among the hospitals being evaluated lead to systematic differences in outcomes. We selected the covariates for the final claims model using a backwards elimination procedure through the generalized linear model (GLM) with a logit link function approach. Because of the large number of patient observations, we chose an exit criterion for a variable with a p.0.01. For each model, we calculated several indices for assessing model performance [19] at the patient level: the area under the Receiver Operating Characteristic (ROC) curve, explained variation as measured by the generalized R2 statistic, over-fitting indices (intercepts and slopes), and the observed outcomes in strata defined by the lowest and highest deciles based on predictive probabilities. Models with ROC areas close to 1, high R2, over-fitting intercepts near 0 and intercepts close to 1, and a large range of observed mortality between the 2 deciles are desirable [19]. We further assessed model fit through examination of Pearson residuals. After identifying covariates for the model, we re-estimated the regression coefficients of the covariates identified from our backward elimination strategy using HGLM. Model Validation Medicare Claims Data. We validated the model by comparing its performance in the derivation set with various validation cohorts, as described above. In each validation dataset we recalibrated the model, so that we used the same variables but fit the model to the data for each specific cohort. Medical Record Data. We compared the results from the claims model with those from a model based on information abstracted from the medical record. Risk factors identified by the Pneumonia Patient Outcomes Research Team (PORT) were used as independent variables in the medical record model [20]. Because some covariates could be missing for patients based on chart review, we added a category for missing values where applicable. For each abstracted variable with missing values, we PLoS ONE | www.plosone.org 3 April 2011 | Volume 6 | Issue 4 | e17401