When computers do the scoring, humans have to define the rules.

www.ccmjournal. org 335 Scores to assess the severity of illness of critically ill patients are used for multiple purposes. Clinical outcome studies in critically ill patients use the scores to assess whether patient groups are well matched with regards to their severity of illness. Because there is a clear relationship between severity of illness and mortality, most severity scoring systems use mathematical formulas to transform a certain score into a risk of mortality. This mortality prediction does not represent the actual risk of dying of individual patients, but rather the expected risk in a similar population with a similar score. The standardized mortality ratio can be calculated from the ratio between observed and predicted mortality, and in this way severity scores can be used to compare the performance of ICUs. The Acute Physiology and Chronic Health Evaluation (1) and Simplified Acute Physiology Score (SAPS) (2) are probably the most frequently used scores. Over the years, they have undergone several adaptations, and the latest generations are Acute Physiology and Chronic Health Evaluation IV (3) and SAPS 3 (4, 5). When used in practice, severity of illness scores have two major problems. First, scoring is not as straightforward as it should be. Some of the rules behind the scoring system are complex, and can be subject to interpretation. Querying the most aberrant physiological parameter at the time of ICU admission depends on the diligence of the person scoring, and the quality of the patient’s chart. The literature on consistency of scoring is relatively scarce (6–8), but suggests that there is considerable interand intrarater variability among nontrained persons filling in the score, which could be circumvented by using qualified trained personnel for scoring. Second, even for trained persons, severity of illness scoring is time consuming, as it often involves detective work in the patient’s medical files, which are often poorly structured or incomplete. SAPS 3 is an internationally developed and well-validated score, and is relatively sparse with regards to the number of used variables. Because it is nonproprietary, investigators are free to use it. SAPS 3 consists of three major elements: patient characteristics before ICU admission; the circumstances of ICU admission; and the degree of physiological derangement within 1 hr before or after ICU admission. In this issue of Critical Care Medicine, Liu and colleagues (9) have created an automated SAPS 3–based ICU severity of illness score, the eSAPS3. They have used a very large American multicenter database from their own hospital network, Kaiser Permanente Northern California (KPNC). The database used comes from a commercially developed general hospital (non-ICU specific) electronic medical record (EMR). A separate development and validation cohort was used to assess discrimination and calibration of the score with regards to hospital mortality predictions. Overall, the eSAPS3 had very good calibration and discrimination, although it tended to over predict hospital mortality. The study has several strengths. First, the score was developed using a large multicenter database of almost 70,000 ICU admissions at 21 hospitals, albeit from the same hospital network. Second, attention was given to erroneous or contradictory entries into their EMR, and computerized algorithms were used, combined with manual revisions in order to clean up the files. Third, although the score was SAPS 3 based, it was not exactly the same as a “traditional” human-scored SAPS 3. Computerized scoring from an EMR requires specific rules with regards to the assumptions that have to be made to convert elements from the chart into a score. This was the case especially in the “reasons for ICU admission” category. Rightfully, they have developed and recalibrated a new eSAPS3 score, and have stressed the distinction from the original SAPS 3. Nevertheless, the study also has some weaknesses. First, the performance of the score was not compared with the original SAPS 3 score in the same population. Second, because the eSAPS3 was developed and validated exclusively in KPNC hospitals, the performance of the score can only be guaranteed within this population. Third, it has not yet been assessed if the score can be calculated from any EMR different from the KPNC EMR, or which adaptations would be necessary to do so. Fourth, some of the data cleaning algorithms used are a little “fishy,” although the number of data that had to be cleaned was relatively low and in the range where they probably only had a minimal effect on the score. The observed increase in the number of ICUs with an EMR in the past decade (10) has not stopped us from scoring manually When Computers Do the Scoring, Humans Have to Define the Rules*