Quality of Benchmarks for Assessment of Care Will Influence Outcome

In 1986, Congress passed a law mandating that quality of surgical care in VA hospitals be determined by comparing outcomes to “national averages.”1 Unfortunately, those national averages were unknown and have remained so until now with the increasingly widespread adoption of the American College of Surgeons’ (ACS) National Surgical Quality Improvement Program (NSQIP). The ACS NSQIP currently has over 140 participating hospitals with another 50+ hospitals in the line for adoption and inclusion in the program. The paper by El-Tamer and associates in this issue points out the paucity of data regarding outcomes for even common operations, and is one of the first reports to make use of data being generated by the ACS program. VA healthcare responded to that 1986 mandate by establishing a novel technique to assess patient risk and to measure outcome. The initial effort was called the “VA Surgical Risk Study” and was led by a group comprising three surgeons (J. Bradley Aust, Paul Ebert, and John Mannick), two health service researchers, and two statisticians. This effort culminated in the development of the VA's NSQIP. NSQIP has become the gold standard for measuring risk-adjusted patient outcomes after surgery; and by doing so, it is the only national database that can offer risk-adjusted statistics for evaluation by the sponsoring hospitals. In the NSQIP, hospitals employ a full-time (or occasionally more than one) nurse abstractor who reviews charts on a sample of surgical patients. Most major cases are reviewed. Certain specialties such as ophthalmology are excluded because of the very low morbidity and mortality rates. Cardiac surgery is also not included because a VA system already existed to measure the outcome of these procedures. Currently, some 50 preoperative variables are collected, including demographics, risk factors, and laboratory data. Intraoperative variables are also collected. Morbidity and mortality data are then reviewed for the 30 days following surgery. Hospital performance is measured by the ratio of observed events to expected events (O/E ratio). Using the system as a whole, the “expected” morbidity or mortality for a given hospital's case mix is calculated. This takes into account the preexisting comorbidities of a given patient and the complexity of the operation performed. The O/E ratio of 1 indicates outcomes that are the same as those predicted by the model, ie, average. Hospitals with a significantly low O/E ratio can be singled out to look for areas of exceptional practice, while those with high O/E ratios can be analyzed for areas that need improvement. Using this process, the VA system as a whole has observed a gratifying reduction in mortality that has continued over the years that NSQIP has been in existence. Interestingly, postoperative morbidity appears to have reached a plateau after an initial reduction.1 While it is impossible to say that NSQIP is responsible for the observed reduction in mortality in VA hospitals, it seems inescapable that constant scrutiny of the worst performers cannot help but improve performance as a whole. The NSQIP has probably saved thousands of lives. A beneficial, but not wholly unanticipated, side effect of NSQIP has been its value in research. A host of papers have been presented based on NSQIP data. With well over a million surgical procedures now in the database, its power in this regard is unmatched. Data from the NSQIP will influence surgical research for at least the next generation of surgeons. Recently, the ACS has used data from NSQIP for intraoperative time and postoperative length of stay to assist in developing physician work values for general surgery Current Procedural Terminology codes for the American Medical Association Relative Value Update Committee. This objective source of data was influential in increasing physician work values to more appropriate levels. The data presented by El-Tamer et al represent the prospective multi-institution collection of 30-day mortality and morbidity from 14 universities and 4 community centers. The negative findings of the study, however, are relatively predictable. That is, no one should be surprised that mortality and morbidity associated with these 2 classes of operations are very low and that the larger the operation the more the complications. Basically, the article found that the mortality and morbidity after breast surgery is low and limits the use of morbidity and mortality as an assessment of quality of care. However, this is a landmark paper in that it represents the first use of NSQIP methodology in the private sector even though the results are a bit underwhelming. The NSQIP was designed to track major life-threatening morbidity, such as a myocardial infarction, and death. While such measures are of unquestioned significance, we may need other measures in breast cancer than those standardly used for abdominal, vascular, or other operations. A total quality assessment for breast cancer care might measure not only standard items such as local control and survival but also timeliness of diagnosis and care, preoperative versus intraoperative diagnosis, frequency of breast preservation when all other factors are controlled, and consultation for and timeliness of adjuvant therapy. Even so, these measures may not be equivalent in the research setting where investigators are trying to determine who does not need adjuvant radiation or chemotherapy. Such worthy attempts to spare a patient unnecessary therapy may fall out in a NSQIP system as inadequate care.2 At a time in breast cancer when many women are overtreated and others are undertreated, it is a particular disease that will be difficult to sort out. The NSQIP tells part of the story. But it is probably too crude a tool to measure quality outcome differences, especially for complex diseases with low morbidity surgeries, such as breast, and will certainly require disease-specific modifications. The real importance of this paper is that it comes at a time when the Center for Medicare and Medicaid Services and many commercial payers are beginning to move toward outcomes-based reimbursement, also called value-based purchasing. The accurate measurement of outcomes and risk-adjusted complications will be essential to provide fair reimbursement of surgeons under this new paradigm of payment we all are facing. The ACS through its efforts at developing valid quality measures for surgeons is a vital part of that effort along with NSQIP. With its ability to take into account the comorbidity differences between various groups, regions, or practice settings and compare those results on a national basis, NSQIP seems to represent the very tool with which outcomes can be fairly monitored and compared. As payers of healthcare costs seek to define quality, it is essential that such risk stratification strategies play a major role. Unadjusted morbidity and mortality figures, such as can be gleaned from administrative databases, are essentially worthless in evaluating performance. Reliance on these data will lead to avoidance of complex care and other “gaming” of the system. An unfortunate down-side to the accurate data collection in NSQIP is the expense. A dedicated nurse abstractor is of varying burden to different size systems. While these costs have generally been borne by the hospitals, it seems logical that payers, who have an economic interest at measuring and ensuring quality, would promote adoption of NSQIP, even to the point of paying its costs. These costs would likely be more than offset by the increased information made available to a hospital or payer to reduce complications and therefore costs.