Don’t Blame the Messenger (2015)

Clinical decision support (CDS) systems that can provide real-time feedback to providers in the form of alerts or messages have the potential to improve quality of care, particularly in a complex and dynamic setting such as the operating room. Such systems have been developed previously and demonstrated to improve documentation accuracy, billing, and quality metrics.1–7 However, the most exciting prospects of such real-time systems lie in their potential ability to improve patient safety, particularly to detect ongoing or impending medically unsafe scenarios and notify the providers in real time to take corrective actions. Although some previous work8,9 has been done, this prospect remains largely unexplored. In this context, the recent study by Panjasawatwong et al.10 presents interesting results worthy of discussion and critique. Their study used a home-grown CDS system to alert anesthesia providers of hypotension (systolic blood pressure [SBP] <80 mm Hg) through a blinking button on the anesthesia information management system (AIMS) screen as well as through text pages to providers involved in the case. Their main finding was that the alerts did not reduce the duration of hypotension. The median time for SBP to return to ≥80 mm Hg was 1 minute with and without alerts, indicating that in the majority of cases, the providers observed the hypotensive episodes and took corrective actions without the alerts or the hypotension resolved on its own. This editorial contains additional comments on their study along with considerations for developing and testing AIMS-based CDS systems. CONSIDERATIONS FOR DEVELOPING AIMS-BASED CDS Traditionally, in an operating room, the patient monitor is the primary device for hemodynamic monitoring of the patient, whereas the AIMS serves a secondary role of a documentation system. Alarms and alerts are built into the patient monitor to indicate out-of-range variables because it is the primary device for patient care. However, with the advent of CDS systems in the operating rooms, the traditional demarcation between a patient monitor being used for patient care and an AIMS being used for clinical documentation is becoming fuzzy. The advantage of an AIMS is that it acquires and links data from multiple sources to create a repository that is information-rich. The real power of an AIMS-based CDS system is in its ability to process and combine this multisourced information to make alerts meaningful and relevant. Thus, the system is bringing to the attention of an anesthesia provider an unsafe or undesirable clinical scenario rather than just an out-of-range variable. In relation to blood pressure management, hypotension with a concurrent high concentration of inhalation agents and hypertension with concurrent infusion of a high dose of phenylephrine are examples of unsafe scenarios that can be detected by an AIMS-based CDS systems.9 In a busy operating room, an unsafe scenario spread across multiple systems or devices is more likely to be overlooked than a variable on a patient monitor being high or low. The hypotension alert generated by Panjasawatwong et al.10 was based on a single variable. In many ways, they duplicated the low blood pressure alarm feature built into the patient monitors. Frequently, the low blood pressure alarm on the patient monitor is disabled by the provider because it does not provide significant added utility as was the case in the study by Panjasawatwong et al. (personal communication with D. I. Sessler, MD). The anesthesia provider is continually monitoring the patient and can easily observe the blood pressure drop on the monitor. Similar to the patient monitor alarm, the providers may have perceived limited benefit for the alerts generated by Panjasawatwong et al., thereby tending to ignore them. Data latency is the biggest nemesis to real-time decision support.11 Data latency in an AIMS/CDS system could be because of several reasons: delayed refresh of the database in certain AIMS configurations, data processing time for an AIMS and CDS system, retrospective manual entry of data, and inherent delays in data interfaces. Delayed availability of data for a CDS system means delayed detection of issues. This has 2 significant undesirable ramifications: (1) delayed notification potentially leading to a prolonged occurrence of an issue; and (2) notification that occurs after an issue has been observed and resolved by the provider. As for provider acceptance of alerts, particularly bothersome is the second ramification. If after the fact, false alerts are generated a majority of the time, they will become a source of annoyance and distraction for the providers. These “cry wolf” alerts are bound to be ignored by the providers. In the study by Panjasawatwong et al., results indicate that hypotension was resolved within 3 minutes (upper quartile value of hypotension recovery time) a majority of the time. With a processing time of 2 minutes for generating their alerts and possibly additional time needed for a page message to reach the provider, it is likely that the notification messages were received after the hypotension events had already been recognized and resolved in most of the cases. A high incidence of false alarms may have led providers to ignore them, thus significantly reducing their intended impact. The only way to avoid the ill effects of data latency is to make the entire system, including AIMS and CDS systems, as real time as possible and to generate alerts at the first instance when information is available. Practically, this is a challenge when an AIMS is interfaced with external devices and systems and when it is not the primary source of much of the data. Under this situation, to reduce false notifications, one can only resort to detect and notify issues that are persistent, possibly after giving the providers a reasonable time period to self-correct the issue. CONSIDERATIONS FOR EVALUATING AIMS-BASED CDS Panjasawatwong et al. should be commended for using an AIMS to conduct a randomized controlled trial in an efficient and cost-effective manner. Although the benefits of a randomized control trial to evaluate the effectiveness of CDS system alerts are undisputed, it is equally important to properly define meaningful metrics for evaluation. The primary objective of a real-time alerting system is to elicit the desired response from the provider in real time. Whether this response translates to a desired clinical outcome is secondary from the perspective of the alert system. Rather than focusing primarily on the clinical outcome, time to recover from hypotension, it would have been more interesting if the authors explored provider response to hypotension with and without alerts. Specifically, whether the providers acknowledged the alert messages and performed steps to treat hypotension (administration of vasopressors, reduction of inhaled agent concentration, etc.) could have been analyzed to determine the effectiveness of the alerts. There was a clear change in behavior because of the alerts. Hypotension was treated with a vasopressor within 10 minutes, 54% of the time in cases using the alert system and 48% of the time without the alert system (P = 0.0027) (personal communication with E. J. Mascha, PhD). Thus, the CDS system in this study significantly impacted clinician behavior, meeting its primary objective. The next issue is whether positive impacts on clinician behavior translate to improved clinical outcomes. In the study by Panjasawatwong et al., the improvement in treatment response rates did not measurably improve recovery from hypotension. In a similar study, Nair et al.12 evaluated the impact of a CDS system on intraoperative blood glucose management. Analogous to the results by Panjasawatwong et al. for treatment of hypotension, the study by Nair et al. showed improved provider adherence to blood glucose measurement and treatment. However, the mean glucose level and other glycemic management variables did not show significant improvement with the CDS system alerts. In the case of hypotension, the clinical issue regularly self-corrected, canceling the impact of improved clinician behavior. In the case of glycemic management, the medical center’s glucose management recommendations did not change blood glucose levels fast enough to demonstrate benefit in the intraoperative period. When comparing clinical outcomes to assess effectiveness of real-time alerts, it is important to exclude scenarios that the alert system is unable to address. For example, in the alerts used by Panjasawatwong et al., system latencies meant that it took >2 minutes for the message to reach the anesthesia provider. The alerts simply could not have helped in reducing hypotension recovery periods <2 minutes. Hence, it would be more appropriate to compare the effects of alerts on prolonged (greater than the data latency of the system) periods of hypotension. Including hypotension recovery times <2 minutes in the control and intervention groups would “wash out” or diminish the ability to detect the changes because of alerts in subsequent analyses, particularly when a significant number of hypotension recovery times were <2 minutes. Another aspect to explore when evaluating CDS system-generated real-time notifications is the possibility of alert fatigue. Generating a large number of frequent notifications can cause alert fatigue resulting in provider indifference. Data on how many alerts were generated during a case and how many were required to elicit provider response should be analyzed to determine the efficacy of alerts. In addition, soliciting provider feedback on whether alerts were useful or distracting would be useful in understanding the reasons for success or failure of certain alerts. Last, in instances where CDS system alerts are targeting clinical issues that can recur multiple times during a case, the effect of the alerts on all the episodes (or at least the episode with the maximum duration) should be evaluated to fully test the effectiveness of CDS system alerts. In the study by Panjasawatwong et al., the primary outcome data were the time for SBP to return to >80 mm Hg after the first episode of hypotension. Patients may have experienced subsequent, more prolonged periods of hypotension after the initial episode and the effect of alerts on these episodes are unknown. FUTURE CONSIDERATIONS FOR CDS With a growing number of institutions either using or installing an AIMS, it is time to tap their potential for enhancing patient care rather than simply using them for documentation purposes. A perfect example would be building and integrating CDS features in an AIMS. CDS system-mediated real-time alerts and guidance hold significant promise in improving quality of care and patient safety. However, such systems should be well designed and implemented to elicit the most optimal response and behavior from the anesthesia providers. The real advantage of an AIMS-based CDS system is the ability to combine multiple pieces of information from disparate sources to provide meaningful alerts. Not only should the decision support logic be thoughtfully constructed, but also the method, timing, and presentation of notification messages be carefully strategized. Adequate evaluation of CDS decision rules to demonstrate their utility should be performed. Evaluation should focus on the provider compliance to the desired response as the primary outcome, and this should be differentiated from the clinical outcome. Observing and measuring provider behavior and soliciting feedback are not only important to gauge success or failure of decision rules, but also to refine and improve the decision support strategy and subsequent improvements in management guidelines. DISCLOSURES Name: Bala G. Nair, PhD. Contribution: This author helped write this manuscript. Attestation: Bala G. Nair approved this manuscript. Name: Howard A. Schwid, MD. Contribution: This author helped write this manuscript. Attestation: Howard A. Schwid approved this manuscript. This manuscript was handled by: Maxime Cannesson, MD.