1/17/2012  Discrete Event Simulation for Process Improvement: Emergency Department Flow VIReC Clinical Informatics Seminar Day, Eugene

----------

Please stand by for realtime captions. 

>> Welcome, everybody.  This session is part of the VA information resource center ongoing clinical informatics server seminar series.  The series aims are to provide information about research and quality improvement application in clinical informatics and also information about approaches for evaluating clinical informatics applications.  Thank you to CIDER for providing technical and promotional support for the series.  Questions will be monitored during the talk in the Q&A portion of GoToWebinar and VIReC will present them to the speaker at the end of his talk.  I brief evaluation questionnaire will appear when you close you're GoToWebinar today.  We would appreciate if you would take a few moments to complete it.  Please let us know if there is a specific topic area or suggested speaker that you would like to -- for us to consider for future sessions.  At this time I would like to introduce our speaker, Eugene Day, Dr. Of science.  Dr. Day is a health system Specialist and HSR&D at St. Louis VA Medical Center.  He's also adjunct assistant professor of emergency medicine at the warren albert medical school of Brown University.  Without further adeu, may I present Dr. Day. 

>> Good morning, everyone. Thank you for attending my VIReC lecture.  I appreciate all of the work that has been done by Heidi and Margaret to get us prepared.  What I'm going to be talking about today is discrete event simulation for process improvement and specifically how it applies to emergency department flow. 

>>> After a brief introduction I'm going to take a couple of steps back and talk about systems engineering in medicine, what is the systems engineering? And how do we look at medical systems as dynamic systems? What is complexity? I will talk about how we model dynamic systems using discrete event simulations and I will briefly touch on a couple of other ways that these things are approached. 

>> Very briefly.  And finally, then I will talk about the simulation of the St. Louis VA emergency department.  Modeling current practice and generating predictions and using those to develop evidence-based implementation of interventions, talk about the results that we did, possible limitations and some acknowledgments. 

>>> Sadly I have no conflict-of-interest to report.  Funding for this project first of all, the project itself was for quality improvement and hence was not funded research.  However, post-hoc analysis was done as research and was declared exempt by the St. Louis VA institutional review board.  Software licenses, computers, etc., were provided by the 2009 VHA innovations award, Greenfield integration number 123 for which I am the innovator. 

>>> So by way of introduction there are a lot of challenges in medicine right now.  First of all, how do we treat all of our patients? We have an enormous country and we have an enormous number, population of veterans that all deserve fine treatment and the question is what the resources that we have, how do we treat them? How do we decide where best to dedicate resources, what performance measures are appropriate for determining how well care is administered?  And systems engineering in medicine provides a framework for a systematic approach to those answers.  Of those questions.  As well as answering questions of a what if an how would type.  What if we rearranged a clinic? What if we hired a new Dr. for the emergency department or how will things change if we implemented a new policy? Systems engineering can also provide us with a different paradigm for hypothesis testing, essentially saying we believe that if we were to hire a new physician for this emergency department, the following would take place.  Then we can test that in simulation. 

>>> Then specifically for this project, how can we reduce the number of long stays at the St. Louis VA emergency department? That was a fundamental question that we were asked. 

>>> So the driving questions for this project are can we reduce the average length of stay in the emergency department and also specifically, can we reduce the proportion of patients with very long stays? By providing provider assessment and disposition from triage.  And there's a decent amount of literature that in fact does suggest so.  That the addition of mid-level providers to triage and discharged from triage and the enhanced use of provider-based fast-track will in fact improve length of stay in the long stays in the emergency department. 

>>> Then can we assess the likelihood of success prior to experimentation? That's one of the critical strengths of simulation is that we can test interventions in a cost free environment and also a risk-free environment anytime we make changes to a medical system we have some, ideally very small, but measurable risk to patients. So we would like to have a way to determine ahead of time which interventions were likely to be effective so that we don't experiment in risky situations unnecessarily. So discrete event simulation provides a platform for doing that. It is been employed in emergency departments before.  It is useful for prediction of patient throughput times comparison of triage practices and economic evaluation and this is of course, just a very small sampling of the sorts of things that discrete event simulation is used for in the emergency department and beyond.  And a PubMed search or Google search will produce hundreds of papers in a number of different fields. 

>>> So what is systems engineering in medicine? Most of the time when you think about- when people think- about what is engineering as it applies to medicine they think of biomedical engineering such as anesthesia pumps or robotic limbs, that sort of thing.  Now systems engineers and biomedical engineers will frequently worked closely together. In fact managing the blood-pressure of a patient on an anesthesia pump is one of the canonical examples of that's used to train the systems engineers. And being able to use robotic systems engineering and robotics are very closely related.  But what I'm going to be talking about today is the simulation aspect of systems engineering and medicine which while it bears a bit of relationship with these in a conceptual way is not the same thing.  What is simulation in medicine?

>> Unfortunately, simulation in medicine has a number of different things that you can refer to and most physicians and administrators will frequently think of simulation in medicine as mannequin simulators.  Sometimes even real patient interviews where you hire actors to function as patients is called simulation as well.  Now systems engineering in fact has a lot to inform mannequin simulators as well. They are a large scale complex system of the sort that I'm about to describe, but what I'm talking about is in silico representation of clinical systems. 

>>> So to back all the way up, what's a system? Fundamentally, a system is simply a collection of objects and relationships between those objects. And the systems engineering is an interdisciplinary approach and means to enable the realization of successful systems so when we have, for example, a clinical system as in an emergency department the objects our patients, exam rooms, computers, physicians, nurses and the relationships are how patients consume resources, that is the facilities available to treat patients at the locations where they are treated.  Then proceed on.  So systems engineering in medicine is the study and design of these large-scale systems and of course, also the improvement of existing large-scale systems. 

>>> Additionally, systems engineering in medicine has some challenges that systems engineering in other fields may not have to face.  Such as the optimization of systems with human actors.  How do decision-makers make decisions? There's an entire field of theory about this called semantic control theory which is essentially modeling systems where humans and computers have to interact in real time.  Some of the first work that was done in that field was the piloting of a damaged fighter aircraft.  Where the pilot has to be informed of potential decisions along the way.  So being able to model how do human decision-makers make decisions is extraordinarily difficult and in many cases I would argue is essentially impossible, but approximations to that can be useful in extending these systems.  The models of the systems. 

>>> We also have to analyze competing objectives in medicine cost and quality are always going to be competing very frequently speed and safety will also be competing objectives.  And our goal is to find an optimal balance between these competing objectives between the these goals in order to provide the best care with resources that our available to us.  So we also in the process of prosecuting these requirements we examine what are the constraints and the bottlenecks and barriers to care which are frequently associated with capacity, but are also often associated with policies and regulations.  The regulations which our designed to improve safety or improve quality will necessarily sometimes increase cost or speed and vice versa. 

>>> So what does it look like when we model medical systems as dynamic systems? A dynamic system is simply a system in which time is an element.  The variables change as time progresses.  So here's a mockup of a basic emergency department model and the software I'm using here is any logic 6.6 and this is an example  of a systems dynamics model.  I use this now even though the model I'm going to talk about of the emergency department is a discrete events simulation model.  I used the system dynamic model for this part because it is very visual and it is useful, easy to describe.  So here we see that the number of emergency patients in the emergency department at any given time is essentially a function of the two flows, the arriving flow and the exiting flow.  In each of those flows is impacted by a number of different things, time of day, time of year, the epidemiology of the population, patient demographics, in terms of the arriving patients.  And also and terms of treatment, how many physicians, nurses, technicians, what consult services you have available to you, exam room capacity, laboratory turnaround, radiology turnaround, that sort of thing that will influence how rapidly you can treat your patients and therefore, how the queuing system functions.  And at first as a large-scale conceptual model this might look like it is a reasonable basic emergency department model.  But as we think about it a little bit more we realize there's actually a lot of things that interact with the emergency room that require us to consider maybe a larger scale system.  So radiology and laboratory turnaround is going to be affected by demand from other clinics for those services.  The outflow is going to depend on the floor bed availability and the transfer speed which is going to depend on other hospitals. 

>>> Consult services are going to depend on demand from other clinics and the time of day and time of year is going to affect radiology, laboratory, all of these things interact in a way which is much more complicated than is necessarily apparent as you first look at an emergency room. 

>>> And of course, it ends up getting a very, very large as we start to look at all of the things which might influence the way an emergency department functions.  So what we have are these large-scale dynamic systems.  There are multiple dependencies, any given value is generally dependent on several other values.  There are very complex interactions between them, like I said dynamic behavior meaning the systems change as time goes on.  They are generally subject to what we called the butterfly effect which anyone who's read Jurassic Park is familiar with, but essentially it means that a very small change anywhere in the system may result in a very large change elsewhere in the system later as time goes on that may not necessarily be obvious.  There tend to be feedback loops so that as demand from other clinics changes it may affect the laboratory turnaround time because there's greater demand there which may affect the treatment which may affect the ability to get patients to the floor which may affect a number of other things leading back to increasing time in the laboratory. 

>>> Then there are also these systems tend to be adaptive and that the easiest way to think of an adaptive systems with a simple predator -- model. If you have a large number of predators suddenly influx into a system they can eat up all of the prey which then causes some of them to either starve to death or have to move to some other area which will allow the prey population to rebound.  We see those sort of automatic adaptation available, take place in these systems.  The systems are often memory dependent.  Memory dependence is a technical term systems engineering and other kinds of engineering as well, but frequently with these large human interactive dynamics systems they can be literally memory dependent.  in the sense that suppose we have a hospital which is not functioning adequately and we go and invest $100 million into it and bring it up to absolute state-of-the-art.  It still may not receive the number of patients arriving that it is capable of processing because people remember that hospital as being bad and therefore, don't want to go to it.  So with human interactive systems we often see types of memory dependence that are beyond the strict engineering concept of memory dependence. 

>>> The State variables are often non linear which means that as you increase the, for example, number of lab samples in a laboratory that need to be processed, the time to process each lab sample may not vary according to a linear -- in a linear fashion with a number of labs samples needed.  As I hope I've been showing from the last three slides, you often have indistinct boundaries to these medical systems and to any complex system that it it's not necessarily clear where the system begins and ends.  And careful decisions have to be made about where you are going to stop the model so that you have a model of a manageable size and can nevertheless get useful information out of it.  Dr. Box who is one of the founders of systems theory is famous for saying that all models are wrong, but some are useful. 

>>> So what is this complexity? I've already described a little bit, but to hit the salient points, these are large systems which have multiple semi- independent subsystems.  For example, the laboratory is its own system and the ED is its own system, but they interact in some ways, but they don't interact in other ways.  But changes to either one can influence the other.  There's the adaptivity, emergent behavior, emergent behavior just means a behavior that is apparent at a large-scale which is not necessarily the obvious consequence of behaviors taken by individual actors within the system.  An easy way to think about that is a flock of birds that all seems to move at the same time despite the fact that they don't seem to communicate in any obvious way that they are intending to move.  Or to change direction. 

>>> Propagating consequences between these subsystems, this is the butterfly effect again that changed somewhere may have an enormous effect elsewhere.  And these systems will frequently exhibit what's called a semi- periodicity which means that the system appears to follow a sort of a sine wave, for example, an emergency department will generally reliably be crowded during the day and less crowded at night.  But it never actually mimics itself.  You cannot say that every day at noon we will have 23 patients.  It is semi- periodic in nature. 

>>> So how do we turn a dynamic system into a model? It is a multistage process.  It generally involves using both direct observation of the system and archival data.  There's multiple methods, the systems dynamics which I've shown you sort of the pictures of before, but which is a large-scale continuous technique usually continuous.  It models sort of stocks and flows as large-scale systems change over time, systems dynamics is really good for modeling economies and climates.  Agent-based modeling which is also used for large-scale models and is generally really, really valuable when you have indivisible quanta that you need to model.  For example, an agent-based model is really good at modeling the flock of birds where an agent is an individual Bird and then you have a large cohort of agents which interact with one another and their environment.  Finally discrete event simulation which is what I'm going to be talking about for the rest of the time here which is useful when you have individuals which need to negotiate a set flow chart of tasks in order to complete the process.  Of course, their stochastic modeling, other types of modeling for dynamic systems which are really good for queuing theory, Markov change and that kind of thing. 

>>> What is discrete event simulation? First of all, it is stochastic which means that some of these State variables are random generally in semi- predictable ways.  We have stochastic processes which will govern things like the arrival rate of patients.  Just like in the real world in the  discrete summation we can say we don't know exactly when the next patient is going to arrive, but we know we will it will happen according to an exponentionally distributed random variable with a mean of such and such.  

They are dynamic which means that the time evolution of the system is important.  We are not just looking at how the system behaves on average the way, for example, queuing theory straightforward queuing theory will do.  But we are looking at, we are trying to mimic the actual hour by hour behavior of the system. And these systems are called discrete event because significant changes occur at discrete time instances.  We say that, for example, the patient goes and sees a Dr. At a particular time and they are done at a particular time.  And the next process can occur following that.  In these models, continuous processes are modeled as very, very small discrete changes in a process that happened very close together in time. 

>>> Some of the advantages of discrete events simulation is they provide a real-time visualization of the dynamic system.  This will allow medical personnel, administrative personnel to critique and improve systems Fidelity without requiring a great deal of sophisticated computer science or engineering knowledge.  And it can give us a visual indication of problem areas that allow non-engineers to identify bottlenecks and proposed solutions.  So in the real-world when you have a large number of patients waiting in a room you cannot necessarily determine what they are waiting for.  However, in discrete simulation, it is relatively easy to have the icons representing these patients literally, physically queue for the service they are waiting for so you can see everybody is waiting for the lab to be completed.  It is critical that these systems allow non-engineers to identify these bottlenecks and propose solutions because one of the key ethical concerns in systems engineering and what I'm absolutely dedicated to is -- is that physicians, nurses, medical providers are the ones that our responsible for the care of the patient.  And no one wants less than i do for engineers to be telling physicians how to treat patients.  The goal of this is so the engineers can identify the valuable interventions that are proposed by physicians, test them in silico this cost free, risk-free environment and then allow the medical personnel to put them into practice. 

>>> How do we capture one of the systems? The first process is to decompose the clinic into its constituent elements.  Locations, where things happen.  Exam rooms, waiting rooms, offices.  But also locations can be virtual, they can be a computer hard drive that allows e-mails to stack up or a phone that allows phone calls to queue. Resources, resources are the objects within a system which do the work.  So doctors, nurses, technicians, equipment.  These are the objects that are needed by patients in order to negotiate a visit to the emergency department.  Then entities, these are the objects upon which work is done, patients, paper records, phone calls. 

>>> To analyze these systems once it's been decomposed we need to develop the flow of the system.  Which is we answered the question, how do entities consume resources at locations? Then how do entities proceed to the next location? And how do resources search for tasks? Frequently an entity will have a number of different resources that it may need if the patient is in an exam room they are going to need to see a nurse, they're going to need to see a physician, then when they're finished  at a location that they may have a choice of several dfferent locations to proceed to next so how do they choose which location and then how do they actually locomote to that position.  How do they -- do they need to be accompanied, escorted or can they go by themselves? And some cases they may, an entity may instantaneously vanish from one location and reappear at another like transferring a phone call.  Then how resources search for tasks is also a critical element of this.  An emergency department will function very, very differently if you always treat the longest waiting patient next versus if you always treat the nearest patient next.  Versus if you always treat the most emergent patient next.  Determining how resources search for tasks can be critical. 

>>> We need to develop the locations and staff capacities.  Sometimes a location can handle only one entity at a time, sometimes several, sometimes infinitely many like e-mails on the hard drive.  Some processes are batchable meaning for example, lab work may not be sent to the lab until there are 25 samples waiting or they may go individually.  And finally, how do we develop a queue.  How do entities wait while desired resources are unavailable? And some cases they may wait where they are and other cases they may have to return to a waiting room or we may for graphical purposes like I described before, we may want to have them wait in the way they would not wait in the real world. 

>>> So in order to inform this flowchart we need to acquire a great deal of data. Often using measured data, for example, in order to determine how long a patient spends with a physician in an exam room we often need to have someone physically there to observe a large number of these data points and build a distribution that we can use as a stochastic process.  That often involves inhabiting the system with a stopwatch and a clipboard and taking that data which can be problematic.  Often it is difficult for patients and for resources to be watched in that way.  So we have to take special pains to make sure that our presence observing the system doesn't influence the system.  Or at least where we can minimize our influence in the system.  Then of course, there's archival data that we used that's critical as well.  Patient encounter sheets or monthly status reports which may tell us how many patients were in the ED each day, etc. 

>>> So how do we know when one of these models is good? There's three fundamental types of verification that we do with discrete event simulation.  In the first place the model can be verified by clinical staff and by input-output identification.  So the clinical staff observing the simulation and comparing it to their own experience of the system their in is called face valitity and that essentially allows them to look at the simulation and say yes, this looks like the emergency department that I live in or no, it doesn't look like the emergency department that I live in. That's a critical first step in verification of the model. 

>>> There's then two other aspects of the verification what we call external validation and internal validation.  External validation for example is the comparison of throughput metrics between the real-world system and discrete event simulation. The patient demography and statistics we want to make sure that the simulated patient arrivals look like the real-world patient arrivals in terms of the distribution of their emergency severity index, perhaps they're age and the services they require.  Throughput metrics may be time to see a physician, time from arrival to departure, overall time in clinic, that sort of thing. 

>>> And internal validation is the internal validation of the computer code itself and of the model itself, essentially, there needs to be a code review, this is best if you can get more than one person who speaks the computer programming language so that we can check each other's work.  Then also what I call system stress test. Essentially, a discrete event simulation generally speaking is a large very complex queuing system.  Queuing systems have their own definitions of stability and instability and the definition of instability for a queue is that the length of the queue goes to infinity so that entities are arriving too fast for the system to process.  And any queuing system can be forced into instability by overwhelming it with patient arrivals.  However, those entities which do make it to the front of the queue should still be processed according to the standard rules of the system and the expected behavior that we've tried to code into the system.  So overwhelming the system with entity of rivals is one way to do internal validation.  Making sure that despite the fact that the system cannot handle the number of arrivals, those that can handle it still behaving the way we  expect it to. 

>>> Finally, I'm going to start talking about the simulation of the St. Louis VA emergency department.  We have a medium-sized emergency department we're a level 3 trauma facility with a, 120 inpatient beds.  We see about 20,000 patient visits per year.  We have 14 emergency patient beds of which two are dedicated from mental health emergency.  At any given time in the emergency department we will have two to four physicians and 4-11 RNs varying by time of day and then associated support staff with nurses aides, technicians, registrars, etc. 

>>> So what is the problem that we are looking to address? There's a national performance measure in the Veterans Administration that no more than 10% of patients should have stayed longer than six hours in the emergency department.  At St. Louis VA we were averaging 19.9% and our daily being length of stay was about 247 minutes so we were approaching a little more than four hours.  And we know that long stays are associated with increased morbidity and mortality and eloping or leaving without being seen or leaving against medical advice in the literature.  Long stays and large  large proportions of patients with very long stays are a real problem not just in terms of policy and performance measure, but also and terms of the safety and quality of the care that we are providing to our veterans. 

>>> What did we propose? The chief of emergency medicine, Dr. Alruby, based on the literature and based on his conversations with other VA emergency physicians proposed an intervention that consisted of a change to triage practices.  This involved the consolidation of fast-track into triage, fast-track being sometimes it is called urgent care, I'm sure there are slight differences between a fast-track and an urgent care that a physician could explain better than I can.  But essentially it is for patients who are not as emergent and can be seen by a mid-level provider without being given an emergency department bed, so the goal was to consolidate Fast Track with triage, to place a physician and a mid-level provider at triage.  Then to treat and disposition patients of emergency severity index of four and five, the two lowest levels, directly from triage. And also about 30% of the ESI three.  The emergency severity index goes from one to five and its associated with the number of resources that the patient is expected to require during their emergency department Stay.  Not necessarily with the acute distress of the patient, although ESI one and two are to my understanding almost invariably in extreme acute distress. 

>>> As I have said, I'm not a physician.  How do we address this problem with a simulation? The first case is to simulate current practice according to the described process that I've gone through.  Validate the simulation against the current practices.  And use the data we have to generate a model and then to use a disjoint data set to validate the model.  It is important you don't validate the model against data used to generate it for obvious reasons.  Then we simulate the proposed change to the system.  It is important to note that these systems, this methodology is most effective for single changes within the system.  Multiple simultaneous changes are not recommended.  The reason for that is that because the interactions between the subsystems are so complex, very frequently minor errors in the modeling or oversights because no model is capable of simulating everything that goes on within a large-scale system.  If you make a large number of changes to the model you risk a degradation of predictive fidelity.  But these models are very effective for single changes.  To which I mean an isolated change to an individual process within the system like a change to triage. 

>>> So the model was informed by data, observation and interview with the persons that work in the system, with the actual nurses, physicians who are there.  We validated the simulation against six weeks of real-world data that were taken from the 18th of September to the 30th of October of  2009.  And then we used the simulation to predict a six-week simulated trial period.  Then with the intention of implementing the intervention based on simulated results. 

>>> The pre-intervention flow of the St. Louis VA triage system and the entire emergency department is that patients will arrive and they will have a pre-triage nurse who may direct them from clinic to pharmacy, etc.  ESI 1 and 2 and ambulance patients are sent directly to the ED triage bad or treatment bed, sorry.  Then ESI three, four, and five would be sent to registration to a more sophisticated triage whereby they would be either sent to a treatment bed or to Fast Track, Fast Track patients would be discharged, ED treatment bed patients will either be transferred, admitted or discharged. 

>>> The post-intervention flow is the same in terms of pre-triage.  But after registration those patients that were sent to triage, the more severe patients would be sent and have a physician plus nurse assessment where as the less severe patients will have a mid-level provider and a nurse assessment.  And of course, those providers will consult and may reassign if the patient that was initially classified as an ESI four or five becomes a three or if the patient who was initially classified a three it turns out is not after seeing a physician it turns out is not as severe as previously thought. 

>>> Then many of those patients will be treated from triage only those patients that can safely and comfortably wait in the waiting room rather than needing an ED bed.  Those patients which need an ED bed are directed there. 

>>> Both models have the same process in terms of what happens when a patient is in an ED bed.  So they would be assigned to either a mental health bed or to a medical examination bed and receive consult if they were in mental health and then nurse physician assessment and orders, nurse process which would include maybe labs and radiology.  That will also include other consults as necessary.  A physician evaluation and disposition and a nurse disposition and instructions and finally, disposition.

>> What does the actual simulation look like? Pardon me for a moment as I change screens.  Here we have an ANYlogic 6.6 professional simulation of the St. Louis VA emergency department.  Before I get this started I'm going to show you basically what it is here.  We have the resources that are used in the emergency department, the clerk, triage, pre-triage, ED nurse, ED physician, exam rooms, mental health exam rooms, we have sliders which allow us to change the number of physicians and number of nurses as time goes on.  The variables showing how many patients we've had in the system at any given time.  Triage, pre-triage and then patients will arrive through these doors, be directed to a bed in any one of these areas which are the exam rooms.  They would be treated by the physicians and nurses that our available on the staff at that time.  We have a graphic here which will show us whether it is daytime or nighttime or the weekend. And we have graphs here which tell us what is the current moving average of the number of -- for the length of stay, what is the current moving average of the proportion of patients over six hours in the system and what is the percent of patients who are admitted who are spending over six hours in the system. 

>>> Here we have the flow.  Patients arrive, they go through pre-triage, to the -- they may go straight to the emergency department if they are extremely severe or maybe sent to pharmacy, etc.  They may check in.  They  check in.  They will be sorted into mental health or non-mental health.  They may be sorted to Fast Track after triage.  They will go through the ED bed process and then be dispositioned.  If we look at the entire flow network what we see here is that based on different times of the day and different days of the week the flow has to be modified.  It all generally looks to same because all includes the same ED bed process and same disposition options, however, there may not be at certain times of the day or days of the week or may not be pre-triage they may present directly to triage and at certain times of the day there may not be Fast Track so all patients may be required to go through emergency department.  The simulation I'm showing here as you can tell but the fact there's Fast Track, is the pre-intervention simulation. I'm not going to show both because they look very similar on the screen and to detail the differences I don't think will necessarily be enlightening. 

>>> But what we can see here then is and I hope the animation is coming across, we can see what happens as patients arrive and are treated.  If we want to we can speed up the simulation greatly and we can see these moving averages change as time goes on.  We see the simulation of the pre-intervention period is showing around 20% greater than six hours which is what we expect from the real world.  We know our real-world timing clinic is generally about 247 minutes and the simulation is showing roughly the same.  If we speed this up a great deal we can run more time off the clock and we can see how these things change over time.  And also see they tend to oscillate right around the same amount of time.  I think it these graphs are showing the moving average of the last 100 patients. 

>>> Here we see that it is currently daytime.  This is the hours on the clock.  We can simulate this very, very rapidly.  That's what the simulation looks like and if we slow it down then we can have the clinical staff actually watch the simulation as one patient arrives and negotiates the entire process and we can validate it that way.

>>> And of course, the simulation will spit out all of this data to an Excel file which can then be used to analyze the output of the simulation in a much more detailed way.  Then just looking at these graphs as they go on.

>>> So how did we validate this model? First of all, we did a comparison of the throughput metrics.  I'm going to describe the external validation.  Face validity was done with Dr. Al Rubly and Bradly Black who was an RN in the emergency department.  And internal code review and stress tests were performed by myself.  The external validation was the comparison of throughput metrics so the simulated daily mean the length of stay and this is based on 10 simulation runs of six weeks each.  Matched up very very closely in distribution with the real world and in fact, a matched up much more closely than I would require for validated, to say that the simulation was valid.  Although there is not at the moment a best practices standard for what is considered validation is essentially expert opinion. We are working on changing that now.  

>>Then in the simulation the percent greater than six hours stays was 9.9% and the real-world pre-intervention was nine -- 19.9% and 19.0% and I show you these P-values here and I want to make it clear that we want there to be nonsignificance.  In order to validate we want to show that there is no statistically significant difference between the behavior of the simulation and at the behavior of the real-world system. 

>>> How accurate was our prediction? Once we made the change and based on the change, the predicted change, the implementation was in fact adopted.  The intervention was adopted for 30 day trial.  From October 1 to October 30 of 2010.  It was subsequently put into place full-time.  But compared against a 30 day trial period the simulation predicted a daily mean length of stay of about 200 minutes down from 247.  The actual daily mean length of stay dropped to 210 minutes.  The predicted drop in % greater than six hours stays was 13.1% and the actual drop was 14.3%.  We found no statistically significant difference between the post-intervention simulation, that is our trial simulation, and the trial real-world data.  What we found was we were able using the simulation to make predictions which were accurate in the real-world. 

>>> How good was the intervention? There was a decrease in the daily mean length of stay of 37 minutes in the real world.  Which was very statistically significant.  We had a relative decrease in the greater than six hours stays, the proportion of 28.2%.  Meaning we dropped from over all patients at 19% to 14.3%.  I want to go back one slide in point out that in both of these cases, the simulation did appear to slightly outperformed the real world.  It wasn't a statistically significant outperformance, but to the naked eye it appears like it is an outperformance.  We expect that because the simulation cannot take into account every individual thing that takes place within an emergency department within a system.  There are always going to be occasional things that take place that make the real-world system perform a little bit slower than the simulation is capable of performing.  Often this is due to perfect information dispersal within the simulation.  For example, when an x-ray is back from radiology and ready to be read, the physician always knows that immediately.  And as soon as the physician has time to read it they will go and read it in simulation where as that is not necessarily the case in the real world.  So and critically of course, according to Dr. AlRubly, he has had no indication of negative effects on patient status or statisfaction as a result of the change in triage policy.  

>>What are the limitations? As we said, couple of times now, it is impossible to capture every possible variation in flow.  The simulation is designed to capture normal practices, the things that happen 95% of the time.  There where slightly limited sample sizes.  The real-world patient cohorts for the study were 2194 and  1699 respectively in the test and trial periods.  The decrease in sample size in the trial period should be noted that it wasn't because we had fewer patients arriving per day, but rather that the trial period was 30 days opposed to the test period which was six weeks.  But comparisons were made on a daily basis. 

>>> Some time distributions that we used in assimilation were estimated from ED staff interviews which introduces a possible recall bias however given the accuracy of the prediction, we don't consider that to have been a significant source of error.  Then because as I mentioned before, this was designated as exempt for the post-hoc analysis, we did not have access to protected health information for the research portion of this project.  So we don't know how important factors like patient age or history might be to length of stay.  But again, given the accuracy of prediction it appears that factors other than the emergency severity Index are likely to be less significant than the PHI related factors because we did have access to the emergency severity index of the arriving patients. We were able to predict the outcome accurately based on that information. 

>>> What hope I've shown you today is that discrete event simulation is a useful tool for examining these complex systems.  They must be thoroughly validated.  They rely on assumptions.  And they're best focused on narrow predictions.  But we can approach these very large scale, very complex systems in a systematic way and we can test interventions and hypotheses in simulation in a way that allows us to make systems improvements while minimizing both financial risk and risk to patients. 

>>> Finally I'd like to knowledge the VA innovations program which provided us with computers and software licenses.  Our chief of emergency medicine at the VA Medical Center, Abdul Rahim Al-Roubaie who designed the intervention and worked closely with me on the development of the simulation.  Bradley Black and Tommie Strong who bradly black is a nurse and emergency department and work with me on simulation development and post-hoc analysis and Tommie Strong whose a data analyst to provided me with the data necessary to build the project. 

>>> That concludes my lecture.  I hope I came close to time but I wasn't looking at the clock.  I'd be happy to answer any questions. 

>> Thank you very much.  It was great.  I think people will probably start typing in the questions now.  I have one question for you, sort of a comment.  If you applied an evidence-based intervention new triage with mid-level providers, why do you need to run a simulation afterwards to see if the simulation was accurate? Seems like a lot of extra work to confirm that you improved flow. 

>> That's part of the research aspect.  Once the flow in the emergency department is improved, then yes, from a quality point of view, the project is over.  We've improved the flow.  From research, one of my goals as a VA HSR&D researcher is to publish and disseminate the efficacy of these methodologies so that other people can mimic what we did here.  And validating, not validating but demonstrating that the simulation was accurate and then hopefully publishing how the work was done is a way to encourage others to hopefully achieve the success we've achieved. 

>> Okay, thank you.  Here's another question.  Where did you purchase the software from? And how much was the estimated cost? 

>> ANYlogic professional, I was not involved in the actual purchase, but if I recall, ANYlogic has a number of different price points for simulation software of varying capabilities.  We have the big fancy one.  I think that it was somewhere around $20,000.  Annual support which is not necessary, but which is very useful is another I think $2500.  Once again, I really want to thank the VA innovations program for committing to systems research and the costs associated with the software. ANYlogic is available through ANYlogic North America and if anybody's interested in contact information I'm sure they can get it through Google but I'd be willing to provide it as well. That said, I don't want to necessarily be endorsing that specific simulation software.  There are a number of other simulation software is such as med- model or Flex-sim which are capable of doing the same analysis. 

>> Okay, thank you.  Next question, what are you're recommendations for facilities that are starting a new emergency room department? How can you account for best practices in a start up when no data is available for simulation? 

>> That's a very good question.  Building these models from the ground up for a new emergency department is not something that I have done before.  I think that it could be valuable.  My recommendation off of the top of my head would be to say try to find an emergency department which you believe is similar to the one that you are attempting to build with regard to the staff complement and the type of patient demography that you are going to see.  And a large number of educated guesses are probably going to need to be made in terms of the length of time and the amount of patient in flow.  But one of the things that these simulations are very good for is a sensitivity analysis.  You can test the simulation with a large variance, for example, rate of patient arrival.  So you have an idea of what sort of behavior you could expect you're system to have. I'm not aware of new facilities which have been designed based on discrete event simulation trials ahead of time.  But it strikes me you could do an interesting analysis, but without prior experience I'm afraid I cannot say precisely what best ought to be done. 

>> Here are two more questions from the same person.  I will put them together.  Would this model the applicable at other VA hospitals and additionally, what was the project timeframe from start to finish? 

>> In terms of the applicability to other hospitals, this specific model was detailed to our particular emergency department.  And therefore, may not be directly applicable to other emergency departments.  There's always a trade-off when designing these simulations that the more specific it is the less generalizable it is.  More generalizable models which might be applicable in a large number of settings are definitely useful but the specificity of predictions that you are going to get out of the them is necessarily not going to be quite as detailed. 

>>> In terms of how long did it take? The to the actual simulation development was probably three or four weeks.   That was on a slightly compressed timeframe.  Because of the manner in which we developed it, like I said, some of those distributions were taken from staff interviews as opposed to actually sitting in the emergency department and observing.  I'm currently working on a very large-scale simulation of a level one trauma facility that sees 150,000 patients a year.  And we're looking at that simulation requiring about a year to develop.  But it will be extraordinarily detailed.  And able to make a wide variety of careful predictions. And I say a year to develop, but while devoting about 25% of my time to it.  Depending on the system fidelity, different time frames for the models are reasonable. 

>> Okay.  Dr. Day, I think we better stop.  It is past the top of the hour.  There are a few more questions and we will get them to you and then we will through CIDER supply the questions and answers to the people who have attended.  I want to thank you very much.  I also want to announce to our audience our next session which is February 21.