**Transcript of Cyberseminar**

**Timely Topics of Interest Series**

**Hybrid Agent-Based/Discrete Event Simulation Models for Analysis of Health Care Delivery Systems**

**Presenter: Dr. Eugene Day**

**November 7, 2013**

*Moderator:*We are at the top of the hour now, so I’d like to introduce our speaker.  We have Dr. Eugene Day.  He is a Senior Improvement Advisor at the Children’s Hospital of Philadelphia and we are very happy to have him presenting for us today.  At this time I’d like to turn it over to you, Eugene.

*Dr. Day*Thank you very much, Molly.  It’s a real pleasure to be here.  Once again, my name is Eugene Day.  I’m a Senior Improvement Advisor and also a principal investigator with the Children’s Hospital of Philadelphia.  The work I’m going to talk about today is agent-based and discrete event simulation models for analysis of healthcare delivery systems.  This is a presentation based on work that was done at the St.  Louis VA Medical Center.By way of introduction, there’ll be a brief introduction followed by I’ll talk a little bit about the background of complex systems in healthcare: what those are, what they mean.  I’ll give a quick background on discrete event simulation and on agent-based models.  I’ll talk about how we integrate the two methods and then describe the results of a pilot study using the methodology to address the question of how long the screening interval between diabetic eye screens for diabetic veterans with no or with background diabetic retinopathy—how does that duration influence the progression of diabetic retinopathy in the population and discuss the ramifications of that study and of the methodology and the conclusions.As I said, this is work that was done for the Veterans Administration St. Louis Health Care System and is the result of a merit review pilot study called “Simulation Modeling for Implementation Analysis.”  Equipment and software for the study was provided by the VHA Innovations.  I have no conflict of interest to report regarding this material.  The data and some figures from this presentation have been previously published or are currently in review, and the study was reviewed and approved by the VA St. Louis Health Care System Institutional Review Board.The questions that I’m hoping that we can address today are: How can simulation inform policy and practice in healthcare?  How can simulation be used to improve health services delivery?  How can simulation be used to improve the population health in general and specifically can we use simulation to analyze the consequences of increasing the duration of the screening interval for diabetic retinopathy?The first poll question, then, is who has any computer simulation modeling experience?  Are you a model developer?  Do you directly use models developed for your institution?  Do not use them by your institution does?  You’ve heard of it perhaps but your institution doesn’t do any yet.  Have you never heard of simulation modeling at all, or perhaps because simulation is used as a term of art in many different disciplines, were you under the impression that we were here to talk about mannequins?

*Moderator:*Great.  Thank you so much, Dr. Day.  It looks like the answers from our audience have just about stopped streaming in, so I’ll go ahead and end the poll now.  Would you like to read through those results real quick?

*Dr. Day*Sure.  It looks like our largest group are people who do not use simulations themselves but their institution does was 17.  A large group of people who have heard of simulation modeling but their institution doesn’t do it and I’m gratified to see that we also have 13 model developers with us, which is definitely gratifying.  How do I get back to the other view?  There we go.  It looks like we have a broad range of interest here.  I want to talk a little bit about complex systems in healthcare and what that means.  On the right we have what could be seen as a very simple systems dynamics model of an emergency department.  If what we’re interested in—our current patient census; our patients arrive and they flow through and at any given time we’ll have a census of patients in our system.  The rate at which patients flow through the system is going to be dependent on the number of exam rooms we have, the number of doctors, the number of nurses, the time required for labs and imaging and of course it’s going to be a consequence of the rate at which patients arrive.We have these multiple dependencies in these systems; that the metrics of interest are dependent upon an extraordinarily large number of factors which may interact with one another in very complex ways.  We describe it as having dynamic behavior because these systems unfold over time.  They’re subject to what of course what was made famous in the movie and book *Jurassic Park*, the butterfly effect, which states that a very small change to initial conditions may have an enormous change in our outcome metrics.  These systems may have feedback loops; they may be adaptive.  An easy example of an adaptive system that we can think of is like a predator/prey system where if you have excess predators in a forest they may eat up all of the prey animals, which then means there’s not enough food left for the predators so their numbers decrease which allows the prey numbers to rebound.  That’s an adaptive, a naturally adaptive system.  Systems can be memory dependent, which means that their future state doesn’t just depend on the current state but may depend on the entire past state of the system.  In this case—especially in the case of healthcare systems—we may literally be talking about memory dependent systems where people’s impressions about a healthcare system may depend upon information which is no longer true.  They’ll have nonlinear state variables, meaning that we can’t just assume that there’s a simple relationship between, for example, the number of exam rooms that we have and the rate at which patients flow through those exam rooms.  Finally—and this could be the most vexing issue dealing with complex systems—is they have indistinct boundaries.  When we look at this very simple diagram on the right, we note, “Well, this doesn’t take into account things like consults from other services.  It doesn’t take into account the time of day or the day of the week.”  When we start sort of expanding our scope and considering what does—what actually goes into the complex dynamics of the system, we can end up with something that looks really alarming and confusing as we realize that the boundaries of this system sort of keep encroaching outwards until they may involve whole other hospital departments.  They may involve other hospitals when you start talking about the transition—sorry, the transfer of patients from one hospital to another.  It’ll depend upon the operating rooms.  In order to change the rate at which patients are treated we may need to hire, which can take a great deal of delay.  The rate is going to depend upon institutional policies which may depend on national regulatory policy.  Pretty soon, in order to model the rate at which patients flow through my own small emergency department, I have to model the entire healthcare system of the United States in order to take into account everything that influences it.  Determining where we’re going to set our boundaries in order to properly model a system of this type can be a very delicate and even an artistic process.For the model that we’re going to discuss today, I wanted to just give the basic classification.  First of all, it’s a hybrid model, which means two different things: one, it means that it’s a combination of multiple methods, in this case agent-based modeling and discrete event simulation.  Second of all, it’s a combination of multiple time measurements.  If you were to go and Google “hybrid systems,”  that’s what you’re going to find, is that hybrid systems traditionally mean that the system itself has both continuous time elements—such as how long patients stay in particular areas—but also discrete measurements as well, such as has a patient been given a screening test, yes or no.  That’s a hybrid system.  It’s an agent-based model because we deal with systems which are best modeled used using agent-based methods—which I’ll describe in more detail in a bit; fundamentally, agent-based models are really, really good at modeling individuals within a large population—and discrete simulation, which is a step-by-step model progression which is generally really, really good at modeling things like flow charts.  Therefore it’s very good at modeling the progression of a patient through a clinical system, for example, or you can think of it as a Model T on a production line.  Finally, it’s a multi-scale model.  Multi-scale modeling refers to the model unfolds on two different time-scales.  In our case we’re looking at the progression of diabetic retinopathy in the population over about a decade, whereas the clinic—the actual clinical system within the model has to unfold on an hour-by-hour basis in order to accurately capture the way a clinic functions.Let’s talk about the second part first: discrete event simulation.  Discrete event simulation is very widely reported in the literature.  It’s a standard tool for planning and for quality improvement in health care at this point.  It’s a computer simulation of real-time processes which occur at specific time intervals as time goes on.  In order to build discrete event simulations, we decompose the system into its basic elements: entities, which are patients, patient records, lab samples—anything upon which we do work in the system; resources, which are the physicians, the nurses, the equipment that are all our assets that we can use to perform the work; locations, which in some software systems are actually a subset of resources, but these are the exam rooms and the queues and they’re also the virtual locations that we can use them—for example e-mail stacking up in a computer—and then the paths, the networks which link the locations together.  In order to build a discrete event simulation, we have to design the process flow, which essentially requires us to answer the question: how do entities consume resources at locations and then proceed along paths to the next location from the beginning of that simulation to the end of the simulation, or as we would say, from the source to the sink?In our case we use a fairly rudimentary simulated eye clinic which performs two functions.  It performs eye screens, so screens with an ophthalmologist or an optometrist to determine an individual entity’s current state of diabetic retinopathy and then laser eye surgeries or panretinal photocoagulation, which is indicated as a treatment for proliferative diabetic retinopathy.  Now I should state as well at this point that I’m not a physician and while I had physician colleagues on this project, this isn’t designed to replace medical judgment.  This is a proof of concept for simulation and using that to model real-world systems.  I’ll talk about a little bit more about that towards the end as well.This discrete event simulation will represent the environment of interaction for the agents.  Essentially this is where the agents—which are our diabetic veterans in the population—compete for clinical time and resources.Agent-based models.  An agent is fundamentally just a computer object, a data structure which has individual rule sets and attributes that allow it to be capable of autonomous decision-making.  An agent will have a statechart or a rule set which governs its behavior in whatever situation it finds itself in.  Agents can interact with each other and with their environment, although we don’t necessarily exploit that capacity here.  As I said, we treat the discrete event simulation as the environment of interaction rather than having agents directly interact with one another the way they might in other agent-based simulations.  As I said before, these are ideal for modeling populations of individuals: birds in a flock, fish in a school, cells in a tumor, cars in traffic—these are all examples that you can find in the literature and in textbooks at this point that our excellent uses of agent-based simulation, agent-based modeling.  It’s really interesting how very few and very simple rules can result in extraordinarily complex behavior.  For example, fish in a school; it takes very little in the way of rule sets to have each agent—which is a fish—function in the population in an interesting way.  Schooling behavior may simply be the result of rules that state every fish tries to stay within six inches of its nearest neighbor, and every fish tries to move orthogonally to a predatory threat and that will naturally result in some of this very, very complex what we call emergent behavior where we see many agents acting in concert in a larger population system.  In our model, each agent represents a single diabetic veteran and is imbued with the individual characteristics of that veteran—so their BMI, recent A1c, blood pressure, age, duration of diabetes, presence of diabetic nephropathy, stage of diabetic retinopathy, et cetera—with an event which updates the agent annually in order to show how these things change over time, these predictors and covariants.  There is then also a rule set which governs how diabetic retinopathy advances and an algorithm which determines the visitation schedule for the clinic for eye screens where the agents will than queue for availability.  Our agent-based veteran includes, like I said, statechart governing the progression of diabetic retinopathy, numerical fields capturing the veteran’s health status and demography, function attached to a life table to predict when the agent expires, an event which updates the veteran annually and then an entity generator which communicates with the DES clinic—I’ll talk a little bit more about that in a minute—and then finally we use multiple regression based on a cross-sectional cohort of real-world patients from the St. Louis VA Eye Clinic to model the DR progression.  This is what the data structure looks like.  We modeled this using AnyLogic Professional and here we see an agent.  On the left is—the large colorful structure is the statechart, which governs the advancement of diabetic retinopathy, possible death of the agent and the state of a clinical visit in each stage.  On the right we see all of the various parameters and variables and covariants that each diabetic veteran has which govern how the diabetic retinopathy progresses and the life table, which determines when the patient might expire.To integrate these models, each agent periodically enters that clinic visit state and generates an entity.  Agents can’t interact directly with a discrete event model of a clinic and so we have to create an entity, pass the information from the agent to that entity and associate the agent and the entity to each other with a unique identifier and then pass that entity to the source of the clinic so that the entity then can negotiate the clinic.  All entity activity is governed by an external flowchart while they’re in the clinic.  The entity receives the indicated care according to whatever individual needs they have—whether or not they need surgery; how frequently they have the eye screen, et cetera—and then that information is updated, passed back to the agent and then the entity exits the discrete event simulation clinic and vanishes.  In this way we can create essentially the functionality of a veteran at home, visiting the eye clinic, having their health status updated and returning home.  To say that again, what we see here, the agent-based veteran—which is imbued with all of the covariants and predictors of retinopathy progression and has this annual update which keeps track of its age and duration of DM, which will change the rate at which retinopathy progresses—and then as time goes on the agent will periodically visit eye screens, perhaps have treatment, update the health status and then return home.  The larger structure—and I apologize that this is a little fuzzier than I intended it to be—the larger structure is that we have a discrete event simulation which replicated the St. Louis VA Eye Clinic, agent-based veteran creates the entity, visits the eye clinic, updates the health status of each individual, seeks treatment as we see here in the overall movement of information in the simulation.Now how do we actually use this large structure that we’ve described to conduct an experiment?  What we did is we created—once we had a template for a veteran, a diabetic veteran—we created, we replicated that 500 times and then conducted 10 simulation runs of 10 years each of these cohorts of 500 veterans.  We did each of those five times for five groups with screening going for the screening of eye—excuse me—screening for diabetic retinopathy or screened patients with no diabetic retinopathy or background diabetic retinopathy from one to five years between screenings.  Once proliferative diabetic retinopathy was noted, then screenings went back to annually or more frequently as necessary.We used that to examine our outcome metric, which was the incidence of vision loss in the population at the end of ten years.  What’s interesting about this is that the simulation really allows us to isolate the contribution of screening intervals.  We don’t assume that the agents changed their behavior in any way based on being screened less frequently or more frequently.  It is only the impact of the screening interval that contributes to accelerated progression of diabetic retinopathy towards vision loss.  When we ran this, what we discovered was—like I said, our primary outcome measure was the number of veterans reaching the state vision loss.  We found no significant difference between the annual and the biannual groups, and that is concordant with most of the real-world literature that’s out there, is we know that for veterans without retinopathy, that there’s large agreement in the ophthalmological community now that biannual screens are quite safe because it’s a very slowly progressing chronic condition.  We did see a significant increase between a one-year interval of the eye screen and a three-year interval from an average of 60 veterans to 65 veterans in each of those groups.  We saw increases as the screening interval extended up to four and five years, but it didn’t rise to the level of significance from 65 to 67 veterans, so it appears as though most of the contribution of extending the duration of the screening interval to vision loss occurs between two years and three years.Additionally, this provided a reduction of more than 75 appointments per year between the annual and biannual screening groups.  That represents an enormous number of appointments, eye screen appointments which would then be freed up, either in terms of cost savings, simply cost savings, or in terms of allowing additional appointments of other types for other veterans.  I think we can all agree that it is generally desirable to adopt policies which provide equivalent or improved care at lower cost or which provide additional care at the same cost.  Of course the VA is not a traditional fee for service-dependent environment, but that doesn’t mean that the budget isn’t important.  We are all constrained by how we are able to provide care and the cost of that care.  Simulation really allows us to identify the candidates for such policies.  Essentially we can test many, many, many different types of changes and implement the ones which are the most promising and so it functions as hypothesis generation, that we can run many, many different ideas in many, many different contexts.  It allows us to determine where the biggest bang for our buck is going to be in terms of policy changes and crucially, I think, it allows us to avoid policy changes which would be expensive but not particularly effective or far worse, those which would be expensive and which would result in worse care.  It allows us to avoid costly change that would be deleterious to our populations.  It can help reduce waste in health delivery by identifying unnecessary outlays, such as in this case we can increase access through the reduction of redundant screens.  This simulation also allows us—the structure allows a very exciting—to me anyway—bidirectional analysis.  We’re not just restricted to examining the consequences of the changes in clinical policy to the effects of the population, although that is certainly one of its great strengths, that changes to clinical policy can be measured at a population-level outcomes instead of at the strict level of saying, “Well, if we do fewer eye screens for patients with no retinopathy or with minimal background retinopathy, then we’re going to save a certain amount of money every year or we’re going to be able to increase access by a certain percentage every year.”  We can actually measure at the level of population-level outcomes, which can help us manage chronic disease.The VA is now—especially in diabetes—is really being looked at as a model organization for management of chronic disease and our performance metrics are better than a lot of the HMOs out there now.  What this allows us also to do is to look in the other direction: how changes to population demographics, changes to predictors and covariants can be measured at the level of demand for services and queue lengths.  For example, if we had some sort of magic intervention which would reduce all of our veterans’ BMIs by five percent, that’s going to—in the simulation that’s very easy for us to incorporate, and that would allow us, then, to influence how diabetic retinopathy is going to progress, because we know that diabetes in general—and diabetic retinopathy specifically—are both sensitive in general to the BMI of the patient.  It may not be a direct action—I’m not an epidemiologist—but we know that those are correlated.  By changing the BMI of our population of diabetic veteran agents, if we know the manner in which that affects the progression of disease, we would be able to predict how that intervention would influence the demand for service of laser eye surgery, the demand for service for additional eye screens.  If patients are progressing more rapidly and need to be screened more frequently when they have proliferative diabetic retinopathy, then that’ll tell us what sorts of resources we’re going to have to devote to those facilities.  This helps us manage healthcare facilities, because we can predict ways that our population demographics are going to change.  We know that diabetes in general is increasing in our population; we know that BMIs are getting larger, but this may also—so this may help us plan for the future that we expect, but it may also allow us to identify and invest in the most effective ameliorations of those things.  If we have interventions that we believe would be effective for BMI or would be as effective for A1c, we can determine which of these are most likely to provide a powerful—a large effect size.  What this does for us is it integrates the populations and institutions in a single comprehensive model.  Additionally, simulation allows us to conduct risky, ambitious and even unethical experiments.  You may have noticed on a previous slide that I underlined including background diabetic retinopathy.  Now as I said before, the ophthalmological community is pretty much in concert now that screening biannually for diabetic patients with no retinopathy is perfectly safe, that that same agreement is not there for background diabetic retinopathy.  Many, many, many ophthalmologists believe that it is important with any exhibition of retinopathy to screen annually rather than biannually, but in simulation I can test biannual screens on patients with background diabetic retinopathy, fearless that I’m putting any real patients at harm.  This can allow us to conduct these risky experiments.  If every member of my cohort goes blind in simulation, then all we’ve learned is that we should really, really, really not consider doing that in the real world ever, but we have to take great care in translating these risky results back to the real-world trials.  Even though this simulation does not predict an increase in vision loss as a result of biannual screens in patients with background diabetic retinopathy, I can’t sit here and tell you that therefore we ought to adopt that and make that the policy of the VA.  There are some significant limitations to this type of work, and most importantly, no one wants less than me for engineers to be telling physicians how to treat patients.  That’s not the goal.  The goal of this is to—as I said before—function as hypothesis generation for health services research, that these are ideas which can come from simulation and then perhaps be conducted as responsible real-world trials with proper physician oversight.  As I said, these results to complement other real world and simulated studies which have shown recently that two-year screens for no retinopathy are safe.As I said, there are some significant limitations.  We use cross-sectional patient data, so any follow-up work would definitely require improved epidemiology, as I said from the outset.  This was a pilot study.  It was designed to develop the simulation methodology, rather than necessarily being indicative of improvements to practice.  Additionally, this is not designed to predict the disease progression in any individual.  Only aggregate information should be considered informative from these types of models.  No agent-based model is going to be able to project the way that an individual’s diabetes is going to respond to treatment or how any individual we can expect to progress in a particular way.  They’re designed so that each individual, when many individuals are combined into a population, that we can make probabilistic predictions at the way the population health will move, not the individual health of individual veterans.  As I said, simulations do not replace medical judgment.  They can provide useful supplemental evidence to incorporate into human studies, but they are not by themselves medical evidence that should be used to make conclusions about human patients should be treated.I hope that we’ve demonstrated at least that simulations can inform policy and practice, that using these methods to model proposed policy interventions even in these very, very complex systems and test them in silico rather than in vivo, supplies evidence that practice changes may be safe and effective for the long-term; like I said, supplying evidence but not supplying proof.  I hope that we’ve seen that simulation can help improve health services delivery by determining optimal screening schedules, by allowing additional appointments to be made for alternative care and by determining how healthcare delivery resources can be strategically positioned, essentially, in order to provide the best care for the veterans and for the populations that depend upon our services.I hope that I’ve shown a little bit about how simulation can improve population health by identifying broad risk categories for appropriate screening intervals, optimizing the delivery of care to the individuals most in need of particular services and targeting interventions likeliest to have large effect sizes and providing data-driven analysis to encourage adoption.  As I said at the very beginning, the grant from which this work arose was “Simulation Modeling for Implementation Analysis.”  Our goal is that we will be able to produce systems which will support decision-making in such a way that they can encourage the adoption of sensible and improved health policy so that we have data to say, “We believe that this information—that these changes are going to have the following effects.”  We don’t have to say, “A bunch of smart people sat around a desk and said ‘We think that this is a good policy for the future.’”  What we have is an actual projection of how the future might look if we adopt particular policies.  I think maybe we’ve shown that simulation can inform practice regarding the screening interval for diabetic retinopathy.  The results that we have do concur with some of the accepted evidence, but we have taken a step further than we’ve seen in other literature by including background diabetic retinopathy in this group.  That means that very special care would have to be taken before we could adopt results; that this may indicate that this is worth additional study, but it is certainly not proof that we can suddenly adopt this policy for patients with background diabetic retinopathy.  We would want to do careful retrospective study, I think, of administrative data to support and confirm the results that perhaps we could with our very, very large data set that we have in the VA go back and find those patients with background diabetic retinopathy who did just happen to have roughly biannual screens and see how that population fared as time went on—maybe looking from 2000 to 2010—in order to support or confirm the results that we found for this, but any prospective trials, I think, would have to be very carefully planned and conducted by physicians.  We don’t want to put any real patients at risk.  It’s fun and sort of ham-handed to put patients at risk in simulation because we know that at the end of the day if something goes terribly wrong nobody’s harmed, but when translating simulated results to the real world, we have to be very, very careful that proper oversight is seen regardless, of the quality of evidence that we have out of the simulation.  It looks like I’m going to end a few minutes early.  We may have a little extra time for questions.  A final poll question then: Is anybody actually interested in this?  Do you feel that this has potential for your institution?  Are you interested perhaps in the future in exploring these options, or are you definitely not interested or are you convinced that the set of mannequins that you have at the time is already sufficient?

*Moderator:*Thank you very much, Dr. Day.  It looks like people are a little more gun shy to answer this one, but we’ll give everybody a little more time to submit their responses.  Just click the circle to the left of your answer choice and you can submit the answer.  Looks like things have just about leveled off.  We’ll give everybody just a little bit more time.  All right.  Eugene, would you like to read through those responses?

*Dr. Day*Sure.  Well, I think the good news—for me at least—is that I have not apparently scared everyone off from simulation.  It looks like our largest group—about 50 percent—says that this has definite potential and our next group at about 41 percent says that they’re interested in exploring the option.  I don’t have anyone who says that this looks like a complete waste of time, so that’s gratifying from a personal perspective.  I’ve included a slide with some further reading if you’re interested in epidemiology using agent-based models; if you’re interested in using discrete event simulation or if you’re interested in more of the deep details of how we devised the agent-based template for diabetic retinopathy in a population of veterans.  I’ve also included a couple of references on the best practices that were proposed last year by the Society for Medical Decision Making for model transparency and validation and for the generation of discrete event simulation.  Then finally, the Günal and Pidd review article in the *Journal of Simulation* is an excellent place to look at sort of a comprehensive treatment of what’s been done using discrete event simulation.I’d like to thank my co-authors, Nathan Ravi, Hong Xian and Ann Brugh and coworkers from the St. Louis VA, Jeffery Scherrer, Angelique Zeringue and Lauren Garfield, who helped me with statistics and talked me through some of the epidemiology and of course for funding, both the VA Office of Research and Development, Health Services Research and Development and the VHA Innovations.  I would be happy to answer questions.

*Moderator:*Great.  Thank you.  We do have about half a dozen pending questions.  For those of you that joined us after the top of the hour, to submit your question or comment now simply type it into the Q&A box located in the upper right-hand corner of your screen and we’ll get to it in the order that it was received.The first one: How will you measure the impact of reduced visits on patient behavior before studying policy based on the model outcome?

*Dr. Day*That’s a really good question and the answer to that is I’m not entirely sure.  Like I said, I’m an engineer.  I have always been involved in the development of the model, but I agree that this model, like I said, is designed to isolate the contribution of the screening interval and we assume that changing the screening interval did not change patient behavior.  That’s probably not an assumption which bears out in real life, at least if the ophthalmologist is the veteran’s only point of contact with the VA, but the chances are that even if you’re delaying the eye screens and you’re only doing a retinopathy screening every other year rather than every year for a diabetic veteran who has not yet exhibited retinopathy, the chances are that you’re not saying, “This is going to be your only contact with a physician who’s in charge.”  They’re still going to be seeing their primary care physician; they’re still going to be seeing their endocrinologist, so I think ideally we would like to say that this would not influence their overall management of their diabetes, because the ophthalmologist is not the only physician that they would be seeing in order to manage their diabetes, one would hope.  But yes, I think that it’s a risk worth stating that if you are seeing—if you’re receiving less healthcare because you don’t need to be screened as often, we would not want that to result in poorer care of diabetes in general.

*Moderator:*Thank you for that reply.  The next question we have: Can we talk through a simple build or high level steps for constructing a five-step process for DES?

*Dr. Day:*Sure.  Let’s go back to that earliest slide.  The process of building a discrete event simulation.  The first thing that you need to do is—I’d say decomposing the system into these basic elements, yes.  We have to identify what are our entities, what are our resources, what are our locations, what are our paths and then lay them out—you lay out your paths on a blueprint and you place your locations where you need them on the blueprint, et cetera.  Generally, the first thing that we have to do is someone like me—a developer—has to go into the system, interview the people that work there and say, “How do you do what you need to do in order to asks required of you to work in this system?”  Through that process of interviews, we end up building that process flow, which is how do entities consume these resources in locations and proceed to the next location.  Then all of that needs to be informed with data from the real world.  We need access to—lots of this can be done with aggregated data and in general, this can be done with de-identified data so that we don’t have protected health information.  In this case for this model, we did use the PHI and that’s why we had an approval rather than an exemption from the IRB, but most of the work that I’ve done in discrete event simulation has been considered IRB-exempt, although caveat, of course: only an IRB can determine if a study is IRB-exempt.  Go into the system.  You learn the system by interviewing the people that work there, by shadowing patients throughout their process, by shadowing resources throughout their process, developing these flowcharts and then by taking that information and coding it into a commercially available discrete event simulation suite.  The software we used here was AnyLogic, but there are many others.  There’s MedModel; there’s FlexSim; there’s Arena, SIMUL8.  You code that into the system.  You code the system into the discrete simulation software.  Then you need to validate that simulation so that you can say that the simulation really does model what the real world does.  There’s about five different types of validation and these are described in those best practices articles that are listed at the end of the simulation.  I would really recommend reading those.  Sure.  The articles by Eddy and the article by Karnon are really good for how to build models and how to validate models.  Fundamentally, for most modeling purposes you need three sets of validation.  You need the face validation, which basically says you take the simulation once you think it it’s done and you show it to the people that work in the system and the people that administrate the system and you ask them to follow the simulation through and say, “Does this do what we think it ought to do?”  Then there’s internal verification or internal validity, where someone other than the developer goes back and reads through the code and makes sure that the code is really doing what the developer wants the code to do.  Then there’s external validation, which states that when we put reasonable inputs into the simulation that the outputs look like the real world output.  For the example of an emergency department, when we have patients of the appropriate acuity, of the appropriate number at the right times of day diving into the system, does our length of stay, does our hourly census actually look in the simulation the same as it does in the real world?  When those things are true, that’s called external validation.  The other two types of validation that are often used are cross-validation, which says that two different models both agree and predictive validation which is sort of the Holy Grail, which says that when we make a change in simulation and then we make the same change to the real world, do the outcomes of those changes agree?  That’s how you build a simulation.  You start by entering the system, learning how the system works and laying out the flow of entities, resources, locations, path networks, building it in a discrete event simulation software suite and validating it.

*Moderator:*Excellent.  Thank you very much.  We do have seven pending questions left.  The next question starts out, “Great presentation.  How much of this work is being used in the VA?  What are some of the barriers in adopting this method?”

*Dr. Day*I’m not entirely certain how much of this work is being done in the VA.  I know that I am not the only person.  There’s a guy in Salt Lake doing agent-based modeling—I want to say his name is Michael Rubin—but there’s not as much as I think that could be valuable.  Simulation, of course, is useful for so much more than just research.  It’s directly applicable.  In my current position with the Children’s Hospital of Philadelphia most of my work is in direct quality improvement using simulation to determine how can we improve our current systems in order to improve care for our cohort of patients that we have from day to day.  That work will invariably lead into interesting research questions as well, but I’m not entirely certain how much quality improvement is being done.  I know that there is some that is being done.  Some of the barriers to the adoption of this work—of this type of work are the software suites aren’t cheap.  I think a simulation license may cost as much as $25,000.00 as an upfront investment, and if you need ongoing technical support it can be a couple of thousand dollars a year.  In order to do it well, you need people that are trained to do it.  Unfortunately, if you read the literature around simulation, you get a lot of discrete event simulation that was attempted to be used to answer individual research questions and it was conducted often by—this is less true today than it was ten years ago—but often conducted by physicians who may have had undergraduate backgrounds in engineering, who are interested and intelligent but aren’t specially trained in this field.  As a result, when you read a lot of the literature you do see a lot of un-validated simulations; you see a lot of simulations with rather dramatic assumptions that had to be made.  Another barrier to this type of work is there needs to be the institutional investment in someone who can conduct, manage and deploy simulation in a responsible and comprehensive way.  Actually, I have a paper on that—exactly on that process—coming out in the *American Journal of Medical Quality* here soon.  It’s in press now, so hopefully you can look for that and it will draw a blueprint.

*Moderator:*Thank you very much for that reply.  Here are some comments from a viewer.  “The simulation seems to be as good as the data from the St. Louis VA Ophthalmology Clinic in this case.  The progression of patients in the simulation is deterministic based on curve fitting, what happens in this particular population at particular epoch.  If we make a policy to affect outcomes in the next ten years based on the last ten, we may see the underlying data change.”

*Dr. Day*Oh, that’s certainly true.  One of the fundamental assumptions that has to be made with simulation—and this is generally true in anything that uses retrospective data—is that the recent past is predictive of the near future.  If we started to see dramatic changes in the way in which diabetic retinopathy progressed or the number of patients exhibiting diabetic retinopathy—which is certainly possible—then the results of this might not hold, which is why I say simulation can inform policy, but it can’t write policy for you.  It is absolutely true that you want a more generalized cohort.  If we were to extrapolate from this and make national policy based on the St. Louis VA population over the last ten years, that may not be responsible.  I mean it certainly wouldn’t be based on the strength of this right now, but in general you would need to verify that the population that any simulation is based on is representative of a larger population.  I think it would be good to update this with a broader cohort from administrative data, but it is true in general in any human studies discipline of any kind that the results are true for the population that was studied.  Extrapolating to other populations may not necessarily yield the same results, but that’s one of—I think one of the ubiquitous challenges of medicine in general.

*Moderator:*Thank you for that reply.  We do still have several more questions coming in.  The next question: Can this be done in SAS?

*Dr. Day*I don’t speak SAS, so I’m afraid I can’t answer that question.  In general, as a person with a background in computer science, I believe that SAS represents a Turing-complete programming language and therefore theoretically anything can be done, but my impression is that it is not the right tool for the job.

*Moderator:*Thank you.  The next question: What regression models of DR progression do you use and did you use the results to inform your agent-based model?

*Dr. Day*The regression modeling was done by Hong Xian.  I’m not a biostatistician.  Like I said, this was cross-sectional data and unfortunately I don’t have the slides on that, but this is described pretty comprehensively in the paper that was published in *PLOS ONE* earlier this year.  That information is available, but often top of my head I’m going to have to pass.

*Moderator:*Thank you.  Here’s a couple comments from another attendee.  “Seeing an ophthalmologist can have a large effect on behavior.  The visits tend to remind the patient about the possibility of losing his or her eyesight.  On average, such a visit may well have an outsized effect on behavior relative to their other MD, RN visits.”

*Dr. Day*That sounds perfectly reasonable.

*Moderator:*Thank you.  “Can you please talk briefly about the process of validating your model?”

*Dr. Day*Sure.  One aspect that we validated—now of course this is prospective simulation and so it’s very difficult to—we can’t validate unless we also do a prospective study in the real world in a 100 percent completely thorough manner.  However, what we established is—using the curve fitting that the other commenter described, we validated that the simulated cohort of patients that we created mimicked in all relevant respects at baseline the population from—the real-world population from the St. Louis VA Medical Center, so we established that we’re capable of creating a cohort that looks the same at baseline.  Then we developed a progression based on the predictors of progression from our cohort in the real world.  We supplemented that in some cases with progression from the literature.  It is fair to say that this particular model should be described as speculative in nature, which is why I feel like I’ve tried to qualify those results at every turn.  In general, the best way to validate the outcome of a model—and in a more comprehensive model that had better access to epidemiology, what would be nice is to set the simulation based on a real-world cohort from, like I said, say, 2000, create a simulated cohort based on that 2000 baseline, run that forward to 2010 and see how well we hit the 2010 data.  With a pilot study and on the short timeframe that we had for this study of one year, we weren’t able to validate at that level, which would really have been a very strong validation metric.  In general what we want to see from simulation is that when you compare the output of a simulation to the output of the real world, that you don’t see statistically significant variation in your metric of interest.  For example, for an emergency room if we were to look at the length of stay for each individual patient, we’d like to compare that to the length of stays for real-world patients and show that those populations were not statistically significantly variant.

*Moderator:*Thank you for that reply.  We do have another comment.  “You did say that these models work better with cancer screenings where the patient has less ability to alter the course through behavior.”

*Dr. Day*I do not recall mentioning cancer.

*Moderator:*All right.  No problem.

*Dr. Day*Sorry, sorry.  Cells in a tumor.  Yes, that is a reference to—so agent-based modeling has been used to model cells in a tumor in the past.  I have not personally done that work, so I can’t comment on its efficacy in that manner.

*Moderator:*Thank you.  I believe this is the final question.  Did you attend the Winter Simulation Conference?

*Dr. Day*I have never had the pleasure of attending the Winter Simulation Conference; however, this year there will be a paper being given by Ola Batarseh.  Dr. Batarseh was a postdoctoral scholar under a colleague of mine, Eric Goldlust, at Brown University, where I am also an adjunct assistant professor in the medical school there.  There will be a paper presented at this year’s Winter Simulation Conference on the simulation that we did at the Rhode Island Hospital emergency department.  If I get the opportunity to go this year, I would love to go.  That is the Holy Grail conference of this discipline, yes.

*Moderator:*Thank you.  It looks like I did miss a question or I may have asked it and I don’t recall.  Do you know if anyone has created a dynamic model of VA outpatient clinic appointment access?

*Dr. Day*VA outpatient appointment clinic access?  No.  I know that one study that we had funded but unfortunately were unable to accept—in part because of my position change—was designed to look at patients who are frequent attenders and no-shows in primary care.  We were intending to build just such a model for primary care in women’s clinics, but to my knowledge I haven’t seen one yet.  However, there are definitely many simulation models in the literature of outpatient clinic access.

*Moderator:*Thank you.  One of our attendees just wrote in and said that they are working on one in Durham.

*Dr. Day*Oh, excellent.  I would love to hear from you.

*Moderator:*Excellent.  Another person writes, “Yes.  There are tons of models being developed by the VERCs in this area.”

*Dr. Day*Yes.  That makes good sense.

*Moderator:*Great.  Well, that is our final question.  Would you like to give any concluding comments?

*Dr. Day*I would just like to say thank you very much for letting me present.  I hope that people found it at least interesting and if nothing else are aware that simulation isn’t just for creepy mannequins.

*Moderator:*Great.  Thank you so much, Dr. Day, for lending your expertise to the field.  I want to thank our attendees also for joining us.  Please wait just a second.  I will close out the meeting and I’d ask you to please fill out the audience survey that’s going to pop up on your screen.  It is your feedback that guides what sessions we have presented.  Thanks again very much to Dr. Day and to our attendees.  Everyone have a nice day.

*Dr. Day*Thank you very much.

*[End of Audio]*

Page 1 of 16