We failed to predict the second wave
By: Ramprasad Saptharishi
Tags: Agent Based Modeling, Simulation, Covid 19, Sir Model, Epidemic Modeling

Modelling: Over-simplifying vs over-fitting

At the end of the day, every model makes assumptions about what is a good simplification of reality, and then chooses parameters that best fit the available data. This data is often just three time series — cases, recovered, fatalities. Therefore, the initial calibration is indeed turning a few “knobs” (parameters) to fit to observed data. Once this calibration is done, you use these parameters to make “predictions” for the future.

However, as mentioned earlier, most models often have a lot more knobs than data to fit to and so one can fit the model to essentially any data. Therefore, it is important to explicitly mention what the calibrated parameters are and, if subsequent adjustments were made, provide justifications for each of those adjustments.

Having more parameters is a double-edged sword. It allows the model to have more nuance but it also makes it easier to fit to data. If there are numerous different ways to fit to data, which of the conclusions (if any) is reliable? To better illustrate this issue, and especially understand the modelling efforts for COVID-19, we need to look a little deeper into how these models and simulation efforts typically work.

Compartmental models

The SIR model is the simplest of models for a disease spread within a population. The SIR model assumes that the population can be neatly split into 3 compartments — susceptible, infective and removed — and eventually boils down to just two parameters often called β and γ . The parameter β captures the likelihood of interactions, and also the transmissibility of the disease from one individual to another. The parameter γ is the removal rate, captures the chance that a person currently infected becomes non-infectious within a fixed period of time (say, the person recovers or dies).

One usually fits such a model to data by, for instance, setting the infective compartment to equal the active reported cases, removed compartment to equal the number of recovered and fatalities, and the susceptible compartment to be everyone else. One then chooses the best choice of β and γ and the initial values of S, I and R to best explains the data observed so far. This sets up the model, which can then be used for predictions.

In this set-up, since there are so few parameters, we usually do not have the problem of an under-determined system. However, this simplistic dynamics is probably not a reliable simplification of the disease progression for an extended period of time. This may be due to a variety of reasons — interaction patterns may be far from homogeneous, or there may be interventions such as lockdowns that change interaction patterns within a population, or there may be improvements in medical care that cause changes in the removal rate, or there may be changes in the number of people tested, etc.

There are many views taken to try and address some of the shortcomings listed above. One approach is to create more compartments. For instance, in the case of COVID-19, it seems clear from various serological surveys that there is a large population of people who get exposed to the disease but show no symptoms and are unlikely to contribute to the reported cases at all. Or maybe individuals, after being infective, do not develop symptoms right away and they stay dormant (possibly silently infectious) for a significant period. Adding more compartments to SIR models results in models such as SEIR (with E being the ‘exposed’ category) or maybe further gradations based on severe or mild symptoms and so on.

The other issue is also the fact that interaction and recovery patterns may not be the same throughout a long duration as there may be “interventions” added by local authorities. One way in which many compartmental models address this is by fragmenting the observed data into shorter time frames and using the best fit for each time frame.

Agent-based models

Agent-based models are on the extreme side of compartmental models where we essentially have a compartment for each individual with their own characteristics. In an agent-based model, the simulator creates an agent for each different individual keeping in mind certain factors relevant to the disease spread. The model introduces an interaction graph where different agents can meet at certain points (say schools, workplaces, community centres, public transport etc.) and each such interaction is a potential for an infectious person to infect a susceptible person.

The primary advantage of this approach is that it allows additional heterogeneity in how disease spread may actually happen. It allows one to study targeted interventions (say, what is the effect of of reducing attendance in workplaces to 20%). These can be incorporated into an agent-based simulator with a little additional work.

Agent-based simulators are therefore more flexible than those based more closely on the SIR model. However, agent-based models come with a host of other issues. For instance, everything from our current understanding suggests that age and presence of co-morbidities do influence the disease progression. But to incorporate this into the simulator, then one needs to feed specific numbers for this — numbers that are often not available! Often one relies on a handful of studies in a completely different context and uses that data to build the age-based transition matrix that is fed into the simulator. Public transportation might have a strong influence on the transmission. But, does the simulator have reliable data on commute patterns in the city? It may be the case that high density areas such as the slums in Mumbai probably see a faster spread as a result of being densely packed, but different publicly available data sources give very different estimates of what fraction of Mumbai stays in these high-density areas. Hence, yes, the agent-based simulator could potentially make use of this nuance, but data at that level of granularity may not be available. These are all places where a model makes additional assumptions.

In other words, the issue of having an under-determined system is a much larger issue here. Agent-based simulators have so many knobs (parameters) that one can potentially fit anything by setting appropriate parameter values. It is therefore a lot more important to identify potential red flags in agent-based models because it is so much easier to fit to data.

(Another issue is also the fact that agent-based simulators tend to involve a much more complicated codebase compared to a simulator for a compartmental model, and hence there is a lot of scope for programming errors as well.)

Choice of parameters in under-determined systems

Both the models mentioned above involves working with under-determined systems. Whatever are the choices for these model parameters, a key question is whether they “make sense.” This is where the domain knowledge becomes very relevant. A person without domain knowledge may only be able to spot clear red flags to say that a certain choice of parameters do not make sense. Large deviations in parameters are one such example of red flags. For instance, say the fit in a month had a much higher recovery rate compared to the next month. Unless there appears to be a plausible explanation for this sudden drop in recovery rate, this is a clear red flag. Or, say, we know that stricter measures were imposed during a period but the best-fit β value seems to be larger than earlier — this is a red flag.

Another way of spotting a red flag is via sensitivity analysis. If one were to change one of the parameters very slightly, without substantially affecting the fit to observed data, does that lead to a substantial change in predictions? If two “equally good” parameter settings lead to drastically different conclusions, we cannot have much confidence in the conclusions.

Even if we are unable to spot any red flags, that does not mean that model or the predictions are correct. There is just no way, without substantial domain knowledge, to confidently assert that parameter choices make sense. Models hence go by some parameter choices that seem reasonable, while continuing to look for potential red flags.

What modellers should do

With the potential concerns when dealing with such under-determined systems, modellers can take steps towards being transparent about the simulator. This would be a step towards alleviating some of the legitimate concerns.