EpidemiOptim: A Toolbox for the Optimization of Control Policies in Epidemiological Models
Authors: Cédric Colas
Boris Hejblum
Sébastien Rouillon
Rodolphe Thiebaut
Pierre-Yves Oudeyer
Clément Moulin-Frier
Mélanie Prague
Venue:  (2021)
Keywords: epidemiology
covid19
reinforcement learning
evolutionary algorithms
multi-objective optimization
decision-making
toolbox

Abstract: Epidemiologists model the dynamics of epidemics in order to propose control strategies based on pharmaceutical and non-pharmaceutical interventions (contact limitation, lock down, vaccination, etc). Hand-designing such strategies is not trivial because of the number of possible interventions and the difficulty to predict long-term effects. This task can be cast as an optimization problem where state-of-the-art machine learning algorithms such as deep reinforcement learning might bring significant value. However, the specificity of each domain - epidemic modelling or solving optimization problems - requires strong collaborations between researchers from different fields of expertise.
This is why we introduce EpidemiOptim, a Python toolbox that facilitates collaborations between researchers in epidemiology and optimization. EpidemiOptim turns epidemiological models and cost functions into optimization problems via a standard interface commonly used by optimization practitioners (OpenAI Gym). Reinforcement learning algorithms based on Q-Learning with deep neural networks (DQN) and evolutionary algorithms (NSGA-II) are already implemented. We illustrate the use of EpidemiOptim to find optimal policies for dynamical on-off lock-down control under the optimization of death toll and economic recess using a Susceptible-Exposed-Infectious-Removed (SEIR) model for SARS-CoV-2/COVID-19. 
Using EpidemiOptim and its interactive visualization platform in Jupyter notebooks, epidemiologists, optimization practitioners and others (e.g. economists) can easily compare epidemiological models, costs functions and optimization algorithms to address important choices to be made by health decision-makers.

Under review as a conference paper at ICLR 2021
EPIDEMIOPTIM: A TOOLBOX FOR THE OPTIMIZATION OF CONTROL POLICIES IN EPIDEMIOLOGICAL MODELS
A Anonymous authors
Paper under double-blind review
ABSTRACT
Epidemiologists model the dynamics of epidemics in order propose control strategies based on pharmaceutical and non-pharmaceutical interventions (contact limitation, lock down, vaccination, etc). Hand-designing such strategies not trivial because of the number of possible interventions and the difficulty to pre- dict long-term effects. This task can be cast as an optimization problem where state-of-the-ard machine learning algorithms such as deep reinforcement learn- ing. might bring significant value. However, the specificity of each domain epidemic modelling solving optimization problem requires strong collabo- rations between researchers from different fields of expertise. This is why we introduce EpidemiOptim, Python toolbox that facilitates collaborations between researchers in epidemiology and optimization. EpidemiOptim turns epidemiolog- ical models and cost functions into optimization problems via a standard inter- face commonly used by optimization practitioners (OpenAl Gym). Reinforcement learning algorithmss based on Q-Learning with deep neural networks (DQN) and evolutionary algorithms (NSGA-II) are already implemented. We illustrate the use of EpidemiOptim to find optimal policies for dynamical on-off lock-down control under the optimization of death toll and economic recess using a Susceptible- Exposed-Infectione inneetiouu-Remooedd (SEIR) model for COVID-19. Using EpidemiOp- tim and its interactive visualization platform in Jupyter notebooks, epidemiolo- gists, optimization practitioners and others (e.g. economists) can easily compare epidemiological models, costs functions and optimization algorithms to address important choices to be made by health decision-makers.
INTRODUCTION
The recent COVID-19 pandemic highlights the destructive potential of infectious diseases in our societies, especially on our health, but also our economy. To mitigate their impact, scientific understanding of their spreading dynamics coupled with methods quantifying the impact of inter- vention strategies along with their associated uncertainty, are key support and optimize informed policy making. For example, in the COVID-19 context, large scale population lock-downs were enforced based analyses and predictions from mathematical epidemiological models (Ferguson) et al. 2005) 2006) Cauchemez et al., 2019 Ferguson et al.) (020). In practice, researchers often con- sider small number of relatively coarse and pre-defined intervention strategies, and run calibrated epidemiological models to predict their impact (Ferguson et al. 2020). This is a difficult problem for several reasons: 1) the space of potential strategies can be large, heterogeneous and multi-scale (Halloran et al., 2008): 2) their impact on the epidemic is often difficult to predict; 3) the problem multi-objective by essence: often involves public health objectives like the minimization of the death toll or the saturation of intensive care units, but also societal and economic sustainability. For these reasons, pre-defined strategies are bound to be suboptimal. Thus, a major challenge consists in leveraging more sophisticated and adaptive approaches to identify optimal strategies.
Machine learning can be used for the optimization of such control policies, with methods ranging from deep reinforcement learning multi-objective evolutionary algorithms. In other domains, they have proven efficient at finding robust control policies, especially in nigh-dimensional non- stationary environments with uncertainty and partial observation of the state of the system (Deb) ett al., 2007; Mnih et al., 2015) Silver et al. 2017]] Haarnoja et al. 2018: Kalashnikov et al.) 2018 Hafner al. 2019). Yet, researchers in epidemiology, in public-health, economies, and in ma-

---

Under review as a conference paper at ICLR 2021
chine learning evolve in communities that rarely cross, and often use different tools, formalizations and terminologies. We believe that tackling the major socictal challenge of epidemic mitigation re- quires interdisciplinary collaborations organized around operational scientific tools and goals, that can be used and contributed by researchers of these various disciplines. To this end, we introduce EpidemiOptim, Python toolbox that provides framework to facilitate collaborations between researchers in epidemiology, economics and machine learning.
EpidemiOptim turns epidemiological models and cost functions into optimization problems via the standard OpenAl Gym (Brockman et al. 2016) interface that is commonly used by optimization practitioners. Conversely, it provides epidemiologists and economists with an easy-to-use access to a variety of deep reinforcement learning and evolutionary algorithmss, capable of handling different forms of multi-objective optimization under constraints. Thus, FpidemiOptin facilitates the inde- pendent update of models by specialists of each topic, while enabling others to leverage implemented models to conduct experimental evaluations. We illustrate the use of EpidemiOptim to find optimal policies for dynamical on-off lock-down control under the optimization of death toll and economic recess using an extended Seceptible-Exppssee Exposed-Infectiouss SereectiousRRemove (SEIR) model for COVID-19 from Prague et al. (2020).
Related Work. We can distinguish two main lines of contributions concerning the optimization of intervention strategies for epidemic response. On the one hand, several contributions focus on providing guidelines and identifying the range of methods available to solve the problem. For exam- ple. Yáñez et al. (2019) framed the problem of finding optimal intervention strategies for disease spread as reinforcement learning problem: /Alamo et al. (2020) provided a road-map that goes from the access to data sources to the final decision-making making step; and Shearer et al. (2020) highlighted that a decision model for epidemic response cannot capture all of the social, political, and ethical considerations that these decisions impact. These contributions reveal major challenge for the com- munity: developing tools that can easily used, configured and interpreted by cetiiiionmmakess On the other hand, computational contributions proposed actual concrete implementations of such opti- mization processes. These contributions mostly differ by their definition of epidemiological models (e.g. SEIR (Yaesoubi et al. 2020) agent-based models (Chandak et al., 2020). of optimization methods (e.g. deterministic rules Tarrataca et al.) (020). Bayesian optimization (Chandak et al.)
2020). Deep RL (Arango & Pelov 2020) or evolutionary optimization (Miikkuliinen et al.) 2020 )
of cost functions (e.g. fixed weighted sum of health and economical costs (Arango & Pelov 2020). possibly adding constraints on the school closure budget (Libin et al.] 2020), multi-objective op- timization (Müikulliinen et al., (0200), of state and action spaces (c.g. using the entire observed epidemic history (Yaesoubi et al. 2020) or an image of the disease outbreak to capture the spa- tial relationships between locations (Probert et al., 2019)), as well as methods for representing the model decisions in a format suitable to decision makers (e.g simple summary state representation (Probert et al.) 2019) real-time surveillance data with decision rules (Yaesoubi & Cohen, 2016)). See Appendix A.2 for a detailed description of the aforementioned papers.
Given this high diversity of potential methods in the field, our approach aims at providing a standard toolbox facilitating the comparison of different configurations along the aforementioned dimensions in order to assist drcision-makerr in the evaluation of the range of possible intervention strategies.
Contributions. This paper makes three contributions. First, we formalize the coupling of epidemi- ological models and optimization algorithms with a particular focus on the multi-objective aspect of such problems (Section 2). Second, based on this formalization, we introduce the EpidemiOptim library, a toolbox that integrates epidemiological models, cost functions, optimization algorithms and experimental tools to easily develop, study and compare epidemic control strategies (Section 3 Third, we demonstrate the utility of the EpidemiOptim library by conducting a case study on the op- timization of lock-down policies for the COVID-19 19 epidemic (Section 4.. We use a recent epidemi- ological model grounded on real data, cost functions based on a standard economical model of GDP loss, as well as state-of-the-art optimization algorithms, all included in the EpidemiOptim library. This is, to our knowledge, the first contribution that provides comparison of different optimiza- tion algorithm performances for the control of intervention strategies on the same epidemiological model. The user of the EpidemiOptim library can interact with the trained policies via Jupyter note- book, exploring the space of cost functions (health cost x economic cost) for a variety of algorithms. The code is made available anonymously at https: / /ttiinnnrrrr com/ ep demi demiopt.51 opt.im
2

---

Under review as a conference paper at ICLR 2021
2 THE EPIDEMIC CONTROL PROBLEM AS AN OPTIMIZATION PROBLEM
In an epidemic control problem, the objective is to find an optimal control strategy to minimize some cost (e.g. health and/or economical cost) related to the evolution of an epidemic. We take the approach of reinforcement learning (RL) (Sutton & Barto 2018]: the control policy is seen as learning agent, that interacts with epidemiological model that is seen as its learning environment, see Figure [ Each of the epidemic is a learning episode, where the agent alternatively interacts with the epidemic through actions and observes the resulting states of the environment and their associated costs. To define this optimization problem, we define three elements: 1) the state and action spaces; 2) the epidemiological model; 3) the cost function.
Epidemiological
Policy
model
(ra)
Learning algorithm
Trajectory
C2
Cost functions
collector
Agent
Environment
Figure I: Epidemic control as an optimiza- tion problem. S. C refer to environment states, control actions, and the 1t cost, while traj. is their collection over an episode. Blue and red arrows match the input and output of the OpenAl Gym step function.
State and action spaces. The state 8, describes the information the control policy can use (e.g. epidemiological model states, cumulative costs up to timestep etc.) We also need to define which actions a can be exerted on the epidemic, and how they affect its parameters. For instance, enforcing a lock-down can reduce the transmission rate of the epidemic and slow it down.
The epidemiological model. The epidemiological model is akin to transition function T: it governs the evolution of the epidemic. From the current state of the epidemic and the current action d₂ exerted by the agent, it generates the next state 81+1: T S,A + 8 where A,A are the state and action spaces respectively. Note that this mapping can be stochastic.
Cost functions and constraints. The design of cost functions central to the creation of opti- mization problems. Epidemic control problems can often be framed multi-objective problems, where algorithms need to optimize for a set of cost functions instead of a unique one. Costs can be defined at the level of timesteps C(s, a) or the level of an episode C. (trajecttory) This cumulative cost measure can be obtained by summing timestep-based costs over an episode. For mathematical convenience, RL practitioners often use discounted sum of costs, where future costs are discounted exponentially: C. (trajectory) = >, 7² c($2,a;) with discount factor (e.g. it avoids infinite sums when the horizon infinite). When faced with multiple cost functions C2 Cilegii..N.) a simple ap- proach is to aggregate them into unique cost function C, computed as the convex combination of the individual costs: C(s, a) = Σ, Br (s, a). Another approach to optimize for the Pareto front, i.e. the set of non-dominated solutions. A solution is said to be non-dominated if no other solution performs better on all costs and strictly better on at least one. In addition, one might want to define constraints on the problem (e.g. a maximum death toll over year).
The epidemic control problem as a Markoy Decision Process. Just like traditional RL prob- lems, our epidemic control problem can be framed as Markov Decision Process: M A, Po, C, 7). where po is the distribution of initial states, see Yáñez al. (219)) for a similar formulation.
Handling time limits. Epidemics are not episodic phenomena, or least it is difficult to predict their end in advance. Optimization algorithms, however, require episodes of finite lengths. This enables the algorithm to regularly obtain fresh data by trying new policies in the environment. This can be a problem, as we do not want the control policy to act as if the end of the episode was the end of the epidemic: the epidemic is growing rapidly right before the end, we want the control policy to care and react. Evolution algorithms (EAs) cannot handle such behavior as they use episodic data (9e, C.). where 9. are the policy parameters used for episode e. RL algorithms, however, can be made time unaware. To this end, the agent should not know the current timestep, and should not be made aware of the last timestep (termination signal). Default RL implementations often ignore this fact, see discussion of time limits in Pardo et al. (0018)
3

---

Under review as a conference paper at ICLR 2021
3 THE EPIDEMIÖPTIM TOOLBOX
3.1 TOOLBOX DESIDERATA
The EpidemiOptim toolbox aims facilitating collaborations between the different fields interested in the control of epidemics. Such library should contain state-of-the-art epidemiological models, optimization algorithms and relevant cost functions. It should enable optimization practitioners to bring their newly-designed optimization algorithm, to plug it to interesting environments based on advanced models grounded in epidemiological knowledge and data, and to reliably compare it to other state-of-the-arr optimization algorithms. On the other hand, it should allow epidemiologists to bring their new epidemiological model, reuse existing cost functions to form a complete epi- demic control problem and use of-t--oftthearr optimization algorithms to solve it. This requires modular approach where algorithms, epidemiological models and cost functions are implemented independently with standardized interface. The toolbox should contain additional tools to facilitate experiment management (result tracking, configuration saving, logging). Good visualization tools would also help the interpretation of results, especially in the context of multi-objective problems. Ideally, users should be able to interact with the trained policies, and to observe the new optimal strategy after the modification of the relative importance of costs in real-time. Last, the toolbox should enable reproducibility (seed management) and facilitate statistical comparisons of results.
3.2 TOOLBOX ORGANIZATION
The EpidemiOptim toolbox is centered around two modules: the environment and the optimization algorithm, see Figure To run an experiment, the user can define its own or select the ones contained in the library. Appendix Figure X presents the main interface of the EpidemiOptim library. The following sections delve into the environment and algorithm modules.
Overview.
Environment: epidemiological models. The Model module wraps around epidemiological mod- els, and allows to sample new model parameters from a distribution of models, to sample initial conditions (pp) and to run the model for n steps. Currently, the EpidemiOptim library contains Python implementation of an extended Susceptible-Exposee fitted to French data, from Prague et al. (2020).
Iereedd-neeecc--eemeeee Eiseetiile-pppppeeeeccccoooooss (SEIR) model
Environment: cost functions and constraints. In the EpidemiOptim toolbox, each cost function is a separate class that can compute ca a)) normalize the cost to [0. IJ and generate constraints on its cumulative value. The Multi-Cost module integrates several cost functions. computes the list of costs (s, a)]sE(L..NN.] and an aggregated cost è parameterized by the mixing weights Beleejt..N.. The current version of EpidemiOptim contains two costs: 1) a health cost computed as the death toll of the epidemie; 2) an economic cost computed as the opportunity loss of GDP resulting from either the epidemic itself (illness and death reduce the workforce), or the control strategies (lock- down also reduces the workforce). In addition, we define two constraints as maximal values that the cumulative measures of the two costs described above can take.
OpenAl Gym interface universal interface. The learning environment de- fines the optimization problem. We use a framework that has become standard in the Reinforcement Learning community in recent years: the OpenAl Gym environment (Brockman al. 2016). Gym environments have the following interface: First, they have a resel function that resets the environ- ment state, effectively starting a new simulation or episode. Second, they have a step function that takes as input the next action from the control policy. updates the internal state of the model and returns the new environment state, the cost associated to the transition and a Boolean signal indicat- ing whether the episode is terminated, see Figure [ This standardized interface allows optimization practitioners (RL researchers especially) to easily plug any algorithm, facilitating the comparison of algorithms on diverse benchmark environments. In our library. this class wraps around an epidemi- ological model and a set of cost functions. It also defines the state and action spaces S, A. So far, contains the EpidemicDisscrtee- environment, a Gym-like environment based on the epidemiolog- ical model and the bi-objective cost function described above. Agents decide every week whether to enforce partial lock-down (binary action) that results in decreased transmission rate.
Environment:
4

---

Under review as a conference paper at ICLR 2021
Optimization algorithms. The optimization algorithm, or algorithm for short, trains learning agents to minimize cumulative functions of costs. Learning agents are characterized by their control policy: function that takes the current state of the environment as input and produces the next action to perform: TT : S + A where TT is parameterized by the vector 0. The objective of the algorithm is, thus, find the optimal set of parameters 6* that minimizes the cost. These algorithms collect data by letting the learning agent interact with the environment. Currently, the EpidemiOp- tim toolbox integrates several RL algorithms based on the famous Deep Q-Network algorithm (Mnih) et al.
2015 as well as the state-of-theear arr multi-objeccive EA NSGA-II Deb et al. (2002).
Utils. We provide additional tools for experimentation: experience management, logging and plots. Algorithms trained on the same environment can be easily compared, e.g. by plotting their Pareto front on the same graph. We also enable users visualize and interact with trained policies via Jupyter notebooks. Users can select solutions from the Pareto front, modify the balance between different cost functions and modify the value of constraints set to the agent via sliders. They visual- ize run of the resulting policy in real-time. We are also committed to reproducibility and robustness of the results. For this reason, we integrate to our framework a library for statistical comparisons designed for RL experiments (Colas et al. 2019).
4 CASE STUDY: LOCK-DOWN ON-OFF POLICY OPTIMIZATION FOR THE COVID-19 EPIDEMIC
The SARS-CoV-2 virus was identified on January. 7th 2020 as the cause of new viral disease named COVID-19. As rapidly spread globally, the World Health Organization (2020) declared pandemic on March 11th 2020. Exceptional measures have been implemented across the globe as an attempt to mitigate the pandemic (Kraemer et al., 2020). In many countries, the government response
total lock-down: self-isolation with social distancing, schools and workplaces clo- sures, cancellation of public events, large and small gatherings prohibition, and travel limitation. In France, lock-down lasted 55 days from March, 17th May, 11th It led to decrease in gross domestic product of, at least, 10.1% % (Mandel & Veetil 2020) but allowed to reduce the death toll by an estimated 690,000 [570,000; 820,000) people (Flaxman al. (2220). In this case study, investigate alternative strategies optimized by RL and EA algorithms over a period of I year.
4.1 ENVIRONMENT
The epidemiological model. We use mathematical structural model understand the large-scale dynamics of the COVID-19 epidemic. We focus on the extended Srreptible-Expoeede Inferctooo-- Removed (SEIR) model from Prague al. (2020). which was fitted to French data. The estimation in this model was region-specific. Here, we use their estimated parameters for the epidemic dy- namies in fl-de-France (Paris and its surrounding region), presented Appendix Table | The model explains the dynamics between Susceptibles (S), Exposed (E), Ascertained Infectious (T), Removed (R), Non-Ascertained Asymptomatic (A) and Hospitalized (H). The underlying system of differential equations and schematic view of the mechanistic model can be found in Appendix Section A.3 To account for uncertainty the model parameters, we create distribution of models resulting from parameter distributions, see details in Appendix Section A.3 and a visualization of 10 sampled models in Appendix Section A.3.2
State and action spaces. The state of the agent is composed of the epidemiological states (SEIRAH) normalized by the population size (N = 12, 278, 210), booleans indicating whether the previous and current state are under lock-down, the two cumulative costs and the level of the transmission rate (from to 4). The two cumulative costs are normalized by N and 150B respec- tively. 150B being the approximate maximal cost due to full year of lockdown. The action occurs weekly and is binary (1 for lock-down, 0 otherwise). Consecutive weeks of lockdown decrease the b in four stages (see model deseription in Appendix Figure 5 while stopping lockdown follows the opposite path (stair-case increase of b).
Cost functions. In this case-study, an optimal policy should minimize two costs: health cost and an economic cost. The health cost is computed as the death toll, it can be evaluated for each environment transition (weekly) or over full-episode (e.g. annually): Ca(() = D(I) = 0.005/((1).
5

---

Under review as a conference paper at ICLR 2021
The economic cost is measured in euros the GDP opportunity cost resulting from diseased dead workers, as well as people unemployed due to the lock-down policy. At any time t. the GDP can be expressed as a Cobb- Sobb-lagllas function F F([(t)) economy before the outbreak that we will suppose constant during the pandemics, L(t) is the active population employed, "Yk is the capital elasticity and A is the exogenous technical progress (Douglas) 1976). Let Lg = AN be the initial employed population, where À is the activity/employment rate. The lock-down leads to partial unemployment and decreases the size of the population able to work (illness, isolation, death), thus L(t) (1 (( - G(t)), where u(t) is the level of partial unemployment due to the lock-down and G(t) I(t) + (t) + 0.005 is the size of the ill, isolated or dead population as defined in our SEIRAH model. The economic cost is defined as the difference between the GDP before (Y)) and after the pandemic, Yo- - F(L(1)) and given by:
AK2 L(t) -(t)->>, where Ko is the capital stock of
Ceco = Yo- A KKO ((1 - =(O))((N -G(1)
Parameters for values of parameters in the economic model are given in Table We also consider the use of constraints on the maximum value of cumulative costs.
4.2 OPTIMIZATION ALGORITHMS
We consider three algorithms: 1) a vanilla implementation of Deep Q-Network (DQN) (Mnih al.
2013); 2) goal-conditioned version of DQN inspired from (Schaul et al. 2015); 3) NSGA-II, state-of-the-ard multi-objective evolutionary algorithm (Deb et al. 2002). The trained policy the same for all algorithms: neural network with one hidden-layer of size 64 and ReLU activations. Further background about these algorithm is provided in Appendix SectionA.3
Optimizing convex combinations of costs: DQN and Goal-DQN. DQN only targets one cost function: ج = (1 B) G + 33 Ceca- Here the health and economic costs are normalized to have similar ranges (scaled by 1/(65 1c3) and 1/1e9 respectively). We train independent policies for each values of 3 in [0., 0.05,.. .., I[. This consists in training a Q-network to estimate the value of performing any given action in any given state s, where the value is defined as the cumulative negative cost - expected in the future. Goal-DQN, however, trains one policy [ss 33 to target any convex combination parameterized by B. We use the method presented in (Badia et al. 2020): we train two Q-networks, one for each of the two costs. The optimal action is then selected as the one maximizing the convex combination of the two Q-values:
[s, .) argmax, & (1 (3) (x( + 1 (8)
(1)
By disentangling the two costs, this method facilitates the representation of values that can have different scales and enables automatic transfer for any value of 3, see justifications in Badia et al. 2020 During training, agents sample the targeted 3 uniformly in [0, 1].
Adding constraints with Goal DQN-C. We design and train variant of Goal DQN handle constraints on maximal values for the cumulative costs: Goal-DQN-C. We train a Q-network for each constraints with cost of 1 each time the constraint is violated, 0 otherwise (Og, Q...). With discount factor = 1, this network estimates the number of transitions that are expected to violate the constraints in the future. The action is selected according
Eq. among the actions that are not expected to lead to constraints violations (Q" a) < 1). If all actions are expected to lead to violations, the agent selects the action that minimizes that violation. During training, agents sample 3 uniformly, and 50% of the time samples uniformly one of the two constraints in (1000, 62000) for the maximum death toll and [20, 160) for the maximum economic cost, in billions.
EAs to optimize a Pareto front of solutions. We also use NSGA-II, a state-of-the-an multi- objective EA algorithm that trains a population of solutions to optimize a Pareto front of non- dominated solutions (Deb et al. 2002). As others EAs, NSGA-II uses episodic data (0e, C.). To obtain reliable measures of Ce in our stochastic environment, we average costs over 30 evaluations. For this reason. NSGA-II requires more samples than traditional gradient-based based algorithms. For fair comparison, we train two NSGA-II algorithm: one with the same budget of samples than DQN variants (le6 environment steps), and the other with 15x as much (after convergence).

---

Under review as a conference paper at ICLR 2021
4.3 RESULTS
Pareto fronts. We aim at providing decision tools to decision-maker and, for this reason, cannot decide ourselves the right value for the mixing parameter B. Instead, we want present Pareto fronts of Pareto-optimal solutions. Only NSGA-II aims at produeing an optimal Pareto front as its result while DQN and Goal DQN optimize unique policy. However, one can build Pareto fronts even with DQN variants, We can build population of policies by aggregating independenlty trained DQN policies, or evaluating single Goal DQN policy on a set of Npureto goals sampled uniformly (100 here). The resulting population can then be filtered to produce the Pareto front. Figure 2a presents the resulting Pareto fronts for all variants. The Pareto fronts made of several DQN policies (yellow) perform better in the regime of low health cost (< 10 deaths) where it leads to save around 20 B euros over NSGA-II and the Goal DQN without constraint. Over 25 x 103 deaths, NSGA-II becomes the best algorithm, as it allows to save 10 to 15 B euros in average, the health cost being kept constant.
1.0
Health cost
Economic
H566-1 (Budget GOAL DQMC
cost (B)
2.50
0.8
0.6-
100
(%
50-
0
10
20
30
40
50
60
0.8
0.2
0.4
0.6
0.5
1.0
Deaths (x10*)
(a)
(b)
Figure 2: Left: Pareto fronts for various optimization algorithms. The closer to the origin point the better. Right: Evolution of the costs when Goal DQN agents are evaluated on various B. We plot the mean and standard error of the mean (left) and standard deviations (right) over 10 seeds.
In the low health cost regime, the DQN policy attempts to break the first wave of the epidemic, then maintains low with periodic lock-down (1 or 2 weeks period) until it is sufficiently low. If a second wave appears, it goes back to that strategy, see Figure for an example. Goal DQN with constraints shows a similar behavior. NSGA-II enters a cyclic 1-week period of lock-down but does not stop even when the number of new cases appears null (Appendix Section A.4p. In the low economic cost regime (below 15 B), NSGA-II waits for the first wave to grow and breaks it by a lock-down of a few weeks (Figure 3bp. DQN- based algorithms sometimes find such policies. Policies in the range of health cost between 102 and 50 x 10x deaths are usually policy that perform low or high health costs depending the particular epidemic and its initial state. Appendix Section 4.3 delves into the different strategies and interesting aspects of the four algorithms.
Optimizing convex combinations. When B is low (respectively high), the cost function is closer to the health (respectively economic) cost (see Eq [T. Figure 2b presents the evolution of the two costs when trained Goal DQN policy (without constraints) is evaluated different B. As expected, the higher 33 the lower the health cost and vice-versa. We see that, outside of the [0.2, 0.8) range, the policy falls in extreme regimes; either it always locks-down (low () or it never does (high 8))
Interactive decision making. The graphs presented above help make sense of the general per- formance of the algorithms, in average over multiple runs. However, it is also relevant to look at particular examples to understand a phenomenon. For this purpose, we include visualization Jupyter Notebooks in the EpidemiOptim toolbox. There are three notebooks for DQN, Goal DQNs and NSGA-II respectively. The two first one help making sense of DQN and Goal DQN algorithms (with or without constraints). After loading a pre-trained model, the user can interactively act on the cost function parameters (mixing weight 3 or constraints on the maximum values of the cumu- lative costs). As the cost function changes, the policy automatically adapts and the evolution of the epidemic and control strategy are displayed. The third notebook helps visualizing a Pareto front of the NSGA-II algorithm. After loading a pre-trained model, the user can click on solutions shown on the Pareto front, which automatically runs the corresponding model and displays the evolution of the epidemic and control strategy. The user can explore these spaces and visualize the strategies that correspond to the different regimes of the algorithm.
7

---

Under review as a conference paper at ICLR 2021
Eco cest: 59.52 B, Death Cast: 6023, Aggregated Cost: 37.17
Eco cost: 13.09 B, Death Cost: 41216, Aggregated Cost: 38.25
Cont
Tranemiesion rate Aggragated Cont
Deaths
2006
Dcathe
9.4
New Deaths
Deathe
10o
20c0
0-2
2000
20co
New GDP Loog (8)
Total GDP Loss (8)
Transmii rate
New GDP Lowe (8)
0.4
Total CDP Loss
80
106
100
days
(a)
(b)
Figure 3: Evolution of cost metries and strategy over a year of simulated epidemic with DQN agent (left) in the low health cost regime and a NSGA-II agent in the low economic cost regime (right),
Discussion. This case study proposed to use set of state-of-the-art algorithms to target the prob- lem of lock-down policy optimization in the context of the COVID-19 epidemic (RL and EA algo- rithms). DQN appears to work better in the regime of low health cost, but requires to train multiple policies independently. As trains only one policy, Goal DQN is faster, but does not bring any per- formance advantage. NSGA-II, which is designed for the purpose of multi-objective optimization, seems perform better in the regime of low economic cost.
DISCUSSION & CONCLUSION
On the use of automatic optimization for decision-making. We do not think that optimization algorithms should ever replace decision- makers, especially in contexts involving the life and death of individuals such as epidemics. However, we think optimization algorithms can provide decision- makers with useful insights by taking into account long-term effects of control policies. To this end, we believe it is important consider a wide range of models and optimization algorithms to target same control problem. This fosters diversity of methods and reduces the negative impacts of model- and algorithm-inducen biases. The EpidemiOptin toolbox is designed for this purpose: provides a one-stop shop centralizing both the modeling and the optimization in place.
The future of EpidemiOptim. This toolbox is designed as a collaborative toolbox that will be SEIR epidemiological models
extended in the future. To this day, it contains one population- and three optimization algorithms. In the future, we plan on extending this library via collaborations with other research labs in epidemiology and machine learning. We plan on integrating agent-based epidemiological models that would allow broader range of action modalities, and to add a variety of mujti-ohjective algorithms from EAs and RI. We could finally imagine the organization of chal- lenges, where optimization practitioners would compete on set of epidemic control environments.
Although this toolbox focuses on the use of optimization algorithm to solve epidemic control poli- cies, similar tools could be used in other domains. Indeed, just like epidemics, large diversity of dynamical systems can be modeled models by ordinary differential equations (e.g. in chemistry, biology, economics, etc.). One could imagine spin-offss of EpidemiOptin for applications other domains. Furthermore, Chen et al. (2018) recently designed a differentiable ODE solver. Such solvers could be integrated to EpidemiOptim to allow differential learning through the model itself.
Conclusion. This paper introduces EpidemiOptim, toolbox that facilitates collaboration between epidemiologists, optimization practitioners and others in the study of epidemic control policies to help decision decision-making. Of course, others have studied the use of optimization algorithms for the control of epidemics. We see this as a strength, as shows the relevance of the approach and the importance of centralizing this research in common framework that allows easy and reproducible comparisons and visualizations. Articulating our understanding epidemies evolution with the so- cietal impacts of potential control strategies on the population (death toll, physical and psychological traumas, ett) and on the economy will help improve the control future epidemics.
8

---

Under review as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
REFERENCES
Teodoro Alamo, Daniel G Reina, and Pablo Millán. and control covid- 19 pandemic: Leveraging data science, epidemiology and control theory. arXiv preprint wrXiv:2006.0173/ 2020.
Data-driven methods monitor, model, forecast
Mauricio Arango and Lyudmil Pelov. Covid-19 pandemic cyclic lockdown optimization using rein- forcement learning. arXiv preprint arXiv:2009.0400 2020.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. arXiv preprint arXiv:2003. /3350000..33., 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1606.0 2016.
Simon Cauchemez, Nathanaël Hoze, Anthony Cousien, Birgit Nikolay, al. How modelling can enhance the analysis of imperfect epidemic data. Trends in parasitology, 55(5):369-379) 2019.
Amit Chandak, Debojyoti Dey, Bhaskar Mukhoty, and Purushottam Kar. Epidemiologically and eociiecconomically optimal policies via bayesian optimization. arXiv preprint 2rxi:00000::000002155 2020.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Cédric Colas, Olivier Sigaud, and Pierre- Oudeyer. A hitchhiker's s guide to statistical compar- isons reinforcement learning algorithms. arXiv preprint arXiv:/904.000 190499697, 2019.
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist mul- tiobjective genetic algorithm: Nsga-ii. IEEE transactions evolutionary computation, 6(2): 182-197, 2002.
Kalyanmoy Deb, S Karthik, et al. Dynamic multi-objective optimization and decision-making using modified nsga-ii: a case study on hydro-thermal power scheduling. In International conference on evolutionary multi-criterion optimization, pp. 803-817. Springer, 2007.
Paul H. Douglas. The cobb- production function once again: Its history, its testing, and some new empirical values. Journal of Political Economy, 84(5)-903-915, 1976. doi: 10. 1086//60888. URL
Neil Ferguson, Daniel Laydon, Gemma Nedjati Gilani, Natsuko Imai, Kylie Ainslie, Marc Baguelin, Sangeeta Bhatia, Adhiratha Boonyasiri, ZULMA Cucunuba Perez, Gina et al. Report 9: Impact of non-pharmaceutical interventions (npis) to reduce 1 mortality and healthcare demand. 2020.
Neil M Ferguson, Derek AT Cummings. Simon Cauchemez, Christophe Fraser, Steven Riley, Aron- rag Mecyai, Sopon lamsirithaworn, and Donald Burke. Strategies for containing an emerging influenza pandemic in southeast asia. Nature, 437(7056):209-211 2005.
Neil M Ferguson, Derek AT Cummings, Christophe Fraser, James C Cajka, Philip C Cooley, and Donald S Burke. Strategies for mitigating an influenza pandemic. Nature, 442(7101):448 452, 2006.
Seth Flaxman, Swapnil Mishra, Axel Gandy, Juliette T Unwin, Thomas A Mellan, Helen Cou- pland, Charles Whittaker, Harrison Zhu. Tresnia Berah, Jeffrey W Eaton, et al. Estimating the effects of non-pharmaccutical interventions on covid-19 19 in europe. Nature, 584(782002577-61, 2020.

---

Under review as a conference paper at ICLR 2021
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of ICML, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In Proceedings of ICLR, 2019.
M Elizabeth Halloran, Neil M Ferguson, Stephen Eubank, Ira M Longini, Derek AT Cummings, Bryan Lewis, Shufu Xu, Christophe Fraser, Anil Vullikanti, Timothy Germann, et al. Modeling targeted layered containment of influenza pandemic in the united states. Proceedings of the National Academy of Sciences, 105(12):46394466 2008.
Karel Havik, Kieran Mc Morrow, Fabrice Orlandi, Christophe Planas, Rafal Raciborski, Werner Röger, Alessandro Rossi, Anna Thum-Thysen, Valerie Vandermeulen, et al. The production function methodology for calculating potential growth rates & output gaps. Technical report, Directorate General Economic and Financial Affairs (DG ECFIN), European... .... 2014.
Insee. Insee website. https: htapps://www.. www. insee. fr/
2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, Scalable deep reinforce- ment learning for vision-basee robotic manipulation. In Conference on Robot Learning, pp. 651- 673, 2018.
Moritz U. G. Kraemer, Chia-Hung Yang. Bernardo Gutierrez, Chieh-Hsi Wu, Brennan Klein, David M. Pigott, Louis du Plessis, Nuno R. Faria, Ruoran Li, William P. Hanage, John Brown- stein, Maylis Layan, Alessandro Vespignani, Huaiyu Tian, Christopher Dye, Oliver G. Pybus, and Samuel V. Scarpino. The effect of human mobility and control measures the COVID-19 epidemic in China. Science, 368(6490):493 497, 2020. doi: 10.1 (0.112//cciecccaabb2llk.
Pieter Libin, Arno Moonens, Timothy Verstracten, Fabian Perez-Sanjines, Niel Hens, Philippe Lemey, and Ann Nowé. Deep reinforcement learning for large-scale epidemic control. arXiv preprint arXiv:2003.13666 2020.
Antoine Mandel and Vipin Veetil. The economic cost of covid lockdowns: An out-of-cqullibrium analysis. Economics of Disasters and Climate Change, 2020. doi: 10.2139/ssm.3588421..
Risto Müikkulaieen, Olivier Francon, Elliot Meyerson, Xin Qiu, Elisa Canzani, and Babak Hodjat. From prediction to prescription: Ai-based optimization of non-pharmaceutical interventions for the covid-19 pandemic. arXiv preprint arXiv:2005. 13766, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529) 533, 2015.
Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. In International Conference Machine Learning, pp.
2018.
Mélanie Prague, Linda Wittkop, Quentin Clairon, Dan Dutartre, Rodolphe Thiébaut, and Boris Pierre Hejblum. Population modeling of early covid-19 19 epidemic dynamics in french re- gions and estimation of the lockdown impact on infection rate. medRxiv, 2020.
J. M. Probert, S. Lakkur, J. Fonnesbeck, K. Shea, M. C. Runge, M. Tildesley, and M. J. Fer- rari. Context matters: using reinforcement learning develop human-readable, dtate-dependent outbreak response policies. Philosophical Transactions of the Royal Society B: Biological Sci- ences, (74(3777))7101888887 2019. doi: 10.1009//sth.2018..m
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320, 2015.
Freya M Shearer, Robert Moss, Jodie Mc Vernonn, Joshua V Ross, and James M McCaw. Infectious disease pandemic planning and response: Incorporating decision analysis. PLoS medicine. 17(1): e1000118, 2020.
10

---

Under review as a conference paper at ICLR 2021
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Luís Tarrataca, CM Dias, Diego Barreto Haddad, and EF Arruda. Flattening the curves: on-off lock-down strategies for covid-19 with an application to brazi. arXiv preprint arXiv:2004 2020.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-22)) 1992.
World Health Organization. Who director-general's opening remarks at the media briefing on COVID-19 11 march 2020, 2020. Accessed: 2020-04-02.
Reza Yaesoubi and Ted Cohen. Identifying cost-effective dynamic policies to control epidemics. Statistics in medicine, 35(28):5189-5222 2016.
Reza Yaesoubi, Joshua Havumaki, Melanie H Chitwood, Nicolas A Menzies, Gregg Gonsalves, Joshua Salomon, A David Paltiel, and Ted Cohen. Adaptive policies for use of physical distancing interventions during the covid-19 19 pandemic. medRxiv, 2020.
Andrea Yáñez, Conor Hayes, and Frank Glavin. Towards the control of epidemic spread: Designing reinforcement learning environments. In AICS, pp. 188-199, 2019.

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1
EPIDEMIOPTIM INTERFACE
Figure f presents the main interface of the EpidemiOptim library.
epidemiop
epudentog -ype seport: get
epodemioo
epidentoo opp import get ilgoriths
froe epodentoptin cont igs, get perenss seport get,
config
guration
parans get ppw.pasarao...... conttg, (duconfig)
epident oggiccal model
get, (noded, Indeenea [dsparass] "nodel_ [_d" paranseparansl "podet_parans"
functton
cost. functton get, cost,
wwt.ctssttttttiiiioooocoo funct cont functton, Ideparamst "cost_fd"), paronsaparansi "cost_paraws"
Creato the ton proclon as Gyn-Ltkk anvuromeent
got oonv om. (deparass lonv, I. cost_ functtorwcsstffne function, lodole, stx, hiu.horkzonpprraw "stin hor.hoorooo scodeparons] ssood
DON alpor Lthn aeranneerraa by beta
alpor pet_alporitther aleo, (deparans atto_ 1, attwonn, porasseparans)
the tralning Loop
algur Lite Learntuun inaing wnw.nnntesssprrraaca parass Frux, stapp
Figure 4: Main running script of the EpidemiOptim library
A.2 EXTENDED RELATED WORKS
Prior to the current COVID-19 pandemics, Yáñez et al. (2019) framed the problem of finding optimal intervention strategies for disease spread reinforcement learning problem, focusing how to design environments in terms disease model, intervention strategy, reward function, and state representations. In Alamo et al.) (2020), the CONCO-Team (CONtrol COvid-19 Team) provides detailed SWOT analysis (Strengths, Weaknesses, Opportunities, Threats) and a roadmap that goes from the access to data sources the final decision-making step, highlighting the interest of standard optimization methods such as Optimal Control Theory, Model Predictive Control, Multi-oba Mujeivobeective control and Reinforcement Learning. However, as argued [Shaarer al. (2020), a decision model for pandemic response cannot capture all of the social, political, and ethical considerations that is is therefore central challenge to propose tools that can be easily used, configured and interpreted by decision-makers.
impact decision-makinn
The aforementioned contributions provide valuable analyses on the range of methods and challenges involved in applying optimization methods to assist decision making during pandemics events. They do not, however, propose concrete implementations of such systems. Several recent contributions have proposed such implementations, Yacsoubi et al. (2020) applied an approximate policy iteration algorithm of their own (Yaesoubi & Cohen 2016) on SEIR model calibrated on the 1918 Influenza Pandemic in San Francisco. An interesting contribution of the paper the development of pragmatic decision tool to characterize adaptive policies that combined real-time surveillance data with clear decision rules guide when to trigger, continue, stop physical distancing interventions. Arango & Pelov 2020) applied a more standard RL algorithm (Double Deep Q-Network) on a SEIR model of the COVID-19 19 spread to optimize cyclic lock-down timings. The reward function was fixed combination of health objective (minimization of overshoots of ICU bed usage above an ICU bed threshold) and an economic objective (minimization the time spent under lock-downs). They compared the action policies optimized by the RL. algorithm with baseline on/off feedback control (fixed-rule), and with different parameters (reproductive number) of the SEIR model. Libin et al. 2020) used more complex SEIR model with coupling between different districts and age groups. They applied the Proximal Policy Optimization algorithm to learn a joint policy that control the districts using reward function quantifying the negative loss in susceptibles over week, with a constraint on the school closure budget. |robert et al. (2019) used Deep Q-Networks (DQN) and Monte- Carlo control on stochastic, addvidual-basel model of the 2001 mouuhh
simulated
12

---

Under review as a conference paper at ICLR 2021
disease outbreak in the UK. An original aspect of their approach was to define the state at time t using an image of the disease outbreak to capture the spatial relationships between farm locations, allowing the use of convolutional neural networks in DQN. Other contributions applied non-RL optimization methods such as deterministic rules (Tarrataca et al. 2020), stochastic approximation algorithms (Yaesoubi al. 2020) or Bayesian optimization (Chandak et al. 2020). This latter paper also proposes a stochastic agent-based model called VIPER (Virus- (Virus-Individual Socaddaal-Pooc---ee Policy-EnviRonmenn allowing to compare the optimization results on variations of the demographies and geographical distribution of population. Finally, Miikkulaienn et al. (2020) proposed an original approach using Prescription (ESP). In this approach, a recurrent neural network (the Predictor) was trained with publicly available data on infections and NPIs in number of countries and applied to predicting how the pandemic will unfold in them in the future. Using the Predictor, an evolutionary algorithm generated large number of candidate strategies (the Prescriptors) in a multi- objective setting to discover Pareto front that represents different tradeoffss between minimizing the number of COVID-19 cases, as well as the number and stringency of NPIs (representing economic impact).
Evolutionary Surrogate-assistee
As we have seen, existing contributions widely differ in their definition of epidemiological models, optimization methods, cost functions, state and action spaces, as well as methods for representing the model decisions in a format suitable to decision-makers. Our approach aims at providing a standard toolbox facilitating the comparison of different configurations along these dimensions in order to assist -akerrs in the evaluation and the interpretation of the range of possible intervention strategies.
A.3 ADDITIONAL METHODS
A.3.1 OPTIMIZATION ALGORITHMS
The Deep Q-Network algorithm. DQN was introduced in (Mnih et al. 2013) as an extension of the Q-learning algorithm (Watkins & Dayan 1992) for policies implemented by deep neural networks. The objective is to train a Q-function to approximate the value of given state-action pair (s,a) a). This value can be understood the cumulative measure of reward that the agent can expect to collect in the future by performing action now and following an optimal policy afterwards. In the Q-learning algorithm, the Q-function is trained by:
Qsa a4) + Q(st. at) + aafre1 + max, Q($++1, a) - Q(ss,, at)].
Here, a is a learning rate, [re+1 +++1+7 max Q(5 a) Q(ss, at)] is called the temporal difference (TD) error with r++1 + max, Q($++1, a) the target. Indeed, the value of the current state-action pair Q(se. dt) should be equal to the immediate reward r++1 plus the value of the next state Q(*++1, 1, 0e+1) discounted by the discount factor y. Because we assume we behave optimally after the first action, action a++1 should be the one that maximizes ((s++1,0 a). Once this function is trained, the agent simply needs to take the action that maximize the Q-value in its current state: a* = max, Q(3t, a).
Deep Q-Network only brings few mechanisms to enable the use of large policies based on deep neural network see Mnih et al, (2015) for details.
coal-parameterizzed DQNs. Schaul al. 2015) introduced Universal Value Function Approxi- mators: the idea that Deep RL agents could be trained to optimize a whole space of cost functions instead of unique one. In their work, cost functions are parameterized by latent codes called goals. An agent in a maze can decide to target any goal position within that maze. The cost function is then function of that goal. To this end, the agent is provided with its current goal. In the traditional setting, the goal was concatenated to the state of the agent. In a recent work, Badia et al. (2020) used a cost function decomposed into two sub-costs. Instead of mixing the two costs the cost/reward level and learning Q-functions from this signal, they proposed to learn separate Q-functions, one for each cost, and to merge them afterwards: Q(s, a) = (1 3) Q1 a) + 3 Q2(8, a). We use this same technique for our multi-objective problems.
Using constraints, In RL, constraints are usually implemented by adding a strong cost whenever the constraint is violated. However, it might be difficult for the Q-function to learn how to model these sudden jumps in the cost signal. Instead, we take inspiration from the work of Badia et al. 2020). We train separate Q-function for each cost, where the associated reward is -I whenever
13

---

Under review as a conference paper at ICLR 2021
the constraint is violated. This Q-function evaluates how many times the constraint is expected to be violated in the future when following an optimal policy. Once this trained, we can use it to filtrate the set of admissible actions: if constriin (s, a) >> 1, then expect action to lead to at least constraint violation. Once these actions are filtered, the action is selected according to the usual maximization of the traditional Q-function.
NSGA-II. Multi-objective algorithms do not require the computation of an aggregated cost but use all costs. The objective of such algorithms is obtain the best Pareto front. NSGA-II (Deb al. 2002) is state-of-ther -art--ff multi-objective algorithm based on EAs (here genetic algorithm). Start- ing from parent population of solutions, parents are selected by tournament selection involving the rank of the Pareto front they belong to (fitness) and a crowding measure (novelty). Offisring then obtained by cross-over of the parent solutions and mutations of the resulting parameters. The offspring population is then evaluated and sorted in Pareto fronts. A new parent population is finally obtained by selecting offspring from the higher-ranked Pareto fronts, prioritizing novel solutions as second metric (low crowding measures).
A.3.2 EPIDEMIOLOGICA MODEL
Figure presents the schematic view of the mechanistic model behind the SEIRAH model from Prague et al. (2020) (left), and its underlying system differential equations (right). See details in the original paper Prague et al. 2020).
6SII aA) d5
di
N
dE
DSII
+ aA)
dt
N
D.
D.il-d)
dI
rE
1
dB
D,
Dv
D1
dR
(T
A)
(I)
dt
=
A
aA
(I
+)E
dt
D.
dH
I
H
D
DA
by exp(Sen litest week lock-doww
Bat lsecond week lock- dows
0.5%6
+ letied
4 But Istareafeer under lnde-Bown)
Figure Description of the SEIRAH cpidemiological model. Adapted from Prague al. (2020)
To account for uncertainty in the model parameters, the epidemiological model is sampled from that distribution. The transition function is thus condi- tioned on latent code (the model parameters) that is unknown the agent. This effectively results in a stochastic transition from the viewpoint of the agent. To build the distribution of models, we assume normal distributions for each of the model parameters, using either the standard deviation from the model inversion for parameters estimated in PPaague et al. (2020) or 10% of the mean value for other parameters. Those values are available in appendix Figure Tigure/6/featinnn 6ffeatures for reproducibility of the experiment. We further add uniformly distributed delay between the epidemic onset and the start of the learning episode (uniform [0, 21 days]). This delay models the reaction time of the decision authorities. This results in the distribution of models shown in Figure
distribution of models. At each episode,
Epidemiological model and economic cost function parameters. Table T lists the parameters of the epidemiological model (from Prague al. (2020)) and the economic cost function (from the National Institute of Statistics and Economical Analysis Insee (2000) and Havik et al. (0144).
Epidemiological model and cost function parameters. Table presents the list of parameters used in the epidemiological model (top) and the economic cost function (bottom).
14

---

Under review as a conference paper at ICLR 2021
Param.
Interpretation
Value "lle-de-France
bu
Transmission rate of ascertained cases before lock-down
2.23
Ascortainement rate
0.043
Q
Ratio of transmission between and
0.55
De
Latent (incubation) period (days)
5.1
Dr
Infectious period (days)
2.3
Da
Duration from I onset H (days)
0.36
Du
Hospitalization period (days)
30
N
Population size
12,278,210
8
Lock-down effects first (3w1), second (Sw2),
(-0.11, -0.50,
third (3wa) and thereafter weeks (f,,.)
-1.36, -1.46
So
Initial number of susceptible Initial number of exposed
Eg
Io Ro
Ao Ho
Eo
5004
To
Initial numher of ascertained
16
Ro
Initial number of removed
0
Initial number of ascertained Initial number of hospitalized Initial number of death
356
Ho
4
Do
0
Ko
Initial capital stock in millions euros
1,388,122 ,122
Number of employed individuals in millions
4.58
feiivitylemlloyment rate (%)
37.4
Yo
Initial GDP in million euros year
424,474
A
Exogenous technical progress
867
u(t)
Level of partial unemployment during lock-down (%)
50
%
Capital elasticity
0.37
Table Parameter values for the fle-de-France France region for the SEIRAH epidemiological model and the economy cost function.
E
b
Blw
Bzw
Baw
B4w
D
D
Dp
D
E 52492
-0,024
-0,14
0,024
0,020
0,008
0,009
0
0
b
-0,024
0,00001
0
0
0
B 1w -0,144
0,004
0
B 2w 0,024
0,003
0
0
0
0
B 3w 0,020
B 4ww 0,008
0
0
0,0002
D
D
0
0
0
0,26
0
D
Ph
0
D
0
0
0
0
0,05
0
0
0
α
0,003
0
0
0
0
0,00002
Figure 6: Variance-covariance matrix for parameters of the epidemiological model.
15

---

Under review as a conference paper at ICLR 2021
A.4
ADDITIONAL RESULTS
A.4.1 A DISTRIBUTION OF MODELS
presents the evolution of the SEIRAH model states for a few models sampled from the distribution models described in 4.1 Here, no lockdown is enforced.
Figure
Eco cost: 1.85 B, Death Cost: 57649, Aggregated Cost: 45.27
Aggregated Cost
New Deaths
Total Deaths
60000
2000
40000
2:
1000
20000
200
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
Transmission rate
2F
0.015
2.3
0.010
0.005
2.2
0.000
100
200
300
0
100
200 days
300
100
200
300
Figure 7: 10 models sampled from the distribution of epidemiologicaa models used in the case-study.
Now, let illustrate in details the results of the case study. To this end, provide several runs of the different algorithms to illustrate their strategies.
16

---

Under review as a conference paper at ICLR 2021
A.4.2
DQN
We first present strategies found by DQN.
1220
20
400
S (xx109)
1200
E (x104)
15
300
10
200
1180
100
1160
0
0
0
100
200
300
100
200
300
100
200
300
80
8-
R (xx10)
60
(x110)
(xx10)
1.0
40
4-
0.5
20
2-
00.
0.0
0
100
200
300
100
200
300
0
100
200
300
days
Eco cost: 155.16 B, Death Cost: 3513, Aggregated Cost: 5.40
Aggregated Cost
New Deaths
150
Total Deaths
0.4
100
2000
0.2
0
0.0
0
100
200
300
0
100
200
300
100
200
300
New GDP Loss (B)
Transmission rate
Total GDP Loss
150
100
0.4276
0.4274
0
0
100
200
300
0
100
200 days
300
100
200
300
Figure 8: DQN trained with 3 0.. For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of 3 in [0, 0.25, 0.5, 0.75, 1]. For O, the agent only cares about the health cost, it always locks-down.
17

---

Under review as a conference paper at ICLR 2021
1225.0
3
60
S (xx10))
E (x109)
1222.5
2-
40
1220.0
1-
20
1217.5
0
100
200
300
100
200
300
0.20
10
R (x104)
A (x104)
1.0
H (x10%)
0.15
0.10
0.05
0.0g
0.00
100
200
300
0
100
200
300
100
200
300
days
Eco cost: 62.81 B, Death Cost: 556, Aggregated Cost: 31.84
Aggregated Cost
New Deaths
Total Deaths
600
0.4
20
400
10
200
0.2
0.0
100
200
300
100
200
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
Transmission rate
60
1
20
000000000000 100
100
200
days
0.5. Here the strategy is cyclical with a one or two weeks pe- riod. For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of 3 in [0, 0.25, 0.5, 0.75, 1].
Figure 9: DQN trained with 3 =
18

---

Under review as a conference paper at ICLR 2021
300
1220
S (xx10%
E (x10*)
200
1200
5
100
1180
0
0
0
100
200
300
100
200
300
100
200
300
60
0.8
(x109)
A (x109)
(x10%)
0.4
20
0.2
0
100
200
300
100
200
300
100
200
300
days
Eco cost: 59.41 B, Death Cost: 2804, Aggregated Cost: 34.86
Aggregated Cost
New Deaths
100
Total Deaths
3000
0.4
2000
50
0.2
1000
0
0
0.0
100
200
300
100
200
300
100
200
300
New GDP Loss (B)
Transmission rate
0.4
Total GDP Loss
60
0.2
20
0.0
0
1L.
0
100
200
300
100
200 days
300
100
200
300
Figure 10: DQN trained with 8 0.55. Here the agent first lock-down stop the first wave, but then stops the cyclical lockdown early which induces second wave later, where the agent also reacts by a lock-down. For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of B in [0, 0.25. 0.5, 0.75, 1).
19

---

Under review as a conference paper at ICLR 2021
1250
1000
200
4000
S (x10%)
E (x100)
750
500
100
2000
250
0
0
100
100
200
300
1000
(x109)
(x104)
H (x104)
750
10:
500
250
0
0
100
200
300
0
100
200
300
0
100
200
300
days
Eco cost: 1.74 B, Death Cost: 56621, Aggregated Cost: 31.62
Aggregated Cost
New Deaths
2000
Total Deaths
w
40000
2-
1000
20000
0
0
100
200
300
0
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
Transmission rate
1.5
2.3
0.010
1.0
0.005
2.2
0.5
0.000
0.0
100
200
300
0
100
200 days
300
100
200
300
Figure 11: DQN trained with 3 0.65. Here the agent mostly cares about the economic cost, which results in no-lockdown policy. For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost shown for various values of in [0, 0.25, 0.5, 0.75, 1].
20

---

Under review as a conference paper at ICLR 2021
A.4.3
GOAL-DQN WITHOUT CONSTRAINTS
Now we present few strategies found by Goal-DQN without constraints.
1220
S 5x1109)
E (x100)
1210
2-
50
1200
0
0
100
200
300
100
200
300
100
200
300
30
0.6
R- (xx10))
A (xx10)
H (x104)
20
2
0.4
10
1-
0.2
0.0
0
100
200
300
100
200
300
0
100
200
300
days
Eco cost: 92.74 B, Death Cost: 1350, Aggregated Cost: 20.21
Aggregated Cost
New Deaths
60
Total Deaths
0.4
1000
40
0.2
500
0-
0
0.0
100
200
300
0
100
200
300
100
200
300
New GDP Loss (B)
Transmission rate
100
0.4
Total GDP Loss
2-
0.2
50
1-
0.0
0-
100
200
300
0
100
200
300
0
100
200
300
days
0.2. Here the agent starts with lasting lockdown, then pursues with cyclical lock-downs, which ensures the absence of second wave and, thus, results in low health cost but high economic cost. For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values 3 in [0, 0.25, 0.5, 0.75, 1).
Figure 12: Goal-DQN without constraints evaluated
21

---

Under review as a conference paper at ICLR 2021
150
5 (xx10%
1220
[xx104)
6
100
1210
2-
50
1200
0
100
200
300
100
200
300
0
100
200
300
R (x10%)
A (x109)
H (x109)
0.4
2
0.2
1-
00
0.0
200
100
300
100
200
300
days
Eco cost: 59.84 B, Death Cost: 1301, Aggregated Cost: 28.03
Aggregaten Cost
New Deaths
Total Deaths
0.4
1000
40
0.2
20
500
0.0
100
200
300
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
Transmission rate
0.4
0.2
1-
20
00000000000000
0.0
100
200
300
0
100
200
300
100
200
300
days
Figure 13: Goal-DQN without constraints evaluated in 3 = 0.65. Here find cyclical strategy equivalent to the one of shown with DQN agent trained with 0.5. For one run, this fig- shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost shown for various values [0, 0.25, 0.5, 0.75, 1].
22

---

Under review as a conference paper at ICLR 2021
4000
200
S (x109)
1000
E [xx10*)
3000
2000
100
500
1000
0.
02
100
200
300
100
200
300
100
200
300
80
15
1000
R (x109)
R (x109)
H (x109)
10:
500
40
20
0
0
100
200
300
0
100
200
300
0
100
200
300
days
Eco cost: 4.66 B, Death Cost: 55507, Aggregated Cost: 28.88
Aggregated Cost
New Deaths
2000
Total Deaths
60000
40000
2
1000
20000
1-
0
0
100
200
300
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
Transmission rate
0.4
0.2
2-
2.1
2.0
0.0
100
200
300
100
200 300 days
100
200
300
Figure 14: Goal-DQN without constraints evaluated in 3 0.7. Here the agent only use a lock- down at the very beginning of the epidemic. It is unclear whether this has an impact on the health cost at all. For one run, this figure shows the evolution model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of 8 in [0, 0.25, 0.5, 0.75, 1].
23

---

Under review as a conference paper at ICLR 2021
A.4.4 GOAL-DQN WITH CONSTRAINTS
Now present few strategies found by Goal-DQN with constraints, and study how reacts to health and economic constraints.
40
1.5
(x10%)
1226
(x10%)
30
1.0
1224
0.5
10
1222
0.0
0.
0
100
200
300
100
200
300
100
200
300
0.8
(x10%)
A (x10%)
0.6
(x109)
0.4
0.05
0.2
0.0
0.00
0
100
200
300
100
200
300
0
100
200
300
days
Eco cost: 80.75 B, Death Cost: 297, Aggregated Cost: 24.55
Transmission rate Esgregated Cost
New Deaths
Total Deaths
300
0.4
10
0.2
100
0.0
100
200
300
0
100
200
300
0
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
0.4
1.5-
1.0
0.2
25
0.5
0.0
100
200
300
0
100
200 days
300
0
100
200
300
Figure 15: Goal-DQN with constraints evaluated in 8 0.3, Meconemic 160B, Mhealth 62000 deaths. This boils down to no constraints as they are maximal values. This leads to a cyclical policy. For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of / in [0, 0.25, 0.5, 0.75. 1).
24

---

Under review as a conference paper at ICLR 2021
1250
S (x109)
1000
E (x104)
3000
100
750
2000
500
1000
00
0
0
100
200
300
0
100
200
300
100
200
300
1000
80
15
R (x109)
750
(x104)
H [xx10)
10
500
40
250
20
0
00
100
200
300
0
100
200
300
100
200
300
days
Eco cost: 42.74 B, Death Cost: 45278, Aggregated Cost: 61.58
Transmission rate Aggregated Cost
New Deaths
1500
Total Deaths
40000
2
20000
1
500
100
200
300
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
0.4
2
0.2
1
0.0
0
LUU
100
200
100 200
300
100
200
300
days
Figure 16: Goal-DQN with constraints evaluated in , = 0.3, M Mecomomic 55 Mheatth 62000 deaths. This is the same 3 Figure [3] (previous page). Now there constraint the number of deaths but a strong constraint on the economic cost. The strategy is not cyclical any- more, the resulting economic cost would be too high. This strategy stays below the economic constraint but still tries to minimize the health cost. For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost shown for various values of 3 in [0, 0.25.. 0.5, 0.75, 1].
25

---

Under review as a conference paper at ICLR 2021
1250
200
4000
1000
S (x10*)
E (x100))
150
3000
750
100
2000
500
1000
0
02
0
100
200
300
100
200
300
100
200
300
80
1000
15
(x109)
750
[xx10)
H (x104)
10
500
40
& 20
250
0
100
200
300
0
100
200
300
100
200
300
days
Eco cost: 1.73 B, Death Cost: 54604, Aggregated Cost: 26.41
Aggregated Cost
New Deaths
Total Deaths
1500
40000
2.
1000
20000
500
0
100
200
300
0
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
Transmission rate
1.5
0.010
2.3
1.0
0.005
0.5
2.2
0.000
0.0
100
200
300
0
100
200 days
300
100
200
300
Figure 17: Goal-DQN with constraints evaluated in 8 0.7, M Mconomiic 160B, Mheatt 62000 deaths. There is no constraint. As the balance favorizes the economic cost, the strategy does implement any lock-down. For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of B [0, 0.25, 0.5, 0.75, 1).
26

---

Under review as a conference paper at ICLR 2021
800
1200
W Wan
S (x109)
[x100%
600
1000
20
400
800
10
200
0
0
100
200
300
100
200
300
100
200
300
600
V M m
15
3-
(x10*))
400
(x10*)
(x109)
2-
200
I
02
0
100
200
300
100
200
300
100
200
300
days
Eco cost: 78.37 B, Death Cost: 27214, Aggregated Cost: 67.42
Aggregated Cost
30000
New Deaths
300
Total Deaths
0.4
200
M
100
10000
0.2
0
0.0
100
200
300
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
Transmission rate
0.4
75
50
0.2
25
1
0.0
00
0
100
200
300
100
200 days
300
0
100
200
300
Figure 18: Goal-DQN with constraints evaluated in / 0.7, M Mecomomicc 160B, Mhealth
30500 deaths. Now we have the same setup as the previous page (Figure 17). except that we have a strong constraint on the number of deaths. The resulting strategy respect the constraint while attempting minimize the economic cost. For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost shown for various values of 8 in (0, 0.25, 0.5, 0.75, 1].
27

---

Under review as a conference paper at ICLR 2021
1200
15
300
((x10*)
E (x100')
1100
10
200
5 1000
100
0
0
0
100
300
100
200
300
100
200
300
2.0
300
6-
R. (x10*)
(x109)
H (x10s)
200
4-
1.0
100
0.5
0.
0.0
0
100
200
300
100
200
300
100
200
300
days
Eco cost: 68.56 B, Death Cost: 15114, Aggregated Cost: 928.43
Aggregated Cost
New Deaths
Total Deaths
15000
0.4
100
10000
0.2
0
0.0
100
200
300
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
Transmission rate
0.4
60
2
40
0.2
1
20
0.0
UUN
0
100
200
300
100
200 days
300
100
200
300
0.3, Meconomic 55B, Mheatth 15000 deaths. Here we have strong constraints on both economic costs. In that case, there is no good solution. This strategy respects the health constraint but violates the economic constraint. For this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of /3 in [0, 0.25, 0.5, 0.75, 1].
Figure 19:
Goal-DQN with constraints evaluated in
28

---

Under review as a conference paper at ICLR 2021
A.4.5
NSGA-II
Now we present few strategies found by NSGA-II.
1227.5
40
1225.0
S (x110)
E (x100)
30
1222.5
1220.0
10
1217.5
0
200
200
300
10.0
R (x109)
(x10*)
H (x109)
0.10
0.05
2.5
0.0
0.0g
0.00
100
200
300
100
200
300
100
200
300
days
Eco cost: 80.76 B, Death Cost: 538, Aggregated Cost: 40.79
Aggregated Cost
New Deaths
Total Deaths
15
0.4
10
0.2
200
5
0
0.0
100
200
300
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
Transmission rate
0.4
2
0.2
25
0.0
100
200
300
100
200 days
300
100
200
300
Figure 20: NSGA-II, the closest point to [8000 deaths, 60B] in the Pareto front. In this low health cost regime, NSGA-II finds cyclical strategy with period of 2 weeks. For one run, this fig- ure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of 3 in [0, 0.25, 0.5, 0.75,
29

---

Under review as a conference paper at ICLR 2021
3000
1250
S (x109)
1000
(x109)
100
2000
750
500
1000
250
0
100
200
300
100
200
300
100
200
300
60
1000
R (x109)
A (x109)
H (x104))
750
500
250
0
0
02
0
100
200
300
100
200
300
100
200
300
days
Eco cost: 13.53 B, Death Cost: 53414, Aggregated Cost: 47.85
Transmission rate Aggregated Cost
New Deaths
Total Deaths
2
1000
40000
1
500
20000
0
100
200
300
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
10
0.2
5-
0.0
100
200
100
200 days
300
100
200
300
Figure 21: NSGA-II, the closest point to [30000 deaths, 40B) in the Pareto front. Here we find that NSGA-II alternative strategies depending the epidemiological models it faces. The average of these strategies ends up close to 30000, although the two strategies either find high health costs (>50000) low ones (< (<00))). These plots show the first alternative, where the strategy aims breaking the first wave of the epidemic. See the other alternative in Figure 22 For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of 3 in [0, 0.25, 0.5, 0.75, 1].
30

---

Under review as a conference paper at ICLR 2021
3000
1250
S (x109)
1000
(x109)
100
2000
750
500
1000
250
0
100
200
300
100
200
300
100
200
300
60
1000
R (x109)
A (x109)
H (x104))
750
500
250
0
0
02
0
100
200
300
100
200
300
100
200
300
days
Eco cost: 13.53 B, Death Cost: 53414, Aggregated Cost: 47.85
Transmission rate Aggregated Cost
New Deaths
Total Deaths
2
1000
40000
1
500
20000
0
100
200
300
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
10
0.2
5-
0.0
100
200
100
200 days
300
100
200
300
Figure 22: NSGA-II, the closest point to [30000 deaths, 40B) in the Pareto front. Here we find that NSGA-II alternative strategies depending the epidemiological models it faces. The average of these strategies ends up close to 30000, although the two strategies either find high health costs (>50000) low ones (<1000) These plots show the second alternative, where the strategy is cyclical and achieved low health costs. See Figure 21 for the first alternative. For one run, this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of 3 in [0, 0.25, 0.5, 0.75, 1].
31

---

Under review as a conference paper at ICLR 2021
1200
200
4000
5 (xx10)
1000
(x10*
3000
800
100
2000
600
1000
400
0
100
200
300
0
100
200
300
0
100
200
300
80
15
750
& [x100)
(x100)
H (x104)
500
40
5-
250
20
00
100
200
300
100
200
300
100
days
Eco cost: 13.26 B, Death Cost: 43612, Aggregated Cost: 40.18
Transmission rate Aggregated Cost
New Deaths
Total Deaths
1500
40000
2-
1000
20000
1
0:
100
200
300
100
200
300
100
200
300
New GDP Loss (B)
Total GDP Loss (B)
0.4
2
10
0.2
1
0.0
100
200
300
100
200 days
300
100
200
300
Figure 23: NSGA-II, the closest point to [45000 deaths, 20B] in the Pareto front. NSGA-II seem to find a robust strategy that consists in a single lock-down of few weeks to break the first wave. For this figure shows the evolution of model states (above) and states relevant for optimization (below). The aggregated cost is shown for various values of 3 in [0, 0.25, 0.5, 0.75, 1).
32


Reviews:
[{"review_id": "yChfkjP3W5V", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper introduces a library that implements 1 infectious disease model and 3 agents in a gym environment. The authors provide analysis for a covid-19 lock down scenario using their library.\n\n**Strengths**\nThe paper addresses an important and timely topic. The paper is easy to read and follow. The authors have open sourced their library and it seems to be well documented.\n\n**Weaknesses**\nThe main weakness of the paper is the strength of the contributions. The authors’ main contributions are a gym environment for a specific epidemiological model.\n\nUnfortunately, the novelty is somewhat lacking. As the authors mention in the conclusion section this line of work has been heavily researched. There are many infectious disease models and simulation environments already out there. E.g. a gym like interface has even been already open sourced earlier this year https://github.com/google/ml-fairness-gym/blob/master/environments/infectious_disease.py .\n\nIn terms of impact, the number of environments and agents is also quite limited at this stage. It’s also unclear if there is likely to be adoption by serious policy makers. Absent such impact statements it remains as one of many simulation frameworks out there.\n\n**What could make the paper better**\na) Many more environments and agents need to be implemented such that this library has the potential to become the standard for infectious disease simulation.\nb) A lot more analysis comparing agents and disease scenarios that truly unearth interesting scientific observations. \nc) Real world impact statements of adoption by policy makers and governments.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mq4m7spuJKf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors provide a python tool able to model epidemics development as optimization problems. This allows easing the work of decision-makers when faced with the problem of deciding new lockdowns. The model has been applied to real-world data to evaluate the consequences, in terms of deaths and per-capita loss, of a new lockdown.\n\nThe paper presents an interesting study on the dynamics of the epidemic. In my opinion, the development of a tool is relevant for medical and decision-making studies but is not enough novel or significant for the ML field. I think that this kind of analysis is better suited for a more applicative venue, while the novelty provided in your work is not enough to justify a publication at ICLR.\n\nThe paper is clear and well written. I appreciate the use of a real case-study and the following analysis. I think that the only problem is that the venue chosen by the authors does not fit its purpose.\n\n\nQuestions:\n- Did you also try your model on different datasets? I think that a more wide experimental campaing might improve the value of what have been proposed here\n- What about other forms of prevention other than lockdowns? For instance, it is possible to model tracing or testing as prevention methods in your modeling? This would instead make the model more flexible and allow decisional organs to act in a more flexible way.\n- Another interesting study might include the use of different methods for contagion prevention and their use in a joint or sequential manner in order to understand what are the policies over time which might be the most promising for a tradeoff healt/GDP.\n\n----------------------------------------------------------------\nAfter rebuttal: the paper seems interesting but, as I already mentioned and as other reviewers pointed out, the main concerns about this paper are novelty and relevance to the ML community.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XgtR106rQ8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces OpenAI Gym environment for RL optimization of epidemic containment policies. The envirnoment currently contains an example SEIR model parameterized for COVID-19, along with a simple economic model to evaluate the lost productivity due to lockdowns. Some experimental results are shown where different deep learning algorithms are used to optimize intermittent lockdown policies. \n\nOn the positive side, connecting the epidemiological and ML communities is definitely an important goal. Developing open-source tools to make this interaction easier is valuable. \n\nI'm not sure that this is an appropriate paper for ICLR though. It mostly takes a preexisting epidemiological model and exposes it in the OpenAI gym interface, without a strong research contribution. I believe that there are technical issues in the development of a platform for health policy optimization which are likely to result in research contributions, for example dealing with uncertainty in models/parameters, developing more efficient methods for multiobjective optimization, or providing explanations of policies (particularly since experts are unlikely to implement a RL policy verbatim, but rather try to synthesize its recommendations with other considerations/sources of information). I hope that the authors continue to refine this platform and tackle these or other issues in the future. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the contributions may not be relevant to the ML research community or perhaps are a poor fit for the venue, but otherwise find the work potentially useful and addressing a timely topic. Because the paper focuses on a simulation environment for existing epidemiological models, reviewers comment that the technical and methodological novelty is limited.
Decision: Reject