Simulating Environments for Evaluating Scarce Resource Allocation Policies
Authors: Jeroen Berrevoets
Alex Chan
Daniel Jarrett
Mihaela van der Schaar
Venue: Submitted to ICLR 2023 (2023)
Keywords: Simulation
Evaluation
Validation

Abstract: Consider the sequential decision problem of allocating a limited supply of resources to a pool of potential recipients: This scarce resource allocation problem arises in a variety of settings characterized by "hard-to-make" tradeoffs– such as assigning organs to transplant patients, or rationing ventilators in overstretched ICUs. Assisting human judgement in these choices are dynamic allocation policies that prescribe how to match available assets to an evolving pool of beneficiaries– such as clinical guidelines that stipulate selection criteria on the basis of recipient and organ attributes. However, while such policies have received increasing attention in recent years, a key challenge lies in pre-deployment evaluation: How might allocation policies behave in the real world? In particular, in addition to conventional backtesting, it is crucial that policies be evaluated on a variety of possible scenarios and sensitivities– such as distributions of recipients and organs that may diverge from historic patterns. In this work, we present AllSim, an open-source framework for performing data-driven simulation of scarce resource allocation policies for pre-deployment evaluation. Simulation environments are modular (i.e. parameterized componentwise), learnable (i.e. on historical data), and customizable (i.e. to unseen conditions), and– upon interaction with a policy –outputs a dataset of simulated outcomes for analysis and benchmarking. Compared to existing work, we believe this approach takes a step towards more methodical evaluation of scarce resource allocation policies.

Under review as a conference paper at ICLR 2023
SIMULATING ENVIRONMENTS FOR EVALUATING SCARCE RESOURCE ALLOCATION POLICIES
Anonymous authors Paper under double-blind review
ABSTRACT
Consider the sequential decision problem of allocating limited supply of sources to pool of potential recipients: This scarce resource allocation problem arises in variety of settings characterized by thard-to-make" tradeoffs such assigning organs to transplant patients, or rationing ventilators in overstretched ICUs. Assisting human judgement in these choices are dynamic allocation policies that prescribe how to match available assets to an evolving pool of beneficiaries- such as clinical guidelines that stipulate selection criteria on the basis recipient and organ attributes. However, while such policies have received increasing at- tention recent years, key challenge lies in pre-deployment evaluation: How might allocation policies behave in the real world? In particular, in addition to conventional backtesting, it crucial that policies be evaluated on a vari- as distributions of recipients
ety of possible scenarios and sensitivities- and organs that may diverge from historic patterns. In this work, we present AllSim, an open-source framework for performing data-driven simulation of scarce resource allocation policies for pre-deployment evaluation. Simulation environments are modular (i.e. parameterized componentwise), learnable (i.e. on historical data), and customizable (i.e. to unseen conditions), and -upon interaction with policy-outputs a dataset of simulated outcomes for analysis and benchmarking. Compared to existing work, we believe this approach takes a step towards more methodical evaluation of scarce resource allocation policies.
INTRODUCTION
The distribution of organs for transplant is prototypical example of the scarce resource allocation problem one with salient "life-or-death" consequences that places significant pressure decision- makers to make implicit but difficult trade-offs. To make the task more manageable, assisting human judgement in these choices are dynamic allocation policies that prescribe how to match each available unit of resource to a potential beneficiary. For instance, the United Network for Organ Sharing (UNOS) stipulates policies for organ allocation according to weighted organ- and patient-specific criteria, such as time on the waiting list, severity of illnesses, human leukocyte antigen matching, prognostic information, and other considerations [1-3]. Likewise, in the machine learning community, variety of data-driven algorithms have been proposed as drop-in dynamic allocation policies, leveraging modern techniques for estimating treatment effects, predicting survival times, and accounting for organ scarcity and often succeed in demonstrating high degrees of improvement in terms of life expectancies when deployed and evaluated on a backtested basis [4-6].
Is such demonstrated backtested performance sufficiently convincing for practitioners to adopt these developed allocation strategies? In many cases the answer is no, since there still no standardised way in which this backtesting is actually undertaken [7]. The variety of methods that do exist share common challenges: First, when testing a target policy different from the actual policy used to generate the data, any offline evaluation method is immediately biased away from the true open-loop tata-generating process [8]. Second, the evaluation methods themselves impute predicted outcomes, often with simple linear models [9, 10], which are not flexible enough properly test more flexible machine learning methods. The compounding effect of these limitations leads to elinicians finding the results unconvincing (1 1J, consequently limiting the use of these potentially very beneficial systems.

---

Under review as a conference paper at ICLR 2023
Historical data is also not the only aspect for which candidate policies should be compared against. In fact, it is crucial that policies be tested on a variety of scenarios and sensitivities for more robust evaluation based plausible possible futures not just conditions that have been seen before. Real-world conditions may change after all, meaning historical data may no longer be representative of the current environment. To this end, desire system that allows us to test policies in such counterfactual scenarios for which we need counterfactual machine learning models [12] (cfr. Sect. 3.4 and our Appendix F)-thereby more systematically evaluating and benchmarking candidate policies before they are put into practice, as well as continuously updating existing policies based on anticipated or unanticipated changes to the environment. We propose to evaluate policies using machine learning as we illustrate in Figure I.
As a motivating example, consider the impact of COVID-19 on the availability of organs for transplant leading in many cases to sudden and severe drop in supply [13]. Is it possible to reasonably measure performance of alloca- tion policies against such theoretical event priori? Not with current satt-bbaed methods that do not contain example of such an event, Conversely, with our simulated model we can in- tervene on the distribution of organs reducing policy roll-out and testing to simple task.
Input
Output new eval. data D
Data (UNOS)
Policies
A118im
Counterf. ML
(2) (3)
Figure I: Conceptual overview of AllSim. We illus- trate Allsim's input and how it relates the desiderata: (2) learnable and (3) customizable. The first desider- ata (modularity) is discussed at length in Sect. 2 where we present Al1Sim's 's design. In Figure 2 we discuss the (Allsim) node in detail. The practitioner can use AllSim to test policy (T) and generate new dataset (D'), used to compute performance metrics interest.
With this in mind, we wish highlight that ac- curate evaluation of such allocation policies is equally. if not more, important than their devel- opment. After all, there is little point in developing policies if they cannot be shown to be beneficial and then never used. Worse, it may be that the current evaluation techniques are causing researchers to optimise for performance that may end up being actively detrimental in real-world deployment. And so, we now continue by considering what exactly an ideal evaluation method may consist of:
Desiderata We argue that good solution should satisfy three key criteria: The environment must be (1) modular, in the sense that it is composed of parameterized components to allow flexible user interaction; (2) learnable, in the sense that it is grounded in the characteristics of real-worke data; and (3) customizable, in the sense that it can test policies in previously unseen environment conditions. In particular, performance estimates should be unbiased when target policies different from the data- generating policies are evaluated. Note that our objective is not limited to estimating simple aggregate descriptive statisties (such mean performance): We may also be interested in evaluating more general impacts of policy deployment-Such as whether it inadvertently discriminates based on gender or ethnicity, or whether patients suffering from particular disease are inadvertently disadvantaged.
Contributions In this work, we present AllSim (Allocation Simulator), a general-purpose open- source framework for performing data-driven simulation of scarce resource allocation policies for pre-deployment evaluation. We use modular environment mechanisms to capture range of environment conditions (e-g. varying arrival rates, sudden shocks, etc.), and provide for com- ponentwise parameters to be learned from historical data, as well as allowing users to further configure parameters for stress testing and sensitivity analysis. Potential outcomes are evalu- ated using unbiased causal effects methods: Upon interaction with policy, Allsim outputs batch dataset detailing all of the simulated outcomes, allowing users to draw their own conclu- sions over the effectiveness of policy. Compared to existing work, we believe this simulation framework takes a step towards more methodical evaluation of scarce resource allocation policies.
2 SIMULATING ENVIRONMENTS
In this section, we explain various design choices we made when building Allsim in light of the three criteria we listed under "desiderata" above. Specifically, we require method for evaluating a policy, not only for the expected outcome (for example recipient life-years), but also with respect to various demographics, aetiologies, resource waste, etc. We will solve this by outputting synthetic dataset where recipient covariates are matched with organ covariates and an outcome. An example of an excerpt of such dataset can be found in Appendix A.
2

---

Under review as a conference paper at ICLR 2023
2.1 PRE- PRE-DEPLYYMENT EVALUATION
Let X e Id denote the feature vector of a recipient, and let (t) denote the arrival process of recipi- ents. At each time t, let X(t == [ =0 ~ X() give the arrival set of (time-varying) size N (t). Likewise, let R E R be the feature vector of a resource, and let R(t) be the arrival process of re- sources. At each time f, let (t)
| (1) ((t) give the arrival set of virniin size (t). While we make no assumptions how recipients are modeled, we assume that resources are immedi- ately perishable-that is, each incoming resource cannot be kept idle, and must be consumed by some recipient in the same time step. In organ transplantation, for instance. the time between harvesting an organ and transplanting it ("cold ischemia time") must be minimized to prevent degradation [14-16].
Let Y+ E R be the outcome of a matched recipient, drawn from the distribution (((X, R) induced by assigning a resource R a X. At each time t. let (t) == {Y+ v(X n. R) R E R(t)} give the set of outcomes that result from matching each incoming R E R((t) with its assigned X Like- wise, let Y- E R be the outcome of an un an-matched recipient, drawn from the distribution )(X, 2). At each time I, let the set of outcomes for recipients who are never assigned resource be given by (() = (Y~ ~ )XX, Ø)= : X E ((), -33' 2 () E R('), X XR)). (Note that we focus on discrete-time settings (e.g. hours or days), and leave continuous time for future work). Then we have:
Definition 1 (Scarce Resource Allocation) Denote an environment with the tuple = (Xt R,
The scarce resource allocation problem is to decide which recipients to assign each incoming resource to-that is, to come up with a dynamic allocation policy π IR P(R)) + R.d, perhaps to optimize some objective defined on the basis of matched/un matched matched outcomes. For instance, if Y is a patient's 's post-transplantation survival time, then objective might be to maximize the average survival time.
R x - M denote evaluation metric mapping a sequence outcomes {Y())) to some space of evaluation outcome Ml (e.g. for the average survival time, this would simply be R). where Yt)) == Y (t) -t) Given problem & and policy n, the pre-deployment evaluation problem is to compute statistics of the distribution Fe.r of evaluation outcomes ((((()) commonly, this would be the mean Eg, ((
Definition 2 (Pre-Deployment Evaluation) Let S: IRI
Note that we have defined I in terms of the sequence of per-period outcomes such that it gives maximum flexibility: Depending on how individual outcomes Y are defined, we can measure point estimates (e.g. the mean survival), compare subpopulations (e.g. whether some types of recipients systematically receive more favorable outcomes), examine trends (e.g. whether outcomes degrade as the types of recipients arriving change), or potentially investigate more complex hypotheses.
2.2 SIMULATION COMPONENTS
Environment Components Clinically validating dynamic allocation policy not only requires estimating how that policy performs under a hypothesized scenario &, but also how sensitive is to changing ceciitiions-smmetiie that is not possible using existing methods. We believe that simulation. By this we mean a simulation that comprises the key components (i.e. recipient arrival process X, resource arrival process R, and outcome generation mechanism y), each parameterized in such way so that a practitioner can easily compose the simulation to mimic scenario of interest. For example, if a practitioner wishes evaluate an allocation policy against a sudden decline of resources, they need only to alter the component that is responsible for resource provision in the simulated environment. With a cowponents- we allow policy to simply interact with each component. Every interaction (i.e. eccppenttresouree pairing) is registered in a simulated dataset. Having such a simulated dataset, the practitioner can specify typical clinical evaluation metrics to properly evaluate the policy.
the solution lies in components-bute
simulation,
Policy Responsibilities A policy has two responsibilities: (i) the policy maintains the set of recipients in need of a resource (being P(X))), the environment only provides new recipients: and (ii) when the environment provides a new resource, the policy is tasked with allocating it to one of the recipients in need. We will denote these two responsibilities as policy. add (X(t)) which allows the pol icy to maintain the arriving recipients for (i), and policy- seelcct (R(t)) which returns set of recipients (one for each in R(t)) for (ii). Note that while policy is not part of the environment, it may naturally act differently depending on the characteristics of the environment. For instance, it can even learn how best to behave through interaction.
3

---

Under review as a conference paper at ICLR 2023
Component
Description
Learned
X(()
Recipients
Azs
of
arrival amount
asiect Y
A113im
H
sample objects
R(t)
Resources
salect
select recipients
(()+
infer outcome
Evaluation datase D
Figure 2: Simulation life-cycle of AllSim. In the leftmost part of this figure, have illustrated the simulation lyfe- In the rightmost part, we list each component and indicate whether or not they are learned from data. Simulation life-cycle. First, we sample the amount of each object arrives on day t. The expected amount is indicated hy X₂ and A- for the recipients, and resources, respectively. The expected amount should he interpreted an arrival rate. Once the amount of objects is specified, they sampled from distribution X()) and ((f).. Next, each objected is presented to the ScRAP, we wish to test, and the ScRAP replies by selecting recipients whom the resources will be presented to. Finally, the simulation evaluates the ScRAP's decision, by providing an inferred outcome. Wrench icons ( and an X's in the table indicate which components can be user-specified, others (with v/ are learned. Finally, X(). R(), and (l) compose dataset D for evaluation.
of
Non-stationary Environments A major shortcoming in simulations used to evaluate resource allocation policies medicine, is that they cannot evaluate policies in non-stationary settings. Hence, we model the environment with a time-indicator, E(). In this subsection, we discuss how we can model a non-stationary environment, while still maintaining a large level of control for the practitioner.
Having sampled a particular recipient or resource, does not influence which is sampled next. However, the probability of sampling a particular recipient, or resource, does change with time. For example, the ever more increasing surgical techniques allows clinicians to attempt surgery in more challenging patients, increasing the amount of patients. Similarly for resources, an organ which would not be considered for transplantation 20 years ago, may be considered today. Conversely, the recent COVID-19 pandemic is shock which has shown an incredible drop in supply [13].
These types of environment changes are modeled through f. With i, we can model both slightly changing drifts of supply and demand (such as those that are the consequence of improving surgical techniques), as well sudden changes (such those caused by drastic events). We particular recipient and a resource to a probability p(X,(t) and p(R,(t) [t) given a time f, and their respective arrival processes, (t) and R(t). A major point in favour of AllS im, is the ability practitioner has to model change. Specifically, a practitioner can learn a model of the arrival rate as a function of time and then provide that model to A. AliSmm m (as we have in Sect. 5), or the practitioner can provide a self-specified function of the arrival rate, and test the policy in an environment, alternative than the real environment.
pair
SIMULATION LIFE-CYCLE
Given the components defined in Sect. 2.2, we are now able to formalise the high-level simulation life- cycle. In simple terms, the simulation loop will model the interaction between & and the pol cyy for each consecutive day (t = .... until some pre-specified end (7). We have organised this section around Algorithm where we connect each important line with a component in Figure 2 and discuss it in a separate subsection. In Figure 2, where we illustrate the interaction between each component, and clarify which parts of the simulation can be learned, and which parts should be parameterised by the practitioner- solving the second (learnable) and third (costumizable) desiderata, respectively. Having a components-basee evaluation strategy solves the first desiderata from our introduction: the simulation is modular, i.e. each component can be reimplemented and changed will.
3.1
(I) Sample secipients and resources
The first step in Algorithm comprises the first and second component in Figure 2, repeated below,
# (1())
arrival amount
sample objects
Essentially, the goal in In 4 of alg. 1 is to translate the arrival rates to a set of recipients and resources, trademarked by their features sets. With above formulation, model the arrival processes
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Main simulation loop. This simulation life-cycle acts as a section overview for Sect. 3. In our main text we discuss how each line is simulated. input :Environment, &; resource allocation policy denoted as pol
output: :Policy runtime summary
Start, f =0;
while simulation rums do
/* iterate time */
X(t). R() ~ E(t); /+ (i) sample recip.s and resources (Sect. 3.1) */
pol. add (X(()) /x (ii) notify policy of recipients (Sect. 3.2) x/
XR()) <pol. select (R(f)); /* (iii) allocate resources (Sect. 3.3) */
Y, V'. ()((R(() R()):
/+ (iv) consume resources (Sect. 3.4) */
end
two-step approach: (1) sample the amount of objects we may expect, (2) sample the objects from a distribution. Indicated in Figure only some of these component are learned, and others user-defined.
Recipients and organs arrive randomly. The arrival processes (t) and R(t) are approximated as,
(t, 0z) 01.x) x Rx (t, 0x,z). R(,, ( x... x (1,
where &g and R2 are modeled by a Poisson arrival process with arrival rate (t) and (t), respec- tively. In Equations (1) and (2) we factorise the arrival processes in K and different sub-arrival processes. Each factor corresponds with some (user-defined) recipient-type (Equation (1)) and resource-type (Equation (2)). Having these factors allows us to model increasing numbers of, for example, older patients entering transplant wait-list. Arrival rates change with and are modeled as,
(t) v4 (0) (x = (O)
(4)
with Au (1), Ae(1)= E RR,,, and un, '1 E R2 as normalising constant such that the sum of all A equal some overall arrival rate dze, and similarly, the sum of all x equal some overall arrival rate dr- Optionally, V/ and by can be kept fixed throughout the simulation such that ag and a, vary as does (t, or it can be recomputed for every step t, such that ag and a, kept fixed throughout the simulation. Lastly, fa and fi are continuous functions that simulate a user-specified drift. Note that these f can also be combination of multiple drift scenarios, Having f. allows practitioners to very accurately describe the non-stationarity they wish to test for.
be shared across different k. L.
As such, have set of arrival rates, A₂ [Ai,..,AAkl with Σ, d ag with 0g R4 as the total arrival rate of recipients. The advantage of splitting Og into multiple Ag. is that we can finetune the arrival of certain recipient types, yet allow comparison between 0g and a, (the total arrival rate for resources). For example, the 4thh recipient type may be completely absent when policy is launched, but over time gradually enters the system, increasing Og as a whole. Naturally, we also model the arrival of resources as we have for recipients, but left it out of discussion for clarity.
The recipient and resource distributions. When recipient a resource arrives we sample them from distribution denoted po... (X) for the recipients, and ps, (R) for the resources. These distributions are either learnt from data, or shared as an open-source (but privatised) distribution.
Let's say we have a recipient type k associated with a region in X, and an arrival rate parameter A E [0, 1). Note that recipient type's support may overlap with other recipient types, and that each recipient type is modeled by distribution p(X (I))). In principle, we wish to keep p( X [e) fixed, and learnt from data; and A (now acting as an arrival rate conditioned on the type, k),, tunable.
3.2 (II) notify policy of recipients
After the simulation has sampled X(t) and R(t), the policy add (X(t)) function is called. Specif- ically, this notifies the policy of the arrival of new recipients, which in Figure 2 corresponds to,
(()), R(!)
5

---

Under review as a conference paper at ICLR 2023
Note that the above is simply service provided by the simulation to the policy, and is thus not learned. The reason for this is, that the organisation of in-need-reet redeiieeteis is entirely up to the policy. For example, more traditional policies may maintain only one priority queue [17], whereas more novel policies may maintain multiple, based on recipient and resource types [6]. As such A Allsim remains agnostic to the tested policy, resulting in more general-purposs framework.
3.3
(III) allocate resources
Like Sect. 3.2, step (iii) is entirely managed by the tested policy, and in Figure 2 corresponds with,
select
Selecting which recipients that get consume the resource, is generally what differentiates policies. When set of recipients is selected (through a poliice seleeeeeeeee select (R(t)) call in the python interface, AllSim retains every piece of information associated with the select-call, which is used evaluate.
Having the simulation retain all this information, allows AllSim to output a dataset of past policy behaviour, much like the original (real-world) dataset we provided to the simulation in order to learn the various components, In essence, with Ailsim we are able to sample a synthetic dataset of counterfactual scenario such that are able
policy of interest, if were already in use.
3.4 (IV) consume resources
The final component in A. 1Sim is the inference- component, yielding an outcome after a resource was consumed by a recipient. Importantly, this component allows us to evaluate a policy, de- spite it deviating from the data used to learn the simulation. In Figure 2, this component is illustrated as,
Y
D oor D
π
Y or Y
Y Y
)
Figure Effect of policy on data. Above illustration, depicts two policies: и and x'. Each policy is tasked Besides re-
The outcome function of the resource and its recipient, making inference hard as some combinations are less observed in the original data. Essentially, the tested policy may make out-of-distribttion combinations, as the simulation is Iearnt from data that was collected under some other policy, illustrated in Figure where two policies, and x result in different datasets D and D'.
with assigning organs to recipients cipients and organs, observe outcome, Y. Despite recipients and organs, different
observing the policy results completely different outcomes.
Counterfactual inference. AllSim handles this by using a counterfactual estimator to infer alloca- tion outcomes they deal with this issue explicitly. Counterfactual methods aim to make an unbiased prediction of the potential outcome, associated with some treatment (or resource). We are interested in counterfactual methods that model the potential outcomes for the recipients when they are/are not allocated a resource. A counterfactual estimator then "completes" the simulated dataset as,
Y(!) B| (R(() (x))))))))))) ((),
where (R(()) the estimated potential outcome, using methodology known in the potential outcomes literature, and (1) are the recipients selected by at time í. Equation (5) provides the recipient-resource pair with an estimated outcome, and presents the practitioner with a dataset:
D= = (XX, R, (R). 1...,,
DE allows to easily calculate clinical measures of performance, as we demonstrate in Sect. 5. More information regarding these counterfactual models is provided in Appendix F.
RELATED WORK
We have summarised work related to ours in four major categories: (a) Off-policy learning and evaluation, (b) causal inference, (c) clinical simulations and trials, and (d) simulations for evaluating reinforcement learning (RL) agents. A high-level summary of these areas can be found in Table 1. For discussion related work in the clinical domain, refer Appendix F.

---

Under review as a conference paper at ICLR 2023
Table 1: Overview of related work. We categorise our related work in four categories: off-policy learning, causal inference, clinical simulations, simulations for evaluating reinforcement learning agents. For each category we provide the most prominent method of calculating policy performance, and whether the categories take into account four major questions: (i) is the method tunable new settings?: (ii) can evaluate using data?; (iii) are the performance estimates unbiased??; and (iv) can we evaluate beyond simple aggregate descriptive statistics?
Citations
Perf. calculation
iii)
(iii)
(iv)
Off-policy learning
[18-23]
P+1(R1X) Putla Y ~ D P() Iy D"
X
/f
d
Causal inference
[24-27]
/
/
X
Clinical simulations
[17, 28-31]
30- [M]]]
x
/
x
V
Reinforcement learning
[32-38]
Einlo(Y) R]
x
A11Sim (ours)
DD' Xxxx YIX]
(a) Off-policy evaluation and (b) causal inference. Data are collected under some active scarce resource allocation policy (ScRAP), which determines how resources are paired with recipients, which in turn determine the outcomes we get to observe. Clearly, our setting is connected to off- policy evaluation, we wish to evaluate a policy (ScRAP) that is different from the policy that collected the available data. Figure 3 illustrates different matches (depicted as different policies resulting in different outcomes: Y or Y', and datasets, D and D'. The difficulty of this situation, is that we observe either D or D', not both, relating to the potential outcomes setup [12, 39, 40]. Using D to evaluate simply means calculating some descriptive statisties. In contrast, evaluating n' on 'D is much more involved, as the same statistics would yield a biased estimate.
made across
One way to calculate, for example the average Y' D. is to weigh each sample: gf Y D. In the literature on off-policy evaluation (and learning), this is known as importance sampling, where each sample is weighted according the likelihood of it belonging in D'. Interestingly, the exact same strategy is also widely known in counterfactual learning and causal inference, under the name inverse propensity weighting (IPW) [20]. The key difference between both, is the target distribution. Where importance sampling transforms the estimate from one policy's is distribution, to another; IPW transforms from a policy's distribution, to an unbiased estimate (i.e. to a N weighting for averages).
Shown in Table 1, neither IPW nor importance sampling, provide a solution to evaluating scarce resource allocation policies. One reason for this is that the outcome is a (weighted) estimate of only one aggregate descriptive statistic (for example, expected outcome or reward). When we want additional estimates, one requires a different weighting scheme [23]. Futhermore, when we wish to test policy in different environment, where for example the recipients' resources' distributions change, evaluating a policy becomes increasingly more difficult if one wishes to rely on IPW or importance sampling due to the increased differences in likelihood density. Furthermore, evaluation goes much beyond aggregate statistics, as we may be interested in, for example, demographic differences between recipients and non-recipients (Appendix A includes an example using AllSim).
(c) Clinical evaluation and trials. A naive method to evaluating these policies, is to model Y' using
through a new policy, n' has an estimated outcome ý' using a regression model, trained on D. Herein lies the problem: the regression model is still biased и. In particular, D simply does not contain pairs that x' would make, and therefor has trouble estimating Y'. By using causal methodology (Sect. 3.4), this is one key area where AllS im improves upon contemporary clinical simulations [9. 17, 41, 42], as these only "replay" the past without the possibility of changing the environment characteristics nor allow counterfactual inference.
simple (linear [17)) regression, Any combination
Clinicians do have another way to account for this: clinical trials [28, 29]. Clinical trials estimate a causal estimand, which could then be used to "complete" the dataset as a naive regression model would above. However, two problems arise: setting up a clinical trial can sometimes be considered unethical (especially when dealing with scarce resources, and more specific research questions) [43], or clinical trials may suffer from compliance issues which will still result in biased outcomes [44, 45].
(d) Simulations for evaluating RL agents. A final set of solutions is that of simulations in which an RL agent can act and Iearn. Naturally, there are differences between policies for RL and scarce resource allocation, such as the importance of future reward which implies correlated consecutive states, and a fixed action space. However, there are also similarities, such as the definition of a policy (being a set of rules which take into account a context or state), and the online nature of the problem.
Literature on RL is blessed with some of the most well-known and actively maintained libraries for evaluating a wide range of RI. algorithms [32, 33, 38). However, other than being inapplicable in
7

---

Under review as a conference paper at ICLR 2023
import allsim as asim
Avg. donor age
bins [30, 45, 60]
avg. age
60
_ lambda age: np digitizee age, bins-bins) .itm
55
50
7 condition asim. www.eee..nndiiiini
avg. BMI
Avg. donor BMI
labels- [= function-f, options-4)
27
26
kde
asim. KDEDensity
25
Al1sim
Real data
condition=concitionn
kde. fit (R) $ R: res. pd. Fa.DaaaFrame
==0
t
300
600
900
(a) Mean arrival of age and BMI..
system
asim. www.sssrrenereses [0)),
arrival rate
25
Arrival rates over time
16
1:
20
15
45
poisson systemssstemmmmmm asim. PoiesonSyytenn
10
45 60
00 <
20
density=kde, systen=system,,
21
alpha-alpha, nermal Tre=))
0
300 1
600
900
(b) Depicted are the arrival rates for each age bracket. Note that, in this instance, we learnt these from data, but one can use any function.
stream DataFrame
(olouns==+R_cooumss) "t"]]
25 For t in T:
sample = peisscn_systeeeeeeeeeeeeessesese stream stream append sappenddsamgg (sample)
Figure 4: Al Alms can generate realistic synthetic data. Using data on donor organs, we let Al1s im model 3 years of organ arrivals and compare it with the actual arrival as reported in the data. In Figures 4a and 4b we show AllSt sim's in's output (in donor age and BMI), given the code on the right. With minimal code, simple condition (4 age brackets), and conservative models (polynomial regression to fit the arrival rates, and Gaussian kernel density to model the organ densities), we find All Allmim accurately model the actual arrival organs.
setting (for the reasons included above), they also have one other major downside: they do not learn from data. While these simulations model some very interesting environments, each aspect of the order to evaluate a policy that is to deployed in real- hol-world setting,
simulation is hard-coded. we have to test it to the specifics of the environment of interest; to be defined by the practitioners developing the policy. In fact, we consider this a major reason for developing and using AllSim
Al1Sim marks significant advance over the current tate-offthhe-rrt currently evaluation technique allows a practitioner to change the behaviour of the evaluation environment, despite the fact that the actual environment most certainly will change in the future; nor do contemporary evaluation techniques account for potential bias from previous policies, resulting in poor performance estimates.
SIMULATION INTERFACE & EXAMPLES
Given the formal definition of A 115m in Sects. 2 and 3, we now introduce A Sim's interface and use directly to provide some experimental results. We split this section in two major parts: first, discuss how model arrival processes and resouree densities (Sect. 5.1); and then, discuss how we can combine these arrival processes in the broader simulation interface, together with a counterfactual outcome model (Sect. 5.2). Both parts provide some empirical validation of AfAlll Allsim!
5.1 ARRIVAL PROCESSES AND RESOURCE DENSITIES
As is indicated in Figure 2 and Sect. 2.2, sampling resources and recipients is two-step process. On a daily basis (or some other discrete time interval in [t)). we first determine how many resources recipients need sample, and then sample them. The former is modeled through a Poisson arrival rate that changes over time, and the second is sampled from some learnt density. Of course, user can implement their own arrival process by inheriting from the abstract arrivalProces class. Importantly, we need to be able to condition both on some pre-specified characteristic of the object of interest. For example, one may be interested in modeling the arrival of harvested organs of older patients distinctly from younger patients. An example of this is provided in Figure 4, where we show the changing resources coming in the system, alongside the code that generated the result.
Note that AllSim is the first method of this kind for henchmarking scarce resource allocation policies.
8

---

Under review as a conference paper at ICLR 2023
Average days alive
Donor arrival rates
<:30
Simulation outcome.
1.3k
2
30
45
Simulation setup.
60
1.1k
Average age
Donar Recipient
.9k
45
MELD
MELD-Na
.kk
HPO
300
600
= 900
t 300
t 600
t 900
Figure 5: Simulated outcomes. Allsim be used to compare policies in user-specified scenario. Above we compare three different organ allocation policies: MELD, MELD-na, and FIFO. The MELD policies are clinically validated and used in practice while FIFO is a naive first-in first-out allocation policy (which should not work at all!). When comparing these policies, our Allsim simulation suggests that the clinically validated policies do outperform naive FIFO policy in terms of average survival (left), is expected. Furthermore, the performance of each policy does seems drop slightly over time. This is expected a setup with increased recipient age over time (right). The reported averages are windowed over samples.
tested the policies
Object densities. Before discussing a temporal arrival rate, we first discuss modeling the object's densities. Consider ins 3-12 in the righthand side of Figure 4. Using this code, first define what we want to condition on, using a Conditi on object: in this case we formulate age brackets. With the KDEDensity class, which is subelass of the abstract Density class, model a density, conditioned on these age brackets. Each Density object implements a fit and sample function, which is used to sample new objects by the System, which we discuss next.
automatically
Arrival processes. Using Density, we move on to ins 14-26, where we first build system of multiple arrival processes, one for each discrete condition as Equations (1) and (2). particular, we define a PoisonProcees rocess for each condition (or age bracket), which is then provided to PoissonSysLem. Using the PoissonSysLent we can sample the arriving objects for each in In 25. Note that we also model a alpha, returning the overall arrival rate, such that the system can calculate an appropriate v. Figure 4a shows that mmmiim accurately models the arriving objects.
5.2 COUNTERFACTUAL OUTCOMES AND SIMULATION
With the arrival processes coded above together with counterfactual Inference object compose Simula at on object- the main interaction interface. In particular, one defines a set of arrival rates (such in Figure 4b) for both recipients and resources (efr. in 13-17) to create simulation:
simulat lation asin. Sim (resource wttssuurreepcccessseeeeee
processssrrcess,steeee inference)
With that simulation, a practitioner can instantiate a Policy, which implements the add and se ect methods. For example, we have implemented the MELD policy [46], which a widely known and used ScRAP for liver allocation. Using the simul atm a on, we can generate simulated dataset:
df s simulatiin inn. simulate (meld policy, T=T)
Where of is Pandas Dat rame [47, 48]. Naturally, df contains an enormous amount of informa- tion w.r.t. the policy's allocations in our environment. As such, we have included only subset of the potential results in Figure 5. Additional results and details can be found in Appendices A and Ultimately, the practitioner determines appropriate analysis, settings, and performance metrics.
6 CONCLUSION
AllSim allows standardised evaluation resource allocation policies. While our main text heavily focuses on organ-transplantation for the sake of exhibition, Appendix E illustrates A a115.. 1.lmmm for COVID-19, an example outside wrgan-transplantation We believe strongly that AllSim's generality and modularity allows for sensible adoption in a wide range of application areas. Furthermore, having standardised evaluation will encourage research in this very important and impactful domain. Naturally, conducting research in clinical resource allocation requires consideration of policy's societal impact. While we believe Allsim will aid this respect also (by offering more than simple aggregate statistics), in Appendix D we provide section this topic specifically.

---

Under review as a conference paper at ICLR 2023
Ethics Statement. We envisage AllSim tool to help accurate and standardised evaluation of organ allocation policies, however emphasise that any finding would need to be further verified by human expert or by clinical trial. Ultimately, the decision whether or not to trust decision making tool is up to the acting clinician and hospital board. We hope that Allsim can help in any way to facilitate that decision, but stress that suggestions evaluation always require critical assessment, as is the case for any research. We also refer the reader to Appendix D for more thorough discussion on the potential societal impact of systems such as AllSim.
Reproducibility Statement. To ensure reproducibility, we have included all our code to reproduce the presented results. Furthermore, we have included a detailed discussion on how to use our simulation in Appendices B and C.
REFERENCES
[1] Dale G Renlund, David O Taylor, Abdallah G Kfoury, and Robert S Shaddy. New unos rules: historical background and implications for transplantation management. The Journal of heart and lung transplantation, 18(1 18(15):00601070, 1999.
[2] Thomas M Egan and Robert M Kotloff. Pro/con debate: lung allocation should be based on medical urgency and transplant survival and not on waiting time. Chest, 128(1):407-415, 2005.
[3] Leslie P Scheunemann and Douglas B White. The ethics and reality of rationing in medicine. Chest, 140(6):1625- 1632, 2011.
[4] Jinsung Yoon, Ahmed Alaa, Martin Cadeiras, and Mihaela Van Der Schaar. Personalized donor-recipient matching for organ transplantation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[5] Jeroen Berrevoets, James Jordon, Ioana Bica, Alexander Gimson, and Mihaela van der Schaar. OrganITE: Optimal transplant donor organ offering using an individual treatment effect. In Advances Neural Information Processing Systems, volume 33, pages 20037-20050. Curran Associates, Inc., 2020.
[6] Jeroen Berrevocts, Ahmed Alaa, Zhaozhi Qian, James Jordon, Alexander ES Gimson, and Mihaela Van Der Schaar. Learning queueing policies for organ transplantation allocation using interpretable counterfactual survival analysis. In International Conference Machine Learning, pages 792-802 802. PMLR, 2021.
[7] Srikant Devaraj, Sushil K Sharma, Dyan J Fausto, Sara Viernes, Hadi Kharrazi, et al. Barriers and facilitators clinical decision support systems adoption: systematic review. Journal of Business Administration Research, 3(2):36, 2014.
[8] Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforce- ment learning. In International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.
[9] Vikram Kilambi, Kevin Bui, and Sanjay Mehrotra. Livsim: an open-source simulation software platform for community research and development for liver allocation policies. Transplantation. 102(2), 2018
[10] David Thompson, Larry Waisanen, Robert Wolfe, Robert M Merion, Keith McCullough, and Ann Rodgers. Simulating the allocation of organs for transplantation. Health care management science, 7(4):331-338, 2004.
[11] John PA loannidis. Why most published research findings are false. PLoS medicine, 2(8) 2005.
[12] Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974.
[13] C Ahn, H Amer, D Anglicheau, N Ascher, C Baan, B Bat-Ireedui, Thierry Berney, MGH Betjes, S Bichu, H Bim, et al. Global transplantation covid report march 2020. Transplantation, 2020.
[14] Agnes Debout, Yohann Foucher, Katy Trébern- Launay, Christophe Legendre, Henri Kreis, Georges Mourad, Valérie Garrigue, Emmanuel Morelon, Fanny Buron, Lionel Rostaing, et al. Each additional hour of cold ischemia time significantly increases the risk of graft failure and mortality following renal transplantation. Kidney international, 87(2):343-344 2015.
10

---

Under review as a conference paper at ICLR 2023
[15] J Adam van der Vliet and Michiel C Warlé. The need to reduce cold ischemia time in kidney transplantation. Current opinion in organ transplantation,, (8(2):174-778, 2013.
[16] James Stahl, Jennifer Kreke, Fawaz Ali Abdul Malek, Andrew J Schaefer, and Joseph Vacanti. Consequences of cold-ischemia time primary nonfunction and patient and graft survival in liver transplantation: a anta-snalysis. PloS one, 3(6):c2468, 2008.
[17] James Neuberger, Alex Gimson, Mervyn Davies, Murat Akyol, John O'Grady, Andrew Bur- roughs, Mark Hudson, UK Blood, et al. Selection of patients for liver transplantation and allocation of donated livers in the uk. Gut, 57(2)2252-257 2008.
[18] Doina Precup.
Eligibility traces for off- ooflpo policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
[19] Philip S Thomas. Safe reinforcement learning. PhD thesis, University of Massachusetts Libraries, 2015.
[20] Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning.
arXiv preprint arXiv: 103.4601, 2011.
[21] John Hammersley. Monte carlo methods. Springer Science & Business Media, 2013.
[22] Michael JD Powell and Swann. Weighted uniform sampling a monte carlo technique for reducing variance. IMA Journal of Applied Mathematics, 0(3):228-236) 1966.
[23] Yash Chandak, Scott Nickum, Bruno Castro da Silva, Erik Learned-Mille Emma Brunskill, and Philip S Thomas. Universal off off-policy evaluation. arXiv preprint arXiv:2 /2820, 2021.
[24] Keisuke Hirano, Guido W Imbens, and Geert Ridder. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 7 71(4): 161-1189, 2003.
[25] Donald B Rubin. Estimating causal effects from large data sets using propensity scores, Annals of internal medicine, 127(8 Part 20.557)):77763. 1997.
[26] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[27] Guido W Imbens and Donald Rubin. Rubin causal model. Microeconometrrcc, pages
229-241. Springer, 2010.
[28] Steven Piantadosi. Clinical trials: a methodologic perspective. John Wiley & Sons, 2017.
[29] Stuart Pocock. Clinical trials: a practical approach. John Wiley & Sons, 2013.
[30] Lawrence M Friedman, Curt D Furberg, David L DeMets, David M Reboussin, and Christo- pher B Granger. Fundamentals of clinical trials. Springer, 2015.
[31] Rehecca DerSimonian and Nan Laird. Meta-analysis in clinical trials. Controlled clinical trials, 177)-177-888 1986.
[32] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[33] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5033. IEEE, 2012.
[34] Joel Z. Leibo, Edgar Dué nez Guzmán, Alexander Sasha Vezhnevets, John P. Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charles Beattie, Igor Mordatch, and Thore Graepel. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International Conference on Machine Learning, pages 6187-6199, PMLR, 2021.
[35] Charles Beattie, Thomas Köppe, Edgar A Duéñez-Guzmmá, and Joel Z Leibo. Deepmind lab2d. arXiv preprint arXiv:2011.07003 2020.
[36] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schritwiieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint 1rXivv1108.04777 2017.
[37] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq. Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv: 16Xiv 03801, 2016.
11

---

Under review as a conference paper at ICLR 2023
[38] Tom Stepleton. The pycolab game engine, 2017.
[39] Jersey Neyman. Sur les applications de la théorie des probabilités aux experiences agricoles: Essai des principes. Roczniki Nauk Rolniczych, 10:1-51, 1923.
[40] Paul W Holland. Statistics and causal inference. Journal of the American statistical Association,
8 81(396):945-96 960, 1986.
[41] Mohd Shoaib, Utkarsh Prabhakar. Sumit Mahlawat, and Varun Ramamohan. A discrete-event simulation model of the kidney transplantation system in rajasthan, india. Health Systems, 1 11(1):30-47, 2022.
[42] Shoaib Mohd, Navonil Mustafee, Karan Madan, and Varun Ramamohan. Leveraging healthcare facility network simulations for capacity planning and facility location in a pandemic. Available SSRN 3794811, 2021.
[43] Cecilia Nardini. The ethies of clinical trials. Ecancernediiaasciinee,, 2014.
[44] Jack Cuzick, Robert Edwards, and Nereo Segnan. Adjusting for non-compliance and contami- nation in randomized clinical trials. Statistics in medicine, 16(9):0017-109,, 1997.
[45] David Maxwell Chickering and Judea Pearl. A clinician's 's tool for analyzing non-compliance. In Proceedings of the National Conference on Artificial Intelligence, pages 1269-12277 1276, 1996.
[46] Patrick S Kamath, Russell H Wiesner, Michael Malinchoc, Walter Kremers, Terry M Therneau, Catherine L Kosberg, Gennaro D' Amico, E Rolland Dickson, and W Ray Kim. A model to predict survival patients with end-stage liver disease. Hepatology. 33(2):464-47,, 2001.
[47] Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference, pages 56- 61, 2010.
[48]
The pandas development team. pandas-dev/pandanl Pandas, February 2020.
[49]
Alicia Curth and Mihaela van der Schaar. Nonparametric estimation of heterogeneous treat- ment effects: From theory to learning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 1810-1818, PMLR, 2021.
[50] Sören R Künzel, Jasject S Sckhon, Peter Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 1 116110)4416-44165 2019.
[51] loana Bica, James Jordon, and Mihaela van der Schaar. Estimating the effects of continuous- valued interventions using generative adversarial networks. Advances in Neural Information Processing Systems, 33: 13:16434-16445 2020.
[52] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-319 2021.
[53] Edward H Kennedy. Optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint arXiv:2004. 2020.
[54] Scott Powers, Junyang Qian, Kenneth Jung, Alejandro Schuler, Nigam H Shah, Trevor Hastie, and Robert Tibshirani. Some methods for heterogeneous treatment effect estimation in high dimensions. Statistics in medicine, 37(1 17:1767-17877 2018.
[55] W Ray Kim, Ajitha Mannalithara, Julie K Heimbach, Patrick S Kamath, Sumeet K Asrani, Scott W Biggins, Nicholas I. Wood, Sommer Gentry, and Allison Kwong. Meld 3.0: the model for end-stage liver disease updated for the modern era. Gastroenterology. 161(6):1887
1895, 2021.
[56] Osvald Nitski, Amirhossein Azhie, Fakhar Ali Qazi-Arisar, Xueqi Wang, Shihao Ma, Leslie Lilly, Kymberly D Watt, Josh Levitsky, Sumeet K Asrani, Douglas S Lee, et al. Long-term mortality risk stratification of liver transplant recipients: real-time application of deep learning algorithms on longitudinal data. The Lancet Digital Health, 3(5):e295-e33) -305, 2021.
[57] Uri Kartoun. Towards optimally replacing the current version of meld. Journal of Hepatology, 2022.
[58] Ina Jochmans, Maricke van Rosmalen, Jacques Pirenne, and Undine Samuel. Adult liver allocation in eurotransplant. Transplantation, 101(7):142-11550 2017.
12

---

Under review as a conference paper at ICLR 2023
[59] David Goldberg, Alejandro Mantero, Craig Newcomb, Cindy Delgado, Kimberly Forde, David Kaplan, Binu John, Nadine Nuchovich, Barbara Dominguez, Ezekiel Emanuel, et al. Develop- ment and validation of model to predict long-term survival after liver transplantation. Liver transplantation, 27(6):797-8077 2021.
[60] Katie L Connor, Eoin O' D'Sullivan, Lorna P Marson, Stephen Wigmore, and Ewen M Harrison. The future role of machine learning in clinical transplantation. Transplantation. 105(4):723-735, 2021.
[61] Dennis Medved, Pierre Nugues, and Johan Nilsson. Simulating the outcome of heart allocation policies using deep neural networks. In 2018 40th Anmual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 6141-6144, IEEE, 2018.
[62] Manuel Dorado-Moreno María Pérez-Ortiz, Pedro A Gutiérrez, Rubén Ciria, Javier Briceño, and César solving an imbalanced liver transplantation problem. Artificial Intelligence Medicine, 77:1-11, 2017.
Dynamically weighted evolutionary ordinal neural network for
[63] Dennis Medved, Mattias Ohisson, Peter Höglund, Bodil Andersson, Pierre Nugues, and Johan Nilsson. Improving prediction of heart transplantation outcome using deep learning techniques. Scientific reports, 8(1) 2018.
[64] Kyung Don Yoo, Junhyug Noh, Hajeong Lee, Dong Ki Kim, Chun Soo Lim, Young Hoon Kim, Jung Pyo Lee, Gunhee Kim, and Yon Su Kim. A machine learning approach using survival statistics to predict graft survival in kidney transplant recipients: a multicenter cohort study, Scientific reports, 7(1):1-12, 2017.
[65] Dimitris Bertsimas, Vivek F Farias, and Nikolaos Trichakis. Fairesss, efficiency, and flexibility in organ allocation for kidney transplantation. Operations Research, 61():733-87 2013.
[66] Dimitris Bertsimas, Vivek F Farias, and Nikolaos Trichakis. On the efficiency-fairness trade-off. Management Science, 58(12):2234-2 2250, 2012.
[67] Amir Elalouf, Yael Perlman, and Uri Yechiali. A double-ended queueing model for dynamic allocation of live organs based on best-fit criterion. Applied Mahematical Modelling, 60:179-
191, 2018.
[68] John P Dickerson, Ariel D Procaccia, and Tuomas Sandholm. Failure-aware kidney exchange. In Proceedings of the fourteenth ACM conference on Electronic commerce, pages 323-340. 2013.
[69] Mustafa Akan, Oguzhan Alagoz, Baris Ata, Fatih Safa Frenay, and Adnan Said. A broader view of designing the liver allocation system. Operations research, 60(4):757-770, 2012.
[70] Andres E Ruf, Walter K Kremers, Lila L Chavez, Valeria Descalzi, Luis G Podesta, and Federico G Villamil. Addition of serum sodium into the meld score predicts waiting list mortality better than meld alone. Liver Transplantation, 11(3):336-343 2005.
[71] Neta Gotlieb, Amirhossein Azhie, Divya Sharma, Ashley Spann, Nan-Ji Suo, Jason Tran, Ani Orchanian- Cheff, Bo Wang, Anna Goldenberg, Michael Chassé, et al. The promise machine learning applications in solid organ transplantation. NPJ digital medicine, 5(1):1-13, 2022.
13

---

Under review as a conference paper at ICLR 2023
A EXTENDED RESULTS
A.1 RUNNING A SIMULATION
A simple example. Consider Figure 6 where the MELD-Na policy in two hypothetical scenarios: one where COVID-19 happens, and one where it doesn't. With Al1Sim we can model each scenario confidently. For this particular example, we fix the seed of Allsim and only change the supply of organs by giving two different resource arrival processes:
Two hypothetical scenarios
1,000
750
500
sp COVIn-11
Aug.
Mar.
Oct
def :t)
if < 600:
COVID-19
return .5
happened
else: return .25
Figure 6: Two hypothetical scenarios. We require to evaluate a policy (e.g. MELD-Na) in hypothetical (contradictory) scenarios.
def (t) return 5
Tuning the simulation. Running and composing simulation, is as simple defining how wish to let the supply of resources and recipients evolve t. Such evolution is expressed a changing arrival rate (crf. Equations (3) and (4). In particular, one only needs to define År,r, normalisation handled by A 1Sim. However, even there does A 1imm offer detailed specification possibilities. Like y., we can specify custom function for 0z,r also. Normalisation, will then respect the value of a at time t. For example, if we wish the total arrival rate of all resources to remain constant at 5, we simply a,- constant function:
alpha_ lambda
This way, no matter how complicated our functions for Xr, are, we know that the total arrival rate, across all conditioned distributions for the resources, we remain fixed at 5.
Inference. One additional component Allsim, is its Inference module. our simulations, we relied on OrganITTE [5]. where we adopted the original source-code provided by the authors, to our Inference module. Having causal model, allows to have unbiased estimates for the resource to recipient pairing's 'ss outcome (column "Y" in Table 2).
Building a custom Inference object, as simple as inheriting from Inference, which requires the user to implement the infer (x: np ad.rddrray, r: np. odaaraar) -> Any function.
A.2 ANALYSING Allsim's OUTPUT.
Running of simulat ion. simulate (policy, T-T) yields a DataFram object, contain- ing each match made by the policy, in the environment simulated by the as. S Simulation object. Consider Table for an excerpt of the output for which we provided some aggregate results in Figure 5. Table 2 is exactly the type of data may expect when learning about policy, learning a new (ML-driven) policy [5].
Given the output of our simulation, over time. For example, one may be interested how each gender evolves w.r.t. allocations, with the changing supply of resources (as in Figure 5). Naturally, this is but one example of what one can achieve in terms of analysis on MELD (or any other policy), using Allsing im. As such, we encourage the reader to browse through the provided code, in order get sense of what the possibilities really are when using Al1Sim.
each variable, both donor as well recipient progress
B DETAILS ON AllSim
While all source code is provided in our submission, we provide some additional detail on A 6.11sin lsin''s functionality, structure, and future goals.
14

---

Under review as a conference paper at ICLR 2023
Table Example output. When donor covariates NaN, know patient died the wait list, they did not receive donor organ. Note that, only display some covariates, we have included code in our submission. should the interested reader want to inspect the complete Dat rame.
Donor covariates
Recipient covariates Sodium
Age
BMI
INR
Creatinine
Gender
Y
Day
26.1628
27.7080
4.8144
138.10
11.351
M
1792.9
1
NaN
NaN
0,5663
137.71
0,4535
5.0
7
NaN
NaN
6.694
135.63
1.7323
F
24.948
26
34.893
21.852
3.8375
134.15
7.9500
M
257.79
0.55
Arrival rates over time
M
F
0.5
0.45
300
600
900
Figure 7: Recipient-gender in function of time. Given the MELD policy, and the changing supply of resources (efr. Figure we find that recipient- seneene-eeeerr changes slightly. Note that, type analysis easily done given A1 $1 m, as we can easily run analysis on the provided arraa object, returned by our simulation.
B. NEAT FUNCTIONALITY
One-hot encoded variables. In heterogeneous data, one may expect mixture of continuous and categorical variables. Good practice to handle categorical variables, is to one-hot encode them into columns of ones and zeroes, for each category. Most density estimation methods, do automatically sample these mixtures of continuous and categorical data. Despite its performance and simplicity, standard Kernel Density estimator, like the one we use our simulation, cannot handle these data out-of-the-bos However, implementation does. One only needs to provide the specific groups of columns which compose the categorical variables.
For example, the gender column for the recipient may be one-hot encoded into: "GENDER_O" and "GENDER_1". When fitting our xDEDensity, we simply provide these columns to the wne.hot_encodee ["GENDER_O", "GENDER_1"] parameter, and the KDEDens automat- ically transforms these columns, in such a way that only one of them (being the one with maximum probability), is translated into
Auto-scaling. Dealing with a normalised arrival rate can be tricky to detirmine up front. As such, Allsim does this automatically. By solely providing alpha: Callable [[bt][[int]]] float], Al18im's Poi reisssnSystem objects will normalise any output that the System's Pois sosssscoeeeseeeeo ses may have. Naturally, normalisation will depend the chosen process, and as such does not propagate up to the general System nor Process. If, for example, on wishes implement alternative point process (not Poisson process), normalisation may look differently.
Automatic conditioning. Another nice functionality provided in Al AllSim automatic conditioning, in case no Condition is provided. A Density does this, by first clustering the data (from which will later learn its density), into clusters. Naturally, one will have to provide the amount of clusters before the Density is able to automatically condition. Having these separate clusters, still allows defining arrival rates and processes for each cluster center, i.e. must match the amount of Processes provided to the System.
B.2 PACKAGE STRUCTURE
The A A1l 1.1.Sim package structure follows:
15

---

Under review as a conference paper at ICLR 2023
infer.py s sim.py py
come
counterfacttall_iffereee
counterfacctua__nodell
policies
base.py
policy.py
The two main components are in infer. py and sim. py, where infer. py contains everything related to Density, Process, and their subclasses; and sim. py concerns the Simulation class.
In out come and policy, we provide basic implementations of some well known policies and coun- terfactual models. Note that these are actually non-essential to Alisim, they can be completely user-defined, as long as they respect the Inference and Policy abstract classes, provided the respective folders.
The base classes (Density, Policy, and Inference) can all be implemented by the user. Consider following class structures:
Inheritance.
class Density: aef init (self
condition: Condition,
below for more details
K: int-1,
drop: www.darraa-p..ddrrr... (I' condition'
@abstractmethod def Cample(self, n: int=1) -> np.ndarray:
10
@abstractmethod
13
def fit (self, D: pd. rdtaataFrame) -> None:
14
15
class Policy:
17
aet init (seff, self, name: str, initial_waitlist, nd.ndarray, dm: OrganDatalloulle) standardised datamodule should the policy want to learn from data
18
19
22
23
@abstractmethod:
25
def sclect (self, resources: np. np.ndarray) Tuple ndarray,,,, np. ndarray]: # returns recipients and resources, matched
26
29
@abstractmethod
30
aef (self,,f, x: np. .darraa) -> None:
31
i allows policy to add recipients to internal waitlist (if
needed)
32
33
34
@abstractmethod
def remove (self, np.ndarray) -> None:
allows policy to recipients, when for example they
dic
37
38
class Inference
def initt (self,
16

---

Under review as a conference paper at ICLR 2023
the Inference class acts wrapper for model: Any, any model type
42
mean: float=0, mean and std are necessary to scale the
outcomes, assuming
std: float=1):
standard scaling
45
46
aef caall (self, x: ad.ndarryy, r: np.ndarray, *args: Any, **
kwargs: Any) ->Any: self. infer E, *args, **kwargs)
49
@abstractmethod
30
def infer (self, x: np ndarryryy,, r: nd.ndarray, *args: Any, "*
kwargs: Any) -> Any:
The Condit inn class then, wraps function from a set of labels in the dataset, to numerical value. One may choose to combine multiple variables, or just one, as we have with the "AGE" variable in Figures 4 and 5. Please find an example initialisation in Figure 4. While one can inherit from the Condition class, we would advice to implement a custom function (as the lambda function we had), and provide it to the Condition object.
Future goals open-source Our goal is S.mm to a benchmarking standard when evaluating ScRAPs. An important milestone for this, is to completely open-source simulation. Doing so, allows other researchers to scrutinise, enhance, and discuss how we should move beyond what we can do today. Below, we provide two points we believe our community should discuss.
C USING A11Sim
Here we provide the basic code, that will generate Figure 5. Naturally, the complete code is provided in the supplemental materials.
load data
X, R, Y custom_ lonttontiooc (data) organite Load orad_oorgaitt_meee model (Location) should Load imp ed Inference object
DENSITY LEARNING RESOURCES
bins_r 130, 45, 60]
def conditien_functionn function.onnnmo (
return np.digitize (age, bins=bins_rd r) .item ()
10
12
conditicn_r infer. Coner.conditiont
13
labels= (' AGE' 1, cunetiion-oondiioo_ffmmeee op +
14
17
18
kde r = infer. iwf.r. www.DeDDDDtttcceeeeeee annbeessttycomditttaaooootd..... r, X=condition r. r. r.options
kde_r.fite (R, one_hotteecooed-eeeeee one_hhot_encoddddrrrcssss.
20
21
22
RECIPIENTS
bins_x [30, 45, 60]
25
def cendition fww.ionnfnnttoooxxtiieo (bilir)
26
return np digitize (bilir, bins-bins .item ()
27
condition infer.Condition!
1abel s- AGE
17

---

Under review as a conference paper at ICLR 2023
function=conditionn functicn_2 options=len (bins_x)
32
33
k infer. LDDDeenstyycconniiiiooooooooo X=condition_x option
35
ide_x.fitt (X, one_hot_encodedde one_hot__nccoded-rrrup....
36
37
BUILD THE SYSTEMS
resource_system, patient_system dict (), dict ()
40
41
def update_lan_O(t)-
return (1 (x+ng.exp (- ))
44
def update aodate_laa___ 1am :
45
return (1 (1+np.cxp ( (t- 350) /150) ))
46
def update 1am lam_2 (t))
(1 (1+np.exp (t-150) ))
49
b - (1 (1+np.exp (t /100)))
50
return (a b)
def update Iam (t))
53
a (1 (1+np exp (t-150) (1500))))111)1 (150)1)
54
b b (1 (ttp.exp((tt650))) (t-650) ((t0)))))000)) )100)))
55
return (a b)
58
resource_systenn [0] infer.PoissonProne iapaatt_lalaaaaaapttttttttee
59
rescurce_aystent [1] infer.PoissonProcess (ipdate_lannuupptte_____aa
60
resource_ayst cm [2] infer.PeissonProcct httpaaaacaccccaaarrauttttaaaaaaa
resource_systemm|3
62
63
64
patient_eystem [3]
patient_system [2] infer scccprocess (updare_lammmppdaee__lml))
patient_system [1]
67
stiient_systemt infer.PoisscnProces (update Lan.attttean-
68
resource_process infer. Polssccyyssnn
71
density=kde_r,
72
@ystemrrresourrc_syytteoe
13
alpha=lambda t: 5,
normalize-Trus)
76
patient_process infer. PoisscnSytten
77
density=kde_x,
78
system--atient__eytte _ystem, alpha-lambda t: 7, normalize-Irue)
81
82
eeraantee.mdee..ce
simulation sim. Sim(
86
87
Inference-organito
90
policy MELD
91
name=' MELD', initial maitiist-simlationn innteetttooonntnneeeee internaL_waitlliee waitlist, dm=
cim
18

---

Under review as a conference paper at ICLR 2023
94
df simulation, simulate (policy, T-1021)
C.I GETTING STARTED
To use AlISim, user requires at least dataset of the following type: D = {(x, Ru, Y): t.u N+ h. with t, u indicating the time of arrival. In principle, this is sufficient to start using AllSim already. In fact, this is exactly what provided AllSim in experiment in Figure 4. Let us elaborate:
AllSim requires to specify the following components:
1. A counterfactual model
2. The patient and resource arrival processes
3. The patient and resource densities
Components 1 and are easily implemented using existing packages like econml (https: : econml. azurewebs tes net) and stable/). /), respectively, They only require to use the model. fit API to be applied to the users' data:
(https: :tsp:p///t /tssssiiiitteatttieeeeeeeeeeeeee org/
counterf econml. BasecatcEstimmttor/ fitt (Y, R, X)
density_X, density_R allsim, innrrmm allsim, infer. KDEDensity() S.nsity_Xffit.l R.msity_R.fit
The only place where we need AllSim specifically is when we create the arrival processes, and the eventual simulation. Creating an arrival process on data, first requires us to regress the arrival probability on time:
from sklearn. preprocessing import PolynomialFeatures from sklearn winear mmocc_ import LinearRegression from caleaan.pipeliie import Pipeline
func_X (Pipelen(iielinee( ('poly', PolymomialFeaturre (degree=))),
('Linear', Iinearfegression (tt.::tttiiitereeeeeeee
with amount () a function that returns how many of x arrived at t in
10
fitt amount (X, t))
11
12
with the arrival processes patient_process = allsim. PoisssoPososssseesss (Fanc_X) _X
resource_process allsim. wnf.r. Pnfer. (func_R)
15
16
patient_oysten allsim. infer htw..ahhffe..esssssststttmmmme (denssstttdddeeeyyy...
patieent_rroess)
resource_system allsim. .nw.rrrPr... Pai www.PPissssssnp. ssonSystem (densi ty_R,
rescurce_process)
The simulation then created as:
simulation asim. sim.
aoouurce_systemerreeooree. system, siiient_system=pptet hetpppatteetttttttteeeeeee inferencee-cunterr
19

---

Under review as a conference paper at ICLR 2023
The only thing left is to actually the policy against created simulation:
df simulatioon simulate (policy, T=100) # thats it!
D SOCIAL IMPACT
is very clear that machine learning has the potential to transform healthcare. Its success both in other domains and already within healthcare very promising. ML-based policies have the potential to extend lives. However, as with almost any other machine learning method, there are risks associated with their deployment in a real healtheare setting. Before any (experimental) ML--based policy (or even non-ML-based policies for that matter) are to be deployed, they requrie thourough testing.
We believe Al Allsim will enable practitioners leverage their data and evaluate their policies in rigorous manner. AllSim iim fully recognises the difficulties associated with evaluating policies that diverge from the data-generating policy and aims to mitigate these difficulties by employing tried and tested methods from causality. Naturally, there are some caveats: A1 1 1S inherits the potential assumptions made by the counterfactual method, furthermore, if real data is used, it is crucial is at least somewhat related the environment the tested policy will end up operating in. If for example, one aims to evaluate a policy on a domain that is completely unrelated, All Allsim's learned simulation will provide the tested policy with unrealistic resources, recipients, and arrivals.
E ALLSIM IN VACCINE DISTRIBUTION SCENARIO
AllSim is a general purpose simulator which evaluates scarce resource allocation policies. While we mainly focus on organ- saann-tassllaatttonn in our main text, we show in this section that AllSim is also applicable in other settings. To illustrate, policy evaluation system AllSim. This use-case will show how few adjustments one has to make with respect to the presented settings in our main text.
show how
implement vaccine distribution
Compared to the organ-allocation problem, in vaccine distribution, each resource is the same and they arrive in batches. Furthermore, the type of patient-in-need is also much broader (in fact, they cover the entire population). Yet, AllSim is perfectly capable of modelling this scenario given the following:
Batch arrival simply requires multiplier. For example, if the Poisson process samples value of 2 on one day, we could simply interpret this as two batches of 1000 doses.
As all vaccines are the same, no longer require density of resources we required for organ allocation. This can be done by implementing a dummy-density that always returns (or the amount of vaccine).
A broader patient-type in AllSim is achieved by retraining the density of recipients over the entire population.
These implementation details relatively simple implement and easily done using AllSim's modular API.
While not necessarily a problem in vaccine distribution, recipient arrival in the ICU in setting of infectious disease (such as COVID-19), is definitely different as compared to the organ-allocation setting. With organ-allocation, we can safely assume a Poisson process for recipient arrival recipi- ents enter the system independently. This is of course not true in an infectious disease scenario: one recipient arriving may indicate higher infection rate. As such, recipients do not arrive independently.
With the above, it is clear that we can no longer rely on Poisson arrival process for recipients entering the system. Instead, to accurately model situation of infectious disease, we recommend using a Hawkes process. To further illustrate, we include some code below showing exactly how one may go about including such a Hawkes process in AllSim.
class HawkesProcess (PoissonProcess))
def
init_
self, 1mm: float=.1,
20

---

Under review as a conference paper at ICLR 2023
update lam: Callable [int], float]=lamd t: E, delta: float=.1, a: float=.2
assert 0, "a should larger than equal assert delta 0, "delta should be larger than 0"
11
12
13
super () init (1am, update_lam)
15
self.a, self.delta a, delta self._sample []
16
17
def get lam unnormalized t: int) -> float:
self. baseline lam np.sum(
20
self.a self.beta ex.exp (-beta (t self. samples]
< (t)))
21
def progress (self, t: int, float=1) -> int: self. lam
vee__lam_unnormal (t). eqs. (5, 6)
24
sample np. random. wecammpppissi peipppissnnnoo (lam=sel lam)
25
salf.ssappess sapppes...ppeed (sample) return sample
26
Naturally, if recipient arrival is indeed dependent on previous arrivals, simply learning the arrival process (as have for figure 3) should model such self-exciting process automatically. Furthermore, the above implementation is simple linear univariate Hawkes process. We refer to hawkes ib (https: github....t com/ www.errtukmmee/hwwweeee. of Hawkes processes.
for implementations of more types
F COUNTERFACTUAL INFERENCE
AllSim relies for large part on counterfactual inference. As real-world data is collected under an active policy, testing an alternative policy would almost immediately diverge from the allocations made in the data. As such, we have infer an outcome from a pair made by the tested policy. the tested policy is indeed different from the active policy, then we are unlikely to find comparable pair the data. The above is question most conisdered in research on counterfactual inference [40].
Of course there are many models that perform counterfactual inference. As such, we provide brief overview of the type of models one may resort to. Naturally, this list is non-exhaustive, may guide user to model fit for their use-case.
Broadly speaking, we recognise a few "meta-categories,, or meta-leurners [49, 50]. The are as follows:
T-Learner. Simplest is to learn model (such neural net random forest) for each treatment. In the vaccine distribution case that would mean learning a model on recipients who haven't received vaccine, and different model recipients who have received vaccine.
S-Learner. Contrasting the above, an S-Learning learns one model, where the treatment is considered part of the covariate set. This allows for more flexible treatment, such as continuous [51] or multivariate treatments [6]. In our main-text we use OrganITE counterfactual model [5], which can be considered an S-Learner.
X-Learner. Used specifically for estimating the treatment effect directly, an X-Leamer first learns the outcome functions (such the T-Learner), then imputes the dataset with the completed treatmet effect (or the estimated counterfactual outcome), and then learns treatment effect with third model directly the completed samples. While useful for treatment effect estimating, performing counterfactual inference with an X-Learner would default back to either S-Learner or T-Learner.
R-Learner. An R-Learner like the X-Learner, estimates the treatment effect directly [52]. Specifically, it leams the outcome functions well as propensity estimate. Using these
21

---

Under review as a conference paper at ICLR 2023
models, the R-Learner optimises custom loss function based on the propensity, the outcome model, and a cross- validation setup.
DR-Learner. The DR-Learner or doubly robust-learner is an iteration of the X-Learner [53]. Like the X-Leamnr the DR-learner first estimates the outcome models, then completes the dataset to learn a treatment effects model using standard supervised learning. Then the DR-Leamner repeats this process using the treatment effects model from the first step.
While there are more meta-learners than what we reported above (e.g. U-Learner [52] or CW-Learner [54]), but they are much less adopted and unlike the S-Learner and T-Learner not fit for estimating the counterfactual outcomes as they, like the X-Learner, R- R-Learner, and DR-Learner, fit the effect function directly. We point the interested reader to the following papers: [49, 50, 52]; or to the following open-source libraries for various implementations: causal mm (https: /github. com/uber/cal causa lml), or econml
www.:/aw..ooooooo...o...o......o BBooML.)
G EXTENDED RELATED WORK
Let us discuss how the literature which introduces novel allocation policies, in particular to donor- organ allocation. From this, we observe that they all come up with their own unique simulation in order to validate their proposal.
Allocation polices from medicine. As an example, consider the following papers introduced the medical domain (we focus here the liver allocation setting this is also where focused in our paper): [17, 55-59]. Each paper, all coming from different research groups, provide a custom simulation to validate their allocation policy. While some focus only gathering test data (such as the recent (author?) [55]), others construct a simulation by simply iterating over the patients in the sequence they arrived in reality (such (author?) [17J). While we would note some flaws in the way these policies are evaulated (e.g. counterfactual trajectories when allocations misalign), the truly striking observation that each evaluation strategy is different! Not one paper reuses the same simulation; AllSim may change this going forward.
Allocation policies from machine learning and OR. The same is true for the ML [4-6, 60-64 and OR [65-69] communities. It seems that both the ML and OR community is focused more and more on this important problem- which is fantastic! But also warrants careful evaluation. Furthermore, if we find that the evaluation strategies in medicine (which generally propose linear combinations of features [46, 70] or simple CoxPH models [17, 55, 57, 59]) have shortcomings, then this certainly the case for much more complicated strategies introduced in ML or OR. In fact, recent survey confirmed exactly that: [71, cfr. Limitations of ML in transplant medicine].
Given the ubove, with respect to novelty, we suggset: had there been purpose built simulator, would everyone be suggesting their own simple evaluation method paper-by baaeeb-y-papr?? We believe there is a clear literature gap which AllSim aims fill.
SimUnet. This brings us to SimUnet, which we thank the reviewer for highlighing. While indeed related, SimUnet is in fact very different from AllSim. In particular, from the online README document (see tttps:/ / unos wwt/wppccoo cono-coonett... htteeennt/aaaaaee
S--page-Simnnee pdf) be seen that SimUnet more concerned simulate offers from the transplant clinician's point of view; not allocations from an overarching healthcare system (such as UNOS, or the NHS).
22


Reviews:
[{"review_id": "kwGe6ww4A6-", "reviewer": "Reviewer_Ga5q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a general module for data-driven simulations for resource allocation. They outline and demonstrate this codebase using an organ allocation scenario.\n", "summary_of_the_review": "This paper is interesting and relevant and well written. The simulation module does seem very useful, and the experiments are compelling. However I'm not sure how to assess novelty here. While the module appears to be very useful, it's not clear that it contributes anything new to the literature. (Aside -- ICLR seems like a strange venue for this paper.) Nevertheless, this seems to me like a strong contribution. \n\nIf the authors can clarify the novelty of this paper (perhaps using additional simulations, discussion, applications, etc.), I would advocate strongly to accept this paper.\n\n\n--- \n\nUpdated this review after reading comments and other reviews. See my comment from Nov. 18.", "strength_and_weaknesses": "\nStrengths:\n- this paper & methods are very clear, and their discussion is very thorough\n- the simulation module seems very sound, and they provide code\n- the experiments are compelling, though only use 1-2 examples. \n\nWeaknesses:\n- it's difficult to assess novelty here. this paper synthesizes and combines many important topics related to resource allocation. but it's not clear whether this contributes any new ideas the literature.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XHWUe05wEjZ", "reviewer": "Reviewer_uUjW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a simulation environment for evaluating scarce resource (e.g., Kidney, Liver) allocation policy. The proposed allocation simulator (AllSim) is modular, learnable and customizable. The simulator provides functionality for generating resource and recipient distribution, allocating resources, and evaluating performance of the allocation. It allows practitioners to properly tune different knobs of the simulator to match their practical needs. The paper also demonstrates the performance of different components experimentally.  ", "summary_of_the_review": "The paper proposed a novel simulator with end-to-end functionalities for evaluating a scarce resource allocation policy, especially in medical domain setting. An accurate simulator would be hugely effective for research progress in the medical domain as well as offline RL space. I appreciate the specific functionalities of resource allocation and counterfactual inference, and their detailed explanation in experimental results. Moreover, the simulator could be easily reproduced as the source codes are shared. Having said that, I still have some reservations regarding the technical/methodological contributions, e.g., how well it generalizes to different organ settings and how much finetuning is required are not clear to me, and its practical adoption, e.g., I expected an human evaluation on how satisfied the practitioners are with the simulator, or how much effort the practitioners need to finetune all the knobs, or even if it is feasible to properly tune all the knobs to generate accurate behavior or outcomes of a patient health?  ", "strength_and_weaknesses": "Scarce resource allocation (e.g., Kidney/Liver matching) is an extremely important and challenging problem that requires online planning. While there has been significant effort in AI and OR communities to find optimal online matching of demand and supplies of these scarce resources, we absolutely need an accurate simulation environment for evaluation of different decisions. The proposed simulator can be used for counterfactual evaluation of policies learned by offline RL methods from a batch dataset. The simulator provides end-to-end functionalities from generating demand and supply of resources, allocating resources and evaluating a policy by supporting counterfactual inference. Having said that, I have some reservations regarding the technical novelties of the paper. There have been efforts for generating simulation environments for kidney or liver matching or transplantation (e.g., SimUNet: https://unos.org/solutions/simunet/). Does the simulator generalize to different settings of organ matching with simple tunable knobs? Moreover, tuning these knobs could be challenging for the practitioners as the outcome might depend on a variety of indicators for individual patients. I think it would be great if the authors can perform a human evaluation study and report statistics in terms of how much value the actual practitioners see in this simulator. If it does not pass the bar of practitioner’s expectation, then the counterfactual evaluation in the simulator might not be of practical use. On the other hand, I was expecting some guidance on how to tune the knobs of the simulator, especially for accelerating research progress in this domain. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5mzVPxQPmD4", "reviewer": "Reviewer_ACC7", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a simple simulation model which they argue can \"Learn\" from the environment. However, the authors seem unaware of teen area of formal methods and the tools in this area such as PEPA, CARMA OR SPN which this work appears to be a re-invention of.", "summary_of_the_review": "The paper contains only a simulation approach which has been widely covered in prior work. There is nothing in this work which is novel in the way of ICLR. The authors need to reflect on tools such as PEPA, CARMA and SPN. A more analytical analysis of the solution is needed.", "strength_and_weaknesses": "Strengths:\n - The English in the paper is clear and well structured\n - The ideas are clearly presented\nWeaknesses:\n - The work presents ideas which have been covered before using formal methods using tools such as PEPA, CARMA AND SPN. Comparison to these tools would seem essential.\n - The results section is only a small number of simple examples and does not compare the work to other approaches.\n - There are many statements in the work which are incorrect. And many which need supporting evidence such as citation to other works.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "-1-ZxaqfQqJ", "reviewer": "Reviewer_a2x4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an open-source framework to be used to allocate scarce resources to a pool of recipient.  The application example presented in the paper is allocation of organs to recipients, an application area where optimizing the allocation is extremely important.  The framework is meant to be used to explore dynamic policies for the allocation. The system used discrete time settings for the simulation, modeling arrival times for resources and for recipients and generating outcome predictions.  The framework, which learns from past data, is designed to be modular and customizable. An important aspect is the ability to do counterfactual reasoning and provide for each pair resource-recipient the potential outcome. ", "summary_of_the_review": "The paper proposes a framework for allocation of scarce resources to a pool of recipients.  The software is open source, making it easy to costumize it and extend it with additional modules.", "strength_and_weaknesses": "Strengths: the problem addressed is an important problem, not just for the application studies but for other potential areas of applications.  The framework proposed has multiple properties, like modularity and customizability. Modularity allows to connect it with external applications, custinmizability enables the users to decide what they need. Counterfactual inference is used to make predictions. The software is open source.\nWeaknesses: there is no indication that the proposed framework has been discussed with potential users, to guarantee it provides a valuable solution for the intended users. It is still up to the practioners to decide how to configure the system, what analysis to do, etc.  If the framework has to become usable by a variety of practioners, much more guidance would be needed for specific cases. This is not difficult and does not need to be done immediately.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject