SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Changing the policy in (i) is done simply by implementing the new policy according to the simulation interface (discussed in the next section). We stress once more that this paper does not provide guidance for allocation policies nor does it propose a new policy of any kind. In fact, the policies used in the following section are tried and tested policies, currently in use in practice. Changing the utility function for (ii) is easily done in AllSim as running the simulation does not depend at all on the chosen utility function! As AllSim provides a completely counterfactual dataset,  $\mathcal{D}'$ , the utility is computed post-hoc which allows us to always fall back on the generated dataset. Finally, perturbing arrivals (iii) is already discussed in Sect. 2.1; the arrival processes are perturbed through  $g$ .
Sampling data. Equations (1) to (4) provides us with  $\mathbf{X}(t)$  and  $\mathbf{R}(t)$ . Equation (5) provides us with an estimated potential outcome given  $X(t)$ ,  $R(t)$  and their allocations using  $\pi'$ . AllSim then carefully indicates a timestamp for each combination and then presents the decision maker with a new dataset:
$$
\mathcal {D} ^ {\prime} := \left\{\left(X, R, \hat {Y} (R), t\right) _ {i}: i = 1, \dots , N \right\}.
$$
Having a new (counterfactual) dataset based on  $\pi^{\prime},\mathcal{D}^{\prime}$ , allows to easily calculate various performance utilities of interest, which the decision maker can use to evaluate the allocation policy, pre-deployment:
Definition 2 (Pre-Deployment Evaluation) Let  $f: \prod_{k} \mathbb{R}^{k} \times \ldots \to \mathbb{M}$  denote an evaluation metric mapping a sequence of outcomes  $\{\mathbf{Y}(t)\}_{t=1,\ldots}$  to some space of evaluation outcome  $\mathbb{M}$  (e.g. for the average survival time, this would simply be  $\mathbb{R}$ ), where  $\mathbf{Y}(t) := \mathbf{Y}_{+}(t) \cup \mathbf{Y}_{-}(t)$ . Given a problem  $\mathcal{E}$  and policy  $\pi$ , the pre-deployment evaluation problem is to compute statistics of the distribution  $\mathcal{F}_{\mathcal{E},\pi}$  of evaluation outcomes  $f(\{\mathbf{Y}(t)\}_{t=1,\ldots})$ ; commonly, this would be the mean  $\mathbb{E}_{\mathcal{E},\pi}[f(\{\mathbf{Y}(t)\}_{t=1,\ldots})]$ .
Note that we have defined  $f$  in terms of the sequence of per-period outcomes such that it gives maximum flexibility: Depending on how individual outcomes  $Y$  are defined, we can measure point estimates (e.g. the mean survival), compare subpopulations (e.g. whether some types of recipients systematically receive more favourable outcomes), examine trends (e.g. whether outcomes degrade as the types of recipients arriving change), or potentially investigate more complex hypotheses.
3 AllSim Interface & Examples
Given the formal definition of AllSim presented in Sect. 2 (and further in Appendix A), we now introduce AllSim's programming interface and use it directly to provide some experimental results. We split this section in two major parts: first, we show the type of analysis AllSim can do for us, as well as how to tailor AllSim to the decision maker's needs; and then, we show how realistic the AllSim simulations are, compared to the factual data; we show that AllSim models realistic systems.
3.1 Example analysis and decision-maker specifications
As a first example, let us showcase an analysis to illustrate the possible impact AllSim may have in practice. Throughout this section, we will use the open-access United Network for Organ Sharing (UNOS) dataset which comprises 25 years of liver-to-patient allocation. Importantly, we had to make zero adjustments to our framework to fully capture these data, showcasing the generality of the
Figure 4: Specifying a simulation using AllSim. In the above, a decision maker defines a set of donor arrival rates, based on age ( $\lambda_{\bullet}$  and  $\lambda_{\bullet \bullet}$ ). Using these very simple, but custom, arrival rates, we see a direct influence in the user and resource distributions ( $p(\bullet)$  and  $p(\bullet \bullet)$ ). These perturbations constitute as perturbations of type (iii) as per Sect. 2.4. Finally, the decision maker tries out three different policies: MELD, MELD-na, and FIFO, which constitute as perturbation type (i). The result of these policies is shown on the right. The reported averages are windowed over 300 samples.
AllSim framework. We only use UNOS data until 2019 which, interestingly, predates the COVID-19 global pandemic. As such, it is impossible to evaluate policies using only these data: we need AllSim to model a counterfactual scenario that mimics what we saw during the pandemic to test a policy.
In Figure 3 we ran the MELD-Na policy in two hypothetical scenarios: one where COVID-19 happens (which resulted in a  $50\%$  drop in the donor liver arrival rates [23-25]), and one where it doesn't. With AllSim we can model each scenario confidently. For this particular example, we fix the seed of AllSim and only change the supply of organs by giving two different resource arrival processes:
1 defcovid(t):   
2 if t  $<  600$  ..   
3 return .5   
4 else:   
5 return .25   
6   
7 def no_covid(t):   
8 return .5
Figure 3: Two hypothetical scenarios. We require AllSim to evaluate a policy (e.g. MELD-Na) in hypothetical (counterfactual) scenarios. The x-axis is time, and the y-axis indicates survival time.
Having illustrated the power of AllSim, let us now show how a decision maker may introduce their perturbations (such as theCOVID and no_covid arrival processes from above) into the AllSim simulator. For this, we will provide the donor-organ system with two specific perturbations: (1) we will change the policies (from MELD, to MELD-na, and a simple FIFO policy), and (2) we will increase the user age and decrease the donor age. These two perturbations respectively illustrate perturbation types (i) and (iii), listed in Sect. 2.4 (recall that perturbation type (ii) was changing the utility which is done after a counterfactual dataset is sampled and hence does not require testing).
Consider Figure 4 which shows the resulting simulation and the found utility when perturbing the organ arrival rates as well as the allocation policies. The take-away from this experiment is not the performance of the policy (although, reassuringly, MELD and MELD-na<sup>1</sup> do outperform FIFO). Instead, we learn that increasing user age results in dropping the survival time, regardless of the donor age which is decreasing. This is not surprising, as older transplant patients simply have less time to live, whether they get an organ or not. Which leads to the next question: is AllSim realistic?
<sup>1</sup>Which are the policies used in Europe and the USA for donor liver allocation.
(a) Mean arrival of age and BMI..
(b) Depicted are the arrival rates for each age bracket. Note that, in this instance, we learnt these from data, but one can use any function.
Figure 5: AllSim (easily) simulates realistic environments. Using real-world data on donor organs, we let AllSim model 3 years of organ arrivals and compare it with the actual arrival as reported in the data. In Figures 5a and 5b we show AllSim's output (in donor age and BMI), given the code on the right. With minimal code, a simple condition (4 age brackets), and conservative models (polynomial regression to fit the arrival rates, and a Gaussian kernel density to model the organ densities), we find that AllSim accurately models the actual (real-world) arrival of organs as reported in the UNOS data.
import allsim as asim  
simulation = asim.init(  $\star \star$  kwargs) # Kwarges: # - resource arrival process # type (defined in API) # - user arrival process type # (defined in API) # - counterfactual model # type (defined in API) # - Object columns (X, R, Y)  
simulation.add_dataset(X, R, Y) # Adding a dataset should # automatically model arrival # processes  
df = simulation.evaluate(policy, T)
3.2 AllSim's realism
In this section, we will learn an AllSim configuration purely from the UNOS data (i.e. without a decision maker's input), such that we can compare AllSim's output side-by-side with what actually happened in UNOS. If they match up, we confirm that AllSim can output realistic scenarios (as UNOS is a real dataset). However, before we do so, we first show how we use AllSim from a programming perspective and configure appropriate arrival processes and densities for this particular use case.
First we determine how many users and resources we need to sample, once we know the amount we sample them from a generative model. The former is modeled through a Poisson arrival rate that changes over time, and the second is sampled from some learnt density. Of course, a user can implement their own arrival process by inheriting from the abstract ArrivalProcess class.
Importantly, we need to be able to condition the density on some pre-specified characteristic of the object of interest. For example, one may be interested in modelling the arrival of harvested organs of older patients distinctly from younger patients. An example of this is provided in Figure 5, where we show the changing resources coming in the system, alongside the code that generated the result.
Object densities. Before discussing a temporal arrival rate, we first discuss modelling the object's densities. Consider lns 3-12 in the righthand side of Figure 5. Using this code, we first define what we want to condition on, using a Condition object: in this case we formulate age brackets. With the KDEDensity class, which is a subclass of the abstract Density class, we can automatically model a density, conditioned on these age brackets. Each Density object implements a fit and sample function, which is used to sample new objects by the System, which we discuss next.
Arrival processes. Using a Density, we move on to lns 14-26, where we first build a system of multiple arrival processes, one for each discrete condition as in Equations (1) and (2). In particular, we define a PoissonProcess for each condition (or age bracket), which is then provided to a PoissonSystem. Using the PoissonSystem, we can sample the arriving objects for each  $t$  in
In 25. Note that we also model alpha, returning the overall arrival rate, such that the system can calculate an appropriate  $\nu$ . Figure 5a shows that AllSim accurately models the arriving objects.
With the arrival processes coded above together with a counterfactual Inference object, we compose a Simulation object—the main interaction interface. In particular, one defines a set of arrival rates (such as in Figure 5b) for both users and resources (cfr. ln 13-17) to create a simulation:
simulation = asim.init(resource_process, patient_process, inference, columns)
With that simulation, a practitioner can instantiate a Policy, which implements the add and select methods. For example, we have implemented the MELD policy [26], which is a widely known and used ScRAP for liver allocation. Using the simulation, we can generate a simulated dataset:
df  $=$  simulation.evaluate(policy=meld_policy，T=T)
Where df is a Pandas DataFrame [27, 28]. Naturally, df contains an enormous amount of information w.r.t. the policy's allocations in our environment. As such, we have included only a subset of the potential results in Figure 4. Additional results and details can be found in Appendices C and H. Ultimately, the practitioner determines appropriate analysis, settings, and performance metrics.
3.3 Beyond Organs
AllSim is a general purpose simulator which evaluates scarce resource allocation policies. While we have mainly focused on organ-transplantation so far, AllSim is also applicable in other settings. To illustrate, we show how one can implement a vaccine distribution policy evaluation system in AllSim. This use-case illustrates the few adjustments one has to make compared to the organ-allocation problem. Specifically, in vaccine distribution, each resource is the same and arrives in batches. Furthermore, the type of patient-in-need is also much broader (in fact, they cover the entire population). Yet, AllSim is perfectly capable of modelling this scenario given the following:
- Batch arrival requires a multiplier: if the Poisson process samples a value of 2 on one day, we could simply interpret this as two batches of 1000 doses, i.e. multiply by batch content.
- We no longer require a resource density as vaccines are not unique, contrasting organ allocation. This is implemented as a dummy-density that always returns 1 (or the vaccine amount).
- The broader patient-type is achieved by retraining the recipient-density on the entire population.
These implementation details are relatively simple to implement using AllSim's modular API. While not necessarily a problem in vaccine distribution, recipient arrival in the ICU in a setting of infectious disease (such as COVID-19), is definitely different compared to the organ-allocation setting. With organ-allocation, we can safely assume a Poisson process for recipient arrival as recipients enter the system independently. This is not true for infectious diseases: one recipient arriving may indicate higher infection rates. As such, recipients do not arrive independently, motivating AllSim's design.
It is clear that above scenario can no longer rely on a Poisson arrival process for new recipients entering the system. Instead, accurately modelling a situation of infectious disease could be done using a Hawkes process. To illustrate, we include some code below showing exactly how one may go about including such a Hawkes process in AllSim (replacing the Poisson processes used earlier).
class HawkesProcess(PoissonProcess): def __init__(self, lam: float=.1, update_lam: Callable[[int], float]=lambda t: t, delta: float=.1, a: float=.2): assert a >= 0, "a should be larger than or equal to 0" assert delta > 0, "delta should be larger than 0" super().__init__(lam, update_lam) self.a, self delta, self._samples = a, delta, []
def get_lam_unnormalized(self, t: int) -> float:
    return self._baseline_lam + np.sum(
        self.a * self.beta * np.exp(-beta * (t - self._samples[ 
            self._samples < t])
        )
    def progress(self, t: int, neu: float=1) -> int:
        self.lam = neu * self.get_lam_unnormalized(t) # eqs. (5, 6)
        sample = np.random.poisson(lamself.lam)
        self._samples.append(sample)
        return sample
Allocation policies from machine learning and OR. It seems that both the ML [1-3, 29-34] and OR [35-41] community is focused more and more on this important class of problems-- which is fantastic! But it also warrants careful evaluation. Furthermore, if we find that the evaluation strategies in medicine (which generally propose linear combinations of features [26, 42] or simple CoxPH models [43-46]) have shortcomings, then this is certainly the case for much more complicated strategies introduced in ML or OR. In fact, a recent survey confirmed exactly this concern: [47, cfr. Limitations of ML in transplant medicine]. It is in these extended scenarios where AllSim could help.
Naturally, problems solved by the OR community concern a variant of the general problem presented in this paper. For example, Balseiro et al. [40] are concerned with distributing a fixed set of resources, to a varying set of incoming users. While different, such problems can still be modelled in AllSim. In the specific case of Balseiro et al. [40], resources are not unique (they represent an amount) and require much less machinery than what we require to model the varying resource scenario. Specifically, one can model the remaining amount of resources as an attribute in our Policy class.
AllSim provides the means to perform standardised evaluation of repeated resource allocation policies in non-steady-state environments. While our experiments focus on organ-transplantation for the sake of exhibition, Appendix G illustrates AllSim for COVID-19 vaccine distribution, an example outside organ-transplantation. We believe that AllSim's generality and modularity allows for sensible adoption in a wide range of application areas. Furthermore, having standardised evaluation will encourage research in this very important and impactful domain spanning many application areas.
Conducting research in repeated resource allocation requires consideration of a policy's societal impact. While we believe AllSim will aid (rather than negatively impact) in this respect (by offering more than simple aggregate statistics), in Appendix F we provide a section dedicated to this topic.
Ethical research. We envisage AllSim as a tool to help accurate and standardised evaluation of repeated resource allocation policies, however emphasise that any finding would need to be further verified by a human expert or in some cases by a clinical trial. Ultimately, the decision on whether or not to trust a decision making tool is up to the acting decision-maker and ethics board. We hope that AllSim can help in any way to facilitate that decision, but stress that suggestions or evaluation always require critical assessment, as is the case for any research. We also refer the reader to Appendix F for a more thorough discussion on the potential societal impact of systems such as AllSim.
Reproducibility. To encourage reproducibility, we have included all our code to reproduce the presented results (as well as those in Appendix C). It should be clear from this paper, that reproducibility is actually one of the main reasons for doing this type of research in the first place. Furthermore, we have included a detailed discussion on how to use our simulation in Appendices D and E.
We would like to thank our many collaborating clinicians, and in particular, Alexander Gimson, for many interesting discussions leading to this work. JB is funded by the W.D. Armstrong Trust, DJ is funded by Alzheimer's Research UK (ARUK), and AC is funded by Microsoft Research.
14   
15   
16   
17   
18   
19   
20   
21   
22
[78] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[79] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[80] Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10):4156-4165, 2019.
[81] Ioana Bica, James Jordon, and Mihaela van der Schaar. Estimating the effects of continuous-valued interventions using generative adversarial networks. Advances in Neural Information Processing Systems, 33:16434-16445, 2020.
[82] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-319, 2021.
[83] Edward H Kennedy. Optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint arXiv:2004.14497, 2020.
[84] Scott Powers, Junyang Qian, Kenneth Jung, Alejandro Schuler, Nigam H Shah, Trevor Hastie, and Robert Tibshirani. Some methods for heterogeneous treatment effect estimation in high dimensions. Statistics in medicine, 37(11):1767-1787, 2018.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments (e.g. for benchmarks)...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] These are non compute-intensive experiments
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] When using the code
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] All used data is public
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]