SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jeroen Berrevoets
Daniel Jarrett
Alex J. Chan
Numerous real-world systems, ranging from healthcare to energy grids, involve users competing for finite and potentially scarce resources. Designing policies for repeated resource allocation in such real-world systems is challenging for many reasons, including the changing nature of user types and their (possibly urgent) need for resources. Researchers have developed numerous machine learning solutions for determining repeated resource allocation policies in these challenging settings. However, a key limitation has been the absence of good methods and test-beds for benchmarking these policies; almost all resource allocation policies are benchmarked in environments which are either completely synthetic or do not allow any deviation from historical data. In this paper we introduce AllSim, which is a benchmarking environment for realistically simulating the impact and utility of policies for resource allocation in systems in which users compete for such scarce resources. Building such a benchmarking environment is challenging because it needs to successfully take into account the entire collective of potential users and the impact a resource allocation policy has on all the other users in the system. AllSim's benchmarking environment is modular (each component being parameterized individually), learnable (informed by historical data), and customizable (adaptable to changing conditions). These, when interacting with an allocation policy, produce a dataset of simulated outcomes for evaluation and comparison of such policies. We believe AllSim is an essential step towards a more systematic evaluation of policies for scarce resource allocation compared to current approaches for benchmarking such methods.
The problem of repeated resource allocation to users with timeliness constraints is ubiquitous in settings ranging from healthcare to engineering systems and even labour markets. This problem becomes even more challenging when the resources are diverse and users may derive different benefits from obtaining a specific resource. In these applications, a resource coordinator or decision maker, is tasked with allocating these diverse resources to a pool of diverse users which arrive or leave over time.
When a new resource arrives, it is the decision maker's task to assign this coveted resource to one of the users in their current pool. Making such a decision has an enormous impact on the complete system: (i) when a resource is allocated to a user, other users now need to wait for the next resource to arrive, thereby impacting their utility (since they are delay-sensitive), (ii) the match between the resource and the user recipient has different valuations, (iii) assigning a resource to a specific user in the pool influences the utilities of all the other users in the pool as well, and thereby impacting any subsequent
Figure 1: AllSim overview. We illustrate how AllSim takes as input a dataset  $(\mathcal{D})$  which comprises a set of users  $(\bullet)$  and resources  $(\bullet)$  and outcomes  $(Y)$ . Each of these objects is confounded by an allocation policy  $\pi$ . From this highly complex dataset, AllSim learns a set of separate components (in green): a distribution for users  $(p(\bullet))$  and resources  $(p(\bullet))$ , an associated arrival time based on  $t$ $(\lambda_{\bullet}$  and  $\lambda_{\bullet})$ , and a utility  $(u)$ . AllSim then exposes an interface where a decision maker can perturb and influence each component separately, included the allocation policy itself. We highlight these perturbations in blue, resulting in a new dataset  $(\mathcal{D}^{\prime})$  used to measure the effect of each perturbation.
allocations. We note that the above problem scenario is incredibly general. To have an idea of the diversity of situations described as such, we refer to Table 1 where we list a few example situations.
Resource allocation. We identify three main challenges one has to overcome when solving problems described by the above. (i) Resources and users are described by multiple (possibly continuous) variables resulting in them being diverse and having complex interactions. (ii) The above are dynamic non-steady-state scenarios, which means that at any time the arrival of resources and users may change, the user-specific as well as system-wide utility may change, and even the users and resources themselves may change. (iii) These are multi-user problems, which means that each decision needs to take into account the resource recipient alongside every other user in the system.
Evaluation. Given the complex interactions between diverse resources and users, more and more we have to rely on machine learning based allocation policies which model these interactions to optimise a (system-wide) utility [1-4] (cfr. Table 1). While these novel policies receive a lot of research attention, the way in which they are evaluated seems to receive much less while being equally important. In fact, literature introducing these new methods fail to evaluate them against the challenges listed above. We believe the reason is the lack of proper evaluation tools; to our knowledge, there only exist tools that: Have no diverse resources/users [5, 6], Remain steady-state [7], or Model single-user systems [8]. None of them capture the challenges described above.
Synthetic versus real data. Another major consideration is the usage of data when evaluating policies. Whenever a real-world dataset is available—comprising users, resources, assignments and outcomes—we have to consider the fact that this dataset is tainted by an observational (in-place) policy. We have illustrated this as  $\mathbf{\Phi}_t - \overline{\mathbf{\Phi}}\rightarrow \mathbf{\Phi}_t$  in Figure 1, where the policy is denoted as  $\pi$ . The moment we want to test a policy which is different from the observational policy (e.g.  $\pi^{\prime}$  in Figure 1), we deviate from the original dataset as the resource to user assignments are (by definition) different.
A solution could be a completely synthetic simulation to evaluate policies. However, given the detailed and diverse descriptions of resources and users, this would introduce too much bias into our evaluation as every detail needs to be manually specified [9]. The latter is of particular importance when testing these novel ML-based policies, as this is exactly what they were built for in the first place.
Our solution is AllSim. Illustrated in Figure 1, AllSim learns separate (unbiased) components from historical data which was biased by a previous observational policy,  $\pi$ . These components are exposed as an interface which a decision maker can use to modify the system to fit their purpose. An obvious example of such a modification is to replace the past policy with a different policy. Other examples could be changing the arrival rates of users, resources, changing user and resource types, or even resource efficacy by changing the outcomes  $(Y)$  while still maintaining detail and realism. These perturbations are illustrated in blue in Figure 1. From AllSim, we sample a new dataset to measure the effect of the practitioner's modifications on utilities such as fairness, survival, waste, etc.
Table 1: Example situations. We list a few example situations that follow the general problem formalism introduced in this paper and the repeated allocation policy used to solve them.
Problem setting	Users	Resources	Allocation policy	Utility
Headhunting	Openings	Applicants	Assignment	Hires (& retention)
Project staffing	Projects	Workers	Staffing	Project success
Organ transplantation	Patients	Donors	Matching	Post/Pre TX survival
Mechanical ventilation	Patients	Ventilators	Triaging	ICU Discharge
Bicycle sharing	Docks	Bicycles	Redistribution	Idle times
Why are such simulators important for ML research? The successes in other subareas in machine learning are driven largely by the existence of capable and qualitative simulators [10-13]. With an easy-to-use interface and easily customisable environments, simulators allow researchers to focus on model development rather than creating their own (often conflicting) evaluation protocols. With AllSim, we hope to drive innovation for resource allocation in multi-user problems in healthcare, engineering, economics, etc. The aforementioned simulators are great examples of systematic evaluation across entire research communities, however, they do not: learn realistic and unbiased simulation objects from data, allow for multi-user simulation, or model dynamic non-steady-state scenarios.
Desiderata. From Figure 1 we identify three important desiderata: (1) A simulation should extract unbiased components from historical data which was tainted by existing policies; (2) The simulation should infer unbiased outcomes despite having access to only these biased data, which includes long-term impact on system-wide utilities since present allocations influence future allocations, requiring counterfactual inference (to determine outcomes under different allocations). (3) Using the extracted components from (1), a user must be able to perturb and change the components to fit their specific needs to evaluate different policies and settings before being deployed in the real world.
Contributions In this work, we present AllSim (Allocation Simulator), a general-purpose open-source framework for performing data-driven simulation of scarce resource allocation policies for pre-deployment evaluation. We use modular environment mechanisms to capture a range of environment conditions (e.g. varying arrival rates, sudden shocks, etc.), and provide for componentwise parameters to be learned from historical data, as well as allowing users to further configure parameters for stress testing and sensitivity analysis. Potential outcomes are evaluated using unbiased causal effects methods: Upon interaction with a policy, AllSim outputs a batch dataset detailing all of the simulated outcomes, allowing users to draw their own conclusions over the effectiveness of a policy. Compared to existing work, we believe this simulation framework takes a step towards more methodical evaluation of scarce resource allocation policies.
In Appendix B we compare against other strategies used to evaluate allocation policies. AllSim's itself is built using ideas from various fields in machine learning which we also review in Appendix B. Furthermore, in Appendix B.1 we review some medical simulations which seem related, but are not.
2 AllSim
Let  $X \in \mathbb{R}^d$  denote the feature vector of a user, and let  $\mathcal{X}(t)$  denote the arrival process of users. At each time  $t$ , let  $\mathbf{X}(t) \coloneqq \{X_i\}_{i=0}^{N(t)} \sim \mathcal{X}(t)$  give the arrival set of (time-varying) size  $N(t)$ . Likewise, let  $R \in \mathbb{R}^e$  be the feature vector of a resource, and let  $\mathcal{R}(t)$  be the arrival process of resources. At each time  $t$ , let  $\mathbf{R}(t) \coloneqq \{R_j\}_{j=0}^{M(t)} \sim \mathcal{R}(t)$  give the arrival set of (time-varying) size  $M(t)$ .
While we make no assumptions on how users are modelled, we assume that resources are immediately perishable—that is, each incoming resource cannot be kept idle, and must be consumed by some user in the same time step. In organ transplantation, for instance, the time between harvesting an organ and transplanting it ("cold ischemia time") must be minimized to prevent degradation [14-16].
Let  $Y_{+} \in \mathbb{R}$  be the outcome of a matched user, drawn from the distribution  $\mathcal{Y}(X,R)$  induced by assigning a resource  $R$  to a user  $X$ . At each time  $t$ , let  $\mathbf{Y}_{+}(t) \coloneqq \{Y_{+} \sim \mathcal{Y}(X_{R},R) : R \in \mathbf{R}(t)\}$  give the set of outcomes that result from matching each incoming  $R \in \mathbf{R}(t)$  with its assigned  $X_{R}$ . Likewise, let  $Y_{-} \in \mathbb{R}$  be the outcome of an un-matched user, drawn from the distribution  $\mathcal{Y}(X,\varnothing)$ . At each time  $t$ , let the set of outcomes for users who are never assigned a resource be given by
$\mathbf{Y}_{-}(t) \coloneqq \{Y_{-} \sim \mathcal{Y}(X, \emptyset): X \in \mathbf{X}(t), \neg (\exists t' \geq t)(R \in \mathbf{R}(t'), X = X_{R})\}$ . (Note that we focus on discrete-time settings (e.g. hours or days), and leave continuous time for future work). Then we have:
Definition 1 (Repeated Resource Allocation) Denote an environment with the tuple  $\mathcal{E} := (\mathcal{X}, \mathcal{R}, \mathcal{Y})$ . The repeated resource allocation problem is to decide which users to assign each incoming resource to—that is, to come up with a repeated allocation policy  $\pi: \mathbb{R}^e \times \mathcal{P}(\mathbb{R}^d) \to \mathbb{R}^d$ , perhaps to optimize some utility defined on the basis of (un-)matched outcomes. For instance, if  $Y$  is a patient's post-transplantation survival time, we might wish to maximize the average survival time.
With the necessary notation, and a formal definition of a policy's input and output in Definition 1, we are equipped to introduce each component of AllSim as illustrated in Figure 1. In Sect. 2.4 we also discuss how AllSim's output can be used to evaluate a new policy (or any other modification from the decision maker). Details regarding the simulation life-cycle can be found in Appendix A.
2.1 Arrival of users and resources
There are two necessary ingredients that comprise the arrival of new users, and new resources: the amount  $(N(t)$  and  $M(t)$ , respectively), and the description  $(X_{i}$  and  $R_{j}$ , respectively). Each is modelled differently. Before we sample the user and resource description, we first sample the amount of each arriving at time  $t$  from an associated arrival process—i.e., in this subsection we will focus on  $N(t)$  and  $M(t)$ . We first introduce the structure of the arrival processes, and explain how their parameters can be learned from data and modified by a decision maker to setup the environment.
First, we stress that  $N(t)$  and  $M(t)$  are not necessarily sampled from constant arrival processes. Instead, we want the user and resource arrivals to change over time either completely or per user/resource type, which we will explain in more detail below. To accommodate this, we split each arrival process into a product of separate arrival processes which we combine into  $\mathcal{X}(t)$  and  $\mathcal{R}(t)$  as:
$$
\hat {\mathcal {X}} (t, \theta_ {x}) = \hat {\mathcal {X}} _ {1} (t, \theta_ {1, x}) \times \dots \times \hat {\mathcal {X}} _ {K} (t, \theta_ {K, x}), \tag {1}
$$
$$
\mathcal {\hat {R}} (t, \theta_ {r}) = \mathcal {\hat {R}} _ {1} (t, \theta_ {1, r}) \times \dots \times \mathcal {\hat {R}} _ {L} (t, \theta_ {L, r}), \tag {2}
$$
where each individual arrival process in  $\hat{\mathcal{X}}_k$  and  $\hat{\mathcal{R}}_l$  is parameterised with (learnable) parameters  $\theta_{k,x}$  and  $\theta_{l,r}$  with  $k\in [K]$  and  $l\in [L]$ , respectively. Each factor corresponds with some (learned or predefined) user-type (Equation (1)) and resource-type (Equation (2)). Having these factors allows us to model increasing numbers of, for example, older/younger patients entering a transplant wait-list.
In order for  $\mathcal{X}(t)$  and  $\mathcal{R}(t)$  to change over time, we let their parameterisation,  $\theta_{x}$  and  $\theta_{r}$ , change in  $t$ . As an example, we can set the arrival processes to Poisson processes (we refer to Appendix E for other examples) with arrival rates  $\theta_{x} = \lambda_{k}(t)$  and  $\theta_{r} = \lambda_{l}(t)$ , which we can both model over time as,
$$
\lambda_ {k} (t) = \nu_ {k} \lambda_ {k} (0) g _ {k} (t), \tag {3}
$$
$$
\lambda_ {l} (t) = \nu_ {l} \lambda_ {l} (0) g _ {l} (t), \tag {4}
$$
with  $\lambda_{k}(t),\lambda_{l}(t)\in \mathbb{R}_{+}$ , and  $\nu_{k},\nu_{l}\in \mathbb{R}_{+}$  as a normalising constant such that the sum of all  $\lambda_{k}$  equal some overall arrival rate  $a_{x}$ , and similarly, the sum of all  $\lambda_{l}$  equal some overall arrival rate  $a_{r}$ . Lastly,  $g_{k}$  and  $g_{l}$  are continuous functions that simulate a user-specified drift. Note that these  $g$  can also be a combination of multiple drift scenarios, or can be shared across different  $k,l$ . Having  $g$ , allows practitioners to very accurately describe the non-stationarity they wish to test for. Optionally,  $\nu_{k}$  and  $\nu_{l}$  can be kept fixed throughout the simulation such that  $a_{x}$  and  $a_{r}$  vary as does  $g_{k,l}(t)$ , or it can be recomputed for every step  $t$ , such that  $a_{x}$  and  $a_{r}$  are kept fixed throughout the simulation.
As such, we have a set of arrival rates,  $\Lambda_{x} = [\lambda_{1},\dots,\lambda_{K}]$ , with  $\sum_{k}\lambda_{k} = \alpha_{x}$  with  $\alpha_{x}\in \mathbb{R}_{+}$  as the total arrival rate of recipients. The advantage of splitting  $\alpha_{x}$  into multiple  $\lambda_{k}$ , is that we can finetune the arrival of certain recipient types, yet allow comparison between  $\alpha_{x}$  and  $\alpha_{r}$  (the total arrival rate for resources). For example, the  $k^{\mathrm{th}}$  recipient type may be completely absent when a policy is launched, but over time it gradually enters the system, increasing  $\alpha_{x}$  as a whole. Naturally, we also model the arrival of resources as we have for recipients, but left it out of discussion for clarity.
Learning  $\theta_{x,r}$  naturally depends on the choice of arrival process. In our setups below we use a Poisson process and have either: (i) learned the dynamic parameters  $\lambda_{k,l}(t)$  as in Equations (3) and (4) using polynomial regression over a time-windowed average of incoming users and resources—over all
data to compute a correct  $\nu_{k,l}$ , as well as per predefined condition; or (ii) have predefined an arrival function and drift functions,  $g$ , to illustrate a scenario where one wishes to test a prespecified scenario.
2.2 New users and resources
$p(\bullet)$  and  $p(\text{串})$
From  $\hat{\mathcal{X}}(t)$  and  $\hat{\mathcal{R}}(t)$  we sample  $N(t)$  and  $M(t)$ , respectively. Of course, we need to provide the tested policies with more than just an amount of users and resources arriving at time  $t$ . Furthermore, when working with user and resource types (using the decomposition in Equations (1) and (2)), we have  $N(t) = \sum_{k} N_k(t)$  and  $M(t) = \sum_{l} M_l(t)$ , where each  $N_k(t)$  and  $M_l(t)$  represents an amount of users and resources per type. As such, we need these types to sample detailed descriptions of each.
When a recipient or a resource arrives, we sample them from a distribution denoted  $p_{\theta_x}(X)$  for the recipients, and  $p_{\theta_r}(R)$  for the resources. These distributions are either learnt from data, or shared as an open-source (but privatised) distribution. For the user-distributions we learn from  $\bigcup_t \mathbf{X}_t$ , and similarly, for the resource-distributions we learn from  $\bigcup_t \mathbf{R}(t)$ . Since both remain independent from the past policy (no policy determines which users and resources arrive in the system), we can use any (conditional) generative model to learn these distributions as we are not required to de-bias these data.
Of course, we need to be able to sample specific user and resource types. For this we require conditional generative models, where the condition corresponds with a type:  $p_{\theta_x}(X)$  becomes  $p_{\theta_x}(X|k)$ , and similarly  $p_{\theta_r}(R)$  becomes  $p_{\theta_r}(R|l)$ . In case we wish to use an unconditional generative model, we can simply learn multiple:  $p_{\theta_{k,x}}(X)$  for each  $k \in [K]$ , and similarly for  $p_{\theta_{l,r}}(R)$  for each  $l \in [L]$ .
Interestingly, we do not need to account for any variability (learned nor specified) over time, since this is completely modelled through the arrival processes in Sect. 2.1. In particular, whenever we wish one type,  $k$  to dominate others, we simply increase  $g_{k}(t)$  in Equations (3) and (4). In case we only want one type to appear after  $t$  in the simulation, we set  $g_{k} = 0$  for  $t' < t$  and increase it for  $t'' > t$ .
2.3 Utility
$$
u (\pi , Y, \mathbf {b} _ {t}, \mathbf {b} _ {t})
$$
The final component in AllSim, as per Figure 1, are the utilities: functions of the policy  $(\pi)$ , the users and resources  $(X$  and  $R$ ), and crucially, the allocation outcomes  $(Y)$ . Given the previous sections, all that remains are the outcomes and how we can combine each element into a new dataset,  $\mathcal{D}'$ , with counterfactual outcomes,  $Y'$ .
As the outcome is a function of the resource and its recipient, inference is a hard problem as allocations suggested by the tested policy deviate from historical data which was collected under a different policy (i.e., they are counterfactual). Consequently, some combinations are less observed in the original data, illustrated in Figure 2. In Figure 2 we illustrate two policies,  $\pi$  and  $\pi^{\prime}$  which result in different datasets  $\mathcal{D}$  and  $\mathcal{D}'$ . The latter  $(\mathcal{D}')$  is what we wish to provide with AllSim, using only data from the former  $(\mathcal{D})$ .
Figure 2: Allocation policies bias data. Above illustration depicts two policies:  $\pi$  and  $\pi'$ . Each policy is tasked with assigning resources to users as per Definition 1. Besides users and resources, we observe an outcome,  $Y$ . Despite observing the same users and resources, a different policy results in completely different outcomes,  $Y'$ , and data,  $\mathcal{D}'$ .
Counterfactual inference. AllSim handles this difficult problem by using a counterfactual estimator. Counterfactual methods correct for allocation bias explicitly [2]. In particular, these methods aim to make an unbiased prediction of the potential outcome, associated with some treatment (or resource). We are interested in counterfactual methods that model the potential outcomes for the recipients when they are (not) allocated a resource. A counterfactual estimator then "completes" the dataset  $(\mathcal{D}^{\prime})$  as,
$$
\mathbf {Y} (t) = \mathbb {E} [ \hat {\mathbf {Y}} (\mathbf {R} (t)) | \mathbf {X} _ {\pi} (t) ], \tag {5}
$$
where  $\hat{\mathbf{Y}} (\mathbf{R}(t))$  is the estimated potential outcome, using methodology known in the potential outcomes literature [17-22], and  $\mathbf{X}_{\pi}(t)$  are the recipients selected by a policy  $\pi$  at time  $t$ . The potential outcome is a random variable depicting the (possibly alternative) outcome when the user receives the resource,  $R(t)$ . Note that this is not the same as simply conditioning the outcome
variables on the users, for the reasons outlined above: conditioning using only biased data will lead to biased estimates for the outcome variable. Hence, literature on counterfactual inference introduced the potential outcomes notation in Equation (5) to differentiate between  $Y(R(t))$  and  $Y|R(t)$ . We provide a comprehensive overview of counterfactual methods and literature in Appendix H.
2.4 Putting it all together
We have now discussed each component in the middle section of Figure 1. What remains are the decision maker's perturbations, and finally, combining each component into a new dataset,  $\mathcal{D}'$ .
Perturbations. From Figure 1 we learn that a decision maker can make three types of perturbations: (i) they can replace the original policy,  $\pi$ , with a new (alternative) policy,  $\pi'$ ; (ii) they can change the utility function,  $u$ , which takes as argument a dataset comprised of users, resources, outcomes, and a policy; and lastly, (iii) they can change the types, as well as the amount, of users and resources entering the system. Given these perturbations, the policy is allowed to act in a different environment.