Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors present a simple simulation model which they argue can "Learn" from the environment. However, the authors seem unaware of teen area of formal methods and the tools in this area such as PEPA, CARMA OR SPN which this work appears to be a re-invention of.
strength_and_weaknesses: Strengths:
 - The English in the paper is clear and well structured
 - The ideas are clearly presented
Weaknesses:
 - The work presents ideas which have been covered before using formal methods using tools such as PEPA, CARMA AND SPN. Comparison to these tools would seem essential.
 - The results section is only a small number of simple examples and does not compare the work to other approaches.
 - There are many statements in the work which are incorrect. And many which need supporting evidence such as citation to other works.
clarity,_quality,_novelty_and_reproducibility: Although the quality of presentation early in the paper is good. The latter parts of the paper become much weaker. Presenting code fragments is not the right level for academic conferences.

The idea is not novel and has been done extensively in the literature.

The authors use their own notation in places with no explanation of what it is or how it works.
summary_of_the_review: The paper contains only a simulation approach which has been widely covered in prior work. There is nothing in this work which is novel in the way of ICLR. The authors need to reflect on tools such as PEPA, CARMA and SPN. A more analytical analysis of the solution is needed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a simulation environment for evaluating scarce resource (e.g., Kidney, Liver) allocation policy. The proposed allocation simulator (AllSim) is modular, learnable and customizable. The simulator provides functionality for generating resource and recipient distribution, allocating resources, and evaluating performance of the allocation. It allows practitioners to properly tune different knobs of the simulator to match their practical needs. The paper also demonstrates the performance of different components experimentally.  
strength_and_weaknesses: Scarce resource allocation (e.g., Kidney/Liver matching) is an extremely important and challenging problem that requires online planning. While there has been significant effort in AI and OR communities to find optimal online matching of demand and supplies of these scarce resources, we absolutely need an accurate simulation environment for evaluation of different decisions. The proposed simulator can be used for counterfactual evaluation of policies learned by offline RL methods from a batch dataset. The simulator provides end-to-end functionalities from generating demand and supply of resources, allocating resources and evaluating a policy by supporting counterfactual inference. Having said that, I have some reservations regarding the technical novelties of the paper. There have been efforts for generating simulation environments for kidney or liver matching or transplantation (e.g., SimUNet: https://unos.org/solutions/simunet/). Does the simulator generalize to different settings of organ matching with simple tunable knobs? Moreover, tuning these knobs could be challenging for the practitioners as the outcome might depend on a variety of indicators for individual patients. I think it would be great if the authors can perform a human evaluation study and report statistics in terms of how much value the actual practitioners see in this simulator. If it does not pass the bar of practitioner’s expectation, then the counterfactual evaluation in the simulator might not be of practical use. On the other hand, I was expecting some guidance on how to tune the knobs of the simulator, especially for accelerating research progress in this domain. 
clarity,_quality,_novelty_and_reproducibility: The paper is generally written well and easy to follow. The motivations, key components of the simulator and their usage are explained clearly. The authors shared the source code of the simulator, so it can be easily reproduced. Related works are exhaustive and covered relevant papers in different settings. My only concern remains with the technical novelties and practical deployment of the simulator in the real-world as mentioned earlier. 
summary_of_the_review: The paper proposed a novel simulator with end-to-end functionalities for evaluating a scarce resource allocation policy, especially in medical domain setting. An accurate simulator would be hugely effective for research progress in the medical domain as well as offline RL space. I appreciate the specific functionalities of resource allocation and counterfactual inference, and their detailed explanation in experimental results. Moreover, the simulator could be easily reproduced as the source codes are shared. Having said that, I still have some reservations regarding the technical/methodological contributions, e.g., how well it generalizes to different organ settings and how much finetuning is required are not clear to me, and its practical adoption, e.g., I expected an human evaluation on how satisfied the practitioners are with the simulator, or how much effort the practitioners need to finetune all the knobs, or even if it is feasible to properly tune all the knobs to generate accurate behavior or outcomes of a patient health?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a general module for data-driven simulations for resource allocation. They outline and demonstrate this codebase using an organ allocation scenario.

strength_and_weaknesses: 
Strengths:
- this paper & methods are very clear, and their discussion is very thorough
- the simulation module seems very sound, and they provide code
- the experiments are compelling, though only use 1-2 examples. 

Weaknesses:
- it's difficult to assess novelty here. this paper synthesizes and combines many important topics related to resource allocation. but it's not clear whether this contributes any new ideas the literature.


clarity,_quality,_novelty_and_reproducibility: 
Clarity
- very clear

Quality
- the simulations and methods seem very sound

Novelty
- not really clear..

Reproducibility
- yes

summary_of_the_review: This paper is interesting and relevant and well written. The simulation module does seem very useful, and the experiments are compelling. However I'm not sure how to assess novelty here. While the module appears to be very useful, it's not clear that it contributes anything new to the literature. (Aside -- ICLR seems like a strange venue for this paper.) Nevertheless, this seems to me like a strong contribution. 

If the authors can clarify the novelty of this paper (perhaps using additional simulations, discussion, applications, etc.), I would advocate strongly to accept this paper.


--- 

Updated this review after reading comments and other reviews. See my comment from Nov. 18.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an open-source framework to be used to allocate scarce resources to a pool of recipient.  The application example presented in the paper is allocation of organs to recipients, an application area where optimizing the allocation is extremely important.  The framework is meant to be used to explore dynamic policies for the allocation. The system used discrete time settings for the simulation, modeling arrival times for resources and for recipients and generating outcome predictions.  The framework, which learns from past data, is designed to be modular and customizable. An important aspect is the ability to do counterfactual reasoning and provide for each pair resource-recipient the potential outcome. 
strength_and_weaknesses: Strengths: the problem addressed is an important problem, not just for the application studies but for other potential areas of applications.  The framework proposed has multiple properties, like modularity and customizability. Modularity allows to connect it with external applications, custinmizability enables the users to decide what they need. Counterfactual inference is used to make predictions. The software is open source.
Weaknesses: there is no indication that the proposed framework has been discussed with potential users, to guarantee it provides a valuable solution for the intended users. It is still up to the practioners to decide how to configure the system, what analysis to do, etc.  If the framework has to become usable by a variety of practioners, much more guidance would be needed for specific cases. This is not difficult and does not need to be done immediately.
clarity,_quality,_novelty_and_reproducibility: The paper is very well written, with enough details and examples.  I am not familiar with any competing system with the same properties.
summary_of_the_review: The paper proposes a framework for allocation of scarce resources to a pool of recipients.  The software is open source, making it easy to costumize it and extend it with additional modules.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper contributes a method for running simulations.  The specific problem addressed of allocating transplants is quite important, and the fact that the system has been designed with medical practitioners can suggest that the system will be adopted and that it satisfies the needs for which it has been designed.  This potential impact is a major strength of the paper.

While the proposed simulation module might be novel for a variety of reasons, the advantages of the proposed method over the status quo are not well demonstrated in the paper.  For example, one could take multiple resource-allocation settings, and
- simulate resource arrivals and distributions in a reasonable way, using available datasets. (make sure that these simulations align with observed behaviors)
- simulate resource allocation policies used in practice, and verify that their behavior/outcomes aligns with real-world observations
- simulate new resource allocation policies (e.g., reinforcement learning)
- use a "status quo" simulation method from the literature, and demonstrate why this is inferior to the proposed method

Also, the subject of the paper is not the core topic of ICLR.  While this does not prohibit publications in ICLR, other communities might evaluate the paper differently.
justification_for_why_not_higher_score: The novelty of the proposed simulation method is unclear, and the advantages of the proposed method over existing methods are not demonstrated.
Decision: Reject