Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This study presents “Medkit-Learn(ing) environment” which gives researchers the ability to generate a variety of synthetic datasets to advance sequential decision making (offline RL, IRL, reward design) in medical machine learning. The study describes the data generating process used to provide datasets that are customized with different environment and policy dynamics which is different from other similar simulators which entangle these two dynamics.Finally, some practical tasks and evaluations are demonstrated using the library.
strengths: This paper is very well-written and provides a new simulator that is much more flexible than many of the ad-hoc simulators that have been created for medical sequential decision making studies in the ML4H community. The data generating process is very clear and the demonstrations provide good insight into potential uses for the simulator. 
weaknesses: The use of differential privacy is mentioned in the construction of the synthetic datasets. This was barely discussed in the paper but seems to play an important part in ensuring that the data is protected. For example what epsilon, delta values were chosen and what is the effect of this on the final synthetic datasets that are generated? Additionally, it would be great to see more empirical comparisons of this simulator to some of the ad-hoc simulators. For example, how trustworthy are results on these synthetic datasets compared to dynamics based simulators used in the sequential decision making ML4H literature? 
correctness: All statements appear to be correct to me.

clarity: This paper was written quite clearly.
relation to prior work: This paper did a great job of comparing the differences of this simulator to other simulators at a high level in the table provided. It would be great to seem some empirical comparisons. 
additional feedback: - Spelling mistake of parameterization on Line 111 Page 3
review_title: Useful simulator for modelling sequential decision making in different medical environments.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present Medkit-Learn(ing) Environment, a modular python package for offline sequential decision making. This package enables the creation of simulated medical setting, with high customizability and controllability. The authors outline each of aspect of their contributions and provide some initial demonstrations. 
strengths: - Very well written manuscript, including the definition of a scenario, domain, environment and policy and the graphical model. 

- The authors have done an excellent job of framing their new package in the context of other comparable ones, and highlighting key different contributions.  The authors have made it clear why their work is significant.

- The authors have created highly customisable scenarios, with detailed descriptions on the different domains, policies (along with the different API aspects), environments (patient history, latent variable models). This increases the relevance to the broader community as well as well accessibility. 

- The authors have considered ethical and social implications, and have presented a clear focus for their work without overstating any claims.
weaknesses: - Brief practical demonstrations. I suggest the authors include demonstrations using the ICU data, and expand on the practicality of their package. Given that minimal comparable works exist for the medical context, I am concerned that the broader community may not be interested in this very interesting work or find it inaccessible otherwise. Demonstrating more practical examples would be one way to mitigate that.

- A lack of complete examples (in code) may limit user adoption and usage. For example, the authors outline a potential workflow in 2.2, but no similar example exists in their code repository as far as I can tell.

- The validation of realism is superficial and not compelling. 
correctness: The package presented here is logically sound and correct. The way the authors have presented this work is sound.

However, there appear to be missing definitions for functions f and g. The first mention of these functions appears on page 3, section 2.1, line 77 as far as I can tell. For completeness, please add in these definitions.
clarity: - Consistency on the capitlization of Markovianity (pg 4, line 144)
- While I appreciate the colour coding in 4.2, some colours are not easily distinguished for those of us that are colour blind. I suggest another form of formatting or different colours. 
- Figures 3 and 4 is relatively hard to read at normal zoom. 

Otherwise, the paper is very well written given the topic matter and the many concepts being introduced. 
relation to prior work: The authors have clearly positioned their package against others, including going beyond medical only applications to look at numerous other works. The authors have then identified key factors to consider for all of those and address the shortcomings in previous contributions. 
additional feedback: - There appear to be multiple contributions here. It's unclear to me which are the most significant and which are incremental in nature. Can the authors elaborate on the relative significance of being a) non-Markovian, b) having a distentangled environment and policy, and c) having a customizable simulated policy? (Relative to one another)

- What would the process be for outside contributors wanting to add new medical domains?
review_title: Interesting data generator for medical decision making with unclear practical use.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This work proposes a benchmark for clinical sequential decision making. The simulated environment of course consists of the underlying dynamics model and policy model, both of which can be independently modeled. The benchmark provides flexibility in terms of policy classes that can be used to sample synthetic data from two types of domains (ICU and Ward). The environment model is captured using various deep learning methods trained on real-world data that can then be used to sample synthetic data under custom policies. Benchmark is evaluated on imitation learning baselines and discriminatory performance comparison of synthetic and real data samples. Policy functions can be customized to modify clinician/behavior policy, markovianity of the policy, practice variability and other factors. Environment is modeled with different choices like LSTMs, recurrent counterfactual networks, latent variable models. 
strengths: 1. ML-based policies and interventions learned on clinical data are fairly challenging to evaluate on the data itself and real-world experimentation is often not possible/unethical. Such a benchmark (which is fairly general) can prove very valuable to the community interested in building models for clinical decision support for proof-of-concept demonstrations of their support models.

2. The methodology of designing the environment is clear and looks reasonable to me, particularly the non-markovian policies and auto-regression (markovian environments). I do suggest authors make that distinction in Table 1 unless I missed something (i.e. only policies can be non-markovian, environment dynamics cannot). 

3. The main utility of the benchmark seems to be in its ability to evaluate RL algorithms and account for realistic clinician behavior in policies (like practice variation etc). 


weaknesses: My main concern is the way environments are modeled. I understand the need for realistic data. Authors have chosen to model environments with methods like recurrent neural networks, "counterfactual" recurrent neural networks and latent variable models like attention-based state-space models and variational auto-encorders. 

Designing the environments in this way clearly biases the benchmark and ability to evaluate them to certain methods over others. I have multiple concerns about this that I have listed below:

i) At a conceptual level, it is probably better to describe variability of environments in terms of statistical variations rather than specific model types used. By that I mean it is more meaningful to users of such a benchmark to be able to claim that a particular policy/method works well on, say i) non-stationarity (and what kind of non-stationarity), ii) heteroskedasticity (and what kind), iii) multi-modality, iv) heterogeneity, instead of saying something general about a neural network based environment.

ii) Model-based methods in RL cannot be realistically and properly evaluated with such environments, all one has to do is create parametric models of the dynamics that match the environment property and they will mis-specify the model far less often compared to those that don't. If the benchmark is not designed for model-based evaluation, authors should clearly say so in the introduction. This can clearly bias which methods do better than others. I strongly suggest authors provide complex parametrized models that add noise over real data to create semi-synthetic data-generators for the environment instead of deep network based environments. It is not even clear whether such deep networks are good proxies for realistic clinicial environments. But statistical notions of choices of environments can certainly help understand limitations of the environment without biasing methods. It will make the benchmark more usable in my opinion.

iii) Relying on semi-synthetic data + specific parametrization of the *distributions* rather than deep learning models of the environment will also significantly avoid bias in the generate samples. If a certain population or disease type is more prevalent in the data, it clearly biases the environment in a manner that's not appealing for strong evaluation. 


correctness: i) Please see my concerns above on weaknesses. I believe this is a very strong limitation of the current benchmark and something to address to improve the utility of the benchmark.

ii) Why exactly will $q(y|.)$ in Equation for $Q_{\pi}$ - \begin{equation}
Q_{\pi}^{\Omega}\left(y_{t} \mid \vec{x}_{t}, \vec{y}_{t-1}\right)=\sum_{i} w_{i} \frac{e^{\beta_{i} q_{i}\left(y_{t} \mid g_{i}\left(\vec{x}_{t}\left\langle\mathcal{X}^{\prime}\right\rangle_{i}, \vec{y}_{t-1}\right)\right)}}{\sum_{y \in \mathcal{Y}} e^{\beta_{i} q_{i}\left(y \mid g_{i}\left(\vec{x}_{t}\left\langle\mathcal{X}^{\prime}\right\rangle_{i}, \vec{y}_{t-1}\right)\right)}}
\end{equation} 
model ground-truth structure that can be customized and verified? Can the authors elaborate a bit? I understand it helps generating data, but doesn't help me understand why it captures structure. 
clarity: The paper is fairly well written and motivated. Please just check for minor typos.
relation to prior work: Related work, existing benchmarks are well contextualized.
additional feedback: If authors can clarify the concerns I have raised above, particularly in weaknesses, in detail, I am happy to reconsider the score.
review_title: This work contributes a benchmark environment for clinical sequential decision making to capture real-world challenges of training models for clinical decision support. The problem is well motivated and the authors have come up with a reasonable solution to create such a challenging benchmark. My main concern is with the modeling choices made for modeling the environments, which I believe could be significantly improved.

Meta-review: The reviewers feel that the new “Medkit-Learn(ing) environment” is well motivated and the authors have come up with a reasonable solution. The discussion has converged to a positive evaluation by the reviewers R1 and R2, but R3 still has concerns about the modeling choices made for modeling the environments, which could be significantly improved. In view of that, the authors may need to expand them in the next version. 
Decision: Reject