Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: the authors present a simulation environment for evaluating medical decision modelling 
strengths: the paper presents and interesting framework and evaluation. I appreciated that the authors trained their medical environments on real data sources and used differential privacy to ensure patient protections. The dynamics of policy interactions with hospital environments is certainly understudied so I appreciated their under-taking 
weaknesses: - the central contribution seems to be an environment for modeling clinical decision making that explicitly disentangles environment and policy so that the parameters governing these two systems can be explicitly controlled. However, there is no analysis or experiment that validates whether medkit-learn actually works in this way. 

- the paper would heavily benefit from an example in which a specific outcome (say sepsis) and intervention is described, and the sensitivity of outcomes to policy interventions quantified and discussed.
correctness: "First, the very nature of healthcare data science is that any learning and testing must be carried out entirely offline, using batch medical datasets that are often limited in size, variety, and accessibility [25, 39]."

- This intro sentence makes a very confusing claim to me. Hospital data science teams routinely train and evaluate clinical decision support algorithms in real time, often through a silent deployment, to collect data about their performance or to calibrate them as data shifts. Perhaps you mean just decision or policy modeling. But if you mean healthcare science, you will find several examples in literature of real-time deployments. 

"Second, directly using methods for time-series synthetic data generation is inadequate, as they simply learn sequential generative models to replicate existing data, making no distinction between environment and policy dynamics."

- To what extent is the environment actually identifiable using the real data sources? Certainly you have shown that classification models trained on the synthetic data are predictive on real data, and that the are difficult to tell apart. But my understanding is that you are attempting to learn a generative model in which the environment and policy are disentangled from data in which they operate in closed loop. Put another way, aren't there a lot of policy scenarios that are not present in real data, that would therefore make the simulation less accurate outside of the operating regime in which the environment was trained? 


clarity: The paper is well written, although it could benefit from being written for a broader audience.
relation to prior work: yes
additional feedback: typos:

- parameteriaation

- In particular BC performs the worst (although there is little between them) in the ends up

- this is since
review_title: interesting simulation of clinical decision support

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present a Medkit-Learn(ing) Environment for benchmarking medical sequential decision making. Medkit offers users the flexibility to obtain synthetic batch datasets for any desired type of parameterization of the decision-making policy and thus evaluate a wide range of methods for modeling sequential decision making.
strengths: The need of the paper is very well motivated in the introductions section; sets up the tone of the paper.

A very good comparison to the related work. Special shoutout to Table-2, which highlights how the work is different from the ones existing in the literature.

The practical demonstrations section is strong. Especially like the way the realism of the synthetic data is validated.
weaknesses: The main issue with the paper is the generalizability of the synthetic dataset. The authors need to address the possibility of bias in synthetic data generation, as the ML models generating the synthetic data, have only been trained on data from a few hospitals.

The authors can make this work even stronger (or consider it as a part of future work) by checking if training on a combination of real+synthetic data actually helps to break some of the existing benchmarks in the field. This can help to get a sense if more data actually help or just provides a redundant signal to the ML algorithm.


correctness: Addressed in the above sections!
clarity: Addressed in the above sections!
relation to prior work: Addressed in the above sections!
additional feedback: One of the potential ways in which synthetic data generation can be helpful is for ensuring the privacy of the patients. Usually, the hospitals are not willing to share the original patient data (even de-identified). Hence, if it can be shown that the ground truth data distribution is comparable, generating synthetic data and sharing it in public can be really helpful.
review_title: Review of "The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulation"

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: They provide Medkit-Lear Environment, which is a data simulator for medical decision modeling.
strengths: - Medical sequential decision making is an important task that lacks a standard benchmark, their paper fills the gap.
- They provide an easy-to-use python package for users to obtain the data.
weaknesses: - The generated data provide the static features, observation features, and actions. However, it is hard to understand the meanings of the provided features (e.g., which number represents the patients' age). This does not influence the algorithm benchmarking but affects the doctors to use or understand the models trained on the data.
correctness: None.
clarity: Yes.
relation to prior work: Yes.
additional feedback: Minor problems:
- parameteriaation -> parameter?
review_title: An easy-to-use simulator for medical decision modeling

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors develop and publicize a new benchmarking suite designed specifically for medical sequential decision modeling, which is called the Medkit-Learn(ing) Environment. It is a publicly available Python package providing simple and easy access to high-fidelity synthetic medical data. It enables users to obtain batch datasets with known ground-truth policy parameterisations that simulate decision making behaviours with various degrees of Markovianity, bounded rationality, confounding, individual consistency and variation in practice.
strengths: The main strength of the paper is to provide a python toolkit that fulfills the following three desirable properties.

(1) a variety of realistic environment models—learned from actual data, to reflect real medical settings), thus allowing simulation of 
(2) a variety of expressive and customisable policy models that represent complex human decision-behaviours
(3) ensuring that the environment and policy components are disentangled

As described in Section 3 and Table 2, these above features well differentiate the proposed package with some relevant ones. The third property is extensively explained in Section 4 with some formulations, which is nice to read. There are also some demonstrations and use cases, which helps readers understand the high-level usage of the package. The used datasets and experimental details are also well documented in the supplementary materials.

weaknesses: Although the paper has sufficient description on the dataset and environment, the package does not have sufficient documentation and example codes. A well documented API and some easy-to-run example tutorials will greatly enhance the usability of the package. Moreover, the repository does not include test codes. Most of its functions also do not have the docstrings. It is very hard to understand the correct types and shapes of the inputs/outputs. The implemented functions also seem not raise errors (such as ValueError/TypeError), it is hard for users to understand why an error occurs when the package doesn’t work. So, the repository looks to be on the way of its development, and I can hardly say that the package is user-friendly. I would love to see some actions and additional efforts to enhance the pacakge during the author feedback period.

I could not find the code to replicate the results in Section 5. Why not publicize it as an example analysis in the GitHub repo? This will ensure the reproducibility of the demonstration and help users understand how to use the package. (I’m sorry if I miss something here)

correctness: The formulation and the design of the environment seem to be well-thought by the authors and sound. The design choices are thoroughly described.
clarity: The paper is well written and easy to follow. However, I found that the package documentation could be improved. Please see the “Documentation” section below.

typo
- p4. L158, “We describe here the the various..”
relation to prior work: Related works are well described in the paper. Table 2 helps readers differentiate this work with the previous ones.
additional feedback: For suggestions for improvement, please the “Weaknesses” and “Documentation” sections.
review_title: Review of the paper

Meta-review: There is uniform consensus among the reviewers for the acceptance of this work.
Decision: Accept