Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a deep-learning approach called Re3Writer to imitate the clinicians’ working patterns for automatically generating patient instructions at the point of discharge from the hospital.
strengths_and_weaknesses: Strengths:

1. Topic is timely
2. The application is interesting
3. The paper is easy to follow

Weaknesses:
1. Lack of technical details (e.g., hyper parameter tuning, information about node transitions/weights)
2. Lack of output validation
3. Novelty is limited

questions: The application was tested using one generated dataset. How did the authors analyse the accuracy of the generated PIs? 
I would suggest performing the comparison of the output of the proposed deep-learning approach with more datasets/examples. Otherwise, it is hard to assess the novelty of the work.

Further technical explanations are needed (e.g., weights between nodes). For example, In Figure 4, under the reasoned medical knowledge box. What does 0.013 mean? 

How the optimum model was selected? Further information regarding hyper-parameter tuning can be useful.

Building a PI dataset is defined as a novelty (contribution). How it is built and tested? Did the authors build the deep-learning approach on this dataset? If so, how can authors make sure its applicability to real-world health data (e.g., clinical validation)? 
limitations: The proposed approach is interesting. However, further technical details, validation and also the comparison of the deep-learning application on more datasets are needed to assess its novelty.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors have proposed a new task of generating patient instructions (PIs) using the MIMIC dataset. The authors also proposed a strong SOTA model for the task Re3Writer which takes into consideration the historic PIs, clinical domain knowledge and refining of the final generated PI for a patient. The authors show that their method can improve the performance of a lot of baseline models by following the three techniques defined in the ReWriter framework.
strengths_and_weaknesses: Strength of the paper: 
1. It tackles an important problem and given it is created with the help of an openly available dataset (MIMIC), other researchers can also access the dataset and build further models.
2. The ReWriter approach is very intuitive as it considers the past PIs which generating the PI for a new patient and then considers the medical knowledge for refining it. This medical knowledge further helps the model in generating clinically coherent PIs.
3. All experiments are well defined but I really like the Table 5 ablation study because it justifies the use of each step in ReWriter. 

Weakness/Questions:
These are some things on which I would like to hear from the authors.
1. During train and test split, did the authors make sure that the same patient is not used in train and test. Even if the HADM (hospital admission ID) is different, a same patient can have similar PIs such as older patients can have same recommendations during multiple visits.
2. Did you consider UMLS for knowledge injection? Because the information of a new patient with a new ICD code that has not been used before in the train set would get ignored. Whereas UMLS can help with that information, if you can run an entityLinker to get different clinical entities and then create your final graph. 
3. The patients were matches via their procedure, diagnostic codes but their other demographic/personal information such as age and gender wasn't considered. A patient with same ICD codes but with different age-group could potentially be assigned different PIs because of their health.
4. Note: I might be wrong here but why not consider a pre-trained model? Is that because of constraint of the note length? 
questions: Weakness/Questions:
These are some things on which I would like to hear from the authors.
1. During train and test split, did the authors make sure that the same patient is not used in train and test. Even if the HADM (hospital admission ID) is different, a same patient can have similar PIs such as older patients can have same recommendations during multiple visits.
2. Did you consider UMLS for knowledge injection? Because the information of a new patient with a new ICD code that has not been used before in the train set would get ignored. Whereas UMLS can help with that information, if you can run an entityLinker to get different clinical entities and then create your final graph. 
3. The patients were matches via their procedure, diagnostic codes but their other demographic/personal information such as age and gender wasn't considered. A patient with same ICD codes but with different age-group could potentially be assigned different PIs because of their health.
4. Note: I might be wrong here but why not consider a pre-trained model? Is that because of constraint of the note length? 
limitations: Currently in this draft, no serious limitations are discussed. It is great to see Fig.4 where the model performs exceptionally well than the baseline model, it would be great to see some examples where ReWriter is consistently wrong.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposed the way (Re$^{3}$Writer) to automatically generate the Patient Instruction (PI) to reduce the workload of clinicians. Re$^{3}$Writer consists of three components: Retrieve (to retrieve instructions of previous patients according to the similarity of diagnosis, medication and procedure), Reason (to reason medical knowledge into the current input patient data with the help of the Knowledge Graph), and Refine (to refine relevant information from the retrieved working experience and reasoned medical knowledge to generate final PIs). The authors built a dataset PI which is based on the MIMIC 3 data and showed experimentally that with the help of their method performance of 6 different SOTA models can be substantially boosted across all metrics (BLEU-4, ROUGE-L, METEOR). They also showed the results from human evaluations to evaluate the approach for the clinical practice routine. 
strengths_and_weaknesses: Strength:

1. Overall the paper is well organised, easy and enjoable to read. The contributions and motivation are clearly put. In my opinion this paper is relevant to the people from Machine Learning for Healthcare domain. The problem of generating the clear PI is very relevant for the clinicians. The attempt of medical reports (not instructions) generation were done before, but most of them used other modalities (i.e images). Here authors provided the way to utilise only the text knowledge.
2. Previous approaches are clearly explained and cited; experimental settings are well explained.
3. With the help of Re$^{3}$Writer the performance was significantly increased (Table 2). Ablation studies clearly shows the impact of each component in the final result (Table 5).

Weaknesses:
1. More details about the PI dataset will be helpful: i.e did authors use discharge summaries? Also about the patients notes - they are sparse in time of the patient stay in hospital (i.e nursing and physicians notes, radiology reports), did author just concatenate all these notes, or did they perform the selection somehow?
2. Additional ablation studies about what type of notes (nursing notes, physicians notes, radiology) contributes more in the Re$^{3}$Writer performance. However, it is a minor comment.
3. Some relevant citations are missing, for example "Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation", Fenglin Liu, Chenyu You, Xian Wu, Shen Ge, Sheng wang, Xu Sun, NeurIPS'21. This paper (and not only this) also utilises the Knowledge Graph structure in order to generate the medical reports. Not the same idea, but it will be helpful for reader to see the full overview.

questions: 1. Will authors publish the PI dataset? Or it will be enough to use the provided code and instructions to generate it?
2. Could authors provide more details about the dataset generation?
3. Does eICU[*] dataset have the text notes? If yes, will it be possible to include the results on this data as well?

*Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eICU collaborative research database, a freely available multi-center database for critical care research. Scientific data, 5(1):1–13, 2018.
limitations: Limitations were discussed.
review_title: null

Meta-review: metareview: The authors propose and evaluate a method to automatically generate "patient instruction" drafts. There was a consensus amongst reviewers that this is an interesting application. 

While the technical innovation here may be modest, the empirical results firmly establish the benefits of the proposed "Re3Writer" approach. The ablations provided (both in the original submission and during author response) further strengthen the contribution. 

Furthermore, the task definition and accompanying dataset (derived from MIMIC) constitute contributions which may lead to follow-up work. That said, in revised versions of the paper the authors should include additional details about the data and be explicit that they will release this (as mentioned by reviewer oy13). 
confidence: Certain
Decision: Accept