Retrieve, Reason, and Refine: Generating Accurate and Faithful Patient Instructions
Authors: Fenglin Liu
Bang Yang
Chenyu You
Xian Wu
Shen Ge
Zhangdaihong Liu
Xu Sun
Yang Yang
David A. Clifton
Venue: NeurIPS 2022 Accept (2022)
Keywords: Natural Language Generation
Medical Text Generation
MIMIC-III
Patient Instruction
Discharge
AI for Healthcare

Abstract: The "Patient Instruction" (PI), which contains critical instructional information provided both to carers and to the patient at the time of discharge, is essential for the patient to manage their condition outside hospital. An accurate and easy-to-follow PI can improve the self-management of patients which can in turn reduce hospital readmission rates. However, writing an appropriate PI can be extremely time consuming for physicians, and is subject to being incomplete or error-prone for (potentially overworked) physicians. Therefore, we propose a new task that can provide an objective means of avoiding incompleteness, while reducing clinical workload: the automatic generation of the PI, which is imagined as being a document that the clinician can review, modify, and approve as necessary (rather than taking the human "out of the loop"). We build a benchmark clinical dataset and propose the Re$^3$Writer, which imitates the working patterns of physicians to first retrieve related working experience from historical PIs written by physicians, then reason related medical knowledge. Finally, it refines the retrieved working experience and reasoned medical knowledge to extract useful information, which is used to generate the PI for previously-unseen patient according to their health records during hospitalization. Our experiments show that, using our method, the performance of 6 different models can be substantially boosted across all metrics, with up to 20%, 11%, and 19% relative improvements in BLEU-4, ROUGE-L, and METEOR, respectively. Meanwhile, we show results from human evaluations to measure the effectiveness in terms of its usefulness for clinical practice. The code is available at https://github.com/AI-in-Health/Patient-Instructions.

Retrieve, Reason, and Refine:
Generating Accurate and Faithful Patient Instructions
Fenglin Liu'; Bang Yang"; Chenyu You',
Xian Wu*, Shen Get,
Zhangdaihong Liu
Xu Sun Yang Yang?! David Clifton
Depaarmenn of Engineering Science, University of Oxford School of ECE, Peking University 'Department of Electrical Engineering, Yale University *Tencent JARVIS Lab, China
'Oxford-Suzhou Centre for Advanced Research, China
MOO Key Lab of Computational Linguisties. School of Computer Science, Peking University "School of Public Health, Shanghai Jíao Tong University School of Medicine, China (fenglin.liu, 1iu, david. seiftton@eeng ox ax. uk, {bangyang, xusun) @pku.. edu. Chenyu.youyaale you@yale edu, jessie jessie. 1iuooxford
edu, on (kevinxwu, shenge)@tencee com, rmma002@sjtu.edde edu. on
Abstract
The "Patient Instruction" (PI), which contains critical instructional information provided both to carers and to the patient at the time of discharge, is essential for accurate and easy-to-
the patient to manage their condition outside hospital. follow PI can improve the melf-management of patients which can in turn reduce hospital readmission rates. However, writing an appropriate can extremely time-consuming for physicians, and is subject to being incomplete or error-provd prone for (potentially overworked) physicians. Therefore, we propose a new task that provide objective means of avoiding incompleteness, while reducing clin- ical workload: the automatic generation of the PI, which imagined as being a review, modify, and approve necessary (rather than taking the human "out of the loop"). We build a benchmark clinical dataset and propose the Re which imitates the working patterns of physicians first retrieve related working experience from historical Pls written by physicians, then reason related medical knowledge. Finally, it refines the retrieved working experience and reasoned medical knowledge to extract usefuu information, which is used to generate the for previously-unseed patient according to their health records during hospitalization. Our experiments show that, using our method, the performance of five different models can be substantially boosted across all metrics, with up to 20%, 1 11% and 19% relative improvements in BLEU-4, ROUGE-L and METÉOR, respectively. Meanwhile, show results from human evaluations measure the effectiveness in terms of its usefulness for clinical practice.
document that the clinician
Introduction
At the time of discharge home, the Patient Instruction (PI), which is rich paragraph of text containing multiple instructions, is provided by the attending clinician to the patient or guardian. PI
'Equal contribution.
i 'Corresponding authors.
'The code is available attp://gttiiccmmAAnnaaaallliliit
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Input (Patient's Health Records)
Admission Notes
Nursing
Radiology Notes
Physician Notes
Medications
Notes
Output (Patient Instruction)
Patient 1: Please shower daily including washing incisions gently with soap, baths swimming, [...] please lotions, cream, powder, or ointments incisions [...] females: please wear bra reduce pulling on incision, avoid rubbing on lower edge.
You were admitted for bleeding from ulcer in your stomach. This ulcer at least partially caused by naproxen. You should stop taking naproxen and take only tylenal for pain. [...] you are scheduled to get repeat endoscopy next week. Prior to the procedure do not have anything drink eat after midnight.
Patient 2:
Figure 1:
Two examples of the Patient Instruction written by physicians which guide the patients how to manage their conditions after discharge based on their health records during hospitalization.
is used for the purpose of facilitating safe and appropriate continuity of care 37 43 As result,
the PI has significant implications for patient management and good medical care, lowering down the risks of hospital readmission, and improving the doctor-patien relationship. As shown in Figure 1
PI typically contains the following three main components from patients' perspective What is my main health condition? (i.e., why was in the hospital?) (2) What do I need to do? (i.e..
45
how do manage home, and how should best care for myself?) (3) Why is it important for me to do this? Of the above, the second section is often considered to be the most important information. For example, when patient has had surgery while the hospital, the PI might tell the patient to keep the wound away from water to avoid infection.
Currently, the following skills are needed for physicians write medical knowledge for interpreting the patient's 's clinical records, including diagnosis, medication, and procedure records; (2) skills for carefully analyzing the extensive and complex patient's 's data acquired during hospitalization, e.g., admission notes, nursing notes, radiology notes, physician notes, medications, and laboratory results; (3) the ability of extracting key information and writing instructions appropriate for the lay reader. Therefore, writing PIs is a necessary but Lime-consuming task for physicians, exacerbating the workload of clinicians who would otherwise focus on patient care [50 41 Besides, physicians need to read lots of patient's ' health records in their daily work, resulting in substantial opportunity for incompleteness or inappropriateness
37
(1) thorough
wording 44 51].
Statistically, countries with abundant healthcare resources, such the United States, have up to 54% of physicians experiencing some sign of burnout in one year of study [42), which further exacerbated in countries with more tightly resource-constrained healthcare resources.
The overloading of physicians systems that can partly automate routine tasks, such as generation of PIs for modificatiion/pprooall by clinicians is an important contribution to healthcare practice. To this end, we propose the novel task of automatic Pl generation, which aims to generate accurate and fluent textual PI given input health records of a previously-unsenn patient during hospitalization. physicians, given the health records of new patient, need only review and modify the generated PI, rather than writing a new PI from scratch, significantly relieving the physicians from the heavy workload and increasing their time and energy spent in meaningful clinical interactions with patients. Such endeavors would be particularly useful in resource-limite countries [46].
well-documented phenomenon
44 51 and Al-related support
this way, intended that
In this paper, we build dataset PI and propose deap-len approach named &ritrr, which imitates the physicians' working patterns to automatically generate PI the point discharge from hospital. Specifically, when patient discharges from the hospital, physicians will carefully analyze the patient's health records terms of diagnosis, medication, and procedure, then accurately write a corresponding Pl based on their working experience and medicul knowledge 5 6 In order to model clinicians' text production, Re &eiter, which introduces three components: Retrieve, Reason, and Refine, (1) first encodes working experience by mining historical Pls, i.c., retrieving instructions

---

of previous patients according to the similarity of diagnosis, medication and procedure; (2) then reasons medical knowledge into the current input patient data by learning knowledge graph. which is constructed to model Jomain-speciifi knowledge structure; (3) at last, refines relevant information from the retrieved working experience and reasoned medical knowledge to generate final Pls.
To prove the effectiveness of our Re Writer we incorporate it into 5 different language generation models: 1) recurrent neural network (RNN)-based model, 2) attention-based model, 3) hierarchical RNN-based model, 4) copy mechanism-bass model, 5) fully-attentive model, i.e., the Transformer
The extensive experiments show that our approach can substantially boost the performance of baselines across all metrics.
47
Overall, the main contributions of this paper are:
We make the first attempt to automatically generate the Patient Instruction (PI), which can reduce the workload of physicians. As a result, it can increase their time and energy spent in meaningful interactions with patients, providing high-quality care for patients and improving doctor-patient relationships.
To address the task, we build dataset PI and propose an approach RerWer, which imitates physicians' working patterns to retrieve working experience and reason medical knowledge, and finally refine them to generate accurate and faithful patient instructions.
We prove the effectiveness and the generalization capabilities of our approach the built PI dataset. After including our approach, performances of the baseline models improve significantly on all metries, with up to 20%, 11%, and 19% relative improvements in BLEU- 4, ROUGE-L, and METHOR, respectively. Moreover, we conduct human evaluations to the generated PI for its quality and usefulness in clinical practice.
2 Related Works
The related works are introduced from two aspects: 1) Natural Language Generation (NLG) and 2) Medical Text Generation.
Natural Language Generation (NLG) It aims automatically generate coherent text in natural- language from given corresponding input data, which can be, c.g., text M. images [4]. video 48], and audio [8. Commonly, approaches use an encoder-decoder framework, where the encoder computes intermediate representations for the input source data and the decoder adopts RNNs or CNNs 14 9. to generate the target sentences given the intermediate representation. The attention mechanism B3 47 [22 27 and the copy mechanism [I]] have been proposed to directly provide the decoder with the source information, enabling a more efficient use of the source data. particular, fully-attentive models, such as the Transformer [47], in which no recurrence is required, have successfully achieved state-of-the-ard performance for multiple natural language generation tasks. However, most existing work focuses the natural text-related data, where research concerning the clinical domain remains relatively under-studied.
Medical Text Generation Recently, language generation in the medical domain has received growing research interest. For example, Jing al. 18] Liu et (29 30
proposed to generate radiology reports from chest X-ray images; Lee (23] proposed to generate the clinical notes of emer- gency department cases from discharge diagnosis codes; Ive et al. [77 proposed to generate mental health records. Meanwhile, some works treated the generation of medical text summarization task. For example, Zhang et al.
(33 proposed to generate clinical impressions by summarizing radiology reports; Scott et al. [39 proposed to generate the patients' histories by summarizing their respective health records; Several works [36] 12 32 I0 40). including HARVEST [T3] a medical summarization tool, proposed to summarize patients' key conditions from their clinical health records, such as Hassanpour and Langlotz [22 and Shing et al. 440 which summarized important clinical entities from radiology reports and hospital health records, respectively.
Despite the encouraging performances of these methods in generating various types of medical text, to the best of our knowledge, none of them has attempted to generate the Patient Instructions, which
has significant implications for supporting care delivery 6 37 43 as noted earlier. We here make the
first attempt to generate the Patient Instructions, which is typically longer than existing NI.G-related output in the medical domain and which covers more diverse topics than previous medical text generation tasks, such clinical impressions [53]. clinical notes (23 and radiology reports [28] 31]
3

---

R'
Wpr
Gpr
PI Decoder
Output Probabilities
Record Encoder
Working Experience
Medical Knowledge
Linear Softmax
Add Norm
Retrivved of Patient
Norm
Forward
Retrieved Pi of Palient
Feed Forward
Add Norm
Norm
Mulli-Head Attention
Retrieved Pl of Patient's
Refine
Word Embedding
Retrieve
Reason
Add & Norm
Mulli-Head Allention
Input (Patient's Health Records)
Admissionn
Nuriing Notes
Radiolog Notes
Physician
Medicaiion
Notes
Notes
Target Instruction
Figure 2:
We take the Transformer 47 as our baseline as an example to illustrate our Writer:: Retrieve, Reason, and Refine, which designed first retrieve related working experience from historical PIs written by physicians and reason related medical knowledge from a medical knowledge graph; then adaptively refine and merge them generate accurate and faithful patient instruction for current previously-unseen patient.
3 Approach
We first define the PI generation problem; then, we describe the proposed Writer in detail.
3.1 PI Generation Problem Definition
When patient is discharged from the hospital, the PI generation system should generate fluent and faithful instruction help the patient or carer to manage their conditions at home. Therefore. the goal of the PI generation task is to generate a target instruction / = {31, ./..... VN. } given the patient's s health records R (r1, .,, (N in terms of diagnoses, medications and procedures performed during hospitalization.
Since the input R including NR words and the output including Ni words are both textual sequences, we adopt the encoder-decoder framework, which is widely-used in natural language generation tasks, perform the Pl generation task. In particular, the encoder-decoder framework includes health record encoder and PI decoder, which can be formulated as:
(I) where R' E Xx denotes the record embeddings encoded by the record encoder, e.g., LSTM 14] or Transformer [4)). Then, R' is fed into the PI decoder (which again could be an LSTM or Transformer), generate the target PI T. During training, given the ground truth PI for the input patient's health records, we can train the model by minimizing the widely-used cross-cntropp loss.
Record Encoder R Rt: Decoder R' + 1,
3.2 The Proposed Re Re Writer
Our Writer consists of three core components: Retrieve, Reason, and Refine.
Formulation the Weitrr
stated above, given the health records R encoded as R' E R-Nxxd, we aim generate a desirable PI T. Figure 2 the detail of our method, which is designed retrieve related working experience Wp, and reason related medical knowledge Gr- from the training corpus for current input patient. Finally, Re3 refines the retrieved working

---

experience Wp and reasoned medical knowledge Gp, to extract useful information to generate a proper Pl:
Record Encoder: R - R'; Retrieve: : R - Wpri Reason : R - Gpai
Refine PI Decoder: : {R', Wpr, Gp.} -> T.
Our method can be incorporated into existing encoder-decoder based models, to boost their perfor- mance with PI generation, as we will later demonstrate. We now describe how to retrieve related working experience and reason related medical knowledge from the training corpus for PI generation.
Retrieve As shown in Figure D a hospitalization typically produces records of diagnoses given, medications used, and procedures performed; our dataset (described later) has 1 1. 1,208 unique clin- ical codes, including 5,973 diagnosis codes, 3,435 medication codes, and 1,800 procedure codes. Therefore, we first extract the one-hot embeddings of all clinical codes. Given a new patient, we represent this patient's 's hospitalization by averaging the associated one-hot embeddings of clinical codes produced during this hospitalization. Then, we collect a set of patients similar to the new patient according to the associated clinical codes. Taking the diagnosis codes as an example, we retrieve Np patients from the training corpus with the highest cosine similarity to the input diagnosis codes of the current patient. The PIs of the top-Np retrieved patients are returned and encoded by BERT encoder 7 (88, followed by a max-pooling layer over all output vectors, and projected to dimensions, generating the related working experience in terms of Diagnosis for the current patient: {I1 I...... IN₂ } e R Nexd Similarly, we use the medication codes and procedure codes to acquire the working experience in terms of Medication and Procedure, respectively. At last, these code-specific working experience representation (Diagnosis, Medication, Procedure) to obtain the final working experience related to current patient Wp, = (I,, I2. ...,, Ja+ Nr ) E TR (3+N)) We also attempted to incorporate age and gender information into our approach to match patients, please see Appendix C for details.
concatenate
Reason
To reason related medical knowledge from the training corpus, we construct an off-the-shelf global medical knowledge graph 9 (V, E) using all clinical codes, i.e., diagnosis, medication, and procedure codes, across all hospitalizations, where V = {ox == R R.Nxx is a set of Nkg nodes and E- = {et. } =:j==:NKo is a set of edges. The G models the domain-specific knowledge structure. In detail, we consider the clinical codes nodes. The weights of the edges are calculated by normalizing the co-occurrence of pairs of nodes in the training corpus. After that, guided by current input patient's health records, the knowledge graph is embedded by graph convolution network (GCN) 34 24 (2), acquiring a set of node embeddings {v1, v/...., '' h. which is regarded as our reasoned medical knowledge Gp, {oi vg..... ................................ D'NNa } IR x
Please refer to Appendix A for the detailed description of our medical knowledge graph. We note that more complex graph structures could be constructed by using larger-scale well-defined medical ontologies. Therefore, our approach is not limited to the currently constructed graph and could provide good basis for future research in this direction.
Refine As shown in Eq. )2. the PI decoder equipped with our method aims to generate the final PI based on the encoded patient's 's health records R' I Nuxd the retrieved working experience Wix E (3+) xd and the reasoned medical knowledge Gpr E N In implementations, we can choose either the LSTM [14] Transformer [47] the decoder. Taking the Transformer decoder as an example: for each decoding step I, the decoder takes the embedding of the current input word r; wi + e, Rd input, where w. and e denote the word embedding and fixed position embedding, respectively; we then generate each word in the target instruction (y1, (2,,,,,--.0 J:
he = MHA(H, maa); Mr Refine(h,, R², Wp., Gpt); Yt Pt Softmax(FFN(M)), + bp) (3)
where the MHA and FFN respectively stand for the Multi-Head Attention and Feed-Forward Network in the original Transformer (see Appendix D): Wp E Rdx DD and b, are the learnable parameters (DD:: vocabulary size); the Refine component then refines the Wp, and Gp. extract useful and correlated information to generate an accurate and faithful PI.
Intuitively, the Pl generation task aims to produce an instruction based on the source patient's health records R', supported with appropriate working experience WA, and medical knowledge Gp. Thus, Wp, and Ge, play auxiliary role during the PI generation. To this end, the Refine component,
5

---

which makes the model adaptively learn to refine correlated information, is designed as follows:
Refine(hu, Rt', Wer, Gp.) MHA R') + X @ MHA((,, Wo,) A2 & MHA Gp.) A1 =a ((he; (H, Wo.)| + baI); X22 a ([hr; MHA (h,, Gp.) W + ba2)
(4)
where Wa1. WA2 E 12ddd and b 2y2 are learnable parameters. O a, and [: denote the element- wise multiplication, the sigmoid function, and the concatenation operation, respectively. The com- puted λ1, 22 E [0, 1] weight the expected importance of Wi: and Gp, for each target word.
In particular, if LSTM is adopted as the PI decoder, we can directly replace the MHA with the LSTM unit and remove the FFN in Eq. (3). In the following sections, we will prove that our proposed Re Writer can provide solid basis for patient instruction generation.
4 Experiment
We first introduce our dataset used the basis for our experiments, as well as the baseline models and settings. We subsequently show evaluations using both automatic and "human" approaches.
4.1 Dataset, Baselines, and Settings
Dataset We propose a benchmark clinical dataset Patient Instruction (PI) with around 35k pairs of input patient's 's health records and output patient instructions. detail, we collect the PI dataset from the publicly-accessible MIMIC-III v1.4 resource 20 19 which integrates de de-identified, comprehensive clinical data for patients admitted to the Intensive Care Unit of the Beth Israel Deaconess Medical enter in Boston, Massachusetts, USA. This resource is an important "benchmark" dataset that promotes easy comparison between studies in the area. For each patient in the MIMIC-III v1.4 resource, the dataset includes various patient's ' health records during hospitalization in terms of diagnoses, medications and procedures, e.g., demographics, laboratory results, admission notes, nursing notes, radiology notes, and physician notes. In our experiments, we found that the discharge summaries contain the abstractive information of patient's health records. Therefore, for clarity, we adopt such abstractive information generate the PI. We concatenate all available patients' health records as the input of our model.
For data preparation, we first exclude entries without patient instruction and entries where the word-counts of patient instructions are less than five. This results in our PI dataset of 28,029 unique patients and 35,851 pairs of health records and patient instructions, summarized in T We randomly partition the dataset into 80%- 10%- 10% cain--vliiattin-tess partitions according patients. Therefore, there is no overlap of patients between train, validation and test sets. Next, we pre-process the records and instructions by tokenizing and converting text to lower-case. Finally, we filter tokens that occur fewer than 20 times in the corpus, resulting in a vocabulary of approximately 19.9k tokens, which covers over 99.5% word occurrences in the dataset.
Table 1: Statistics of the built Patient
Instruction (PI) dataset.
Statistics
TRAIN
VAL
TEST
Number Instructions
28,673
3,557
3,621
Number of Patients
22,423
2,803
2,803
Avg. Instruction Length
162.5
164.5
162.8
Avg. Record Length
2147.1
2144.9
2124.3
Baselines We choose five representative language generation models with different structures as baseline models, i.c., 1) RaN- model (LSTM) [4) 2) attention- model (Seq2Seq) []
3) hierarchical RNN-based model (HRNN) 26), 4) copy mechanism based model (CopyNet) II]) 5) fully-attentive model (Transformer)
47 We prove the effectiveness of our Re Writer by comparing the performance of the various baseline models with and without the Writer..
Settings The model size d is set to 512. For a patient, we directly concatenate all available patient's health records during hospitalization as input, e.g., admission notes, nursing notes, radiology notes. For example, if a patient only has admission notes, our model will just rely on the available admission notes to generate the PI. We adopt Transformer [47] as the record encoder. Based on the average performance on the validation set, the number of retrieved previous PIs, Np. is set to 20 for all three codes (see Appendix B). During training, we use the Adam optimizer (21] with a batch size of 128 and learning rate of 10 for parameter optimization. We perform early stopping based on BLEU-4 with maximum 150 epochs. During testing, we apply beam search of size 2 and repetition penalty

---

Table 2: Performance of automatic evaluation built benchmark dataset PL. Higher is better in all columns. We conducted 5 runs with different seeds for all experiments, the t-tests indicate that < 0.01. The (+Number) denotes the absolute improvements. As see, all the baseline models with significantly different structures enjoy a comfortable improvement with our Re Writer approach.
Dataset: Patient Instruction (PI)
Methods
METEOR
ROUGE-1
ROUGE-2
ROUGE-L
BLEU-1
BLEU-2
BLEU-3
BLEU-4
LSTM
16.5
35,9
17.9
33.2
34,4
26.3
23.1
21.0
with Writer
19.6 (+3.1)
39.4 (+3.5)
20.5 (+2.6)
37.0 (+3.8)
40.8 (+6.4)
31.5 (+5.2)
27.6 (+4.5)
25,3 (+4.3)
Seq2Seq 0
19.9
39.0
20.3
37.1
41.6
32,5
27.9
25.1
with Writer
20.9 (+1.0)
40.8 (+1.8)
21.9 (+1.6)
38.6 (+1.5)
43.2 (+1.6)
34.2 (+1.7)
29.7 (+1.8)
26.8 (+1.7)
IRRNN 26
40.1
36,9
33.7
25.6
with R Writer
21.6 (+1.3)
42.5 (+2.4)
22.1 (+1.6)
39.0 (+2.1)
47.2 (+3.7)
36.9 (+3.2)
31.5 (+2.7)
27.8 (+2.2)
CopyNet M
19.5
38.3
19.9
36,5
40.4
31,6
27.0
24.4
with Writer
20.6 (+1.1)
39,9 (+1.6)
20.9 (+1.0)
37.8 (+1.3)
42.7 (+2.3)
33.6 (+2.0)
28.7 (+1.7)
26.0 (+1.6)
foansfomeer 47
21.8
42,1
21.6
38.9
47.1
36.8
31.4
27.3
with Re Re Writer
23.7 (+1.9)
45.8 (+3.7)
24.4 (+2.8)
42.2 (+3.3)
52.4 (+5.3)
41.2 (+4.4)
35.0 (+3.6) 30.5 (+3.2)
Table 3:
Performance of human evaluation for comparing our method (baselines with Re Writer) with baselines in terms of the fluency of generated Pls, the comprehensiveness of the generated true PIs and the faithfulness to the ground truth PIs. All values are reported in percentage (%).
Table 4: Evaluation of how many times physicians would have deemed the generated re- sult as "helpful" vs. "unhelp- ful" in terms of assisting them in writing Pls. Baseline denotes the Transformer [47]
Seq2Seq I
Transformer
47
Metries
Baseline Win Tie
Ours Win Baseline Win
Tic
Ours Win
Methods Helpful Unhelpful
Fluency
10.5
69.5
20.0
27.0
40.5
32.5
Faithfulness
22.5
38.0
39,5
25.5
31.0
43.5
Baseline Ours 74%
32%
68%
Comprehensiveneess
17.5
48.0
34.5
21.0
24.0
55.0
26%
2.5. For all baselines, we keep the inner structure of baselines untouched and preserve the same training/testing settings for experiments.
4.2 Automatic Evaluation
Metries We measure the performance by adopting widely-used natural language generation metrics,
i.e. BLEU-1, -2, -3, -4 (5), METEOR (2 and ROUGE-1, -2, -L (25), which are calculated by
the evaluation toolkit 3 and measure the match between the generated instructions and reference instructions annotated by professional clinicians.
Results Table[] that our Writer can consistently boost all baselines across all metrics, with relative improvement of 7%,~0%% 4%%~1% and 59% BLEU-4, ROUGE-L, and METEOR, respectively. The improved performance proves the validity of our approach in retrieving working experience, reasoning medical knowledge, and refining them for PI generation. The performance gains over all of the five baseline models also indicate that our approach is less prone to the variations of model structures and hyper-parameters, proving that the generalization capabilities of our approach are robust over wide range of models. In Section 4.5 we verify the robustness of our approach to various data/examples.
4.3 Human Evaluation
Metrics We further conduct human evaluation to verify the effectiveness of our approach WeiWer in clinical practice. To successfully assist physicians and reduce their workloads, important to generate accurate patient instructions (faithfulness, precision), such that the model does not generate instructions that "do not exist" according to doctors. is also necessary to provide comprehensive true instructions (comprehensiveness, recall), i.e., the model does not leave out the important instructions. For example, given the ground truth PI [Instruction_ A, Instruction B] written by physicians, the PI generated by Model_ is [Instruction _A, Instruction B, Instruction CJ, and the PI generated by
7

---

Table Ablation study of our RerWer,er, which includes three components: Retrieve, Reason, and Refine, two representative baseline models, i... Seq2Seq and Transformer 47 Full Model denotes the baseline model with our R Writer.
Settings
Diagnose Medication Procedure
Retrieve:
Working Experience
Knowledge
Reason:
Reline
METEOR
ROUGE-
Dataset: Patient Instraction (PI)
ROUGE22
ROUGEE I. BLEU-- BLEU
Seq25eq (a)
19.9
39.0
20.3
37.1
27.9
25.1
20.1
39.1
20.5
37.2
28.3
25.5
20.7
39.8
21.2
37.8
29.0
26.1
(c)
20.1
39.2
20.6
37.2
28.4
25.6
(d)
J
20.6
40.1
21.1
38.0
29.2
26.3
(c)
40.5
21.6
38.4
29.4
26.5
(1)
20.7
40.5
21.7
38.2
29.5
26.6
Full Model
20.9
40.8
21.9
38.6
29.7
26.8
Transformer
21.8
42.1
21.6
38.9
31.4
27.3
22.2
43.1
22.3
39.7
32.3
28.0
(b)
22.7
44,2
23.0
40.7
33.4
28.9
22.7
32.7
(d)
23.1
44.5
23.6
41.0
33.6
29.2
23.2
44.8
23.8
41.4
34.0
29.4
45.2
24.1
41.8
34.3
29.9
Full Model
23.7
45.8
24.4
42.2
35.0
30.5
Model_2 _ is [Instruction_]] _. Model_ better than Model_2 in terms of comprehensiveness, while worse than Model in terms of faithfulness. Finally, unacceptable generate repeated or otherwise unreadable instructions (fluency). Therefore, we randomly select 200 samples from our PI dataset. The human evaluation conducted by two junior annotators (medical students) and senior annotator (clinician). All three annotators have sufficient medical knowledge. By giving ground-rre -trut Pls, each junior annotator was asked to independently compare the outputs of our approach and that of the baseline models, in terms of the perceived quality of the outputs including fluency, comprehensiveness. and faithfulness outputs compared to the corresponding ground-trus instructions. The senior annotator re-evaluated those cases that junior annotators have difficulties deciding. The annotators were blinded to the model that generated the output instructions.
Results We select representative baseline, Seq2Seq, and competitive baseline, Transformer, to show the human evaluation results (Table As may seen from the results, our Wrrter better than baselines with improved performance in terms of the three aspects, i.e., fluency, comprehen- siveness and faithfulness. The human evaluation results show that the instructions generated by our approach are of higher clinical quality than the competitive baselines, which proves the advantage of our approach in clinical practice. In particular, by using our Re Writrr,,r, the winning chances increased by maximum of 43.5 25.5 18 points and 55 21 34 points in terms of the faithfulness metric (precision) and comprehensiveness metric (recall), respectively. At last, we further evaluate how many times physicians would have deemed the generated result as "helpful" vs. "unhelpfull in terms assisting them in writing a PI. The results are reported Table 4. As we can see, our approach can generate more accurate PIs than the baselines, improving the usefulness of AI systems in better assisting physicians in clinical decision decision-makings and reducing their workload.
4.4 Ablation Study
We conduct a quantitative analysis using the Seq2Seq and the Transformer for the purposes of evaluating the contribution of each proposed component: Retrieve, Reason, and Refine.
Efffet of the 'Retrieve' Component Table (a,b.c) shows that the Diagnosis, Medication, and Pro- cedure elements of the model all contribute to a boost in performance, which proves the effectiveness of our approach in retrieving similar patient instructions from the available repository of historical PIs to aid in the generation of new patient instructions. Among these three elements, Medication leads to the best improvements, which may explained by the fact that the PI is more relevant to medications from historical experience [45. By combining the three elements (d), we observe an overall improvement. As result, retrieving related working experience can boost the performance of baselines: 25.1 + 26.3 and 27.3 29.2 in BLEU-4 for Seq2Seq and Transformer, respectively.

---

Table 6: Performance of approach the three sub-datasets: Gender, Age, Disease.
Female
Male
Gender
METEOR
ROUGE-L
BLEU-4
METEOR
ROUGE-L
BLEU-4
Seq2Seq
19.8
35.9
25.0
20.0
38.0
25.2
with Writer
20.6
37.6
26.3
21.1
39.5
27.2
Transformer 21.5
38.1
26.9
22.0
39.6
27.6
Wniiri
23.2
41.3
30.1
24.1
43.0
30.8
Age-55
55<=Age<70
Age
Age Group
METEOR ROUGE-L BLEU-4
METEOR
ROUGE-L BLEU-4
METEOR ROUGE-L BLEU-4
Seq2Seq
18.2
34.7
21.9
20.7
39.5
26.7
20.7
37.1
26.5
with Writer
19.2
35.6
23.7
21.8
41.2
28.4
21.5
38,9
28.1
Transforner
20.2
36.9
24.4
23.1
41.3
28.5
22.8
39.0
28.
with Re Writer
22.9
40.1
28.5
26.2
45.0
31.8
24.3
42.7
31.2
Hypertension
Hyperlipidemia
Anemia
Disease
METEOR ROUGE-L BLEU-4
METEOR ROUGE-L BLEU-4 METEOR ROUGE-L BLEU-4
Seq2Seq
21.3
39.8
27.9
21.3
41.7
27.2
18.0
36.4
20.7
with Re Wrritrr
22.6
41.4
30.4
22.5
43.9
29.5
18.8
37.6
22.3
Transformer 22.8
42.5
30.7
23.0
44.7
30.3
19.6
38.2
23.4
with Re" Writer 24.6
45.1
33.5
24.9
46.4
33.8
21.8
41.3
27.4
Effeet of the 'Reason' Component Table 5 (c) shows that the Reason component can further improve the performance by learning the enriched medical knowledge. By comparing (d) and (e), we can observe that the reasoned medical knowledge leads to similar improvements as the retrieved working experience does. We attribute this to the fact that learning conventional and general writing styles for PIs in deep seaming-bbsed approach is important incorporating accurate medical knowledge. In this way, the retrieved patient instructions of previous patients can be treated as templates, providing solid basis to generate accurate Pls.
Effect of the 'Refine' Component As shown in Table 5 (f and Full Model), it is clear that the model with our Refine component performs better than the model without it, which directly verifies the effectiveness of our approach. To further understand the ability of adaptively refining useful information, we summarize the average refining weight values \ and X of Eq. (4). We can find that the average value of X 0.42 is larger than Å₁ 0.26. It indicates that the medical knowledge plays prominent role in PI generation; it is in accordance with the results of (d) and (e). We conclude that our method is capable of learning to efficiently refine the most related useful information to generate accurate PIs. Overall, our three components improve the performance from different perspectives, and combining them can lead an overall improvement.
4.5 Robustness Analysis
In this section, we evaluate the performance of our approach with more fine-grained datasets. Specifi- cally, we further divide our PI dataset into three sub-datasets according to Gender, Age, and Disease. Specifically, to ensure an even distribution of the data, we divide the ages into three age-groups: Age < 55 (29.9%), 55 <= Age 70 (30.5%), and Age >= (39.7%). Table B shows the results of our approach on the three sub-datasets. As we can see, the proposed approach can consistently boost baselines across different genders, ages, and diseases on all evaluation metrics, proving the generalization capability and the effecttivnees of our method to different datasets/examples.
4.6 Qualitative Analysis
Figure shows that our approach is significantly better aligned with ground truth instructions than the baseline. For example, our approach correctly generates the key instructions ("you were found to have an infection your gallbladder" and "you also had a urinary tract infection which was treated with antibiotics" (Bluc-colored text)), and the personalised medication note "start oxycodone 5 mg

---

Ground Truth: dear ms. [name], you were admitted the hospital with abdominal pain, nausea, vomiting and inability you were diugnosed with a urinury tract infection, and treuted with untibiotics, your puin did improve, and you were diagnosed with chronic cholecystitis (inflammation of your gallbladder). your gallbladder was removed. changes to your home medications include: for pain, you can take acetaminophen. you can also take oxycodone 5 10mg every 4 6 hours for pain. your pain should be improving, you should require less medication over time.
Retrieved Working Experience
Baseline: dear ms. [name], it was pleasure taking care of you at [hospital]] you were admitted to the hospital for abdominal pain nausea vomiting, you were found have infoction your blood stream which treated with antibiotics. you were also started on antibiotics for this infection. you will need follow- up with [name] as listed below.
I.I.. the lovers and urinary tract infection
treated with untiviatics [..]
2. [-.] you will also be preseribed on antibiotic
your urinary infection [..]
Reasoned Medical Knowledge
with Re dear ms. [name], it was a pleasure taking care of you at [hospital], you were admitted to the hospital with abdominal pain and found have an infection in your gallbladder. you were treated with antibiotics for this in infection. you also had a urinary tract infection which was treated with antibiotics. we made the following changes to your medications :- start oxycodone mg every 6 hours as needed for pain.
Addociiaal
IL013 D.013
LERNEE BIXE infoction
pan
Iny codenda
pillMender
Figure 3: An example of the PI generated by baseline and our approach (i.c., baseline with Re Writrit Underlined text denotes alignment between the ground truth text and the generated text. Red colored text denotes unfavorable results. The Blue and Green colored text respectively denote the retrieved working experience and reasoned medical knowledge when generating corresponding sentences.
every 6 hours needed for pain" (Green-colored text)), for the patient, who was admitted to the hospital with abdominal pain. The baseline can only generate the correct reason for the patient's s admission, however, it also generates serious wrong instruction (Red-colored text). It proves the effectiveness of our approach in retrieving associated working experience and reasoning associated medical knowledge to aid PI generation. Meanwhile, note that our method produces interpretable refining weight values that may help understand the contribution of working experience and medical knowledge towards PI generation: when generating words similar to those that appear in the retrieved working experience (Blue-colored text) and reasoned medical knowledge (Green-colored text), the corresponding a and the A2 would be significantly larger than their means, λι and X2, efficiently refine the relevant information from working experience and medical knowledge, respectively.
5
Conclusion
We propose new task of Patient Instruction (PI) generation which attempts to generate accurate and faithful Pls. To address this task, we present an effective approach ReiWri::r: Retrieve, Reason, and Refine, which imitates the working patterns of physicians. The experiments on our built benchmark clinical dataset verify the effectiveness and the generalization capabilities of our approach. In particular, our approach not only consistently boosts the performance across all metrics for wide range of baseline models with substantially different model structures, but also generates meaningful and desirable PIs regarded by clinicians. It shows that our approach has the potential to assist physicians and reduce their workload. In the future, it can be interesting to generate personalized PIs by taking into account the patient's 's cognitive status, health literacy, and other barriers to self-care.
Limitations and Societal Impacts: The training of our approach relies on a large volume of existing Pls. The current model performance may be limited by the size of the built dataset. This might be alleviated in the future by using techniques such as knowledge distillations from publicly- available pre-trained models, e.g., ClinicalBERT [15]. Although alleviating the heavy workload of physicians, it is possible that some physicians directly give the generated PI the patients or guardians without quality check. Also for less experienced physicians, they may not be able to correct the errors in the machine-generatee PI. To best assist the physicians via our approach, it is required to add process control to avoid unintended usage.
approach has the potential of
Acknowledgments
This work was supported in part by the National Institute for Health Research (NIHR) Oxford Biomed- ical Research Centre; an InnoHK Project at the Hong Kong Centre for Cerebro-caddiovascula Health Engineering: and the Pandemic Sciences Institute, University of Oxford, Oxford, UK. Fenglin Liu gratefully acknowledges funding from the Clarendon Fund and the Magdalen Graduate Scholarship.
10

---

References
[1] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
[2] S. Banerjee and A. Lavie. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In IEEvaluation (aciioo@@@AL. @ACL, pages 65-72, 2005.
[3] X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitmick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv: 1544.00300 2015.
[4] K. Cho, B. van Merrienboer, Ç. Gülgehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.
[5] A. Chugh, M. V. Williams, J. Grigsby, and E. A. Coleman. Better transitions: improving comprehension of discharge instructions. Frontiers of Health Services Management, 25(3):11, 2009.
[6] E. A. Coleman, A. Chugh, M. Williams, J. Grigsby, J. Glasheen, M. McKenzie, and S.-J. Min. Understanding and execution of discharge instructions. American Journal of Medical Quality, 2885)3383-31, 2013.
[7] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HILT, pages 4171-4186, 2019.
[8] K. Drossos, S. Adavanne, and T. Virtanen. Automated audio captioning with recurrent neural networks. WASPAA, pages 374-378, 2017.
[9] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. Convolutional sequence to sequence learning. In ICML, pages 1243-1252, 2017.
[10] J. J. Goff and T. W. Lochfelm. Automated radiology report summarization using an open-source natural language processing pipeline. Journal of Digital Imaging, 31(2):185-199 2018.
[11] J. Gu, Z. Lu, H. Li, and V. O. K. Li. Incorporating copying mechanism in sequence-to -ooeence-t-ssquenee learning. In ACL, 2016.
[12] S. Hassanpour and C. P. Langlotz. Information extraction from multi-institutional radiology reports. Artificial Intelligence in Medicine, 66:29-39, 2016.
[13] S. Hirsch, J. S. Tanenbaum, S. Lipsky Gorman, C. Liu, E. Schmitz, D. Hashorva, A. Ervits, D. Vawdrey, M. Sturm, and N. Elhadad. Harvest, longitudinal patient record summarizer. Journal of the American Medical Informatics Association, 22(2):263-274, 2015.
[14] S. Hochreiter and Schmidhuber. Long short-term memory. Neural Comput., 183:1775-1 1997.
[15] K. Huang, Altosaar, and R. Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint (rXXi::9944.03342. 2019.
[16] Institute for Hcalthcare Improvement. Ask me 3: Good questions for your good health. National Patient Safety Foundation, 2016.
[17] J. Ive, N. Viani, J. Kam, L. Yin, S. Verma, S. Puntis, R. N. Cardinal, A. Roberts, R. Stewart, and Velupillai. Generation and evaluation of artificial mental health records for natural language processing. NPJ Digital Medicine, 3(1):1-9, 2020.
[18] B. Jing, Xie, and E. Xing. On the automatic generation of medical imaging reports. In ACL, pages 2577-2586, 2018.
[19] A. F. W. Johnson, J. Pollard, and R. G. Mark. MIMIC-III clinical database (version 1.4). PhysioNer, 2016. URL :/doii////o/o// org/10.//// 13026/C22226

---

[20] A. E. W. Johnson, T. J. Pollard, L. Shen, L. wei H. Lehman, M. Feng, M. M. Ghassemi, B. Moody, Szolovits, L. A. Celi, and R. G. Mark. MIMIC-III, freely accessible critical care database. Scientific Data, 3, 2016.
[21] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
[22] N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
[23] S. Lee. Natural language generation for electronic health records. arXiv preprint arXiv:180..01333 2018.
[24] Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel. Gated graph sequence neural networks. In ICLR, 2016.
[25] C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries, In ACL, pages 74-81, 2004.
[26] R. Lin, Liu, M. Yang, M. Li, M. Zhou, and S. Li. Hierarchical recurrent neural network for document modeling. In EMNLP, pages 899-907, 907,2015,
[27] Liu, Liu, X. Ren, X. He, and X. Sun. Aligning visual regions and textual concepts for semantic-groume grounded image representations. In NeurlPS, 2019.
[28] F. Liu, S. Ge, and X. Wu. Competence-base based multimodal curriculum learning for medical report generation. In ACL/JJCNPP, 2021.
[29] F. Liu, X. Wu, S. Ge, W. Fan, and Y. Zou. Exploring and distilling posterior and prior knowledge for radiology report generation. CVPR, 2021.
[30] F. Liu, C. You, X. Wu, S. Ge, S. Wang, and X. Sun. Auto-encoding knowledge graph for unsupervised medical report generation. In NeurlPS, 2021.
[31] G. Liu, T. H. Hsu, M. B. A. McDermott, W. Boag, W. Weng, P. Szolovits, and M. Ghassemi. Clinically accurate chest x-ray report generation. In MLHC, pages 249-269, 2019.
[32] H. Liu and C. Friedman. Cliniviewer: tool for viewing electronic medical records based natural language processing and xml. In MEDINFO 2004, pages 639-643. IOS Press, 2004.
[33] T. Luong, H. Pham, and C. D. Manning. Effective approaches to attention-based neural machine translation. In EMNLP, pages 1412-1421, 2015.
[34] D. Marcheggiani and I. Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In EMNLP, pages 1506-1515, 2017.
[35] K. Papineni, S. Roukos, T. Ward, and W. Zhu. BLEU: a Method for automatic evaluation of machine translation. In ACL, pages 31 1 1-118, 2002.
[36] R. Pivovarov and N. Elhadad. Automated methods for the summarization of electronic health records. Journal of the American Medical Informatics Association, 22(5):938-9) 00799999777 2015.
[37] R. D. Powers. Emergency department patient literacy and the readability of antient-directed materials. Annals of Emergency Medicine, 17(2):124 124-1114,, 1988.
[38] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP/IJCNLP.. pages 3980-3990, 2019.
[39] D. Scott, C. Hallett, and R. Fettiplace. Data-to-text summarisation of patient records: Using computer-generated summaries to access patient histories. Patient Education and Counseling, 92(2):153-159, 2013.
[40] H. Shing, C. Shivade, N. Pourdamghani, F. Nan, P. Resnik, D. W. Oard, and P. Bhatia. Towards clinical encounter summarization: Learning to compose discharge summaries from prior notes. arXiv preprint 2104422104.13498, 2021.
12

---

[41] C. Sinsky, L. Colligan, L. Li, M. Prgomet, S. Reynolds, L. Goeders, J. Westbrook, M. Tutty, and G. Blike. Allocation of physician time in umbulatory practice: a time and motion study in specialties. Annals of Internal Medicine, (65(11):753--27 2016.
[42] C. Sinsky, M. Tutty, and L. Colligan. Allocation of physician time in ambulatory practice. Annals of Internal Medicine, 166(9):683-684, 2017.
[43] J. M. Spandorfer, D. J. Karras, L. A. Hughes, and C. Caputo. Comprehension of discharge instructions by patients in an urban emergency department. Annals of Emergency Medicine, 25 (1):71-74, 1995.
[44] D. Tawfik, J. Profit, T. Morgenthaler, D. V. Satele, C. A. Sinsky, L. N. Dyrbye, M. A. Tutty, C. P. West, and T. D. Shanafelt. Physician burnout, well-being, and work unit safery grades in relationship reported medical errors. In Mayo Clinic Proceedings, pages 1571-1580, 2018.
[45] D. M. Taylor and P. A. Cameron. Discharge instructions for emergency department patients: what should we provide? Emergency Medicine Journal, 17(2):86-90, 2000.
[46] The World Bank. Physicians per 1,000 people. 2018. URL httph: httplab//aaa...orlobbb org/ indicator/Sh MED. WWW.PHYS........................ ZS
[47] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. NIPS, pages 5998-6008, 2017.
[48] S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. J. Mooney, and K. Saenko. Translating videos natural language using deep recurrent neural networks. In HLT-NAACL, pages 1494-1500 1504, 2015.
[49] O. Vinyals, A. Toshev, S. Bengio. and D. Erhan. Show and tell: A neural image caption generator. In CVPR, pages 3156-3164, 2015.
[50] M. Weiner and P. Biondich. The influence of information technology on relationships. Journal of General Internal Medicine, 21(1):35-39, 2006.
[51] C. P. West, L. N. Dyrbye, and T. D. Shanafelt. Physician burnout: contributors, consequences and solutions. Journal of Internal Medicine, (83(6) 2018.
[52] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, S. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pages 2048-2057, 2015.
[53] Y. Zhang, D. Y. Ding, T. Qian, C. D. Manning, and C. P. Langlotz. Learning to summarize radiology findings. In Proceedings of the Ninth International Workshop Health Text Mining and Information Analysis, Louhi @ EMNLP 2018, pages 204-213, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes)
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and ensured that our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
13

---

(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as a URL)? [Yes] The code is available at https://giihub.coo////aaa
http://iiiaaaaaommaaaaa-aaaaaaaa Patt-aetenntnsttuctionn Instructions
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section [4.1
(c) Did you report error bars (c.g., with respect to the random seed after running ex- periments multiple times)? [Yes] We conducted runs with different seeds for all experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] PhysioNet Credentialed Health Data License 1.5.0 htttps://pysiione:: org content//////////ooo/ ciamiccccrr/vieeee wwwcc///e/viiieeeeeeeeeeeeeeeee
2.0.0/} used.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] The code is available https://github... http://tthuuu..om/AI// Hcalth/Patient Instructitnsttnnnnss
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [Yes] All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] All protected health information was de- identified. De-identification was performed in compliance with Health Insurance Portability and Accountability Act (HIPAA) standards in order to facilitate public access to the datasets. Deletion of protected health information (PHI) from structured data sources (c.g.. database fields that provide patient name date of birth) was straightforward. All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "h9brTxqaQDY", "reviewer": "Reviewer_uXD1", "summary": "In this paper, the authors have proposed a new task of generating patient instructions (PIs) using the MIMIC dataset. The authors also proposed a strong SOTA model for the task Re3Writer which takes into consideration the historic PIs, clinical domain knowledge and refining of the final generated PI for a patient. The authors show that their method can improve the performance of a lot of baseline models by following the three techniques defined in the ReWriter framework.", "questions": "Weakness/Questions:\nThese are some things on which I would like to hear from the authors.\n1. During train and test split, did the authors make sure that the same patient is not used in train and test. Even if the HADM (hospital admission ID) is different, a same patient can have similar PIs such as older patients can have same recommendations during multiple visits.\n2. Did you consider UMLS for knowledge injection? Because the information of a new patient with a new ICD code that has not been used before in the train set would get ignored. Whereas UMLS can help with that information, if you can run an entityLinker to get different clinical entities and then create your final graph. \n3. The patients were matches via their procedure, diagnostic codes but their other demographic/personal information such as age and gender wasn't considered. A patient with same ICD codes but with different age-group could potentially be assigned different PIs because of their health.\n4. Note: I might be wrong here but why not consider a pre-trained model? Is that because of constraint of the note length? ", "limitations": "Currently in this draft, no serious limitations are discussed. It is great to see Fig.4 where the model performs exceptionally well than the baseline model, it would be great to see some examples where ReWriter is consistently wrong.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength of the paper: \n1. It tackles an important problem and given it is created with the help of an openly available dataset (MIMIC), other researchers can also access the dataset and build further models.\n2. The ReWriter approach is very intuitive as it considers the past PIs which generating the PI for a new patient and then considers the medical knowledge for refining it. This medical knowledge further helps the model in generating clinically coherent PIs.\n3. All experiments are well defined but I really like the Table 5 ablation study because it justifies the use of each step in ReWriter. \n\nWeakness/Questions:\nThese are some things on which I would like to hear from the authors.\n1. During train and test split, did the authors make sure that the same patient is not used in train and test. Even if the HADM (hospital admission ID) is different, a same patient can have similar PIs such as older patients can have same recommendations during multiple visits.\n2. Did you consider UMLS for knowledge injection? Because the information of a new patient with a new ICD code that has not been used before in the train set would get ignored. Whereas UMLS can help with that information, if you can run an entityLinker to get different clinical entities and then create your final graph. \n3. The patients were matches via their procedure, diagnostic codes but their other demographic/personal information such as age and gender wasn't considered. A patient with same ICD codes but with different age-group could potentially be assigned different PIs because of their health.\n4. Note: I might be wrong here but why not consider a pre-trained model? Is that because of constraint of the note length? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cMxZ9vWjEqo", "reviewer": "Reviewer_oy13", "summary": "The paper proposed the way (Re$^{3}$Writer) to automatically generate the Patient Instruction (PI) to reduce the workload of clinicians. Re$^{3}$Writer consists of three components: Retrieve (to retrieve instructions of previous patients according to the similarity of diagnosis, medication and procedure), Reason (to reason medical knowledge into the current input patient data with the help of the Knowledge Graph), and Refine (to refine relevant information from the retrieved working experience and reasoned medical knowledge to generate final PIs). The authors built a dataset PI which is based on the MIMIC 3 data and showed experimentally that with the help of their method performance of 6 different SOTA models can be substantially boosted across all metrics (BLEU-4, ROUGE-L, METEOR). They also showed the results from human evaluations to evaluate the approach for the clinical practice routine. ", "questions": "1. Will authors publish the PI dataset? Or it will be enough to use the provided code and instructions to generate it?\n2. Could authors provide more details about the dataset generation?\n3. Does eICU[*] dataset have the text notes? If yes, will it be possible to include the results on this data as well?\n\n*Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eICU collaborative research database, a freely available multi-center database for critical care research. Scientific data, 5(1):1–13, 2018.", "limitations": "Limitations were discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. Overall the paper is well organised, easy and enjoable to read. The contributions and motivation are clearly put. In my opinion this paper is relevant to the people from Machine Learning for Healthcare domain. The problem of generating the clear PI is very relevant for the clinicians. The attempt of medical reports (not instructions) generation were done before, but most of them used other modalities (i.e images). Here authors provided the way to utilise only the text knowledge.\n2. Previous approaches are clearly explained and cited; experimental settings are well explained.\n3. With the help of Re$^{3}$Writer the performance was significantly increased (Table 2). Ablation studies clearly shows the impact of each component in the final result (Table 5).\n\nWeaknesses:\n1. More details about the PI dataset will be helpful: i.e did authors use discharge summaries? Also about the patients notes - they are sparse in time of the patient stay in hospital (i.e nursing and physicians notes, radiology reports), did author just concatenate all these notes, or did they perform the selection somehow?\n2. Additional ablation studies about what type of notes (nursing notes, physicians notes, radiology) contributes more in the Re$^{3}$Writer performance. However, it is a minor comment.\n3. Some relevant citations are missing, for example \"Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation\", Fenglin Liu, Chenyu You, Xian Wu, Shen Ge, Sheng wang, Xu Sun, NeurIPS'21. This paper (and not only this) also utilises the Knowledge Graph structure in order to generate the medical reports. Not the same idea, but it will be helpful for reader to see the full overview.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VEChKQ5masm", "reviewer": "Reviewer_Wpt8", "summary": "This paper introduces a deep-learning approach called Re3Writer to imitate the clinicians’ working patterns for automatically generating patient instructions at the point of discharge from the hospital.", "questions": "The application was tested using one generated dataset. How did the authors analyse the accuracy of the generated PIs? \nI would suggest performing the comparison of the output of the proposed deep-learning approach with more datasets/examples. Otherwise, it is hard to assess the novelty of the work.\n\nFurther technical explanations are needed (e.g., weights between nodes). For example, In Figure 4, under the reasoned medical knowledge box. What does 0.013 mean? \n\nHow the optimum model was selected? Further information regarding hyper-parameter tuning can be useful.\n\nBuilding a PI dataset is defined as a novelty (contribution). How it is built and tested? Did the authors build the deep-learning approach on this dataset? If so, how can authors make sure its applicability to real-world health data (e.g., clinical validation)? ", "limitations": "The proposed approach is interesting. However, further technical details, validation and also the comparison of the deep-learning application on more datasets are needed to assess its novelty. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Topic is timely\n2. The application is interesting\n3. The paper is easy to follow\n\nWeaknesses:\n1. Lack of technical details (e.g., hyper parameter tuning, information about node transitions/weights)\n2. Lack of output validation\n3. Novelty is limited\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept