SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As can be seen from Table 2, REFINE demonstrates superior performance compared to the baseline methods, with statistically significant differences  $(p$ -values  $< 0.05)$  based on a one-way ANOVA test [8]. The improved performance of REFINE can be attributed to its ability to account for medication dosage and lab test response trends across patient visits. Further, REFINE also outperforms the baselines in terms of wDDI, and is even lower than the wDDI present in MIMIC-IV (0.0448) and PRIVATE (0.1159) datasets. This suggests that REFINE can effectively reduce the potential risk of adverse drug interactions, which is a critical aspect of ensuring patient safety in clinical practice.
Table 3 shows the performance on a subset of patient instances where at least one change of medication occurs between consecutive visits. As more than  $90\%$  of the instances in MIMIC-IV already involve medication changes, we consider the performance reported in Table 2 to be representative of such
Table 3: Results for instances with medication change.
Methods	PRIVATE
AUC	F1	Jaccard	wDDI
LR	0.605 ± 0.006	0.536 ± 0.004	0.379 ± 0.008	0.1239 ± 0.0006
RF	0.607 ± 0.003	0.539 ± 0.005	0.381 ± 0.005	0.1234 ± 0.0002
BART	0.609 ± 0.002	0.541 ± 0.003	0.385 ± 0.004	0.1236 ± 0.0004
LEAP	0.602 ± 0.003	0.527 ± 0.005	0.374 ± 0.006	0.1111 ± 0.0002
DMNC	0.606 ± 0.006	0.532 ± 0.002	0.387 ± 0.005	0.1243 ± 0.0003
GameNet	0.619 ± 0.003	0.545 ± 0.005	0.392 ± 0.003	0.1124 ± 0.0006
CompNet	0.614 ± 0.002	0.552 ± 0.006	0.401 ± 0.008	0.1138 ± 0.0003
PREMIER	0.638 ± 0.001	0.576 ± 0.002	0.419 ± 0.005	0.1119 ± 0.0001
MICRON	0.647 ± 0.003	0.581 ± 0.006	0.422 ± 0.002	0.1114 ± 0.0004
SafeDrug	0.648 ± 0.004	0.591 ± 0.002	0.405 ± 0.006	0.1107 ± 0.0001
COGNet	0.665 ± 0.007	0.605 ± 0.002	0.437 ± 0.005	0.1129 ± 0.0007
MoleRec	0.667 ± 0.003	0.610 ± 0.001	0.442 ± 0.003	0.1106 ± 0.0002
REFINE	0.707 ± 0.002*	0.645 ± 0.001*	0.494 ± 0.003*	0.1097 ± 0.0001*
* indicates that the result is statistically significant when compared to the second best with p-value  $< {0.05}$  .
Table 4: Average number of recommended, extra, and missed medications.
Methods	MIMIC-IV	PRIVATE
# Average	# Extra	# Missed	# Average	# Extra	# Missed
PREMIER	9.85	4.60	3.47	5.13	2.3	1.35
MICRON	7.50	2.90	4.12	3.08	0.91	2.01
SafeDrug	7.01	2.57	4.29	2.93	0.77	2.02
COGNet	10.31	4.86	3.27	5.27	2.36	1.27
MoleRec	10.27	4.77	3.22	5.22	2.29	1.25
REFINE	9.48	3.70	2.94	4.91	1.93	1.20
cases. Therefore, we focus our analysis on the PRIVATE dataset, where around  $50\%$  of the instances involve medication changes between visits. We observe a drop in performance for all methods. This is expected as a change in treatment regime often implies a different patient response, making accurate predictions more challenging. Nevertheless, REFINE still outperforms the baseline methods in this challenging scenario. This indicates that even when faced with more complex situations involving medication changes, REFINE's integrated approach provides valuable insights and maintains a higher level of performance compared to other methods.
We also analyze the average number of recommended medications, average number of extra medications recommended, and average number of missed medications per visit for REFINE and the best performing baselines, as shown in Table 4. The average number of medications in the ground truth for the test set is 8.72 and 4.18 for MIMIC-IV and PRIVATE dataset respectively. We see that COGNet, PREMIER, and MoleRec generally recommend more medications whereas SafeDrug, MICRON tend to recommend fewer medications; whereas the average number of medications recommended by REFINE is closely aligned with the ground truth for both datasets. We find that COGNet has the highest average of extra medications in the recommended set. In contrast, REFINE recommends relatively lower extra medications. This can be attributed to the effective learning of patient condition from lab test and medication dosage trends. For the average number of missed medications, SafeDrug has the worst performance. On the contrary, REFINE misses the least number of medications from the ground truth set. This indicates that not weighing the benefits and potential interactions of drug pairs together may lead the system to omit medications necessary for managing a patient's condition.
We conduct an ablation study to evaluate the effect of trend information and drug interaction severity on the performance of REFINE. We have four variants: (a) REFINE w/o med trend excludes variance and slope of the medication dosages, (b) REFINE w/o lab trend omits the variance and slope of the lab test responses, (c) REFINE w/o any trends does not incorporate any trend information, and (d) REFINE w/o severity does not incorporate severity-based drug interaction.
Table 5 shows the results. We observe that REFINE w/o any trend has the lowest accuracy, indicating that trend information is crucial for making accurate fine-grained medication recommendations. There is a significant drop in performance for REFINE w/o severity, highlighting the importance of considering drug interaction severity when recommending medications.
Table 5: Results for ablation study.
Methods	MIMIC-IV	PRIVATE
AUC	F1	wDDI	AUC	F1	wDDI
w/o med trend	0.701 ± 0.001	0.618 ± 0.003	0.044 ± 0.0001	0.718 ± 0.003	0.645 ± 0.004	0.112 ± 0.0002
w/o lab trend	0.695 ± 0.002	0.613 ± 0.001	0.044 ± 0.0003	0.714 ± 0.004	0.643 ± 0.001	0.112 ± 0.0005
w/o any trends	0.682 ± 0.002	0.602 ± 0.002	0.045 ± 0.0002	0.703 ± 0.002	0.630 ± 0.001	0.112 ± 0.0004
w/o severity	0.694 ± 0.002	0.617 ± 0.001	0.046 ± 0.0005	0.713 ± 0.004	0.639 ± 0.002	0.115 ± 0.0003
REFINE	0.708 ± 0.001*	0.635 ± 0.003*	0.043 ± 0.0001*	0.729 ± 0.004*	0.656 ± 0.003*	0.111 ± 0.0002*
* indicates that the result is statistically significant when compared to the second best with p-value  $< {0.05}$  .
Finally, we present case studies to highlight REFINE's ability to provide accurate medication recommendations which ultimately contributes to better patient care. Figure 4(a) shows a sample patient from the PRIVATE dataset who has been diagnosed with Type 2 Diabetes Mellitus and Hyperlipidemia, and the HbA1c ranges from 7.4 to 9.8. In the second visit, we observe that a diabetes medication Glipizide is added to the Metformin treatment to manage the increased HbA1c value. REFINE successfully recommends the addition of Glipizide, while SafeDrug and COGNet do not. This can be attributed to REFINE's ability to learn lab test trends and incorporate balanced drug interaction information. REFINE identifies the increase in HbA1c value for the current visit as the influencing factor for Glipizide, which is consistent with clinical knowledge. Further, the decrease in HbA1c in the third visit indicates that certain medication combinations with some risk of interaction may indeed be necessary to effectively manage a patient's condition.
Figure 4(b) shows another patient from the PRIVATE dataset who has been diagnosed with Diabetes Mellitus, Hypertension, and Hyperlipidemia in visit 1, followed by chronic kidney disease in visit 2. We see that the medication Simvastatin is replaced by Atorvastatin in the second visit as it is preferred over other hyperlipidemia medications for patients with chronic kidney disease  $^{2}$ . REFINE closely follows the prescribed medication set and accurately recommends the medication change, indicating REFINE's ability to model co-morbidities. In contrast, SafeDrug and COGNet are unable to effectively capture this aspect.
Figure 4(c) shows a patient from the MIMIC-IV dataset who has been diagnosed with multiple diseases in visit 1, and is diagnosed with nontoxic multinodular goiter, and nausea with vomiting in visit 2. We observe that REFINE is able to recommend medications that matches the prescribed medications more closely compared to COGNet and SafeDrug. A closer inspection reveals that SafeDrug tends to under-recommend to avoid the interacting medication pairs, whereas COGNet tends to over-recommend many drugs outside the prescribed set.
We have described a unique fine-grained medication recommender system that offers personalized medication suggestions while considering the severity of potential drug interactions and explicitly modeling the trends in medication dosage titrations and lab test responses over visits. To the best of our knowledge, REFINE is the only system that factors in these trends while also assessing the varying severity of drug interactions. Experiments on real-world datasets and case studies show the effectiveness of REFINE in providing personalized, accurate, and safe medication recommendation.
The current system can be extended to include additional rich data from EHR such as clinician notes to further refine the system's recommendations. While we have shown that the severity of drug interactions is a crucial factor, the dosage strength of the drugs could also have a significant impact. A lower dosage of an interacting drug pair may sometimes be clinically acceptable and beneficial. Future work could explore how dosage strength influences the acceptability of interacting drug pairs.
This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-GC-2019-001-2A).
$^{2}$ https://tinyurl.com/3deyr6bp
Visit 1	Visit 2	Visit 3
Diagnosis	Diabetes mellitus, Hyperlipidemia	Diabetes mellitus, Hyperlipidemia	Diabetes mellitus, Hyperlipidemia
Lab Tests	HbA1c	Cholesterol HDL, Cholesterol LDL, Cholesterol Total, HbA1c ↑, Triglycerides	ALT test, Cholesterol HDL, Cholesterol LDL, Cholesterol Total, HbA1c ↓, Triglycerides
Prescribed Medications	Metformin, Simvastatin	Metformin, Glipizide, Simvastatin	Metformin ↓, Glipizide, Simvastatin
Recommended Medications	SafeDrug	Metformin, Simvastatin	Metformin, Atorvastatin	Metformin, Simvastatin
COGNet	Metformin, Simvastatin	Metformin, Simvastatin, Atorvastatin	Metformin, Simvastatin
REFINE	Metformin, Simvastatin	Metformin, Glipizide, Simvastatin	Metformin, Glipizide, Simvastatin
[Recommended Medications, Influencing Factors]	[Metformin, Diabetes mellitus], [Simvastatin, Hyperlipidemia]	[Metformin, Diabetes mellitus], [Glipizide, HbA1c ↑], [Simvastatin, Hyperlipidemia]	[Metformin, HbA1c ↓], [Glipizide, HbA1c ↓], [Simvastatin, Cholesterol LDL Calculated]
(a) Patient with fluctuating HbA1c values.
Visit 1	Visit 2
Diagnosis	Diabetes mellitus, Hypertension, Hyperlipidemia	Diabetes mellitus, Hypertension, Hyperlipidemia, Diabetic chronic kidney disease
Lab Tests	ALT test, Chloride serum, Cholesterol HDL, Cholesterol LDL, Cholesterol Total, Creatinine, Glucose (fasting) plasma, HbA1c, Potassium, Sodium, Triglycerides	HbA1c ↓
Prescribed Medications	Metformin ,Glipizide, Simvastatin, Lisinopril	Metformin, Glipizide, Atorvastatin, Lisinopril
Recommended Medications	SafeDrug	Metformin , Simvastatin, Atenolol	Metformin , Simvastatin, Atenolol, Lisinopril
COGNet	Metformin, Simvastatin, Enalapril	Metformin , Simvastatin, Atenolol, Lisinopril
REFINE	Metformin, Glipizide, Simvastatin, Atenolol	Metformin, Glipizide, Atorvastatin, Lisinopril
[Recommended Medications, Influencing Factors]	[Metformin, Diabetes mellitus], [Glipizide, HbA1c], [Simvastatin, Hyperlipidemia], [Atenolol, Hypertension]	[Metformin, Diabetes mellitus], [Glipizide, Diabetes mellitus], [Atorvastatin, Hyperlipidemia], [Lisinopril, Hypertension]
(b) Patient with changes in co-morbidities.
Visit 1	Visit 2
Diagnosis	Morbid (severe) obesity, Obstructive sleep apnea, Prediabetes, Postprocedural hypothyroidism, Vitamin D deficiency, Iron deficiency anemia, Hyperlipidemia, Hyperuricemia, Localized edema, Nausea	Nontoxic multinodular goiter, Nausea with vomiting
Lab Tests	Hematocrit	Parathyroid Hormone, Calcium
Prescribed Medications	Heparin, Lactate, Acetaminophen, Cefazolin, Levothyroxine, Famotidine	Heparin, Calcium carbonate, Hydromorphone, Calcitriol, Levothyroxine ↓, Scopolamine
Recommended Medications	SafeDrug	Heparin, Lactate, Acetaminophen, Levothyroxine, Simvastatin	Heparin, Levothyroxine, Scopolamine, Lactate
COGNet	Lactate, Ibuprofen, Cefazolin, Levothyroxine, Famotidine, Simvastatin	Heparin, Calcium carbonate, Hydromorphone, , Levothyroxine, Famotidine, Lactate
REFINE	Heparin, Lactate, Acetaminophen, Cefazolin, Levothyroxine, Famotidine, Simvastatin	Heparin, Calcium carbonate, Hydromorphone, Levothyroxine, Scopolamine
[Recommended Medications, Influencing Factors]	[Heparin, Postprocedural hypothyroidism], [Lactate, Nausea], [Acetaminophen, Postprocedural hypothyroidism], [Cefazolin, Postprocedural hypothyroidism], [Levothyroxine, Postprocedural hypothyroidism], [Famotidine, Nausea], [Simvastatin, Hyperlipidemia]	[Heparin, Postprocedural hypothyroidism], [Calcium carbonate, Calcium], [Hydromorphone, Postprocedural hypothyroidism], [Levothyroxine, Nontoxic multinodular goiter], [Scopolamine, Nausea with vomiting]
(c) Patient from MIMIC-IV with multiple diagnosis.
Figure 4: Medications recommended for sample patients. Red color indicates interacting medications. Symbols  $\uparrow$  and  $\downarrow$  depict an increase and decrease in the value of dosage or lab test response. The top influencing factor for each recommended medication provide insights into why REFINE recommends it.
[19] Qingxiong Tan, Andy Jinhua Ma, et al. Ua-crnn: Uncertainty-aware convolutional recurrent neural network for mortality risk prediction. In CIKM, pages 109-118, 2019.
[20] Ashish Vaswani, Noam Shazeer, et al. Attention is all you need. In NeurIPS, pages 6000-6010, 2017.
[21] Shanshan Wang, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Jun Ma, and Maarten de Rijke. Order-free Medicine Combination Prediction With Graph Convolutional Reinforcement Learning. In CIKM 2019, 2019.
[22] Rui Wu, Zhaopeng Qiu, Jiacheng Jiang, Guilin Qi, and Xian Wu. Conditional generation net for medication recommendation. In Proceedings of the ACM Web Conference 2022, pages 935-945, 2022.
[23] Guoli Xiong, Zhijiang Yang, Jiacai Yi, Ningning Wang, Lei Wang, Huimin Zhu, Chengkun Wu, Aiping Lu, Xiang Chen, Shao Liu, et al. Ddinter: an online drug-drug interaction database towards improving clinical decision-making and patient safety. Nucleic acids research, 50(D1):D1200-D1207, 2022.
[24] Chaoqi Yang, Cao Xiao, Lucas Glass, and Jimeng Sun. Change matters: Medication change prediction with recurrent residual networks. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3728-3734, 8 2021.
[25] Chaoqi Yang, Cao Xiao, Fenglong Ma, Lucas Glass, and Jimeng Sun. Safedrug: Dual molecular graph encoders for recommending effective and safe drug combinations. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3735–3741, 8 2021.
[26] Nianzu Yang, Kaipeng Zeng, Qitian Wu, and Junchi Yan. Molerec: Combinatorial drug recommendation with substructure-aware molecular representation learning. In Proceedings of the ACM Web Conference 2023, pages 4075-4085, 2023.
[27] Yutao Zhang, Robert Chen, Jie Tang, Walter F Stewart, and Jimeng Sun. Leap: learning to prescribe effective and safe treatment combinations for multimorbidity. In SIGKDD, pages 1315-1324. ACM, 2017.