PERSONALIZED LAB TEST RESPONSE PREDICTION WITH KNOWLEDGE AUGMENTATION

Abstract: Personalized medical systems are rapidly gaining traction as opposed to “one size
fits all” systems. The ability to predict patients’ lab test responses and provide justification for the predictions would serve as an important decision support tool and
help clinicians tailor treatment regimes for patients. This requires one to model
the complex interactions among different medications, diseases, and lab tests. We
also need to learn a strong patient representation, capturing both the sequential
information accumulated over the visits and information from other similar patients. Further, we model the drug-lab interactions and diagnosis-lab interactions
in the form of graphs and design a knowledge-augmented approach to predict patients’ response to a target lab result. We also take into consideration patients'
past lab responses to personalize the prediction. Experiments on the benchmark
MIMIC-III and a real-world outpatient dataset demonstrate the effectiveness of
the proposed solution in reducing prediction errors by a significant margin. Case
studies show that the identified top factors for influencing the predicted lab results
are consistent with the clinicians' understanding.

Reviews and author responses:

 Dear authors, 
Thank you for your response, and apologies for the late reply.  I believe your responses address my remaining concerns.  One note is that you should ideally make it clearer in the manuscript (and perhaps the table legend) that BEHRT was also provided with the medication data.
Good luck! 


This work presents a method for predicting HbA1c lab test results from electronic health record (EHR) data.  The method builds on lots of prior work that uses encoders to summarize longitudinal patient data, and adds some interesting elements.  Most notably, it incorporates various bits of domain specific knowledge about drug-lab test interactions (e.g., drug A tends to increase HbA1c lab measurements), patient similarity, and per-patient historical information about lab test results. The overall system is evaluated in MIMIC-3, comprising ICU stays, and a proprietary outpatient dataset, and seems to perform markedly better than baseline methods as measured by MSE, etc.  The authors present a pretty thorough ablation study, which has some interesting results.  Patient data in EHRs is complex - longitudinal, irregularly sampled, and missingness is very often informative.  There have been very many prior works applying the deep neural net architecture of the day, most recently transformer-like encoders, to this sort of data, usually for problems such as predicting mortality at various timeframes.  This work is a bit of a departure from the typical paper in that it focuses on a somewhat unusual target - HbA1c - which is useful as a marker for metabolic disease such as diabetes.  The authors claim that this is useful because such the management of chronic conditions like metabolic disease often involve empirically adjusting doses of medications to achieve target lab test values.  In this review I will leave aside questions of whether this is truly a clinically useful task and focus on the novelty and effectiveness of the methods presented. 

Task formulation
The problem is set up as a regression problem (HbA1c lab test results are numeric - percentages), and formulated in an autoregressive manner - to predict a lab test result at time T, all patient information from time points 0,...,T-1 can be used.  The space in which this paper operates is: how best can do we distill patient information prior to prediction time T into a feature vector that allows us to estimate the lab test result.  Each patient has at least 2 visits, and two HbA1c lab test results.  Note that there is no prediction on the first visit.  

Novelty
This work presents a method called KALP. With respect to the models used to encode patient data, the authors essentially take a sequential prediction approach using transformer-like models as encoders.  For reasons that are not entirely convincing, this work uses separate encoders for diagnoses and medications; the resulting embeddings are added together to get an embedding for each time point for each patient, $\textbf{r}_1, \dots, \textbf{r}_{T-1}$. This is not new ground, and indeed the strongest baseline used in the work (BEHRT) is essentially the same thing (with a couple notable differences discussed shortly).  The paper does, however, add quite a bit of side information that is more interesting: 

1. Patient similarity - a fully connected graph whose vertices are patients is constructed; edges are weighted with a similarity, which is calculated as a cosine similarity between patient data on the first visit.  This graph is encoded using a GAT, and the resulting node representations are used to construct an additional embedding per patient that is added to the time-point specific embedding. The resulting embedding is added to $\textbf{r}_t$.   
2. Diagnosis/Medication-lab test information - this work also uses a GAT to encode a bipartite graph of diagnosis/medication-lab test effects, with edges having a direction (causes increase or causes decrease).  These edges do not denote quantitative information - just directional.  This results in an additional embedding $\textbf{c}$. 
3. Past lab test results - previous lab test results are combined as a weighted sum, with weights derived from inner products of the patient representations $\textbf{r}_t$ for each of the previous time steps.  This results in an additional input $\textbf{f}_{T-1}$.  

The final representation is the concatenation $\[\textbf{r}_{T-1}, \textbf{c}, \textbf{f}_{T-1}\]$.  This is passed through a linear layer, and the entire thing is trained jointly, with  squared error loss. 

This method is applied to two datasets: MIMIC-3, and ICU dataset, and a proprietary outpatient dataset.  Compared to a range of baselines, this method  achieves the lowest error by a pretty large margin in ten-fold cross validation.  The appendix also presents a quite thorough ablation study in which various components of the model are taken out to assess their contribution to performance. The appendix also contains a lot of information about the tuning of the models and baselines; again, this work seems quite thorough.  Kudos to the authors!  

Overall, I liked this work because it presented some (to me) relatively novel (in the EHR-predictions space) methods to a potentially useful task.  The methods are, for the most part, not new but I appreciate their efforts to incorporate side data to improve predictions a lot, as well as the clarity and thoroughness of the presentation. 

However, I find the evaluation lacking in a couple ways.  First, I would note that it's always a good idea to present a 'carry last value forward' baseline for this sort of task - it can be surprisingly hard to beat sometimes.  Second, and more seriously, I would note that the characterization of BEHRT, the strongest baseline, as originally presented (and as described in Related Works) uses only demographics and diagnoses.  BEHRT achieves an MSE in the MIMIC dataset of 1.57, higher than the 1.15 for KALP.  But in the ablation study presented in the appendix, KALP without medication information suffers a sharp degradation of performance - 1.55, which is pretty much the same as BEHRT, which as described does not have access to medication information.  Thus, in my mind, in order to convince the reader that the complexity of KALP actually matters very much, I would want to see a BEHRT baseline with medication information.  I can't see any particular reason this is technically infeasible.  The basic question for this paper is - does all that patient similarity, diagnosis/medication-lab test interactions, and past lab test results stuff really matter over a relatively simple baseline with access to some pretty critical information.  For me, this is the key question - without knowing this, I would not feel comfortable concluding that all that juicy side data is really helping very much!  

One final remark - in the discussion, the authors say, "This allows clinicians to simulate the effect of different treatment regimes before finalizing the best treatment option for the patient".  This is a very strong claim, and effectively says that KALP is getting good causal effect estimates from intrinsically observational data.  I strongly recommend backing off this claim because that is not at all supported by the work presented.  

 This work is does not present any fundamentally new methods, but rather tries to show the value of using largely pre-existing methods to incorporate additional information to the task of predicting HbA1c lab test results.  That is, in my view, a perfectly fine kind of paper for ICLR.  But given the above question about how BEHRT (a baseline) fares with the addition of medication data, I am reluctant to recommend acceptance for this work because that is a key piece of evidence regarding whether the complexity of KALP is really necessary.  I would reconsider if this was added to the paper.  


 We would like to thank all the reviewers for their valuable comments and suggestions. The consolidated list of changes to the paper are mentioned below:
1)	Addition of standard deviation along with the mean values in Table 1 and Table 2. 
2)	Addition of patient characteristics of both the datasets in Table 4 in Appendix A.2 of the paper.
3)	Inclusion of additional classical and recent works in literature survey in Section 2.
4)	Comparative analysis with two additional baselines “Previous Value” and “HiTANet” in Table 1, Table 2, Section 4.
5)	Correction of typos. 

 We would like to thank the reviewer for the thoughtful reviews. We have addressed some of the queries and have summarized as follows:

1)	The authors fail to report confidence bounds around the point estimates. Without these numbers the significance of the results cannot be established.

We have added standard deviation along with the mean values in Table 1 and Table 2. 

2)	To properly evaluate such methods, one needs to make sure there are no data leakage across folds. Thus more information is needed around how the folds were created e.g. were they split based on the data points or unique patients. Were they validated in a temporal cross-validation manner?

The data is split based on unique patients. We did not validate in a temporal cross-validation manner since the splits were based on unique patients and not on temporal data points.

3)	The authors provide some high level information about the datasets. However, a proper Table 1 for the data is missing - this is crucial to understand the data characteristics especially for the PRIVATE data to understand the importance of the results .

We have added patient characteristics in Table 4 in Appendix A2 of the paper.

4)	The paper also seems doesn't analyze why and how the two datasets are useful dataset for this problem. For example, MIMIC-III mainly contains ICU data where data is collected at regular interval and are mostly dense. On the other hand, the authors mention that PRIVATE is an outpatient setting - is the data sparse and irregular in that case? Did the authors analyze this aspect? If so how did they normalize the different data streams that may be arriving at different interval (A setting which DMNC explicitly models).

We validate our proposed model on these two datasets since both display distinctly different characteristics as depicted in Table 3 in Appendix A2. For example, MIMIC-III has a higher number of diagnosis (more co-morbidities) along with higher number prescribed medications per patient as compared to PRIVATE dataset. On the other hand, PRIVATE has a higher number of hospital visits per patient compared to MIMIC-III.

In the current setting, we use positional embedding [1](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) to explicitly model the sequential nature of visits and do not model the time between visits. The time interval between visits can be incorporated by replacing the visit number with the timestamp in the positional embedding calculation. We will investigate incorporating the information regarding the different time gap between visits as our future work.

5)	Presented literature survey on this topic is also limited. For example, there has been both classical as well as more recent work on using intra and inter- patient dependencies for similar class of problems [2](https://www.ijcai.org/proceedings/2021/0486.pdf)

We have updated our literature survey to include both classical and recent works (see Section 2).

6)	Finally, the authors didn't mention whether they intend to publish their code.

Yes, we intend to publish our code on Github.

7)	In the introduction section and subsequently the authors claim that the attention values lends itself for easy interpretability. It is crucial to note that recent literature has debated this practice and the debate about "whether attention is explanation" and how "attention can be adversarially affected" is an active topic. While it is acceptable to use the attention method for this problem, the critique needs to be acknowledged.

We use attention scores to identify the top factors that may have influenced the predicted lab test results. We have not made any claim on interpretability nor explainability. Nonetheless, we have added a statement in Section 3.3 to make the distinction between attention weights and interpretability/explainability. 

8)	The authors may consider describing in more details the effort and scale involved in encoding the domain knowledge beyond what was presented. e.g. was any expert involved in this process? The authors describe HBA1c as their target problem - can these methods scale to other populations?

Yes, experts (clinicians) were involved in the process of data curation, and validating the results. Yes, the proposed method can be easily scaled to other populations with different target lab tests such as creatinine of chronic kidney disease patients, LDL/HDL of patients with high cholesterol.

Do let us know if you have any other queries. Thank you for your time.

 We would like to thank the reviewer for pointing different aspects to improve our work. We have addressed the queries as follows:

1)	Although achieving better results than baseline methods for future lab test prediction, the contribution of this work is negligible in terms of technical perspective. In particular, Transformer and GAT are well-known architecture coming from previous works. The only clear contribution of this work is to utilize external knowledge such as patient similarity (some previous works did that) and drug-lab and disease-lab interactions.

Though Transformer and GAT are well-known architectures, using them to model disease-lab interaction, medication-lab interaction, as well as patient similarity, requires adaption to allow weighted graphs. Further, the use of fine-grained dosage information has been largely ignored in existing work. 

2)	The baseline methods are not competitive. There're many well-known models for EHR prediction such as Dipole, TLSTM, HiTaNet, etc., authors should adapt these models to their problem and compare them with their proposed models beside conventional models such as linear regression, k-nearest neighbor, and recurrent neural network.

We have adapted HiTaNet for the task of lab test response prediction and have included the comparative results in Table 1 and Table 2. We do not compare with TLSTM and Dipole since these methods do not outperform HiTANet [1](https://www.kdd.org/kdd2020/accepted-papers/view/hitanet-hierarchical-time-aware-attention-networks-for-risk-prediction-on-e).


3)	It's trivial that the proposed model achieves better results compared to baseline methods because it uses external information. Authors need to emphasize their claims by comparing their model with the baseline methods using this external information. It's trivial to incorporate this information, for example, using binary vectors to represent the positive and negative impacts of drugs and diseases to lab tests and concatenating these vectors to drug and disease representations.

In the ablation study in Appendix A.4 we show the performance of KALP without the external information (“KALP w/o lab interactions”) outperforms all the other baselines by a decent margin. 

4)	The motivation for using patient similarity is not clear. In particular, the model encodes information from the patient similarity matrix constructed from their demographic information only without using lab test information from these patients. Thus, it is not clear how this information source can help to improve performance.

For this work, patient similarity graph is constructed using demographic, diagnosis, prescribed medications and lab test response of the patient for the first visit. This helps the model to create similar representations for similar patients and make predictions accordingly. This aligns with the clinicians’ observation that patients (with similar demographic information) belonging to chronic and non-chronic categories generally respond to treatments differently whereas patients (with similar demographic information) belonging to the same category (for ex. Chronic category) generally respond to treatments in a similar way. Here, we use the first visit to establish the baseline measurement for patients and to align with how clinicians assess new patients before they personalize the treatments for subsequent visits.

5)	Minor comments: Some notations are used without any introduction (e.g., ni, nj, nk in equation (3); E in equation (5)) 

We have corrected these in the paper. 

6)	EHR record is temporal data by nature. Authors should consider incorporating time information into their model.

We will incorporate the time interval between visits in our future work. 

Do let us know if you have any queries. Thank you for your time.

 We would like to thank the reviewer for the thorough analysis and comments on our work. We have addressed the queries as follows:

1)	First, I would note that it's always a good idea to present a 'carry last value forward' baseline for this sort of task - it can be surprisingly hard to beat sometimes.

This is a good suggestion and we have carried out additional experiments using “Previous Value” as baseline. The results are added in Table 1 and Table 2.

2)	Second, and more seriously, I would note that the characterization of BEHRT, the strongest baseline, as originally presented (and as described in Related Works) uses only demographics and diagnoses. BEHRT achieves an MSE in the MIMIC dataset of 1.57, higher than the 1.15 for KALP. But in the ablation study presented in the appendix, KALP without medication information suffers a sharp degradation of performance - 1.55, which is pretty much the same as BEHRT, which as described does not have access to medication information. Thus, in my mind, in order to convince the reader that the complexity of KALP actually matters very much, I would want to see a BEHRT baseline with medication information. I can't see any particular reason this is technically infeasible. The basic question for this paper is - does all that patient similarity, diagnosis/medication-lab test interactions, and past lab test results stuff really matter over a relatively simple baseline with access to some pretty critical information. For me, this is the key question - without knowing this, I would not feel comfortable concluding that all that juicy side data is really helping very much!

BEHRT originally takes as input demographics and diagnoses, but we have adapted it to provide lab response prediction to use medication information in addition to demographics and diagnosis. In other words, BEHRT performance of 1.57 includes medication information. Even then, the performance of BEHRT is worse than KALP without medication information.


3)	One final remark - in the discussion, the authors say, "This allows clinicians to simulate the effect of different treatment regimes before finalizing the best treatment option for the patient". This is a very strong claim, and effectively says that KALP is getting good causal effect estimates from intrinsically observational data. I strongly recommend backing off this claim because that is not at all supported by the work presented.

We agree and have removed it from the paper.

Do let us know if you have any queries. Thank you for your time and effort.

 We would like to thank the reviewer for highlighting the strong points and the shortcomings of our work. We have addressed the queries as follows:

1)	The novelty of the proposed representation is rather limited. It is a combination of well-known models that have been previously applied in similar context. 

Although GAT is popular choice to obtain node representation from a graph, it has been seen to represent nodes with limited expressivity which is overcome by GATv2 [1](https://arxiv.org/pdf/2105.14491.pdf). This work fills the gap of using GATv2 to model domain knowledge such as disease-lab interaction, medication-lab interaction as well as patient similarity. We also adapt GATv2 to model nodes in weighted patient similarity graph (Equation 3). Naively combining the models might not be able to capture the complex relationships between diagnosis, demographics, medications and lab response results for providing personalized predictions. Also, we specifically introduce personalization by attention based modelling of patients’ previous known lab response result. The combination of the models used is non-trivial in the medical setting shown and leads to good performance boost (as is demonstrated in the paper).

2)	The model capacity/size is not assessed vs. SOTA model sizes. It is unclear whether the choice of the representation or the model size is driving the prediction performance. 

The number of parameters for KALP is 7352069 (\~7.3M) with an RMSE of 1.15 and for BEHRT is 5545883 (\~5.5M) with an RMSE of 1.57 for MIMIC-III. Also, KALP w/o lab interactions (i.e. without external information) has 5627851 (\~5.6M) parameters with an RMSE of 1.35. The number of parameters of  KALP w/o lab interactions is similar to BEHRT but performs better that BEHRT as shown in the table below. Although, modeling external information increases the number of parameters it helps boost the performance of the system. 

| Method                                          |  # of parameters	| RMSE   |
| -------------------------------------------- | ------------------------ | -----------|
| __KALP__	                                |   ~7.3M	                |      1.15 |
| __KALP w/o lab interactions__	|  ~5.6M	                |      1.35 |
| __BEHRT__	                                |   ~5.5M	                |      1.57 |


3)	The choice of hyper-parameters in the model is not well justified: Why the factor for adjusting the fusion positive and negative lab interactions is set as a global hyper-parameters ?

We set the fusion parameter as a global hyper parameter since the negative and positive lab interaction graphs are general domain knowledge.

4)	Patient information includes demographics information and the first visit. This is not well justified. The first visit does not necessarily represent a global/static view of patient state. Clarification on this point would be needed. 

We agree that the first visit does not necessarily represent a global/static view of the patient state, and one can use any number of visit(s) including the most recent visit to create the patient similarity graph. Here, we use the first visit to establish the baseline measurement for patients and to align with how clinicians assess new patients before they personalize the treatments for subsequent visits.


5)	I was expecting that the performance for the inpatient dataset (MIMIC-III) would be much better compared to the outpatient dataset (PRIVATE). Inpatient is a well controlled environment compared to outpatient. For example, mortality prediction in inpatient is highly accurate compared to outptient models. This is not the case in the experimental results of the paper. This needs clarifications.

There are two reasons for the lower performance for MIMIC-III. 

(a)	Patients in MIMIC-III are generally diagnosed with more number of diseases and are prescribed more number of medication on an average as compared to the PRIVATE dataset (see Table 3). This leads to more co-morbidities, more medication combinations leading to more complex patient-lab response relationships in MIMIC-III as compared to PRIVATE dataset. 

(b)	Although the number of unique patients for both datasets for HbA1c cohort are similar, PRIVATE has strikingly more number of visits per patient giving more data points for training, leading to better personalized predictions.  

6)	Last line of page 8: We observe see that -> we observe that . Figure 5, the arrows (up and down) needs to be explained (negative vs. positive interaction) in the figure legend.

Thank you for pointing these out. We have updated the draft based on these suggestions.

Do let us know if you have any queries. We will try our best to clarify them. Thank you for your time.

The paper proposes a representation learning model from EHR data for the prediction of patients' lab test response.  The proposed model, named KALP (Knowledge-Augment Lab test Prediction), consists of two modules: 1) Patient Representation ($\bold{r}_{T-1}$) where T is the number of visits): It is an aggregation of two transformers: The first (transformer encoder) for patient-specific data and the second (the modified graph attention GATv2) for modeling similar patients. The output of the patient representation is passed to 2) Knowledge Augmentation ($\bold{c}$): It consists of two modules: a) Lab Interaction Module: GATv2 is used to extract a representation of drug-lab interactions and disease-lab interactions from two external domain knowledge databases. Separate graphs are defined for positive (increase of lab test value) and negative interactions. b) Past Lab Test Response Module ($\bold{f}_{T-1}$): The patient representation previously obtained ($\bold{r}_{T-1}$) is used to compute an attention representation from the different patient visits. The final representation is obtained as a concatenation of the three components $\bold{r}_{T-1}$, $\bold{c}$ and $\bold{f}_{T-1}$. A last linear layer is used to fit the lab test result as a regression task. Attention weights and the last linear layer are used to define what is called influential factors as to make the model interpretable. The model is evaluated on two EHR datasets (MIMIC-III and PRIVATE). The obtained results in terms of regression metrics (RMSE, MAE and MAPE ) are better (statistically significant) than baselines and SOTA such as BEHRT, DMNC, RNN. Case studies are given to show how the model can be used to interpret the model output. Pros:
- The proposed model KALP incorporates dosage information which usually neglected in the literature.
- The proposed model outperforms SOTA signficantly on two real-world EHR datasets.

Cons: 
- The novelty of the proposed representation is rather limited. It is a combination of well-known models that have been previously applied in similar context. 
- The model capacity/size is not assessed vs. SOTA model sizes. It is unclear whether the choice of the representation or the model size is driving the prediction performance.
- The choice of hyper-parameters in the model is not well justified: Why the factor $\lambda$ for adjusting the fusion positive and negative lab interactions is set as a global hyper-parameters ? While the novelty in terms of individual components of the proposed model rather is limited, the overall method seems to provide good improvement compared to SOTA on real-world datasets. 

Additional comments:
- Patient information includes  demographics information and the first visit. This is not well justified. The first visit does not necessarily represent a global/static view of patient state. Clarification on this point would be needed.
-  PRIVATE dataset has a slightly smaller cohort (6321 from Table 3) contrary to what mentioned in the paper while MIMIC-III has 6412 patients.
I was expecting that the performance for the inpatient dataset (MIMIC-III) would be much better compared to the outpatient dataset (PRIVATE).  Inpatient is a well controlled environment compared to outpatient. For example, mortality prediction in inpatient is highly accurate compared to outptient models. This is not the case in the experimental results of the paper. This needs clarifications
- last line of page 8: We observe see  that -> we observe that
-  Figure 5, the arrows (up and down) needs to be explained (negative vs. positive interaction) in the figure legend.

The paper proposes a novel architecture to predict an individual patients lab response and provide supporting evidence about predicted risk for use in a clinical decision support tool. The authors also claim to model the drug-lab interactions and diagnosis-lab interactions in the form of graphs and used this to create a knowledge augmented individual risk prediction model. The authors experimented on MIMIC-III dataset and another closed real dataset to analyze the efficacy of the solution. They also provided a case study report on the consistency of the top discovered factors with a clinician's understanding.  Some of the stronger aspects of this paper are as follows:

- Predicting individual patients' lab response can be of crucial importance in a clinical decision support system. The authors propose a method that incorporates both external knowledge as well as auto-correction based on patients' past lab response.
- The presented results, in both the main paper and the appendix section, seems to provide some justification about the efficacy of the method via point estimates
- The authors conducted several case studies (distributed over the main paper and the appendix) to uncover some of the important aspects of the patients


The paper can be improved upon by addressing the following aspects
- One of the biggest shortcomings of the paper lies in the experimental settings. The authors fail to report confidence bounds around the point estimates. Without these numbers the significance of the results cannot be established
- Furthermore, the proposed method seems to learn from a sequential setting. To properly evaluate such methods, one needs to make sure there are no data leakage across folds. Thus more information is needed around how the folds were created e.g. were they split based on the data points or unique patients. Were they validated in a temporal cross validation manner? 
- The authors provide some high level information about the datasets. However, a proper ``Table 1`` [1] for the data is missing - this is crucial to understand the data characteristics especially for the PRIVATE data to understand the importance of the results 
- The paper also seems doesn't analyze why and how the two datasets are useful dataset for this problem. For example, MIMIC-III mainly contains ICU data where data is collected at regular interval and are mostly dense. On the other hand, the authors mention that PRIVATE is an outpatient setting - is the data sparse and irregular in that case? Did the authors analyze this aspect? If so how did they normalize the different data streams that may be arriving at different interval (A setting which DMNC explicitly models)
- The presented literature survey on this topic is also limited.  For example there has been both classical as well as more recent work on using intra- and inter- patient dependencies for similar class of problems [2] 
- Finally, the authors didn't mention whether they intend to publish their code

Some other comments

- This is an interesting insight from the paper `we observe that patients with similar age, weight, gender, diagnosis, etc. tend to have similar lab test responses` 
- In the introduction section and subsequently the authors claim that the attention values lends itself for easy interpretability. It is crucial to note that recent literature has debated this practice and the debate about "whether attention is explanation" and how "attention can be adversarially affected" is an active topic. While it is acceptable to use the attention method for this problem, the critique needs to be acknowledged. 
- The authors may consider describing in more details the effort and scale involved in encoding the domain knowledge beyond what was presented. e.g. was any expert involved in this process? The authors describe HBA1c as their target problem - can these methods scale to other populations?
- The ablation analysis should be in main paper



[1]: https://www.sciencedirect.com/science/article/abs/pii/S0895435618309867
[2]: https://www.ijcai.org/proceedings/2021/0486.pdf
 The paper targets an interesting problem and the provided results seem to support the efficacy of the model. However, the paper's experimental section needs to be improved upon to understand the importance of the results. The paper can be improved upon by providing a more thoughtful analysis of why the model works and where can such models be used.

This paper proposes a deep learning architecture for predicting future lab tests, in particular HbA1c based on EHR records. The proposed model leverages (1) attention mechanism (i.e., Transformer) to model sequence of diagnosis and medication, (2) graph attention network (GAT) to capture patient similarity, and drug-lab and disease-lab interactions. The experimental results show that this model outperforms other baselines on the two EHR datasets (i.e., MIMIC-III and private dataset) for lab test prediction. Strengths:
+ The proposed model outperforms other baselines by incorporating external knowledge including drug-lab and disease-lab interactions.
+ The paper shows the case study for interpreting its prediction by analyzing attention weights to know influential factors. 

Weaknesses:
- Although achieving better results than baseline methods for future lab test prediction, the contribution of this work is negligible in terms of technical perspective. In particular, Transformer and GAT are well-known architecture coming from previous works. The only clear contribution of this work is to utilize external knowledge such as patient similarity (some previous works did that) and drug-lab and disease-lab interactions.
- There are some concerns about the experimental settings with respect to the baseline methods. 
-- The baseline methods are not competitive. There're many well-known models for EHR prediction such as Dipole, TLSTM, HiTaNet, etc., authors should adapt these models to their problem and compare them with their proposed models beside conventional models such as linear regression, k-nearest neighbor, and recurrent neural network. 
-- It's trivial that the proposed model achieves better results compared to baseline methods because it uses external information. Authors need to emphasize their claims by comparing their model with the baseline methods using this external information. It's trivial to incorporate this information, for example, using binary vectors to represent the positive and negative impacts of drugs and diseases to lab tests and concatenating these vectors to drug and disease representations.
- The motivation for using patient similarity is not clear. In particular, the model encodes information from the patient similarity matrix constructed from their demographic information only without using lab test information from these patients. Thus, it is not clear how this information source can help to improve performance.
- Minor comments:
-- Some notations are used without any introduction (e.g., ni, nj, nk in equation (3); E in equation (5))
-- EHR record is temporal data by nature. Authors should consider incorporating time information into their model.
 A paper with an interesting idea but with some problems

Meta-review: The paper uses several types of information to predict one specific
lab test response for patients. The predictions are made by combining
and tailoring mainly existing techniques.

The reviewers raised a number of concerns, and the authors clarified
many of them and provided additional results. In particular the
following issues were discussed: Comparing to state-of-the-art methods
and methods having the same information available, specifics of
empirical evaluations and of methodological novelty, choice of the
particular data sets, and justifiability of conclusions.

The main remaining weakness is the limited novelty, which should not
be interpreted as the contributions of the paper being trivial.

In contrast, the solid engineering work done by the authors in this
paper will be valuable in developing clinical decision support tools,
and the authors are encouraged to incorporate the new results and
feedback in future work and submissions.

Decision: Reject