Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel architecture to predict an individual patients lab response and provide supporting evidence about predicted risk for use in a clinical decision support tool. The authors also claim to model the drug-lab interactions and diagnosis-lab interactions in the form of graphs and used this to create a knowledge augmented individual risk prediction model. The authors experimented on MIMIC-III dataset and another closed real dataset to analyze the efficacy of the solution. They also provided a case study report on the consistency of the top discovered factors with a clinician's understanding. 
main_review: Some of the stronger aspects of this paper are as follows:

- Predicting individual patients' lab response can be of crucial importance in a clinical decision support system. The authors propose a method that incorporates both external knowledge as well as auto-correction based on patients' past lab response.
- The presented results, in both the main paper and the appendix section, seems to provide some justification about the efficacy of the method via point estimates
- The authors conducted several case studies (distributed over the main paper and the appendix) to uncover some of the important aspects of the patients


The paper can be improved upon by addressing the following aspects
- One of the biggest shortcomings of the paper lies in the experimental settings. The authors fail to report confidence bounds around the point estimates. Without these numbers the significance of the results cannot be established
- Furthermore, the proposed method seems to learn from a sequential setting. To properly evaluate such methods, one needs to make sure there are no data leakage across folds. Thus more information is needed around how the folds were created e.g. were they split based on the data points or unique patients. Were they validated in a temporal cross validation manner? 
- The authors provide some high level information about the datasets. However, a proper ``Table 1`` [1] for the data is missing - this is crucial to understand the data characteristics especially for the PRIVATE data to understand the importance of the results 
- The paper also seems doesn't analyze why and how the two datasets are useful dataset for this problem. For example, MIMIC-III mainly contains ICU data where data is collected at regular interval and are mostly dense. On the other hand, the authors mention that PRIVATE is an outpatient setting - is the data sparse and irregular in that case? Did the authors analyze this aspect? If so how did they normalize the different data streams that may be arriving at different interval (A setting which DMNC explicitly models)
- The presented literature survey on this topic is also limited.  For example there has been both classical as well as more recent work on using intra- and inter- patient dependencies for similar class of problems [2] 
- Finally, the authors didn't mention whether they intend to publish their code

Some other comments

- This is an interesting insight from the paper `we observe that patients with similar age, weight, gender, diagnosis, etc. tend to have similar lab test responses` 
- In the introduction section and subsequently the authors claim that the attention values lends itself for easy interpretability. It is crucial to note that recent literature has debated this practice and the debate about "whether attention is explanation" and how "attention can be adversarially affected" is an active topic. While it is acceptable to use the attention method for this problem, the critique needs to be acknowledged. 
- The authors may consider describing in more details the effort and scale involved in encoding the domain knowledge beyond what was presented. e.g. was any expert involved in this process? The authors describe HBA1c as their target problem - can these methods scale to other populations?
- The ablation analysis should be in main paper



[1]: https://www.sciencedirect.com/science/article/abs/pii/S0895435618309867
[2]: https://www.ijcai.org/proceedings/2021/0486.pdf

summary_of_the_review: The paper targets an interesting problem and the provided results seem to support the efficacy of the model. However, the paper's experimental section needs to be improved upon to understand the importance of the results. The paper can be improved upon by providing a more thoughtful analysis of why the model works and where can such models be used.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a representation learning model from EHR data for the prediction of patients' lab test response.  The proposed model, named KALP (Knowledge-Augment Lab test Prediction), consists of two modules: 1) Patient Representation ($\bold{r}_{T-1}$) where T is the number of visits): It is an aggregation of two transformers: The first (transformer encoder) for patient-specific data and the second (the modified graph attention GATv2) for modeling similar patients. The output of the patient representation is passed to 2) Knowledge Augmentation ($\bold{c}$): It consists of two modules: a) Lab Interaction Module: GATv2 is used to extract a representation of drug-lab interactions and disease-lab interactions from two external domain knowledge databases. Separate graphs are defined for positive (increase of lab test value) and negative interactions. b) Past Lab Test Response Module ($\bold{f}_{T-1}$): The patient representation previously obtained ($\bold{r}_{T-1}$) is used to compute an attention representation from the different patient visits. The final representation is obtained as a concatenation of the three components $\bold{r}_{T-1}$, $\bold{c}$ and $\bold{f}_{T-1}$. A last linear layer is used to fit the lab test result as a regression task. Attention weights and the last linear layer are used to define what is called influential factors as to make the model interpretable. The model is evaluated on two EHR datasets (MIMIC-III and PRIVATE). The obtained results in terms of regression metrics (RMSE, MAE and MAPE ) are better (statistically significant) than baselines and SOTA such as BEHRT, DMNC, RNN. Case studies are given to show how the model can be used to interpret the model output.
main_review: Pros:
- The proposed model KALP incorporates dosage information which usually neglected in the literature.
- The proposed model outperforms SOTA signficantly on two real-world EHR datasets.

Cons: 
- The novelty of the proposed representation is rather limited. It is a combination of well-known models that have been previously applied in similar context. 
- The model capacity/size is not assessed vs. SOTA model sizes. It is unclear whether the choice of the representation or the model size is driving the prediction performance.
- The choice of hyper-parameters in the model is not well justified: Why the factor $\lambda$ for adjusting the fusion positive and negative lab interactions is set as a global hyper-parameters ?
summary_of_the_review: While the novelty in terms of individual components of the proposed model rather is limited, the overall method seems to provide good improvement compared to SOTA on real-world datasets. 

Additional comments:
- Patient information includes  demographics information and the first visit. This is not well justified. The first visit does not necessarily represent a global/static view of patient state. Clarification on this point would be needed.
-  PRIVATE dataset has a slightly smaller cohort (6321 from Table 3) contrary to what mentioned in the paper while MIMIC-III has 6412 patients.
I was expecting that the performance for the inpatient dataset (MIMIC-III) would be much better compared to the outpatient dataset (PRIVATE).  Inpatient is a well controlled environment compared to outpatient. For example, mortality prediction in inpatient is highly accurate compared to outptient models. This is not the case in the experimental results of the paper. This needs clarifications
- last line of page 8: We observe see  that -> we observe that
-  Figure 5, the arrows (up and down) needs to be explained (negative vs. positive interaction) in the figure legend.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a method for predicting HbA1c lab test results from electronic health record (EHR) data.  The method builds on lots of prior work that uses encoders to summarize longitudinal patient data, and adds some interesting elements.  Most notably, it incorporates various bits of domain specific knowledge about drug-lab test interactions (e.g., drug A tends to increase HbA1c lab measurements), patient similarity, and per-patient historical information about lab test results. The overall system is evaluated in MIMIC-3, comprising ICU stays, and a proprietary outpatient dataset, and seems to perform markedly better than baseline methods as measured by MSE, etc.  The authors present a pretty thorough ablation study, which has some interesting results. 
main_review: Patient data in EHRs is complex - longitudinal, irregularly sampled, and missingness is very often informative.  There have been very many prior works applying the deep neural net architecture of the day, most recently transformer-like encoders, to this sort of data, usually for problems such as predicting mortality at various timeframes.  This work is a bit of a departure from the typical paper in that it focuses on a somewhat unusual target - HbA1c - which is useful as a marker for metabolic disease such as diabetes.  The authors claim that this is useful because such the management of chronic conditions like metabolic disease often involve empirically adjusting doses of medications to achieve target lab test values.  In this review I will leave aside questions of whether this is truly a clinically useful task and focus on the novelty and effectiveness of the methods presented. 

Task formulation
The problem is set up as a regression problem (HbA1c lab test results are numeric - percentages), and formulated in an autoregressive manner - to predict a lab test result at time T, all patient information from time points 0,...,T-1 can be used.  The space in which this paper operates is: how best can do we distill patient information prior to prediction time T into a feature vector that allows us to estimate the lab test result.  Each patient has at least 2 visits, and two HbA1c lab test results.  Note that there is no prediction on the first visit.  

Novelty
This work presents a method called KALP. With respect to the models used to encode patient data, the authors essentially take a sequential prediction approach using transformer-like models as encoders.  For reasons that are not entirely convincing, this work uses separate encoders for diagnoses and medications; the resulting embeddings are added together to get an embedding for each time point for each patient, $\textbf{r}_1, \dots, \textbf{r}_{T-1}$. This is not new ground, and indeed the strongest baseline used in the work (BEHRT) is essentially the same thing (with a couple notable differences discussed shortly).  The paper does, however, add quite a bit of side information that is more interesting: 

1. Patient similarity - a fully connected graph whose vertices are patients is constructed; edges are weighted with a similarity, which is calculated as a cosine similarity between patient data on the first visit.  This graph is encoded using a GAT, and the resulting node representations are used to construct an additional embedding per patient that is added to the time-point specific embedding. The resulting embedding is added to $\textbf{r}_t$.   
2. Diagnosis/Medication-lab test information - this work also uses a GAT to encode a bipartite graph of diagnosis/medication-lab test effects, with edges having a direction (causes increase or causes decrease).  These edges do not denote quantitative information - just directional.  This results in an additional embedding $\textbf{c}$. 
3. Past lab test results - previous lab test results are combined as a weighted sum, with weights derived from inner products of the patient representations $\textbf{r}_t$ for each of the previous time steps.  This results in an additional input $\textbf{f}_{T-1}$.  

The final representation is the concatenation $\[\textbf{r}_{T-1}, \textbf{c}, \textbf{f}_{T-1}\]$.  This is passed through a linear layer, and the entire thing is trained jointly, with  squared error loss. 

This method is applied to two datasets: MIMIC-3, and ICU dataset, and a proprietary outpatient dataset.  Compared to a range of baselines, this method  achieves the lowest error by a pretty large margin in ten-fold cross validation.  The appendix also presents a quite thorough ablation study in which various components of the model are taken out to assess their contribution to performance. The appendix also contains a lot of information about the tuning of the models and baselines; again, this work seems quite thorough.  Kudos to the authors!  

Overall, I liked this work because it presented some (to me) relatively novel (in the EHR-predictions space) methods to a potentially useful task.  The methods are, for the most part, not new but I appreciate their efforts to incorporate side data to improve predictions a lot, as well as the clarity and thoroughness of the presentation. 

However, I find the evaluation lacking in a couple ways.  First, I would note that it's always a good idea to present a 'carry last value forward' baseline for this sort of task - it can be surprisingly hard to beat sometimes.  Second, and more seriously, I would note that the characterization of BEHRT, the strongest baseline, as originally presented (and as described in Related Works) uses only demographics and diagnoses.  BEHRT achieves an MSE in the MIMIC dataset of 1.57, higher than the 1.15 for KALP.  But in the ablation study presented in the appendix, KALP without medication information suffers a sharp degradation of performance - 1.55, which is pretty much the same as BEHRT, which as described does not have access to medication information.  Thus, in my mind, in order to convince the reader that the complexity of KALP actually matters very much, I would want to see a BEHRT baseline with medication information.  I can't see any particular reason this is technically infeasible.  The basic question for this paper is - does all that patient similarity, diagnosis/medication-lab test interactions, and past lab test results stuff really matter over a relatively simple baseline with access to some pretty critical information.  For me, this is the key question - without knowing this, I would not feel comfortable concluding that all that juicy side data is really helping very much!  

One final remark - in the discussion, the authors say, "This allows clinicians to simulate the effect of different treatment regimes before finalizing the best treatment option for the patient".  This is a very strong claim, and effectively says that KALP is getting good causal effect estimates from intrinsically observational data.  I strongly recommend backing off this claim because that is not at all supported by the work presented.  


summary_of_the_review: This work is does not present any fundamentally new methods, but rather tries to show the value of using largely pre-existing methods to incorporate additional information to the task of predicting HbA1c lab test results.  That is, in my view, a perfectly fine kind of paper for ICLR.  But given the above question about how BEHRT (a baseline) fares with the addition of medication data, I am reluctant to recommend acceptance for this work because that is a key piece of evidence regarding whether the complexity of KALP is really necessary.  I would reconsider if this was added to the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a deep learning architecture for predicting future lab tests, in particular HbA1c based on EHR records. The proposed model leverages (1) attention mechanism (i.e., Transformer) to model sequence of diagnosis and medication, (2) graph attention network (GAT) to capture patient similarity, and drug-lab and disease-lab interactions. The experimental results show that this model outperforms other baselines on the two EHR datasets (i.e., MIMIC-III and private dataset) for lab test prediction.
main_review: Strengths:
+ The proposed model outperforms other baselines by incorporating external knowledge including drug-lab and disease-lab interactions.
+ The paper shows the case study for interpreting its prediction by analyzing attention weights to know influential factors. 

Weaknesses:
- Although achieving better results than baseline methods for future lab test prediction, the contribution of this work is negligible in terms of technical perspective. In particular, Transformer and GAT are well-known architecture coming from previous works. The only clear contribution of this work is to utilize external knowledge such as patient similarity (some previous works did that) and drug-lab and disease-lab interactions.
- There are some concerns about the experimental settings with respect to the baseline methods. 
-- The baseline methods are not competitive. There're many well-known models for EHR prediction such as Dipole, TLSTM, HiTaNet, etc., authors should adapt these models to their problem and compare them with their proposed models beside conventional models such as linear regression, k-nearest neighbor, and recurrent neural network. 
-- It's trivial that the proposed model achieves better results compared to baseline methods because it uses external information. Authors need to emphasize their claims by comparing their model with the baseline methods using this external information. It's trivial to incorporate this information, for example, using binary vectors to represent the positive and negative impacts of drugs and diseases to lab tests and concatenating these vectors to drug and disease representations.
- The motivation for using patient similarity is not clear. In particular, the model encodes information from the patient similarity matrix constructed from their demographic information only without using lab test information from these patients. Thus, it is not clear how this information source can help to improve performance.
- Minor comments:
-- Some notations are used without any introduction (e.g., ni, nj, nk in equation (3); E in equation (5))
-- EHR record is temporal data by nature. Authors should consider incorporating time information into their model.

summary_of_the_review: A paper with an interesting idea but with some problems
review_title: null

Meta-review: The paper uses several types of information to predict one specific
lab test response for patients. The predictions are made by combining
and tailoring mainly existing techniques.

The reviewers raised a number of concerns, and the authors clarified
many of them and provided additional results. In particular the
following issues were discussed: Comparing to state-of-the-art methods
and methods having the same information available, specifics of
empirical evaluations and of methodological novelty, choice of the
particular data sets, and justifiability of conclusions.

The main remaining weakness is the limited novelty, which should not
be interpreted as the contributions of the paper being trivial.

In contrast, the solid engineering work done by the authors in this
paper will be valuable in developing clinical decision support tools,
and the authors are encouraged to incorporate the new results and
feedback in future work and submissions.
Decision: Reject