Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a method to learn representation that is invariant to sensitive groups (or domains as claimed by the authors). This is achieved through a network structure that has a transformer to generate time-wise attention, domain-specific feature-wise attention (to extract features that are specific to each domain/subgroup), and finally all these representations are concatenated and passed to a network to generate a universal representation that can be used for downstream tasks.
strength_and_weaknesses: Strength: 
- The problem is very important especially in health care domain. 

Weakness: 
- The paper is not easy to follow 
- I think using the word "domain" might by non-intuitive in this context. It is usually referred to as sensitive or protected groups. 
- The method seems to be adhoc without a principle way to motivate the structure of the network. 
- It is claimed in the paper that "To the best of our knowledge, MTATE is the first model that automatically trains and determines optimal and fair data representations." There are many papers about learning fair representation. The one that needs to be highlighted here is "Creager, Elliot, et al. "Flexibly fair representation learning by disentanglement." International conference on machine learning. PMLR, 2019." The method in that paper learns representation that is invariant to the sensitive groups, which is the main objective of the current paper. 
- Missing important baselines to compare to. 
clarity,_quality,_novelty_and_reproducibility: Not easy to follow. Method novelty is minor and very adhoc. 
summary_of_the_review: Read the strength and weakness section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose an attention-based encoder model to learn data representations that are fair for prespecified subpopulations. The proposed method was tested on real data to learn EHR representations and predict 72-hour mortality after dialysis.
strength_and_weaknesses: The experiment was well conducted. The proposed model has shown advantages in both prediction performance and fairness across prespecified subgroups. 

More explanation and intuitions can be provided for the proposed method, for example, the rationale for introducing each model part and their connection with existing methods. Also, the reason why the model can achieve desired fairness can be better described.


clarity,_quality,_novelty_and_reproducibility: The method was presented clearly with formulas, and the experiments were well conducted. It connects domain-specific and domain-invariant representations to achieve fairness and better model prediction performance. 

However, the code was not provided for reproducibility purposes. Also, are there any tuning parameters for the proposed method? How were they selected and how much do they affect the model results?

summary_of_the_review: The method in the paper is proposed to balance domain-specific and domain-invariant representations with the aim of learning fair and unbiased representation. The experiment has shown its advantage empirically. Overall, I think the proposed method could be useful. But more details can be provided in the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the fairness problem in a healthcare setting involving outcome prediction from electronic health records (EHR). To this end, the authors propose the Masked Triple Attention Transformer Encoder (MTATE), an adaptive multi-task learning model capable of automatically learning and selecting optimal yet fair data representations without explicit domain adaptation or domain-specific bias correction. The experimental results suggest that MTATE learns representations of diverse subpopulations that lead to improvements in predictive performance and fairness on the task of mortality risk prediction for critically ill Acute Kidney Injury requiring Dialysis (AKI-D) patients.
strength_and_weaknesses: Strengths:

* The problem addressed in this paper is of considerable importance as approaches that account for patients’ backgrounds and subpopulations are quite relevant to personalized medicine and reliable prediction of patient outcomes.

* The proposed MTATE model is capable of learning optimal yet fair representations of EHR sequences, which is of great importance to outcome prediction for patients with different demographic characteristics and clinical histories.

* MTATE incorporates time-wise, feature-wise and representation-wise attention mechanisms. This allows for (1) capturing associations among timesteps (which are quite informative in the case of EHR data), associations among features, and (2) summarizing feature-attended representations for downstream patient outcome prediction.

* While it is unclear whether domain-invariant or domain-specific data representations should be used for prediction tasks concerning EHR data, the authors claim that MTATE learns EHR representations that can be domain-specific, domain-invariant, or even a mixture of the two.

* Experiments conducted on a real-world EHR dataset demonstrate that MTATE outperforms several outcome prediction baselines on mortality risk prediction for critically ill AKI-D patients, yielding improvements in both classification metrics as well as fairness metrics, thus mitigating bias towards different subpopulations in the data.

* A few traditional classifiers (XGboost, SVM and RF) have achieved performance comparable to that of MTATE and outperformed some of the compared baseline methods; which suggests that MTATE’s representations can also serve as pre-trained EHR data representations for various classifiers applied to downstream prediction tasks.

* The authors conducted an insightful analysis of the relationship between the patient outcome loss, the domain loss and the representation-wise attention.

-------------------------------------------------------------------------------------------------

Weaknesses:

* The proposed MTATE model appears to resemble a multi-task attention classifier with an output layer that combines the representations learned for all ‘tasks’ (i.e. domains) into a single outcome. Due to this resemblance, I believe that a comparison of MTATE to multi-task classification models should have been carried out. I would encourage the authors to consider such a comparison or provide their reasoning as to why multi-task classification models should not be considered among the baselines.

* In Section 2.3, the authors state the following: “Since features for the different domains are not equally important, we randomly masked out some number of latent features along all time steps differently for each FR-Attention module. In return, the masking procedure forces each sub-module to learn different feature focuses for each domain to generate unique domain representations.” However, no experiment has been conducted to inspect the impact of the random masking on the learned feature representations and thus on the overall patient outcome prediction performance. I am aware of the ablation study (Table 3) conducted to assess the performance with and without masking, nevertheless, such a study does not provide an insight into the effect of different random masking trials. I would suggest that the authors elaborate on this aspect of the random marking in their response.

* One of my main concerns is that the resulting representations learned for each domain can be domain-specific or domain-invariant, depending on the domain classification loss. Thus, target domains that are “easier to classify” will tend to have smaller loss values. On the other hand, other target domains might be difficult-to-classify yet contain more informative patterns, in which case such domains will have higher loss values and may need many iterations for their losses to be substantially minimized. This case does not appear to be accounted for in the model design. I would encourage the authors to clarify the potential behavior of MTATE in the aforementioned scenario.

* In Section 2.6, the authors mention that “the final representation C of EHR is concatenated with all static features, such as demographics and comorbidity, followed by a linear layer with a sigmoid function for the outcome binary prediction”. I assume that the demographics and comorbidity features are one-hot encoded, in which case it would be desirable that the range of the learned representations’ values is around [0, 1]. Is a certain normalization technique applied to the final representations to scale them within the same (or similar) range as that of the static features? If this is not the case, I would like to ask the authors to provide their thoughts on this point in their response.

* The authors have generated 30 samples from each patient’s EHR history with a random start and end times. Nevertheless, I am wondering whether there is a specific reason behind this data preprocessing choice. Why not consider the entire EHR sequence for each patient? Were the sequence samples generated just for data augmentation purposes, i.e. so that representations can be learned from a larger dataset created by generating multiple samples from a single EHR sequence?

* In Section 3.2, the authors mention that all models have been compared on both imbalanced and balanced sets having positive/negative ratios of 1:4 and 1:1, respectively. However, I am not certain if 1:4 can be considered imbalanced (of course, the ratio itself is not a balanced ratio, but the classification problem cannot be considered a typical case of an imbalanced classification problem, or at least not a highly imbalanced classification problem).

* In addition to the considered baselines, I believe that MTATE should have been compared to several recently published state-of-the-art representation learning methods for EHR data including, but not limited to, the methods proposed in:

    Darabi, S., Kachuee, M., Fazeli, S., & Sarrafzadeh, M. (2020). Taper: Time-aware patient ehr representation. IEEE journal of biomedical and health informatics, 24(11), 3268-3275.

    Bang, S. J., Wang, Y., & Yang, Y. (2020). Phased-lstm based predictive model for longitudinal ehr data with missing values.

    Bai, T., Zhang, S., Egleston, B. L., & Vucetic, S. (2018, July). Interpretable representation learning for healthcare via capturing disease progression through time. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 43-51).

-------------------------------------------------------------------------------------------------

Minor weaknesses:
There are also certain grammatical and typographical errors and remarks that require attention. Some of them are summarized as follows:
- First paragraph on page 4: The sentence “Thus, the attention weights computed from the query and key represent how much focus each time step is associated with other time steps.” is rather unclear and should be revised accordingly.
- Page 4: In the last sentence of Section 2.4, “generate” should be replaced with “generating” and a word seems to be missing between “for” and “in”.
- Page 5: “C” should be bolded in the paragraph after Eq. (9).
- Last paragraph on page 6: In the sentence “LSTM is the most competitive method since it has the same highest accuracy”, I assume that the authors meant “same highest ROCAUC” instead of “same highest accuracy”.
- Page 7, last paragraph of Section 4.1: I believe that “Figure 4.1” should be replaced with “Figure 3”. Later in the same paragraph, “Figure A.3.2” should be replaced with “Figure A4”.
- In Section 4.3: replace “negative correlated” with “negatively correlated”.

clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is well written, organized, and technically detailed to a satisfactory extent. The notation is clear and consistent throughout the paper.

Quality:
The design and justifications for the proposed Masked Triple Attention Transformer Encoder are technically sound. The observations made regarding the improvements in patient outcome prediction performance introduced by the proposed MTATE are empirically well supported. Overall, the work seems to be fairly well developed in terms of technical quality.

Novelty:
From a methodological perspective, the contribution of this work can be considered rather incremental. In essence, the authors leveraged several existing attention-based modules and integrated them into a single architecture for the purposes of learning EHR representations that preserve different types of associations within the data. Nevertheless, each of these attention mechanisms are already established in the literature which limits the novelty of this work.

Reproducibility:
The reproducibility of this work is somewhat limited since all experiments have been conducted on, what appears to be, a proprietary dataset which has not been made publicly available by the authors. Even if the authors could not make this dataset available, they should have evaluated MTATE and the baselines on at least one widely-used publicly available EHR dataset (e.g., MIMIC-III [1]). As for the code for the proposed MTATE model, it is also not made available by the authors, however, the architecture of the model is clearly explained in the paper, thus it is safe to say that one should be able to implement MTATE by following its description in Section 2.

[1] Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L. W. H., Feng, M., Ghassemi, M., ... & Mark, R. G. (2016). MIMIC-III, a freely accessible critical care database. Scientific data, 3(1), 1-9.

summary_of_the_review: Overall, this paper proposes an adaptive multi-task learning model capable of learning optimal and fair data representations without explicit domain adaptation or domain-specific bias correction. Addressing fairness in a healthcare setting involving patient outcome prediction is of considerable importance. Nevertheless, the work lacks methodological novelty and is rather incremental in that respect considering that it combines already established attention mechanisms. That being said, I recommend that this work is not considered for acceptance at ICLR due to reasons including, but not limited to, the following:

(1) the degree to which this work is application-oriented rather than a work that makes novel methodological contributions in representation learning;

(2) the absence of a comparison of the proposed model to more recently published models for representation learning from EHR data;

(3) unjustified data preprocessing decisions (outlined in the list of weaknesses);

(4) experiments conducted solely on proprietary data despite the availability of public EHR datasets that also contain records of AKI-D 
patient among other conditions.

Despite the above reasons, I am still looking forward to the authors’ response and I would be willing to adjust my score in case I have misunderstood or misinterpreted certain aspects of the work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a transformer-encoder framework for multipleThe authors present a encoder/transformer method for learning representations of EHR data,  for multiple prediction tasks. They demonstrate that their method out-performs other DL methods on one prediction task using MIMIC-III data.

strength_and_weaknesses: 
Strengths:
- This paper is well written
- it focuses on an important problem (they also motivate the problem very well)
- the methods seem sound, though I am not an expert
- the experiments use relevant data, and the results demonstrate that their method works well
- their experiments use public data

Weaknesses
- There has been a lot of work on this topic, and the authors don't cite or compare with existing methods. Here is a short list of papers that look relevant to this topic, but were not mentioned or compared with in the paper:
-- https://doi.org/10.1109/ICDM50108.2020.00050
-- https://doi.org/10.1145/3394486.3403129
-- https://doi.org/10.1109/ICDM51629.2021.00060
-- here is a relevant review: https://doi.org/10.1016/j.jbi.2020.103671
- there is no comparison against state-of-the-art risk prediction models in medicine, or any non-DL models at all. there are several such models used in medicine (which don't use DL), which would be good baselines to compare against here. 
- the authors don't provide any code


clarity,_quality,_novelty_and_reproducibility: 
Clarity
- this paper is very clear

Quality
- the methods and experiments all seem sound

Novelty
- I think there are issues with the novelty of this work (see "weaknesses" above)

Reproducibility
- The authors don't mention that they will share code, but their explanations of their methods are fairly clear, so I think someone could independently reproduce their method.


summary_of_the_review: This is a well-written paper on an interesting and important topic. The experiments are somewhat compelling andt he methods appear to be sound. The main issue is that the authors don't disucss or compare with other methods from the literature. There has been a lot of work in the encoder/transformer space for EHR data, and the authors don't engage with this literature. Since they are presenting their method as a general approach for EHR data (and not a method specific to AKI), they should make sure to engage & compare with exisitng work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studied how to learn unbiased and fair representations of different subpopulations from their electrical health records (EHR). The referees all agree the paper addressed an important and interesting problem. However, they also raised a series of concerns, including insufficient comparisons with the methods in the literature, and insufficient explanation or intuitions about the procedure. There is still insufficient enthusiasm from the referees to accept this paper after rebuttals.
justification_for_why_not_higher_score: There is still insufficient enthusiasm from the referees to accept this paper after rebuttals.
Decision: Reject