Interpretable Pre-Trained Transformers for Heart Time-Series Data

Authors: Harry J Davies, James Monsen, Danilo Mandic
Primary area: interpretability and explainable AI
Keywords: biosignals, interpretability, healthcare, transformers

Abstract:
Interpretability of artificial intelligence models is vital in healthcare, as a poorly informed decision can directly impact the health and well-being of patients. This means that, owing to their black box nature, deep-learning solutions that may even yield high accuracy often fail to be adopted in real-world healthcare settings. To this end, we employ the generative pre-trained transformer (GPT) framework to clinical heart time-series data, to create two pre-trained general purpose cardiac models, termed PPG-PT and ECG-PT. We place a special emphasis on making both such pre-trained models fully interpretable. This is achieved firstly through aggregate attention maps which show that, in order to make predictions, the model focuses on similar points in previous cardiac cycles and gradually broadens its attention in deeper layers. Next, we show that tokens with the same value, which occur at different distinct points in the electrocardiography (ECG) and photoplethysmography (PPG) cycle, form separate clusters in a high dimensional space. Such clusters are formed according to the phase of the cardiac cycle, as the tokens propagate through the transformer blocks. Finally, we highlight that individual attention heads correspond to specific physiologically relevant features, such as the dicrotic notch in PPG and the P-wave in ECG. Importantly, it is also demonstrated that these pre-trained models are straightforward to fine-tune for tasks such as the classification of atrial fibrillation (AF), and beat detection in photoplethysmography. The so introduced PPG-PT and ECG-PT models achieve accuracy comparable to the state-of-the-art for both tasks, whilst crucially retaining their interpretability and explainability. This is demonstrated in the AF-screening fine-tuned model, with attention clearly shifting to regions in the context that are strongly indicative of atrial fibrillation.

Decision: reject
Reviewer scores: 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730031446101

thread_type:
official_review

<thread_object>
creator: reviewer_fhb9

<content>
summary:
The paper aims to use generative pre-trained transformer (GPT) framework to predict next token for ECG and PPG signals with a focus in interpreting the decision-making of these large language models. This was done through observing the behavior of the attention heads in different transformer layers and aggregating them. The experiments reveal that the next token generation looks cycles in the nearest past more than that of the farthest past in the context. Qualitative interpretation reveals the ECG and PPG GPT models look into important signal components. These GPT models were fine tuned for a downstream task of atrial fibrillation classification with interpretation highlighting irregular occurrence of a beat w.r.t. the previous beat which is interesting. Overall, the problem formulation is logical, methods are well designed and explained, and the discussion clearly articulates the explainability the GPT models can offer.

strengths:
- Related GPT based interpretation studies for ECG and PPG exists in literature, but the experimentation carried out in this study is rigour and systematic in revealing the explanations for firstly, the generation of next token and finally, extend the idea for a downstream classification task.

- The idea of decoding attention head for explanations are common in the CV and NLP domain, but extending it to physiological time-series can be seen a contribution.

- Data split follows a subject-wise separation to avoid data leak in training and testing.

weaknesses:
Major:
- The explainability method of GPT models was shown to focus on previous cycles meaning that it can observe a beat w.r.t the previous one which is why the downstream task interprets well where distance between two consecutive beat is important criteria. However, this might not be the case for other common tasks such as sleep staging where the input in 30 second ECG or PPG signal and separating sleep stages such as wake, light sleep, deep sleep and REM sleep manifests from HR variability. Another experimentation for a downstream task would be interesting to see if the proposed explainability idea fits well to provide clinical domain specific interpretation, given the fact that GPT models were found to be useful in these downstream tasks.

- Few related GPT interpretability studies were referenced. More references should be included to have a broad picture based on GPT interpretablity. For example, 
a. Creswell, A., Shanahan, M., & Higgins, I. (2022). Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712.
b. Ben Melech Stan, G., Aflalo, E., Rohekar, R. Y., Bhiwandiwalla, A., Tseng, S. Y., Olson, M. L., ... & Lal, V. (2024). LVLM-Intrepret: an interpretability tool for large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8182-8187).

Minor:
- Figure 3 captions needs to be corrected a-d.

questions:
- 10 second segments were extracted from recordings from where tokens were generated, are these tokens represent whole segment data or beats were isolated to form tokens?

- The prediction or generation of tokens from GPT models is unclear. Experiment says a single token is generated for a given context length, then it needs to be clearly mentioned how the generation process continued to generate to produce, say Figure 1, of 15 second PPG or ECG data which was compared against the original signal?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730561765031

thread_type:
official_review

<thread_object>
creator: reviewer_85mv

<content>
summary:
This manuscript presents a novel approach to representation learning for heart time-series data (typically PPG and ECG in this work) using generative pre-trained transformers (GPTs). The idea comes from the NLP domain where GPTs have shown remarkable capabilities in learning representations from text data, and the fact that the (pre-)training process for GPTs is very simple, only next-token prediction, also motivated this work, as claimed by the authors. The authors conducted a very detailed attention mechanism analysis to interpret the learned representations of the pre-trained GPTs.

strengths:
1. To the best of my knowledge, this is the first work that applies generative pre-training to representation learning for heart time-series data, where previously contrastive learning methods were the most popular (also inherited from other domains like voice recognition). This is a novel and interesting idea.

2. The attention mechanism analysis is very detailed and provides a clear understanding of the learned representations, and how the transformer layers work on the heart time-series data.

weaknesses:
1. The datasets used in the experiments are not comprehensive enough. The authors used the CinC2020 dataset, which is superseded by the CinC2021 dataset. The latter dataset is more comprehensive and contains more data. Moreover, there are other larger datasets available, such as the CODE-15% dataset (https://zenodo.org/records/4916206), etc.

2. The authors did not compare their method in their numerical experiments with other representation learning methods, such as contrastive learning-based methods (for example CLOCS (https://proceedings.mlr.press/v139/kiyasseh21a/kiyasseh21a.pdf)), to show the effectiveness of their method.

3. Not enough downstream tasks are conducted to evaluate the learned representations (or the pre-trained GPTs).

questions:
See the "Weaknesses" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686125469

thread_type:
official_review

<thread_object>
creator: reviewer_wrus

<content>
summary:
This paper introduces two interpretable pre-trained transformer models (PPG-PT and ECG-PT) adapted from the GPT framework for analyzing heart time-series data from PPG and ECG signals.  The authors demonstrate that these models develop interpretable attention patterns corresponding to physiologically meaningful cardiac cycle features.

strengths:
- Interpretability analysis on physiological time series
- The model was tested on two types of physiological time series: PPG and single-lead ECG.

weaknesses:
1. The model's novelty is questionable, as the PPG-GPT has previously been explored across multiple tasks (Chen et al., 2024). It will be helpful to specify how the current work differs from it.
2. The manuscript does not compare models on different PPG or ECG tasks compared to previous PPG-GPT (Chen et al., 2024). The comparison with one baseline per PPG and ECG is mentioned only in the appendix. Importantly, the models are compared on performance using different setups.
3. The experiments have not been performed across multiple runs, so the variability (std or IQR) of the performance is not clear. Consider 5-fold or 5 seeds.
4. Consider additional metrics, for example, sensitivity, specificity, false-positive rate, false-negative rate, and F1 score for AF.
5. The majority of the paper focuses on interpretability. However, it is difficult for the machine-learning community to validate clinical utility claims that require experience in reading ECG or PPG. Have you collaborated with clinical experts to validate the interpretability claims? Could you provide more evidence that the examples you have shown guarantee the presence of AF?

- Chen, Zhaoliang, et al. "Adapting a Generative Pretrained Transformer Achieves SOTA Performance in Assessing Diverse Physiological Functions Using Only Photoplethysmography Signals: A GPT-PPG Approach." AAAI 2024 Spring Symposium on Clinical Foundation Models. 2024.

questions:
1: You have considered very clean datasets and samples; seeing the behavior across different noisy scenarios would be highly important. Since we could achieve very high performance on high-quality data with an F1 score of 0.96 with DeepBeat (Torres-Soto et al., 2020).

2: Attention maps as an interpretability tool have been successfully explored (Zhao et al., 2023). It needs to be clarified that you did something new methodologically; it seems more of an application. For example, you have details on using the `findpeaks` function in Matlab, but it is poorly structured overall. It would be great if you would summarize the methodology as a pseudocode and suggest choices.

- Zhao, Haiyan, et al. "Explainability for Large Language Models: A Survey." arXiv preprint arXiv:2309.01029 (2023).
- Torres-Soto, Jessica, and Euan A. Ashley. "Multi-task deep learning for cardiac rhythm detection in wearable devices." NPJ digital medicine 3.1 (2020): 116.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732571691292

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of our revisions and response to reviewers:

comment:
We would like to thank all the reviewers for their insightful feedback and supportiveness of our work. The motivation for this manuscript is a lack of clearly interpretable AI in the domain of physiological time-series analysis, and to this end we are grateful that the reviewers all agreed that the core strength of our work is the clearly articulated interpretability of our network, with `[R-Wrus]` *“Strengths: interpretability analysis on physiological time series”*, `[R-85Mv]` *“The attention mechanism analysis is very detailed and provides a clear understanding of the learned representations”* and `[R-FHB9]` *“the experimentation carried out in this study is rigorous and systematic”*. We have addressed all suggestions and made the following changes, which we believe have improved the quality of our manuscript:

-  **Addition of more downstream tasks:** We have added more downstream tasks in the form of 4-class arrythmia detection from single lead ECG, and premature ventricular beat detection in ECG. This is in addition to the existing tasks of AF detection in ECG and PPG and beat detection in PPG. Through these downstream tasks, we have also integrated more recent datasets into our models, such as the Chapman dataset from the CinC 2021 challenge.

-  **Extended the analysis of interpretability for downstream tasks:** We have added more interpretability-related analysis in the classification of premature ventricular beats. Our fine-tuned model clearly shifts attention to premature ventricular beats when they occur (Appendix A.10, Figure 10). This task, like AF, was explicitly chosen given that the arrythmia is straightforward to see visually for clinicians and non-clinicians alike, meaning that we can be certain of the interpretability.

-  **Clarifying the contribution of our work against previous works:** We have extended our background literature review to include comparisons with GPT-PPG (Chen 2024), CLOCS (Kiyasseh 2020) and DeepBeat (Torres-Soto 2020). Importantly, none of the aforementioned works contain detailed interpretability analysis, which is the core contribution of our work and is of the utmost importance when applying AI tools in clinical settings. 

-  **Improved robustness of the fine-tuning analysis:** We repeated all fine-tuning cross-validations with 5 different random seeds and extended the number of reported metrics whilst including a mean and standard deviation for each one.

- **Improved discussion about prior works on interpretability through attention:** We have added further discussion in our previous works section to include more references to interpretability in LLMs, such as Creswell 2022 and Zhao 2023, as well as prior literature on interpretability in large vision models (Ben Melech Stan 2024). 

- **Highlighting our contribution to attention analysis methods:** We have added explanation for how we averaged attention weights across time to determine that different attention heads correspond to different features in ECG and PPG. This explanation is given in Figure 12, provided in Appendix A.14. We have also further detailed how our attention analysis in downstream tasks involves the shift in attention from pre-trained next token prediction to the fine-tuned task. In LLMs an analogous scenario would be to switch from next token prediction to sentiment analysis, and then to analyse the corresponding shift in attention.

- **Including detailed analysis on the effects of noise:** We have analysed the effects of signal quality on classification performance in AF (PPG), by using the average beat confidence from our independently fine-tuned beat detector as a proxy for signal quality. By excluding the bottom 15% of signal qualities, we have demonstrated that the false positive rate was halved. We have included visualisations of the distributions of signal quality against incorrect and correct classification, as well as an example workflow showing that signal quality estimates could be used in combination with classification probabilities to reduce false positives for clinicians. (Appendix A.12, Figure 11).

- **Addition of a future work section:** In this section we describe future avenues for research, such as scaling the model up for longer context lengths, to handle tasks such as automatic sleep staging involving heart rate variability. We also discuss the need to scale up the model to work with different morphologies of PPG, such as in-ear PPG, and lower quality ECG such as the wearable ear-ECG (Appendix A.13).
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: Although this paper received positive reviews and one borderline review but I do not believe it to be above the bar for acceptance at ICLR. This paper builds two transformer based models, one for ECGs and another for PPGs (signals used for non-invasive). These are trained via next token prediction. The work goes on to interpret what the model has learned by looking at the attention weights onto the space of inputs. They find that peak predictions are most sensitive to prior peaks, the cosine similarities of patterns such as rising and falling slopes are shared across time and that for each modality, various attention heads latch on to different parts of the signal and studying how the model's insights change when fine tuning to predict atrial fibrilation. Overall, this reads as a well written report that builds two transformers and does a study on how to interpret the model's predictions on two data modalities relevant for healthcare. My concern is not with the choice of topic, which I do think is interesting, but rather with the claims of the paper, the organization, the lack of engagement with other work in the literature on interpretability and finally the lack of clarity on what a practitioner in ML and healthcare should do with these results.

Limitations: First, it is unclear what the core technical contributions from the perspective of learning. If there are no new technical contributions, this should be stated up-front and the manuscript should instead highlight the exploratory nature of the work on a problem of interest. Given the simple methods (training transformers, using simple methods for interpreting attention weights) used here, if the contribution of this work is that transformers capture clinically relevant signals, then this should be verified via a panel of clinicians to demonstrate that using this form of interpretability improves trust and/or utility. Second, there is a large body of in deep learning at the intersection of ML and healthcare for cardiology that this work does not do much to engage with in terms of how similar the models are and this, in turn, hurts the generalizability of the findings. Finally, there is a large literature around the interpretability of transformer based models the relationship to which has not been well articulated or compared to (e.g. https://arxiv.org/abs/2005.00928, https://arxiv.org/abs/1908.07442, https://arxiv.org/abs/2406.00426,https://arxiv.org/abs/2212.14776). Is this the only method for interpretability of such models, a good one, or even the best one? I think its important to contend with these topics for this work to have value for a learning focused venue like ICLR.