# Fusion of Dwi Image and Clinical Variables for Stroke Outcome Prediction Using Tabular Transformer

Mingtian Liu, Nima Hatami, Laura Mechtouff, Tae-Hee Cho, Carole Lartizien, Carole Frindel

► To cite this version:

Mingtian Liu, Nima Hatami, Laura Mechtouff, Tae-Hee Cho, Carole Lartizien, et al. Fusion of Dwi Image and Clinical Variables for Stroke Outcome Prediction Using Tabular Transformer. 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI), Apr 2025, Houston, France. pp.1-5, 10.1109/ISBI60581.2025.10981076. hal-05303799

HAL Id: hal-05303799

https://hal.science/hal-05303799v1

Submitted on 8 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
FUSION OF DWI IMAGE AND CLINICAL VARIABLES FOR STROKE OUTCOME
PREDICTION USING TABULAR TRANSFORMER

Mingtian Liu¹, Nima Hatami¹, Laura Mechtouff²,³, Tae-Hee Cho²,³,
Carole Lartizien¹, Carole Frindel¹,⁴

¹CREATIS, CNRS UMR5220, INSERM U1294, Université Lyon 1, INSA-Lyon, France

²Stroke Department, Hospices Civils de Lyon, France

³CarMeN, INSERM U1060, INRA U1397, Université Lyon 1, INSA-Lyon, France

⁴Institut universitaire de France (IUF)

ABSTRACT

Ischemic stroke is a major cause of global disability, making
outcome prediction crucial for effective management. In this
study, we propose an approach using a tabular transformer to
fuse diffusion weighted MRI and clinical variables for stroke
outcome prediction. Radiomics are employed to extract quan-
titative features from regions of interest drawn on early diffu-
sion weighted MRI acquired at acute stage in intensive care
unit. The 3-month modified Rankin Scale (mRS) score serves
as the ground truth for forecasting patient outcomes. Per-
formance of the proposed model was evaluated on a private
database encompassing 119 exams of stroke patients treated
with thrombectomy (66% of which had a favorable prognos-
sis). The proposed fusion approach achieves strong predic-
tive performance, with an AUC of 0.75 and an accuracy of
0.72. Furthermore, analyzing feature importance contributed
to a deeper understanding of clinical relevance.

***Index Terms***— Multimodal fusion, Radiomics, Tabular transformer, Stroke, Clinical outcome prediction

# 1. INTRODUCTION

Stroke is a leading cause of long-term disability, with is-
chemic stroke accounting for nearly 87% of all cases [1].
While the advent of reperfusion therapies has significantly
improved ischemic stroke management [2], patient outcomes
remain highly variable. This underscores the critical need
for enhanced predictive models to better anticipate recovery
trajectories. In this study, we focus on the modified Rankin
Scale (mRS), a widely recognized clinical measure of stroke
outcomes, with particular attention to the 3-month mRS.
This metric serves as a key benchmark for evaluating both
recovery and the effectiveness of interventions [3].

Traditional machine learning models have shown utility
to predict stroke outcome based on clinical data [4]. But they
often struggle with the complexity of multi-modal data, due
to their limited ability to model non-linear interactions across

diverse data types. Recent work has focused on deep learn-
ing (DL) approaches, especially CNNs and vision transform-
ers using MRI images, which offer improved accuracy but
frequently lack sufficient interpretability [5, 6]. Nima et al.
[7, 8] combined CNNs with LSTM to improve stroke progno-
sis by incorporating sequential data and weighting the output
based on a single clinical variable. Although promising, this
model's limited integration of clinical data highlights an op-
portunity to more fully leverage patient information.

Building on the success of DL in processing images
and text, there is growing interest in extending these meth-
ods to tabular data. However, tabular data presents unique
challenges, including heterogeneous feature types and high
cardinality in categorical features [9]. The Feature Tok-
enizer Transformer (FT-Transformer) addresses these issues
by leveraging self-attention mechanisms to effectively handle
diverse feature types in tabular data [9](Figure 2). Another
significant challenge in tabular learning is achieving general-
ization across datasets with varying column structures. XTab
[10] addresses this by pretraining on the OpenML-AutoML
benchmark [11], a diverse collection of tabular datasets. This
pretraining enables the model to learn transferable features
using the FT-Transformer as its backbone. The use of XTab
has been shown to substantially improve performance, par-
ticularly in tasks involving the fusion of multimodal data
[12].

In this paper, we aim to improve stroke outcome predic-
tion by leveraging radiomics to quantify imaging features [13]
and integrating clinical data through a transformer-based ar-
chitecture [10]. Diffusion Weighted Imaging (DWI), a crit-
ical MRI modality for early stroke detection, is key to our
approach. To effectively capture features that characterize le-
sion patterns in DWI MRI, we address the challenge of barely
visible lesions at the acute stage by refining the lesion mask's
shape and size. This approach combining radiomics and clin-
ical features, improves prediction accuracy while maintaining
interpretability, ultimately supporting more informed clinical
decision-making.
## 2. METHOD

Our method involves two main stages: (1) extracting image-based features from DWI MRI using radiomics and (2) fusing these features with clinical data in a tabular transformer-based model to predict the mRS, a 7-point disability scale binarized in this study, where 0–2 indicates a favorable prognosis.

### 2.1. Radiomics feature extraction

As shown in Figure 1, our method starts by defining the region of interest (ROI), centered on lesion masks in DWI MRI acquired at the acute stage. In some cases, early lesions appear faint and small, resulting in minimal ROIs that make radiomic feature extraction more challenging due to noise, instability, or missing information, potentially affecting model performance. To address this, we applied morphological dilation based on lesion size. For lesions smaller than a threshold $T$ (measured in voxel count), we performed 6 iterations of dilation to expand the ROI, using a 3 × 3 structuring element. Each iteration expanded the lesion by one pixel in all directions. We experimented with various thresholds $T \in \{200, 500, 800, 1000\}$ and number of iterations $\in \{4, 5, 6, 7\}$. The optimal configuration, determined on the validation set, was a threshold of $T = 500$ with 6 iterations. Approximately 10% of the lesions required dilation processing, while the rest remained unchanged.

Once the ROI was defined, we used Python library PyRadiomics [14] to extract a variety of descriptors:

1. **Shape descriptors:** Measure geometric properties of ROI, such as volume and surface area.

2. **First-order intensity descriptors:** Capture fundamental statistical measures of the intensity values within ROI.

3. **Texture descriptors:** Describe the spatial distribution and relationships of pixel intensities, providing insights into the structural arrangement of tissues.

### 2.2. Fusion approach through tabular transformer

Tabular transformers have proven effective at integrating different feature types (numerical and categorical) within a single model. Categorical features are transformed into tokens through an embedding layer, while numerical features are tokenized using a linear layer. This distinction is essential because categorical features have discrete values, whereas numerical features are continuous and can be processed directly after a simple transformation that maintains their quantitative relationships. Figure 2 illustrates how each feature type is converted into tokens and subsequently processed. To develop task-specific tokenizers, we initialize the model with XTab pretrained weights (Figure 1). These tokenizers are trained to transform clinical variables and radiomics descriptors into embeddings that the transformer can effectively utilize. The development process for the tokenizers involves iteratively refining these transformations based on validation

performance. By leveraging self-attention mechanisms, the model can focus on the most relevant features, providing more accurate predictions.

## 3. EXPERIMENTS

### 3.1. Dataset and preprocessing

**Clinical data:** The cohort consists of 119 ischemic stroke patients enrolled in an ongoing single-center observational study [15]. Inclusion criteria were as follows: (1) patients with an anterior circulation stroke due to a proximal intracranial occlusion; (2) baseline imaging that included diffusion and perfusion MRI; and (3) patients who underwent thrombectomy. Clinical data were extracted from hospital records, comprising 63 variables chosen by clinicians for their relevance to patient outcomes. This dataset includes demographic information, neurological assessments using NIHSS scores [16], medical history, and treatment details. It also encompasses numerical clinical indicators, including blood tests results and radiological scores, as well as time-related data from admission to various treatment stages. All clinical variables were included in the analysis. Different strategies were used to handle missing data. Missing values in basic patient information, like weight, were imputed with the mean of other patients. For binary categorical variables, the most frequent value was used. Missing treatment-related data were assigned to "missing" category. Categorical variables were then converted to numerical form, and the data were normalized to ensure feature comparability during model training.

**Image data:** All patients underwent a standardized protocol (IRB number:00009118) that included MR DWI acquisition at the acute stage, followed by follow-up scans 3 months after the stroke event, which included FLAIR MRI. The 3D volumes were resliced to dimensions of 192×192 pixels with 27 slices along the z-axis. Skull stripping was performed using FSL [17], and image intensities were normalized for consistency across patients. Lesions were manually annotated on the early scan by expert neuroradiologists. As described in section 2.1, 130 radiomic features characterizing the lesion pattern were extracted from various masks : 1) a *whole-brain mask* covering the entire brain region; 2) a *dilated mask* described in section 2.1; and 3) a *FLAIR late lesion mask* manually drawn on the follow-up FLAIR MRI scan, which serves as the ground truth for final lesion identification.

### 3.2. Implementation

For testing radiomics features alone, we used all extracted features directly. When integrating with clinical data, we applied the LASSO algorithm [18] to select radiomics features, reducing the dimensionality to approximately 20 columns. Despite the ability of DL models to automatically extract features, having too many features with a limited dataset can lead
<figure><img src="image_2.png" /><figcaption>Fig. 1. Fusion approach proposed for stroke outcome prediction.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2. Illustration of feature tokenization in FT-Transformer for numerical and categorical data and Xtab weights we used in experiments.</figcaption></figure>

to learn noise and result in overfitting. For the tabular transformer, we utilized pretrained weights provided by XTab and adapted their AutoGluon-based code [19]. Key parameters were set as follows: `batch_size` = 32, `max_epochs` = 1000, and `seed` = 42 while all other configurations remained consistent with the original paper [10]. We used 10-fold cross-validation, where each fold included training and test sets, with a validation subset separated from the training set. Probability outputs for each test fold were recorded and combined to generate overall predictions. The model's final output is a probability score, and a threshold of 0.36 (determined on the validation set) is used for binary mRS prediction. We assess performance of our proposed model, evaluate the impact of the masks used to extract radiomic features, conduct an ablation study to determine the added value of the fused image and clinical data, and finally compare the results with various reference models.

## 4. RESULTS AND ANALYSIS

<table><caption>Table 1. AUC(Area Under the ROC Curve) Scores for Various Masks Utilized in Radiomic Feature Extraction</caption><thead><tr><td>Input masks</td><td>AUC</td></tr></thead><tbody><tr><td>Original DWI mask</td><td>0.53</td></tr><tr><td>Whole brain mask</td><td>0.57</td></tr><tr><td>Dilated lesion DWI mask</td><td>0.64</td></tr><tr><td>Late FLAIR lesion mask</td><td><strong>0.70</strong></td></tr></tbody></table>

Table 1 presents the changes in AUC with different masks. The original DWI mask underperformed due to the inclusion of nearly invisible lesions. Similarly, whole brain masks yielded poor results as they aggregate all information, leading to the incorporation of irrelevant regions and noise. Although our proposed dilation method effectively addressed
<table><caption>**Table 2.** Performance (mean ± standard deviation over 10 different random seeds) of the fusion of radiomics and clinical data compared to various baselines using different types of input data on the same cohort. The MRI modalities considered include DWI and other MRI sequences.</caption><thead><tr><th>Input Data</th><th>Model</th><th>AUC</th><th>Sens.</th><th>Spec.</th><th>F1 score</th><th>MAE</th><th>Acc.</th></tr></thead><tbody><tr><td>MRI modalities</td><td>3D-CNN</td><td>0.66±0.02</td><td>0.64±0.02</td><td>0.64±0.02</td><td>0.60±0.03</td><td>0.43±0.00</td><td>0.65±0.02</td></tr><tr><td>MRI modalities + Age/NHISS</td><td>CNN-LSTM[7]</td><td>0.68±0.05</td><td>0.66±0.03</td><td>0.65±0.03</td><td>0.66±0.03</td><td>0.38±0.01</td><td>0.69±0.02</td></tr><tr><td>MRI modalities</td><td>AE<sup>2</sup>-LSTM[8]</td><td>0.71±0.03</td><td>0.67±0.03</td><td>0.67±0.03</td><td><strong>0.68</strong>±0.03</td><td>0.34±0.01</td><td><strong>0.72</strong>±0.03</td></tr><tr><td>DWI modality + Clinical</td><td>Ours</td><td><strong>0.75</strong>±0.02</td><td><strong>0.68</strong>±0.05</td><td><strong>0.73</strong>±0.05</td><td>0.63±0.02</td><td><strong>0.30</strong>±0.02</td><td><strong>0.72</strong>±0.02</td></tr></tbody></table>

the challenges posed by nearly invisible lesions, its performance remained lower compared to the FLAIR lesion masks, which are derived from follow-up scans. FLAIR MRI is used as ground truth in stroke studies to identify and delineate ischemic lesions, particularly in the subacute phase, by detecting hyperintense signals indicative of tissue injury or edema. Since FLAIR lesion masks are not available in advance, we consistently applied our dilation method in subsequent experiments. This approach strokes a balance between capturing relevant information and minimizing noise, resulting in a significant improvement over the original masks.

<table><caption>**Table 3.** Comparison of tabular Transformer performance using different inputs, with radiomics features extracted using our proposed dilation mask. "Radiomics + Clinical" refers to the fusion of radiomics and processed clinical data.</caption><thead><tr><th>Input</th><th>AUC</th><th>Sens.</th><th>Spec.</th><th>F1</th><th>MAE</th><th>Acc.</th></tr></thead><tbody><tr><td>Radiomics</td><td>0.64</td><td>0.52</td><td><strong>0.73</strong></td><td>0.5</td><td>0.35</td><td>0.65</td></tr><tr><td>Raw Clinical</td><td>0.67</td><td>0.69</td><td>0.62</td><td>0.58</td><td>0.35</td><td>0.65</td></tr><tr><td>Processed Clinical</td><td>0.72</td><td>0.60</td><td>0.71</td><td>0.56</td><td>0.33</td><td>0.67</td></tr><tr><td>Radiomics + Clinical</td><td><strong>0.74</strong></td><td><strong>0.71</strong></td><td>0.70</td><td><strong>0.63</strong></td><td><strong>0.29</strong></td><td><strong>0.71</strong></td></tr></tbody></table>

Table 3 highlights the benefits of combining radiomics and clinical data for improved performance, with Table 2 comparing results across models using comprehensive metrics such as AUC-ROC, specificity, sensitivity, F1 score, mean absolute error (MAE), and accuracy.

To identify the most significant variables for predicting stroke outcomes, we assess feature importance by measuring the performance drop when the values of a feature are randomly shuffled. Our analysis revealed that texture features associated with GLCM (gray-level co-occurrence matrix) are the most critical among the radiomic features. The key clinical features identified include hypertension (HTA), the National Institutes of Health Stroke Scale (NIHSS), age, and precedents of stroke (ATCD). The importance of these clinical features aligns with established clinical knowledge, enhancing the model's reliability for clinical applications. Specifically, HTA and age are recognized as significant risks factors impacting outcomes, while NIHSS serves as indicator of stroke severity. Additionally, ATCD helps in estimating the risk of recurrence. These findings enhance clinical value and provide valuable insights, helping clinicians identify potentially significant factors that may influence patient outcomes from a different perspective.

## 5. CONCLUSION

Radiomics features, linked to physical properties, enhance the model's decision-making, rendering it more explainable and understandable to clinicians, thereby increasing trust. Furthermore, the transformer's attention mechanism captures non-linear relationships between clinical and imaging data, aligning with our goal of integrating clinical insights with imaging data. Our comprehensive analysis across different data modalities demonstrates the effectiveness of our methodology, suggesting a pathway toward more personalized healthcare solutions.

While our method for expanding the DWI mask effectively addresses the challenge of nearly invisible lesions, it may introduce irrelevant regions, leading to discrepancies when compared to the actual FLAIR lesions. To enhance accuracy, we plan to train segmentation models that can predict the final lesion [15] and utilize these predictions for feature extraction. Additionally, future research will investigate the relationship between each radiomics-extracted biomarker and clinical outcomes through detailed quantitative experiments, potentially yielding new insights into clinical prognosis.

## 6. ACKNOWLEDGMENTS

This work was supported by the RHU MARVELOUS (ANR-16-RHUS-0009) of Universite Claude Bernard Lyon-1 and by the RHU BOOSTER (ANR-18-RHUS-0001), within the program "Investissements d'Avenir" operated by the French National Research Agency (ANR). The authors acknowledge the support of the ANR, under grant ANR-20-CE23-0013 (project PMR).
# 7. COMPLIANCE WITH ETHICAL STANDARDS

This study followed the principles of the Declaration of Helsinki, and was approved by the local ethics committee (IRB number: 00009118, September 8th, 2016). All subjects or their relatives signed an informed consent form.

# 8. REFERENCES

[1] V Feigin et al., “Global, regional, and national burden of stroke and its risk factors, 1990–2019: A systematic analysis for the global burden of disease study 2019,” *The Lancet Neurology*, vol. 20, no. 10, pp. 795–820, 2021.

[2] M Goyal et al., “Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials,” *The Lancet*, vol. 387, no. 10029, pp. 1723–1731, 2016.

[3] J. Saver and et al., “Improving the reliability of stroke disability grading in clinical trials and clinical practice: the rankin focused assessment (rfa),” *Stroke*, vol. 41, no. 5, pp. 992–995, 2010.

[4] HJA Van Os et al., “Predicting outcome of endovascular treatment for acute ischemic stroke: potential value of machine learning algorithms,” *Frontiers in neurology*, vol. 9, pp. 784, 2018.

[5] S Bacchi et al., “Deep learning in the prediction of ischaemic stroke thrombolysis functional outcomes: a pilot study,” *Academic Radiology*, vol. 27, no. 2, pp. 19–23, 2020.

[6] D Ma et al., “Transformer-based classification outcome prediction for multimodal stroke treatment,” *arXiv preprint arXiv:2404.12634*, 2024.

[7] N Hatami et al., “Cnn-lstm based multimodal mri and clinical data fusion for predicting functional outcome in stroke patients,” in *International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)*, Jul 2022, pp. 3430–3434.

[8] N Hatami et al., “A novel autoencoders-lstm model for stroke outcome prediction using multimodal mri data,” in *2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI)*, Apr 2023, pp. 1–5.

[9] Y Gorishniy et al., “Revisiting deep learning models for tabular data,” *Advances in Neural Information Processing Systems*, vol. 34, pp. 18932–18943, 2021.

[10] B Zhu et al., “Xtab: Cross-table pretraining for tabular transformers,” in *Proceedings of the International Conference on Machine Learning (ICML)*, 2023, pp. 181–204.

[11] P Gijsbers et al., “Amlb: an automl benchmark,” *Journal of Machine Learning Research*, vol. 25, no. 101, pp. 1–65, 2024.

[12] N Painchaud et al., “Fusing echocardiography images and medical records for continuous patient stratification,” *arXiv preprint arXiv:2401.07796*, 2024.

[13] J van Timmeren et al., “Radiomics in medical imaging—“how-to” guide and critical reflection,” *Insights into Imaging*, vol. 11, pp. 91, 2020.

[14] Radiomics Documentation, “pyradiomics: Documentation,” https://pyradiomics.readthedocs.io/en/latest/, 2024.

[15] N Debs et al., “Impact of the reperfusion status for predicting the final stroke infarct using deep learning,” *NeuroImage: Clinical*, vol. 29, pp. 102548, 2021.

[16] GA Ortiz and RL Sacco, “National institutes of health stroke scale (nihss),” *Wiley StatsRef: statistics reference online*, 2014.

[17] SM Smith et al., “Advances in functional and structural mr image analysis and implementation as fsl,” *NeuroImage*, vol. 23, no. Suppl 1, pp. S208–S219, 2004.

[18] R Tibshirani, “Regression shrinkage and selection via the lasso,” *Journal of the Royal Statistical Society: Series B (Methodological)*, vol. 58, no. 1, pp. 267–288, 1996.

[19] N Erickson et al., “Autogluon-tabular: Robust and accurate automl for structured data,” *arXiv preprint arXiv:2003.06505*, 2020.