# Design and optimization of an automatic deep learning-based cerebral reperfusion scoring (TICI) using thrombus localization

Arthur Folcher, Jérémy Piters, Daphné Wallach, Gwenael Guillard, Julien Ognard, Jean-Christophe Gentric

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Arthur Folcher, Jérémy Piters, Daphné Wallach, Gwenael Guillard, Julien Ognard, et al.. Design and optimization of an automatic deep learning-based cerebral reperfusion scoring (TICI) using thrombus localization. Journal de Neuroradiologie / Journal of Neuroradiology, 2025, 52 (5), pp.101366. 10.1016/j.neurad.2025.101366. hal-05288144

HAL Id: hal-05288144

https://hal.univ-brest.fr/hal-05288144v1

Submitted on 29 Sep 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

Original Article

# Design and optimization of an automatic deep learning-based cerebral reperfusion scoring (TICI) using thrombus localization

<figure><img src="image_4.png" /></figure>

Arthur Folcher<sup>a,*</sup>, Jérémy Piters<sup>a</sup>, Daphné Wallach<sup>b</sup>, Gwenael Guillard<sup>b</sup>, Julien Ognard<sup>c,d</sup>, Jean-Christophe Gentric<sup>a</sup>

ARTICLE INFO

Keywords:

Artificial intelligence

Deep learning

Thrombectomy

Stroke

Thrombus

TICI

ABSTRACT

**Background:** The Thrombolysis in Cerebral Infarction (TICI) scale is widely used to assess angiographic outcomes of mechanical thrombectomy despite significant variability. Our objective was to create and optimize an artificial intelligence (AI)-based classification model for digital subtraction angiography (DSA) TICI scoring.

**Methods:** Using a monocentric DSA dataset of thrombectomies, and a platform for medical image analysis, independent readers labeled each series according to TICI score and marked each thrombus. A convolutional neural network (CNN) classification model was created to classify TICI scores, into 2 groups (TICI 0,1 or 2a versus TICI 2b, 2c or 3) and 3 groups (TICI 0,1 or 2a versus TICI 2b versus TICI 2c or 3). The algorithm was first tested alone, and then thrombi positions were introduced to the algorithm by manual placement firstly, then after using a thrombus detection module.

**Results:** A total of 422 patients were enrolled in the study. 2492 thrombi were annotated on the TICI-labeled series. The model trained on a total of 1609 DSA series. The classification model into two classes had a specificity of 0.97 ± 0.01 and a sensibility of 0.86 ± 0.01. The 3-class models showed insufficient performance, even when combined with the true thrombi positions, with, respectively, F1 scores for TICI 2b classification of 0.50 and 0.55 ± 0.07. The automatic thrombus detection module did not enhance the performance of the 3-class model, with a F1 score for the TICI 2b class measured at 0.50 ± 0.07.

**Conclusion:** The AI model provided a reproducible 2-class (TICI 0,1 or 2a versus 2b, 2c or 3) classification according to TICI scale. Its performance in distinguishing three classes (TICI 0,1 or 2a versus 2b versus 2c or 3) remains insufficient for clinical practice. Automatic thrombus detection did not improve the model's performance.

Introduction

Mechanical thrombectomy (MT) is standard of care for acute ischemic stroke (AIS) with proximal vessel occlusion.¹ The spectrum of MT indications enlarges²,³ and Thrombolysis in Cerebral Infarction (TICI) score is the used endpoint in clinical practice and in research to assess revascularization on digital subtraction angiography (DSA) images.⁴,⁵ This scale, derived from Thrombolysis in Myocardial Infarction (TIMI) scale, proposed by Higashida et al. in 20,03⁶ is constructed from TICI 0 (no reperfusion) to TICI 3 (complete reperfusion) (Table 1).

The TICI 2 class is the most heterogeneous and scores 2a and 2b have been introduced (less and more of 50 % of the middle cerebral artery (MCA) territory respectively)(7). Relevant differences in clinical outcomes exist between contiguous scores as 2a and 2b, or 2b and 2c.⁸,⁹ Significant inter- and intraobserver variability has been proven, and explained by the number of grades (5 to 6 grades) and the analysis of the images.¹⁰ Operators tend to overestimate the TICI and their accuracy of TICI scoring decreases during nighttime.¹¹

These difficulties and variability of TICI scoring complicate the prediction of clinical outcomes and the decision-making process. AI could

**Abbreviation:** ACA, Anterior Cerebral Artery; AI, Artificial Intelligence; AIS, Acute Ischemic Stroke; AP, AnteroPosterior; API, Angiographic Parametric Imaging; AUC, Area Under the Curve; CNN, Convolutional neural network; DICOM, Digital Imaging and Communications in Medicine; DSA, Digital Subtraction Angiography; FOV, Field Of View; ISO, International Organization for Standardization; Lat, Lateral; MCA, Middle cerebral artery; MT, Mechanical thrombectomy; PACS, Picture Archiving and Communication System; ROC, Receiver Operating Characteristic; SD, Standard Deviation; TICI, Thrombolysis in Cerebral Infarction; TIMI, Thrombolysis in Myocardial Infarction

*Corresponding author at: Radiology Department, Brest University Hospital, Brest, France.

E-mail address: arthur.folcher@chu-brest.fr (A. Folcher).

<figure><img src="image_5.png" /></figure>

<footnote><sup>a</sup> Neuroradiology Department, Brest University Hospital, Brest, France</footnote>

<footnote><sup>b</sup> Research and Development Department, Intradys, Brest, France</footnote>

<footnote><sup>c</sup> Radiology, Mayo Clinic, Rochester, MN, USA</footnote>

<footnote><sup>d</sup> LaTIM INSERM UMR1101, University Hospital of Brest, Brest, France</footnote>
<table><caption>**Table 1**
Revised TICI score as defined by Mohammed A Almekhlafi et al.²⁰.</caption><thead><tr><th>Score</th><th>Revised Thrombolysis in Cerebral Infarction Scale</th></tr></thead><tbody><tr><td>0</td><td>No perfusion or anterograde flow beyond the site of occlusion.</td></tr><tr><td>1</td><td>Penetration but not perfusion. Contrast penetration exists past the initial obstruction but with minimal filling of the normal territory.</td></tr><tr><td>2</td><td>Incomplete perfusion wherein the contrast passes the occlusion and opacifies the distal arterial bed but rate of entry or clearance from the bed is slower or incomplete when compared to non-involved territories:<br>2a Some perfusion with distal branch filling of &lt; 50 % of territory visualized<br>2b Substantial perfusion with distal branch filling of ≥ 50 % of territory visualized<br>2c Near complete perfusion except for slow flow in a few distal cortical vessels, or presence of small distal cortical emboli</td></tr><tr><td>3</td><td>Complete perfusion with normal filling of all distal branches</td></tr></tbody></table>

be a help to reduce variability and to provide a reproducible rating. Deep learning-based algorithms have already shown their effectiveness in enhancing human detection performance in various radiological fields.¹²,¹³ Semi-automatic assessment of TICI has shown similar performance compared to an independent core-lab TICI scoring in predicting favorable functional outcomes. However, this method is time-consuming and subject to variability, as it relies on operator labeling of the target downstream territory.¹⁴ An automatic quantitative evaluation, autoTICI, using a ResNet-based model demonstrated a strong correlation with the TICI score and functional prognosis, although the correlation was weaker for M2 occlusion and incomplete reperfusion, as the model did not account for occlusion detection.¹⁵

Studies suggest that a fully automated evaluation by a convolutional neural network (CNN) could assess reperfusion and provide proof of principle for deep-learning-based automated TICI scoring in AIS with proximal occlusion, with interobserver variability comparable to human expert.¹⁶,¹⁷ Although studies have demonstrated the feasibility of automatic detection of thrombus position in large vessel occlusion on cerebral CT angiography, to our knowledge, clot localization has never been automatized and utilized in a deep learning algorithm to help in predicting TICI scores.¹⁸,¹⁹

The aim of this study is to design a deep learning-based model to automate TICI scoring, and to improve its performance by adding thrombus detection.

## Materials and methods

### Patients and images

DSA series came from the department's anonymized database, with approval from a human subject protection committee (approval number N°29BRC20.0032). All patients who underwent MT in Brest between June 2015 and June 2020 were enrolled. A biplane angiographic sensor (Siemens C-arm Artis Zee for interventional neuroradiology) was used. Patients without available DSA series or those who underwent multiple MT procedures were excluded. DSA series were anonymized, and identified. DICOM images were retrieved from the Brest University Hospital's picture archiving and communication system (PACS) and uploaded to an online platform for medical image analysis and reading: MD.ai. (https://matrix.md.ai/hub) (Appendix).

### Labeling TICI

DSA series were retrospectively rated according to the revised TICI score (Table 1) using the MD.ai platform. TICI scoring was provided by two readers, Reader 1 and Reader 2, with 12 and 4 years of experience, who annotated all datasets using specially designed labels. Each DSA was rated based on the reader's own judgment, blind to clinical outcomes, by choosing one label from a set of 12 possibilities including

TICI scores (TICI0 TICI1 TICI2a TICI2b TIC2c TICI3) but also labels in case of inability to score the images (No DSA images; No middle cerebral artery territory; Movements; Unsatisfactory field of view (FOV); Selective catheterization; Others).

The Reader 3 (9 years of experience) rated the series subject to key discrepancies between Reader 1 and Reader 2, defined as discrepancies with a significant outcome (i.e. excluding discrepancies between TICI 0 and 1, and 2c and 3). Engineering team conducted data cleaning, using assigned labels and a set of quality criteria (number of images per DSA series, angulation of images, number of frames, quality of subtraction, unsupported files by the program, DSA series lacking anteroposterior and lateral axes). This process resulted in datasets containing images of comparable quality, constituting the ground truth for training the model.

### Labeling thrombi

All TICI-labeled DSA series were analyzed by Reader 4 (4 years of experience), and each thrombus localization was marked by a cross on each anteroposterior (AP) and lateral (Lat) DSA using the Mdai platform (Fig. 1). In cases of thrombus-labeling uncertainty, Reader 1 adjudicated. Non-TICI labeled DSA series were excluded from thrombus annotation. Proximal end of each thrombus was marked on each view, differentiating thrombi involving the MCA territory (localization including terminal internal carotid artery and MCA) from thrombi involving only the anterior cerebral artery (ACA).

A visible thrombus was defined as an abnormal sudden stop of the contrast agent within an artery or as an abnormal intra-arterial opacity with decreased opacification of the downstream arterial network. Occlusive thrombi (O) from non-occlusive thrombi (NO) were distinguished. A clot visible in one view was matched with the clot in the other view. If no clot was visible in the other view, a general label "Thrombus not seen" was assigned with a suffix identifying the absent thrombus. In the absence of thrombi in the anterior cerebral circulation, the label "No thrombus" was assigned. A final automatic verification of accurate labeling was conducted. Following the correction of any remaining errors, no discrepancies were detected during the final dataset control.

## Artificial intelligence model development

### Classification model

Input data initially consisted in DSA series with their TICI label, and then combined with manual thrombi localization. Multiple datasets of series were generated; 20 % of the whole dataset was used for test. The remaining 80 % was used for training and validation using a five-fold validation scheme. Each fold (20 % of the remaining data, i.e. 16 % of the whole dataset) was used in turn for validation, while the remaining four folds (80 % of the remaining data, i.e. 64 % of the whole dataset) were used for training. Data separation is implemented in such a way that series from the same patient are never part of two different datasets, and that each dataset has roughly the same distribution of scores.

A convolutional neural network (CNN) has been developed, implemented in Python, and adheres to quality standards specified by the International Organization for Standardization (ISO 13485 and IEC 62304). In a similar manner to the one described in the study conducted by Bhurwani et al.,¹⁶ angiographic parametric imaging (API) was used along with temporal and hemodynamic data to extract dynamic information from each point in the DSA series. No other information extracted from DICOM data was incorporated. One neural network per dynamic parameter was trained, i.e. its parameters were optimized so that the predicted score most closely matched the real score on the training dataset. The predictions of each neural network were then linearly combined using a weighted sum. The contributing weight of each single network was optimized on the validation dataset of the first fold. The model was trained to classify series into two and three classes. The 2-class model
<figure><img src="image_6.png" /><figcaption>Fig. 1. DSA series AP (left) and Lat (right).</figcaption></figure>

These series were labeled TICI 0. The cross indicates the proximal end of the occlusive thrombus on the AP view (MCA-O-AP1). This clot could not be distinguished on the lateral view, therefore the label “MCA-O-Lat1 not seen” was used.

distinguishes between TICI 0, 1, 2a and 2b, 2c, 3. The 3-class model distinguishes a lowest class, corresponding to TICI 0, 1 and 2a, a middle class, corresponds to TICI 2b, a highest class corresponding to scores 2c and 3. The ground truth for the model was defined by Reader 1 or by consensus with Reader 3 in cases of key discrepancies.

The CNN used for the automatic score prediction was based on the EfficientNetB2 architecture. The model was pretrained on the ImageNet dataset. The last dense layer of the network was then discarded, and replaced by a global average pooling layer, followed by a dense layer containing two neurons for the two-class model, and three neurons for the three-class model. Only the last layer was trained at first, then all layers were retrained. The cross-entropy cost function was used with the Adam optimizer.

Both pretraining and retraining steps used a learning rate scheduler dividing the learning rate by 10 when the accuracy on the validation

data stagnates for more than 10 epochs. The initial learning rate was set to $10^{-3}$ for the first step and $10^{-4}$ for the second step. The number of epochs was fixed at 20 for both steps. All hyperparameters were optimized on the validation dataset.

As a first approach, the first 12 frames of each series phases were used as model input. A CNN was then trained to automatically distinguish the arterial phase. This CNN is applied to a given frame independently, and outputs the probability that this frame belongs to the arterial phase. A post-processing step ensures venous phase frames follow arterial phase frames in the series. A sensitivity analysis (Fig. 2) revealed that mistakenly excluding arterial frames when training the TICI classification model leads to a decreased performance, whereas mistakenly including an extra venous frame had no effect. Therefore, the number of frames used was chosen as the number of detected arterial frames plus two frames.

<figure><img src="image_7.png" /><figcaption>Fig. 2. Sensitivity analysis, showing the evolution of different metrics according to the number of missing arterial frames.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>**Fig. 3.** Illustration of the segmented analysis by the detection model.</figcaption></figure>

The clot detection model was trained on cropped 128 × 128 pixels images to achieve an acceptable training time. The analysis of entire images was made possible by downsampling them to 512 × 512 pixels and applying a sliding window of 128 × 128 pixels (yellow square), progressively analyzing the whole image (blue squares).

### Detection module combined with classification model

Analogous to the classification model, a CNN was employed to automate thrombus detection. Training and validation also used a five-fold validation scheme, in such a way that each dataset has roughly the same distribution of thrombi. Input data consisted in DSA series along with no, one or several thrombi positions given by Reader 4, considered as the ground truth. Input images were automatically spatially segmented for analysis, with multiple analysis over central segments (Fig. 3). The module made localization predictions, filtering out less probable predictions and those that were too close to each other, to provide only the most probable final predictions (Fig. 4). Detectors positions were calculated using DICOM tags, and predictions of each view were projected onto the other view.

Predictions which did not correspond to any prediction on the other view were ruled out. Several post-processing pipelines were tested, and results are provided for the pipeline with the best performances. Parameters of the detection module were adjusted through backpropagation, according to the comparison with the ground truth. The input data of the final classification model were structured as three-channel images. The first channel contained parameter maps derived from the DSA series. The second and third channels contained the presence of thrombi detected by the module in the MCA and ACA, respectively. Thrombi were represented as square regions measuring 12 mm per side, with pixel intensities of 255 for occlusive thrombi and 128 for non-occlusive thrombi, using a standard 8-bit grayscale scale.

<figure><img src="image_9.png" /><figcaption>**Fig. 4.** Post-processing procedure for thrombus detection.
The hollow squares represent a model prediction. The filled square represents the ground truth.
Each step of the post-processing procedure rules out some predictions, finally retaining the predictions with the highest confidence score.</figcaption></figure>
## Statistical analysis

The statistical analysis of inter-operator variability was calculated using Cohen's Kappa index.

For each of the five folds, various metrics were computed on the test set, such as sensitivity, specificity, area under the receiver operating characteristic curve (AUC) score, accuracy, and F1 score (harmonic mean of recall and precision).

The mean and the standard deviation (SD) of each metric were then computed over all folds. For the 2-class model, a true positive was defined as a predicted TICI score of 2b-2c-3, consistent with the ground truth. For the 3-class, a true positive was considered as the prediction of the relevant class, compared to all the other classes. The model's performance was compared to the performance of Reader 2 relative to Reader 1. The detection model performance metrics were computed on the validation set of the first fold. Using confusion matrix, the number of true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN) was computed. The threshold to reach by the model was determined as an F1 score greater than 0.6 for any grade.

## Results

### Patients and images

465 patients were initially enrolled in the study. Following the exclusion of patients who underwent multiple MT and those whose imaging data were either inaccessible or unrecorded, the final pool comprised 422 patients (F/M ratio = 0.83, mean age = 74 ± 14.5 years). The initial dataset encompassed 96,951 images derived from 4032 series, as shown in Fig. 5.

### Labeling TICI

According to the readers, between 40,6 % and 48,4 % of series were non-TICI labeled. Among the TICI labels, the most common rating by Reader 1 was TICI 2a (14,8 % of all series), the same for Reader 2 (17,7 %). TICI 2b represented between 10,8 % to 15,6 % of the series.

1940 DSA series were TICI labeled, among which unusable images were removed, resulting in a total of 1609 valid DSA series. The contingency table analysis identified a total of 1057 disagreements between

<figure><img src="image_10.png" /><figcaption>Fig. 5. Flow Chart.</figcaption></figure>
<table><caption>**Table 2**
Distribution of thrombi labels.</caption><thead><tr><th></th><th>MCA-O</th><th>MCA-NO</th><th>ACA-NO</th><th>ACA-NO</th></tr></thead><tbody><tr><th>Lat</th><td>784</td><td>364</td><td>92</td><td>29</td></tr><tr><th>AP</th><td>854</td><td>265</td><td>79</td><td>25</td></tr></tbody></table>

Anteroposterior (AP); Lateral (Lat); Middle Cerebral Artery Occlusive (MCA-O); Middle Cerebral Artery Non Occlusive (MCA-NO); Anterior Cerebral Artery Occlusive (ACA-O); Anterior Cerebral Artery Non Occlusive (ACA-NO).

Reader 1 and 2, out of 4032 ratings, representing 26.2 % of judgments. The Cohen's Kappa index was measured at 0,69 between Reader 1 and Reader 2. A total of 308 key discrepancies were analyzed by Reader 3 to establish a consensus, resulting in a disagreement with the first two readers in 27 % of the DSA series (13 % of non-TICI labeled DSA series, and 14 % of different TICI labels). The Cohen's Kappa coefficient was calculated at 0.12 between Reader 1 and Reader 3, and 0.25 between Reader 2 and Reader 3.

### Labeling thrombi

A total of 2492 thrombi were annotated, among which those in the MCA represented 91.4 %. Occlusive thrombi accounted for 70.2 % of the annotated thrombi. 34.8 % of MCA thrombi visible on the AP view were absent on the lateral view, and 24.2 % of thrombi seen laterally were missing from the AP view. No thrombi were found in 27.8 % of the series (Table 2).

### Artificial intelligence model development

#### Classification model

The testing dataset comprised 286 series, and the remaining series was distributed between training and validation datasets. The 2-class model showed a specificity of 0.97 ± 0.01 (mean ± SD), a sensitivity of 0.86 ± 0.01, and an accuracy of 0.92 ± 0.01 (Table 3). Its performance was equivalent to Reader 2 for the 2-class model. The 3-class model showed inferior performance in detecting TICI 2b, with an F1 score of 0.50 compared to 0.66, and lower accuracy 0.81 versus 0.83. After adding manual thrombus position, the 3-class model increased its accuracy measured at 0.85 and its F1 score for TICI 2b at 0.55. Regarding the Receiver Operating Characteristic (ROC) curves (Fig. 6), the lowest values of AUC were measured for the distinction between TICI 2b and high scores, at 0.66 for the classifier alone and 0.85 for the classifier with manual thrombus position, meanwhile the AUC for the separation between low and high score were measured between 0.96 and 0.99.

#### Detection module combined with classification model

The detection module detected 228 thrombi on the first fold, of which 149 were false positives, and 255 true thrombi were not detected by the module (Tables 4.1 and 4.2). Compared to the classification

model alone, the 2-class model's performance remains similar after adding the thrombus detection module, with an accuracy measured at 0.91 vs 0.92. Similarly, the 3-class model was not improved as shown by an F1 score for TICI 2b measured at 0.49 vs 0.50. AUC for TICI 2b vs. low score was similar at 0.96 vs 0.96, and the AUC for TICI 2b vs. high score increased to 0,69 vs 0,66 (See Appendix Table A. for more details). Almost all metrics decreased after adding automatic thrombi detection compared to true thrombi position, as evidenced by the accuracy measured at 0.80 vs 0.85.

### Discussion

We presented an automatic model for cerebral reperfusion scoring, combined with an automatic thrombus detection. 4032 series were included before data cleaning, comparable to the literature, with for example 1857 DSA series included in a study conducted by Ruisheng Su et al.²¹ The TICI and thrombi annotation were effective using the Md.ai platform. Inter-observer variability shown by a Cohen's Kappa coefficient was measured at 0.69, considered to be a substantial agreement²² and may be attributed to a monocentric bias. Concerning key discrepancies, the high inter-observer variability between Reader 3 and Readers 1 and 2 highlights the grading difficulty of these borderline cases.

The minimum desired performance of our model was set to be at least equivalent to that of a human operator in peroperative conditions, as provided in Zhang's 2018 study,¹¹ which analyzed 1182 patients. The 2-class model alone demonstrated superior performance compared to human classification, with an F1 score greater than 0.9 for any grade, vs between 0.70 and 0.89.¹¹ It can be noted that the AUC for 2-class model reached 0.98, versus 0.90 for the automatic 2-class TICI in the study conducted by Ruisheng Su et al.,²¹ whom demonstrated that automated TICI can achieve a comparable predictive ability for clinical outcomes, for DSA with M1 MCA or internal carotid artery occlusions. The 3-class model showed insufficient performance in classifying TICI 2b, with F1 scores of 0.5 for the classification model alone but remains better than human results in Zhang's 2018 study that showed an F1 score at 0,40.¹¹ Before the revised TICI scale, TICI 2b could be seen by operators as the minimal target to reach to declare the procedure a success. TICI 2b has covered a wide spectrum of clinical outcomes, resulting in creation of TICI 2c, describing near complete reperfusion.²³ Distinguishing a TICI 2b score is crucial due to the differences in clinical outcomes between 2b and 2c.⁸ We hypothesized that the model's poor performance in predicting TICI 2b was partly due to the AI model limitations, but also to the variability in the application and definition of the score among readers and series. Scoring can sometimes be based on parenchymography, indicating cerebral reperfusion, or recanalization, or a variable proportion of both.¹⁰

The model associated with manual thrombus position shows superior performance with an F1 score measured at 0.55 versus 0.50 for TICI 2b. These results remain moderate, as it represents the maximum performance that can be expected from a model with an automatic thrombi detection. Adding automatic thrombi detection did not improve model's

<table><caption>**Table 3**
Performance overview of the classification model alone, with true thrombi positions and combined with automatic thrombi detection (mean ±SD).</caption><thead><tr><th rowspan="3">TICI</th><th colspan="3">2-Class Model</th><th colspan="3">3-Class Model</th></tr><tr><th>0-1-2a</th><th>2b-2c-3</th><th rowspan="2">Accuracy</th><th>0-1-2a</th><th>2b</th><th rowspan="2">Accuracy</th></tr><tr><th>F1</th><th>F1</th><th>F1</th><th>F1</th></tr></thead><tbody><tr><td>A posteriori score by Reader 2</td><td>0.84</td><td>0.83</td><td>0.86</td><td>0.84</td><td>0.66</td><td>0.79</td><td>0.83</td></tr><tr><td>Classification model</td><td>0.94 ± 0.01</td><td>0.91 ± 0.01</td><td>0.92 ± 0.01</td><td>0.91 ± 0.01</td><td>0.50 ± 0.02</td><td>0.74 ± 0.04</td><td>0.81 ± 0.01</td></tr><tr><td>Classification model with thrombi positions</td><td>0.92 ± 0.01</td><td>0.90 ± 0.01</td><td>0.91 ± 0.01</td><td>0.91 ± 0.01</td><td>0.55 ± 0.07</td><td>0.85 ± 0.02</td><td>0.85 ± 0.02</td></tr><tr><td>Classification model combined with automatic thrombi detection</td><td>0.92 ± 0.01</td><td>0.90 ± 0.01</td><td>0.91 ± 0.01</td><td>0.91 ± 0.01</td><td>0.49 ± 0.03</td><td>0.73 ± 0.03</td><td>0.80 ± 0.01</td></tr></tbody></table>
<figure><img src="image_11.png" /><figcaption>Figure 6.1.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 6.2.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 6.3.</figcaption></figure>

<caption>**Fig. 6.** ROC curves of the 3-class model, for classification only (Figure 6.1), with true thrombi positions (Figure 6.2) and with automatic thrombi detection (Figure 6.3).</caption>

Low score corresponds to TICI 0,1 or 2a, med score to TICI 2b, and high score to TICI 2c or 3. (See Appendix Figure B. for more details).

performance and was particularly less favorable in the 2-class model, highlighting that the contribution of our automatic thrombus detection, with numerous false positives, was erroneous. The ROC curve for the separation between TICI 2b and the high grades concerning the classification model alone and combined with automatic thrombus detection showed an AUC of 0.66 and 0.69, respectively, emphasizing insufficient performance for practical application. As a single training dataset was

used for both classification and detection models, the performance is therefore overestimated, compared to what would be achieved under real-life conditions. In fact, the dataset was too small to be separated into two independent training sets and to achieve an unbiased approach.

The automatic thrombus detection is affected by the subjectivity in defining and localizing a thrombus. This raises the issue of differential

<caption>**Table 4.1**</caption>

Confusion matrix of the detection module on the first fold, indicating the number of thrombi, AP view.

<table><caption>**Table 4.2**</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="5">Predicted label</th></tr><tr><th>MCA-O</th><th>MCA-NO</th><th>ACA-O</th><th>ACA-NO</th><th>Not detected</th></tr></thead><tbody><tr><td rowspan="5">True label</td><td>MCA-O</td><td>36</td><td>0</td><td>0</td><td>0</td><td>75</td></tr><tr><td>MCA-NO</td><td>2</td><td>6</td><td>0</td><td>0</td><td>31</td></tr><tr><td>ACA-O</td><td>1</td><td>0</td><td>3</td><td>0</td><td>18</td></tr><tr><td>ACA-NO</td><td>0</td><td>0</td><td>0</td><td>1</td><td>3</td></tr><tr><td>No thrombus</td><td>68</td><td>13</td><td>2</td><td>2</td><td>0</td></tr></tbody></table>

Confusion matrix of the detection module on the first fold, indicating the number of thrombi, Lat view.

<table><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="5">Predicted label</th></tr><tr><th>MCA-O</th><th>MCA-NO</th><th>ACA-O</th><th>ACA-NO</th><th>Not detected</th></tr></thead><tbody><tr><td rowspan="5">True label</td><td>MCA-O</td><td>23</td><td>0</td><td>0</td><td>0</td><td>71</td></tr><tr><td>MCA-NO</td><td>0</td><td>4</td><td>0</td><td>0</td><td>31</td></tr><tr><td>ACA-O</td><td>1</td><td>0</td><td>0</td><td>1</td><td>19</td></tr><tr><td>ACA-NO</td><td>0</td><td>0</td><td>0</td><td>1</td><td>7</td></tr><tr><td>No thrombus</td><td>51</td><td>10</td><td>1</td><td>2</td><td>0</td></tr></tbody></table>
diagnosis, including dissections, angiographic spasms, or flow stasis. Furthermore, identifying thrombi in TICI 2b series can be challenging, as it often involves multiple or distal thrombi. Finally, the training of the detection module may also have been biased by the data annotation, performed by a single reader who had access to all the series. However, interobserver variability in position annotation of the proximal endpoint of the clot was measured at 2% ±3% of image width in Mittmann et al. 2022 study,²⁴ showing the limited subjectivity of these annotations. Automatic thrombi detection is rarely described in literature. A study shows that automatic detection of large vessel occlusion in CT angiography can highlight areas of decreased contrast, leading to clot localization.¹⁸ Mittmann et al. demonstrated acceptable performance of a deep learning algorithm in classifying DSA series into thrombus and non-thrombus groups, without providing TICI classification.²⁴ In clinical practice, it correctly classifies large vessel occlusions in the M1 segment, which remains insufficient.

There is a tendency to underestimate the concept of reperfusion, which represents the best criterion for effective revascularization.⁷ TICI, which is a cerebral perfusion score, may be overestimated during proper recanalization despite persistence of a cerebral perfusion defect.²¹ This highlights a limitation of the thrombus annotation strategy for model optimization, whereas recanalization assessment appears to be helpful to measure the direct action of the MT.⁷ Specific recanalization scores, such as the Arterial Occlusive Lesion (AOL) scale, exist and could be automated,⁷ but are rarely used in routine practice. New scales have emerged, notably the expanded TICI with 7 classes, which has been suggested by some studies to be more relevant for predicting clinical outcome.²⁵ The increase in the number of classes in the TICI score, however, must be balanced against induced variability and the growing difficulty of the AI model. Additionally, TICI score can be defined as a relative measure comparing perfusion before and after MT.²¹ Our rating is based on a single series at a time, always relative to the entire MCA territory, as we wanted each series to be scorable without any additional context. A model incorporating the pre-treatment series might help improve automatic scoring. Beyond TICI scoring, automatic analysis of flat panel CT perfusion could help reduce the subjectivity of perfusion assessment.²⁶ However, since this technique is less frequently performed during the procedure, it appears to be a less practical alternative than DSA series cotation for decision-making.

Our results could also be partially explained by a lack of power of the CNN model. In the future, normalizing the FOV of the image using fixed landmarks may facilitate AI analysis. While a Transformer-type AI model could have been used, the need for a very large quantity of training data makes this type of AI model less relevant for medical imaging use. Our AI model delivered the classification in a matter of seconds, and the computational weight of the model was sufficiently low to run on a Central Processing Unit (CPU) making it applicable to clinical practice. Another main question is ethical limits in the use of an artificial intelligence model in medicine, as evidenced in the judicial system by the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) software.²⁷ This study adheres to the TRIPOD-AI guidelines.²⁸ Our study focused on creating a high-performing model, without conducting performance calculations for different groups to check for discrimination bias. However blind reading of clinical data helps to limit

this risk. Despite the necessary questioning around confidence in AI, establishing a reproducible TICI scoring is crucial given its role in stroke therapeutic research.²⁹ This objective evaluation could also provide greater transparency among the various members of the medical team and serve as a decision support tool for the operator during the MT procedure.

## Conclusion

We have presented an automatic classification model for cerebral reperfusion assessment using the TICI score, adding an automatic thrombi detection. While the results of a 2-class model were satisfactory, the 3-class model showed insufficient efficacy, particularly concerning TICI 2b Automatic thrombi detection did not improve the model's performance. Manual positions of thrombi slightly improved the model results, suggesting that it could be beneficial to implement some adjustments, to achieve an acceptable level of effectiveness for clinical practice.

## Human and animal rights

The authors declare that the work described has been carried out in accordance with the Declaration of Helsinki of the World Medical Association revised in 2013 for experiments involving humans as well as in accordance with the EU Directive 2010/63/EU for animal experiments.

## Informed consent and patient details

The authors declare that this report does not contain any personal information that could lead to the identification of the patient(s) and/or volunteers.

## Funding

This work did not receive any grant from funding agencies in the public, commercial, or not-for-profit sectors.

## Author contributions

All authors attest that they meet the current International Committee of Medical Journal Editors (ICMJE) criteria for Authorship.

## Declaration of competing interest

Dr. Julien OGNARD received research grants from the French Society of Neuroradiology SFNR, the French network of University Hospitals HUGO ('Hôpitaux Universitaires du Grand Ouest'), INNOVEO-Chu De Brest, Medtronic, Phenox, Institut Servier, Philippe Foundation, Philips, and received consulting fees from Balt.

Dr. Daphné WALLACH and Gwenael GUILLARD are affiliated with Intradys. The company had no role in the data collection or interpretation, nor in the decision to submit the manuscript for publication.
Appendix

<figure><img src="image_14.png" /><figcaption>Fig. A1. Md.ai Platform.</figcaption></figure>
<caption>**Fig. B1.** ROC curves of the 3-class model, for classification only (Fig. B.1), with true thrombi positions (Figure B.2) and with automatic thrombi detection (Figure B.3), showing results for one score versus rest classification results.</caption>

Low score corresponds to TICI 0,1 or 2a, med score to TICI 2b, and high score to TICI 2c or 3.

<figure><img src="image_15.png" /><figcaption>Figure B.1</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure B.2</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure B.3</figcaption></figure>

**Table A.1**

Confusion matrix of the 2-class classification model with thrombus detection.

<table><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="2">Predicted label</th></tr><tr><th>TICI 0-1-2a</th><th>TICI 2b-2c-3</th></tr></thead><tbody><tr><th rowspan="2">True label</th><th>TICI 0-1-2a</th><td>147</td><td>14</td></tr><tr><th>TICI 2b-2c-3</th><td>8</td><td>117</td></tr></tbody></table>

**Table A.2**

Confusion matrix of the 3-class classification model with thrombus detection.

<table><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">Predicted label</th></tr><tr><th>TICI 0-1-2a</th><th>TICI 2b</th><th>TICI 2c-3</th></tr></thead><tbody><tr><th rowspan="3">True label</th><th>TICI 0-1-2a</th><td>156</td><td>5</td><td>0</td></tr><tr><th>TICI 2b</th><td>16</td><td>28</td><td>3</td></tr><tr><th>TICI 2c-3</th><td>14</td><td>21</td><td>43</td></tr></tbody></table>
References

1. Goyal M, Menon BK, Van Zwam WH, Dippel DWJ, Mitchell PJ, Demchuk AM, et al. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. *Lancet*. 2016;387(10029):1723-1731.

2. Ducroux D, Derex L, Nourredine M, Haesebaert J, Buisson M, Alesefir W, et al. Successful thrombectomy is beneficial in patients with pre-stroke disability: results from an international multicenter cohort study. *J Neuroradiol*. 2020;50(1):59-64.

3. Nogueira RG, Jadhav AP, Haussen DC, Bonafe A, Budzik RF, Bhuva P, et al. Thrombectomy 6 to 24 hours after stroke with a mismatch between deficit and infarct. *N Engl J Med*. 2018;378(1):11-21.

4. Powers WJ, Rabinstein AA, Ackerson T, Adeoye OM, Bambakidis NC, Becker K, et al. 2018 Guidelines for the early management of patients with acute ischemic Stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. *Stroke [Internet]*. 2018;49(3). Mar [cited 2024 Sep 24]Available from; https://www.ahajournals.org/doi/10.1161/STR.0000000000000158.

5. Turc G, Bhogal P, Fischer U, Khatri P, Lobotesis K, Mazighi M, et al. European Stroke Organisation (ESO) - European Society for Minimally Invasive Neurological Therapy (ESMINT) Guidelines on mechanical thrombectomy in acute ischemic Stroke. *J NeuroInterv Surg*. 2023;15(8):e8..-e8.

6. Higashida RT, Furlan AJ, Roberts H, Tomsick T, Connors B, Barr J, et al. Trial design and reporting standards for intra-arterial cerebral thrombolysis for acute ischemic stroke. *Stroke*. 2003;34(8):e109-e137.

7. Zaidat OO, Yoo AJ, Khatri P, Tomsick TA, von Kummer R, Saver JL, et al. Recommendations on angiographic revascularization grading standards for acute ischemic stroke: a consensus statement. *Stroke*. 2013;44(9):2650-2663.

8. Jang KM, Nam TK, Ko MJ, Choi HH, Kwon JT, Park SW, et al. Thrombolysis in cerebral infarction grade 2C or 3 represents a better outcome than 2B for endovascular thrombectomy in acute ischemic stroke: a network meta-analysis. *World Neurosurg*. 2020;136:e419-e439.

9. Karlsson A, Jood K, Björkman-Burtscher IM, Rentzos A. Extended treatment in cerebral ischemia score 2c or 3 as goal of successful endovascular treatment is associated with clinical benefit. *J Neuroradiol*. 2024;51(2):190-195.

10. Fugate JE, Klunder AM, Kallmes DF. What is meant by "TICI"? *AJNR Am J Neuroradiol*. 2013;34(9):1792-1797.

11. Zhang G, Treurniet KM, Jansen IGH, Emmer BJ, Van Den Berg R, Marquering HA, et al. Operator versus core lab adjudication of reperfusion after endovascular treatment of acute ischemic stroke. *Stroke*. 2018;49(10):2376-2382.

12. Kalmet PHS, Sanduleanu S, Primakov S, Wu G, Jochems A, Refaee T, et al. Deep learning in fracture detection: a narrative review. *Acta Orthop*. 2020;91(2):215-220.

13. Bathla G, Dhruba DD, Soni N, Liu Y, Larson NB, Kassmeyer BA, et al. AI-based classification of three common malignant tumors in neuro-oncology: a multi-institutional comparison of machine learning and deep learning methods. *J Neuroradiol*. 2024;51(3):258-264.

14. Prasetya H, Ramos LA, Epema T, Treurniet KM, Emmer BJ, Van Den Wijngaard IR, et al. qTICI: quantitative assessment of brain tissue reperfusion on digital subtraction angiograms of acute ischemic stroke patients. *Int J Stroke*. 2021;16(2):207-216.

15. Van der Sluijs PM, Su R, Cornelissen S, van Es ACGM, Lycklama A, Nijeholt GJ, van Doormaal PJ, et al. Assessment of automated TICI scoring during endovascular treatment in patients with an ischemic stroke. *J Neurointerv Surg*. 2025. jnis-2024-021892.

16. Shiraz Bhurwani MM, Snyder KV, Waqas M, Mokin M, Rava RA, Podgorsak AR, et al. Use of biplane quantitative angiographic imaging with ensemble neural networks to assess reperfusion status during mechanical thrombectomy. In: Drukker K, Mazurowski MA, eds. *Medical Imaging 2021: Computer-Aided Diagnosis*. United States: SPIE; 2021:48.. [Internet]. Online Only [cited 2024 Sep 24].

17. Nielsen M, Waldmann M, Frölich AM, Flottmann F, Hristova E, Bendszus M, et al. Deep Learning-Based automated thrombolysis in cerebral infarction scoring: a timely proof-of-principle study. *Stroke*. 2021;52(11):3497-3504.

18. Amukotuwa SA, Straka M, Dehkharghani S, Bammer R. Fast automatic detection of large vessel occlusions on CT angiography. *Stroke*. 2019;50(12):3431-3438.

19. Ghozy S, Azzam AY, Kallmes KM, Matsoukas S, Fifi JT, Luijten SPR, et al. The diagnostic performance of artificial intelligence algorithms for identifying M2 segment middle cerebral artery occlusions: a systematic review and meta-analysis. *J Neuroradiol*. 2023;50(4):449-454.

20. Almekhlafi MA, Mishra S, Desai JA, Nambiar V, Volny O, Goel A, et al. Not all "successful" angiographic reperfusion patients are an equal validation of a modified TICI scoring system. *Interv Neuroradiol*. 2014;20(1):21-27.

21. Su R, Cornelissen SAP, Van Der Sluijs M, Van Es ACGM, Van Zwam WH, Dippel DWJ, et al. autoTICI: automatic brain tissue reperfusion scoring on 2D DSA images of acute ischemic stroke patients. *IEEE Trans Med Imaging*. 2021;40(9):2380-2391.

22. Landis JR, Koch GG. The measurement of Observer agreement for categorical data. *Biometrics*. 1977;33(1):159.

23. Tung EL, McTaggart RA, Baird GL, Yaghi S, Hemendinger M, Dibiasio EL, et al. Rethinking thrombolysis in cerebral infarction 2b: which thrombolysis in cerebral infarction scales best define near complete recanalization in the modern thrombectomy era? *Stroke*. 2017;48(9):2488-2493.

24. Mittmann BJ, Braun M, Runck F, Schmitz B, Tran TN, Yamlahi A, et al. Deep learning-based classification of DSA image sequences of patients with acute ischemic stroke. *Int J CARS*. 2022;17(9):1633-1641.

25. Liebeskind DS, Bracard S, Guillemin F, Jahan R, Jovin TG, Majoie CB, et al. eTICI reperfusion: defining success in endovascular stroke therapy. *J NeuroInterv Surg*. 2019;11(5):433-438.

26. Mujanovic A, Kurmann CC, Manhart M, Piechowiak EI, Pilgram-Pastor SM, Serrallach BL, et al. Value of immediate flat panel perfusion imaging after endovascular therapy (AFTERMATH): a proof of concept study. *AJNR Am J Neuroradiol*. 2024;45(2):163-170.

27. Larson J, Mattu S, Kirchner L, Angwin J. How we analyzed the COMPAS recidivism algorithm. *Propublica*. 2016.

28. Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, et al. TRIPOD + AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. *BMJ*. 2024e078378.

29. Zarzour A, Batot C, Boisseau W, Cho TH, Guillon B, Richard S, et al. Tenecteplase versus Alteplase before thrombectomy: a comprehensive evaluation of clinical and angiographic impact: insights from the ETIS registry. *J Neuroradiol*. 2024;51(4):101189.