# Trustworthy AI for stage IV non-small cell lung cancer: automatic segmentation and uncertainty quantification

Sacha Dedeken, Pierre-Henri Conze, Vera Damerjian Pieters, Olivier Gallinato, Jérôme Faure, Thierry Colin, Dimitris Visvikis

<figure><img src="image_1.png" /></figure>

## To cite this version:

Sacha Dedeken, Pierre-Henri Conze, Vera Damerjian Pieters, Olivier Gallinato, Jérôme Faure, et al.. Trustworthy AI for stage IV non-small cell lung cancer: automatic segmentation and uncertainty quantification. Computerized Medical Imaging and Graphics, 2025, 123 (July 2025), pp.102567. 10.1016/j.compmedimag.2025.102567. hal-05049800

HAL Id: hal-05049800

https://imt-atlantique.hal.science/hal-05049800v1

Submitted on 12 Aug 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

# Trustworthy AI for stage IV non-small cell lung cancer: Automatic segmentation and uncertainty quantification

Sacha Dedeken<sup>a,b,c</sup>, Pierre-Henri Conze<sup>a,d,e</sup>,*, Vera Damerjian Pieters<sup>c</sup>, Olivier Gallinato<sup>c</sup>, Jérôme Faure<sup>c</sup>, Thierry Colin<sup>c</sup>, Dimitris Visvikis<sup>a,e</sup>

## ARTICLE INFO

### Keywords:

- Deep learning

- Oncology

- Stage IV NSCLC

- Tumor segmentation

- Uncertainty estimation

## ABSTRACT

Accurate segmentation of lung tumors is essential for advancing personalized medicine in non-small cell lung cancer (NSCLC). However, stage IV NSCLC presents significant challenges due to heterogeneous tumor morphology and the presence of associated conditions including infection, atelectasis and pleural effusion. The complexity of multicentric datasets further complicates robust segmentation across diverse clinical settings. In this study, we evaluate deep-learning-based approaches for automated segmentation of advanced-stage lung tumors using 3D architectures on 387 CT scans from the Deep-Lung-IV study. Through comprehensive experiments, we assess the impact of model design, HU windowing, and dataset size on delineation performance, providing practical guidelines for robust implementation. Additionally, we propose a confidence score using deep ensembles to quantify prediction uncertainty and automate the identification of complex cases that require further review. Our results demonstrate the potential of attention-based architectures and specific preprocessing strategies to improve segmentation quality in such a challenging clinical scenario, while emphasizing the importance of uncertainty estimation to build trustworthy AI systems in medical imaging. Code is available at: https://github.com/Sacha-Dedeken/SegStageIVNSCLC.

## 1. Introduction

### 1.1. Overview

Lung cancer constitutes a major public health burden. In 2020, lung cancer was estimated to be responsible for ~2.2 million new cancer diagnoses worldwide and constituted the leading cause of cancer-related mortality with ~1.8 million lung-cancer-related deaths (Travis et al., 2015). Among the different types of lung cancer, non-small cell lung cancer (NSCLC) is by far the most frequent, representing ~85% of total lung cancer cases. Within NSCLC, cases are further classified along three main categories according to WHO: adenocarcinoma, squamous cell carcinoma, and large cell carcinoma (Travis et al., 2015). The overall prognosis of lung cancer remains bleak, with an estimated five-year overall survival rate after diagnosis of ~20% only (Duma et al., 2019). For NSCLC specifically, five-year survival is estimated at ~25% across all stages, but less than 10% for stage IV disease (the most advanced stage) (Anon, 2000). For advanced lung cancer patients not eligible for targeted therapies, the main treatment options are chemotherapy and

immunotherapy. Immunotherapy has dramatically altered the treatment landscape of stage IV NSCLC these past years. However, while immunotherapy offers significant benefits, only a subset of patients experience durable responses, whereas all are exposed to potentially severe side effects. Additionally, immune checkpoint inhibitors are particularly expensive and constitute a significant financial burden for the healthcare systems. Developing reliable biomarkers to predict treatment response is therefore crucial to allow significant progress towards truly personalized medicine approach. Consequently, the best available treatment could be offered to a specific patient based on his or her biological characteristics.

Imaging plays a critical role throughout the patient care pathway. CT scans are utilized for initial diagnosis, assessment of treatment response, and longitudinal follow-up. The signal in CT imaging, directly linked to tissue density, provides valuable morphological information about tumor characteristics. Radiomics has emerged as a tool for quantifying solid tumor phenotypes through the extraction of quantitative radiographic features (Gillies et al., 2016). There is a growing body of

<footnote><sup>a</sup> LaTIM UMR 1101, Inserm, Brest, France</footnote>

<footnote><sup>b</sup> University of Western Brittany, Brest, France</footnote>

<footnote><sup>c</sup> SOPHiA GENETICS, Pessac, France</footnote>

<footnote><sup>d</sup> IMT Atlantique, Brest, France</footnote>

<footnote>* Corresponding author at: IMT Atlantique, Brest, France.
E-mail address: pierre-henri.conze@imt-atlantique.fr (P.-H. Conze).</footnote>
evidence pointing to the prognostic value of such features as well as their utility in stratifying patients (Bortolotto et al., 2021). Traditionally, radiomics has focused on extracting hand-crafted imaging features explicitly designed to capture specific patterns in medical images. However, recent advancements have shifted towards deep learning approaches, particularly convolutional neural networks (CNNs), which automatically learn representative multi-scale features directly from the data (Chen et al., 2021).

There is a growing need for robust and reproducible radiomics markers to improve multimodal analyses aimed at discovering predictive biomarkers, particularly concerning treatment responses in patients with metastatic NSCLC. Achieving this requires accurate lung tumor segmentation methods that can perform reliably across heterogeneous, multicentric datasets and in advanced disease settings. However, certain cases remain challenging to delineate due to imaging artifacts or comorbidities. In this context, incorporating a confidence score into the segmentation process is essential to automatically detect uncertain cases and flag them for further review, thereby ensuring more reliable downstream analyses.

## 1.2. Related work

Medical image segmentation has evolved significantly over the years. Classical approaches, including thresholding and region growing, as well as model-based methods, such as active shape and active contour models, have been widely used (Purushwalkam et al., 2013; Van Ginneken et al., 2002; Conze et al., 2017). These techniques perform well for simple structures but struggle with complex anatomical variability and noise. The introduction of deep learning in automated image analysis marked a turning point for all data modalities (Suzuki, 2017; Iqbal et al., 2021, 2025), with the U-Net architecture becoming foundational in image segmentation tasks (Ronneberger et al., 2015). The U-Net's encoder-decoder structure with skip connections is effective even with limited training data (Ronneberger et al., 2015). Numerous U-Net variations have followed, including 3D U-Net which considers volumetric data and Attention U-Net (Oktay et al., 2018) which incorporates attention mechanisms. Recent advancements involving Transformers (Hatamizadeh et al., 2021) or frameworks such as nn-UNet (Isensee et al., 2018) further refine segmentation pipelines through improved architectures and pre/post-processing strategies (Conze et al., 2023).

Segmenting NSCLC tumors from 3D CT scans poses unique challenges. Lung cancer segmentation faces complexity due to intricate tumor morphology, significant inter-patient variability, poor contrast with adjacent normal tissues (especially in the mediastinum and the diaphragm), and the presence of medical conditions with similar densities (e.g. atelectasis, inflammation) (Wang et al., 2023). Early efforts relied on 2D methods limiting global context capture. Modern pipelines can use a 2D slice-by-slice segmentation before aggregation to obtain volumes or directly employ a patch-based approach (Primakov et al., 2022; Jiang et al., 2019). Preprocessing steps such as lung cropping and intensity normalization are crucial to handle large CT volumes efficiently while preserving relevant anatomical information. These methods have significantly improved tumor segmentation accuracy, which is essential for both diagnostic and prognostic purposes. However, most studies focus on early-stage (I-III) datasets, which are easier to segment and directly relevant for radiotherapy planning (Wang et al., 2024). To our knowledge, there remains a notable gap in research addressing the specific challenges of segmenting advanced (stage IV) NSCLC tumors.

Moreover, uncertainty quantification is critical in medical imaging, as model reliability plays a key role in supporting clinical decision-making. By identifying cases with uncertain predictions, related methods help clinicians prioritize reviews and mitigate risks associated with over-confidence in erroneous outputs (Lambert et al., 2022). Calibration techniques align predicted probabilities with actual outcomes. For

instance, if a tumor segmentation model assigns a probability of 0.3 to 100 voxels, it is well calibrated if 30 of those voxels truly belong to the tumor. This ensures that the model's probability outputs accurately reflect prediction reliability (Nixon et al., 2020). For its part, Monte Carlo dropout offers a practical Bayesian approximation by applying dropout during inference, thereby generating multiple predictions per input to estimate variability and thus uncertainty (Gal and Ghahramani, 2016; Wickstrøm et al., 2020). Another approach, deep ensembles, involves training multiple models with different initializations, using the variability in their predictions to capture uncertainty. This is particularly useful for classification tasks (Lakshminarayanan et al., 2017). Deep ensembles are simple, scalable and have been used in various contexts including PET scan segmentation (Arabi and Zaidi, 2022). However, most existing approaches focus on pixel-wise uncertainty, therefore estimating confidence for individual voxels. Evaluating uncertainty at the image level would more directly impact high-level clinical decisions and patient management.

## 1.3. Contributions

This study addresses key challenges in the delineation of advanced-stage NSCLC tumors from heterogeneous multicentric CT datasets. Our main contributions are:

1. **Stage IV Tumor Segmentation:** We propose a robust 3D segmentation approach adapted to stage IV NSCLC tumors, achieving strong performance despite the complexity of the dataset.

2. **Guidelines for Model Design:** Through extensive experiments, we provide practical guidelines for architectural choices, HU windowing, and dataset size to optimize segmentation performance.

3. **Confidence Score Development:** We introduce a new confidence score, which allows an automatic ranking of the reliability of predicted delineations. This, in turn, helps identify complex cases where model predictions require further review, thereby improving trustworthiness in clinical workflows.

The source code for our experiments, including data preprocessing, model training, and evaluation scripts, is available at https://github.com/Sacha-Dedeken/SegStageIVNSCLC. This ensures transparency and allows other researchers to reproduce our results and build upon our work.

# 2. Materials and methods

## 2.1. Dataset

The data used in this work originates from Deep-Lung-IV (Sophia Genetics SAS, 2024), a multicenter observational cohort study, both retrospective and prospective, focusing on patients with stage IV NSCLC treated in the first-line setting. The study, initiated in January 2020 and whose progressive data collection ended in September 2024, aims originally to enroll 4000 patients across ~30 investigational sites around the world. Patients are treated with one of three treatments: pembrolizumab monotherapy (Cohort A), a combination of chemotherapy and pembrolizumab (Cohort B), or a chemotherapy doublet (Cohort C).

In this paper, we focus on a subset of Deep-Lung-IV with 387 patients, 218 from treatment arm B and 169 from C. These images have available ground truth tumor delineations performed by internal experts then validated and corrected by two radiologists with more than 10 years of experience. Data is acquired from 13 European centers using 46 different device models under various acquisition conditions. The slice thickness ranges from 0.6 to 3.0 mm. All patients received contrast injections, with 269 images acquired at portal phase and 118 images at arterial phase. Some scans cover the full body, some include the lungs and the abdomen, while others are limited to the lung area only. An
<table><caption>**Table 1**
Baseline characteristics of the patient cohort.</caption><thead><tr><td>Characteristic</td><td>Value</td></tr></thead><tbody><tr><td>Gender</td><td>71% male</td></tr><tr><td>Smoking status</td><td>55% current, 40% former</td></tr><tr><td>Stage at diagnosis</td><td>91% stage IV</td></tr><tr><td>Subtype</td><td>67% stage IVb</td></tr><tr><td>ECOG performance status</td><td>24% ECOG 0, 56% ECOG 1, 17% ECOG 2, 2% ECOG 3</td></tr><tr><td>Histopathology</td><td>75% adenocarcinoma, 21% squamous</td></tr><tr><td>Progression-Free Survival (PFS)</td><td>8 months (mean), 4.8 months (median)</td></tr><tr><td>Overall Survival (OS)</td><td>14 months (mean), 9.5 months (median)</td></tr><tr><td>PD-L1 expression</td><td>88% with PD-L1 &lt;50%</td></tr><tr><td>Age</td><td>64 years (mean)</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>**Fig. 1.** Illustration of advanced lung cancer heterogeneity on CT scan: (a) large mass in lung, (b) multiple lesions in lung, (c) atelectasis and pleural effusion, (d) mediastinal lesion, (e) small lesions with metastasis in liver.</figcaption></figure>

overview of patient baseline characteristics is provided in Table 1. A list of the different hospitals and device models included in the study is presented in Appendix Table A.1.

Stage IV NSCLC is characterized by extreme heterogeneity in tumor size, shape, number, location, vascularization, and the presence of associated conditions such as infection, atelectasis and/or pleural effusion. In some cases, reliable ground-truth tumor segmentation is not feasible due to this complexity. Given the variability in imaging protocols, conventional harmonization methods, such as ComBat Fortin et al. (2017), are inadequate, since overlap in categories complicates harmonization. Individual hospitals may use multiple devices, some of which might be similar to those in other hospitals. However, even if those devices are identical, their settings can greatly vary across sites.

The aforementioned complexity conditions make a fully accurate segmentation algorithm extremely challenging. Therefore, we identify the need for a robust algorithm that is able to provide reliable and exploitable lung tumor delimitations in most cases. It can also supply an automatic ranking regarding the segmentations' reliability where inaccurate model predictions are identified.

## 2.2. Segmentation methods

**Preprocessing.** All images selected for the study were acquired using a soft reconstruction filter. They were systematically reoriented to the axial plane and cropped around the lungs using an automatic deep-learning-based lung detector previously developed by our team. The choice to use a bounding box around the lung segmentation rather than inside the lungs was motivated by the tendency of tumors to extend into the mediastinum or beyond the lungs. Following bounding-box cropping, the images were resized to a standardized resolution of 128³ voxels.

To ensure consistency in intensity values, the images were clipped using the abdominal window, with Hounsfield units (HU) restricted to

the range [−100, 220]. Voxel intensities outside of these upper and lower limits were assigned the value of the closest limit. Finally, intensity normalization was applied, thereby scaling the pixel values between 0 and 1.

**Segmentation networks.** Three network architectures were investigated: U-Net (Ronneberger et al., 2015), Attention U-Net (Oktay et al., 2018), and UNET Transformer (UNETR) (Hatamizadeh et al., 2021). U-Net is the core CNN for biomedical image segmentation, featuring an encoder-decoder structure with skip connections that preserve spatial details by linking corresponding layers (Ronneberger et al., 2015). The Attention U-Net builds on this design by adding attention gates, which enhance focus on relevant regions, improving segmentation accuracy for complex and variable structures (Oktay et al., 2018). In contrast, UNETR replaces the CNN encoder with a Transformer-based architecture, leveraging self-attention mechanisms to capture global context (Hatamizadeh et al., 2021). We also investigated the nnU-Net framework, which automatically configures preprocessing, architecture, and training strategies to optimize performance across diverse datasets (Isensee et al., 2018). These models have been widely used in the literature, with no single architecture consistently outperforming others across all medical imaging tasks and datasets (Armand et al., 2024).

## 2.3. Confidence score

In specific cases, lung tumor delineation may remain inaccurate due to the inherent complexity of the images. The tumor can be small or hard to segment due to atelectasis, pleural effusion or acquisition artifacts (Fig. 1). For these images, even the concept of a ground truth becomes questionable, as healthcare professionals are often unable to confidently delineate which regions of the image correspond to tumor tissue. To develop a more robust segmentation model, a useful
approach is to associate predictions with confidence scores. The higher the scores, the more accurate are the segmentations. On the other hand, lower scores indicate increased uncertainty about the segmentation quality. In contrast to approaches that provide voxel-wise uncertainty metrics, we propose to estimate segmentation uncertainty at the image scale by generating slight variations of the same model. This is performed by altering the random initialization of weights when starting the training.

N versions of the model $\phi_n(\cdot)$ are trained using identical hyperparameters but with different seeds for data shuffling and weight initialization. For a given image $I$, these models generate N independent predictions. The confidence score $conf(I)$ is computed by averaging the Dice similarity coefficients between all $\binom{N}{2}$ unique pairs of predicted segmentations:

$$conf(I) = \frac{1}{\binom{N}{2}} \sum_{I}^{N} \sum_{J \neq I}^{N} \text{Dice}(\phi_{I}(I), \phi_{J}(I)) \quad (1)$$

The score ranges from 0 to 1, with values near 1 indicating high consistency among predictions and values near 0 reflecting significant variability, suggesting uncertainty and low confidence in the segmentation.

The final segmentation is the pixel-wise majority vote of the N segmentations. In other words, for a given voxel if more than half of the segmentations predict a true value, then the voxel belongs to a tumor.

# 3. Experiments and results

## 3.1. Training procedure

For the task of segmenting stage IV lung tumors, we used the 3D CT scans, cropped and preprocessed as described in Section 2.2. Unlike many state-of-the-art convolutional neural network (CNN) frameworks, which often utilize 2D slices or smaller 3D patches, our 3D approach imposes significant memory constraints, requiring input downsampling. However, we argue that this design choice allows the model to capture necessary semantic and spatial context, as stage IV tumors can be highly scattered throughout the lung tissue. By maintaining the full 3D structure, the model can better identify and localize these diffuse tumor regions.

To train the model and evaluate its performance, we employed a 5-fold cross-validation strategy. In each fold, 20% of the full dataset (387 patients) was reserved as unseen test set (77 patients). The remaining 80% of the dataset was further split, with 20% used as a validation set (62 patients) and the remaining 80% as a training set (248 patients).

After achieving satisfactory Dice results during training, we adopted a partial grid search strategy for hyperparameter optimization. Indeed, a full exhaustive grid search was impractical due to time and computational resource constraints. Starting from a baseline set of hyperparameters, we systematically explored variations by selecting possible values for each hyperparameter. Once the most optimal value for one hyperparameter was identified, it was fixed as the baseline for exploring the next hyperparameter, and so on. This process produced Table 2, where the hyperparameters are outlined in order of exploration along with their corresponding search ranges.

Following this hyperparameter tuning process, the final optimized model chosen for our experiments was an Attention U-Net architecture, using instance normalization and a dropout rate of 0.2. The model was trained with a weight decay of $10^{-3}$, using Adam Kingma and Ba (2017) with an initial learning rate of $10^{-4}$, and a batch size of 2. This batch size was the maximum we could use with the given input size and memory constraints. The loss function employed was the DiceFocal loss, particularly effective for segmentation tasks and more robust than non-compound losses (Ma et al., 2021). The chosen input data shape was $128^3$ voxels. The larger shape of $160^3$ did not show any increase in performance while higher sizes could not fit in memory. The cross-validation results on the test set are shown in Fig. 2, and an example

<table><caption>**Table 2**
Hyperparameters and corresponding value ranges for Attention U-Net Oktay et al. (2018).</caption><thead><tr><th>Constant parameter</th><th>Value range</th></tr></thead><tbody><tr><td>Batch size</td><td>1, 2</td></tr><tr><td>Interlayer normalization</td><td>Batch Normalization (Ioffe and Szegedy, 2015), Instance Normalization (Wu and He, 2018)</td></tr><tr><td>Accumulation steps</td><td>1, 4, 16</td></tr><tr><td>Loss function</td><td>Dice, Focal, DiceFocal</td></tr><tr><td>Input size</td><td>$128^3$, $160^3$</td></tr><tr><td>Weight decay</td><td>0.0, $10^{-3}$, $10^{-2}$, $10^{-1}$</td></tr><tr><td>Learning rate</td><td>$10^{-5}$, $10^{-4}$, $10^{-3}$, $10^{-2}$</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>**Fig. 2.** Dice score on test set of the best model following partial grid search.</figcaption></figure>

of the model's segmentation compared to ground-truth delineation are illustrated in Fig. 3.

Experiments were performed on Azure Machine Learning using an Nvidia Tesla M60 GPU, 112 GB of RAM, and PyTorch v2.2.2. Each cross-validation fold required around 19 h to complete, with a maximum of 200 epochs.

## 3.2. Ablation study

Following the partial grid search to optimize the Attention U-Net hyperparameters, we conducted additional experiments to evaluate the influence of different types of windowing. While the advantages and limitations of narrow abdominal windowing are discussed in Section 4, we present in this section two other clipping possibilities. We therefore trained the model in the same way as described above, while considering 3 clipping ranges: abdominal windowing [−100;220], mediastinal windowing [−1350;150], no windowing. The results are displayed in Fig. 4.

We also evaluated the Attention U-Net's performance against two widely used architectures and a baseline framework. First, we trained a classical U-Net using the same cross-validation setup to assess the impact of attention gates. Second, we trained the UNETR model as a Transformer-based alternative, enabling comparison of global context modeling capabilities. Finally, we deployed the nnU-Net framework (Isensee et al., 2021) that selects custom preprocessing and a self-configuring architecture based on 3D patches followed by a sliding window with overlap during inference. Results are displayed in Fig. 5.

We also assessed the impact of the size of the training dataset on the model performance. A 5-fold cross-validation was performed using 5 different training set sizes: 50, 100, 150, 200, and 250. Results are presented in Fig. 6.

Overall, the aim of these experiments was to compare common alternatives in lung tumor delineation literature, namely preprocessing,
<figure><img src="image_7.png" /><figcaption>Fig. 3. Illustration of segmentation results, each column corresponding to a different patient case: (Left) a large tumor invading the mediastinum, (Middle) a tumor with three connected components partially invading the pleura, and (Right) a challenging case with a proximal tumor causing atelectasis due to bronchial obstruction, where the algorithm captures a large portion of the atelectasis. The first row shows the 3D view of the model segmentation, the second row presents the 3D ground-truth delineations, and the third row displays a 2D slice with both segmentations for comparison.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 4. Influence of clip choice (windowing) on test Dice score.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 5. Influence of model architecture on test Dice score.</figcaption></figure>

windowing and model architecture. Varying the size of the training dataset allows us to evaluate the minimum number of images to be considered in order to obtain satisfying performances. In conclusion, the hyperparameters selected in Section 3.1 and the performances of the final model (Fig. 2) were indeed optimal among the variations explored.

### 3.3. Confidence score

We evaluated the proposed confidence score on the same dataset of 387 patients used in the previous experiments and described in Section 2.1. The confidence score was computed using predictions from the Deep Ensemble of the five model versions, as discussed previously. The

Dice similarity coefficient was also calculated between the individual model predictions and the ground truth on the respective test sets. The Pearson correlation coefficient between the confidence score and the Dice coefficient was 0.86, indicating a strong linear relationship. Given a Dice standard deviation of 0.19, we can predict the Dice using the confidence score with a root mean square error (RMSE) below 0.1 (Fig. 7). This opened up the possibility to clearly distinguish reliable delineations from uncertain ones that needed further review. We also implemented Monte Carlo Dropout (MCDO) as an alternative way to compute this uncertainty by implementing 5 forward passes with dropout during inference, and then computing the confidence score as shown in Eq. (1). Applying this method using MCDO produced a Pearson correlation coefficient with a Dice coefficient of 0.77, which
<figure><img src="image_10.png" /><figcaption>Fig. 6. Influence of train set size on test Dice score.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 7. Correlation between confidence score and Dice.</figcaption></figure>

was satisfying but below our results, thereby suggesting that the Deep Ensemble method was more reliable.

We note that lower confidence scores were observed in challenging cases like small tumor volumes, high number of lesions, or the presence of infection, pleural effusion, and/or atelectasis (Fig. 8, 9, 10), all of which are known to complicate the tumor delineation task. These findings support the utility of the confidence score in assessing model reliability.

## 4. Discussion

This study aimed to evaluate the feasibility of achieving high-performance tumor segmentation using supervised learning on multicentric 3D CT scans of metastatic NSCLC. The cross-validated results of the final model using majority voting were promising, with a mean (±std) test Dice score of 0.76 (±0.20) and the first quartile at 0.7. The model performances across subsets (scores by gender in Table A.2, by smoking status in Table A.3, and by histology type in Table A.4) suggest that the model is reliable in all kind of setting, but variability can be hard to explain because it is always multifactorial and some subsets have a very low number of samples. Lower Dice scores were observed in cases with small tumor volumes or complex scenarios such as atelectasis

and pleural effusion. We conducted a partial grid search to benchmark various hyperparameters, assessing their impact on both performance and training duration. This process allowed us to identify the optimal model configuration within the search space.

Inside the ablation study, we fixed our optimal hyperparameter choices and compared three architectures: Attention U-Net, U-Net, and UNETR. Attention U-Net demonstrated the highest performance with a mean (± std) Dice score of 0.76 (±0.20), followed by U-Net at 0.66 (±0.23), and UNETR at 0.60 (±0.23). U-Net performed well despite the dataset's heterogeneity, but attention gates in the Attention U-Net significantly improved segmentation by focusing on tumor regions. UNETR's lower performance may have resulted from its higher parameter count, requiring larger datasets or fine-tuning to realize its potential. In addition to the previously mentioned architectures, we applied the nnUNet framework for further comparison. With its customized preprocessing and 3D patch approach, nnUnet obtained a mean Dice score of 0.73 (±0.22), which was better than the classic U-Net, but not as performant as Attention U-Net.

We also evaluated the impact of windowing during preprocessing: no windowing, large mediastinal windowing and narrow abdominal windowing. While the abdominal window excludes ground-glass opacity regions, it improves precision for segmenting the solid components of the tumor. This approach would be inadequate in the context of lung cancer screening for early diagnosis. However, for advanced-stage disease, it is sufficient to extract the most relevant information. Note that the ground-truth tumor delineations were determined by experts with the clinical a priori of focusing on solid parts for what concerns stage IV NSCLC. This includes the tumor, adenopathy and pleural metastasis inside the whole thorax region.

The influence of the size of training data on performance was also examined. Results indicated that increasing the dataset size enhanced the mean (± std) Dice score from 0.64 (±0.24) to 0.76 (±0.20). The results obtained with just 50 training samples showed the impressive ability of UNet architectures to learn even with a very small amount of data. The performance showed an asymptotic improvement as the dataset size approached approximately 200 images. Beyond this threshold, performance gains became less pronounced. Nevertheless, additional data would still capture new details that were not directly reflected in the global Dice metric. Indeed, this could be reflected in other aspects of the segmentation quality, such as a better detection of multiple lesions or a higher performance in rare or unusual cases.

The confidence score from the Deep Ensembles method does more than assess segmentation stability. It effectively predicts cases where the segmentation is likely to be accurate or at risk of failure. This score has potential practical applications in clinical workflows. For instance, a high confidence threshold could be used to select images with reliable segmentations, thereby ensuring robust results in automated pipelines. Conversely, a low confidence threshold could serve to alert users or to demand additional verification through human review. This enhances safety and reliability in largely automated processes.

**Limitations.** This study has several limitations that highlight opportunities for future research.

The number of patients with available ground-truth is dependent to time and cost constraints, but the rest of the patients has unlabeled images that could be use for semi-supervised setting as a pre-training pre-text task. Since ground truth preparation is time-consuming and expensive, semi-supervised approaches could offer a more resource-efficient alternative.

The grid search for hyperparameter optimization was limited due to time and resource constraints. Moreover, model training, validation, and evaluation relied on the mean Dice-Sørensen coefficient, a standard metric in medical segmentation. However, this metric can be influenced by the overall volume of the segmented region and may hide performance variations related to tumor size or shape.

Furthermore, due to abdominal windowing, some details, such as lymphatic infiltration, which may be important for identifying relapse,
<figure><img src="image_12.png" /><figcaption>Fig. 8. Distribution of confidence scores by volumes.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 9. Distribution of confidence scores by number of lesions.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Fig. 10. Distribution of confidence scores for images with or without pleural effusion.</figcaption></figure>

could be missed. Nevertheless, this trade-off was made to address the challenge of handling highly heterogeneous phenotypes with the clinical a priori of focusing on solid parts only, as mentioned previously (Fig. 1).

Finally, low confidence scores may arise from two distinct scenarios: uncertainty about including specific regions or failure to detect any lesions. While the former case totally corresponds to a low confidence

score, the latter might refer to the absence of lesions in the image or the presence of little ones, undetectable by the model. The lack of tumors is possible in metastatic NSCLC and should not correspond to low confidence scores. Future studies should therefore investigate these scenarios separately.

## 5. Conclusion

Lung cancer remains a major public health challenge with advanced NSCLC presenting significant difficulties due to its heterogeneous morphology and poor prognosis. Accurate tumor segmentation from CT scans is essential for extracting reliable imaging biomarkers, which, in turn, can support clinical decision-making and personalized treatment strategies. The insights we provide in this work contribute to the growing body of knowledge on deep-learning-based tumor segmentation in lung CT scans, particularly for stage IV NSCLC patients. Through extensive experiments on hyperparameters, architectures, and dataset sizes, we provide practical guidelines to enhance the efficiency, robustness, and time management of future research in this field. Additionally, we apply a simple yet effective Deep Ensembles approach for measuring uncertainty at the image level using a confidence score. The next step is to deploy these algorithms in the Deep-Lung-IV clinical study involving several thousand patients worldwide and to identify multimodal biomarkers predictive of treatment response. We are currently conducting investigations to assess the performance of the segmentation model on stage III NSCLC patients undergoing radiotherapy and chemotherapy to evaluate its generalizability across different disease stages. This work aligns with the need for developing trustworthy AI algorithms, offering insights and tools to accelerate the integration of AI in medical imaging.
<caption>**Table A.1**</caption>

Summary of hospital contributions to the dataset, including number of patients, imaging devices, segmentation performance (mean test Dice score), and device usage metrics (mean and max numbers of patients by device).

<table><caption>**Table A.2**</caption><thead><tr><th>Hospital name</th><th>Patients</th><th>Devices</th><th>Mean Dice</th><th>Patients/Device</th><th>Max Patients/Device</th></tr></thead><tbody><tr><td>CHU Lyon</td><td>172</td><td>26</td><td>0.7642</td><td>6.62</td><td>36</td></tr><tr><td>Hôpital Tenon - APHP</td><td>48</td><td>10</td><td>0.7270</td><td>4.80</td><td>31</td></tr><tr><td>Puerta de Hierro Hospital</td><td>41</td><td>5</td><td>0.7854</td><td>8.20</td><td>32</td></tr><tr><td>CHU Bordeaux</td><td>34</td><td>9</td><td>0.7975</td><td>3.78</td><td>13</td></tr><tr><td>Hôpital Foch</td><td>23</td><td>8</td><td>0.7517</td><td>2.88</td><td>14</td></tr><tr><td>Ambroise Paré - APHP</td><td>17</td><td>2</td><td>0.8282</td><td>8.50</td><td>13</td></tr><tr><td>CH Toulon</td><td>15</td><td>2</td><td>0.7052</td><td>7.50</td><td>14</td></tr><tr><td>University of Leipzig</td><td>10</td><td>4</td><td>0.8131</td><td>2.50</td><td>5</td></tr><tr><td>Shaare Zedek Medical Center</td><td>7</td><td>2</td><td>0.6474</td><td>3.50</td><td>5</td></tr><tr><td>La Pitié Salpêtrière</td><td>6</td><td>4</td><td>0.8450</td><td>1.50</td><td>2</td></tr><tr><td>Hôpital Avicenne - APHP</td><td>5</td><td>3</td><td>0.8493</td><td>1.67</td><td>2</td></tr><tr><td>CHU Nantes</td><td>4</td><td>4</td><td>0.8340</td><td>1.00</td><td>1</td></tr><tr><td>IVO</td><td>2</td><td>2</td><td>0.7988</td><td>1.00</td><td>1</td></tr></tbody></table>

Mean Dice score by gender.

<table><caption>**Table A.3**</caption><thead><tr><th>Gender</th><th>Mean DSC</th><th>Patients</th></tr></thead><tbody><tr><td>Male</td><td>0.7638</td><td>278</td></tr><tr><td>Female</td><td>0.7748</td><td>109</td></tr></tbody></table>

Mean Dice scores by smoking status.

<table><caption>**Table A.4**</caption><thead><tr><th>Smoking Status</th><th>Mean DSC</th><th>Patients</th></tr></thead><tbody><tr><td>Current smoker</td><td>0.7776</td><td>209</td></tr><tr><td>Former smoker</td><td>0.7695</td><td>156</td></tr><tr><td>Never smoker</td><td>0.6435</td><td>19</td></tr><tr><td>Unknown</td><td>0.6737</td><td>3</td></tr></tbody></table>

Mean Dice scores by histopathology type at diagnosis.

<table><thead><tr><th>Histopathology type</th><th>Mean Dice</th><th>Patients</th></tr></thead><tbody><tr><td>Adenocarcinoma</td><td>0.7594</td><td>278</td></tr><tr><td>Squamous cell/epidermoid</td><td>0.7793</td><td>79</td></tr><tr><td>Unknown/Not available</td><td>0.7425</td><td>17</td></tr><tr><td>Large cell carcinoma</td><td>0.8795</td><td>8</td></tr><tr><td>Other</td><td>0.8898</td><td>4</td></tr><tr><td>Sarcomatoid</td><td>0.8817</td><td>1</td></tr></tbody></table>

### CRediT authorship contribution statement

**Sacha Dedeken:** Writing – original draft, Methodology, Investigation, Formal analysis, Data curation. **Pierre-Henri Conze:** Writing – original draft, Supervision, Methodology, Investigation, Conceptualization. **Vera Damerjian Pieters:** Writing – original draft, Supervision, Methodology, Investigation, Conceptualization. **Olivier Gallinato:** Supervision, Project administration, Funding acquisition, Conceptualization. **Jérôme Faure:** Software, Resources, Methodology, Investigation. **Thierry Colin:** Supervision, Project administration, Investigation, Funding acquisition, Data curation, Conceptualization. **Dimitris Visvikis:** Supervision, Project administration, Funding acquisition, Conceptualization.

### Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

### Acknowledgments

This work was partially supported by the industrial chair CYBAILE, partly funded by a European and Brittany region grant (reference 1FEDER-76568).

### Appendix

See Tables A.1–A.4.

### Data availability

The data that has been used is confidential.

### References

* Anon, 0000. SEER Cancer Statistics Review, 1975–2017, SEER URL https://seer.cancer.gov/csr/1975_2017/index.html.

* Arabi, H., Zaidi, H., 2022. Confidence score estimation in machine learning-based automated lesion segmentation from PET images. In: 2022 IEEE Nuclear Science Symposium and Medical Imaging Conference. NSS/MIC, pp. 1–3. http://dx.doi.org/10.1109/NSS/MIC44845.2022.10399079, ISSN: 2577-0829 URL https://ieeexplore.ieee.org/abstract/document/10399079.

* Armand, T.P.T., Bhattacharjee, S., Choi, H.-K., Kim, H.-C., 2024. Transformers effectiveness in medical image segmentation: A comparative analysis of unet-based architectures. In: 2024 International Conference on Artificial Intelligence in Information and Communication. ICAIIC, pp. 238–242. http://dx.doi.org/10.1109/ICAIIC60209.2024.10463435.

* Bortolotto, C., Lancia, A., Stelitano, C., Montesano, M., Merizzoli, E., Agustoni, F., Stella, G., Preda, L., Filippi, A.R., 2021. Radiomics features as predictive and prognostic biomarkers in NSCLC. Expert. Rev. Anticancer. Ther. 21 (3), 257–266. http://dx.doi.org/10.1080/14737140.2021.1852935, Publisher: Taylor & Francis URL https://www.tandfonline.com/doi/abs/10.1080/14737140.2021.1852935.

* Chen, W., Hou, X., Hu, Y., Huang, G., Ye, X., Nie, S., 2021. A deep learning- and CT image-based prognostic model for the prediction of survival in non-small cell lung cancer. Med. Phys. 48 (12), 7946–7958. http://dx.doi.org/10.1002/mp.15302, _epprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/mp.15302 URL https://onlinelibrary.wiley.com/doi/abs/10.1002/mp.15302.

* Conze, P.-H., Andrade-Miranda, G., Singh, V.K., Jaouen, V., Visvikis, D., 2023. Current and emerging trends in medical image segmentation with deep learning. IEEE Trans. Radiat. Plasma Med. Sci. 7 (6), 545–569. http://dx.doi.org/10.1109/TRPMS.2023.3265863, Conference Name: IEEE Transactions on Radiation and Plasma Medical Sciences URL https://ieeexplore.ieee.org/document/10098158.

* Conze, P.-H., Noblet, V., Rousseau, F., Heitz, F., de Blasi, V., Memeo, R., Pessaux, P., 2017. Scale-adaptive supervoxel-based random forests for liver tumor segmentation in dynamic contrast-enhanced CT scans. Int. J. Comput. Assist. Radiol. Surg. 12 (2), 223–233. http://dx.doi.org/10.1007/s11548-016-1493-1.

* Duma, N., Santana-Davila, R., Jr., Molina, 2019. Non-small cell lung cancer: Epidemiology, screening, diagnosis, and treatment. Mayo Clin. Proc. 94 (8), http://dx.doi.org/10.1016/j.mayocp.2019.01.013, Publisher: Mayo Clin Proc URL https://pubmed.ncbi.nlm.nih.gov/31378236/.

* Fortin, J.-P., Parker, D., Tunç, B., Watanabe, T., Elliott, M.A., Ruparel, K., Roalf, D.R., Satterthwaite, T.D., Gur, R.C., Gur, R.E., Schultz, R.T., Verma, R., Shinohara, R.T., 2017. Harmonization of multi-site diffusion tensor imaging data. NeuroImage 161, 149–170. http://dx.doi.org/10.1016/j.neuroimage.2017.08.047, URL https://www.sciencedirect.com/science/article/pii/S1053811917306948.

* Gal, Y., Ghahramani, Z., 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. http://dx.doi.org/10.48550/arXiv.1506.02142, arXiv URL http://arxiv.org/abs/1506.02142 [stat].

* Gillies, R.J., Kinahan, P.E., Hricak, H., 2016. Radiomics: Images are more than pictures, they are data. Radiology 278 (2), 563–577. http://dx.doi.org/10.1148/radiol.2015151169.
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H., Xu, D., 2021. UNETR: Transformers for 3D medical image segmentation. http://dx.doi.org/10.48550/arXiv.2103.10504, arXiv URL http://arxiv.org/abs/2103.10504.

Ioffe, S., Szegedy, C., 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. URL http://arxiv.org/abs/1502.03167 [cs].

Iqbal, I., Ullah, I., Peng, T., Wang, W., Ma, N., 2025. An end-to-end deep convolutional neural network-based data-driven fusion framework for identification of human induced pluripotent stem cell-derived endothelial cells in photomicrographs. Eng. Appl. Artif. Intell. 139, 109573. http://dx.doi.org/10.1016/j.engappai.2024.109573, URL https://www.sciencedirect.com/science/article/pii/S0952197624017317.

Iqbal, I., Younus, M., Walayat, K., Kakar, M.U., Ma, J., 2021. Automated multi-class classification of skin lesions through deep convolutional neural network with dermoscopic images. Comput. Med. Imaging Graph. 88, 101843. http://dx.doi.org/10.1016/j.compmedimag.2020.101843, URL https://www.sciencedirect.com/science/article/pii/S0895611120301385.

Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H., 2021. Nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18 (2), 203–211. http://dx.doi.org/10.1038/s41592-020-01008-z, Publisher: Nature Publishing Group URL https://www.nature.com/articles/s41592-020-01008-z.

Isensee, F., Petersen, J., Klein, A., Zimmerer, D., Jaeger, P.F., Kohl, S., Wasserthal, J., Koehler, G., Norajitra, T., Wirkert, S., Maier-Hein, K.H., 2018. Nnu-net: Self-adapting framework for U-net-based medical image segmentation. arXiv URL http://arxiv.org/abs/1809.10486.

Jiang, J., Hu, Y.-C., Liu, C.-J., Halpenny, D., Hellmann, M.D., Deasy, J.O., Mageras, G., Veeraraghavan, H., 2019. Multiple resolution residually connected feature streams for automatic lung tumor segmentation from CT images. IEEE Trans. Med. Imaging 38 (1), 134–144. http://dx.doi.org/10.1109/TMI.2018.2857800, Conference Name: IEEE Transactions on Medical Imaging URL https://ieeexplore.ieee.org/document/8417454.

Kingma, D.P., Ba, J., 2017. Adam: A method for stochastic optimization. http://dx.doi.org/10.48550/arXiv.1412.6980, arXiv URL http://arxiv.org/abs/1412.6980 [cs].

Lakshminarayanan, B., Pritzel, A., Blundell, C., 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. http://dx.doi.org/10.48550/arXiv.1612.01474, arXiv URL http://arxiv.org/abs/1612.01474 [stat].

Lambert, B., Forbes, F., Tucholka, A., Doyle, S., Dehaene, H., Dojat, M., 2022. Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis. http://dx.doi.org/10.48550/arXiv.2210.03736, arXiv URL http://arxiv.org/abs/2210.03736 [cs, eess].

Ma, J., Chen, J., Ng, M., Huang, R., Li, Y., Li, C., Yang, X., Martel, A.L., 2021. Loss odyssey in medical image segmentation. Med. Image Anal. 71, 102035. http://dx.doi.org/10.1016/j.media.2021.102035, URL https://www.sciencedirect.com/science/article/pii/S1361841521000815.

Nixon, J., Dusenberry, M., Jerfel, G., Nguyen, T., Liu, J., Zhang, L., Tran, D., 2020. Measuring calibration in deep learning. http://dx.doi.org/10.48550/arXiv.1904.01685, arXiv URL http://arxiv.org/abs/1904.01685 [cs].

Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., Kainz, B., Glocker, B., Rueckert, D., 2018. Attention U-net: Learning where to look for the pancreas. http://dx.doi.org/10.48550/arXiv.1804.03999, arXiv URL http://arxiv.org/abs/1804.03999 [cs].

Primakov, S.P., Ibrahim, A., van Timmeren, J.E., Wu, G., Keek, S.A., Beuque, M., Granzier, R.W.Y., Lavrova, E., Scrivener, M., Sanduleanu, S., Kayan, E., Halilaj, I., Lenaers, A., Wu, J., Monshouwer, R., Geets, X., Gietema, H.A., Hendriks, L.E.L., Morin, O., Jochems, A., Woodruff, H.C., Lambin, P., 2022. Automated detection and segmentation of non-small cell lung cancer computed tomography images. Nat. Commun. 13 (1), 3423. http://dx.doi.org/10.1038/s41467-022-30841-3, Publisher: Nature Publishing Group URL https://www.nature.com/articles/s41467-022-30841-3.

Purushwalkam, S., Li, B., Meng, Q., McPhee, J., 2013. Automatic segmentation of adipose tissue from thigh magnetic resonance images. In: Kamel, M., Campilho, A. (Eds.), Image Analysis and Recognition. Springer, Berlin, Heidelberg, pp. 451–458. http://dx.doi.org/10.1007/978-3-642-39094-4_51.

Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: Convolutional networks for biomedical image segmentation. http://dx.doi.org/10.48550/arXiv.1505.04597, arXiv URL http://arxiv.org/abs/1505.04597.

Sophia Genetics SAS, 2024. Deep learning-enabled exploration of predictive signatures in a multicenter retrospective and prospective observational study allowing the analysis of the aggregation of multimodal clinical, biological, genomic and radiomics data associated with the response to treatment and prognosis of patients with stage IV non-small cell lung cancer. Clinical trial registration NCT04994795, clinicaltrials.gov, submitted: 2021-07-15 URL https://clinicaltrials.gov/study/NCT04994795.

Suzuki, K., 2017. Overview of deep learning in medical imaging. Radiol. Phys. Technol. 10 (3), 257–273. http://dx.doi.org/10.1007/s12194-017-0406-5.

Travis, W.D., Brambilla, E., Nicholson, A.G., Yatabe, Y., Austin, J.H.M., Beasley, M.B., Chirieac, L.R., Dacic, S., Duhig, E., Flieder, D.B., Geisinger, K., Hirsch, F.R., Ishikawa, Y., Kerr, K.M., Noguchi, M., Pelosi, G., Powell, C.A., Tsao, M.S., Wistuba, I., WHO Panel, 2015. The 2015 world health organization classification of lung tumors: Impact of genetic, clinical and radiologic advances since the 2004 classification. J. Thorac. Oncol.: Off. Publ. Int. Assoc. the Study Lung Cancer 10 (9), 1243–1260. http://dx.doi.org/10.1097/JTO.0000000000000630.

Van Ginneken, B., Frangi, A., Staal, J., Ter Haar Romeny, B., Viergever, M., 2002. Active shape model segmentation with optimal features. IEEE Trans. Med. Imaging 21 (8), 924–933. http://dx.doi.org/10.1109/TMI.2002.803121, URL http://ieeexplore.ieee.org/document/1076037/.

Wang, T.-W., Hong, J.-S., Huang, J.-W., Liao, C.-Y., Lu, C.-F., Wu, Y.-T., 2024. Systematic review and meta-analysis of deep learning applications in computed tomography lung cancer segmentation. Radiother. Oncol. 110344.

Wang, S., Mahon, R., Weiss, E., Jan, N., Taylor, R.J., McDonagh, P.R., Quinn, B., Yuan, L., 2023. Automated lung cancer segmentation using a PET and CT dual-modality deep learning neural network. Int. J. Radiat. Oncology* Biology* Phys. 115 (2), 529–539. http://dx.doi.org/10.1016/j.ijrobp.2022.07.2312, URL https://www.sciencedirect.com/science/article/pii/S036030162203067X.

Wickstrøm, K., Kampffmeyer, M., Jensen, R., 2020. Uncertainty and interpretability in convolutional neural networks for semantic segmentation of colorectal polyps. Med. Image Anal. 60, URL https://www.sciencedirect.com/science/article/pii/S1361841519301574.

Wu, Y., He, K., 2018. Group normalization. http://dx.doi.org/10.48550/arXiv.1803.08494, arXiv URL http://arxiv.org/abs/1803.08494 [cs].