# Multi-modal medical Transformers: A meta-analysis for medical image segmentation in oncology

Gustavo Andrade-Miranda, Vincent Jaouen, Olena Tankyevych, Catherine

Cheze Le Rest, Dimitris Visvikis, Pierre-Henri Conze

► To cite this version:

Gustavo Andrade-Miranda, Vincent Jaouen, Olena Tankyevych, Catherine Cheze Le Rest, Dimitris Visvikis, et al.. Multi-modal medical Transformers: A meta-analysis for medical image segmentation in oncology. Computerized Medical Imaging and Graphics, 2023, 110, pp.102308. 10.1016/j.compmedimag.2023.102308. hal-04266909

HAL Id: hal-04266909

https://imt-atlantique.hal.science/hal-04266909v1

Submitted on 15 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Multi-modal medical Transformers: A meta-analysis for medical image segmentation in oncology

Gustavo Andrade-Miranda<sup>a,*</sup>, Vincent Jaouen<sup>a,b</sup>, Olena Tankyevych<sup>a,c</sup>,
Catherine Cheze Le Rest<sup>a,c</sup>, Dimitris Visvikis<sup>a</sup>, Pierre-Henri Conze<sup>a,b</sup>

## Abstract

Multi-modal medical image segmentation is a crucial task in oncology that enables the precise localization and quantification of tumors. The aim of this work is to present a meta-analysis of the use of multi-modal medical Transformers for medical image segmentation in oncology, specifically focusing on multi-parametric MR brain tumor segmentation (BraTS2021), and head and neck tumor segmentation using PET-CT images (HECKTOR2021). The multi-modal medical Transformer architectures presented in this work exploit the idea of modality interaction schemes based on visio-linguistic representations: (i) single-stream, where modalities are jointly processed by one Transformer encoder, and (ii) multiple-stream, where the inputs are encoded separately before being jointly modelled. A total of fourteen multi-modal architectures are evaluated using different ranking strategies based on dice similarity coefficient (DSC) and average symmetric surface distance (ASSD) metrics. In addition, cost indicators such as the number of trainable parameters and the number of multiply-accumulate operations (MACs) are reported. The results demonstrate that multi-path hybrid CNN-Transformer-based models improve segmentation accuracy when compared to traditional methods, but come at the cost of increased computation time and potentially larger model size.

**Keywords:** Medical imaging, multi-modality, tumor segmentation, vision Transformers, CNN, oncology

<footnote><sup>a</sup>LaTIM UMR 1101, Inserm, Brest, France</footnote>

<footnote><sup>b</sup>IMT Atlantique, Brest, France</footnote>

<footnote><sup>c</sup>Nuclear Medicine, University Hospital of Poitiers, Poitiers, France</footnote>

<footnote>Manuscript received February 7, 2023.</footnote>

<footnote>*Corresponding author</footnote>

<footnote>Email addresses: gustavo-xavier.andrade-miranda@inserm.fr (Gustavo Andrade-Miranda), vincent.jaouen@imt-atlantique.fr (Vincent Jaouen), olena.tankyevych@chu-poitiers.fr (Olena Tankyevych), catherine.cheze-le-rest@chu-poitiers.fr (Catherine Cheze Le Rest), visvikis@univ-brest.fr (Dimitris Visvikis), pierre-henri.conze@imt-atlantique.fr (Pierre-Henri Conze)</footnote>
# 1. Introduction

Transformer, as an attention-based structure Vaswani et al. (2017), has first demonstrated its efficacy in natural language processing (NLP) Devlin et al. (2019); Radford et al. (2018); Yang et al. (2019) and has gradually emerged as a predominant deep learning tool for multi-modal learning tasks Girdhar et al. (2022); Akbari et al. (2021); Nagrani et al. (2021); Hu & Singh (2021). The most recent dominant models combine self-supervised pre-training with Transformers He et al. (2021); Bao et al. (2022); Chen et al. (2020a) followed by supervised fine-tuning on scarce and specific downstream tasks. Mainly used to learn task-agnostic joint representations of image content and natural language, this scheme gave birth to the category of vision-and-language pre-training (VLP) models Lu et al. (2019); Kim et al. (2021). While language sequences can be straightforwardly fed into VLP models, image pixels need to be initially embedded in a dense form before being put alongside language tokens (words). Since the seminal work of Krizhevsky et al. Krizhevsky et al. (2012), this embedding step, like many vision-based tasks, has been universally conducted by convolutional neural networks (CNN).

CNN attributes as sparse interactions, parameter sharing, and translation equivariance built-in several inductive biases, which make them suitable for a wide variety of computer vision applications. Inspired by the great success of self-attention mechanisms in NLP, CNN architectures have started to up-weight the relevance of certain information through attention layers either in spatial or channel dimensions Wang et al. (2017); Woo et al. (2018); Park et al. (2018); Wang et al. (2018); Cao et al. (2019); Bello et al. (2019); Yin et al. (2020); Cor- donnier et al. (2020). Recently, vision Transformers (ViT) models Dosovitskiy et al. (2021) have emerged as the most popular option to replace or complement CNNs for computer vision applications Carion et al. (2020); Touvron et al. (2021). Their popularity is now also rapidly growing in medical image analysis Jun et al. (2021), especially for medical image segmentation (MIS) with an ex- ponential growth of related publications Shamshad et al. (2022). Indeed, MIS plays an important role in clinical applications such as computer-aided diagnosis Conze et al. (2021); Iantsen et al. (2020); Duran et al. (2022), surgery planning Kavur et al. (2021), treatment follow-up Islam et al. (2022), image-guided in- terventions Malik & Vidyarthi (2022), or radiotherapy Jin et al. (2019); Lei et al. (2021). While MIS is usually performed using a single modality, such as computed tomography (CT), magnetic resonance (MR), or positron emis- sion tomography (PET) imaging, practical clinical protocols often involve the analysis of multiple imaging modalities to capture various aspects of the same biological process. Particularly in oncology, combining different types of scans can help the model gain a better understanding of the body's structures and cancer-related issues, leading to increased accuracy and reliability in tumor seg- mentation Dana et al. (2020). By leveraging the strengths of each scan, these models can provide a more detailed picture of the tumor's location and size Mazzucchi et al. (2022); Ning et al. (2021). This not only assists doctors in di- agnosing and planning treatments more accurately but also holds the potential
to detect cancer at an earlier stage and monitor treatment effectiveness. Consequently, multi-modal learning approaches have the potential to enhance overall segmentation performance by leveraging both complementary and redundant information across modalities Zhou et al. (2019); Zhang et al. (2021d, 2022a).

Unfortunately, the majority of multi-modal medical image segmentation (MMIS) methods are limited to a straightforward modality concatenation at the input level Zhou et al. (2018a); Shapey et al. (2021); Wang et al. (2021); Chen et al. (2021c); Hatamizadeh et al. (2022a) (i.e., one-pathway encoder). This makes it hard to discover highly non-linear relationships between the low-level features of different modalities, especially when the modalities have significantly different statistical properties Oreiller et al. (2022). As an alternative, some authors choose to employ multiple encoders to handle multi-modal information separately Chen et al. (2020b); Zhang et al. (2021b,a); Zhou et al. (2021b). Then, the multi-pathways are mid or late-fused to take advantage of the latent correlation between modalities. Despite the success of ViT, only a few works have been inspired by their intrinsic advantages and scalability in modeling different modalities Li et al. (2022c); Zhang et al. (2022a). Concretely, they could encompass multiple sequences of tokens where each sequence's attribute represents a different modality, thus allowing a multi-modal learning framework without architectural modification Devlin et al. (2019). Furthermore, learning per-modal specificity and inter-modal correlation can be achieved by manipulating self-attention input patterns Lu et al. (2019).

The benefits of using one or multi-pathway CNN encoders, as well as how they can be efficiently linked with Transformers to leverage all of their multi-modal properties, are currently unclear. Furthermore, the question of whether using or not a pure Transformer-based encoder Hatamizadeh et al. (2022b,a) to solve MMIS tasks remains unanswered. This work attempts to address these concerns by presenting several hybrid-based (CNN-ViT) and purely Transformers-based models and empirically comparing their differences in a controlled environment based on well-labelled and aligned cross-modal datasets. We believe that this comparative analysis is crucial to better understand and guide future research on MMIS. We specifically looked into whether differences in MMIS performance can be explained by: (i) convolution or Transformed stems; (ii) one-path or multi-path CNN-based encoder, and (iii) Transformer modality interaction scheme. In summary, our contributions in this paper are four-fold. First, we introduce a unified mathematical framework to describe current Transformer-based architectures and is readily extensible to new ones. Second, we conduct a series of controlled experiments to analyze different multi-modal segmentation networks. Third, we explore the gain variation in performance depending on one-path versus multi-path CNN-based encoder. Fourth, we study the impact of single- and dual-stream joint representations that have been successfully implemented in vision and language processing fields.

The rest of the paper is organized as follows. Sect. 2 details related works and explains the current state-of-the-art in MMIS. Sect. 3 presents the mathematical framework of each multi-modal Transformer building block. Sect. 4 illustrates how to unify the multi-modal Transformer blocks to build diverse hybrid and
pure-Transformer models. Sect. 5 describes the experimental setup including the datasets, implementation details, competing methods, and ranking. The presentation and in-depth analysis of the results are provided in Sect. 6. Sect. 7 concludes our study and suggests possible avenues to be further investigated in the near future.

## 2. Related works

We mainly examine approaches that use Transformers to improve multi-modal medical image segmentation where U-Net-like shape architecture Ronneberger et al. (2015); Isensee et al. (2020) remains the preferred option. From them and as illustrated in Fig. 1, three categories can be identified: pure Transformer-based encoder, hybrid CNN-ViT encoder, and full-Transformer-based network.

The first category exploits the global context modeling capacity of Transformers to effectively encode relationships between spatially distant voxels. The idea is to feed-forward flattened 3D patches (also called tokens) to the Transformers, whose outputs are subsequently reorganized into 3D tensors to match with segmentation masks. For instance, Hatamizadeh et al. and Karimi et al. employ in Hatamizadeh et al. (2022b) and Karimi et al. (2021) a 3D ViT as an encoder where the last feature map is reshaped and up-sampled by a factor of 2. Then, the decoding branch towards the segmentation mask space is handled using CNN up-sampling with multi-level feature aggregation. This process is repeated until the initial resolution is reached. Despite the competitive results obtained, vanilla ViT has conclusive defects in dense predictions due to the lack of prior information from images. To overcome these challenges, hierarchical ViT such as CSWin Dong et al. (2021b), Swin Hatamizadeh et al. (2022a) and PVT Dong et al. (2021a) Transformers have been introduced. They inject more-specific inductive bias derived from CNN-like features into the Transformers as they compute local attention with shifted windows, starting from small-sized patches and gradually merging neighboring patches in the subsequent layers. Xin Yu et al. proposed in Xin Yu et al. (2022) a 3D U-shape model inspired by nested hierarchical Transformer Zhang et al. (2022b) to minimize the design complexity of classic hierarchical ViT. They retain the idea of global self-attention but restrict it to smaller non-overlapping 3D blocks. Cross-block self-attention communication was achieved by hierarchically nesting Transformers and connecting them with an aggregation function.

The hybrid CNN-ViT encoder, on the other hand, combines the inductive bias of CNN with the global context modeling capability of Transformers Chen et al. (2021c); Wang et al. (2021); Li et al. (2022b). In this case, CNN layers serve as multi-scale feature extractors meanwhile Transformers encapsulate the long-term dependencies among features. In the end, a CNN decoder projects the Transformer output back into a 4D feature map that is gradually up-sampled to recover the entire segmentation mask. Others approaches modified the traditional attention blocks using deformable Transformers Xie et al. (2021) or by
<figure><img src="image_2.png" /><figcaption>Figure 1: Typical Transformer-based model architectures for medical image segmentation: (a) hybrid CNN-Transformer encoder, (b) pure Transformer-based encoder, (c) full Transformer-based network. Adapted from Li et al. (2022a).</figcaption></figure>

squeeze-and-expansion Transformers layers Li et al. (2021). Chen et al. proposed in Chen et al. (2021a) a 2D hybrid network that combines two independent self-attention blocks to model the long-range interactions and global spatial relationships. Furthermore, at the skip connection level, a multi-resolution technique is used to produce more discriminative representations. PC-SwinMorph Liu et al. (2022) performed registration-based segmentation using CNN and patch-based contrastive strategy followed by a Swin transformer which enforces to capture not only global but also local anatomical representations. Another type of hybrid Transformer-CNN encoders is multi-branch fusion like Transfuse Zhang et al. (2021c), or CrossTeaching Luo et al. (2022). Commonly, the two

135

140
parallel branches, one for CNN and the other for Transformers, are fused, letting the network benefiting from the two learning paradigms. CNN layers give the model the ability to focus on local information meanwhile Transformers learn the long-range relationship between voxels.

Lastly, pure Transformers networks are built in an end-to-end Transformer-based fashion Zhou et al. (2021a); Wu et al. (2022). Cao et al. proposed in Cao et al. (2022) a Swin 2D U-Net network with a patch expansion layer for up-sampling the feature map of the decoder. This design outperformed bilinear up-sampling-based decoders in terms of collecting fine details. Lin et al. went a step further in Lin et al. (2021) by adopting a dual-scale encoder based on a 2D Swin-Transformer to extract the coarse and fine-grained feature representations at different scales. They also included an interactive fusion module that uses self-attention to successfully establish global dependencies between features of varying sizes. To better leverage multi-scale feature hierarchies, Huang et al. proposed in Huang et al. (2021) a hierarchical encoder-decoder 2D architecture named MISSFormer. The primary feature was the incorporation of an enhanced Transformer context bridge to capture the correlation and local context of multi-scale information generated by the hierarchical Transformer encoder. Peiris et al. proposed in Peiris et al. (2022) VT-UNet that works directly in 3D volume using Swin Transformers. The Swin decoder combines parallel cross-attention and self-attention to bridge the gap between queries from the decoder and keys and values from the encoder. This parallelization retains the entire global context during the decoding process, which is critical for the delineation task.

Table 1 provides an extensive overview of important oncology multi-modal datasets. These datasets include various types of cancer and use different imaging methods. They range from MRI sequences like T2-weighted (T2W), apparent diffusion coefficient (ADC), and diffusion-weighted imaging (DWI), to positron emission tomography combined with computerized tomography (PET/CT). This diverse collection has been used in various challenges to assess the effectiveness of deep learning models, particularly in tasks like tumor detection, segmentation, and diagnosis. Researchers have leveraged these datasets to benchmark the performance of state-of-the-art algorithms and to develop more accurate and reliable techniques for processing oncological images.

The current literature on multi-modal medical image Transformers shares two common attributes: a U-shape design and the re-introduction of numerous CNN priors either by directly incorporating CNN blocks or via self-attention-shifted windows. Furthermore, only a few works Sun et al. (2021); Li et al. (2022c); Zhang et al. (2022a); Lin et al. (2022) included multi-pathway cross-modal interactions (e.g., fusion, alignment) which might be beneficial to complement information, particularly in circumstances of missing modalities that are common in clinical practice.
<table><caption>Table 1: Summary of multi-modal oncology datasets.</caption><thead><tr><th>Dataset</th><th>Modalities</th><th>Task</th><th>Paired</th><th>Reference</th></tr></thead><tbody><tr><td>PICAI'22</td><td>T2W/ADC/DWI</td><td>csPCa detection and diagnosis</td><td>Yes</td><td>Saha et al. (2023)</td></tr><tr><td>ISLES'22</td><td>DWI/ADC/T2-FLAIR</td><td>Segmentation of stroke lesions</td><td>Yes</td><td>Petzsche et al. (2022)</td></tr><tr><td>AutoPET II</td><td>PET/CT</td><td>Detection and segmentation of FDG-avid tumor lesions</td><td>Yes</td><td>Gatidis et al. (2023)</td></tr><tr><td>Bonbid-Hie2023</td><td>ADC/Zadc</td><td>Hypoxic ischemic encephalopathy segmentation</td><td>Yes</td><td>Bao et al. (2023)</td></tr><tr><td>BraTS2021</td><td>T1/T1Gd/T2/T2-FLAIR</td><td>Brain tumor segmentation</td><td>Yes</td><td>Baid et al. (2021)</td></tr><tr><td>HECKTOR2021</td><td>PET/CT</td><td>Head and neck tumor segmentation</td><td>Yes</td><td>Oreiller et al. (2022)</td></tr><tr><td>crossMoDa2022</td><td>ceT1/hrT2</td><td>Vestibular schwannoma and cochlea segmentation</td><td>No</td><td>Dorent et al. (2023)</td></tr></tbody></table>

### 3. Multi-modal transformers building blocks

In this section, we first review how the input modalities are tokenized and represented in feature space (Sect. 3.1). Then, we explain the mathematical framework of vanilla and hierarchical Transformers (Sect. 3.2). Finally, we present alternative ways to fuse multi-modal information (Sect. 3.3) and possible decoders (Sect. 3.4) for MMIS.

Let $\mathcal{X}$ be a set of $M$ input modalities $\mathbf{X}^{(m)} \in \mathbb{R}^{H \times W \times D}$ with spatial resolution $H \times W \times D$ where $D$ is the number of slices, and $\mathbf{Y} \subset [0, 1]^{H \times W \times D \times C}$ a ground truth annotation mask, where $C$ represents the number of classes. Then, the multi-modal image segmentation problem can be formulated as finding the function $\phi: \mathcal{X} \to \phi(\mathcal{X}; \mathbf{\Theta}_{\phi}) = \hat{\mathbf{Y}}$ with weights $\mathbf{\Theta}_{\phi}$ that produce the best mapping between the $M$ inputs modalities $\mathbf{X}^{(m)}$ and the label $\mathbf{Y}$ from $N$ training samples by optimizing a loss function $\mathcal{L}_{\phi}(\mathbf{Y}, \hat{\mathbf{Y}})$.

#### 3.1. Multi-modal tokenization and embedding

Because Transformer-based architectures operate over 1D sequences, the $\mathbf{X}^{(m)}$ modalities have to be tokenized and represented in an embedding space before feeding them to the Transformer layers. In practice, tokenizing inputs and embeddings for the tokens are vital for Transformers but highly flexible Xu et al. (2022). Tokenization has two main theoretical benefits: it is a more flexible approach to processing multi-modal data (e.g., concatenation, weighted summation) and it is compatible with task-specific customized tokens as in classification Dosovitskiy et al. (2021); Touvron et al. (2021) or masked image modelling (MIM) Bao et al. (2022); He et al. (2021). Tokenization is often carried out by splitting a 3D input volume into a set of non-overlapped 3D patches Arnab et al. (2021); Hatamizadeh et al. (2022b) or by using extracted feature maps from 3D CNN backbone as tokens Wang et al. (2021); Tu et al. (2022); Ryoo et al. (2021). Given a tokenization plan, the subsequent embedding approach is usually performed by linearly projecting tokens into a $d$-dimensional
vector space. The mathematical framework behind both tokenization and token embedding is described as what follows.

### 3.1.1. 3D CNN feature tokenization

Given an input modality $\mathbf{X}^{(m)} \in \mathbb{R}^{H \times W \times D}$ and a 3D CNN feature ex-tractor function $f_{\Theta}$, a low-resolution/high-level feature representation $\mathbf{F}^{(m)} = f_{\Theta}(\mathbf{X}^{(m)})$, $\mathbf{F}^{(m)} \in \mathbb{R}^{H' \times W' \times F}$ is computed where $F$ represents the number of encoded feature maps. Then, a tokenizer function $T(\cdot)$ maps $\mathbf{F}^{(m)}$ into $S = H'W'D'/P^2F$ 3D non-overlapping patches of resolution $(P, P, F)$. Lastly, the patches are flattened to a series of 1D vectors $\mathbf{Z}_{l=0}^{(m)} = [z_{li}^{(m)}]_{i=1}^{S}$ with $z_{0i}^{(m)} \in \mathbb{R}^{F}$ before being passed thought the Transformer layers. In practice, $P$ is usually set to 1. $H'$, $W'$ and $D'$ represent a fraction of the input dimension that depends on the number of down-sampling operations in $f_{\Theta}$. $l \in [0, L]$ denotes the layers of the Transformer and is set to 0 in $\mathbf{Z}_{l}^{(m)}$ since it represents the Transformer input.

In a multi-modal framework, the 3D CNN feature tokenization is accomplished either by concatenating together $(\|)$ the $M$ modalities at the channel level (i.e., one-path), $\mathbf{X} \in \mathbb{R}^{H \times W \times D \times M}$ before passing through a single $f_{\Theta}$ (Eq. 1) or by using multiple $f_{\Theta}$ (i.e., multi-path) to extract modality-independent features (Eq. 2). Fig. 2 depicts 3D CNN feature tokenization methods for the case of $M$ modalities.

$$ \mathbf{Z}_0 = [z_{0i}]_{i=1}^{S} = T(\mathbf{F}) = T(f_{\Theta}([\mathbf{X}^{(1)} \| \dots \| \mathbf{X}^{(M)}])) \quad (1) $$

$$ \begin{aligned} \mathbf{Z}_{0}^{(m)} &= [z_{0i}^{(m)}]_{i=1}^{S} = T^{(m)}(\mathbf{F}^{(m)}) ; \quad m \in [0, M] \\ &= T^{(m)}(f_{\Theta}^{(m)}(\mathbf{X}^{(m)})) \end{aligned} \quad (2) $$

### 3.1.2. Patchify tokenization

Given an input modality $\mathbf{X}^{(m)} \in \mathbb{R}^{H \times W \times D}$, a tokenizer function $T(\cdot)$ directly splits $\mathbf{X}^{(m)}$ into $S = HWD/P^3$ 3D non-overlapping sub-tensors of resolution $(P, P, P)$. Since Transformers layers operate over fixed size 1D set of vectors, the $S$ patches are flattened such that $\mathbf{Z}_{0}^{(m)} = [z_{0i}^{(m)}]_{i=1}^{S}$ with $z_{0i}^{(m)} \in \mathbb{R}^{P^3}$. The size of the patches $P$ is inversely related to the Transformer's sequence length, making models with small patch sizes computationally more costly. As a result, using cubic patches of size $P = 16$ or $P = 14$ is a good trade-off between performance and complexity. In a multi-modal context, the patchify tokenization is conducted by either directly combining the $M$ modalities at the channel level (Eq. 3) or by performing independent tokenization from each modality separately (Eq. 4). Fig. 3 depicts the procedure of patchify tokenization when $M$ modalities are available.

$$ \mathbf{Z}_0 = [z_{0i}]_{i=1}^{S} = T(\mathbf{X}) = T([\mathbf{X}^{(1)} \| \dots \| \mathbf{X}^{(M)}]) \quad (3) $$
<figure><img src="image_3.png" /><figcaption>Figure 2: Multi-modal 3D CNN feature tokenization procedure for M-modalities: (a) one-path 3D-CNN based-tokenization, (b) multi-path 3D-CNN based-tokenization.</figcaption></figure>

$$ \mathbf{Z}_{0}^{(m)} = [z_{0i}^{(m)}]_{i=1}^{S} = T^{(m)}(\mathbf{X}^{(m)}); \quad m \in [0, M] \qquad (4) $$

### 3.1.3. Linear projection

Following the tokenization step, the set of S tokens are embedded in a d-dimensional space via a trainable linear projection **E**. Depending on the previous tokenization procedure, a common linear projection (Eq. 5) Hatamizadeh et al. (2022a,b) or either a modality-specific projection (Eq. 6) Zhang et al. (2022a) can be formulated. Fig. 2 illustrates the embedding procedure carried out from a sequence of tokens coming from a 3D CNN feature tokenization whereas Fig. 3 depicts an embedding obtained given a patchify tokenization.

$$ \mathbf{Z}_{0} = [[z_{0i}]_{i=1}^{S} \cdot \mathbf{E}]; \quad \mathbf{E} \in \mathbb{R}^{(P^3 \cdot M) \times d} \qquad (5) $$

$$ \mathbf{Z}_{0}^{(m)} = [z_{0i}^{(m)}]_{i=1}^{S} \cdot \mathbf{E}^{(m)}; \quad \mathbf{E}^{(m)} \in \mathbb{R}^{P^3 \times d} \qquad (6) $$

### 3.1.4. Positional embedding

Since Transformers operates over sets, a positional learnable embedding $\mathbf{E}_{pos} \in \mathbb{R}^{S \times d}$ is added to the tokens embedding to retain positional information (Eq. 7). At the end, the embedding $\mathbf{Z}_{0} = [z_{0i}]_{i=1}^{S}$ with $\mathbf{Z}_{0} \in \mathbb{R}^{S \times d}$ represents the effective input of the Transformer layers.
<figure><img src="image_4.png" /><figcaption>Figure 3: Multi-modal patchify tokenization process for M-modalities: (a) one-path 3D patchify tokenization, (b) multi-path 3D patchify tokenization.</figcaption></figure>

$$Z_{0}^{(m)} = Z_{0}^{(m)} + E_{pos} ; \quad E_{pos} \in \mathbb{R}^{S \times d} \qquad (7)$$

## 3.2. Transformer blocks

A Transformer basic block consists of alternating multi-head self-attention (MHSA) and multi-layer perceptron (MLP) blocks. In addition, a layer normalization (LN) is applied before each block and a residual connection after each block (Fig. 4a). There are two main ways to build a Transformers-based architecture, whether using fixed scale feature maps (i.e., vanilla Transformer Dosovitskiy et al. (2021)) or by constructing hierarchical feature maps (i.e., Swin Transformer Liu et al. (2021)).

### 3.2.1. Vanilla Transformer

The vanilla Transformer (ViT$_{v}$) introduces no image-specific inductive bias. The only difference with original NLP Transformers relies on the tokenization process. The computation of a vanilla Transformer block for a particular layer l can be formulated as:

$$ \begin{aligned} Z'_{l} &= \text{MHSA}(\text{LN}(Z_{l-1})) + Z_{l-1} & l = 1, \dots, L, \\ Z_{l} &= \text{MLP}(\text{LN}(Z'_{l})) + Z'_{l} & l = 1, \dots, L, \end{aligned} \qquad (8) $$
<figure><img src="image_5.png" /><figcaption>Figure 4: Illustration of Transformer blocks: (a) vanilla Transformer block, (b) Swin Transformer block.</figcaption></figure>

The MHSA block consists of $h$ parallel self-attention (SA) heads, where each SA attends to information from different representation sub-spaces at different positions using a scoring function $\mathcal{A}$. To achieve this goal, an input sequence $\mathbf{Z}_0 \in \mathbb{R}^{S \times d}$ is mapped into a set of query $(Q \in \mathbb{R}^{S \times d_k})$, key $(K \in \mathbb{R}^{S \times d_k})$ and value $(V \in \mathbb{R}^{S \times d_v})$ matrices using three learnable parameters: $W_q \in \mathbb{R}^{d \times d_k}$, $W_k \in \mathbb{R}^{d \times d_k}$, and $W_v \in \mathbb{R}^{d \times d_v}$, as is shown in Eq. 9. Then, the attention distribution function is computed (Eq. 10) and the resulting attention weights are applied to the $V$ matrix obtaining the SA maps (Eq. 11).

$$ 
\begin{aligned}
Q &= \mathbf{Z}W_q \\
K &= \mathbf{Z}W_k \\
V &= \mathbf{Z}W_v
\end{aligned} 
\qquad (9) 
$$

$$ \mathcal{A}(Q, K) = \mathrm{Softmax} \left( \frac{Q \cdot K^T}{\sqrt{d_k}} \right) \qquad (10) $$

$$ \mathrm{SA}(Q, K, V) = \mathcal{A}(Q, K) \times V \qquad (11) $$

For MHSA, $Q$, $K$, and $V$ are computed once for each $j$-head using $h$ different learnable parameters ($W_j^{q,k,v}$), and the final attention map result from the concatenation of the $h$ heads multiplied by a learnable aggregation matrix $W^o \in \mathbb{R}^{hd_v \times d}$. The computational cost of one single-head attention with full $d$-dimension is maintained by setting $d_k$ and $d_v$ equal to $d/h$.
$$
\begin{align}
\text{MHSA}(Q, K, V) &= (\text{head}_1 \| \dots \| \text{head}_h)W^o \quad \text{where} \nonumber \\
\text{head}_j &= \text{SA}(Q_j, K_j, V_j) \tag{12} \\
&= \text{SA}(\mathbf{Z}_{l-1}W_q, \mathbf{Z}_{l-1}W_k, \mathbf{Z}_{l-1}W_v) \nonumber
\end{align}
$$

3.2.2. Swin Transformer

Unlike vanilla Transformer, Swin Transformers introduce a sliding window
strategy to only compute the attention within local windows, thus reducing the
computational cost of global MHSA Liu et al. (2021). As opposed to vanilla
Transformers, Swin Transformers gradually decrease the initial number of to-
kens by implementing patch merging layers as the network gets deeper. This
attribute delivers a hierarchical representation similar to the way CNN behaves.
In practice, the Swin Transformer consists of four stages and performs 2 × 2 spa-
tial down-sampling in the patch merging layer of each stage. The merging layer
concatenates the features of each group of 2 × 2 neighboring patches and applied
a linear transformation to reduce the number of concatenated features to half
of their dimension.

In Swin Transformer, the attention is computed in two-step. In the first one, given as an input $H' \times W' \times D'$ 3D token and a 3D window size of ($P'$, $P'$, $P'$), a multi-head self-attention denoted as W-MHSA is performed within non-overlapping 3D windows of size $H/P' \times W/P' \times D/P'$. Since multi-head self-attention is applied separately for each window, there is a lack of cross-connections. Therefore, the window partition is shifted along height, width, and depth axes by ($P'/2 \times P'/2 \times P'/2$) tokens from that of the preceding block (W-MHSA) and self-attention is re-computed (SW-MHSA). Fig. 4b illustrates two successive W-MHSA and SW-MHSA blocks and the computation of both attention blocks is described below:

$$
\begin{align}
\mathbf{Z}'_l &= \text{W-MHSA}(\text{LN}(\mathbf{Z}_{l-1})) + \mathbf{Z}_{l-1}, \nonumber \\
\mathbf{Z}_l &= \text{MLP}(\text{LN}(\mathbf{Z}'_l)) + \mathbf{Z}'_l, \nonumber \\
\mathbf{Z}'_{l+1} &= \text{SW-MHSA}(\text{LN}(\mathbf{Z}_l)) + \mathbf{Z}_l, \nonumber \\
\mathbf{Z}_{l+1} &= \text{MLP}(\text{LN}(\mathbf{Z}'_{l+1})) + \mathbf{Z}'_{l+1} \tag{13}
\end{align}
$$

3.3. Multi-modal Transformer fusion

Fusing information across multiple modalities has been conventionally achieved at three different levels: input (i.e., early fusion), intermediate representation (i.e., middle fusion), and prediction (i.e., late fusion) Zhou et al. (2019). However, such cross-modal interactions in multi-modal Transformers are essentially processed by the self-attention mechanism. Then, from the standpoint of self-attention designs, two main types of cross-modal interactions using Transformers are possible: (1) Transformers with single-stream interaction (or all-attention) (ViT<sub>s</sub>), and (2) Transformers with multi-stream interaction (or cross-attention) Xu et al. (2022) (ViT<sub>m</sub>). For simplicity, we will state and compare the mathematical formulations of cross-modal interactions for two modal cases. Given two inputs modalities **X**<sup>(1)</sup> and **X**<sup>(2)</sup> with their respective token
embeddings at layer $l-1$, $\mathbf{Z}_{l-1}^{(1)}$ and $\mathbf{Z}_{l-1}^{(2)}$, the token embedding $\mathbf{Z}_l$ represents the multi-modal interaction in layer $l$ obtained after having processed the input tokens by a Transformer block $Tg(\cdot)$.

### 3.3.1. Transformers with single-stream interaction

The tokens are processed by a single-stream set of $L$ stacked Transformer blocks with cross-modal interaction occurring at the Transformer input level ($l=1$). There are two ways to perform multi-modal token embedding: early summation Xu & Zhu (2021) or early concatenation Akbari et al. (2021); Kim et al. (2021).

Early summation is a straightforward and effective multi-modal interaction, where the token embeddings from various modalities are weighted and summed at each token position before being processed:

$$ \mathbf{Z}_l = Tg(\alpha\mathbf{Z}_{l-1}^{(1)} \oplus \beta\mathbf{Z}_{l-1}^{(2)}) \\ \text{where } \text{MHSA}(Q_{(12)}, K_{(12)}, V_{(12)}) \\ \left\{ \begin{array}{l} Q^{(12)} = (\alpha\mathbf{Z}_{l-1}^{(1)} \oplus \beta\mathbf{Z}_{l-1}^{(2)})W_q \\ K^{(12)} = (\alpha\mathbf{Z}_{l-1}^{(1)} \oplus \beta\mathbf{Z}_{l-1}^{(2)})W_k \\ V^{(12)} = (\alpha\mathbf{Z}_{l-1}^{(1)} \oplus \beta\mathbf{Z}_{l-1}^{(2)})W_v \end{array} \right. \qquad (14) $$

where $\oplus$ is element-wise sum, and $\alpha$ and $\beta$ are weightings. Its main benefit is that it does not increase computing complexity. However, its drawbacks stem from the usage of manually adjusted weights.

On the other hand, early concatenation originally introduced on BERT model Devlin et al. (2019) allows for an early and unconstrained fusion of cross-modal information. First, a learnable modal-type embedding $\mathbf{E}_T^{(m)}$ is added to each of the modal input embedding $\mathbf{Z}_{l-1}^{(m)}$ to indicate its belonging to the $m$-modality. This gives the model a piece of additional information to differentiate among the $m$-modalities. Then, the token embedding sequences from the multiple modalities are concatenated and input into the Transformer's layers.

$$ \mathbf{Z}_l = Tg([\mathbf{Z}_{l-1}^{(1)} \oplus \mathbf{E}_T^{(1)} \parallel \mathbf{Z}_{l-1}^{(2)} \oplus \mathbf{E}_T^{(2)}]) \qquad (15) $$

The fundamental advantage of early concatenation formulation is that all multi-modal token positions can be attended as a whole sequence, allowing each modality's positions to be well encoded by conditioning the context of other modalities. However, the longer the input sequence after concatenation, the greater the computational complexity.

### 3.3.2. Transformers with multi-stream interaction

First proposed by Lu et al. in Lu et al. (2019), this method consists of $m$-streams (one for each modality) that communicate through cross-Transformer blocks. This structure can accommodate the different processing needs of each
modality by alternating intra-modal interactions (self-attention) with inter-modal interactions (cross-attention) at varying representation depths Chen et al. (2021b). The cross-attention is achieved by only swapping the queries $Q^{(m)}$ before carrying out the MHSA computation, which condition each other modality without increasing the computational complexity. For the two-stream case, each embedded modality ($\mathbf{Z}_{l-1}^{(1)}$ and $\mathbf{Z}_{l-1}^{(2)}$) is passed independently through $L_m$ Transformer blocks to be later fused together $L_c$ times via cross-attention.

$$L_m \times \begin{cases} \mathbf{Z}_{l'}^{(1)} = Tg(\mathbf{Z}_{l-1}^{(1)}) \\ \mathbf{Z}_{l'}^{(2)} = Tg(\mathbf{Z}_{l-1}^{(2)}) \end{cases} \\ L_c \times \begin{cases} \mathbf{Z}_l^{(1)} = Tg_1 \rightarrow 2(\mathbf{Z}_{l'}^{(1)}) = \text{MHSA}(Q^{(2)}, K^{(1)}, V^{(1)}) \\ \mathbf{Z}_l^{(2)} = Tg_2 \rightarrow 1(\mathbf{Z}_{l'}^{(1)}) = \text{MHSA}(Q^{(1)}, K^{(2)}, V^{(2)}) \end{cases} \quad (16)$$

The aforementioned procedure illustrates the computation of a single layer $l$. The whole multi-stream strategy is obtained by repeating such a block $L$ times. Another option is to use a type of hierarchically cross-modal interaction Tsai et al. (2019) in which cross-attention can be concatenated and processed by another Transformer to represent the global context. This type of hierarchically cross-modal interaction combines cross-attention and all-attention strategies. Fig. 5 illustrates the aforementioned self-attention variants used to perform multi-modal fusion.

$$\left\{ \begin{aligned} \mathbf{Z}_{l'}^{(1)} &= \text{MHSA}(Q_{(2)}, K_{(1)}, V_{(1)}) \\ \mathbf{Z}_{l'}^{(2)} &= \text{MHSA}(Q_{(1)}, K_{(2)}, V_{(2)}) \\ \mathbf{Z}_l &= Tg([\mathbf{Z}_{l'}^{(1)} \parallel \mathbf{Z}_{l'}^{(2)}]) \end{aligned} \right. \quad (17)$$

### 3.4. Decoder

After having encoded the image to a high-dimensional feature representation, the decoder aims at reducing the number of feature maps while increasing their dimensions by means of successive up-sampling operations. Then, the feature maps from the down-sampling are concatenated to the up-sampling part, thus helping the propagation of spatial information. Finally, a $1 \times 1$ convolution processes the final feature maps to generate a dense prediction with per-pixel class scores.

The decoder approach proposed by Ronneberger et al. in Ronneberger et al. (2015) has been almost unaltered with only a few attempts to replace it. For instance, Zhou et al. (2018b) directly replaces the plain skip connections with dense skip connectivity, allowing the interaction at different scale feature maps. Other strategies focus on strengthening the skip-connections by introducing additional non-linear transformations on the features propagating from the encoder stage, which allows to focus only on target structures Oktay et al. (2018). With the advent of Transformers, modern architectures have also opted to include them (Fig. 6) in the expanding path by using multiple up-sampling
<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /><figcaption>Figure 5: Self-attention variants for multi-modal fusion. First row corresponds to single-stream interaction (all-attention) whereas second row depicts multi-stream interaction (cross-attention). Adapted from Xu et al. (2022).</figcaption></figure>

Transformers blocks with plain skip connections Cao et al. (2022); Zhou et al. (2021a). Other works Huang et al. (2022); Wang et al. (2022) replace the plain independent skip-connections with a collaborative learning framework based on cross-attention layers to transfer the multi-scale context from the encoder to the decoder. In spite of the existence of current state-of-the-art decoder variants, the original progressive CNN up-sampling strategy with plain skip-connections remains the preferred choice due to its simplicity and the excellent results reached in several medical image segmentation benchmarks Isensee et al. (2020).
<figure><img src="image_8.png" /><figcaption>Figure 6: Different skip-connection schemes Wang et al. (2022): (a) plain skip-connections (Unet), (b) skip-connections with non-linearity, (c) multi-scale skip-connection (Unet++), and (d) multi-scale fusion with cross-attention blocks (UCTransNet).</figcaption></figure>

## 4. Hybrid and pure-Transformer models

In this section, we illustrate how to unify the previously mentioned multi-modal blocks to build diverse multi-modal segmentation architectures. The networks introduced in this section follow a U-shaped design in which the extracted feature representations from encoder layers are fused with their decoder counterparts through plain skip-connections, towards finer segmentation masks with richer spatial details. This study only considers CNN-based decoders implemented with sequential CNN up-sampling operations and plain skip connections. First, we go through some specific configuration details concerning the 3D CNN feature extractor and Transformer blocks. Then, based on the categorization presented in Sect. 2, pure Transformer-based and hybrid CNN-ViT encoder networks are extended to a multi-modal setting by incorporating self-attention modality interaction schemes.

### 4.1. 3D CNN feature extractor setup

The 3D CNN feature extractor $f_{\Theta}$ is made of ResNet blocks (Fig. 7a) to capture compact features at multiple scales. It has 4 resolution stages. Each resolution stage is a sequence of {conv3D, instance norm, leaky ReLU}, where all the convolution kernels are set to [3, 3, 3]. The stride for down-sampling is set to [2, 2, 2] but only to the first convolution of each resolution stage. The initial number of feature maps is set to 16 and doubled with each down-sampling operation. Therefore, the down-sampling gradually encode the 3D input into a low-resolution feature representation $F^{(m)} \in \mathbb{R}^{\frac{H}{16} \times \frac{W}{16} \times \frac{D}{16} \times F}$ which is $\frac{1}{16}$ of input dimensions. Furthermore, such backbone can be extended to a multi-path 3D CNN feature extractor, where independent encoders learn intra-modality feature representations meanwhile the Transformer block carries out the inter-modality fusion operation.
<figure><img src="image_9.png" /><figcaption>Figure 7: ResNet block used to build the 3D CNN feature extractor $f_{\Theta}$: (a) down- and (b) up-sampling blocks.</figcaption></figure>

## 4.2. ViT and Swin Transformer setup

The ViT blocks follow the original setup of the base vision Transformer model Dosovitskiy et al. (2021): $L$ = 12 as number of layers, a hidden dimension size of $d$ = 768, 3072 as MLP size, and $h$ = 12 heads. To represent the different types of modality interaction, we extend the original ViT-B/P notation to ViT-$\tau$-B/P where the subscript $\tau$ represents the type of multi-modal fusion and $P$ the input patch size. For instance, a Transformer-based network ViT$_{s}$-B/16 represents the base Transformer variant with input patch size $16 \times 16 \times 16$, and single-stream interaction. For single- and multi-stream interactions, only early concatenation and cross-attention are implemented since they have been widely deployed for multi-modal learning.

On the other hand, the Swin blocks follow the configuration presented in Hatamizadeh et al. (2022a). With a patch size of $P = 2 \times 2 \times 2$, the feature dimension of each patch is equal to $2 \times 2 \times 2 \times M$. Then, the linear embedding layer projects it to a hidden dimension $d = 48$. The Swin encoder has four stages with two Transformer blocks per layer. Hence, the total number of layers $L$ is 8. The hierarchical representations of the Swin encoder are obtained by decreasing the resolution of the feature representation by a factor of two in each stage, giving as results on the bottleneck a resolution of $\frac{H}{16} \times \frac{W}{16} \times \frac{D}{16} \times 4d$. As in the original Swin paper Liu et al. (2021), the self-attention is computed within local windows of size $7 \times 7 \times 7$.

## 4.3. Hybrid CNN-ViT encoder networks

The 3D CNN feature tokenization and embedding step serves as starting point to build a total of six hybrid CNN-ViT encoder networks. The first three models are derived from the one-path 3D CNN feature tokenization block and named as CNN+ViT$_{v}$-B/1 3D-Transfuse and 3D-Swinfuse (see Fig. 1a). CNN+ViT$_{v}$-B/1 is similar to Wang et al. (2021) and only differs in the use of a base Transformer model at the bottleneck level. The 3D CNN feature maps effectively capture the local 3D context information, meanwhile the vanilla Transformer (ViT$_{v}$-B/P) with effective patch size of $1 \times 1 \times 1$ models the global feature
<figure><img src="image_10.png" /><figcaption>Figure 8: Hybrid CNN-ViT encoder networks with multi-path 3D-CNN-based tokenization.</figcaption></figure>

representation. On the other hand, we extended the original Transfuse presented in Zhang et al. (2021c) to a 3D context by using a 3D-ResNet50 and ViT$_v$-B/16 encoders. 3D-Transfuse is a parallel-in branch architecture that combines CNN and Transformer in a parallel style, enabling the capture of both global dependency (ViT$_v$-B/16) and low-level spatial details (3D ResNet50). The independent branches are fused at the different multi-level features by a special BiFusion module that incorporates channel and spatial attention mechanisms. Lastly, the 3D-Swinfuse network keeps the same attributes as the 3D-Transfuse but replaces the vanilla for a Swin Transformer. It is worth mentioning that no modality interaction is applied due to the use of a one-path 3D CNN feature tokenization step which already fuses the modalities at the input level.

The remaining three versions have on the top the multi-path 3D CNN feature tokenization block and are denoted MCNN+ViT$_v$-B/1, MCNN+ViT$_s$-B/1, and MCNN+ViT$_m$-B/1. The multi-stream aspect of these networks (MCNN) captures both inter- and intra-modality interactions. In addition, they lead to more versatile networks where not only modality-specific feature representations are learned, but also false adaptation among modalities is avoided. MCNN+ViT$_v$-B/1 employs a vanilla Transformer where the feature maps extracted across paths are concatenated and passed directly through the transformer block ViT$_v$-B/1. On the other hand, MCNN+ViT$_s$-B/1 and MCNN+ViT$_m$-B/1 carry out single and multi-stream interaction giving to the networks the ability to better model the global multi-modal information either via all-attention or cross-attention. Fig. 8 shows MCNN+ViT$_v$-B/1 architectures meanwhile the first two columns of Tab. 2 summarizes hybrid CNN-ViT encoders.
<figure><img src="image_11.png" /><figcaption>Figure 9: Flowchart detailing the steps followed to build multi-modal Transformers networks for medical image segmentation.</figcaption></figure>

4.4. *Pure Transformer-based encoder networks*

One-path 3D patchify tokenization block is built on the top of pure Transformer-
based encoder networks. Hence, a total of four models can be derived: UNETR<sub>v</sub>,
UNETR<sub>s</sub>, UNETR<sub>m</sub> and Swin UNETR (Fig. 1b. Models derived from multi-
path 3D patchify tokenization are not taken into consideration due to the high
computational cost inquired. UNETR<sub>s</sub> and UNETR<sub>m</sub> models extend previous
works (UNETR<sub>v</sub>) Hatamizadeh et al. (2022b) to a multi-modal context by re-
placing the vanilla Transformer (ViT<sub>v</sub>-B/16) with single and multi-stream inter-
action Transformers (ViT<sub>s</sub>-B/16 and ViT<sub>m</sub>-B/16). For its part, Swin UNETR
takes advantage of the ability of Swin Transformer to handle various scales which
is well-suitable for dense prediction tasks. The pure Transformer-based encoder
networks are summarized in Tab. 2 meanwhile Fig. 9 illustrates a flowchart de-
tailing the sequence of steps required to build the multi-modal medical image
segmentation networks.

5. Experiments

5.1. *Datasets*

To evaluate the effectiveness of the models, two publicly-available datasets
with different imaging modalities were considered: BraTS2021 Baid et al. (2021),
and HECKTOR2021 Oreiller et al. (2022).
<table><caption>Table 2: Hybrids and pure Transformers-based encoder networks. The two first (resp. last) columns correspond to hybrid CNN-ViT (resp. pure Transformers-based) encoder networks.</caption><thead><tr><th colspan="2">Hybrid CNN-ViT encoder</th><th colspan="2">Pure Transformer encoder</th></tr><tr><th colspan="2">3D CNN Tokenization</th><th colspan="2">3D patchify tokenization</th></tr><tr><th>One-path</th><th>Multi-path</th><th>One-path</th><th></th></tr></thead><tbody><tr><td>CNN+VIT<sub>v</sub>-B/1</td><td>MCNN+ViT<sub>v</sub>-B/1</td><td>UNETR<sub>v</sub></td><td>Swin UNETR</td></tr><tr><td>3D-Transfuse</td><td>MCNN+ViT<sub>s</sub>-B/1</td><td>UNETR<sub>s</sub></td><td></td></tr><tr><td>3D-Swinfuse</td><td>MCNN+ViT<sub>m</sub>-B/1</td><td>UNETR<sub>m</sub></td><td></td></tr></tbody></table>

The BraTS2021 dataset Baid et al. (2021) is made up of a 2040 cases of multi-parametric MR scans (mpMRI) with brain tumors, divided into 1251 cases for training, 219 for validation, and 570 for testing. MR scans were obtained from different institutions using various imaging techniques and equipment under standard clinical conditions, resulting in a highly heterogeneous image quality that reflects varying clinical practices across institutions Baid et al. (2021); Menze et al. (2014); Bakas et al. (2017). BraTS2021 stands as a widely recognized dataset in the field of medical imaging. Given its extensive size and complexity, it offers an ideal platform for delving into the potential of multi-modal transformer-based networks. The 3D MR scans within this dataset encompass four distinct modalities: native (T1), post-contrast T1-weighted (T1Gd), T2-weighted (T2), and T2 fluid attenuated inversion recovery (T2-FLAIR). Annotations include gadolinium-enhancing tumor region, peritumoral edematous/in-vaded tissues, and the necrotic tumor core that are combined to get three sub-regions to segment: tumor core (TC), whole tumor (WT), and enhanced tumor (ET). All 3D volumes were skull-striped and re-sampled to 1mm³ isotropic resolution with an input image size of 240×240×155 voxels. Since ground truth annotations are only available on the training set, only this subset was taken into account to carry out experiments. The effective dataset contains 1241 data divided into 1188 training and 63 testing cases (ratio 95:5).

<table><caption>Table 3: Overview of BraTS2021 Baid et al. (2021) and HECKTOR2021 Oreiller et al. (2022) datasets.</caption><thead><tr><th>Dataset name</th><th>Modalities</th><th>Number of centers</th><th>Segmentation task</th><th>Number of classes</th><th>Number of cases Training Testing</th><th>Median shape (spacing in mm)</th></tr></thead><tbody><tr><td>BraTS2021 Baid et al. (2021)</td><td>mpMRI (T1, T1Gd, T2, FLAIR)</td><td>21</td><td>Tumor core (TC), whole tumor (WT) and enhanced tumor (ET)</td><td>3</td><td>1188</td><td>240 × 240 × 155 (1.1.1)</td></tr><tr><td>HECKTOR2021 Oreiller et al. (2022)</td><td>(FDG)-PET, CT</td><td>6</td><td>Primary gross tumor volume (GTVt)</td><td>1</td><td>241</td><td>144 × 144 × 144 (1.1.1)</td></tr></tbody></table>

HECKTOR2021 Oreiller et al. (2022) is a moderately sized dataset with a distinctive feature - it comprises PET-CT multi-modal data. This combination allows for comprehensive assessments of various medical conditions by revealing metabolic activity alongside precise anatomical structures. Moreover, its size corresponds to dimensions commonly employed in clinical trials or public evaluations, rendering it particularly useful for assessing the capabilities of transformer-based networks. The dataset includes 325 fluorodeoxyglucose (FDG)-PET and CT scans from six clinical centers of patients with head and
<figure><img src="image_12.png" /><figcaption>Figure 10: HECKTOR2021 Oreiller et al. (2022) (a) and BraTS2021 Baid et al. (2021) (b) datasets. (a) 2D sagittal slice, from left to right: PET, CT, and fused PET/CT images with corresponding GTVt ground truth segmentation. (b) Axial view of glioma sub-regions, from left to right: whole tumor (WT, cyan) visible in T2-FLAIR, tumor core (TC, magenta) visible in T2, and enhancing tumor (ET, yellow) visible in T1Gd. The last panel depicts the combined delineations generating the final tumor.</figcaption></figure>

neck cancer in the oropharynx region. The goal was to segment the primary Gross Tumor Volume (GTVt) to avoid the costly and error-prone manual or semi-automatic procedure that is required for both radiotherapy planning and radiomics analysis. Training and testing cases represent one 3D FDG-PET volume (in SUV) registered with a 3D CT volume of the head and neck area, as well as the annotated GTVt ground truth. The training data comprises 224 cases from five centers meanwhile 101 cases from two centers were used for testing Oreiller et al. (2022). Since the original data contains regions varying from the head only to the whole body, we limited the analysis to the oropharynx region using a bounding box of size 144×144 ×144 mm³, as suggested by the organizer of the related HECKTOR2021 challenge.

Tab. 3 summarizes the description of the datasets and Fig. 10 displays the imaging modalities per dataset with the corresponding ground truth annotations.
## 5.2. Competing methods

Overall, we compared six Hybrid CNN-VIT encoder and four pure Transformer encoder networks against four state-of-the-art medical image segmentation methods. The baseline methods included 3 convolutional models: U-Net Ronneberger et al. (2015), MU-Net, and nn-UNet Isensee et al. (2020) as well as one full Transformer-based model, nnFormer Zhou et al. (2021a). The U-Net design was made up of four down-sampling and four up-sampling layers, with the encoder and decoder based on the 3D CNN feature extractor described in Sect. 4.1. MU-Net, for its part, extended the U-Net architecture by integrating dedicated encoders for each modality. Then, the individual feature maps were concatenated and processed by the up-sampling CNN blocks. The last convolutional model is the self-configure nn-UNet framework Isensee et al. (2020) which has demonstrated state-of-the-art performance over different datasets. The success of nn-UNet is not leaned on the network, loss function, or training scheme but on the fact to distill knowledge information from a large pool of datasets in a set of heuristic rules. On the other hand, nnFormer is a 3D full Transformer architecture for volumetric medical image segmentation Zhou et al. (2021a). nnFormer introduces local and global volume-based self-attention mechanisms to learn volume representations and replaces the traditional concatenation/-summation operations with skip-connections exploiting attention mechanisms. Tab. 4 summarizes the main features of the fourteen competing models, including the type of model interaction or fusion used as well as the employed architecture.

<table><caption>Table 4: Synthetic comparison of competing methods.</caption><thead><tr><th>Model</th><th>Modality interaction/ fusion</th><th>Category</th><th>Description</th><th>Reference</th></tr></thead><tbody><tr><td>U-Net</td><td>Early fusion</td><td>CNN-based</td><td>Encoder-decoder network made of ResNet blocks with four down-sampling and four up-sampling.</td><td>Ronneberger et al. (2015)</td></tr><tr><td>MU-Net</td><td>Middle fusion</td><td>CNN-based</td><td>Multi-path encoder-decoder network constructed with a CNN encoder per modality.</td><td>—</td></tr><tr><td>nn-UNet</td><td>Early fusion</td><td>CNN-based</td><td>Self-configure U-Net that uses a set of fixed, rule-based and empirical parameters.</td><td>Isensee et al. (2020)</td></tr><tr><td>nnFormer</td><td>Early fusion</td><td>Full Transformer-based</td><td>U-shape network with local and global attention both in the encoder and decoder.</td><td>Zhou et al. (2021a)</td></tr><tr><td>CNN+ViT<sub>3</sub>-B/1</td><td>Early fusion</td><td>Hybrid CNN-VIT encoder</td><td>One-path 3D CNN feature tokenization with vanilla Transformer on the bottleneck.</td><td>Chen et al. (2021c); Wang et al. (2021)</td></tr><tr><td>3D-Transfuse</td><td>Early fusion</td><td>Hybrid CNN-VIT encoder</td><td>Paralell-in-branch 3D ResNet50 and ViT<sub>3</sub>-B/16 encoders.</td><td>Zhang et al. (2021c)</td></tr><tr><td>3D-Swinfuse</td><td>Early fusion</td><td>Hybrid CNN-VIT Encoder</td><td>Paralell-in-branch 3D ResNet50 and Swin encoders.</td><td>—</td></tr><tr><td>MCNN+ViT<sub>3</sub>-B/1</td><td>Middle fusion</td><td>Hybrid CNN-VIT encoder</td><td>Multi-path 3D CNN feature tokenization with vanilla Transformer on the bottleneck.</td><td>—</td></tr><tr><td>MCNN+ViT<sub>3</sub>-B/1</td><td>Single-stream</td><td>Hybrid CNN-VIT encoder</td><td>Multi-path 3D CNN feature tokenization with all attention on the bottleneck.</td><td>—</td></tr><tr><td>MCNN+ViT<sub>m</sub>-B/1</td><td>Multiple-stream</td><td>Hybrid CNN-VIT Encoder</td><td>Multi-path 3D CNN feature tokenization with cross-attention on the bottleneck.</td><td>—</td></tr><tr><td>UNET<sub>r</sub></td><td>Early fusion</td><td>Pure Transformer encoder</td><td>One-path 3D patchify tokenization based on ViT<sub>3</sub>-B/16 Transformer.</td><td>Hatamizadeh et al. (2022b)</td></tr><tr><td>UNET<sub>r</sub></td><td>Single-stream</td><td>Pure Transformer encoder</td><td>One-path 3D patchify tokenization based on ViT<sub>3</sub>-B/16 Transformer.</td><td>—</td></tr><tr><td>UNET<sub>m</sub></td><td>Multiple-stream</td><td>Pure Transformer encoder</td><td>One-path 3D patchify tokenization based on ViT<sub>m</sub>-B/16 Transformer.</td><td>—</td></tr><tr><td>Swin UNET</td><td>Early fusion</td><td>Pure Transformer encoder</td><td>One-path 3D patchify tokenization based on Swin Transformer blocks.</td><td>Hatamizadeh et al. (2022a)</td></tr></tbody></table>
### 5.3. Pre-processing and augmentation strategies

To make a fair comparison among all the competing models, we followed the nn-UNet pre-processing and planning strategy Isensee et al. (2020). The pre-processing step started by cropping all the training cases to their non-zero region, reducing the image size of datasets and improving the computational efficiency. Then, a dataset fingerprint was created to capture dataset-specific properties such as image sizes, voxel spacings, and intensity information (mean, standard deviation, 0.5 and 99.5 percentiles of the foreground regions). During training and inference, each image was normalized independently using z-scoring. All images were re-sampled to the same target spacing using either third order Spline, linear, or nearest interpolation. The target spacing was obtained by collecting all spacings within the training data. For each axis, the median value was selected as default target spacing. Data augmentation followed nn-UNet default strategy, where techniques such as random rotation, scaling, flipping, Gaussian noising, Gaussian blurring, brightness and contrast adjusting, simulation of low resolution, and Gamma transformation were applied on-the-fly Isensee et al. (2019).

### 5.4. Training details

The models were trained for a total of 150 epochs with a batch size of 2. Except for nn-UNet, the hyper-parameters were selected based on the guidance provided in the DeiT paper Touvron et al. (2021), resulting in successful convergence for all architectures. The models used NovoGrad as optimizer, a learning rate of $1 \times 10^{-3}$, a weight decay of 0.05, a dropout rate of 0.1, and a cosine learning rate schedule with a warmup of 30 epochs. For both segmentation tasks, we randomly sampled the input images based on the 3D full-resolution planning of the nn-UNet framework. The size of the random samples for both datasets was (128, 128, 128), with one sample taken from a random location and the other containing at least a foreground class (due to batch size of 2). In the testing stage, we employed the sliding window inference strategy where the window size equals the training sample size, including an overlap portion of 0.5 between the contiguous regions. The segmentation loss computed both Dice and cross-entropy loss and returns their weighted sum.

Results on test data were produced with an averaged ensembling of five models trained and validated on different splits of the training set. For HECKTOR2021, the models were built using leave-one-center-out cross-validation such that the data from four centers were used for training and the data from the fifth center was held out for validation. For BraTS2021, the five models were fitted on the whole dataset's random training/validation splits. In this study, no pre-trained weights were considered, neither for CNN nor the ViT blocks.

### 5.5. Evaluation

The assessment of 3D delineations considered both per-task and multiple-task performance, where HECKTOR2021 and BraTS2021 respectively correspond to Task 1 ($T_1$) and Task 2 ($T_2$). Inspired by the LiTS Bilic et al. (2023),
<table><caption>Table 5: Ingredients and hyper-parameters for competing methods.</caption><thead><tr><th colspan="3">Implementation details</th></tr><tr><th>Dataset</th><th>BraTS2021</th><th>HECKTOR2021</th></tr></thead><tbody><tr><td>Sample size</td><td>(128, 128, 128)</td><td>(128, 128, 128)</td></tr><tr><td>Target spacing (mm)</td><td>1 × 1 × 1</td><td>1 × 1 × 1</td></tr><tr><td>Normalization</td><td colspan="2">Each image is normalized independently by subtracting its mean and dividing by its standard deviation</td></tr><tr><td>Epochs</td><td>150</td><td>150</td></tr><tr><td>Batch size</td><td>2</td><td>2</td></tr><tr><td>Optimizer</td><td>Novograd</td><td>Novograd</td></tr><tr><td>LR</td><td>1 × 10<sup>-3</sup></td><td>1 × 10<sup>-3</sup></td></tr><tr><td>LR decay</td><td>cosine</td><td>cosine</td></tr><tr><td>Decay rate</td><td>—</td><td>—</td></tr><tr><td>Decay epochs</td><td>—</td><td>—</td></tr><tr><td>Weight decay</td><td>0.05</td><td>0.05</td></tr><tr><td>Warmup epochs</td><td>30</td><td>30</td></tr><tr><td>Dropout</td><td>0.1</td><td>0.1</td></tr><tr><td>Stoch. depth</td><td>✗</td><td>✗</td></tr><tr><td>Gradient clip</td><td>✗</td><td>✗</td></tr><tr><td>nnUNet aug</td><td>✓</td><td>✓</td></tr><tr><td>EMA</td><td>✗</td><td>✗</td></tr><tr><td>Mixed precision</td><td>✓</td><td>✓</td></tr><tr><td>Loss</td><td>Dice+CE</td><td>Dice+CE</td></tr></tbody></table>

HECKTOR2021 Oreiller et al. (2022); Wiesenfarth et al. (2021), and BraTS2021 Baid et al. (2021) medical image segmentation challenges; performance was assessed as described below.

### 5.5.1. Segmentation metrics

For data analysis and visualization of model performance, we mainly followed the metrics used in the related challenges. The Dice similarity coefficient (DSC) and the average symmetric surface distance (ASSD) were used for $T_1$, whereas only DSC was employed for $T_2$. The metrics were illustrated using boxplots for each model separately, representing the descriptive statistics over all cases (median, quartiles, and outliers), where the horizontal out-of-the-box dots and whiskers represent the outliers.

### 5.5.2. Ranking strategy

The design choices of the ranking scheme on one or multiple metric(s) are as follows: whether to perform metric-based (aggregate, then rank) or case-based (rank, then aggregate) and whether to take the mean or the median for aggregation. In this work, the models were ranked using the Rank-Then-Mean strategy. First, a rank was computed for each test case for each model ('rank first'), obtaining the ranking per task by aggregating each of the test cases' ranks using the mean. In the end, the consensus ranking across tasks was obtained by averaging the per-task cumulative ranks and re-ranking them Wiesenfarth et al. (2021). To evaluate the robustness of rankings, we employed line plots to
examine how rankings changed when different ranking methods were applied. For these purposes, we consider three additional ranking strategies; rank-then-median, mean-then-rank, and median-then-rank. In these plots, each model was represented by a unique colored line. The x-axis illustrates the various ranking methods, and the y-axis indicates the corresponding rank of the model. If all of the lines are parallel, it suggests that the rankings are consistent across methods. However, if the lines intersect, it indicates that the rankings are sensitive to the ranking method employed. These line plots provided us with a clear and concise visual representation of the ranking robustness for a given task.

### 5.5.3. Significance testing

The pair-wise statistical comparison of the fourteen models was illustrated by a significance matrix computed with a one-sided Wilcoxon signed-rank test at 5% significance with adjustment for multiple testing according to Holm Wiesenfarth et al. (2021). The statistical significance of the relative rankings between each pair of models was conducted by permutation testing. Specifically, for each pair of models, we repeatedly and randomly permuted the cumulative Ranks for each subject ($b = 10000$ times). For each permutation, we calculated the difference in the final ranking between the pair of models. The proportion of times the difference in the final ranking calculated using the permuted data exceeded the observed difference in the final ranking was used as a p-value to determine the statistical significance of their relative rankings. These p-values were reported in an upper triangular matrix, thus providing insights into statistically significant differences across each pair of participating models Baid et al. (2021). Yellow shading indicates that the metric values from the algorithm on the x-axis are significantly superior to those from the algorithm on the y-axis. Blue shading indicates no significant difference. In an ideal scenario, all models with smaller ranks would be significantly better than models with larger ranks, resulting in a yellow area above the diagonal and a blue area below.

### 5.5.4. Cost indicators

The trade-off between quality and cost is a major consideration in the design of neural network architectures. Aside from model quality, the efficiency of a model is a crucial factor to consider Dehghani et al. (2022). There are several widely accepted methods for evaluating and reporting model efficiency. Here, we focused on two of the most commonly used methods: the number of trainable parameters and the number of multiply-accumulate (MACs). These indicators take different points of view. For instance, the number of trainable parameters gives insights into the computational complexity as well as memory usage (during inference) whereas MACs bring information about the computational cost of a given model.

### 5.5.5. Predicted segmentation masks

A visual comparison of predicted segmentation masks and attention maps was provided. This information provides insights into the relevant features that
each hybrid and Transformer-based models focus on to perform 3D segmenta-
tion. Specifically, we visualized the attention maps extracted by self-, all-, and
cross-attention blocks with the original image resolution.

6. Results and discussion

The results are reported and discussed, taking into account performance
metrics, ranking strategies, significance testing analysis, and cost indicators.
This detailed examination provides a full assessment of the model performance
and enables to make informed decisions about their suitability for various use-
cases.

6.1. Overall segmentation performance

For $T_1$ task the models range from an average DSC of 68.0 to 76.0%, obtaining the best overall DSC with nn-UNet. On the other hand, ASSD median values range from 0.97 to 2.24mm with both MCNN+ViT$_v$-B/1 and nn-UNet achieving the minimal value of 0.97mm among all models. We chose to report the median for ASSD since a value of $+\infty$ is attributed when the prediction is null. Beyond the multi-path hybrid CNN-ViT encoder networks, CNN+ViT$_v$-B/1 has the best DSC and ASSD scores, followed by 3D-Swinfuse and 3D-Transfuse networks. When only pure transformer-based encoder networks are considered, the hierarchical Transformer Swin UNETR outperforms their competitors, achieving an overall DSC of 75.5% and a median ASSD of 1.01mm. UNETR$_v$ comes in second with 74.8% and 1.14mm overall DSC and median ASSD, respectively. The full Transformer-based network nnFormer performs poorly, obtaining values of 72.6% and 1.41mm for DSC and ASSD, respectively. Besides from nn-UNet, the multi-path CNN-based network MU-Net achieves competing results in both metrics (DSC=75.9% and ASSD=1.03mm), highlighting the importance of multi-modal approaches. Lastly, the worst DSC and ASSD were obtained for UNETR$_m$ and UNETR$_s$ models. The distributions of DSC and ASSD across patients and models are illustrated in Fig. 11a and 11b, respectively.

For the case of task $T_2$, we report the average DSC independently for each of
the three sub-regions to segment (TC, WT, and ET). The average DSC for class
TC ranges from 73.6 to 88.8%, with the highest overall value reached with both
MCNN+ViT$_s$-B/1 and nn-UNet. The average DSC for class WT ranges from
81.7 to 93.5%, with the best average DSC obtained with MCNN+ViT$_v$-B/1.
Lastly, DSC for class ET ranges from 76.7 to 87.2% with MCNN+ViT$_m$-B/1
obtaining the highest value. It is important to mention that the performance
of models differs more in $T_2$ than in $T_1$. For instance, the difference between
models in $T_1$ is found out around 6% for DSC, whereas in $T_2$, depending on the
classes, we might achieve a maximum difference of 15.2%. This might be con-
nected to the intrinsic characteristics of both datasets, specifically its multi-class
and multi-modal nature. As in $T_1$, Swin UNETR has the best DSC performance
among the pure Transformer-based encoder networks, with a significant advan-
tage over the second (UNETR$_v$). nnFormer, on the other hand, has made sig-
nificant progress, with overall DSC values of 86.4, 91.5, and 85.2% for TC, WT,
<figure><img src="image_13.png" /><figcaption>Figure 11: Box plots corresponding to DSC (a) and ASSD (b) distributions for task $T_1$, ordered by decreased Rank-Then-Mean consensus.</figcaption></figure>

and ET, respectively. Such a performance might be related to the increasing number of training examples used in $T_2$. The worst DSC performance was also obtained with UNETR modality interaction variants, UNETR<sub>s</sub> and UNETR<sub>m</sub>. The distributions of DSC across patients and models are illustrated in Fig. 12 for TC, WT and ET classes.

In terms of Rank-Then-Mean strategy, the top three models for both DSC and ASSD in $T_1$ are MCNN+ViT<sub>s</sub>-B/1, followed by MCNN+ViT<sub>v</sub>-B/1, and Swin UNETR. As a result, the ranking consensus (R) across metrics for $T_1$ keeps the same top three models in the same order as the best performers. The distribution of the cumulative rankings CR across metrics are shown in Fig. 13a. We can observe from the box plots that, except for Swin UNETR, MCNN+ViT<sub>m</sub>-B/1, and MU-net, all models have ranked last at least once.
<figure><img src="image_14.png" /><figcaption>(a) DSC distribution for TC</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>(b) DSC distribution for WT</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>(c) DSC distribution for ET</figcaption></figure>

Figure 12: Box plots of DSC distributions for task $T_2$ for each of the three sub-regions to segment (TC, WT, and ET), ordered by decreased Rank-Then-Mean consensus (R).
<figure><img src="image_17.png" /><figcaption>Figure 13: Box plots distributions of cumulative rankings for $\mathcal{T}_1$ (a) and $\mathcal{T}_2$ (b) tasks, ordered by decreased Rank-Then-Mean consensus (R).</figcaption></figure>

Furthermore, we notice that nn-UNet has ranked first the most times, but it is also the model with the most intra-variability among the ranking of the test cases. In general, hybrid models with multi-path 3D CNN feature tokenization performed the best, closely followed by Swin UNETR and CNN-based models (nn-UNet and MU-net).

Apart from the MCNN+ViT$_s$-B/1 model that ranks first for the three sub-regions, task $\mathcal{T}_2$ presents more disparity among the ranking depending on the sub-region to segment. MCNN+ViT$_v$-B/1 and MU-Net rank second and third in the TC sub-region, respectively. In the WT sub-region, nn-UNet reaches the
second rank, followed by MCNN+ViT<sub>v</sub>-B/1. Finally, for the ET sub-region, MCNN+ViT<sub>v</sub>-B/1 and Swin UNETR rank second and third, respectively. As a result, the top three models obtained by ranking consensus (R) across sub-regions are MCNN+ViT<sub>v</sub>-B/1, followed by MCNN+ViT<sub>v</sub>-B/1, and nn-UNet. The final ranking for T<sub>2</sub> can be derived by simply looking at the cumulative rankings CR plot in Fig. 13b. However, this is not the case for T<sub>1</sub> (Fig. 13a). Tab. 6 summarizes the model's metrics, average cumulative ranking per-metric/sub-regions CR, average cumulative ranking per-task CR, ranking consensus per-metric/sub-region (R), ranking consensus per-task (R), and the final ranking across tasks.

<table><caption>Table 6: Ranking results based on the Rank-Then-Mean strategy. Metrics, average cumulative ranking per metric/sub-regions CR, average cumulative ranking per task (CR), ranking consensus per-metric/sub-region (R), ranking consensus per task (R), and the final ranking are reported. The top three models are highlighted in yellow boxes based on ranking and metrics values.</caption><thead><tr><th rowspan="3">Models</th><th colspan="6">HECKTOR2021 (T<sub>1</sub>)</th><th colspan="6">BraTS2021 (T<sub>2</sub>)</th></tr><tr><th colspan="3">Rank-Then-Mean</th><th rowspan="2">Ranking<br>consensus</th><th colspan="3">Rank-Then-Mean</th><th colspan="3">Ranking<br>consensus</th><th rowspan="2">Final<br>Ranking</th></tr><tr><th>DSC<br>Mean</th><th>CR<br>(R)</th><th>Median<br>CR<br>(R)</th><th>TC DSC<br>Mean</th><th>WT DSC<br>Mean</th><th>ET DSC<br>Mean</th><th>CR<br>(R)</th><th>CR<br>(R)</th></tr></thead><tbody><tr><td>● MCNN+ViT<sub>v</sub>-B/1</td><td>75.9</td><td>5.86</td><td>(1)</td><td>1.01</td><td>5.53</td><td>(1)</td><td>5.70</td><td>(1)</td><td>88.8</td><td>3.44</td><td>(1)</td><td>93.3</td><td>3.16</td><td>(1)</td><td>86.4</td><td>2.90</td><td>(1)</td><td>3.17</td><td>(1)</td><td>4.43</td><td>(1)</td></tr><tr><td>● MCNN+ViT<sub>v</sub>-B/1</td><td>75.9</td><td>6.03</td><td>(2)</td><td>0.97</td><td>5.61</td><td>(2)</td><td>5.82</td><td>(2)</td><td>87.6</td><td>4.03</td><td>(2)</td><td>93.5</td><td>3.71</td><td>(3)</td><td>86.2</td><td>3.83</td><td>(2)</td><td>3.86</td><td>(2)</td><td>4.84</td><td>(2)</td></tr><tr><td>● Swin UNETR</td><td>75.5</td><td>6.08</td><td>(3)</td><td>1.01</td><td>5.93</td><td>(3)</td><td>6.00</td><td>(3)</td><td>87.7</td><td>4.49</td><td>(5)</td><td>92.7</td><td>4.73</td><td>(5)</td><td>85.5</td><td>3.95</td><td>(3)</td><td>4.39</td><td>(4)</td><td>5.20</td><td>(3)</td></tr><tr><td>● MCNN+ViT<sub>v</sub>-B/1</td><td>75.5</td><td>6.40</td><td>(6)</td><td>1.02</td><td>6.11</td><td>(4)</td><td>6.25</td><td>(4)</td><td>88.7</td><td>4.63</td><td>(6)</td><td>92.8</td><td>4.65</td><td>(4)</td><td>87.2</td><td>4.51</td><td>(4)</td><td>4.60</td><td>(6)</td><td>5.43</td><td>(4)</td></tr><tr><td>● CNN+ViT<sub>v</sub>-B/1</td><td>75.5</td><td>6.37</td><td>(5)</td><td>1.03</td><td>6.22</td><td>(5)</td><td>6.29</td><td>(5)</td><td>86.0</td><td>5.98</td><td>(7)</td><td>90.7</td><td>7.43</td><td>(7)</td><td>84.1</td><td>6.14</td><td>(7)</td><td>6.52</td><td>(7)</td><td>6.41</td><td>(7)</td></tr><tr><td>● MU-Net</td><td>75.9</td><td>6.36</td><td>(4)</td><td>1.03</td><td>6.35</td><td>(6)</td><td>6.35</td><td>(6)</td><td>87.8</td><td>4.19</td><td>(3)</td><td>92.3</td><td>5.05</td><td>(6)</td><td>86.6</td><td>4.51</td><td>(4)</td><td>4.58</td><td>(5)</td><td>5.47</td><td>(5)</td></tr><tr><td>● nn-UNet</td><td>76.0</td><td>7.01</td><td>(7)</td><td>0.97</td><td>6.48</td><td>(7)</td><td>6.74</td><td>(7)</td><td>88.8</td><td>4.27</td><td>(4)</td><td>92.8</td><td>3.48</td><td>(2)</td><td>86.6</td><td>4.87</td><td>(6)</td><td>4.21</td><td>(3)</td><td>5.47</td><td>(5)</td></tr><tr><td>● UNETR<sub>s</sub></td><td>74.8</td><td>7.19</td><td>(8)</td><td>1.14</td><td>7.29</td><td>(8)</td><td>7.24</td><td>(8)</td><td>80.7</td><td>10.14</td><td>(11)</td><td>87.5</td><td>9.86</td><td>(10)</td><td>81.3</td><td>8.84</td><td>(10)</td><td>9.61</td><td>(10)</td><td>8.43</td><td>(9)</td></tr><tr><td>● U-Net</td><td>73.2</td><td>7.76</td><td>(9)</td><td>1.15</td><td>7.84</td><td>(9)</td><td>7.80</td><td>(9)</td><td>84.7</td><td>8.70</td><td>(9)</td><td>89.8</td><td>8.37</td><td>(9)</td><td>83.1</td><td>8.17</td><td>(9)</td><td>8.41</td><td>(9)</td><td>8.11</td><td>(8)</td></tr><tr><td>● 3D-Swinfuse</td><td>75.0</td><td>8.21</td><td>(11)</td><td>1.15</td><td>8.25</td><td>(10)</td><td>8.23</td><td>(10)</td><td>84.4</td><td>10.32</td><td>(12)</td><td>89.9</td><td>10.79</td><td>(12)</td><td>81.6</td><td>11.38</td><td>(13)</td><td>10.83</td><td>(12)</td><td>9.53</td><td>(12)</td></tr><tr><td>● nnFormer</td><td>72.6</td><td>8.19</td><td>(10)</td><td>1.41</td><td>9.22</td><td>(12)</td><td>8.70</td><td>(11)</td><td>86.4</td><td>8.00</td><td>(8)</td><td>91.5</td><td>8.11</td><td>(8)</td><td>85.2</td><td>8.03</td><td>(8)</td><td>8.05</td><td>(8)</td><td>8.38</td><td>(9)</td></tr><tr><td>● 3D-Transfuse</td><td>74.0</td><td>8.44</td><td>(12)</td><td>1.12</td><td>8.61</td><td>(11)</td><td>8.52</td><td>(12)</td><td>85.0</td><td>9.52</td><td>(10)</td><td>90.3</td><td>10.30</td><td>(11)</td><td>81.5</td><td>10.76</td><td>(11)</td><td>10.20</td><td>(11)</td><td>9.36</td><td>(11)</td></tr><tr><td>● UNETR<sub>s</sub></td><td>69.4</td><td>9.58</td><td>(13)</td><td>1.71</td><td>10.41</td><td>(13)</td><td>10.00</td><td>(13)</td><td>73.6</td><td>12.29</td><td>(14)</td><td>81.7</td><td>12.59</td><td>(13)</td><td>76.7</td><td>11.29</td><td>(12)</td><td>12.05</td><td>(13)</td><td>11.02</td><td>(13)</td></tr><tr><td>● UNETR<sub>m</sub></td><td>68.0</td><td>10.66</td><td>(14)</td><td>2.24</td><td>11.16</td><td>(14)</td><td>10.91</td><td>(14)</td><td>74.5</td><td>12.08</td><td>(13)</td><td>82.2</td><td>12.76</td><td>(14)</td><td>76.7</td><td>11.44</td><td>(14)</td><td>12.10</td><td>(14)</td><td>11.50</td><td>(14)</td></tr></tbody></table>

## 6.2. Ranking robustness

Since rankings are largely dependent on the ranking method applied, we illustrate in Fig. 14a and 14b the ranking robustness with respect to different ranking methods for task T<sub>1</sub> and T<sub>2</sub>, respectively. That is, if one model outperforms the others, then whatever ranking method used should yield the same ranking position. In an ideal situation, the best model would remain first regardless of the ranking method applied. Previous findings in Sect. 6.1 suggest a more evident panorama of the rankings in T<sub>2</sub> compared with T<sub>1</sub> (Fig. 13). This fact is corroborated by Fig. 14, where models in T<sub>2</sub> are substantially more robust across ranking methods than in T<sub>1</sub>. Furthermore, MCNN+ViT<sub>s</sub>-B/1 in T<sub>2</sub> scores best in three out of four ranking techniques, indicating its robustness across different ranking approaches. This is also true for MCNN+ViT<sub>v</sub>-B/1 and nn-UNet, which rank second and third in three of the four ranking techniques,
respectively. On the other hand, rankings in $T_1$ depend far more on the ranking method. One significant observation is that the MCNN+ViT$_v$-B/1 model appears more stable, ranking top and second across all ranking techniques. Interestingly, nn-UNet ranks first in both tasks when the Mean-Then-Rank strategy is used, suggesting that the outlying test cases has less impact on the model ranking on average.

<figure><img src="image_18.png" /><figcaption>Figure 14: Line plots for visualizing ranking robustness across different ranking methods; R1 (Rank-Then-Mean (Ours)), R2 (Rank-Then-Median) R3 (Mean-Then-Rank) and R4 (Median-Then-Rank). Each model is represented by one colored line that varies according to the rank (y-axis) achieved with each ranking method (x-axis).</figcaption></figure>

### 6.3. Significance map analysis

The significance map in Fig. 16 confirms that task $T_2$ provides a clear ranking of the algorithms with the top-ranked models separating from the remaining algorithms, while in $T_1$ the uncertainty is too large to provide a meaningful
ranking (Fig. 15). It is worst to mention that the fact that MCNN+ViT<sub>s</sub>-B/1 ranks higher than the other models in terms of significance in T<sub>1</sub> does not imply that MCNN+ViT<sub>s</sub>-B/1 is substantially superior to them, as revealed by the significance map. But, it is also true that MCNN+ViT<sub>s</sub>-B/1 has more models with a substantial difference. For instance, it is significantly better in 5 out of 13 available models (Fig. 15a) compared to the only two achieved with nn-UNet and U-Net CNN-based models. Another interesting finding shows that MCNN+ViT<sub>s</sub>-B/1 has a significance difference with a plain U-Net and a full Transformer-based network (nnFormer). Similar findings can be observed with MCNN+ViT<sub>v</sub>-B/1, besides the fact that it is only significantly better in four models and there is no significant difference with U-Net in case of DSC. Lastly, most of the models show a considerable difference compared to UNETR<sub>s</sub> and UNETR<sub>m</sub>, confirming their poor performance.

The panorama is clearer for task $\mathcal{T}_2$, and especially when focusing on DSC for class ET. The top five ranked models (MCNN+ViT<sub>s</sub>-B/1, MCNN+ViT<sub>v</sub>-B/1, Swin UNETR, MU-Net, and MCNN+ViT<sub>m</sub>-B/1) have a substantial difference with the remainder of the models (excluding nn-UNet). A similar conclusion can be drawn from Fig. 16b for the case of DSC in TC. Fig. 16c shows a noteworthy discovery for DSC in WT where MCNN+ViT<sub>s</sub>-B/1 reveals a significant difference in 11 of the 13 existing models, especially outstanding when compared to models such as Swin UNETR, MU-Net and MCNN+ViT<sub>m</sub>-B/1. In addition, multi-path 3D CNN-ViT encoder networks differ significantly from their one-path 3D counterparts, which clearly emphasizes the significance of including multiple encoders, one for each modality. In regards to pure Transformer-based encoder networks, Swin UNETR outperforms in eight out of thirteen models, making it the best in its class. In contrast, a full Transformer-based network such as nnFormer outperforms 3D-Swinfuse, 3D-Transfuse, UNETR<sub>v</sub> as well as its variants (UNETR<sub>s</sub> and UNETR<sub>m</sub>), indicating that the performance may be connected to the number of training samples.

### 6.4. *Impact of cost indicators*

To balance model performance and cost, we examine the number of parameters as well as the number of MACs operations required by each model (Fig. 17). The number of parameters in a model tells us about the amount of memory required to store weights and biases. As the number of parameters increases, so does the amount of memory required to store the models. Therefore, training and inference time increase, as well as higher storage and computing expenses, are thus required. On the other hand, the number of MAC operations shows how much processing the models need. As the number of MAC operations grows, so does the calculation required by the models, which has an immediate impact on computing costs as well as training and inference time.

The number of parameters in millions and the number of Tera MACs used per model are depicted in Fig. 17a and Fig. 17a, respectively. Each model is color-coded according to the notation presented in Tab. 4. In addition, the area of the colored circles serves as a visual representation of the model size and MAC operations. From a close look at the different model architectures (x-axis),
<figure><img src="image_19.png" /><figcaption>Figure 15: Visualization of significance testing maps for $T_1$.</figcaption></figure>

805

specifically the number of parameters, we note that the only hybrid CNN-ViT model impacted by the growing number of modalities among tasks $T_1$ and $T_2$ is MCNN+ViT$_m$-B/1 (●) (from 226.65 to 533.09 M). The same effect is observed on the pure ViT-based models where UNETR$_m$ (●) go from 213.10 to 518.05 million of parameters. Consequently, models with cross-attention blocks require
a significantly greater amount of memory to hold the model’s weights and biases. Lastly, full-Transformer and CNN-based models do not exhibit any appreciable variation in the number of parameters when the number of modalities increases.

On the other hand, adding modalities appears to have a considerable im-
pact on Tera MACs computation on multi-path 3D CNN-ViT encoder net-
works. Models such as MCNN+ViT<sub>s</sub>-B/1 (●), MCNN+ViT<sub>m</sub>-B/1 (●), and
MCNN+ViT<sub>v</sub>-B/1 (●) reveal a significant increase in computation, passing
from 1.04 to 3.53 Tera MACs. A similar large disparity can be seen on MU-Net
(●) (0.83 to 3.02 Tera MACs). This difference is less significant in models like
UNETR<sub>s</sub> (●) and UNETR<sub>m</sub> (●), where the computation ranges around 0.48 to
0.78 Tera MACs. This discovery suggests that most of the computation comes
from the architecture building process (i.e., one encoder per modality). Other
models do not present meaningful differences when the number of modalities
increases.

6.5. *Qualitative segmentation results*

We first visually assess the different models' architectures for the primary gross tumor segmentation task (Fig. 18). Each column represents a different kind of architecture, whereas the rows show 2D sagittal slices coming from four test cases. In general, all of the models exhibit over- and under-segmentation errors as compared to the ground truth mask (magenta line), which is most visible in CHUP052/93 (over-segmented). U-Net (●) seems to be the worst around the CNN-based models. Meanwhile, MU-Net (●) and nnUNet (●) tend to perform similarly. Among the pure ViT-based models, UNETR<sub>s</sub> (●) and UNETR<sub>m</sub> (●) appear to be more prone to over-segmentation issues. This phenomenon is seen notably in CHUP042/59 and CHUV027/85 cases, where both models hallucinate tumor regions. However, UNETR (●), Swim UNETR (●) and full Transformer model nnFormer (●) do not exhibit such a problem, implying that this concern could be tied to the modality interaction block. Multi-path hybrid CNN-ViT models (MCNN+ViT<sub>v</sub>-B/1 (●), MCNN+ViT<sub>m</sub>-B/1 (●) and MCNN+ViT<sub>s</sub>-B/1 (●)) behave similarly without hallucinating regions. Regardless of whether single or multiple modality interaction techniques are used, which could be attributable to the incorporation of local inductive bias in the form of multi-path 3D CNN feature tokenization. Finally, no significant visual differences are found between the CNN+ViT<sub>v</sub>-B/1 (●), 3D-Transfuse (●), and 3D-Swinfuse (●).

Visual comparison of the five top-ranked model approaches (MCNN+ViT<sub>s</sub>-B/1, MCNN+ViT<sub>v</sub>-B/P, nn-UNet, Swin UNETR, and MU-Net) provided visual in-
sights on the segmentation quality of whole tumor, tumor core, and enhanc-
ing tumor glioblastoma sub-regions (Fig.19). Overall, the top five models out-
put well-delineated segmentation for all three sub-regions. Only minor dif-
erences are seen based on the test case examined. For instance, the models
do not completely cover the whole tumor region in T<sub>1</sub>-01608/95. In contrast,
the models over-delineate the core tumor area in T<sub>1</sub>-01621/88, and the whole
tumor in T<sub>1</sub>-01636/80. Furthermore, we notice that all of the models par-
tially fail in segmenting the enhanced tumor sub-region, while a couple of them
(MCNN+ViT<sub>v</sub>-B/P and Swin UNETR) fail to completely delineate the core tumor.

7. Conclusion

In this work, we introduced a complete unified mathematical framework to build diverse multi-modal medical Transformers for medical image segmentation in oncology. We conducted a series of controlled studies to contrast and better understand the difference in the performance of various models: six hybrid CNN-ViT encoder-based models, four pure Transformer encoder networks, one full Transformer-based model, and three CNN-based models. We found out that the multi-path hybrid model named MCNN+ViT<sub>s</sub>-B/1 rank first in 3 out of 4 ranking strategies, while being only surpassed in the mean-and-then ranking method by nn-UNet network. However, depending on the task, MCNN+ViT<sub>s</sub>-B/1 reflects more or less significant differences regarding the rest of the models. This conclusion is comparable with those seen in other models when comparing the performance obtained from HECKTOR2021 and BraTS2021 datasets. For instance, in CNN-based models, a simple implementation of U-Net displayed a significant decrease in performance in comparison to more elaborate models such as nn-UNet and MU-Net. The same results are reported when comparing one-path encoder architectures to multi-path encoder frameworks. These results suggest that under large, multi-modal, multi-label, and high-quality datasets, it is possible to benefit from more elaborate models with less inductive bias based on Transformers or multi-path CNN encoder networks.

Concerning the use of 3D CNN feature tokenization (hybrid CNN-ViT en-
coder) or 3D patchify tokenization (pure Transformers-based encoder) strategy,
there is no question that hybrid approaches outperform pure Transformers-
based models. Despite full Transformers-based models like nnFormer consid-
erably boost their performance when moving from T₁ to T₂ task. However,
we are aware that the performance advantages of hybrid CNN-Transformers
networks are primarily due to the multi-path CNN encoders rather than the
Transformer itself. This is demonstrated in the statistical significance analysis,
where MCNN+ViT<sub>s</sub>-B/1 only outperform MU-Net on whole tumor segmenta-
tion in task T₂. A similar discovery was made recently in Roy et al. (2022),
where Transformer blocks were removed and replaced with a linear projection.
Therefore, we suggest caution while designing hybrid CNN-Transformers en-
coders networks, particularly given the influence of spatial downsampling on
Transformers' capacity to describe long-range dependencies. There were no no-
table differences in the modality interaction method between single- and dual-
stream models, either for hybrid CNN-ViT encoder or pure Transformer-based
networks.

On the other hand, we observed that the number of MACs operation in multi-
path hybrid models increases significantly with the number of modalities which
is not the case for CNN-based models (nn-UNet and U-Net). This demonstrates
two essential points: a well-trained U-Net as nn-UNet is still a good enough
network to solve multi-modal medical image segmentation. As a result, given
the trade-off quality cost, we believe that a solution based on CNN models is still
the paradigm to follow in the case of multi-modal medical image segmentation
with medium to small datasets. Because of advancements in GPU memory
capacity and processing speed, larger models and batch sizes are now possible,
also patch size, and target spacing can be easily extended which could also
improve 3D CNN-based models.

We recommend the use of nn-UNet pipeline (pre-processing and training scheme) as a starting point to evaluate the performance of different medical image segmentation architectures. We learned that having a well-established pipeline reduces most design problems and allows to focus on architecture more. Since medicine is inherently a multi-modal discipline. We believe that, in addition to considering imaging modalities, other data such as clinical notes, lab tests, electronic health records, genomics, and more can significantly enhance medical diagnostics, particularly in oncology. Concurrently, the capacities and capabilities of large language models (LLMs) have advanced to the point where they demonstrate comprehension and expertise in medical knowledge. Therefore, we believe they should be leveraged to enhance the capabilities of current learning approaches. Last but certainly not least, it is crucial to develop new and more robust foundation models trained on extensive datasets using self-supervised learning. They will serve as the foundation for new multi-modal medical image segmentation models.

Our findings, we believe, will inspire additional controlled analyses of novel
proposed architectures based on Transformers for multi-modal medical image
segmentation in oncology.

CRediT author statement

**Gustavo Andrade-Miranda:** Conceptualization, methodology, software, validation, formal analysis, investigation, writing - original draft, visualization.

**Vincent Jaouen:** Conceptualization, methodology, writing - original draft, supervision, funding acquisition.

**Olena Tankyevych:** Data curation, resources.

**Catherine Cheze Le Rest:** Data curation, resources.
**Dimitris Visvikis:** Supervision, project administration.
**Pierre-Henri Conze:** Conceptualization, methodology, writing - original draft, writing - reviewing and editing, supervision, funding acquisition.

Acknowledgments

This work benefited from state aid managed by the National Research Agency under the Future Investment Program bearing the reference ANR-17-RHUS-0005 (FollowKnee project). This work was also partially funded by France Life Imaging (grant ANR-11-INBS-0006).
References

Akbari, H., Yuan, L., Qian, R., Chuang, W.-H., Chang, S.-F., Cui, Y., & Gong, B. (2021). Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In *Advances in Neural Information Processing Systems*.

940

Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., & Schmid, C. (2021). Vivit: A video vision transformer. In *International Conference on Computer Vision (ICCV)* (pp. 6836–6846).

945

Baid, U., Ghodasara, S., Bilello, M., Mohan, S., Calabrese, E., Colak, E., Farahani, K., & et al., J. K. (2021). The RSNA-ASNR-MICCAI BraTS 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv:2107.02314.

950

Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J. S., Frey- mann, J. B., Farahani, K., & Davatzikos, C. (2017). Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. *Scientific data*, 4, 170117.

Bao, H., Dong, L., & Wei, F. (2022). BEiT: BERT pre-training of image transformers. In *International Conference on Learning Representations (ICLR)*.

955

Bao, R., Song, Y., Bates, S. V., Weiss, R. J., Foster, A. N., Cobos, C. J., Sotardi, S., Zhang, Y., Gollub, R. L., Grant, P. E., & Ou, Y. (2023). Boston neonatal brain injury dataset for hypoxic ischemic encephalopathy (bonbid-hie): Part i. mri and manual lesion annotation. *bioRxiv*, .

960

Bello, I., Zoph, B., Le, Q., Vaswani, A., & Shlens, J. (2019). Attention augmented convolutional networks. In *International Conference on Computer Vision (ICCV)* (pp. 3285–3294).

Bilic, P., Christ, P., Li, H. B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szék skin, A., Jacobs, C., Mamani, G. E. H., Chartrand, G., Lohöfer, F., Holch, J. W., Sommer, W., Hofmann, F. et al. (2023). The liver tumor segmentation benchmark (lits). *Medical Image Analysis*, 84, 102680.

965

Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., & Wang, M. (2022). Swin-unet: Unet-like pure transformer for medical image segmentation. In *European Conference on Computer Vision (ECCV)*.

970

Cao, Y., Xu, J., Lin, S., Wei, F., & Hu, H. (2019). Gcnet: Non-local networks meet squeeze-excitation networks and beyond. *International Conference on Computer Vision (ICCV)*, (pp. 1971–1980).

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-end object detection with transformers. In *European Conference on Computer Vision (ECCV)* (pp. 213–229).

975
Chen, B., Liu, Y., Zhang, Z., Lu, G., & Zhang, D. (2021a). Transattunet: Multi-level attention-guided u-net with transformer for medical image segmentation. CoRR, abs/2107.05274.

Chen, C.-F. R., Fan, Q., & Panda, R. (2021b). Crossvit: Cross-attention multi-scale vision transformer for image classification. In International Conference on Computer Vision (ICCV) (pp. 357–366).

Chen, J. et al. (2021c). Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306, .

Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., & Sutskever, I. (2020a). Generative pretraining from pixels. In H. D. III, & A. Singh (Eds.), International Conference on Machine Learning (ICML) (pp. 1691–1703). PMLR volume 119.

Chen, Y., Chen, J., Wei, D., Yuexiang, L., & Yefeng, Z. (2020b). Octopusnet: A deep learning segmentation network for multi-modal medical images. In International Workshop on Multiscale Multimodal Medical Imaging (pp. 17–25).

Conze, P.-H., Kavur, A. E., Cornec-Le Gall, E., Gezer, N. S., Le Meur, Y., Selver, M. A., & Rousseau, F. (2021). Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks. Artificial Intelligence in Medicine, 117, 102109.

Cordonnier, J., Loukas, A., & Jaggi, M. (2020). On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations (ICLR).

Dana, J., Agnus, V., Ouhmich, F., & Gallix, B. (2020). Multimodality imaging and artificial intelligence for tumor characterization: Current status and future perspective. Seminars in Nuclear Medicine, 50, 541–548.

Dehghani, M., Tay, Y., Arnab, A., Beyer, L., & Vaswani, A. (2022). The efficiency misnomer. In International Conference on Learning Representations (ICLR).

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional Transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4171–4186).

Dong, B., Wang, W., Fan, D.-P., Li, J., Fu, H., & Shao, L. (2021a). Polyp-pvt: Polyp segmentation with pyramid vision transformers.

Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., & Guo, B. (2021b). Cswin transformer: A general vision transformer backbone with cross-shaped windows.
Dorent, R., Kujawa, A., Ivory, M., Bakas, S., Rieke, N., Joutard, S., Glocker, B., Cardoso, J., & et al. (2023). Crossmoda 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation. *Medical Image Analysis*, 83, 102628.

1015

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In *International Conference on Learning Representations (ICLR)*.

1020

Duran, A., Dussert, G., Rouvière, O., Jaouen, T., Jodoin, P.-M., & Lartizien, C. (2022). ProstAttention-Net: A deep attention model for prostate cancer segmentation by aggressiveness in MRI scans. *Medical Image Analysis*, 77, 102347.

1025

Gatidis, S., Früh, M., Fabritius, M., & et al. (2023). The autopet challenge: Towards fully automated lesion segmentation in oncologic pet/ct imaging, PREPRINT (Version 1) available at Research Square.

1030

Girdhar, R., Singh, M., Ravi, N., van der Maaten, L., Joulin, A., & Misra, I. (2022). Omnivore: A Single Model for Many Visual Modalities. In *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.

Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H., & Xu, D. (2022a). Swin UNETR: Swin Transformers for semantic segmentation of brain tumors in MRI images. In *Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries*.

1035

Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H. R., & Xu, D. (2022b). UNETR: Transformers for 3D medical image segmentation. In *IEEE/CVF Winter Conference on Applications of Computer Vision* (pp. 272–284).

1040

He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2021). Masked autoencoders are scalable vision learners. In *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.

Hu, R., & Singh, A. (2021). Unit: Multimodal multitask learning with a unified transformer. In *International Conference on Computer Vision (ICCV)* (pp. 1439–1449).

1045

Huang, H., Xie, S., Lin, L., Iwamoto, Y., Han, X., Chen, Y.-W., & Tong, R. (2022). Scaleformer: Revisiting the transformer-based backbones from a scale-wise perspective for medical image segmentation. In *International Joint Conferences on Artificial Intelligence (IJCAI)*.

1050

Huang, X., Deng, Z., Li, D., & Yuan, X. (2021). Missformer: An effective medical image segmentation transformer. *CoRR*, abs/2109.07162.
Iantsen, A., Jaouen, V., Visvikis, D., & Hatt, M. (2020). Squeeze-and-excitation normalization for brain tumor segmentation. In *Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (MICCAI)* (pp. 366–373).

1055

Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2020). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. *Nature Methods*, **18**, 203–211.

Isensee, F., Jäger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2019). Automated design of deep learning methods for biomedical image segmentation. arXiv:1904.08128.

1060

Islam, M. M., Badic, B., Aparicio, T., Tougeron, D., Tasu, J.-P., Visvikis, D., & Conze, P.-H. (2022). Deep treatment response assessment and prediction of colorectal cancer liver metastases. In *International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)* (pp. 482–491).

1065

Jin, D., Guo, D., Ho, T.-Y., Harrison, A. P., Xiao, J., Tseng, C.-k., & Lu, L. (2019). Deep esophageal clinical target volume delineation using encoded 3D spatial context of tumors, lymph nodes, and organs at risk. In *International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)* (pp. 603–612).

1070

Jun, E., Jeong, S., Heo, D., & Suk, H. (2021). Medical Transformer: Universal brain encoder for 3D MRI analysis. arXiv:2104.13633.

Karimi, D., Vasylechko, S., & Gholipour, A. (2021). Convolution-free medical image segmentation using transformers. In *Medical Image Computing and Computer Assisted Intervention (MICCAI)* (pp. 78–88).

1075

Kavur, A. E., Gezer, N. S., Barış, M., Aslan, S., Conze, P.-H., Groza, V., Pham, D. D., Chatterjee, S., Ernst, P., Özkan, S. et al. (2021). CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation. *Medical Image Analysis*, **69**, 101950.

1080

Kim, W., Son, B., & Kim, I. (2021). Vilt: Vision-and-language transformer without convolution or region supervision. In M. Meila, & T. Zhang (Eds.), *International Conference on Machine Learning (ICML)* (pp. 5583–5594). PMLR volume 139 of *Proceedings of Machine Learning Research*.

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, & K. Q. Weinberger (Eds.), *Advances in Neural Information Processing Systems*. Curran Associates, Inc. volume 25.

1085

Lei, W., Mei, H., Sun, Z., Ye, S., Gu, R., Wang, H., Huang, R., Zhang, S., Zhang, S., & Wang, G. (2021). Automatic segmentation of organs-at-risk from head-and-neck CT using separable convolutional neural network with hard-region-weighted loss. *Neurocomputing*, **442**, 184–199.

1090
Li, J., Chen, J., Tang, Y., Landman, B. A., & Zhou, S. K. (2022a). Transforming medical imaging with transformers? a comparative review of key properties, current progresses, and future perspectives. arXiv:2206.01136.

1095

Li, J., Wang, W., Chen, C., Zhang, T., Zha, S., Wang, J., & Yu, H. (2022b). Transbtsv2: Towards better and more efficient volumetric segmentation of medical images.

Li, S., Sui, X., Luo, X., Xu, X., Liu, Y., & Goh, R. S. M. (2021). Medical image segmentation using squeeze-and-expansion transformers. In *International Joint Conferences on Artificial Intelligence (IJCAI)*. volume abs/2105.09511.

1100

Li, X., Ma, S., Tang, J., & Guo, F. (2022c). Transiam: Fusing multimodal visual features using transformer for medical image segmentation.

Lin, A., Chen, B., Xu, J., Zhang, Z., & Lu, G. (2021). Ds-transunet: Dual swin transformer u-net for medical image segmentation. *CoRR*, abs/2106.06716.

1105

Lin, J., Lin, J., Lu, C., Chen, H., Lin, H., Zhao, B., Shi, Z., Qiu, B., Pan, X., Xu, Z., Huang, B., Liang, C., Han, G., Liu, Z., & Han, C. (2022). Ckd-transbts: Clinical knowledge-driven hybrid transformer with modality-correlated cross-attention for brain tumor segmentation. *CoRR*, abs/2207.07370.

Liu, L., Huang, Z., Liò, P., Schönlieb, C.-B., & Aviles-Rivero, A. I. (2022). Pc-swinmorph: Patch representation for unsupervised medical image registration and segmentation.

1110

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical vision Transformer using shifted windows. In *International Conference on Computer Vision (ICCV)* (pp. 10012-10022).

1115

Lu, J., Batra, D., Parikh, D., & Lee, S. (2019). VilBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In *Advances in Neural Information Processing Systems*. volume 32.

Luo, X., Hu, M., Song, T., Wang, G., & Zhang, S. (2022). Semi-supervised medical image segmentation via cross teaching between CNN and transformer. In *Medical Imaging with Deep Learning*.

1120

Malik, P., & Vidyarthi, A. (2022). A deep learning assisted image-guided framework for differentiation among tumors and hemorrhages in head imaging. *Computers and Electrical Engineering*, 103, 108330.

Mazzucchi, E., La Rocca, G., Hiepe, P., Pignotti, F., Galieri, G., Policicchio, D., Boccaletti, R., Rinaldi, P., Gaudino, S., Ius, T., & Sabatino, G. (2022). Intraoperative integration of multimodal imaging to improve neuronavigation: A technical note. *World Neurosurgery*, 164, 330-340.

1125
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R. et al. (2014). The multimodal brain tumor image segmentation benchmark (BraTS). *IEEE Transactions on Medical Imaging, 34*, 1993–2024.

1130

Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., & Sun, C. (2021). Attention bottlenecks for multimodal fusion. In *Advances in Neural Information Processing Systems (NeurIPS)*.

1135

Ning, Z., Luo, J., Xiao, Q., Cai, L., Chen, Y., Yu, X., Wang, J., & Zhang, Y. (2021). Multi-modal magnetic resonance imaging-based grading analysis for gliomas by integrating radiomics and deep features. *Annals of Translational Medicine, 9*.

1140

Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M. J., Heinrich, M. P., Misawa, K., Mori, K., McDonagh, S. G., Hammerla, N. Y., Kainz, B., Glocker, B., & Rueckert, D. (2018). Attention u-net: Learning where to look for the pancreas. In *Medical Imaging with Deep Learning*.

Oreiller, V., Andrearczyk, V., Jreige, M., Boughdad, S., Elhalawani, H., Castelli, J., Vallières, M., Zhu, S., Xie, J., Peng, Y., Iantsen, A., Hatt, M., Yuan, Y., & et al. (2022). Head and neck tumor segmentation in pet/ct: The hecktor challenge. *Medical Image Analysis, 77*, 102336.

1145

Park, J., Woo, S., Lee, J.-Y., & Kweon, I. S. (2018). Bam: Bottleneck attention module. In *British Machine Vision Conference (IBMVC)*.

Peiris, H., Hayat, M., Chen, Z., Egan, G., & Harandi, M. (2022). A robust volumetric transformer for accurate 3d tumor segmentation. In *Medical Image Computing and Computer Assisted Intervention (MICCAI)* (pp. 162–172).

1150

Petzsche, H., de la Rosa, E., Hanning, U., Wiest, R., Valenzuela, W., & et al. (2022). Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset. *Scientific data, 9*, 762.

1155

Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training (2018).

Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In *International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)* (pp. 234–241). Springer.

1160

Roy, S., Köhler, G., Baumgartner, M., Ulrich, C., Petersen, J., Isensee, F., & Maier-Hein, K. H. (2022). Transformer utilization in medical image segmentation networks. In *International Conference Med-neurips*.

Ryoo, M. S., Piergiovanni, A., Arnab, A., Dehghani, M., & Angelova, A. (2021). Tokenlearner: What can 8 learned tokens do for images and videos? *arXiv:2106.11297*.

1165
Saha, A., Bosma, J., Twilt, J., van Ginneken, B., Yakar, D., Elschot, M., Veltman, J., Fütterer, J., de Rooij, M., & henkjan huisman (2023). Artificial intelligence and radiologists at prostate cancer detection in MRI — the PI-CAI challenge. In *Medical Imaging with Deep Learning, short paper track*.

1170

Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2022). Transformers in medical imaging: A survey. arXiv:2201.09873.

Shapey, J., Wang, G., Dorent, R., Dimitriadis, A., Li, W., Paddick, I., Kitchen, N., Bisdas, S., Saeed, S. R., Ourselin, S., Bradford, R., & Vercauteren, T. (2021). An artificial intelligence framework for automatic segmentation and volumetry of vestibular schwannomas from contrast-enhanced T1-weighted and high-resolution T2-weighted MRI. *Journal of Neurosurgery, 134*, 171 – 179.

1175

1180

Sun, Q., Fang, N., Liu, Z., Zhao, L., Wen, Y., & Lin, H. (2021). Hybridc-trm: Bridging cnn and transformer for multimodal brain image segmentation. *Journal of Healthcare Engineering, 2021*, 1–10.

Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jegou, H. (2021). Training data-efficient image transformers & and distillation through attention. In *International Conference on Machine Learning (ICML)* (pp. 10347–10357). volume 139.

1185

Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019). Multimodal transformer for unaligned multimodal language sequences. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*. Florence, Italy.

1190

Tu, D., Sun, W., Min, X., Zhai, G., & Shen, W. (2022). Video-based human-object interaction detection from tubelet tokens.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., & Polosukhin, I. (2017). Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), *Advances in Neural Information Processing Systems*. volume 30.

1195

Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017). Residual attention network for image classification. arXiv:1704.06904.

1200

Wang, H., Cao, P., Wang, J., & Zaiane, O. R. (2022). Uctransnet: Rethinking the skip connections in u-net from a channel-wise perspective with transformer. *Proceedings of the AAAI Conference on Artificial Intelligence, 36*, 2441–2449.

Wang, W., Chen, C., Ding, M., Li, J., Yu, H., & Zha, S. (2021). TransBTS: Multimodal brain tumor segmentation using Transformer. In *International*

1205
Conference on Medical Image Computing and Computer Assisted Intervention.

Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.

1210

Wiesenfarth, M., Reinke, A., Landman, B. A., Eisenmann, M., Saiz, L. A., Cardoso, M. J., Maier-Hein, L., & Kopp-Schneider, A. (2021). Methods and open-source toolkit for analyzing and visualizing challenge results. *Scientific reports*, 11, 1-15.

1215

Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018). Cbam: Convolutional block attention module. In *European Conference on Computer Vision (ECCV)*.

Wu, Y., Liao, K., Chen, J., Wang, J., Chen, D. Z., Gao, H., & Wu, J. (2022). Dformer: A u-shaped dilated transformer for 3d medical image segmentation. *CoRR*, abs/2201.00462.

1220

Xie, Y., Zhang, J., Shen, C., & Xia, Y. (2021). Cotr: Efficiently bridging cnn and transformer for 3d medical image segmentation. In *International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)*.

1225

Xin Yu, Yucheng Tang, Yinchi Zhou, Riqiang Gao, Qi Yang, Ho Hin Lee, Thomas Li, Shunxing Bao, Yuankai Huo, Zhoubing Xu, Thomas A. Lasko, Richard G. Abramson, & Bennett A. Landman (2022). Characterizing renal structures with 3d block aggregate transformers. arXiv:2203.02430.

Xu, P., & Zhu, X. (2021). Deepchange: A long-term person re-identification benchmark. arXiv:2105.14685.

1230

Xu, P., Zhu, X., & Clifton, D. A. (2022). Multimodal learning with transformers: A survey. arXiv:2206.06488.

Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., & Le, Q. V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, & R. Garnett (Eds.), *Advances in Neural Information Processing Systems*. volume 32.

1235

Yin, M., Yao, Z., Cao, Y., Li, X., Zhang, Z., Lin, S., & Hu, H. (2020). Disentangled non-local neural networks. In *European Conference on Computer Vision (ECCV)* (p. 191-207).

1240

Zhang, D., Huang, G., Zhang, Q., Han, J., Han, J., & Yu, Y. (2021a). Cross-modality deep feature learning for brain tumor segmentation. *Pattern Recognition*, 110, 107562.
1245

Zhang, G., Shen, X., Zhang, Y., Luo, Y., Luo, J., Zhu, D., Yang, H., Wang, W.,
Zhao, B., & Lu, J. (2021b). Cross-modal prostate cancer segmentation via self-
attention distillation. *IEEE Journal of Biomedical and Health Informatics*,

1250

Zhang, Y., He, N., Yang, J., Li, Y., Wei, D., Huang, Y., Zhang, Y., He, Z., & Zheng, Y. (2022a). mmformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation. In *Medical Image Computing and Computer Assisted Intervention (MICCAI)*.

Zhang, Y., Liu, H., & Hu, Q. (2021c). Transfuse: Fusing transformers and cnns for medical image segmentation, arXiv:2102.08005.

1255

Zhang, Y., Yang, J., Tian, J., Shi, Z., Zhong, C., Zhang, Y., & He, Z. (2021d). Modality-aware mutual learning for multi-modal medical image segmentation. In *International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)* (pp. 589-599). Springer.

1260

Zhang, Z., Zhang, H., Zhao, L., Chen, T., Arik, S. O., & Pfister, T. (2022b). Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding. *Proceedings of the AAAI Conference on Artificial Intelligence*, 36, 3417-3425.

Zhou, C., Ding, C., Lu, Z., Wang, X., & Tao, D. (2018a). One-pass multi-task convolutional neural networks for efficient brain tumor segmentation. In *Medical Image Computing and Computer Assisted Intervention* (pp. 637-645).

1265

Zhou, H., Guo, J., Zhang, Y., Yu, L., Wang, L., & Yu, Y. (2021a). nnformer: Interleaved transformer for volumetric segmentation. *CoRR*, abs/2109.03201.

Zhou, T., Canu, S., Vera, P., & Ruan, S. (2021b). Feature-enhanced generation and multi-modality fusion based deep neural network for brain tumor segmentation with missing mr modalities. *Neurocomputing*, 466, 102-112.

1270

Zhou, T., Ruan, S., & Canu, S. (2019). A review: Deep learning for medical image segmentation using multi-modality fusion. *Array*, 3-4, 100004.

Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., & Liang, J. (2018b). Unet++: A nested u-net architecture for medical image segmentation. In *Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support* (pp. 3-11). Springer International Publishing.
<figure><img src="image_20.png" /><figcaption>(a) DSC for ET</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>(b) DSC for TC</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>(c) DSC for WT</figcaption></figure>

<caption>Figure 16: Visualization of significance testing maps for $\mathcal{T}_2$.</caption>
<figure><img src="image_23.png" /><figcaption>Figure 17: Illustration of numbers of parameters and Tera MACs for each model.</figcaption></figure>
<figure><img src="image_24.png" /><figcaption>Figure 18: Visual segmentation results of the fourteen total models for the HECKTOR2021 dataset: (from left to right) CNN-based models, pure Transformer-based encoder networks, full Transformer models, multi-path 3D CNN-transformer encoder architectures and one-path 3D CNN-Transformer encoder networks. Ground truth is color-coded using magenta mean-while the architectures are according to the notation presented in Tab. 4. Rows represent the test cases under study with their respective name in the format Case/#Slice.</figcaption></figure>
<figure><img src="image_25.png" /><figcaption>Figure 19: Visual segmentation results of the top five ranked models for the BraTS2021 dataset: (from left to right) ground truth, MCNN+ViT$_s$-B/1, MCNN+ViT$_v$-B/P, nn-UNet, Swin UNETR, and MU-Net. The green, red and yellow colors respectively correspond to WT, TC, and ET glioblastoma sub-regions. Rows represent the test cases under evaluation with their respective name in the format Case/#Slice.</figcaption></figure>
Dear editor and reviewers,

Please find below the highlights for our manuscript:

**Title:** Multi-modal medical Transformers: A meta-analysis for medical image segmentation in oncology.

**Corresponding author:** Gustavo Andrade-Miranda

**Authors:** Gustavo Andrade-Miranda, Vincent Jaouen, Olena Tankyevych, Catherine Cheze Le Rest, Dimitris Visvikis and Pierre-Henri Conze

**Highlights:**

☐ A meta-analysis of multi-modal medical image segmentation in oncology is provided.

☐ A unified framework describing various Transformer-based architectures is introduced.

☐ Variations in performance are explored between one-path versus multi-path encoders.

☐ The impact of multiple stream interactions is studied for multi-modal Transformers.

☐ Assessment comprises various ranking schemes, cost indicators, and significance tests.

<figure><img src="image_26.png" /><figcaption>Gustavo Andrade-Miranda</figcaption></figure>
Multi-modal datasets

<figure><img src="image_27.png" /><figcaption>HECKTOR2021</figcaption></figure>

<figure><img src="image_28.png" /></figure>

Pure Transformer-based
encoder

Swin Transformer
ViT<sub>m</sub>-B/P
ViT<sub>s</sub>-B/P
ViT<sub>v</sub>-B/P

Hybrid CNN-Transformer encoder

<figure><img src="image_29.png" /><figcaption>One-path</figcaption></figure>

ViT<sub>v</sub>-B/P

<figure><img src="image_30.png" /><figcaption>Multi-path</figcaption></figure>

ViT<sub>m</sub>-B/P
ViT<sub>s</sub>-B/P

<figure><img src="image_31.png" /></figure>

Assessment

<figure><img src="image_32.png" /><figcaption>Ranking analysis</figcaption></figure>

<figure><img src="image_33.png" /><figcaption>Significance test</figcaption></figure>

<figure><img src="image_34.png" /></figure>
CRediT author statement

**Gustavo Andrade-Miranda:** Conceptualization, methodology, software, validation, formal analysis, investigation, writing - original draft, visualization. **Vincent Jaouen:** Conceptualization, methodology, writing - original draft, supervision, funding acquisition.

**Olena Tankyevych:** Data curation, resources. **Catherine Cheze Le Rest:** Data curation, resources. **Dimitris Visvikis:** Supervision, project administration. **Pierre-Henri Conze:** Conceptualization, methodology, writing - original draft, writing - reviewing and editing, supervision, funding acquisition.
**Declaration of interests**

☐ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

☒ The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:

Pierre-Henri Conze reports financial support was provided by French National Research Agency.