Class-Aware Adversarial Transformers for Medical Image Segmentation 
Authors: Chenyu You
Ruihan Zhao
Fenglin Liu
Siyuan Dong
Sandeep P. Chinchali
ufuk topcu
Lawrence Hamilton Staib
James s Duncan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Medical Image Segmentation
Generative Adversarial Network
vision Transformer

Abstract: Transformers have made remarkable progress towards modeling long-range dependencies within the medical image analysis domain. However, current transformer-based models suffer from several disadvantages: (1) existing methods fail to capture the important features of the images due to the naive tokenization scheme; (2) the models suffer from information loss because they only consider single-scale feature representations; and (3) the segmentation label maps generated by the models are not accurate enough without considering rich semantic contexts and anatomical textures. In this work, we present CASTformer, a novel type of adversarial transformers, for 2D medical image segmentation. First, we take advantage of the pyramid structure to construct multi-scale representations and handle multi-scale variations. We then design a novel class-aware transformer module to better learn the discriminative regions of objects with semantic structures. Lastly, we utilize an adversarial training strategy that boosts segmentation accuracy and correspondingly allows a transformer-based discriminator to capture high-level semantically correlated contents and low-level anatomical features. Our experiments demonstrate that CASTformer dramatically outperforms previous state-of-the-art transformer-based approaches on three benchmarks, obtaining 2.54%-5.88% absolute improvements in Dice over previous models. Further qualitative experiments provide a more detailed picture of the model’s inner workings, shed light on the challenges in improved transparency, and demonstrate that transfer learning can greatly improve performance and reduce the size of medical image datasets in training, making CASTformer a strong starting point for downstream medical image analysis tasks.

Class-Aware Adversarial Transformers for Medical Image Segmentation
Chenyu You!
Ruihan Zhao
Fenglin Liu
Siyuan Dong'
Sandeep Chinchali
Ufuk Topeu²
Lawrence Staíb
James Duncan
I Yale University
UT Austin
University of Oxford
Abstract
Transformers have made remarkable progress towards modeling long-range depen- dencies within the medical image analysis domain. However, current transformer- based models suffer from several disadvantages: (1) existing methods fail capture the important features of the images due to the naive tokenization scheme; (2) the models suffer from information loss because they only consider single- sealle feature representations; and (3) the segmentation label maps generated by the models are not accurate enough without considering rich semantic contexts and anatom- ical textures. In this work, we present CASTformer, novel type of adversarial transformers, for 2D medical image segmentation. First, we take advantage of the pyramid structure to construct multi-scale representations and handle multi-scale variations. We then design novel class-aware transformer module to better learn the discriminative regions of objects with semantic structures. Lastly, utilize adversarial training strategy that boosts segmentation accuracy and correspondingly allows transformer-bant based discriminator capture high-level semantically corre- lated contents and low-level anatomical features. Our experiments demonstrate that CASTf ormer dramatically outperforms previous state-of-the-ard transformer-based approaches on three benchmarks, obtaining 2.54%-5.88. absolute improvements in Dice previous models. Further qualitative experiments provide detailed picture of the model's inner workings, shed light on the challenges in improved transparency, and demonstrate that transfer learning can greatly improve performance and reduce the size of medical image datasets in training, making CASTf ormer strong starting point for downstream medical image analysis tasks.
Introduction
Accurate and consistent measurements of anatomical features and functional information medical images can greatly assist radiologists in making accurate and reliable diagnoses, treatment plamning, and post-treatment evaluation [1]. Convolutional neural networks (CNNs) have been the de-facto standard for medical image analysis tasks. However, such methods fail in explicitly modeling long-range dependencies due to the intrinsic locality and weight sharing of the receptive fields in convolution operations. Such a deficiency in context modeling at multiple scales often yields sub-optimal segmentation capability capturing rich anatomical features of variable shapes and
scales (e.g., tumor regions with different structures and sizes) [2 3. Moreover, using transformers
has been shown be more promising in computer vision 4 for utilizing long-range dependencies
than other, traditional CNN-based methods. In parallel, transformers with powerful global relation modeling abilities have become the standard starting point for training on wide range of downstream
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Clecneet Plukis
Tankone
Encoder
[RaaNec]
Traders
Traakonr Exode Hobld
Encoder (ResNee)
Tranofomer Encoder Močsk
Decodar
Head
Exods Foodd
Raal/ Falce!
Generator
Discriminatar
Figure I: Our proposed CASTf ormer consists of a transformer-based generator (i.e., CATformer) and a discriminator.
medical imaging analysis tasks, such as image segmentation 10-13 image synthesis 14
and image enhancement [1726
Medical image semantic segmentation can be formulated as typical dense prediction problem. which aims at performing pixel-level classification on the feature maps. Recently, Chen et al. 7 introduced TransUNet, which inherits the advantage both UNet [27] and Transf ormers to exploit high-resolution informative representations in the spatial dimension by CNNs and the powerful global relation modeling by Transformers. Although existing transformer- based approaches have proved promising in the medical image segmentation task, there remain several formidable challenges, because (1) the model outputs single-scale and low-resolution feature representation; (2) prior work mainly adopts the standard tokenization scheme, hard splitting an image into a sequence of image patches of size 16, which may fail to capture inherent object structures and fine-grained spatial details for the downstream dense prediction task; (3) compared to the standard convolution, the transformer architecture requires a grid structure, and thus lacks the capability to localize regions that contain objects of interest instead of the uninteresting background; and (4) existing methods are usually deficient in ensuring the performance without capturing both global and local contextual relations among pixels. We argue that Hansformer-basee segmentation models not yet robust enough to replace CNN-based methods, and investigate several bhove-mentions key challenges transformer-basee segmentation models still face.
Inspired by recent success of vision transformer networks more practical scenario in which we only assume access to pre-trained models on public computer vision datasets,
we make a step towards
relatively small medical dataset, which we can use weights of the pre-trained models achieve higher accuracy in the medical image analysis tasks. These settings particularly appealing as (1) such models can be easily adopted on typical medical datasets; (2) such a setting only requires limited training data and annotations; and (3) transfer learning typically leads to better performance 38) Inspired by such findings, we propose several novel strategies for expanding setting, considering both multi-scale anatomical feature representations of interesting objects and transfer learning in the medical imaging domain.
its learning abilities
First, we aim to model multi-scale variations by learning feature maps of different resolutions. Thus, we propose to incorporate the pyramid structure into the transformer framework for medical image segmentation, which enables the model to capture rich global spatial information and local multi-scale context information. Additionally, we consider medical semantic segmentation as scquence-to- sequence prediction task. The standard patch tokenization scheme in [4] is an art- art-splitting it into several fixed-size patches and linearly embedding them into input tokens. Even if signíficant progress is achieved, model performance is likely to be sub-optimal. We address this issue by introducing a novel class-aware transformer module, drawing inspiration from a progressive sampling strategy in image classification (39], to adaptively and selectively learn interesting parts of objects. This essentially allows us to obtain effective anatomical features from spatial attended regions within the medical images,
guide the segmentation of objects or entities.
Second, we adopt the idea of Generative Adversarial Networks (GANs) to improve segmentation performance and correspondingly enable transformer- based discriminator to learn low-level anatom-

---

(2
12)
MLP
Offsets
Blinear Interpolatin
W
Muli-ioe SoilFouunion
MLP
H
Sempled Feotures
(a) Class-Aware Transformer Module
(b) Transformer Network
Figure 2: An illustration of (a) our class-aware transformer module and (b) transformer network. Our class-aware transformer module iteratively samples discriminative locations. A group of points is initialized in the regularly-spaces location. At each step, given the feature map F,, we iteratively update its sampling locations by adding them with the estimated offsets of the last step. Note that only 4 points shown for clear presentation, and there more points in the actual implementation.
ical features and high-level semantics. The standard GANs are not guaranteed to prioritize the most informative demonstrations on interesting anatomical regions, and mixing irrelevant regions (i.e., background) creates inferior contexts, which drastically underperform segmentation performance 2 40 3. Additionally, it is well-known that they are notoriously difficult train and prone model collapse [41] Training vision transformers is also tedious and requires large amounts of labeled data, which largely limits the training quality. We use more refined strategy, where, for each input, combine it with the predicted segmentation mask to create the image with anatomical demonstrations. We also leverage the pre-traines trained checkpoints compensate the need of large-dataset training. thereby providing a good starting point with more discriminative visual demonstrations. We refer to our approach as CASTformer, class- aware advergarial transformers: a strong transformer-basee method for 2D medical image segmentation. Our contributions are summarized follows:
Novel Network Architecture: We make the first attempt to build a GAN using transformer- based architecture for the 2D medical image segmentation task. We incorporate the pyramid structure into the generator learn rich global and local multi-scale spatial representa- tions, and also devise a novel class-aware transformer module by progressively learning the interesting regions correlated with semantic structures of images. To the best of knowledge, we are the first work to explore these techniques the context of medical imaging segmentation.
Better Understanding Inner Workings: We conduct careful analyses to understand the model's inner workings, how the sampling strategy works, and how different training factors lead the final performance. We highlight that it more effective progressively learn distinct contextual representations with the class-aware transformer module, resulting in more accurate and robust models that applied better to a variety of downstream medical image analysis tasks (See Appendix ?? and ??).
Remarkable Performance Improvements: CASTformer contributes towards dramatic improvement across three datasets we evaluate on. For instance, we achieve Dice scores of 82.55% and 73.82% by obtaining gains to 5.88% absolute improvement compared to prior methods on the Synapse multi-organ dataset. We illustrate the benefits of leveraging pre-trained models from the computer vision domain, and provide suggestions for future research that could be less susceptible the confounding effects of training data from the natural image domain.
2 Related Work
CNN-based Segmentation Networks Before the emergence of transformer-based methods, CNNs were the de facto methods medical image segmentation tasks [42) For example, Ron- neberger et al. [27] proposed deep 2D UNet architecture, combining skip connections between

---

opposing convolution and deconvolution layers to achieve promising performance on a diverse set of medical segmentation tasks. Han al. [62 developed a 2.5D 24-layer Fully Convolutional Network (FCN) for liver segmentation tasks where the residual block was incorporated into the model. To further improve segmentation aceuracy, Kamnitsas el al. 63 proposed dual pathway 1 1-layer 31) CNN, and also employed a 3D fully connected conditional random field (CRF) [64] as an additional pairwise constraint between neighboring pixels for the challenging task of brain lesion segmentation.
65H70] have focused on developing transformer- baseed methods for medical image analysis tasks. Recently, Chen et al. [ proposed TransUNet, which takes advantage of both UNet and Transf Tranffomers, to exploit high- resolution informative information in the spatial dimension by CNNs and the global dependencies by Transformers. Cao et al. [T0] explored how to use a pure transformer for medical image analysis tasks. However, the results do not lead to better performance. These works mainly utilized hard splitting some highly semantically correlated regions without capturing the inherent object structures. In this work, beyond simply using the naive tokenization scheme in 4 7.
Transformers in Medical Image Segmentation Recent studies
we aim at enabling the
transformer to capture global information flow to estimate offsets towards regions of interest.
Transformer in Generative Adversarial Networks Adversarial learning has proved to be very useful and widely applicable technique for learning generative models of arbitrarily complex data distributions in the medical domain. As the discriminator D differentiates between real and fake samples, the adversarial loss serves as the regularization constraint to enforce the generator G to predict more realistic samples. Inspired by such recent success 7 22 75 20 23
76 77].
Jiang ef al. (11 proposed to build GAN pipeline with two pure transformer-based a in synthesizing high-resolution images. Esser al. [3 first used convolutional GAN model to learn codebook of context-rich visual features, followed by transformer architecture to learn the compositional parts. Hudson et al. [3] proposed a bipartite self-attention on StyleGAN to propagate latent variables to the evolving visual features. Despite such success, it requires high computation costs due to the quadratic complexity, which fundamentally hinders its applicability to the real world. Besides the image generation task, we seek to take step forward in tackling the challenging task of 2D medical image segmentation.
3 Method
Our proposed approach is presented in Figure 1 Given the input image x E xxx similar to TransUNet architecture [7] our proposed generator network G, termed CATformer, is comprised of four key components: encoder (feature extractor) module, class-aware transformer module, transformer encoder module, and decoder module. As shown in Figure [ our generator has four stages with four parallel subnetworks. All stages share a similar architecture, which contains a patch embedding layer, class-aware layer, and L; Transformer encoder layers.
Encoder Module. Our method adopts a CNN-Transformer hybrid model design instead of using a pure transformer, which uses 40 convolutional layers, to generate multi-scale feature maps. Such a convolutional stem setting provides two advantages: (1) using convolutional stem helps transformers perform better in the downstream vision tasks 84]; (2) it provides high-resolution feature maps with parallel medium- and low-resolution feature maps to help boost better representations. way, we can construct the feature pyramid for the Transformers, and utilize the multi-scale feature maps for the downstream medical segmentation task. With the aid of feature maps of different resolutions, our model is capable of modeling multi-resolution spatially local contexts.
this
Hierarchical Feature Representation. Inspired by recent success object detection deviate from TransUNet by generating single-resolution feature map, and our focus is extracting CNN-like multi-level features F,, where E 21, 3, 4), to achieve high segmentation accuracy by leveraging high-resolution features and low-resolution features. More precisely, in the first stage, we utilize the encoder module to obtain the dense feature map F1 E I # x Y xC1, where H W C1) is the spatial feature resolution and the number of feature channels. In a similar way, we can formulate the following feature maps as follows: F2 E R x x(C -4) Fa E R #x (C) and HW F4 E R # " (C),-12), Then, we divide F, into I6 patches with the patch size P of 16 x 16x3. and feed the flattened patches into a learnable linear transformation to obtain the patch embeddings x C₁.
85 we
2
of size HW
162

---

Class-Aware Transformer Module. The class-aware transformer module (CAT) is designed to adaptively focus on useful regions of objects (e.g., g. the underlying anatomical features and structural information). Our CAT module is largely inspired by the recent success for image classification 39) but we deviate from theirs as follows: (1) we remove the vision transformer module in (39] to alleviate the computation and memory usage; (2) we use 4 separate Transformer Encoder Modules (TEM), which will be introduced below; (3) we incorporate M CAT modules on multi-scale representations to allow for contextual information of anatomical features to propagate into the representations. Our class-aware transformer module is an iterative optimization process. In particular, we apply the class-aware transformer module to obtain the sequence of tokens I E [RC((x))), where (n n) and M are the number of samples each feature map and the total iterative number, respectively. As shown in Figure 2 given the feature map F,, we iteratively update its sampling locations by adding them with the estimated offset vectors of the last step, which can be formulated as follows:
S1+1 Se + o,, LE E {I,.... M - 1
(1)
where S, (xxxn) and o (xxx)))) are the sampling location and the predicted offset vector at t-th step. Specifically, the s₁ is initialized at the regularly spaced sampling grid. The i-th sampling location s is defined as follows:
s 13 Th. + Tn/2, Tur + /2]
where B- Li/n], 35 = 8 The step sizes in the (row index) and (column index) directions denote Th = H/n and The = W/n, respectively. [.] is the floor operation. We can define the initial token on the input feature map in the following form: I = (s:))) where t E {I,..., M and I, (x(xx)) denotes the initial sampled tokens at 1-th step. We set the sampling function as the bilinear interpolation, since it is differentiable with respect to both the sampling locations 8; and the input feature map F,, We do an clement-wist addition of the current positional embedding of the sampling locations, the initial sampled tokens, and the estimated tokens of the last step, and then we can obtain the output tokens at each step:
S Wss St
V. L @ S + I-1
I = Transformer (V..) t (1....,My M)
where W. E RCx2 the learnable matrix that embeds s, to positional embedding S, (xxx)))) and @ is the element-wise addition. Transformer(-) is the transformer encoder layer, we will show the following paragraphs. We can compute the estimated sampling location offsets as:
o, (L)),, e (1, (L...., M --1) 1}:
where (.(-) E 2xx(xx)))) is the learnable linear mapping for the estimated sampling offset vectors. It is worth noting that these operations are all differentiable, thus the model can be learned in an end-to-end fashion.
Transformer Encoder Module. The transformer encoder module (TEM) is designed to model long-range contextual information by aggregating global contextual information from the complete sequences of input image patches embedding. In implementations. the transformer encoder module follows the architecture in ViT 44, which composed of Multi-head Self Self-Antention (MSA), and MLP blocks, which can be formulated as:
E.- = Hx)H: x H; X H| + Hpos
(5)
E': = MSA((L((E-.)) + E,-1,
(6)
E, = -MMP(LL(EE)) .)) +E',,
(7)
where i = 1...M.MM M, and LN(-) is the layer normalization. H E (P and Hpos E R N denote the patch embedding projection and the position embedding.
Decoder Module. The decoder is designed to generate the segmentation mask based on four output feature maps of different resolutions. In implementations, rather than designing a hand-crafted decoder module that requires high computational demand, we incorporate a lightweight AII-MLP decoder (86), and such simple design allows us yicld powerful representation much more
5

---

right Eddnay
stamadh
Ground Truch
UNet
AcUNeet
R50+UNe:
SETR
Col:
TransLNNet
Swin-UNkt
CATormerr
CAST
Figure 3: Visual comparisons with other methods on Synapse dataset. observed, CASTformer achieves superior performance with detailed anatomical features and the boundary information of different organs.
efficiently. The decoder includes the following criteria: 1) the channel dimension of multi-scale features is unified through the MLP layers; 2) we up-sample the features to 1/4th and concatenate them together; 3) we utilize the MLP layer to fuse the concatenated features, and then predict the multi-class segmentation mask y' from the fused features.
Discriminator Network. We use the R50+ //1- hybrid model pre-trained on ImageNet from VIT 4 as a starting point for our discriminator design, in this case using the pre-traince strategies to learn effectively the limited size target task data. Then, simply apply a two-layer multi-layere perceptron (MLP) to make a prediction about the identity of the class-aware image. Following previous work (2]. first utilize the ground truth image and the predicted segmentation mask y' to obtain the class-aware image x (i.e., pixel-wise multiplication of x and y'). important that this construction re-uses the pre-trained weights and does not introduce any additional parameters. D seeks to classify between real and fake samples [8)) G and /) compete with each other through attempting reach equilibrium point of the minimax game. Using this structure enables the discriminator to model long-range dependencies, making it better assess the medical image fidelity. This also essentially endows the model with a more holistic understanding of the anatomical visual modality (categorical features).
Training Objective. As to the loss function and training configurations, we adopt the settings used in Wasserstein GAN (WGAN) [88], and use WGAN-GP loss [89]. We jointly use the segmentation loss 7 and WGAN-GP loss train G. Concretely, the segmentation loss includes the dice loss and cross-entropy loss. Hence, the training process of CASTformer can be formulated as:
LG CnC + C2 + >2L
where \1, A2, As determine the importance of each loss term. See Appendix ?? for an ablation study.
4 Experimental Setup
Datasets. We experiment on multiple challenging benchmark datasets: Synapse LITS, and MP-MRI. More details can be found in Appendix ??.
Implementation Details. We utilize the AdamW optimizer (90 in all our experiments. For training our generator and discriminator, we use a learning rate of 5e with batch size of 6, and train each model for 300 epochs for all datasets. We the sampling number n on each feature map and the total iterative number M as 16 and 4, respectively. See Appendix ??, ??, ?? for details on the training configuration, model architecture and hyperparameters. We also adopt the input resolution and patch size P as 224x 224 and 14, respectively. We set \ 0.5, X2 0.5. and X3 0.1 in this experiments. the testing stage, we adopt four metrics to evaluate the segmentation performance: Dice coefficient (Dice), Jaccard Index (Jaccard), 95% Hausdorff Distance (95HD), and Average
https: fttkps://www.snaee http://wwwaaaaoo...eee.ee.oeaaa/ hatp444sSSpppppssyy133333////2/2

---

Table 1: Quantitative segmentation results on the Synapse multi-organ CT dataset.
Framework
Average
Aorta Gallbladder Kidney (L) Kidney (R) Liver Pancreas Spleen Stomach
Encoder
Decoder
DSC
Jaccard 95HD
ASD
Uilet
70.11
59.39
44.69
14.41
84,00
56.70
72.41
62.64
86.98
48.73
81.48
67.96
ActnUNet: [9]
71.70
61.38
34.47
10.00
82.61
61 94
76.07
70.42
87.54
46.70
80.67
67.66
ReaNet550
73.51
63.81
29.65
8.83
82.21
55.06
76.71
73.07
89.36
53.52
84,91
73.22
Reatlet50
Attalliet 01
74.74
62.69
33.04
9.49
83.68
58.63
79.08
74.53
90.81
53.76
83,80
71.68
SETR [
66.30
54.1
29:09
7.16
66.63
38.34
74.45
68.49
92. 18
35.91
83.01
71.41
cHtl- encoder
54.82
42.49
20.37
63.22
67.10
88.48
15.46
60.74
CoTr [13
72.60
61.25
41.55
12.42
83.27
60.41
79.58
73.01
91.93
45.07
82.84
64.67
TransliKat:
77.48
64.78
8.46
87.23
77.02
55.86
85.08
SwinUKet [00
76.33
65. 64
27.16
8.32
85.47
66.53
83.28
79.61
94.29
56.58
90.66
76.60
CATformer (ours)
82.17
73.22
16.20
4.28
88.98
67.16
85.72
81.69
95.34
66.53
90.74
81.20
CASTformer (ours)
82.55
74.69
22.73
5.81
89.05
67.48
86.05
82.17
95.61
67.49
91.00
81.55
Symmetric Surface Distance (ASD). All our experiments are implemented in Pytorch 1.7.0. We train all models on single NVIDIA GeForce RTX 3090 GPU with 24GB of memory.
5 Results
We compare approaches (i.e., CATformer and CASTformer) with previous state-of-the-ait Transformer-based segmentation methods, including UNet [27]. AttnUNet [1]] ResNet50+UNet
27 Resllet50+AttnüNett (91), SETR [92], CoTr w/o CNN-encoder [3] CoTr [I3, TransUNet
SwinUnet [10 on Synapse, LiTS, and MP-MRI datasets. More results are in Appendix ?? and
??.
Synapse Multi-organ. The quantitative results on the Synapse dataset are shown in Table[[ The results visualized in Figure It can be observed that our CATformer outperforms the previous best model by a large margin and achieves a 4.69% 8.44% absolute improvement in Dice and Jaccard, respectively. Our CASTforme achieves the best performance of 82.55% and 74.69%, dramatically improving the previous state-of-the-art model (TransUNet) by +5.07% and +9.91 %, in terms of both Dice and Jaccard scores. This shows that the anatomical visual information is useful for the model to gain finer control in localizing local semantic regions. As also shown in Table | our CASTformer achieves absolute Dice improvements of +2.77%, +2.51 %, +1.5%, +4.95% on large organs (i.e., left kidney, right kidney, liver, stomach) respectively. Such improvements demonstrate the effectiveness of learning the evolving anatomical features of the image, as well as accurately identifying the boundary information of large organs. We also observed similar trends that, compared the previous state-of-the-an results, our CASTformer obtains 89 99.05% 67. 67.49% in terms of Dice on small organs (i.e., aorta, gallbladder, pancreas) respectively, which yields big improvements of +1.2%, +0.95%, + 11.9919 %. This clearly demonstrates the superiority of our models, allowing for spatially finer control over the segmentation process.
Experiments:
Experiments: LITS. To further evaluate the effectiveness of our proposed approaches, we compare our models on the LITS dataset. Experimental results on the LiTS CT dataset are summarized in Appendix Table ??. As is shown, we observe that CATformer yields 72.39% Dice score, outperforming all other methods. Moreover, our CASTformer significantly outperforms all previous approaches, including the previous best TransUNet, and establishes new state-of-the-an of 73.82% and 64.91% % in terms of Dice and Jaccard, which are 5.88% and 4.66% absolute improvements better than TransUNet. For example, our CASTf achieves the best performance of 95. 95.88% Dice on the liver region by 2.48%, while it dramatically increases the result from 42.49% to 51.76% on the tumor region, demonstrating that our model achieves competitive performance on liver and tumor segmentation. As shown in Figure our method is capable of predicting high- object segmentation, considering the fact that the improvement in such a setting is challenging. This demonstrates: (1) the necessity of adaptively focusing on the region of interests; and (2) the efficacy of semantically correlated information. Compared to previously gegh-performing models, our two approaches achieve significant improvements on all datasets, demonstrating their effectiveness.
7

---

Ground Truth
UNet
AwUUNet
R50+UNet
SETR
CoTr
TransUNet
Swin-UNet
CATfrmar
CASTformrr
Figure 4: Visual comparisons with other methods LiTS dataset. As observed, CASTf ormer achieves superior performance with detailed anatomical information (e.g., .. the tumor regions red).
Analysis
In this section, we conduct thorough analyses of our CASTf AAsTrormer in the following aspects: transfer learning, model components, effects of iteration number N (Appendix ??). sampling number n (Ap- pendix ??). hyperparameter selection (Appendix ??). different GAN-based loss functions (Appendix ??), and gain a better understanding of the model's s inner workings (Appendix ?? and ??).
leverage the pre-trained model commonly used in computer vision literature 4 to provide more evidence for the beneficial impact of our network performance. We use CATformes (G) the baseline and evaluate all the settings on the Synapse multi-organ dataset. To put our results in perspective, we compare with six ways of using pre-trained R50+ViT-B/16 hybrid model from VIT for transfer learning, namely (1) CATformer: w/o pre- trained and w/ pre- trained;; (2) CASTformers both w/o pre-trained, only w/ pre-trained D, only w/ pre-trained G, and both w/ pre-trained G and D.
Transfer Learning. We consider whether
Table 2: Effect of transfer learning in CATformer and CASTformer on the Synapse multi-organ dataset.
Table 3: Ablation model component: Baseline; CATf w/o CAT, CATformer w/o TEM; and CATformer.
Model
DSC
Jaccard
95HD
ASD
Model
DSC
Jaccard
95HD
ASD
CAffornor (wdo pre-trained)
7481
65.61
31.81
7.23
Baseline
77.48
64.78
31.69
8.46
82.17
73.22
4.28
CAfforner w/o CAT
80.00
70.56
25.62
7.30
CASTforner thoth pre-trinnd))
CATforuer we
81.35
72.66
24.43
7.17
CASTfexmer tonly pre trained D)
78.87
69.36
30.54
9.17
CATformer
82.17
73.22
16.20
4.28
CASTfermer tonly w/ pre-trained C1
81.46
71
27.36
6.91
CASTforner thoth pre-tminedd)
82,55
74.69
22.13
5.81
The results are in Table 2 Overall, we observe that using "ww/ pre-trained" leads to higher accuracy than "w/o pre-trained", with significant improvements for the smaller sizes of datasets, suggesting that using "ww pre-trained" provides good set of initial parameters for the downstream tasks. With using pre-trained weights, CATformer outperforms the setting without using pre-trained weights by large margin and achieves 7.33% and 7.61 % absolute improvements in terms of Dice and Jaccard. CASTformer ("both w/ pre-trained") also yields big improvements (+8.91% and -12.01%) in Dice and Jaccard. This suggests that CASTforme is better at both initializing from the pre-trained models and better at gathering the anatomical information a more adaptive and selective manner. As shown in Table 2] surprisingly, there is significant discrepancy between only using "w/ pre-trained D" and "ww/ traitrained G" for example, CASTf ormer achieves 78.87% Dice with only w/ pre-trained D, while CASTf ormer achieves 81.46% if only G uses the pre- pre-trained weights. This demonstrates that only using me-ttrined weight D might be the culprit for the exploitation of anatomical information.
Our results suggest that (1) utilizing pre-trained models in the computer vision domain can help the model quickly adapt to new downstream medical segmentation tasks without re-building billions of anatomical representations; (2) we find that leveraging pre-trained weights can further boost the

---

performance because it can mitigate the discrepancy between training and inference; and (3) it also creates a possibility to adapt our model to the typical medical dataset with the smaller size.
Ablation of Model Components. Our key observation is that it is crucial to build high-quality anatomical representations through each model component. To show the strengths of our approach, we examine the following variants and inspect each key component on the Synapse multi-organ segmentation dataset: (1) Baseline: we remove the class-aware transformer module and the trans- former encoder module in our CATformer as the baseline, similar to TransUNet defined in IZ]: (2) CATformer w/o CAT: we only remove the class-aware transformer module in our CATformer; (3) CATformer w/o TEM: we only remove the transformer encoder module in our orTorr, (4) CATformer: this is our G model; and (5) CASTf ormer: this our final model described in Section 3 Table 3 summarizes the results of all the variants. As is shown, we observe that compared to baseline model, both CATformer w/o CAT and CATformer w/o TEM are able to develop a better holistic understanding of global shapes/structures and fine anatomical details, thus leading to large performance improvements (+ 2. 2.61% % and +3.87%) in terms of Dice. Our results show the class-aware transformer module is useful in improving the segmentation performance, suggesting that the discriminative regions of medical images are particularly effective. Finally, one thing worth noticing is that incorporating both the class-aware transformer module and the transformer encoder module performs better than only using a single module, highlighting the importance of two modules in our CATformer.
7 Conclusion and Discussion of Broader Impact
In this work, we have introduced CASTformer, a simple yet effective type of adversarial transformers, for 2D medical image segmentation. The key insight is to integrate the multi-scale pyramid structure to capture rich global spatial information and local multi-scale context information. Furthermore, CASTformer also benefits from our proposed class-aware transformer module progressively and selectively learn interesting parts of the objects. Lastly, the inveratoo-discriminaa design is used to boost segmentation performance and correspondingly enable the transformer-basee discriminator to capture low-level anatomical features and high-level semantics. Comprehensive experiments demonstrate that our CASTf outperforms the previous state-of-the-ar on three popular medical datasets considerably. We conduct extensive analyses to study the robustness of our approach, and form a more detailed understanding of desirable properties in the medical domain (i.e., transparency and data efficiency).
Overall, we hope that this model can serve solid baseline for 2D medical image segmentation and motivate further research in medical image analysis tasks. It also provides a new perspective on transfer learning in medical domain, and initially shed novel insights towards understanding neural network behavior. As such pattern is hard to quantify, we expect more mechanistic explanations for clinical practise. We also plan to optimize the transformer- medical image analysis tasks both in terms of data and model parameters.
architectures for the downstream
Broader Impact. We acknowledge that our research will not pose significant risks of societal harm to society. Our work is scientific nature and will have the potential to positively contribute to a number of real-world clinical applications that establish high-quality and end-to-end medical image segmentation systems. We expect our approach to contribute to the grand goal of building more secured and trustworthy clinical Al.
References
[1] Mehrdad Moghbel, Syamsiah Mashohor, Rozi Mahmud, and M Iqbal Bin Saripan. Review of liver segmentation and computer assisted detection/diagnosis methods in computed tomography. Artificial Intelligence, 2017.
[2] Yuan Xue, Tao Xu, Han Zhang, L Rodney Long, and Xiaolei Huang. Segan: Adversarial network with scate loss for medical image segmentation. Neuroinformatics, 2018
[3] Drew A Hudson and Larry Zitnick. Generative adversarial transformers. In International Conference on Machine Learning (ICML), 2021.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghnn,, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is

---

worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2020.
[5] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang. Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.1 2021.
[6] Aditya Desai, Zhaozhuo Xu, Menal Gupta, Anu Chandran, Antoine Vial-Aussavy, and Anshumali Shri- vastava. Raw nav-merge scismic data to subsurface properties with mip based multi-modal information unscrambler. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
[7] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2021.
[8] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
[9] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[10] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-unct: Unet-like pure transformer for medical image segmentation. arXiv preprint arXiv:2 100.05537,, 2021.
[11] Wenxuan Wang, Chen Chen, Meng Ding. Hong Yu, Sen Zha, and Jiangyun Li. Transbis: Multimodal brain tumor segmentation using transformer. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAT), 2021.
[12] Jeya Maria Jose Valanarasu, Poojan Oza, llker Hacihaliloglu, and Vishal M Patel. Medical transformer: Gated axial-attention for medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2021.
[13] Yutong Xie, Jianpeng Zhang, Chunhua Shen, and Yong Xia. Cotr: Efficiently bridging eon and transformer for 3d medical image segmentation. In International Conference Medical Image Computing and Computer-Assivee Intervention (MICCAT), 2021.
[14] Lingke Kong, Chenyu Lian, Detian Huang, Yanle Hu, Qichao Zhou, et Breaking the dilemma of medical image-to-image translation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[15] Nicolae- Cattaln Ristea, Andreca- Indreca-Iuliana Miron, Olivian Savencu, Mariana- Iuliana Georgescu, Nicolae Verga, Fahad Shahbaz Khan, and Radu Tudor Tonescu. Cytran: Cycle-consistent transformers for non- contrast to contrast translation. arXiv preprint arXiv:21 10.06400, 2021.
[16] Onat Dalmaz, Mahmut Yurt, and Tolga Çukur. Resvit: Residual vision transformers for multi-modal medical image synthesis. arxiv preprint arXiv:2106. 16031, 2021.
[17] Yilmaz Korkmaz, Salman UH Dar, Mahmut Yurt, Muzaffer Özbey, and Tolga Çukur. Unsupervised mri reconstruction via zero-shot learned adversarial transformers. arXiv preprint arXiv:2105.00 08059, 2021.
[18] Zhicheng Zhang, Lequan Yu, Xiaokun Liang, Wei Zhao, and Lei Xing. Transct: Dual-path transformer for low dose computed tomography. In International Conference Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2021.
[19] Qing Lyu, Chenyu You, Hongming Shan, and Ge Wang. Super-resolution mri through deep learning. arXiv preprint 11Xiv:1810.06776 2018
[20] Qing Lyu, Chenyu You, Hongming Shan, Yi Zhang, and Ge Wang. Super-resolution mri and et through gan-circle. In Developments in X-ruy tomography XII, volume 11113, page 111 International Society for Optics and Photonics, 2019.
[21] Indranil Guha, Syed Ahmed Nadeem, Chenyu You, Xiaoliu Zhang, Steven M Levy, Ge Wang, James Torner, Punam K Saha. Deep learning based high- resolution reconstruction trabecular bone microstructures from low-resolution et seans using circle. In Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging. International Society for Optics and Photonics, 2020.
10

---

[22] Chenyu You, Qingsong Yang, Hongming Shan, Lars Cjesteby, Guang Li, Shenghong Ju, Zhuiyang Zhang. Zhen Zhao, Yi Zhang, Wenxiang Cong, et al. Structurally-sensitivee multi-scale deep neural network for low-dose et denoising, IEEE access, 2018
[23] Chenyu You, Linfeng Yang. Zhang, and Ge Wang. Low-dose et via deep ean with skip connection and network-in-netwonk In Developments in X-Ray tomography XII. International Society for Optics and Photonics, 2019.
[24] Chenyu You, Lianyi Han, Aosong Feng, Ruihan Zhao, Hui Tang, and Wei Fan. Megan: Memory enhanced graph atlention network for space-time video super-resolution. In In Proceedings of WACV 2022, 2021.
[25] Achleshwar Luthra, Harsh Sulakhe, Tanish Mittal, Abhishek Iyer, and Santosh Yadav, Eformer: Edge 2021.
enhancement based transformer for medical image denoising. arXiv preprint arXiv:2
[26] Dayang Wang, Zhan Wu, and Hengyong Yu. Ted-net: Convolution- frree 12t vision transformer-based encoder-decoder dilation network for low- ct denoising. arXiv preprint arXiv:2106.04660 2021.
[27] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.
[28] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
[29] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), 2020.
[30] Yanhong Zeng, Jianlong Fu, and Hongyang Chao. Learning joint pratial-tempora transformations for video inpainting. In European Conference on Computer Vision (ECCV), 2020.
[31] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 103. 2021.
[33] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
[34] Hugo Touvron, Matthiu Cord, Alexandre Sablayrolles. Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2 /72399 2021.
[35] S. Kevin Zhou, Hayit Greenspan, Christos Davatzikos, James S. Duncan, Bram Van Ginneken, Anant Madabhushi, Jerry L. Prince, Daniel Rueckert, and Ronald M. Summers. review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. Proceedings of the IEEE, 2021.
[36] Gonglei Shi. Li Xiao, Yang Chen. and S Kevin Zhou. Marginal loss and exclusion loss for partially supervised multi-organ segmentation. Medical Image Analysis, 2021.
[37] Qingsong Yao, Li Xiao, Peihang Liu, and S Kevin Zhou. Label-free segmentation of covid-19 lesions in lung ct. IEEE Transactions on Medical Imaging, 2021.
[38] Jiuwen Zhu, Yuexiang Li. Yifan Hu, Kai Ma, S Kevin Zhou, and Yefeng Zheng. Rubik's cube+: A self-supervised feature learning framework for 3d medical image analysis. Medical Image Analysis, 2020.
[39] Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, Philip Torr, Wayne Zhang, and Dahua Lin. Vision transformer with progressive sampling. In IEEE International Conference on Computer Vision (ICCV), 2021.
[40] Pauline Luc, Camille Couprie, Soumith Chintala, and Jakob Verbeek. Semantic segmentation using 2016.
adversarial networks. arXiv preprint arXiv:
[41] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
11

---

[42] Hayit Greenspan, Bram Van Ginneken, and Ronald M Summers, Guest editorial deep learning in medical imaging: Overview and future promise of exciting new technique. IEEE Transactions Medicul Imaging, 2016.
[43] Geert Litjens, Thijs Kooi, Bahak Fhteshami Bejnordi, Amaud Arindra Adiyoso Sctio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen AWM van der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey on deep learning in medical image analysis. Medical Image Analysis, 2017.
[44] Yizhe Zhang, Lin Yang. Jianxu Chen, Maridel Fredericksen, David Hughes, and Danny Z Chen. Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In International Conference on Medical Image Computing and Computer-Assise Intervention (MICCAI), 2017.
[45] Xiaomeng Li, Lequan Yu, Hao Chen, Chi-Win Thi-Wing Fu, and Pheng-Am Ann Heng. Semi-supervised skin lesion segmentation via transformation consistent self-ensem model. arXiv preprint arXiv: 03887,
2018.
[46] Dong Nie, Yaozong Gao, Wang, and Dinggang Shen. Asdnet: Attention based supervisee deep networks for medical image segmentation. In International Conference on Medical Image Computing and Computer-Asiisted Intervention (MICCAI), 2018.
[47] Yicheng Wu, Yong Xia, Yang Song, Yanning Zhang, and Weidong Cai. Multiscale network followed network model for retinal vessel segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAT), 2018.
[48] Lequan Yu, Shujun Wang, Xiaomeng Li, Chi- Chi-Wing Fu, and Pheng-Ann Heng. Uncertainty-aware self- ensembling model for semi- semi-supervised 3d left atrium segmentation. In International Conference on Medical Image Computing and Computer- Assistte Intervention (MICCAI). 2019.
[49] Yu Zeng, Yunzhi Zhuge, Huchuan Lu, and Lihe Zhang. Joint learning of saliency detection and weakly supervised semantic segmentation. In IEEE International Conferencee on Computer Vision (ICCV), 2019.
[50] Gerda Bortsova, Florian Dubost, Laurens Hogeweg, loannis Katramados, and Marleen de Bruijne. Semi- supervised medical image segmentation via learning consistency under transformations. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2019.
[51] Yicheng Wu, Yong Xia, Yang Song, Donghao Zhang. Dongnan Liu, Chaoyi Zhang, and Weidong Cai. Vessel-net: retinal vessel segmentation under multi supervision. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2019.
[52] Chenyu You, Junlin Yang, Julius Chapiro, and James S. Duncan. Unsupervised wasserstein distance guided domain adaptation for 3d multi-domain liver segmentation. In Interpretable and Annotation-Effiiinn Learning for Medical Image Computing, pages 155-163, Springer International Publishing, 2020.
[53] Linfeng Yang, Rajarshi P Ghosh, J Matthew Franklin, Simon Chen, Chenyu You, Raja R Narayan, Mare L Melcher, and Jan T Liphardt. Nuset: deep learning tool for reliably separating and analyzing crowded cells. PLoS computational biology, 2020.
[54] Shanlin Sun, Kun Han, Deying Kong, Chenyu You, and Xiaohui Xic. Mirnf Medical image registration via neural fields. arXiv preprint 1.Xiv.2206.03111 2022.
[55] Xiaoran Zhang, Chenyu You, Shawn Ahn, Juntang Zhuang, Lawrence Staib, and James Duncan. Learning correspondences of cardiac motion from images using biomechanics-infore -informme modeling. arXiv preprint arXiv:2209.0077 (0726, 2022.
[56] Chenyu You, Ruihan Zhao, Lawrence Staib, and James S Dunean. Momentum contrastive voxel wise representation learning for semi-supervised volumetric medical image segmentation. arXiv preprint arXiv:2105.07059 2021.
[57] Chenyu You, Yuan Zhou, Ruihan Zhao, Lawrence Staib, and James S Duncan. Simevd: Simple contrastive voxel-wise representation distillation for cup-suueee medical image segmentation. IEEE Transactions on Medical Imaging, 2022.
[58] Chenyu You, Jinlin Xiang, Kun Su, Xiaoran Zhang, Siyuan Dong, John Onofrey, Lawrence Staib, and James S Duncan. Incremental learning meets transfer learning: Application to multi-site prostate mri segmentation. arXiv preprint arXiv:2206.01300 2022.
[59] Chenyu You, Weicheng Dai, Lawrence Staib, and James S Duncan. Bootstrapping semi-supervised medical image segmentation with anatomical aware contrastive distillation. arXiv preprint 220::22000003307, 2022.
aware
12

---

[60] Chenyu You, Weicheng Dai, Fenglin Liu, Haoran Su, Xiaoran Zhang, Lawrence Staib, and James S Duncan. Mine your own anatomy: Revisiting medical image segmentation with extremely limited labels. arxivv preprint arXiv:2209. 13476, 2022.
[61] Krishna Chaitanya, Frtunc Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[62] Yuexing Han, Xiaolong Li, Bing Wang, and Lu Wang. Boundary neural networks approach for segmentation: A case study of the liver and tumor on computed tomography. Algorithms, 2021.
2.5 d fully convolutional
[63] Konstantinos Kamnitsas, Christian I. Ledig. Virginia FJ Newcombe, Joanna P Simpson, Andrew D) Kane, David K Menon. Daniel Rueckert, and Ben Glocker. Efficient multi-scale 3d enn with fully connected crf for accurate brain lesion segmentation. Medical Image Analysis. 2017.
[64] John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data, In International Conference on Machine Learning (ICML), 2001.
[65] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang. Andriy Myronenko, Bennett Landman, Holger Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. arXiv preprint arXiv:2103. 10504, 2021.
[66] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: self-configuring method for deep learning-hased biomedical image segmentation. Nature methods, 2021.
[67] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, al. Rethinking semantic segmentation from a sequenee-to-sequee perspective with transformers. arXiv preprint arXiv:2012. 158400 2020.
[68] Georg Hille, Shubham Agrawal, Christian Wybranski, Macicj Pech, Alexey Surov, and Sylvia Saalfeld. Joint liver and hepatic lesion segmentation using a hybrid enn with transformer layers. arXiv preprint arXiv:2201. 10981, 2022.
[69] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In IEEE Winter Conference Applications of Computer Vision (WACV), 2022.
[70] Yucheng Tang, Dong Yang, Wenqi Li. Holger Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. sule-supe pre-training swin transformers for 3d medical image analysis. arXiv preprint ar Xiv:2111. 14791, 2021.
[71] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two transformers can make one strong gan. arXiv preprint 2:Xiv:2002.0700 2021.
[72] Yu Zeng, Zhe Lin, and Vishal M Patel. Sketchedit: Mask-free local image manipulation with partial sketches. arXiv preprint arXiv:2111. 15078, 2021.
[73] Yu Zeng, The Lin, Huchuan Lu, and Vishal M Patel. Cr-fill: Generative image inpainting with auxiliary contextual reconstruction. In IEFE International Conference on Computer Vision (ICCV), 2021.
[74] Dor Arad Hudson and Larry Zütnick. Compositional transformers for scene generation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[75] Chenyu You, Guang Li, Yi Zhang, Xiaoliu Zhang, Hongming Shan, Mengzhou Li, Shenghong Ju, Zhen Zhao, Zhuiyang Zhang, Wenxiang Cong, al. CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (gan-circle). IEEE Transactions on Medical Imaging, 2019.
[76] Long Zhao, Zizhao Zhang, Ting Chen, Dimitris N Metaxas, and Han Zhang. Improved transformer for high-resolution gans. arXiv preprint arXiv:2106.0760 2021.
[77] Yanhong Zeng, Huan Yang, Hongyang Chao, Jianbo Wang, and Jianlong Fu. Improving visual quality of image synthesis by basen generator with transformers. In Advances in Neural Information Processing Systems (NeurIPS). 2021.
[78] Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. arXiv preprint X1X::206600000 2021.
13

---

[79] Kaiming He. Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[80] Liang-Chieh Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In European Conference on Computer Vision (ECCV), 2018.
[81] Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kemnl matters-improve semantic segmentation by global convolutional network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[82] Fisher Yu and Vladlen Koltun. Multi-scalc context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07111 2015.
[83] Tete Xiao, Piotr Dollar, Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick. Early convolutions help transformers see better. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[84] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEF International Conference on Computer Vision (ICCV), 2017.
[85] Tsung-Ye Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[86] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers, arXiv preprint 205. 2021.
[87] Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
[88] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning (ICML), 2017.
[89] Ishaan Guirajani. Faruk Ahmed, Martin Arjovsky. Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[90] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
[91] Jo Schlemper, Ozan Oktay, Michiel Schaap, Mattias Heinrich, Bernhard Kainz, Ben Glocker, and Daniel Rueckert. Attention gated networks: Learning to leverage salient regions in medical images. Medical Image Analysis, 2019.
[92] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang. Philip HS Torr, al. Rethinking semantic segmentation from soquencc-to-sqqueee perspective with transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. - You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
14

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section and Broader Impact on Page 9.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section
7 and Broader Impact on Page 9.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] See Sections and supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sections ( and supplemental material.
(c) Did you report error bars (c.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Sections 4 and supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sections and supplemental material.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No] using/curating? [Yes] See Sections and supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] See Sections and supplemental material.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "FbG4KGpMrQ", "reviewer": "Reviewer_YJsa", "summary": "The authors introduce an effective generative adversarial transformer called CASTformer for 2D medical image segmentation. The main idea behind the design principle is to integrate the multi-scale pyramid structure to capture rich global spatial information as well as local multi-scale context information. The suggested class-aware transformer module enables CASTformer to discover useful aspects of objects incrementally and selectively. The generator-discriminator design is used to improve segmentation performance, enabling the transformer-based discriminator to capture both low-level anatomical information and high-level semantics.", "questions": "- Could you please check the reference papers and make comments on it with respect to your work?", "limitations": "Due to nature of the work, as per authors, research will not pose significant risks of societal harm to society. Rather, the study has the potential to positively contribute to a number of real-world clinical applications possibly.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed generative adversarial transformer called CASTformer for 2D medical image segmentation sounds interesting and useful. Both quantitative and qualitative results show the effectiveness of the proposed model. A number of ablations are performed to show that the suggested mechanisms are worth considering.\n\nThe authors provide clear justification for their work. This is a well-written manuscript and easy to follow indeed. The supplementary improves the overall attractiveness of the manuscript as well. The suggested mechanism is straightforward but effective. The authors validate their claims through extensive experiments and significant performance gain.\n\nThe study leverages pyramid structure to construct multi-scale representations and handle multi-scale variations. Some of the ideas here may be familiar to readers of the following papers [1, 2]. If these studies have any relevance to the topic at hand, it would be great if the authors would highlight it.\n\n[1] T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In CVPR, 2017.\n\n[2] T. -Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, \"Focal Loss for Dense Object Detection,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 2, pp. 318-327, 1 Feb. 2020, doi: 10.1109/TPAMI.2018.2858826.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bm0igmgj4dw", "reviewer": "Reviewer_oVPV", "summary": "This paper proposed Transformer based segmentation model for medical image called CASTformer. Compared to the other segmentation algorithms, it utilizes 1) the multi-scale approach with CNN+Transformer hybrid architecture, 2) the class-aware transformer module for progressive sampling strategy, and the adversarial training scheme. Also, the pre-trained model in computer vision dataset helps to start their training in good starting point. The results show that CASTformer improves the segmentation performance impressively.", "questions": "- Please refer and explain the differences between your model and the deformable sampling CNN/Transformers: DCN(Deformable convolutional networks), DAT(Vision Transformer with Deformable Attention), Deformable DETR(Deformable DETR: Deformable transformers for end-to-end object detection), and etc. They are using same concept for feature sampling with irregular grid. The comparison results with the other deformable sampling module and the reason why it works better might be helpful for the readers.\n\n- Is there any performance drop using the bilinear interpolation in the feature domain? How much the performance drop for discrete sampling such as nearest neighbor sampling? I have some worries about the existence of continuity in 2D grid features. Please, refer some theoretical background proof or practical evidence for that. \n\n- Fair comparison between pre-trained model and from-scratch model.\nI wonder the epoch numbers for the training of from-pretrained model and from-scratch model is same or not. \n\n- Ablation study.\nAs you shown in the analysis, the proposed train processes (such as pre-training on natural image dataset and adversarial loss) seems very effective to improve the performance dramatically. Then, to support the proposed architectural excellence, it is necessary to compare between SwinUnet(or other SOTA architecture) with the proposed  training process(adversarial loss+pre-training with computer vision dataset) and the proposed CASTformer. It is necessary to show that the proposed architecture (class-aware Transformer module, Multi-scale feature extraction, etc) is superior to others.\n", "limitations": "\n- Why the term ‘generative’ is used for your model? Does your model generate the image which not exists in the input image? The proposed method utilizes discriminator to improve the performance of segmentation with adversarial auxiliary task for the ROI reconstruction not the generation of some parts not in the input. The outputs are the reconstruction of the attended region of input image, not the generation. In my opinion, the term ‘generative’ should be used at a minimum to maximize the persuasive power of your claim because some (quiet large portion of the medical field) peoples in real clinic have a lot of worries to use generative model for the medical imaging. \n\n\n- The term ‘class-aware’ is confusing. \nSome readers in computer vision field or machine learning field might expect that the proposed ‘class-aware’ module dynamically/adaptively utilizes the class information like in StyleGAN. However, the authors used the term ‘class-aware’ for the module which samples discriminative locations by iteratively estimated the sampling offsets based on the current input features. \nThe author might say the class-aware Transformer module samples different point according to the type of organ. However, does the class-aware module have ability to distinguish kidney/liver/spleen? I think the sampling module does not distinguish the type of organ. It focuses on the current feature pixel is related/connected to the other feature pixel or not according to the global information. \nTo support your claim, the author should provide the experimental support which adding a linear layer on the proposed class-aware module with the freezed weight has some ability to classify the organ type class.\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- (significance) CASTformer shows the dramatic improvement of segmentation performance with large gap to the other algorithms. The author proposed better training processes to utilized the adversarial loss and the computer vision dataset as pre-training to improve the performance of segmentation task for the (relatively-) limited medical dataset.\n\n- (originality & quality) Some reader might claim that they combined many concepts already exists in computer vision field such as multi-scale feature extraction, feature sampling, adversarial loss and transfer-learning. However, it is important that they make that work and it reaches the dramatic performance improvement on their task.\n\n- (clarity)  They support various ablation studies and comparison with many deep network algorithms. However, there are some lack of clarity and I asked those questions in the followings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "45GIIcbQkHl", "reviewer": "Reviewer_aCmr", "summary": "This paper proposed a transformer-based model for medical image segmentation, which adopts the pyramid structure and adversarial training. The proposed method is validated on three benchmark datasets, and obtains advantaged results.", "questions": "As indicated in this paper, it is a pre-trained model on public computer vision datasets. In the experiments, which computer vision dataset is employed? For medical image segmentation task, is the pre-trained model necessary?", "limitations": "see weakness", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. the class-aware transformer module is interesting. \n2. the paper is well written, and there are careful experimental analysis conducted.\n\nWeakness:\n1. the novelty and contribution are limited. The key contribution is the class-aware transformer module, a revised transformer, to learn the class aware context. Pyramid structure and adversarial training are common approaches used in medical image analysis.\n2.The motivation is unclear. Why the adversarial network is needed in this model?\n3. the comparison of experimental results is unfair. The proposed model is equipped with newly-added CAT and GAN, which is a bigger model than others. Even the pre-trained model is compared with other models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept