E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation
Authors: Boqian Wu
Qiao Xiao
Shiwei Liu
Lu Yin
Mykola Pechenizkiy
Decebal Constantin Mocanu
Maurice van Keulen
Elena Mocanu
Venue: NeurIPS 2024 poster (2024)
Keywords: Medical Image Segmentation
Sparse Training
Feature Fusion
Primary area: machine_learning_for_healthcare

Abstract: Deep neural networks have evolved as the leading approach in 3D medical image segmentation due to their outstanding performance. However, the ever-increasing model size and computational cost of deep neural networks have become the primary barriers to deploying them on real-world, resource-limited hardware. To achieve both segmentation accuracy and efficiency, we propose a 3D medical image segmentation model called Efficient to Efficient Network (E2ENet), which incorporates two parametrically and computationally efficient designs. i. Dynamic sparse feature fusion (DSFF) mechanism: it adaptively learns to fuse informative multi-scale features while reducing redundancy. ii. Restricted depth-shift in 3D convolution: it leverages the 3D spatial information while keeping the model and computational complexity as 2D-based methods. We conduct extensive experiments on AMOS, Brain Tumor Segmentation and BTCV Challenge, demonstrating that E2ENet consistently achieves a superior trade-off between accuracy and efficiency than prior arts across various resource constraints. %In particular, with a single model and single scale, E2ENet achieves comparable accuracy on the large-scale challenge AMOS-CT, while saving over 69% parameter count and 27% FLOPs in the inference phase, compared with the previous
best-performing method. Our code has been made available at: https://github.com/boqian333/E2ENet-Medical.

E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation
Boqian Wu Qiao Xiao² Shiwei Liu', Lu Yin', Mykola Pechenizkiy",
Decebal Constantin Mocanu Maurice van Keulen', Ellna Mocanu
University of Twente, University of Luxembourg. Eindhoven University of Technology,
University of Oxford, University of Surrey
{b. wu, m. vankeulen,
mocanu)/utwentter
{q xiaxi m. ww.ppchenizkiyyeeeee nl, shiwei liu@maths. ac. yinissuurryy. ac. uk, decebal www.mmmmmi..i....mmm
Abstract
Deep neural networks have evolved as the leading approach in 3D medical image segmentation due to their outstanding performance. However, the ever-incrent werrinnceassigg model size and computational cost of deep neural networks have become the pri- mary barriers to deploying them real-world, resource-limited hardware. To achieve both segmentation accuracy and efficiency, we propose a 3D medical image segmentation model called Efficient to Efficient Network (E2ENet), which incorporates two parametrically and computationally efficient designs. i. Dynamic sparse feature fusion (DSFF) mechanism: it adaptively Iearns to fuse informa- tive multi-scale features while reducing redundancy. ii. Restricted depth-shift in 3D convolution: it leverages the 3D spatial information while keeping the model and computational complexity as 2D-based methods. We conduct exten- sive experiments AMOS, Brain Tumor Segmentation and BTCV Challenge, demonstrating that E2ENet consistently achieves a superior trade-off between accuracy and efficiency than prior arts across various resource constraints. E2ENet achieves comparable accuracy on the large-scale challenge AMOS-CT, while sav- ing over 69% parameter count and 27% FLOPs in the inference phase, compared with the previous best berformmngg method, Our code has been made available at: http://ssttbbbooommopppm///////e
Introduction
3D medical image segmentation plays essential role in numerous clinical applications, including computer-aided diagnosis (Yu et al. 2020) and image-guided surgery systems [Ronneberger et al.
2015]. Over the past decade, the rapid development of deep neural networks has achieved tremendous
breakthroughs and significantly boosted the progress in this area [Zhou et al. 2018 Huang et al.) 2020 Isensee et al.) 2021]. However, the model sizes and computational costs are also exploding. which deter their deployment in many real-world applications, especially for 3D models where resource consumption scales cubically (Hu al. 2021 Valanarasu and Patel 2022). This naturally raises research question: Can we design a 3D medical image segmentation method that trades off accuracy and efficiency better, subjected to different resource availability?
Accurately segmenting organs is a challenging task in 3D medical image segmentation due to the variability in size and shape even among the same type of organ, caused by factors such patient anatomy and disease stage. This diversity amplifies the difficulty to accurately identify and distinguish
38th Conference Neural Information Processing Systems (NeurIPS 2024).
'Equal contribution.

---

a+ ++
Bictors
Seaton
Sostore
DeTrons Dessder
bastore
Rositon
UN
EINN
Figure I: A comparison of feature fusion schemes. The purple nodes depict features extracted from the backbone, while the green nodes depict the fused features. In particular, in DiNTS (d), red lines indicate information flow paths determined through neural architecture search techniques. In E2ENet (e), the red lines with different widths represent sparse information flows determined by the DSFF mechanism, allowing for efficient feature fusion. E2ENet is capable of dynamically learning how many of the features to fuse are derived from the backbone.
the boundaries of different organs, leading potential errors in segmentation. One of the main solutions for accurate medical image segmentation is to favorably leverage the multi-scale features extracted by the backbone network, but this remains long-standing problem. Pioneering work UNet |Ronebberger et al., 2015] utilizes skip connection to propagate detailed information to high-level features. More recently, UNet++ (Zhou al. 2018). CoTr [Xie al. 2021], and DiNTS [He et al. 2021] have introduced more complex neural wetw architectures (e.g., dense skip connection and attention mechanism) and optimization techniques (e.g.. neural architecture search (NAS) (Elsken et al. 2 for sross-scale feature fusion. WAS-based methods search for network topology and operators (e.g., 3x3 Convolution and Maxpool) and subsequently optimize model weights. However, these methods typically require substantial computational resources to explore network topologies and evaluate numerous candidate architectures, making them time-consuming. For instance, C2FNAS Yuu et al.) 2020 requires nearly one GPU-year to discover a single 3D segmentation architecture, while DINTS [He et al.j 202T] improves search efficiency but still needs 5.8 GPU days find a single architecture. In contrast to NAS approaches, method does not require costly architecture search time and instead directly optimizes and searches for sparse topologies within the predefined
architecture.
In this paper, we propose the Efficient Efficient Network (E2ENet), model that efficiently incorporates both bottom-up up and top-down features from the backbone network in dynamically sparse patter, achieving an improved accuracy-efficiene trade-off. As shown in Figure [ (e). E2ENet incorporates multi-scale features from the backhone into the final output by gradually fusing adjacent features, allowing the network to fully utilize information across various scales. To prevent unnecessary information aggregation, we propose dynamic sparse feature fusion (DSFF) mechanism and embed it each fusion node. The DSFF mechanism adaptively integrates informative multi-scale features and filters out unnecessary ones during the course of the training process, significantly reducing the computational overhead without sacrificing performance. Additionally, to further improve efficiency, our E2ENet employs restricted depth-shift strategy within 3D convolutions, derived from the temporal shift [Lin et al.. 2019) and 3D-shift [Fan et al.) 2020 strategies used in efficient video action recognition. This allows the 3D convolution operation with a kernel size of (1,3,3) to capture 3D spatial relationships while maintaining the parameter complexity of 2D convolutions and reducing computational cost. To evaluate the performance of our proposed E2ENet, we conducted extensive experiments on the AMOS-CT [Ji et al.] [2022] Brain Tumor Segmentation in the Medical Segmentation Decathlon (MSD) [Antonelli al. [022]] J, and Multi-Atlas Labeling Beyond the Cranial Vault (BTCV) [Landman et al.) 2015 challenges. We found that E2ENet can effectively trade-off between segmentation accuracy and efficiency compared to both convolution-base challenge, E2ENet achieves competitive accuracy with a mDice of 90.0%, while being 69% smaller and using 27% fewer FLOPs during inference. With the DSFF mechanism filtering out 90% of feature connections, E2ENet further reduces resource costs without significantly sacrificing accuracy.
and transformer-base architectures. In particular, on the AMOS-CT
2 Methodology
In this section, we first present the overall architecture of E2ENet, which allows for the fusion of multi-scale features from three directions. Next, we describe the proposed DSFF mechanism, which adaptively selects informative features and filters out unnecessary ones during training. Lastly, to further increase efficiency, we introduce the use of restricted depth-shift in 3D convolution.

---

Lakon operation
stage
3
5
Intercaherr
level
SNICCNN
Cusput Module
level
lovel
Dewnaarpl
fnwoid
level 5
Jysampling
Shit CVV dlekk
lovel 6
Paved Teature
Efficient backbone
Pyramid
Dynamic Sparse Feature Fusian
Output
Paskua
Figure 2: The overall architecture of the proposed E2ENet consists of a CNN backhone that extracts multiple levels of features. These features are then gradually aggregated through several stages, during which the multi-scale features are fused using a fusion operation.
2.1 The Architecture
Figure 2 provides overview of the proposed architecture. Given the input 3D image Jin RDxH XW the CNN backbone extracts feature maps at multiple scales, represented by x" = (x".1, 0,2 ....., x".),, where L is the total number of feature scales. The feature level i, 0 is tensor with dimensions d, x h4 X w; X C, where d; h, wy and represent the depth, height, width, and number of channels of the feature maps level respectively. It worth mentioning that the spatial resolution of the feature maps decreases as the level increases, while the number of channels increases up maximum of 320. To fully exploit the hierarchical features extracted by the CNN backbone, these features are aggregated across multiple stages, as depicted in Figure (Dynamic Sparse Feature Fusion section). At stage j (0 < fusing the adjacent features from the previous stage along three directions:
1), the features level are obtained by
"Downward flow" (in yellow): The high-resolution feature -i-i.i-1 which provides richer visual details, passed downward; "Upward flow" (in blue): The low-resolution feature x/-1,6+1, which captures more global context, is passed upward; 3. "Forward flow" (in red): The features -11 which maintain their spatial resolution, are passed forward for further information integration. The fused ccoss-scale feature maps at the i-th level of the j-th stage can be formulated as:
x ( 1111 U(x 1:2))), x -(x
i= others,
where (.) fusion operation, consisting of a convolution operation followed by Instance Normalization (IN) [Ulyanov et al. 2016) and Leaky Rectified Linear Unit (LeakyReLU) (Maas] ct al. 2013 (1()) and D(.) denote the up-sampling and down- wwwn-sampling, respectively. [-] denotes concatenation operation. In the convolution operation, the input feature maps, which have channel number of Cy. (Cr,** G-1 G + c++1 C; C2+1), are fused and processed produce output feature maps with channel number of C (C c:). Unlike the UNet++ model, which only considers bottom-up up information flows for image segmentation. our proposed E2ENet architecture incorporates both bottom-up and top-down information flows. This allows E2ENet to make use of both high-level contextual information and fine-grained details in order to produce more accurate segmentation maps (experimental results can found Table 2).
Dynamic Sparse Feature Fusion
Such multi-stage cross-level feature propagation provides more comprehensive understanding of the images but can also introduce redundant information, necessitating careful handling in feature fusion to ensure efficient interaction between scales stages. Our proposed Dynamic Sparse Feature Fusion

---

(DSFF) mechanism addresses the issue of multi-scale feature fusion in an intuitive and effective way. optimizes the process by enabling selective and adaptive use of features from different levels during training. This results in more efficient feature fusion process with lower computational and memory overhead.
The DSFF mechanism is applied in each fusion operation, allowing the fusion operation F-)(() to select the informative feature maps from the input fused features. The feature map selection process is controlled by a binary mask (.)( E {0, 1 Ch xC which are trained to filter out S unnecessary feature map connections by zeroing out the corresponding kernels in F5i((). With the DSFF mechanism, the output of the fusion operation is then computed as:
x Coueifsii
a (I c=0
(8 Cuii
(M C,Cout C.out
Aii
where xei cunt
6 the input fused feature map at the c-th channel. connection, as in Figure 3 that connects the c-th input feature map to Cout output feature map, is a convolution operation, is the product of a scalar (i.e., c,Cou! and matrix (i.e., 01.
chfouchhtt the kernel (feature map
C.Cout
M
C.Cout denotes the existence or absence of a connec- tion between the c-th input feature map and the Cour-th output feature map. I(.) and (.) denote the Instance Normalization and LeakyReLU, respectively. Thus, the core of DSFF mechanism is the learning of binary masks during the training. At initialization, each binary mask is initialized randomly, with the number of zero en- tries ||M llo equals (1 S) x C.A x Cá. Here, S (0 S < is hyperparameter called the feature spar- sity level, which determines the percentage of feature map connections that are inactivated. The activated connec- tions can be updated throughout the course of training, while the feature sparsity level S remains constant. This enables efficiency in both testing and training, as the ex- ploitation of connections remains sparse throughout the training process.
reactivate
absciete fesure mep
input feature map
resuting feature map
training
evoludion
AT-1
AT epochs
training
exclution
AF
2
2 PT apocha
T Tepochs
Every AT training epoch, the activated connections with lower importance are removed, while the same randomly reac-
Figure 3: Illustration of our Dynamic Sparse Feature Fusion (DSFF) mecha- nism. The fusion operation starts from sparse feature connections and allows the connectivity to be evolved after train- ing for AT epochs. During each evo- lution stage, a fraction of kernels with smaller L₁ norms will be zeroed out (red dotted line), while the same fraction of other inactivated connections will be re- activated randomly, keeping the feature sparsity S constant during training (blue solid line).
number of deactivated connections tivated, as shown in Figure 3 The importance of activated connection is determined by the Li norm of the corre- sponding kernel. That is, for M the importance score of connection between -iythh input feature and Cour-th output is ||02. cin..coott..... 1I. Our intuition is simple: if a fea- ture map connection is more important (i.e., has a larger effect on output accuracy) than others, the L1 norm of the corresponding kernel should be larger. However, the im- portance score is suboptimal during training, as the kernels are gradually optimized. Randomly reactivating previous "obsolete" connections with re-initialization can avoid un- recoverable feature map abandonment and thoroughly explore the representative feature maps that contribute to the final performance.
The DSFF mechanism allows for the exploration and exploitation of multi-scaled sparse features through a sparse-to-
training method. Our method selects features at multiple levels, as opposed to relying solely on feature selection at the input layer [Sokar et al., 2022 Atashgahi et al., 2022 Additionally, it is plastic approach to fuse features that can adapt to changing conditions
'The computation cost of the masking operation is negligible. For instance. consider the feature map x with size D x H x W, the kernel 02-1 fz.i........ with size of 3 x 3 x 3, and the number of kernels as C.,, x Cout- The computation cost of the masking operation alone is C.n x Cous, whereas the combined cost of the masking operation and convolution operation is D x H W x 3 3 3 x Cin x Cout-

---

during training, opposed to static methods that rely on one-shot feature selection, as shown in the comparison of dynamic and static sparsity in Table of (Mostafa and Wang 2019]. More details of the training process are elaborated in the Algorithm in Appendix A.3
2.3 Restricted Depth-Shift in 3D Convolution
In 3D medical image segmentation, the 2D-based methods (such as the 2D nneNet [[sensee al. 202T)), which apply the 2D convolution to each slice of the 3D image, are computationally efficient but can not fully capture the relationships between slices. To overcome this limitation, we take inspiration from the temporal-shift [Lin et al. [019] and 3D-shift [Fan ct al., 2020] efficient video action recognition, and the ahial-shift [Lian et al., 2021] in efficient MLP architecture for vision. Our proposed E2ENet incor- porates depth-shift strategy in 3D convolution op- erations, which facilitates inter-slice information ex- change and captures 3D spatial relationships while retaining the simplicity and computational effi- ciency of 2D-based method. Temporal-shifft (Lin ict al. 2019) requires selecting a Shift Proportion (the proportion of channels conduct temporal shift). Axial-shift [Lian et al. 2021] and 3D-Shift optimize the learnable 3D spatiotemporal shift. We have made refinements to the channel shifting technique by shift- ing along the depth dimension and incorporating con- straints on the shift size. This adaptation is thought- fully designed to align with the distinctive needs of sparse models employed in medical image segmenta- tion.
C1
ahin -f
Ciry
ahit
2rCin
Cout
shift 1
Figure 4: Illustration of restricted depth-shift in 3D Convolution of our E2ENet. The input features (left) are firstly split into 3 parts along the channel dimension, and then shifted by {-1, O, }} units along the depth dimension respectively (middle). After that, 3D CNNs with kernel size x are performed on the feature maps (middle) to generate the output features (right).
In our method, we employ a simple depth shift technique by shifting all channels while constraining the shift size to be either +1, 0 or, as shown in Figure This choice is motivated using dynamic sparse feature fusion, where the feature maps contain sparse information. If the shift magnitude is too large, it can result an insufficient representation of channels or an excessive representation of depth information, which can have a negative impact on the effectiveness of the shift operation (experimental results can be found in Table 3).
3 Experiments
In this section, we compare the performance of our E2ENet model to baseline methods and report results in terms of both segmentation qual- ity and efficiency. In addition, we will perform ablation studies investigate the behavior of each component in the E2ENet model. To fur-
Table 1: Quantitative comparisons of segmentation performance on the validation set of AMOS-CT dataset. denotes the results with postprocessing.
Methods
mDice (8) f
Params (M) 1
FLOPs !
score
mNSD
(G)
1
(%)
77.1
41.87
1510.5
1.07
inFoommr
85.6
150.14
1543.65
1.12
74.2
ther analyze the performance our model, we will examine its generalizability and the effect of model capacity, and present qualitative results by visualizing the predicted segmentations on sample images. We will also visualize the fea- ture fusion ratios to gain insights into which fea- tures play an important role the segmentation process. The description of the dataset and ex- perimental setup can be found in Appendix A.4 and A.3
UNETR
78.3
93.02
391.03
1.41
61.5
Swin UNETR
86.4
62.83
1562 99
1.14
75.3
VNei
82.0
45.65
1737.57
1.10
67.9
nnUNet
30.76
noUNer"
90.5
30.76
1067.8% 066.89
1.31
83.0
EXFNNer (s=0.)
90.3
11.23
969.32 32
1.54
82.7
ENENNe (s=0.8)
90.3
9.44
778. 74
1.65
82.5
E2FNeet (s=0.9)
89.9
7.64
492.22 29
1.89
81.8
(s=0.7)
11.23
969.55
E2ENet (s=0.8)
90.0
9.44
778.74
1.65
82,3
E2ENet (s=0.9)
89.6
7.64
492.29
1.88
81.4
The inference [LLOPs are calculated based on the patch sizes of without considering postpracessing
128 128
5

---

3.1 Comparison with SOTA methods
AMOS-CT Challenge. To comprehensively validate our method, we compare it to several state-
of-the-art CNN-based models (e.g. nnUNet [Isensee al. 2021], and Vnet (Milletari et al.] 2016))
and transformer-based models (e.g. CoTr [Xie et al. 2021 nnFormer [Zhou et al. 2021]. J. UNETR
2022 We record the mDice (class-wise (Hatamizadeh et al., 2022]. and Swin UNETR [Tang al.
Dice can be found in Table 10 of Appendix), Params, inference FLOPs, and PT score on the
validation set set" of the AMOS-CT challenge in Table E2ENet with feature sparsity (S) of 0.8 achieves comparable performance, with mDice of 90.3%, while being 3 x smaller in model size and requiring less computational cost during the inference phase compared to the top-performing lightweight model, nnUNet. As the feature sparsity of E2ENet increases, the number of model parameters and inference FLOPs can be further reduced without significantly compromising seg- mentation performance. This indicates that there is potential to trade-off performance and efficiency by adjusting the feature sparsity of E2ENet. We also report the results of mean normalized surface dice (mNSD), the official segmentation metric for the AMOS challenge, to provide supplementary information on boundary segmentation quality.
BraTS Challenge in MSD. E2ENet demonstrates superior performance in terms of mDice compared to other state-of-the-ar methods (DiNTS (He
Table 2: 5-fold cross-validation of segmentation performance on the BraTS Challenge training set in the MSD. Note: ED, ET, and NET denote edema, enhancing tumor, and non- enhancing tumor, respectively.
2021]. UNet++ [Zoo et al. 2018 and nnUNet [[sensee et al.
et al.
Methods
Dice (%) 1
miDice ($)
Params (M)
FLOPs (G).
PT score
ED
ET
NET
2021). Additionally, it is resource-
DINTS
80.2
61.1
77.6
73.0
1
1
/
efficient network that is competitive with the baselines, as evidenced by its small model size and low inference FLOPs. Specifically, E2ENet model a feature sparsity level of 90% has only 7.63 M parameters, which is sig- nificantly smaller than other models yet still outperforms them in terms of mDice. The results on the BraTTS dataset, which are MRT images, further validate the effectiveness and efficiency of E2ENet across both CT and MRI medical image analysis.
UNet++
80.5
62.5
79.2
74.1
58.38
3938.25
1.12
nnUNet
81.0
62.0
79.3
74.1
31.20
1076.62
1.35
E2E (8 0.7)
81.2
62.7
79,5
74.5
11.24
1067 066
1.57
E2ENet (8 0.8)
81.0
79.0
780.97
1.72
E2E (8 0.9
80.9
62.5
79.4
74.3
7.63
494.52
2.00
The inference FLOPs are calculated based on patch sizes of x 128 x 128 x 128.
The numher of parameters and inference FLOPs for DINTS are not reported because calculating them time-conumiing. for DINTS is not readily available for the dutaset and must he found using Neural Architecture Seurch (NAS).
due to the
the architecture
Due to the limited space and the similarity between the BTCV and AMOS-CT challenges, both of which involve multi-organ segmentation in CT images, the results of the BTCV challenge are provided in Appendix A.8.2
3.2 Ablation Studies
In this section, we investigate the im- pact two factors on the performance of E2ENet: (i) the DSFF mechanism and (ii) restricted depth-shift in 3D
Table 3: Ablation study on the effects of the DSFF mech- anism and restricted depth-shift in 3D convolution the validation set of the AMOS-CT Challenge.
shift
kernel
Params (M) L
FLOPs (G)
mNSD (%6
convolution. Specifically, we con- sider the following scenarios: (#1)
DSFF
Bers)
shift
(%)
(-1,0,1
90.2
23.90
3069.55
82.6
w/ DSFF: DSFF mechanism is used to dynamically activate feature map, otherwise, all feature maps are acti- vated during training: (#2) w/ shift: restricted depth/heighh/width shifting is applied on feature maps before the
(1 x 3 3)
88.6
23.90
3069.55
78.6
(-1,0,1
x 3
90.1
11.23
969.32
82.3
1200
X
x 1/3xx
88.2
11.23
969.32
79.4
1200
(-2,0,2)
(1 x
3)
89.8
11.23
969.32
82.0
(-3,0,3)
[1 3
3)
11.23
969.32
81.6
1200
(-
(1 x
87.6
11.23
969.32
77.5
1200
(3 x 3 3)
90.3
$2.54
4511.57
82.5
x
90.1
27.97
1778.55
82.1
convolution operation, otherwise, the convolution is performed directly as standard 3D convolution without shifting operation.
600
(-1.0,1
(1
90.0
11.23
969.32
82.2
1800
(-1.0.1
(1 1 3xx3xx3
89.9
11.23
969.32
82.0
'The explanation of metrics can found in Appendix A.7
'The validation set includes 100 images and was the previous test set for the first stage of the challenge. In the training of E2ENet, we did not use the validation set and treated it as the test set.

---

Effect of DSFF Mechanism. Table B (rows and 3) demonstrates that the E2ENet with the DSFF mechanism, achieved comparable performance while using three times fewer parameters and inference FLOPs. Table 4 shows that removing the DSFF mechanism caused the mDice score to drop from 74.5 % (row 4) to 74.1 % (row 2) on the BraTTS challenge, while also increasing the number of parameters by more than 2x and the inference FLOPs by nearly 3x. These ablation study results highlight that dynamic sparse feature fusion (DSFF) helps reduce resource costs while maintaining segmentation performance.
To further investigate the impact of the DSFF mechanism, we compared E2ENet with other multi-scale med- ical image segmentation methods on the AMOS-CT validation dataset, as These methods in-
Table 4: Ablation study on the effects of the DSFF mecha- nism and restricted depth-shift in 3D convolution, evaluated through 5-fold cross-validation of segmentation performance on the BraTS Challenge training set in the MSD.
w/ DSFF
kernel size
Dice (S)
mDice (9)
Params
FLOPSs
shown in Table [
shill
ED
ET
NET
(M) 4
(G)
clude Deepl.abv3 [Chen et al. [217]]
X
(1 x 3)
80.4
62.4
79.0
73.9
23.89
3071.78
20221]. and Med- Former [Gao et al. 2022]. 3eepLabv3
CoTr [Xie et al.
(1 1 3)
81.0
62.3
79.0
74.1
23.89
3071.78
V
(1 [1x3x80 33)
80.3
62.5
79.0
73.9
11.24
1067.06
uses atrous convolution to capture multi-scale context, CoTr integrates multi-scale features using attention, and MedFormer employs all-to-all at- tention for comprchensive multi-scale fusion, addressing both semantic and spatial aspects.
V
x1x3 3)
81.2
62.7
79.5
74.5
11.24
1067.06
X
(3 3)
80.9
61.9
79.1
74.0
52.55
4519.26
x 3)
81.0
62.2
79.1
74.1
28.02
2023.52
V
3)
80.3
61.7
78.7
73.6
11.24
1067.06
V
(3 1)
80.5
61.9
78.7
73.7
11.24
1067.06
x3x x 3)
81.1
62.4
78.9
74.1
11.24
1067.06
(3
1)
81.0
62.3
79.4
74.2
11.24
1067.06
We observed that with a sparsity ratio of 0.9, E2ENet outperforms DeepLabv3 in terms of mDice while significantly reducing computational and memory costs by more than 3x and nearly 10x, respectively. Additionally, E2ENet with a sparsity ratio of 0.8 matches MedFormer's performance while significantly reducing computational and memory costs by 3x and 4x, respectively. This demonstrates the benefits of multi-scaa feature aggregation for medical image segmentation, with E2ENet's DSFF mechanism being much more efficient than other multi-scale methods.
Moreover, as shown in Table 2 E2ENet with the DSFF mechanism outperforms other feature fusion architectures, such as UNet+ and DINTS, on the BraTS challenge. Overall, the results from the BraTS and AMOS- CT challenges demonstrate that the DSFF mechanism provides an effective and efficient approach for feature fusion.
Finally, we also studied the impact of topology update frequency (AT) in the DSFF mechanism and observed that our algorithm is not sensitive to the hyperparameter AT,
Effect of Restricted Depth-Shift in 3D Con- volution. We evaluated the effectiveness of our restricted shift strategy by comparing the per- formance of E2ENet and its variants. Without restricted depth-shift, the mDice decreased from 74.5% (row 4) to 73.9% (row 3) for the BraTS challenge as shown in Table 4 and decreased from 90.1 % (row 4) to 88.2% 2% (row 3) for AMOS- CT (as shown in Table 3) In Table 3 when comparing E2ENet with kernel sizes of 3x3x3 (rows and 9) to E2ENet with kernel sizes of Lx3x3 combined with a depth shift (rows 1 and 3).
Table 5: Comparison with other multi-scale med- ical image segmentation methods on the AMOS- CT validation dataset. The mDice score for Med- Former is sourced from [Goo et al. 2022).
method mDice (%)
Params (M) L FLOPs (G)
DeepLabe3
89.0
74,68
1546.25
Col's
77.1
41.87
1510.53
MedFormer
90.1
39.59
2332.75
E2ENet (S=0.8)
90.0
9.44
778.74
E2ENet (S=0.9)
89.6
7.64
492.29
their segmentation accuracy remains the same, whether DSFF is used or not. In Table g we find that E2ENet with kernel sizes of 1x3x3 combined with depth shift (rows 2 and 4) even improves segmentation accuracy. whether without DSFF or with DSFF, when compared to kernel sizes of 3x3x3 (rows and 6). This further demonstrates that our proposed efficient Restricted Depth-Shift 3D Convolutional layer, which utilizes a 1x3x3 kernel with restricted depth shift, is equivalent to a 3x3x3 kernel in terms of segmentation accuracy. Moreover, it offers significant savings computational and memory resources. This demonstrates that the use of a 1x3x3 kernel with restricted depth shift is functionally equivalent to 3x3x3 kernel in terms of segmentation accuracy, all the while offering savings in computational and memory resources.
7

---

Rows 9 and 10 of Table 4 demonstrate that when restricted shift is applied to the height or width dimensions, the model's performance decreases compared E2ENet (4th row) with restricted shift on the depth dimension, called as restricted depth-shift. We also observed that the performance of 3D convolution with kernel sizes of x 3 x 3 (row 3), 3 x 3 (row 7), and 3 decreased significantly without restricted shift compared to their counterparts with restricted shift on the corresponding dimensions (rows 4, 9, and 10, respectively). These phenomena demonstrate the effectiveness of restricted shift, especially restricted depth-shift, for medical image segmentation.
1 (row 8)
Furthermore, we evaluated the impact of different shift sizes on the model's performance. As shown in Table we compared the performance of E2ENet with shift sizes of (-1, O, (-2, 0, 2),
Table 6: Evaluating the generalizability of E2ENet on the
AMOS-MRI dataset. CT - MRI: pretrained on the CT dataset and fine-tuned on the MRI dataset; MRI: trained solely on the MRI dataset; CT+MRI: trained on both the CT and MRI datasets. Results for all models, except E2ENet and
(-3, 0, 3) and (-7, 0, 7), and observed a decrease in performance from 90.1% to 89.8%, 89.7% and 87.6%, respec- tively, as the shift size increased. In- creasing the shift size means consid- ering more depth- -ise information at the expense of channel-wis informa- tion, leading to an insufficient repre- sentation of channels, discussed in Section 2.3
nnUNet (CT + MRI), are sourced from the AMOS website
method
mDiic (9t)
mNSD (S)
MRI
MRI
CT +
MRI
CT
87.8
nntiNer
87.4
87.1
87:7
83.3
83.1
82.7
VNet
83.9
75.4
65.8
65,6
74.0
66.63
CoTr
77.5
73.46
78.0
65,35
Swin UNTER UNTER
75.7
77.52
65.3
69.10
70.1
Additionally, large shift size (equivalent to large kernel size) leads to a loss of local spatial relationships, which are crucial for segmentation. This results in a blurring effect that reduces the precision of boundary alignment, particularly affecting metrics like mNSD, which rely heavily on accurate boundary information. Therefore, we use a shift size of (-1, 1) in our restricted depth-shift strategy as the default setting in our experiments. Finally, we plot a critical distance diagram to show the statistical significance of our modules in Appendix A.8.3
3.3 Generalizability Analysis
To evaluate the generalizability of the resulting architectures, we compared our model, pre-trained on AMOS-CT and fine-tuned on AMOS-MRI, with nnUNet, which was pre-trained on AMOS-CT and fine-tuned on AMOS- MRI, and both models are fine-tuned for 250 epochs. is worth noting that the topology (weight connections) is determined through AMOS-CT pre- training and remains fixed during the fine-tuning phase. From Table we discovered that the E2ENet architec- ture, initially designed for AMOS-CT, can be effectively applied to AMOS- MRI as well. This adaptation resulted in improved performance compared to UNet and other baselines. nnUNet and these baselines were either transferred from CT, trained solely on MRI data, or jointly on MRI and CT datasets.
Table 7: Quantitative comparisons of segmentation perfor- mance on AMOS-CT test dataset. 1 and denote the results with and without postprocessing that reproduced by us. indicates the results with postprocessing.
Methods
mDice
Params
FLOPs
PT score
mNSD (5) f
(%)*
(M) 4
(G)
t
CoTr
$0.9
41.87
1510.53
1.11
66.3
nnFormer
85.6
150.14
1343.65
1.11
72.5
UNETR
79.4
93.02
391.03
1.41
60.8
Swin-UNETR
86.3
62.83
1562.99
1.13
73.8
VNet
82.9
45.65
1737.57
1.II
67.6
nnUNett
90.6
30.76
1067.89
1.31
82.0
unNNet
91.0
30.76
1067.89
1.31
82.6
E2ENer (s=0.7)
90.7
11.23
969.32
1.54
82.2
E2FINet* (s=0.8)
90.7
9.44
778.74
1.65
82.1
E2HNet" (s=0.9)
90.4
7.64
492.29
1.89
81.4
E2 (s=0.7)
90.6
11.23
969.32
1.54
82.0
E2ENet (s=0.8)
90.4
9.44
778.74
1.65
81.8
E2ENet (s=0.9)
90.1
7.64
492.29
1.89
80.7
In the AMOS-CT dataset, there is domain shift between the training and test datasets due to variations in the image acquisition scanners (Ji et al., 2022]. Thus, we also assess the generalizability of our approach by evaluating its performance on the AMOS-CT test dataset. As demonstrated in Table 7 E2ENet exhibits comparable performance even in the presence of domain shift when compared to nnUNet. This is achieved while maintaining lower computational and memory costs.
8

---

3,4 Model Capacity Analysis
To ensure a fair comparison, we scale down nnUNet by reducing the number of channels at level 1 from 32 to 27 and decreasing the total number of feature scales from 5 to 4, resulting modified version referred to as nnUNet (-). Additionally, we scale up E2ENet by increasing the number of channels at level from 48 to 58, creating modified version referred to as E2ENet (+). We evaluated the performance of these models, and the results can be found in Table X It is worth noting that scaling down nnNe resulted in decreased performance in terms of mDice and mNSD, while scaling up E2ENet led to an increase in mDice and comparable performance in terms of mNSD. This indicates that the comparable performance of the memory and computation- attributed to the dataset's 's requirement for a small number of parameters and computations.
E2ENet not
3.5 Qualitative Results
In this section, we compare the proposed E2ENet and nnUNet qualitatively on three chal- lenges. To make the comparison easier, we highlight detailed segmentation results with red dashed boxes.
Table 8: A comparison under a similar FLOPs
budget on the AMOS-CY CT challenge in two cases: when nnUNet is scaled down and when E2ENet is scaled up.
Method
mDice (%)
Params (M) 4
FLOPPs
mNSD
pressnts qualitative comparison of our proposed E2ENet with unUNet on the AMOS-
(G) 4
(%) t
EZENet (S=0.8)
90.0
9,43
778.74
82.3
nnUNet
90.0
31.20
1067.89
82.1
CT challenge. As widely used baseline, nnUNet has been evaluated on multiple med-
nnUNeL
89.7
12.96
755.79
81.9
F2RNCT (+)
90.4
10.37
1119.17
82.2
ical image segmentation challenges. Our results
demonstrate that, on certain samples, E2ENet can improve segmentation quality and overcome some of the challenges faced on the AMOS- -T challenge. For example, as shown in the first column, E2ENet more accurately distinguishes between the stomach and the esophagus, which is challenging due to their close proximity. In the second column, E2ENet better differentiates the duodenum from the background, while in the third column, E2ENet accurately identifies the precise boundaries of the liver, a structure that is prone to wver-eegmentation. These examples demonstrate the potential of our proposed method to improve the accuracy of medical image segmentation to some extent, especially in challenging cases such as distinguishing between closely located organs or accurately segmenting complex shapes. The Qualitative Results of BTCV and BraTS Challenge can be found in Appendix A.9.I
3.6
Feature Fusion Visualization
In this section, we explore how the DSFF mechanism fuses features from three directions (upward, forward, and downward) during training. To shed light on this mecha- nism, specifically analyze the proportions F of feature map connections from different directions to a specific fused feature node, considering feature sparsity level of 0.8. After training, we observed in Figure 6 (d) that the DSFF module learned to assign greater importance to features from the "forward" directions for most fused feature nodes. For example, in the first feature level, the flow and processing of features can be observed as if they were passing through fully convolutional neural network (FCN), which preserves the spatial dimensions of the input image. At the second feature level, the original image is downsampled by a factor of 1/2 and flows through another FCN. Therefore, from this perspective, E2ENet can take advantage of FCN to effectively incorporate and preserve multi-scale information.
Image
Draund Trush
C
Polls
Figure 5: Qualitative comparison of E2ENet and nnUNet on the AMOS-CT challenges.
'We calculated these proportions by dividing the number of non zero connections in a given direction by the total number of non-zero connections within each fused feature.

---

Simultancously, the complementary feature flows from the 'upward' and 'downward' directions pro- vide richer information. At the later fusion step of the lower feature levels, the 'upward' information becomes more dominant than the 'downward' information. This prioritization of upward feature flow is similar to the design of the decoders in UNet and UNet++. While E2ENet alleviates the semantic gap more effectively, the proportion of feature map connections in the 'downward" direction always has a certain ratio. This also allows the network to better preserve low-level information and integrate it with high-level information, leading to improved performance in capturing fine details. Interestingly, this trend becomes increasingly apparent during training, as illustrated Figure (a), (c) and (d).
(c) opoch
Figure 6: The proportions feature connections during training with the DSFF mechanism at feature sparsity level of 0.8 on the AMOS-CT challenge.
4 Conclusion
In this work, our aim is to address the challenge of designing a 3D medical image segmentation method that both accurate and efficient. By proposing dynamic sparse feature fusion mechanism and incorporating restricted depth-shift in 3D convolution, our E2ENet improves performance on 3D medical image segmentation tasks while significantly reducing computational and memory overhead. The dynamic sparse feature fusion mechanism demonstrates its ability to adaptively learn the importance of each feature map, zeroing out the less important ones. This results in more efficient feature representation without compromising performance. Additionally, the experiments demonstrate that restricted depth-shift in 3D convolution enables the model to capture spatial information more effectively and efficiently. Extensive experiments on three benchmarks demonstrate that E2ENet consistently achieves a superior trade-off between accuracy and efficiency compared to previous state-ot of-the-art baselines. While E2ENet provides promising solution for balancing accuracy and computational cost, future work could explore the potential of learnable shift offset, which may further improve performance. Additionally, plug-and- components, the DSFF mechanism and restricted depth-shift could interesting to apply to other segmentation models explore their potential in the future. Furthermore, future advancements in hardware support for sparse neural networks could fully unlock the potential of sparse training methods.
Acknowledgements
Qiao Xiao is supported by the research program 'MegaMind- Measuring, Gathering, Mining and Integrating Data for Self-management the Edge the Electricity System', (partly) financed by the Dutch Research Council (NWO) through the Perspectief program under number P19-25, Elena Mocanu is partly supported by the Modular Integrated Sustainable Datacenter (MISD) project funded by the Dutch Ministry of Economic Affairs and Climate under the European Important Projects of Common European Interest Cloud Infrastructure and Services (IPCEI-CIS) program for 2024-2029, This research used the Dutch national e- e-infrastructure with the support of the SURF Cooperative, using grant no. EINF-7499.
10

---

References
Michela Antonelli, Annika Reinke, and Annette Kopp-Schneider Bennett A. Landman Geert Litjens Bjoem Menze Olaf Ronneberger Ronald M. Summers Bram van Ginneken Michel Bilello Patrick Bilic Patrick F. Christ Richard K. G. Do Mare J. Gollub Stephan H. Heckers Henkjan Huisman William R. Jarnagin Maureen
K. McHugo Sandy Napel Jennifer S. Golia Pernicka Kawal Rhode Catalina Tobon-Gomed Eugene Vorontsov
James A. Meakin Sebastien Ourselin Manuel Wiesenfarth Pablo Arbeláez Byeonguk Bae Sihong Chen Laura Daza Jianjiang Feng Baochun He Fabian Isensee Yuanfeng Fucang Jia Ildoo Kim Klaus Maier-Hein Dorit Merhof Akshay Pai Beomhee Park Mathias Perslev Ramin Rezaifar Oliver Rippel Ignacio Sarasua Wei Shen Jaemin Son Christian Wachinger Liansheng Wang Yan Wang Yingda Xia Daguang Xu Zhanwei Xu Yefeng Zheng Amber L. Simpson Lena Maier-Hein M. Jorge Cardoso Spyridon Bakas, Keyvan Farahani. The medical segmentation decathlon. Nature Communications, 2022.
Zahra Atashgahi, Ghada Sokar, Tim van Lee, Flena Mocanu, Decebal Constantin Mocanu, Raymond Veldhuis, and Mykola Pechenizkiy. Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders. Machine Learning, 11 111(1):377-414, 2022.
Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-Unen Unet-like pure transformer for medical image segmentation. arXiv: 100:2205.000 055005577 2021.
Liang- ChingCChihhh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/1706.05587, 2017.
Yinpeng Chen, Xiyang Dai. Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In 2020 IEEF//CV Conference on Computer Vision and Pattern Recogni- tion, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 1007--10036 Computer Vision Foundation / IEEE, 2020.
Özgün Çişek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention, pages 424432. Springer, 2016.
Zihang Dai, Hanxiao Liu, Quec V Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. Advances Neural Information Processing Systems, 34:3965-3977, 2021.
Stéphane Ascoli, Hugo Touvron, Matthew Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. ConViT: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pages 2286-2296. PMLR, 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, The Journal of Machine Learning Research, 20(1):1997-28 2017, 2019.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen, Rigging the lottery: Making all tickets winners. In International Conference Machine Learning, pages 2943-2952 PMLR, R, 2020.
Linxi Fan, Shyamal Buch, Guanzhi Wang, Ryan Cao, Yuke Zhu, Juan Carlos Niebles, and Li Fei-Fei. RubiksNet: Learnable 3D-shift for efficient video action recognition. European Conference on Computer Vision, pages 505-521. Springer, 2020.
Y. Gao, M. Zhou, D. Liu, and D. Metaxas. A data-scalable transformer for medical image segmentation: Architecture, model efficiency, and benchmark. arXiv preprint arX1v:2203.00131, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.0073 2023.
Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang. Andriy Myronenko, Bennett Landman, Holger Roth, and Daguang Xu. UNETR: Transformers for 3D medical image segmentation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 574-584, 2022.
Yufan He, Dong Yang, Holger Roth, Can Zhao, and Daguang Xu. DiNTS: Differentiable neural network topology search for 3D medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5841-5850, 2021.
Torsten Hoefler, Dan Alistarh, Tal Nen-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22(241): 1-124, 2021.
11

---

Xiaowan Hu, Rujjun Ma. Zhihong Liu, Yuanhao Cai. Xiaole Zhao. Yulun Zhang, and Haoqian Wang. Pseudo 3D auto-correlation network for real image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16175-16184, 2021.
Huimin Huang, Lanfen Lin, Ruofeng Tong, Hongjie Hu, Qiaowei Zhang. Yutaro Iwamoto, Xianhua Han. Yen-Wei Chen, and Jian Wu. UNet 3+: A full-scale connected UNet for medical image segmentation. In ICASSP 2020-2020 JEEE International Conference on Acousticy, Speech and Signal Processing (ICASSP).
pages 1055-1059, IEEE, 2020.
Nabil Ibtehaz and M Sohel Rahman. MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Networks, 121:74-87, 87, 2020.
Fabian Isensee. Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier- Hein. nnU-Net: a self- configuring method for deep learning carning-based biomedical image segmentation. Nature Methods, (8(2):203-211 2021.
Fabian Isensee, Constantin Ulrich, Tassilo Wald, and Klaus H Maier- Extending nnU-Net is all you need. arXiv:2208. 2022.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-KAST: Top-K always sparse training. Advances in Neural Information Processing Systems, 33:20744-20754 2020.
Debesh Jha, Pia H Smedsrud. Michael A Riegler, Dag Johansen, Thomas De Lange, Pål Halvorsen, and Håvard D Johansen. ResUNet++: An advanced architecture for medical image segmentation. In 2019 IEEE International Symposium on Multimedia (ISM), pages 225-2255. IEEE, 2019.
Yuanfeng Ji. Haotian Bai, Chongjian GE, Jie Yang. Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng. Wanling Ma, Xiang Wan, and Ping Luo. AMOS: A large-scale abdominal ougti-organ benchmark for versatile medical image segmentation. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Davood Karimi, Serge Didenko Vasylechko, and Ali Gholipour. Convolution- free medical image segmentation using transformers. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pages 78-88. Springer, 2021.
Landman, Z Xu, J Igelsias, M Styner, T Langerak, and A Klein. Miccai multi-ater atlas labeling beyond the cranial vault workshop and challenge. In Proc. MICCAI Multi Atlas Labeling Beyond Cranial Vault Workshop Challenge, 2015.
Namhoon Lec, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2018.
Chen Li, Yusong Tan, Wei Chen, Xin Luo, Yuanming Gao, Xiaogang Jia, and Zhiying Wang. Attention Unet++: A nested attention-awar U-Net for liver et image segmentation. In 2020 IEEE International Conference on Image Processing (ICIP), pages 345-349. IEEE, 2020.
Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. AS-MLP: An axial shifted mlp architecture for vision. In International Conference on Learning Representations, 2021.
Ji Lin, Chuang Gan, and Song Han. TSM: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7083-7093, 2019.
Shiwei Liu and Zhangyang Wang. Ten lessons we have learned in the new" sparseland": A short handbook for sparse neural network researchers. arXiv preprint arXiv:2302.02556 2023.
Shiwei Liu, Tianlong Chen, Xiaohan Chea, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with ncuroregeneration. Advances in Neural Information Processing Systems, 1199908-99221 2021a.
Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary deep learning with over one million artificial neurons commodity hardware. Neural Computing and Applications, 33(7):2589-266 22044 2021b.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over- parameterization? in-time over parameterization in sparse training. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 6989-7000. PMLR, 2021c
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang. More ConvNets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv:2207.0363 03620, 2022.
12

---

Andrew L Maas, Awni Y Hannun, Andrew Y Ng. et al. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the 30th International Conference Machine Learning, number 1, page 3. Atlanta, Georgia, USA, 2013.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D vision (3DV), pages 565-571. IEEE, 2016.
Decebal Constantin Mocanu, Elena Mocanu, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. A topological insight into restricted boltzmann machines. Machine Learning, 104(2):24322) 270, 2016.
Decebal Constantia Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibeseu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 1-1: 2018.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pages 4646-4655. PMLR, 2019.
Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, Ben Glocker, and Daniel Rucckert, Attention U-Net: Learning where to look for the pancreas. In Medical Imaging with Deep Learning, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyToorc: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer assisted Intervention, pages 234 241. Springer, 2015.
Jiacheng Ruan and Suncheng Xiang. Vm-unet: Vision mamba unet for medical image segmentation. arXiv preprint arXiv:2402. 02491, 2024.
Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello. Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, et al. large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv: arXXiv:9002.000 190200903, 2019.
Ghada Sokar, Zahra Atashgahi, Mykola Pechenizkiy, and Decebal Constantia Mocanu. Where to pay attention in sparse training for feature selection? In Advances in Neural Information Processing Systems, 2022.
Zhuo Su, Linpu Fang, Wenxiong Kang, Dewen Hu, Matii Pietikäinen, and Li Liu. Dynamic group convolution for accelerating convolutional neural networks. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm, editors, Computer Vision ECCV 2020- 16th European Conference, Glaxgow, UK, August 23-28, 2020, Proceedings, Part VI, 2020.
Mingxing Tan, Ruoming Pang, and Quoc V Le. EfficientDet: Scalable and efficient object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10781-10790, 2020.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow, Advances in Neural Information Processing Systems, 05:6377-6389 2020.
Yucheng Tang, Riqiang Gao, Ho Hin Lee, Shizhong Han, Yunqiang Chen, Dashan Gao, Vishwesh Nath. Camilo Bermudez, Michael R Savona, Richard G Abramson, et al. High-resolution 3D abdominal segmentation with random patch network fusion. Medical Image Analysis, 69:101894, 2021.
Yucheng Tang, Dong Yang, Wenqi Li, Holger Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3D medical image analysis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 20740, 2022.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky, Instance normalization: The missing ingredient for fast stylization. urXiv:1607.08022. 2016.
Jeya Maria Jose Valanarasu and Vishal M Patel. UNeXt: MLP based rapid medical image segmentation network. arXiv:2203.04990 2022.
13

---

Chaoqi Wang, Guodong Zhang. and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2019.
Haonan Wang, Peng Cao, Jiaqi Wang, and Osmar R Zaiane. UCTransNet: rethinking the skip connections in U-Net from channel-wise perspective with transfomer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2441-2449, 2022.
Wenxuan Wang, Chen Chen, Meng Ding, Hong Yu, Sen Zha, and Jiangyun Li. TransBTS: Multimodal brain tumor segmentation using transformer. International Conference on Medical Image Computing and Computer- Assissted Intervention, pages 109-100 119. Springer, 2021.
Ziyang Wang. Hian-Qing Zheng, Yichi Zhang, Ge Cui, and Lei Manba-unet: Unet-like pure visual mamba for medical image segmentation. arXiv preprint urXiv:2402.0500 2024.
Yu Weng. Tianbao Zhou, Yujie Li. Xiaoyu Qiu. NAS- UnstUnet: Neural architecture search for medical image segmentation. IEEE Access, 7:44247 44257, 2019.
Qiao Xiao, Boqian Wu, Zhang, Shiwei Liu, Mykola Pechenizkiy, Elena Mocanu, and Decebal Constantin Mocamu. Dynamic sparse network for time series classification: Learning what to "see". In Advances in Neural Information Processing Systems, 2022.
Yutong Xie, Jianpeng Zhang, Chunhua Shen, and Yong Xia. CoTr: Efficiently bridging CNN and transformer for 3D medical image segmentation. In International Conference on Medical Image Computing and Computer- assisted Intervention, pages 171-180 Springer, 2021.
Zhaohu Xing, Tian Ye. Yijun Yang, Guang Liu, and Lei Zhu. Segmamba: Long-range range sequential modeling mamba for 3d medical image segmentation. arXiv preprint arXiv:2401.13566 2024.
Qihang Yu, Dong Yang, Holger Roth, Yutong Bai, Yixiao Zhang, Alan L. Yuille, and Daguang Xu. C2FNAS: coarse-to to-sic- neural architecture scarch for 3d medical image segmentation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 4125 4134. Computer Vision Foundation / JEEE, 2020.
Geng Yuan. Xiaolong Ma, Wei Niu. Zhengang Zhenglun Kong. Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang Hc, Qing Jin, et al. MEST: Accurate and fast memory- sparse training framework on the edge. Advances in Neural Information Processing Systems, 34:20838-20888 2021.
Hong- Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang. and Yizhou Yu. nnFormer: Interlcaved transformer for volumetric segmentation. arXiv:2109.0322 2021.
Yuyin Zhou, Zhe Li, Song Bai, Chong Wang, Xinlei Chen, Mci Han, Elliot Fishman, and Alan L Yuille. Prior- aware neural network for partiall--supervise multi-organ segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10672 10681, 2019.
Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. UNet++: A nested U-Net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 3-11. Springer, 2018
14

---

A Appendix.
A.1 Impact Statement
In an era dominated by parapaammetere models, designing resource-award Al models is becoming increasingly important, especially for time-consuming tasks like medical segmentation. Our insights into model efficiency training have the potential to broaden the application of deep neural networks in this area. Overall, this work advances our fundamental understanding of dynamic sparse training and offers future perspectives for scalable and efficient Al models. We do not anticipate any negative societal impacts resulting from this research.
A.2 Related Work
A.2.1 3D Medical Image Segmentation
Convolutional neural networks (CNNs) have become the dominant architecture for 3D medical
image segmentation recent years (e.g. 3D UNet (Ciekk et al. (0T6). UNet++ [Zhou et al. 2018), UNet3+ [Huang et al., 2020). PaNN [Zhou et al., 2019 and nn [Isensee et al. [02 duc to their ability to capture local and weight-sharing dependencies [d d''Ascoli et al. 202 Dai et al.
2021]. However, some recent methods have attempted to incorporate transformer modules into CNNs
(e.g. CoTr Xie et al. 2021]. TransBTS [Wang et al. 2021)) D. or use pure transformer architectures (e.g. Convit [Karimi et al. (021). anFormer [hoou et al. 2021), Swin UNet [Cao et al., (022]). in
order to capture long range dependencies. These transformer based approaches often require large amounts of training data, longer training times, or specialized training techniques, and can also be computationally expensive. Most recently, a novel architecture called Mamba (Gu and Dao 2023
has shown potential for computational efficiency as State Space model in handling long sequences and has been applied to medical image segmentation tasks [Kuan and Xiang 2024]] Xing et al., 2024 Wang et al. 2024). However, has led underwhelming performance compared state-of-the-an convolutional models. In this paper, we propose an alternative method for efficiently incorporating 3D contextual information using restricted depth-shift strategy in 3D convolutions, and further improving performance through adaptive multi-scale feature fusion.
A.2.2 Feature Fusion in Medical Image Segmentation
Multi-scale feature fusion a crucial technique in medical image segmentation that allows a model to detect objects across range of scales, while also recovering spatial information that is lost during pooling [Wang et al., 2022 Xie et al. (202T). However, effectively representing and processing multi-scale hierarchy features can be challenging, and simply summing them up without distinction
can lead to semantic gaps and degraded performance [Wang et al. [202]] Tan et al. 2020). To address this issue, various approaches have been proposed. including adding learnable operations to reduce the gap with residuals [Ibtchaz and Rahman 2020), attention blocks [Oktay et al., 2018). More recently, UNet++ [Zhou et al.) 2018 and its variants [Li et al.) 2020 Huang al.) 2020 Tha et al.) 2019] have adapted the gating signal to dense nesting levels, taking into account as many feature levels possible. NAS-UNet [Weng et al. 2019 tries automatically search for better feature fusion topology. While these methods have achieved better performance, they can also incur significant computational and information redundancy. Dynamic convolution (Su et al.) 2020 Chen al. 2020) utilizes coefficient prediction or attention modules to dynamically aggregate convolution kernels, thereby reducing computation costs. In our paper, we propose an intuitive approach to optimizing multi-scale feature fusion, which enables selective leveraging of sparse feature representations from fine-grained to semantic levels through the proposed dynamic sparse feature fusion mechanism.
A.2.3 Sparse Training
Recently, sparse training techniques have shown the possibility of training an efficient network with sparse connections that match (or even outperform) the performance of dense counterparts with lower computational cost [Mocanu et al. 2018 Liu et al. 2021b]. Beginning with [Mocanu et al. 2016)]. it has been demonstrated that initializing a static sparse network without optimizing its topology during training can also yield comparable performance in certain situations [Lee et al. 2018 Tanaka et al. 2020 Wang et al. 2019]. However, Dynamic Sparse Training (DST), also known as sparse training with dynamic sparsity [Mocanu et al. 2018), offers different approach by jointly optimizing the
15

---

Algorithm The Training Process of Dynamic Sparse Feature Fusion (DSFF)
Require: Dataset X' with label y; feature sparsity S; backbone fe(:;; Output Module: fouti Total training epochs: T:
evolution period: AT; connection updating number; fdecay (AT; a, T) ž cos ( A a represents the number of updated connections during the initial topology update, which is set to 1/2; Loss function: [(); fusion operation: (.) with convolution kernels 8 where the numbers of input and output channel are C Cou..
T
1: M f- random initialize masks for all levels and stages, satisfying that | equals (1
S) x Cd.
in x C
2: for to T do
3: Sample a batch 4. Y ~ X.
4: Generate multi-scaled features: (x x0.2 ....,,x11) fe(
5: for each stage j = to L - do
6: for each level 1 to L j do
7:
if i then
8:
x F.. ([x) ((xxx((x1)))
9:
else
= = -15-1111111111 x 1, u(xx-11ttt))
1,5
11: end if
end for
13:
end for
4
4 4/7L(foou (x!-111),Y3
+
2/7L(fou (x1-22) D(Y:))
1/7C(four (x ,(D((X)))
15:
if (t mod AT) = 0 then
16:
for each stage j = 1 to L do
17:
for each level
to L j do
u1 = (C in
C. Cout)deccay a, T) (1 -
19:
IS +- importance score (L1 Norm of corresponding kernel) for activated each feature connection
20:
Lactivate Randomk K (linactivates u) linactivate = ArgTophe ooo (-IS, u) M t- Update M using linctivate and lactivate
21:
22:
23:
end for
24:
end for
25: else
26:
Training the E2ENet using SGD optimizer
27:
end if
end for
sparse topology and weights during the training process starting from a sparse network [Lu ct al.
2021a 2022 Evei et al., 2020 Jayakumar al. 2020)) Mostafa and Wang 2019 Yuan et al. 2021 This alwwws the model's, s sparse connections to gradually evolve in prune-and-gun grow scheme, leading to improved performance compared to naively training static sparse network [Liu et al. 2021c
Xiao et al. 2022). In contrast to prior methods that aim to find sparse networks that can match the performance corresponding dense networks, we aim to leverage DST to adaptively fuse multi-scale features in computationally efficient manner for 3D medical image segmentation.
A.3 Algorithm
A.4 Datasets and Experiment Setup
AMOS-CT: The Abdominal Multi-Organ Segmentation Challenge (AMOS) [Ji et al. [022] task 1 consists 500 computerized tomography (CT) cases, including 200 scans for training, 100 for validation, and 200 for testing. These cases have been collected from a diverse patient population and include annotations of 15 organs. The scans are from multiple centers, vendors, modalities, phases, and discases.
16

---

BTCV:
The Beyond the Cranial Vault (BTCV) abdomen challenge dataset consists of 30 CT scan images for training and 20 for testing. These images have been annotated by interpreters under the supervision of radiologists, and include labels for 13 organs.
BraTS: The Brain Tumor Segmentation Challenge in the Medical Segmentation Decathlon (MSD) [Attonelli et al., [022 Simpson et al. 2019) consists of 484 MRI images from 19 different institutions. These images contain three different tumor regions of interest (ROIs): edema (ED), non-enhancing tumor (NET) and enhancing tumor (ET). The goal of the challenge is to segment these ROIs in the images accurately.
A.5 Implementation Details
In our work, we utilized the PyTorch toolkit [Paszke et al., 2019 NVIDIA A100 GPU for all our experimental evaluations. We also used the nnet codebase [ssessee et al. 2021 to pre- pre-process data before training our proposed E2ENet model. For the AMOS dataset, used the nnUNet codebase as the benchmark implementation.
For training, we use the stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.01, which is gradually decreased through a "poly" decay schedule. The optimizer configured with momentum of 0.99 and a weight decay of 3 x 10-5, The maximum number of training epochs 1000, with 250 iterations per epoch. For the loss function, we combine both cross-entropy loss and Dice loss in [Isensee et al. 2021]. To improve performance, various data augmentation techniques such as random rotation, scaling, flipping, adding Gaussian noise, blurring, adjusting brightness and contrast, simulating low resolution, and Gamma transformation are used before training.
We employ a 5-fold cross-validatte model from each fold and simply averaging their outputs for the final segmentation predictions. In the testing stage, we employ the sliding window strategy, where the window sizes are equal to the size of the training patches. Additionally, post-processing methods outlined in [[sensee et al.) 2022 are applied for the AMOS-CT dataset during the testing phase.
strategy on the training set for all experiments, selecting the final
A.6 The Architecture
The backbone generates a total of 7, 6 multi-scale feature levels, each with specified number of channels: [cl, c2, c3, c4, c5, c6] = [48, 96, 192, 320, 320, 320]. At each level of feature generation, there are two convolution layers with kernel size of (1, 3, 3), followed by instance normalization and the application of leaky ReLU activation. The down-sampling ratios for each level are as follows: ((1, 2, 2), (2, 2), (2, 2), (2, 2, 2), (2, 2, 2), (2, 2, 2), (2, 2)).
A.7 Evaluation Metries
A.7.1 Mean Dice Similarity Coefficient
To assess the quality of the segmentation results, use the mean Dice similarity coefficient (mDice), which is a widely used metric in medical image segmentation. The mDice is calculated as follows:
1 N
2 ý, (yg + 15:1)*
mDice
where N is the number of classes, is the pointwise multiplication, y, and ý, represent the ground truth and predicted masks of the j-th class, respectively, which are encoded in one-hot format. 8yl the Dice of j-th class, which measures the overlap between the predicted and ground Tyys +)
truth segmentation masks for that class.
A.7.2 Number of Parameters
The size of the network can be estimated by summing the number of non-zero parameters (Params), which includes the parameters of activated sparse feature connections (kernels) and parameters the
17

---

backbone. The calculation is given by the following equation:
7-14- ΣΣΣ /=1 Gan
C2
C
Params 110||o +
M Cous
1
Con Coue llo-
Here, ④ is the parameter from backbone, L is the total number of feature levels, M is a matrix of indicates whether the kernel Gäd Cin.fout connecting the Gin -th input and Coue-th output feature map exist or not. The Lo norm ||0 Com provides the number of non-zero entries of (2.1
size C Co and M Con.Cout
Cis Cons"
A.7.3 Float Point Operations
Floating point operations (FLOPs) is commonly used metric to compare the computational cost of In our comparison, it is calculated
sparse model to that of a dense counterpart (Hoefler et al. 2021] 7 by counting the number of multiplications and additions performed in only one forward pass of the inference process without considering postprocessing. The inference FLOPs are estimated layer by layer and depend on the sparsity level of the network. For each convolution or transposed convolution layer, the inference FLOPs is calculated as follows:
FLOPs = (2K, K, C. -1- -) S) + 1) x HWD,
(5)
where Ka, Kn and Kw are the kernel sizes in depth, height and width; S is the feature sparsity level, for layers that are not part of the DSFF mechanism, S = is used; Csn and Cout the numbers of input feature and output feature; H. and D are the height, width and depth of output features. For each fully connected layer, the inference FLOPs is calculated follows:
FLOPsfe (2C (1 8) + 1) Cout.
A.7.4 Performance Trade-Off Score
The accuracy-efiin accuracy-efficiency trade-offs could be further analyzed, from comparing resource requirements to describing holistic behaviours (including mDice, Params and inference FL.OPs) for the 3D image segmentation methods. To quantify these trade-offs, we introduce the Performance Trade-Off (PT)
score, which defined as follows:
m Dice
Paramsmin FLOPsmin FLOPs
PT = 01 m Dicemas
+ 02
Params
where a and 0g are weighting factors, which control the trade-off between accuracy performance and resource requirements, and mDicemmax Paramsmin- and FLOPsmin denote the highest mDice score, the smallest number of parameters, and the lowest inference FLOPs among the compared methods for a specific dataset, respectively. The term Dice measures the segmentation accuracy, mice
while Paremsmus. FLOPEmin measures the resource cost.
Porams
PLOP=
In most cases, we consider both segmentation accuracy and resource cost to be equally important, thus we set a1 = and aı 1/2 in the following experiments. However, we also explore the impact of different choices of and a22 as detailed in Section A.9 The PT score serves as valuable metric for evaluating the trade-offs between segmentation accuracy and efficiency.
A.7.5 Discussion on Running Time
Given the relatively restricted support for sparse operations in current off-the-shelf commodity GPUs and TPUs without aparsity-aware accelerators, we did not attempt to achieve practical speedup during training. Instead, we chose to implement our models with binary masks in work. The promising benefits of dynamic sparsity presented in this study have not yet translated into actual speedup. Accelerating training time will be focus for next work.
Thii because current sparse training methods often use masks on dense weights to stimulate sparsity. This done because most deep learning hardware is optimized for dense matrix operations. As a result, using these prototypes doesn't it accurately reflect the true memory and speed benefits of a truly sparse network (Hoefler et al. 2021
18

---

Although not the focus of our current work, it would be interesting for future work to examine the speedup results of sparse operation during training. using such specialized hardware accelerators. For example, at high unstructured sparsity levels, XNNPACK 8 has already shown significant speedups over dense baselines on smartphone processors.
Although the support for unstructured sparsity on GPUs remains relatively limited, its practical relevance has been widely demonstrated on non-
Table 9: Comparison of inference speeds between E2t and nnt.
Methods
noUNeer
E2FNet((=O.8)
GPU hardware, such as CPUs and customized accelerators. For example, FPGA accelerators have demonstrated significant acceleration and energy efficiency for unstructured sparse RNNs, outperforming commercial CPUs and GPUs by effectively leveraging the FPGA's embedded resources. Additionally, DeepSparse has shown impressive results in deploying large-scale, BERT-level sparse models on modern Intel CPUs. This approach achieved 10x compression in model size with under a 1% accuracy loss, I Oxx increase in CPU inference speed with less than 2% accuracy reduction, and an impressive 29x CPU inference speedup with a maximum accuracy drop of 7.5% [Liu and Wang 2023
Latency(ms) 10.07
8.02
7.28
Thrughhat(temmske)) speedup
99.19
137.13
1.0%
1.26x
1,38
Inspired by these advancements, we adopted an approach based on DeepSparse. As shown in Table 9 we conducted experiments with patches of images sized 32x32x32 as input, comparing the CPU wall-clock timings for online inference between our proposed E2ENet and mnUNet on an Intel Xeon Platinum 8360Y CPU with 18 cores. We acknowledge that, while our proposed models with sparsity do achieve speedups in practical inference, they are not as pronounced as those observed with BERT-level sparse models. This is primarily due to the nature of segmentation and 3D convolution operations. However, this presents promising avenue for our future work.
A.8
More Experimental Results
A.8.1 Class-wise Dice of AMOS-CT
Table 10: Quantitative comparisons (class-wise Dice (%) f. mDice(%)1. t, Params(M)4. inference
FL., PT scoref and mNSD($))) of segmentation performance on the validation set of AMOS- CT dataset. Bold indicates the best and underline indicates the second best. Note: Spl: spleen, RKid:
right kidney, LKid: left kidney, Gall: gallbladder, Eso: esophagus, Liv: liver, Sto: stomach, Aor: aorta IVC: inferior vena cava, Pan: pancreas, RAG: right adrenal gland, LAG: left adrenal gland, Duo: duodenum, Bla: bladder, Pro/Uth: prostate/uterus. The wise Dice, mDice and mNSD results of baselines, except for uneNet, are collected from the [Ji et al. 2022). indicates the results without postprocessing that are collected from the AMOS website denotes the results with postprocessing that are reproduced by us.
indicates the results with postprocessing.
Metbods
Spi
RX1d
LKd
Gdll
Eio
Lis
Sto
Aor
IVC
Fun
RAG
LAG
Duo
Bla
ProtU
mDDice
Parstis
FLO8,
PT score
nEND
Coli
91.1
81.2
86.4
60.5
$0.5
91.6
80.1
45.7
$7.7
76.3
73.7
71.7
08.0
67.4
40.5
77.1
41.87
1510U5
1.07
64.2
mivemner
94.
76.5
$1.1
94.3
86.2
65.0
75.0
75.9
78.5
$3.9
74.6
150.14
1345.85
1.12
74.2
Swis UNETE
66.5
41.8
06.5
71.3
63.0
94.7
69.6
66.9
72.3
76.3
78.6
65.6
71.4
62.85
1562 59
1.16
25.3
VSSi
$6.2
91.
02.7
752
79.0
64.7
84.8
93.5
$1.4
50.5
72.6
73.2
71.3
71.5
65.6
92.0
45.65
1737.57
1.10
61.5
mit miNN
97.1
06.4
062
81.3
67.6
97.6
97.2
96.8
52.5
68.6
61.2
51.7
35.0
00.5
$1.0
9000
30:76
1067 8M
1.50
$2.1
mi Nast
95.1
97.8
07.1
85.6
67.7
92.9
92.4
96.8
92.7
68.8
81.6
32.1
25.0
90.6
85.2
98.4
30.76
1067 8M
1.51
83.8
EdENet (p
97.1
253
97.1
2500
32.6
999
12.3
0557
22.3
50.0
BLS
30.4
25.2
2003
35.3
13:3
11:25
10033
1.54
$2.1
(6-0.3)
07:0
92.1
54.5
90.1
776.74
RJENN"
97.1
1538
15.3
51.4
1000
51.
107.12
1.54
97.1
95.6
06.5
81.4
17.5
07.5
03.3
05.0
92.1
80.0
81.3
80.0
84.5
90.1
84.8
10.0
0.64
776.74
1.65
83.3
ETENee (seit
56.7
95.4
9644
87.6
66.9
97
05.6
93.0
58.6
80.9
61.7
84.0
$9.9
$1.5
89.6
7.51
093.29
1.88
81.4
NTNNeetattti su0.9)
87.6
$4.0
83.8
The inference FL,0Ps caloooed based
of 125
125 withost postprocesiing
A.8.2 BTCV Challenge
We compare the performance of our E2ENet model to several baselines (CoTr Xie et al. 2021
RandomPatch [Tang al. 1 PaNN (Zhou al. 2 UNETR (Hatamizadeh et al. 2022 and nnUNet [Tsensee et al.) 202TT) on the test set of B TTCV challenge, and report class-wiss wise Dice, mDice, Params and inference FLOPs on the test set in Table II is worth noting that nneN is strong performer that uses an automatic model configuration strategy to select and ensemble
hanns//gghhnn.co///ogglXXXXP hhunss/gitttoooommmmcrraaanggggo
19

---

two best of multiple U-Net models (2D, 3D and 3D cascade) based on cross-validation results. In contrast, E2ENet is designed to be computationally and memory efficient, using consistent 3D network configuration. Swin UNETR (Tang et al., 2022] is among the best on the leaderboard for this challenge. However, we do not include it in our comparison because employs self-supervised learning with extra data. This falls outside of our goal of trading off training efficiency and accuracy without using extra data.
Our proposed E2ENet, a single 3D architecture without cascade, has achieved comparable perfor- mance to nn une,, with mDice of 88.3%. Additionally, it has a significantly smaller number of parameters, 11.25 M, compared to other methods such as nnUNet (30.76 M), CoTr (41.87 M), and UNETR (92.78 M).
Table 11: Quantitative comparisons of segmentation performance BTCV test set. Note: Spl: spleen, RKid: right kidney, LKid: left kidney, Gall: gallbladder, Eso: esophagus, Liv: liver, Sto: stomach, Aor: aorta IVC: inferior vena cava, Veins: portal and splenic veins, Pan: pancreas, AG: adrenal gland. The results (class-wise Dice and mDice) for these baselines are from HHatamizadehi et al. 2022]. denotes that the training of UNETR* is without using any extra data outside the challenge. The results of nnUNet*, E2ENet and Hausdorff Distance (HD) of UNETR are from the standard leaderboard of BTCV challenge, while the results of nnUNet are from the free leaderboard
Marhods
RKid
1.Kid
IVC
Veins
miDic
FLOPs/
HD
Calr
7544
78.7
65634
RandomPatch
96.3
91.2
92.1
74.9
76.0
96.2
87.0
83.9
84.6
78.6
76.2
71.2
84.4
PaNN
96.6
92.7
95.2
73:2
79.1
97.3
89.1
91.4
85.0
50.5
50.2
65.2
85.4
UNEIR"
96.8
92.4
94.1
75.0
76.5
97.1
91.3
$9.0
84.7
78.8
76.7
74.1
85.6
92.79
164.91
1.53
23.4
unNNeel
97.2
91.8
95.8
75.3
81.1
97.7
92.2
92.9
88.1
81.2
85.2
27.8
88.4
31.18
$16.73
1.38
15.6
Nee
96.5
91.7
95.8
78.5
84.2
97.4
91.5
92.1
56.9
33.1
$4.9
73.5
88.0
31.18
416.73
1.38
16.9
12 (o 0.73
96.5
91.3
95.7
78.1
84.5
97.5
91.5
922
50.7
83.4
84.8
77.9
88.3
11:25
$49.00
1.68
16.1
The inference PLOPPeee calcalatal hased th juich devernine their model size and inderence PLOPs.
BandsonPach PaNS
publidy smiilabie possisl:
A.8.3 Statistical Significance of Designed Modules
To demonstrate the advantages of individual modules, we plot a critical distance diagram using the Nemenyi post-hoc test with p-value of 0.05 to establish the statistical significance of our modules. In Figure 7 the top line represents the axis along which the meth- ods' average ranks, and lower value indicates better performance. Methods joined by thick horizontal black lines are considered not statistically different. E2ENet with depth shift significantly outperforms E2ENet without depth shift. Additionally, the incorporation of dynamic sparse feature fusion into E2ENet results in a substantial reduction in both the number of FL.OPs (from 23.90M to 11.23M) and parameters (from 3069.55C; to 969.32G) while maintaining comparable performance, without any significant performance degradation.
Accurecy
0ST7 WAY ahi
DSFT stitt
Figure 7: The critical distance diagram on the AMOS-CT validation dataset, with the evaluation metric being mDice.
From the diagram, we can clearly observe that
A.9 The Impact of Weighting Factors a on PT Score
In this section, we investigate the impact of the weighting factors ai and a on the Performance Trade-off Score for the AMOS-C -T challenge, as defined in Equation 7 These factors are used to balance the trade-off between accuracy and resource cost. A higher value of a1 prioritizes accuracy, while a lower value emphasizes resource cost. Figure g shows that as 0r decreases, the gaps in the Performance Trade-off Score between E2ENet and other methods become larger, indicating that our method is more advantageous when prioritizing resource cost. However, even when Q1 is set to be 20 times greater than 022 which prioritizes accuracy over resource cost, the Performance Trade-off Score of E2ENet remains superior to other baselines. This result indicates that our proposed E2ENet architecture is highly efficient in terms of computational cost and memory usage while achieving excellent segmentation performance on the AMOS-CT challenge.
A.9.1 Qualitative Results
BTCV Challenge In Figure (b), we present a qualitative comparison of our proposed E2F.Net method with nne as bascline model on the BTCV challenge. Our results demonstrate the
20

---

Trate AMOOICT Dallergeiy
COMPCC SIMICA
Parberaacc
AMCS
DIN0-1 SIMPCE SIAPCE
Perfentornn Traik Soore AMDS www.s/tT-hhooooooooooooooo Chalmgrt Mam-
035
BRMP-CC ShalrOR
Figure 8: Comparison Performance Trade-OF score between E2ENet and other models on AMOS- CT challenge with varying a and a2 values. E2ENet outperforms other baselines in achieving better trade-off between accuracy and efficiency across different preferences.
effectiveness of our proposed method in addressing segmentation. For example, as shown in the first and third columns, our E2ENet method accurately distinguishes the stomach from the background without over- under-segmennatiio, which be difficult due to the low contrast in the image. the second column, E2ENet performs well in differentiating the stomach from the spleen. These examples suggest that our DSFF module can effectively encode feature information for improved performance in medical image segmentation.
of the challenges of medical image
Ground
USG
MC
Pan
RAG
LAG
Duc
Piclth
NCC
Velno
Pan
RWG
UAG
(4)
(b)
Figure 9: Qualitative comparison the proposed E2ENet and nnUNet on AMOS CT and BTCV challenges.
BraTS Challenge in MSD Figure 10 presents qualitative comparison our proposed E2F.ett method with the unUN the BraTS challenge with highly variable shapes of the segmentation
21

---

CICNN
Greand Fouds
Figure 10: Qualitative comparison of the proposed E2ENet and nnUNet on BraTS Challenge in MSD.
targets. Based on the results of the baseline model, unUNet, we observed that accurately distinguishing the edema (ED) from the background is difficult, as the edema tends to have less smooth boundaries. Our results suggest that E2ENet may have some potential improve the distinguishability of the edema boundaries,
evidenced by the relatively better segmentation results in the first, second, and fourth columns. Moreover, E2ËNet accurately differentiates the enhanced tumor (ET) from the edema, as shown in the third column, which is challenging task due to the similarity in appearance between these two regions, and the dispersive distribution of ET. These findings suggest that E2ENet is a promising method for accurately segmenting brain tumors in challenging scenarios.
A.10 Convergence Analysis
In this section, we analyze the convergence behavior E2ENet by examining the loss changes during topology updating (kernel activation//deactivttooo epochs), comparing it with the best pest-performing baseline nneNet, and studying the impact of topology update frequency. From Figure 11 we observed that the aetivation/deacciiation of weights initially led to an increase in training loss. However, over the long term, the training converged. Additionally, we compared the learning curve of E2ENet with that of Unet and found that E2ËNet converged even faster than nnUNet, as shown in the subplot in (a). To account for the effect of the number of parameters, we scaled down nnUNet have a similar number of parameters as E2ENet and observed that it converged even more slowly than the original UNet. We also studied the impact of topology update frequency. As shown in Figure [2 (b), when the topology updating frequency is increased, the convergence speed may decrease slightly,
Figure 12 (a).
but the impact is not significant.
CON
Figure 11: The learning curve E2ENet on AMOS-CT, with green dotted vertical lines indicating the epochs of weight activation and deactivation. The blue line represents the ratio of weight deactivation/reacciittion throughout the training process.
22

---

Train Loss (600)
0.00
Trein Loss
0.5
Train Loss (1800)
Train Loss (1200)
0.25
200
400
600
800
Epech
Valtaatonn Line
0.0
Va dieia Lask shwe)
chem covec
Validetion Loss
Validation Loss (000)
Validation Loss (1200)
Validation Loos (1800)
022
200
400
600
800
033
Epoch
tooch
(a)
(b)
Figure 12: (a) Comparing the learning curve of E2ENet with that of nnUNet and scaled-down nn UNet (referred to nnUNet (-)); (b) Comparing the learning curve of E2ENet with different topology update frequencies.
A.11 Organ Volume Statistics and Class-wise Results Visualization
154
714
meg
Regle]
Figure 13: (a) The organ volume statistics AMOS- CT training dataset. (b) Class-wise Dice of unUNet without postprocessing (visualization of Table (c) Class-wise Dice differences between E2ENet with feature sparsity 0.7 without postprocessing and nnUNet without postprocessing on AMOS-CT validation dataset. The positive value means that E2ENet outperforms nnUNet, and vice versa.)
13
Figure 14: (a) The organ volume statistics of BTCV training dataset. (b) Class-wise Dice of unee (visualization of Table II). Note that AG denotes the average of the right and left adrenal glands (RAG and LAG). (c) Class-wise Dice differences between E2ENet with feature sparsity 0.7 and nnUNet on BTCV test dataset. The positive value means that E2ENet outperforms nnUNet, and vice versa.
In this section, analyzed the relationship between organ volume and segmentation accuracy on the
AMOS-CT BTCV, and BraTS challenges. The results, depicted in Figures [3] 14 and [5] showed
that small organs with relatively low segmentation accuracy. For the AMOS-CT challenge, RAG
23

---

kigh
Figure 15: (a) The organ volume statistics of BraTS training dataset. (b) Class-wise Dice of UneNet (visualization of Table 2 (c) Class-wise Dice differences between E2ENet with feature sparsity 0.7 and nnUNet on 5-fold cross-validation of the training dataset. The positive value means that E2ENet outperforms nnUNet, and vice versa.
(right adrenal gland), LAG (left adrenal gland), Gall (gallbladder), and Eso (esophagus) more challenging to accurately segment. This may be due to the fact that smaller organ volumes provide less visual information for the segmentation algorithm work with. However, our proposed method, E2ENet, also demonstrated comparable (or better) performance on these small organs, particularly for the organ "LAG", in which the Dice improved from 81.7% to 82.4%. On the BTCV challenge, the Dice of "Gall", which is considered to be the most challenging organ, improves from 75.3% to 78.1% when using E2ENet compared to UNet. For the BraTs challenge, E2ENet demonstrates the most significant improvement in the Dice score of the "ET" region, which is considered the most challenging class, with an increase of 0.7%.
These results indicate that by applying the DSFF mechanism, E2ENet is able to effectively utilize multi-scale information, potentially leading improved performance segmenting small organs. It is important to note that other factors, such as the quality and resolution of the medical images, well as the complexity of the anatomy being imaged, may also impact the performance of the segmentation algorithms. Future work could focus on further exploring the potential impact of these factors on segmentation accuracy.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have claimed that our proposed Efficient Efficient Network (E2ENet) in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The efficiency in this paper is theoretical, we still need future work for hard work support, which we claim in conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [No]
Justification: We don't include theoretical results for proof
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide implementation details in A.5 and the code and dataset links in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The dataset we use is accessible online, and the code provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
26

---

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide implementation details in A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Section A.8.3 plot critical distance diagram using the Nemeny post-hoc test with p-value of 0.05 to establish the statistical significance of our modules.
Guidelines:
The answer NA
that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the type of hardware used for the implementation in Appendix
A.5
Guidelines:
27

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We include border impact in Section A.1
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
28

---

Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The Licenses names for existing assets are CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
29

---

Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "ZIpg7Attb0", "reviewer": "Reviewer_xwUb", "summary": "In this paper, the authors propose a novel architecture that addresses the challenges observed in increasing the model size and computational complexity of neural network architectures. This leads to concerns in the deployment stage, mainly because of resource-limited hardware. The authors propose a 3D medical image segmentation model named Efficient to Efficient Network (E2ENet). They incorporated two designs to make the model efficient while preserving accuracy: Dynamic sparse feature fusion (DSFF) mechanism and Restricted depth-shift in 3D convolution.  Extensive experiments on three benchmarks show that E2ENet consistently achieves a superior trade-off between accuracy and efficiency compared to prior state-of-the-art baselines.", "questions": "Can the proposed method accelerate training speed in terms of time to achieve SOTA accuracies? \nCould the authors provide information on the inference times?", "limitations": "The authors didn't discuss the limitations of the proposed methodology.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized and well-written. \nThe paper reads well.\nThe motivation behind the study is clear.\nThe proposed restricted depth shift method is interesting and somewhat novel.", "weaknesses": "Tables 3 and 4 don’t provide consistent results. It seems that the combination that works for CT does not optimally work for MRIs.\nMissing discussion on limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer xwUb\n\nWe sincerely thank you for your time and effort in reviewing our paper. We are glad that you find our work interesting and novel. We address your comments below.\n\n> Tables 3 and 4 don’t provide consistent results. It seems that the combination that works for CT does not optimally work for MRIs. \n\nFor the mDice score, in Table 3, the shift operation in E2ENet with and without DSFF combination scores 90.1 and 90.2, respectively. In Table 4, with and without DSFF scores 81.2 and 81.0, respectively. These results rank among the top in these tables and show marginal differences in performance.\n\nHowever, with the DSFF combination, while it does not significantly improve the mDice scores, the computational cost in terms of parameters is notably reduced from 23.9M to 11.2M, a **2-fold reduction** for both Table 3 and Table 4.\n\n- Performance: The mDice scores show marginal differences, indicating that the shift operation combined with DSFF **maintains comparable performance** to the shift operation without DSFF.\n- Efficiency: The incorporation of DSFF into E2ENet results in a substantial **reduction in the number of parameters and FLOPs**, achieving a significant computational cost reduction.\n- Conclusion: As discussed in Section 3.2, our main conclusion is that dynamic sparse feature fusion significantly reduces computational resources while maintaining high performance without any significant degradation.\n\nFurthermore, in Appendix B.7.3, we provide additional verification on the **statistical significance of the designed modules: DSFF and depth-shift operation**.\n\n\n> Can the proposed method accelerate training speed in terms of time to achieve SOTA accuracies? Could the authors provide information on the inference times?\n\nThank you for your enthusiasm for our paper and for sharing your concerns with us.\nGiven the relatively restricted support for sparse operations in current off-the-shelf commodity GPUs and TPUs without sparsity-aware accelerators, we did not attempt to achieve practical speedup during training. Instead, we chose to implement our models with binary masks in our work. As mentioned extensively in the conclusion and appendix, the promising benefits of dynamic sparsity presented in this study have not yet translated into actual speedup. Accelerating training time will be a focus for our next work. \n\nAlthough not the focus of our current work, it would be interesting for future work to examine the speedup results of sparse operation during training, using such specialized hardware accelerators, as we see much improvement room of promise here. For example, at high unstructured sparsity levels, XNNPACK (https://github.com/google/XNNPACK) has already shown significant speedups over dense baselines on smartphone processors.\n\n\n**For the inference time:**\n\nAlthough the support for unstructured sparsity on GPUs remains relatively limited, its practical relevance has been widely demonstrated on non-GPU hardware, such as CPUs and customized accelerators. For instance, FPGA accelerators for an unstructured sparse RNN achieved high acceleration and energy efficiency compared to commercial CPUs and GPUs by maximizing the use of embedded multiply resources on the FPGA. Another notable success is DeepSparse (https://github.com/neuralmagic/deepsparse), which successfully deploys large-scale BERT-level sparse models on modern Intel CPUs, achieving a 10× model size compression with less than 1% accuracy drop, a 10× CPU-inference speedup with less than 2% accuracy drop, and a 29× CPU-inference speedup with less than 7.5% accuracy drop.\n\nInspired by these advancements, we adopted an approach based on DeepSparse. We conducted experiments with patches of images sized 32×32×32 as input, comparing the CPU wall-clock timings for online inference between our proposed E2ENet and nnUNet on an Intel Xeon Platinum 8360Y CPU with 18 cores. We acknowledge that, while **our proposed models with sparsity do achieve speedups in practical inference**, they are not as pronounced as those observed with BERT-level sparse models. This is primarily due to the nature of segmentation and 3D convolution operations. However, this presents a promising avenue for our future work.\n\n\n| Methods | nnUNet | E2ENet(s=0.8) | E2ENet(s=0.9)|\n| -------- | -------- | -------- | --- |\n| Latency(ms)     | 10.07     | 8.02     |  **7.28**  |\n| Throughput(items/sec)     | 99.19     | 124.60     |  **137.13**   |\n| speedup     | 1.0x | 1.26x | **1.38x** |\n\n\n> Missing discussion on limitations.\n\nWe appreciate the reviewer pointing this out. I will make it more clear in the final version. Our proposed model leverages unstructured dynamic sparsity; however, due to the relatively restricted support for unstructured sparsity in current off-the-shelf commodity GPUs, we did not attempt to achieve practical speedup during training on GPU. This limitation presents a potential direction for our future work.\n\nWe thank you again for the time and effort you've taken to participate in the review of our paper. If you have further questions, we are more than happy to discuss with you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8u6ZGHrbVS", "reviewer": "Reviewer_JUDh", "summary": "The paper introduces E2ENet, a novel neural network designed for 3D medical image segmentation, which emphasizes efficiency in computational resource usage without compromising accuracy. This paper introduces a Dynamic Sparse Feature Fusion (DSFF) mechanism that adaptively learns to integrate multi-scale features effectively and a novel application of restricted depth-shift in 3D convolution that aligns with the computational simplicity of 2D methods. The model demonstrates superior performance on various benchmarks like AMOS-CT challenge and BraTS Challenge in MSD, showcasing significant reductions in parameters and computational costs while maintaining competitive accuracy.", "questions": "For Section 3.2 Table 3, it seems that for the different shift size, the mDice score does not vary much from each other. What are authors’ insights about this? Based on this, how do authors justify in Section 2.3 that the shift magnitude would have a negative impact on the effectiveness of the shift operation?", "limitations": "Yes, authors have addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(1) DFSS mechanism proposed provides a more efficient feature fusion process while reducing the computational and memory overhead. (2) E2ENet integrates depth-shift strategy in 3D convolution networks, enabling the ability for network to capture 3D spatial relationships. (3) E2ENet significantly reduces parameter size to 7.63 M minimally.", "weaknesses": "Section 3.2 Ablation Studies lacks of more insights about Table 3, detail question will be shown in Question part below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer JUDh\n\nThank you very much for taking the time to review our paper and for your helpful comments. We provide detailed responses to your constructive feedback below.\n\n> **For Section 3.2 Table 3, it seems that for the different shift size, the mDice score does not vary much from each other. What are authors’ insights about this?**\n\n\nThank you for your enthusiasm for our paper and for sharing your concerns with us. Indeed, we observe a marginal difference in mDice, with a decrease of 0.4 when the shift size magnitude is increased from 1 to 3. However, for mNSD, there is a significant performance drop from 82.3 to 81.6. This is because the mDice score primarily measures the **overlap between the predicted segmentation and the ground truth**, whereas mNSD focuses on the **distance between the surfaces (boundaries)**. This demonstrates that shift size is more impactful on boundary alignment. \n\nFurthermore, in additional experiments where we extended the shift size magnitude to 7, we observed a further decrease in performance by 2.5 for mDice and 4.8 for mNSD, compared to a shift size magnitude of 1. The detailed results are shown below. In summary, the shift operation is beneficial, resulting in a 1.9 improvement in mDice compared to not using the shift operation. However, our experiment results indicate that the magnitude should not be set too large. We will explain more below.\n\n\n| w/ shift | shift size | kernel size  | mDice  |mNSD |\n| ----- | ----------- | --- | --- | --- | \n|   No    |   --  | 1x3x3    |  88.6   |   78.6 |\n|   Yes    |   (−1, 0, 1)  | 1x3x3    |  **90.1**   |   **82.3** |\n|   Yes    |   (−2, 0, 2)  | 1x3x3    |   89.8  | 82.0     |\n|   Yes    |   (−3, 0, 3)  | 1x3x3    |  89.7   |   81.6   |\n|   Yes    |   (−7, 0, 7)  | 1x3x3    |   87.6  |   77.5   |\n\n\n> **How do authors justify in Section 2.3 that the shift magnitude would have a negative impact on the effectiveness of the shift operation?**\n\nIn Section 2.3, we propose the Restricted Depth-Shift operation, which aims to capture depth-wise information while maintaining a 2D computation cost. However, increasing the shift size means considering more depth-wise information **at the expense of channel-wise information**, leading to an insufficient representation of channels, as discussed in Section 3. Additionally, a large shift size causes a **loss of local spatial relationships**, which are crucial for segmentation. This results in a blurring effect that reduces the precision of boundary alignment, particularly affecting metrics like mNSD, which rely heavily on accurate boundary information.\n\nWe would like to thank you again for your time and effort. If you have further questions, we are more than happy to discuss them with you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IpAwBNOpcy", "reviewer": "Reviewer_SYnH", "summary": "This paper introduces E2ENet, a 3D medical image segmentation model designed for efficiency and performance. E2ENet incorporates Dynamic Sparse Feature Fusion (DSFF) to adaptively fuse informative multi-scale features and a Restricted Depth-Shift mechanism in 3D convolution to maintain low model complexity. Extensive experiments demonstrate that E2ENet achieves a superior trade-off between accuracy and efficiency, significantly reducing parameter count and FLOPs compared to previous methods, particularly in large-scale datasets like AMOS-CT.", "questions": "1. Clarification on the Backbone Network:\nQuestion: Can you provide a more detailed description of the backbone network used in E2ENet? Specifically, is it an EfficientNet-based backbone, or is it composed of a different set of CNN layers?\nSuggestion: Consider revising Figure 2 and the corresponding text to clearly define the architecture of the backbone network. Providing explicit details will enhance the readers' understanding and reduce ambiguity.\n2. Kernel Size Generalization:\nQuestion: Why was the 3x3x3 kernel size not included in the ablation study results for the BraTS dataset in Table 4?\nSuggestion: It would be beneficial to include the results for the 3x3x3 kernel size in the ablation study for the BraTS dataset. This would help in understanding the generalizability of the DSFF mechanism across different datasets and ensure that the chosen kernel sizes are suitable for various segmentation tasks.\n3. Comparison with Recent Lightweight Networks:\nQuestion: Have you considered comparing E2ENet with other recent lightweight network structures specifically designed for medical image analysis?\nSuggestion: Including comparisons with recent lightweight networks would strengthen the paper by providing a more comprehensive evaluation of E2ENet's performance. This could involve benchmarking against models that are known for their efficiency and effectiveness in medical image segmentation.", "limitations": "No", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Improved Efficiency: E2ENet significantly reduces the parameter count and FLOPs, making it more computationally efficient and suitable for deployment on resource-limited hardware without compromising on performance.\nInnovative Mechanisms: The introduction of Dynamic Sparse Feature Fusion (DSFF) and Restricted Depth-Shift in 3D convolution effectively balances the need for high accuracy with lower computational complexity, offering a novel approach to 3D medical image segmentation.\nRobust Validation: Extensive experiments on multiple challenging datasets demonstrate E2ENet's consistent performance, ensuring its reliability and applicability in various medical imaging scenarios, especially for exceeding nnUNet.", "weaknesses": "1. Unclear Backbone Network: The backbone network of E2ENet is not clearly defined. Figure 2 labels the left part as an \"efficient backbone,\" but it is ambiguous whether this refers to an EfficientNet-based backbone or simply a group of CNN layers. This lack of clarity can confuse readers and detract from the paper's overall comprehensibility.\n2. Segmentation Performance and Kernel Size: The segmentation performance is primarily evaluated on the BraTS and AMOS datasets. However, the ablation study on AMOS does not adequately address the suitability of the DSFF kernel size for BraTS. Table 4 lacks results for a kernel size of 3x3x3, which raises questions about the generalizability of the chosen kernel sizes across different datasets.\n3. Lack of Comparison with Recent Lightweight Networks: The paper does not include comparisons with recent lightweight network structures specifically designed for medical image analysis. This omission limits the assessment of how E2ENet stands relative to other contemporary, efficient models and reduces the comprehensiveness of the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer SYnH\n\nWe would like to thank the reviewer for your thoughtful and detailed comments. We are glad that you appreciate the efficiency improvements and robust validation presented in our paper. We address your comments below.\n\n> Unclear Backbone Network: The backbone network of E2ENet is not clearly defined. Figure 2 labels the left part as an \"efficient backbone,\" but it is ambiguous whether this refers to an EfficientNet-based backbone or simply a group of CNN layers. This lack of clarity can confuse readers and detract from the paper's overall comprehensibility.\n\nThank you for pointing out the description of the backbone network, which indeed helps to enhance the readers' understanding and reduce ambiguity. We will provide this information below and add it to our paper.\n\nThe efficient backbone network consists of several levels as in Figure 2, each comprising two consecutive blocks. Each block includes our proposed efficient Restricted Depth-Shift 3D Convolutional layer, followed by instance normalization and ReLU activation (termed **conv–norm–relu**). After each level, the downsampling is performed using a strided convolution operation in the second block of that level (the convolution in the second block of the new resolution has a stride >1). \nTable: Backbone Network – Each row describes level $i$, with input resolution $H^i$, $W^i$, $D^i$, and output channels $C^i$, given an input size of 128×128×128.\n\n| level $i$ | Operator | Resolution $H^i$, $W^i$, $D^i$  |  Channels $C^i$   |\n| -------- | -------- | -------- | --- |\n| 1     | conv–norm–relu + conv–norm–relu      | 128x128x128     |  48  |\n| 2     | conv–norm–relu + conv–norm–relu      | 64x64x64     |  96  |\n| 3     | conv–norm–relu + conv–norm–relu      | 32x32x32     |  192  |\n| 4     | conv–norm–relu + conv–norm–relu      | 16x16x16     |  320  |\n| 5     | conv–norm–relu + conv–norm–relu      | 8x8x8     |  320  |\n\n> Kernel Size Generalization: Question: Why was the 3x3x3 kernel size not included in the ablation study results for the BraTS dataset in Table 4? Suggestion: It would be beneficial to include the results for the 3x3x3 kernel size in the ablation study for the BraTS dataset.\n\nThank you for your valuable suggestion. Indeed, a 3x3x3 kernel size is helpful in the ablation study for the BraTS dataset as well. This will aid in understanding the generalizability of the DSFF mechanism across different datasets. Here, we update these results as follows and include them in Table 4 of the paper:\n\n| w/ DSFF | shift | kernel size | ED  | ET  | NET | mDice | Params | FLOPs |\n| ------- | ----- | ----------- | --- | --- | --- | ----- | ------ | ----- |\n|   No |    No   |      3x3x3   |  80.9   | 61.9    |   79.1  |   74.0    |   52.55     | 4519.26| \n|   Yes    |   No    |    3x3x3 | 81.0    | 62.2    |  79.1  |  74.1     |   28.02     | 2023.52 | \n|   No |    Yes   |      1x3x3   |  81.0    |  62.3   |  79.0   |   74.1    | 23.89 |  3071.78| \n|   Yes    |   Yes    |  1x3x3 |   **81.2**  |  **62.7**   |  **79.5**    |   **74.5**     | 11.24| 1067.06|\n\nWe find that, compared to kernel sizes of 3x3x3, E2ENet with kernel sizes of 1x3x3 combined with a depth shift (3rd and 4th row) maintains or even improves segmentation accuracy, whether with DSFF or without DSFF. This further demonstrates that our **proposed efficient Restricted Depth-Shift 3D Convolutional layer, which utilizes a 1x3x3 kernel with restricted depth shift, is equivalent to a 3x3x3 kernel in terms of segmentation accuracy.** Moreover, it offers significant **savings in computational and memory resources**, as observed in the AMOS-CT dataset in Table 3.\n\n> Comparison with Recent Lightweight Networks: Question: Have you considered comparing E2ENet with other recent lightweight network structures specifically designed for medical image analysis? Suggestion: Including comparisons with recent lightweight networks would strengthen the paper by providing a more comprehensive evaluation of E2ENet's performance. \n\nThank you for your suggestion. We agree that comparisons with recent lightweight networks would strengthen the paper. We have included another recent efficient lightweight network, UNETR++ [1], as our baseline. UNETR++ offers both high-quality segmentation accuracy and efficiency. From the results below, we can see that E2ENet achieves better mDice scores. This further verifies the efficiency and accuracy of our proposed model for 3D medical segmentation.\n\nMoreover, we conducted an extended ablation study by integrating our proposed DSFF into UNETR++[1], referred to as **UNETR++ w/ DSFF** in the table below. Specifically, we introduced dynamic sparsity into UNETR++ by sparsifying its connections: **less important activated connections are removed, while the same number of deactivated connections are randomly reactivated during training**. The results show that parameters can be further reduced while maintaining stable mDice performance. This further demonstrates the effectiveness of our proposed DSFF, showing that it can be **easily integrated into recent lightweight networks to potentially enhance efficiency while maintaining mDice performance.**\n\n| model | ED  | ET  | NET | mDice | Params\n----------- | --- | --- | --- | ----- | ------ |\n|  UNETR++ |   80.2   |  61.0  |   78.7  |  73.3   |  33.8 | \n| UNETR++ w/ DSFF | 80.4    | 61.4  |  78.4   | 73.4 | 5.11 | \n|  E2ENet   |   81.2  |  62.7   |  79.5    |   74.5  | 11.24|\n\n[1] Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang and Fahad Shahbaz Khan. UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentation. IEEE Transactions on Medical Imaging, 2024.\n\nWe thank you again for the time and effort you've taken to participate in the review of our paper. If you have further questions and concerns, we are more than happy to discuss with you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## To All Reviewers:

We thank the reviewers for their constructive suggestions and in-depth analysis, which is helpful for our work. We are humbled by such a positive response, and we truly appreciate it.

We are delighted to note that all reviewers recognized the novelty of our research, found the idea **well-motivated**, **interesting**, and **novel** (Reviewers xwUb & SYnH), appreciated our contributions towards **improving efficiency** (Reviewers JUDh & SYnH), and acknowledged the **robust validation** (Reviewer SYnH). Additionally, the reviewer xwUb found our work **well-organized** and **well-written**.

In response to the reviewers' valuable feedbacks, we have made earnest efforts to address all raised concerns. A summary of our responses is as follows:

- Explained the results in Tables 3 and 4. (Reviewers xwUb)
- Reported the inference time. (Reviewers xwUb)
- Provided more insight for Table 3. (Reviewers JUDh)
- Explained our backbone (Reviewer SYnH)
- Added the results for kernel size of 3x3x3 on the BraTS dataset (Reviewer SYnH)
- Compared with recent lightweight networks (Reviewer SYnH)

Should there be any points that remain unclear or require further clarification, please do not hesitate to bring them to our attention. We are open to any additional feedback, comments, or suggestions, and we sincerely appreciate your continued engagement in enhancing the quality of our work.

Meta-review: This paper receives all positive reviews. Reviewers recognize the novelty of the paper and enjoy the efficiency of the proposed method. The AC suggests acceptance and encourages the authors to include the discussion in the rebuttal into the final version.
Decision: Accept (poster)