E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation

Authors: boqian  wu
qiao  xiao
shiwei  liu
lu  yin
mykola  pechenizkiy
decebal constantin mocanu
maurice  van keulen
elena  mocanu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 00:23:19

Abstract: Deep neural networks have evolved as the leading approach in 3D medical image segmentation due to their outstanding performance. However, the ever-increasing model size and computational cost of deep neural networks have become the primary barriers to deploying them on real-world, resource-limited hardware. To achieve both segmentation accuracy and efficiency, we propose a 3D medical image segmentation model called Efficient to Efficient Network (E2ENet), which incorporates two parametrically and computationally efficient designs. i. Dynamic sparse feature fusion (DSFF) mechanism: it adaptively learns to fuse informative multi-scale features while reducing redundancy. ii. Restricted depth-shift in 3D convolution: it leverages the 3D spatial information while keeping the model and computational complexity as 2D-based methods. We conduct extensive experiments on AMOS, Brain Tumor Segmentation and BTCV Challenge, demonstrating that E2ENet consistently achieves a superior trade-off between accuracy and efficiency than prior arts across various resource constraints. %In particular, with a single model and single scale, E2ENet achieves comparable accuracy on the large-scale challenge AMOS-CT, while saving over 69% parameter count and 27% FLOPs in the inference phase, compared with the previous
best-performing method. Our code has been made available at: https://github.com/boqian333/E2ENet-Medical.

Main paper: 
 introduction: 
 3D medical image segmentation plays an essential role in numerous clinical applications, including computer-aided diagnosis [Yu et al., 2020] and image-guided surgery systems [Ronneberger et al., 2015] . Over the past decade, the rapid development of deep neural networks has achieved tremendous breakthroughs and significantly boosted the progress in this area [Zhou et al., 2018 , Huang et al., 2020 , Isensee et al., 2021] . However, the model sizes and computational costs are also exploding, which deter their deployment in many real-world applications, especially for 3D models where resource consumption scales cubically [Hu et al., 2021, Valanarasu and Patel, 2022] . This naturally raises a research question: Can we design a 3D medical image segmentation method that trades off accuracy and efficiency better, subjected to different resource availability?Accurately segmenting organs is a challenging task in 3D medical image segmentation due to the variability in size and shape even among the same type of organ, caused by factors such as patient anatomy and disease stage. This diversity amplifies the difficulty to accurately identify and distinguish Figure 1 : A comparison of feature fusion schemes. The purple nodes depict features extracted from the backbone, while the green nodes depict the fused features. In particular, in DiNTS (d), red lines indicate information flow paths determined through neural architecture search techniques. In E2ENet (e), the red lines with different widths represent sparse information flows determined by the DSFF mechanism, allowing for efficient feature fusion. E2ENet is capable of dynamically learning how many of the features to fuse are derived from the backbone.the boundaries of different organs, leading to potential errors in segmentation. One of the main solutions for accurate medical image segmentation is to favorably leverage the multi-scale features extracted by the backbone network, but this remains a long-standing problem. Pioneering work UNet [Ronneberger et al., 2015] utilizes skip connection to propagate detailed information to high-level features. More recently, UNet++ [Zhou et al., 2018] , CoTr [Xie et al., 2021] , and DiNTS [He et al., 2021] have introduced more complex neural network architectures (e.g., dense skip connection and attention mechanism) and optimization techniques (e.g., neural architecture search (NAS) [Elsken et al., 2019] ) for cross-scale feature fusion. NAS-based methods search for network topology and operators (e.g., 3x3 Convolution and Maxpool) and subsequently optimize model weights. However, these methods typically require substantial computational resources to explore network topologies and evaluate numerous candidate architectures, making them time-consuming. For instance, C2FNAS [Yu et al., 2020] requires nearly one GPU-year to discover a single 3D segmentation architecture, while DiNTS [He et al., 2021] improves search efficiency but still needs 5.8 GPU days to find a single architecture. In contrast to NAS approaches, our method does not require costly architecture search time and instead directly optimizes and searches for sparse topologies within the predefined architecture.In this paper, we propose the Efficient to Efficient Network (E2ENet), a model that efficiently incorporates both bottom-up and top-down features from the backbone network in a dynamically sparse pattern, achieving an improved accuracy-efficiency trade-off. As shown in Figure 1 (e), E2ENet incorporates multi-scale features from the backbone into the final output by gradually fusing adjacent features, allowing the network to fully utilize information across various scales. To prevent unnecessary information aggregation, we propose a dynamic sparse feature fusion (DSFF) mechanism and embed it in each fusion node. The DSFF mechanism adaptively integrates informative multi-scale features and filters out unnecessary ones during the course of the training process, significantly reducing the computational overhead without sacrificing performance. Additionally, to further improve efficiency, our E2ENet employs a restricted depth-shift strategy within 3D convolutions, derived from the temporal shift [Lin et al., 2019] and 3D-shift [Fan et al., 2020] strategies used in efficient video action recognition. This allows the 3D convolution operation with a kernel size of (1, 3, 3) to capture 3D spatial relationships while maintaining the parameter complexity of 2D convolutions and reducing computational cost. To evaluate the performance of our proposed E2ENet, we conducted extensive experiments on the AMOS-CT [Ji et al., 2022] , Brain Tumor Segmentation in the Medical Segmentation Decathlon (MSD) [Antonelli et al., 2022] , and Multi-Atlas Labeling Beyond the Cranial Vault (BTCV) [Landman et al., 2015] challenges. We found that E2ENet can effectively trade-off between segmentation accuracy and efficiency compared to both convolution-based and transformer-based architectures. In particular, on the AMOS-CT challenge, E2ENet achieves competitive accuracy with a mDice of 90.0%, while being 69% smaller and using 27% fewer FLOPs during inference. With the DSFF mechanism filtering out 90% of feature connections, E2ENet further reduces resource costs without significantly sacrificing accuracy.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We have claimed that our proposed Efficient to Efficient Network (E2ENet) in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
methodology: 
 In this section, we first present the overall architecture of E2ENet, which allows for the fusion of multi-scale features from three directions. Next, we describe the proposed DSFF mechanism, which adaptively selects informative features and filters out unnecessary ones during training. Lastly, to further increase efficiency, we introduce the use of restricted depth-shift in 3D convolution. Figure 2 : The overall architecture of the proposed E2ENet consists of a CNN backbone that extracts multiple levels of features. These features are then gradually aggregated through several stages, during which the multi-scale features are fused using a fusion operation.Figure 2 provides an overview of the proposed architecture. Given the input 3D image I in ∈ R D×H×W , the CNN backbone extracts feature maps at multiple scales, represented by x 0 = x 0,1 , x 0,2 , . . . , x 0,L , where L is the total number of feature scales. The feature at level i, x 0,i is a tensor with dimensions d i × h i × w i × c i , where d i , h i , w i and c i represent the depth, height, width, and number of channels of the feature maps at level i, respectively. It is worth mentioning that the spatial resolution of the feature maps decreases as the level increases, while the number of channels increases up to a maximum of 320. To fully exploit the hierarchical features extracted by the CNN backbone, these features are aggregated across multiple stages, as depicted in Figure 2 (Dynamic Sparse Feature Fusion section). At stage j (0 < j ≤ L -1), the features at level i are obtained by fusing the adjacent features from the previous stage along three directions:1. "Downward flow" (in yellow): The high-resolution feature x j-1,i-1 , which provides richer visual details, is passed downward; 2. "Upward flow" (in blue): The low-resolution feature x j-1,i+1 , which captures more global context, is passed upward; 3. "Forward flow" (in red): The features x j-1,i , which maintain their spatial resolution, are passed forward for further information integration. The fused cross-scale feature maps at the i-th level of the j-th stage can be formulated as:EQUATIONwhere F j,i (.) is a fusion operation, consisting of a convolution operation followed by Instance Normalization (IN) [Ulyanov et al., 2016] and a Leaky Rectified Linear Unit (LeakyReLU) [Maas et al., 2013] . U(.) and D(.) denote the up-sampling and down-sampling, respectively.[.] denotes the concatenation operation. In the convolution operation, the input feature maps, which have a channel number ofC j,i in (C j,i in = c i-1 + c i + c i+1 or c i + c i+1 ), are fused and processed to produce output feature maps with a channel number of C j,i out (C j,i out = c i ). Unlike the UNet++ model, which only considers bottom-up information flows for image segmentation, our proposed E2ENet architecture incorporates both bottom-up and top-down information flows. This allows E2ENet to make use of both high-level contextual information and fine-grained details in order to produce more accurate segmentation maps (experimental results can be found in Table 2 ).Such multi-stage cross-level feature propagation provides a more comprehensive understanding of the images but can also introduce redundant information, necessitating careful handling in feature fusion to ensure efficient interaction between scales or stages. Our proposed Dynamic Sparse Feature Fusion (DSFF) mechanism addresses the issue of multi-scale feature fusion in an intuitive and effective way. It optimizes the process by enabling selective and adaptive use of features from different levels during training. This results in a more efficient feature fusion process with lower computational and memory overhead.The DSFF mechanism is applied in each fusion operation, allowing the fusion operation F j,i (•) to select the informative feature maps from the input fused features. The feature map selection process is controlled by a binary mask M j,i (•) ∈ {0, 1} C j,i in ×C j,i out , which are trained to filter out S unnecessary feature map connections by zeroing out the corresponding kernels in F j,i (•). With the DSFF mechanism, the output of the fusion operation is then computed as:x j,i cout,:,:,: = σ(I(EQUATIONwhere xj,i c,:,:,: is the input fused feature map at the c-th channel. θ j,i c,cout,:,:,: is the kernel (feature map connection, as in Figure 3 ) that connects the c-th input feature map to c out -th output feature map, * is a convolution operation, • is the product of a scalar (i.e., M j,i c,cout ) and a matrix (i.e., θ j,i c,cout,:,:,: )2 . M j,i c,cout denotes the existence or absence of a connection between the c-th input feature map and the c out -th output feature map. I(.) and σ(.) denote the Instance Normalization and LeakyReLU, respectively. Thus, the core of DSFF mechanism is the learning of binary masks during the training. At initialization, each binary mask is initialized randomly, with the number of non-zero entries ∥M j,i ∥ 0 equals (1 -S) × C j,i in × C j,i out . Here, S (0 < S < 1) is a hyperparameter called the feature sparsity level, which determines the percentage of feature map connections that are inactivated. The activated connections can be updated throughout the course of training, while the feature sparsity level S remains constant. This enables efficiency in both testing and training, as the exploitation of connections remains sparse throughout the training process.Every ∆T training epoch, the activated connections with lower importance are removed, while the same number of deactivated connections are randomly reactivated, as shown in Figure 3 . The importance of activated connection is determined by the L 1 norm of the corresponding kernel. That is, for M j,i , the importance score of connection between c in -th input feature and c out -th output is ∥θ j,i cin,cout,:,:,: ∥ 1 . Our intuition is simple: if a feature map connection is more important (i.e., has a larger effect on output accuracy) than others, the L 1 norm of the corresponding kernel should be larger. However, the importance score is suboptimal during training, as the kernels are gradually optimized. Randomly reactivating previous "obsolete" connections with re-initialization can avoid unrecoverable feature map abandonment and thoroughly explore the representative feature maps that contribute to the final performance.The DSFF mechanism allows for the exploration and exploitation of multi-scaled sparse features through a sparse-to-sparse training method. Our method selects features at multiple levels, as opposed to relying solely on feature selection at the input layer [Sokar et al., 2022 , Atashgahi et al., 2022] . Additionally, it is a plastic approach to fuse features that can adapt to changing conditions during training, as opposed to static methods that rely on one-shot feature selection, as shown in the comparison of dynamic and static sparsity in Table 2 of [Mostafa and Wang, 2019] . More details of the training process are elaborated in the Algorithm 1 in Appendix A.3. In 3D medical image segmentation, the 2D-based methods (such as the 2D nnUNet [Isensee et al., 2021] ), which apply the 2D convolution to each slice of the 3D image, are computationally efficient but can not fully capture the relationships between slices. To overcome this limitation, we take inspiration from the temporal-shift [Lin et al., 2019] and 3D-shift [Fan et al., 2020] in efficient video action recognition, and the axial-shift [Lian et al., 2021] in efficient MLP architecture for vision. Our proposed E2ENet incorporates a depth-shift strategy in 3D convolution operations, which facilitates inter-slice information exchange and captures 3D spatial relationships while retaining the simplicity and computational efficiency of 2D-based method. Temporal-shift [Lin et al., 2019] requires selecting a Shift Proportion (the proportion of channels to conduct temporal shift). Axial-shift [Lian et al., 2021] and 3D-Shift optimize the learnable 3D spatiotemporal shift. We have made refinements to the channel shifting technique by shifting along the depth dimension and incorporating constraints on the shift size. This adaptation is thoughtfully designed to align with the distinctive needs of sparse models employed in medical image segmentation.In our method, we employ a simple depth shift technique by shifting all channels while constraining the shift size to be either +1, 0 or -1, as shown in Figure 4 . This choice is motivated using dynamic sparse feature fusion, where the feature maps contain sparse information. If the shift magnitude is too large, it can result in an insufficient representation of channels or an excessive representation of depth information, which can have a negative impact on the effectiveness of the shift operation (experimental results can be found in Table 3 ). In this section, we compare the performance of our E2ENet model to baseline methods and report results in terms of both segmentation quality and efficiency. In addition, we will perform ablation studies to investigate the behavior of each component in the E2ENet model. To further analyze the performance of our model, we will examine its generalizability and the effect of model capacity, and present qualitative results by visualizing the predicted segmentations on sample images. We will also visualize the feature fusion ratios to gain insights into which features play an important role in the segmentation process. The description of the dataset and experimental setup can be found in Appendix A.4 and A.5.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The efficiency in this paper is theoretical, we still need future work for hard work support, which we claim in conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
comparison with sota methods: 
 AMOS-CT Challenge. To comprehensively validate our method, we compare it to several stateof-the-art CNN-based models (e.g. nnUNet [Isensee et al., 2021] , and Vnet [Milletari et al., 2016] ) and transformer-based models (e.g. CoTr [Xie et al., 2021] , nnFormer [Zhou et al., 2021] , UNETR [Hatamizadeh et al., 2022] , and Swin UNETR [Tang et al., 2022] ). We record the mDice (class-wise Dice can be found in Table 10 of Appendix), Params, inference FLOPs, and PT score3 on the validation set4 of the AMOS-CT challenge in Table 1 . E2ENet with a feature sparsity (s) of 0.8 achieves comparable performance, with a mDice of 90.3%, while being 3× smaller in model size and requiring less computational cost during the inference phase compared to the top-performing lightweight model, nnUNet. As the feature sparsity of E2ENet increases, the number of model parameters and inference FLOPs can be further reduced without significantly compromising segmentation performance. This indicates that there is potential to trade-off performance and efficiency by adjusting the feature sparsity of E2ENet. We also report the results of mean normalized surface dice (mNSD), the official segmentation metric for the AMOS challenge, to provide supplementary information on boundary segmentation quality. The number of parameters and inference FLOPs for DiNTS are not reported because calculating them is time-consuming. This is due to the fact that the architecture for DiNTS is not readily available for the dataset and must be found using Neural Architecture Search (NAS).BraTS Challenge in MSD. E2ENet demonstrates superior performance in terms of mDice compared to other state-of-the-art methods (DiNTS [He et al., 2021] , UNet++ [Zhou et al., 2018] and nnUNet [Isensee et al., 2021] ). Additionally, it is a resourceefficient network that is competitive with the baselines, as evidenced by its small model size and low inference FLOPs. Specifically, E2ENet model a feature sparsity level of 90% has only 7.63 M parameters, which is significantly smaller than other models yet still outperforms them in terms of mDice. The results on the BraTS dataset, which are MRI images, further validate the effectiveness and efficiency of E2ENet across both CT and MRI medical image analysis.Due to the limited space and the similarity between the BTCV and AMOS-CT challenges, both of which involve multi-organ segmentation in CT images, the results of the BTCV challenge are provided in Appendix A.8.2.Table 3 : Ablation study on the effects of the DSFF mechanism and restricted depth-shift in 3D convolution on the validation set of the AMOS-CT Challenge.w/ ∆T w/ shift size kernel size mDice Params FLOPs mNSD In this section, we investigate the impact of two factors on the performance of E2ENet: (i) the DSFF mechanism and (ii) restricted depth-shift in 3D convolution. Specifically, we consider the following scenarios: (#1) w/ DSFF: DSFF mechanism is used to dynamically activate feature map, otherwise, all feature maps are activated during training; (#2) w/ shift: restricted depth/height/width shifting is applied on feature maps before the convolution operation, otherwise, the convolution is performed directly as a standard 3D convolution without shifting operation.EQUATIONEffect of DSFF Mechanism. Table 3 (rows 1 and 3) demonstrates that the E2ENet with the DSFF mechanism, achieved comparable performance while using three times fewer parameters and inference FLOPs. Table 4 shows that removing the DSFF mechanism caused the mDice score to drop from 74.5 % (row 4) to 74.1 % (row 2) on the BraTS challenge, while also increasing the number of parameters by more than 2× and the inference FLOPs by nearly 3×. These ablation study results highlight that dynamic sparse feature fusion (DSFF) helps reduce resource costs while maintaining segmentation performance. To further investigate the impact of the DSFF mechanism, we compared E2ENet with other multi-scale medical image segmentation methods on the AMOS-CT validation dataset, as shown in Table 5 . These methods include DeepLabv3 [Chen et al., 2017] ,CoTr [Xie et al., 2021] , and Med-Former [Gao et al., 2022] . DeepLabv3 uses atrous convolution to capture multi-scale context, CoTr integrates multi-scale features using attention, and MedFormer employs all-to-all attention for comprehensive multi-scale fusion, addressing both semantic and spatial aspects.We observed that with a sparsity ratio of 0.9, E2ENet outperforms DeepLabv3 in terms of mDice while significantly reducing computational and memory costs by more than 3× and nearly 10×, respectively. Additionally, E2ENet with a sparsity ratio of 0.8 matches MedFormer's performance while significantly reducing computational and memory costs by 3× and 4×, respectively. This demonstrates the benefits of multi-scale feature aggregation for medical image segmentation, with E2ENet's DSFF mechanism being much more efficient than other multi-scale methods.Moreover, as shown in Table 2 , E2ENet with the DSFF mechanism outperforms other feature fusion architectures, such as UNet++ and DiNTS, on the BraTS challenge. Overall, the results from the BraTS and AMOS-CT challenges demonstrate that the DSFF mechanism provides an effective and efficient approach for feature fusion.Finally, we also studied the impact of topology update frequency (∆T ) in the DSFF mechanism and observed that our algorithm is not sensitive to the hyperparameter ∆T . Effect of Restricted Depth-Shift in 3D Convolution. We evaluated the effectiveness of our restricted shift strategy by comparing the performance of E2ENet and its variants. Without restricted depth-shift, the mDice decreased from 74.5% (row 4) to 73.9% (row 3) for the BraTS challenge as shown in Table 4 , and decreased from 90.1% (row 4) to 88.2% (row 3) for AMOS-CT (as shown in Table 3 ). In Table 3 , when comparing E2ENet with kernel sizes of 3x3x3 (rows 8 and 9) to E2ENet with kernel sizes of 1x3x3 combined with a depth shift (rows 1 and 3), their segmentation accuracy remains the same, whether DSFF is used or not. In Table 4 , we find that E2ENet with kernel sizes of 1x3x3 combined with a depth shift (rows 2 and 4) even improves segmentation accuracy, whether without DSFF or with DSFF, when compared to kernel sizes of 3x3x3 (rows 5 and 6). This further demonstrates that our proposed efficient Restricted Depth-Shift 3D Convolutional layer, which utilizes a 1x3x3 kernel with restricted depth shift, is equivalent to a 3x3x3 kernel in terms of segmentation accuracy. Moreover, it offers significant savings in computational and memory resources. This demonstrates that the use of a 1x3x3 kernel with restricted depth shift is functionally equivalent to a 3x3x3 kernel in terms of segmentation accuracy, all the while offering savings in computational and memory resources.Rows 9 and 10 of Table 4 demonstrate that when restricted shift is applied to the height or width dimensions, the model's performance decreases compared to E2ENet (4th row) with restricted shift on the depth dimension, called as restricted depth-shift. We also observed that the performance of 3D convolution with kernel sizes of 1 × 3 × 3 (row 3), 3 × 1 × 3 (row 7), and 3 × 3 × 1 (row 8) decreased significantly without restricted shift compared to their counterparts with restricted shift on the corresponding dimensions (rows 4, 9, and 10, respectively). These phenomena demonstrate the effectiveness of restricted shift, especially restricted depth-shift, for medical image segmentation. Furthermore, we evaluated the impact of different shift sizes on the model's performance. As shown in Table 3 , we compared the performance of E2ENet with shift sizes of (-1, 0, 1) to (-2, 0, 2), (-3, 0, 3) and (-7, 0, 7), and observed a decrease in performance from 90.1% to 89.8%, 89.7% and 87.6%, respectively, as the shift size increased. Increasing the shift size means considering more depth-wise information at the expense of channel-wise information, leading to an insufficient representation of channels, as discussed in Section 2.3. Additionally, a large shift size (equivalent to a large kernel size) leads to a loss of local spatial relationships, which are crucial for segmentation. This results in a blurring effect that reduces the precision of boundary alignment, particularly affecting metrics like mNSD, which rely heavily on accurate boundary information. Therefore, we use a shift size of (-1, 0, 1) in our restricted depth-shift strategy as the default setting in our experiments. Finally, we plot a critical distance diagram to show the statistical significance of our modules in Appendix A.8.3. In the AMOS-CT dataset, there is a domain shift between the training and test datasets due to variations in the image acquisition scanners [Ji et al., 2022] . Thus, we also assess the generalizability of our approach by evaluating its performance on the AMOS-CT test dataset. As demonstrated in Table 7 , E2ENet exhibits comparable performance even in the presence of domain shift when compared to nnUNet. This is achieved while maintaining lower computational and memory costs.To ensure a fair comparison, we scale down nnUNet by reducing the number of channels at level 1 from 32 to 27 and decreasing the total number of feature scales from 5 to 4, resulting in a modified version referred to as nnUNet (-). Additionally, we scale up E2ENet by increasing the number of channels at level 1 from 48 to 58, creating a modified version referred to as E2ENet (+). We evaluated the performance of these models, and the results can be found in Table 8 . It is worth noting that scaling down nnUNet resulted in decreased performance in terms of mDice and mNSD, while scaling up E2ENet led to an increase in mDice and comparable performance in terms of mNSD. This indicates that the comparable performance of the memory and computation-efficient E2ENet is not attributed to the dataset's requirement for a small number of parameters and computations. In this section, we compare the proposed E2ENet and nnUNet qualitatively on three challenges. To make the comparison easier, we highlight detailed segmentation results with red dashed boxes.Figure 5 presents a qualitative comparison of our proposed E2ENet with nnUNet on the AMOS-CT challenge. As a widely used baseline, nnUNet has been evaluated on multiple medical image segmentation challenges. Our results demonstrate that, on certain samples, E2ENet can improve segmentation quality and overcome some of the challenges faced on the AMOS-CT challenge. For example, as shown in the first column, E2ENet more accurately distinguishes between the stomach and the esophagus, which is challenging due to their close proximity. In the second column, E2ENet better differentiates the duodenum from the background, while in the third column, E2ENet accurately identifies the precise boundaries of the liver, a structure that is prone to over-segmentation. These examples demonstrate the potential of our proposed method to improve the accuracy of medical image segmentation to some extent, especially in challenging cases such as distinguishing between closely located organs or accurately segmenting complex shapes. The Qualitative Results of BTCV and BraTS Challenge can be found in Appendix A.9.1. In this section, we explore how the DSFF mechanism fuses features from three directions (upward, forward, and downward) during training. To shed light on this mechanism, we specifically analyze the proportions5 of feature map connections from different directions to a specific fused feature node, considering a feature sparsity level of 0.8. After training, we observed in Figure 6 (d) that the DSFF module learned to assign greater importance to features from the "forward" directions for most fused feature nodes. For example, in the first feature level, the flow and processing of features can be observed as if they were passing through a fully convolutional neural network (FCN), which preserves the spatial dimensions of the input image. At the second feature level, the original image is downsampled by a factor of 1/2 and flows through another FCN. Therefore, from this perspective, E2ENet can take advantage of FCN to effectively incorporate and preserve multi-scale information.Simultaneously, the complementary feature flows from the 'upward' and 'downward' directions provide richer information. At the later fusion step of the lower feature levels, the 'upward' information becomes more dominant than the 'downward' information. This prioritization of upward feature flow is similar to the design of the decoders in UNet and UNet++. While E2ENet alleviates the semantic gap more effectively, the proportion of feature map connections in the 'downward' direction always has a certain ratio. This also allows the network to better preserve low-level information and integrate it with high-level information, leading to improved performance in capturing fine details. Interestingly, this trend becomes increasingly apparent during training, as illustrated in Figure 6 (a), (c) and (d). Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [No] Justification: We don't include theoretical results for proof 4. Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide implementation details in A.5, and the code and dataset links in the supplementary materials. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
conclusion: 
 In this work, our aim is to address the challenge of designing a 3D medical image segmentation method that is both accurate and efficient. By proposing a dynamic sparse feature fusion mechanism and incorporating restricted depth-shift in 3D convolution, our E2ENet improves performance on 3D medical image segmentation tasks while significantly reducing computational and memory overhead. The dynamic sparse feature fusion mechanism demonstrates its ability to adaptively learn the importance of each feature map, zeroing out the less important ones. This results in a more efficient feature representation without compromising performance. Additionally, the experiments demonstrate that restricted depth-shift in 3D convolution enables the model to capture spatial information more effectively and efficiently. Extensive experiments on three benchmarks demonstrate that E2ENet consistently achieves a superior trade-off between accuracy and efficiency compared to previous state-of-the-art baselines. While E2ENet provides a promising solution for balancing accuracy and computational cost, future work could explore the potential of a learnable shift offset, which may further improve performance. Additionally, as plug-and-play components, the DSFF mechanism and restricted depth-shift could be interesting to apply to other 3D segmentation models to explore their potential in the future. Furthermore, future advancements in hardware support for sparse neural networks could fully unlock the potential of sparse training methods. A Appendix. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The dataset we use is accessible online, and the code is provided in the supplementary materials. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 6. Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We provide implementation details in A.5. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: In Section A.8.3, we plot a critical distance diagram using the Nemeny post-hoc test with a p-value of 0.05 to establish the statistical significance of our modules. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide the type of hardware used for the implementation in Appendix A.5. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We have reviewed the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We include border impact in Section A.1.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]  
 
 Appendix: In an era dominated by over-parameterized models, designing resource-aware AI models is becoming increasingly important, especially for time-consuming tasks like medical segmentation. Our insights into model efficiency training have the potential to broaden the application of deep neural networks in this area. Overall, this work advances our fundamental understanding of dynamic sparse training and offers future perspectives for scalable and efficient AI models. We do not anticipate any negative societal impacts resulting from this research.Convolutional neural networks (CNNs) have become the dominant architecture for 3D medical image segmentation in recent years (e.g. 3D UNet [Çiçek et al., 2016] , UNet++ [Zhou et al., 2018] , UNet3+ [Huang et al., 2020] , PaNN [Zhou et al., 2019] and nnUNet [Isensee et al., 2021] ), due to their ability to capture local and weight-sharing dependencies [d'Ascoli et al., 2021 , Dai et al., 2021] . However, some recent methods have attempted to incorporate transformer modules into CNNs (e.g. CoTr [Xie et al., 2021] , TransBTS [Wang et al., 2021] ), or use pure transformer architectures (e.g. ConvIt [Karimi et al., 2021] , nnFormer [Zhou et al., 2021] , Swin UNet [Cao et al., 2021] ), in order to capture long-range dependencies. These transformer-based approaches often require large amounts of training data, longer training times, or specialized training techniques, and can also be computationally expensive. Most recently, a novel architecture called Mamba [Gu and Dao, 2023] has shown potential for computational efficiency as a State Space model in handling long sequences and has been applied to medical image segmentation tasks [Ruan and Xiang, 2024 , Xing et al., 2024 , Wang et al., 2024] . However, it has led to underwhelming performance compared to state-of-the-art convolutional models. In this paper, we propose an alternative method for efficiently incorporating 3D contextual information using a restricted depth-shift strategy in 3D convolutions, and further improving performance through adaptive multi-scale feature fusion.Multi-scale feature fusion is a crucial technique in medical image segmentation that allows a model to detect objects across a range of scales, while also recovering spatial information that is lost during pooling [Wang et al., 2022 , Xie et al., 2021] . However, effectively representing and processing multi-scale hierarchy features can be challenging, and simply summing them up without distinction can lead to semantic gaps and degraded performance [Wang et al., 2022 , Tan et al., 2020] . To address this issue, various approaches have been proposed, including adding learnable operations to reduce the gap with residuals [Ibtehaz and Rahman, 2020] , attention blocks [Oktay et al., 2018] . More recently, UNet++ [Zhou et al., 2018] and its variants [Li et al., 2020 , Huang et al., 2020 , Jha et al., 2019] have adapted the gating signal to dense nesting levels, taking into account as many feature levels as possible. NAS-UNet [Weng et al., 2019] tries to automatically search for better feature fusion topology. While these methods have achieved better performance, they can also incur significant computational and information redundancy. Dynamic convolution [Su et al., 2020 , Chen et al., 2020] utilizes coefficient prediction or attention modules to dynamically aggregate convolution kernels, thereby reducing computation costs. In our paper, we propose an intuitive approach to optimizing multi-scale feature fusion, which enables selective leveraging of sparse feature representations from fine-grained to semantic levels through the proposed dynamic sparse feature fusion mechanism.Recently, sparse training techniques have shown the possibility of training an efficient network with sparse connections that match (or even outperform) the performance of dense counterparts with lower computational cost [Mocanu et al., 2018 , Liu et al., 2021b] . Beginning with [Mocanu et al., 2016] , it has been demonstrated that initializing a static sparse network without optimizing its topology during training can also yield comparable performance in certain situations [Lee et al., 2018 , Tanaka et al., 2020 , Wang et al., 2019] . However, Dynamic Sparse Training (DST), also known as sparse training with dynamic sparsity [Mocanu et al., 2018] , offers a different approach by jointly optimizing the Algorithm 1 The Training Process of Dynamic Sparse Feature Fusion (DSFF) Require: Dataset X with label Y; feature sparsity S; backbone f Θ (.); Output Module: f out ; Total training epochs: T ; evolution period: ∆T ; connection updating number: f decay (∆T ; α, T ) = α 2 1 + cos ∆T π T , α represents the number of updated connections during the initial topology update, which is set to 1/2; Loss function: L(.); fusion operation: F j,i (•) with convolution kernels θ j,i , where the numbers of input and output channel are C j,i in , C j,i out . 1: M j,i ← random initialize masks for all levels and stages, satisfying that ∥M j,i ∥ 0 equals (1 -S) × C j,i in × C j,i out 2: for t = 1 to T do3: Sample a batch I t , Y t ∼ X , Y 4:Generate multi-scaled features: x 0,1 , x 0,2 , . . . , x 0,L = f Θ (I t )5:for each stage j = 1 to L -1 do 6:for each level i = 1 to L -j do 7:if i = 1 then 8:x j,i = F j,i ([x j-1,1 , U(x j-1,2 )])9:else 10:x j,i = F j,i ([D(x j-1,i-1 ), x j-1,i , U(x j-1,i+1 )])11:end if 12:end for 13:end for 14:l t = 4/7L(f out x L-1,1 , Y i ) + 2/7L(f out x L-2,2 , D(Y i )) + 1/7L(f out x L-3,3 , D(D(Y i ))) 15: if (t mod ∆T ) == 0 then 16:for each stage j = 1 to L -1 do 17: Training the E2ENet using SGD optimizer 27:for each level i = 1 to L -j do 18: u = (C j,i in × C j,i out )f decay (t; α, T ) (1 -S)end if 28: end for sparse topology and weights during the training process starting from a sparse network [Liu et al., 2021a , 2022 , Evci et al., 2020 , Jayakumar et al., 2020 , Mostafa and Wang, 2019 , Yuan et al., 2021] . This allows the model's sparse connections to gradually evolve in a prune-and-grow scheme, leading to improved performance compared to naively training a static sparse network [Liu et al., 2021c , Xiao et al., 2022] . In contrast to prior methods that aim to find sparse networks that can match the performance of corresponding dense networks, we aim to leverage DST to adaptively fuse multi-scale features in a computationally efficient manner for 3D medical image segmentation.AMOS-CT: The Abdominal Multi-Organ Segmentation Challenge (AMOS) [Ji et al., 2022] task 1 consists of 500 computerized tomography (CT) cases, including 200 scans for training, 100 for validation, and 200 for testing. These cases have been collected from a diverse patient population and include annotations of 15 organs. The scans are from multiple centers, vendors, modalities, phases, and diseases.The Beyond the Cranial Vault (BTCV) abdomen challenge dataset6 consists of 30 CT scan images for training and 20 for testing. These images have been annotated by interpreters under the supervision of radiologists, and include labels for 13 organs.The Brain Tumor Segmentation Challenge in the Medical Segmentation Decathlon (MSD) [Antonelli et al., 2022 , Simpson et al., 2019] consists of 484 MRI images from 19 different institutions. These images contain three different tumor regions of interest (ROIs): edema (ED), non-enhancing tumor (NET) and enhancing tumor (ET). The goal of the challenge is to segment these ROIs in the images accurately.In our work, we utilized the PyTorch toolkit [Paszke et al., 2019] on an NVIDIA A100 GPU for all our experimental evaluations. We also used the nnUNet codebase [Isensee et al., 2021] to pre-process data before training our proposed E2ENet model. For the AMOS dataset, we used the nnUNet codebase as the benchmark implementation.For training, we use the stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.01, which is gradually decreased through a "poly" decay schedule. The optimizer is configured with a momentum of 0.99 and a weight decay of 3 × 10 -5 . The maximum number of training epochs is 1000, with 250 iterations per epoch. For the loss function, we combine both cross-entropy loss and Dice loss as in [Isensee et al., 2021] . To improve performance, various data augmentation techniques such as random rotation, scaling, flipping, adding Gaussian noise, blurring, adjusting brightness and contrast, simulating low resolution, and Gamma transformation are used before training.We employ a 5-fold cross-validation strategy on the training set for all experiments, selecting the final model from each fold and simply averaging their outputs for the final segmentation predictions. In the testing stage, we employ the sliding window strategy, where the window sizes are equal to the size of the training patches. Additionally, post-processing methods outlined in [Isensee et al., 2022] are applied for the AMOS-CT dataset during the testing phase.The backbone generates a total of L = 6 multi-scale feature levels, each with a specified number of channels : [c1, c2, c3, c4, c5, c6] = [48, 96, 192, 320, 320, 320] . At each level of feature generation, there are two convolution layers with a kernel size of (1, 3, 3), followed by instance normalization and the application of leaky ReLU activation. The down-sampling ratios for each level are as follows: ((1, 2, 2), (2, 2, 2), (2, 2, 2), (2, 2, 2), (2, 2, 2), (2, 2, 2), (2, 2, 2)).To assess the quality of the segmentation results, we use the mean Dice similarity coefficient (mDice), which is a widely used metric in medical image segmentation. The mDice is calculated as follows:EQUATION)where N is the number of classes, • is the pointwise multiplication, y j and ŷj represent the ground truth and predicted masks of the j-th class, respectively, which are encoded in one-hot format.2|yj •ŷj | (|yj |+|ŷj |)is the Dice of j-th class, which measures the overlap between the predicted and ground truth segmentation masks for that class.The size of the network can be estimated by summing the number of non-zero parameters (Params), which includes the parameters of activated sparse feature connections (kernels) and parameters of the backbone. The calculation is given by the following equation:P arams = ∥Θ∥ 0 + L-1 j=1 L-j i=1 C j,i in cin=1 C j,i out cout=1 M j,i cin,cout ∥θ j,i cin,cout ∥ 0 . (4)Here, Θ is the parameter from backbone, L is the total number of feature levels, M j,i is a matrix of size C j,i in × C j,i out , and M j,i cin,cout indicates whether the kernel θ j,i cin,cout connecting the c in -th input and c out -th output feature map exist or not. The L 0 norm ∥θ j,i cin,cout ∥ 0 provides the number of non-zero entries of θ j,i cin,cout .Floating point operations (FLOPs) is a commonly used metric to compare the computational cost of a sparse model to that of a dense counterpart [Hoefler et al., 2021] 7 . In our comparison, it is calculated by counting the number of multiplications and additions performed in only one forward pass of the inference process without considering postprocessing. The inference FLOPs are estimated layer by layer and depend on the sparsity level of the network. For each convolution or transposed convolution layer, the inference FLOPs is calculated as follows:EQUATIONwhere K d , K h and K w are the kernel sizes in depth, height and width; S is the feature sparsity level, for layers that are not part of the DSFF mechanism, S = 0 is used; C in and C out are the numbers of input feature and output feature; H, W and D are the height, width and depth of output features. For each fully connected layer, the inference FLOPs is calculated as follows:EQUATIONThe accuracy-efficiency trade-offs could be further analyzed, from comparing resource requirements to describing holistic behaviours (including mDice, Params and inference FLOPs) for the 3D image segmentation methods. To quantify these trade-offs, we introduce the Performance Trade-Off (PT) score, which is defined as follows:EQUATIONwhere α 1 and α 2 are weighting factors, which control the trade-off between accuracy performance and resource requirements, and mDice max , P arams min , and F LOP s min denote the highest mDice score, the smallest number of parameters, and the lowest inference FLOPs among the compared methods for a specific dataset, respectively. The term mDice mDicemax measures the segmentation accuracy, while P aramsmin P arams + F LOP smin F LOP s measures the resource cost.In most cases, we consider both segmentation accuracy and resource cost to be equally important, thus we set α 1 = 1 and α 2 = 1/2 in the following experiments. However, we also explore the impact of different choices of α 1 and α 2 , as detailed in Section A.9. The PT score serves as a valuable metric for evaluating the trade-offs between segmentation accuracy and efficiency.Given the relatively restricted support for sparse operations in current off-the-shelf commodity GPUs and TPUs without sparsity-aware accelerators, we did not attempt to achieve practical speedup during training. Instead, we chose to implement our models with binary masks in our work. The promising benefits of dynamic sparsity presented in this study have not yet translated into actual speedup.Accelerating training time will be a focus for our next work.Although not the focus of our current work, it would be interesting for future work to examine the speedup results of sparse operation during training, using such specialized hardware accelerators. For example, at high unstructured sparsity levels, XNNPACK8 has already shown significant speedups over dense baselines on smartphone processors. Although the support for unstructured sparsity on GPUs remains relatively limited, its practical relevance has been widely demonstrated on non-GPU hardware, such as CPUs and customized accelerators. For example, FPGA accelerators have demonstrated significant acceleration and energy efficiency for unstructured sparse RNNs, outperforming commercial CPUs and GPUs by effectively leveraging the FPGA's embedded resources. Additionally, DeepSparse9 has shown impressive results in deploying large-scale, BERT-level sparse models on modern Intel CPUs. This approach achieved a 10× compression in model size with under a 1% accuracy loss, a 10× increase in CPU inference speed with less than a 2% accuracy reduction, and an impressive 29× CPU inference speedup with a maximum accuracy drop of 7.5% [Liu and Wang, 2023] .Inspired by these advancements, we adopted an approach based on DeepSparse. As shown in Table 9 , we conducted experiments with patches of images sized 32×32×32 as input, comparing the CPU wall-clock timings for online inference between our proposed E2ENet and nnUNet on an Intel Xeon Platinum 8360Y CPU with 18 cores. We acknowledge that, while our proposed models with sparsity do achieve speedups in practical inference, they are not as pronounced as those observed with BERT-level sparse models. This is primarily due to the nature of segmentation and 3D convolution operations. However, this presents a promising avenue for our future work.A.8.1 Class-wise Dice of AMOS-CT We compare the performance of our E2ENet model to several baselines (CoTr [Xie et al., 2021] , RandomPatch [Tang et al., 2021] , PaNN [Zhou et al., 2019] , UNETR [Hatamizadeh et al., 2022] , and nnUNet [Isensee et al., 2021] ) on the test set of BTCV challenge, and report class-wise Dice, mDice, Params and inference FLOPs on the test set in Table 11 . It is worth noting that nnUNet is a strong performer that uses an automatic model configuration strategy to select and ensemble two best of multiple U-Net models (2D, 3D and 3D cascade) based on cross-validation results. In contrast, E2ENet is designed to be computationally and memory efficient, using a consistent 3D network configuration. Swin UNETR [Tang et al., 2022] is among the best on the leaderboard for this challenge. However, we do not include it in our comparison because it employs self-supervised learning with extra data. This falls outside of our goal of trading off training efficiency and accuracy without using extra data.Our proposed E2ENet, a single 3D architecture without cascade, has achieved comparable performance to nnUNet, with mDice of 88.3%. Additionally, it has a significantly smaller number of parameters, 11.25 M, compared to other methods such as nnUNet (30.76 M), CoTr (41.87 M), and UNETR (92.78 M). To demonstrate the advantages of individual modules, we plot a critical distance diagram using the Nemenyi post-hoc test with a p-value of 0.05 to establish the statistical significance of our modules. In Figure 7 , the top line represents the axis along which the methods' average ranks, and a lower value indicates better performance. Methods joined by thick horizontal black lines are considered not statistically different. From the diagram, we can clearly observe that E2ENet with depth shift significantly outperforms E2ENet without depth shift. Additionally, the incorporation of dynamic sparse feature fusion into E2ENet results in a substantial reduction in both the number of FLOPs (from 23.90M to 11.23M) and parameters (from 3069.55G to 969.32G) while maintaining comparable performance, without any significant performance degradation.A.9 The Impact of Weighting Factors α on PT ScoreIn this section, we investigate the impact of the weighting factors α 1 and α 2 on the Performance Trade-off Score for the AMOS-CT challenge, as defined in Equation 7. These factors are used to balance the trade-off between accuracy and resource cost. A higher value of α 1 prioritizes accuracy, while a lower value emphasizes resource cost. Figure 8 shows that as α 1 decreases, the gaps in the Performance Trade-off Score between E2ENet and other methods become larger, indicating that our method is more advantageous when prioritizing resource cost. However, even when α 1 is set to be 20 times greater than α 2 , which prioritizes accuracy over resource cost, the Performance Trade-off Score of E2ENet remains superior to other baselines. This result indicates that our proposed E2ENet architecture is highly efficient in terms of computational cost and memory usage while achieving excellent segmentation performance on the AMOS-CT challenge.A.9.1 Qualitative Results targets. Based on the results of the baseline model, nnUNet, we observed that accurately distinguishing the edema (ED) from the background is difficult, as the edema tends to have less smooth boundaries.Our results suggest that E2ENet may have some potential to improve the distinguishability of the edema boundaries, as evidenced by the relatively better segmentation results in the first, second, and fourth columns. Moreover, E2ENet accurately differentiates the enhanced tumor (ET) from the edema, as shown in the third column, which is a challenging task due to the similarity in appearance between these two regions, and the dispersive distribution of ET. These findings suggest that E2ENet is a promising method for accurately segmenting brain tumors in challenging scenarios.In this section, we analyze the convergence behavior of E2ENet by examining the loss changes during topology updating (kernel activation/deactivation epochs), comparing it with the best-performing baseline nnUNet, and studying the impact of topology update frequency. From Figure 11 , we observed that the activation/deactivation of weights initially led to an increase in training loss. However, over the long term, the training converged. Additionally, we compared the learning curve of E2ENet with that of nnUNet and found that E2ENet converged even faster than nnUNet, as shown in the subplot in Figure 12 (a). To account for the effect of the number of parameters, we scaled down nnUNet to have a similar number of parameters as E2ENet and observed that it converged even more slowly than the original nnUNet. We also studied the impact of topology update frequency. As shown in Figure 12 (b), when the topology updating frequency is increased, the convergence speed may decrease slightly, but the impact is not significant. (right adrenal gland), LAG (left adrenal gland), Gall (gallbladder), and Eso (esophagus) are more challenging to accurately segment. This may be due to the fact that smaller organ volumes provide less visual information for the segmentation algorithm to work with. However, our proposed method, E2ENet, also demonstrated comparable (or better) performance on these small organs, particularly for the organ "LAG", in which the Dice improved from 81.7% to 82.4%. On the BTCV challenge, the Dice of "Gall", which is considered to be the most challenging organ, improves from 75.3% to 78.1% when using E2ENet compared to nnUNet. For the BraTs challenge, E2ENet demonstrates the most significant improvement in the Dice score of the "ET" region, which is considered the most challenging class, with an increase of 0.7%.These results indicate that by applying the DSFF mechanism, E2ENet is able to effectively utilize multi-scale information, potentially leading to improved performance in segmenting small organs.It is important to note that other factors, such as the quality and resolution of the medical images, as well as the complexity of the anatomy being imaged, may also impact the performance of the segmentation algorithms. Future work could focus on further exploring the potential impact of these factors on segmentation accuracy.The computation cost of the masking operation is negligible. For instance, consider the feature map x j,i c,:,:,: with size D × H × W , the kernel θ j,i c,c out ,:,:,: with size of× 3 × 3, and the number of kernels as Cin × Cout. The computation cost of the masking operation alone is Cin × Cout, whereas the combined cost of the masking operation and convolution operation isD × H × W × 3 × 3 × 3 × Cin × Cout.The explanation of metrics can be found in Appendix A.7.The validation set includes 100 images and was the previous test set for the first stage of the challenge. In the training of E2ENet, we did not use the validation set and treated it as the test set.We calculated these proportions by dividing the number of non-zero connections in a given direction by the total number of non-zero connections within each fused feature.https://www.synapse.org/#!Synapse:syn3193805/wiki/89480This is because current sparse training methods often use masks on dense weights to stimulate sparsity. This is done because most deep learning hardware is optimized for dense matrix operations. As a result, using these prototypes doesn't accurately reflect the true memory and speed benefits of a truly sparse network[Hoefler et al., 2021].https://github.com/google/XNNPACK.https://github.com/neuralmagic/deepsparse. 


Decision: This paper receives all positive reviews. Reviewers recognize the novelty of the paper and enjoy the efficiency of the proposed method. The AC suggests acceptance and encourages the authors to include the discussion in the rebuttal into the final version.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose a novel architecture that addresses the challenges observed in increasing the model size and computational complexity of neural network architectures. This leads to concerns in the deployment stage, mainly because of resource-limited hardware. The authors propose a 3D medical image segmentation model named Efficient to Efficient Network (E2ENet). They incorporated two designs to make the model efficient while preserving accuracy: Dynamic sparse feature fusion (DSFF) mechanism and Restricted depth-shift in 3D convolution.  Extensive experiments on three benchmarks show that E2ENet consistently achieves a superior trade-off between accuracy and efficiency compared to prior state-of-the-art baselines.
strength_weakness: strength_weakness: Tables 3 and 4 don’t provide consistent results. It seems that the combination that works for CT does not optimally work for MRIs.
Missing discussion on limitations.
questions: questions: Can the proposed method accelerate training speed in terms of time to achieve SOTA accuracies? 
Could the authors provide information on the inference times?
limitations: limitations: The authors didn't discuss the limitations of the proposed methodology.
paper_summary: paper_summary: The paper introduces E2ENet, a novel neural network designed for 3D medical image segmentation, which emphasizes efficiency in computational resource usage without compromising accuracy. This paper introduces a Dynamic Sparse Feature Fusion (DSFF) mechanism that adaptively learns to integrate multi-scale features effectively and a novel application of restricted depth-shift in 3D convolution that aligns with the computational simplicity of 2D methods. The model demonstrates superior performance on various benchmarks like AMOS-CT challenge and BraTS Challenge in MSD, showcasing significant reductions in parameters and computational costs while maintaining competitive accuracy.
strength_weakness: strength_weakness: Section 3.2 Ablation Studies lacks of more insights about Table 3, detail question will be shown in Question part below.
questions: questions: For Section 3.2 Table 3, it seems that for the different shift size, the mDice score does not vary much from each other. What are authors’ insights about this? Based on this, how do authors justify in Section 2.3 that the shift magnitude would have a negative impact on the effectiveness of the shift operation?
limitations: limitations: Yes, authors have addressed the limitations.
paper_summary: paper_summary: This paper introduces E2ENet, a 3D medical image segmentation model designed for efficiency and performance. E2ENet incorporates Dynamic Sparse Feature Fusion (DSFF) to adaptively fuse informative multi-scale features and a Restricted Depth-Shift mechanism in 3D convolution to maintain low model complexity. Extensive experiments demonstrate that E2ENet achieves a superior trade-off between accuracy and efficiency, significantly reducing parameter count and FLOPs compared to previous methods, particularly in large-scale datasets like AMOS-CT.
strength_weakness: strength_weakness: 1. Unclear Backbone Network: The backbone network of E2ENet is not clearly defined. Figure 2 labels the left part as an "efficient backbone," but it is ambiguous whether this refers to an EfficientNet-based backbone or simply a group of CNN layers. This lack of clarity can confuse readers and detract from the paper's overall comprehensibility.
2. Segmentation Performance and Kernel Size: The segmentation performance is primarily evaluated on the BraTS and AMOS datasets. However, the ablation study on AMOS does not adequately address the suitability of the DSFF kernel size for BraTS. Table 4 lacks results for a kernel size of 3x3x3, which raises questions about the generalizability of the chosen kernel sizes across different datasets.
3. Lack of Comparison with Recent Lightweight Networks: The paper does not include comparisons with recent lightweight network structures specifically designed for medical image analysis. This omission limits the assessment of how E2ENet stands relative to other contemporary, efficient models and reduces the comprehensiveness of the evaluation.
questions: questions: 1. Clarification on the Backbone Network:
Question: Can you provide a more detailed description of the backbone network used in E2ENet? Specifically, is it an EfficientNet-based backbone, or is it composed of a different set of CNN layers?
Suggestion: Consider revising Figure 2 and the corresponding text to clearly define the architecture of the backbone network. Providing explicit details will enhance the readers' understanding and reduce ambiguity.
2. Kernel Size Generalization:
Question: Why was the 3x3x3 kernel size not included in the ablation study results for the BraTS dataset in Table 4?
Suggestion: It would be beneficial to include the results for the 3x3x3 kernel size in the ablation study for the BraTS dataset. This would help in understanding the generalizability of the DSFF mechanism across different datasets and ensure that the chosen kernel sizes are suitable for various segmentation tasks.
3. Comparison with Recent Lightweight Networks:
Question: Have you considered comparing E2ENet with other recent lightweight network structures specifically designed for medical image analysis?
Suggestion: Including comparisons with recent lightweight networks would strengthen the paper by providing a more comprehensive evaluation of E2ENet's performance. This could involve benchmarking against models that are known for their efficiency and effectiveness in medical image segmentation.
limitations: limitations: No

Comments: title: Thank you for your support!
Response
Author-reviewer discussion
comment: Dear Reviewer JUDh,

We sincerely thank you for your response and positive support!

Best,
Authors
Authors, thank you so much for your detailed response. I will keep my score.
Dear reviewer JUDh, we appreciate your valuable feedback and constructive comments. Since there is only one day left in the rebuttal process, we want to know if our response addresses your concerns.
Dear Reviewer xwUb, we appreciate your constructive comments and feedback! Thank you for your positive support!
Thank you, authors, for the detailed response feedback. After carefully considering your comments, I believe that the initial score accurately reflects the strengths of this work.
Dear Reviewer SYnH,

We sincerely thank you for your constructive comments and support! 

Best,

Authors
Thank you for addressing my concerns, i would raise my score to week accept.
Dear reviewers,

Since the authors provided their responses, please read the responses, respond to them on in the discussion, and discuss points of disagreement if necessary by Aug 13.

Best regards,

AC