Upping the Game: How 2D U-Net Skip Connections Flip 3D Segmentation

Authors: xingru  huang
yihao  guo
jian  huang
tianyun  zhang
hong  he
shaowei  jiang
yaoqi  sun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:10:56

Abstract: In the present study, we introduce an innovative structure for 3D medical image segmentation that effectively integrates 2D U-Net-derived skip connections into the architecture of 3D convolutional neural networks (3D CNNs). Conventional 3D segmentation techniques predominantly depend on isotropic 3D convolutions for the extraction of volumetric features, which frequently engenders inefficiencies due to the varying information density across the three orthogonal axes in medical imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI). This disparity leads to a decline in axial-slice plane feature extraction efficiency, with slice plane features being comparatively underutilized relative to features in the time-axial. To address this issue, we introduce the U-shaped Connection (uC), utilizing simplified 2D U-Net in place of standard skip connections to augment the extraction of the axial-slice plane features while concurrently preserving the volumetric context afforded by 3D convolutions. Based on uC, we further present uC 3DU-Net, an enhanced 3D U-Net backbone that integrates the uC approach to facilitate optimal axial-slice plane feature utilization. Through rigorous experimental validation on five publicly accessible datasets—FLARE2021, OIMHS, FeTA2021, AbdomenCT-1K, and BTCV, the proposed method surpasses contemporary state-of-the-art models. Notably, this performance is achieved while reducing the number of parameters and computational complexity. This investigation underscores the efficacy of incorporating 2D convolutions within the framework of 3D CNNs to overcome the intrinsic limitations of volumetric segmentation, thereby potentially expanding the frontiers of medical image analysis. Our implementation is available at https://github.com/IMOP-lab/U-Shaped-Connection.

Main paper: 
 introduction: 
 3D volumetric data segmentation extensively relies on the utilization of axial symmetrical 3D convolutions to extract features based on a volumetric representation. Imaging modalities such as CT [1, 2, 3, 4] and MRI [5, 6, 7, 8, 9] yield high-precision images along the slice plane and repeat this process along the temporal axis, resulting in non-uniform information density across the three axes. We visualize the difference in information density between the axial and the slice plane in 3D medical imaging in Fig. 1 . The information density variance arising from non-simultaneous imaging in three dimensions of volumetric data engenders distinctiveness between time-axial and slice plane features, rendering them unsuitable for the same processing. Typically, features extracted from the slice plane possess higher information density and are more adept at delineating the finegrained, voxel-level neighboring structures. These localized features are essential in understanding the precise 3D structure of tissues, thereby surpassing the utility of time-axial features. However, traditional 3D convolutions [10, 11, 12] , although excelling in time-axial information extraction, significantly increase computational complexity to accommodate time-axial information, leading to axial-slice plane performance drop-off in 3D CNNs. Thus, the structural constraints of 3D convolutions inherently weaken the rich local features of the slice plane. When confronted with sequences abundant in slice plane features, symmetric 3D convolutions face substantial inefficiencies and performance degradation.To address the axial-slice plane performance drop-off issue, an efficient strategy involves increasing channel depth to enhance feature extraction performance. The depth of the channels, indicative of the high-dimensional feature richness, directly influences network performance. Increasing channel depth can enhance the network's capacity to capture more high-dimensional information and improve feature representation quality. However, the relationship between channel depth and network performance in 3D CNNs is intricate and nonlinear. While initially increasing channel depth improves model performance by introducing more detailed and abstract features, this improvement comes at the expense of ever-increasing computational overhead. Moreover, an excessive number of channel depths can complicate the model's information processing, potentially introducing noise and redundancy, culminating in dimensional explosion and performance decline. The linear relationship between channel depths and computational overhead means that increasing channel depth significantly inflates computational costs, leading to reduced model efficiency under identical hardware constraints. Conversely, 2D convolutions, with their focus on slice-wise information extraction, exhibit higher efficiency and performance peaks, boasting a superior parameter performance ratio. For a kernel size K, a 2D convolution requires only 1 K of the parameters compared to its 3D counterpart, thereby significantly enhancing local feature extraction efficiency within a slice. Consequently, 2D convolutions present a key solution to the low parameter-to-performance ratio of 3D CNN architectures, markedly increasing axial-slice plane feature extraction efficiency while minimizing parameter count and computational load. Given the excellent computational efficiency and parameter-to-performance ratio of 2D convolutions for axial-slice plane feature utilization, and considering the prevalent use of skip connections in 3D medical image segmentation architectures to integrate early layer details, we propose employing 2D convolutions within skip connections to supplement axial-slice plane information. For this purpose, we introduce a plug-and-play U-shaped Connection (uC), leveraging a simplified 2D U-Net to replace skip connections in 3D segmentation architectures for improving the utilization efficiency of axialslice plane features. Using classical 3D U-Net as the backbone, we further propose the uC 3DU-Net, which substitutes the original skip connections with uC, thereby enhancing the network's ability to comprehend slice plane information and efficiently utilize initial encoded layer details. For more effective integration of 3D spatial features from 3D CNNs and 2D slice plane features introduced by uC, we employ a Dual Feature Integration (DFi) module to combine multi-dimensional features. Experimental evaluations demonstrate the incorporation of uC significantly enhances segmentation performance even with reduced channels and surpasses parameter-to-performance ratios of state-ofthe-art networks. Comparative analyses on five publicly available datasets-FLARE2021, OIMHS, FeTA2021, AbdomenCT-1K, and BTCV-corroborate the superiority of proposed uC 3DU-Net over existing state-of-the-art methods, achieving persuasive performance with reduced computational complexity and model parameters. The principal contributions of our work can be summarized as follows:1. We proposed uC, utilizing simplified 2D U-Net to replace traditional skip connections in 3D segmentation backbones, enhancing model capacity to capture axial-slice plane features.2. uC 3DU-Net is further proposed, adopting uC to replace original skip connections, and applying the DFi module to effectively merge 3D sequential spatial features with 2D axial-slice plane features, thus improving feature utilization efficiency.3. We explore the complementary relationship between uC and 3D CNNs, revealing that incorporating 2d convolutions in 3D CNNs can achieve superior performance with fewer parameters, striking perfect balances between efficiency and performance in volumetric segmentation. 4 . Comparative experiments on five public datasets-FLARE2021, OIMHS, FeTA2021, AbdomenCT-1K, and BTCV-demonstrate that uC 3DU-Net surpasses all previous models, achieving SOTA performance with fewer parameters and lower computational cost.2 Related Work 
3d medical image segmentation: 
 The realm of 3D medical image segmentation has experienced substantial advancements, primarily propelled by the progressive evolution of deep learning paradigms and the augmentation of computational prowess [13, 14] . At the key of these advancements lies the continuous enhancement of CNNs [15, 16, 17, 18, 19] and the advent of Vision Transformers (ViTs) [20] , both of which have significantly promoted segmentation methodologies [21, 22, 23, 24, 25] . The 3D U-Net [26] , a foundational model in this field, the subsequent iterations have incorporated attention mechanisms, augmenting feature extraction capabilities by prioritizing important regions while mitigating noise interference [27, 28] . Transformer-based architectures, exemplified by the UNETR framework [29, 30] , leverage multi-head self-attention mechanisms to capture long-range dependencies, thus facilitating more precise and meticulous segmentation. Hybrid models that combine CNNs with Transformers, such as TransUNet [31] and TransBTS [32] , effectively balance local spatial feature extraction with global contextual understanding, culminating in superior segmentation performance [33, 34] . The nnU-Net [35] , an exemplary self-adaptive framework, has demonstrated exceptional performance across a diverse array of medical imaging tasks by autonomously configuring itself to specific datasets.Beyond architectural innovations, a multitude of research endeavors have concentrated on refining 3D medical image segmentation mask techniques. Chen et al. [36] propose a novel method integrating Active Appearance Models (AAM) with live wire (LW) and Graph Cuts (GC) techniques, thereby enhancing the efficacy of 3D medical image segmentation. Zhang et al. [21] develop the 3D Context Residual Network (ConResNet), which employs a context residual module to interlink the segmentation decoder with the context residual decoder, explicitly learning inter-slice contextual information to improve segmentation accuracy. Advanced loss functions and optimization strategies, such as extensive implementations of Dice loss [37] , have been designed to address class imbalance issues, ensuring a more balanced learning process between easy and hard examples. Furthermore, data augmentation and transfer learning techniques have been beneficial in enhancing model robustness and generalization capabilities [38, 39] . These unremitting innovations promise increasingly precise and reliable analysis of 3D medical images, significantly advancing the fields of medical diagnostics and therapeutic planning.Innovations in skip connection structures have further augmented the performance and computational efficiency of CNNs [40, 41, 42] in the domain of medical image segmentation. Skip connections are important in facilitating direct gradient propagation, thereby mitigating the vanishing gradient issue and enabling the training of substantially deeper networks. This method permits unimpeded gradient flow throughout the network, effectively circumventing intermediate layers, minimizing information degradation, and achieving more stable and efficient optimization. Prominently exemplified in the ResNet [43] architecture, skip connections mitigate the vanishing gradient dilemma, thereby fostering the training of deep neural networks. In the U-Net [44] architecture, skip connections are crucial in the transmission of initial detailed spatial information from the encoder to the decoder, thereby preserving high-resolution details indispensable for precise segmentation tasks.Recent endeavors have extensively refined traditional skip connections to augment their functional efficacy. Attention U-Net [45] replaces the normal application of skip connections by incorporating attention mechanisms that dynamically emphasize important features while attenuating irrelevant information. Dense U-Net [46] leverages densely connected convolutional networks (DenseNet) [47] to facilitate feature reuse and improve gradient flow. Interconnecting each layer with every other layer in a feed-forward manner facilitates the assimilation of richer and more diverse feature representations. Skip connections in the Residual Channel Attention Network (RCAN) [48] network mitigate the training difficulty of deep neural networks and enhance feature reuse and information flow efficiency, thereby improving the network's ability to learn high-frequency information. Hybrid Densely Connected UNet (H-DenseUNet) [49] incorporates skip connections within dense blocks, combining the strengths of DenseNet and U-Net. By harnessing advanced feature reuse, optimized gradient flow facilitated by dense connectivity, and efficient multi-scale feature fusion, it produces a robust architecture adept at capturing intricate details and nuanced contextual information.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [No] Justification: We have not identified any apparent deficiencies in the proposed method at this time.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 3.1 Preliminaries: 3D convolutional segmentation networks and skip connections Current 3D medical image segmentation networks primarily utilize architectures based on 3D convolutions, as exemplified by classic models such as 3D U-Net, SegResNet, and the recent 3D UX-Net. These networks adopt an end-to-end encoder-decoder framework with skip connections. The encoder progressively increases channel depth while downsampling spatial feature maps to abstract and consolidate contextual information, thereby reducing parameter complexity and improving computational efficiency. In contrast, the decoder gradually upsamples these encoded features to restore the original resolution, refining segmentation boundaries and overall segmentation accuracy. Skip connections effectively reintroduce fine-grained details from the original images during the upsampling stage, aiding the refinement of segmentation results. Several 3D medical image segmentation networks, such as UNETR, TransBTS, and SwinUNETR, have incorporated multi-head self-attention and advanced skip connections within the 3D Conv-based encoder-decoder framework. These hybrid approaches effectively capture long-range dependencies and facilitate multi-scale feature utilization. Multi-head self-attention ensures global consistency, while 3D convolution operations excel in preserving local spatial details. The combination of self-attention with 3D convolutions ensures the model's ability to retain image details while comprehending and processing global information more effectively. However, relying solely on transformer architectures, such as Vision Transformers, would result in a substantial increase in parameters and computational load. Additionally, without skip connections, it is difficult to achieve satisfactory segmentation details. Hence, 3D convolutions and skip connections remain crucial for achieving optimal segmentation performance in 3D medical image segmentation.Considering the objective of skip connections to capture detailed features of original images, it is essential to note that 3D medical images fundamentally represent a sequence of 2D images. Given the anisotropic nature of medical images, the 2D slice plane shows richer feature information compared to the temporal sequence axis. Although basic skip connection methods like cat and addition can supplement original feature information to some extent, these methods fail to fully utilize the rich axial-slice plane information in 3D medical images. To address the challenges posed by the anisotropy of 3D medical images, the U-shaped Connection(uC) is proposed. It offers a solution to anisotropy in medical imaging. This approach employs a simplified 2D U-Net to implement skip connections, thereby supplementing the rich original 2D slice plane feature information, and enhancing the 2D slice plane feature extraction capabilities of any 3D image segmentation network. The uC combines features extracted by 2D convolution with those extracted by 3D upsampling, offering more efficient feature extraction compared to pure 3D convolution, as detailed structure in Sec3.3. The fundamental structure of uC is based on a 2D U-Net. In comparison to the basic 2D U-Net, uC omits the initial and final conv layers, retaining only the downsampling and upsampling layers to minimize computational load and enhance the extraction efficiency of original slice plane features. Each downsampling layer comprises an average pooling layer and two conv layers, each consisting of 2D conv, Group Normalization (GN), and ReLU. The average pooling layer reduces the feature map size by half, and the subsequent conv layers double the channel number, with the group number of GN set to half the current channel number. Each upsampling layer includes a transposed convolution layer and two conv layers, where the transposed convolution layer restores the feature map to its original size, and the conv layers reduce the channel number back to its half. Based on uC, we proposed uC 3DU-Net, which employs the 3D U-Net as the backbone, incorporating uC with a dual feature integration (DFi) module. The network architecture is illustrated in Fig. 2 .Given the input 3D volumetric image data denoted as x, we divide the encoder of the 3D U-Net into five stages. Stage 1 is the initial Two Conv layer, comprising two 3D convolutions, expanding the input feature channel depth to 24. Stages 2 to 5 are downsampling layers, each comprising a max-pooling operation and two 3D convolutions, doubling the feature channels while halving the feature map dimensions. The outputs of stages 1 to 5 are denoted as x (θ:1) to x (θ:5) . The decoder is also divided into five stages. Stages 5 to 2 utilize transposed convolutions to upsample, effectively doubling the spatial dimensions of the feature maps. Stages 5 to 3 apply DFi to effectively halve the feature channel depth, while Stage 2 retains two convolution layers. Stage 1 is the Final Conv layer, converting the feature channels to the number of categories by a 3D convolution. All 3D convolution operations are subsequently followed by Instance Normalization and a LeakyReLU activation function, with the negative slope set to 0.1.Considering the parameter-efficiency trade-off, we employ the uC in only stages 1 to 3, denoted as uC 1 to uC 3 . The inputs x (θ:1) to x (θ:3) are reshaped into 4D dimensions by stacking slices along the batch dimension using a rearrange operation, resulting in x(θ:1) to x(θ:3) . The structure of uC 1 to uC 3 resembles that of a simplified version of 2D U-Net, with detailed descriptions of the uC architecture provided in Section 3.2. uC 1 , uC 2 , and uC 3 consist of 4, 3, and 2 downsampling and upsampling layers, respectively, to match the dimensions of the input 4D tensor. The outputs of x(θ:1) to x(θ:3) after passing through the uC are x(θ:1) to x(θ:3) , which are reshaped back to 5D dimensions using a rearrange operation, resulting in x(ϕ:1) to x(ϕ:3) .We utilize the dual feature integration (DFi) module to seamlessly integrate the 3D spatial features extracted by the 3D CNN with the rich axial-slice plane features introduced by the uC. Specifically, x (θ:4) and x (θ:5) are fused using the DFi block to obtain x µ , x µ and x(ϕ:3) are fused to obtain x υ , and x υ and x(ϕ:2) are fused to obtain x ξ . Finally, x ξ and x(ϕ:1) are fused using the DFi block and undergo the final convolutional block to obtain the final output y.Dual Feature Integration. In conventional 3D U-Net architectures, skip connections integrate the original feature information from the encoder during the upsampling process through cat, followed by two 3x3 convolutions to reduce the number of channels and consolidate useful information. However, this basic feature fusion approach may lead to inefficiencies when reconciling the disparity between 3D spatial features and 2D slice plane features. Therefore, we devise a streamlined, efficient multi-scale feature fusion module DFi to better integrate the axial-slice plane information introduced by uC. The 2D slice plane features extracted by uC and the 3D spatial features from the 3D CNN encoder are cat to retain maximal original feature information. However, subsequent convolution operations to reduce channel depths are computationally intensive and inefficient. Utilizing addition for feature fusion could result in information loss due to inherent discrepancies between the two types of features. Hence, an approach combining cat, addition, and subsequent convolution layers, with the aim of minimizing parameter count and FLOPs, resulting in the design of the DFi module. For the 2D slice plane features F1 extracted by uC and the 3D spatial features F2 obtained during 3D U-Net upsampling, branch 1 of the DFi module cats F1 and F2, then employs a 1x1 conv to halve the channel count, preserving important features to produce feature map Fc. Branch 2 initially extracts critical information from F1 and F2 using 1x1 convs, followed by addition and sigmoid activation to generate the attention map Fa. The element-wise multiplication of Fc and Fa adjusts and integrates the feature map Fc from branch 1, thereby learning the relative importance of F1 and F2 during the fusion process. The proposed DFi module integrates and simplifies addition, cat, and subsequent convolution layers, achieving a more lightweight and efficient fusion of 2D and 3D spatial features.4 Experiments and resultsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: The paper meticulously details the assumptions and provides comprehensive proofs for all theoretical results, ensuring clarity and correctness. This includes the full set of mathematical assumptions and derivations presented in Section 4 and Appendix, which underpin the proposed uC 3DU-Net architecture's performance improvements.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
datasets: 
 To validate the efficacy of the proposed method, multiple experiments are conducted on five widelyused, publicly accessible datasets: FLARE2021 [50] , OIMHS [51] , FeTA2021 [52] , AbdomenCT-1K [53] , and BTCV [54] . To ensure experimental rigor and fairness, we apply identical data preprocessing and hyperparameter settings across all datasets, uniformly utilizing the L DiceCE for training.FLARE2021 GeForce RTX 4090 GPUs and 128GB of memory. The framework employs Python 3.9, PyTorch 2.0.0, and MONAI 0.9.0 within a Distributed Data-Parallel (DDP) training framework. Evaluation metrics. We utilize IoU/mIoU, Dice, ASSD, HD, HD95, VOE, and AdjRand as evaluation metrics to comprehensively assess segmentation performance. IoU and mIoU measure the overlap accuracy between predicted and ground truth regions, providing robustness across multiple classes.Dice is particularly effective for medical images, especially in small target regions. ASSD calculates average surface distances, while HD and HD95 assess boundary accuracy, with HD95 focusing on the 95th percentile to reduce outliers. VOE examines volumetric overlap and AdjRand evaluates clustering similarity, ensuring a thorough assessment of both overlap and boundary precision.Comparative experiments are conducted on proposed uC 3DU-Net and previous state-of-the-art methods across five diverse, publicly available 3D medical image segmentation datasets: FLARE2021, OIMHS, FeTA2021, AbdomenCT-1K, and BTCV. The results are presented in Table 1 , Table 2 , Table 3 , and Table 13 (provided as supplementary material in the Appendix). The FLARE2021, FeTA2021, and OIMHS datasets are evaluated using five-fold cross-validations. The results from all five folds pass the Wilcoxon signed-rank test with a significance level of α=0.05, indicating statistical equivalence in data distributions. Additionally, the results from the AbdomenCT-1K dataset all pass the one-tailed Wilcoxon signed-rank test with p<0.05, thereby validating the SOTA performance of the uC 3D U-Net. Overall, the result indicates that the proposed uC 3DU-Net achieved SOTA performance with fewer model parameters and reduced computational complexity compared to previous segmentation methods. On the FLARE2021 and FeTA datasets, uC 3DU-Net's parameter count and FLOPs are only 40.9% and 44.7% of the suboptimal model, 3D UX-Net, yet the average Dice scores improve by 0.7% and 0.4%, respectively, with a notable 3.1% increase in the pancreas category. On the OIMHS and AbdomenCT-1K datasets, uC 3DU-Net outperformed previous models by at least 0.6% and 0.99% in Dice scores, respectively, highlighting its superior overall segmentation performance. Additionally, uC 3DU-Net showed significant improvements in boundary performance metrics such as ASSD and HD95 on the OIMHS and AbdomenCT-1K datasets, demonstrating its enhanced ability to identify and correct boundary pixels by focusing on axial-slice plane features. These findings substantiate that the 2D Conv-based uC effectively compensates for the limitations of 3D convolutions in extracting in-plane slice information to enhance model performance. Notably, the performance improvement observed on the anisotropic datasets surpasses that on the isotropic FeTA dataset, further highlighting the efficiency of the proposed uC in enhancing in-slice plane information extraction, especially for anisotropic datasets enriched with axial-slice plane features. Overall, uC significantly improves in-plane feature extraction efficiency while maintaining robust axial spatial feature representation, representing the pinnacle of current performance and parameter efficiency. Image Reconstruction Fidelity Comparison. We evaluate three methods: 3D U-Net, 3D U-Net + 3D uC, and 3D U-Net + 2D uC, on their ability to reconstruct axial-slice plane images from input 3D sequence images, with the results presented in Table 4 . The highest PSNR achieved by 3D U-Net + 2D uC confirms the proposed 2D uC's superior slice-plane feature extraction and reconstruction abilities compared to 3D Conv. Impact of Convolution Dimensionality on uC Performance and Parameters. We conduct experiments on the OIMHS dataset using 3D U-Net, 3D U-Net + 3D uC, and 3D U-Net + 2D uC across varying channel depths to evaluate the impact of 2D and 3D convolutions on uC performance and parameter count. As shown in Table 5 and Fig. 5a (provided as supplementary material in the Appendix along with Fig. 5b ), the 2D uC consistently achieves a superior parameter-to-performance ratio. This performance difference, independent of skip connections, is directly attributed to the differing feature extraction efficiencies of the 2D and 3D convolutions within the uC structure. This result is consistent with Table 4 , further confirming the effectiveness of 2D convolutions for axial-slice plane feature extraction. Moreover, Fig. 5b indicates that the 2D uC converges faster and achieves a higher performance ceiling in the validation set, highlighting its greater performance in axial-slice plane feature extraction. To assess uC's efficacy in overcoming axial-slice plane performance drop-off, we conducted quantitative analysis experiments on the FLARE2021 dataset using numerous state-of-the-art models.As depicted in Table 6 and Fig. 3 , replacing traditional skip connections with uC can significantly enhance axial-slice feature extraction capabilities and thereby improve overall performance. Incorporating uC exhibited improvements in mIoU and Dice scores by 0.79%-1.83% and 0.65%-1.21%, respectively. Integrating uC into high-performing 3D medical image segmentation models such as 3D UX-Net, which initially achieved a 93.43% Dice score, resulted in a 1.0% mIoU and 0.77% Dice score improvement. This demonstrates that even state-of-the-art models encounter inherent axial-slice plane performance drop-off issues, which uC effectively mitigates, thereby enhancing axial-slice plane feature extraction and overall performance. is consistently observed across different layers of uC, highlighting its robust and efficient feature fusion capabilities. The focus of the proposed DFi thus shifts towards optimizing feature fusion efficiency rather than merely achieving performance improvements. Continuous improvement efforts will prioritize refining 2D and 3D feature integration while maintaining computational efficiency.The low parameter performance ratio observed in 3D CNNs is largely due to their reliance on axially symmetric 3D convolutions, which extract spatial features well but struggle to capture critical axialslice plane details. Introducing the uC structure significantly enhances the capability of 3D CNNs to extract axial-slice plane features. Can incorporating uC allow a reduction in channel count, thereby decreasing computational cost? To explore this, we conduct an analytical experiment utilizing 3D UX-Net as the backbone of FLARE2021 and OIMHS datasets to investigate the impact of channel depth and uC integration on segmentation performance. As illustrated in Table 8 , achieving a 0.66% Dice improvement with the 3D UX-Net on FLARE2021 requires a tenfold model increase, highlighting the challenge of enhancing performance with 3D CNNs. The incorporation of uC mitigates the axial-slice plane performance drop-off, resulting in performance enhancements across 3D UX-Net with different channel depths. Notably, a 24-channel depth uC 3D UX-Net surpasses the original 48-channel depth 3D UX-Net on both datasets, with parameter count and FLOPs reduced to only 29% and 34.3%, respectively. This highlights uC's capability to achieve superior performance with fewer parameters, suggesting that substituting traditional skip connections with uC and reducing channel depth is a viable approach to achieve substantial reductions in parameter count and computational load while maintaining or even improving performance. The qualitative results are further illustrated in Fig. 4 . Channel Depth Configuration. We experiment with channel depths of 16, 24, and 32 for the proposed uC 3DU-Net, all surpassing the original 32-channel U-Net, as shown in Table 10 . Performance improved with increased channels, but efficiency dropped drastically at 32 channels compared to 24. Notably, the 16-channel uC 3DU-Net performed better than the 32-channel U-Net with similar parameter counts and FLOPs, demonstrating a 1.74% Dice score improvement and a 4.95 HD95 reduction. This underscores that efficiently capturing slice plane features with uC significantly enhances computational efficiency in 3D CNN architecture. Thus, replacing skip connections with uC and reducing channel depth is a straightforward strategy to achieve substantial parameter and computational load reductions while maintaining or enhancing performance.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Specific experimental setups, dataset descriptions, hyperparameters, and detailed architecture of uC 3DU-Net are provided in the paper. The code is accessible on GitHub, and all five datasets used are publicly available. 
conclusion and future work: 
 In this paper, we propose a U-shaped Connection (uC) for enhancing 3D CNN-based medical image segmentation architecture. This approach is specifically designed to address the inherent axial-slice plane performance drop-off in 3D CNNs, characterized by their inefficiency in extracting high-density axial-slice plane features, which are crucial for accurate 3D medical image segmentation. By replacing traditional skip connections in 3D U-Net with uC, we further develop the uC 3DU-Net, capitalizing on more efficient feature extraction capabilities of both 3D sequential spatial features and 2D axialslice plane features, reaching the best segmentation accuracy and computational efficiency ratio among all previous SoTA methods. Empirical evaluations on diverse datasets demonstrate that uC 3DU-Net consistently outperforms previous SoTA 3D medical segmentation methods, with notably reduced parameters and FLOPs. This underscores the transformative potential of the uC structure in revolutionizing medical volumetric segmentation by breaking through the intrinsic limitations of 3D convolutions. Future research will extend the application of the uC structure to a broader range of volumetric segmentation tasks, with the aim of continually advancing the performance and efficiency of 3D image segmentation models. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]Justification:The paper provides open access to its code on GitHub, along with detailed instructions and descriptions. All data utilized in this study is widely recognized and publicly accessible.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Justification: The paper meticulously delineates the training and testing specifics, including data splits, hyperparameter choices, optimizer type, and the rationale behind these selections.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. Guidelines:• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: The paper outlines rigorous validation on public datasets, ensuring reproducibility and mitigating misuse risks with transparent methodologies and ethical considerations for model release and application.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: This paper thoroughly acknowledges the original creators of all utilized assets, including code, datasets, and models, within the text. Each asset's license and terms of use are meticulously observed, ensuring full compliance with all legal and ethical standards. For further details, refer to the 'Datasets' and 'Implementation Details' sections where attributions are explicitly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The proposed U-shaped Connection (uC) and its integration into the 3DU-Net are comprehensively detailed in the method chapter, including their architecture, implementation, and the datasets used for validation.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper focuses on the technical advancements in medical image segmentation using the uC 3DU-Net architecture and does not involve any crowdsourcing experiments or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This study focuses on the development of a novel 3D medical image segmentation technique. As it does not involve direct engagement with study participants, IRB approval is not required.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Here, we add two recently proposed models as baselines for comparative experiments across four publicly available datasets: FLARE2021, OIMHS, FeTA2021, and AbdomenCT-1K. The experimental results in Table 14 demonstrate that our proposed uC 3D-UNet maintains superior performance compared to the latest models. To validate the effectiveness of the proposed uC in enhancing axial-slice plane feature extraction, we conduct experiments with 3D U-Net + 2D uC by slicing input tensors along different dimensions. As shown in Table 15 , although the 2D slicing augmentation with uC consistently enhances performance across all planes, slicing along the time-axial slice plane dimension yields the best results. This result reflects the denser information available in the axial-slice plane of 3D medical images, which improves performance when using the proposed uC method. Furthermore, these findings support why 2D uC achieves superior parameter efficiency in 3D image segmentation tasks compared to purely 3D convolutions. In this section, we present the details of our experimental results. For comparative experiments, as the baselines for FLARE2021 and FeTA2021 utilize data from [56] , we provide detailed experimental results for all categories in the OIMHS and AbdomenCT-1K datasets (with part of OIMHS comparative data comes from [25] , which are not included here), as shown in Tables 16 and 17 . Tables 18, 19 , 20, 21, and 22 present detailed data for all categories in the analytical experiments, while Tables 23 and 24 display the detailed data for all categories in the ablation studies. The comprehensive results across all categories further demonstrate that integrating the uC module significantly enhances the backbone models' ability to extract 2D plane information from axial slices, while retaining the extensive volumetric feature extraction capabilities of 3D convolutions. This integration effectively addresses the challenge of anisotropic medical imaging, achieving superior performance with reduced computational complexity and fewer parameters compared to previous models. As a result, the proposed approach achieves an optimal parameter-to-performance ratio. In this section, we present additional qualitative results. Fig. 6 , Fig. 7 , Fig. 8 , and Fig. 9 further show the superior segmentation accuracy of the proposed uC 3D U-Net compared to previous models on the FLARE2021, FeTA2021, OIMHS, and AbdomenCT-1K datasets. Fig. 10 visualizes the impact of integrating the uC module with backbone models on the FLARE2021 dataset. These visualizations demonstrate that the integration of uC effectively enhances the model's ability to extract axial-slice plane information while retaining effective volumetric feature extraction, addressing challenges related to anisotropic medical imaging. This integration achieves improved performance with reduced computational cost and fewer parameters, resulting in an optimal parameter-to-performance ratio compared to previous models. Justification: The primary claims focus on the integration of 2D U-Net-derived skip connections into 3D CNN architectures, specifically addressing the inefficiencies in volumetric feature extraction due to varying information densities along different axes, presented as the uC and the enhanced uC 3DU-Net, is rigorously validated through various experiments on multiple datasets, improving performance with reduced computational complexity.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This paper provides a an approach to utilize 2D features via 2D skip connections for 3D CNN U-net segmentation.
The reviewers all appreciate the paper, in particular the simplicity of the idea combined with convincing experiments.
The authors should take care to include their code in the final paper, as well as to utilize the detailed feedback from the reviewers to improve their paper.

Reviews: review: paper_summary: paper_summary: This paper proposes an approach to combine 3D and 2D features for medical image segmentation. The key observation is that medical images often have high in-plane resolution (2D) and lower through-plane resolution (across 2D slices). They introduce a U-shaped connection which uses 2D convolutions in place of standard skip connections in the widely used UNet architecture. They demonstrate segmentation results on four public datasets across a variety of imaging modalities and organs. Their method is computationally more efficient as there are fewer parameters needed by pushing some computations to 2D.
strength_weakness: strength_weakness: The method is a fairly simple architectural tweak that improves performance very slightly, on already well-studied segmentation tasks. Consequently, there should be substantially more experimental evaluation to properly conclude the benefit of this approach over other methods, especially attention-based approaches. 

The main claim of this work is that using an isotropic 3D convolution will ignore or will not accurately learn in-slice features. Additional experiments should have been done to demonstrate this. For example, how would this model perform on a highly nonistropic dataset, or a high-resolution, isotropic one (such as brain MRI)?

Are the table results shown on the test set or the validation set? How was the validation set used?

For the baseline models, I believe additional details should have been provided:

- how was the hyperparameter tuning done? Was there a proper grid search to determine model depth, regularization parameters, etc...? For example, the very poor performance of nnFormer in Table 10 and of TransBTS in Table 11 are concerning.
- The set of data augmentations is quite limited, and may have hurt the models with a larger number of parameters (e.g. transformer-based). Additional nonlinear augmentations should have been performed. 

The ablation studies also reveal negligible improvement when adding the uC to an existing model (Table 5). This is perhaps the most revealing experiment of this work. 

All experiments should have associated error bars. The differences are far too small to make any meaningful conclusions. 

I am also not convinced that the main claim of the paper is proven:
*While 3D convolutions capture sequential spatial feature information, their efficiency in extracting 2D plane information is inferior to that of 2D convolutions applied to individual slice images.* (lines 169-170)
questions: questions: - How are the images reshaped to be run in the 2D UNet model? Is the 3D image tiled to create a 2D image?
- How would network performance change for a high resolution, isotropic dataset? For a dataset with low out-of-plane resolution?
- Why does the nnUNet model have substantially more parameters than your model?
limitations: limitations: Minimal limitations are mentioned.
paper_summary: paper_summary: The study introduces uC 3DU-Net, a 3D medical image segmentation method combining 2D U-Net skip connections with 3D CNNs.
strength_weakness: strength_weakness: **Motivation:** The justification provided for the selection of a 2D UNet over 3D convolutions in Section 3.2 is somewhat insufficient. The authors argue that 3D convolutions are less efficient at extracting 2D plane information; however, this assertion does not consider that using 2D convolutions on 3D medical images can compromise the spatial relationships between slices, thereby limiting the capture of intra-slice features. Could the authors elaborate on their rationale behind this choice? Furthermore, the novelty of the proposed approach appears primarily limited to the incorporation of a 2D UNet as a skip connection. A substantial portion of the paper (approximately four pages) is devoted to discussing previous methodologies, while the method is described only in Section 3.3 and this makes it hard to see the contributions of this work among the extensive discussion of other research. 

Additionally, the reference to 3D convolutions in Line 195 seems inconsistent with the dedicated use of a 2D UNet and requires further clarification. The main figure (Figure 2) also lacks a detailed representation of the blocks used in the 3D UNet, which obscures understanding of the architecture.

**Experiments:** To robustly support the choice of a 2D UNet, a comparative analysis with at least one 3D UNet or 3D convolutional model would be highly informative. This would provide a stronger justification for the choice of model architecture.  Furthermore, testing the models on more challenging datasets, such as the BTCV 13 class dataset, would better demonstrate their efficacy and robustness. There is also a discrepancy between the number of parameters listed for SegResNet in Table 1 and Table 4 that needs to be addressed for clarity and accuracy in the reported results. 

Since the addition of the DFi module results in only minor improvements, the performance gains claimed could be attributed to random variation, given the very small difference in the provided metrics. This minimal change does not convincingly demonstrate the effective integration of 2D and 3D features.

**Minor comments:** 
Fix the AdamW citation in line 246.
questions: questions: Please visit Weaknesses.
limitations: limitations: Please visit Weaknesses.
paper_summary: paper_summary: The manuscript proposes using 2D U-Net connections to enhance 3D U-Net segmentation. The innovative approach demonstrates excellent performance across different imaging modalities and achieves state-of-the-art (SOTA) results with fewer parameters and lower computational costs on four public datasets. The manuscript thoroughly describes the challenges associated with 3D segmentation.
strength_weakness: strength_weakness: 1. The principle behind the proposed DFI is not well-explained.
2. In the ablation studies section, the HD95 metric for +cU(stage 1, 2, 3)+DFI is slightly higher than for +cU(stage 1, 2, 3).
questions: questions: 1. Can authors discuss more about the principle of DFI?
2. Why is the HD95 metric for +cU(stage 1, 2, 3)+DFI slightly higher than for +cU(stage 1, 2, 3)?
limitations: limitations: The impact of DFI on various 3D segmentation backbones is not explored.
paper_summary: paper_summary: This paper proposes a novel element for 3D medical image segmentation, called uC (U-shaped connection) that replaces standard skip connection with 2D convolutions.
strength_weakness: strength_weakness: - Paper is hard to read at times. The language is sometimes complex, using rare words I have never seen used in that context. 
- Section 3.2 is essentially empty: there is no clear description of the U-Shaped connection.
questions: questions: The authors claim their code is available on github. However, I did not find the code, nor a link at https://anonymous.4open.science or some similar one. 

Line 246, the reference to AdamW is missing. 
Line 182, why 5D volumetric data image?
limitations: limitations: The authors have not highlighted any limitations, nor potential negative societal impact.

Comments: title: Official Comment by Reviewer WWao
comment: Thank you for the detailed response and the additional experimental validation. I have raised my score accordingly. Please ensure to add error bars to all tables in the updated paper. Please also conduct tests of statistical significance.
Thank you for providing the clarification and the requested experiments. Given these clarifications, I will revise my score. However, the paper requires major revision in terms of clearly describing its contributions, as much of the manuscript is devoted to discussing related works.
Thank you for your response. However, while your rebuttal has clarified certain aspects of the work, it does not alter my overall assessment of the paper. Therefore, I have decided to maintain my original score.
Thank you for your response. 

I believe you could have added the anonymous GitHub link to the response, for everyone and not just for the AC. 

At this stage, I see no reason to change my score.