Toward Real Ultra Image Segmentation: Leveraging Surrounding Context to Cultivate General Segmentation Model

Authors: sai  wang
yutian  lin
yu  wu
bo  du
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 06:45:42

Abstract: Existing ultra image segmentation methods suffer from two major challenges, namely the scalability issue (i.e. they lack the stability and generality of standard segmentation models, as they are tailored to specific datasets), and the architectural issue (i.e. they are incompatible with real-world ultra image scenes, as they compromise between image size and computing resources).
To tackle these issues, we revisit the classic sliding inference framework, upon which we propose a Surrounding Guided Segmentation  framework  (SGNet) for ultra image segmentation. 
The SGNet leverages a larger area around each image patch to refine the general segmentation results of local patches.
Specifically, we propose a surrounding context integration module to absorb surrounding context information and extract specific features that are beneficial to local patches. Note that, SGNet can be seamlessly integrated to any general segmentation model.
Extensive experiments on five datasets demonstrate that SGNet achieves competitive performance and consistent improvements across a variety of general segmentation models, surpassing the traditional ultra image segmentation methods by a large margin.

Main paper: 
 introduction: 
 With the rapid development of computing and imaging equipment, ultra-high resolution images with millions or even billions of pixels emerge in endlessly and driving the need for more advanced analytical techniques. Accurate understanding of information conveyed by images [1, 47, 43] has become imperative in diverse fields such as remote sensing [29, 48, 44, 50] and medical analysis [37, 36, 31] , and ultra image segmentation has emerged as an essential tool for achieving this goal.To achieve ultra image segmentation, most of the methods adhere to the concept of integrating global and local information, utilizing the global context clues to aid local region refinement [6, 20] . Typically, these networks have two branches: one receives the down-sampled ultra image and extracts global context information of whole image, while the other extracts local information from sliced patches or the complete image, as shown in Fig. 1(a) . The final result is generated by fusing the global and local features. Despite numerous improvements made to ultra image segmentation, we observe that it still suffer from issues in both their scalability and architecture:(1) Scalability Issue. The existing ultra image segmentation methods (UIS) often rely on datasetspecific parameters, which limit their model capacity and scalability. The UIS methods struggle to scale up to larger image sizes, with performance significantly degrading as image resolution increases. Moreover, UIS has complex training procedures that need multi-stage parameter tuning [8, 24] . In contrast, general semantic segmentation (GSS) methods are more effective and demonstrate greater scalability compared to UIS. With a straightforward training process, GSS can be adaptable to various ultra image datasets.(2) Architectural Issue. The architecture of existing UIS is not suitable for processing ultra image scenes in real-world scenarios, as it compromises between dataset size and computing resources. Most methods utilize either the entire image or the downsampled version to capture global information. However, the two strategies are both constrained by the size of input image. When the input image is excessively large, the former strategy cannot process it directly, while the later will suffer great information loss during compression.Building upon the above discussions, a simple solution is to introduce general segmentation model [27, 12, 18, 39] into the ultra image segmentation task using the sliding window approach, which takes only isolated patches as input due to memory limitations. However, directly adapting GSS to ultra image scenes also raises two challenges: (1) Information bottleneck. Using isolated patches as input prevents the model from capturing the correlations between patches, which blocks the information flow and affects the model's perception of the surrounding information. (2) Fragmentation phenomenon. With isolated patch input, even if overlapped patches are used, the prediction results of each patch are independent. This leads to inconsistent prediction between patches, especially at the edges of adjacent patches, as shown in Fig. 2 .Based on above considerations, we propose an end-to-end framework called Surrounding Guided Segmentation Framework (SGNet), which takes advantage of the general segmentation method and utilizes surrounding information near the local patch to guide the model. As shown in Fig. 1 (b), SGNet includes two decoupled parts: a general segmentation module and a surrounding contextguided branch. Specifically, SGNet takes both the local patch and the corresponding surrounding patch as input. The surrounding patch covers a larger area around the local patch and provides more context information. In the surrounding context-guided branch, we model the context information required to segment the local patch from a larger perspective to drive the flow of information between regions of whole image. Besides, to alleviate the fragmentation phenomenon, we propose a boundary consistency loss to improve the consistency of the prediction results of adjacent patches, thereby alleviating the inconsistent predictions. Note that, unlike existing tightly coupled ultra images segmentation methods, our method can be seamlessly incorporated into any general segmentation models, and brings stable performance improvements.Our contributions are summarized as follows:• We excavate two essential but largely overlooked issues in UIS, which hold great value for the community. In addressing these challenges, we are the first to tackle the ultra image segmentation task from the general segmentation model perspective.• We present a novel end-to-end ultra image segmentation framework named SGNet, which leverage surrounding context information to guide patch-based model segmentation. Our method is flexible to be added to any general segmentation model.• Experiments show that our method achieves competitive performance and consistently improves over different general segmentation models on five public datasets, outperforming previous methods by a large margin.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The main claims accurately summarize the our contributions.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
general semantic segmentation: 
 With the development of deep learning [33, 32] , the two mainstream methods based on convolution neural network [52, 35, 34, 2] and Transformer [42, 17, 51, 28] have achieved excellent performance. FCN [27] is the first fully convolutional architecture and a lot of work has been extended, such as DeeplabV3 [3] and HRNet [38] . Compared with CNN-based methods, representative Transformerbased works include SegFormer [46] , Mask2Foremr [7] and SAM [22] . BiSeNetV1 [49] and STDC [12] are designed for real-time segmentation to reduce the computational overhead. The general segmentation model shows excellent stability and scalability, and we aim to leverage its strengths to address both the Scalability and Architectural issues in existing UIS methods.GLNet [6] introduces a novel global-local architecture. PPN [45] builds on the top of GLNet by integrating a classification network to distinguish valuable patches. Furthermore, Magnet [16] employs a multi-stage pipeline where each stage corresponds to a specific magnification level. FCtL [24] exploits locality-aware contextual correlation to effectively integrate and associate contextual information of local patches. Compared to previous methods, ISDNet [14] proposes a novel framework that combines shallow and deep networks, enabling directly whole-image inference, thereby improving the segmentation effect while increasing the speed. ElegantSeg [5] proposes a end-to-end holistic learning framework from the perspective of engineering optimization. Recently, WSDNet [20] follows the architecture of ISDNet, using DWT-IWT to preserve spatial details. GPWFormer [19] also employs the global-local architecture, and propose the wavelet transformer to model semantic relations. However, the aforementioned methods all suffer from inherent framework flaws, and cannot be applied to ultra images of extremely large scale. On the contrary, our simple yet generalized solution is more adaptable and applicable than specific UIS methods in real-world scenarios.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: In Conclusion, we discuss the limitations of the proposed method.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 In this section, we present SGNet, a novel framework that enables existing general segmentation models for ultra image segmentation. As shown in Fig. 3 , SGNet consists of two major components, the general segmentation module and surrounding context-guided branch (SCB). The two branches take the local patch and its surrounding larger area as inputs, respectively and extract their features (Section Architecture). In the surrounding context-guided branch, we introduce a surrounding context integration module (Section Surrounding Context Integration Module) to enable interaction between local and surrounding features, and selectively learn contextual information that is helpful for patch segmentation. Furthermore, we propose a boundary consistency loss to maintain the consistency of prediction results across adjacent patches in Section Loss Function.The general segmentation approaches include global inference and slide inference. The former approaches could result in a significant quality drop during the image resolution compression. Therefore, we attempt to adapt the slide inference based methods into ultra image segmentation tasks. Given an ultra image I ∈ R H×W ×C , we divide it into N non-overlapping local patches I local ∈ R h×w×c and feed them into general segmentation module for prediction as our target. Next, we start from a random position within the local patch as the center and obtain a surrounding patch that is α(α > 1) times larger than its side length. The surrounding patch includes more contextual details, which is helpful to guide the local patch training. Afterward, the general segmentation module receives the local patch to extract features. Meanwhile, the surrounding context-guided branch (SCB) receives the surrounding patch to extract the surrounding information to guide the segmentation of local patch. In order to boost the processing speed of the SCB, a lightweight backbone is employed for feature extraction. The resulting feature map are subsequently transmitted to the surrounding context integration module for further relationship modeling and focused acquisition of contextual information essential for local patch prediction.The output feature map is aligned using coordinate relative relationships, retaining only the portion corresponding to the local patch. This portion is then combined with the feature map generated by the general segmentation module to obtain the final prediction through a standard segmentation head.To achieve a more complete optimization of the SCB, we add an extra auxiliary segmentation head to predict the result of the surrounding patch. This helps us to further calculate the boundary consistency loss between the local and surrounding patches based on this prediction.Compared to previous ultra image segmentation methods [6, 45, 24, 14, 19 ] that cannot handle extremely large images well, our architecture is more flexible and integrated, as it can handle an ANY size large image, equip with ANY general segmentation model, thereby well-suited for real-world scenarios.The information contained in the surrounding patch can serve as an extension of the local patch, providing it with more abundant decision-making guidance. Therefore, we introduce the Surrounding Context Integration Module (SCI), from the perspective of absorbing the information contained in each region, and integrating context across all the windows of surrounding patch. The structure of SCI is shown in Fig. 3 .Following [26, 25, 13] , the feature map F ∈ R H×W ×C is partitioned into a set of R × R nonoverlapping regions asF region ∈ R H R × W R ×R 2 ×C, with each region being subsequently subdivided into w × w non-overlapping windows asF w ∈ R H Rw × W Rw ×R 2 ×w 2 ×C .To facilitate information exchange within the region as well as absorb the contextual information that is helpful for segmentation, we perform layer normalization and window-based multi-head self-attention (W-MSA) on all patch tokens within F w as follows:EQUATIONCurrently, the information of all patch tokens within the region have been fully incorporated into F ′ w . Subsequently, we apply global average pooling (GAP) to acquire the global feature representationF global w ∈ R H Rw × W Rw ×R 2 ×1×Cfor each window, so as to facilitate the information exchange among the windows in later stage:EQUATIONThis process enables the absorption and integration of information from every position within the window into a unified vector representation that encapsulates the overall information of the window.To leverage the complementarity of information and encourage information sharing among windows within the surrounding patch, we utilize self-attention (SA) mechanism to capture the interdependence among the global feature representations F global w of each window:EQUATIONUpon completion of the information exchange, the global feature representation of the current window incorporates the information of other windows. Subsequently, the global feature representation F global ′ w is added to F ′ w by broadcasting, while the information of the remaining windows is transferred toEQUATIONTo enhance the compatibility and generalization for subsequent operation, we apply feed forward layer (FFN) to further refine F global :EQUATIONAt this time, F ′ global integrates the context information of the surrounding patch, which can strengthen the features of the local patch region as a complement and solve the challenge of information bottleneck.Despite the incorporation of local patch features and surrounding contextual information, there are still inconsistent prediction results between adjacent patches due to the lack of explicit constraints. Therefore, we propose a boundary consistency loss L Consistency to improve the consistency of prediction results in neighboring regions and alleviate the fragmentation phenomenon. L Consistency compels the prediction results of both the general segmentation module and SCB to be as similar as possible, thus promoting consistency in results of neighboring regions across different patches. This helps create a smoother transition between predictions of adjacent patches, harmonizing the prediction of the entire ultra image.Concretely, we crop the predicted mask P ′ global corresponding to the local patch from the prediction of the surrounding patch P global . Then, we apply L1 constraints on both P ′ global and the prediction result of local patch P local to encourage similarity between them, even when different contexts are utilized. The loss function is defined as follows:EQUATIONThe cross-entropy loss is used for both the general segmentation module (L CE ) and SCB (L SCB ).The overall loss L is a weighted sum of all the losses mentioned above: 4 ExperimentsEQUATIONQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: We have included our core code in an anonymized zip as part of the supplementary, and will release the complete code upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
datasets: 
 To comprehensively evaluate our method, we conduct experiments on five public ultra image datasets involving general, medical and remote sensing scenarios: Cityscapes [10] , DeepGlobe [11] , Inria Aerial [30] , Five-Billion-Pixels [40] , and Gleason [21] . Since we do not have data partition details from [16] , we randomly split the Gleason dataset into a training set of 195 images and a testing set of 49 images, and retrained the relevant models. All other datasets followed the official division.In both training and testing, the local patch has a size of 512×512 in all datasets, while the surrounding patch is twice as large without any resizing. During sliding inference process, we do not preserve any overlapping regions and the center of local patch and surrounding patch are the same. The first four stages of STDC, used as the lightweight backbone following [14] , are initialized with ImageNet weights. For all experiments, we set λ 1 = 1, λ 2 = 0.4, λ 3 = 0.1. For the DeepGlobe dataset, the "unknown" category is ignored during training as it is not included in the evaluation [11] . We also exclude "unlabeled" category in the FBP dataset following [5] .We adopt MMSegmentation [9] as our toolbox and use AdamW optimizer, which initial learning rate is set to 2 × 10 -4 . All the models are trained on 4 Tesla V100 GPUs with batch size of 8, except for the Inria Aerial and Gleason dataset, which are trained for 10k iterations while the rest are trained for 30k iterations. Apart from regular operations such as multi-scale training, flipping, and rotating, we do not do any special data augmentation, and the final result does not use any test time augmentation.We classify our comparison methods into two groups: general semantic segmentation and ultra image segmentation. To perform a comprehensive evaluation, we select semantic segmentation methods that rely on CNN (FCN, DeepLabV3Plus, HRNet), Transformer (SegFormer), and lightweight architecture (STDC) for comparison. In addition, we verify the performance of classic ultra image segmentation methods, including GLNet, ISDNet, GPWFoermer, etc. For fair comparison, we adopt ResNet-50 [15] or its equivalent parameter amount as the backbone for all the methods we compared.According to the results presented in Table 1 , UIS methods only work well on specific datasets, while GSS methods achieve relatively satisfying performance on all datasets. This confirms the aforementioned scalability issue. The existing UIS methods show a noticeable performance decrease when transitioning from handling images in DeepGlobe dataset (2448×2448) to larger ultra images ). We use DeepLabV3Plus as our general segmentation model and add SCB on top of it. We apply red color to mark regions where the background is misclassified as foreground and green color to denote regions where foreground is misclassified as background.Table 2 : Efficacy of proposed module. Align, SCI, Aux, Loss respectively correspond to the feature alignment operation, surrounding context integration module, auxiliary head and boundary consistency loss.Group Backbone Align SCI Aux Loss mIoU like FBP (6800×7200), particularly for those utilizing whole images as input, such as ISDNet. This demonstrates that the current UIS architecture lacks flexibility and faces a trade-off between models and datasets. This reveals the aforementioned architectural issue.A - - - - - 73.22 B ✓ - - - - 73.76 C ✓ ✓ - - - 73.88 D ✓ ✓ ✓ - - 74.76 E ✓ ✓ ✓ ✓ - 75.23 F ✓ ✓ ✓ ✓ ✓ 75.44Compared with above methods, our method obtains consistent gains upon all general segmentation methods. Since our method can seamlessly be incorporated into GSS and utilizes the sliding inference strategy for prediction, it can handle ultra images of any scale while leveraging the advantages of scalability from GSS. This perfectly resolves the scalability issue and architectural issue. Taking the DeepGlobe dataset as an example, our method yields a minimum improvement of 1.69% across all general models, and a maximum of 75.44% mIoU, which is 2.22% higher than using only DeepLabV3Plus. This demonstrates that our method can make full use of the contextual information in the surrounding patch to guide local patch segmentation. It is worth emphasizing that our method is flexible and applicable to any general segmentation models. It also should be noted that all the methods reported in Table 1 are sliding window without overlap. A sliding window with overlap is essentially a test time augmentation method, where multiple predictions on overlapping regions are averaged to enhance the model's performance. By using overlapping regions that cover half of the input patch for predictions, our method can further improve the performance by 0.24% to 75.68% mIoU on DeepGlobe dataset.Qualitative results of representative works are shown in Fig. 4 , while the results of WSDNet and GPWFormer are not available due to no public code. Since we leverage the contextual clues around the current patch as a guide, our method exhibits fewer false positives and delivers more precise segmentation results than other methods. Finally, group F incorporates the boundary consistency loss from group E, and we can conclude that this loss can effectively enhance the consistency between the bordering regions of adjacent patches, up to 75.44% mIoU. Additionally, when combining the logits maps from the surrounding branch and the local branch, the performance is further improved to 75.59% mIoU. The improvement in results essentially belongs to a model ensembling method. This indicates that our surrounding branch has learned effective and complementary information to the local branch, further demonstrating the validity of our approach.Other than analyzing the effectiveness of individual components within the module, we also demonstrate its generality across different methods, as shown in Fig. 5 . As observed, on the general dataset such as Cityscapes, our method still lead to significant and consistent improvements upon various models. In addition, we select objects whose area is less than 900 pixels (0.0036% of the whole image) from Inria Aerial dataset to verify the effect on small objects. Adding SCB improved the mIoU from 57.12% to 57.73%, which shows SCB also improve the tiny objects.To explore how the attention mechanism and global interaction affect the performance, we conduct ablation studies on surrounding context integration module in Table 3 . We first analyze the difference between the naive self-attention (Naive SA) and the window self-attention (W-MSA) in modeling the attention information of the surrounding patch. We apply six residual layers to replace attention module as the baseline, achieving 74.40% mIoU. Building on this, we further utilize Naive SA and W-MSA, which improve the mIoU by 0.46% and 0.32%, respectively. This indicates that the attention mechanism can capture the correlation between different positions, promoting the flow of surrounding information within the feature map. As naive self-attention operates on every pixel of the feature map, it inherently incorporates surrounding interaction operation. We further analyze the impact of incorporating surrounding information into the W-MSA mechanism, and we utilize convolution and GAP operations to extract surrounding information from the windows. After adopting either convolution or GAP, we can get consistent improvement, indicating that introducing surrounding context is essential for W-MSA, which enables the current window to acquire information of other regions. In comparison to the convolution, using GAP to extract the surrounding representation of the window is more suitable for W-MSA, which can reach up to 75.44% mIoU.We perform experiments to analyze the scheme of feature fusion of SCB and general segmentation module, as shown in Table 4 . The fusion position including before the decoder head (early fusion) and after the decoder head (late fusion). The fusion method including ADD and CONCAT. We observe that the performance of late fusion is better than early fusion. We believe that late fusion is more flexible, shielding the complex processing of different models in the input part of the decoder head.Both ADD and CONCAT operations yield satisfactory outcomes, where the latter have a leading of 0.51%. We believe that the segmentation head captures a greater amount of information, allowing it to dynamically select the optimal feature for prediction. Hence, modifying the segmentation head of GSS can further enhance the performance of model, but it deviates from our starting point of simplicity and is outside the scope of our method.We perform ablation studies on the surrounding patch size in Table 5 . We observe that, as the surrounding size increases, the performance initially improves and then gradually decreases. This is attributed to the fact that only those close nearby area could provide valuable contextual guidance. Too far-away pixels in the entire images are not relevant to the local patch and may introduce additional noise. This confirms the motivation that contextual information around the local patch is beneficial.Table 6 : Comparison of speed on DeepGlobe. We measure GPU memory using the command line tool "gpustat". " * " represents results we reproduced in our setting. "-" indicates that there is no publicly available result or code.Method mIoU FPS Memory(MB) GLNet [6] 71.60 0.17 1865 CascadePSP [8] 68.50 0.11 3236 PPN [45] 71.90 12.90 1193 PointRend [23] 71.78 6.25 1593 MagNet [16] 72.96 0.80 1559 MagNet-Fast [16] 71.85 3.40 1559 FCtL [24] 72.76 0.13 4332 ISDNet [14] 73.30 22.67 * 1948 GPWFormer [19] 75.80 -2380 DeepLabV3Plus [4] 73 We conduct experiments to examine the speed of different methods. Frames-per-second (FPS) and Memory are measured on a Tesla V100 GPU with a batch size of 1. The variation in FPS among different methods is primarily attributed to the inference framework, that is, whole inference and slide inference. Without considering the overlap, the latter theoretically requires W w × H h more operations than the former. Methods such as ISDNet primarily focus on model efficiency and achieve higher FPS, based on a shallow-deep architecture that directly employs the whole image for inference. Consequently, we also develop a fast version named SGNet (ISDNet-Style) by incorporating a modified SCB for a fair comparison. SGNet (ISDNet-Style) applies a single surrounding context integration module to the output feature map of the last stage of STDC to model global information. To maintain consistency with ISDNet and exclude speed differences caused by inference frameworks, we set our local patch size as same as ISDNet to simulate the whole inference. As shown in Table 6 , the fast version not only surpasses the mIoU of ISDNet but also achieves a higher FPS of up to 25.59.We compressed the DeepGlobe dataset to 10% of its original image quality using JPEG compression and retrained SGNet and DeepLabV3Plus on it. As shown in Table 7 , the results show that SGNet significantly outperforms DeepLabV3Plus by 3.92% (from 64.16% mIoU versus 60.24% mIoU), and this improvement is almost twice that of normal images (from 73.22% mIoU to 75.44% mIoU). This indicates that our method is relatively insensitive to noise compared to baseline models and can use surrounding information to infer damaged pixel information within the object. It also demonstrates that our method is particularly robust in scenarios involving image degradation.To show our advantage to alleviate the fragmentation phenomenon, we compare our module against GSS in Fig. 6 . We simulate all predicted patches (non-overlapping) generated during slide inference process with white lines. It is evident that GSS result in a steep change in the boundary between adjacent patches, and the predicted results of each patch are relatively independent, lacking coherence. Our method is capable of modeling the correlation between patches, leading to smoother prediction results for adjacent patches. 
conclusion: 
 In this paper, we propose the Surrounding Guided Segmentation Framework (SGNet) to address the scalability and architectural issues in existing UIS methods. SGNet leverages surrounding context to guide local patch segmentation and can be incorporated into any general segmentation model. Our method consistently improves performance across five datasets and demonstrates greater adaptability and applicability than existing methods in real-world scenarios. This work not only contributes a novel solution to the UIS domain but also emphasizes the potential of integrating general segmentation techniques to advance the field. We hope that SGNet inspires further exploration in ultra image segmentation, fostering innovations that enhance model performance and scalability.Limitations. Noise and artifacts in ultra-high-resolution images can hinder segmentation accuracy, necessitating further research to address these challenges. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Not applicable.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We have conducted.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the societal impacts in conclusion.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite all the original papers that provided the code and dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The proposed method has the potential to advance various fields, including medical image analysis and remote sensing image processing.In Figure 7 , we show the detail architectures of existing ultra image segmentation methods. Whole inference 7(a) and slide inference 7(b) are the two most fundamental ultra image segmentation architectures. While whole inference can be applied directly to general segmentation models [27, 4, 38, 46, 12, 49] , it utilizes the down-sampled image as input, which results in a loss of essential information. The latter divides the image into multiple patches by means of sliding window, and predicts the patches one by one, aggregating the results together. However, due to the isolated prediction of patches and the lack of global information, it is prone to cause information bottleneck and fragmentation phenomenon. To address the issues above and leverage the benefits of both whole inference and slice inference, some studies [6, 45, 24, 16, 19] have introduced the Global & Local architecture 7(c). This architecture involves one branch that takes in the down-sampled ultra image as the global cue, while the other branch extracts local information from sliced patches. However, these tasks usually require sequential processing of each patch and complex integration strategies to combine global information, resulting in poor scalability of the architecture. Alternatively, some study [14, 20] have introduced the Shallow & Deep architecture 7(d), which inputs the original image and the resized image into different branches for processing. Despite its fast speed, it is essentially a trade-off in dataset size and computing resources, and cannot be used for real ultra image segmentation scenarios.The architectures depicted in Figure 7 (c) and Figure 7 (d) both use the original image after resize in an input branch, which may limit the capability of the architecture to process ultra images. On the one hand, when the image is extremely large, the resized image will inevitably lose a lot of detail information to fit the GPU memory. On the other hand, the resized ultra image cannot guarantee to provide specific global information that is helpful for local patch segmentation. To address these issues, we designed a novel and effective ultra image segmentation architecture 7(e) based on slide window. Our architecture can handle ultra images of any scale and provide surrounding information that aids in local patch segmentation. Furthermore, our architecture is highly flexible, not restricted by computing resources, and can be seamlessly integrated to any encoder-decoder based segmentation model.To comprehensively evaluate our method, we conduct experiments on five public ultra image datasets involving general, medical and remote sensing scenarios:Cityscapes [10] . The Cityscapes dataset is a popular street scene dataset for generic semantic segmentation. It contains 3475 images with the resolution of 1024×2048 and a total of 19 categories. We use 2975 images for training and 500 images for testing.Gleason [21] . The Gleason dataset is a high resolution medical image dataset with the resolution of 5120×5120. It contains 244 H&E-stained histopathology images for automatic Gleason grading of prostate cancer. Since we do not have data partition details from [16] , we randomly split dataset into training and testing set with 195 and 49 images.DeepGlobe [11] . DeepGlobe is a satellite image dataset that contains 803 ultra-high resolution images (2448×2448 pixels). It contains 7 categories, of which the class named "unknown" is excluded. Following [6] , we divide the training, validation, and testing sets into 454, 207, and 142 images, respectively.Inria Aerial [30] . The Inria Aerial dataset comprises 180 images, each with a resolution of 5000×5000 pixels. Following [6] , we divide the training, validation, and testing sets into 126, 27, and 27 images, respectively.Five-Billion-Pixels [40] . The FBP dataset includes 150 high-resolution images, each with a size of 7200×6800 pixels and labeled with 24 categories. We use the same test set as in [40] , and randomly divide the remaining images into training set and validation set, including 90 and 30 images.In order to demonstrate the proposed SCB Branch's efficacy, we used sixteen conventional convolution layers followed by four transformer blocks to form a trivial replacement branch for extracting surrounding image feature. It serving as a functionally analogous replacement for the proposed SCB branch. As shown in Table 8 , we added this trivial replacement branch to all general segmentation models in Table 1 and conducted experiments on the DeepGlobe dataset using the same settings. The results show that our proposed SCB branch significantly outperforms this branch, proving the efficacy of our proposed component. We conducted experiments on the well-known CelebAMask-HQ dataset to further verify the effectiveness of our method in large scale human subject segmentation. Due to the lack of extremely high-resolution human datasets, we simulate ultra-high resolution by resizing the images from the CelebAMask-HQ dataset from 1024 to 2448 pixels. We compared our SGNet with DeepLabV3Plus, which is a highly popular and widely used image segmentation model across various domains. The performance of our method significantly outperformed DeepLabV3Plus by 1.61% (from 62.93% mIoU to 64.54% mIoU). We provide more examples of qualitative results in Figure 8 . In Figure 9 , we provide more examples of qualitative results between our method and existing ultra image segmentation methods like FCtL [24] and ISDNet [14] . These results indicate that our method is capable of achieving satisfactory quality on various challenging datasets [11, 30] . Figure 10 and Figure 11 illustrate the effectiveness of SCB by comparing it to general segmentation models such as DeepLabV3Plus and HRNet on Cityscapes [10] and Gleason [41] . NeurIPS Paper Checklist 


Decision: According to 3 out of 4 reviewers, this paper makes a solid contribution to the area of segmenting ultra highres images, and presents convincing results to show improvement over the state-of-the-art. One reviewer is an outlier, with a low score (3) and a very very short review. The authors alledge that review was written by an LLM. While the Area Chair was not able to validate that claim, the review is rather unprofessional and written in vague terms, well below the level of detail of the other reviews. Therefore the Area Chair largely discounted it and recommends this paper for acceptance.

Reviews: review: paper_summary: paper_summary: This paper focus on the generalization and architectural issues of the ultra image segmentation methods, and proposes SGNet which consists of two branches for processing surrounding patch and local patch, respectively. The motivation is to leverage the surrounding context for refining the segmentation results of local patches. The results show a significant improvement over previous methods. Overall, the paper is well-written with good results.
strength_weakness: strength_weakness: (1) The description of the surrounding branch is not clear enough (see question (1)).

(2) The evaluation of speed was insufficient (only included one of the lightest versions, i.e., ISDNet-Style). More adequate results could help to better understand the overhead of the method.
questions: questions: (1) What kind of backbone network is used in surrounding branch? Is it the same as that in local branch? Does the surrounding branch resize its input?

(2) A sliding window without overlap is used for inference, does this (with or without overlap) make a difference to the gain of the method?

(3) According to Figure 1, the surrounding branch’s output is discarded during inference. Is it possible to further improve the performance by combining it with the result of the local branch?

(4) There is a typo at last of the caption of Figure 1.
limitations: limitations: The limitations and social Impact have been discussed.
paper_summary: paper_summary: To overcome the challenges in generalization and compatibility with real-world ultra images, SGNet revisits a classic sliding inference approach and incorporates a surrounding context module to refine local patch segmentation. SGNet is compatible with various segmentation models and achieves significant performance improvements.
strength_weakness: strength_weakness: 1. The authors emphasize in lines 29 to 35 that general segmentation models possess superior generalization capabilities. However, from the results presented in Table 1, it can be observed that even general segmentation algorithms exhibit significant performance fluctuations across different datasets. This is particularly evident on the Cityscapes dataset.
2. Regarding the experiments detailed in Table 1, several general segmentation methods outperform existing specialized ultra image segmentation approaches on DeepGlobe, FBP, and Aerial Inria datasets without introducing the proposed SGNet.
    1. The authors attribute their performance superiority to the generalizability or architecture. However, they overlook the impact of different inference modes. Based on the description provided in the paper, all general segmentation methods employ sliding inference. However, this is not consistent across existing methods. This discrepancy precludes a direct conclusion that "general segmentation algorithms have better generalization."
    2. Given the diversity of inference modes, the authors should supplement Table 1 with specific inference settings for each method to identify more critical factors.
3. The authors mention in line 489 that due to the inaccessibility of the data partition details from [1], they conducted their own dataset division. Yet, in Table 1, only ISDNet [4] presents results for Gleason. This is logically inconsistent. Did the authors retrain ISDNet using your own data split?
4. The validation of the proposed component's efficacy is not rigorous. Since the component is pluggable, the performance evaluation should compare two variants: 1) removing the proposed component directly as shown in Table 1, and 2) replacing it with a trivial operation of similar function, such as some simple convolutional layers with the same number of parameters, or components with analogous functions from existing methods. The latter was overlooked by the authors.
5. The design of the surrounding context integration module appears strikingly similar to that in [2]. The authors should provide a detailed comparison.
6. There is a citation error for the Five-Billion-Pixels dataset. It should be referenced as [3].
7. For the experiments in Table 2:
    1. The order is illogical. It would be natural to integrate the alignment operation into the model before introducing SCI.
    2. The boundary consistency loss yields a limited performance gain.
8. What is the specific architecture of the first model variant listed in Table 3?
questions: questions: See Weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: The paper introduces SGNet to address the limitations of existing UIS methods, i.e., generalization issues and architectural constraints. SGNet leverages a larger context around image patches to refine segmentation results, making it compatible with general segmentation models and enhancing their performance on high-resolution images. The framework is validated on five diverse datasets, demonstrating significant improvements in segmentation accuracy and efficiency.
strength_weakness: strength_weakness: (1) The challenges in Contribution 1 have been proposed in many segmentation tasks. Therefore, C1 is weak.
(2) The contribution of the SGNet is incremental, including the consistency loss and the local and global branches.
(3) If the author wants to verify the SGNet is plug-and-play, the author should compare it with other plug-and-play methods rather than compare it with other task-specific strategies.
(4) The computational cost is expensive.
questions: questions: see weakness
limitations: limitations: see weakness
paper_summary: paper_summary: The paper discusses the challenges of ultra image segmentation and proposes a Surrounding Guided Segmentation framework (SGNet) to address these challenges by leveraging surrounding context information.
strength_weakness: strength_weakness: 1) For some reason, there seems to be no evaluation done on human subjects specifically. I would have liked to see some segmentation results on human subjects, especially around fingers, ear etc.
2) The discussion on limitations is extremely limited. Please read my comment in that section.
questions: questions: 1) In equation 7 for the loss function, the paper uses 3 parameters, where we could only do with 2 after normalization. 
2) It is typical to represent L1 loss with a subscript for clarity, so I would add it in equation 6.
3) The paper uses the word performance in several instances to discuss quality. It is an important distinction, as the usage of performance should be limited to quantify speed, and quality should be used for discuss the improvements in segmentation mask quality.
4) Table 6 discusses the memory footprint of the method, but it is quite unclear how this is measured or what it represents. Is it peak memory usage? Average memory usage during inference? 
5) In the efficiency study, it is unclear what device is the method evaluated on.
6) " The method can handle images of any scale.", this is only partially true as you are still going to be restricted by your compute resources. This needs to be clarified.
limitations: limitations: The discussion on limitations is extremely limited. Can this method be used for matting for instance? Or very thin peninsular regions? Will it improve or reduce the quality of segmentation if the images have jpeg artifacts? Can it be used on difficult objects like jewelry? These questions could be discussed in the limitations section.

Comments: title: Missing References in My Review
Official Comment by Reviewer HefE
Request for clarification for W5 "reference [2]"
comment: Dear Reviewer MJRN:

We greatly appreciate your suggestions!

Your comments are highly valuable and have provided excellent insights for our paper. 
We will definitely incorporate all the discussed suggestions in the revised version, which will further highlight the strengths of our work.

Sincerely, Authors
Thank you to the authors for responding to my questions and basically put my concerns to rest. Considering the good results and reference value of this context-enhanced design, I am willing to raise the score.

In fact, the existing review comments reflect the deficiencies that exist in the current version of the paper.
Regardless of eventual acceptance into the conference, I hope that the authors will improve and add these discussed contents in the revised version.
> W5: Provide a detailed comparison with [2].

First, our surrounding patch is designed to assist the local patch. It is centered around the local patch and its area is twice as large. The surrounding patch dynamically changes with the local patch, providing sufficient contextual information and preventing fragmentation.
In contrast, the regions in RegionViT [2] are predefined based on the size of the image.
Dynamically adjusting the surrounding patch provides better information, while the fixed division in RegionViT can lead to even more fragmented prediction results. This is especially problematic for ultra image segmentation tasks, such as remote sensing images, where regional features can vary significantly.

Second, the RegionViT integrates local tokens using self-attention with a single corresponding regional token. In contrast, we use the ROI Align operation to associate and merge information, which is designed for pixel-level understanding tasks, allowing precise alignment of information between the surrounding patch and the local patch.

Third, our approach employs the global average pooling and broadcasting operations for exchanging information. It is non-parametric and simpler compared to RegionViT, which introduces additional attention parameters for local self-attention.

Technically, we have not emphasized that the design of specific modules is our contribution, although the utilization of context information is similar across methods. The core contribution of our paper is addressing ultra image segmentation from the perspective of general segmentation models and proposing a plug-and-play approach that leverages surrounding information to guide a general segmentation model in segmenting local patches.
Thanks to the author for the response. After reading the response, most of my concerns have been addressed. I've decided to maintain my current score.
Our listed contributions do not include anything related to "the consistency loss and the local and global branches". 
Instead, we emphasize the framework that leverages surrounding context information to guide a general segmentation model in segmenting local patches. It also can be integrated into any general segmentation model.
Furthermore, we are the first to address the UIS task from the perspective of a general segmentation model, which is different from all previous works.

We've already explained W3 (insufficient experiments) in our rebuttal, clarifying that we are the first to propose a "plug-and-play" method for ultra image segmentation, and there are no other "plug-and-play methods" for this task. Please list any references if we missed and specify what "insufficient experiments" includes.
REF:

* [1]: Progressive semantic segmentation, CVPR 2021
* [2]: RegionViT: Regional-to-Local Attention for Vision Transformers, ICLR 2022
* [3]: Enabling Country-Scale Land Cover Mapping with Meter-Resolution Satellite Imagery, ISPRS Journal of Photogrammetry and Remote Sensing 2023
* [4]: Isdnet: Integrating shallow and deep networks for efficient ultra-high resolution segmentation, CVPR 2022
I appreciate the author's feedback. I am inclined to maintain my score because my major concerns, including incremental contributions and insufficient experiments, haven't been addressed.
Dear Reviewer,

We are currently in the process of drafting our rebuttal response and would greatly appreciate your clarification on a point mentioned in Weakness 5.

Could you please provide the specific title of the reference [2] so that we can address this point more accurately in our response.

Thank you very much for your assistance!

Sincerely,
Authors