Toward Real Ultra Image Segmentation: Leveraging Surrounding Context to Cultivate General Segmentation Model
Authors: Sai Wang
Yutian Lin
Yu Wu
Bo Du
Venue: NeurIPS 2024 poster (2024)
Keywords: Ultra Image Segmentation
Primary area: machine_vision

Abstract: Existing ultra image segmentation methods suffer from two major challenges, namely the scalability issue (i.e. they lack the stability and generality of standard segmentation models, as they are tailored to specific datasets), and the architectural issue (i.e. they are incompatible with real-world ultra image scenes, as they compromise between image size and computing resources).
To tackle these issues, we revisit the classic sliding inference framework, upon which we propose a Surrounding Guided Segmentation  framework  (SGNet) for ultra image segmentation. 
The SGNet leverages a larger area around each image patch to refine the general segmentation results of local patches.
Specifically, we propose a surrounding context integration module to absorb surrounding context information and extract specific features that are beneficial to local patches. Note that, SGNet can be seamlessly integrated to any general segmentation model.
Extensive experiments on five datasets demonstrate that SGNet achieves competitive performance and consistent improvements across a variety of general segmentation models, surpassing the traditional ultra image segmentation methods by a large margin.

Toward Real Ultra Image Segmentation: Leveraging Surrounding Context to Cultivate General Segmentation Model
Sai Wang, Yutian Lin, Yu Wu; Bo Du
School of Computer Science, Wuhan University [aaggaai23, wuyucs, yutian. dubo) @ @whu. edu. en
Abstract
Existing ultra image segmentation methods suffer from two major challenges, namely the scalability issue (i.e. they lack the stability and generality of standard segmentation models, as they are tailored to specific datasets), and the architec- tural issue (i.e. they are incompatible with real-world ultra image scenes, as they compromise between image size and computing resources). To tackle these issues, we revisit the classic sliding inference framework, upon which we propose a Sur- rounding Guided Segmentation framework (SGNet) for ultra image segmentation. The SGNet leverages a larger area around each image patch to refine the general segmentation results of local patches. Specifically, context integration module to absorb surrounding context information and extract specific features that beneficial to local patches. Note that, SGNet can be seam- lessly integrated to any general segmentation model. Extensive experiments on five datasets demonstrate that SGNet achieves competitive performance and consistent improvements across a variety of general segmentation models, surpassing the traditional ultra image segmentation methods by large margin.
propose surrounding
Introduction
With the rapid development of computing and imaging equipment, ultra-high resolution images with millions or even billions of pixels emerge endlessly and driving the need for more advanced analytical techniques. Accurate understanding of information conveyed by images [1, 47, 43] has become imperative diverse fields such as remote sensing [29, 48, 44, 50) and medical analysis [37, 36, 31], and ultra image segmentation has emerged as essential tool for achieving this goal.
To achieve ultra image segmentation, most of the methods adhere the concept of integrating global and local information, utilizing the global context clues to aid local region refinement [6, 20]. Typically, these networks have two branches: one receives the nown-sampled ultra image and extracts global context information of whole image, while the other extracts local information from sliced patches the complete image, as shown in Fig. 1(a). The final result is generated by fusing the global and local features. Despite numerous improvements made to ultra image segmentation, we observe that still suffer from issues in both their scalability and architecture:
(I) Scalability Issue. The existing ultra image segmentation methods (UIS) often rely on dataset- specific parameters, which limit their model capacity and scalability. The UIS methods struggle to scale up to larger image sizes, with performance significantly degrading as image resolution increases. Moreover, UIS has complex training procedures that need multi-stage parameter tuning [8, 24J. In contrast, general semantic segmentation (GSS) methods are more effective and demonstrate greater
'CCoresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Uin Imaze
Ulta Iang?
Organl
Cop tado
Gleas Dop thash
Gocal Stulko
Protidhar
Production
(a) Specific method
(b) General model
(a) Fragmentation
(b) Ground Truth
Figure I: Comparison with specific (a) and (b). The Figure 2: Directly adapting the general segmentation previous architecture all suffered from Scalability Issue model to ultra image scene will cause fragmentation and Architectural Issue due specific designs and phenomenon (a): prediction results of the edge lack tailored context, whereas ours is designed to areas between adjacent patches (even in overlapped leverage any general segmentation model to address patches) are inconsistent. ultra image segmentation.
scalability compared UIS. With straightforward training process, GSS can be adaptable to various ultra image datasets.
(2) Architectural Issue. The architecture of existing UIS is not suitable for processing ultra image scenes real-world scenarios, as it compromises between dataset size and computing resources. Most methods utilize either the entire image the downsampled version to capture global information. However, the two strategies are both constrained by the size of input image. When the input image is excessively large, the former strategy cannot process it directly, while the later will suffer great information loss during compression.
Building upon the above discussions, a simple solution is to introduce general segmentation model [27, 12, 18, 39] into the ultra image segmentation task using the sliding window approach, which takes only isolated patches as input due to memory limitations. However, directly adapting GSS to ultra image scenes also raises two challenges: (1) Information bottleneck. Using isolated patches input prevents the model from capturing the correlations between patches, which blocks the infor- mation flow and affects the model's perception of the surrounding information. (2) Fragmentation phenomenon. With isolated patch input, even if overlapped patches are used, the prediction results of each patch are independent. This leads to inconsistent prediction between patches, especially at the edges of adjacent patches, as shown in Fig. 2.
Based on above considerations, we propose an end end-to-end framework called Surrounding Guided Segmentation Framework (SGNet), which takes advantage of the general segmentation method and utilizes surrounding information near the local patch to guide the model. As shown in Fig. 1(b), SGNet includes two decoupled parts: a general segmentation module and surrounding context- guided branch. Specifically, SGNet takes both the local patch and the corresponding surrounding patch input. The surrounding patch covers larger area around the local patch and provides more context information. required to segment the local patch from a larger perspective to drive the flow of information between regions of whole image. Besides, to alleviate the fragmentation phenomenon, we propose a boundary consistency loss to improve the consistency of the prediction results of adjacent patches, thereby alleviating the inconsistent predictions. Note that, unlike existing tightly coupled ultra images segmentation methods, our method can be seamlessly incorporated into any general segmentation models, and brings stable performance improvements.
the surrounding context-guided branch, we model the context information
Our contributions are summarized as follows:
We excavate two essential but largely overlooked issues in UIS, which hold great value for the community. In addressing these challenges, we are the first to tackle the ultra image segmentation task from the general segmentation model perspective.
We present a novel end-to-end ultra image segmentation framework named SGNet, which leverage surrounding context information to guide patch-based model segmentation. Our method is flexible to be added to any general segmentation model.

---

Experiments show that our method achieves competitive performance and consistently improves over different general segmentation models on five public datasets, outperforming previous methods by large margin.
Related Work
2.1
General Semantic Segmentation
With the development of deep learning [33, 32], the two mainstream methods based on convolution
neural network [52, 35, 34, 2] and Transformer [42, 17, 51, 28] have achieved excellent performance. FCN [27] is the first fully convolutional architecture and a lot of work has been extended, such as D3eplabV3 [3] and HRNet [38]. Compared with CNN-based methods, representative Transformer- based works include SegFormer [46], Mask2Foremr [7] and SAM [22]. BiSeNetV1 [49] and STDC [12] are designed for real-time segmentation to reduce the computational overhead. The general segmentation model shows excellent stability and scalability, and we aim to leverage its strengths to address both the Scalability and Architectural issues in existing UIS methods.
2.2 Ultra Image Semantic Segmentation
GLNet [6] introduces novel global-local architecture. PPN [45] builds on the top of GLNet by inte- grating a classification network to distinguish valuable patches. Furthermore, Magnet [16] employs multi-stage pipeline where each stage corresponds to a specific magnification level. FCIL [24] exploits locality-awel contextual correlation to effectively integrate and associate contextual informa- tion of local patches. Compared previous methods, ISDNet [14] proposes novel framework that combines shallow and deep networks, enabling directly whole-image inference, thereby improving
the segmentation effect while increasing the speed. ElegantSeg [5] proposes a end-to-ent holistic learning framework from the perspective of engineering optimization. Recently, WSDNet [20] follows the architecture ISDNet, using DWT-IWT to preserve spatial details. GPWFormer [19]
also employs the global-local architecture, and propose the wavelet transformer to model semantic relations. However, the aforementioned methods all suffer from inherent framework flaws, and cannot be applied to ultra images of extremely large scale. On the contrary, our simple yet generalized solution is more adaptable and applicable than specific UIS methods real-world scenarios.
3 Methodology
3.1 Overview
In this section, we present SGNet, novel framework that enables existing general segmentation models for ultra image segmentation. As shown in Fig. 3, SGNet consists of two major components, the general segmentation module and surrounding context-guided branch (SCB). The two branches take the local patch and its surrounding larger area as inputs, respectively and extract their features (Section Architecture). In the surrounding context-guided branch, we introduce surrounding context integration module (Section Surrounding Context Integration Module) to enable interaction between local and surrounding features, and selectively learn contextual information that is helpful for patch segmentation. Furthermore, we propose boundary consistency loss to maintain the consistency of prediction results across adjacent patches in Section Loss Function.
3.2 Architecture
The general segmentation approaches include global inference and slide inference. The former approaches could result in significant quality drop during the image resolution compression. Therefore, we attempt to adapt the slide inference based methods into ultra image segmentation tasks. Given an ultra image I I XxX we divide it into N non-overlapping local patches Irera R xxxx and feed them into general segmentation module for prediction as our target. Next, we start from a random position within the local patch as the center and obtain surrounding patch that is a( > 1) times larger than its side length. The surrounding patch includes more contextual details, which is helpful to guide the local patch training.
3

---

Surrounding Context-guided
Sarroundnn Patch
GAP
Breadeast
Ultra high reselution
IN W-MSA
Food Feeward
Sig Hold
Ssca
sany pnd
Coordinase 7
Consistencc
DONE Rog
Backbone
Decoder
-Cn
Ultra Image!
General Segmentation Medule
Concatenation
Remeved dusing inference
Figure 3: The Architecture of SGNet. An ultra image I randomly cropped to obtain a local patch Tracot and a surrounding patch Lytobal containing more context information of any size, which are respectively the general segmentation module and the surrounding context-guides branch extract features. The resulting features are aggregated through simple concatenation and used to generate high-qualij, predictions. General segmentation module can be applied to any segmentation model, and surrounding context-guided branch consistently achieves stable improvements on it. I.N, W-MSA, and GAP stand for layer normalization, window-based multi-head self-attention, and global average pooling, respectively.
Afterward, the general segmentation module receives the local patch to extract features. Meanwhile, the surrounding context-guie guided branch (SCB) receives the surrounding patch to extract the surround- ing information guide the segmentation of local patch. In order boost the processing speed of the SCB, lightweight backbone is employed for feature extraction. The resulting feature map subsequently transmitted to the surrounding context integration module for further relationship modeling and focused acquisition of contextual information essential for local patch prediction.
The output feature map aligned using coordinate relative relationships, retaining only the portion corresponding to the local patch. This portion is then combined with the feature map generated by the general segmentation module obtain the final prediction through standard segmentation head. To achieve more complete optimization of the SCB, we add an extra auxiliary segmentation head to predict the result of the surrounding patch. This helps us further calculate the boundary consistency loss between the local and surrounding patches based on this prediction.
Compared to previous ultra image segmentation methods [6, 45, 24, 14, 19] that cannot handle extremely large images well, our architecture is more flexible and integrated, as it can handle ANY size large image, equip with ANY general segmentation model, thereby well-suited for real-world scenarios.
3.3 Surrounding Context Integration Module
The information contained in the surrounding patch serve extension of the local patch, providing with more abundant decision-making guidance. Therefore, we introduce the Surrounding Context Integration Module (SCI), from the perspective of absorbing the information contained in each region, and integrating context across all the windows of surrounding patch. The structure of SCI is shown Fig. 3.
Following [26, 25, 13], the feature map F Hx is partitioned into a set of R x R non- overlapping regions as Fregion E R # x XC with each region being subsequently subdivided into non-overlapping windows F... R
Rw R2 xxxxxxx
To facilitate information exchange within the region as well as absorb the contextual information that is helpful for segmentation, perform layer normalization and window-based multi-head self-attention (W-MSA) on all patch tokens within F., as follows:
F (.MSA(LN(F.)) F..
(I)

---

Currently, the information of all patch tokens within the region have been fully incorporated into F Subsequently, we apply global average pooling (GAP) acquire the global feature representation Figlbal E R y the R² xlxC for each window, so as to facilitate the information exchange among the windows in later stage:
Fglobal GAP(FF))
(2)
This process enables the absorption and integration of information from every position within the window into unified vector representation that encapsulates the overall information of the window.
To leverage the complementarity of information and encourage information sharing among windows within the surrounding patch, we utilize self-attention (SA) mechanism to capture the interdependence among the global feature representations Fglobal of each window:
pglobal" -SA( (7ylobai))
Upon completion of the information exchange, the global feature representation the current window incorporates the information of other windows. Subsequently, the global feature represen- tation @ Fglobal is added to F by broadcasting, while the information of the remaining windows is transferred Fr:
Fglobal
F
F.glbbll"
To enhance the compatibility and generalization for subsequent operation, we apply feed forward layer (FFN) to further refine Folobal:
F Flobbat FFN (FFgoba)) globat) + F global-
At this time, F' Fgtobal integrates the context information of the surrounding patch, which can strengthen the features of the local patch region complement and solve the challenge of information bottleneck.
3.4
Loss Function
3.4.1 Boundary Consistency Loss
Despite the incorporation of local patch features and surrounding contextual information, there still inconsistent prediction results between adjacent patches due to the lack of explicit constraints. Therefore, we propose boundary consistency loss LConsisterey improve the consistency of prediction results in neighboring regions and alleviate the fragmentation phenomenon. LConsistency compels the prediction results of both the general segmentation module and SCB be as similar as possible, thus promoting consistency in results of neighboring regions across different patches. This helps create smoother transition between predictions of adjacent patches, harmonizing the prediction of the entire ultra image.
Concretely, we crop the predicted mask P' of the surrounding patch Pglobel. Then, we apply L1 constraints on both P global and the prediction result of local patch Ptocat encourage similarity between them, even when different contexts utilized. The loss function is defined as follows:
global corresponding to the local patch from the prediction
LConsistency 1|| global Piocat ||-
(6)
3.4.2 Overall Loss
The cross-entropy loss is used for both the general segmentation module (LCE) and SCB (Lson).
The overall loss L is weighted sum of all the losses mentioned above:
AiLoE AxCsOB + XyLCComsistency.
(7)
5

---

Table I: Comparison with baseline on five datasets. The "Mode" column denotes the specific inference modes associated with each method. The "Test size" column relates to the DeepGlobe dataset.
Method
Mode
Backbone
Test size
DeepGlobe 2448 2448
FBF
Accial Inzta 5000x00000
Glesson S120x5120
Citsscaps 2048x 1024
7200
Image Seguentation Methods
GLNet [6]
Slide+Whule
R50
308
71.60
42.05
71:20
PPN
Slide+ Whole
R50
512
71.90
75.20
MagNet [16]
Slide+Whole
R50
72.96
67.57
FCLL [24]
Slide
VGGL6
508
72.76
48.28
72.87
ISDNet [14]
Whole
74.23
76.02
ElegamSeg [5]
Whole
W48
2448
74.32
61.62
WSDNer [20]
Whole
RS0
2448
74.10
75.20
GPWFormer [19]
While
76.50
General Semantic Segmentation Methody
FCN [27]
Slide
RS0
512
72.38
59.97
86:35
52.03
72.39
+eGNer
Slide
R50
512
75.28 (+2.90)
61.85 (+1.88)
80.81 (+0.46)
58.46 (+5.81)
75.84 (+3.45)
DeepLabV3Plns [4]
Slide
R50
512
73.22
61.85
80.88
55.45
75.24
+ SGGNrr
Slide
R50
512
75.44 (+2.22)
63.18 (+1.33)
81.21 (+0.33)
61.21 (+5.76)
76,72 (+148)
HRNeT
Slide
79.17
+ SGNer
Slide
WIS
512
75.25 (+2.38)
61.49 (+2.94)
80.08 (+0.91)
60.50 (+6.19)
73.06 (+1.86
SeyFormer [46]
Slide
Mit-h0
512
72.96
$7.56
76.26
49.62
67.08
SGGNe
Slide
Mir-h0
69)
60:35 (42.79)
77.880 (+2.54)
$4.86
(+5.22 24)
770.42 (+3.34)
STDC [12)
Slide
R50
512
72.59
54,38
75.20
$4,51
66.24
+ Ner
Slide
R50
512
74.51
(+1.92)
59.40 (+5.02)
77.25 (+2.05)
60.36 (+5.85)
69.36 (+3.12)
Experiments
4.1 Datasets
To comprehensively evaluate our method, we conduct experiments on five public ultra image datasets involving general, medical and remote sensing scenarios: Cityscapes [10], DeepGlobe [11]. Inria Aerial [30], LieeBBllionnPiiele [40], and Gleason [21]. Since we do not have data partition details from [16], we randomly split the Gleason dataset into training set of 195 images and a testing set of 49 images, and retrained the relevant models. All other datasets followed the official division.
4.2 Implementation Details
In both training and testing, the local patch has a size of patch is twice large without any resizing. During sliding inference process, do preserve any overlapping regions and the center of local patch and surrounding patch are the same. The first four stages of STDC, used as the lightweight backbone following [14], are initialized with ImageNet weights. For all experiments, we set λ = 1, A2 = 0.4, A3 = 0.1. For the DeepGlobe dataset, the "unknown" category is ignored during training exclude "unlabeled" category in the FBP dataset following [5].
in all datasets, while the surrounding
included in the evaluation [I J... We also
We adopt MMSegmennation [9] our toolbox and use AdamW optimizer, which initial learning rate is set to 2 x 10 All the models are trained on 4 Tesla V100 GPUs with batch size of 8, except for the Inria Aerial and Gleason dataset, which are trained for 10k iterations while the 30k iterations. Apart from regular operations such as multi-scale training, flipping, and rotating, we do not do any special data augmentation, and the final result does not use any test time augmentation.
trained for
4.3 Comparison Results
We classify our comparison methods into two groups: general semantic segmentation and ultra image segmentation. perform a comprehensive evaluation, we select semantic segmentation methods that rely CNN (FCN, DeepLabV3Plus, HRNet), Transformer (SegFormer), and lightweight architecture (STDC) for comparison. In addition, we verify the performance of classic ultra image segmentation methods, including GLNet, ISDNet, GPWFoermer, etc. For fair comparison, adopt ResNet-50 [15] or its equivalent parameter amount as the backbone for all the methods we compared.
4.3.1 General VS Ultra Image Segmentation Methods
According the results presented in Table 1, UIS methods only work well on specific datasets, while GSS methods achieve relatively satisfying performance on all datasets. This confirms the aforementioned scalability issue. The existing UIS methods show noticeable performance decrease when transitioning from handling images in DeepGlobe dataset (2448 x 22488 to larger ultra images

---

(a) Image
(b) Ground Truth
(c) FCIL
(d) ISDNet
(c) Ours
Figure 4: The visualization of different methods in DeepGlobe (top row, 2448 2448) and Inria Aerial (bottom row, 5000x We DeepLabV3Plrr apply red color mark regions where the background is misclassified foreground and green color denote regions where foreground is misclassified background.
general segmentation model and add SCB top of it. We
Table 2: Efficacy of proposed module. Align, SCL, Aux, Loss respectively correspond to the fcature align- ment operation, surrounding context integration mod- ulc, auxiliary head and boundary consistency loss. Group Backbone
Table 3: Analysis of Surrounding Context Integration.
Allentio
Global Interaction
Nnive
W.MSA
Cony
GAP
mloD
Align
SCT
Aux
Loss
mioli
74.86
73.22
75.17
73.76
75.44
73.88
74.76
75.23
like FBP (6800x7200), particularly for those utilizing whole images input, such as ISDNet. This demonstrates that the current UIS architecture lacks flexibility and faces trade-off between models and datasets. This reveals the aforementioned architectural issue.
4.3.2 Improvement over General Segmentation Model
Compared with above methods, our method obtains consistent gains upon all general segmentation methods. Since our method can seamlessly be incorporated into GSS and utilizes the sliding inference strategy for prediction, can handle ultra images of any scale while leveraging the advantages of scalability from GSS. This perfectly resolves the scalability issue and architectural issue. Taking the DeepGlobe dataset as an example, our method yields a minimum improvement of 1.69% across all general models, and maximum of 75.44% mIoU, which is 2.22% higher than using only DeepLab V3Plus. This demonstrates that our method can make full use of the contextual information in the surrounding patch to guide local patch segmentation. It is worth emphasizing that our method is flexible and applicable to any general segmentation models. also should be noted that all the methods reported in Table I are sliding window without overlap. A sliding window with overlap is essentially test time augmentation method, where multiple predictions on overlapping regions are averaged to enhance the model's performance. By using overlapping regions that cover half of the input patch for predictions, our method can further improve the performance by 0.24% to 75.68% % mloU DeepGlobe dataset.
Qualitative results of representative works are shown Fig. 4, while the results WSDNet and Formor are not available due to no public code. Since we leverage the contextual clues around the current patch as guide, our method exhibits fewer false positives and delivers more precise segmentation results than other methods.

---

(a) Image
(b) HRNet
(c) Deeplab
(a) w/o SCB
(b) w/ SCB
(d) GT
(c) HRNeHSCB (f) Deeplab+SCB
Figure 5: Comparison of adding SCB on different mod- Figure 6: Comparison of adding SCB on different mod- els (from top bottom are DeeplabV3Plus. HRNet els in Gleason (5120x5120). The predicted patches of and FCN in (()) and 5(b)) in Cityscapes (2048 x 1024). model are divided by the white line, while the red box The mloU values are calculated on each shown images. indicates the area where the model failed to predict.
Table 4: Analysis of feature fusion scheme.
Table S: Analysis of surrounding patch size.
Tarly Fusion
Lale Fusion
ADD
CONCAT
mloU
Scule (a) Surrounding Patch Size
mloU
74.68
512
74.75
74.87
2
1024
75.44
1536
74.96
75.44
2048
74.78
4.4
Ablation Study
4.4.1
Effectiveness of Proposed Module
To thoroughly confirm the efficacy of each module, we substitute SGNet (Deepl.at V3Plus version) with six variations denoted as Group A ABCDE. As shown Table we employ only lightweight backbone individually extract features from surrounding patch, and then concatenated them last features in group B. Group B achieves a mloU of 73.76% which exceeds group Å that only used general segmentation module by 0.54%. In group C, we utilize the crop operation align features, resulting in an improvement of O. 1.12% compared to B. By applying the surrounding context integration module to model the context information, and the auxiliary head enhance the convergence of SCB in groups and E, we are able to improve performance by 0.88% and 1.35% compared to group C. Finally, group FF incorporates the boundary consistency loss from group E, and this loss can effectively enhance the consistency between the bordering regions of adjacent patches, up 75. 54.4% mloU. Additionally, when combining the logits maps from the surrounding branch and the local branch, the performance further improved to 75.59% mloU. The improvement in results essentially belongs model ensembling method. This indicates that our surrounding branch has learned effective and complementary information to the local branch, further demonstrating the validity of our approach.
conclude that
Other than analyzing the effectiveness individual components within the module, we also demon- strate its generality across different methods, shown in Fig. 5. As observed, the general dataset such as Cityscapes, our method still lead to significant and consistent improvements upon various models. In addition, select objects whose area is less than 900 pixels (0.0036% of the whole image) from Inria Aerial dataset to verify the effect on small objects. Adding SCB improved the mloU from 57.12% to 57.73%, which shows SCB also improve the tiny objects.
4.4.2 Efficacy of Surrounding Context Integration Module
To explore how the attention mechanism and global interaction affect the performance, we conduct ablation studies on surrounding context integration module in Table 3. We first analyze the difference between the naive self-attention (Naive SA) and the window self-attention (W-MSA) in modeling the attention information of the surrounding patch. We apply six residual layers to replace attention module as the baseline, achieving 74.40% 40% mloU. Building on this, we further utilize Naive SA and W-MSA, which improve the mloU by 0.46% and 0.32%, respectively. This indicates that the attention

---

mechanism can capture the correlation between different positions, promoting the flow of surrounding information within the feature map. As naive self-attention operates on every pixel of the feature map, it inherently incorporates surrounding interaction operation. We further analyze the impact of incorporating surrounding information into the W-MSA mechanism, and we utilize convolution and GAP operations to extract surrounding information from the windows. After adopting either convolution or GAP, we can get consistent improvement, indicating that introducing surrounding context is essential for W-MSA, which enables the current window to acquire information of other regions. In comparison to the convolution, using GAP to extract the surrounding representation of the window is more suitable for W-MSA, which reach up 75.44% mloU.
4.4.3 Efficacy of Feature Fusion
We perform experiments to analyze the scheme of feature fusion of SCB and general segmentation module, as shown in Table The fusion position including before the decoder head (early fusion) and after the decoder head (late fusion). The fusion method including ADD and CONCAT. We observe that the performance of late fusion is better than early fusion. We believe that late fusion more flexible, shielding the complex processing of different models in the input part of the decoder head. Both ADD and CONCAT operations yield satisfactory outcomes, where the latter have a leading of 0.51%. We believe that the segmentation head captures greater amount of information, allowing to dynamically select the optimal feature for prediction. Hence, modifying the segmentation head of GSS further enhance the performance of model, but it deviates from our starting point of simplicity and is outside the scope of our method.
4.4.4 Comparison of Surrounding Patch Size
We perform ablation studies on the surrounding patch size Table We observe that, as the surrounding size increases, the performance initially improves and then gradually decreases. This is attributed the fact that only those close nearby area could provide valuable contextual guidance. Too far-away pixels in the entire images are not relevant to the local patch and may introduce additional noise. This confirms the motivation that contextual information around the local patch is beneficial.
4.4.5 Efficiency Study
We conduct experiments to examine the speed of different methods. Frames- pennee--er--econdd (FPS) and Memory are measured on a Tesla V100 GPU with a batch size of 1. The variation in FPS among different methods is primarily attributed to the inference framework, that is, whole inference and slide inference. Without considering the overlap, the latter theoretically requires W x H!
Table 6: Comparison of speed on DeepGlobe. We mea- sure GPU memory using the command line tool "gpus- tat". "*" represents results reproduced in our setting. w_" indicates that there is publicly available result code.
Method
mloU
FPS
Mcmory(MB)
GLNet [6]
71.60
0.17
1865
h more operations than the former. Methods such as ISDNet primarily fo- cus on model efficiency and achieve higher FPS, based on shallow-deep architecture that di- rectly employs the whole image for inference. Consequently, also develop fast version named SGNet (ISDNet-Style) by incorporating modified SCB for a fair comparison. SGNet (ISDNet-Style) applies a single surrounding con- text integration module the output feature map of the last stage of STDC to model global infor- mation. To maintain consistency with ISDNet and exclude speed differences caused by infer- ence frameworks, we set our local patch size as same as ISDNet to simulate the whole inference. shown in Table 6, the fast version not only surpasses the mloU of ISDNet but also achieves higher FPS of up 25.59.
CuscadePSP [8]
68.50
0.11
3236
PPN
71.90
12.90
1193
PointRend [23]
71.78
6.25
1593
Mag/Net [16]
72.96
0.80
1559
MagNet-Fast [16]
71.85
3.40
1559
[C4.
72.76
0.13
4332
ISDNet [14]
73.30
22.67*
1948
GPWFormer [19]
75.80
2380
Deepl 3Pus [4]
73.22
1.14
1279
DeepLabV3Plus SGNet
75.44
0.66
2187
SGNer (1SyNettStyle)
74.28
25.59
2043
Table 7: Comparison of normal image and JPRG com-
pressed image.
Normal
JPEG compression
DeePLabV3Plus
13,22
60.24
SGNer
75.44 (42.22)
64.16 (+3.92)

---

4.4.6 Robustness Study
We compressed the DeepGlobe dataset to 10% of its original image quality using JPEG compression and retrained SGNet and DeepLabV3Plun on it. As shown in Table 7, the results show that SGNet significantly outperforms ReepLab33Phur by 3.92% (from 64.16% mloU versus 60.24% mloU), and this improvement almost twice that of normal images (from 73.22% mloU to 75.44% mloU). This indicates that method is relatively insensitive noise compared to baseline models and can surrounding information to infer damaged pixel information within the object. also demonstrates that method is particularly robust in scenarios involving image degradation.
4.4.7 Fragmentation Phenomenon Study
To show our advantage to alleviate the fragmentation phenomenon, we compare our module against GSS in Fig. 6. We simulate all predicted patches von-overlappinng generated during slide inference process with white lines. adjacent patches, and the predicted results of each patch are relatively independent, lacking coherence. Our method is capable of modeling the correlation between patches, leading smoother prediction results for adjacent patches.
evident that GSS result in steep change in the boundary between
Conclusion
In this paper, we propose the Surrounding Guided Segmentation Framework (SGNet) to address the scalability and architectural issues in existing UIS methods. SGNet leverages surrounding context to guide local patch segmentation and can be incorporated into any general segmentation model. Our method consistently improves performance across five datasets and demonstrates greater adaptability and applicability than existing methods real-world scenarios. This work not only contributes novel solution to the UIS domain but also emphasizes the potential of integrating general segmentation techniques to advance the field. We hope that SGNet inspires further exploration in ultra image segmentation, fostering innovations that enhance model performance and scalability.
Limitations. Noise and artifacts ultra-high-resolutton images can hinder segmentation accuracy, necessitating further research to address these challenges.
Social Impact. The proposed method has the potential to advance various fields, including medical image analysis and remote sensing image processing.
Acknowledgment. This work was supported in part by the National Key Research and Development Program of China under grant 2023YFC2705700, in part by the National Natural Science Foundation of China under grant 62372341, and in part by the Fundamental Research Funds for the Central Universities under grant 2042024k10040.
References
[1] Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, and Alex Hauptmann. comprehensive survey of scene graphs: Generation and application. IEEE TPAMI, 45(1):1-26. 2021.
[2] Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-local attention for vision transformers. arXiv preprint arXiv:2106.02668 2021.
[3] Liang-Chiel Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arxiv 2017. arXiv preprint arXiv:1706005588 2, 2019.
[4] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, pages 801-818, 2018.
[5] Wei Chen, Yansheng Li, Dang, Yongjun Zhang. Flegantseg: End-to-end holistic learning for extra-large image semantic segmentation. arXiv preprint arXiv:2211.11316 2022.
10

---

[6] Wuyang Chen, Ziyu Jiang, Zhangyang Wang, Kexin Cui, and Xiaoning Qian. Collaborative global-local networks for memory-efficee segmentation of ultra-high resolution images. In CVPR, pages 8924-8933, 2019.
[7] Bowen Cheng, Ishan Mísra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, pages 1290- 1299, 2022.
[8] Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, and Chi-Keung Tang. Cascadepsp: Toward class- agnostic and very high-resolution segmentation via global and local refinement. In CVPR, pages 8890-8899, 2020.
[9]
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github. com/ open- www.mmmmsssmmmmm
2020.
[10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213-3223, 2016.
[11] like Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang. Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. Deepglobe 2018: challenge to parse the earth through satellite images. In CVPRW, pages 172-181, 2018.
[12] Mingyuan Fan, Shenqi Lai, Junshi Huang, Xiaoming Wei, Zhenhua Chai, Junfeng Luo, and Xiaolin Wei. Rethinking bisenet for real-time semantic segmentation. In CVPR, pages 9716- 9725, 2021.
[13] Jiemin Fang, Lingxi Xie, Xinggang Wang, Xiaopeng Zhang, Wenyu Liu, and Qi Tian. Msg- transformer: Exchanging local spatial information by manipulating messenger tokens. In CVPR, pages 220721 2022.
[14] Shaohua Guo, Liang Liu, Zhenye Gan, Yabiao Wang, Wuhao Zhang, Chengjie Wang, Guannan Jiang, Wei Zhang, Ran Yi, Lizhuang Ma, et al. Isdnet: Integrating shallow and deep networks for efficient ultra-high resolution segmentation. In CVPR, pages 4361-4370 2022.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[16] Chuong Huynh, Anh Tuan Tran, Khoa Luu, and Minh Hoai. Progressive semantic segmentation. In CVPR, pages 16755-16764, 2021.
[17] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani. Nikita Orlov, and Humphrey Shi. One- former: One transformer to rule universal image segmentation. In CVPR, pages 2989-29999 2023.
[18) Jitesh Jain, Anukriti Singh, Nikita Orlov, Zilong Huang, Jiachen Li, Steven Walton, and Humphrey Shi. Semask: Semantically masked transformers for semantic segmentation. In ICCV, pages 752-761, 2023.
[19] Deyi Ji, Feng Zhao, and Hongtao Lu. Guided patch-grouping wavelet transformer with spatial congruence for ultra-high resolution segmentation. In IJCAI, pages 920-928, 2023.
[20] Deyi Ji, Feng Zhao, Hongtao Lu, Mingyuan Tao, and Jieping Ye. Ulta-high resolution segmentation with ultra-rich context: A novel benchmark. In CVPR, pages 23621-23630, 2023,
[21] Davood Karimi, Guy Nir, Ladan Fazli, Peter C Black, Larry Goldenberg, and Septimiu E Salcudean. Deep learning-bar based gleason grading of prostate cancer from histopathology images-rou role of multiscale decision aggregation and data augmentation. JBHI,, 24(5):1413 1426, 2019.
[22] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, al. Segment anything. arXiv preprint 23X1v:2304.02643, 2023.

---

[23] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. CVPR, pages 9799-9808, 2020.
[24] Qi Li, Weixiang Yang, Wenxi Liu, Yuanlong Yu, and Shengfeng He. From contexts to locality: Ultra-high resolution image segmentation via locality-aware contextual correlation. ICCV, pages 7252-7261, 2021.
[25] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, pages 12009-12019, 2022.
[26] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pages 3431-3440, 2015.
[28] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segelip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In ICML, pages 23033-23044. PMLR, 2023.
[29] Lei Ma, Yu Liu, Xueliang Zhang, Yuanxin Yc, Gaofei Yin, and Brian Alan Johnson. Deep learning in remote sensing applications: A meta-analysis and review. ISPRS, 152:166-177, 2019.
[30] Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez. Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark. In IGARSS, pages 2229.2229. IEEE, 2017.
[31] Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang. Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis, 89: 102918, 2023.
[32] Gaurav Menghani. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Computing Surveys, 55(12):1-37, 2023.
[33] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE TPAMI, 44(7):3523- 3542, 2021.
[34] Tuan Due Ngo, Binh-Son Hua, and Khoi Nguyen. Isbnet: 3d point cloud instance segmentation network with instance-awe aware sampling and box-aware dynamic convolution. In CVPR, pages 15550-13559, 2023.
[35] Yaolei Qi, Yuting He, Xiaoming Qi, Yuan Zhang, and Guanyu Yang. Dynamic snake convolution based on topological geometric constraints for tubular structure segmentation. In ICCV, pages 4070-6079, 2023.
[36] Muhammad Imran Razzak, Saceda Naz, and Ahmad Zaib. Deep learning for medical image processing: Overview, challenges and the future. Classification in BioApps: Automation of Decision Making, pages 323-350, 2018
[37] Dinggang Shen, Guorong Wu, and Heung-ll Suk. Deep learning in medical image analysis. Annu Rev Biomed Eng, 19:221-248, 2017.
[38] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, pages 5693-5703, 2019.
[39] Hans Thisanke, Chamli Deshan, Kavindu Chamith, Sachith Seneviratne, Rajith Vidanaarachchi, and Damayanthi Herath. Semantic segmentation using vision transformers: survey. FAAI, 126: 106669, 2023.
12

---

[40] Xin-Y Yi Tong, Gui-Song Xia, and Xiao Xiang Zhu. Enabling country- scale land cover mapping with meterr-esolution satellite imagery. ISPRS Journal of Photogrammetry and Remote Sensing. 196:178- 1011178-9669 196, 2023.
[41] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The Nam10000 dataset, large collection of multi soure- dermatoscopic images of common pigmented skin lesions. SCI DATA, 5(1):1-9, 2018.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 30, 2017.
[43] Sai Wang, Yutian Lin, and Yu Wu. Omni-q: Omni-directional scene understanding for unsuper- vised visual grounding. In CVPR, pages 14261-14270, 2024.
[44] Yi Wang, Syed Muhammad Arsalan Bashir, Mahrukh Khan, Qudrat Ullah, Rui Wang, Yilin Song, Zhe Guo, and Yilong Niu. Remote sensing image super-resolution and object detection: Benchmark and state of the art. Expert Syst. Appl, page 116793, 2022.
[45] Tong Wu, Zhenzhen Lei, Bingqian Lin, Cuihua Li, Yanyun Qu, and Yuan Xie. Patch proposal network for fast semantic segmentation of high-resolution images. In AAAI, volume 34, pages 12402-12409, 2020.
[46] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NeurlPS, 34: 14:12077- 2021.
[47] Shuo Xu, Sai Wang, Xinyue Hu, Yutian Lin, Bo Du, and Yu Wu. Mac: A benchmark for multiple attributes compositional zero-shot learning. arXiv preprint arXiv:2406. 12757, 2024.
[48] Huang Yao, Rongjun Qin, and Xiaoyu Chen. Unmanned aerial vehicle for remote sensing applications-a review, Remote Sens, 11(12):1443, 2019.
[49] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In ECCV, pages 325-341, 2018.
[50] Haitao Yuan, Sai Wang, Zhifeng Bao, and Shangguang Wang. Automatic road extraction with multi-source data revisited: completeness, smoothness and discrimination. Proceedings of the VLDB Endowment, 16 (6(11))00043017, 2023.
[51] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li. Jianwei Yang, and Lei Zhang, simple framework for open-vocabulary segmentation and detection. In ICCV, pages 1020-1031, 2023.
[52] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, pages 288 1-2900, 2017.
13

---

A Existing Architecture Analysis
In Figure 7. we show the detail architectures of existing ultra image segmentation methods. Whole inference 7(a) and slide inference 7(b) are the two most fundamental ultra image segmentation architectures. While whole inference can be applied directly to general segmentation models [27, 38, 46, 12, 49], it utilizes the down-sampled image as input, which results in a loss of essential information. The latter divides the image into multiple patches by means of sliding window, and predicts the patches one by one, aggregating the results together. However, due the isolated prediction of patches and the lack of global information, it is prone to cause information bottleneck and fragmentation phenomenon.
Ultra Image
Ultra Image
Resize
Slide window
Resized image
Crop patches
General Model
General Model
Prediction
Prediction
(a) Whole Inference
(b) Slide Inference
Ultra Image
Ultra Image
Ulm Image
Resize
Resize
Slide window
Slide window
Resized image
Crop patches
Resizod image
Crop patches
Global Branch
Local Branch
Deep Branch
Shallow
Shallow Branch
General Model
Fusson Medule
Fusion Module
Fusion Module
Prediction
Prediction
Prodiction
(c) Global & Local
(d) Shallow & Deep
(e) Ours
Figure 7: Comparison of existing architectures of ultra image segmentation.
To address the issues above and leverage the benefits of both whole inference and slice inference, some studies [6, 45, 24, 16, 19] have introduced the Global & Local architecture 7(c). This architecture involves one branch that takes in the down-sampled ultra image as the global cue, while the other branch extracts local information from sliced patches. However, these tasks usually require sequential processing of each patch and complex integration strategies to combine global information, resulting in poor scalability of the architecture. Alternatively, some study [14, 20] have introduced the Shallow & Deep architecture 7(d), which inputs the original image and the resized image into different branches for processing. Despite its fast speed, is essentially a trade-off in dataset size and computing resources, and cannot be used for real ultra image segmentation scenarios.
The architectures depicted in Figure 7(c) and Figure 7(d) both use the original image after resize in an input branch, which may limit the capability of the architecture to process ultra images. On the one hand, when the image is extremely large, the resized image will inevitably lose a lot of detail information to fit the GPU memory. On the other hand, the resized ultra image cannot guarantee to provide specific global information that is helpful for local patch segmentation. To address these issues, we designed novel and effective ultra image segmentation architecture 7(c) based on slide
14

---

window. Our architecture can handle ultra images of any scale and provide surrounding information that aids in local patch segmentation. Furthermore, our architecture highly flexible, not restricted by computing resources, and can be seamlessly integrated to any encoder-decoder based segmentation model.
B
Dataset Details
To comprchensively evaluate our method, we conduct experiments on five public ultra image datasets involving general, medical and remote sensing scenarios:
Cityscapes [10]. The Cityscapes dataset is a popular street scene dataset for generic semantic segmentation. It contains 3475 images with the resolution of x024x 2048 and a total of 19 categories. We use 2975 images for training and 500 images for testing.
Gleason [21]. The Gleason dataset is a high resolution medical image dataset with the resolution of 5120x5120. It contains 244 H&E-stained histopathology images for automatic Gleason grading of prostate cancer. Since we do not have data partition details from [16]. we randomly split dataset into training and testing set with 195 and 49 images.
DeepGlobe [11]. DeepGlobe is satellite image dataset that contains 803 ultra-high resolution images (2448 2448 pixels). It contains 7 categories, of which the class named "unknown" is excluded. Following [6], we divide the training, validation, and testing sets into 454, 207, and 142 images, respectively.
Inria Aerial [30]. The Inria Aerial dataset comprises 180 images, each with a resolution of 5000x5000 pixels. Following [6], we divide the training, validation, and testing sets into 126, 27, and 27 images, respectively.
Five-Bil Billion-Pixels [40]. The FBP dataset includes 150 high- reghlutsolttion images, each with size of 7200 x 6800 pixels and labeled with 24 categories. We use the same test set as [40], and randomly divide the remaining images into training set and validation set, including 90 and 30 images.
C Comparative Analysis of SCB Branch
In order to demonstrate the proposed SCB Branch's 's efficacy, we used sixteen conventional convolution layers followed by four transformer blocks to form a trivial replacement branch for extracting surrounding image feature. It serving as a functionally analogous replacement for the proposed SCB branch. As shown in Table 8, we added this trivial replacement branch to all general segmentation models in Table 1 and conducted experiments the DeepGlobe dataset using the same settings. The results show that our proposed SCB branch significantly outperforms this branch, proving the efficacy of our proposed component.
Table 8: Comparison of SCB Branch with trivial replacement.
Original
Trivial Branch
SGNet
FCN
72.38
72.56
72.56(+0.28) (+0.28)
75.28 (+2.90)
DeepLab V3Plus
73.22
73.77 (+0.55)
75.44 (+2.22)
HRNet
72.87
73.24 (+0.37)
75.25 (+2.38)
SegFormer
72.96
73.56 (+0.60)
74.65 (+1.69)
STDC
72.59
72.88 (+0.29)
74.51 (+1.92)
D Large Scale Human Subject Segmentation Study
We conducted experiments on the well-known CelebAMask-HHe dataset to further verify the effec- tiveness of our method in large scale human subject segmentation. Due to the lack of extremely high-resolution human datasets, we simulate ultra-high resolution by resizing the images from the CelebA Mask- Mask-HQ dataset from 1024 to 2448 pixels. We compared our SGNet with DeepLabV3Plus which is highly popular and widely used image segmentation model across various domains. The performance of our method significantly outperformed DeepL.abV3PI by 1.61% % (from 62.93% mloU to 64.54% mloU). We provide more examples of qualitative results in Figure 8.
15

---

Image
Ground Truth
SGNet
DeepLab Plulus
Figure 8: The visualization of different methods in CelcbAMask-HOO. We resized the images from 1024 2448 pixels to simulate ultra-high resolution and used sliding window of size $12 without overlap inference.
E More Qualitative Result
In Figure 9, we provide more examples of qualitative results between our method and existing ultra image segmentation methods like FCtL [24] and ISDNet [14]. These results indicate that our method capable of achieving satisfactory quality on various challenging datasets [11, 30]. Figure 10 and Figure 11 illustrate the effectiveness of SCB by comparing to general segmentation models such as DecpLab and HRNet on Cityscapes [10] and Gleason [41].
16

---

(a) Image
(b) Ground Truth
(c) FCIL
(d) ISDNNt
(c) Ours
Figure 9: The qualitative results of different methods in DeepGlobe (lop row, 2448 x 2448) and Inria Aerial (bottom row, 5000 x 5000).
(a) Image
(b) Ground Truth
(c) DeepLab
(d) DeepLab SCB
(e) HRRNet
(I) HRNett + SCB
Figure 10: The comparison of adding SCB on different models in Cityscapes (2048 1024).
(a) Image
(b) HRNet
(c) DeepLab V3Plus
(d) Ground Truth
(e) HRNet + SCB
(1) DeepLab + SCB
Figure 11: The comparison of adding SCB on different models in Gleason (5120x5120).
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims accurately summarize the our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Conclusion, we discuss the limitations of the proposed method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the necessary information required to replicate the main experi- mental results, including the model architecture and experimental settings.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: We have included our core code in an anonymized zip as part of the supple- mentary, and will release the complete code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines Pottps://nise cc/
public/ guides/ CodeSubmi ss casioooollc for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: i ps cc/ hccppbll c/gu des / CodeSubmi i ss onPol 1 ayy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are provided in Sec. 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 4.2, we provided detailed information about the specific computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:// /ttprri////urpps../ http://ppp..cctttttccso....oooo.
Answer: [Yes]
Justification:
We have conducted.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: discuss societal impacts in conclusion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all the original papers that provided the code and dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers' naww aw. thcode com/ data setss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "i3oSArlxjJ", "reviewer": "Reviewer_1ux8", "summary": "This paper focus on the generalization and architectural issues of the ultra image segmentation methods, and proposes SGNet which consists of two branches for processing surrounding patch and local patch, respectively. The motivation is to leverage the surrounding context for refining the segmentation results of local patches. The results show a significant improvement over previous methods. Overall, the paper is well-written with good results.", "questions": "(1) What kind of backbone network is used in surrounding branch? Is it the same as that in local branch? Does the surrounding branch resize its input?\n\n(2) A sliding window without overlap is used for inference, does this (with or without overlap) make a difference to the gain of the method?\n\n(3) According to Figure 1, the surrounding branch’s output is discarded during inference. Is it possible to further improve the performance by combining it with the result of the local branch?\n\n(4) There is a typo at last of the caption of Figure 1.", "limitations": "The limitations and social Impact have been discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper is well-written and the figures and tables are well-organized and clear. \n\n(2) The analysis of problems in the previous work is clear and logical, which is easy to follow.\n\n(3) The proposed method is simple and effective and shows advantages on various datasets with various general segmentation models.", "weaknesses": "(1) The description of the surrounding branch is not clear enough (see question (1)).\n\n(2) The evaluation of speed was insufficient (only included one of the lightest versions, i.e., ISDNet-Style). More adequate results could help to better understand the overhead of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W2: The evaluation of speed was insufficient (only included one of the lightest versions, i.e., ISDNet-Style). More adequate results could help to better understand the overhead of the method.\n\nWe have given the full-version (SGNet) speed comparison in Table 6 of the original manuscript (L281-299). It is attached to the DeeplabV3Plus model as backbone.\n\nSome works, such as ISDNet, directly take the entire image or a downsampled version for prediction to achieve faster speed, but face the architectural issue (L36-41). When the image is excessively large (e.g., 7200x6800), the former (entire image input) is infeasible due to limited GPU memory, while the latter (downsampled image) suffers significant information loss during compression, making them unsuitable for ultra image segmentation in real-world scenarios.\n\n> Q1: What kind of backbone network is used in surrounding branch? Is it the same as that in local branch? Does the surrounding branch resize its input?\n\n(1) The surrounding branch and local branch use different backbones.\nWe employ a lightweight backbone for the surrounding branch, i.e., the first four layers of STDC, in order to boost processing speed (L120-121, L190-191).\n\n(2) The surrounding branch does not resize the input, which helps to better enhance spatial details.\nIt always maintains the original resolution, which is different from previous works that use downsampled images and suffer information loss when scaled up to extremely large resolution.\n\n> Q2: A sliding window without overlap is used for inference, does this (with or without overlap) make a difference to the gain of the method?\n\nAll the methods reported in Table 1 of the original manuscript are sliding window without overlap, ensuring a fair comparison (L189-190).\n\nA sliding window with overlap is essentially a test time augmentation method, where multiple predictions on overlapping regions are averaged to enhance the model's performance.\nBy using overlapping regions that cover half of the input patch for predictions, our method can further improve the performance by 0.24 (from 75.44 mIoU to 75.68 mIoU).\n\n> Q3: According to Figure 1, the surrounding branch’s output is discarded during inference. Is it possible to further improve the performance by combining it with the result of the local branch?\n\nYes, it will.\n\nWhen combining the logits maps from the surrounding branch and the local branch, the performance is further improved by 0.15 (from 75.44 mIoU to 75.59 mIoU) on the DeepGlobe dataset.\nThe improvement in results essentially belongs to a model ensembling method.\nThis indicates that our surrounding branch has learned effective and complementary information to the local branch, further demonstrating the validity of our approach.\n\n> Q4: There is a typo at last of the caption of Figure 1.\n\nThank you for pointing out this issue and we have fixed it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mx8xzLPe87", "reviewer": "Reviewer_MJRN", "summary": "To overcome the challenges in generalization and compatibility with real-world ultra images, SGNet revisits a classic sliding inference approach and incorporates a surrounding context module to refine local patch segmentation. SGNet is compatible with various segmentation models and achieves significant performance improvements.", "questions": "See Weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The authors address ultra image segmentation from the perspective of general segmentation models. This is an interesting approach to the problem.\n* The proposed SGNet leverages surrounding context information to enhance the segmentation results obtained from analyzing image patches.\n* This framework can be integrated with various general segmentation models and achieves competitive performance across different general segmentation models on five publicly available datasets.", "weaknesses": "1. The authors emphasize in lines 29 to 35 that general segmentation models possess superior generalization capabilities. However, from the results presented in Table 1, it can be observed that even general segmentation algorithms exhibit significant performance fluctuations across different datasets. This is particularly evident on the Cityscapes dataset.\n2. Regarding the experiments detailed in Table 1, several general segmentation methods outperform existing specialized ultra image segmentation approaches on DeepGlobe, FBP, and Aerial Inria datasets without introducing the proposed SGNet.\n    1. The authors attribute their performance superiority to the generalizability or architecture. However, they overlook the impact of different inference modes. Based on the description provided in the paper, all general segmentation methods employ sliding inference. However, this is not consistent across existing methods. This discrepancy precludes a direct conclusion that \"general segmentation algorithms have better generalization.\"\n    2. Given the diversity of inference modes, the authors should supplement Table 1 with specific inference settings for each method to identify more critical factors.\n3. The authors mention in line 489 that due to the inaccessibility of the data partition details from [1], they conducted their own dataset division. Yet, in Table 1, only ISDNet [4] presents results for Gleason. This is logically inconsistent. Did the authors retrain ISDNet using your own data split?\n4. The validation of the proposed component's efficacy is not rigorous. Since the component is pluggable, the performance evaluation should compare two variants: 1) removing the proposed component directly as shown in Table 1, and 2) replacing it with a trivial operation of similar function, such as some simple convolutional layers with the same number of parameters, or components with analogous functions from existing methods. The latter was overlooked by the authors.\n5. The design of the surrounding context integration module appears strikingly similar to that in [2]. The authors should provide a detailed comparison.\n6. There is a citation error for the Five-Billion-Pixels dataset. It should be referenced as [3].\n7. For the experiments in Table 2:\n    1. The order is illogical. It would be natural to integrate the alignment operation into the model before introducing SCI.\n    2. The boundary consistency loss yields a limited performance gain.\n8. What is the specific architecture of the first model variant listed in Table 3?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: The authors emphasize that general segmentation models possess superior generalization capabilities, but they still exhibit significant performance fluctuations.\n\nSorry for misleading.\n\nWe use the term \"generalization ability\" to represent a model's capability to scale up to larger resolutions, rather than performance fluctuations across datasets.\nOur intention was to highlight that ultra image segmentation (UIS) methods struggle to scale up to larger image sizes, rather than the variance of the same method across different datasets.\nAs image resolution increases (e.g., from 2448x2448 to 6800x7200), the performance of UIS methods significantly degrades (GLNet, a typical UIS method, drops from 71.60 to 42.05, due to processing the entire image information).\nIn contrast, our method is designed to handle extremely high resolutions effectively and is less affected by image size (e.g., from 75.44 at 2448x2448 resolution to 63.18 at 6800x7200 resolution), making it more suitable for extremely large image.\nIn the revised version, we will use more precise terms to avoid ambiguity.\n\n> W2: The inference mode of general segmentation methods is not consistent across existing UIS methods and affects the performance.\n\nWe have supplemented Table 1 of the original manuscript with specific inference modes for each method (see attached pdf) and extracted UIS results on the DeepGlobe dataset as follows.\n\nMost UIS methods in Table 1 use sliding window inference, so there is no issue with inconsistency in inference modes.\nFrom the comparison of the nine UIS methods, the sliding window mode does not have an advantage over whole inference.\nIn addition, we reproduced the results of ISDNet by changing from whole inference to sliding window inference, and its performance even dropped.\nMoreover, the sliding window mode is more suitable in real applications, as it can scale up to extremely large images.\n\n| Method     | Mode        | DeepGlobe |\n| ---------- | ----------- | --------- |\n| ISDNet     | Whole       | 73.30     |\n| WSDNet     | Whole       | 74.10     |\n| ElegantSeg | Whole       | 74.32     |\n| ISDNet\\*   | Slide       | 68.82     |\n| FCtL       | Slide       | 72.76     |\n| GLNet      | Slide+Whole | 71.60     |\n| PPN        | Slide+Whole | 71.90     |\n| MagNet     | Slide+Whole | 72.96     |\n| GPWFormer  | Slide+Whole | 75.80     |\n\nTable 1: Comparison with UIS methods. \"\\*\" represents results we reproduced in the sliding window mode.\n\n> W3: Did the authors retrain ISDNet using your own data split.\n\nYes, we retrained ISDNet using the same data split as ours.\n\nWe apologize for the omission of specific experimental settings. ISDNet is the state-of-the-art model among all the **_open-sourced_** methods.\nWe followed the original settings of ISDNet and retrained it on the Gleason dataset using our own data split.\n\n> W4: Extra experiments using trivial operations.\n\nThank you for pointing out this issue. We have supplemented this comparative experiment to demonstrate the proposed component's efficacy in the below table.\nSpecifically, we used sixteen conventional convolution layers followed by four transformer blocks to form a trivial replacement branch for extracting surrounding image feature, serving as a functionally analogous replacement for the proposed SCB branch.\n\nWe added this trivial replacement branch to all general segmentation models in Table 1 of the original manusript and conducted experiments on the DeepGlobe dataset using the same settings.\nThe results show that our proposed SCB branch significantly outperforms this branch, proving the efficacy of our proposed component.\n\n|               | Original | + Trivial replacement branch |    + SGNet    |\n| :-----------: | :------: | :--------------------------: | :-----------: |\n|      FCN      |  72.38   |        72.56 (+0.28)         | 75.28 (+2.90) |\n| DeepLabV3Plus |  73.22   |        73.77 (+0.55)         | 75.44 (+2.22) |\n|     HRNet     |  72.87   |        73.24 (+0.37)         | 75.25 (+2.38) |\n|   SegFormer   |  72.96   |        73.56 (+0.60)         | 74.65 (+1.69) |\n|     STDC      |  72.59   |        72.88 (+0.29)         | 74.51 (+1.92) |\n\n> W5: Provide a detailed comparison with [2].\n\nSorry, we don’t know which paper [2] refers to. Could you provide the specific title? We are happy to provide more detailed comparisons.\n\n> W6: Citation error for the Five-Billion-Pixels dataset.\n\nThank you for pointing out this error.\nWe have updated the reference to \"Enabling country-scale land cover mapping with meter-resolution satellite imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 196:178–196, 2023.\"\n\n> W7: Ablation order and the performance of the boundary consistency loss.\n\n(1) Thank you for pointing out this issue. We conducted an experiment using the backbone with the alignment operation and achieved a 73.88 mIoU.\nWe will add this comparison in the final version.\n\n(2) This is an auxiliary loss, which we did not list as a contribution. We initially hoped would encourage consistency in predictions across different branches for the same region. In practice, its constraint effect under supervised conditions is relatively limited.\n\n> W8: The architecture of the first model in Table 3.\n\nThanks for remindering.\nThe first model in Table 3 uses six residual layers (Conv-BN-ReLU) to replace the attention module. It takes feature maps divided into non-overlapping regions as input, allowing for simple and direct integration of surrounding features.\nIn contrast, using attention mechanisms and global interaction allows us to selectively learn contextual information that benefits local patch segmentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zJukvCFkwH", "reviewer": "Reviewer_HefE", "summary": "The paper introduces SGNet to address the limitations of existing UIS methods, i.e., generalization issues and architectural constraints. SGNet leverages a larger context around image patches to refine segmentation results, making it compatible with general segmentation models and enhancing their performance on high-resolution images. The framework is validated on five diverse datasets, demonstrating significant improvements in segmentation accuracy and efficiency.", "questions": "see weakness", "limitations": "see weakness", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This method proposes SGNet to address two intrinsic problems in UIS.\n(2) The method is evaluated on a wide range of datasets, including Cityscapes, DeepGlobe, Inria Aerial, Five-Billion-Pixels, and Gleason, showcasing its effectiveness across various scenarios, including medical and remote sensing applications.", "weaknesses": "(1) The challenges in Contribution 1 have been proposed in many segmentation tasks. Therefore, C1 is weak.\n(2) The contribution of the SGNet is incremental, including the consistency loss and the local and global branches.\n(3) If the author wants to verify the SGNet is plug-and-play, the author should compare it with other plug-and-play methods rather than compare it with other task-specific strategies.\n(4) The computational cost is expensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: The challenges in Contribution 1 have been proposed in many segmentation tasks. Therefore, C1 is weak.\n\nThe contribution 1 is: \"We excavate two essential but largely overlooked issues in UIS, which hold great value for the community. In addressing these challenges, we are the first to tackle the ultra image segmentation task from the general segmentation model perspective\" (L65-67).\nThe two issues represent the **Generalization issue** and **Architectural issue**.\n\nPrevious works have all been specialized UIS methods (Generalization issue). We are the first to address the task of ultra image segmentation from the perspective of general segmentation model.\nThe architectures of previous works are not suitable for handling images of extremely large scales in real-world scenarios (Architectural issue).\n\nTherefore, the two issues presented in Contribution 1 are unique to the task of ultra image segmentation.\nAs far as we know, we are the first to propose the above viewpoint. Please provide any reference if we miss.\n\n> W2: The contribution of the SGNet is incremental, including the consistency loss and the local and global branches.\n\nNot agreed.\n\nThe contribution of SGNet is that we are the first to address ultra image segmentation from the perspective of general segmentation models, capable of handling images of extremely large scale.\nIt is plug-and-play, can be integrated into any general segmentation model (we tested five different GSS models), and has consistently improved performance across multiple datasets.\n\nThis has been epoched by the other reviewers that our method is \"an interesting approach to the problem\" (Reviewer-MJRN) and \"quite novel\" (Reviewer-xcUS). They noted that SGNet \"is simple and effective\" (Reviewer-1ux8), and \"the biggest plus\" (Reviewer-xcUS) is that it \"can be integrated with various general segmentation models\" (Reviewer-MJRN).\n\n> W3: If the author wants to verify the SGNet is plug-and-play, the author should compare it with other plug-and-play methods rather than compare it with other task-specific strategies.\n\nOur proposed SGNet can be integrated into any segmentation model, and we have conducted experiments on segmentation models with five different models, including CNN, Transformer, and lightweight models architectures (L202-204), to demonstrate that our method is \"plug-and-play\".\n\nOur paper targets the ultra image segmentation task.\nThus, we proposed SGNet to migrate general segmentation models to this task due to its better scale-up ability.\nTherefore, it is appropriate for us to compare our method with other UIS task-specific methods.\n\nWe are the first to offer a \"plug-and-play\" method for ultra image segmentation. As far as we know, there are no \"other plug-and-play methods\" for this task.\n\nPlease list any reference if you found.\n\n> W4: The computational cost is expensive.\n\nOur main speed bottleneck lies in the sliding window architecture.\nAlthough using the entire image as input processes the image only once,\nit can only handle images of limited size.\nIn contrast, our method processes the same image multiple times but it is the only way to handle images of extremely large scales (L36-43).\nWe also develop a fast version for a fair comparison with those methods that emphasize speed, and we surpass the comparison methods in both speed and performance (L292-299).\n\nFurthermore, we believe that the UIS community prefer prioritizes usability and quality in real-world scenarios over speed requirements like real-time.\nIn practical applications, there are numerous engineering optimization techniques, such as model quantization and multi-GPU parallelism, that can enhance speed.\n\nAt the same time, UIS methods typically have more complex architectures and present deployment challenges (L29-32).\nIn contrast, our method can be seamlessly integrated into general segmentation models, making deployment faster and easier.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tSZF6qb7nx", "reviewer": "Reviewer_xcUS", "summary": "The paper discusses the challenges of ultra image segmentation and proposes a Surrounding Guided Segmentation framework (SGNet) to address these challenges by leveraging surrounding context information.", "questions": "1) In equation 7 for the loss function, the paper uses 3 parameters, where we could only do with 2 after normalization. \n2) It is typical to represent L1 loss with a subscript for clarity, so I would add it in equation 6.\n3) The paper uses the word performance in several instances to discuss quality. It is an important distinction, as the usage of performance should be limited to quantify speed, and quality should be used for discuss the improvements in segmentation mask quality.\n4) Table 6 discusses the memory footprint of the method, but it is quite unclear how this is measured or what it represents. Is it peak memory usage? Average memory usage during inference? \n5) In the efficiency study, it is unclear what device is the method evaluated on.\n6) \" The method can handle images of any scale.\", this is only partially true as you are still going to be restricted by your compute resources. This needs to be clarified.", "limitations": "The discussion on limitations is extremely limited. Can this method be used for matting for instance? Or very thin peninsular regions? Will it improve or reduce the quality of segmentation if the images have jpeg artifacts? Can it be used on difficult objects like jewelry? These questions could be discussed in the limitations section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) Novelty: The paper identifies two specific issues to ultra image segmentation, namely generalization and architectural issues, and proposes a solution to solve both. The proposed SGNet is quite novel and the biggest plus is that it can be used alone with any segmentation model to improve the quality of the results. This can be beneficial in a variety of image segmentation applications.\n2) Evaluation: The proposed method has been validated on a variety of datasets, which helps build confidence in the method. The method also consistently seems to outperform previous methods.\n3) The contribution resulting from each individual part of the module has been clearly evaluated, which explains the efficacy of each.\n4) The method can handle images of any scale.\n\nOverall it seems to be a solid paper in my opinion with sufficient novelty.", "weaknesses": "1) For some reason, there seems to be no evaluation done on human subjects specifically. I would have liked to see some segmentation results on human subjects, especially around fingers, ear etc.\n2) The discussion on limitations is extremely limited. Please read my comment in that section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: I would like to see results on human subjects.\n\nWe aim to address the task of ultra image segmentation and common academic datasets are typically in remote sensing or medical imaging (L183-186).\nTo further verify the effectiveness of our method in large scale human subject segmentation, we conducted experiments on the well-known CelebAMask-HQ dataset[1].\n\nDue to the lack of extremely high-resolution human datasets, we simulate ultra-high resolution by resizing the images from the CelebAMask-HQ dataset from 1024 to 2448 pixels.\nWe compared our SGNet with DeepLabV3Plus, which is a highly popular and widely used image segmentation model across various domains.\n\nThe performance of our method significantly outperformed DeepLabV3Plus by 1.61 (from 62.93 mIoU to 64.54 mIoU).\nPlease refer to our attached pdf for visualization.\n\n[1]: Lee, Cheng-Han, et al. \"Maskgan: Towards diverse and interactive facial image manipulation.\" CVPR. 2020.\n\n> Q1: In equation 7 for the loss function, the paper uses 3 parameters, where we could only do with 2 after normalization.\n\nThank you for your suggestion. Yes, it can be reformulated with two hyper-parameters. We will revise this function in the final version.\n\n> Q2: It is typical to represent L1 loss with a subscript for clarity, so I would add it in equation 6.\n\nThank you for pointing out this issue. We have revised Equation 6 to the following form: ${L} _ {consistency}=||{P}^{'} _ {global}-{P} _ {local}{||} _ {1}$.\n\n> Q3: The paper uses the word performance in several instances to discuss quality.\n\nWe apologize for the inaccurate use of the terms 'performance' and 'quality'. We have revised the relevant sections and will update the final version.\n\n> Q4: Explanation of 'memory' in Table 6，peak or average?\n\nApologies for the missing explanation. 'Memory' refers to the maximum GPU memory consumption during model inference. We measure GPU memory using the command line tool 'gpustat', following previous works such as ISDNet.\n\n> Q5: what device is the method evaluated on.\n\nWe evaluated on a Tesla V100 GPU with a batch size of 1.\n\n> Q6: The method can handle images of any scale.\", this is only partially true as you are still going to be restricted by your compute resources. This needs to be clarified.\n\nNot agreed.\n\nOne of our highlights is the ability to handle images of any scale. Our approach only uses sliding window mode for predictions, making it independent of computational resource limitations. Our method uses the same amount of GPU memory for handling 2448x2448 and 6800x7200 images.\n\n> L1: Could this method be used for image matting, very thin peninsular regions, or difficult objects like jewelry? Will it improve segmentation quality with JPEG artifacts？ Need to be discussed in the limitation section.\n\n(1) Our method is primarily designed for ultra image segmentation, with its core approach utilizing surrounding information to guide local patch segmentation in the sliding window mode.\n\n(2) For image matting tasks, accurate segmentation of instance boundaries is crucial. Our method, leveraging surrounding information, effectively enhances boundary segmentation quality, as demonstrated in Figure 6 of the original manuscript. It is also applicable to matting for ultra image and addresses the issue of inconsistent foreground instances between adjacent image patches.\nMoreover, our method can be seamlessly integrated into most image matting models.\n\n(3) Our method is suitable for handling narrow peninsula regions, which also appear in the DeepGlobe and FBP datasets.\n\n(4) We compressed the DeepGlobe dataset to 10\\% of its original image quality using JPEG compression and retrained SGNet and DeepLabV3Plus on it.\nThe results show that SGNet significantly outperforms DeepLabV3Plus by 3.92 (from 64.16 mIoU versus 60.24 mIoU), and this improvement is almost twice that of normal images (from 73.22 mIoU to 75.44 mIoU).\nThis indicates that our method is relatively insensitive to noise compared to baseline models and can use surrounding information to infer damaged pixel information within the object. It also demonstrates that our method is particularly effective in scenarios involving JPEG compression.\n\n(5) Our work is independent of the difficulty of the objects and is more focused on their performance in ultra images. Thus we may not be different from other models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We supplemented Table 1 of the original manuscript with specific inference modes for each methods (@Reviewer-MJRN).

And we have also presented the visualization results on the CelebAMask-HQ dataset (@Reviewer-xcUS).

Please refer to the attached pdf.

Meta-review: According to 3 out of 4 reviewers, this paper makes a solid contribution to the area of segmenting ultra highres images, and presents convincing results to show improvement over the state-of-the-art. One reviewer is an outlier, with a low score (3) and a very very short review. The authors alledge that review was written by an LLM. While the Area Chair was not able to validate that claim, the review is rather unprofessional and written in vague terms, well below the level of detail of the other reviews. Therefore the Area Chair largely discounted it and recommends this paper for acceptance.
Decision: Accept (poster)