Title: Toward Real Ultra Image Segmentation: Leveraging Surrounding Context to Cultivate General Segmentation Model
Venue: NeurIPS 2024
Accepted: yes
Keywords: Ultra Image Segmentation
Abstract: Existing ultra image segmentation methods suffer from two major challenges, namely the scalability issue (i.e. they lack the stability and generality of standard segmentation models, as they are tailored to specific datasets), and the architectural issue (i.e. they are incompatible with real-world ultra image scenes, as they compromise between image size and computing resources).
To tackle these issues, we revisit the classic sliding inference framework, upon which we propose a Surrounding Guided Segmentation  framework  (SGNet) for ultra image segmentation. 
The SGNet leverages a larger area around each image patch to refine the general segmentation results of local patches.
Specifically, we propose a surrounding context integration module to absorb surrounding context information and extract specific features that are beneficial to local patches. Note that, SGNet can be seamlessly integrated to any general segmentation model.
Extensive experiments on five datasets demonstrate that SGNet achieves competitive performance and consistent improvements across a variety of general segmentation models, surpassing the traditional ultra image segmentation methods by a large margin.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the challenges of ultra image segmentation and proposes a Surrounding Guided Segmentation framework (SGNet) to address these challenges by leveraging surrounding context information.\n\nSTRENGTHS:\n1) Novelty: The paper identifies two specific issues to ultra image segmentation, namely generalization and architectural issues, and proposes a solution to solve both. The proposed SGNet is quite novel and the biggest plus is that it can be used alone with any segmentation model to improve the quality of the results. This can be beneficial in a variety of image segmentation applications.\n2) Evaluation: The proposed method has been validated on a variety of datasets, which helps build confidence in the method. The method also consistently seems to outperform previous methods.\n3) The contribution resulting from each individual part of the module has been clearly evaluated, which explains the efficacy of each.\n4) The method can handle images of any scale.\n\nOverall it seems to be a solid paper in my opinion with sufficient novelty.\n\nWEAKNESSES:\n1) For some reason, there seems to be no evaluation done on human subjects specifically. I would have liked to see some segmentation results on human subjects, especially around fingers, ear etc.\n2) The discussion on limitations is extremely limited. Please read my comment in that section.\n\nQUESTIONS:\n1) In equation 7 for the loss function, the paper uses 3 parameters, where we could only do with 2 after normalization. \n2) It is typical to represent L1 loss with a subscript for clarity, so I would add it in equation 6.\n3) The paper uses the word performance in several instances to discuss quality. It is an important distinction, as the usage of performance should be limited to quantify speed, and quality should be used for discuss the improvements in segmentation mask quality.\n4) Table 6 discusses the memory footprint of the method, but it is quite unclear how this is measured or what it represents. Is it peak memory usage? Average memory usage during inference? \n5) In the efficiency study, it is unclear what device is the method evaluated on.\n6) \" The method can handle images of any scale.\", this is only partially true as you are still going to be restricted by your compute resources. This needs to be clarified.\n\nLIMITATIONS:\nThe discussion on limitations is extremely limited. Can this method be used for matting for instance? Or very thin peninsular regions? Will it improve or reduce the quality of segmentation if the images have jpeg artifacts? Can it be used on difficult objects like jewelry? These questions could be discussed in the limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-01"}

comments: SUMMARY:
The paper introduces SGNet to address the limitations of existing UIS methods, i.e., generalization issues and architectural constraints. SGNet leverages a larger context around image patches to refine segmentation results, making it compatible with general segmentation models and enhancing their performance on high-resolution images. The framework is validated on five diverse datasets, demonstrating significant improvements in segmentation accuracy and efficiency.

STRENGTHS:
(1) This method proposes SGNet to address two intrinsic problems in UIS.
(2) The method is evaluated on a wide range of datasets, including Cityscapes, DeepGlobe, Inria Aerial, Five-Billion-Pixels, and Gleason, showcasing its effectiveness across various scenarios, including medical and remote sensing applications.

WEAKNESSES:
(1) The challenges in Contribution 1 have been proposed in many segmentation tasks. Therefore, C1 is weak.
(2) The contribution of the SGNet is incremental, including the consistency loss and the local and global branches.
(3) If the author wants to verify the SGNet is plug-and-play, the author should compare it with other plug-and-play methods rather than compare it with other task-specific strategies.
(4) The computational cost is expensive.

QUESTIONS:
see weakness

LIMITATIONS:
see weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo overcome the challenges in generalization and compatibility with real-world ultra images, SGNet revisits a classic sliding inference approach and incorporates a surrounding context module to refine local patch segmentation. SGNet is compatible with various segmentation models and achieves significant performance improvements.\n\nSTRENGTHS:\n* The authors address ultra image segmentation from the perspective of general segmentation models. This is an interesting approach to the problem.\n* The proposed SGNet leverages surrounding context information to enhance the segmentation results obtained from analyzing image patches.\n* This framework can be integrated with various general segmentation models and achieves competitive performance across different general segmentation models on five publicly available datasets.\n\nWEAKNESSES:\n1. The authors emphasize in lines 29 to 35 that general segmentation models possess superior generalization capabilities. However, from the results presented in Table 1, it can be observed that even general segmentation algorithms exhibit significant performance fluctuations across different datasets. This is particularly evident on the Cityscapes dataset.\n2. Regarding the experiments detailed in Table 1, several general segmentation methods outperform existing specialized ultra image segmentation approaches on DeepGlobe, FBP, and Aerial Inria datasets without introducing the proposed SGNet.\n    1. The authors attribute their performance superiority to the generalizability or architecture. However, they overlook the impact of different inference modes. Based on the description provided in the paper, all general segmentation methods employ sliding inference. However, this is not consistent across existing methods. This discrepancy precludes a direct conclusion that \"general segmentation algorithms have better generalization.\"\n    2. Given the diversity of inference modes, the authors should supplement Table 1 with specific inference settings for each method to identify more critical factors.\n3. The authors mention in line 489 that due to the inaccessibility of the data partition details from [1], they conducted their own dataset division. Yet, in Table 1, only ISDNet [4] presents results for Gleason. This is logically inconsistent. Did the authors retrain ISDNet using your own data split?\n4. The validation of the proposed component's efficacy is not rigorous. Since the component is pluggable, the performance evaluation should compare two variants: 1) removing the proposed component directly as shown in Table 1, and 2) replacing it with a trivial operation of similar function, such as some simple convolutional layers with the same number of parameters, or components with analogous functions from existing methods. The latter was overlooked by the authors.\n5. The design of the surrounding context integration module appears strikingly similar to that in [2]. The authors should provide a detailed comparison.\n6. There is a citation error for the Five-Billion-Pixels dataset. It should be referenced as [3].\n7. For the experiments in Table 2:\n    1. The order is illogical. It would be natural to integrate the alignment operation into the model before introducing SCI.\n    2. The boundary consistency loss yields a limited performance gain.\n8. What is the specific architecture of the first model variant listed in Table 3?\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper focus on the generalization and architectural issues of the ultra image segmentation methods, and proposes SGNet which consists of two branches for processing surrounding patch and local patch, respectively. The motivation is to leverage the surrounding context for refining the segmentation results of local patches. The results show a significant improvement over previous methods. Overall, the paper is well-written with good results.

STRENGTHS:
(1) The paper is well-written and the figures and tables are well-organized and clear. 

(2) The analysis of problems in the previous work is clear and logical, which is easy to follow.

(3) The proposed method is simple and effective and shows advantages on various datasets with various general segmentation models.

WEAKNESSES:
(1) The description of the surrounding branch is not clear enough (see question (1)).

(2) The evaluation of speed was insufficient (only included one of the lightest versions, i.e., ISDNet-Style). More adequate results could help to better understand the overhead of the method.

QUESTIONS:
(1) What kind of backbone network is used in surrounding branch? Is it the same as that in local branch? Does the surrounding branch resize its input?

(2) A sliding window without overlap is used for inference, does this (with or without overlap) make a difference to the gain of the method?

(3) According to Figure 1, the surrounding branch’s output is discarded during inference. Is it possible to further improve the performance by combining it with the result of the local branch?

(4) There is a typo at last of the caption of Figure 1.

LIMITATIONS:
The limitations and social Impact have been discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12