AIMS: All-Inclusive Multi-Level Segmentation for Anything
Authors: Lu Qi
Jason Kuen
Weidong Guo
Jiuxiang Gu
Zhe Lin
Bo Du
Yu Xu
Ming-Hsuan Yang
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Image Segmentation

Abstract: Despite the progress of image segmentation for accurate visual entity segmentation, completing the diverse requirements of image editing applications for different-level region-of-interest selections remains unsolved. In this paper, we propose a new task, All-Inclusive Multi-Level Segmentation (AIMS), which segments visual regions into three levels: part, entity, and relation (two entities with some semantic relationships). We also build a unified AIMS model through multi-dataset multi-task training to address the two major challenges of annotation inconsistency and task correlation. Specifically, we propose task complementarity, association, and prompt mask encoder for three-level predictions. Extensive experiments demonstrate the effectiveness and generalization capacity of our method compared to other state-of-the-art methods on a single dataset or the concurrent work on segment anything. We will make our code and training model publicly available.

AIMS: All-Inclusive Multi-Level Segmentation for Anything
Lu Qi' Jason Kuen? Weidong Guo
Jiuxiang Gu²
Zhe Lin"
Bo Du
Yu Xu
Ming-Hsuan Yang
UC Merced 2 Adobe Research
SQQ Brower Lab, Tencent
4 Wuhan University
Google Research
Abstract
Despite the progress of image segmentation for accurate visual entity segmentation, completing the diverse requirements of image editing applications for different- level region-of-interes selections remains unsolved. In this paper, we propose a new task, All-Inclusive Multi-Level Segmentation (AIMS), which segments visual regions into three levels: part, entity, and relation (two entities with some semantic relationships). We also build unified AIMS model through multi-dataset multi-task training address the two major challenges of annotation inconsistency propose task complementarity, association,
and task correlation. Specifically, and prompt mask encoder for three-level predictions. Extensive experiments demonstrate the effectiveness and generalization capacity of our method compared to other state-of-the-ar methods on a single dataset or the concurrent work on segment anything. We will make our code and training model publicly available.
Introduction
Decomposing an image into semantically meaningful regions a key goal of image segmentation. With the recent emergence of photorealistic and high-quality image generation technologies [1, 2], image segmentation has gained considerable attention for its ability select visual entities of interest within images. Combining image segmentation with image generation has enabled variety of image editing and manipulation applications, such as altering the style of specific object or eliminating a distracting element from a scene. Compared to bounding box-based object detection, segmentation methods preserve the integrity of pixels outside the edited or manipulated region.
Panoptic segmentation [3], with its instance awareness and dense segmentation capabilities, is main limitations remain when using it: (1) it struggles to handle classes encountered 'in the wild' but absent in the training set [4, 5]: and (2) existing methods typically do not explore the more general possibilities of segmentation masks across multiple abstraction levels (e.g., car's 's wheel, the whole car, both a person and car combined), with the exception of part-aware panoptic segmentation [6. 7] and panoptic scene graph [8]. These tasks also fail to generalize to classes beyond those in the training set, especially the pert-level task where annotations are sparse.
promising solution for various editing and manipulation applications. However,
In this paper, we address the constraints inherent to different-level segmentation tasks by introducing AIMS (All-Inclusive Multi-Level Segmentation). AIMS represents comprehensive framework designed to segment images across various hierarchical levels (part, entity, relation) using multiple query-driven image decoders. Drawing inspiration from recent advancements in Entity Segmen- tation [4. 5], which showeased remarkable generalization capabilities on unseen classes, AIMS performs class-agnostic segmentation across different hierarchical levels enhance generalization.
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

One-stop Infereace
Awexralion
ParbLeud 82)
Entlyy-
Relation-
Prompt Inference
Relelion-lo- Fabil
Figure 1: The overview our All-Inclusive Multi-Level Segmentation (AIMS) task.
Thanks its hierarchical modeling and class-agnostic learning approach, AIMS provides more level-flexible and all-inclusive segmentation masks for users to select from, thereby enabling wider range of image editing and manipulation applications. In Figure I, the AIMS task has two modes of inference: one-step inference and prompt inference. In one-step inference, the model segments the image directly into three levels (part, entity, and relation level). Mcanwhile, in prompt inference, the model separates the image further based on the provided mask. Notably, one-step inference necessitates an association of the results between any two neighboring levels.
Unlike existing part-aware panoptic or panoptic scene graph methods, AIMS model is not confind training on single dataset that has limited hierarchical levels. Instead, we adopt the strategy of multi-dataset training to build unified AIMS model for all levels. To maintain consistency across multiple hierarchical levels sourced from different datasets, we introduce the Task Complementarity Module and the Association Module to AIMS model. These respectively facilitate feature interactions between different levels and establish explicit relationships between predictions at adjacent levels (e.g., a car's part mask should correspond with the overall car mask, and car mask should belong to a mask with car and driver.).
We train our AIMS model on existing segmentation datasets such as Pascal Panoptic Parts [6], COCO- PSG [8], PACO [9], and EntitySeg [5]. However, due to partial or incomplete annotations over the full image, with inconsistencies in unannotated areas across different datasets, sub-optimal supervision signals are generated during training. To address this issue, we propose a novel Mask Prompt Encoder for our AIMS model. This encoder segments an image based on a given mask prompt, which specifies region slated for segmentation. We train AIMS accommodate a diverse set mask prompts, derived from various hierarchical levels and datasets while excluding unannotated regions. This encoder offers two distinct benefits: (/) it mitigates the inconsistency problem inherent in training with partial annotations from multiple datasets; (2) it exhibits strong generalizability to arbitrary mask prompts during inference, including those stemming from unseen classes and in-the-wild visual entities, shown in Figure 5.
The main contributions of this work are follows:
We introduce the AIMS task, which segments images into three distinct levels: part, entity, and relation. AIMS is capable performing a one-step inference for the entire image and prompt inference for specified mask area within the image, which has great potential for image editing and manipulation applications.
We propose a unified AIMS model for this task, which incorporates multi-dataset and multi-task training. We have meticulously designed several modules, including the task complementary, association, and mask prompt encoder, to maintain the consistency of tasks or data annotations.
Extensive experiments and in-depth analysis showcase the effectiveness of our proposed AIMS model across various segmentation tasks and settings. We compare the results from our AIMS model with the popular segmentation models (e.g., SAM[10]), demonstrating the fine-grained segmentation capabilities and generalization to both seen and unseen categories.

---

lurgg Proupt
Prompt
All-inclusive segmentation for Anything
Encoder
Image Decoder
Expp-an
Rdiake-kdd
Image
Encuder
Figure 2:
The outline of our AIMS framework encompasses an image encoder, a mask prompt encoder, and an image decoder. The outputs generated by AIMS consist of five distinct components, including segmentation results across three levels, and two association metrics.
2 Related Work
Significant advancements [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21] have been made in image segmentation [22, 23, 3, 24, 25, 26] in recent years, with numerous methods and techniques emerging to handle various challenges through supervised learning. Notable works include FCN [11], DeepLab [12]. PSPNet [13], Mask R-CNN [14], PANet [15]. SOLO [16], CondInst [17], PanopticFPN [18], PanopticFCN [19], Max- Deeplab [20], and Mask2Former [21] for semantic, instance, and panoptic segmentation. Additionally, specialized tasks tailored to specific scenarios, such as part- pamoptic segmentation [6. 27, 7, 28] and panoptic scene graphs [8, 29], have been developed. However, these tasks typically focus single target and do not incorporate hierarchical concepts that are more in line with human behavior. This paper aims to address this problem by proposing unified segmentation model that can organize pixels three different levels: part, entity, and relation. We also introduce an interactive stage to decompose unseen regions into smaller entities or parts in an open-world setting.
Image Segmentation
Multi-Dataset Training Multi-dataset training has gained considerable attention in recent years, as it allows models to draw from multiple data sources to improve performance and generaliza- tion [30, 31, 32]. This approach has significantly benefited domain adaptation [33, 34, 35]. dataset distillation [36, 37], curriculum learning [38, 39, 40, 41], data augmentation [42, 43, 44], federated learning [45, 46], and transfer learning [47, 48] in the context of image segmentation. This paper addresses the annotation inconsistency among multiple datasets for generalized hierarchical segmen- tation. Specifically, we tackle the issue of incomplete annotations single dataset across three
levels using a prompt structure.
3 Method
As shown in Figure 1, given image E x3 the AIMS task segments image into three segmentation maps Sy E {0, Nr at part, entity, and relation levels and two association maps Agp E {0, NNN. (entity and part levels) and ARE E N).1)N.x.N, (relation and entity levels), where and W are the height and width of the image, N2 indicates N₂ prediction results class- agnostic binary format and T = {p, e, r]. The two association maps AFP and ArE required to build the association relationships between two level prediction results. We chose these three levels since they align with the human brain's hierarchical information processing system [49, 50, 51] and fulfill the majority of requirements for image editing tools [52]. The outputs of AIMS are akin to those of related tasks like part-aware panoptic segmentation and panoptic scene graphs, but the AIMS task incorporates more levels and operates in a class-agnostic manner.
Like the existing Transformer-based segmentation methods [21]. our AIMS model employs an image encoder and decoder to extract robust image features and decode them into segmentation masks. However, it differs in that it addresses the challenges of multi-datane illustrated in Figure 3, we first establish a task complementarity module and an association module to build the correlation of each task, followed by mask prompt encoder tackle the partial annotation problem among different datasets and enable generalized inference.
multi-task training. As
Next, we introduce our base framework and then elaborate our three proposed components.

---

Imuge Decoder
Task Correiation Module
Reletion-eee
Decoder
Infationieesll Dereder
FPN
Complrarata Medale
FFS
Complementarty
Association
Muse
E-P E.PCoiiirnnattll
Deccdeer
CONCAT
SA
SKir
Pari-bas Decedar
Complementarity
Aunciation
FFN
Association Module
Qp
thx
Decoder
Complommmtariy Madilo
Figure 3: Overview of image decoder of our All-Inclusive Multi-Level Segmentation framework. The terms 'E-P Correlation Module' and 'E-P Association Module' refer the task correlation and association structure between the entity and part level, respectively. This explanation can also be applied to the RRE Correlation Module' and 'R-E Association Module". The rightward arrows denote the outputs from the respective modules. "CONCAT" or 'SPLIT" refers to the joining of two tensors or the division of a single tensor into two tensors, respectively, occurring along the dimension the query number.
3.1 Base Framework
The base framework of AIMS shares the same image encoder O and has three independent decoders for different- predictions. The image encoder consists of backbone and pixel decoder, extracting multi-scale image features F, process as:
IR " x256 and
{2, 4, 5). We denote this image-encoding
F. = (I
(1)
For the three independent decoders Q, each decoder has 9 cascaded attention blocks, where each block includes masked cross- aoss-attention, self-attention. and feedforward network. Each block of decoder iteratively refines query embeddings ET E RR x256 by interacting with multi-scale features F, where Er is initialized by group of learnable parameters Qr. This iterative decoding process is denoted as
ET PT Fs)
II i=1
E
F
II
Ri, (AS (RxAur (E Fs)))
where 4r and i indicate the decoder of T type and its attention blocks. indicates the 2th attention block. For each attention block, it includes cross-attention R self-attention R and feed- forward network R At last, ET used for part-ness/entity-o neseettitynnss/rlltttoon ness prediction and pixel-level mask prediction using the low-level image feature F2 (derived from the image encoder). The prediction process can be formulated as:
U U = PredHeady F2)
where U and Up denote the part-ness/entity www.enttttynnsssssaaaoonnneeeeeo prediction and pixel-level mask outputs.
During training, we employ the losses Lr for different level predictions, with respect to the corre- sponding ground truth GT. The overall training loss for our model is defined as:
Lhase
C U G
C
(U
G
+ Cdice (
G
[E(P...)
where L denote binary cross-entropy loss for part-/entity-rrelttie and Cdice denote the binary cross-entropy and dice loss for different level mask prediction.
prediction. Similarly, Lhee

---

Level
coco PPP COCOLPSG PACO Eniityeg
Information
COCO
PPP
COCO-PSG
PACO
EnritySeg.
Relation
Annotation Area
91.7%
90.4%
99.9%
Entity
Category Numbers
133
20
133
634
Part
Image Numaeers
115k
9.7K
46k
35k
32k
(a) Table I: The statistics of the datasets used for training including COCO [53], EntitySeg [5], PascalVOC: Part (PPP) [6], PACO [9), and COCO-PSG [8]. The (a) indicates whether the dataset has
(b)
the annotations for some level. The (b) statisfy the information of each dataset.
3.2 Task Complementarity Module
In our base method, we utilize three independent decoders for predictions different hierarchy levels, which merely share the image encoder. To better exploit cross-task knowledge, we design the task complementarity module (TCM) that fuses task information within each attention block of the decoders. To provide clearer explanation, we use the task complementarity module between the entity and part tasks as an example, and it works identically for the relation and entity tasks. As depicted in Figure 3, we concatenate the embeddings of two tasks following the self-attention of the attention block between two level decoders. Subsequently, we feed the concatenated embedding into self-attention GAsa and reew-fr network SAn and then separate processed embeddings back into their original dimensions:
E E
(SS EE E,, 1)))
(5)
where the Ee, and Ep, denote the embedding processed by self-attention in the attention block. After that, we acquire the final embedding by performing a straightforward addition between the fused and original embeddings:
E = E E E.p.. E.P + E
(6)
Ultimately, feed the enhanced embeddings each level into the original feedforward network of the corresponding decoder:
El+1 Rip (ET,,))
3.3 Association Module
Aside from generating predictions at various levels, the AIMS model also strives to establish the association between low-level and higher-level results. For example, the part masks a car should be explicitly associated with the entire car's 's mask. In image editing and manipulation, this can provide more intuitive mask selection experience. While it possible to associate the masks across different levels by computing mask intersection, such association knowledge cannot be instilled into the model. Instead, we introduce an association module to AIMS to learn cross-level associations by applying an association loss to the similarity between embeddings. Besides having the association graph as an outcome, this also helps the model be aware the cross-level association relationships and thus benefiting the individual-level task performance. The association loss Lasso is defined as:
(8) where FC indicates a fullly-ommected layer and denotes dot product. The Gep E (0, N|N,NNp and Gre E {0, 1 N.N. are the binary association ground truth of entity-part and relation- eenttty levels. Finally, the overall training loss for AIMS
Lasso Lice (FF(EE))((C((E)))) Gep) + Lhce (FC(E,) (FC(E.)) Gre)
L Lrase
Lasso
3.4 Mask Prompt Encoder
As aforementioned, we train AIMS with multiple existing segmentation datasets that each covers one or more hierarchy levels, whose details are provided in Table Due the use of predefined classes, most of the datasets suffer from the problem of partial annotations and have unannotated areas on the images. Furthermore, the unannotated areas appear and happen inconsistently among the multiple datasets (e.g., 91.6% of PACO is unannotated, while it is 0.01 % for EntitySeg). This leads to undesirable supervision signals that confuse the model because something that is treated as background in one dataset/level may qualify foreground in another dataset/level.
5

---

Prongt
Target
Prongt
Target
Prunpt
Terget
Full Image
Fusly
Multiple Entilies
Part
Eailyy
Relution
Doooddee
Deodkke
One Entity
Part
Two Entilies
Feilyy
DOoOER
Ingut Image
Figure 4: Illustration of the task prompt types with different combinations between mask prompts and decoders. each sub-figure, the right part is our target of the network.
We design the Mask Prompt Encoder (MPE), which allows our method to segment images conditioned on specific mask cant-contdolled task prompts. As depicted in Figure 4, our approach integrates four types of mask prompts: full image, single entity, dual entities and multiple entities (where the upper limit is the entire annotated area). By employing diverse mask prompts along with multi-level mask decoders, AIMS can accomplish different segmentation goals. The mask prompts, covering either the full or partial image area, drive the model to segment the entire image target area at part, entity, or relation levels using the corresponding decoders. This approach enables to provide the model with trustworthy supervision signal for the region to be segmented, even when dealing with datasets that lack full image annotations. Morcover, during inference, the mask prompt encoder can be applied to unrestricted mask prompts. To some extent, the model can subdivide the masks of unseen classes and/or in-the-wild visual entities into reasonable sub-masks, even they are absent training data.
In particular, the mask encoder accepts binary mask Mp, which designates the area to be segmented. Our mask encoder utilizes series of cascaded convolution layers downscale Mp, resulting in M2 4 W. xCm.
M MaskEncoder(MHy
(10)
where the detailed structure of the MaskEncoder consists of three operation blocks including CONV2+LN+GELU, CONV2+LN+GELU and CONV1. The 'CONV2', 'CONVI', 'LN', and 'GELU" represent convolution layer with stride and a kernel size of 2 layer normalization. and Gaussian error linear units, respectively. Subsequently, we directly downscale M2 by 2x, 4x, and 8x obtain M, where E 3, 4, 5, using bilinear interpolation.
Ultimately, we acquire multi-scale fusion features F's by summing the encoded image and mask features Fs and M.. respectively. For AIMS with MPE, replace F, in Eq. 2 with F.;
F = F M,
(11)
Given MPE, we delve deeper into the sampling strategy implemented across various mask prompts and subtasks. To effectively balance contributions from each subtask and prompt type, adopt uniform sampling approach capable of considering multiple factors, such as subtask difficulty and the distribution of different prompt types within the training data. Specifically, in each training iteration, we have five subtasks: segmenting the entire annotated area (either the full image or multiple entities) based three hierarchy levels, subdividing single entity parts, and subdividing two merged entities with relationships to two independent entities. Additionally, each subtask has its own data pool for sampling, taking into account the statistical annotation information of each dataset. For more details, please refer to our supplementary file.
4 Experiments
We construct our training set by aggregating images from five segmentation datasets, including COCO [53], EntitySeg [5], PascalVOC Part (PPP) [6], PACO [9], and COCO-PSG [8]. Given that

---

IPPerence(PPP)
Inference(COCOO
Method
Train Data
Backbone
AP
AP
APTs
AP
APto
AP?;
AR
APM
AP AP
AP
AP
AP AR
PPP [6]
24.3
54.3
18.1
48.6
70.0
49.8
65.8
25.7
53.1
34.1
COCO-PSO [8] (Sec. 3,1)
Baselino
Swin Tan
44.5
65.6
46.1
39.6
59.7
41.3
41.7
67.3
424
5188
354
182
534
75.4
69.7
389
59.0
65.0
41.2
AIMS
All
26.5
59.0
202
56.1
78.5
58.5
723
403
60.6
423
42.
67.
43.
53.
Swin-Largo
29.3
63.0
23,1
60.0
81.6 63.1
75.6
44.2
65.1
45.3
46.8
70.4
48.7
56.9
Table 2: Comparison among single dataset training, baseline, and our proposed method AIMS on multi-dataset training.
each dataset was originally created for a distinct task, we split each dataset into training and evaluation, and subsequently merge them to form unified training and evaluation sets. Overall, our training and evaluation sets have about 236.7K and 2.1K images. Please refer to our supplementary file for details on the split criteria. With the exception of COCO and EntitySeg, which are solely for entity-level prediction with full or partial image mask prompts, PPP dataset is utilized for entity/part prediction with full or partial image mask prompts and the entity-to-pai -orrttt task. COCO-PSG is employed for relation/entity prediction with full or partial image mask prompts and a relation-to-entiee keep only the relationship annotations in COCO-PSG that involve physical contact, be consistent with typical part-whole relationships. PACO dataset is exclusively used for the entity-to setitytto-part task.
task. We
Training settings. We train our model for 36,000 iterations using a base learning rate of 0.0001 and weights pre-trained on COCO-Entity [3] with the exception of images contained in our validation set. The longer edge size of the images is set 1,333 pixels, while the shorter edge size is randomly sampled between 640 and 800 pixels, with a stride of 32 pixels. The learning rate is decayed by factor of 0.1 after 28,000 and 33,000 iterations, respectively. During each training iteration, sample the data and tasks as introduced in the sampling strategy of Section 3.4, with batch size of 64 on 8 A100 GPUs.
Federated evaluation. We build our evaluation set by collecting images from PPP [6] and COCO- PSG [8] dataset, where each image only has two level annotations. Similar to LVIS [46], our evaluation images have non-ex -ohauuxaastive annotations. To better approach this, we organize the eval- uation set into two subsets: one for entity- part and another for relation-entity. For each subset, evaluate our model by the instance segmentation metric mean average precision (mAP) for each level and recall AR @ 100 for cross-level association predictions. For brief illustration, APP, APe, and AP" are mAP for part-, entity- and relation- predictions. The A @ 100 and ART @ 100 are metrics for entity-part and relation-entity association performance.
In the following, we first ablate our experiments with a Swin-Tiny [54] backbone and then report the quantitative performance and visualization results of our best model with the Swin-Large [54] backbone to other ofett-e----rrr methods.
4.1 Ablation Study
Multi-dataset training. Table 2 demonstrates the overall performance improvement by introducing multiple datasets. In the 1st and 2nd rows, we directly apply the base framework (baseline) with single dataset training and separate decoders. From the inference results, we observe that the model trained on the PPP dataset yields lower performance the entity level when evaluated on the COCO-PSG dataset. This is primarily due to the fact that the COCO-PSC dataset contains 4 times more images than the PPP dataset, which does not provide sufficient entity-level annotations for the model. This issue can be alleviated through multi-dataset training, as depicted in the third row.
When we train our baseline method with all datasets, there are slight improvements or even degra- dation in part-level prediction for the PPP dataset, and relation and entity-level prediction for the COCO-PSG dataset. This can be attributed to annotation inconsistencies and incompleteness among multiple datasets, which results in ambiguous training signals for the model. By exploiting task complementarity (with Task Complementarity Module) and mitigating annotation issues (with MPE) .our proposed AIMS framework can improve performance on both evaluation datasets.
Improvement trajectory of the proposed method. In Table 3(a), we show the component wise improvements over the baseline. The 1st and last rows represent the performance of our baseline (three separate decoders for three levels) and the full AIMS framework. The middle three rows display the performance enhancement over the baseline method when integrated with different modules. It is evident that both MPE and TCM contribute to improvements in performance across all three
7

---

PPP
COCO-PSG
MPE TCM AM
AP"
AP
PPP
AR
AP
COCO-PSG
AP
AR
FUM PAM REM EPM
APE
AR
APR
AP AR
24.5
53.2
69.6
38.9
40.3
51.0
24.3
53,4
69.7
38.9
40.4
50.9
24.5
53.4
69.7
38,9
40.4
50,9
25.7
54.8
71.0
39.5
41.2
52.0
0
24.3
25.9
53.1
55.0
70.6
71.2
39.0
39.9
40.5
41.6
52.0
52.3
V
V
25.6
51.8
54.9
70.4
35.8
39.6
41.3
52.1
20.4
49.7
62.3
32.5
35.6
44.8
V
26.5
56.1
72.3
40.5
42.1
53.1
26.5
56.1
72.3
40.5
42.1
53.1
(8)
(b)
Table 3: Ablation studies. (a) Improvement trajectory of the proposed modules. The abbreviations "MPE', 'TCM', and AM' represent the mask prompt encoder, task complementarity module, and association module, respectively. (b) Ablation study concerning the utilization of mask prompt types. The terms 'FUM', 'PAM', 'REM', and 'EPM' denote the mask types for the full image, partial image with multiple entities, dual entities' mask, and single entity mask, respectively. The checkmark symbol (V) and the circle symbol (o) indicate whether the ablated module is used or not.
PPP
COCO PSG
PPP
COCO PSG
Structure
APÊ
AR
APR
AP
AR
SA FF ADD
APÊ
AR
APR APÊ
ARRE
53.4
38.9
50,9
53.4
50,9
(CONNI
26.5
56.1
72.3
40.5
42,1
53.1
25.5
54.8
70.7
39.1
40.9
51.7
(CONVIXXIHCONVIIK) x2)
26.4
56.1
72.3
40.5
42.1
53.1
V
25.7
54.3
70.8
39.7
41.2
52,3
(
25,9
54.7
71.1
39.6
41.6
52.5
26.5
56.1
72.3
40.5
42.1
53.1
(h)
Table 4: Ablation studies. (a) The structure design of prompt mask encoder. The 'CONV2B' and 'CONVI' indicates convolution block operation with kernel and stride size 2 followed by layer normalization and GELU and convolution with kernel and stride size 1. The x2 means the two sequential structures. (b) The structure design of the task correlation module. The terms 'SA', 'FF' and 'ADD' denote self-attention feed-forward, and addition respectively.
levels (mAP and association recall) on the two evaluation subsets, highlighting the effectiveness of these proposed modules all levels. With the addition of the association module, more significant improvement in association performance can be achieved while maintaining similar level of segmentation performance compared the baseline. This indicates that our proposed association module does not negatively impact the segmentation performance. Furthermore, all three modules mutually benefit each other, giving the best performance shown in the last row.
Mask Prompt Encoder Table 3(b), we demonstrate the impact of using different mask prompt types during training. Note that only full-image mask prompts are used during inference. The 1st and 2nd rows show that full-image mask prompts do not bring any impact. In the third row, solely using a mask prompt based on the partial annotation area of the image results in a significant performance decline due to the mask prompt inconsistency between the training and inference stages, because inference only uses full-image prompts. However, when both full and partial image prompts used, as shown in the 4th row, better performance achieved by simultancously mitigating the incomplete annotation problem and maintaining maining-inference consistency. As shown by the last row, the introduction of dual-entity or single-entity mask prompts further improves performance, by pushing immediate adjacent levels (relation-to-
the model to learn better how to split masks between entities & entity-to- to-parts). This is different than (but complements) using full- or -intall-imaee mask prompts which impose large gaps between the source and target levels (e.g.. image-to to-tttts).
In Table 4(a), we examine the influence of various mask encoder architectures The 1st row represents our baseline without prompt mask encoder. As demonstrated in the 2nd and 4th rows, we find that adding a 'CONV2' and 'CONVI' block with convolution kernel and stride sizes of 2 and is sufficient for the mask encoder architecture. Incorporating more 'CONVI blocks does not lead to a significant improvement in the final performance, as shown in the third row. This because a single "CONVI" layer is enough to encode mask information after the feature that has been already processed by two 'CONV2' blocks, as evident in the last two rows.
Task Complementarity Module. Table 4(b) presents the ablation study on the design of TCM. It evident that the self-attention block capable of fusing information from each level enhance each respective task, as demonstrated in the 2nd row. Moreover, adding the original level-specific embeddings back fused embeddings is erucial for improving the final performance. This because the level-specific embeddings carries unique information specific to each task.
8

---

Prompt Mask
SAM
Prompt Mask
SAM
Ous
Figure 5: Qualitative comparison of SAM [10] and our method in interactive segmentation. The prompt mask indicates entity mask that is provided the network for further separation the next low level. For the SAM model, we use hyperr-aaaaeetess for more fine- tine-grained segmentation, which will be introduced in our supplementary material. The images in the first row and the last two rows are from PACO [9] (in-domain) and ImageNet [56] (out-of-domain)
Method
PPP PQ ParPO PWO 20 60 R/ 100
COCO-PSG
Method
Train VG- (5-)
Data
AR
AR
PACO
AR
AR AR*
ADE20K
PanoopicPatFormsree PSG/R [8]
68.3
Eniily Sey
28.2/15.4
32.1/20.3 35.3/21.5
SAM [10]
I1M
86.2 $2.3
82.3 49.7
Ours
36.7/22.9
39.4/25.6 42.3/28.2
Ours
237K
67.4
52.9
83.0 50.4
Table 5: Ablation studies. (a) Comparison with state-of-the-ar (SOTA) methods in panoptic part segmentation and panoptic scene graph tasks. (b) Performance comparison of segmentation gener- alization for images in the real world. The AR AR and AR represent recall @ 100 at relation, entity, and part levels, respectively.
4.2 Comparison to state-of-the-art methods
In the following comparison, we evaluate our model against other state-of-the-ard methods. To ensure fair comparison, all methods employ Swin-Large backbone, with the exception of SAM, which uses a ViT-Huge backbone [55].
Category-awars segmentation. We transfer our AIMS method to other class- aware segmentation tasks, including panoptic part segmentation and panoptic scene graph on PPP and COCO-PSG datasets, respectively. To equip our method with class-aware capability, we directly use our AIMS model as pre-trained weights fine-tune the corresponding class-aware training dataset. Table 5(a), we observe that our AIMS model outperforms the corresponding sfate-of-the-art methods on both tasks. This because our AIMS model can utilize more training data from various datasets by employing proposed prompt mask encoder to address annotation inconsistencies among them. Moreover, our AIMS model benefits from the two-stage process of class-agnostic pretraining and class-aware fine-tuning [38], which has been demonstrated to improve the performance of the second stage class-aware fine-tuning when class-agnostic pretraining is performed with more data.
Interactive segmentation. In Table 5(b), we compare AIMS to another popular interactive seg- mentation method, SAM [10], on the PACO and ADE20K [57] datasets, where ADE20K for zero-shot evaluation at entity and part levels. Neither SAM nor AIMS uses ADE20K for training, and ADE20K's 's classes are fairly dissimilar to PassalVOC's [58]. The row represents our baseline method with purely class-agnostic training without interactive segmentation, which outperforms SAM at the pure entity level even though the SAM model is trained on 11 million images. This indicates that the zero-shot generalization ability is attributed to class-agnostic rather than interactive training. On the other hand, SAM's interactive segmentation can divide masks into more fine- fine-grained segments, which our AIMS is also capable of. On both evaluation datasets, AIMS remarkably outperforms SAM, using merely a 273K training set that is far smaller than SAM's 1M training set. Our proposed MPE and TCM allow model to leverage more desirable training signals and multi-level task information relation-level predictions, which higher than part and entity levels and absent in the SAM model.
lata-efficient manner. Furthermore, our AIMS model can make
Figure provides a visual comparison between the SAM [10] model and our model real-world scenario. Like AIMS, the SAM model also supports image segmentation three levels (object,

---

(9)
Dhe
One Eadiy Pouter
Eatle Mark
Figure 6: The illustration of the flexibility of AIMS tasks to tackle the subjective annotation issues in existing datasets.
part, and sub-part level). For an equitable comparison, we use the same entity mask prompt for further delineation. As shown in Figure 5, both models capably segment parts of the entity with distinct approaches. The SAM model leans towards identifying parts based on color and texture clues, whereas our AIMS model relies more on semantic understanding.
The flexibility of our AIMS task. Figure illustrates how our proposed AIMS task provides the flexibility for segmenting anything. This is to address the subjective annotation issues in existing datasets, For instance, the image (a), the throw pillows and the sofa are predicted as a single entity, which aligns with the ground truth annotation. Nonetheless, in certain scenarios, user might wish to edit the throw pillows independently. With our AIMS method, utilizing a full image prompt for part-level prediction does not yield these separated masks. However, by selecting this entity mask as an entity prompt for part-level prediction, we can successfully differentiate the three throw pillows.
Additionally, in image (b), user may want to segment the windows into several components. However, original ground truth annotations typically consider the two windows as single entity. initially utilize a full image prompt and entity encoder to identify
To meet this requirement, the mask of the whole windows. Following this, we apply this two-entity mask prompt and relation Decoder to split them into two independent window masks. Ultimately, we can select one window further segmentation into two parts: the window edge and curtains.
Conclusion
This paper presents AIMS, unified segmentation model that parses images at three levels: part, entity, and relation. To address the absence of a dataset with annotations for all three levels, we construct a unified dataset by aggregating multiple existing datasets, each providing annotations for one two levels. Our base method uses three separate decoders for different-leve predictions. To handle the annotation inconsistency issue. we propose Mask Prompt Encoder, which offers more accurate task signal to the model, and Task Complementarity Module to enhance each level's prediction. Extensive experiments demonstrate the effectiveness of our method in both closed-world and zero-shot settings. Our work can serve as springboard for further research on this new segmentation problem.
6 Acknowledgement
This research is based upon work supported by the Office of the Director National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract 1022-2110210000
No. 2022-21 102 The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies endorsements, either expressed or implied, of the ODNI. IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.
10

---

References
[1] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. In arXiv, 2022.
[2] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. NeurIPS, 2022.
[3] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019. 3,
[4] Lu Qi. Jason Kuen, Yi Wang, Jiuxiang Gu, Hengshuang Zhao, Zhe Lin, Philip Tomr, and Jiaya Jia. Open-world entity segmentation. TPAMI, 2022.
[5] Lu Qi. Jason Kuen, Weidong Guo, Tiancheng Shen, Jiuxiang Gu, Wenbo Li, Jiaya Jia, Zhe Lin, and Ming-Hsun Hing-Hsuan Yang. Fine-grained entity segmentation. 2022. 6, 9,
[6] Daan de Geus, Panagiotis Meletis, Chenyang Lu, Xiaoxiao Wen, and Gijs Dubbelman. Part-aware panoptic segmentation. In CVPR, 2021. 2, 6, 7,
[7] Xiangtai Li, Shilin Xu, Yibo Yang, Haobo Yuan, Guangliang Cheng, Yunhai Tong, Zhouchen Lin, and Dacheng Tao. Panopticpartfrmer+:: A unified and decoupled view for panoptic part segmentation. In arXiv, 2023. 1,3,9
[8] Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph generation. ECCV, 2022. 1, 3, 7, 9,
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes common objects. In arXiv, 2023. 2, 5, 6, 9, 14
[10] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Speneer Whitchead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv, 2023. 9, 14, 19
[11] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. CVPR, 2015.
[12] Liang-Chiee Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. In TPAMI, 2017. 3
[13] Hengshuang Zhao, Jianping Shi. Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017. 3
[14] Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick. Mask r-enn. ICCV, 2017. 3
[15] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmenta- tion. CVPR, 2018.
[16] Xinlong Wang. Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In ECCV, 2020. 3
[17] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV, 2020. 3
[18] Alexander Kirillov, Ross Ginshick, Kaiming He, and Piotr Dollár, Panoptic feature pyramid networks. In
CVPR, 3019. 3
[19] Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. Fully convolutional networks for panoptic segmentation. In CVPR, 2021.
[20] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In CVPR, 2021. 3
[21] Bowen Cheng, Ishan Misra, Alexander G Schwing. Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 3
[22] Hongshan Yu, Zhengeng Yang, Lei Tan, Yaonan Wang, Wei Sun, Mingui Sun, and Yandong Tang. Methods and datasets semantic segmentation: review. Neurocomputing, 2018.
11

---

[23] Abdul Mueed Hafiz and Ghulam Mohiuddin Bhat. A survey on instance segmentation: state of the art. International journal of multimedia information retrieval, 2020. 3
[24] Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia, Amodal instance segmentation with kins dataset. In CVPR, 2019. 3
[25] Tiancheng Shen, Yucchen Zhang, Lu Qi, Jason Kuen, Xingyu Xie, Jianlong Wu, Zhe Lin, and Jiaya Jia. High quality segmentation for ultra high-resolution images. In CVPR, 2022. 3
[26] Liulei Li. Tianfei Zhou, Wenguan Wang, Jianwu Li, and Yi Yang. Deep hierarchical semantic segmentation. In CVPR, 2022. 3
[27] Xiangtai Li, Shilin Xu, Yibo Yang. Guangliang Cheng, Yunhai Tong, and Dacheng Tao. Panoptic-
partformer: Learning a unified model for panoptic part segmentation. In ECCV, 2022.
[28] Fangting Xia, Peng Wang, Xianjic Chen, and Alan L. Yuille. Joint multi-person pose estimation and semantic part segmentation. CVPR, 2017.
[29] Zijian Zhou, Miaojing Shi. and Holger Caesar. Hilo: Exploiting high low frequency relations for unbiased panoptic scene graph generation. In arXiv, 2023. 3
[30] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbüühl. Simple multi-dataset detection. In CVPR, 2022.
[31] Georgios Kapidis, Ronald Poppe, and Remeo C Veltkamp. Multi-dataset, multitask learning of egocentric vision tasks. TPAMI, 2021. 3
[32] Qiang Zhou, Yuang Liu, Chaohui Yu, Jingliang Li, Zhibin Wang, and Fan Wang. Lmseg: Language-guider multi-dataset segmentation. In arXiv, 2023. 3
[33] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan, Universal domain adaptation. In CVPR, 2019.
[34] Antoine Saporta, Arthur Douillard, Hun-HHug Vu, Patrick Pérez, and Matthiuu Cord. Multi-head distillation for continual unsupervised domain adaptation in semantic segmentation. In CVPR, 2022. 3
[35] Duo Peng, Yinjie Lei, Munawar Hayat, Yulan Guo, and Wen Li. Semantic-aware domain generalized segmentation. In CVPR, 2022. 3
[36] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In ICML, 2019. 3
[37] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexci A Efros. Dataset distillation. In arXiv, 2018.
[38] Lu Qi, Jason Kuen, Zhe Lin, Jiuxiang Gu, Fengyun Rao, Dian Li, Weidong Guo, Zhen Wen, Ming-Hsuan Yang. and Jiaya Jia. Ca-ssl: Class alnss-ag semi-supervised learning for detection and segmentation. In ECCV, 2022. 3, 9
[39] Yang Zhang, Philip David, and Boqing Gong. Curriculum domain adaptation for semantic segmentation of urban scenes. /CCV, 2017. 3
[40] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Guided curriculum model adaptation and uncertainty- aware evaluation for semantic nighttime image segmentation. In ICCV, 2019.
[41] Lu Qi. Shu Liu, Jianping Shi, and Jiaya Jia. Sequential context encoding for duplicate removal, In Advances in Neural Information Processing Systems, 2018. 3
[42] Yukang Chen, Peizhen Zhang. Tao Kong, Yanwei Li. Xiangyu Zhang, Lu Qi, Jian Sun, and Jiaya Jia. Scale-aware automatic augmentations for object detection with dynamic training. In TPAMI, 2022. 3
[43] Yukang Chen, Yanwei Li, Tao Kong. Qi, Ruihang Chu, Lei Li, and Jiaya Jia. Scale-aware automatic augmentation for object detection. In CVPR, 2021. 3
[44] Lu Qi, Jason Kuen, Jiuxiang Gu, Zhe Lin, Yi Wang, Yukang Chen, Yanwei Li, and Jiaya Jia. Multi-scale aligned distillation for low-rcsolution detection. In CVPR, 2021. 3
[45] Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ana Heng. Feddg: Federated domain generaliza- tion on medical image segmentation via episodic learning in continuous frequency space. In CVPR, 2021.
12

---

[46] Agrim Gupta. Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR. 2019. 7
[47] Davood Karimi, Simon K Warfield, and Ali Gholipour. Transfer learning in medical image segmentation: New insights from analysis of the dynamics of model parameters and learned representations. Artificial intelligence in medicine, 2021.
[48] Yen Tsai, Bidan Huang, Yao Guo, and Guang-Zhong Yang. Transfer learning for surgical task segmentation. In ICRA, 2019.
[49] David Marr. Vision: computational investigation into the human representation and processing of visual information. Francisco: W.H. Freeman. Print., 1982.
[50] Leslie G Ungerleider and James V Haxby. 'what'ane and 'where' in the human brain. Current opinion in neurobiology, 4(2):557-665 1994.
[51] Danielle Bassett and Michael Gazzaniga. Understanding complexity in the human brain. Trends in cognitive sciencex, 15(5):200220, 2011.
[52] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without lest-time finetuning. In CVPR, 2023.
[53] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. 5, 6,
[54] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[55] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers for image recognition at scale. In arXiv, 2020. 9
16x16 words:
[56] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012. 9
[57] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 9
[58] Mengde Xu, Zheng Zhang, Fangyun Wei, Hu, Xiang Bai. Side adapter network open- vocabulary semantic segmentation. In arXiv, 2023. 9
[59] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images datasct v4, IJCV, 2020. 16, 20
[60] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Muliis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of elip-filtered 400 million image-text pairs. In NeurlPS Workshop. 2021. 16, 20
13

---

Appendix
In this supplementary file, provide more experimental details and empirical results to further demonstrate the benefits of our proposed All-Inclusive Multi-Level segmentation task (AIMS).
Experimental details on dataset split, sampling strategy and SAM's [10] configuration.
More ablation studies sample strategy and dataset usage.
More visualization results of our AIMS model with various inference modes.
User study of AIMS and the concurrent work SAM [10]
The code and models to reproduce our experiments will be released.
A Experimental Details
Dataset Split As outlined in the main paper, all models are trained using five datasets: COCO [53], EntitySeg [5]. PascalVOC Part (PPP)[6), PACO[9], and COCO-PSG [8]. Given that the original training and validation splits of these datasets are tailored for single tasks, collate the images and reorganize them to suit our AIMS task. Initially, we select 1069 and 1000 validation images from PPP [6] (which covers the part and entity levels) and COCO-PSG [8] (which covers the entity and relation levels) respectively. Following this, we eliminate any duplicate images in the unified training set that are present the validation images, resulting in refined training set comprised of approximately 236.7K unique images.
Sampling Strategy In Table 7, we present the eight different sample types, labeled as Sample II) 1 through 8, for each iteration, assuming batch size of 8. Additionally, images for each of these sample types are uniformly selected from the datasets mentioned in the 'Dataset' column.
SAM's configuration To obtain more fine finennrained part-level predictions, parameters found on SAM's GitHub page: points per side (2), pred iou thresh (0.86), stabil- ity score thresh (0.92), crop n layers (I), crop n points downscale factor (2), min mask region area (100).
follow the hyper-
per_side
B Ablation Studies
Sample Strategy Table 7 presents an ablation study on various sample strategies by examining the impact of each sample type during each iteration. The first row serves our baseline, adhering to our base framework with a split decoder for varying-leve predictions, and it excludes the usage of a prompt mask encoder. The remaining rows illustrate the influence on performance when task prompts are employed during each training iteration.
For instance, the second row reveals that the exclusive usage of full image task prompt brings marginal performance improvement over the baseline. Since full-image mask prompts are always the same, injecting them into the original image features is identical to not using any mask prompts. Conversely, the third row shows that employing solely partial-image mask prompts can considerably deteriorate performance, which inconsistent with the inference process that involves full-image mask prompts.
In the fourth row, observe that using both full and partial mask prompts further enhance performance by providing more accurate signals to the network. Additionally, the introduction of mask prompts the entity-to-part and relation-to-entity levels consistently yields performance improvements, as depicted in subsequent rows.
Dataset Usage Table 8 displays an ablation study focusing on the utilization of various datasets. The first three rows present the model performance when trained on single dataset. This results in degraded performance on cross-datases evaluation due to inconsistencies in annotations across different datasets. As shown the fourth row, combining the three datasets for training enhances the overall validation performance. This improvement is due to the consistency maintained between the training and validation splits. Incorporating PACO and Entityseg datasets at part and entity levels further improve the performance for the respective levels.
14

---

Sample ID
Mask Prompt
Decoder
Dotaset
Entily
COCO, Eniiyyeg. PPP, CSC
Full Image
Part
PPP
Relation
POCO-PSG
Eatity
COCO, EntrySog. COCO-PSG, PACO
Partial Image
Pari
PPP. PACO
COCO.
One Eabiiy
Part
PPP. PACO
Two Entilies
Relation
COCO-PSG
Table 6: The illustration of eight sample types each training iteration.
Sample ID
PPP (Inference)
COCO-PSG (Inference)
2
3
4
5
6
7
8
APP
APE
AR
APR
APF
AR
o
o
0
o
0
0
24.5
53.4
69.7
38.9
40.4
50,9
25.0
33.5
39.2
40.8
51.3
22.5
51.4
67.3
37.1
37.6
48,9
7
7
0
25.7
54.4
70.9
39.6
41.2
51.8
8
5
55.9
41.3
51.8
25.7
54.3
71.0
40.4
42.0
53.0
7 7
26.5
56.1
72.3
40.5
42.1
53.1
Table 7: The ablation studies on eight distinct sample types used in each training iteration. The v and o symbols are employed to indicate whether a specific sample type is utilized. For fair comparison, we adjust the learning rate linearly in relation to the batch size. The default learning rate is 1e-8 for batch size of 8.
Dataset
PPP (Inference)
COCO-PSG (Inference)
COCO
COCO-PSG
PPP
PACO
Eotityog
AP
AR
APR
AR
44.4
41.2
44.5
39.6
41.7
51.8
0
0
24.3
48.6
65.8
25.7
70.9
8
26.4
55.9
72.1
39.6
41.2
51.8
4
0
23.7
54.3
71.0
40.4
42.0
53.0
72.3
Table 8: The ablation study of performance influence with dataset usage. Similar to Table 7, we adjust the learning rate linearly considering the lack of some task prompts due to the non-provided dataset.
C Visualization
The flexibility of our AIMS task. Figure illustrates how our proposed AIMS task provides the flexibility for segmenting anything. This is to address the subjective annotation issues in existing datasets. For instance, the image (a), the throw pillows and the sofa are predicted as a single entity, which aligns with the ground truth annotation. Nonetheless, in certain scenarios, user might wish to edit the throw pillows independently. With our AIMS method, utilizing a full image prompt for part-level prediction does not yield these separated masks. However, by selecting this entity mask as an entity prompt for part-level prediction, we can successfully differentiate the three throw pillows.
Additionally, in image (b), user may want to segment the windows into several components. single entity.
However, original ground truth annotations typically consider the two windows To meet this requirement, we initially utilize a full image prompt and an entity encoder to identify the mask of the whole windows. Following this, we apply this two-entity mask prompt and relation Decoder to split them into two independent window masks. Ultimately, we can select one window for further segmentation into two parts: the window edge and curtains.
Prediction consistency. We investigate the prediction consistency across two inference modes: full and partial image prompts. Figure 8 displays an example where our AIMS model fails segment anything at the part level with full image prompt, but it is able to break down an entity-level mask prompt into a more detailed level, similar to the sofa and throw pillows in the image. Given that the pillows have never been labeled in our utilized dataset, inconsistent prediction results for unseen classes when different prompts are used. Using entity-level mask prompts could help our model to drill down to the next level.
surmise that our AIMS model might yield
15

---

Inll Prmpp
Sekert Fadty Nark
Oue Eaily Pranul ParrEaeelr
Easly
Image Prompt
Fan Enender
Das the Danwee Anneration
(b)
Jany Taceeer
Anctee Lande
One Eatio Presep.. Pert Enosler
Soket Extin Makk
Figure 7: The illustration of the flexibility of AIMS task to tackle the subjective annotation issues in existing datasets.
mmps porpe
Enily level Reslls
Furtbre Roals
city proegg
Figure 8: The example of prediction inconsistency between two prompt types on unseen classes.
However, the scenario changes when we turn our attention to the known classes. Figure 9 illustrates that three cars yield similar part-level prediction results, regardless whether full image single entity prompt is used. This indicates that our AIMS model can maintain prediction consistency for known classes.
The three-level predictions Figure 10 displays the three-level prediction outcomes of our AIMS model. Despite the potential subjectivity defining the three levels, our AIMS model shows strong promises in fulfilling various user needs and intentions in image editing. For instance, the red region in the image is not detected at the entity-level prediction results, as is merged into the tray. However, by injecting entity-level mask prompt for the tray region into the part encoder, that the separate red region can be obtained, as shown the second column of the 'part-level results'. Additionally, the relation-level masks of any two entities that share some semantic relationship can be predicted, and all relation-level prediction results can be represented in a scene graph.
More visualization results. In the Fig. 13, we show more visualization results entity and part level in the wild (Open-Image [59] dataset), manifesting the generalization ability of our proposed AIMS model the entity and part level. Considering the entity level is the principal part of image editing, also show some cases Laion400M [60] to show the effectiveness of our AIMS model.
16

---

full image prompt
one entity prompt
Entity-level Results
Part-level Results
Figure 9: The example of prediction consistency between two prompt types on seen classes.
D User Study
We conducted study in which there were 480 individuals who were identified as Adobe Photoshop users who regularly used the software for image manipulation/editnng
this user study, randomly selected 40 images in the wild and provided the users with visualization of the three- level prediction results of our AIMS model, as shown in Figure 10. For each image, we asked each user about his/her degree of satisfaction with treating semantically meaningful and -coherent segments for three levels, with respect to their relevance to and suitability for image manipulation/ediitn applications. The satisfactory scores aggregated from all users for the individual images. We find that the average score of each image is large than 7.8 on the condition that the maximum score is 10. Most of the selected images' scores larger than 6.0, This confirms that the users are highly satisfied with our model's 's prediction results in the context of image manipulation/cdiite user study's 's findings, we summarize the data of Table 9. The image IDs with the minimum, median, and maximum user scores are 20, 19, and 35. We show these images and their corresponding entity annotations in Fig. 12.
To better present the
In Fig. 12, we provide statistical information about our user study. The sub-figure (a) is a summary of Table 9, indicating that over 90% of users rated our prediction results higher than 6.0. Sub-figure (b) contrasts user preferences between our results and those from the SAM model. Despite being trained with fewer data, our model garners an appreciation comparable to that of SAM, can be observed in the sub-figure (b).
17

---

Part-level Results
Entity-level Results
Relation level Results (Seene Graph)
Figure 10: The illustration of how the AIMS model can be used to obtain culti-inteenionn segmentation
results, including a scene graph for physical-toue relations.
(8.00)
Image 32 (8:25)
Imago-ID 15 (8.05)
Imsge-1D 20 (7.83)
Image-ID 22 (7 96)
Image-ID 34 27)
Figure 11: The selected images used in our user study. For brevity, we only show the entity- and part-level predictions the user most care.
18

---

The statistical visualizations of the data from our user and survey studies. (a) The histogram here represents the distribution the users based on their given scores in the three-level predictions. (b) Each horizontal bar here indicates the proportions of votes given by the survey participants to SAM [10] (green) and ours (orange) on each the 40 images.
Figure 12:
Image
P1
P2
P4
P6
P1o
AVG
0.00
0.00
1888
3.13
8.13
11.25
11.25
16888
38.13
8.06
0.63
0.00
3.13
3.75
5.63
15.00
13.75
13.13
10.00
35.00
7.84
1.25
0.63
1.88
1.88
10.63
10.63
15.00
12.50
8.13
37.50
7,86
0.63
0.63
0.63
3.13
7.50
14.38
13.13
14.38
8.75
36.88
7.95
5
0,00
0.63
2.50
1.25
7.50
11.88
13.13
18.13
7.50
37.50
8.02
6
0.63
0.63
1.25
1.25
6.88
15.00
14.38
20.00
7.50
32.50
7.88
0.00
0.00
0.63
3.13
6.25
13.75
12.50
20.63
10.63
32.50
8.01
8
0.00
0.00
1.25
1.88
8.13
11.88
13.63
13.75
9.38
38.13
8.08
9
0.00
0.63
1.25
3.75
8.13
10.63
15.63
18.13
8.13
33.75
7.90
10
0.00
0.63
3.13
2.50
5.00
16.88
15.00
14.38
6.88
35.63
7.85
11
0.00
0.00
0.00
1.25
5.63
17.50
10.00
19.38
13.75
32.50
8.12
12
0.63
0.63
0.63
2,50
5.00
10.63
16.88
22.50
11.87
31.88
8.00
13
0.00
0.00
0.00
2,50
5.00
14.38
15.00
13.63
12.50
35.00
14
0.00
0.63
1.25
3.75
6.25
13.75
13.75
19.38
12.50
28.75
7.85
15
0.00
0.00
0.63
1.88
8.13
13.75
11.25
20.63
9.38
34.38
8.05
16
0.63
0.63
1.25
2.50
7.50
11.88
13.75
18.13
11.25
32.50
7.91
17
0.00
0.00
0.63
1.88
4.38
13.75
18.13
16.25
11.88
33.13
8.09
18
0,00
0.00
0.00
3.13
5.63
16.88
15,00
13.75
11.25
34.38
8.02
19
0.00
0.63
0.63
1.88
6.25
15.00
16.88
14.38
8.75
35.63
8.00
20
0.00
0.00
0.63
3.13
6.25
10.63
12.50
20.63
10.63
32.50
7.83
21
0.00
1.88
0.63
0.63
5.00
11.88
15.63
22.50
8.13
33.75
8.05
22
0.00
0.63
1.25
3.13
8.75
10.63
13.75
16.25
11.88
33.75
7.96
23
0.00
0.63
0.63
1,88
5.63
13.75
13.75
18.75
9.38
35.63
8.08
24
0.00
0.63
0.63
1,88
4.38
11.33
13.63
20.00
11.88
33.13
8.11
25
0.00
1.88
0.63
1.88
7.50
11.88
11.88
19.38
11.25
33.75
8.00
26
0.00
1.25
0.63
2.50
5.00
11.25
16.88
18.75
34.38
8.03
27
0.00
0.00
1.25
7.50
12.50
13.13
20.63
34.38
28
0.00
0.00
0.63
4.38
3.75
12.50
13.13
20.00
13.13
32.50
8.08
29
0.00
0.00
0.63
3.13
5.00
8.13
16.88
18.75
13.75
33.75
8.18
30
0.00
1.25
1.25
5.00
5.63
11.25
11.25
18.13
[1.88
34.38
7.96
31
0.00
0.63
1.88
0.63
5.00
12.50
12.50
20.00
12.50
34.38
8.13
32
0.00
0.63
1.88
0.00
4.38
11.25
15.00
18.75
9.38
38.75
8.23
33
0.00
0.63
0.63
2.50
5.00
13.13
9.38
23.75
9.38
35.66
8.13
34
0.00
1.25
0.63
2.50
5.00
11.88
6.88
18.75
13.63
37.50
8.25
35
0.00
0.63
0.63
1.25
4.38
10.00
12.50
21.88
13.13
35.63
8.27
36
0.00
0.63
0.63
3.13
5.00
10.63
13.13
18.13
14.38
34.38
8.14
37
0.00
0.00
0.63
2.50
3.75
10.00
18.13
18.75
16.25
30.00
8.14
38
0.63
0.00
0.63
2.50
10.00
10.63
13.13
23.13
10.00
29.38
7.84
39
1.25
4.38
10.63
18.13
17.50
13.75
30.63
8.04
40
0.00
0.00
0.63
2.50
6.88
11.88
11.25
16.88
13.75
36.25
8.08
Table 9: Data from the user study on the three-level prediction results of our AIMS model. Pz indicates the percentage of users who give "x" as the score to represent their degrees of satisfaction. "x" ranges from 1 to 10, and 10 represents the highest degree of satisfaction. "AVG" indicates the the score averaged across all users for each image.
19

---

Figure 13: More visualization results on Dataset [59] in the 'wild'.
Figure 14: More visualization results on Laion400M [60] in the 'wild'.
20


Reviews:
[{"review_id": "deFQbJTwkV", "reviewer": "Reviewer_mZLg", "summary": "This work proposes a unified multi-level segmentation (AIMS) approach. For better generalisation, the model is concurrently trained on multiple dataset consisting varied hierarchical level annotations across parts, entities and relations. In order to utilise signals from multiple hierarchical level as well as infuse model with level-awareness for each training sample, AIMS proposes three modules - task complementarity, association and mask prompt encoder. Experimental evaluation shows AIMS performing better than other state-of-the-art class-aware as well as class-agnostic segmentation.", "questions": "AIMS model have three separate output while other models output single mask irrespective of entity our parts. So in table 5(b), how do one measure seperate AR for parts and entity segmentation. Do you run seperate inference with different prompt for part level and entity level ? Or full-image mask prompt is applied ?", "limitations": "Authors haven't addressed any limitations. Some of the listed weaknesses above can be elaborated to address the limitations the paper. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\nAlthough model is inspired from Mask2former, the inclusion of other novel modules packages AIMS for multi-level hierarchical segmentation. Thus I note sufficient novelty in the proposed architecture. \n\nQuality\nThe paper is well-written and easy to follow. Authors can be more clearer upfront about AIMS being complete class-agnostic model (i.e., mask proposal model) and there is no semantic labelling.\n\nSignificance\n1. As far as I am aware of, AIMS is the first segmentation model to be trained with multi level semantic understanding. Though SAM is trained with fine-grained annotations it lacks part- and relation-level understanding and thus certain fine-grained semantic features may not be fully segmented out.\n2. Moreover, training with multiple datasets enables AIMS to be deployed for range of image editing and manipulation applications. \n3. AIMS achieves these advantages with as minimal data as possible (237K vs 11M in SAM).", "weaknesses": "1. As pointed earlier, AIMS is class-agnostic. Hence it cannot label the output segments. \n2. AIMS use interactive settings by deploying mask prompt encoder (MPE) to specify model what to segment. However, MPE usage is limited to providing unmask region of an input image for model to act on. There is no provision to use text, bounding box, or points as inputs. This is very limited interactive settings.\n3. For the same reason though powerful, AIMS cannot extract referring expression segmentation. This is despite the fact that AIMS is trained with semantic of relation understanding.\n4. Overall, the model is novel but the evaluation lacks distinguishing aspects that AIMS in the very first place was designed for.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and insightful comments.\n\n### Q1: AIMS is class-agnostic model and cannot label the output segments.\n\nThanks for the suggestion. Although mask labeling is related to our work, the main goal of our work is to develop an all-inclusive segmentation model capable of delivering high-quality multi-level mask proposals for various downstream tasks, including class-aware ones. That's because class-agnostic segmentation has shown great effectiveness to the unseen categories or image domain than class-aware one in entity segmentation [1,2] or SAM [3]. In Table 5(a) of our paper, we demonstrate the advantages of fine-tuning the AIMS pre-trained model for class-aware tasks such as panoptic part segmentation and scene graph. This approach leads to performance that outperforms the state-of-the-art methods of each task, showcasing the flexibility and effectiveness of our model even on class-aware tasks.\n\nFurthermore, AIMS works complementarily with existing mask labeling methods [4,5,6] that can be applied to external mask proposals.\n\n[1] Open World Entity Segmentation. TPAMI 2022.\n\n[2] High-Quality Entity Segmentation. ICCV 2023.\n\n[3] Segment Anything. ICCV 2023.\n\n[4] Open-vocabulary semantic segmentation with mask-adapted clip\n. CVPR 2023.\n\n[5] Open-Vocabulary Universal Image Segmentation with MaskCLIP. ICML 2023.\n\n[6] Scaling Open-Vocabulary Image Segmentation with Image-Level Labels\n. ECCV 2022.\n\n\n---\n### Q2: Limited interactive settings like text, bounding box or points.\n\nOur AIMS model is not designed for interactive segmentation. Instead, we focus on automatic image segmentation at multiple explicit levels in an all-inclusive manner, without requiring intricate prompt engineering and parameter tuning during inference to obtain desired results, as SAM needs. SAM's weakness is that it does not enforce explicit levels of segmentation and only selects the prediction that has the smallest loss against ground truth for training. Therefore, our setting and the proposed method is more desirable for large-scale automated segmentation mask generation without costly human intervention and correction.\n\nInteractive settings are usually sensitive to geometric prompts. For instance, using two-point prompts with slightly different locations in SAM can give very different mask predictions. Furthermore, for many applications that do not rely on user inputs (e.g., automated image content analysis), using a regular grid of points as prompts is unfavorable since the model can fail to segment objects that do not overlap with those points.\n\nWhile our proposed framework can accept mask prompts, this feature is part of our design to address annotation inconsistencies among datasets. It encourages the network to learn how to perform further separation on a given mask instead of functioning as an interactive segmentation component. We do not expect users to be able to draw out sufficiently good mask prompts for our model.\n\n\n---\n### Q3: Lacking ability for referring expression segmentation.\n\nSimilar to the first question, one of AIMS's roles is to obtain good pretrained weights for finetuning on downstream tasks. We test AIMS model in generalized referring expression segmentation [7] on gRefCOCO dataset.\n\nThe detailed structure design proceeds as follows: First, we obtain the text embedding using the BERT model. Within each level of the decoder, we integrate three additional transformer blocks. These blocks include a cross-attention layer between the query and text embedding, a cross-attention layer between the query embedding and image features, self-attention, and a feedforward network. For each cross-attention, query embedding is the query and other keys and values. These additional layers are appended to the default nine transformer layers. Ultimately, the three-level results are merged for bi-partite matching. During inference, we directly select the mask with the highest score. To ensure a fair comparison, we employ a similar number of training iterations to the ReLA method [7], utilizing the Swin-Tiny backbone.\n\nThe comparison results are presented in the table below, demonstrating the generalization ability of the AIMS model in the context of referring expression segmentation.\n\n| method | cIoU | gIoU |\n| :-----: | :----: | :----: |\n| ReLA [7] | 52.26 | 54.44|\n| Ours | 52.83 | 54.96|\n\n[7] GRES: Generalized Referring Expression Segmentation. CVPR 2023.\n\n\n---\n### Q4: Evaluation Metrics.\n\nOur evaluation metrics face the complex challenge of assessing three performance levels simultaneously, making it difficult to devise a unified metric that accurately balances the weight across these levels. Consequently, we opt for decoupled evaluation metrics tailored to each level, ensuring a better model demonstrates enhanced performance across all three levels. This approach not only allows us to pinpoint specific areas of performance improvement when implementing new designs but also helps validate our method's effectiveness. \n\nFor unified evaluation metrics, we explore it in downstream tasks such as PartPQ and PWQ for panoptic part segmentation and R/mR@100 for panoptic scene graph construction, as shown in Table 5(a) of our paper. That shows our model also performs better than state-of-the-art methods by using unified evaluation metrics.\n\n---\n### Q5: The detailed inference setting in Table 5(b).\nUsing a full-image mask prompt, we compare the entity- and part-level results between SAM and our model.  To ensure a fair comparison in the part-level results, we show the result comparison in AR\\@100 in the following table by utilizing the ground-truth entity-level masks as mask prompts for part-level inference.\n\n| method | PACO | ADE20K|\n| :-----: | :----: | :----: |\n| SAM [3] | 54.4 | 50.6 |\n| Ours | 55.0 | 51.3 |"}, {"review_id": "JedTdFROYm", "reviewer": "Reviewer_b8VN", "summary": "This paper introduces a novel task called All-Inclusive Multi-Level Segmentation (AIMS) and proposes a unified AIMS model to address the challenges of annotation inconsistency and task correlation. The model consists of a shared image encoder and three independent decoders for part, entity, and relation predictions. It incorporates a Task Complementarity Module (TCM) to fuse task information and an Association Module to establish associations between different levels of segmentation. The model is trained using a **combination** of existing segmentation datasets. The model also incorporates a Mask Prompt Encoder (MPE) to provide supervision signals. \n\nThe proposed method outperforms SOTA methods on both panoptic part segmentation and panoptic scene graph estimation tasks. The proposed method also outperforms the Segment Anything model (SAM) on entity-level segmentation, even though the proposed method was trained on much less data than SAM.", "questions": "* The authors mention that Swin-Large backbone provides the best results, but can you provide more details on the decoder architecture? Also, what is the total model size and computational requirement (FLOPs) of the model?\n* The legend in Figure 2 is missing the symbol for \"Add\"", "limitations": "No limitations section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Novel approach to a multi-level segmentation task\n* The paper introduces a novel task called All-Inclusive Multi-Level Segmentation (AIMS) and proposes a unified model to address the challenges of annotation inconsistency and task correlation. This task formulation and the proposed model are unique and provide a fresh perspective on multi-level segmentation.\n\n### Comprehensive experiments with detailed ablations and good results\n* The paper presents a comprehensive evaluation of the proposed method, comparing it with state-of-the-art methods and conducting ablation studies. \n* The experiments are well-designed, and the results demonstrate the effectiveness and generalization capacity of the proposed method.\n* The authors also provide detailed explanations of the model components and training settings, ensuring reproducibility.\n\nI also appreciate the clarity of writing in this paper which has a thorough analysis of problems in existing segmentation frameworks motivating the need for the AIMS task.", "weaknesses": "### Missing performance analysis on images with a large number of objects\n* SAM has shown impressive performance on images with a large number of objects. How does the proposed method fare in such conditions? It would be useful to break down the performance on the basis of the number of objects and compare it with SAM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and insightful comments.\n\n### Q1: Missing performance analysis on images with a large number of obejcts.\n\nWe present the recall@100 comparison of the entity-level results of between SAM and our model on selected LVIS validation data that includes more than 20 objects, as well as on crowd-human validation data. In the statistics, the average object numbers in our LVIS and crowd-human are 21 and 23, respectively. The table below demonstrates that our model can achieve performance comparable to the SAM model on images containing a large number of objects.\n| method | LVIS | crowd-human |\n| :-----: | :----: | :----: |\n| SAM | 37.2 | 97.5 |\n| Ours | 36.9 | 97.8 |\n\nFurthermore, we compare our method with the SAM model on the object clutter indoor segmentation for robotic grasping (OCID) and the few-shot segmentation (FSS) dataset on recall@100. These comparisons highlight some advantages of our method's separated-decoder design. This is because the SAM model does not explicitly divide the results into three levels and only selects the minimum loss corresponding to the ground truth for training. This approach leads to strong requirements for user selection effort at certain levels, whereas our design mitigates this limitation.\n\n| method | OCID | FSS |\n| :-----: | :----: | :----: |\n| SAM | 55.17 | 73.60 |\n| Ours | 76.63 | 86.26 |\n\n\n---\n### Q2: The details of decoder structure.\n\nWe adhere to the transformer decoder design in the Mask2Former method, encompassing a set of learnable queries and nine transformer blocks. Each block comprises a cross-attention layer, a self-attention layer, and a feed-forward network. In our AIMS decoder, we begin with three distinct transformer decoders designated for different levels of segmentation results, forming our baseline approach.\n\nBuilding upon this baseline, we introduce the task complementarity module to facilitate information fusion across different levels, such as the fusion between entity-part and relation-entity decoders. This innovative design is inspired by our observation that separate transformer decoders consistently outperform a shared transformer decoder; such observation is demonstrated in the following table. Essentially, our experimental findings support the notion that decoders at different levels possess distinct unique features compared to other decoders. The values reported in PPP dataset are $AP^P$, $AP^E$ and $AR^{ER}$. The values reported in COCO-PSG dataset are $AP^R$, $AP^E$ and $AR^{RE}$. \n\n| Decoder Design | PPP | COCO-PSG |\n| :-----: | :----: | :----: |\n| Shared | 23.2/52.0/67.9 | 37.2/39.1/49.2 |\n| Separated | 24.5/53.4/69.7 | 38.9/40.4/50.9 |\n\nOur proposed task complementarity module capitalizes on these individual strengths, enabling each level decoder to interact with and be enriched by the others. This synergistic approach improves performance compared to our baseline, highlighting the efficacy of tailored, level-specific interactions within the decoder architecture.\n\nFor the association module, we directly sample positive embeddings from each level generated by the final transformer block to establish associations between two adjacent levels. It's important to note that these sampled positive embeddings are pre-processed through a fully connected (FC) layer.\n\n\n---\n### Q3: Model size and computational requirement.\n\nThe table below provides the parameter size of our largest model with the Swin-Large backbone. Notably, the new proposed modules, including task complementarity module (TCM), mask encoder prompt（ME）, and association module (AE), only introduce a minor increase of parameters compared to our baseline model (backbone and baseline decoder). This is due to the efficient shared weight design implemented in the three proposed modules. Overall, the number of parameters in AIMS with the Swin-Large backbone is 246,236,372 (200845932+42626304+2630144+1384+132608), compared to the 641,090,864 number of parameters in SAM with VIT huge backbone.\n\n| Backbone | Baseline Decoder  | TCM | ME | AM  |\n| :----:  | :----:  | :----:  | :-----:  | :----:  |\n| 200,845,932 | 42,626,304 | 2,630,144 | 1,384 | 132,608 |\n\nFor the computation cost, conducting full image inference under (800, 1333) image size in three levels requires 854.0 GFlops and takes approximately 0.142 seconds per image on A100 40G. This computation cost is comparable to the baseline method, which consumes about 801.0 GFlops and has an inference time of 0.138 seconds per image. For the SAM model we compare, it would cost 0.253 seconds per image within the same setting.\n"}, {"review_id": "use1RtCmUR", "reviewer": "Reviewer_u59B", "summary": "This work presents AIMS, a multi-level image segmentation model with levels representing parts, instance, and relation. Further, a curated dataset is created from several existing segmentation datasets. AIMS outperforms the baselines on the curated dataset.", "questions": "1. Does the training set aggregation make any modification of the original datasets/annotations? Would the curated dataset (split) be released?\n2. The comparison in Figure 4 only shows part segmentation. What would the comparison be when it comes to multi-instance level?\n3. In table 5(b), shouldn't the $AR^E$ score for EntitySeg be bold since 87.2 > 87.1?", "limitations": "Limitations are not discussed by the authors. Apart from potential limitations in Weaknesses, I see there are these limitations:\n1. The computational cost is unspecified.\n2. The complementarity module and association module are designed to couple with the base framework, thus not obviously usable by other architectures.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed architecture reasonably bridges information across levels.\n2. The quantitative and qualitative performance are good.\n3. Experiments, especially the ablations, are comprehensive.", "weaknesses": "1. It's unclear how the prompt mask is obtained and used during inference. When decoding with such a mask, would its experiments against baselines unfair comparisons since baselines might not have such additional localization information?\n2. The comparison with SAM is partially done. Especially that SAM would provide fine segmentation masks with mask prompt with low-accuracy, while AIMS has not mentioned explicitly the quality of the mask prompt -- based on Figure 3 the masks seem to be accurate.\n3. The presentation can be generally improved. Figure 2 provides details that distracts readers without further knowledge of more details, whilst the mask prompt encoder can be motivated earlier.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and insightful comments.\n\n### Q1: Unclear about the prompt mask used in inference.\n\nFor a fair comparison, we conduct all three levels of segmentation using the full-image mask prompts throughout our experiments, without unfairly introducing any specific localization or region information. In Table 5(b), we show the entity- and part-level results between the SAM model and our method, employing only full-image mask prompts for the evaluation.\n\nTo further substantiate the efficacy of our approach, we present a comparative analysis of the results in terms of AR@100. This comparison is conducted by prompting both SAM and our AIMS models with the same ground-truth entity mask for part-level inference. The outcomes, detailed in the following table, serve as additional evidence of the strengths of our method.\n\n| Method | PACO | ADE20K|\n| :-----: | :----: | :----: |\n| SAM | 54.4 | 50.6 |\n| Ours | 55.0 | 51.3 |\n\n---\n### Q2: The ablation study on the quality of mask prompt.\n\nIn the following two tables, we show the ablation studies about the quality of mask prompt on PACO dataset where we degrade the ground truth entity masks with different levels including the part and entity level respectively.\n\nThe performance comparison of part-level results is as follows:\n| Method | Ground-Truth Mask| Bounding Box | Randomly Extended Box | Randomly Created Mask |\n| ----- | ---- | ---- | ---- | ---- |\n| SAM | 54.4 | 53.5 | 52.8 | 52.5 |\n| Ours | 55.0 | 54.2 | 53.7 | 53.4 |\n\nThe performance comparison of entity-level results is as follows:\n| Method | Ground-Truth Mask| Bounding Box | Randomly Extended Box | Randomly Created Mask |\n| ----- | ---- | ---- | ---- | ---- |\n| SAM | 100.0 | 98.7 | 98.6 | 98.9 |\n| Ours | 100.0 | 99.1 | 98.9 | 98.8 |\n\nFor the extended bounding box, we randomly perturbed the four corners of the original bounding boxes. For the arbitrary mask, we randomly added polygon points to the extended bounding boxes to create random masks. We can provide the corresponding code to generate these boxes and masks in the next round discussion due to the limited characters.\n\n\n---\n### Q3: The improved presentation in Figure2.\n\nWe have improved Figure2 with a better presentation. Please check the figure 1 and 2 in our submitted PDF.\n\n\n---\n### Q4: Training Set.\n\nWe report about it in our supplementary file. All models are trained using five datasets: COCO, EntitySeg, PascalVOC Part, PACO, and COCO-PSG. Given that these datasets' original training and validation splits are tailored for single tasks, we collate the images and reorganize them to suit our AIMS task. Initially, we select 1069 and 1000 validation images from PPP (which covers the part and entity levels) and COCO-PSG (which covers the entity and relation levels), respectively. Following this, we eliminate any duplicate images in the unified training set that are present in the validation images, resulting in a refined training set comprised of approximately 236.7K unique images. In a nutshell, we only reorganized existing datasets and did not introduce new annotations.\n\nWe will release the curated dataset to the community for future research and reproducibility. \n\n\n---\n### Q5: The wrong bold location.\n\nThanks for pointing this out. We will correct it in our paper.\n\n\n---\n### Q6: Computation cost.\n\nThe table below provides the parameter size of our largest model with the Swin-Large backbone. Notably, the new proposed modules, including the task complementarity module (TCM), mask encoder prompt (ME), and association module (AE), only introduce a minor increase of parameters compared to our baseline model (backbone and baseline decoder). This is due to the efficient shared weight design implemented in the three proposed modules. Overall, the number of parameters in AIMS with the Swin-Large backbone is 246,236,372 (200845932+42626304+2630144+1384+132608), compared to the 641,090,864 number of parameters in SAM with VIT huge backbone.\n\n| Backbone | Baseline Decoder  | TCM | ME | AM  |\n| :-----:  | :----:  | :----:  | :-----:  | :----:  |\n| 200,845,932 | 42,626,304 | 2,630,144 | 1,384 | 132,608 |\n\nFor the computation cost, conducting full image inference under (800, 1333) image size in three levels requires 854.0 GFlops and takes approximately 0.142 seconds per image on A100 40G. This computation cost is comparable to the baseline method, which consumes about 801.0 GFlops and has an inference time of 0.138 seconds per image. For the SAM model we compare, it would cost 0.253 seconds per image within the same setting.\n\n\n---\n### Q7: The universality of proposed complementarity and association module.\n\nWe introduce the task complementarity and association modules, drawing inspiration from the state-of-the-art segmentation decoder design, which employs learnable queries and transformer blocks. This approach allows for adaptability with minimal modification, aligning with designs commonly found in recent transformer-based segmentation methods [1,2,3]. For instance, we integrate these two modules into the improved Mask DINO [1] by replicating the decoders three times for three-level segmentation (our baseline). The incorporation of these modules in Mask DINO is carried out in a manner akin to their implementation in Mask2Former, demonstrating the versatility of our design.\n\n| Method | PPP  | COCO-PSG  |\n| :-----:  | :----:  | :----:  |\n| Mask DINO  | 23.9/52.7/68.9 | 38.0/39.2/48.7 |\n| Mask DINO with TCM and AM | 25.3/54.2/71.5 | 38.9/40.3/49.9 |\n\n[1] Mask dino: Towards a unified transformer-based framework for object detection and segmentation. CVPR 2023.\n\n[2] MaX-DeepLab: End-to-End Panoptic Segmentation With Mask Transformers. CVPR 2021.\n\n[3] k-means Mask Transformer. ECCV 2022."}, {"review_id": "WDH0aYZpGm", "reviewer": "Reviewer_Tt8E", "summary": "This paper introduces a new task and a model for All-Inclusive Multi-Level Segmentation (AIMS), which segments images into three levels: part, entity, and relation. AIMS can also segment images based on mask prompts, which specify the region of interest. The paper proposes a unified AIMS model that uses multi-dataset multi-task training, task complementarity, association, and mask prompt encoder modules to address the challenges of annotation inconsistency and task correlation. The paper shows that the proposed method achieves better performance and generalization than existing methods on various segmentation tasks and datasets.", "questions": "Please refer to weakness.", "limitations": "Limitations such as failure cases and border impact should be discussed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The AIMS task setup is interesting, which also includes the deduction of relationships between instances. \n2. The prompt-based segmentation also makes sense, and it solves the multi-dataset training problem to some extent.\n3. The proposed method achieves SOTA performance.", "weaknesses": "1. From the initial description of the proposed new task, it was difficult to understand why it would make sense to segment visual regions in relation level. Please clarify it.\n2. The base framework consists of three separate decoders with a task complementarity module, However, the framework is relatively simple and lacks task-specific designs.\n3. Regarding the operation in Formula 5, does G_split mean decompose the embedded feature into two features? The meaning of the symbol should be clearly described.\n4. In Line 131, there is a mistake. \"L_ce denote binary cross-entropy loss\", It should be L_bce\n5. The ablations in In Table 3 is incomplete. Compared to the baseline with three separate decoders for three levels, the addition of the association module improve the association performance and has some negative impact on segmentation performance on PPP. To verify the effectiveness of association module, I suggest the authors to experiment with both MPE and TCM modules and report the results. \n6. It is not clear how entity-part and relation-entity association affect the final result separately. This should be included in the ablations.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments to improve our paper further.\n\n### Q1: Why it would make sense to segment visual regions in relation level?\n\nWe introduce relation-level segmentation results in entity pairs for two key reasons. \n\nFirst, entity pairs represent the minimal relational constructs that can depict the entire scene graph of an image. This representation facilitates downstream tasks associated with scene graph analysis or generalized referring expression segmentation. As demonstrated in Table 5(a) of our paper and the subsequent table, the AIMS model contributes to enhanced performance, outperforming state-of-the-art methods in both panoptic scene graph (COCOPSG dataset) and generalized referring expression segmentation task (gRefCOCO) [1].\n\n| method | cIoU | gIoU |\n| :-----: | :----: | :----: |\n| ReLA [1] | 52.26 | 54.44|\n| Ours | 52.83 | 54.96|\n\nSecond, the ability of our model to learn relation-level data allows it to learn richer and higher-level contextual features that can be used to discern visual entities. For example, given an image of a person sitting on a chair, the model can better segment the person and chair into two entity-level masks, if the model understands that some relation associates the person and chair. Although it is a relation-level subtask, it is beneficial to entity-level segmentation.  Our proposed task complementarity module can effectively propagate such useful relation-level information to the entity-level decoder. Furthermore, entity pairs are used to construct “pseudo” relation-level masks that serve as a form of mask prompt for our model to learn to split such relation-level masks into individual entity-level masks. This improves our model’s capability to split or subdivide images into masks. Table 1, 2 and 3 of our supplementary file show the benefits to the entity-level results by introducing the relation-level task.\n\n[1] GRES: Generalized Referring Expression Segmentation. CVPR 2023.\n\n\n---\n### Q2: The relative simplicity of the task complementarity module.\n\nOur baseline approach employs three distinct decoders for segmenting different levels, allowing each decoder to develop task-specific features for its particular task. The effectiveness of this separation, as opposed to using shared decoders, is substantiated in the comparison provided in the subsequent table. This comparison illustrates the advantages of our segmented approach, highlighting the strong need for separate network parameters for learning level-specific features. The values reported in PPP dataset are $AP^P$, $AP^E$ and $AR^{ER}$. The values reported in COCO-PSG dataset are $AP^R$, $AP^E$ and $AR^{RE}$. \n\n| Decoder Design | PPP | COCO-PSG |\n| :-----: | :----: | :----: |\n| Shared | 23.2/52.0/67.9 | 37.2/39.1/49.2 |\n| Separated | 24.5/53.4/69.7 | 38.9/40.4/50.9 |\n\nBuilding on the foundation of separated task-specific decoders, we introduce the task complementarity module to enable the sharing of each task's specific features with one another. Although this module is relatively simple, it proves efficient for multi-level segmentation in the Table 3(a) of our paper. Moreover, its streamlined design enhances adaptability across various base frameworks. For example, we successfully incorporate the task complementarity module into a Mask DINO by replicating the decoders three times for three-level segmentation (which serves as our baseline). The integration of this module in Mask DINO [2] follows a pattern similar to its implementation in Mask2Former, thereby showcasing the flexibility and wide applicability of our design. The values reported in PPP dataset are $AP^P$, $AP^E$ and $AR^{ER}$. The values reported in COCO-PSG dataset are $AP^R$, $AP^E$ and $AR^{RE}$. \n\n| Method | PPP | COCO-PSG |\n| ----- | :----: | :----: |\n| Mask DINO | 23.9/52.7/68.9 | 38.0/39.2/48.7 |\n| Mask DINO with TCM | 25.1/54.3/70.1 | 38.8/40.2/49.3 |\n\n[2] Mask dino: Towards a unified transformer-based framework for object detection and segmentation. CVPR 2023.\n\n\n---\n### Q3: Clarity issue on Formula 5.\n\nThe $G_{split}$ means splitting the tensor along the channel dimension, which is the opposite of the concatenation operation.\n\n\n---\n### Q4: The modification from L_ce to L_bce.\n\nThanks for pointing out this mistake. We will correct it in our paper.\n\n\n---\n### Q5: The ablation study on the baseline with TCM and MPE in Table 3(a)\n\nThe table below presents an ablation study of the association module. It can be observed that the association module primarily enhances the association performance, having only a positive impact on the cross-level (entity-part/EP and relation-entity/RE) results. This finding aligns with the observations made in Table 3(a), where only slight variations in performance were noted. The values reported in PPP dataset are $AP^P$, $AP^E$ and $AR^{ER}$. The values reported in COCO-PSG dataset are $AP^R$, $AP^E$ and $AR^{RE}$. \n\n| Design | PPP | COCO-PSG |\n| ----- | ---- | ---- |\n| Baseline | 24.5/53.4/69.7 | 38.9/40.4/50.9|\n| Basline with TCM and MPE | 26.4/55.9/71.4 | 40.5/42.0/52.6|\n| Basline with TCM, MPE and AM | 26.5/56.1/72.3 | 40.5/42.1/53.1 |\n\n\n---\n### Q6: Independent influence of entity-part and relation-entity association.\n\nThe table below illustrates the impact of two association-dependent modules on our final segmentation results. Notably, each association module exclusively influences the association performance between its corresponding two levels, without affecting the others. The values reported in PPP dataset are $AP^P$, $AP^E$ and $AR^{ER}$. The values reported in COCO-PSG dataset are $AP^R$, $AP^E$ and $AR^{RE}$. \n\n| Design | PPP | COCO-PSG |\n| ----- | ---- | ---- |\n| Baseline | 24.5/53.4/69.7 | 38.9/40.4/50.9|\n| Basline with EP association | 24.2/53.1/70.5 | 39.2/40.5/51.1 |\n| Basline with RE association | 24.4/53.3/69.9 | 39.0/40.5/52.0 |\n| Basline with EP and RE association | 24.3/53.1/70.6 | 39.0/40.5/52.0 |"}, {"review_id": "bjUgqxTrZO", "reviewer": "Reviewer_S47w", "summary": "This paper proposes a new task that segments visual regions into three levels: part, entity, and their relation. The authors build a unified AIMS model through multi-dataset multi-task training to address the two major challenges of annotation inconsistency and task correlation. Extensive experiments show the effectiveness of proposed approach.\n\nOverall, this paper is good to me in case of new concepts and unified modeling.", "questions": "Also, I still have some questions that require the author’s clarification.\n1, Could the author report the performance influence of using different architectural choices for Mask Prompt Encoder?\n\n2, Could the author split the part-level datasets into seen and unseen categories and test the model’s generalization ability to unseen categories?\n\n3. There are SAM models with different model sizes. Which SAM model is used for comparison? \n\nOverall, I appreciate and like the paper’s ideas and solution for all-inclusive multi-level segmentation. The proposed method can solve the annotation inconsistency among various datasets. Compared to using many annotators to label large-scale images at three levels, the AIMS method is more efficient in utilizing multiple datasets with different-level annotations. The paper does suffer from some weaknesses, but I think its benefits outweigh its weaknesses. I hope to see those issues addressed by the authors in the rebuttal.\n", "limitations": "None", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n-\tThe proposed AIMS task is interesting and novel to me, which unify several recent tasks, including entity segmentation, panoptic-part segmentation, and panoptic scene graph generation. It is a trend to unify multi-level and multi-dataset segmentation using a transformer architecture.  \n\n-\tThe proposed module including Task Correlation Module and R-E Complementarity Module are proven effectively in experiment part.\n\n-\tThe design of mask prompt is interesting for multi-dataset segmentation and part-whole segmentation.  The proposed method of multi-dataset multi-task training is novel. Especially the method proposes the mask prompt encoder to solve the incomplete annotations on three levels among datasets. This encoder can receive eight mask prompts and specify the corresponding region slated for segmentation. This way, the proposed method can relieve the annotation ambiguity among datasets.\n\n-\tThe ablation studies are good. The performance looks good compared with recent PanopticPartFormer++ and PSGTR. It also shows better generation than SAM. \n\n", "weaknesses": "\n\n- Presentation issues:\n   Several citations are missing. For example, L-118, “a masked cross-attention, self-attention, and feedforward network”，To be my knowledge, masked cross-attention is proposed by Mask2Former, it is better add a citation to the origin work. Also, is the pixel decoder the default decoder in Mask2Former? Moreover, several details are missing. It is hard to know the benefits \n\n- Several papers are missing:\n\t-Part-whole modeling paper:\n\n\t\t[1], Hierarchical Human Parsing with Typed Part-Relation Reasoning,CVPR-2020\n\n\t\t[2], Differentiable Multi-Granularity Human Representation Learning for Instance-Aware Human Semantic Parsing,CVPR-2021\n        Both works aims at solving part-whole segmentation, which is close to this work. \n\n\t-Multi-task query interaction paper:\n\n\t\t[1], Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition, ECCV-2022\n\n\t\t[2], PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation, ECCV-2022.\n\n\n\tBoth works adopt multiple task queries for joint reasoning and attention, which is similar to the proposed method in this work. \n\n        Moreover, some related works to unsupervised grouping. For example, \n        [1], Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers. CVPR2022.\n\n        [2], LASSIE:  Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery. NeurlPS2022.\n\n    These works should be discussed in the related work part. Following the last item, the writing of this work can be improved in the next drat. \n\n- The teaser could be a little fancier since it introduces a new concept. For example, it would be better to show the difference with previous segmentation settings. \n\n- Following the last item, an overview of different property settings including panoptic segmentation, entity segmentation, multi-dataset training, prompting can be compared in one table. \n\n- Is this possible to obtain the real results of PACO (MAP_{f1})? If so, how to extend this transformer architecture to achieve that. \n\n- Lacking a more comprehensive ablation study on the proposed modules (Table 3a). The authors only evaluated a few combinations. For example, what is the performance if MCE and TCM are present but AM is not? \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and insightful comments.\n\n### Q1: Clarity issue on the decoder.\n\nWe adhere to the transformer decoder design in the Mask2Former method, encompassing a set of learnable queries and nine transformer blocks. Each block is composed of a cross-attention layer, a self-attention layer, and a feed-forward network. In our AIMS decoder, we begin with three distinct transformer decoders designated for different levels of segmentation results, forming our baseline approach.\n\nBuilding upon this foundation, we introduce the task complementarity module to facilitate information fusion between the various levels, such as entity-part and relation-entity decoders. This innovative design was inspired by our observation that separated transformer decoders consistently outperform a shared transformer decoders, a finding that is demonstrated in our table. Essentially, this implies that different level decoders benefit from having unique features distinct from each other.\n\n\n---\n### Q2: Missing some related works.\n\nWe will include the following related works and discuss them: Papers [1] and [2] propose a hierarchical segmentation structure for part- and entity-level segmentation, but they are exclusively targeted at human parsing, not general objects. Although [3] and [4] utilize separate queries to enhance the results of different tasks, they cannot associate mask predictions of multiple levels and prompt the outcomes. Furthermore, some unsupervised methods, such as those proposed in [5,6], group the pixels using an unsupervised clustering approach. Despite their innovative approach, they still have a significant performance gap compared to supervised methods.\n\n[1] Hierarchical Human Parsing with Typed Part-Relation Reasoning, CVPR 2020.\n\n[2] Differentiable Multi-Granularity Human Representation Learning for Instance-Aware Human Semantic Parsing, CVPR 2021.\n\n[3] Fashionformer: A Simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition, ECCV 2022.\n\n[4] PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation, ECCV 2022.\n\n[5] Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers. CVPR 2022.\n\n[6] LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery. NeurlPS 2022.\n\n\n---\n### Q3: Changing the teaser figure for better illustration.\n\nPlease check figure 1 in our submitted pdf to manifest our multi-level targets against the single-level segmentation tasks.\n\n\n---\n### Q4: Adding the comparison to previous segmentation settings.\n\nThe table below summarizes the comparison of various settings, illustrating that our AIMS model encompasses all three levels.\n\n| Task | relation-level | entity-level | part-level |\n| :----: | :----: | :----: | :----: |\n| Part Segmentation | &cross; | &cross; | &check; |\n| Panoptic/Instance/Entity Segmentation | &cross; | &check; | &cross; |\n| Scene Graph | &check; | &cross; | &cross; |\n| Segment Anthing | &cross; | &check; | &check; |\n| AIMS (Ours) | &check; | &check; | &check; |\n\n\n---\n### Q5: More results on PACO.\nWe fine-tune our pretrained AIMS model directly on the class-aware PACO dataset, achieving 45.6 $AP^{obj}$ and 18.9 $AP^{opart}$. This compares favorably with the 43.4 $AP^{obj}$ and 17.7 $AP^{opart}$ obtained using the cascaded ViT-L FPN as reported in [7]. These results suggest that our AIMS model may serve as a superior pretraining model for fine-tuning in downstream tasks.\n\n[7] PACO: Parts and Attributes of Common Objects. arXiv 2023.\n\n\n---\n### Q6: Lacking an ablation study in Table 3(a).\n\nThe table below presents an ablation study of the association module. It can be observed that the association module primarily enhances the association performance, having only a positive impact on the segmentation results. This finding aligns with the observations made in Table 3(a), where only slight variations in performance were noted. The values reported in PPP dataset are $AP^P$, $AP^E$ and $AR^{ER}$. The values reported in COCO-PSG dataset are $AP^R$, $AP^E$ and $AR^{RE}$. \n\n| Design | PPP | COCO-PSG |\n| ----- | ---- | ---- |\n| Baseline | 24.5/53.4/69.7 | 38.9/40.4/50.9 |\n| Basline with TCM and MPE | 26.4/55.9/71.4 | 40.5/42.0/52.6 |\n| Basline with TCM, MPE and AM | 26.5/56.1/72.3 | 40.5/42.1/53.1 |\n\n\n---\n### Q7: The ablation study on mask prompt encoder structure.\n\nWe ablate the structure design of the mask prompt encoder in Table 4(a) of our paper.\n\n\n---\n### Q8: The experiments on part segmentation with seen/unseen data split.\n\nWe reorganized the PACO and PPP datasets, dividing them into 68 seen and 10 unseen object categories, and then trained only on the part with seen objects. In the following table, we compare the part segmentation results between our method and SAM on the validation dataset, including both seen and unseen categories. It becomes evident from the comparison that our model achieves performance comparable to SAM's, but with the advantage of utilizing much less training data.\n\n| Method | seen (AR@100) | unseen (AR@100) |\n| :-----: | :----: | :----: |\n| SAM | 46.6 | 46.2 |\n| Ours | 47.0 | 46.4 |\n\n\n---\n### Q9: Clarity of using the SAM model.\n\nFor the SAM model, we use the one with a VIT-Huge backbone, which is much larger than the Swin-Large backbone our model use.\n\nFor the computation cost, conducting full image inference under (800, 1333) image size in three levels requires 854.0 GFlops and takes approximately 0.142 seconds per image on A100 40G. This computation cost is comparable to the baseline method, which consumes about 801.0 GFlops and has an inference time of 0.138 seconds per image. For the SAM model we compare, it would cost 0.253 seconds per image within the same setting."}]

Author rebuttal:

We thank the reviewers for the insightful comments regarding our work. We have carefully addressed each of your concerns, and our responses can be found in the respective rebuttal sections for each reviewer. 


In addition, we have included a PDF containing three figures, as requested by some of the reviewers. We will refine our paper in light of your insightful advice. 


Meta-review: This submission received five reviews all of which are positive and propose acceptance. The authors address technical questions in the discussion to the satisfaction of the reviewers and some increased the scores. 

All reviewers acknowledge this paper as novel and they find the idea of combining a segmentation to include different tasks, especially also semantic relationship as interesting. The experiments are reported and appreciated as being better than current SOTA, comparison with current best performing models are included, (e.g. SAM) and the segmentation results come out better. 
In the weakness section of the reviews are mostly technical points that have been explained in the rebuttal. There is no major shortcoming that would stand in the way of accepting this version already. 

With five votes for acceptance this is a clear contribution to the field and a solid paper.
Decision: Accept (spotlight)