Title: AIMS: All-Inclusive Multi-Level Segmentation for Anything
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image Segmentation
Abstract: Despite the progress of image segmentation for accurate visual entity segmentation, completing the diverse requirements of image editing applications for different-level region-of-interest selections remains unsolved. In this paper, we propose a new task, All-Inclusive Multi-Level Segmentation (AIMS), which segments visual regions into three levels: part, entity, and relation (two entities with some semantic relationships). We also build a unified AIMS model through multi-dataset multi-task training to address the two major challenges of annotation inconsistency and task correlation. Specifically, we propose task complementarity, association, and prompt mask encoder for three-level predictions. Extensive experiments demonstrate the effectiveness and generalization capacity of our method compared to other state-of-the-art methods on a single dataset or the concurrent work on segment anything. We will make our code and training model publicly available.

AIMS: All-Inclusive Multi-Level Segmentation for Anything

Abstract

Despite the progress of image segmentation for accurate visual entity segmentation, completing the diverse requirements of image editing applications for differentlevel region-of-interest selections remains unsolved. In this paper, we propose a new task, All-Inclusive Multi-Level Segmentation (AIMS), which segments visual regions into three levels: part, entity, and relation (two entities with some semantic relationships). We also build a unified AIMS model through multi-dataset multi-task training to address the two major challenges of annotation inconsistency and task correlation. Specifically, we propose task complementarity, association, and prompt mask encoder for three-level predictions. Extensive experiments demonstrate the effectiveness and generalization capacity of our method compared to other state-of-the-art methods on a single dataset or the concurrent work on segment anything. We will make our code and training model publicly available.

1 Introduction

Decomposing an image into semantically meaningful regions is a key goal of image segmentation. With the recent emergence of photorealistic and high-quality image generation technologies [1, 2], image segmentation has gained considerable attention for its ability to select visual entities of interest within images. Combining image segmentation with image generation has enabled a variety of image editing and manipulation applications, such as altering the style of a specific object or eliminating a distracting element from a scene. Compared to bounding box-based object detection, segmentation methods preserve the integrity of pixels outside the edited or manipulated region.

Panoptic segmentation [3], with its instance awareness and dense segmentation capabilities, is a promising solution for various editing and manipulation applications. However, two main limitations remain when using it: (1) it struggles to handle classes encountered 'in the wild' but absent in the training set [4, 5]; and (2) existing methods typically do not explore the more general possibilities of segmentation masks across multiple abstraction levels ( e.g., a car's wheel , the whole car , both a person and car combined), with the exception of part-aware panoptic segmentation [6, 7] and panoptic scene graph [8]. These tasks also fail to generalize to classes beyond those in the training set, especially at the part-level task where annotations are sparse.

In this paper, we address the constraints inherent to different-level segmentation tasks by introducing AIMS (All-Inclusive Multi-Level Segmentation). AIMS represents a comprehensive framework designed to segment images across various hierarchical levels ( part , entity , relation ) using multiple query-driven image decoders. Drawing inspiration from recent advancements in Entity Segmentation [4, 5], which showcased remarkable generalization capabilities on unseen classes, AIMS performs class-agnostic segmentation across different hierarchical levels to enhance generalization.

Entity-Level (

𝑺𝒆

)

Relation-Level (

𝑺𝒓

)

ride

on

on

on

…

…

ID

Association

ID

Association

Relation-to-Entity

Entity-to-Part

Prompt

Inference

One-step

Inference

…

Part-Level (

𝑺𝒑

)

Image (

𝐈

)

Thanks to its hierarchical modeling and class-agnostic learning approach, AIMS provides more level-flexible and all-inclusive segmentation masks for users to select from, thereby enabling a wider range of image editing and manipulation applications. In Figure 1, the AIMS task has two modes of inference: one-step inference and prompt inference. In one-step inference, the model segments the image directly into three levels (part, entity, and relation level). Meanwhile, in prompt inference, the model separates the image further based on the provided mask. Notably, one-step inference necessitates an association of the results between any two neighboring levels.

Unlike existing part-aware panoptic or panoptic scene graph methods, AIMS model is not confined to training on a single dataset that has limited hierarchical levels. Instead, we adopt the strategy of multi-dataset training to build a unified AIMS model for all levels. To maintain consistency across multiple hierarchical levels sourced from different datasets, we introduce the Task Complementarity Module and the Association Module to AIMS model. These respectively facilitate feature interactions between different levels and establish explicit relationships between predictions at adjacent levels ( e.g., a car's part mask should correspond with the overall car mask, and a car mask should belong to a mask with a car and driver.).

We train our AIMS model on existing segmentation datasets such as Pascal Panoptic Parts [6], COCOPSG [8], PACO [9], and EntitySeg [5]. However, due to partial or incomplete annotations over the full image, with inconsistencies in unannotated areas across different datasets, sub-optimal supervision signals are generated during training. To address this issue, we propose a novel Mask Prompt Encoder for our AIMS model. This encoder segments an image based on a given mask prompt, which specifies the region slated for segmentation. We train AIMS to accommodate a diverse set of mask prompts, derived from various hierarchical levels and datasets while excluding unannotated regions. This encoder offers two distinct benefits: (1) it mitigates the inconsistency problem inherent in training with partial annotations from multiple datasets; (2) it exhibits strong generalizability to arbitrary mask prompts during inference, including those stemming from unseen classes and in-the-wild visual entities, as shown in Figure 5.

The main contributions of this work are as follows:

We introduce the AIMS task, which segments images into three distinct levels: part, entity, and relation. AIMS is capable of performing a one-step inference for the entire image and a prompt inference for a specified mask area within the image, which has great potential for image editing and manipulation applications.

We propose a unified AIMS model for this task, which incorporates multi-dataset and multi-task training. We have meticulously designed several modules, including the task complementary, association, and mask prompt encoder, to maintain the consistency of tasks or data annotations.

Extensive experiments and in-depth analysis showcase the effectiveness of our proposed AIMS model across various segmentation tasks and settings. We compare the results from our AIMS model with the popular segmentation models ( e.g ., SAM[10]), demonstrating the fine-grained segmentation capabilities and generalization to both seen and unseen categories.

Mask

Prompt

Encoder

Image

Encoder

𝐹 𝑠

′

Full Image

Mask Prompt

Image

Decoder

…

Part-level

Entity-level

Relation-level

All-inclusive segmentation for Anything

Association

2 Related Work

Image Segmentation Significant advancements [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21] have been made in image segmentation [22, 23, 3, 24, 25, 26] in recent years, with numerous methods and techniques emerging to handle various challenges through supervised learning. Notable works include FCN [11], DeepLab [12], PSPNet [13], Mask R-CNN [14], PANet [15], SOLO [16], CondInst [17], PanopticFPN [18], PanopticFCN [19], Max-Deeplab [20], and Mask2Former [21] for semantic, instance, and panoptic segmentation. Additionally, specialized tasks tailored to specific scenarios, such as part-aware panoptic segmentation [6, 27, 7, 28] and panoptic scene graphs [8, 29], have been developed. However, these tasks typically focus on single target and do not incorporate hierarchical concepts that are more in line with human behavior. This paper aims to address this problem by proposing a unified segmentation model that can organize pixels at three different levels: part, entity, and relation. We also introduce an interactive stage to decompose unseen regions into smaller entities or parts in an open-world setting.

Multi-Dataset Training Multi-dataset training has gained considerable attention in recent years, as it allows models to draw from multiple data sources to improve performance and generalization [30, 31, 32]. This approach has significantly benefited domain adaptation [33, 34, 35], dataset distillation [36, 37], curriculum learning [38, 39, 40, 41], data augmentation [42, 43, 44], federated learning [45, 46], and transfer learning [47, 48] in the context of image segmentation. This paper addresses the annotation inconsistency among multiple datasets for generalized hierarchical segmentation. Specifically, we tackle the issue of incomplete annotations in a single dataset across three levels using a prompt structure.

3 Method

As shown in Figure 1, given an image I ∈ R H × W × 3 , the AIMS task segments an image into three segmentation maps S T ∈ { 0 , 1 } N T × H × W at part, entity, and relation levels and two association maps A EP ∈ { 0 , 1 } N e × N p (entity and part levels) and A RE ∈ { 0 , 1 } N r × N e (relation and entity levels), where H and W are the height and width of the image, N T indicates N T prediction results in classagnostic binary format and T = { p , e , r } . The two association maps A EP and A RE are required to build the association relationships between two level prediction results. We chose these three levels since they align with the human brain's hierarchical information processing system [49, 50, 51] and fulfill the majority of requirements for image editing tools [52]. The outputs of AIMS are akin to those of related tasks like part-aware panoptic segmentation and panoptic scene graphs, but the AIMS task incorporates more levels and operates in a class-agnostic manner.

Like the existing Transformer-based segmentation methods [21], our AIMS model employs an image encoder and decoder to extract robust image features and decode them into segmentation masks. However, it differs in that it addresses the challenges of multi-dataset multi-task training. As illustrated in Figure 3, we first establish a task complementarity module and an association module to build the correlation of each task, followed by a mask prompt encoder to tackle the partial annotation problem among different datasets and enable generalized inference.

Next, we introduce our base framework and then elaborate on our three proposed components.

𝑄𝑟

𝑄e

𝑄𝑝

Image Decoder

𝐹 𝑠

′

𝐹 𝑠

′

𝐹 𝑠

′

𝐸𝑟

𝐸𝑝

𝐿𝑏𝑎𝑠𝑒

𝑟

𝐿𝑏𝑎𝑠𝑒

e

𝐿 𝑏𝑎𝑠𝑒

p

Relation-level

Decoder

R-E Complementarity

Module

Entity-level

Decoder

E-P Complementarity

Module

Part-level

Decoder

R-E Association

Module

E-P Association

Module

Task Complementarity Module

𝐿𝑎𝑠𝑠𝑜

𝑟𝑒

𝐿𝑎𝑠𝑠𝑜

𝑒𝑝

CONCAT

SA

SPLIT

FFN

R-E Complementarity Module

×9

CA

SA

×9

Entity-level Decoder

CA

SA

×9

FFN

Relation-level Decoder

CONCAT

SA

SPLIT

FFN

E-P Complementarity Module

×9

FFN

CA

SA

×9

Part-level Decoder

FFN

Task Correlation Module

𝐹 𝑠

′

𝑄𝑟

𝑄e

𝑄𝑝

𝐸𝑝

Association Module

FC

𝐸𝑟

FC

FC

Legend

：

Add

：

Multiplication

CA

：

Cross Attention

SA

：

Self Attention

FFN

：

feed-forward

3.1 Base Framework

The base framework of AIMS shares the same image encoder Θ I and has three independent decoders for different-level predictions. The image encoder consists of a backbone and pixel decoder, extracting multi-scale image features F s ∈ R H 2 s × W 2 s × 256 and s = { 2 , 3 , 4 , 5 } . We denote this image-encoding process as:

For the three independent decoders Φ , each decoder has 9 cascaded attention blocks, where each block includes a masked cross-attention, self-attention, and feedforward network. Each block of decoder iteratively refines query embeddings E T ∈ R N × 256 by interacting with multi-scale features F , where E T is initialized by a group of learnable parameters Q T . This iterative decoding process is denoted as

where Φ T and Φ i T indicate the decoder of T type and its attention blocks. i indicates the i th attention block. For each attention block, it includes cross-attention R i XAtt T , self-attention R i SAtt T , and feedforward network R i ff T . At last, E T is used for part-ness/entity-ness/relation-ness prediction and pixel-level mask prediction using the low-level image feature F 2 (derived from the image encoder). The prediction process can be formulated as:

where U c T and U m T denote the part-ness/entity-ness/relation-ness prediction and pixel-level mask outputs.

During training, we employ the losses L T for different level predictions, with respect to the corresponding ground truth G T . The overall training loss for our model is defined as:

where L ce T denote binary cross-entropy loss for part-/entity-/relation-ness prediction. Similarly, L bce T and L dice T denote the binary cross-entropy and dice loss for different level mask prediction.

Level

Relation

Entity

Part

COCO PPP COCO-PSG PACO EntitySeg

◦

✓

◦

✓

◦

◦

✓

✓

✓

◦

(a)

(b)

3.2 Task Complementarity Module

In our base method, we utilize three independent decoders for predictions at different hierarchy levels, which merely share the image encoder. To better exploit cross-task knowledge, we design the task complementarity module (TCM) that fuses task information within each attention block of the decoders. To provide a clearer explanation, we use the task complementarity module between the entity and part tasks as an example, and it works identically for the relation and entity tasks. As depicted in Figure 3, we concatenate the embeddings of two tasks following the self-attention of the attention block between two level decoders. Subsequently, we feed the concatenated embedding into a self-attention G i SAtt and feed-forward network G i SAtt and then separate processed embeddings back into their original dimensions:

where the E i e s and E i p s denote the embedding processed by self-attention in the attention block. After that, we acquire the final embedding by performing a straightforward addition between the fused and original embeddings:

Ultimately, we feed the enhanced embeddings at each level into the original feedforward network of the corresponding decoder:

✓

✓

◦

✓

◦

3.3 Association Module

Aside from generating predictions at various levels, the AIMS model also strives to establish the association between low-level and higher-level results. For example, the part masks of a car should be explicitly associated with the entire car's mask. In image editing and manipulation, this can provide a more intuitive mask selection experience. While it is possible to associate the masks across different levels by computing mask intersection, such association knowledge cannot be instilled into the model. Instead, we introduce an association module to AIMS to learn cross-level associations by applying an association loss to the similarity between embeddings. Besides having the association graph as an outcome, this also helps the model be aware of the cross-level association relationships and thus benefiting the individual-level task performance. The association loss L asso is defined as:

where FC indicates a fully-connected layer and · denotes dot product. The G ep ∈ { 0 , 1 } N e × N p and G re ∈ { 0 , 1 } N r × N e are the binary association ground truth of entity-part and relation-entity levels. Finally, the overall training loss for AIMS is

3.4 Mask Prompt Encoder

As aforementioned, we train AIMS with multiple existing segmentation datasets that each covers one or more hierarchy levels, whose details are provided in Table 1. Due to the use of predefined classes, most of the datasets suffer from the problem of partial annotations and have unannotated areas on the images. Furthermore, the unannotated areas appear and happen inconsistently among the multiple datasets ( e.g., 91.6% of PACO is unannotated, while it is 0.01% for EntitySeg). This leads to undesirable supervision signals that confuse the model because something that is treated as background in one dataset/level may qualify as foreground in another dataset/level.

Full

Image

…

Part

Decoder

Entity

Decoder

Relation

Decoder

Part

Decoder

Entity

Decoder

Relation

Decoder

Part

Decoder

Part

Decoder

Multiple

Entities

One

Entity

Entity

Decoder

Two

Entities

Entity

Decoder

Prompt

Target

Prompt

Target

Prompt

Target

117310 Wedesign the Mask Prompt Encoder (MPE), which allows our method to segment images conditioned on specific mask-controlled task prompts. As depicted in Figure 4, our approach integrates four types of mask prompts: full image, single entity, dual entities and multiple entities (where the upper limit is the entire annotated area). By employing diverse mask prompts along with multi-level mask decoders, AIMS can accomplish different segmentation goals. The mask prompts, covering either the full or partial image area, drive the model to segment the entire image or target area at part, entity, or relation levels using the corresponding decoders. This approach enables us to provide the model with a trustworthy supervision signal for the region to be segmented, even when dealing with datasets that lack full image annotations. Moreover, during inference, the mask prompt encoder can be applied to unrestricted mask prompts. To some extent, the model can subdivide the masks of unseen classes and/or in-the-wild visual entities into reasonable sub-masks, even if they are absent in training data.

In particular, the mask encoder accepts a binary mask M P , which designates the area to be segmented. Our mask encoder utilizes a series of cascaded convolution layers to downscale M P , resulting in M 2 ∈ R H 4 × W 4 × C m .

where the detailed structure of the MaskEncoder consists of three operation blocks including CONV2+LN+GELU, CONV2+LN+GELU and CONV1. The 'CONV2', 'CONV1', 'LN', and 'GELU' represent a convolution layer with stride and a kernel size of 2 or 1, layer normalization, and Gaussian error linear units, respectively. Subsequently, we directly downscale M 2 by 2 × , 4 × , and 8 × to obtain M s , where s ∈ 3 , 4 , 5 , using bilinear interpolation.

Ultimately, we acquire multi-scale fusion features F ′ s by summing the encoded image and mask features F s and M s , respectively. For AIMS with MPE, we replace F s in Eq. 2 with F ′ s :

Given MPE, we delve deeper into the sampling strategy implemented across various mask prompts and subtasks. To effectively balance contributions from each subtask and prompt type, we adopt a uniform sampling approach capable of considering multiple factors, such as subtask difficulty and the distribution of different prompt types within the training data. Specifically, in each training iteration, we have five subtasks: segmenting the entire annotated area (either the full image or multiple entities) based on three hierarchy levels, subdividing a single entity to parts, and subdividing two merged entities with relationships to two independent entities. Additionally, each subtask has its own data pool for sampling, taking into account the statistical annotation information of each dataset. For more details, please refer to our supplementary file.

4 Experiments

We construct our training set by aggregating images from five segmentation datasets, including COCO [53], EntitySeg [5], PascalVOC Part (PPP) [6], PACO [9], and COCO-PSG [8]. Given that

each dataset was originally created for a distinct task, we split each dataset into training and evaluation, and subsequently merge them to form unified training and evaluation sets. Overall, our training and evaluation sets have about 236.7K and 2.1K images. Please refer to our supplementary file for details on the split criteria. With the exception of COCO and EntitySeg, which are solely for entity-level prediction with full or partial image mask prompts, PPP dataset is utilized for entity/part prediction with full or partial image mask prompts and the entity-to-part task. COCO-PSG is employed for relation/entity prediction with full or partial image mask prompts and a relation-to-entities task. We keep only the relationship annotations in COCO-PSG that involve physical contact, to be consistent with typical part-whole relationships. PACO dataset is exclusively used for the entity-to-part task.

Training settings. We train our model for 36,000 iterations using a base learning rate of 0.0001 and weights pre-trained on COCO-Entity [3] with the exception of images contained in our validation set. The longer edge size of the images is set to 1,333 pixels, while the shorter edge size is randomly sampled between 640 and 800 pixels, with a stride of 32 pixels. The learning rate is decayed by a factor of 0.1 after 28,000 and 33,000 iterations, respectively. During each training iteration, we sample the data and tasks as introduced in the sampling strategy of Section 3.4, with a batch size of 64 on 8 A100 GPUs.

Federated evaluation. We build our evaluation set by collecting images from PPP [6] and COCOPSG [8] dataset, where each image only has two level annotations. Similar to LVIS [46], our evaluation images have non-exhaustive annotations. To better approach this, we organize the evaluation set into two subsets: one for entity-part and another for relation-entity. For each subset, we evaluate our model by the instance segmentation metric mean average precision (mAP) for each level and recall AR@100 for cross-level association predictions. For brief illustration, AP p , AP e , and AP r are mAP for part-, entity- and relation- predictions. The AR ep @100 and AR re @100 are metrics for entity-part and relation-entity association performance.

In the following, we first ablate our experiments with a Swin-Tiny [54] backbone and then report the quantitative performance and visualization results of our best model with the Swin-Large [54] backbone to other state-of-the-art methods.

4.1 Ablation Study

Multi-dataset training. Table 2 demonstrates the overall performance improvement by introducing multiple datasets. In the 1 st and 2 nd rows, we directly apply the base framework (baseline) with single dataset training and separate decoders. From the inference results, we observe that the model trained on the PPP dataset yields lower performance at the entity level when evaluated on the COCO-PSG dataset. This is primarily due to the fact that the COCO-PSG dataset contains 4 times more images than the PPP dataset, which does not provide sufficient entity-level annotations for the model. This issue can be alleviated through multi-dataset training, as depicted in the third row.

When we train our baseline method with all datasets, there are slight improvements or even degradation in part-level prediction for the PPP dataset, and relation and entity-level prediction for the COCO-PSG dataset. This can be attributed to annotation inconsistencies and incompleteness among multiple datasets, which results in ambiguous training signals for the model. By exploiting task complementarity (with Task Complementarity Module) and mitigating annotation issues (with MPE) , our proposed AIMS framework can improve performance on both evaluation datasets.

Improvement trajectory of the proposed method. In Table 3(a), we show the component-wise improvements over the baseline. The 1 st and last rows represent the performance of our baseline (three separate decoders for three levels) and the full AIMS framework. The middle three rows display the performance enhancement over the baseline method when integrated with different modules. It is evident that both MPE and TCM contribute to improvements in performance across all three

(b)

(a)

(b)

levels (mAP and association recall) on the two evaluation subsets, highlighting the effectiveness of these proposed modules on all levels. With the addition of the association module, a more significant improvement in association performance can be achieved while maintaining a similar level of segmentation performance compared to the baseline. This indicates that our proposed association module does not negatively impact the segmentation performance. Furthermore, all three modules can mutually benefit each other, giving the best performance as shown in the last row.

Mask Prompt Encoder In Table 3(b), we demonstrate the impact of using different mask prompt types during training. Note that only full-image mask prompts are used during inference. The 1 st and 2 nd rows show that full-image mask prompts do not bring any impact. In the third row, solely using a mask prompt based on the partial annotation area of the image results in a significant performance decline due to the mask prompt inconsistency between the training and inference stages, because inference only uses full-image prompts. However, when both full and partial image prompts are used, as shown in the 4 th row, better performance is achieved by simultaneously mitigating the incomplete annotation problem and maintaining training-inference consistency. As shown by the last row, the introduction of dual-entity or single-entity mask prompts further improves performance, by pushing the model to learn better how to split masks between two immediate adjacent levels (relation-toentities & entity-to-parts). This is different than (but complements) using full- or partial-image mask prompts which impose large gaps between the source and target levels ( e.g., image-to-parts).

In Table 4(a), we examine the influence of various mask encoder architectures The 1 st row represents our baseline without a prompt mask encoder. As demonstrated in the 2 nd and 4 th rows, we find that adding a 'CONV2' and 'CONV1' block with convolution kernel and stride sizes of 2 and 1 is sufficient for the mask encoder architecture. Incorporating more 'CONV1' blocks does not lead to a significant improvement in the final performance, as shown in the third row. This is because a single 'CONV1' layer is enough to encode mask information after the feature that has been already processed by two 'CONV2' blocks, as evident in the last two rows.

Task Complementarity Module . Table 4(b) presents the ablation study on the design of TCM. It is evident that the self-attention block is capable of fusing information from each level to enhance each respective task, as demonstrated in the 2 nd row. Moreover, adding the original level-specific embeddings back to fused embeddings is crucial for improving the final performance. This is because the level-specific embeddings carries unique information specific to each task.

Image

Prompt Mask

SAM

Ours

Image

Prompt Mask

SAM

Ours

(a)

(b)

4.2 Comparison to state-of-the-art methods

In the following comparison, we evaluate our model against other state-of-the-art methods. To ensure a fair comparison, all methods employ Swin-Large backbone, with the exception of SAM, which uses a ViT-Huge backbone [55].

Category-aware segmentation. We transfer our AIMS method to other class-aware segmentation tasks, including panoptic part segmentation and panoptic scene graph on PPP and COCO-PSG datasets, respectively. To equip our method with class-aware capability, we directly use our AIMS model as pre-trained weights to fine-tune the corresponding class-aware training dataset. In Table 5(a), we observe that our AIMS model outperforms the corresponding state-of-the-art methods on both tasks. This is because our AIMS model can utilize more training data from various datasets by employing our proposed prompt mask encoder to address annotation inconsistencies among them. Moreover, our AIMS model benefits from the two-stage process of class-agnostic pretraining and class-aware fine-tuning [38], which has been demonstrated to improve the performance of the second stage class-aware fine-tuning when class-agnostic pretraining is performed with more data.

Interactive segmentation. In Table 5(b), we compare AIMS to another popular interactive segmentation method, SAM [10], on the PACO and ADE20K [57] datasets, where ADE20K is for zero-shot evaluation at entity and part levels. Neither SAM nor AIMS uses ADE20K for training, and ADE20K's classes are fairly dissimilar to PascalVOC's [58]. The 1 st row represents our baseline method with purely class-agnostic training without interactive segmentation, which outperforms SAM at the pure entity level even though the SAM model is trained on 11 million images. This indicates that the zero-shot generalization ability is attributed to class-agnostic rather than interactive training. On the other hand, SAM's interactive segmentation can divide masks into more fine-grained segments, which our AIMS is also capable of. On both evaluation datasets, AIMS remarkably outperforms SAM, using merely a 273K training set that is far smaller than SAM's 11M training set. Our proposed MPE and TCM allow our model to leverage more desirable training signals and multi-level task information in a data-efficient manner. Furthermore, our AIMS model can make relation-level predictions, which are higher than part and entity levels and absent in the SAM model.

Figure 5 provides a visual comparison between the SAM [10] model and our model in a real-world scenario. Like AIMS, the SAM model also supports image segmentation at three levels (object,

Select Entity Mask

Full Image Prompt

Entity Encoder

One Entity Prompt

Part Encoder

Full Image Prompt

Part Encoder

Full Image Prompt

Entity Encoder

Select Relation Mask

Two Entities Prompt

Relation Encoder

Select Entity Mask

One Entity Prompt

Part Encoder

Due to the Dataset Annotation

(a)

(b)

part, and sub-part level). For an equitable comparison, we use the same entity mask prompt for further delineation. As shown in Figure 5, both models capably segment parts of the entity with distinct approaches. The SAM model leans towards identifying parts based on color and texture clues, whereas our AIMS model relies more on semantic understanding.

The flexibility of our AIMS task. Figure 7 illustrates how our proposed AIMS task provides the flexibility for segmenting anything. This is to address the subjective annotation issues in existing datasets. For instance, in the image (a), the throw pillows and the sofa are predicted as a single entity, which aligns with the ground truth annotation. Nonetheless, in certain scenarios, a user might wish to edit the throw pillows independently. With our AIMS method, utilizing a full image prompt for part-level prediction does not yield these separated masks. However, by selecting this entity mask as an entity prompt for part-level prediction, we can successfully differentiate the three throw pillows.

Additionally, in image (b), a user may want to segment the windows into several components. However, original ground truth annotations typically consider the two windows as a single entity. To meet this requirement, we initially utilize a full image prompt and an entity encoder to identify the mask of the whole windows. Following this, we apply this two-entity mask prompt and relation Decoder to split them into two independent window masks. Ultimately, we can select one window for further segmentation into two parts: the window edge and curtains.

5 Conclusion

This paper presents AIMS, a unified segmentation model that parses images at three levels: part, entity, and relation. To address the absence of a dataset with annotations for all three levels, we construct a unified dataset by aggregating multiple existing datasets, each providing annotations for one or two levels. Our base method uses three separate decoders for different-level predictions. To handle the annotation inconsistency issue, we propose Mask Prompt Encoder, which offers a more accurate task signal to the model, and Task Complementarity Module to enhance each level's prediction. Extensive experiments demonstrate the effectiveness of our method in both closed-world and zero-shot settings. Our work can serve as a springboard for further research on this new segmentation problem.

References

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. In arXiv , 2022. 1

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. In NeurlPS , 2022. 1

Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Doll´ ar. Panoptic segmentation. In CVPR , 2019. 1, 3, 7

Lu Qi, Jason Kuen, Yi Wang, Jiuxiang Gu, Hengshuang Zhao, Zhe Lin, Philip Torr, and Jiaya Jia. Open-world entity segmentation. In TPAMI , 2022. 1

Lu Qi, Jason Kuen, Weidong Guo, Tiancheng Shen, Jiuxiang Gu, Wenbo Li, Jiaya Jia, Zhe Lin, and Ming-Hsuan Yang. Fine-grained entity segmentation. 2022. 1, 2, 5, 6, 9, 14

Daan de Geus, Panagiotis Meletis, Chenyang Lu, Xiaoxiao Wen, and Gijs Dubbelman. Part-aware panoptic segmentation. In CVPR , 2021. 1, 2, 3, 5, 6, 7, 14

Xiangtai Li, Shilin Xu, Yibo Yang, Haobo Yuan, Guangliang Cheng, Yunhai Tong, Zhouchen Lin, and Dacheng Tao. Panopticpartformer++: A unified and decoupled view for panoptic part segmentation. In arXiv , 2023. 1, 3, 9

Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph generation. In ECCV , 2022. 1, 2, 3, 5, 6, 7, 9, 14

Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In arXiv , 2023. 2, 5, 6, 9, 14

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In arXiv , 2023. 2, 9, 14, 19

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR , 2015. 3

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. In TPAMI , 2017. 3

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR , 2017. 3

Kaiming He, Georgia Gkioxari, Piotr Doll´ ar, and Ross Girshick. Mask r-cnn. In ICCV , 2017. 3

Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In CVPR , 2018. 3

Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In ECCV , 2020. 3

Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV , 2020. 3

Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Doll´ ar. Panoptic feature pyramid networks. In CVPR , 2019. 3

Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. Fully convolutional networks for panoptic segmentation. In CVPR , 2021. 3

Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In CVPR , 2021. 3

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Maskedattention mask transformer for universal image segmentation. In CVPR , 2022. 3

Hongshan Yu, Zhengeng Yang, Lei Tan, Yaonan Wang, Wei Sun, Mingui Sun, and Yandong Tang. Methods and datasets on semantic segmentation: A review. Neurocomputing , 2018. 3

Abdul Mueed Hafiz and Ghulam Mohiuddin Bhat. A survey on instance segmentation: state of the art. International journal of multimedia information retrieval , 2020. 3

Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Amodal instance segmentation with kins dataset. In CVPR , 2019. 3

Tiancheng Shen, Yuechen Zhang, Lu Qi, Jason Kuen, Xingyu Xie, Jianlong Wu, Zhe Lin, and Jiaya Jia. High quality segmentation for ultra high-resolution images. In CVPR , 2022. 3

Liulei Li, Tianfei Zhou, Wenguan Wang, Jianwu Li, and Yi Yang. Deep hierarchical semantic segmentation. In CVPR , 2022. 3

Xiangtai Li, Shilin Xu, Yibo Yang, Guangliang Cheng, Yunhai Tong, and Dacheng Tao. Panopticpartformer: Learning a unified model for panoptic part segmentation. In ECCV , 2022. 3

Fangting Xia, Peng Wang, Xianjie Chen, and Alan L Yuille. Joint multi-person pose estimation and semantic part segmentation. In CVPR , 2017. 3

Zijian Zhou, Miaojing Shi, and Holger Caesar. Hilo: Exploiting high low frequency relations for unbiased panoptic scene graph generation. In arXiv , 2023. 3

Xingyi Zhou, Vladlen Koltun, and Philipp Kr¨ ahenb¨ uhl. Simple multi-dataset detection. In CVPR , 2022. 3

Georgios Kapidis, Ronald Poppe, and Remco C Veltkamp. Multi-dataset, multitask learning of egocentric vision tasks. TPAMI , 2021. 3

Qiang Zhou, Yuang Liu, Chaohui Yu, Jingliang Li, Zhibin Wang, and Fan Wang. Lmseg: Language-guided multi-dataset segmentation. In arXiv , 2023. 3

Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Universal domain adaptation. In CVPR , 2019. 3

Antoine Saporta, Arthur Douillard, Tuan-Hung Vu, Patrick P´ erez, and Matthieu Cord. Multi-head distillation for continual unsupervised domain adaptation in semantic segmentation. In CVPR , 2022. 3

Duo Peng, Yinjie Lei, Munawar Hayat, Yulan Guo, and Wen Li. Semantic-aware domain generalized segmentation. In CVPR , 2022. 3

Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In ICML , 2019. 3

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. In arXiv , 2018. 3

Lu Qi, Jason Kuen, Zhe Lin, Jiuxiang Gu, Fengyun Rao, Dian Li, Weidong Guo, Zhen Wen, Ming-Hsuan Yang, and Jiaya Jia. Ca-ssl: Class-agnostic semi-supervised learning for detection and segmentation. In ECCV , 2022. 3, 9

Yang Zhang, Philip David, and Boqing Gong. Curriculum domain adaptation for semantic segmentation of urban scenes. In ICCV , 2017. 3

Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Guided curriculum model adaptation and uncertaintyaware evaluation for semantic nighttime image segmentation. In ICCV , 2019. 3

Lu Qi, Shu Liu, Jianping Shi, and Jiaya Jia. Sequential context encoding for duplicate removal. In Advances in Neural Information Processing Systems , 2018. 3

Yukang Chen, Peizhen Zhang, Tao Kong, Yanwei Li, Xiangyu Zhang, Lu Qi, Jian Sun, and Jiaya Jia. Scale-aware automatic augmentations for object detection with dynamic training. In TPAMI , 2022. 3

Yukang Chen, Yanwei Li, Tao Kong, Lu Qi, Ruihang Chu, Lei Li, and Jiaya Jia. Scale-aware automatic augmentation for object detection. In CVPR , 2021. 3

Lu Qi, Jason Kuen, Jiuxiang Gu, Zhe Lin, Yi Wang, Yukang Chen, Yanwei Li, and Jiaya Jia. Multi-scale aligned distillation for low-resolution detection. In CVPR , 2021. 3

Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In CVPR , 2021. 3

Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR , 2019. 3, 7

Davood Karimi, Simon K Warfield, and Ali Gholipour. Transfer learning in medical image segmentation: New insights from analysis of the dynamics of model parameters and learned representations. Artificial intelligence in medicine , 2021. 3

Ya-Yen Tsai, Bidan Huang, Yao Guo, and Guang-Zhong Yang. Transfer learning for surgical task segmentation. In ICRA , 2019. 3

David Marr. Vision: A computational investigation into the human representation and processing of visual information . San Francisco: W.H. Freeman. Print., 1982. 3

Leslie G Ungerleider and James V Haxby. 'what'and 'where'in the human brain. Current opinion in neurobiology , 4(2):157-165, 1994. 3

Danielle S Bassett and Michael S Gazzaniga. Understanding complexity in the human brain. Trends in cognitive sciences , 15(5):200-209, 2011. 3

Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. In CVPR , 2023. 3

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014. 5, 6, 14

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , 2021. 7

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In arXiv , 2020. 9

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS , 2012. 9

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR , 2017. 9

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for openvocabulary semantic segmentation. In arXiv , 2023. 9

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. IJCV , 2020. 16, 20

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In NeurIPS Workshop , 2021. 16, 20

Appendix

In this supplementary file, we provide more experimental details and empirical results to further demonstrate the benefits of our proposed All-Inclusive Multi-Level segmentation task (AIMS).

Experimental details on dataset split, sampling strategy and SAM's [10] configuration.

More ablation studies on sample strategy and dataset usage.

More visualization results of our AIMS model with various inference modes.

User study of AIMS and the concurrent work SAM [10]

The code and models to reproduce our experiments will be released.

A Experimental Details

Dataset Split As outlined in the main paper, all models are trained using five datasets: COCO [53], EntitySeg [5], PascalVOC Part (PPP)[6], PACO[9], and COCO-PSG [8]. Given that the original training and validation splits of these datasets are tailored for single tasks, we collate the images and reorganize them to suit our AIMS task. Initially, we select 1069 and 1000 validation images from PPP [6] (which covers the part and entity levels) and COCO-PSG [8] (which covers the entity and relation levels) respectively. Following this, we eliminate any duplicate images in the unified training set that are present in the validation images, resulting in a refined training set comprised of approximately 236.7K unique images.

Sampling Strategy In Table 7, we present the eight different sample types, labeled as Sample ID 1 through 8, for each iteration, assuming a batch size of 8. Additionally, images for each of these sample types are uniformly selected from the datasets mentioned in the 'Dataset' column.

SAM's configuration To obtain more fine-grained part-level predictions, we follow the hyperparameters found on SAM's GitHub page: points per side (2), pred iou thresh (0.86), stability score thresh (0.92), crop n layers (1), crop n points downscale factor (2), min mask region area (100).

B Ablation Studies

Sample Strategy Table 7 presents an ablation study on various sample strategies by examining the impact of each sample type during each iteration. The first row serves as our baseline, adhering to our base framework with a split decoder for varying-level predictions, and it excludes the usage of a prompt mask encoder. The remaining rows illustrate the influence on performance when task prompts are employed during each training iteration.

For instance, the second row reveals that the exclusive usage of a full image task prompt brings a marginal performance improvement over the baseline. Since full-image mask prompts are always the same, injecting them into the original image features is identical to not using any mask prompts. Conversely, the third row shows that employing solely partial-image mask prompts can considerably deteriorate performance, which is inconsistent with the inference process that involves full-image mask prompts.

In the fourth row, we observe that using both full and partial mask prompts can further enhance performance by providing more accurate signals to the network. Additionally, the introduction of mask prompts at the entity-to-part and relation-to-entity levels consistently yields performance improvements, as depicted in subsequent rows.

Dataset Usage Table 8 displays an ablation study focusing on the utilization of various datasets. The first three rows present the model performance when trained on a single dataset. This results in degraded performance on cross-dataset evaluation due to inconsistencies in annotations across different datasets. As shown in the fourth row, combining the three datasets for training enhances the overall validation performance. This improvement is due to the consistency maintained between the training and validation splits. Incorporating PACO and Entityseg datasets at part and entity levels further improve the performance for the respective levels.

C Visualization

The flexibility of our AIMS task. Figure 7 illustrates how our proposed AIMS task provides the flexibility for segmenting anything. This is to address the subjective annotation issues in existing datasets. For instance, in the image (a), the throw pillows and the sofa are predicted as a single entity, which aligns with the ground truth annotation. Nonetheless, in certain scenarios, a user might wish to edit the throw pillows independently. With our AIMS method, utilizing a full image prompt for part-level prediction does not yield these separated masks. However, by selecting this entity mask as an entity prompt for part-level prediction, we can successfully differentiate the three throw pillows.

Additionally, in image (b), a user may want to segment the windows into several components. However, original ground truth annotations typically consider the two windows as a single entity. To meet this requirement, we initially utilize a full image prompt and an entity encoder to identify the mask of the whole windows. Following this, we apply this two-entity mask prompt and relation Decoder to split them into two independent window masks. Ultimately, we can select one window for further segmentation into two parts: the window edge and curtains.

Prediction consistency. We investigate the prediction consistency across two inference modes: full and partial image prompts. Figure 8 displays an example where our AIMS model fails to segment anything at the part level with a full image prompt, but it is able to break down an entity-level mask prompt into a more detailed level, similar to the sofa and throw pillows in the image. Given that the pillows have never been labeled in our utilized dataset, we surmise that our AIMS model might yield inconsistent prediction results for unseen classes when different prompts are used. Using entity-level mask prompts could help our model to drill down to the next level.

Select Entity Mask

Full Image Prompt

Entity Encoder

One Entity Prompt

Part Encoder

Full Image Prompt

Part Encoder

Full Image Prompt

Entity Encoder

Select Relation Mask

Two Entities Prompt

Relation Encoder

Select Entity Mask

One Entity Prompt

Part Encoder

Due to the Dataset Annotation

(a)

(b)

Entity-level Results

Part-level Results

full image prompt

one entity prompt

However, the scenario changes when we turn our attention to the known classes. Figure 9 illustrates that three cars yield similar part-level prediction results, regardless of whether a full image or a single entity prompt is used. This indicates that our AIMS model can maintain prediction consistency for known classes.

The three-level predictions Figure 10 displays the three-level prediction outcomes of our AIMS model. Despite the potential subjectivity in defining the three levels, our AIMS model shows strong promises in fulfilling various user needs and intentions in image editing. For instance, the red region in the image is not detected at the entity-level prediction results, as it is merged into the tray. However, by injecting an entity-level mask prompt for the tray region into the part encoder, we see that the separate red region can be obtained, as shown in the second column of the 'part-level results'. Additionally, the relation-level masks of any two entities that share some semantic relationship can be predicted, and all relation-level prediction results can be represented in a scene graph.

More visualization results. In the Fig. 13, we show more visualization results at entity and part level in the wild (Open-Image [59] dataset), manifesting the generalization ability of our proposed AIMS model at the entity and part level. Considering the entity level is the principal part of image editing, we also show some cases on Laion400M [60] to show the effectiveness of our AIMS model.

Entity-level Results

Part-level Results

full image prompt

one entity prompt

一→

一→

→

D User Study

We conducted a user study in which there were 480 individuals who were identified as Adobe Photoshop users who regularly used the software for image manipulation/editing. In this user study, we randomly selected 40 images in the wild and provided the users with a visualization of the threelevel prediction results of our AIMS model, as shown in Figure 10. For each image, we asked each user about his/her degree of satisfaction with treating semantically meaningful and -coherent segments for three levels, with respect to their relevance to and suitability for image manipulation/editing applications. The satisfactory scores are aggregated from all users for the individual images. We find that the average score of each image is large than 7.8 on the condition that the maximum score is 10. Most of the selected images' scores are larger than 6.0, This confirms that the users are highly satisfied with our model's prediction results in the context of image manipulation/editing. To better present the user study's findings, we summarize the data of Table 9. The image IDs with the minimum, median, and maximum user scores are 20, 19, and 35. We show these images and their corresponding entity annotations in Fig. 12.

In Fig. 12, we provide statistical information about our user study. The sub-figure (a) is a summary of Table 9, indicating that over 90% of users rated our prediction results higher than 6.0. Sub-figure (b) contrasts user preferences between our results and those from the SAM model. Despite being trained with fewer data, our model garners an appreciation comparable to that of SAM, as can be observed in the sub-figure (b).

Entity-level Results

Part-level Results

Relation-level Results (Scene Graph)

n

Image-ID 20 (7.83)

Image-ID 34 (8.27)

Image-ID 19 (8.00)

Image-ID 15 (8.05)

Image-ID 22 (7.96)

Image-ID 32 (8.23)

TL-450Jatoba

The Score Distribution

ThePreference toSAM orours

35

E'TE

100

Ours

SAM

30

80

25

(%) ogey

20

(%) oge8

60

18.3

15

12.4

13.9

40

10

11.0

6.1

20

0.1

0.5

1.0

[001°06] (06°08] (08°01 (0°09] (09°05] (09°0]  (0+0℃] (00'02] (0°01] (01°0)

Score Range

10

11

Image ID

电

口

111

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new task that segments visual regions into three levels: part, entity, and their relation. The authors build a unified AIMS model through multi-dataset multi-task training to address the two major challenges of annotation inconsistency and task correlation. Extensive experiments show the effectiveness of proposed approach.\n\nOverall, this paper is good to me in case of new concepts and unified modeling.\n\nSTRENGTHS:\n-\tThe proposed AIMS task is interesting and novel to me, which unify several recent tasks, including entity segmentation, panoptic-part segmentation, and panoptic scene graph generation. It is a trend to unify multi-level and multi-dataset segmentation using a transformer architecture.  \n\n-\tThe proposed module including Task Correlation Module and R-E Complementarity Module are proven effectively in experiment part.\n\n-\tThe design of mask prompt is interesting for multi-dataset segmentation and part-whole segmentation.  The proposed method of multi-dataset multi-task training is novel. Especially the method proposes the mask prompt encoder to solve the incomplete annotations on three levels among datasets. This encoder can receive eight mask prompts and specify the corresponding region slated for segmentation. This way, the proposed method can relieve the annotation ambiguity among datasets.\n\n-\tThe ablation studies are good. The performance looks good compared with recent PanopticPartFormer++ and PSGTR. It also shows better generation than SAM.\n\nWEAKNESSES:\n- Presentation issues:\n   Several citations are missing. For example, L-118, “a masked cross-attention, self-attention, and feedforward network”，To be my knowledge, masked cross-attention is proposed by Mask2Former, it is better add a citation to the origin work. Also, is the pixel decoder the default decoder in Mask2Former? Moreover, several details are missing. It is hard to know the benefits \n\n- Several papers are missing:\n\t-Part-whole modeling paper:\n\n\t\t[1], Hierarchical Human Parsing with Typed Part-Relation Reasoning,CVPR-2020\n\n\t\t[2], Differentiable Multi-Granularity Human Representation Learning for Instance-Aware Human Semantic Parsing,CVPR-2021\n        Both works aims at solving part-whole segmentation, which is close to this work. \n\n\t-Multi-task query interaction paper:\n\n\t\t[1], Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition, ECCV-2022\n\n\t\t[2], PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation, ECCV-2022.\n\n\n\tBoth works adopt multiple task queries for joint reasoning and attention, which is similar to the proposed method in this work. \n\n        Moreover, some related works to unsupervised grouping. For example, \n        [1], Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers. CVPR2022.\n\n        [2], LASSIE:  Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery. NeurlPS2022.\n\n    These works should be discussed in the related work part. Following the last item, the writing of this work can be improved in the next drat. \n\n- The teaser could be a little fancier since it introduces a new concept. For example, it would be better to show the difference with previous segmentation settings. \n\n- Following the last item, an overview of different property settings including panoptic segmentation, entity segmentation, multi-dataset training, prompting can be compared in one table. \n\n- Is this possible to obtain the real results of PACO (MAP_{f1})? If so, how to extend this transformer architecture to achieve that. \n\n- Lacking a more comprehensive ablation study on the proposed modules (Table 3a). The authors only evaluated a few combinations. For example, what is the performance if MCE and TCM are present but AM is not?\n\nQUESTIONS:\nAlso, I still have some questions that require the author’s clarification.\n1, Could the author report the performance influence of using different architectural choices for Mask Prompt Encoder?\n\n2, Could the author split the part-level datasets into seen and unseen categories and test the model’s generalization ability to unseen categories?\n\n3. There are SAM models with different model sizes. Which SAM model is used for comparison? \n\nOverall, I appreciate and like the paper’s ideas and solution for all-inclusive multi-level segmentation. The proposed method can solve the annotation inconsistency among various datasets. Compared to using many annotators to label large-scale images at three levels, the AIMS method is more efficient in utilizing multiple datasets with different-level annotations. The paper does suffer from some weaknesses, but I think its benefits outweigh its weaknesses. I hope to see those issues addressed by the authors in the rebuttal.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new task and a model for All-Inclusive Multi-Level Segmentation (AIMS), which segments images into three levels: part, entity, and relation. AIMS can also segment images based on mask prompts, which specify the region of interest. The paper proposes a unified AIMS model that uses multi-dataset multi-task training, task complementarity, association, and mask prompt encoder modules to address the challenges of annotation inconsistency and task correlation. The paper shows that the proposed method achieves better performance and generalization than existing methods on various segmentation tasks and datasets.\n\nSTRENGTHS:\n1. The AIMS task setup is interesting, which also includes the deduction of relationships between instances. \n2. The prompt-based segmentation also makes sense, and it solves the multi-dataset training problem to some extent.\n3. The proposed method achieves SOTA performance.\n\nWEAKNESSES:\n1. From the initial description of the proposed new task, it was difficult to understand why it would make sense to segment visual regions in relation level. Please clarify it.\n2. The base framework consists of three separate decoders with a task complementarity module, However, the framework is relatively simple and lacks task-specific designs.\n3. Regarding the operation in Formula 5, does G_split mean decompose the embedded feature into two features? The meaning of the symbol should be clearly described.\n4. In Line 131, there is a mistake. \"L_ce denote binary cross-entropy loss\", It should be L_bce\n5. The ablations in In Table 3 is incomplete. Compared to the baseline with three separate decoders for three levels, the addition of the association module improve the association performance and has some negative impact on segmentation performance on PPP. To verify the effectiveness of association module, I suggest the authors to experiment with both MPE and TCM modules and report the results. \n6. It is not clear how entity-part and relation-entity association affect the final result separately. This should be included in the ablations.\n\nQUESTIONS:\nPlease refer to weakness.\n\nLIMITATIONS:\nLimitations such as failure cases and border impact should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents AIMS, a multi-level image segmentation model with levels representing parts, instance, and relation. Further, a curated dataset is created from several existing segmentation datasets. AIMS outperforms the baselines on the curated dataset.\n\nSTRENGTHS:\n1. The proposed architecture reasonably bridges information across levels.\n2. The quantitative and qualitative performance are good.\n3. Experiments, especially the ablations, are comprehensive.\n\nWEAKNESSES:\n1. It's unclear how the prompt mask is obtained and used during inference. When decoding with such a mask, would its experiments against baselines unfair comparisons since baselines might not have such additional localization information?\n2. The comparison with SAM is partially done. Especially that SAM would provide fine segmentation masks with mask prompt with low-accuracy, while AIMS has not mentioned explicitly the quality of the mask prompt -- based on Figure 3 the masks seem to be accurate.\n3. The presentation can be generally improved. Figure 2 provides details that distracts readers without further knowledge of more details, whilst the mask prompt encoder can be motivated earlier.\n\nQUESTIONS:\n1. Does the training set aggregation make any modification of the original datasets/annotations? Would the curated dataset (split) be released?\n2. The comparison in Figure 4 only shows part segmentation. What would the comparison be when it comes to multi-instance level?\n3. In table 5(b), shouldn't the $AR^E$ score for EntitySeg be bold since 87.2 > 87.1?\n\nLIMITATIONS:\nLimitations are not discussed by the authors. Apart from potential limitations in Weaknesses, I see there are these limitations:\n1. The computational cost is unspecified.\n2. The complementarity module and association module are designed to couple with the base framework, thus not obviously usable by other architectures.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel task called All-Inclusive Multi-Level Segmentation (AIMS) and proposes a unified AIMS model to address the challenges of annotation inconsistency and task correlation. The model consists of a shared image encoder and three independent decoders for part, entity, and relation predictions. It incorporates a Task Complementarity Module (TCM) to fuse task information and an Association Module to establish associations between different levels of segmentation. The model is trained using a **combination** of existing segmentation datasets. The model also incorporates a Mask Prompt Encoder (MPE) to provide supervision signals. \n\nThe proposed method outperforms SOTA methods on both panoptic part segmentation and panoptic scene graph estimation tasks. The proposed method also outperforms the Segment Anything model (SAM) on entity-level segmentation, even though the proposed method was trained on much less data than SAM.\n\nSTRENGTHS:\n### Novel approach to a multi-level segmentation task\n* The paper introduces a novel task called All-Inclusive Multi-Level Segmentation (AIMS) and proposes a unified model to address the challenges of annotation inconsistency and task correlation. This task formulation and the proposed model are unique and provide a fresh perspective on multi-level segmentation.\n\n### Comprehensive experiments with detailed ablations and good results\n* The paper presents a comprehensive evaluation of the proposed method, comparing it with state-of-the-art methods and conducting ablation studies. \n* The experiments are well-designed, and the results demonstrate the effectiveness and generalization capacity of the proposed method.\n* The authors also provide detailed explanations of the model components and training settings, ensuring reproducibility.\n\nI also appreciate the clarity of writing in this paper which has a thorough analysis of problems in existing segmentation frameworks motivating the need for the AIMS task.\n\nWEAKNESSES:\n### Missing performance analysis on images with a large number of objects\n* SAM has shown impressive performance on images with a large number of objects. How does the proposed method fare in such conditions? It would be useful to break down the performance on the basis of the number of objects and compare it with SAM.\n\nQUESTIONS:\n* The authors mention that Swin-Large backbone provides the best results, but can you provide more details on the decoder architecture? Also, what is the total model size and computational requirement (FLOPs) of the model?\n* The legend in Figure 2 is missing the symbol for \"Add\"\n\nLIMITATIONS:\nNo limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a unified multi-level segmentation (AIMS) approach. For better generalisation, the model is concurrently trained on multiple dataset consisting varied hierarchical level annotations across parts, entities and relations. In order to utilise signals from multiple hierarchical level as well as infuse model with level-awareness for each training sample, AIMS proposes three modules - task complementarity, association and mask prompt encoder. Experimental evaluation shows AIMS performing better than other state-of-the-art class-aware as well as class-agnostic segmentation.\n\nSTRENGTHS:\nOriginality\nAlthough model is inspired from Mask2former, the inclusion of other novel modules packages AIMS for multi-level hierarchical segmentation. Thus I note sufficient novelty in the proposed architecture. \n\nQuality\nThe paper is well-written and easy to follow. Authors can be more clearer upfront about AIMS being complete class-agnostic model (i.e., mask proposal model) and there is no semantic labelling.\n\nSignificance\n1. As far as I am aware of, AIMS is the first segmentation model to be trained with multi level semantic understanding. Though SAM is trained with fine-grained annotations it lacks part- and relation-level understanding and thus certain fine-grained semantic features may not be fully segmented out.\n2. Moreover, training with multiple datasets enables AIMS to be deployed for range of image editing and manipulation applications. \n3. AIMS achieves these advantages with as minimal data as possible (237K vs 11M in SAM).\n\nWEAKNESSES:\n1. As pointed earlier, AIMS is class-agnostic. Hence it cannot label the output segments. \n2. AIMS use interactive settings by deploying mask prompt encoder (MPE) to specify model what to segment. However, MPE usage is limited to providing unmask region of an input image for model to act on. There is no provision to use text, bounding box, or points as inputs. This is very limited interactive settings.\n3. For the same reason though powerful, AIMS cannot extract referring expression segmentation. This is despite the fact that AIMS is trained with semantic of relation understanding.\n4. Overall, the model is novel but the evaluation lacks distinguishing aspects that AIMS in the very first place was designed for.\n\nQUESTIONS:\nAIMS model have three separate output while other models output single mask irrespective of entity our parts. So in table 5(b), how do one measure seperate AR for parts and entity segmentation. Do you run seperate inference with different prompt for part level and entity level ? Or full-image mask prompt is applied ?\n\nLIMITATIONS:\nAuthors haven't addressed any limitations. Some of the listed weaknesses above can be elaborated to address the limitations the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}