DI-MaskDINO: A Joint Object Detection and Instance Segmentation Model

Authors: zhixiong  nan
xianghong  li
tao  xiang
jifeng  dai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 04:44:00

Abstract: This paper is motivated by an interesting phenomenon: the performance of object detection lags behind that of instance segmentation (i.e., performance imbalance) when investigating the intermediate results from the beginning transformer decoder layer of MaskDINO (i.e., the SOTA model for joint detection and segmentation). This phenomenon inspires us to think about a question: will the performance imbalance at the beginning layer of transformer decoder constrain the upper bound of the final performance? With this question in mind, we further conduct qualitative and quantitative pre-experiments, which validate the negative impact of detection-segmentation imbalance issue on the model performance. To address this issue, this paper proposes DI-MaskDINO model, the core idea of which is to improve the final performance by alleviating the detection-segmentation imbalance. DI-MaskDINO is implemented by configuring our proposed De-Imbalance (DI) module and Balance-Aware Tokens Optimization (BATO) module to MaskDINO. DI is responsible for generating balance-aware query, and BATO uses the balance-aware query to guide the optimization of the initial feature tokens. The balance-aware query and optimized feature tokens are respectively taken as the Query and Key&Value of transformer decoder to perform joint object detection and instance segmentation. DI-MaskDINO outperforms existing joint object detection and instance segmentation models on COCO and BDD100K benchmarks, achieving +1.2 $AP^{box}$ and +0.9 $AP^{mask}$ improvements compared to SOTA joint detection and segmentation model MaskDINO. In addition, DI-MaskDINO also obtains +1.0 $AP^{box}$ improvement compared to SOTA object detection model DINO and +3.0 $AP^{mask}$ improvement compared to SOTA segmentation model Mask2Former.

Main paper: 
 introduction: 
 Object detection and instance segmentation are two fundamental tasks in the computer vision community. Intuitively, the two tasks are closely-related and mutually-beneficial. However, in the current time, specialized detection or segmentation gains more focuses, and the amount of works studying the specialized task is significantly larger than that for joint tasks. One typical explanatory is that the multi-task training even hurts the performance of the individual task.Confronting current research situations, we think about whether there are some essential cruxes that are ignored in previous works and these cruxes hinder the cooperation of object detection and instance segmentation tasks, which further constrains the breakthrough of the performance upper bound. This paper reveals one of cruxes is the imbalance of object detection and instance segmentation. As shown in Fig. 1 , when investigating the results at the first layer of transformer decoder of MaskDINO model [25] , an interesting phenomenon is found that there exists the performance imbalance between object detection and instance segmentation, as qualitatively illustrated in Fig. 1a and quantitatively illustrated in the first bar of Fig. 1c . After considering the imbalance issue, the performance gap at the first layer is narrowed as illustrated in the first bar of Fig. 1d , and the final performance upper bounds are improved (i.e., 28.1 to 29.5 for detection and 25.3 to 25.7 for segmentation) as illustrated in the second bar of Fig. 1d . The qualitative results are also optimized, the detection bounding boxes closely surround segmentation masks, as illustrated in Fig. 1b .According to the above analysis, we can find the detection-segmentation imbalance at the beginning layer is one of essential cruxes that hinders the cooperation of object detection and instance segmentation. Therefore, we reviewed the previous works to investigate whether there are works that have been aware of this issue. The idea of many classical and excellent methods [16, 2, 5, 11] is combining two tasks by adding a segmentation branch to an object detector. These detect-then-segment methods make the performance of segmentation task to be limited by the performance of the object detector. Thanks to the thriving of transformer [44] and DETR [3] , recent research attention has been geared towards transformer-based methods, which make giant contributions to the community. For example, [10, 50, 25] use the unified query representation for object detection and instance segmentation tasks based on transformer architecture. However, to our best knowledge, there is no existing work to solve the above mentioned detectionsegmentation imbalance issue. Factually, the imbalance issue naturally exists, which is determined by the individual characteristics of detection and segmentation tasks and also derived from the supervision manners for the two tasks. Firstly, segmentation is a pixel-level grouping and classification task [16, 46] , thus local detailed information is important for this task. In contrast, detection is a region-level task to locate and regress the object bounding box [13, 38] , which requires global information focusing on the complete object. The query at the beginning decoder layer conveys relatively local features, which is more beneficial for the segmentation task, thus the detection task tends to achieve lower performance at the beginning layer. Secondly, supervision manners for detection and segmentation are distinctly different. The segmentation is densely supervised by all pixels of the GT mask, while detection is sparsely supervised by a 4D vector (i.e., x, y, w, and h) of GT bounding box. The dense supervision provides richer and stronger information than the sparse supervision during the optimization procedure. Therefore, the optimization speeds of the two tasks are not synchronous, which will lead to the imbalance issue.Based on two above-analyzed reasons for the detection-segmentation imbalance issue, it is straightforward that the performance of detection task will lag behind at the beginning layer. Considering existing methods share a unified query for detection and segmentation tasks, the performance of a task will be negatively affected by another poorly-performed task, leading to that the multi-task joint training even hurts the performance of the individual task. Therefore, addressing the detection-segmentation imbalance issue is significant for designing a joint object detection and instance segmentation model. To address the detection-segmentation imbalance issue, we propose DI-MaskDINO model, which is implemented by configuring our proposed De-Imbalance (DI) module and Balance-Aware Tokens Optimization (BATO) module to MaskDINO. DI module is responsible for generating balance-aware query. Specifically, DI module strengthens the detection at the beginning decoder layer to balance the performance of the two tasks, and the core of DI module is our proposed residual double-selection mechanism. In this mechanism, the token interaction based double-selection structure learns the global geometric, contextual, and semantic patch-to-patch relations to update initial feature tokens, and high-confidence tokens are selected to benefit the detection task since the selected tokens have learned global semantics during the token interaction procedure. In addition, this mechanism makes use of initial feature tokens by the residual structure, which is the necessary compensation for the information loss occurring during double-selection. Apart from DI module, we also design BATO module, which uses the balance-aware query to guide the optimization of initial feature tokens. The balance-aware query and optimized feature tokens are respectively taken as the Query and Key&Value of transformer decoder to perform joint object detection and instance segmentation.The contributions of this paper are as follows: i) to our best knowledge, this paper for the first time focuses on the detection-segmentation imbalance issue and proposes DI module with the residual double-selection mechanism to alleviate the imbalance; ii) DI-MaskDINO outperforms existing SOTA joint object detection and instance segmentation model MaskDINO (+1.2 AP box and +0.9 AP mask on COCO, using ResNet50 backbone with 12 training epochs), SOTA object detection model DINO (+1.0 AP box on COCO), and SOTA segmentation model Mask2Former(+3.0 AP mask on COCO). 
related work: 
 Object Detection. Classical object detection methods [13, 38, 27, 37, 5, 43, 42] have achieved significant success. In recent years, transformer-based methods such as DETR [3] make a giant contribution to object detection by introducing the concept of object queries and the one-to-one matching mechanism. The success of DETR has sparked a boom in query-based end-to-end detectors, and numerous excellent variants are proposed [56, 33, 52, 28, 23, 6, 47, 51, 24, 32, 21, 54, 18] . Specifically, to enhance the convergence speed of DETR, Deformable DETR [56] proposes deformable attention mechanism that learns sparse feature sampling and aggregates multi-scale features accelerating model convergence and improving performance. From the interpretability of object queries, DAB-DETR [28] formulates the queries as 4D anchor boxes and dynamically updates them in each decoder layer.Instance Segmentation. CNN-based instance segmentation methods are categorized into top-down and bottom-up paradigms. The top-down paradigm [16, 2, 11, 19, 7, 4, 1] firstly generates bounding boxes by object detectors, and then segments the masks. The bottom-up paradigm [35, 9, 29, 12] treats instance segmentation as a labeling-clustering problem, where pixels are firstly labeled as a class or embedded into a feature space and then clustered into each object. Recently, many transformer-based instance segmentation methods [8, 14, 20, 53, 17, 10, 50, 25] are proposed. The transformer-based methods treat the instance segmentation task as a mask classification problem that associates the instance categories with a set of predicted binary masks.Joint Object Detection and Instance Segmentation. The goal of joint object detection and instance segmentation is to carry out the two tasks simultaneously [45, 34, 41, 36] . The traditional joint object detection and instance segmentation methods [16, 2, 5, 11] are usually implemented by adding a mask branch to a strong object detector. For example, the classical model Mask RCNN [16] achieves joint object detection and instance segmentation by adding a mask branch to Faster RCNN [39] . Recently, the proposal of the transformer-based framework has promoted the development of joint object detection and instance segmentation. Following DETR [3] , SOLQ [10] proposes a unified query representation for simultaneous detection and segmentation tasks. The recent MaskDINO [25] achieves optimal performance with the unified query representation on both detection and segmentation tasks. Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: This paper discusses the limitations of the work in § 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed method: 
 In response to the naturally-existing but commonly-ignored imbalance issue between object detection and instance segmentation, we propose DI-MaskDINO model, which is based on MaskDINO [25] . To better understand our proposed DI-MaskDINO, we firstly review MaskDINO ( § 3.1), and then introduce DI-MaskDINO ( § 3.2).MaskDINO is a unified object detection and segmentation framework, which adds a mask prediction branch on the structure of DINO [52] . MaskDINO (grey shaded part in Fig. 2 ) is composed of a backbone, a transformer encoder, a transformer decoder, and detection and segmentation heads (i.e., "Prediction" in Fig. 2 ). Position embeddings and the flattened multi-scale features (extracted by backbone) are inputted to the transformer encoder to generate the initial feature tokens (T i ). Note that in MaskDINO, the top-ranked feature tokens selected from T i directly serve as the Query of transformer decoder, while we design De-Imbalance module with a residual double-selection mechanism to firstly alleviate the detection-segmentation imbalance and then obtain the balance-aware query Q bal to serve as the Query of transformer decoder. In addition, we design Balance-Aware Tokens Optimization module to optimize T i and generate the balance-aware feature tokens T bal to serve as the Key&Value of transformer decoder. Token and query are specialized terms, and their explanations are provided in Appendix A.Fig. 2 illustrates the overview of DI-MaskDINO, which consists of four modules, including Feature Tokens Extractor (FTE), De-Imbalance (DI), Balance-Aware Tokens Optimization (BATO), and Transformer Decoder (TD). FTE extracts the initial feature tokens T i from the input image using backbone and MaskDINO encoder. The goal of DI is to generate the balance-aware query Q bal , which is implemented by applying our proposed residual double-selection mechanism on the initial feature tokens T i . BATO utilizes Q bal to optimize T i , generating the balance-aware feature tokens T bal . TD takes T bal as the Key&Value and Q bal as the Query to perform joint object detection and instance segmentation.Given an image I ∈ R H×W ×3 , the backbone (e.g., ResNet [15] ) firstly extracts multi-scale features, which are then flattened and concatenated to serve as the input of transformer encoder comprising six multi-scale deformable attention layers [56] , obtaining the initial feature tokens T i that is composed of6 i=3 ( H 2 i × W 2 i) tokens, where each token expresses the feature of the corresponding patch in I.There exists the detection-segmentation imbalance at the beginning layer of transformer decoder, which might constrain the upper bound of model performance. To handle this issue, we design DI module to alleviate the imbalance, instead of directly providing T i to the transformer decoder as MaskDINO does. Specifically, detection-segmentation imbalance means that the performance of object detection lags behind that of instance segmentation at the beginning layer of transformer decoder. Therefore, we propose the residual double-selection mechanism to strengthen the performance of object detection.The double-selection consists of the first selection and the second selection. In the first selection, we select top-k 1 ranked feature tokens in T i , based on their category classification scores:EQUATIONwhere T s1 represents the firstly-selected feature tokens, S denotes the selection operator. The first selection mainly filters out most of the tokens conveying background information, making T s1 focus on the objects.Before the second selection, a token interaction network comprising two self-attention layers is applied on T s1 :EQUATIONwhere MHSA is Multi-Head Self-Attention and T sa s1 indicates the feature tokens processed by MHSA. The token interaction is the key point to make sure that the secondly-selected tokens are beneficial for detection, we explain its rationality as follows. As we know, each token actually corresponds to a patch (remarkably smaller than an object in most cases) in the image [55] . The bounding box of an object is regressed by integrating the multiple patches (belonging to the same object) that have global patch-to-patch spatial relations, thus it is really needed for the detection task to learn the interaction relation between patches. In contrast, the dense all-pixel supervision for the segmentation task mainly focuses on local pixel-level similarity with GT mask [25] , hence the segmentation task is not particularly depend on the patch-to-patch relation as the detection task. By self-attention layers, different tokens representing the patches (belonging to the same object) can interact with each other to learn the global geometric, contextual, and semantic patch-to-patch relations, benefiting the perception of object bounding boxes. Therefore, executing token interaction before the second selection makes DI module to be more beneficial for detection. In addition, verified by some studies (e.g., [32] ), the tokens with higher category scores correspond to higher IOU scores of object bounding boxes. Therefore, the second selection further strengthens the object detection and alleviates the detection-segmentation imbalance.In the second selection, we select the top-k 2 ranked feature tokens in T sa s1 to obtain the secondlyselected feature tokens T s2 :EQUATIONThe residual double-selection is inspired by the residual idea in [15] , and the residual is the necessary compensation for double-selection since the information loss occurs in the selection procedures. The formulation of this mechanism is combining T i with T s2 by the Multi-Head Cross-Attention network (MHCA, a self-attention layer and a FFN layer are omitted here), generating Q bal :EQUATIONQ bal conveys the balance-aware information, thus it is named as balance-aware query. Subsequently, Q bal is fed to BATO to guide the optimization of initial feature tokens T i . It is noted that the tokens in Q bal have become significantly different from the tokens in T i . Through Eq. 1-4, the tokens in Q bal have obtained larger receptive field and considered the interaction with other tokens, thus they are better understood as object/instance candidates. Correspondingly, the denotation has been changed from T to Q.In MaskDINO, initial feature tokens T i directly serve as the Key&Value of TD. Instead, we design BATO module that makes use of both balance-aware query Q bal and T i to generate the Key&Value of TD. T i contains a large number (≈ 20k) of tokens conveying detailed local information for both background and objects/instances, while Q bal consists of a small number (=300) of high-confidence tokens mainly focusing on objects/instances. In addition, benefiting from the token interaction (i.e., Eq. 2), Q bal has learned rich semantic and contextual interaction relations. Therefore, Q bal is used to guide the optimization of T i . The optimized feature tokens (denoted as T bal ) is taken as the Key&Value of TD.Firstly, to provide guidance for both detection and segmentation, the mask network and box network are separately applied on Q bal to generate mask guiding token T mask g and box guiding token T box g :EQUATIONEQUATIONwhere N mask and N box indicate the mask network and box network, respectively. Both N mask and N box consist of a mlp network.Then, the overall guiding token T g is obtained by fusing T mask g and T box g :EQUATIONFinally, T g guides the optimization of the initial feature tokens T i through a Multi-Head Cross-Attention. The motivation is straightforward. Same with Q bal , each token in T g corresponds to an object/instance candidate. When T i interacts with T g , the tokens (in T i ) that belong to the same object/instance will be aggregated, enhancing the foreground information. For a better comprehension, a token in T g could be assumed as the center of a "cluster", and the tokens (in T i ) belonging to the same object/instance could be assumed as the points in the "cluster". The points move towards the "cluster" center, realizing the optimization of T i . This procedure is formulated as follows:EQUATIONgenerating balance-aware feature tokens T bal (also called optimized feature tokens), which serve as the Key&Value of TD.TD is responsible for the predictions of instance mask, object box, and class. TD consists of decoder layers and each layer contains a self-attention, a cross-attention, and a FFN. The inputs of TD are T bal (in Eq. 8) and Q bal (in Eq. 4). Q bal interacts with T bal in the decoder layers, continuously refining the query:EQUATIONwhere Q ref is the refined query, and N de denotes the transformer decoder network. Subsequently, we follow the detection head and segmentation head structures of MaskDINO to perform object detection and instance segmentation. For object detection, Q ref is used to predict the categories c and bounding boxes b:EQUATIONwhere N det denotes the detection head network. For instance segmentation, Q ref , T i , and the 1/4 resolution CNN feature F cnn are used to predict the instance masks m:EQUATIONwhere N seg denotes the segmentation head network.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
settings: 
 We conduct extensive experiments on COCO [26] and BDD100K [49] datasets using ResNet50 [15] backbone pretrained on ImageNet-1k [40] as well as SwinL [30] backbone pretrained on ImageNet-22k. NVIDIA RTX3090 GPUs are used when the backbone is ResNet50. Due to the large memory requirement of SwinL, NVIDIA RTX A6000 GPUs are used when the backbone is SwinL. More implementation details are in Appendix B. MaskDINO is the SOTA model for joint object detection and instance segmentation, thus we mainly compare our model with MaskDINO under different backbones (ResNet50 and SwinL). Additionally, our model is compared with some classical (i.e., Mask RCNN [16] ) and recently-proposed (i.e., HTC [5] and SOLQ [10] ) joint object detection and instance segmentation models. Furthermore, our model is compared with SOTA model that is specifically designed for object detection (i.e., DINO [52] ) and instance segmentation (i.e., Mask2Former [8] ). The comparison results on COCO dataset are summarized in Tab. 1. It is noted that the experiments with the Swin-L backbone are conducted on the A6000 GPUs with the batch size of 4 (the maximum bacth size that 4 A6000 GPUs supports). The batch size is smaller than that in MaskDINO paper (i.e., batch size = 16) and the 4 A6000 GPUs present weaker computation power than 4 A100 GPUs, thus the results we reproduced are lower than those in the original MaskDINO paper. We can observe that 1) our model surpasses MaskDINO under different training conditions (epoch = 12, 24, and 50). Notably, our model presents more significant advantage with the training condition of epoch = 12, which potentially reveals that our model reaches the convergence with a faster speed; 2) under the Swin-L backbone, DI-MaskDINO exhibits significant superiority over MaskDINO, further confirming the effectiveness of our model; 3) the performance of our model on individual detection and segmentation tasks is simultaneously higher than that of SOTA models specifically designed for detection (i.e., DINO) and segmentation (i.e., Mask2Former) when they are fully trained (epoch = 36 or 50), which is really hard-won since the single-task model usually designs the specific module for the specific task (e.g., DINO uses the tailored query formulation to improve the detection performance and Mask2Former proposes tailored masked attention module to improve the segmentation performance).Existing joint detection and segmentation models like [5, 10, 50] only conduct the experiments on COCO dataset. In this paper, to further verify the robustness and generalization of our model, additional experiments are conducted on more complex traffic scene dataset BDD100K [49] using ResNet50 and Swin-L backbones, and the results we reproduce are shown in Tab. 2. Due to the complexity of traffic scenes, the overall performance is lower than the performance on COCO dataset, and the model asks for more training epochs (epoch = 68) to reach the convergence. It can be observed that our model still exhibits superiority over MaskDINO, DINO, and Mask2Former, which presents the robustness and generalization of our model. It should be noted that the performance of MaskDINO on detection task is lower than that of the specialized object detection model DINO, indicating that DINO still exhibits the advantage in complex traffic scene datasets. In contrast, our model improves DINO by 0.6 AP box , further demonstrating the effectiveness of our model.There exists the natural imbalance between object detection and instance segmentation, and we are interested in how will a model perform if the imbalance condition is worsened. Therefore, we conduct the imbalance tolerance test by designing two severe imbalance conditions: 1) loss weight constraint, which is implemented by constraining the weight of detection loss to 1/10 of the default value while the weight of segmentation loss remains unchanged; 2) position token constraint, position token conveys important cues of object locations, thus constraining position token will generate disturbing location information to confuse detection task. The position token constraint is implemented by randomly initializing position token of Q bal (composed of position token and content token) in Eq. 4. DI module is mainly responsible for alleviating imbalance issue, thus the imbalance tolerance test on DI-MaskDINO only enables DI module. The experiments are conducted on more complex BDD100K dataset, because the results on the more complex dataset can better reflect the performance of imbalance tolerance. Considering the imbalance issue is severe at the beginning decoder layer, thus the experiments utilize models configured with 3 decoder layers. The results of imbalance tolerance test are summarized in Tab. 3, and the percentage of performance drops (compared with standard condition) is highlighted in colors. We can observe that 1) the imbalance between detection and segmentation has remarkable affect on the upper bound of model performance, potentially indicating the significance of our work; 2) the effects of imbalance conditions on detection task are larger than that on segmentation task, because the two imbalance conditions are implemented to mainly constrain the detection task to simulate the natural detection-segmentation imbalance; 3) even the performance of SOTA model MaskDINO is largely affected by the imbalance conditions (e.g., -21.8% AP box degradation on the condition of position token constraint), which potentially reflects that De-Imbalance deserves the research focus; 4) compared with MaskDINO, the performance degradation of our model is slighter (i.e., -10.2% v.s. -2.9% and -21.8% v.s. -14.7% on the AP box metric, -3.0% v.s. +1.2% and -7.6% v.s. -4.8% on the AP mask metric), which demonstrates the effectiveness of our model; 5) from a comprehensive perspective, we think the standard condition is still a detection-segmentation imbalance condition (which is commonly treated as a regular condition in previous works), and we claim the imbalance is one of the cruxes that limit the upper bound of model performance, hence it should be further studied.To test the effects of main modules in our model (i.e., DI and BATO), we test the performance of our model under four configurations: #1 exclusion of both DI and BATO; #2 exclusion of BATO; #3 exclusion of DI; #4 inclusion of both DI and BATO. To make the results solid, the experiments are conducted on both BDD100K and COCO datasets, and the results are reported in Tab. 4. In comparison with #1, the model under the configuration of #2 or #3 yields higher performance on both datasets, and the optimum results are achieved when both DI and BATO are enabled (#4). These results demonstrate the effectiveness of DI and BATO. The results are explainable. DI module alleviates the imbalance between detection and segmentation, generating balance-aware query Q bal , which is then fed to BATO to further make use of balance-aware information, contributing to performance improvement.The core of our model is DI, which improves the model performance by mitigating the imbalance between detection and segmentation. DI is realized by applying the residual double-selection mechanism on T i , generating T s1 (firstly-selected tokens), T s2 (secondly-selected tokens), and Q bal (balance-aware query). To analyze DI module, we design the fine-grained ablation experiments by respectively using T i , T s1 , T s2 , and Q bal as the guidance for BATO (i.e., gui. = T i , gui. = T s1 , gui. = T s2 , and gui. = Q bal ) and examine the corresponding performance. The experiment results on BDD100K and COCO datasets are reported in Tab. 5. gui. = T i actually represents the situation when DI module is disabled, which serves as the baseline for other situations. Firstly, we can observe P(gui. = T s2 ) > P(gui. = T s1 ) > P(gui. = T i ) where P( * ) denotes the performance of the model under the configuration * , demonstrating our double-selection mechanism is effective. The reason is intuitive, by applying double-selection mechanism, the tokens with high confidence are selected, and highconfidence tokens indicate the location of objects more clearly than other tokens, thus benefiting the object detection task (i.e., mitigating the imbalance between detection and segmentation). Secondly, the highest performance is achieved when gui. = Q bal , validating the effectiveness of our residual double-selection mechanism. In DI module, apart from the secondly-selected tokens T s2 , the initial feature tokens T i is also used to compute Q bal , which could be coarsely formulated as Q bal = T i + S(T i ). This residual structure enables the model to make use of the information in both the initial feature tokens and the selected feature tokens, hence reaching the optimal performance. BATO targets to use the balance-aware query Q bal to guide the optimization of the initial feature tokens T i . The effectiveness of BATO has been validated in Tab. 4. We further conduct experiments to validate the effect of the proposed guiding token generation (GTG). The GTG is designed to provide guidance for both detection and segmentation, generating mask guiding token and box guiding token that are closely related to mask instances and object boxes through the mask network and box network, respectively. These guiding tokens can provide more global and semantic guiding information for the optimization of the initial feature tokens T i . As shown in Tab. 6, the model with GTG performs better, which demonstrates the effect of GTG. 
conclusion: 
 In this paper, we investigate the naturally-existing but commonly-ignore detection-segmentation imbalance issue. The imbalance means that the performance of object detection lags behind that of instance segmentation at the beginning layer of transformer decoder, which is one of cruxes that hurt the cooperation of object detection and instance segmentation tasks and might constrain the breakthrough of the performance upper bound. To address the issue, we propose DI-MaskDINO model with the residual double-selection mechanism to alleviate the imbalance, achieving significant performance improvements compared with SOTA joint object detection and instance segmentation model MaskDINO, SOTA object detection model DINO, and SOTA segmentation model Mask2Former.Limitations. This paper focuses on the task of joint object detection and instance segmentation, thus the model is not applicable for other segmentation tasks such as semantic segmentation and panoptic segmentation.The token numbers of T s1 and T s2 are 600 and 300, respectively. Unless otherwise specified, the feature channels in both encoder and decoder are set to 256, and the hidden dimension of FFN is set to 2048. The mask network and box network in BATO are both three-layer mlp networks. We use the same loss function as MaskDINO (i.e., L1 loss and GIOU loss for box loss, focal loss for classification loss, and cross-entropy loss and dice loss for mask loss). Under ResNet50 pretrained on ImageNet [40] , our model is trained on NVIDIA RTX3090 GPUs. For Swin-L backbone, NVIDIA RTX A6000 GPUs are used for training and validating. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: Our research focuses only on improving the overall performance of joint object detection and instance segmentation models. As such, it does not have direct societal impacts beyond the scope of enhancing these technical capabilities.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper cites the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We will release the source code of our model at the following URL: https: //anonymous.4open.science/r/DI-MaskDINO-12E4.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Token selection is a crucial step in our proposed residual double-selection mechanism. The number of token selection and the amount of selected tokens may affect the performance. We conduct experiments to verify their impacts. The experimental results are summarized in Tab. 7 and Tab. 8.The number of token selection. Single-selection actually represents the situation when disabling DI module. Double-selection corresponds to our proposed method. Additionally, we add a token selection on T s2 for triple-selection. For fair comparison, we set the amount of lastly-selected tokens to the same value (i.e., 300) for single-, double-, and triple-selection. From Tab. 7, we draw two observations: 1) the results of single-selection are significantly lower than those in other situations, indicating the crucial role of DI module; 2) our method achieves the optimal performance with doubleselection. The results are explainable. There exists information loss in each selection procedure. Therefore, triple-selection introduces more information loss, leading to a lower performance than double-selection.The amount of selected tokens. Three k 1 and k 2 settings in the double-selection mechanism are tested, and their maximum performance gap of AP box on BDD100K dataset is 0.4 (i.e., 29.5-29.1). Similar results are exhibited on the metric of AP mask . These results demonstrate that our method is not sensitive to the hyper-parameters k 1 and k 2 . Furthermore, the experiments on COCO dataset also exhibit the similar results, indicating that our method is robust. At last, we explain that the settings of k 1 and k 2 are infinite since they can be set as any value from 1 to 20k. SOTA models such as [18, 25, 24, 14] take 300 as the query number of transformer decoder. In the experiments, k 1 and k 2 are set as 300 or the multiple of 300 to align with the settings of the SOTA models. AP mask BDD100K k 1 =300,k 2 =300 29.1 25.5 k 1 =600,k 2 =600 29.3 25.3 k 1 =600,k 2 =300 29.5 25.7 COCO k 1 =300,k 2 =300 46.6 42.3 k 1 =600,k 2 =600 46.8 42.4 k 1 =600,k 2 =30046.9 42.3We study the effect of different decoder layer numbers for the model performance, and the results are summarized in Tab. 9. We can observe the following: 1) increasing the number of decoder layers from 6 to 9 on BDD100K dataset results in the performance degradation, which can be explained by the inconsistency between the complexity of the model and the dataset. The BDD100K dataset only contains 7k training sets and 1k validation sets. The size of BDD100K is small and the model with 9 decoder layers is relatively more complex, leading to the overfitting of the training set; 2) increasing the number of decoder layers will contribute to both detection and segmentation on COCO. However, the model configured with 9 decoder layers only achieves a slight improvement and introduces more computation cost. Therefore, we use 6 decoder layers in our model; 3) our model has achieved comparable performance in the configuration with 3 decoder layers compared to MaskDINO with 9 decoder layers (e.g., 45.8 v.s. 45.7 on the AP box metric and 41.3 v.s. 41.4 on the AP mask metric on COCO), demonstrating that our model greatly improves the efficiency of the decoder. We visualize the predictions of MaskDINO and DI-MaskDINO to show qualitative comparison on BDD100K dataset. As shown in Fig. 3 , MaskDINO produces boxes that do not tightly encompass the objects (i.e., Fig. 3a ) or do not fully surround the objects (i.e., Fig. 3b ). Compared to MaskDINO, our model produces perfectly-fitting boxes, demonstrating the effectiveness of DI-MaskDINO. In addition, our model focuses attention on the foreground objects with high category scores through the residual double-selection mechanism that avoids mispredicting the background as a foreground object. Fig. 3c suggests that our proposed residual double-selection mechanism is effective. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This work proposes DI-MaskDINO, aiming to improve the detection-segmentation imbalance of the state-of-the-art model MaskDINO. Specifically, the proposed De-Imbalance (DI) module and Balance-Aware Tokens Optimization (BATO) module are integrated into MaskDINO, where DI generates balance-aware query and BATO uses the query to guide the optimization. Promising results have been demonstrated on COCO and BDD100K datasets.

Initially, the reviewers were concerned about Swin-L experiments, more in-depth analysis of the imbalance problem, and so on. During the reviewer-author discussion period, the author rebuttal successfully assuaged the reviewer concerns. In the end, the paper received all accept recommendations (3 borderline accepts and 1 accept). After considering the author rebuttal and reviewer discussion/comments, the area chair agrees with this recommendation.

The authors are encouraged to incorporate the rebuttal and reviewer suggestions to their final camera-ready version.

Reviews: review: paper_summary: paper_summary: The paper starts from an observation regarding imbalance in the intermediate results between detection and instance segmentation, which motivates the authors to propose DI-MaskDINO, which tries to improve the imbalance through the DE-Imbalance module and Balance-Aware Tokens Optimization module. Evaluated on COCO and BDD100K benchmarks, the proposed DI-MaskDINO achieves better results than MaskDINO baseline.
strength_weakness: strength_weakness: - The paper claims improvement over SOTA results, yet it seems that the MaskDINO-SwinL is only reported under 12 epochs setting, while in MaskDINO paper they report results under 50 epochs setting with AP_mask=52.3, AP_box=59.0. I do not see why in tab.1 epochs = 12/24/50 are all reported for R50 backbone but only epochs = 12 are reported for SwinL backbone, making the SOTA claim less comprehensive and less convincing.


- The motivation stems from the observation on det/seg imbalance. Yet, I do not see why performance gap between det and seg from the first layer can illustrate the imbalance. If the absolute AP value is used to measure the imbalance between det and seg, then why AP_box < AP_mask at first layer yet AP_box > AP_mask at the last layer? Furthermore, it is also possible that DI-MaskDINO just improves the performance of both branches which naturally reduce the performance gap (e.g., improving det from 10 to 20 may be in similar difficulty of improving seg from 15 to 22, but the absolute gap is reduced.) In short, I believe there lacks a solid definition and study on the "imbalance" problem, and the current comparison based on absolute AP value from first layer is not convincing to me.


- Minor: I see the reported FPS is significantly different from the ones reported in MaskDINO, which I assume could be the hardware differences. Please explain how the FPS is measured.

 
- Minor: The paper provides an anonymous link to the code/model, yet the provided repo is empty.
questions: questions: Though I hold several concerns as illustrated in the weakness, I feel the paper is generally technically good and with performance improvement over strong baseline MaskDINO. Thus my initial rating is boarderline accept. My major concerns lies in the motivation on imbalance is not so convincing to me, I look forward to the author's rebuttal for further illustration or other more convincing measurement.
limitations: limitations: No other limitations as far as I know
paper_summary: paper_summary: This paper initially observes that in the current state-of-the-art model MaskDINO, the performance of object detection lags behind instance segmentation at the initial layer of the transformer decoder, resulting in a performance imbalance phenomenon.

To explore whether this "performance imbalance issue" is a factor that restricts the effectiveness of the detector, the authors propose the DI-MaskDINO model, which introduces two key components: the De-Imbalance (DI) module and the Balance-Aware Tokens Optimization (BATO) module, to alleviate the performance imbalance between detection and segmentation tasks.
strength_weakness: strength_weakness: - The lack of in-depth analysis of "performance imbalance problem" in this paper, only through the experimental phenomenon, there may be a certain coincidence.
questions: questions: - In Experiment Table 1, why were AP_S, AP_M, and AP_L not tested separately for MaskDINO at epochs 12 and 24?
- In Experiment Table 2, why is there no comparison on the BDD100K validation set for the SwinL backbone?
limitations: limitations: - Although the paper has proposed the "performance imbalance issue," it only conducted experimental observations on MaskDINO. Can other detectors also improve model performance by addressing the "performance imbalance issue"? The paper does not generalize this issue to a more general level, making it more universally applicable.
- The paper's discussion of the "performance imbalance issue" is only focused on the initial layer of the transformer decoder and does not explore whether other layers of the decoder may also have the "performance imbalance issue."
paper_summary: paper_summary: This paper focuses on the detection-segmentation imbalance issue and proposes DI module with the residual double-selection mechanism to alleviate the imbalance; moreover, Balance-Aware Tokens Optimization (BATO) is proposed to guide the optimization of the initial feature tokens. The proposed method termed DI-MaskDINO, achieves SOTA results in both object detection and instance segmentation.
strength_weakness: strength_weakness: The technological innovation is somewhat limited.
questions: questions: The authors implement De-Imbalance module by stacking several self-attention and Multi-Head Cross-Attention. However, it could be seen as one-layer transformer encoder / decoder. The current phenomenon cannot clarify whether the performance improvement stems primarily from mitigating the detection-segmentation imbalance issue or from the increase in parameters/layer. The authors need to provide evidence to support this claim. For example, they could show that a 6-layer DI-MaskDINO outperforms a 7-layer MaskDINO, etc.
limitations: limitations: Yes.
paper_summary: paper_summary: This work investigates the detection-segmentation imbalance issue in MaskDINO. It proposes DI-MaskDINO model with the residual double-selection mechanism to alleviate the imbalance. The framework mainly includes De-Imbalance and Balance-Aware Tokens Optimization. Experiments prove the effectiveness.
strength_weakness: strength_weakness: 1. The motivation of De-imbalance module requires more verification. The authors claim "The token interaction is the key point to make sure that the secondly-selected tokens are beneficial for detection" in L169. But there are no experiments or theory to prove this claim or design. This makes the reviewer confused why the token interaction can play such a role. It's better to add some experiments or heatmap visualization.
2. The Balance-Aware Tokens Optimization module contains several components. It's essential to give experimental analysis of each design.
3. This work is built on MaskDINO, it is important to generalize this proposed manner to other decoder-based methods. It can further validate the generality of the proposed approach.
4. Experimental results on COCO test set should be provided for fair comparisons.
questions: questions: My main concern is the motivation and analysis of the designed component. Please refer to the Weakness section.
limitations: limitations: Limitations are discussed in the work.

Comments: comment: The author's rebuttal has solved most of my concern, and I'd like to change my rating to Borderline accept.
Thanks for the author's rebuttal. I have read the rebuttal and other reviews. It solved most of my concern. So I'd like to change my rating to Borderline accept.
Thanks for the authors responce, my concerns are resolved. I rise the score to Accept.