DI-MaskDINO: A Joint Object Detection and Instance Segmentation Model
Authors: Zhixiong Nan
Xianghong Li
Tao Xiang
Jifeng Dai
Venue: NeurIPS 2024 poster (2024)
Keywords: object detection
instance segmentation
Primary area: machine_vision

Abstract: This paper is motivated by an interesting phenomenon: the performance of object detection lags behind that of instance segmentation (i.e., performance imbalance) when investigating the intermediate results from the beginning transformer decoder layer of MaskDINO (i.e., the SOTA model for joint detection and segmentation). This phenomenon inspires us to think about a question: will the performance imbalance at the beginning layer of transformer decoder constrain the upper bound of the final performance? With this question in mind, we further conduct qualitative and quantitative pre-experiments, which validate the negative impact of detection-segmentation imbalance issue on the model performance. To address this issue, this paper proposes DI-MaskDINO model, the core idea of which is to improve the final performance by alleviating the detection-segmentation imbalance. DI-MaskDINO is implemented by configuring our proposed De-Imbalance (DI) module and Balance-Aware Tokens Optimization (BATO) module to MaskDINO. DI is responsible for generating balance-aware query, and BATO uses the balance-aware query to guide the optimization of the initial feature tokens. The balance-aware query and optimized feature tokens are respectively taken as the Query and Key&Value of transformer decoder to perform joint object detection and instance segmentation. DI-MaskDINO outperforms existing joint object detection and instance segmentation models on COCO and BDD100K benchmarks, achieving +1.2 $AP^{box}$ and +0.9 $AP^{mask}$ improvements compared to SOTA joint detection and segmentation model MaskDINO. In addition, DI-MaskDINO also obtains +1.0 $AP^{box}$ improvement compared to SOTA object detection model DINO and +3.0 $AP^{mask}$ improvement compared to SOTA segmentation model Mask2Former.

DI-MaskDINO: A Joint Object Detection and Instance Segmentation Model
Zhixiong Nan', Xianghong Li!, Tao Xiang "1 and Jifeng Dai
Colleee of Computer Science, Chongqing University, Chongqing, China.
DDpartment of Electronic Engineering, Tsinghua University, Beijing, China.
nanzxêcqu. edu. cn, lixi Lixiinngoongstu.co cqu.equu.. edducn, en, exiing@cquueddu edu. en, daijifengetsinghii edu.cn
Abstract
This paper is motivated by interesting phenomenon: the performance of object detection lags behind that of instance segmentation (i.e., performance imbalance) when investigating the intermediate results from the beginning transformer decoder layer of MaskDINO (i.e.. the SOTA model for joint detection and segmentation). This phenomenon inspires to think about question: will the performance imbalance at the beginning layer transformer decoder constrain the upper bound of the final performance? With this question in mind, we further conduct qualitative and quantitative pre-experiments, which validate the negative impact of detection- segmentation imbalance issue the model performance. To address this issue, this paper proposes DI-MaskDINO model, the core idea of which is to improve the final performance by alleviating the delectioon-segmenntaiion imbalance. DI- MaskDINO is implemented by configuring our proposed De-Imbalance (DI) module and Balance-Aware Tokens Optimization (BATO) module MaskDINO. DI is responsible for generating balance-aware query, and BATO uses the balance- aware query to guide the optimization of the initial feature tokens. The balance- aware query and optimized feature tokens are respectively taken as the Query and Key& Value of transformer decoder to perform joint object detection and instance segmentation. DI-MaskDINO outperforms existing joint object detection and instance segmentation models on COCO and BDD100K benchmarks, achieving +.22 Apbor and +0.9 A pmask improvements compared to SOTA joint detection and segmentation model MaskDINO. In addition, DINO also obtains +1.0 Apbor improvement compared to SOTA object detection model DINO and +3.0 A Pmask improvement compared to SOTA segmentation model Mask2Former.
1 Introduction
Object detection and instance segmentation are two fundamental tasks in the computer vision community. Intuitively, the two tasks are closely-related and mutualll- However, in the current time, specialized detection or segmentation gains more focuses, and the amount of works studying the specialized task is significantly larger than that for joint tasks. One typical explanatory that the multi-task training even hurts the performance the individual task.
Confronting current research situations, we think about whether there are some essential cruxes that are ignored previous works and these cruxes hinder the cooperation of object detection and instance
Corresponding author.
This work is supported by Chongqing Natural Science Foundation Innovation and Development Joint Fund (CTTB202NSCQ--220000))
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Imbalance
Balance
Imbalance
Balance
AP
AP
29.5
28.1
25.3
25.7
21.6 22.1
222 segmentttion
detection
segmentation
sogmentation
detection
segmentation
detection
Lerection
first layer last layer
first layer last layer
(c)
(d)
Figure I: Qualitatively, (a) shows that the detection bounding boxes predicted by the query/feature the first decoder layer of MaskDINO do not fit well with segmentation masks, and (b) exhibits that the corresponding results of DI-MaskDINO are optimized and the detection bounding boxes closely surround segmentation masks. Quantitatively, (c) displays that there exists a significant performance gap between detection and segmentation at the first decoder layer of MaskDINO, and (d) demonstrates DI-MaskDINO not only alleviates the performance imbalance at the first layer but also improves the performance upper bound.
segmentation tasks, which further constrains the breakthrough of the performance upper bound. This paper reveals one of cruxes is the imbalance of object detection and instance segmentation. As shown in Fig. 1, when investigating the results at the first layer of transformer decoder MaskDINO model [25], an interesting phenomenon is found that there exists the performance imbalance between object detection and instance segmentation, as qualitatively illustrated in Fig. la and quantitatively illustrated in the first bar of Fig. Ic. After considering the imbalance issue, the performance gap at the first layer is narrowed as illustrated in the first bar of Fig. 1d, and the final performance upper bounds are improved (i.e., 28.1 to 29.5 for detection and 25.3 to 25.7 for segmentation) as illustrated in the second bar of Fig. 1d. The qualitative results are also optimized, the detection bounding boxes closely surround segmentation masks, illustrated in Fig. lb.
According to the above analysis, we can find the detection seneetioon-semmmetatttoon imbalance at the beginning layer is one of essential cruxes that hinders the cooperation of object detection and instance segmen- tation. Therefore, we reviewed the previous works to investigate whether there are works that have been aware of this issue. The idea of many classical and excellent methods [16, 2, 5, 11] is combining two tasks by adding a segmentation branch to an object detector. These detect-then-segment methods make the performance of segmentation task to be limited by the performance of the object detector. Thanks to the thriving of transformer [44] and DETR [3], recent research attention has been geared towards transformer- baseerr--bddddeeee methods, which make giant contributions to the community. For example, [10, 50, 25] use the unified query representation for object detection and instance segmentation tasks based on transformer architecture.
However, to our best knowledge, there no existing work to solve the above mentioned detection- segmentation imbalance issue. Factually, the imbalance issue naturally exists, which is determined by the individual characteristics of detection and segmentation tasks and also derived from the supervision manners for the two tasks. Firstly, segmentation is pixel-level grouping and classifi- cation task [16, 46), thus local detailed information important for this task. contrast, detection is region-level task to locate and regress the object bounding box [13,38] which requires global information focusing on the complete object. The query at the beginning decoder layer conveys relatively local features, which is more beneficial for the segmentation task, thus the detection task tends to achieve lower performance at the beginning layer. Secondly, supervision manners for detection and segmentation are distinctly different. The segmentation is densely supervised by all pixels of the GT mask, while detection is sparsely supervised by a 4D vector (i.e., x, y, w, and h) of GT bounding box. The dense supervision provides richer and stronger information than the sparse supervision during the optimization procedure. Therefore, the optimization speeds of the two tasks not synchronous, which will lead to the imbalance issue.
Based on two above-analyzed reasons for the estection-segmentaiie imbalance issue, is straightfor- ward that the performance of detection task will lag behind at the beginning layer. Considering existing

---

methods share unified query for detection and segmentation tasks, the performance of a task will be negatively affected by another perfoorer task. leading to that the multi-task joint training even hurts the performance of the individual task. Therefore, addressing the detection-segmentatiin imbalance issue significant for designing a joint object detection and instance segmentation model. To address the detection-segnan segmentation imbalance issue, we propose DI-MaskDINO model, which is implemented by configuring our proposed De-Imbalance (DI) module and Balance-Aware Tokens Optimization (BATO) module to MaskDINO. DI module is responsible for generating balance-aware query. Specifically, DI module strengthens the detection at the beginning decoder layer to balance the performance of the two tasks, and the core of DI module is our proposed residual double-selection mechanism. In this mechanism, the token interaction based double-selection structure learns the global geometric, contextual, and semantic patch-to- srach-tt--ptchhh relations to update initial feature tokens, and high-confidence tokens are selected to benefit the detection task since the selected tokens have learned global semantics during the token interaction procedure. In addition, this mechanism makes use of initial feature tokens by the residual structure, which is the necessary compensation for the information loss occurring during double-selectine module, which uses the balance-aware query to guide the optimization of initial feature tokens. The balance-aware query and optimized feature tokens are respectively taken as the Query and Key& Value of transformer decoder to perform joint object detection and instance segmentation.
Apart from DI module, we also design BATO
The contributions of this paper are follows: i) to our best knowledge, this paper for the first time focuses on the cetection-segmennte double-selection mechanism to alleviate the imbalance; iř) DI-MaskDINO outperforms existing SOTA joint object detection and instance segmentation model MaskDINO (+1.2 A pboz and +0.9 A Ammask COCO, using ResNet50 backbone with 12 training epochs), SOTA object detection model DINO (+1.0 Abooe on COCO), and SOTA segmentation model Mask2Former(+3.0 A Pmmak on COCO).
imbalance issue and proposes DI module with the residual
2 Related Work
Object Detection. Classical object detection methods [13, 38, 27, 37, 5, 43, 42] have achieved significant success. In recent years, transformer-based methods such as DETR [3] make a giant contribution to object detection by introducing the concept of object queries and the one-to-one matching mechanism. The success of DETR has sparked boom in query-based end-to end-to-end detectors, and numerous excellent variants are proposed [56, 33, 52, 28, 23, 6, 47, 51, 24, 32, 21, 54, 18]. Specifically, to enhance the convergence speed of DETR, Deformable DETR [56] proposes de- formable attention mechanism that learns sparse feature sampling and aggregates multi-scale features accelerating model convergence and improving performance. From the interpretability of object queries, DAB-DETR [28] formulates the queries as 4D anchor boxes and dynamically updates them in each decoder layer.
Instance Segmentation. CNN-based instance segmentation methods are categorized into top-down and bottom-up paradigms. The top-down paradigm [16, 2, 11, 19, 7, 4, 1] firstly generates bounding boxes by object detectors, and then segments the masks. The bottom-up paradigm [35, 9, 29, 12] treats instance segmentation as a cllutering problem, where pixels are firstly labeled as a class or embedded into feature space and then clustered into each object. Recently, many transformer- based instance segmentation methods [8, 14, 20, 53, 17, 10, 50, 25] are proposed. The transformer-bar based methods treat the instance segmentation task as a mask classification problem that associates the instance categories with a set of predicted binary masks.
Joint Object Detection and Instance Segmentation. The goal of joint object detection and instance segmentation is to carry out the two tasks simultancously [45, 34, 41, 36). The traditional joint object detection and instance segmentation methods [16, 2, 5, 11] are usually implemented by adding mask branch to a strong object detector. For example, the classical model Mask RCNN [16] achieves joint object detection and instance segmentation by adding a mask branch to Faster RCNN [39]. Recently, the proposal of the transformer-base detection and instance segmentation. Following DETR [3], SOLQ [10] proposes unified query rep- resentation for simultaneous detection and segmentation tasks. The recent MaskDINO [25] achieves optimal performance with the unified query representation on both detection and segmentation tasks.
framework has promoted the development of joint object
3

---

initial feature token
Transformer Decoder
Balance-Aware Tokens
selected feature token
Optimization
balance-award query
Prediction
Transformer
KeyA Faller
T
Decoder Layers
guiding token
ballncc--waree feature token
Query
Multi-Hese Heat Cross-Atication
multi-scale features
Feature Tokens
De-Imbulance
Qost
Extractor
Multi-Head Attesttttonionn
residnal druble deuble-sclettinn seleccion
1/100/32 Lis
mask guiding token
box guiding token
MaskDINO Encoder
token solection
Mask Network
Token Interaction
Netwock
flatten
guidance
position embeddings
solection
content token
position token
The overview DI-MaskDINO model based on MaskDINO (grey shaded), with the extensions (green shaded) of De-Imbalance and Balance-Aware Tokens Optimization. For simplicity. content token and position token are merged in De-Imbalance (i.e.. T;, T,1. Ts2, and Q contain both content and position token) in presentation. GTG is short for guiding token generation.
Figure 2:
3 Proposed Method
In response to the naturally-existing but commonly-ignored imbalance issue between object detection and instance segmentation, we propose DI-MaskDINO model, which based on MaskDINO [25]. To better understand our proposed DI-MaskDINO, we firstly review MaskDINO ($ 3.1), and then introduce DI-MaskDINO ($ 3.2).
3.1 Preliminaries: MaskDINO
MaskDINO is unified object detection and segmentation framework, which adds mask prediction branch on the structure of DINO [52]. MaskDINO (grey shaded part Fig. 2) is composed of backbone, transformer encoder, transformer decoder, and detection and segmentation heads (i.e.. "Prediction" in Fig. 2). Position embeddings and the flattened multi-scanl features (extracted by backbone) inputted to the transformer encoder generate the initial feature tokens (T.). Note that MaskDINO, the top-ranked feature tokens selected from T, directly serve as the Query of transformer decoder, while design De-Imbalance module with residual double-selection mechanism to firstly alleviate the cetection-seemmentation imbalance and then obtain the balance-aware query Qbal serve the Query transformer decoder. In addition, design Balance-Aware Tokens Optimization module to optimize T; and generate the balance-aware feature tokens That to serve the Key& Value of transformer decoder. Token and query specialized terms, and their explanations are provided in Appendix A.
3.2 Our Method: DI-MaskDINO
Fig. 2 illustrates the overview of DI-MaskDINO, which consists of four modules, including Feature Tokens Extractor (FTE), De-Imbalance (DI), Balance-Aware Tokens Optimization (BATO), and Transformer Decoder (TD). FTE extracts the initial feature tokens from the input image using backbone and MaskDINO encoder. The goal of DI is to generate the balance- query Q bal- which is implemented by applying our proposed residual double-selection mechanism on the initial feature tokens T, BATO utilizes Qin! to optimize T,, generating the balance-aware feature tokens That. TD takes That as the Key& Value and Quat as the Query to perform joint object detection and instance segmentation.

---

3.2.1 Feature Tokens Extractor
Given an image I E RH x3 the backbone (e.g.. ResNet [15]) firstly extracts multi-scale features, which are then flattened and concatenated to serve as the input of transformer encoder comprising six multi-scale deformable attention layers [56], obtaining the initial feature tokens T; that is composed of E 3
W tokens, where each token expresses the feature of the corresponding patch in I.
2
3.2.2 De-mmbalance
There exists the detection detection--egmenntee -eggentationn imbalance at the beginning layer of transformer decoder. which might constrain the upper bound of model performance. To handle this issue, we design DI module alleviate the imbalance, instead of directly providing T; to the transformer decoder as MaskDINO does. Specifically, ject detection lags behind that of instance segmentation at the beginning layer of transformer decoder. Therefore, we propose the residual double-selection mechanism to strengthen the performance of object detection.
imbalance means that the performance of ob-
The double-selection consists of the first selection and the second selection. In the first selection, we select top-k1 ranked feature tokens in T, based on their category classification scores:
T31 S(T,, k1),
(I)
where T. represents the firstly-selected feature tokens, S denotes the selection operator. The first selection mainly filters out most of the tokens conveying background information, making T. focus on the objects.
Before the second selection, a token interaction network comprising two self-attention layers is applied on Te
T MHSA(T.1),
(2)
where MHSA is Multi-Head Self-Attentio and T. indicates the feature tokens processed by MHSA.
The token interaction is the key point to make sure that the secondly-selccter tokens are beneficial for detection, we explain its rationality as follows. As we know, each token actually corresponds to a patch (remarkably smaller than an object in most cases) in the image [55]. The bounding box of an object is regressed by integrating the multiple patches (belonging to the same object) that have global patch-to- sitch-to-patch spatial relations, thus it is really needed for the detection task to learn the interaction relation between patches. In contrast, the dense all all-pixel supervision for the segmentation task mainly focuses on local pixel-level similarity with GT mask [25], hence the segmentation task not particularly depend on the patch-to-patch relation as the detection task. By self-attention layers, different tokens representing the patches (belonging to the same object) can interact with each other to learn the global geometric, contextual, and semantic patch-to-patch relations, benefiting the perception of object bounding boxes. Therefore, executing token interaction before the second selection makes DI module to be more beneficial for detection. In addition, verified by some studies (e.g.. [32]), the tokens with higher category scores correspond to higher IOU scores of object bounding boxes. Therefore, the second selection further strengthens the object detection and alleviates the detection-segmentatioo imbalance.
In the second selection, we select the top-k2 ranked feature tokens in T to obtain the secondly- selected feature tokens Ts2:
Ts = S k2).
The residual double-selection is inspired by the residual idea in [15], and the residual is the necessary compensation for double-selection since the information loss occurs in the selection procedures. The formulation this mechanism is combining T; with T.2 by the Multi-Head Attes-AAtention network (MHCA, self-attention layer and a FFN layer are omitted here), generating Qial:
Q bal MHCA(T.2. T.).
Qut conveys the balance- aware information, thus it is named as balance-aware query. Subsequently, Qbt is fed to BATO to guide the optimization of initial feature tokens T. It is noted that the tokens in Quat have become significantly different from the tokens in T, Through Eq. 1-4, the tokens in Qun! have obtained larger receptive field and considered the interaction with other tokens, thus they are better understood as object/instance candidates. Correspondingly, the denotation has been changed from T to Q.
5

---

3.2.3 Balance-Aware Tokens Optimization
In MaskDINO, initial feature tokens T; directly serve as the Key& Value of TD. Instead, we design BATO module that makes use of both balance-aware query Qaal and T; to generate the Key Value of TD. T. contains a large number (= 20k) of tokens conveying detailed local information for both background and objects/instances, while Qbal consists of a small number (=300) of high-confidence tokens mainly focusing on objects/instances. In addition, benefiting from the token interaction (i.e., Eq. 2), baai has learned rich semantic and contextual interaction relations. Therefore, Qual used to guide the optimization of T. The optimized feature tokens (denoted as Tha!) is taken as the Key& Value of TD.
Firstly, to provide guidance for both detection and segmentation, the mask network and box network are separately applied on bat to generate mask guiding token Tmask and box guiding token Thoo,
Tyaak N (mask(Qbai),
Thoo Nobar (Qbal);
(6)
where Nmask and Noor indicate the mask network and box network, respectively. Both N Wassk and Nbox consist of mlp network.
Then, the overall guiding token T2 is obtained by fusing Timae and Thoo
Tg = Tmaak + TT
=
Finally, T, guides the optimization of the initial feature tokens T. through a Multi-Head Cross- Attention. The motivation is straightforward. Same with Qbal. each token in T2 corresponds an object/instance candidate. When T; interacts with Tg. the tokens (in Tς) that belong to the same object/instance will be aggregated, enhancing the foreground information. For better comprehension, token in T3 could be assumed as the center of "cluster", and the tokens (in T,) belonging to the same object/instance could be assumed as the points the "cluster". The points move towards "cluster" center, realizing the optimization of T. This procedure is formulated as follows:
T Bot = (HCA(T,, T,),
(8)
generating balance-an aware feature tokens Tout (also called optimized feature tokens), which serve as the Key& Value of TD.
3.2.4 Transformer Decoder
TD is responsible for the predictions of instance mask, object box, and class. TD consists of decoder layers and each layer contains self-attention, a cross-attention. and FFN. The inputs of TD are That (in Eq. 8) and Quat (in Eq. 4). Qbal interacts with That in the decoder layers, continuously refining the query:
QreA= = Nie (Qual: Toat);
where Qref is the refined query, and No denotes the transformer decoder network.
Subsequently, we follow the detection head and segmentation head structures of MaskDINO to perform object detection and instance segmentation. For object detection, Qref is used to predict the categories e and bounding boxes b:
{c, = -NeesQure))
(10)
where Note denotes the detection head network. For instance segmentation, Qrefi T, and the 1/4 resolution CNN feature Fenn. are used to predict the instance masks m.
m Noep Qrefi Tú, Fonn);
(11)
where Neeg denotes the segmentation head network.
4
Experiments
4.1
Settings
We conduct extensive experiments on COCO [26] and BDD100K [49] datasets using ResNet50 [15] backbone pretrained on ImageNet- 1k [40] as well as Swinl. [30] backbone pretrained on ImageNet- 22k. NVIDIA RTX3090 GPUs used when the backbone is ResNet50. Due to the large memory

---

requirement of SwinL, NVIDIA RTX A6000 GPUs are used when the backbone is SwinL. More implementation details are Appendix B.
Table Comparison with other methods on the COCO validation set.
Methods
Epochs
Apaox
AP
APAr
A
A Apmask
AP A
APp
FPS
ResNet50 backbone
Mask RCNN [16]
36
41.0
24.9
43,9
53.3
37.2
18.6
39.5
53.3
21.0
HTC [S]
36
44.9
39.7
22.6
42.2
50.6
5.5
SOLQ [10]
50
48.5
30.1
51.6
64.8
40.1
20.9
43.7
59.4
7.0
DINO [52]
36
50.9
34.6
54.1
64.6
6.8
Mask2Former [8]
50
43.7
23.4
47.2
64.8
5.3
MaskDINO [25]
12
45.7
41.4
21.1
44.2
61.4
8.0
DI-MaskDINO (Our))
12
46.9 (+1.2)
28.8
49.5
62.9
42.3 (+0.9)
22
44.8
62.8
7.7
MaskDINO [25]
24
48.4
44.2
23.9
47.0
64.0
8.0
MaskDINO (Ours)
49.6 (+1.2)
31.7
52.6
65.3
44.8 (+0.6)
24.3
47.8
65.0
7.7
MaskDINO [25]
51.7
34.2
54.7
46.3
26.1
49.3
66.1
DI-MaskDINO (Ours)
51.9 (+0.2)
36.3
54.7
46.7 (+0.4)
27.5
66.2
SwinL backbone
MaskDINO [25]
12
52.2
34.8
55.6
69.9
47.2
26.3
50.3
69.1
3,4
DI-MacDDINO (Ours)
53.3 (+1.1)
36.7
56.7
70.4
47.9 (+0.7)
27.7
51.1
69.3
MaskDINO [25]
56.8
40.2
60.2
72.3
51.0
31.3
71.2
3.4
(Ours)
(+1.0)
61.2
(+0.8)
31.8
72.2
Table 2: Comparison with other methods on the BDD100K validation set.
Methods
Epochs
Aplox
A
APhas
APh
Apmask
APg
St APpast
FPS
ResNet50 backbone
Mask RCNN [16]
25.5
15.7
32.8
56.1
20.7
15.7
26.6
49.1
20.1
HTC [5]
26.9
15.7
35.4
55.3
21.1
11.0
26.7
46.4
4.8
SOLQ [10]
16.5
35.3
19.6
10.1
37.4
DINO [52]
28.9
18.0
37.0
48.1
6.1
Mask2Fonmer [8]
19.6
8.4
25.9
41.0
4.7
MaskDINO [25]
28.1
17.4
36.1
47.9
25.3
14.2
31.8
48.1
6.7
DI-MaskDINO (Ours)
29.5 (+1.4)
18.0
37.4
50.4
25.7 (+0.4)
14.5
32.1
48.1
Swinl backbone
MaskDINO
68
30.2
19.0
37.5
48.6
27.0
15.4
32.6
50.5
3.2
Di-MaskDINO (Ours)
68
31.4 (+1.2)
19.4
40.4
48.7
27.9 (+0.9)
16.6
34.1
51.2
2.8
4.2
Comparison Experiments
MaskDINO the SOTA model for joint object detection and instance segmentation, thus we mainly compare our model with MaskDINO under different backbones (ResNet50 and SwinL). Additionally, our model is compared with some classical (i.c., Mask RCNN [16]) and recently-proposed (i.c., HTC [5] and SOLQ [10]) joint object detection and instance segmentation models. Furthermore, our model is compared with SOTA model that is specifically designed for object detection (i.e., DINO [52]) and instance segmentation (i.e., Mask2Former [8]). The comparison results COCO dataset are summarized in Tab. 1. is noted that the experiments with the Swin-L backbone are conducted the A6000 GPUs with the batch size of 4 (the maximum bacth size that A6000 GPUs supports). The batch size is smaller than that in MaskDINO paper (i.e., batch size = 16) and the 4 A6000 GPUs present weaker computation power than 4 A GPUs, thus the results we reproduced are lower than those in the original MaskDINO paper. We can observe that 1) our model surpasses MaskDINO under different training conditions (epoch 12, 24, and 50). Notably, our model presents more significant ad advantage with the training condition of epoch = 12. which potentially reveals that our model reaches the convergence with a faster speed; 2) under the Swin-L backbone, DI-MaskDINOO exhibits significant superiority over MaskDINO, further confirming the effectiveness of our model; the performance of our model on individual detection and segmentation tasks is simultaneously higher than that of SOTA models specifically designed for detection (i.e., DINO) and segmentation (i.e., Mask2Former) when they are fully trained (epoch 36 or 50), which is really hard-won since the single-task model usually designs the specific module for the specific task ge. DINO uses the tailored query formulation to improve the detection performance and Fosk2Former proposes tailored masked attention module improve the segmentation performance).
7

---

Existing joint detection and segmentation models like [5, 10, 50] only conduct the experiments on COCO dataset. In this paper, to further verify the robustness and generalization of our model, additional experiments are conducted on more complex traffic scene dataset BDD100K [49] using ResNer50 and Swin-I. backbones, and the results we reproduce are shown in Tab. Due to the complexity of traffic scenes, the overall performance is lower than the performance on COCO dataset, and the model asks for more training epochs (epoch = 68) to reach the convergence. can be observed that our model still exhibits superiority over MaskDINO, DINO, and Mask2Former, which presents the robustness and generalization of our model. It should be noted that the performance of MaskDINO detection task is lower than that of the specialized object detection model DINO, indicating that DINO still exhibits the advantage in complex traffic scene datasets. In contrast, our model improves DINO by 0.6 Aphor, further demonstrating the effectiveness of our model.
4.3 Diagnostic Experiments
4.3.1 Imbalance Tolerance Test
There exists the natural imbalance between object detection and instance segmentation, and we are interested in how will a model perform if the imbalance condition worsened. Therefore, we conduct the imbalance tolerance test by designing two severe imbalance conditions: 1) loss weight constraint, which is implemented by constraining the weight of detection loss to 1/10 of the default value while the weight of segmentation loss remains unchanged; 2) position token constraint, position token conveys important cues of object locations, thus constraining position token will generate disturbing location information to confuse detection task. The position token constraint is implemented by randomly initializing position token of Qbal (composed of position token and content token) in Eq. 4. DI module is mainly responsible for alleviating imbalance issue, thus the imbalance tolerance test on DI-MaskDINO only enables DI module. The experiments are conducted on more complex BDD100K dataset, because the results on the more complex dataset can better reflect the performance of imbalance tolerance. Considering the imbalance issue is severe at the beginning decoder layer, thus the experiments utilize models configured with 3 decoder layers.
Table 3: Imbalance tolerance test comparison of MaskDINO and DI-MaskDINO.
MaskDINO
DI-MaskDINO (Ours)
Imbalance conditions
A
Appass
Aploz
Apmash
standard
27.5
23.7
27.9
24.9
loss weight constraint
(-10.2%)
(-3.0%)
(+122%)
position token constraint
21.5 (-218%)
(-7.6%)
23.8 (-14.7%)
23.7 (-4.8%)
The results of imbalance tolerance test are summarized in Tab. 3. and the percentage of performance drops (compared with standard condition) is highlighted in colors. We can observe that 1) the imbalance between detection and segmentation has remarkable affect on the upper bound of model performance, potentially indicating the significance of our work; 2) the effects of imbalance conditions on detection task are larger than that on segmentation task, because the two imbalance conditions are implemented to mainly constrain the detection task to simulate the natural detection -egrriitt-semmttttttiionn imbalance; 3) even the performance of SOTA model MaskDINO is largely affected by the imbalance conditions (e.g.. -21.8% % Abbes degradation on the condition of position token constraint), which potentially reflects that De-Imbalance deserves the research focus; 4) compared with MaskDINO, the performance degradation of our model is slighter (i.c., -1.22% -2.9% % and -21.8% v.s. -14.7% % on the Abboz metric, -3.0% v.s. +1.2% % and -7.6% v.s. -4.8% on the Apmask metric), which demonstrates the effectiveness of our model; 5) from comprehensive perspective, we think the standard condition is still a sestettion-eemmetaationn imbalance condition (which is commonly treated regular condition in previous works), and claim the imbalance is one of the cruxes that limit the upper bound of model performance, hence it should be further studied.
4.3.2 Diagnostic Experiments on Main Modules
To test the effects of main modules in our model (i.e., DI and BATO), we test the performance of our model under four configurations: #1 exclusion of both DI and BATO; #2 exclusion of BATO; #3 exclusion of DI: #4 inclusion of both DI and BATO. To make the results solid, the experiments are conducted on both BDD 100K and COCO datasets, and the results are reported in Tab. 4.
8

---

Table The results of diagnostic experiments on main modules. The experiments are conducted on the BDD100K dataset with 68 training epochs and on COCO dataset with 12 training epochs. The results in Tab. 5 and Tab. 6 are also obtained under the same experiment settings.
BDD100K
COCO
ID
DI
BATO
Aphar
Apmosk
Aplor
Apmoxt
#1
27.8
24.4
45.6
41.2
#2
/
28.8
25.2
46.4
42.1
43
/
28.3
24.9
46.2
41.8
#4
/
/
29.5
25.7
46.9
42.3
In comparison with #1, the model under the configuration of #2 or #3 yields higher performance on both datasets, and the optimum results are achieved when both DI and BATO are enabled (#4). These results demonstrate the effectiveness of DI and BATO. The results module alleviates the imbalance between detection and segmentation, generating balance-aware query Qbat. which is then fed to BATO to further make use of balance-award aware information, contributing to performance improvement.
explainable. DI
4.3.3 Diagnostic Experiments on DI Module
The core of our model is DI, which improves the model performance by mitigating the imbalance between detection and segmentation. DI is realized by applying the residual double-selection mechanism on Tū generating T. (firstl--slectted tokens), T,2 (secondly-sclected tokens), and Qual (balance-aware query). To analyze DI module, we design the fine fine-grained ablation experiments by respectively using T. T,1, T22, and Qbal as the guidance for BATO (i.e., gui. = Ti, gui. = T.1. gui. T,2, and gui. Qbat) and examine the corresponding performance.
The experiment results on BDD100K and COCO datasets are reported in Tab. 5. gui. = Ti actually represents the situation when DI module is disabled, which serves as the baseline for other situations. Firstly, we can observe Pggui. = T.2) > P(gui. = T.1)= > P(gui. T:) where P=*) denotes the performance of the model under the configuration *, demonstrating our double-selection mechanism is effective. The reason is intuitive, by applying double-selection mechanism, the tokens with high confidence are selected, and high- confidence tokens indicate the location of objects more clearly than other tokens, thus benefiting the object detection task (i.e., mitigating the imbalance be- tween detection and segmentation). Secondly, the highest performance is achieved when gui. = =Q Qhal- validating the effectiveness of our residual double-selection mechanism. In DI module, apart from the recondly-selecter tokens T22, the initial feature tokens T; is also used to compute Quat. which could be coarsely formulated as Qbal- T; + ST,)). This residual structure enables the model to make use of the information in both the initial feature tokens and the selected feature tokens, hence reaching the optimal performance.
Table 5:
The results of diagnostic exper- iments on DI module, gui. denotes the guidance in Fig. 2.
BDD100K
COCO
Guidance
Appox
A Appeak
Aplos
A Apmosh
gui. T:
28.3
24.9
46.2
41.8
gui. Ta
28.5
24.9
46.6
42.0
gui. T,2
28.9
25.6
46.7
42.2
qui. Quai
29.5
25.7
46.9
42.3
4.3.4 Diagnostic Experiments BATO Module
BATO targets to the balance-aware query Qual to guide the optimization of the initial feature tokens T,.
Table 6: The results of diagnostic experi- ments on BATO module.
The effectiveness of BATO has been validated in Tab. 4. We further conduct experiments to validate the effect of the proposed guiding token generation (GTG). The GTG is designed to provide guidance for both detection and segmentation, generating mask guiding token and box guiding token that are closely related to mask in- stances and object boxes through the mask network and box network, respectively. These guiding tokens can provide more global and semantic guiding information for the optimization of the initial feature tokens T,. As shown in Tab. 6, the model with GTG performs better, which demonstrates the effect of GTG.
BDD10OK
COCO
Configurations
Apior
Apmash
Apbox A p.maa
w/o GTT
28.6
25.4
46.5
42.2
w/ GTG
29.5
25.7
46.9
42,3

---

5 Conclusion
In this paper, we investigate the naturally-existine but commonly-ignan dntectioo-seemmenttie imbalance issue. The imbalance means that the performance of object detection lags behind that of instance segmentation at the beginning layer of transformer decoder, which is one cruxes that hurt the cooperation of object detection and instance segmentation tasks and might constrain the break- through of the performance upper bound. To address the issue, we propose DI-MaskDINO model with the residual double-selection mechanism to alleviate the imbalance, achieving significant perfor- mance improvements compared with SOTA joint object detection and instance segmentation model MaskDINO, SOTA object detection model DINO, and SOTA segmentation model Mask2Former.
This paper focuses on the task of joint object detection and instance segmentation, thus the model is not applicable for other segmentation tasks such as semantic segmentation and panoptic segmentation.
Limitations.
References
[1] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 9157-9166, 2019.
[2] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1483- 1498, 2019.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference Computer Vision, pages 213-229, 2020.
[4] Hao Chen, Kunyang Sun, Zhi Tian, Chunhua Shen, Yongming Huang, and Youliang Yan. Blendmask: Top-down meets bottom-up for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8573-8581, 2020.
[5] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, al. Hybrid task cascade for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974-4983, 2019.
[6] Qiang Chen, Xiaokang Chen, Gang Zeng, and Jingdong Wang. Group detr: Fast training convergence with decoupled one-to-many label assignment. arXiv preprint arXiv: 13085, 2022.
[7] Xinlei Chen, Ross Girshick, Kaiming He, and Piotr Dollár. Tensormask: A foundation for dense object segmentation. Proceedings of the JEEF/CVF International Conference on Computer Vision, pages 2061-2069, 2019.
[8] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290-1299 2022.
[9] Bert De Brabandere, Davy Neven, and Lue Van Gool. Semantic instance segmentation with a discriminative loss function. arXiv preprint 17Xiv:1708.02551, 2017.
[10] Bin Dong, Fangao Zeng, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Solq: Segmenting objects by learning queries. In Proceedings of the Neural Information Processing Systems, volume 34, pages 21898 299099, 2021.
[11] Yuxin Fang, Shusheng Yang, Xinggang Wang, Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances as queries. Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 69 6910-6919, 2021.
10

---

[12] Naiyu Gao, Yanhu Shan, Yupei Wang, Xin Zhao, Yinan Yu, Ming Yang. and Kaiqi Huang. Ssap: Single-shot instance segmentation with affinity pyramid. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 642-651, 2019.
[13] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 1440-1448, 2015.
[14] Junjie Hc, Pengyu Li, Yifeng Geng, and Xuansong Xie. Fastinst: A simple query- model for real-time instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-778, 2016.
[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961 1261-2999,211 2017.
[17] Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji. Istr: End-to-end instance segmentation with transformers. arXiv preprint arXiv:2105.00637, 2021.
[18] Zhengdong Hu, Yifan Sun, Jingdong Wang. and Yi Yang. Dac-detr: Divide the attention layers and conquer. Proceedings of the Neural Information Processing Systems, 36, 2024.
[19] Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6409-6418, 18, 2019.
[20] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani. Nikita Orlov, and Humphrey Shi. One- former: One transformer to rule universal image segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2989-2998, 2023.
[21] Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19702-19712, 2023.
[22] Lei Ke, Martin Danelljan, Xia Li, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Mask transfiner for high-quality instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4412-4421, 2022.
[23] Feng Li. Hao Zhang. Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang, Du-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13619-13627, 2022.
[24] Feng Li, Ailing Zeng, Shilong Liu, Hao Zhang, Hongyang Li, Lei Zhang, and Lionel M Ni Lite detr: An interleaved multi-scale encoder for efficient detr. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 18558-18566 2023.
[25] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung- Yeung Shum. Mask dino: Towards a unified transformer-base framework for object detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3041-3050 2023.
[26] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, pages 740-755, 2014.
[27] Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2988, 2017.
[28] Shilong Liu, Feng Li, Hao Zhang. Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. In Proceedings of the International Conference Learning Representations, 2022.

---

[29] Shu Liu, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. Sgn: Sequential grouping networks for instance segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 3496-3504, 2017.
[30] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[31] Nya Loshehilov and Frank Hutter, Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations, 2019.
[32] Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingging Dang, and Yi Liu. Detrs beat yolos on real-time object detection. arXiv preprint arXiv:2334.08066 2023.
[33] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3651-3660, 2021.
[34] Zhixiong Nan, Jizhi Peng, Jingjing Jiang, Hui Chen, Ben Yang, Jingmin Xin, and Nanning Zheng. A joint object detection and semantic segmentation model with cross-attention and inner-attention mechanisms. Neurocomputing, 463:212-223 225, 2021.
[35] Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. In Proceedings of the Neural Information Processing Systems, volume 30, 2017.
[36] Jizhi Peng. Zhixiong Nan, Linhai Xu, Jingmin Xin, and Nanning Zheng. A deep model for joint object detection and semantic segmentation in traffic scenes. In Proceedings of the International Joint Conference on Neural Networks, pages 1-8, 2020.
[37] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint 1rXiv:1804.02760 2018.
[38] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 779-788, 2016.
[39] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of the Neural Information Processing Systems, volume 28, 2015.
[40] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2015.
[41] Ganesh Sistu, Isabelle Leang, and Senthil Yogamani. Real-time joint object detection and semantic segmentation network for automated driving. arXiv preprint w:Xiv:1901.09911 2019.
[42] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 14454-14463, 2021.
[43] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 9627-9636, 2019.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need, In Proceedings of the Neural Information Processing Systems, volume 30, 2017.
12

---

[45] Shaoru Wang, Yongchao Gong, Junliang Xing, Lichao Huang, Chang Huang, and Weiming Hu. Rdsnet: new deep architecture forreciprocal object detection and instance segmentation. In Proceedings of the Association for the Advancement of Artificial Intelligence, volume 34, pages 12208-12215, 2020.
[46] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In Proceedings of the IEEE/CVF European Conference Computer Vision, pages 649-665, 2020.
[47] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. In Proceedings the Association for the Advancement of Artificial Intelligence, volume 36, pages 2567-2575, 2022.
[48] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. https: / com/ facebookresearchctcooooooc www.ch//eeeeetteee.ec. 2019.
[49] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2636-2645, 2020.
[50] Xiaodong Yu, Dahu Shi, Xing Wei, Ye Ren, Tingqun Ye, and Wenming Tan. Soit: Segmenting objects with instance-aware transformers. Proceedings of the Association for the Advancement of Artificial Intelligence, volume 36, pages 1188-3196, 2022.
[51] Gongjie Zhang, Zhipeng Luo, Yingchen Yu, Kaiwen Cui, and Shijian Lu. Accelerating detr convergence via semantic-aligned matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 949-958, 2022.
[52] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In Proceedings of the International Conference on Learning Representations, 2023.
[53] Hao Zhang. Feng Li, Huaizhe Xu, Shijia Huang. Shilong Liu, Lionel M Ni, and Lei Zhang Mp-former: Mask-piloted transformer for image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18074-18083, 2023.
[54] Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, and Yunhe Wang. Less is more: Focus attention for efficient detr. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6674-6683. 2023.
[55] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequence-to-seqeecce perspective with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6881 16890, 2021.
[56] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to tn- object detection. arXiv preprint arX1v:0010.04159, 2020.
13

---

Appendix
Due to the space limitation of the main text, we provide more results and discussions in the appendix, which are organized follows:
Section A: Prior Knowledge.
Section B: Experiment Settings.
Section B.1: Datasets and Metrics.
-Section B.2: Implementation Details.
Section Additional Diagnostic Experiments.
Section C.I: Diagnostic Experiments on Token Selection.
Section C.2: Diagnostic Experiments on the Number of Decoder Layer.
Section D: Visualization Analysis.
A
Prior Knowledge
Encoder and decoder of DETR-like models. MaskDINO is a type of DETR-like models [24, 54, 23, 47, 56, 51, 18]. A DETR-like model usually contains backbone, an encoder, a decoder, and multiple prediction heads. The encoder is composed of multiple transformer encoder layers, and each encoder layer contains multi-heaa self-attention and FFN. The decoder consists of multiple transformer decoder layers, and each decoder layer has an extra multi-heat cross-attention compared to the encoder layer.
Token and query. Both token and query are used to represent features. The concept of query comes from the original DETR [3]. Commonly, it denotes the feature of the object/instance in the decoder of DETR- like models. The token is a concept in the field of NLP (Natural Language Processing). In the computer vision domain, token corresponds to patch in image. Feature tokens in this paper represent the features of image patches.
How to obtain the intermediate results from the beginning transformer decoder layer. The transformer decoder in MaskDINO is composed of multiple decoder layers, and MaskDINO attaches prediction heads after each decoder layer. Therefore, decoder layer, which are called intermediate results. The intermediate results from the beginning transformer decoder layer obtained by applying prediction heads the O-th decoder layer.
obtain prediction results from any
B Experiment Settings
B.1 Datasets and Metrics
COCO [26] is the most widely used dataset for the object detection and instance segmentation tasks, and many well-known models such [16, 3, 52, 8, 25] evaluated on the COCO dataset. Following the common practice, we use the COCO train2017 split (118k images) for training and the val2017 split (5k images) for validation. In addition, considering autonomous driving is typical and practical application of object detection and instance segmentation, the experiments are also conducted on BDD100K [49] dataset, which is composed of 10k high-quality instance masks and bounding boxes annotations for 8 classes. The training set and validation set are divided following the standard in [49]. Consistent with previous researches [S. 10, 11, 50, 45, 25], report the metrics of pboz and Apmesk for performance evaluation.
B.2 Implementation Details
We implement DI-MaskDINO based on Detectron2 [48], using AdamW [31] optimizer with step learning rate schedule. The initial learning rate is set as 0.0001. Following MaskDINO, DI- MaskDINO is trained for 50 epochs on COCO with the batch size 16, decaying the initial learning rate at fractions 0.9 and 0.95 of the total training iterations by factor of 0.1. For BDD100K, following the setting in [22]. we train our model for 68 epochs with the batch size of 8 and the learning rate drops at the 50-th epoch. The number of transformer encoder and decoder layers is 6.
14

---

The token numbers of Te1 and T2 are 600 and 300, respectively. Unless otherwise specified, the feature channels in both encoder and decoder are set to 256, and the hidden dimension of FFN is set to 2048. The mask network and box network in BATO are both three-layer mlp networks. We use the same loss function as MaskDINO (i.e., I.1 loss and GIOU loss for box loss, focal loss for classification loss, and cross-entropy loss and dice loss for mask loss). Under ResNet50 pretrained on ImageNet [40], our model is trained on NVIDIA RTX3090 GPUs. For Swin-L backbone, NVIDIA RTX A6000 GPUs are used for training and validating.
C Additional Diagnostic Experiments
C.1 Diagnostic Experiments Token Selection
Token selection is a crucial step in our proposed residual double-selection mechanism. The number of token selection and the amount of selected tokens may affect the performance. We conduct experiments to verify their impacts. The experimental results are summarized in Tab. and Tab. 8.
The number of token selection. Single-sclection actually represents the situation when disabling DI module. Double-selection corresponds to our proposed method. Additionally, we add token selection on T₂2 for triple-selection. For fair comparison, we set the amount of lastly-selected tokens to the same value (i.e., 300) for single-, double-, and triple-selection. From Tab. 7, we draw two observations: 1) the results of single-selection are significantly lower than those in other situations, indicating the crucial role of DI module; 2) our method achieves the optimal performance with double- selection. The results are explainable. There exists information loss in each selection procedure. Therefore, triple-selection introduces more information loss, leading to lower performance than double-selection.
The amount of selected tokens. Three k and l22 settings in the double-selection mechanism are tested, and their maximum performance gap of A.boo on BDD 100K dataset is 0.4 (i.e., 29.5-29, 1). Similar results are exhibited on the metric A pmask These results demonstrate that our method not sensitive to the hyper- apper-prrameters k and lg2 Furthermore, the experiments on COCO dataset also exhibit the similar results, indicating that our method is robust. At last, we explain that the settings of j and 22 are infinite since they be set as any value from I to 20k. SOTA models such as [18, 25, 24, 14] take 300 as the query number of transformer decoder. In the experiments, y and lo2 set as 300 or the multiple of 300 to align with the settings of the SOTA models.
Table 7: The results of diagnostic experiments the number of token selection. Single-, double-, and triple-selection are represented as sing., doub., and trip., respectively. k, E [1,2, 3) denotes the amount of selected tokens.
Datasets
Contigurations
Aplox
A peossk
sing.
A,=300
28.3
24.9
BDD100K
doub.
kj=600,62=000
29.5
25.7
trip.
k k==600.,=4450kk=33000
29.3
25.5
sing.
k,=300
46.2
41.8
COCO
doub.
kg=600,63=000
46.9
trip.
ky k==000.6=4450..y33300
46.6
42.2
Table 8: The results of diagnostic experiments on the amount of selected tokens with proposed double-selection mechanism.
Datasets
Configurations
A phox
A ppmas
k1=300,6g=300 300
29.1
25.5
BDD100K
ky=600.b,=600
29.3
25.3
k,=000
25.7
k,=300,6;=300
46.6
42.3
COCO
ky=600,6y=600
46.8
42.4
k;=600,,
46.9
42.3
C.2 Diagnostic Experiments the Number of Decoder Layer
We study the effect of different decoder layer numbers for the model performance, and the results are summarized in Tab. 9. We can observe the following: 1) increasing the number of decoder layers
15

---

from 6 to 9 on BDDI0OK dataset results in the performance degradation, which can be explained by the inconsistency between the complexity of the model and the dataset. The BDDI00K dataset only contains 7k training sets and 1K validation sets. The size of BDD100K is small and the model with decoder layers is relatively more complex, leading to the overfitting of the training set; 2) increasing the number of decoder layers will contribute to both detection and segmentation on COCO. However, the model configured with decoder layers only achieves a slight improvement and introduces more computation cost. Therefore, use 6 decoder layers in model; 3) our model has achieved comparable performance the config configuration with decoder layers compared to MaskDINO with decoder layers (e.g., 45.8 45.7 the Aphoz metric and 41.3 v.s. 41.4 on the Apmask metric COCO), demonstrating that our model greatly improves the efficiency of the decoder.
Table 9; The results of diagnostic experiments the number of decoder layer.
BDD10OK
COCO
Decoder layer
Aphox
Apmask
Aphoc
A Apmssk
3
28.7
25.3
45.8
41.3
25.7
42.3
9
28.9
25.6
46.9
42.5
D Visualization Analysis
We visualize the predictions of MaskDINO and DI-MaskDINO to show qualitative comparison on BDD100K dataset. As shown in Fig. 3, MaskDINO produces boxes that do not tightly encompass the objects (i.e., Fig. 3a) do not fully surround the objects (i.e., Fig. 3b). Compared to MaskDINO, our model produces perfectly-fittee addition, our model focuses attention the foreground objects with high category scores through the residual double-selection mechanism that avoids mispredicting the background as foreground object. Fig. 3c suggests that proposed residual double-selectin mechanism effective.
boxes, demonstrating the effectiveness of DI-MaskDINO. In
MaskDINO
DH-MaskDINO
(c)
Figure 3: Qualitative comparison between MaskDINO and DI-MaskDINO on BDD100K dataset.
Suggest zooming in to view this figure for a clearer view of details.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions and scope accurately made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses the limitations of the work in
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper provides a detailed description our proposed DI- MaskDINO model in $ and comprchensive implementation details in $ 4.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
This paper presents a detailed description of our proposed DI-MaskDINO model in 3, along with comprehensive implementation details provided in Appendix B. We will release the source code of our model at the following URL: https: : /githuti com/ CQU-ADHRIE L DII/DDIDMMMMNO
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /ttp://nipippoo cc/
public/ /ggudgesssseeeeee Code SrodeSubmiii ssionPolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cc/ public/ ///gguideee dede/de//// Codee (CodeSuuaaa Submi ss Poooooioiiyiyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper specifies all the training and test details in 4 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive and the previous researches in the field of object detection and instance segmentation did not report enror bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources needed to reproduce the experiments in $ and Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: hete:://nnuuue /neurips....... wt/ppbbccc/ttcccculueeeeeeeoeeee
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: Our research focuses only on improving the overall performance of joint object detection and instance segmentation models. As such, it does not have direct societal impacts beyond the scope of enhancing these technical capabilities.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper cites the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw swr..ww. thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the source code of our model the following URL: https:
/ anonymous. 4open, science / M/DIIMIMMMMM aKDINO-
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "sHWTnIIFrQ", "reviewer": "Reviewer_aN5C", "summary": "The paper starts from an observation regarding imbalance in the intermediate results between detection and instance segmentation, which motivates the authors to propose DI-MaskDINO, which tries to improve the imbalance through the DE-Imbalance module and Balance-Aware Tokens Optimization module. Evaluated on COCO and BDD100K benchmarks, the proposed DI-MaskDINO achieves better results than MaskDINO baseline.", "questions": "Though I hold several concerns as illustrated in the weakness, I feel the paper is generally technically good and with performance improvement over strong baseline MaskDINO. Thus my initial rating is boarderline accept. My major concerns lies in the motivation on imbalance is not so convincing to me, I look forward to the author's rebuttal for further illustration or other more convincing measurement.", "limitations": "No other limitations as far as I know", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well-written and technically solid\n\n- The starting point of det/seg imbalance sounds interesting\n\n- Extensive experiments and achieve improvement over strong baseline MaskDINO", "weaknesses": "- The paper claims improvement over SOTA results, yet it seems that the MaskDINO-SwinL is only reported under 12 epochs setting, while in MaskDINO paper they report results under 50 epochs setting with AP_mask=52.3, AP_box=59.0. I do not see why in tab.1 epochs = 12/24/50 are all reported for R50 backbone but only epochs = 12 are reported for SwinL backbone, making the SOTA claim less comprehensive and less convincing.\n\n\n- The motivation stems from the observation on det/seg imbalance. Yet, I do not see why performance gap between det and seg from the first layer can illustrate the imbalance. If the absolute AP value is used to measure the imbalance between det and seg, then why AP_box < AP_mask at first layer yet AP_box > AP_mask at the last layer? Furthermore, it is also possible that DI-MaskDINO just improves the performance of both branches which naturally reduce the performance gap (e.g., improving det from 10 to 20 may be in similar difficulty of improving seg from 15 to 22, but the absolute gap is reduced.) In short, I believe there lacks a solid definition and study on the \"imbalance\" problem, and the current comparison based on absolute AP value from first layer is not convincing to me.\n\n\n- Minor: I see the reported FPS is significantly different from the ones reported in MaskDINO, which I assume could be the hardware differences. Please explain how the FPS is measured.\n\n \n- Minor: The paper provides an anonymous link to the code/model, yet the provided repo is empty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are greatly encouraged by your positive comments, including \"paper is generally well-written and technically solid\" and \"the starting point of det/seg imbalance sounds interesting\".\n\n**[W1]** We supplement the experiment under the condition of epoch = 50 when the backbone is Swin-L. The results are shown in Tab.R1 of the rebuttal PDF file, indicating that DI-MaskDINO exhibits the superiority over MaskDINO. It is noted that the experiments with the Swin-L backbone are conducted on the same 4 A6000 GPUs with the batch size of 4 (the maximum bacth size that 4 A6000 GPUs supports). The batch size is smaller than that in MaskDINO paper (i.e., batch size = 16) and the 4 A6000 GPUs present weaker computation power than 4 A100 GPUs, thus the results we reproduced are lower than those in the original MaskDINO paper (i.e., $AP^{box}=59.0$, $AP^{mask}=52.3$). In the original version, DI-MaskDINO is compared with MaskDINO under three different settings of epochs = 12/24/50 when the backbone is ResNet, which have demonstrated the superiority of DI-MaskDINO to the certain extent. Therefore, when the backbone is Swin-L, we only test one setting of epoch = 12.\n\n**[W2]** To better answer your question, we firstly explain the definition of \"imbalance\" and the reason for using the absolute AP to measure the \"imbalance\". Then, we analyze why $AP^{box}$ is smaller than $AP^{mask}$ at the beginning layer. Finally, we explain why $AP^{box}$ is larger than $AP^{mask}$ at the last layer.\n\n(1) Explanation of \"imbalance\"\n\nSpecifically, the \"imbalance'' summarizes the phenomenon that the performance of object detection lags behind that of instance segmentation at the beginning layer of transformer decoder. Therefore, \"imbalance'' could be better understood as a concise summary for the above mentioned phenomenon. To quantitatively evaluate the \"imbalance'', we previously think about the idea to align $AP^{box}$ and $AP^{mask}$ to (0,1) to better clarify relative AP value. However, it might be controversial to decide the AP value corresponding to 1. If we define that AP=100 is corresponding to 1, then the relative AP equals to absolute AP. Therefore, we directly use the absolute AP to measure the \"imbalance\".\n\n(2) Why is $AP^{box}$ smaller than $AP^{mask}$ at the beginning layer?\n\nFirstly, the individual characteristics and supervision manners of detection and segmentation tasks lead to the \"imbalance''. Object detection is a region-level regression task to locate the bounding box of an object, while instance segmentation is a pixel-level classification task to classify each pixel of an object. Therefore, object detection relies on more global information that reveals the contextual relations of different patches in an object. However, at the beginning layer, the global information is limited, thus the performance of object detection lags behind.\n\nIn addition, the supervision for object detection is sparse (i.e., a 4D vector of x, y, w, and h), while the supervision for instance segmentation is dense (i.e., hundreds or thousands dimension vector of GT mask pixels). The sparse supervision is weak, but it encourages a model to learn the relatively global contexts, which is challenging at the beginning layer. In contrast, the dense supervision is strong and suitable for local pixel-level classification. \n\n(3) Why is $AP^{box}$ larger than $AP^{mask}$ at the last layer?\n\nFirstly, as mentioned above, object detection relies on more global information. With the increasing of decoder layers, the global information is richer to benefit object detection.\n\nSecondly, object detection is a coarse-grained task that only needs to locate the four vertices of an object bounding box, while instance segmentation is a fine-grained task that require to accurately classify a large number of pixels of an object mask. Therefore, in an overall view, object detection is relatively simpler. As shown in Tab.1 of the original version, for various models (Mask RCNN, HTC, SOLQ, and MaskDINO), $AP^{box}$ is much higher than $AP^{mask}$, indicating it is easier for object detection to achieve higher final performance. Therefore, the performance of object detection is larger than that of instance segmentation at the last layer.\n\n(4) The significance of our work\n\nThe above analysis could potentially reveal the significance of our work. The detection branch and segmentation branch share the unified query, thus the two tasks are interactive and mutually affected. At the beginning layer, as shown in Fig.1(a) in the original version, object bounding boxes do not fit well with object mask, which will hinder the cooperation of the two tasks, leading to the negative interactive effect. By narrowing the imbalance between object detection and instance segmentation at the beginning layer, the two tasks could rapidly reach a mutually-beneficial state, which contributes to improve the performance, as validated by our experiment results in Tab.1 of the original version. In addition, our model also contributes to reduce the parameters of model, as validated by the supplemented experiment results in Tab.R3 of the rebuttal PDF file, which shows that our model configured with 3 decoder layers could achieve the similar performance with MaskDINO configured with 9 decoder layers, and 4.5M (52.1-47.6) parameters are reduced at the same time.\n\n**[W3]** For the fairness, the FPS of all models is tested on the same 4 RTX3090 GPUs when the backbone is ResNet50. When the backbone is Swin-L, the FPS is tested on the same 4 A6000 GPUs, since 4 RTX3090 GPUs could not support the Swin-L backbone.\n\n**[W4]** Despite using the anonymous link, we are still concerned about violating the double-blind reviewing policy. Therefore, we only provide an empty repository to clarify our willingness of releasing the code/model. We will definitely release the code/model if the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eNWWLzrPpp", "reviewer": "Reviewer_5ayP", "summary": "This paper initially observes that in the current state-of-the-art model MaskDINO, the performance of object detection lags behind instance segmentation at the initial layer of the transformer decoder, resulting in a performance imbalance phenomenon.\n\nTo explore whether this \"performance imbalance issue\" is a factor that restricts the effectiveness of the detector, the authors propose the DI-MaskDINO model, which introduces two key components: the De-Imbalance (DI) module and the Balance-Aware Tokens Optimization (BATO) module, to alleviate the performance imbalance between detection and segmentation tasks.", "questions": "- In Experiment Table 1, why were AP_S, AP_M, and AP_L not tested separately for MaskDINO at epochs 12 and 24?\n- In Experiment Table 2, why is there no comparison on the BDD100K validation set for the SwinL backbone?", "limitations": "- Although the paper has proposed the \"performance imbalance issue,\" it only conducted experimental observations on MaskDINO. Can other detectors also improve model performance by addressing the \"performance imbalance issue\"? The paper does not generalize this issue to a more general level, making it more universally applicable.\n- The paper's discussion of the \"performance imbalance issue\" is only focused on the initial layer of the transformer decoder and does not explore whether other layers of the decoder may also have the \"performance imbalance issue.\"", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies a novel issue: the performance imbalance that exists between object detection and instance segmentation, a problem that is less discussed in existing literature. This work is of significant importance for advancing research in the fields of object detection and instance segmentation, particularly by providing new perspectives and solutions for dealing with performance imbalance issues.\n- The proposed model demonstrates certain improvements over the baseline on the COCO and BDD100K benchmark tests, which to some extent indicates that the \"performance imbalance issue\" is a factor that restricts the effectiveness of detectors, and the logic of the paper is coherent.", "weaknesses": "- The lack of in-depth analysis of \"performance imbalance problem\" in this paper, only through the experimental phenomenon, there may be a certain coincidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1]** We are greatly encouraged by your positive comments \"the paper identifies a novel issue, and this work is of significant importance for advancing research in the fields of object detection and instance segmentation, particularly by providing new perspectives and solutions for dealing with performance imbalance issues\". The experimental analyses for the \"imbalance'' have been conducted, as shown in $\\S$ 4.3.1 in the original version. To further solve your concern regarding the \"imbalance'', we make the in-depth analysis from the theoretical perspective. The analysis begins with the explanation for the performance imbalance at the beginning layer of transformer decoder, followed by analyzing essential reasons causing the \"imbalance''.\n\nThe \"imbalance'' in our paper is a concise summary for the phenomenon that the performance of object detection lags behind that of instance segmentation at the beginning layer of transformer decoder. The reasons are multi-fold.\n\nFirstly, the individual characteristics and supervision manners of detection and segmentation tasks lead to the \"imbalance''. Object detection is a coarse-grained region-level regression task to accurately locate the bounding box of an object, while instance segmentation is a fine-grained pixel-level classification task to correctly classify each pixel of an object. Therefore, object detection relies on more global information that reveals geometric, contextual, and semantic relations of different patches belonging to an object. However, at the beginning layer, the global information is limited, thus the performance of object detection lags behind.\n\nIn addition, the supervision for object detection is sparse (i.e., a 4D vector of x, y, w, and h), while the supervision for instance segmentation is dense (i.e., hundreds or thousands dimension vector of GT mask pixels). The sparse supervision is weak, but it encourages a model to learn the relatively global contexts of different patches belonging to an object, which is challenging at the beginning layer. In contrast, the dense supervision is strong, and it is suitable for a model to achieve local pixel-level classification. Therefore, the supervision manners also sharpen the \"imbalance'' at the beginning layer.\n\n**[Q1]** Thank your very much for kindly pointing out this detail. We fetch the results from [25] (i.e., MaskDINO) at epochs 12 and 24, and $AP^{box}\\_{S}$, $AP^{box}\\_{M}$, and $AP^{box}\\_{L}$ are not provided in [25]. We are also unable to get the model weights for testing since the weight files are not available.\n\n[25] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeung Shum. Mask dino: Towards a unified transformer-based framework for object detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3041–3050, 2023.\n\n**[Q2]** Thank you very much for the insightful observation. In the original version, the experiments on BDD100K dataset are treated as additional experiments, thus we only conduct the experiments using ResNet50 backbone on BDD100K dataset. According to your comment, we conducted additional experiments on BDD100K using Swin-L backbone. The results are reported in Tab.R2 of the rebuttal PDF file, from which we can observe that our model presents the advantage.\n\n**[L1L2]** Thank you very much for pointing out the limitations, which inspire us to optimize the model. Regarding the generality, we do not extend the imbalance issue to other models, since the prior goal of our work is to achieve SOTA performance. In the future, we will further study the generality issue. Regarding the imbalance issue in other decoder layers, the imbalance will gradually be weakened with the increasing of decoder layers, thus the imbalance issue is not prominent in other decoder layers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uo7nlSyoEq", "reviewer": "Reviewer_TyXi", "summary": "This paper focuses on the detection-segmentation imbalance issue and proposes DI module with the residual double-selection mechanism to alleviate the imbalance; moreover, Balance-Aware Tokens Optimization (BATO) is proposed to guide the optimization of the initial feature tokens. The proposed method termed DI-MaskDINO, achieves SOTA results in both object detection and instance segmentation.", "questions": "The authors implement De-Imbalance module by stacking several self-attention and Multi-Head Cross-Attention. However, it could be seen as one-layer transformer encoder / decoder. The current phenomenon cannot clarify whether the performance improvement stems primarily from mitigating the detection-segmentation imbalance issue or from the increase in parameters/layer. The authors need to provide evidence to support this claim. For example, they could show that a 6-layer DI-MaskDINO outperforms a 7-layer MaskDINO, etc.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The authors claim that the performance of object detection lags behind that of instance segmentation from the beginning transformer decoder is interesting;\n2. The proposed method achieves SOTA results in both object detection and instance segmentation;\n3. The paper is clear, and the experimental results are detailed.", "weaknesses": "The technological innovation is somewhat limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments on our work, such as \"the proposed method achieves SOTA results\" and \"the paper is clear and the experimental results are detailed\". There are two concerns regarding technological innovation and whether the performance improvement stems primarily from mitigating the detection-segmentation imbalance issue. Our responses to the two concerns are as follows.\n\n**[W1Q1]** \"The technological innovation is somewhat limited. De-Imbalance module could be seen as one-layer transformer encoder / decoder.\"\n\n**Response:** From the technological view, De-Imbalance module is composed of several self-attention and cross-attention. However, we carefully explain that self-attention and cross-attention are basic units to build our De-Imbalance structure. Therefore, **the technological innovations of De-Imbalance module mainly lie in its explanation, simplicity, and effectiveness, which contributes new inspiring thought to the community to push forward the study of fundamental object detection and instance segmentation tasks.** The detailed analyses are as follows.\n\n(1) De-Imbalance module is explainable and effective\n\nIn transformer-based joint object detection and instance segmentation models, the two tasks are closely-related and mutually-affected, and the mutual effects could be positive or negative. The imbalance brings the negative impact on model performance. As shown in Fig.1(a) of the original version, at the beginning layer, object bounding boxes do not fit well with object mask, which will hinder the cooperation of the two tasks, leading to the negative effect. In contrast, the interaction under a balanced state makes the two tasks mutually-beneficial.\n\nHowever, most joint object detection and instance segmentation models (e.g., SOLQ and SOIT) do not focus on how the interaction between the two tasks affects the model performance. We focus on the imbalance issue through studying SOTA joint object detection and instance segmentation model MaskDINO. Furthermore, we propose De-Imbalance module to alleviate the imbalance. It is noted that the detection-segmentation imbalance means that the performance of object detection lags behind that of instance segmentation at the beginning layer of decoder. Therefore, the core idea of De-Imbalance module is to strengthen the performance of detection to alleviate the imbalance at the beginning layer of decoder. By narrowing the imbalance between the two tasks at the beginning layer, the two tasks could rapidly reach a mutually-beneficial state, which contributes to improve the performance. In addition, a large number of experiments (e.g., Tab.3-5 in the original version) have proven the effectiveness of De-Imbalance module.\n\nIn summary, De-Imbalance module contribute to handle the long-standing and naturally-existing imbalance issue between object detection and instance segmentation, which is commonly ignored in previous works. Although De-Imbalance module is composed of basic network units (e.g., self attention), they have been proven to be explainable and effective.\n\n(2) The simplicity of our model\n\nTo analyze the simplicity of our model, we compare the parameters of DI-MaskDINO configured with different numbers of decoder layers with that of MaskDINO, and the results are summarized in Tab.R3 of the rebuttal PDF file. It is worth noting that our model has only 6 decoder layers (52.3M), while MaskDINO contains 9 decoder layers (52.1M). Our model achieves higher performance, at the cost of involving only 0.2M parameters. We can also observe that our model with 3 decoder layers has achieved comparable performance compared to MaskDINO with 9 decoder layers (i.e., 45.8 v.s. 45.7 on $AP^{box}$ and 41.3 v.s. 41.4 on $AP^{mask}$), saving 4.5M parameters at the same time. \n\nIn addition, alleviating the imbalance contributes to accelerate the convergence. As shown in Tab.1 of the original version, DI-MaskDINO presents significant advantage under the training condition of epoch = 12, which potentially reveals that our model reaches the convergence with a faster speed. This is attributed to that alleviating the imbalance issue promotes the collaboration between the two tasks, thus the two tasks could rapidly reach a mutually-beneficial state.\n\n**[Q1]** \"Whether the performance improvement stems primarily from mitigating the detection-segmentation imbalance issue or from the increase in parameters/layer.\"\n\n**Response:** Thank you very much for the insightful comment. According to your comment, we conduct the experiments to validate that the performance improvement does not stem from the increase in parameters/layer. In detail, we test the performance of DI-MaskDINO configured with 3, 6, and 9 decoder layers and compute the parameters of corresponding configurations, respectively. The results are reported in Tab.R3 of the rebuttal PDF file, indicating that the performance improvement does not stem from the increase in parameters or decoder layers. Specifically, our model with 3 decoder layers has achieved comparable performance with MaskDINO, and 4.5M (52.1-47.6) parameters are reduced at the same time. In contrast, the performance improvement stems from mitigating the detection-segmentation imbalance issue, which could be evidenced by the experiment results in Tab.4 of the original version. When enabling De-Imbalance module, the overall performance significantly increases (i.e., from 45.6 to 46.4 on $AP^{box}$ and from 41.2 to 42.1 on $AP^{mask}$).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3JplsH1wS8", "reviewer": "Reviewer_KbFf", "summary": "This work investigates the detection-segmentation imbalance issue in MaskDINO. It proposes DI-MaskDINO model with the residual double-selection mechanism to alleviate the imbalance. The framework mainly includes De-Imbalance and Balance-Aware Tokens Optimization. Experiments prove the effectiveness.", "questions": "My main concern is the motivation and analysis of the designed component. Please refer to the Weakness section.", "limitations": "Limitations are discussed in the work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The finding of detection and segmentation imbalance at the beginning of MaskDINO is interesting.\n2. Experiments prove the effectiveness on various benchmarks.\n3. Overall, the whole framework is clear and easy to follow.", "weaknesses": "1. The motivation of De-imbalance module requires more verification. The authors claim \"The token interaction is the key point to make sure that the secondly-selected tokens are beneficial for detection\" in L169. But there are no experiments or theory to prove this claim or design. This makes the reviewer confused why the token interaction can play such a role. It's better to add some experiments or heatmap visualization.\n2. The Balance-Aware Tokens Optimization module contains several components. It's essential to give experimental analysis of each design.\n3. This work is built on MaskDINO, it is important to generalize this proposed manner to other decoder-based methods. It can further validate the generality of the proposed approach.\n4. Experimental results on COCO test set should be provided for fair comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are greatly encouraged by your positive comments, including \"the finding of detection and segmentation imbalance at the beginning of MaskDINO is interesting\" and \"the whole framework is clear and easy to follow\".\n\n**[W1]** Thank you very much for the insightful observation. To clearer clarify the effect of token interaction, we firstly make the analysis from the theoretical view, and then prove its effect by experiments.\n\n(1) Theoretical analysis\n\nEach token actually corresponds to a patch (remarkably smaller than an object in most cases) in the image [52], and the bounding box of an object is regressed by integrating the multiple patches (belonging to the same object) that have global patch-to-patch spatial relations, thus it is really needed for the detection task to learn the interaction relation between patches. In contrast, the dense all-pixel supervision for the segmentation task mainly focuses on local pixel-level similarity with GT mask [25], hence the segmentation task is not particularly depend on the patch-to-patch relation as the detection task. Via token interaction, different tokens representing the patches (belonging to the same object) can interact with each other to learn the global geometric, contextual, and semantic patch-to-patch relations, benefiting the perception of object bounding boxes. Therefore, executing token interaction makes De-Imbalance module to be more beneficial for detection.\n\n[25] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeung Shum. Mask dino: Towards a unified transformer-based framework for object detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3041–3050, 2023.\n\n[52] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6881–6890, 2021.\n\n(2) Experimental analysis\n\nTo validate the effectiveness of token interaction, we test the performance of our model under two configurations: w/o token interaction and w/ token interaction. The experiment results are reported in Tab.R4 of the rebuttal PDF file. The configuration with token interaction yields higher performance, demonstrating the effect of token interaction.\n\n**[W2]** Thanks for your suggestion. In DI-MaskDINO, De-Imbalance (DI) module is the critical design to alleviate the detection-segmentation imbalance, generating balance-aware query $Q\\_{bal}$ in Eq.4. Balance-Aware Tokens Optimization (BATO) module serves as an auxiliary design to utilize $Q\\_{bal}$ to optimize the initial feature token $T\\_{i}$ and finally compute balance-aware feature tokens $T\\_{bal}$. The effect of BATO is validated by the diagnostic experiments on main modules, and the results are reported in Tab.4 of the original version. We can observe that, when BATO is enabled, the performance obtains significant improvement, demonstrating the effectiveness of BATO.\n\nThe main component of BATO is the Guiding Token Generation (GTG), which uses $Q\\_{bal}$ to generate overall guiding token $T\\_{g}$. $T\\_{g}$ is then used to optimize $T\\_{i}$ via Multi-Head Cross-Attention. Multi-Head Cross-Attention is a fundamental module that could not be disabled (i.e., if it is disabled, $T\\_{i}$ will be directly inputted to the transformer decoder as MaskDINO does). Therefore, in the diagnostic experiments on BATO module, we only evaluate the effect of GTG by testing the performance of model under the configurations with/without GTG. The experiment results are reported in Tab.6 of the original version.\n\n**[W3]** Thank you very much for this comment. Based on the structure characteristic of existing transformer-based joint object detection and instance segmentation models, our proposed De-Imbalance module (the core of our model) could be friendly applied on other models. Transformer-based joint object detection and instance segmentation models have a similar model architecture, consisting of backbone, transformer encoder, transformer decoder, and prediction heads. De-Imbalance module takes the output of transformer encoder as the input, and the output of De-Imbalance module is then taken as the input of transformer decoder. Therefore, adding the De-Imbalance module to a existing model will not hurt its main structure. Similarly, our proposed Balance-Aware Tokens Optimization module can also be directly added between De-Imbalance module and transformer decoder. Therefore, our proposed modules can be easily applied to other transformer-based joint object detection and instance segmentation models. We would like to carefully explain why the generality is not prioritized in the original version. Though the generality of a model is important, the prior goal of our work is to make the model achieve SOTA results.\n\n**[W4]** Thank you very much for the suggestion. According to your suggestion, we compare DI-MaskDINO with MaskDINO on COCO test-dev. The results are summarized in Tab.R5 of the rebuttal PDF file, which demonstrates the effectiveness and robustness of DI-MaskDINO. We note that the test-dev evaluation server is only available to object detection task, since the testing needs to upload a prediction results file of json format to the evaluation server provided by the COCO dataset website (https://cocodataset.org/?spm=a2c6h.12873639.article-detail.137.34fa30e89NSdTI#upload), but the website does not provide evaluation server for instance segmentation task. Therefore, the performance of instance segmentation is not reported.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers for their constructive comments and suggestions.

We are particularly encouraged that the reviewers unanimously acknowledge **our work is interesting** (aN5C, 5ayP, TyXi, and KbFf). Reviewers commend us for **achieving state-of-the-art results** (aN5C, 5ayP, TyXi, and KbFf). Reviewers comment positively on **writing and presentation** (aN5C, TyXi, and KbFf). Specifically, reviewer 5ayP remarks that "**the work is of significant importance for advancing research in the fields of object detection and instance segmentation**". Reviewer aN5C notes that our paper is "**technically solid**". Reviewer KbFf finds "**the whole framework is clear and easy to follow**".

Meta-review: This work proposes DI-MaskDINO, aiming to improve the detection-segmentation imbalance of the state-of-the-art model MaskDINO. Specifically, the proposed De-Imbalance (DI) module and Balance-Aware Tokens Optimization (BATO) module are integrated into MaskDINO, where DI generates balance-aware query and BATO uses the query to guide the optimization. Promising results have been demonstrated on COCO and BDD100K datasets.

Initially, the reviewers were concerned about Swin-L experiments, more in-depth analysis of the imbalance problem, and so on. During the reviewer-author discussion period, the author rebuttal successfully assuaged the reviewer concerns. In the end, the paper received all accept recommendations (3 borderline accepts and 1 accept). After considering the author rebuttal and reviewer discussion/comments, the area chair agrees with this recommendation.

The authors are encouraged to incorporate the rebuttal and reviewer suggestions to their final camera-ready version.
Decision: Accept (poster)