Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks
Authors: Zhiyang Chen
Yousong Zhu
Zhaowen Li
Fan Yang
Wei Li
Haixin Wang
Chaoyang Zhao
Liwei Wu
Rui Zhao
Jinqiao Wang
Ming Tang
Venue: NeurIPS 2022 Accept (2022)
Keywords: transformer
general visual framework
sequence prediction
multi-task

Abstract: Visual tasks vary a lot in their output formats and concerned contents, therefore it is hard to process them with an identical structure. One main obstacle lies in the high-dimensional outputs in object-level visual tasks. In this paper, we propose an object-centric vision framework, Obj2Seq. Obj2Seq takes objects as basic units, and regards most object-level visual tasks as sequence generation problems of objects. Therefore, these visual tasks can be decoupled into two steps. First recognize objects of given categories, and then generate a sequence for each of these objects. The definition of the output sequences varies for different tasks, and the model is supervised by matching these sequences with ground-truth targets. Obj2Seq is able to flexibly determine input categories to satisfy customized requirements, and be easily extended to different visual tasks. When experimenting on MS COCO, Obj2Seq achieves 45.7% AP on object detection, 89.0% AP on multi-label classification and 65.0% AP on human pose estimation. These results demonstrate its potential to be generally applied to different visual tasks. Code has been made available at: https://github.com/CASIA-IVA-Lab/Obj2Seq.

Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks
Zhiyang Chen
Yousong Zhu'
Zhaowen Li Fan Yang
Wei Li
Haixin Wang
Chaoyang Zhao?
Liwei Wu Rui Zhao Jinqiao Wang!.2. Ming Tang"
National Laboratory of Pattern Recognition, Institute of Automation,
Chinese Academy of Sciences
"ccooo of Artificial Intelligence, University of Chinese Academy of Sciences *eng Cheng Laboratory
SSenseTime Research
(zhiyang. chen,yousong yousong. zhu, zhaowen. 1i haaixii www.xiin.wwnn)nnll.......... ac. ac cn yangfar 2anifan2200 20222222222..a ac. (liwei1, wuliwei, zhaorui Aaennietsnennnneeeeee com {chaoyang, zhao, tangm, jqwang) ia ac. on
Abstract
Visual tasks vary a lot in their output formats and concerned contents, therefore it is hard to process them with an identical structure. One main obstacle lies in the high-dimensional outputs in object-level visual tasks. In this paper, propose an object-centric vision framework, Obj2Seq. Obj2Seq takes objects basic units, and regards most object-level visual tasks as sequence generation problems of objects. Therefore, these visual tasks can be decoupled into two steps. First recognize objects of given categories, and then generate sequence for each of these objects. The definition of the output sequences varies for different tasks, and the model is supervised by matching these sequences with ground-truth targets. Obj2Seq is able to flexibly determine input categories to satisfy customized requirements. and be easily extended different visual tasks. When experimenting on MS COCO, Obj2Seq achieves 45, 7% AP on object detection, 89.0% AP on multi-label classification and 65.0% 65.0% AP on human pose estimation. These results demonstrate its potential to be generally applied to different visual tasks. Code has been made available at: hhtpps: https:///gitubte com/CASIA IVA LLb/ DDbbb/bbeeegeee
Person;
Class: Raseball
(thject Derection
Keuse 22, Yeare
B. 62,
Class: Trunk; Task: Classificttion
store 06 Sanry 10,
45,
No Trunks!
Obj2Seq
Parson
Baseball Bat
Trunk
Figure 1: Obj2Seq is unified framework for visual tasks. It takes categories as inputs, and generates desired output sequences according to different tasks.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Introduction
Deep learning has made great progress in the field of computer vision 20 35 19 26], and recently researchers are seeking for a unified framework for various visual tasks. This idea has been widely discussed in Natural Language Processing (NLP), which results in the emergence of general language models. In NLP, all tasks are formatted as sequence-to-ssqquence problems in order to be processed by a single transformer [43. Meanwhile, prompt learning [16] is utilized so that the general model is compelled to perform specific task. With the help of large-scale text data, numerous pre-trained language models are developed [3] of language tasks.
49
and they achieve excellent performance wide range
With these encouraging achievements in NLP, it is natural to ask whether can implement similar framework in computer vision as well. One of the two obstacles to realize this is that visual tasks defined in various formats. They can hardly be unified like sequence-to-seoe -es-uueeeeeeeeeeeeeeeee problems in NLP, especially those object-level tasks with high-dimensional outputs. The other is that the model may encounter diverse categories even for identical task. If the model is not aware of what it will process in advance, we cannot expect it to satisfy customized requirements flexibly. There are some works taking a step forward to solve these problems. M-DETR [X takes sentences as inputs in addition to the images, so that it is able to flexibly detect specific targets that mentioned and realize target-aware inference. However, M-DETR is designed for object detection only, and hard to extend for general usage. Pix2Seq [9] is a forward looking work to unify the formats of different tasks. It turns object detection into a sequence generation task, and directly applies next token generation to yield bounding boxes and categories. Even so, Pix2Seq is not aware of the desired targets before inference, and the output sequence might become extremely long under sophisticated scenes.
In this paper, we propose a unified visual framework, Obj2Seq. in order to adapt the sequence generation framework for object-level visual tasks. Obj2Seq formats these tasks into two-step process. First, with images and class prompts as inputs, Obj2Seq determines whether each given category exists not in an image, and detects objects of these categories. Then sequence is generated as the output for each object in response to the desired task. Many visual tasks can be filled in this template by changing class prompts and the definition of output sequences. For each specific task, class prompts provide additional instructions to impel the framework to attend to objects of specified categories, and the output sequences are defined accordingly to include information required by the task. This gives the credit to two new modules, Prompted Visual Indicator that receives class prompts from the inputs, and General Sequence Predictor that outputs sequences in unified format for each object. Obj2Seq has two main advantages. On one hand, the behavior of Obj2Seq can be controlled and manipulated. By taking different class prompts as inputs, Obj2Seq is able to focus on different objects and flexibly satisfy specific requirements under practical scenes. On the other hand, the sequences generated by Obj2Seq are of unified format to describe objects, which makes it ready to be extended for wide range of object-level visual tasks.
In order to validate the versatility of our framework, we conduct experiments on MS COCO (28) for some common visual tasks. Obj2Seq achieves 45.7% AP on object detection and 89.0% AP on multi-label classification when trained for object detection, while it also achieves 65.0% AP on human pose estimation with keypoint annotations. These results indicate that our framework is effective for different visual tasks, and has potential for general usage.
2 Related Works
2.1 General Frameworks for Visual Tasks
The idea of pretraining a general model is firstly proposed in NLP 13 1 Currently, similar trend appears in computer vision as well. Researchers first borrow transformer as the base model 30 45 [2]. and then design novel self-supervised algorithms to pretrain them with the help
of huge amounts of unlabeled data and modern hardware [10] 5
18 g 27|. The pretrained models capable of extracting robust feature for downstream tasks. However, since only the pretrained backbone is provided, downstream tasks still have to design their specific head structures and need further training to get desired results.
25
Recently, the academic raises a higher requirement to train a general framework that can be directly applied in different tasks. It should be trained in an end-to-end way, and does not need finetuning

---

for specific task. Multi Task Learning (MTL) is one related field [6] 38 17 It constructs shared backbone and multiple heads to process a fixed combination of tasks. However, since MTL needs individually designed heads and sufficient labels for every sample, it is expansive and unable to be extended to other tasks. Recently, works like MI--Decoder [37 and X-DETR ( attempt to construct general framework for a set of some similar tasks. However, they can hardly be generalized for tasks with totally different definitions. Pix2Seq ! provides a forward solution to convert an image into sequence of words. However, it lacks interface to flexibly adopt for different scenes. In addition, the whole image is sometimes too complex to be represented in a single sequence.
Different from previous work, we take objects as the basic unit in visual tasks, and unify them as sequence generation for each single object. Additional class prompts are used specify the objects in need.
2.2 Prompt Learning
Prompt learning [16] is firstly develop in NLP in order to directly apply pre-trained language models in differrnt tasks. It does not need finetuning model parameters, but rather modifies the inputs and uses a template to indicate the proposed task. Therefore, the model handles all tasks a unified manner.
In order to train vision models with easily available image-text pair data, the concept of prompt is firstly introduced to construct training targets with label text in computer vision [34 48). These algorithmss help supervision with large-scale data only for a certain task. The prompts are used as labels, and they cannot control the inference. VPT [22 and FP-DETR 44 add prompt parameters in their framework. These parameters are trained in the finetune stage for each task. However, they do not make use of language embeddings, and the effect of these modules is hard to explain. ML-Decoder [37] and MDETR [23] use text to impel the model to focus on specific categories or objects. It allows us interfere the model behavior, however, these frameworks are limited to group of similar tasks. In this paper, class prompts are used to give instructions about what categories to detect, and Obj2Seq is able to perform a wide range of vision tasks.
2.3 Sequence Prediction
Sequence prediction is basic formulation for various NLP tasks, since the words in sentences are sequential naturally. It has the characteristic that the forward process at time i depends features or outputs from previous time steps. Currently transformer [43] is widely utilized to solve these problems. In order to generate sequence, the output token at time is taken the input token of the next step. Some up-to-date pre-train algorithm use the input sentence as the target output, in order to perform next-token prediction for unsupervised training [ 49
As to computer vision, there is no preset order the input image. Sequence prediction only available in image generation since they generate output image pixel by pixel [22 When transformer becomes popular, it is firstly applied to process a group of image patches [14 or an unordered set of objects [4. Recently, the idea of next-token prediction is borrowed in order to develop novel self-supervised training algorithms 7 21 However, these algorithms are still closely related image generation, they predict the image patches in sequence. Pix2Seq [9] is the first attempt to apply sequential prediction as general method in a traditional visual task, and achieves decent performance. It regards the whole detection results semantic structures in the original image. On the contrary, our framework considers a more basic unit, objects, in the field of computer vision. This makes Obj2Seq more suitable for visual tasks, and achieves better results.
single sequence, which abandons
3 Methods
3.1 Overview
Obj2Seq is a unified visual framework for object-level tasks. It takes an image I E 3x group K class prompts C [c())_..c(k3] K'd as inputs, and outputs a set of objects, each with sequence to describe it. It contains four main parts listed as below:
Hox W. and a
3

---

hiporttic matching loss
GLA sydtt Sit
Sequence Predictus
Image Fenture Encoder
Fj
Fo
Object
Transforerr
Decoder
Person
Prompted Visual
Horse
Dog
Indicator
No Dogs
(a)
Initialized Onject Querier
Claseificatio
Result
Resulls
Keypoinl
Classifeer
W
h
Aeose Muace
Manktez
Iuage Foutuues
Cross Artn
Add Norm
Self Artn
FFN
Add & Norm
Object Query
10
Image Features
Cross Attn
Tash Pas Endh
Class Embesidings
Forward
Forward and Stog
(b)
Figure (a) The overall framework of Obj2Seq. (b) Illustration of Prompted Visual Indicator. (c) Illustration how General Sequence Predictor generates sequence for an object. We take object detection and human pose estimation examples. The gray part is utilized only for human pose estimation.
Image Feature Encoder & to extract image features.
Prompted Visual Indicator to receive class prompts from inputs and initialize object queries.
Object Transformer Decoder D to encode object-related features into object queries.
General Sequence Predictor P generate sequence of desired attributes for each object.
When the inputs I and C are provided, Obj2Seq first extracts image features F₁ with Image Feature Encoder E. Then, Prompted Visual Indicator receives class prompts. judges the existence of each category for classification, and initializes class-aware object queries Fo. Each object query fo E I xxd; is responsible for detecting and describing one object of a specific class. Afterwards, into Object Transformer Decoder D. The decoder extracts object-related
these object queries features from F, according to Fo, and transforms Fo into well-encoded object queries Fo. These new queries contain sufficient information to describe their corresponding objects as the task requires. Finally, General Sequence Predictor generates sequence for each object query, which contains the desired attributes and constitutes the final result. The whole pipeline is shown in Figure 2a
In the following sections, mainly elaborate on Prompted Visual Indicator I and General Sequence Predictor P, and then supply additional details other parts our implemention.
3.2 Prompted Visual Indicator
Different from processing with complete category set, human beings usually know what categories focus on in advance each task. This prior information makes their behavior more compact and accurate. Also, the categories can be flexibly changed according specific requirements. Therefore,

---

is it possible that we provide not only the images but also additional instructions as inputs, so that the model is forced focus on objects of specified categories?
Prompted Visual Indicator borrows the idea of prompt learning from NLP 16 It receives additional class prompts inputs, therefore it can classify the input image with specific categories, and generate input-dependent object queries to focus on desired objects, As shown in Figure 2b each class prompt a vector representing a certain category ( E R These input vectors are sent into prompt blocks with cross-atiention modules that can extract class-related features from Fj. After processing by these blocks, the output vectors f(( represent features about specified categories in the input image. These vectors can be classified by a class-related binary classifier in Eq. ① The classifier compares them with original class vectors, and outputs score ( to indicate the existence each category. Moreover, these features are utilized to generate class-aware object queries for further inference. We retain K' categories that are more likely to exist with certain retention policy, and perform Cartesian product with set of N position embeddings to generate N object queries for each retained class. Each object query f. is the sum of class vector f(²) and position embedding. These object queries are sent into Object Transformer Decoder for further process. The retention policy is detailed in Section 3.4
sigmaid((neea((f.")) (kk) /V
With the help of Prompted Visual Indicator, Obj2Seq can take not only images but also desired categories as inputs. The model can flexibly classify and detect desired objects with given categories, and therefore is able to achieve better performance. Moreover, each category is processed in an isolated way. Obj2Seq is able to take any category subsets the input, without any influence in results. This makes our framework flexible and easy to implement different scenes.
3.3 General Sequence Predictor
In the field of NLP, task-agnostic language model usually converts all language tasks into a sequence- co-sequence format. This is intuitive, because we always answer questions by speaking a sentence out. As to visual tasks, we can also describe the input image with a sentence. Since the image contains more sophisticated information, the sentence should be better organized with a more basic visual unit, objects. In order to satisfy different requirements, we may need the bounding box, keypoint offsets or some other attributes in this sentence to describe an object. Therefore, we construct General Sequence Predictor, that generates a sequence of attributes to describe an object. This paradigm can be generalized among a wide range of visual tasks.
General Sequence Predictor P takes object queries processed by Object Transformer Decoder as inputs, and outputs sequence of attributes for each object. We illustrate how it works in Figure 2cc taking object detection and human pose estimation as examples. For simplicity. we only elaborate on the prediction process for one object query, and omit the superscript (i). The predictor P first takes the object query fo the input Zin,0 of the first time step. The input is then sent into self- and cross-attention layers to get the output feature 2out,0- For any other time step 1, P takes the previous output feature 1 as the input Zin.t, and the self-attention layers always take all previous inputs Zin,0:t to calculate keys and values. After the output feature zoat, is obtained, a línear layer is utilized to convert it into a logit 2,. Finally, logits from all time steps are further interpreted into specific attributes in a non-parametric way, according to different tasks. We formulate this process as in Eq. 2
fo.
t= 0
Zin,t
Zoat,t-1, t>0
Zhidden,t SelfAAll(zzzn Alln(Zin, Wei WQizin,1: WKV)
Zont, CrossAllni Allnsnnstt( 22hidden, Ff)
2t i
General Sequence Predictor provides unified way to generate output attributes for each object. It can handle different oeject-level visual tasks with identical structure, which makes Obj2Seq ready to be implemented for general usage.
5

---

3.4 Other Details
Image Feature Encoder Image Feature Encoder & consists of a resnet backbone and 6 multi-scall deformable encoder layers. which is exactly the same as [52]. The backbone extracts multi-scale feature maps {Fi) H from C3 to Cg and additional downsampled scale Cs. These feature maps are transformed to have d 256 channels, and then processed by deformable encoder layers to obtain Fy.
Object Transformer Decoder Object Transformer Decoder 'D consists of 4 multi-scale deformable decoder layers. A decoder layer contains self-attention and multi-scale deformable attention modules, which is elaborated in [52]. Here we only implement 4 decoder layers for fair comparison in computation with other methods.
Retention policy Here we discuss about how Prompted Visual Indicator retains categories and generates class-aware object queries. During training, categories with ground truth objects first guaranteed to be retained. Then we select categories with the highest scores to fullfill a fixed number of K' classes. While in inference, we can retain either top-K' categories or all categories over fixed threshold. The influence of these policies is discussed in Section 4.4.3
Objectness branch Besides describing an object with sequence, implement separate objectness branch recognize if the object exists. This branch provides score s for each object as in Eq. (3)- We first compute the probability of each object with given image and categories, and then multiply it with its corresponding class existence (k) In Section 4.4.2 we will elaborate on the efficacy of this objectness branch.
S
(()) = P(CIT) P(O)I,C) sigmoid(linear(ff cc
(k)
/vd/
Training Criterion The outputs of Obj2Seq is a set of objects with specified categories. These objects are equivalent, and do not appear certain order. Therefore, we use bipartite matching loss for supervision [4] It matches object queries with ground truth so that the loss of matched pairs minimized, and then optimizes this optimal match. This loss makes sure that each object is only predicted with one object query, eliminating the need of non-maximum suppression. The definition of the detailed loss function varies for tasks, which will be elaborated in Appendix. As to the classifier in Prompted Visual Indicator, we regard it as an auxiliary multi-label classification problem, and supervise it with asymmetric loss [36].
Experiments
In this section, we first conduct experiments three different visual tasks separately, including object detection, multi-label classification and human pose estimation. These results demonstrate how Obj2Seq solves visual tasks in unified way. After that, some ablation studies are provided to better comprehend the detailed options in Obj2Seq. Here we only provide overall results. Implementation details please refer to Appendix.
4.1 Experiments Object Detection
In Table I we compare Obj2Seq with a group of popular detection frameworks on MS COCO (8 We mainly compare it with different variations of DETR [4) here, because they also end-to-end object detection frameworks without NMS, and Obj2Seq shares some similarity with Deformable DETR (2). All variations use multi-scale features dilated convolution in stage 5 (DC5) for similar computation cost. Meanwhile, we also present some representative convolution- based frameworks 35 40 too, and they are trained with longer schedulers reported in former papers.
Obj2Seq achieves 45.7% mAP with R50 as the backbone, without bells and whistles. It is +1.2% higher than Deformable DETR using the same multi-scale deformable attention, and outperforms other detection frameworks as well, no matter DETR-liK or convolution-base frameworks with longer scheduler. With R101 the backbone, Obj2Seq also achieves 46.1%. We notice that

---

Table Object detection results on MS COCO. All frameworks takes R50 (the upper half) and R101
(the lower half) as the backbone. Results with indicates they use DC5-variation of resnet.
Method
Epochs
GFlops
Params
AP
AP
APTs
APs
APM
APt
Faster RCNN+FPN
108
180
42M
42.0
62.1
45.5
26.6
45.4
53.4
TSP-RCNT 40
96
188
64M
45.0
64.5
49.6
29.7
47.7
58.0
Pix2Seq
300
38M
43.2
61.0
46.1
26.6
47.0
58.6
DETR
500
187
41M
43.3
63.1
45.9
22.5
47.3
61.1
SMCA
15
50
152
40M
43.7
63.6
47.2
24.2
47.0
60.4
Conditional DETR
195
44M
43.8
64.4
46.7
24.0
Anchor DETR!
46
50
171
37M
44.2
64.7
47.5
24.7
48.2
60.6
DAB-DETR 29
202
44M
44.5
47.7
25.3
62.3
Deformable DETR
52
50
171
40M
44.5
63.5
48.8
27.1
47.6
59.6
Obj2Seq-R50 (Ours)
50
184
40M
45.7
64.8
49.5
28.0
48.8
60.2
Faster RCNN+FPN
108
246
60M
44.0
63.9
47.8
27.2
48.1
56.0
TSP-RCNN 40
96
254
83M
46.5
66.0
51.2
29.9
49.7
59.2
Pix2Seq
300
57M
45.0
63.2
48.6
28.2
48.9
60.4
DETR
500
253
60M
44.9
64.7
47.7
23.7
49.5
62.3
SMCA
15
50
218
58M
44.4
65.2
48.0
24.3
48.5
61.0
Conditional DETR
262
63M
45.0
65.5
48.4
26.1
48.9
62.8
Anchor DETR
46
50
237
56M
45.1
65.7
48.8
25,8
49.4
61.6
DAB-DETR
29
50
282
63M
45.8
65.9
49.3
27.0
49.8
63.8
Obj2Seq-R100 (Ours)
251
59M
46.1
65.3
50.0
27.7
50.0
61.4
DAB-DETR achieves significantly higher APL than Obj2Seq. This possibly attributes to its enhanced position embeddings, since reference points with learnable wh may better fit objects with extreme scales. In general, these results indicates that Obj2Seq is an effective framework for object detection. It achieves excellent performance.
4.2 Experiments on Multi-Label Classfication
Prompted Visual Indicator does not only generate object queries for Object Transformer Decoder, also classify if the input categories exist in the image, and output scores for them. Therefore, we able to perform multi-label classification as well with the outputs from Prompted Visual Indicator. We evaluate the model trained in Section 4o MS COCO multi-label benchmark. The classifier in Prompted Visual Indicator achieves 89.0 mAP, which comparable to most multi-label classification solutions. We will elaborate on its detailed behavior in Appendix.
4.3 Experiments on Human Pose Estimation
Current human pose estimation algorithms are generally divided into three categories, bottom-up methods, top-down methods with image crop, and top-down methods in an end-to-end way. All of them involves sophisticate structures, such as image crop 50 or Rol Align 24 19 for
There are few works directly obtaining keypoints from the input image 39) In Obj2Seq. General Sequence Predictor can directly read the coordinates out from object queries without any task-specific model structures. We also provide a bascline for better comparision. The only diffcrence between the bascline and Obj2Seq is that the baseline model use simple MLP instead of General Sequence Predictor. The details in hyperparameters and training configurations are claborated in Appendix.
top-down methods and group post-processing for bottom-up methods 33 3
Our experiments are conducted on the COCO keypoint challange 28 and we mainly list other end-to-end top-down methods Table for comparison. Obj2Seq achieves 60.1% mAP with 50-epoch training scheduler. It reaches reasonable performance as an end-to-end framework with no delicate structures. The result with General Sequence Predictor +2.9% higher than the MLP-head baseline, which indicates that it is more effective predictor. This improvement may attribute to the interdependence between different keypoints. Knowing the exact position of some keypoints benefits the prediction of others. We also attempt to train Obj2Seq with longer training schedule. Training

---

Table Human pose estimation results on MS COCO. We mainly compare with end-to-end frame-
works here. Results with indicates they utilize pre- pre-trained detector.
Method
Backbone
Epochs
AP
APso
AP7s
APM
APr
AR
Mask-RCNN 19
ResNet50
63.1
87.3
68.7
57.8
71.4
CenterNet
51
Hourglass- 104
150
63.0
86.8
69.6
58.9
70.4
DirectPose
41
ResNet50
100
63.1
85.6
68.8
57.7
71.3
PRTR
24
HRNet-W448
64,9
87.0
60.2
72.5
74.1
POET
39
ResNet50
250
53.6
82.2
57.6
42.5
68.1
61.4
baseline
ResNet50
57.2
83.3
63.7
51.5
66.3
65.7
Ohj2Seq
ResNe150
50
60.1
83.9
66.2
54.1
69.5
68.0
Obj2Seq
ResNet50
150
65.0
86.5
71.8
59.6
74.0
72.7
Table 3: Experiments on different parts of Obj2Seq.
Class Prompt
Sequential Predictor
mAP
AP50
AP75
APs
APM
APL
44.5
63.5
48.8
27.1
47.6
59.6
45.1
64.4
49.4
27.4
48.5
60.0
w/o separate Ohjectness
45.1
63.8
48.9
26.8
48.5
60.0
w/ separate Objectness
45.7
64.8
49.5
28.0
48.8
60.2
for 150 epochs can further improves mAP to 65.0%. art methods (listed in Appendix), but has potential for further improvements. Nevertheless, these experiments validate that Obj2Seq can be implemented in human pose estimation, and we believe it also available for other visual tasks.
Obj2Seq currently falls behind of start-of
4.4
Ablation Studies
In this section, we examine different parts and ablate on some important options in Obj2Seq. We conduct these experiments on object detection for demonstration.
4.4.1 Overall Structure
We mainly elaborate the existence of two modules in Obj2Seq, Prompted Visual Indicator and General Sequence Predictor. Removing them leading framework similar to Deformable DETR. Therefore, we directly takes Deformable DETR as the baseline. Table shows that both modules provide steady improvements for Obj2Seq. With the help of Prompted Visual Indicator, Obj2Seq able process objects of different categories separately, and distinguish important class-aware features with given prompts. This makes Obj2Seq work more efficiently, leading to an improvement of +0.6%. General Sequence Predictor is initially designed for better generality. However, the precision is also increased by +0.6%, which actually surprises us. We suppose that attributes of most object-level tasks are not independent. Some known attributes may help with the prediction of the others. For example, it would be more accuracy to estimate the scale of box if its center is determined.
4.4.2 Objectness Branch
In Section 3.4 we have introduced the additional objectness branch. is constructed in order to recognize if the object exists, rather than describe it. Meanwhile, also valid way if merge objectness into the output sequence of General Sequence Predictor, which means inserting an additional word representing the objectness score before the original sequence. We validate the necessity of the objectness branch in Table Predictor without separate objectness branch achieves the same performance as the original MLP. However, separate objectness branch further improves mAP by +0.6%. Both options are extensible for different tasks. We explain this phenomenon in two aspects. On one hand, single linear layer is not that good at judging the existence of objects of different categories. It needs help from the class vectors. On the other hand, checking the existence of an object different from describing it. Their functions may conflict the same head. Therefore, it is better to build separate objectness branch.

---

we are able to Table 4: Experiments on prompt vectors.
Clip Initialization
Fixed Prompt Vectors
m.A
AP50
AP75
APs
APM
APL
45,4
64.5
49.3
27.9
48,9
60.3
45.7
64.8
49.5
28.0
48.8
60.2
45.2
64.7
49.0
26.6
48.7
60.0
46
46
220
230
45
45
210
210
44
44
42-
Compunaton oalogeiks
180
42-
nAP wil citepocis
180
Computazin
41
30
50
70
170
41
0.5
170
Number catogoriis retained
Thruchold to relain canegories
(a)
(b)
Figure 3: We plot how the model behavior changes when different retention policies are utilized in evaluation. The figures drawn with (a) top-K' policies and (b) fixed threshold policies. The curves demonstrate how mAP fluctuates while the colored areas represents computation cost.
4.4.3 Ablation of Prompted Visual Indicator
Input prompt vector Obj2Seq receives class prompts as inputs to detect objects of certain cate- gories, while how generate class prompts remains a question. The simplest way is learning a prompt vector for each class from scratch, just like how object queries are learned DETR 4 As shown forward solution achieves 45.4%. CLIP (34 is robust feature extractor pretrained with a huge amount of image-text data pairs. It can provide better representations for label text of each category. Initializing prompt vectors with CLIP features achieves +0.3% improvement. However, fixing these prompt vectors leads to a slight decrease in precision. Vectors generated by CLIP may not be well aligned with image features. It needs be tuned in the training process.
in Table 4 this straight-forward
Prompted Visual Indicator is only used for classification, but also to initialize object queries for retained categories. If one category is filtered out, it means that Obj2Seq will not detect any objects of this category. It may be doubtful whether Obj2Seq is sensitive to the retention policy. Here we demonstrate how mAP changes when we use different policies in evaluation. As shown Figure 3 we mainly test with two policies, retaining the top-K' categories or any categories over threshold. Obj2Seq achieves steady results with both policies (see the blue curves). The evaluation metrie fluctuates within a range of 0.2% when the number of remaining classes changes from 10 to 80, and mAP is also stable when the threshold is lower than 0.3. These results indicate that Obj2Seq is insensitive the retention policy.
Retention Policies As mentioned in Section 3.2
In Eq. (3. the scores for class existence se further multiplied in the objectness branch to obtain final scores for objects. It does not improves the accuracy much if we use the same retention policy in training and evaluation, but makes Obj2Seq a more robust framework with different policies in inference. Figure 3 provides both curves with and without the participance of class existence $6 Without 8c evaluation. This phenomenon emerges because Object Transformer Decoder and General Sequence Predictor receives inputs in different distributions when the policy changes. Inputs of some categories filtered out by Prompted Visual Indicator during training, therefore Obj2Seq cannot make accurate predictions they are seen in inference. By multiplying with the class existence s make it up to some extent.
(k) the precision drops more significantly when we modify the retention policy in

---

Conclusion
In this paper, we propose a controllable and versatile framework for object-level visual tasks, Obj2Seq. This framework takes class prompts to determine the objects it should attend to, and generates sequence of attributes to describe each object in the image. Obj2Seq can be applied in different visual tasks, such as object detection, human pose estimation and image classification, and achieves comparable results with other task-specific methods.
There are few problems unsolved in Obj2Seq. which need further discussions. Firstly, Obj2Seq mainly focuses on unified definition of different visual tasks, but barely pays attention to the detailed structure of the feature extractor. Designing a new feature extractor for general usage may achieve better performance. Secondly, Obj2Seq can be implemented Obj2Seq only takes class prompts to guide the model. In order to satisfy finer requirements, more detailed indicator is worth developing. All all, Obj2Seq points a promising direction for general vision models, and it may inspire further studies.
a wide range of other tasks. Finally,
Acknowledgments and Disclosure of Funding
This work supported by National Key R&D Program of China under Grant No. No.2021ZD00 10403, National Natural Science Foundation of China (No.62002357, No.61976210, 76252, No.61876086, No.62076235), the InnoHK program and also sponsored by Zhejiang Lab NN.22211KHABB07
References
[1] Brown, T., Mann, B., Ryder, N., Subbiah. M., Kaplan, J.D., Dhariwal, P., Neelakantan, A.. Shyam. P., Sastry, G., Askell, A., al.: Language models are few-shot learners, Advances in neural information processing systems 33, 1877-1901 (2020)
[2] Cai, 7., Kwon, G., Ravichandran, A., Bas, E., Tu, 7., Bhotika, R., Soatto, S.: X-detr: A versatile architecture for instance-wise vision-language tasks. arXiv preprint arXiv:2204.0556: 05562 (2022)
[3] Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE conference computer vision and pattern recognition. pp. 7291-7299 (2017)
[4] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European conference on computer vision. pp. 213-229. Springer (2020)
[5] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerg- ing properties in self-supervised vision transformers. In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)
[6] Caruana, R.: Multitask learning. Machine learning 28(1), 41-75 (1997)
[7] Chen, M., Radford, A., Child, R., Wu, Jun, H., Luan, D., Sutskever, I.: Generative pretraining from pixels. In: International Conference on Machine Learning, pp. 1691-1703. PMLR (2020)
[8] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference machine learning. pp. 1597-1607. PMLR (2020)
[9] Chen, T., Saxena, S., Li, L.,, Fleet, D.J., Hinton, G.: Pix2seq: A language modeling framework
for object detection. In: International Conference on Learning Representations (2022), https //openreview
[10] Chen, X., Xie, S., He, K.: An empirical study of training self self-supervised vision transformers, In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649 (2021)
[11] Chen, Y., Wang, Z., Peng. Y., Zhang, Z., Yu, G., Sun, J.: Cascaded pyramid network for multi-person pose estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7103-7112 (2018)
10

---

[12] Chen, Z., Zhu, Y., Zhao, C., Hu, G., Zeng, W., Wang, J., Tang, M.: Dpt: Deformable patch- transformer for visual recognition. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 2899-2901 (2021)
[13] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 10.04805 (2018)
[14] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al.: An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19929 (2020)
[15] Gao, P., Zheng, M., Wang, X., Dai, J., Li, H.: Fast convergence of detr with spatially modulated co-attention. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3621-3630 (2021)
[16] Gao, T., Fisch, A., Chen, D.: Making pre-trained language models better few-shot learners, In: Association for Computational Linguistics (ACL) (2021)
[17] Ghiasi, G., Zoph, B., Cubuk, E.D., Le, Q.V., Lin, T.Y.: Multi-task self-training for learning gen- cral representations. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8856-8865 (2021)
[18] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. arXiv:2111 111211111.677 (2021)
[19] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE
international conference on computer vision. pp. 2961-2969 (2017)
[20] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778 (2016)
[21] Hua, T. Tian, Y., Ren, S., Zhao, H., Sigal, L.: Self-supervision through random segments with autoregressive coding (randsac). arXiv preprint arXiv:2203. 12054 (2022)
[22] Jia, M., Tang, L., Chen, B.C., Cardie, C., Belongie, S., Hariharan, B., Lim, S.N.: Visual prompt tuning. arXiv preprint arXiv:2203.12119 (2022)
[23] Kamath, A., Singh, M., LeCun, Y. Synnaeve, G., Misra, I., Carion, N.: Mdetr-modulated detec- tion for end-to-ent multi-modal understanding, In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1780-1777777 1790 (2021)
[24] Li, K., Wang. S., Zhang, X., Xu, Y., Xu, W., Tu, Z.: Pose recognition with cascade transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1944-1953 (2021)
[25] Li, Z., Chen, Z., Yang, F., Li, W., Zhu, Y., Zhao, C., Deng, R., Wu, L., Zhao, R., Tang, M., al.: Mst: Masked self-supervised transformer for visual representation. Advances in Neural Information Processing Systems 34 (2021)
[26] Li, Z., Zhao, X., Zhao. C., Tang, M., Wang, J.: Transfering low-frequency features for domain adaptation. In: 2022 IEEE International Conference on Multimedia and Expo (ICME). pp.
01-06. IEEE (2022)
[27] Li, Z.,, Zhu, Y., Yang, F., Li, W., Zhao, C., Chen, Y., Chen, Z., Xie, J., Wu, I.,, Zhao, R., et al.: Univip: A unified framework for self-supervised visual pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14627-14636 (2022)
[28] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740-755. Springer (2014)
[29] Liu, S., Li, F., Zhang, H., Yang, X., Qi, X., Su, H., Zhu, J., Zhang, L.: DAB-DETR: Dy- namic anchor boxes are better queries for DETR. In: International Conference on Learning Representations (2022), htttps://openrviiii
[30] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10012-10022 (2021)
11

---

[31] Meng, D., Chen, X., Fan, Z., Zeng, G., Li, H.,, Yuan, Y.. Sun, L.,, Wang, J.: Conditional detr for fast training convergence. In: Proceedings the IEEE/CVF International Conference on Computer Vision. pp. 3651-3660 (2021)
[32] Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al.: Conditional image generation with pixelenn decoders. Advances in neural information processing systems 29 (2016)
[33] Papandreou, G., Zhu, T., Chen, L.C., Gidaris, S., Tompson, J., Murphy, K.: Personlab: Person pose estimation and instance segmentation with bottom-up, part-based, geometric embedding model. In: Proceedings of the European conference on computer vision (ECCV). pp. 269-286 (2018)
[34] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748-8763, PMLR (2021)
[35] Ren, S., He, K. Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015)
[36] Ridnik, T., Ben-Baruch, E., Zamir, N., Noy, A., Friedman, I., Protter, M., Zelnik-Manor, L.::
Asymmetric loss for multi-label classification. In: Proceedings of the IEEE/CVF International
Conference on Computer Vision. pp. 82-91 (2021)
[37] Ridnik, T., Sharir, G., Ben-Cohen. A., Ben- Ben-Baruch, E., Noy, A.: Ml-decoder: Scalable and versatile classification head. arXiv preprint arXiv:2111.12933 (2021)
[38] Ruder, S.: An overview of multi-task learning in deep neural networks. arXiv preprint
MeXiv:17060050%0 (2017)
[39] Stoffl, L., Vidal, M., Mathis, A.: End-to-end trainable multi-instance pose estimation with transformers, arXiv preprint 1XXi::2103.12111 (2021)
[40] Sun, Z., Cao, S., Yang, Y., Kitani, K.M.: Rethinking transformer-bast -based set prediction for object
detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp.
3611-3620 (2021)
[41] Tian, Z., Chen, H., Shen, C.: Directpose: Direct end-to-end multi-person pose estimation. arXiv preprint 1r1iv:191 1.07451 (2019)
[42] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient
image transformers & distillation through attention. In: International Conference on Machine Learning. pp. 10347-10357. PMLR (2021)
[43] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, L:: Attention all you need, Advances in neural information processing systems 30 (2017)
[44] Wang. W., Cao, Y., Zhang, J., Tao, D.: Fp-detr: Detection transformer advanced by fully pre-training. In: International Conference on Learning Representations (2021)
[45] Wang, W., Xie, E.. Li, X., Fan, D.P., Song, K., Liang, D., Lu. T., Luo, P., Shao, L.: Pyra- mid vision transformer: A versatile backbone for dense prediction without convolutions. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 568-578 (2021)
[46] Wang, Y., Zhang, X., Yang, T., Sun, J.: Anchor detr: Query design for transformer-based detector. arXiv preprint arXiv:2109.07107 (2021)
[47] Xiao, B., Wu, H., Wei, Y.: Simple baselines for human pose estimation and tracking. In: Proceedings of the European conference on computer vision (ECCV). pp. 466-481 (2018)
[48] Xu, J., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., Wang, X.: Groupvit: Semantic
segmentation emerges from text supervision. arXiv preprint arXiv:2202.1 1094 (2022)
[49] Yang, Z., Dai, Z., Yang. Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: Xlnet: Generalized au- toregressive pretraining for language understanding. Advances in neural information processing systems 32 (2019)
[50] Zhang, F., Zhu, X., Dai, H., Ye, M., Zhu, C.: Distribution-aware coordinate representation for human pose estimation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 7093-7102 (2020)
12

---

[51] Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. arXiv preprint urXiv:1904.077 (2019)
[52] Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers for end-to-end object detection, arXiv preprint arXiv:2010.04159 (2020)
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work proposes framework for visual tasks. It does not involve social impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The code is included in the supplemental material. We will also release it later.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix and the code.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Appendix.
(c) Did you include any assets either in the supplemental material URL? [Yes]
Our code is included in the supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "hRfuoR9g1p", "reviewer": "Reviewer_LxeJ", "summary": "The paper introduces a unified architecture similar to M-DETR, but takes object class labels as input, and outputs three components in the same architecture: \n1. the existence of object for multi-label classification \n2. the box coordinates for object detection \n3. keypoint coordinates for human pose estimation. \n\nThe key component of this unified architecture is the sequence decoder, from which the first four outputs are always trained to be box coordinates and the rest outputs are trained to be human pose keypoints. \n", "questions": "1. Are the Figure (3) a & b actually the same figure? I.e. the x-axes have one-to-one mapping and can be plotted in the same figure. \n2. What if there are multiple objects of the same category in the image? From Figure 2 it looks like there are 3 object queries l for each class embedding output. Does it work if the image has more than 3 people? Could the authors add more details about how to handle these multi-instance cases? \n", "limitations": "The paper discusses three limitations: (1) not exploring the design of the feature extractor, (2) not exploring other tasks, (3) only taking object class as input and not exploring more detailed indicators. I agree with these limitations as potential future works, but I think point (2) has a higher priority given the paper claims a \"unified framework\". \n\nAdditionally, I think one limitation of the model is that the object categories are limited by the input categories, and the model cannot discover new categories. \n", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The novelty of the paper is limited. The paper claims a \"unified framework\" (e.g. Figure 1) and \"a wide range of vision tasks\" (L91) but the experiments only show the tasks of outputting spatial coordinates (box coordinate and keypoint). While some recent work OFA (I’m aware it was not officially published at the submission time) shows a seq2seq framework can be applied to a much wider range of tasks including detection, caption and VQA. Overall I mean more vision tasks are required to support the claim of a “unified framework”. Second, the authors design an architecture that is similar to DETR-family, and the previous work M-DETR uses the architecture for language grounding tasks. This paper takes the class labels as input and the model outputs the bounding box, which feels more like a simplified version of a visual grounding task rather than object detection. Can the authors elaborate more on the difference between the proposed setting and visual grounding tasks?\n\nThe overall quality of the paper is good. The experiments are clearly explained and discussed. However, I think the comparison with previous works (Table 1) is unfair since the method uses a retention policy to pre-filter some probable object categories while the normal detection method does not benefit from this. Can the authors discuss this comparison more?\n\nIn general, the paper is very clearly written and well organized. The figures are clear and easy to understand. \n\nThe experiments in object detection show the proposed method outperforms previous methods but due to the retention policy (mentioned above), I doubt the fairness of this comparison and more discussion is required. The experiments in human pose estimation show the proposed method is effective when compared with previous works. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LkjsjbQ-RWz", "reviewer": "Reviewer_7jWi", "summary": "This paper presents a new object-centered framework to unify visual tasks including object detection, key point detection, and multi-label classification. Different from the previous Pix2Seq method, this paper divides the sequence generation problem into an object query generation sub-problem and an attribute sequence generation sub-problem. The method achieves good results on multiple visual recognition tasks on COCO.", "questions": "Overall, I think this is a good paper with several new and insightful technical contributions and decent results. Although I still have concerns about some designs, I think the proposed object-centered framework might be a new and interesting direction for several visual recognition tasks. Therefore, I lean toward accepting this paper. This paper can be stronger if the questions mentioned in the Weaknesses subsection  can be addressed.    ", "limitations": "The limitations and potential societal impact of the method have been discussed. ", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The idea of decoupling the sequence generation process into two steps is new and insightful. \n\n- The object query generation process allows the use of bipartite matching loss for supervision, which improves the optimization of the original pix2seq and helps the framework achieve competitive results with state-of-the-art methods.\n\n-  The method is evaluated on multiple popular benchmarks and achieves good results. Ablation studies in Section 4.4 make the paper solid.\n\nWeaknesses:\n\n- Since the method uses bipartite matching loss from DETR [4] for supervision and generates many redundant queries (100 queries for each candidate category according to Appendix A) like [4], I think the overall framework can be viewed as an improved DETR with class conditioned query generator, which is close to several previous methods like [28, 43, 26, 49]. The object-centered design also makes the method less general compared to the concurrent work pix2seq v2 [r1] which can be extended to the image captioning task.\n\n-  The two-step design makes the necessity of the sequence generation framework questionable. The sequence generation task in Pix2Seq is design to unify vision and NLP tasks. However, due to the existence of the Prompted Visual Indicator and bipartite matching loss, the proposed method is totally different from sequence-to-sequence autoregression framework in NLP. Therefore, is that possible to further simplify the General Sequence Predictor as a 2-token transformers model where the object query is combined with detection and keypoint task tokens as the inputs and use MLP heads like DETR to obtain the final prediction? \n\n- The paper emphasizes a unified framework is designed for various visual tasks. However, the experiments didn't show the potential of the proposed framework to perform multiple challenging tasks like [r1]. Is it possible for the proposed framework to perform general object detection and keypoint detection with a single model? Will the model benefit from joint training?\n\n[r1] A Unified Sequence Interface for Vision Tasks, arXiv 2206.07669", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I2YT3QKJXSp", "reviewer": "Reviewer_tovC", "summary": "This paper is a nice try towards unified interface for various object related computer vision tasks. It use image and class as prompt, achieving SOTA detection and pose estimation performance in the same interface. The architecture designs of the Prompted Visual Indicator and Sequence Predictor are sound. The major debates lie in the usage of class as prompt and the object-level query. These two might be beneficial to get SOTA object-level tasks, but might also be the blocker for more general vision task interface.", "questions": "The limitation of Pix2seq mentioned in this paper that \"Pix2Seq is not aware of the desired targets before inference, and the output sequence might become extremely long under sophisticated scenes.\" This is not convincing enough. We could argue this actually shows the flexibility of Pix2seq. May I bother the author to elaborate more on this? What are the specific scenarios Pix2Seq may meet trouble and Obj2Seq is better.", "limitations": "Yes, the author mentioned the limitation of this work.\nClass as prompt is not only a limitation to satisfy finer requirements, but also cause trouble on open-vocabulary applications.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This effort addresses an ambitious goal of getting a unified interface for computer vision tasks. Obj2seq unifies object detection, pose estimation and image classification.\n2. The designed architecture achieves SOTA performance on object detection and pose estimation.\n\nWeaknesses:\n1. Class as prompt blocks its way to generalize to open-vocabulary applications. The goal of the unified interface is to utilize the scaling law of large scale model to utilize large dataset and address real-world open-vocabulary challenges.\nDebatable:\n1. The architecture keeps the object query, while pix2seq has already gotten rid of that. This is debatable. I don't have an obvious answer whether that part is necessary to achieve SOTA performance on several object-level tasks considering object is the unit that interaction or reasoning works on.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2nBLDGYJB2O", "reviewer": "Reviewer_BccU", "summary": "This paper proposes a new framework for vision tasks, called Obj2Seq. It takes prompts of classes as input then filters the non-exist categories and keeps the most confident K' categories for each image by the Prompted Visual Indicator module. After that, they will be passed to the Object Transformer Decoder module together with the encoded image features to get the object queries. The final General Sequence Predictor takes these queries as input to generate a sequence of desired results, e.g., the location (x, y), size (w, h), or even the other for fine-grained information like keypoints (x_nose, y_nose, x_eye1, y_eye1, .....). The experiments on MS COCO demonstrate that the proposed Obj2Seq can achieve comparable results with other task-specific methods.\n", "questions": "According to the authors, the main contribution of this paper is to provide a unified visual framework, but I only perceive it as an object-level prediction framework, which is equivalent to the conventional Mask R-CNN framework in terms of potential capacity. Therefore, I want to know some specific examples of tasks that can only be formulated by Obj2Seq but not Mask R-CNN during rebuttal.", "limitations": "The authors have already addressed the limitations.", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[+] This paper proposes a new framework for object-level visual prediction tasks, e.g., object detection, multi-label classification or human pose estimation.\n[+] According to the experimental results, the proposed Obj2Seq can achieve comparable performances on each task with their task-specific methods.\n\n[-] However, I don't think Obj2Seq is a unified vision framework for most vision tasks, which is mainly designed to solve the object-level prediction. In my opinion, the capacity of the proposed Obj2Seq is equal to the conventional Mask R-CNN framework, because Mask R-CNN is also able to equip new prediction heads based on the downstream tasks. Therefore, I prefer to consider Obj2Seq as another version of \"Mask R-CNN\" rather than a unified framework for vision tasks. To better understand what a unified vision framework is supposed to look like, please refer to the [1, 2], which are capable of both localization tasks and understanding tasks. Note that I'm NOT suggesting the authors to compare the proposed Obj2Seq with these methods, as [2] and this paper are contemporary works. What I want to say is that the proposed Obj2Seq is far from a unified vision framework. \n\n\n[1] Lu, Jiasen, et al. \"12-in-1: Multi-task vision and language representation learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[2] Zhang, Haotian, et al. \"GLIPv2: Unifying Localization and Vision-Language Understanding.\" arXiv preprint arXiv:2206.05836 (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept